From s.lanthaler at gmail.com Fri Jun 1 08:21:00 2018 From: s.lanthaler at gmail.com (Samuel Lanthaler) Date: Fri, 1 Jun 2018 15:21:00 +0200 Subject: [petsc-users] MatPtAP In-Reply-To: References: <1153922A-8876-492B-8AD9-F41DD718FD21@anl.gov> <58D5E959-0E3C-4EF2-8270-882D1EA6EA38@mcs.anl.gov> Message-ID: <1db3b230-7243-4664-e846-20ceb0132eea@gmail.com> Hi, I was wondering what the most efficient way to use MatPtAP would be in the following situation: I am discretizing a PDE system. The discretization yields a matrix A that has a band structure (with k upper and lower bands, say). In order to implement the boundary conditions, I use a transformation matrix P which is essentially the unit matrix, except for the entries P_{ij} where i,j References: <1153922A-8876-492B-8AD9-F41DD718FD21@anl.gov> <58D5E959-0E3C-4EF2-8270-882D1EA6EA38@mcs.anl.gov> Message-ID: Marius: Current PETSc interface supports sequential sparse multiple right-hand side, but not distributed. It turns out that mumps does not support distributed sparse multiple right-hand sides at the moment (see attached email). Jean-Yves invites you to communicate with him directly. Let me know what we can help on this matter, e.g., add support for parallel implementation of sparse multiple right-hand side with centralized rhs input? Hong ---------------------- Jean-Yves LExcellent 5:14 AM (3 hours ago) to Hong, mumps-dev Hello, We do not support distributed sparse multiple right-hand sides at the moment. From the feedback we have from applications, the right-hand sides are often very sparse, and having them distributed did not seem critical. Since we are specifying a distributed right-hand sides feature at the moment, could you let us know in more detail the need regarding distributed sparse right-hand side (e.g., do all columns have the same nonzero structure in that case) or put us in contact with the user who needs this? Thanks, Jean-Yves and Patrick Thanks a lot guys, very helpful. > > > > > I see MUMPS http://mumps.enseeiht.fr/ > > Sparse multiple right-hand side, distributed solution; Exploitation of > sparsity in the right-hand sidesPETSc interface computes mumps distributed > solution as default (this is not new) (ICNTL(21) = 1) > > I will add support for Sparse multiple right-hand side. > > Hong > > On Thu, May 31, 2018 at 11:25 AM, Smith, Barry F. [mailto:bsmith at mcs.anl.gov]> wrote: > Hong, > > Can you see about adding support for distributed right hand side? > > Thanks > > Barry > > > On May 31, 2018, at 2:37 AM, Marius Buerkle mbuerkle at web.de]> wrote: > > > > The fix for MAT_NEW_NONZERO_LOCATIONS, thanks again. > > > > I have yet another question, sorry. The recent version of MUMPS supports > distributed and sparse RHS is there any chance that this will be supported > in PETSc in the near future? > > > > > > > > > >> On May 30, 2018, at 6:55 PM, Marius Buerkle mbuerkle at web.de]> wrote: > >> > >> Thanks for the quick fix, I will test it and report back. > >> I have another maybe related question, if MAT_NEW_NONZERO_LOCATIONS is > true and let's say 1 new nonzero position is created it does not allocated > 1 but several new nonzeros but only use 1. > > > > Correct > > > >> I think that is normal, right? > > > > Yes > > > >> But, at least as far as I understand the manual, a subsequent call of > mat assemble with > >> MAT_FINAL_ASSEMBLY should compress out the unused allocations and > release the memory, is this correct? > > > > It "compresses it out" (by shifting all the nonzero entries to the > beginning of the internal i, j, and a arrays), but does NOT release any > memory. Since the values are stored in one big contiguous array (obtained > with a single malloc) it cannot just free part of the array, so the extra > locations just sit harmlessly at the end if the array unused. > > > >> If so, this did not work for me, even after doing > >> MAT_FINAL_ASSEMBLY the unused nonzero allocations remain. Is this > normal? > > > > Yes, > > > > Barry > > > >> > >>> > >>> Fixed in the branch barry/fix-mat-new-nonzero-locations/maint > >>> > >>> Once this passes testing it will go into the maint branch and then the > next patch release but you can use it now in the branch > barry/fix-mat-new-nonzero-locations/maint > >>> > >>> Thanks for the report and reproducible example > >>> > >>> Barry > >>> > >>> > >>>> On May 29, 2018, at 7:51 PM, Marius Buerkle mbuerkle at web.de]> wrote: > >>>> > >>>> Sure, I made a small reproducer, it is Fortran though I hope that is > ok. If MAT_NEW_NONZERO_LOCATIONS is set to false I get an error, if it is > set to true the new nonzero element is inserted, if > MAT_NEW_NONZERO_LOCATIONS is false and either MAT_NEW_NONZERO_LOCATION_ERR > or MAT_NEW_NONZERO_ALLOCATION_ERR is set to false afterwards then the new > nonzero is also created without an error, but if MAT_NEW_NONZERO_LOCATIONS > is set to false after MAT_NEW_NONZERO_LOCATION_ERR/MAT_NEW_NONZERO_ALLOCATION_ERR > have been set to false I get an error again. > >>>> > >>>> > >>>> program newnonzero > >>>> #include > >>>> use petscmat > >>>> implicit none > >>>> > >>>> Mat :: A > >>>> PetscInt :: dnnz,onnz,n,m,idxm(1),idxn(1),nl1,nl2 > >>>> PetscScalar :: v(1) > >>>> PetscReal :: info(MAT_INFO_SIZE) > >>>> PetscErrorCode :: ierr > >>>> > >>>> integer :: nproc,iproc,i > >>>> > >>>> call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > >>>> > >>>> call MPI_COMM_SIZE(PETSC_COMM_WORLD, nproc,ierr) > >>>> > >>>> call MPI_Comm_rank( PETSC_COMM_WORLD, iproc, ierr ) > >>>> > >>>> n=3 > >>>> m=n > >>>> call MatCreateAIJ(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,n,m, > 1,PETSC_NULL_INTEGER,0,PETSC_NULL_INTEGER,A,ierr) > >>>> > >>>> > >>>> call MatGetOwnershipRange(A,nl1,nl2,ierr) > >>>> do i=nl1,nl2-1 > >>>> idxn(1)=i > >>>> idxm(1)=i > >>>> v(1)=1d0 > >>>> call MatSetValues(A,1,idxn,1,idxm, v,INSERT_VALUES,ierr) > >>>> end do > >>>> call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) > >>>> call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) > >>>> > >>>> call MatSetOption(A,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE,ierr) > >>>> !~ call MatSetOption(A,MAT_NEW_NONZERO_LOCATION_ERR,PETSC_FALSE,ierr) > >>>> !~ call MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR > ,PETSC_FALSE,ierr) > >>>> !~ call MatSetOption(A,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE,ierr) > >>>> > >>>> > >>>> idxn(1)=0 > >>>> idxm(1)=n-1 > >>>> if ((idxn(1).ge.nl1).and.(idxn(1).le.nl2-1)) then > >>>> v(1)=2d0 > >>>> call MatSetValues(A,1,idxn,1,idxm, v,INSERT_VALUES,ierr) > >>>> end if > >>>> call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) > >>>> call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) > >>>> > >>>> if ((idxn(1).ge.nl1).and.(idxn(1).le.nl2-1)) then > >>>> v(1)=2d0 > >>>> call MatGetValues(A,1,idxn,1,idxm, v,ierr) > >>>> write(6,*) v > >>>> end if > >>>> > >>>> call PetscFinalize(ierr) > >>>> > >>>> end program newnonzero > >>>> > >>>> > >>>> > >>>> $ mpiexec.hydra -n 3 ./a.out > >>>> [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > >>>> [0]PETSC ERROR: Argument out of range > >>>> [0]PETSC ERROR: Inserting a new nonzero at global row/column (0, 2) > into matrix > >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/ > documentation/faq.html[http://www.mcs.anl.gov/petsc/ > documentation/faq.html] for trouble shooting. > >>>> [0]PETSC ERROR: Petsc Release Version 3.9.2, May, 20, 2018 > >>>> [0]PETSC ERROR: ./a.out on a named tono-hpc1 by marius Wed May 30 > 09:42:40 2018 > >>>> [0]PETSC ERROR: Configure options --prefix=/home/marius/prog/petsc/3.9.2 > --download-elemental=yes --download-metis=yes --download-parmetis=yes > --download-mumps=yes --with-scalapack-lib="/home/ > marius/intel/compilers_and_libraries_2018.2.199/linux/ > mkl/lib/intel64/libmkl_scalapack_lp64.a -Wl,--start-group > /home/marius/intel/compilers_and_libraries_2018.2.199/ > linux/mkl/lib/intel64/libmkl_intel_lp64.a /home/marius/intel/compilers_ > and_libraries_2018.2.199/linux/mkl/lib/intel64/libmkl_sequential.a > /home/marius/intel/compilers_and_libraries_2018.2.199/ > linux/mkl/lib/intel64/libmkl_core.a /home/marius/intel/compilers_ > and_libraries_2018.2.199/linux/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.a > -Wl,--end-group -lpthread -lm -ldl" --FC=mpiifort --CC=mpicc --CXX=mpicxx > --with-scalar-type=complex --with-mpi-dir= --with-blaslapack-lib="/home/ > marius/intel/compilers_and_libraries_2018.2.199/linux/ > mkl/lib/intel64/libmkl_scalapack_lp64.a -Wl,--start-group > /home/marius/intel/compilers_and_libraries_2018.2.199/ > linux/mkl/lib/intel64/libmkl_intel_lp64.a /home/marius/intel/compilers_ > and_libraries_2018.2.199/linux/mkl/lib/intel64/libmkl_sequential.a > /home/marius/intel/compilers_and_libraries_2018.2.199/ > linux/mkl/lib/intel64/libmkl_core.a /home/marius/intel/compilers_ > and_libraries_2018.2.199/linux/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.a > -Wl,--end-group -lpthread -lm -ldl" --with-cxx-dialect=C++11 > --download-superlu_dist=yes --download-ptscotch=yes --with-x > --with-debugging=1 --download-superlu=yes --with-mkl_cpardiso=1 > --with-mkl_pardiso=1 --with-scalapack=1 > >>>> [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 607 in > /home/marius/prog/petsc/petsc-3.9.2/src/mat/impls/aij/mpi/mpiaij.c > >>>> [0]PETSC ERROR: #2 MatSetValues() line 1312 in > /home/marius/prog/petsc/petsc-3.9.2/src/mat/interface/matrix.c > >>>> (0.000000000000000E+000,0.000000000000000E+000) > >>>> > >>>> > >>>> > >>>> Please send complete error message; type of matrix used etc. Ideally > code that demonstrates the problem. > >>>> > >>>> Barry > >>>> > >>>> > >>>>> On May 29, 2018, at 3:31 AM, Marius Buerkle mbuerkle at web.de]> wrote: > >>>>> > >>>>> > >>>>> Hi, > >>>>> > >>>>> I tried to set MAT_NEW_NONZERO_LOCATIONS to false, as far as I > understood MatSetValues should simply ignore entries which would give rise > to new nonzero values not creating a new entry and not cause an error, but > I get "[1]PETSC ERROR: Inserting a new nonzero at global row/column". Is > this option supposed to work or not? > >>>> > >>> > >>> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jun 1 08:42:25 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 1 Jun 2018 09:42:25 -0400 Subject: [petsc-users] MatPtAP In-Reply-To: <1db3b230-7243-4664-e846-20ceb0132eea@gmail.com> References: <1153922A-8876-492B-8AD9-F41DD718FD21@anl.gov> <58D5E959-0E3C-4EF2-8270-882D1EA6EA38@mcs.anl.gov> <1db3b230-7243-4664-e846-20ceb0132eea@gmail.com> Message-ID: On Fri, Jun 1, 2018 at 9:21 AM, Samuel Lanthaler wrote: > Hi, > > I was wondering what the most efficient way to use MatPtAP would be in the > following situation: I am discretizing a PDE system. The discretization > yields a matrix A that has a band structure (with k upper and lower bands, > say). In order to implement the boundary conditions, I use a transformation > matrix P which is essentially the unit matrix, except for the entries > P_{ij} where i,j > P = [ B, 0, 0, 0, ..., 0, 0 ] > [ 0, 1, 0, 0, ..., 0, 0 ] > [ ] > [ ] > [ ..., 1, 0 ] > [ 0, 0, 0, 0, ..., 0, C ] > > with B,C are (k-by-k) matrices. > Right now, I'm simply constructing A, P and calling > > CALL MatPtAP(petsc_matA,petsc_matP,MAT_INITIAL_MATRIX,PETSC_DEFAU > LT_REAL,petsc_matPtAP,ierr) > > where I haven't done anything to pestc_matPtAP, prior to this call. Is > this the way to do it? > > I'm asking because, currently, setting up the matrices A and P takes very > little time, whereas the operation MatPtAP is taking quite long, which > seems very odd... The matrices are of type MPIAIJ. In my problem, the total > matrix dimension is around 10'000 and the matrix blocks (B,C) are of size > ~100. > Are you sure this is what you want to do? Usually BC are local, since by definition PDE are local, and are applied pointwise. What kind of BC do you have here? Thanks, Matt > Thanks in advance for any ideas. > > Cheers, > Samuel > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Jun 1 09:55:07 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 1 Jun 2018 14:55:07 +0000 Subject: [petsc-users] MatPtAP In-Reply-To: <1db3b230-7243-4664-e846-20ceb0132eea@gmail.com> References: <1153922A-8876-492B-8AD9-F41DD718FD21@anl.gov> <58D5E959-0E3C-4EF2-8270-882D1EA6EA38@mcs.anl.gov> <1db3b230-7243-4664-e846-20ceb0132eea@gmail.com> Message-ID: <8C838EE8-9DE9-40D3-8C5C-24EBF1EF8538@anl.gov> How do you know it is exactly the MatPtAP() routine that is taking the large time? Please send the output of a run with -log_view so we can see where the time is being spent. Barry > On Jun 1, 2018, at 8:21 AM, Samuel Lanthaler wrote: > > Hi, > > I was wondering what the most efficient way to use MatPtAP would be in the following situation: I am discretizing a PDE system. The discretization yields a matrix A that has a band structure (with k upper and lower bands, say). In order to implement the boundary conditions, I use a transformation matrix P which is essentially the unit matrix, except for the entries P_{ij} where i,j > P = [ B, 0, 0, 0, ..., 0, 0 ] > [ 0, 1, 0, 0, ..., 0, 0 ] > [ ] > [ ] > [ ..., 1, 0 ] > [ 0, 0, 0, 0, ..., 0, C ] > > with B,C are (k-by-k) matrices. > Right now, I'm simply constructing A, P and calling > > CALL MatPtAP(petsc_matA,petsc_matP,MAT_INITIAL_MATRIX,PETSC_DEFAULT_REAL,petsc_matPtAP,ierr) > > where I haven't done anything to pestc_matPtAP, prior to this call. Is this the way to do it? > > I'm asking because, currently, setting up the matrices A and P takes very little time, whereas the operation MatPtAP is taking quite long, which seems very odd... The matrices are of type MPIAIJ. In my problem, the total matrix dimension is around 10'000 and the matrix blocks (B,C) are of size ~100. > > Thanks in advance for any ideas. > > Cheers, > Samuel From lukas.drinkt.thee at gmail.com Fri Jun 1 10:01:52 2018 From: lukas.drinkt.thee at gmail.com (Lukas van de Wiel) Date: Fri, 1 Jun 2018 17:01:52 +0200 Subject: [petsc-users] downloading hypre Message-ID: Hi all, for years I have been installing PETSc on machines, almost always without any issue to speak off. Compliments for the solid configure script. However, now I see an issue I cannot easily solve. When getting HYPRE in the configuration options, the output gives ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Error during download/extract/detection of HYPRE: Unable to download hypre Could not execute "git clone https://github.com/LLNL/hypre /net/home/gtecton/flops/petsc-3.8.1/linux-gnu-x86_64/externalpackages/git.hypre": Cloning into '/net/home/gtecton/flops/petsc-3.8.1/linux-gnu-x86_64/externalpackages/git.hypre'... fatal: unable to access 'https://github.com/LLNL/hypre/': error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version Unable to download package HYPRE from: git://https://github.com/LLNL/hypre Has anybody else seen this? It seems to get stuck on SSL, but web security is sadly not my forte... Thanks a lot, Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jun 1 10:08:37 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 1 Jun 2018 11:08:37 -0400 Subject: [petsc-users] downloading hypre In-Reply-To: References: Message-ID: On Fri, Jun 1, 2018 at 11:01 AM, Lukas van de Wiel < lukas.drinkt.thee at gmail.com> wrote: > Hi all, > > for years I have been installing PETSc on machines, almost always without > any issue to speak off. Compliments for the solid configure script. > However, now I see an issue I cannot easily solve. > > When getting HYPRE in the configuration options, the output gives > > > ************************************************************ > ******************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > details): > ------------------------------------------------------------ > ------------------- > Error during download/extract/detection of HYPRE: > Unable to download hypre > Could not execute "git clone https://github.com/LLNL/hypre > /net/home/gtecton/flops/petsc-3.8.1/linux-gnu-x86_64/ > externalpackages/git.hypre": > Cloning into '/net/home/gtecton/flops/petsc-3.8.1/linux-gnu-x86_64/ > externalpackages/git.hypre'... > fatal: unable to access 'https://github.com/LLNL/hypre/': > error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol > version > Unable to download package HYPRE from: git://https://github.com/LLNL/hypre > > Has anybody else seen this? It seems to get stuck on SSL, but web security > is sadly not my forte... > I have definitely had issues similar to this. It came up with Firedrake. I built my own Python, and had an early version of libssh/libcrypto. I think you need at least 1.0.1 to get the SSL version that github is now requiring. You can use ldd (or otool -L) to check the version for your Python. Thanks, Matt > Thanks a lot, > > Lukas > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Fri Jun 1 10:09:13 2018 From: mfadams at lbl.gov (Mark Adams) Date: Fri, 1 Jun 2018 11:09:13 -0400 Subject: [petsc-users] viewing question Message-ID: I have: ierr = PetscObjectSetName((PetscObject) uproj,"u");CHKERRQ(ierr); ierr = PetscObjectViewFromOptions((PetscObject)uproj, NULL, "-vec_view");CHKERRQ(ierr); The name that I see in the viewer (Visit) is "u_PetscFE_0x840000 ...". Can I get rid of all the crap that PETSc added and just get "u". Thanks, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas.drinkt.thee at gmail.com Fri Jun 1 10:11:54 2018 From: lukas.drinkt.thee at gmail.com (Lukas van de Wiel) Date: Fri, 1 Jun 2018 17:11:54 +0200 Subject: [petsc-users] downloading hypre In-Reply-To: References: Message-ID: Thanks Matt. That was fast! :-) The university server has: [17:04 gtecton at pbsserv petsc-3.8.1] > openssl version OpenSSL 0.9.8zh 3 Dec 2015 And a friend with OpenSSL 1.0.2g 1 Mar 2016 Can execute the git clone command without trouble. Cheers and have a great weekend! Lukas On Fri, Jun 1, 2018 at 5:08 PM, Matthew Knepley wrote: > On Fri, Jun 1, 2018 at 11:01 AM, Lukas van de Wiel < > lukas.drinkt.thee at gmail.com> wrote: > >> Hi all, >> >> for years I have been installing PETSc on machines, almost always without >> any issue to speak off. Compliments for the solid configure script. >> However, now I see an issue I cannot easily solve. >> >> When getting HYPRE in the configuration options, the output gives >> >> >> ************************************************************ >> ******************* >> UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for >> details): >> ------------------------------------------------------------ >> ------------------- >> Error during download/extract/detection of HYPRE: >> Unable to download hypre >> Could not execute "git clone https://github.com/LLNL/hypre >> /net/home/gtecton/flops/petsc-3.8.1/linux-gnu-x86_64/externa >> lpackages/git.hypre": >> Cloning into '/net/home/gtecton/flops/petsc-3.8.1/linux-gnu-x86_64/extern >> alpackages/git.hypre'... >> fatal: unable to access 'https://github.com/LLNL/hypre/': >> error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol >> version >> Unable to download package HYPRE from: git://https://github.com/LLNL/ >> hypre >> >> Has anybody else seen this? It seems to get stuck on SSL, but web >> security is sadly not my forte... >> > > I have definitely had issues similar to this. It came up with Firedrake. I > built my own Python, and had > an early version of libssh/libcrypto. I think you need at least 1.0.1 to > get the SSL version that github > is now requiring. You can use ldd (or otool -L) to check the version for > your Python. > > Thanks, > > Matt > > >> Thanks a lot, >> >> Lukas >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jun 1 10:12:45 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 1 Jun 2018 11:12:45 -0400 Subject: [petsc-users] viewing question In-Reply-To: References: Message-ID: On Fri, Jun 1, 2018 at 11:09 AM, Mark Adams wrote: > I have: > > ierr = PetscObjectSetName((PetscObject) uproj,"u");CHKERRQ(ierr); > ierr = PetscObjectViewFromOptions((PetscObject)uproj, NULL, > "-vec_view");CHKERRQ(ierr); > > The name that I see in the viewer (Visit) is "u_PetscFE_0x840000 ...". Can > I get rid of all the crap that PETSc added and just get "u". > I am guessing this is HDF5? If you are using PetscFE, give your FE a name. Matt > Thanks, > Mark > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jun 1 10:16:09 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 1 Jun 2018 11:16:09 -0400 Subject: [petsc-users] downloading hypre In-Reply-To: References: Message-ID: On Fri, Jun 1, 2018 at 11:11 AM, Lukas van de Wiel < lukas.drinkt.thee at gmail.com> wrote: > Thanks Matt. That was fast! :-) > > The university server has: > > [17:04 gtecton at pbsserv petsc-3.8.1] > openssl version > OpenSSL 0.9.8zh 3 Dec 2015 > > And a friend with > > OpenSSL 1.0.2g 1 Mar 2016 > > Can execute the git clone command without trouble. > > Cheers and have a great weekend! > Great! Also if you are bored next week, we are having the PETSc Meeting in London. Only a short train ride by Eurostar :) Matt > Lukas > > > > > On Fri, Jun 1, 2018 at 5:08 PM, Matthew Knepley wrote: > >> On Fri, Jun 1, 2018 at 11:01 AM, Lukas van de Wiel < >> lukas.drinkt.thee at gmail.com> wrote: >> >>> Hi all, >>> >>> for years I have been installing PETSc on machines, almost always >>> without any issue to speak off. Compliments for the solid configure script. >>> However, now I see an issue I cannot easily solve. >>> >>> When getting HYPRE in the configuration options, the output gives >>> >>> >>> ************************************************************ >>> ******************* >>> UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log >>> for details): >>> ------------------------------------------------------------ >>> ------------------- >>> Error during download/extract/detection of HYPRE: >>> Unable to download hypre >>> Could not execute "git clone https://github.com/LLNL/hypre >>> /net/home/gtecton/flops/petsc-3.8.1/linux-gnu-x86_64/externa >>> lpackages/git.hypre": >>> Cloning into '/net/home/gtecton/flops/petsc >>> -3.8.1/linux-gnu-x86_64/externalpackages/git.hypre'... >>> fatal: unable to access 'https://github.com/LLNL/hypre/': >>> error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol >>> version >>> Unable to download package HYPRE from: git://https://github.com/LLNL/ >>> hypre >>> >>> Has anybody else seen this? It seems to get stuck on SSL, but web >>> security is sadly not my forte... >>> >> >> I have definitely had issues similar to this. It came up with Firedrake. >> I built my own Python, and had >> an early version of libssh/libcrypto. I think you need at least 1.0.1 to >> get the SSL version that github >> is now requiring. You can use ldd (or otool -L) to check the version for >> your Python. >> >> Thanks, >> >> Matt >> >> >>> Thanks a lot, >>> >>> Lukas >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Fri Jun 1 10:45:40 2018 From: mfadams at lbl.gov (Mark Adams) Date: Fri, 1 Jun 2018 11:45:40 -0400 Subject: [petsc-users] viewing question In-Reply-To: References: Message-ID: > > > I am guessing this is HDF5? > yep, > If you are using PetscFE, give your FE a name. > > thanks, -------------- next part -------------- An HTML attachment was scrubbed... URL: From s.lanthaler at gmail.com Fri Jun 1 11:07:09 2018 From: s.lanthaler at gmail.com (Samuel Lanthaler) Date: Fri, 1 Jun 2018 18:07:09 +0200 Subject: [petsc-users] MatPtAP In-Reply-To: References: <1153922A-8876-492B-8AD9-F41DD718FD21@anl.gov> <58D5E959-0E3C-4EF2-8270-882D1EA6EA38@mcs.anl.gov> <1db3b230-7243-4664-e846-20ceb0132eea@gmail.com> Message-ID: <078b8767-adc2-bafa-ee78-6df98c8cb4ab@gmail.com> On 06/01/2018 03:42 PM, Matthew Knepley wrote: > On Fri, Jun 1, 2018 at 9:21 AM, Samuel Lanthaler > > wrote: > > Hi, > > I was wondering what the most efficient way to use MatPtAP would > be in the following situation: I am discretizing a PDE system. The > discretization yields a matrix A that has a band structure (with k > upper and lower bands, say). In order to implement the boundary > conditions, I use a transformation matrix P which is essentially > the unit matrix, except for the entries P_{ij} where i,j n-i,n-j > P = [ B, 0, 0, 0, ..., 0, 0 ] > [ 0, 1, 0, 0, ..., 0, 0 ] > [ ] > [ ] > [ ..., 1, 0 ] > [ 0, 0, 0, 0, ..., 0, C ] > > with B,C are (k-by-k) matrices. > Right now, I'm simply constructing A, P and calling > > CALL > MatPtAP(petsc_matA,petsc_matP,MAT_INITIAL_MATRIX,PETSC_DEFAULT_REAL,petsc_matPtAP,ierr) > > where I haven't done anything to pestc_matPtAP, prior to this > call. Is this the way to do it? > > I'm asking because, currently, setting up the matrices A and P > takes very little time, whereas the operation MatPtAP is taking > quite long, which seems very odd... The matrices are of type > MPIAIJ. In my problem, the total matrix dimension is around 10'000 > and the matrix blocks (B,C) are of size ~100. > > > Are you sure this is what you want to do? Usually BC are local, since > by definition PDE are local, and > are applied pointwise. What kind of BC do you have here? > The boundary conditions are a mixture of Dirichlet and Neumann; in my case, the PDE is a system involving 8 variables on a disk, where the periodic direction is discretized using a Fourier series expansion, the radial direction uses B-splines. In reality, I have two matrices A,B, and want to solve the eigenvalue problem \lambda*B*x = A*x. I found it quite convenient to use a transformation P to a different set of variables y, such that x=P*y and x satisfies the BC iff certain components of y are 0. The latter is enforced by inserting spurious eigenvalues at the relevant components of y in the transformed eigenvalue problem \lambda*Pt*B*P*y=Pt*A*P*y. After solving the EVP in terms of y, I get back x=P*y. Is this an inherently bad/inefficient way of enforcing BC's? Thanks. > Thanks, > > Matt > > Thanks in advance for any ideas. > > Cheers, > Samuel > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From s.lanthaler at gmail.com Fri Jun 1 11:11:32 2018 From: s.lanthaler at gmail.com (Samuel Lanthaler) Date: Fri, 1 Jun 2018 18:11:32 +0200 Subject: [petsc-users] MatPtAP In-Reply-To: <8C838EE8-9DE9-40D3-8C5C-24EBF1EF8538@anl.gov> References: <1153922A-8876-492B-8AD9-F41DD718FD21@anl.gov> <58D5E959-0E3C-4EF2-8270-882D1EA6EA38@mcs.anl.gov> <1db3b230-7243-4664-e846-20ceb0132eea@gmail.com> <8C838EE8-9DE9-40D3-8C5C-24EBF1EF8538@anl.gov> Message-ID: <3e58c15f-14da-f23d-2986-63f163f4107e@gmail.com> On 06/01/2018 04:55 PM, Smith, Barry F. wrote: > How do you know it is exactly the MatPtAP() routine that is taking the large time? At least according to a simple timing (CALL CPU_TIME(...)), the call to MatPtAP appears to consume more time than the actual discretization of the PDE. Let me attach the output of -log_view. > > Please send the output of a run with -log_view so we can see where the time is being spent. > > Barry > > >> On Jun 1, 2018, at 8:21 AM, Samuel Lanthaler wrote: >> >> Hi, >> >> I was wondering what the most efficient way to use MatPtAP would be in the following situation: I am discretizing a PDE system. The discretization yields a matrix A that has a band structure (with k upper and lower bands, say). In order to implement the boundary conditions, I use a transformation matrix P which is essentially the unit matrix, except for the entries P_{ij} where i,j> >> P = [ B, 0, 0, 0, ..., 0, 0 ] >> [ 0, 1, 0, 0, ..., 0, 0 ] >> [ ] >> [ ] >> [ ..., 1, 0 ] >> [ 0, 0, 0, 0, ..., 0, C ] >> >> with B,C are (k-by-k) matrices. >> Right now, I'm simply constructing A, P and calling >> >> CALL MatPtAP(petsc_matA,petsc_matP,MAT_INITIAL_MATRIX,PETSC_DEFAULT_REAL,petsc_matPtAP,ierr) >> >> where I haven't done anything to pestc_matPtAP, prior to this call. Is this the way to do it? >> >> I'm asking because, currently, setting up the matrices A and P takes very little time, whereas the operation MatPtAP is taking quite long, which seems very odd... The matrices are of type MPIAIJ. In my problem, the total matrix dimension is around 10'000 and the matrix blocks (B,C) are of size ~100. >> >> Thanks in advance for any ideas. >> >> Cheers, >> Samuel -------------- next part -------------- ***** VENUS-LEVIS: rev Unversioned directory ***** ------------------------------ DELTA-F COMPUTATION FOR HYBRID KINETIC-MHD ------------------------------ ****************************** Cleaning sub-directories and creating diagnostic tree ------------------- ..... reading simulation data ..... reading equilibrium data ..... reading IO specification synchronize nparts,nprocs! nparts must be multiple of nprocs nparts,nprocs 1 4 Resetting values to satisfy constraint: nparts,nprocs 4 4 ..... initializing magnetic equilibrium Loading SFL (STRAIGHT-FIELD LINE) equilibrium - Plasma volume 197.391538791518 ---------------------------------------------- | magnetic equilibrium: | --------------------------------------------- | B0 = 0.984245824471487 | R0 = 10.1237829388870 | alfven velocity = 4809804.25531587 | alfven frequency = 475099.504241706 ---------------------------------------------- ---------------------------- Profiles related to flow (on axis): R0: 10.1237829388870 mOmp: 2.00000000000000 T0 [keV]: 1.10111199122785 Omega0 [Hz]: 0.000000000000000E+000 Mach0: 0.000000000000000E+000 ---------------------------- ..... initializing analytic distributions ..... bcasting analytic profiles data ..... calling test_analytic_profiles ..... reading equilibrium distribution data ..... bcasting equilibrium distribution data - TAE updaters ..... dumping analytic distribution; Ne,Np,Nmu 100 100 5 - TAE updaters ..... initializing delta-f computation ..... reading delta-f grid data ..... bcasting delta-f grid data ------------- WARNING: Using df_coll_freq to set ftime%dE_prefix. ftime%dE_prefix (original): (0.000000000000000E+000,0.000000000000000E+000) ftime%dE_prefix (set): (4.75099504241706,237.549752120853) ------------- ftime%grate = 4.75099504241706 ftime%freq = 237.549752120853 ftime%dE_prefix = (4.75099504241706,237.549752120853) ----------------------------- ftoro%n = 1 ftoro%dE_prefix = (0.000000000000000E+000,-1.00000000000000) ***** STARTING SIMULATION ***** over 4 processors with 4 orbits ****************************** initializing from MHD in ... setting type PDE.f90--> Chosen case: ********************* MHDflow ... initializing elements initializing with bunching on rational surfaces: q = 1.00000000000000 q = 2.00000000000000 ------ -------------------- q = qvals found for: q = 1.00001265158867 at s = 0.272160000000131 q = 2.00000720104897 at s = 0.666679999999601 -------------------- Nrad = 180 mmin = -3 mmax = 5 ntor = -1 test_mode = 7 Initializing operator setting natural units in MHDflow 8 setting natural units in MHDflow 8 setting natural units in MHDflow 8 setting natural units in MHDflow 8 Discretizing operator Discretization in progress (computation of matrices) Assembly of matA,matB done. Accounting for use of mixed finite elements. Add boundary conditions Adding boundary conditions: Neumann/Dirichlet ... setting up BC ... Setting up R/T (transformation) matrices ... Applying BC to A/B ... A->RtAR, B->RtBR ----------------------------------- time MatPtAP: 17.3900000000000 time MatZeroRowsColumns: 0.670999999999999 time MatAssembly: 4.000000000001336E-003 ------------------------- ----------------------------------- time MatPtAP: 16.3570000000000 time MatZeroRowsColumns: 0.631999999999998 time MatAssembly: 3.000000000000114E-003 ------------------------- ... writing out matrices in boundary conditions... destroying local objects time set BC: 43.9070000000000 Discretization done. writing out matrices Writing matA/matB to: PDE/ writing out operator options ***** WRITING AND UNLOADING MEMORY ***** ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./testing.x on a named spcpc340 with 4 processors, by lanthale Fri Jun 1 17:55:08 2018 Using Petsc Release Version 3.9.1, Apr, 29, 2018 Max Max/Min Avg Total Time (sec): 4.793e+01 1.00000 4.793e+01 Objects: 7.980e+02 1.00000 7.980e+02 Flop: 1.549e+10 1.05057 1.512e+10 6.048e+10 Flop/sec: 3.232e+08 1.05057 3.154e+08 1.262e+09 MPI Messages: 1.176e+03 1.34554 1.020e+03 4.078e+03 MPI Message Lengths: 5.629e+08 2.85940 2.993e+05 1.221e+09 MPI Reductions: 8.430e+02 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flop and VecAXPY() for complex vectors of length N --> 8N flop Summary of Stages: ----- Time ------ ----- Flop ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 4.7933e+01 100.0% 6.0476e+10 100.0% 4.078e+03 100.0% 2.993e+05 100.0% 8.330e+02 98.8% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flop: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flop in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flop --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage BuildTwoSided 234 1.0 5.9497e-03 1.1 0.00e+00 0.0 4.6e+02 4.0e+00 0.0e+00 0 0 11 0 0 0 0 11 0 0 0 BuildTwoSidedF 18 1.0 2.7772e+00 7.1 0.00e+00 0.0 8.0e+01 9.2e+05 0.0e+00 4 0 2 6 0 4 0 2 6 0 0 VecView 2 1.0 3.5000e-04 1.1 0.00e+00 0.0 6.0e+00 2.6e+04 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 243 1.0 5.0569e-04 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyBegin 3 1.0 1.3790e-03 1.6 0.00e+00 0.0 6.0e+01 1.6e+04 0.0e+00 0 0 1 0 0 0 0 1 0 0 0 VecAssemblyEnd 3 1.0 7.1049e-05 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 235 1.0 2.3565e-03 1.5 0.00e+00 0.0 1.4e+03 5.1e+03 0.0e+00 0 0 35 1 0 0 0 35 1 0 0 VecScatterEnd 235 1.0 1.8451e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatConvert 1 1.0 1.3385e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatScale 6 1.0 4.4126e-02 1.1 1.14e+07 1.0 6.0e+00 2.3e+03 0.0e+00 0 0 0 0 0 0 0 0 0 0 1026 MatAssemblyBegin 35 1.0 3.1616e+00 2.2 0.00e+00 0.0 2.0e+01 3.7e+06 0.0e+00 5 0 0 6 0 5 0 0 6 0 0 MatAssemblyEnd 35 1.0 3.3615e-01 1.0 0.00e+00 0.0 7.2e+01 1.1e+03 4.8e+01 1 0 2 0 6 1 0 2 0 6 0 MatZeroEntries 2 1.0 3.1345e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatView 8 1.0 1.8869e+00 1.0 0.00e+00 0.0 1.2e+02 9.0e+06 2.4e+01 4 0 3 88 3 4 0 3 88 3 0 MatMatMult 1 1.0 7.2594e+00 1.0 2.21e+09 1.0 3.6e+01 5.6e+05 1.2e+01 15 14 1 2 1 15 14 1 2 1 1195 MatMatMultSym 1 1.0 5.2231e+00 1.0 0.00e+00 0.0 3.0e+01 4.0e+05 1.2e+01 11 0 1 1 1 11 0 1 1 1 0 MatMatMultNum 1 1.0 2.0366e+00 1.0 2.21e+09 1.0 6.0e+00 1.3e+06 0.0e+00 4 14 0 1 0 4 14 0 1 0 4258 MatPtAP 2 1.0 3.3743e+01 1.0 1.33e+10 1.1 1.1e+02 1.0e+06 3.0e+01 70 86 3 9 4 70 86 3 9 4 1534 MatPtAPSymbolic 2 1.0 1.7436e+01 1.0 0.00e+00 0.0 7.2e+01 5.9e+05 1.4e+01 36 0 2 3 2 36 0 2 3 2 0 MatPtAPNumeric 2 1.0 1.6306e+01 1.0 1.33e+10 1.1 4.2e+01 1.7e+06 1.6e+01 34 86 1 6 2 34 86 1 6 2 3174 MatGetLocalMat 4 1.0 5.9245e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetBrAoCol 4 1.0 1.8033e-02 1.4 0.00e+00 0.0 6.0e+01 7.4e+05 0.0e+00 0 0 1 4 0 0 0 1 4 0 0 SFSetGraph 234 1.0 5.5790e-05 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SFSetUp 234 1.0 7.6420e-03 1.1 0.00e+00 0.0 1.4e+03 4.0e+00 0.0e+00 0 0 34 0 0 0 0 34 0 0 0 SFReduceBegin 234 1.0 1.0035e-03 1.3 0.00e+00 0.0 9.2e+02 4.0e+00 0.0e+00 0 0 23 0 0 0 0 23 0 0 0 SFReduceEnd 234 1.0 4.1008e-04 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Vector 487 485 8328944 0. Matrix 37 31 309181604 0. Index Set 12 12 22220 0. Vec Scatter 7 5 6160 0. Star Forest Graph 234 234 202176 0. Viewer 21 10 8400 0. ======================================================================================================================== Average time to get PetscTime(): 0. Average time for MPI_Barrier(): 9.53674e-07 Average time for zero size MPI_Send(): 2.02656e-06 #PETSc Option Table entries: -log_view -no_signal_handler #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: --prefix=/usr/local/petsc-3.9.1/linux_intel17.0 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --with-debugging=0 --with-blaslapack-dir=/usr/local/intel/17.0/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64 --with-scalapack=1 --with-scalapack-lib="-L/usr/local/intel/17.0/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64 -liomp5 -lpthread -lm" --with-scalapack-include=/usr/local/intel/17.0/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64/../../include --with-blacs=1 --with-blacs-lib="-L/usr/local/intel/17.0/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64 -liomp5 -lpthread -lm" --with-blacs-include=/usr/local/intel/17.0/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64/../../include --with-hypre=1 --download-hypre=1 --with-ml=1 --download-ml=1 --download-mumps=1 --download-parmetis=1 --download-metis=1 --download-superlu ----------------------------------------- Libraries compiled on 2018-05-30 11:33:16 on medusa Machine characteristics: Linux-4.4.114-42-default-x86_64-with-SuSE-42.3-x86_64 Using PETSc directory: /usr/local/petsc-3.9.1/linux_intel17.0 Using PETSc arch: ----------------------------------------- Using C compiler: mpicc -fPIC -wd1572 -g -O3 Using Fortran compiler: mpif90 -fPIC -g -O3 ----------------------------------------- Using include paths: -I/usr/local/petsc-3.9.1//include -I/usr/local/petsc-3.9.1///include -I/usr/local/petsc-3.9.1//include ----------------------------------------- Using C linker: mpicc Using Fortran linker: mpif90 Using libraries: -Wl,-rpath,/usr/local/petsc-3.9.1/linux_intel17.0/lib -L/usr/local/petsc-3.9.1/linux_intel17.0/lib -lpetsc -Wl,-rpath,/usr/local/petsc-3.9.1/linux_intel17.0/lib -L/usr/local/petsc-3.9.1/linux_intel17.0/lib -L/usr/local/intel/17.0/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64 -Wl,-rpath,/usr/local/intel/17.0/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64 -Wl,-rpath,/usr/local/intel/17.0/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/debug_mt -L/usr/local/intel/17.0/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/debug_mt -Wl,-rpath,/usr/local/intel/17.0/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib -L/usr/local/intel/17.0/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib -Wl,-rpath,/usr/local/hdf5-1.8.18-intel17.0/lib64 -L/usr/local/hdf5-1.8.18-intel17.0/lib64 -Wl,-rpath,/usr/local/intel/17.0/compilers_and_libraries_2017.0.098/linux/compiler/lib/intel64_lin -L/usr/local/intel/17.0/compilers_and_libraries_2017.0.098/linux/compiler/lib/intel64_lin -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/4.8 -L/usr/lib64/gcc/x86_64-suse-linux/4.8 -Wl,-rpath,/usr/x86_64-suse-linux/lib -L/usr/x86_64-suse-linux/lib -Wl,-rpath,/opt/intel/mpi-rt/2017.0.0/intel64/lib/debug_mt -Wl,-rpath,/opt/intel/mpi-rt/2017.0.0/intel64/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64 -liomp5 -lpthread -lm -lsuperlu -lHYPRE -lml -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lparmetis -lmetis -lstdc++ -ldl -lmpifort -lmpi -lmpigi -lrt -lpthread -lifport -lifcoremt_pic -limf -lsvml -lm -lipgo -lirc -lgcc_s -lirc_s -lstdc++ -ldl ----------------------------------------- ***** TERMINATED NORMALLY ***** From knepley at gmail.com Fri Jun 1 11:59:53 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 1 Jun 2018 12:59:53 -0400 Subject: [petsc-users] MatPtAP In-Reply-To: <078b8767-adc2-bafa-ee78-6df98c8cb4ab@gmail.com> References: <1153922A-8876-492B-8AD9-F41DD718FD21@anl.gov> <58D5E959-0E3C-4EF2-8270-882D1EA6EA38@mcs.anl.gov> <1db3b230-7243-4664-e846-20ceb0132eea@gmail.com> <078b8767-adc2-bafa-ee78-6df98c8cb4ab@gmail.com> Message-ID: On Fri, Jun 1, 2018 at 12:07 PM, Samuel Lanthaler wrote: > On 06/01/2018 03:42 PM, Matthew Knepley wrote: > > On Fri, Jun 1, 2018 at 9:21 AM, Samuel Lanthaler > wrote: > >> Hi, >> >> I was wondering what the most efficient way to use MatPtAP would be in >> the following situation: I am discretizing a PDE system. The discretization >> yields a matrix A that has a band structure (with k upper and lower bands, >> say). In order to implement the boundary conditions, I use a transformation >> matrix P which is essentially the unit matrix, except for the entries >> P_{ij} where i,j> >> P = [ B, 0, 0, 0, ..., 0, 0 ] >> [ 0, 1, 0, 0, ..., 0, 0 ] >> [ ] >> [ ] >> [ ..., 1, 0 ] >> [ 0, 0, 0, 0, ..., 0, C ] >> >> with B,C are (k-by-k) matrices. >> Right now, I'm simply constructing A, P and calling >> >> CALL MatPtAP(petsc_matA,petsc_matP,MAT_INITIAL_MATRIX,PETSC_DEFAU >> LT_REAL,petsc_matPtAP,ierr) >> >> where I haven't done anything to pestc_matPtAP, prior to this call. Is >> this the way to do it? >> >> I'm asking because, currently, setting up the matrices A and P takes very >> little time, whereas the operation MatPtAP is taking quite long, which >> seems very odd... The matrices are of type MPIAIJ. In my problem, the total >> matrix dimension is around 10'000 and the matrix blocks (B,C) are of size >> ~100. >> > > Are you sure this is what you want to do? Usually BC are local, since by > definition PDE are local, and > are applied pointwise. What kind of BC do you have here? > > > The boundary conditions are a mixture of Dirichlet and Neumann; in my > case, the PDE is a system involving 8 variables on a disk, where the > periodic direction is discretized using a Fourier series expansion, the > radial direction uses B-splines. > > In reality, I have two matrices A,B, and want to solve the eigenvalue > problem \lambda*B*x = A*x. > I found it quite convenient to use a transformation P to a different set > of variables y, such that x=P*y and x satisfies the BC iff certain > components of y are 0. The latter is enforced by inserting spurious > eigenvalues at the relevant components of y in the transformed eigenvalue > problem \lambda*Pt*B*P*y=Pt*A*P*y. After solving the EVP in terms of y, I > get back x=P*y. > Is this an inherently bad/inefficient way of enforcing BC's? Thanks. > I cannot quite see what is happening. However it appears that you are coupling all the unknowns on a boundary to enforce the boundary condition. If this is true, its pretty expensive. For example, in 2D you have N^2 unknowns for N boundary unknowns. However, dense coupling on the boundary produces N^2 storage and N^3 computation, which is not great. Am I missing something? Its fine to do this for some sizes (2D BEM guys do it all the time), but for bigger stuff it gets untenable. Thanks, Matt > > > Thanks, > > Matt > > >> Thanks in advance for any ideas. >> >> Cheers, >> Samuel >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From danyang.su at gmail.com Fri Jun 1 12:07:45 2018 From: danyang.su at gmail.com (Danyang Su) Date: Fri, 1 Jun 2018 10:07:45 -0700 Subject: [petsc-users] Makefile for mixed C++ and Fortran code Message-ID: <164761f9-80cc-042e-6a4f-558dc90992b7@gmail.com> Hi All, My code needs to link to an external C++ library (CGAL). The code is written in Fortran and I have already written interface to let Fortran call C++ function. For the sequential version without PETSc, it can be compiled without problem using the following makefile. The parallel version without CGAL can also be compiled successfully. However, when I tried to use PETSc together with CGAL library, I cannot compile the code. My questions is: How can I modify the makefile? Do I need to reconfigure PETSc with special flags? All the makefile samples are shown below. #makefile for sequential version FC = gfortran #FC = ifort CXX = g++ -std=c++11 DLIB = -lstdc++ -lm -L/usr/local/lib -rdynamic /usr/local/lib/libmpfr.so /usr/local/lib/libgmp.so /usr/local/lib/libCGAL_ImageIO.so.11.0.1 /usr/local/lib/libCGAL.so.11.0.1 /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_system.so -lpthread -lGLU -lGL -lX11 -lXext -lz /usr/local/lib/libCGAL_ImageIO.so.11.0.1 /usr/local/lib/libCGAL.so.11.0.1 /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_system.so -lpthread -lGLU -lGL -lX11 -lXext -lz /usr/local/lib/libmpfr.so /usr/local/lib/libgmp.so /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_system.so -lpthread -Wl,-rpath,/usr/local/lib FFLAGS = -O3 CXXFLAGS = -O3 FPPFLAGS = -DUSECGAL SRC =./../../ SOURCES = $(SRC)usg/math_common.o\ $(SRC)usg/geometry_definition.o\ $(SRC)usg/cgal_common.o\ ... executable: $(SOURCES) $(FC) $(FFLAGS) $(FPPFLAGS) -o executable.out $(SOURCES) ${LIS_LIB} $(DLIB) %.o:%.F90 $(FC) $(FFLAGS) $(FPPFLAGS) -c -frounding-math $< -o $@ %.o:%.cpp $(CXX) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ #makefile for parallel version with PETSc, without CGAL #FC = ifort #FC = gfortran DLIB = -lm FFLAGS = -O3 FPPFLAGS = -DUSEPETSC SRC =./../../ SOURCES = $(SRC)usg/math_common.o\ $(SRC)usg/geometry_definition.o\ $(SRC)usg/cgal_common.o\ ... executable: $(SOURCES) chkopts -${FLINKER} $(FFLAGS) $(FPPFLAGS) $(CPPFLAGS) -o executable.out $(SOURCES) ${PETSC_LIB} #makefile for parallel version with PETSc, with CGAL, CANNOT work #FC = ifort #FC = gfortran DLIB = -lm FFLAGS = -O3 FPPFLAGS = -DUSEPETSC -DUSECGAL SRC =./../../ SOURCES = $(SRC)usg/math_common.o\ $(SRC)usg/geometry_definition.o\ $(SRC)usg/cgal_common.o\ ... executable: $(SOURCES) chkopts -${FLINKER} $(FFLAGS) $(FPPFLAGS) $(CPPFLAGS) -o executable.out $(SOURCES) ${PETSC_LIB} %.o:%.F90 $(FC) $(FFLAGS) $(FPPFLAGS) -c -frounding-math $< -o $@ %.o:%.cpp $(CXX) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ Thanks, Danyang From bsmith at mcs.anl.gov Fri Jun 1 12:29:15 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 1 Jun 2018 17:29:15 +0000 Subject: [petsc-users] Makefile for mixed C++ and Fortran code In-Reply-To: <164761f9-80cc-042e-6a4f-558dc90992b7@gmail.com> References: <164761f9-80cc-042e-6a4f-558dc90992b7@gmail.com> Message-ID: What happens if you add ${DLIB} to the end of the line > executable: $(SOURCES) chkopts > -${FLINKER} $(FFLAGS) $(FPPFLAGS) $(CPPFLAGS) -o executable.out $(SOURCES) ${PETSC_LIB} Send all the output from trying this. Barry > On Jun 1, 2018, at 12:07 PM, Danyang Su wrote: > > Hi All, > > My code needs to link to an external C++ library (CGAL). The code is written in Fortran and I have already written interface to let Fortran call C++ function. For the sequential version without PETSc, it can be compiled without problem using the following makefile. The parallel version without CGAL can also be compiled successfully. However, when I tried to use PETSc together with CGAL library, I cannot compile the code. My questions is: How can I modify the makefile? Do I need to reconfigure PETSc with special flags? All the makefile samples are shown below. > > #makefile for sequential version > > FC = gfortran > #FC = ifort > CXX = g++ -std=c++11 > > DLIB = -lstdc++ -lm -L/usr/local/lib -rdynamic /usr/local/lib/libmpfr.so /usr/local/lib/libgmp.so /usr/local/lib/libCGAL_ImageIO.so.11.0.1 /usr/local/lib/libCGAL.so.11.0.1 /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_system.so -lpthread -lGLU -lGL -lX11 -lXext -lz /usr/local/lib/libCGAL_ImageIO.so.11.0.1 /usr/local/lib/libCGAL.so.11.0.1 /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_system.so -lpthread -lGLU -lGL -lX11 -lXext -lz /usr/local/lib/libmpfr.so /usr/local/lib/libgmp.so /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_system.so -lpthread -Wl,-rpath,/usr/local/lib > > FFLAGS = -O3 > CXXFLAGS = -O3 > > FPPFLAGS = -DUSECGAL > > SRC =./../../ > > SOURCES = $(SRC)usg/math_common.o\ > $(SRC)usg/geometry_definition.o\ > $(SRC)usg/cgal_common.o\ > > ... > > executable: $(SOURCES) > $(FC) $(FFLAGS) $(FPPFLAGS) -o executable.out $(SOURCES) ${LIS_LIB} $(DLIB) > %.o:%.F90 > $(FC) $(FFLAGS) $(FPPFLAGS) -c -frounding-math $< -o $@ > %.o:%.cpp > $(CXX) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ > > > #makefile for parallel version with PETSc, without CGAL > > #FC = ifort > #FC = gfortran > > DLIB = -lm > > FFLAGS = -O3 > > FPPFLAGS = -DUSEPETSC > > SRC =./../../ > > SOURCES = $(SRC)usg/math_common.o\ > $(SRC)usg/geometry_definition.o\ > $(SRC)usg/cgal_common.o\ > > ... > > executable: $(SOURCES) chkopts > -${FLINKER} $(FFLAGS) $(FPPFLAGS) $(CPPFLAGS) -o executable.out $(SOURCES) ${PETSC_LIB} > > > #makefile for parallel version with PETSc, with CGAL, CANNOT work > > #FC = ifort > #FC = gfortran > > DLIB = -lm > > FFLAGS = -O3 > > FPPFLAGS = -DUSEPETSC -DUSECGAL > > SRC =./../../ > > SOURCES = $(SRC)usg/math_common.o\ > $(SRC)usg/geometry_definition.o\ > $(SRC)usg/cgal_common.o\ > > ... > > executable: $(SOURCES) chkopts > -${FLINKER} $(FFLAGS) $(FPPFLAGS) $(CPPFLAGS) -o executable.out $(SOURCES) ${PETSC_LIB} > > %.o:%.F90 > $(FC) $(FFLAGS) $(FPPFLAGS) -c -frounding-math $< -o $@ > %.o:%.cpp > $(CXX) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ > > > Thanks, > > Danyang > From bsmith at mcs.anl.gov Fri Jun 1 12:35:33 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 1 Jun 2018 17:35:33 +0000 Subject: [petsc-users] MatPtAP In-Reply-To: <078b8767-adc2-bafa-ee78-6df98c8cb4ab@gmail.com> References: <1153922A-8876-492B-8AD9-F41DD718FD21@anl.gov> <58D5E959-0E3C-4EF2-8270-882D1EA6EA38@mcs.anl.gov> <1db3b230-7243-4664-e846-20ceb0132eea@gmail.com> <078b8767-adc2-bafa-ee78-6df98c8cb4ab@gmail.com> Message-ID: Could you save the P matrix with MatView() using a binary viewer and the A matrix with MatView() and the binary viewer and email them to petsc-maint at mcs.anl.gov ? Then we can run the code in the profiler with your matrices and see if there is any way to speed up the computation. Barry > On Jun 1, 2018, at 11:07 AM, Samuel Lanthaler wrote: > > On 06/01/2018 03:42 PM, Matthew Knepley wrote: >> On Fri, Jun 1, 2018 at 9:21 AM, Samuel Lanthaler wrote: >> Hi, >> >> I was wondering what the most efficient way to use MatPtAP would be in the following situation: I am discretizing a PDE system. The discretization yields a matrix A that has a band structure (with k upper and lower bands, say). In order to implement the boundary conditions, I use a transformation matrix P which is essentially the unit matrix, except for the entries P_{ij} where i,j> >> P = [ B, 0, 0, 0, ..., 0, 0 ] >> [ 0, 1, 0, 0, ..., 0, 0 ] >> [ ] >> [ ] >> [ ..., 1, 0 ] >> [ 0, 0, 0, 0, ..., 0, C ] >> >> with B,C are (k-by-k) matrices. >> Right now, I'm simply constructing A, P and calling >> >> CALL MatPtAP(petsc_matA,petsc_matP,MAT_INITIAL_MATRIX,PETSC_DEFAULT_REAL,petsc_matPtAP,ierr) >> >> where I haven't done anything to pestc_matPtAP, prior to this call. Is this the way to do it? >> >> I'm asking because, currently, setting up the matrices A and P takes very little time, whereas the operation MatPtAP is taking quite long, which seems very odd... The matrices are of type MPIAIJ. In my problem, the total matrix dimension is around 10'000 and the matrix blocks (B,C) are of size ~100. >> >> Are you sure this is what you want to do? Usually BC are local, since by definition PDE are local, and >> are applied pointwise. What kind of BC do you have here? >> > > The boundary conditions are a mixture of Dirichlet and Neumann; in my case, the PDE is a system involving 8 variables on a disk, where the periodic direction is discretized using a Fourier series expansion, the radial direction uses B-splines. > > In reality, I have two matrices A,B, and want to solve the eigenvalue problem \lambda*B*x = A*x. > I found it quite convenient to use a transformation P to a different set of variables y, such that x=P*y and x satisfies the BC iff certain components of y are 0. The latter is enforced by inserting spurious eigenvalues at the relevant components of y in the transformed eigenvalue problem \lambda*Pt*B*P*y=Pt*A*P*y. After solving the EVP in terms of y, I get back x=P*y. > Is this an inherently bad/inefficient way of enforcing BC's? Thanks. > > > > >> Thanks, >> >> Matt >> >> Thanks in advance for any ideas. >> >> Cheers, >> Samuel >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ > From danyang.su at gmail.com Fri Jun 1 12:37:21 2018 From: danyang.su at gmail.com (Danyang Su) Date: Fri, 1 Jun 2018 10:37:21 -0700 Subject: [petsc-users] Makefile for mixed C++ and Fortran code In-Reply-To: <164761f9-80cc-042e-6a4f-558dc90992b7@gmail.com> References: <164761f9-80cc-042e-6a4f-558dc90992b7@gmail.com> Message-ID: Follow up: With following command executable: $(SOURCES) chkopts -${FLINKER} $(FFLAGS) $(FPPFLAGS) $(CPPFLAGS) -o executable.out $(SOURCES) ${PETSC_LIB} %.o:%.F90 $(FLINKER) $(FFLAGS) $(FPPFLAGS) -c -frounding-math $< -o $@ %.o:%.cpp $(CLINKER) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ The compiler return error: no match function. ../../usg/cgal_triangulation_2d.cpp: In function ?void outputTriangulation2d(in\ t, const char*, int, const char*)?: ../../usg/cgal_triangulation_2d.cpp:485:20: error: no matching function for cal\ l to ?std::basic_ofstream::open(std::string&)? out.open(strfile); Thanks, Danyang On 18-06-01 10:07 AM, Danyang Su wrote: > Hi All, > > My code needs to link to an external C++ library (CGAL). The code is > written in Fortran and I have already written interface to let Fortran > call C++ function. For the sequential version without PETSc, it can be > compiled without problem using the following makefile. The parallel > version without CGAL can also be compiled successfully. However, when > I tried to use PETSc together with CGAL library, I cannot compile the > code. My questions is: How can I modify the makefile? Do I need to > reconfigure PETSc with special flags? All the makefile samples are > shown below. > > #makefile for sequential version > > FC = gfortran > #FC = ifort > CXX = g++ -std=c++11 > > DLIB = -lstdc++ -lm -L/usr/local/lib -rdynamic > /usr/local/lib/libmpfr.so /usr/local/lib/libgmp.so > /usr/local/lib/libCGAL_ImageIO.so.11.0.1 > /usr/local/lib/libCGAL.so.11.0.1 /usr/local/lib/libboost_thread.so > /usr/local/lib/libboost_system.so -lpthread -lGLU -lGL -lX11 -lXext > -lz /usr/local/lib/libCGAL_ImageIO.so.11.0.1 > /usr/local/lib/libCGAL.so.11.0.1 /usr/local/lib/libboost_thread.so > /usr/local/lib/libboost_system.so -lpthread -lGLU -lGL -lX11 -lXext > -lz /usr/local/lib/libmpfr.so /usr/local/lib/libgmp.so > /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_system.so > -lpthread -Wl,-rpath,/usr/local/lib > > FFLAGS = -O3 > CXXFLAGS = -O3 > > FPPFLAGS = -DUSECGAL > > SRC =./../../ > > SOURCES = $(SRC)usg/math_common.o\ > $(SRC)usg/geometry_definition.o\ > $(SRC)usg/cgal_common.o\ > > ... > > executable: $(SOURCES) > $(FC) $(FFLAGS) $(FPPFLAGS) -o executable.out $(SOURCES) > ${LIS_LIB} $(DLIB) > %.o:%.F90 > $(FC) $(FFLAGS) $(FPPFLAGS) -c -frounding-math $< -o $@ > %.o:%.cpp > $(CXX) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ > > > #makefile for parallel version with PETSc, without CGAL > > #FC = ifort > #FC = gfortran > > DLIB = -lm > > FFLAGS = -O3 > > FPPFLAGS = -DUSEPETSC > > SRC =./../../ > > SOURCES = $(SRC)usg/math_common.o\ > $(SRC)usg/geometry_definition.o\ > $(SRC)usg/cgal_common.o\ > > ... > > executable: $(SOURCES) chkopts > -${FLINKER} $(FFLAGS) $(FPPFLAGS) $(CPPFLAGS) -o executable.out > $(SOURCES) ${PETSC_LIB} > > > #makefile for parallel version with PETSc, with CGAL, CANNOT work > > #FC = ifort > #FC = gfortran > > DLIB = -lm > > FFLAGS = -O3 > > FPPFLAGS = -DUSEPETSC -DUSECGAL > > SRC =./../../ > > SOURCES = $(SRC)usg/math_common.o\ > $(SRC)usg/geometry_definition.o\ > $(SRC)usg/cgal_common.o\ > > ... > > executable: $(SOURCES) chkopts > -${FLINKER} $(FFLAGS) $(FPPFLAGS) $(CPPFLAGS) -o executable.out > $(SOURCES) ${PETSC_LIB} > > %.o:%.F90 > $(FC) $(FFLAGS) $(FPPFLAGS) -c -frounding-math $< -o $@ > %.o:%.cpp > $(CXX) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ > > > Thanks, > > Danyang > From bsmith at mcs.anl.gov Fri Jun 1 12:41:07 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 1 Jun 2018 17:41:07 +0000 Subject: [petsc-users] Makefile for mixed C++ and Fortran code In-Reply-To: References: <164761f9-80cc-042e-6a4f-558dc90992b7@gmail.com> Message-ID: <0335466E-9658-4FC5-A6FA-0886F57DE5BF@anl.gov> You need to determine exactly what flags are passed to the C++ compiler for your compile that works and make sure those same flags are used in "PETSc version" of the makefile. You could add the flags directly to the rule > %.o:%.cpp > $(CLINKER) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ Barry > On Jun 1, 2018, at 12:37 PM, Danyang Su wrote: > > Follow up: > > With following command > > executable: $(SOURCES) chkopts > -${FLINKER} $(FFLAGS) $(FPPFLAGS) $(CPPFLAGS) -o executable.out $(SOURCES) ${PETSC_LIB} > > %.o:%.F90 > $(FLINKER) $(FFLAGS) $(FPPFLAGS) -c -frounding-math $< -o $@ > %.o:%.cpp > $(CLINKER) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ > > The compiler return error: no match function. > > ../../usg/cgal_triangulation_2d.cpp: In function ?void outputTriangulation2d(in\ > t, const char*, int, const char*)?: > ../../usg/cgal_triangulation_2d.cpp:485:20: error: no matching function for cal\ > l to ?std::basic_ofstream::open(std::string&)? > out.open(strfile); > > Thanks, > > > Danyang > > On 18-06-01 10:07 AM, Danyang Su wrote: >> Hi All, >> >> My code needs to link to an external C++ library (CGAL). The code is written in Fortran and I have already written interface to let Fortran call C++ function. For the sequential version without PETSc, it can be compiled without problem using the following makefile. The parallel version without CGAL can also be compiled successfully. However, when I tried to use PETSc together with CGAL library, I cannot compile the code. My questions is: How can I modify the makefile? Do I need to reconfigure PETSc with special flags? All the makefile samples are shown below. >> >> #makefile for sequential version >> >> FC = gfortran >> #FC = ifort >> CXX = g++ -std=c++11 >> >> DLIB = -lstdc++ -lm -L/usr/local/lib -rdynamic /usr/local/lib/libmpfr.so /usr/local/lib/libgmp.so /usr/local/lib/libCGAL_ImageIO.so.11.0.1 /usr/local/lib/libCGAL.so.11.0.1 /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_system.so -lpthread -lGLU -lGL -lX11 -lXext -lz /usr/local/lib/libCGAL_ImageIO.so.11.0.1 /usr/local/lib/libCGAL.so.11.0.1 /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_system.so -lpthread -lGLU -lGL -lX11 -lXext -lz /usr/local/lib/libmpfr.so /usr/local/lib/libgmp.so /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_system.so -lpthread -Wl,-rpath,/usr/local/lib >> >> FFLAGS = -O3 >> CXXFLAGS = -O3 >> >> FPPFLAGS = -DUSECGAL >> >> SRC =./../../ >> >> SOURCES = $(SRC)usg/math_common.o\ >> $(SRC)usg/geometry_definition.o\ >> $(SRC)usg/cgal_common.o\ >> >> ... >> >> executable: $(SOURCES) >> $(FC) $(FFLAGS) $(FPPFLAGS) -o executable.out $(SOURCES) ${LIS_LIB} $(DLIB) >> %.o:%.F90 >> $(FC) $(FFLAGS) $(FPPFLAGS) -c -frounding-math $< -o $@ >> %.o:%.cpp >> $(CXX) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ >> >> >> #makefile for parallel version with PETSc, without CGAL >> >> #FC = ifort >> #FC = gfortran >> >> DLIB = -lm >> >> FFLAGS = -O3 >> >> FPPFLAGS = -DUSEPETSC >> >> SRC =./../../ >> >> SOURCES = $(SRC)usg/math_common.o\ >> $(SRC)usg/geometry_definition.o\ >> $(SRC)usg/cgal_common.o\ >> >> ... >> >> executable: $(SOURCES) chkopts >> -${FLINKER} $(FFLAGS) $(FPPFLAGS) $(CPPFLAGS) -o executable.out $(SOURCES) ${PETSC_LIB} >> >> >> #makefile for parallel version with PETSc, with CGAL, CANNOT work >> >> #FC = ifort >> #FC = gfortran >> >> DLIB = -lm >> >> FFLAGS = -O3 >> >> FPPFLAGS = -DUSEPETSC -DUSECGAL >> >> SRC =./../../ >> >> SOURCES = $(SRC)usg/math_common.o\ >> $(SRC)usg/geometry_definition.o\ >> $(SRC)usg/cgal_common.o\ >> >> ... >> >> executable: $(SOURCES) chkopts >> -${FLINKER} $(FFLAGS) $(FPPFLAGS) $(CPPFLAGS) -o executable.out $(SOURCES) ${PETSC_LIB} >> >> %.o:%.F90 >> $(FC) $(FFLAGS) $(FPPFLAGS) -c -frounding-math $< -o $@ >> %.o:%.cpp >> $(CXX) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ >> >> >> Thanks, >> >> Danyang >> > From danyang.su at gmail.com Fri Jun 1 12:42:37 2018 From: danyang.su at gmail.com (Danyang Su) Date: Fri, 1 Jun 2018 10:42:37 -0700 Subject: [petsc-users] Makefile for mixed C++ and Fortran code In-Reply-To: References: <164761f9-80cc-042e-6a4f-558dc90992b7@gmail.com> Message-ID: <5effd043-4ac3-7d16-48f0-c8fef85dd356@gmail.com> On 18-06-01 10:29 AM, Smith, Barry F. wrote: > What happens if you add ${DLIB} to the end of the line > >> executable: $(SOURCES) chkopts >> -${FLINKER} $(FFLAGS) $(FPPFLAGS) $(CPPFLAGS) -o executable.out $(SOURCES) ${PETSC_LIB} > Send all the output from trying this. > > Barry Thanks, Barry. Still not work after including ${DLIB} > > > > >> On Jun 1, 2018, at 12:07 PM, Danyang Su wrote: >> >> Hi All, >> >> My code needs to link to an external C++ library (CGAL). The code is written in Fortran and I have already written interface to let Fortran call C++ function. For the sequential version without PETSc, it can be compiled without problem using the following makefile. The parallel version without CGAL can also be compiled successfully. However, when I tried to use PETSc together with CGAL library, I cannot compile the code. My questions is: How can I modify the makefile? Do I need to reconfigure PETSc with special flags? All the makefile samples are shown below. >> >> #makefile for sequential version >> >> FC = gfortran >> #FC = ifort >> CXX = g++ -std=c++11 >> >> DLIB = -lstdc++ -lm -L/usr/local/lib -rdynamic /usr/local/lib/libmpfr.so /usr/local/lib/libgmp.so /usr/local/lib/libCGAL_ImageIO.so.11.0.1 /usr/local/lib/libCGAL.so.11.0.1 /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_system.so -lpthread -lGLU -lGL -lX11 -lXext -lz /usr/local/lib/libCGAL_ImageIO.so.11.0.1 /usr/local/lib/libCGAL.so.11.0.1 /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_system.so -lpthread -lGLU -lGL -lX11 -lXext -lz /usr/local/lib/libmpfr.so /usr/local/lib/libgmp.so /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_system.so -lpthread -Wl,-rpath,/usr/local/lib >> >> FFLAGS = -O3 >> CXXFLAGS = -O3 >> >> FPPFLAGS = -DUSECGAL >> >> SRC =./../../ >> >> SOURCES = $(SRC)usg/math_common.o\ >> $(SRC)usg/geometry_definition.o\ >> $(SRC)usg/cgal_common.o\ >> >> ... >> >> executable: $(SOURCES) >> $(FC) $(FFLAGS) $(FPPFLAGS) -o executable.out $(SOURCES) ${LIS_LIB} $(DLIB) >> %.o:%.F90 >> $(FC) $(FFLAGS) $(FPPFLAGS) -c -frounding-math $< -o $@ >> %.o:%.cpp >> $(CXX) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ >> >> >> #makefile for parallel version with PETSc, without CGAL >> >> #FC = ifort >> #FC = gfortran >> >> DLIB = -lm >> >> FFLAGS = -O3 >> >> FPPFLAGS = -DUSEPETSC >> >> SRC =./../../ >> >> SOURCES = $(SRC)usg/math_common.o\ >> $(SRC)usg/geometry_definition.o\ >> $(SRC)usg/cgal_common.o\ >> >> ... >> >> executable: $(SOURCES) chkopts >> -${FLINKER} $(FFLAGS) $(FPPFLAGS) $(CPPFLAGS) -o executable.out $(SOURCES) ${PETSC_LIB} >> >> >> #makefile for parallel version with PETSc, with CGAL, CANNOT work >> >> #FC = ifort >> #FC = gfortran >> >> DLIB = -lm >> >> FFLAGS = -O3 >> >> FPPFLAGS = -DUSEPETSC -DUSECGAL >> >> SRC =./../../ >> >> SOURCES = $(SRC)usg/math_common.o\ >> $(SRC)usg/geometry_definition.o\ >> $(SRC)usg/cgal_common.o\ >> >> ... >> >> executable: $(SOURCES) chkopts >> -${FLINKER} $(FFLAGS) $(FPPFLAGS) $(CPPFLAGS) -o executable.out $(SOURCES) ${PETSC_LIB} >> >> %.o:%.F90 >> $(FC) $(FFLAGS) $(FPPFLAGS) -c -frounding-math $< -o $@ >> %.o:%.cpp >> $(CXX) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ >> >> >> Thanks, >> >> Danyang >> From hzhang at mcs.anl.gov Fri Jun 1 14:04:04 2018 From: hzhang at mcs.anl.gov (Hong) Date: Fri, 1 Jun 2018 14:04:04 -0500 Subject: [petsc-users] MatPtAP In-Reply-To: References: <1153922A-8876-492B-8AD9-F41DD718FD21@anl.gov> <58D5E959-0E3C-4EF2-8270-882D1EA6EA38@mcs.anl.gov> <1db3b230-7243-4664-e846-20ceb0132eea@gmail.com> <078b8767-adc2-bafa-ee78-6df98c8cb4ab@gmail.com> Message-ID: Samuel, I have following questions: 1) Why solving \lambda*Pt*B*P*y=Pt*A*P*y is better than solving original \lambda*B*x = A*x? 2) Does your eigen solver require matrix factorization? If not, i.e., only uses mat-vec multiplication, then you may implement z = (Pt*(A*(P*y))) in your eigensolver instead of using mat-mat-mat multi plication. 3) petsc PtAP() was implemented for multigrid applications, in which the product C = PtAP is a denser but a much smaller matrix. I have not seen the use of your case, that P is square with same size as A. If C is much denser than A, then PtAP consumes a large portion of time is anticipated. Hong On Fri, Jun 1, 2018 at 12:35 PM, Smith, Barry F. wrote: > > Could you save the P matrix with MatView() using a binary viewer and the > A matrix with MatView() and the binary viewer and email them to > petsc-maint at mcs.anl.gov ? Then we can run the code in the profiler with > your matrices and see if there is any way to speed up the computation. > > Barry > > > > On Jun 1, 2018, at 11:07 AM, Samuel Lanthaler > wrote: > > > > On 06/01/2018 03:42 PM, Matthew Knepley wrote: > >> On Fri, Jun 1, 2018 at 9:21 AM, Samuel Lanthaler > wrote: > >> Hi, > >> > >> I was wondering what the most efficient way to use MatPtAP would be in > the following situation: I am discretizing a PDE system. The discretization > yields a matrix A that has a band structure (with k upper and lower bands, > say). In order to implement the boundary conditions, I use a transformation > matrix P which is essentially the unit matrix, except for the entries > P_{ij} where i,j >> > >> P = [ B, 0, 0, 0, ..., 0, 0 ] > >> [ 0, 1, 0, 0, ..., 0, 0 ] > >> [ ] > >> [ ] > >> [ ..., 1, 0 ] > >> [ 0, 0, 0, 0, ..., 0, C ] > >> > >> with B,C are (k-by-k) matrices. > >> Right now, I'm simply constructing A, P and calling > >> > >> CALL MatPtAP(petsc_matA,petsc_matP,MAT_INITIAL_MATRIX,PETSC_ > DEFAULT_REAL,petsc_matPtAP,ierr) > >> > >> where I haven't done anything to pestc_matPtAP, prior to this call. Is > this the way to do it? > >> > >> I'm asking because, currently, setting up the matrices A and P takes > very little time, whereas the operation MatPtAP is taking quite long, which > seems very odd... The matrices are of type MPIAIJ. In my problem, the total > matrix dimension is around 10'000 and the matrix blocks (B,C) are of size > ~100. > >> > >> Are you sure this is what you want to do? Usually BC are local, since > by definition PDE are local, and > >> are applied pointwise. What kind of BC do you have here? > >> > > > > The boundary conditions are a mixture of Dirichlet and Neumann; in my > case, the PDE is a system involving 8 variables on a disk, where the > periodic direction is discretized using a Fourier series expansion, the > radial direction uses B-splines. > > > > In reality, I have two matrices A,B, and want to solve the eigenvalue > problem \lambda*B*x = A*x. > > I found it quite convenient to use a transformation P to a different set > of variables y, such that x=P*y and x satisfies the BC iff certain > components of y are 0. The latter is enforced by inserting spurious > eigenvalues at the relevant components of y in the transformed eigenvalue > problem \lambda*Pt*B*P*y=Pt*A*P*y. After solving the EVP in terms of y, I > get back x=P*y. > > Is this an inherently bad/inefficient way of enforcing BC's? Thanks. > > > > > > > > > >> Thanks, > >> > >> Matt > >> > >> Thanks in advance for any ideas. > >> > >> Cheers, > >> Samuel > >> > >> > >> > >> -- > >> What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > >> -- Norbert Wiener > >> > >> https://www.cse.buffalo.edu/~knepley/ > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mbuerkle at web.de Fri Jun 1 21:50:18 2018 From: mbuerkle at web.de (Marius Buerkle) Date: Sat, 2 Jun 2018 04:50:18 +0200 Subject: [petsc-users] MAT_NEW_NONZERO_LOCATIONS working? In-Reply-To: References: <1153922A-8876-492B-8AD9-F41DD718FD21@anl.gov> <58D5E959-0E3C-4EF2-8270-882D1EA6EA38@mcs.anl.gov> Message-ID: An HTML attachment was scrubbed... URL: From jczhang at mcs.anl.gov Fri Jun 1 22:20:28 2018 From: jczhang at mcs.anl.gov (Junchao Zhang) Date: Fri, 1 Jun 2018 22:20:28 -0500 Subject: [petsc-users] Poor weak scaling when solvingsuccessivelinearsystems In-Reply-To: References: <6d9b6488-3957-0d74-148a-323c78d42f6a@physik.uni-giessen.de> <079D8B4D-C57B-4054-B9BD-05644E52C3A4@anl.gov> Message-ID: Hi,Michael, You can add -log_sync besides -log_view, which adds barriers to certain events but measures barrier time separately from the events. I find this option makes it easier to interpret log_view output. --Junchao Zhang On Wed, May 30, 2018 at 3:27 AM, Michael Becker < Michael.Becker at physik.uni-giessen.de> wrote: > Barry: On its way. Could take a couple days again. > > Junchao: I unfortunately don't have access to a cluster with a faster > network. This one has a mixed 4X QDR-FDR InfiniBand 2:1 blocking fat-tree > network, which I realize causes parallel slowdown if the nodes are not > connected to the same switch. Each node has 24 processors (2x12/socket) and > four NUMA domains (two for each socket). > The ranks are usually not distributed perfectly even, i.e. for 125 > processes, of the six required nodes, five would use 21 cores and one 20. > Would using another CPU type make a difference communication-wise? I could > switch to faster ones (on the same network), but I always assumed this > would only improve performance of the stuff that is unrelated to > communication. > > Michael > > > > The log files have something like "Average time for zero size MPI_Send(): > 1.84231e-05". It looks you ran on a cluster with a very slow network. A > typical machine should give less than 1/10 of the latency you have. An easy > way to try is just running the code on a machine with a faster network and > see what happens. > > Also, how many cores & numa domains does a compute node have? I could not > figure out how you distributed the 125 MPI ranks evenly. > > --Junchao Zhang > > On Tue, May 29, 2018 at 6:18 AM, Michael Becker < > Michael.Becker at physik.uni-giessen.de> wrote: > >> Hello again, >> >> here are the updated log_view files for 125 and 1000 processors. I ran >> both problems twice, the first time with all processors per node allocated >> ("-1.txt"), the second with only half on twice the number of nodes >> ("-2.txt"). >> >> On May 24, 2018, at 12:24 AM, Michael Becker wrote: >> >> I noticed that for every individual KSP iteration, six vector objects are created and destroyed (with CG, more with e.g. GMRES). >> >> Hmm, it is certainly not intended at vectors be created and destroyed within each KSPSolve() could you please point us to the code that makes you think they are being created and destroyed? We create all the work vectors at KSPSetUp() and destroy them in KSPReset() not during the solve. Not that this would be a measurable distance. >> >> >> I mean this, right in the log_view output: >> >> Memory usage is given in bytes: >> >> Object Type Creations Destructions Memory Descendants' Mem. >> Reports information only for process 0. >> >> --- Event Stage 0: Main Stage >> >> ... >> >> --- Event Stage 1: First Solve >> >> ... >> >> --- Event Stage 2: Remaining Solves >> >> Vector 23904 23904 1295501184 0. >> >> I logged the exact number of KSP iterations over the 999 timesteps and >> its exactly 23904/6 = 3984. >> >> Michael >> >> >> >> Am 24.05.2018 um 19:50 schrieb Smith, Barry F.: >> >> Please send the log file for 1000 with cg as the solver. >> >> You should make a bar chart of each event for the two cases to see which ones are taking more time and which are taking less (we cannot tell with the two logs you sent us since they are for different solvers.) >> >> >> >> >> On May 24, 2018, at 12:24 AM, Michael Becker wrote: >> >> I noticed that for every individual KSP iteration, six vector objects are created and destroyed (with CG, more with e.g. GMRES). >> >> Hmm, it is certainly not intended at vectors be created and destroyed within each KSPSolve() could you please point us to the code that makes you think they are being created and destroyed? We create all the work vectors at KSPSetUp() and destroy them in KSPReset() not during the solve. Not that this would be a measurable distance. >> >> >> >> >> This seems kind of wasteful, is this supposed to be like this? Is this even the reason for my problems? Apart from that, everything seems quite normal to me (but I'm not the expert here). >> >> >> Thanks in advance. >> >> Michael >> >> >> >> >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mbuerkle at web.de Fri Jun 1 23:14:00 2018 From: mbuerkle at web.de (Marius Buerkle) Date: Sat, 2 Jun 2018 06:14:00 +0200 Subject: [petsc-users] MAT_NEW_NONZERO_LOCATIONS working? In-Reply-To: References: <1153922A-8876-492B-8AD9-F41DD718FD21@anl.gov> <58D5E959-0E3C-4EF2-8270-882D1EA6EA38@mcs.anl.gov> Message-ID: An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Sat Jun 2 06:40:00 2018 From: mfadams at lbl.gov (Mark Adams) Date: Sat, 2 Jun 2018 07:40:00 -0400 Subject: [petsc-users] Poor weak scaling when solvingsuccessivelinearsystems In-Reply-To: References: <6d9b6488-3957-0d74-148a-323c78d42f6a@physik.uni-giessen.de> <079D8B4D-C57B-4054-B9BD-05644E52C3A4@anl.gov> Message-ID: On Fri, Jun 1, 2018 at 11:20 PM, Junchao Zhang wrote: > Hi,Michael, > You can add -log_sync besides -log_view, which adds barriers to certain > events but measures barrier time separately from the events. I find this > option makes it easier to interpret log_view output. > That is great (good to know). This should give us a better idea if your large VecScatter costs are from slow communication or if it catching some sort of load imbalance. > > --Junchao Zhang > > On Wed, May 30, 2018 at 3:27 AM, Michael Becker < > Michael.Becker at physik.uni-giessen.de> wrote: > >> Barry: On its way. Could take a couple days again. >> >> Junchao: I unfortunately don't have access to a cluster with a faster >> network. This one has a mixed 4X QDR-FDR InfiniBand 2:1 blocking fat-tree >> network, which I realize causes parallel slowdown if the nodes are not >> connected to the same switch. Each node has 24 processors (2x12/socket) and >> four NUMA domains (two for each socket). >> The ranks are usually not distributed perfectly even, i.e. for 125 >> processes, of the six required nodes, five would use 21 cores and one 20. >> Would using another CPU type make a difference communication-wise? I >> could switch to faster ones (on the same network), but I always assumed >> this would only improve performance of the stuff that is unrelated to >> communication. >> >> Michael >> >> >> >> The log files have something like "Average time for zero size MPI_Send(): >> 1.84231e-05". It looks you ran on a cluster with a very slow network. A >> typical machine should give less than 1/10 of the latency you have. An easy >> way to try is just running the code on a machine with a faster network and >> see what happens. >> >> Also, how many cores & numa domains does a compute node have? I could not >> figure out how you distributed the 125 MPI ranks evenly. >> >> --Junchao Zhang >> >> On Tue, May 29, 2018 at 6:18 AM, Michael Becker < >> Michael.Becker at physik.uni-giessen.de> wrote: >> >>> Hello again, >>> >>> here are the updated log_view files for 125 and 1000 processors. I ran >>> both problems twice, the first time with all processors per node allocated >>> ("-1.txt"), the second with only half on twice the number of nodes >>> ("-2.txt"). >>> >>> On May 24, 2018, at 12:24 AM, Michael Becker wrote: >>> >>> I noticed that for every individual KSP iteration, six vector objects are created and destroyed (with CG, more with e.g. GMRES). >>> >>> Hmm, it is certainly not intended at vectors be created and destroyed within each KSPSolve() could you please point us to the code that makes you think they are being created and destroyed? We create all the work vectors at KSPSetUp() and destroy them in KSPReset() not during the solve. Not that this would be a measurable distance. >>> >>> >>> I mean this, right in the log_view output: >>> >>> Memory usage is given in bytes: >>> >>> Object Type Creations Destructions Memory Descendants' Mem. >>> Reports information only for process 0. >>> >>> --- Event Stage 0: Main Stage >>> >>> ... >>> >>> --- Event Stage 1: First Solve >>> >>> ... >>> >>> --- Event Stage 2: Remaining Solves >>> >>> Vector 23904 23904 1295501184 0. >>> >>> I logged the exact number of KSP iterations over the 999 timesteps and >>> its exactly 23904/6 = 3984. >>> >>> Michael >>> >>> >>> >>> Am 24.05.2018 um 19:50 schrieb Smith, Barry F.: >>> >>> Please send the log file for 1000 with cg as the solver. >>> >>> You should make a bar chart of each event for the two cases to see which ones are taking more time and which are taking less (we cannot tell with the two logs you sent us since they are for different solvers.) >>> >>> >>> >>> >>> On May 24, 2018, at 12:24 AM, Michael Becker wrote: >>> >>> I noticed that for every individual KSP iteration, six vector objects are created and destroyed (with CG, more with e.g. GMRES). >>> >>> Hmm, it is certainly not intended at vectors be created and destroyed within each KSPSolve() could you please point us to the code that makes you think they are being created and destroyed? We create all the work vectors at KSPSetUp() and destroy them in KSPReset() not during the solve. Not that this would be a measurable distance. >>> >>> >>> >>> >>> This seems kind of wasteful, is this supposed to be like this? Is this even the reason for my problems? Apart from that, everything seems quite normal to me (but I'm not the expert here). >>> >>> >>> Thanks in advance. >>> >>> Michael >>> >>> >>> >>> >>> >>> >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From danyang.su at gmail.com Sat Jun 2 13:36:06 2018 From: danyang.su at gmail.com (Danyang Su) Date: Sat, 2 Jun 2018 11:36:06 -0700 Subject: [petsc-users] Makefile for mixed C++ and Fortran code In-Reply-To: <0335466E-9658-4FC5-A6FA-0886F57DE5BF@anl.gov> References: <164761f9-80cc-042e-6a4f-558dc90992b7@gmail.com> <0335466E-9658-4FC5-A6FA-0886F57DE5BF@anl.gov> Message-ID: <903af1d8-2b4a-3f1b-173b-ed96e0a1b4b3@gmail.com> Hi Barry, For the code without PETSc, the rules used to compile the code with CGAL is DLIB = -lstdc++ -lmetis -lm -L/usr/local/lib -rdynamic /usr/local/lib/libmpfr.so /usr/local/lib/libgmp.so /usr/local/lib/libCGAL_ImageIO.so.11.0.1 /usr/local/lib/libCGAL.so.11.0.1 /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_system.so -lpthread -lGLU -lGL -lX11 -lXext -lz /usr/local/lib/libCGAL_ImageIO.so.11.0.1 /usr/local/lib/libCGAL.so.11.0.1 /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_system.so -lpthread -lGLU -lGL -lX11 -lXext -lz /usr/local/lib/libmpfr.so /usr/local/lib/libgmp.so /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_system.so -lpthread -Wl,-rpath,/usr/local/lib FFLAGS = -O3 -I$(LIS_INC) CXXFLAGS = -std=c++11 -O3 -I$(LIS_INC) However, after adding these to the makefile using PETSc, I got error telling me that all the petsc include files cannot be found. ../../solver/solver_snes_common.F90:27:0: fatal error: petscversion.h: No such \ file or directory #include ../../solver/solver_snes_common.F90:27:0: fatal error: petscversion.h: No such \ file or directory #include Similar for other head files. However, If I change the file path to the the full path, the code cannot compiled. Does the rule I use destroy the PETSc relative path/ The make commands I use is executable: $(SOURCES) chkopts -${FLINKER} $(FFLAGS) $(FPPFLAGS) $(CPPFLAGS) -o executable.out $(SOURCES) ${PETSC_LIB} ${LIS_LIB} ${DLIB} %.o:%.F90 $(FLINKER) $(FFLAGS) $(FPPFLAGS) -c -frounding-math $< -o $@ %.o:%.cpp $(CLINKER) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ Thanks, Danyang On 18-06-01 10:41 AM, Smith, Barry F. wrote: > You need to determine exactly what flags are passed to the C++ compiler for your compile that works and make sure those same flags are used in "PETSc version" of the makefile. You could add the flags directly to the rule > >> %.o:%.cpp >> $(CLINKER) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ > Barry > > >> On Jun 1, 2018, at 12:37 PM, Danyang Su wrote: >> >> Follow up: >> >> With following command >> >> executable: $(SOURCES) chkopts >> -${FLINKER} $(FFLAGS) $(FPPFLAGS) $(CPPFLAGS) -o executable.out $(SOURCES) ${PETSC_LIB} >> >> %.o:%.F90 >> $(FLINKER) $(FFLAGS) $(FPPFLAGS) -c -frounding-math $< -o $@ >> %.o:%.cpp >> $(CLINKER) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ >> >> The compiler return error: no match function. >> >> ../../usg/cgal_triangulation_2d.cpp: In function ?void outputTriangulation2d(in\ >> t, const char*, int, const char*)?: >> ../../usg/cgal_triangulation_2d.cpp:485:20: error: no matching function for cal\ >> l to ?std::basic_ofstream::open(std::string&)? >> out.open(strfile); >> >> Thanks, >> >> >> Danyang >> >> On 18-06-01 10:07 AM, Danyang Su wrote: >>> Hi All, >>> >>> My code needs to link to an external C++ library (CGAL). The code is written in Fortran and I have already written interface to let Fortran call C++ function. For the sequential version without PETSc, it can be compiled without problem using the following makefile. The parallel version without CGAL can also be compiled successfully. However, when I tried to use PETSc together with CGAL library, I cannot compile the code. My questions is: How can I modify the makefile? Do I need to reconfigure PETSc with special flags? All the makefile samples are shown below. >>> >>> #makefile for sequential version >>> >>> FC = gfortran >>> #FC = ifort >>> CXX = g++ -std=c++11 >>> >>> DLIB = -lstdc++ -lm -L/usr/local/lib -rdynamic /usr/local/lib/libmpfr.so /usr/local/lib/libgmp.so /usr/local/lib/libCGAL_ImageIO.so.11.0.1 /usr/local/lib/libCGAL.so.11.0.1 /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_system.so -lpthread -lGLU -lGL -lX11 -lXext -lz /usr/local/lib/libCGAL_ImageIO.so.11.0.1 /usr/local/lib/libCGAL.so.11.0.1 /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_system.so -lpthread -lGLU -lGL -lX11 -lXext -lz /usr/local/lib/libmpfr.so /usr/local/lib/libgmp.so /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_system.so -lpthread -Wl,-rpath,/usr/local/lib >>> >>> FFLAGS = -O3 >>> CXXFLAGS = -O3 >>> >>> FPPFLAGS = -DUSECGAL >>> >>> SRC =./../../ >>> >>> SOURCES = $(SRC)usg/math_common.o\ >>> $(SRC)usg/geometry_definition.o\ >>> $(SRC)usg/cgal_common.o\ >>> >>> ... >>> >>> executable: $(SOURCES) >>> $(FC) $(FFLAGS) $(FPPFLAGS) -o executable.out $(SOURCES) ${LIS_LIB} $(DLIB) >>> %.o:%.F90 >>> $(FC) $(FFLAGS) $(FPPFLAGS) -c -frounding-math $< -o $@ >>> %.o:%.cpp >>> $(CXX) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ >>> >>> >>> #makefile for parallel version with PETSc, without CGAL >>> >>> #FC = ifort >>> #FC = gfortran >>> >>> DLIB = -lm >>> >>> FFLAGS = -O3 >>> >>> FPPFLAGS = -DUSEPETSC >>> >>> SRC =./../../ >>> >>> SOURCES = $(SRC)usg/math_common.o\ >>> $(SRC)usg/geometry_definition.o\ >>> $(SRC)usg/cgal_common.o\ >>> >>> ... >>> >>> executable: $(SOURCES) chkopts >>> -${FLINKER} $(FFLAGS) $(FPPFLAGS) $(CPPFLAGS) -o executable.out $(SOURCES) ${PETSC_LIB} >>> >>> >>> #makefile for parallel version with PETSc, with CGAL, CANNOT work >>> >>> #FC = ifort >>> #FC = gfortran >>> >>> DLIB = -lm >>> >>> FFLAGS = -O3 >>> >>> FPPFLAGS = -DUSEPETSC -DUSECGAL >>> >>> SRC =./../../ >>> >>> SOURCES = $(SRC)usg/math_common.o\ >>> $(SRC)usg/geometry_definition.o\ >>> $(SRC)usg/cgal_common.o\ >>> >>> ... >>> >>> executable: $(SOURCES) chkopts >>> -${FLINKER} $(FFLAGS) $(FPPFLAGS) $(CPPFLAGS) -o executable.out $(SOURCES) ${PETSC_LIB} >>> >>> %.o:%.F90 >>> $(FC) $(FFLAGS) $(FPPFLAGS) -c -frounding-math $< -o $@ >>> %.o:%.cpp >>> $(CXX) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ >>> >>> >>> Thanks, >>> >>> Danyang >>> From balay at mcs.anl.gov Sat Jun 2 16:51:15 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Sat, 2 Jun 2018 16:51:15 -0500 Subject: [petsc-users] Makefile for mixed C++ and Fortran code In-Reply-To: <903af1d8-2b4a-3f1b-173b-ed96e0a1b4b3@gmail.com> References: <164761f9-80cc-042e-6a4f-558dc90992b7@gmail.com> <0335466E-9658-4FC5-A6FA-0886F57DE5BF@anl.gov> <903af1d8-2b4a-3f1b-173b-ed96e0a1b4b3@gmail.com> Message-ID: Try the attached makefile. [with correct PETSC_DIR and PETSC_ARCH values] If you have issues - send the complete makefiles - and complete error log.. On Sat, 2 Jun 2018, Danyang Su wrote: > Hi Barry, > > For the code without PETSc, the rules used to compile the code with CGAL is note: DLIB can probably be simlified - and a fewof the options eliminated. > DLIB = -lstdc++ -lmetis -lm -L/usr/local/lib -rdynamic -lstdc++ is setup by petsc -lmetis can be a dependency of petsc - so care should be taken to have only one copy of metis > /usr/local/lib/libmpfr.so /usr/local/lib/libgmp.so Why are these here? Normally these are dependencies of gcc/gfortran [and PETSc configure picks up the correct ones] > /usr/local/lib/libCGAL_ImageIO.so.11.0.1 /usr/local/lib/libCGAL.so.11.0.1 I have no idea what these are.. Satish > /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_system.so -lpthread > -lGLU -lGL -lX11 -lXext -lz /usr/local/lib/libCGAL_ImageIO.so.11.0.1 > /usr/local/lib/libCGAL.so.11.0.1 /usr/local/lib/libboost_thread.so > /usr/local/lib/libboost_system.so -lpthread -lGLU -lGL -lX11 -lXext -lz > /usr/local/lib/libmpfr.so /usr/local/lib/libgmp.so > /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_system.so -lpthread > -Wl,-rpath,/usr/local/lib > > FFLAGS = -O3 -I$(LIS_INC) > CXXFLAGS = -std=c++11 -O3 -I$(LIS_INC) > > However, after adding these to the makefile using PETSc, I got error telling > me that all the petsc include files cannot be found. > > ../../solver/solver_snes_common.F90:27:0: fatal error: petscversion.h: No such > \ > file or directory > #include > > ../../solver/solver_snes_common.F90:27:0: fatal error: petscversion.h: No such > \ > file or directory > #include > > Similar for other head files. However, If I change the file path to the the > full path, the code cannot compiled. Does the rule I use destroy the PETSc > relative path/ > > The make commands I use is > > executable: $(SOURCES) chkopts > -${FLINKER} $(FFLAGS) $(FPPFLAGS) $(CPPFLAGS) -o executable.out > $(SOURCES) ${PETSC_LIB} ${LIS_LIB} ${DLIB} > %.o:%.F90 > $(FLINKER) $(FFLAGS) $(FPPFLAGS) -c -frounding-math $< -o $@ > %.o:%.cpp > $(CLINKER) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ > > Thanks, > > Danyang > > On 18-06-01 10:41 AM, Smith, Barry F. wrote: > > You need to determine exactly what flags are passed to the C++ compiler > > for your compile that works and make sure those same flags are used in > > "PETSc version" of the makefile. You could add the flags directly to the > > rule > > > >> %.o:%.cpp > >> $(CLINKER) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ > > Barry > > > > > >> On Jun 1, 2018, at 12:37 PM, Danyang Su wrote: > >> > >> Follow up: > >> > >> With following command > >> > >> executable: $(SOURCES) chkopts > >> -${FLINKER} $(FFLAGS) $(FPPFLAGS) $(CPPFLAGS) -o executable.out > >> $(SOURCES) ${PETSC_LIB} > >> > >> %.o:%.F90 > >> $(FLINKER) $(FFLAGS) $(FPPFLAGS) -c -frounding-math $< -o $@ > >> %.o:%.cpp > >> $(CLINKER) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ > >> > >> The compiler return error: no match function. > >> > >> ../../usg/cgal_triangulation_2d.cpp: In function ?void > >> outputTriangulation2d(in\ > >> t, const char*, int, const char*)?: > >> ../../usg/cgal_triangulation_2d.cpp:485:20: error: no matching function for > >> cal\ > >> l to ?std::basic_ofstream::open(std::string&)? > >> out.open(strfile); > >> > >> Thanks, > >> > >> > >> Danyang > >> > >> On 18-06-01 10:07 AM, Danyang Su wrote: > >>> Hi All, > >>> > >>> My code needs to link to an external C++ library (CGAL). The code is > >>> written in Fortran and I have already written interface to let Fortran > >>> call C++ function. For the sequential version without PETSc, it can be > >>> compiled without problem using the following makefile. The parallel > >>> version without CGAL can also be compiled successfully. However, when I > >>> tried to use PETSc together with CGAL library, I cannot compile the code. > >>> My questions is: How can I modify the makefile? Do I need to reconfigure > >>> PETSc with special flags? All the makefile samples are shown below. > >>> > >>> #makefile for sequential version > >>> > >>> FC = gfortran > >>> #FC = ifort > >>> CXX = g++ -std=c++11 > >>> > >>> DLIB = -lstdc++ -lm -L/usr/local/lib -rdynamic /usr/local/lib/libmpfr.so > >>> /usr/local/lib/libgmp.so /usr/local/lib/libCGAL_ImageIO.so.11.0.1 > >>> /usr/local/lib/libCGAL.so.11.0.1 /usr/local/lib/libboost_thread.so > >>> /usr/local/lib/libboost_system.so -lpthread -lGLU -lGL -lX11 -lXext -lz > >>> /usr/local/lib/libCGAL_ImageIO.so.11.0.1 /usr/local/lib/libCGAL.so.11.0.1 > >>> /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_system.so > >>> -lpthread -lGLU -lGL -lX11 -lXext -lz /usr/local/lib/libmpfr.so > >>> /usr/local/lib/libgmp.so /usr/local/lib/libboost_thread.so > >>> /usr/local/lib/libboost_system.so -lpthread -Wl,-rpath,/usr/local/lib > >>> > >>> FFLAGS = -O3 > >>> CXXFLAGS = -O3 > >>> > >>> FPPFLAGS = -DUSECGAL > >>> > >>> SRC =./../../ > >>> > >>> SOURCES = $(SRC)usg/math_common.o\ > >>> $(SRC)usg/geometry_definition.o\ > >>> $(SRC)usg/cgal_common.o\ > >>> > >>> ... > >>> > >>> executable: $(SOURCES) > >>> $(FC) $(FFLAGS) $(FPPFLAGS) -o executable.out $(SOURCES) ${LIS_LIB} > >>> $(DLIB) > >>> %.o:%.F90 > >>> $(FC) $(FFLAGS) $(FPPFLAGS) -c -frounding-math $< -o $@ > >>> %.o:%.cpp > >>> $(CXX) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ > >>> > >>> > >>> #makefile for parallel version with PETSc, without CGAL > >>> > >>> #FC = ifort > >>> #FC = gfortran > >>> > >>> DLIB = -lm > >>> > >>> FFLAGS = -O3 > >>> > >>> FPPFLAGS = -DUSEPETSC > >>> > >>> SRC =./../../ > >>> > >>> SOURCES = $(SRC)usg/math_common.o\ > >>> $(SRC)usg/geometry_definition.o\ > >>> $(SRC)usg/cgal_common.o\ > >>> > >>> ... > >>> > >>> executable: $(SOURCES) chkopts > >>> -${FLINKER} $(FFLAGS) $(FPPFLAGS) $(CPPFLAGS) -o executable.out > >>> $(SOURCES) ${PETSC_LIB} > >>> > >>> > >>> #makefile for parallel version with PETSc, with CGAL, CANNOT work > >>> > >>> #FC = ifort > >>> #FC = gfortran > >>> > >>> DLIB = -lm > >>> > >>> FFLAGS = -O3 > >>> > >>> FPPFLAGS = -DUSEPETSC -DUSECGAL > >>> > >>> SRC =./../../ > >>> > >>> SOURCES = $(SRC)usg/math_common.o\ > >>> $(SRC)usg/geometry_definition.o\ > >>> $(SRC)usg/cgal_common.o\ > >>> > >>> ... > >>> > >>> executable: $(SOURCES) chkopts > >>> -${FLINKER} $(FFLAGS) $(FPPFLAGS) $(CPPFLAGS) -o executable.out > >>> $(SOURCES) ${PETSC_LIB} > >>> > >>> %.o:%.F90 > >>> $(FC) $(FFLAGS) $(FPPFLAGS) -c -frounding-math $< -o $@ > >>> %.o:%.cpp > >>> $(CXX) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ > >>> > >>> > >>> Thanks, > >>> > >>> Danyang > >>> > > -------------- next part -------------- CFLAGS = FFLAGS = -frounding-math CPPFLAGS = FPPFLAGS = -DUSECGAL CXXFLAGS = -std=c++11 CLEANFILES = executable.out include ${PETSC_DIR}/lib/petsc/conf/variables include ${PETSC_DIR}/lib/petsc/conf/rules SRC =./../../ OBJS = $(SRC)usg/math_common.o\ $(SRC)usg/geometry_definition.o\ $(SRC)usg/cgal_common.o\ DLIB = -lstdc++ -lm -L/usr/local/lib -rdynamic /usr/local/lib/libmpfr.so /usr/local/lib/libgmp.so /usr/local/lib/libCGAL_ImageIO.so.11.0.1 \ /usr/local/lib/libCGAL.so.11.0.1 /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_system.so -lpthread -lGLU -lGL -lX11 -lXext -lz \ /usr/local/lib/libCGAL_ImageIO.so.11.0.1 /usr/local/lib/libCGAL.so.11.0.1 /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_system.so \ -lpthread -lGLU -lGL -lX11 -lXext -lz /usr/local/lib/libmpfr.so /usr/local/lib/libgmp.so /usr/local/lib/libboost_thread.so \ /usr/local/lib/libboost_system.so -lpthread -Wl,-rpath,/usr/local/lib executable: $(OBJS) -$(FLINKER) -o executable.out $(OBJS) $(PETSC_LIB) $(DLIB) From danyang.su at gmail.com Sun Jun 3 01:14:44 2018 From: danyang.su at gmail.com (Danyang Su) Date: Sat, 2 Jun 2018 23:14:44 -0700 Subject: [petsc-users] Makefile for mixed C++ and Fortran code In-Reply-To: References: <164761f9-80cc-042e-6a4f-558dc90992b7@gmail.com> <0335466E-9658-4FC5-A6FA-0886F57DE5BF@anl.gov> <903af1d8-2b4a-3f1b-173b-ed96e0a1b4b3@gmail.com> Message-ID: <77919213-e915-ba38-715d-4b7697674d23@gmail.com> Hi Satish, Your makefile works. Please ignore the CGAL dependency. I just modified from CGAL example and it needs to be further optimized with the dependency that is required for the algorithm I use. Thanks, Danyang On 18-06-02 02:51 PM, Satish Balay wrote: > Try the attached makefile. [with correct PETSC_DIR and PETSC_ARCH values] > > If you have issues - send the complete makefiles - and complete error log.. > > > On Sat, 2 Jun 2018, Danyang Su wrote: > >> Hi Barry, >> >> For the code without PETSc, the rules used to compile the code with CGAL is > note: DLIB can probably be simlified - and a fewof the options eliminated. > > >> DLIB = -lstdc++ -lmetis -lm -L/usr/local/lib -rdynamic > -lstdc++ is setup by petsc > > -lmetis can be a dependency of petsc - so care should be taken to have only one copy of metis > >> /usr/local/lib/libmpfr.so /usr/local/lib/libgmp.so > Why are these here? Normally these are dependencies of gcc/gfortran [and PETSc configure picks up the correct ones] > >> /usr/local/lib/libCGAL_ImageIO.so.11.0.1 /usr/local/lib/libCGAL.so.11.0.1 > I have no idea what these are.. > > Satish > >> /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_system.so -lpthread >> -lGLU -lGL -lX11 -lXext -lz /usr/local/lib/libCGAL_ImageIO.so.11.0.1 >> /usr/local/lib/libCGAL.so.11.0.1 /usr/local/lib/libboost_thread.so >> /usr/local/lib/libboost_system.so -lpthread -lGLU -lGL -lX11 -lXext -lz >> /usr/local/lib/libmpfr.so /usr/local/lib/libgmp.so >> /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_system.so -lpthread >> -Wl,-rpath,/usr/local/lib >> >> FFLAGS = -O3 -I$(LIS_INC) >> CXXFLAGS = -std=c++11 -O3 -I$(LIS_INC) >> >> However, after adding these to the makefile using PETSc, I got error telling >> me that all the petsc include files cannot be found. >> >> ../../solver/solver_snes_common.F90:27:0: fatal error: petscversion.h: No such >> \ >> file or directory >> #include >> >> ../../solver/solver_snes_common.F90:27:0: fatal error: petscversion.h: No such >> \ >> file or directory >> #include >> >> Similar for other head files. However, If I change the file path to the the >> full path, the code cannot compiled. Does the rule I use destroy the PETSc >> relative path/ >> >> The make commands I use is >> >> executable: $(SOURCES) chkopts >> -${FLINKER} $(FFLAGS) $(FPPFLAGS) $(CPPFLAGS) -o executable.out >> $(SOURCES) ${PETSC_LIB} ${LIS_LIB} ${DLIB} >> %.o:%.F90 >> $(FLINKER) $(FFLAGS) $(FPPFLAGS) -c -frounding-math $< -o $@ >> %.o:%.cpp >> $(CLINKER) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ >> >> Thanks, >> >> Danyang >> >> On 18-06-01 10:41 AM, Smith, Barry F. wrote: >>> You need to determine exactly what flags are passed to the C++ compiler >>> for your compile that works and make sure those same flags are used in >>> "PETSc version" of the makefile. You could add the flags directly to the >>> rule >>> >>>> %.o:%.cpp >>>> $(CLINKER) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ >>> Barry >>> >>> >>>> On Jun 1, 2018, at 12:37 PM, Danyang Su wrote: >>>> >>>> Follow up: >>>> >>>> With following command >>>> >>>> executable: $(SOURCES) chkopts >>>> -${FLINKER} $(FFLAGS) $(FPPFLAGS) $(CPPFLAGS) -o executable.out >>>> $(SOURCES) ${PETSC_LIB} >>>> >>>> %.o:%.F90 >>>> $(FLINKER) $(FFLAGS) $(FPPFLAGS) -c -frounding-math $< -o $@ >>>> %.o:%.cpp >>>> $(CLINKER) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ >>>> >>>> The compiler return error: no match function. >>>> >>>> ../../usg/cgal_triangulation_2d.cpp: In function ?void >>>> outputTriangulation2d(in\ >>>> t, const char*, int, const char*)?: >>>> ../../usg/cgal_triangulation_2d.cpp:485:20: error: no matching function for >>>> cal\ >>>> l to ?std::basic_ofstream::open(std::string&)? >>>> out.open(strfile); >>>> >>>> Thanks, >>>> >>>> >>>> Danyang >>>> >>>> On 18-06-01 10:07 AM, Danyang Su wrote: >>>>> Hi All, >>>>> >>>>> My code needs to link to an external C++ library (CGAL). The code is >>>>> written in Fortran and I have already written interface to let Fortran >>>>> call C++ function. For the sequential version without PETSc, it can be >>>>> compiled without problem using the following makefile. The parallel >>>>> version without CGAL can also be compiled successfully. However, when I >>>>> tried to use PETSc together with CGAL library, I cannot compile the code. >>>>> My questions is: How can I modify the makefile? Do I need to reconfigure >>>>> PETSc with special flags? All the makefile samples are shown below. >>>>> >>>>> #makefile for sequential version >>>>> >>>>> FC = gfortran >>>>> #FC = ifort >>>>> CXX = g++ -std=c++11 >>>>> >>>>> DLIB = -lstdc++ -lm -L/usr/local/lib -rdynamic /usr/local/lib/libmpfr.so >>>>> /usr/local/lib/libgmp.so /usr/local/lib/libCGAL_ImageIO.so.11.0.1 >>>>> /usr/local/lib/libCGAL.so.11.0.1 /usr/local/lib/libboost_thread.so >>>>> /usr/local/lib/libboost_system.so -lpthread -lGLU -lGL -lX11 -lXext -lz >>>>> /usr/local/lib/libCGAL_ImageIO.so.11.0.1 /usr/local/lib/libCGAL.so.11.0.1 >>>>> /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_system.so >>>>> -lpthread -lGLU -lGL -lX11 -lXext -lz /usr/local/lib/libmpfr.so >>>>> /usr/local/lib/libgmp.so /usr/local/lib/libboost_thread.so >>>>> /usr/local/lib/libboost_system.so -lpthread -Wl,-rpath,/usr/local/lib >>>>> >>>>> FFLAGS = -O3 >>>>> CXXFLAGS = -O3 >>>>> >>>>> FPPFLAGS = -DUSECGAL >>>>> >>>>> SRC =./../../ >>>>> >>>>> SOURCES = $(SRC)usg/math_common.o\ >>>>> $(SRC)usg/geometry_definition.o\ >>>>> $(SRC)usg/cgal_common.o\ >>>>> >>>>> ... >>>>> >>>>> executable: $(SOURCES) >>>>> $(FC) $(FFLAGS) $(FPPFLAGS) -o executable.out $(SOURCES) ${LIS_LIB} >>>>> $(DLIB) >>>>> %.o:%.F90 >>>>> $(FC) $(FFLAGS) $(FPPFLAGS) -c -frounding-math $< -o $@ >>>>> %.o:%.cpp >>>>> $(CXX) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ >>>>> >>>>> >>>>> #makefile for parallel version with PETSc, without CGAL >>>>> >>>>> #FC = ifort >>>>> #FC = gfortran >>>>> >>>>> DLIB = -lm >>>>> >>>>> FFLAGS = -O3 >>>>> >>>>> FPPFLAGS = -DUSEPETSC >>>>> >>>>> SRC =./../../ >>>>> >>>>> SOURCES = $(SRC)usg/math_common.o\ >>>>> $(SRC)usg/geometry_definition.o\ >>>>> $(SRC)usg/cgal_common.o\ >>>>> >>>>> ... >>>>> >>>>> executable: $(SOURCES) chkopts >>>>> -${FLINKER} $(FFLAGS) $(FPPFLAGS) $(CPPFLAGS) -o executable.out >>>>> $(SOURCES) ${PETSC_LIB} >>>>> >>>>> >>>>> #makefile for parallel version with PETSc, with CGAL, CANNOT work >>>>> >>>>> #FC = ifort >>>>> #FC = gfortran >>>>> >>>>> DLIB = -lm >>>>> >>>>> FFLAGS = -O3 >>>>> >>>>> FPPFLAGS = -DUSEPETSC -DUSECGAL >>>>> >>>>> SRC =./../../ >>>>> >>>>> SOURCES = $(SRC)usg/math_common.o\ >>>>> $(SRC)usg/geometry_definition.o\ >>>>> $(SRC)usg/cgal_common.o\ >>>>> >>>>> ... >>>>> >>>>> executable: $(SOURCES) chkopts >>>>> -${FLINKER} $(FFLAGS) $(FPPFLAGS) $(CPPFLAGS) -o executable.out >>>>> $(SOURCES) ${PETSC_LIB} >>>>> >>>>> %.o:%.F90 >>>>> $(FC) $(FFLAGS) $(FPPFLAGS) -c -frounding-math $< -o $@ >>>>> %.o:%.cpp >>>>> $(CXX) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Danyang >>>>> >> From balay at mcs.anl.gov Sun Jun 3 09:55:47 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Sun, 3 Jun 2018 09:55:47 -0500 Subject: [petsc-users] Makefile for mixed C++ and Fortran code In-Reply-To: <77919213-e915-ba38-715d-4b7697674d23@gmail.com> References: <164761f9-80cc-042e-6a4f-558dc90992b7@gmail.com> <0335466E-9658-4FC5-A6FA-0886F57DE5BF@anl.gov> <903af1d8-2b4a-3f1b-173b-ed96e0a1b4b3@gmail.com> <77919213-e915-ba38-715d-4b7697674d23@gmail.com> Message-ID: Glad it works. Thanks for the update. Satish On Sun, 3 Jun 2018, Danyang Su wrote: > Hi Satish, > > Your makefile works. > > Please ignore the CGAL dependency. I just modified from CGAL example and it > needs to be further optimized with the dependency that is required for the > algorithm I use. > > Thanks, > > Danyang > > > On 18-06-02 02:51 PM, Satish Balay wrote: > > Try the attached makefile. [with correct PETSC_DIR and PETSC_ARCH values] > > > > If you have issues - send the complete makefiles - and complete error log.. > > > > > > On Sat, 2 Jun 2018, Danyang Su wrote: > > > >> Hi Barry, > >> > >> For the code without PETSc, the rules used to compile the code with CGAL is > > note: DLIB can probably be simlified - and a fewof the options eliminated. > > > > > >> DLIB = -lstdc++ -lmetis -lm -L/usr/local/lib -rdynamic > > -lstdc++ is setup by petsc > > > > -lmetis can be a dependency of petsc - so care should be taken to have only > > one copy of metis > > > >> /usr/local/lib/libmpfr.so /usr/local/lib/libgmp.so > > Why are these here? Normally these are dependencies of gcc/gfortran [and > > PETSc configure picks up the correct ones] > > > >> /usr/local/lib/libCGAL_ImageIO.so.11.0.1 /usr/local/lib/libCGAL.so.11.0.1 > > I have no idea what these are.. > > > > Satish > > > >> /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_system.so > >> -lpthread > >> -lGLU -lGL -lX11 -lXext -lz /usr/local/lib/libCGAL_ImageIO.so.11.0.1 > >> /usr/local/lib/libCGAL.so.11.0.1 /usr/local/lib/libboost_thread.so > >> /usr/local/lib/libboost_system.so -lpthread -lGLU -lGL -lX11 -lXext -lz > >> /usr/local/lib/libmpfr.so /usr/local/lib/libgmp.so > >> /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_system.so > >> -lpthread > >> -Wl,-rpath,/usr/local/lib > >> > >> FFLAGS = -O3 -I$(LIS_INC) > >> CXXFLAGS = -std=c++11 -O3 -I$(LIS_INC) > >> > >> However, after adding these to the makefile using PETSc, I got error > >> telling > >> me that all the petsc include files cannot be found. > >> > >> ../../solver/solver_snes_common.F90:27:0: fatal error: petscversion.h: No > >> such > >> \ > >> file or directory > >> #include > >> > >> ../../solver/solver_snes_common.F90:27:0: fatal error: petscversion.h: No > >> such > >> \ > >> file or directory > >> #include > >> > >> Similar for other head files. However, If I change the file path to the the > >> full path, the code cannot compiled. Does the rule I use destroy the PETSc > >> relative path/ > >> > >> The make commands I use is > >> > >> executable: $(SOURCES) chkopts > >> -${FLINKER} $(FFLAGS) $(FPPFLAGS) $(CPPFLAGS) -o executable.out > >> $(SOURCES) ${PETSC_LIB} ${LIS_LIB} ${DLIB} > >> %.o:%.F90 > >> $(FLINKER) $(FFLAGS) $(FPPFLAGS) -c -frounding-math $< -o $@ > >> %.o:%.cpp > >> $(CLINKER) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ > >> > >> Thanks, > >> > >> Danyang > >> > >> On 18-06-01 10:41 AM, Smith, Barry F. wrote: > >>> You need to determine exactly what flags are passed to the C++ > >>> compiler > >>> for your compile that works and make sure those same flags are used > >>> in > >>> "PETSc version" of the makefile. You could add the flags directly to > >>> the > >>> rule > >>> > >>>> %.o:%.cpp > >>>> $(CLINKER) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ > >>> Barry > >>> > >>> > >>>> On Jun 1, 2018, at 12:37 PM, Danyang Su wrote: > >>>> > >>>> Follow up: > >>>> > >>>> With following command > >>>> > >>>> executable: $(SOURCES) chkopts > >>>> -${FLINKER} $(FFLAGS) $(FPPFLAGS) $(CPPFLAGS) -o executable.out > >>>> $(SOURCES) ${PETSC_LIB} > >>>> > >>>> %.o:%.F90 > >>>> $(FLINKER) $(FFLAGS) $(FPPFLAGS) -c -frounding-math $< -o $@ > >>>> %.o:%.cpp > >>>> $(CLINKER) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ > >>>> > >>>> The compiler return error: no match function. > >>>> > >>>> ../../usg/cgal_triangulation_2d.cpp: In function ?void > >>>> outputTriangulation2d(in\ > >>>> t, const char*, int, const char*)?: > >>>> ../../usg/cgal_triangulation_2d.cpp:485:20: error: no matching function > >>>> for > >>>> cal\ > >>>> l to ?std::basic_ofstream::open(std::string&)? > >>>> out.open(strfile); > >>>> > >>>> Thanks, > >>>> > >>>> > >>>> Danyang > >>>> > >>>> On 18-06-01 10:07 AM, Danyang Su wrote: > >>>>> Hi All, > >>>>> > >>>>> My code needs to link to an external C++ library (CGAL). The code is > >>>>> written in Fortran and I have already written interface to let Fortran > >>>>> call C++ function. For the sequential version without PETSc, it can be > >>>>> compiled without problem using the following makefile. The parallel > >>>>> version without CGAL can also be compiled successfully. However, when I > >>>>> tried to use PETSc together with CGAL library, I cannot compile the > >>>>> code. > >>>>> My questions is: How can I modify the makefile? Do I need to reconfigure > >>>>> PETSc with special flags? All the makefile samples are shown below. > >>>>> > >>>>> #makefile for sequential version > >>>>> > >>>>> FC = gfortran > >>>>> #FC = ifort > >>>>> CXX = g++ -std=c++11 > >>>>> > >>>>> DLIB = -lstdc++ -lm -L/usr/local/lib -rdynamic /usr/local/lib/libmpfr.so > >>>>> /usr/local/lib/libgmp.so /usr/local/lib/libCGAL_ImageIO.so.11.0.1 > >>>>> /usr/local/lib/libCGAL.so.11.0.1 /usr/local/lib/libboost_thread.so > >>>>> /usr/local/lib/libboost_system.so -lpthread -lGLU -lGL -lX11 -lXext -lz > >>>>> /usr/local/lib/libCGAL_ImageIO.so.11.0.1 > >>>>> /usr/local/lib/libCGAL.so.11.0.1 > >>>>> /usr/local/lib/libboost_thread.so /usr/local/lib/libboost_system.so > >>>>> -lpthread -lGLU -lGL -lX11 -lXext -lz /usr/local/lib/libmpfr.so > >>>>> /usr/local/lib/libgmp.so /usr/local/lib/libboost_thread.so > >>>>> /usr/local/lib/libboost_system.so -lpthread -Wl,-rpath,/usr/local/lib > >>>>> > >>>>> FFLAGS = -O3 > >>>>> CXXFLAGS = -O3 > >>>>> > >>>>> FPPFLAGS = -DUSECGAL > >>>>> > >>>>> SRC =./../../ > >>>>> > >>>>> SOURCES = $(SRC)usg/math_common.o\ > >>>>> $(SRC)usg/geometry_definition.o\ > >>>>> $(SRC)usg/cgal_common.o\ > >>>>> > >>>>> ... > >>>>> > >>>>> executable: $(SOURCES) > >>>>> $(FC) $(FFLAGS) $(FPPFLAGS) -o executable.out $(SOURCES) ${LIS_LIB} > >>>>> $(DLIB) > >>>>> %.o:%.F90 > >>>>> $(FC) $(FFLAGS) $(FPPFLAGS) -c -frounding-math $< -o $@ > >>>>> %.o:%.cpp > >>>>> $(CXX) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ > >>>>> > >>>>> > >>>>> #makefile for parallel version with PETSc, without CGAL > >>>>> > >>>>> #FC = ifort > >>>>> #FC = gfortran > >>>>> > >>>>> DLIB = -lm > >>>>> > >>>>> FFLAGS = -O3 > >>>>> > >>>>> FPPFLAGS = -DUSEPETSC > >>>>> > >>>>> SRC =./../../ > >>>>> > >>>>> SOURCES = $(SRC)usg/math_common.o\ > >>>>> $(SRC)usg/geometry_definition.o\ > >>>>> $(SRC)usg/cgal_common.o\ > >>>>> > >>>>> ... > >>>>> > >>>>> executable: $(SOURCES) chkopts > >>>>> -${FLINKER} $(FFLAGS) $(FPPFLAGS) $(CPPFLAGS) -o executable.out > >>>>> $(SOURCES) ${PETSC_LIB} > >>>>> > >>>>> > >>>>> #makefile for parallel version with PETSc, with CGAL, CANNOT work > >>>>> > >>>>> #FC = ifort > >>>>> #FC = gfortran > >>>>> > >>>>> DLIB = -lm > >>>>> > >>>>> FFLAGS = -O3 > >>>>> > >>>>> FPPFLAGS = -DUSEPETSC -DUSECGAL > >>>>> > >>>>> SRC =./../../ > >>>>> > >>>>> SOURCES = $(SRC)usg/math_common.o\ > >>>>> $(SRC)usg/geometry_definition.o\ > >>>>> $(SRC)usg/cgal_common.o\ > >>>>> > >>>>> ... > >>>>> > >>>>> executable: $(SOURCES) chkopts > >>>>> -${FLINKER} $(FFLAGS) $(FPPFLAGS) $(CPPFLAGS) -o executable.out > >>>>> $(SOURCES) ${PETSC_LIB} > >>>>> > >>>>> %.o:%.F90 > >>>>> $(FC) $(FFLAGS) $(FPPFLAGS) -c -frounding-math $< -o $@ > >>>>> %.o:%.cpp > >>>>> $(CXX) $(CXXFLAGS) $(CPPFLAGS) -c -frounding-math $< -o $@ > >>>>> > >>>>> > >>>>> Thanks, > >>>>> > >>>>> Danyang > >>>>> > >> > > > From michael.wick.1980 at gmail.com Sun Jun 3 13:31:27 2018 From: michael.wick.1980 at gmail.com (Mike Wick) Date: Sun, 3 Jun 2018 11:31:27 -0700 Subject: [petsc-users] understand the restart number and total iteration number in GMRES Message-ID: Hi: I am trying to understand the KSPGMRES a little more. In most books (or, Algorithm 4 in Saad's 1986 paper), I found that the GMRES(m) algorithm tries to construct a Krylov subspace with size m first, then seek for a solution that minimize over this space. Therefore, the total iteration number should be a multiple of the restart number. Apparently this is not the case in KSPGRMES. I wander if there is any stopping condition inside the restart algorithm, or if I misunderstand some part. Thanks! Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Sun Jun 3 13:42:57 2018 From: jed at jedbrown.org (Jed Brown) Date: Sun, 03 Jun 2018 12:42:57 -0600 Subject: [petsc-users] understand the restart number and total iteration number in GMRES In-Reply-To: References: Message-ID: <87in6zhfta.fsf@jedbrown.org> GMRES provides a cheap estimate of the residual norm at each iteration. In some cases ("happy breakdown") the iteration can't even proceed. So there is no point building a subspace of size m, then looking for a solution. Instead, just run until the estimated residual norm satisfies the criteria, restarting with the current estimate of the solution as initial guess any time the subspace would exceed dimension m. This is how everyone does restarted GMRES. Mike Wick writes: > Hi: > > I am trying to understand the KSPGMRES a little more. In most books (or, > Algorithm 4 in Saad's 1986 paper), I found that the GMRES(m) algorithm > tries to construct a Krylov subspace with size m first, then seek for a > solution that minimize over this space. Therefore, the total iteration > number should be a multiple of the restart number. > > Apparently this is not the case in KSPGRMES. I wander if there is any > stopping condition inside the restart algorithm, or if I misunderstand some > part. > > Thanks! > > Mike From lukas.drinkt.thee at gmail.com Mon Jun 4 02:50:29 2018 From: lukas.drinkt.thee at gmail.com (Lukas van de Wiel) Date: Mon, 4 Jun 2018 09:50:29 +0200 Subject: [petsc-users] downloading hypre In-Reply-To: References: Message-ID: Hi Matt, Whoa, thanks for the invite, but that is a bit too short of a notice for the circumstances. I will gladly come to the next one. And if it is far away, I have good excuse to prepend a holiday to it. :-) WIll it be in the beginning of June in 2019 as well? Cheers Lukas On Fri, Jun 1, 2018 at 5:16 PM, Matthew Knepley wrote: > On Fri, Jun 1, 2018 at 11:11 AM, Lukas van de Wiel < > lukas.drinkt.thee at gmail.com> wrote: > >> Thanks Matt. That was fast! :-) >> >> The university server has: >> >> [17:04 gtecton at pbsserv petsc-3.8.1] > openssl version >> OpenSSL 0.9.8zh 3 Dec 2015 >> >> And a friend with >> >> OpenSSL 1.0.2g 1 Mar 2016 >> >> Can execute the git clone command without trouble. >> >> Cheers and have a great weekend! >> > > Great! Also if you are bored next week, we are having the PETSc Meeting in > London. > Only a short train ride by Eurostar :) > > Matt > > >> Lukas >> >> >> >> >> On Fri, Jun 1, 2018 at 5:08 PM, Matthew Knepley >> wrote: >> >>> On Fri, Jun 1, 2018 at 11:01 AM, Lukas van de Wiel < >>> lukas.drinkt.thee at gmail.com> wrote: >>> >>>> Hi all, >>>> >>>> for years I have been installing PETSc on machines, almost always >>>> without any issue to speak off. Compliments for the solid configure script. >>>> However, now I see an issue I cannot easily solve. >>>> >>>> When getting HYPRE in the configuration options, the output gives >>>> >>>> >>>> ************************************************************ >>>> ******************* >>>> UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log >>>> for details): >>>> ------------------------------------------------------------ >>>> ------------------- >>>> Error during download/extract/detection of HYPRE: >>>> Unable to download hypre >>>> Could not execute "git clone https://github.com/LLNL/hypre >>>> /net/home/gtecton/flops/petsc-3.8.1/linux-gnu-x86_64/externa >>>> lpackages/git.hypre": >>>> Cloning into '/net/home/gtecton/flops/petsc >>>> -3.8.1/linux-gnu-x86_64/externalpackages/git.hypre'... >>>> fatal: unable to access 'https://github.com/LLNL/hypre/': >>>> error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert >>>> protocol version >>>> Unable to download package HYPRE from: git://https://github.com/LLNL/ >>>> hypre >>>> >>>> Has anybody else seen this? It seems to get stuck on SSL, but web >>>> security is sadly not my forte... >>>> >>> >>> I have definitely had issues similar to this. It came up with Firedrake. >>> I built my own Python, and had >>> an early version of libssh/libcrypto. I think you need at least 1.0.1 to >>> get the SSL version that github >>> is now requiring. You can use ldd (or otool -L) to check the version for >>> your Python. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thanks a lot, >>>> >>>> Lukas >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jun 4 03:04:30 2018 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 4 Jun 2018 04:04:30 -0400 Subject: [petsc-users] downloading hypre In-Reply-To: References: Message-ID: On Mon, Jun 4, 2018 at 3:50 AM, Lukas van de Wiel < lukas.drinkt.thee at gmail.com> wrote: > Hi Matt, > > Whoa, thanks for the invite, but that is a bit too short of a notice for > the circumstances. > I will gladly come to the next one. And if it is far away, I have good > excuse to prepend a holiday to it. :-) > > WIll it be in the beginning of June in 2019 as well? > It usually is. We try to work around other conferences. Generally we get it set 6 months in advance. Thanks, Matt > Cheers > Lukas > > On Fri, Jun 1, 2018 at 5:16 PM, Matthew Knepley wrote: > >> On Fri, Jun 1, 2018 at 11:11 AM, Lukas van de Wiel < >> lukas.drinkt.thee at gmail.com> wrote: >> >>> Thanks Matt. That was fast! :-) >>> >>> The university server has: >>> >>> [17:04 gtecton at pbsserv petsc-3.8.1] > openssl version >>> OpenSSL 0.9.8zh 3 Dec 2015 >>> >>> And a friend with >>> >>> OpenSSL 1.0.2g 1 Mar 2016 >>> >>> Can execute the git clone command without trouble. >>> >>> Cheers and have a great weekend! >>> >> >> Great! Also if you are bored next week, we are having the PETSc Meeting >> in London. >> Only a short train ride by Eurostar :) >> >> Matt >> >> >>> Lukas >>> >>> >>> >>> >>> On Fri, Jun 1, 2018 at 5:08 PM, Matthew Knepley >>> wrote: >>> >>>> On Fri, Jun 1, 2018 at 11:01 AM, Lukas van de Wiel < >>>> lukas.drinkt.thee at gmail.com> wrote: >>>> >>>>> Hi all, >>>>> >>>>> for years I have been installing PETSc on machines, almost always >>>>> without any issue to speak off. Compliments for the solid configure script. >>>>> However, now I see an issue I cannot easily solve. >>>>> >>>>> When getting HYPRE in the configuration options, the output gives >>>>> >>>>> >>>>> ************************************************************ >>>>> ******************* >>>>> UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log >>>>> for details): >>>>> ------------------------------------------------------------ >>>>> ------------------- >>>>> Error during download/extract/detection of HYPRE: >>>>> Unable to download hypre >>>>> Could not execute "git clone https://github.com/LLNL/hypre >>>>> /net/home/gtecton/flops/petsc-3.8.1/linux-gnu-x86_64/externa >>>>> lpackages/git.hypre": >>>>> Cloning into '/net/home/gtecton/flops/petsc >>>>> -3.8.1/linux-gnu-x86_64/externalpackages/git.hypre'... >>>>> fatal: unable to access 'https://github.com/LLNL/hypre/': >>>>> error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert >>>>> protocol version >>>>> Unable to download package HYPRE from: git://https://github.com/LLNL/ >>>>> hypre >>>>> >>>>> Has anybody else seen this? It seems to get stuck on SSL, but web >>>>> security is sadly not my forte... >>>>> >>>> >>>> I have definitely had issues similar to this. It came up with >>>> Firedrake. I built my own Python, and had >>>> an early version of libssh/libcrypto. I think you need at least 1.0.1 >>>> to get the SSL version that github >>>> is now requiring. You can use ldd (or otool -L) to check the version >>>> for your Python. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Thanks a lot, >>>>> >>>>> Lukas >>>>> >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From Jean-Yves.L.Excellent at ens-lyon.fr Mon Jun 4 07:03:48 2018 From: Jean-Yves.L.Excellent at ens-lyon.fr (Jean-Yves LExcellent) Date: Mon, 4 Jun 2018 14:03:48 +0200 Subject: [petsc-users] MAT_NEW_NONZERO_LOCATIONS working? In-Reply-To: References: <1153922A-8876-492B-8AD9-F41DD718FD21@anl.gov> <58D5E959-0E3C-4EF2-8270-882D1EA6EA38@mcs.anl.gov> Message-ID: Thanks for the details of your needs. For the first application, the sparse RHS feature with distributed solution should effectively be fine. For the second one, a future distributed RHS feature (not currently available in MUMPS) might help if the centralized sparse RHS is too memory consuming, depending on the size of B1. Regards, Jean-Yves, for the MUMPS developers > ?I want to? invert a rather large sparse matrix for this using a > sparse rhs with centralized input would be ok as long as the solution > is distributed. > and the second application I have in mind is solving a system of the > from AX=B where A and B are sparse and B is given by a block matrix of > the form B=[B1 0, 0 0] where B1 is dense but the dimension is (much) > smaller than that of the whole matrix B. > Marius: > Current PETSc interface supports sequential sparse multiple right-hand > side, but not distributed. > It turns out that mumps does not support distributed sparse multiple > right-hand sides at > the moment (see attached email). > Jean-Yves invites you to communicate with him directly. > Let me know what we can help on this matter, > e.g., add support for parallel implementation of sparse multiple > right-hand side with > centralized rhs input? > Hong > ---------------------- > > > Jean-Yves LExcellent > > > > 5:14 AM (3 hours ago) > > toHong,mumps-dev > > Hello, > > We do not support distributed sparse multiple right-hand sides at > the moment. From the feedback we have from applications, the > right-hand sides are often very sparse, and having them distributed > did not seem critical. > > Since we are specifying a distributed right-hand sides feature at the > moment, could you let us know in more detail the need regarding > distributed sparse right-hand side (e.g., do all columns have the same > nonzero structure in that case) or put us in contact with the user who > needs this? > > Thanks, > Jean-Yves and Patrick > > Thanks a lot guys, very helpful. > > > > > I see MUMPS http://mumps.enseeiht.fr/ > > Sparse multiple right-hand side, distributed solution; > Exploitation of sparsity in the right-hand sidesPETSc interface > computes mumps?distributed solution as default (this is not new) > (ICNTL(21) = 1) > > I will add support for?Sparse multiple right-hand side. > > Hong > > On Thu, May 31, 2018 at 11:25 AM, Smith, Barry F. > [mailto:bsmith at mcs.anl.gov > ]> wrote: > ? Hong, > > ? ? Can you see about adding support for distributed right hand side? > > ? ? Thanks > > ? ? ? Barry > > > On May 31, 2018, at 2:37 AM, Marius Buerkle [mailto:mbuerkle at web.de > ]> wrote: > > > > The fix for MAT_NEW_NONZERO_LOCATIONS, thanks again. > > > > I have yet another question, sorry. The recent version of MUMPS > supports distributed and sparse RHS is there any chance that this > will be supported in PETSc in the near future? > > > > > > > > > >> On May 30, 2018, at 6:55 PM, Marius Buerkle [mailto:mbuerkle at web.de > ]> wrote: > >> > >> Thanks for the quick fix, I will test it and report back. > >> I have another maybe related question, if > MAT_NEW_NONZERO_LOCATIONS is true and let's say 1 new nonzero > position is created it does not allocated 1 but several new > nonzeros but only use 1. > > > > Correct > > > >> I think that is normal, right? > > > > Yes > > > >> But, at least as far as I understand the manual, a subsequent > call of mat assemble with > >> MAT_FINAL_ASSEMBLY should compress out the unused allocations > and release the memory, is this correct? > > > > It "compresses it out" (by shifting all the nonzero entries to > the beginning of the internal i, j, and a arrays), but does NOT > release any memory. Since the values are stored in one big > contiguous array (obtained with a single malloc) it cannot just > free part of the array, so the extra locations just sit harmlessly > at the end if the array unused. > > > >> If so, this did not work for me, even after doing > >> MAT_FINAL_ASSEMBLY the unused nonzero allocations remain. Is > this normal? > > > > Yes, > > > > Barry > > > >> > >>> > >>> Fixed in the branch barry/fix-mat-new-nonzero-locations/maint > >>> > >>> Once this passes testing it will go into the maint branch and > then the next patch release but you can use it now in the branch > barry/fix-mat-new-nonzero-locations/maint > >>> > >>> Thanks for the report and reproducible example > >>> > >>> Barry > >>> > >>> > >>>> On May 29, 2018, at 7:51 PM, Marius Buerkle [mailto:mbuerkle at web.de > ]> wrote: > >>>> > >>>> Sure, I made a small reproducer, it is Fortran though I hope > that is ok. If MAT_NEW_NONZERO_LOCATIONS is set to false I get an > error, if it is set to true the new nonzero element is inserted, > if MAT_NEW_NONZERO_LOCATIONS is false and either > MAT_NEW_NONZERO_LOCATION_ERR or MAT_NEW_NONZERO_ALLOCATION_ERR is > set to false afterwards then the new nonzero is also created > without an error, but if MAT_NEW_NONZERO_LOCATIONS is set to false > after MAT_NEW_NONZERO_LOCATION_ERR/MAT_NEW_NONZERO_ALLOCATION_ERR > have been set to false I get an error again. > >>>> > >>>> > >>>> program newnonzero > >>>> #include > >>>> use petscmat > >>>> implicit none > >>>> > >>>> Mat :: A > >>>> PetscInt :: dnnz,onnz,n,m,idxm(1),idxn(1),nl1,nl2 > >>>> PetscScalar :: v(1) > >>>> PetscReal :: info(MAT_INFO_SIZE) > >>>> PetscErrorCode :: ierr > >>>> > >>>> integer :: nproc,iproc,i > >>>> > >>>> call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > >>>> > >>>> call MPI_COMM_SIZE(PETSC_COMM_WORLD, nproc,ierr) > >>>> > >>>> call MPI_Comm_rank( PETSC_COMM_WORLD, iproc, ierr ) > >>>> > >>>> n=3 > >>>> m=n > >>>> call > MatCreateAIJ(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,n,m,1,PETSC_NULL_INTEGER,0,PETSC_NULL_INTEGER,A,ierr) > >>>> > >>>> > >>>> call MatGetOwnershipRange(A,nl1,nl2,ierr) > >>>> do i=nl1,nl2-1 > >>>> idxn(1)=i > >>>> idxm(1)=i > >>>> v(1)=1d0 > >>>> call MatSetValues(A,1,idxn,1,idxm, v,INSERT_VALUES,ierr) > >>>> end do > >>>> call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) > >>>> call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) > >>>> > >>>> call MatSetOption(A,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE,ierr) > >>>> !~ call > MatSetOption(A,MAT_NEW_NONZERO_LOCATION_ERR,PETSC_FALSE,ierr) > >>>> !~ call MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR > ,PETSC_FALSE,ierr) > >>>> !~ call > MatSetOption(A,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE,ierr) > >>>> > >>>> > >>>> idxn(1)=0 > >>>> idxm(1)=n-1 > >>>> if ((idxn(1).ge.nl1).and.(idxn(1).le.nl2-1)) then > >>>> v(1)=2d0 > >>>> call MatSetValues(A,1,idxn,1,idxm, v,INSERT_VALUES,ierr) > >>>> end if > >>>> call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) > >>>> call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) > >>>> > >>>> if ((idxn(1).ge.nl1).and.(idxn(1).le.nl2-1)) then > >>>> v(1)=2d0 > >>>> call MatGetValues(A,1,idxn,1,idxm, v,ierr) > >>>> write(6,*) v > >>>> end if > >>>> > >>>> call PetscFinalize(ierr) > >>>> > >>>> end program newnonzero > >>>> > >>>> > >>>> > >>>> $ mpiexec.hydra -n 3 ./a.out > >>>> [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > >>>> [0]PETSC ERROR: Argument out of range > >>>> [0]PETSC ERROR: Inserting a new nonzero at global row/column > (0, 2) into matrix > >>>> [0]PETSC ERROR: See > http://www.mcs.anl.gov/petsc/documentation/faq.html[http://www.mcs.anl.gov/petsc/documentation/faq.html] > for trouble shooting. > >>>> [0]PETSC ERROR: Petsc Release Version 3.9.2, May, 20, 2018 > >>>> [0]PETSC ERROR: ./a.out on a named tono-hpc1 by marius Wed > May 30 09:42:40 2018 > >>>> [0]PETSC ERROR: Configure options > --prefix=/home/marius/prog/petsc/3.9.2 --download-elemental=yes > --download-metis=yes --download-parmetis=yes --download-mumps=yes > --with-scalapack-lib="/home/marius/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64/libmkl_scalapack_lp64.a > -Wl,--start-group > /home/marius/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64/libmkl_intel_lp64.a > /home/marius/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64/libmkl_sequential.a > /home/marius/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64/libmkl_core.a > /home/marius/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.a > -Wl,--end-group -lpthread -lm -ldl" --FC=mpiifort --CC=mpicc > --CXX=mpicxx --with-scalar-type=complex --with-mpi-dir= > --with-blaslapack-lib="/home/marius/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64/libmkl_scalapack_lp64.a > -Wl,--start-group > /home/marius/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64/libmkl_intel_lp64.a > /home/marius/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64/libmkl_sequential.a > /home/marius/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64/libmkl_core.a > /home/marius/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.a > -Wl,--end-group -lpthread -lm -ldl" --with-cxx-dialect=C++11 > --download-superlu_dist=yes --download-ptscotch=yes --with-x > --with-debugging=1 --download-superlu=yes --with-mkl_cpardiso=1 > --with-mkl_pardiso=1 --with-scalapack=1 > >>>> [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 607 in > /home/marius/prog/petsc/petsc-3.9.2/src/mat/impls/aij/mpi/mpiaij.c > >>>> [0]PETSC ERROR: #2 MatSetValues() line 1312 in > /home/marius/prog/petsc/petsc-3.9.2/src/mat/interface/matrix.c > >>>> (0.000000000000000E+000,0.000000000000000E+000) > >>>> > >>>> > >>>> > >>>> Please send complete error message; type of matrix used etc. > Ideally code that demonstrates the problem. > >>>> > >>>> Barry > >>>> > >>>> > >>>>> On May 29, 2018, at 3:31 AM, Marius Buerkle [mailto:mbuerkle at web.de > ]> wrote: > >>>>> > >>>>> > >>>>> Hi, > >>>>> > >>>>> I tried to set MAT_NEW_NONZERO_LOCATIONS to false, as far as > I understood MatSetValues should simply ignore entries which would > give rise to new nonzero values not creating a new entry and not > cause an error, but I get "[1]PETSC ERROR: Inserting a new nonzero > at global row/column". Is this option supposed to work or not? > >>>> > >>> > >>> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From s.lanthaler at gmail.com Mon Jun 4 07:56:36 2018 From: s.lanthaler at gmail.com (Samuel Lanthaler) Date: Mon, 4 Jun 2018 14:56:36 +0200 Subject: [petsc-users] MatPtAP In-Reply-To: References: <1153922A-8876-492B-8AD9-F41DD718FD21@anl.gov> <58D5E959-0E3C-4EF2-8270-882D1EA6EA38@mcs.anl.gov> <1db3b230-7243-4664-e846-20ceb0132eea@gmail.com> <078b8767-adc2-bafa-ee78-6df98c8cb4ab@gmail.com> Message-ID: <9060dd82-51bd-347c-68bf-0ed3d32982f0@gmail.com> Ok, I now realize that I had implemented the boundary conditions in an unnecessarily complicated way... As you pointed out, I can just manipulate individual matrix rows to enforce the BC's. In that way, I never have to call MatPtAP, or do any expensive operations. Probably that's what is commonly done. I've changed my code, and it seems to work fine and is much faster. Thanks a lot for your help, everyone! This has been very educational for me. Samuel On 06/01/2018 09:04 PM, Hong wrote: > Samuel, > I have following questions: > 1) Why solving \lambda*Pt*B*P*y=Pt*A*P*y is better than solving > original \lambda*B*x = A*x? > > 2) Does your eigen solver require matrix factorization? If not, i.e., > only uses mat-vec multiplication, then you may implement > z = (Pt*(A*(P*y))) in your eigensolver instead of using mat-mat-mat > multiplication. > > 3) petsc PtAP() was implemented for multigrid applications, in which > the product C = PtAP is a denser but a much smaller matrix. > I have not seen the use of your case, that P is square with same size > as A. If C is much denser than A, then PtAP consumes a large portion > of time is anticipated. > > Hong > > On Fri, Jun 1, 2018 at 12:35 PM, Smith, Barry F. > wrote: > > > Could you save the P matrix with MatView() using a binary viewer > and the A matrix with MatView() and the binary viewer and email > them to petsc-maint at mcs.anl.gov ? > Then we can run the code in the profiler with your matrices and > see if there is any way to speed up the computation. > > Barry > > > > On Jun 1, 2018, at 11:07 AM, Samuel Lanthaler > > wrote: > > > > On 06/01/2018 03:42 PM, Matthew Knepley wrote: > >> On Fri, Jun 1, 2018 at 9:21 AM, Samuel Lanthaler > > wrote: > >> Hi, > >> > >> I was wondering what the most efficient way to use MatPtAP > would be in the following situation: I am discretizing a PDE > system. The discretization yields a matrix A that has a band > structure (with k upper and lower bands, say). In order to > implement the boundary conditions, I use a transformation matrix P > which is essentially the unit matrix, except for the entries > P_{ij} where i,j >> > >> P = [ B, 0, 0, 0, ..., 0, 0 ] > >> [ 0, 1, 0, 0, ..., 0, 0 ] > >> [ ] > >> [ ] > >> [ ..., 1, 0 ] > >> [ 0, 0, 0, 0, ..., 0, C ] > >> > >> with B,C are (k-by-k) matrices. > >> Right now, I'm simply constructing A, P and calling > >> > >> CALL > MatPtAP(petsc_matA,petsc_matP,MAT_INITIAL_MATRIX,PETSC_DEFAULT_REAL,petsc_matPtAP,ierr) > >> > >> where I haven't done anything to pestc_matPtAP, prior to this > call. Is this the way to do it? > >> > >> I'm asking because, currently, setting up the matrices A and P > takes very little time, whereas the operation MatPtAP is taking > quite long, which seems very odd... The matrices are of type > MPIAIJ. In my problem, the total matrix dimension is around 10'000 > and the matrix blocks (B,C) are of size ~100. > >> > >> Are you sure this is what you want to do? Usually BC are local, > since by definition PDE are local, and > >> are applied pointwise. What kind of BC do you have here? > >> > > > > The boundary conditions are a mixture of Dirichlet and Neumann; > in my case, the PDE is a system involving 8 variables on a disk, > where the periodic direction is discretized using a Fourier series > expansion, the radial direction uses B-splines. > > > > In reality, I have two matrices A,B, and want to solve the > eigenvalue problem \lambda*B*x = A*x. > > I found it quite convenient to use a transformation P to a > different set of variables y, such that x=P*y and x satisfies the > BC iff certain components of y are 0. The latter is enforced by > inserting spurious eigenvalues at the relevant components of y in > the transformed eigenvalue problem \lambda*Pt*B*P*y=Pt*A*P*y. > After solving the EVP in terms of y, I get back x=P*y. > > Is this an inherently bad/inefficient way of enforcing BC's? Thanks. > > > > > > > > > >> Thanks, > >> > >> Matt > >> > >> Thanks in advance for any ideas. > >> > >> Cheers, > >> Samuel > >> > >> > >> > >> -- > >> What most experimenters take for granted before they begin > their experiments is infinitely more interesting than any results > to which their experiments lead. > >> -- Norbert Wiener > >> > >> https://www.cse.buffalo.edu/~knepley/ > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Mon Jun 4 09:54:28 2018 From: hzhang at mcs.anl.gov (Hong) Date: Mon, 4 Jun 2018 15:54:28 +0100 Subject: [petsc-users] MAT_NEW_NONZERO_LOCATIONS working? In-Reply-To: References: <1153922A-8876-492B-8AD9-F41DD718FD21@anl.gov> <58D5E959-0E3C-4EF2-8270-882D1EA6EA38@mcs.anl.gov> Message-ID: On Mon, Jun 4, 2018 at 1:03 PM, Jean-Yves LExcellent < Jean-Yves.L.Excellent at ens-lyon.fr> wrote: > > Thanks for the details of your needs. > > For the first application, the sparse RHS feature with distributed > solution should effectively be fine. > I'll add parallel support of this feature in petsc after I'm back the ANL after June 14th. Hong > > For the second one, a future distributed RHS feature (not currently > available in MUMPS) might help if the centralized sparse RHS is too > memory consuming, depending on the size of B1. > > Regards, > Jean-Yves, for the MUMPS developers > > > I want to invert a rather large sparse matrix for this using a sparse > rhs with centralized input would be ok as long as the solution is > distributed. > > and the second application I have in mind is solving a system of the from > AX=B where A and B are sparse and B is given by a block matrix of the form > B=[B1 0, 0 0] where B1 is dense but the dimension is (much) smaller than > that of the whole matrix B. > > Marius: > Current PETSc interface supports sequential sparse multiple right-hand > side, but not distributed. > It turns out that mumps does not support distributed sparse multiple > right-hand sides at > the moment (see attached email). > > Jean-Yves invites you to communicate with him directly. > Let me know what we can help on this matter, > e.g., add support for parallel implementation of sparse multiple > right-hand side with > centralized rhs input? > > Hong > > ---------------------- > Jean-Yves LExcellent > 5:14 AM (3 hours ago) > > to Hong, mumps-dev > > > > > Hello, > > We do not support distributed sparse multiple right-hand sides at > the moment. From the feedback we have from applications, the > right-hand sides are often very sparse, and having them distributed > did not seem critical. > > Since we are specifying a distributed right-hand sides feature at the > moment, could you let us know in more detail the need regarding > distributed sparse right-hand side (e.g., do all columns have the same > nonzero structure in that case) or put us in contact with the user who > needs this? > > Thanks, > Jean-Yves and Patrick > >> >> Thanks a lot guys, very helpful. >> >> >> >> >> I see MUMPS http://mumps.enseeiht.fr/ >> >> Sparse multiple right-hand side, distributed solution; Exploitation of >> sparsity in the right-hand sidesPETSc interface computes mumps distributed >> solution as default (this is not new) (ICNTL(21) = 1) >> >> I will add support for Sparse multiple right-hand side. >> >> Hong >> >> On Thu, May 31, 2018 at 11:25 AM, Smith, Barry F. > [mailto:bsmith at mcs.anl.gov]> wrote: >> Hong, >> >> Can you see about adding support for distributed right hand side? >> >> Thanks >> >> Barry >> >> > On May 31, 2018, at 2:37 AM, Marius Buerkle > mbuerkle at web.de]> wrote: >> > >> > The fix for MAT_NEW_NONZERO_LOCATIONS, thanks again. >> > >> > I have yet another question, sorry. The recent version of MUMPS >> supports distributed and sparse RHS is there any chance that this will be >> supported in PETSc in the near future? >> > >> > >> > >> > >> >> On May 30, 2018, at 6:55 PM, Marius Buerkle > mbuerkle at web.de]> wrote: >> >> >> >> Thanks for the quick fix, I will test it and report back. >> >> I have another maybe related question, if MAT_NEW_NONZERO_LOCATIONS is >> true and let's say 1 new nonzero position is created it does not allocated >> 1 but several new nonzeros but only use 1. >> > >> > Correct >> > >> >> I think that is normal, right? >> > >> > Yes >> > >> >> But, at least as far as I understand the manual, a subsequent call of >> mat assemble with >> >> MAT_FINAL_ASSEMBLY should compress out the unused allocations and >> release the memory, is this correct? >> > >> > It "compresses it out" (by shifting all the nonzero entries to the >> beginning of the internal i, j, and a arrays), but does NOT release any >> memory. Since the values are stored in one big contiguous array (obtained >> with a single malloc) it cannot just free part of the array, so the extra >> locations just sit harmlessly at the end if the array unused. >> > >> >> If so, this did not work for me, even after doing >> >> MAT_FINAL_ASSEMBLY the unused nonzero allocations remain. Is this >> normal? >> > >> > Yes, >> > >> > Barry >> > >> >> >> >>> >> >>> Fixed in the branch barry/fix-mat-new-nonzero-locations/maint >> >>> >> >>> Once this passes testing it will go into the maint branch and then >> the next patch release but you can use it now in the branch >> barry/fix-mat-new-nonzero-locations/maint >> >>> >> >>> Thanks for the report and reproducible example >> >>> >> >>> Barry >> >>> >> >>> >> >>>> On May 29, 2018, at 7:51 PM, Marius Buerkle > mbuerkle at web.de]> wrote: >> >>>> >> >>>> Sure, I made a small reproducer, it is Fortran though I hope that is >> ok. If MAT_NEW_NONZERO_LOCATIONS is set to false I get an error, if it is >> set to true the new nonzero element is inserted, if >> MAT_NEW_NONZERO_LOCATIONS is false and either MAT_NEW_NONZERO_LOCATION_ERR >> or MAT_NEW_NONZERO_ALLOCATION_ERR is set to false afterwards then the new >> nonzero is also created without an error, but if MAT_NEW_NONZERO_LOCATIONS >> is set to false after MAT_NEW_NONZERO_LOCATION_ERR/MAT_NEW_NONZERO_ALLOCATION_ERR >> have been set to false I get an error again. >> >>>> >> >>>> >> >>>> program newnonzero >> >>>> #include >> >>>> use petscmat >> >>>> implicit none >> >>>> >> >>>> Mat :: A >> >>>> PetscInt :: dnnz,onnz,n,m,idxm(1),idxn(1),nl1,nl2 >> >>>> PetscScalar :: v(1) >> >>>> PetscReal :: info(MAT_INFO_SIZE) >> >>>> PetscErrorCode :: ierr >> >>>> >> >>>> integer :: nproc,iproc,i >> >>>> >> >>>> call PetscInitialize(PETSC_NULL_CHARACTER,ierr) >> >>>> >> >>>> call MPI_COMM_SIZE(PETSC_COMM_WORLD, nproc,ierr) >> >>>> >> >>>> call MPI_Comm_rank( PETSC_COMM_WORLD, iproc, ierr ) >> >>>> >> >>>> n=3 >> >>>> m=n >> >>>> call MatCreateAIJ(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,n,m, >> 1,PETSC_NULL_INTEGER,0,PETSC_NULL_INTEGER,A,ierr) >> >>>> >> >>>> >> >>>> call MatGetOwnershipRange(A,nl1,nl2,ierr) >> >>>> do i=nl1,nl2-1 >> >>>> idxn(1)=i >> >>>> idxm(1)=i >> >>>> v(1)=1d0 >> >>>> call MatSetValues(A,1,idxn,1,idxm, v,INSERT_VALUES,ierr) >> >>>> end do >> >>>> call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) >> >>>> call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) >> >>>> >> >>>> call MatSetOption(A,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE,ierr) >> >>>> !~ call MatSetOption(A,MAT_NEW_NONZERO_LOCATION_ERR,PETSC_ >> FALSE,ierr) >> >>>> !~ call MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR >> ,PETSC_FALSE,ierr) >> >>>> !~ call MatSetOption(A,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE,ierr) >> >>>> >> >>>> >> >>>> idxn(1)=0 >> >>>> idxm(1)=n-1 >> >>>> if ((idxn(1).ge.nl1).and.(idxn(1).le.nl2-1)) then >> >>>> v(1)=2d0 >> >>>> call MatSetValues(A,1,idxn,1,idxm, v,INSERT_VALUES,ierr) >> >>>> end if >> >>>> call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) >> >>>> call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) >> >>>> >> >>>> if ((idxn(1).ge.nl1).and.(idxn(1).le.nl2-1)) then >> >>>> v(1)=2d0 >> >>>> call MatGetValues(A,1,idxn,1,idxm, v,ierr) >> >>>> write(6,*) v >> >>>> end if >> >>>> >> >>>> call PetscFinalize(ierr) >> >>>> >> >>>> end program newnonzero >> >>>> >> >>>> >> >>>> >> >>>> $ mpiexec.hydra -n 3 ./a.out >> >>>> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> >>>> [0]PETSC ERROR: Argument out of range >> >>>> [0]PETSC ERROR: Inserting a new nonzero at global row/column (0, 2) >> into matrix >> >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/ >> documentation/faq.html[http://www.mcs.anl.gov/petsc/ >> documentation/faq.html] for trouble shooting. >> >>>> [0]PETSC ERROR: Petsc Release Version 3.9.2, May, 20, 2018 >> >>>> [0]PETSC ERROR: ./a.out on a named tono-hpc1 by marius Wed May 30 >> 09:42:40 2018 >> >>>> [0]PETSC ERROR: Configure options --prefix=/home/marius/prog/petsc/3.9.2 >> --download-elemental=yes --download-metis=yes --download-parmetis=yes >> --download-mumps=yes --with-scalapack-lib="/home/ >> marius/intel/compilers_and_libraries_2018.2.199/linux/ >> mkl/lib/intel64/libmkl_scalapack_lp64.a -Wl,--start-group >> /home/marius/intel/compilers_and_libraries_2018.2.199/ >> linux/mkl/lib/intel64/libmkl_intel_lp64.a /home/marius/intel/compilers_ >> and_libraries_2018.2.199/linux/mkl/lib/intel64/libmkl_sequential.a >> /home/marius/intel/compilers_and_libraries_2018.2.199/ >> linux/mkl/lib/intel64/libmkl_core.a /home/marius/intel/compilers_ >> and_libraries_2018.2.199/linux/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.a >> -Wl,--end-group -lpthread -lm -ldl" --FC=mpiifort --CC=mpicc --CXX=mpicxx >> --with-scalar-type=complex --with-mpi-dir= --with-blaslapack-lib="/home/ >> marius/intel/compilers_and_libraries_2018.2.199/linux/ >> mkl/lib/intel64/libmkl_scalapack_lp64.a -Wl,--start-group >> /home/marius/intel/compilers_and_libraries_2018.2.199/ >> linux/mkl/lib/intel64/libmkl_intel_lp64.a /home/marius/intel/compilers_ >> and_libraries_2018.2.199/linux/mkl/lib/intel64/libmkl_sequential.a >> /home/marius/intel/compilers_and_libraries_2018.2.199/ >> linux/mkl/lib/intel64/libmkl_core.a /home/marius/intel/compilers_ >> and_libraries_2018.2.199/linux/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.a >> -Wl,--end-group -lpthread -lm -ldl" --with-cxx-dialect=C++11 >> --download-superlu_dist=yes --download-ptscotch=yes --with-x >> --with-debugging=1 --download-superlu=yes --with-mkl_cpardiso=1 >> --with-mkl_pardiso=1 --with-scalapack=1 >> >>>> [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 607 in >> /home/marius/prog/petsc/petsc-3.9.2/src/mat/impls/aij/mpi/mpiaij.c >> >>>> [0]PETSC ERROR: #2 MatSetValues() line 1312 in >> /home/marius/prog/petsc/petsc-3.9.2/src/mat/interface/matrix.c >> >>>> (0.000000000000000E+000,0.000000000000000E+000) >> >>>> >> >>>> >> >>>> >> >>>> Please send complete error message; type of matrix used etc. Ideally >> code that demonstrates the problem. >> >>>> >> >>>> Barry >> >>>> >> >>>> >> >>>>> On May 29, 2018, at 3:31 AM, Marius Buerkle > [mailto:mbuerkle at web.de]> wrote: >> >>>>> >> >>>>> >> >>>>> Hi, >> >>>>> >> >>>>> I tried to set MAT_NEW_NONZERO_LOCATIONS to false, as far as I >> understood MatSetValues should simply ignore entries which would give rise >> to new nonzero values not creating a new entry and not cause an error, but >> I get "[1]PETSC ERROR: Inserting a new nonzero at global row/column". Is >> this option supposed to work or not? >> >>>> >> >>> >> >>> >> > >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Michael.Becker at physik.uni-giessen.de Mon Jun 4 11:59:40 2018 From: Michael.Becker at physik.uni-giessen.de (Michael Becker) Date: Mon, 4 Jun 2018 18:59:40 +0200 Subject: [petsc-users] Poor weak scaling when solving successive linearsystems In-Reply-To: References: <6d9b6488-3957-0d74-148a-323c78d42f6a@physik.uni-giessen.de> <079D8B4D-C57B-4054-B9BD-05644E52C3A4@anl.gov> Message-ID: Hello again, this took me longer than I anticipated, but here we go. I did reruns of the cases where only half the processes per node were used (without -log_sync): 125 procs,1st?????????? 125 procs,2nd????????? 1000 procs,1st????????? 1000 procs,2nd ?????????????????? Max??????? Ratio Max??????? RatioMax??????? RatioMax??????? Ratio KSPSolve 1.203E+02??? 1.0??? ??? 1.210E+02??? 1.0??? ??? 1.399E+02 1.1??? ??? 1.365E+02??? 1.0 VecTDot??????????? 6.376E+00??? 3.7??? ??? 6.551E+00??? 4.0 7.885E+00??? 2.9??? ??? 7.175E+00??? 3.4 VecNorm??????????? 4.579E+00??? 7.1??? ??? 5.803E+00?? 10.2 8.534E+00??? 6.9??? ??? 6.026E+00??? 4.9 VecScale?????????? 1.070E-01??? 2.1??? ??? 1.129E-01??? 2.2 1.301E-01??? 2.5??? ??? 1.270E-01??? 2.4 VecCopy??????????? 1.123E-01??? 1.3??? ??? 1.149E-01??? 1.3 1.301E-01??? 1.6??? ??? 1.359E-01??? 1.6 VecSet???????????? 7.063E-01??? 1.7??? ??? 6.968E-01??? 1.7 7.432E-01??? 1.8??? ??? 7.425E-01??? 1.8 VecAXPY??????????? 1.166E+00??? 1.4??? ??? 1.167E+00??? 1.4 1.221E+00??? 1.5??? ??? 1.279E+00??? 1.6 VecAYPX??????????? 1.317E+00??? 1.6??? ??? 1.290E+00??? 1.6 1.536E+00??? 1.9??? ??? 1.499E+00??? 2.0 VecScatterBegin??? 6.142E+00??? 3.2??? ??? 5.974E+00??? 2.8 6.448E+00??? 3.0??? ??? 6.472E+00??? 2.9 VecScatterEnd????? 3.606E+01??? 4.2??? ??? 3.551E+01??? 4.0 5.244E+01??? 2.7??? ??? 4.995E+01??? 2.7 MatMult??????????? 3.561E+01??? 1.6??? ??? 3.403E+01??? 1.5 3.435E+01??? 1.4??? ??? 3.332E+01??? 1.4 MatMultAdd???????? 1.124E+01??? 2.0??? ??? 1.130E+01??? 2.1 2.093E+01??? 2.9??? ??? 1.995E+01??? 2.7 MatMultTranspose?? 1.372E+01??? 2.5??? ??? 1.388E+01??? 2.6 1.477E+01??? 2.2??? ??? 1.381E+01??? 2.1 MatSolve?????????? 1.949E-02??? 0.0??? ??? 1.653E-02??? 0.0 4.789E-02??? 0.0??? ??? 4.466E-02??? 0.0 MatSOR???????????? 6.610E+01??? 1.3??? ??? 6.673E+01??? 1.3 7.111E+01??? 1.3??? ??? 7.105E+01??? 1.3 MatResidual??????? 2.647E+01??? 1.7??? ??? 2.667E+01??? 1.7 2.446E+01??? 1.4??? ??? 2.467E+01??? 1.5 PCSetUpOnBlocks??? 5.266E-03??? 1.4??? ??? 5.295E-03??? 1.4 5.427E-03??? 1.5??? ??? 5.289E-03??? 1.4 PCApply??????????? 1.031E+02??? 1.0??? ??? 1.035E+02??? 1.0 1.180E+02??? 1.0??? ??? 1.164E+02??? 1.0 I also slimmed down my code and basically wrote a simple weak scaling test (source files attached) so you can profile it yourself. I appreciate the offer Junchao, thank you. You can adjust the system size per processor at runtime via "-nodes_per_proc 30" and the number of repeated calls to the function containing KSPsolve() via "-iterations 1000". The physical problem is simply calculating the electric potential from a homogeneous charge distribution, done multiple times to accumulate time in KSPsolve(). A job would be started using something like mpirun -n 125 ~/petsc_ws/ws_test -nodes_per_proc 30 -mesh_size 1E-4 -iterations 1000 \\ ?-ksp_rtol 1E-6 \ ?-log_view -log_sync\ ?-pc_type gamg -pc_gamg_type classical\ ?-ksp_type cg \ ?-ksp_norm_type unpreconditioned \ ?-mg_levels_ksp_type richardson \ ?-mg_levels_ksp_norm_type none \ ?-mg_levels_pc_type sor \ ?-mg_levels_ksp_max_it 1 \ ?-mg_levels_pc_sor_its 1 \ ?-mg_levels_esteig_ksp_type cg \ ?-mg_levels_esteig_ksp_max_it 10 \ ?-gamg_est_ksp_type cg , ideally started on a cube number of processes for a cubical process grid. Using 125 processes and 10.000 iterations I get the output in "log_view_125_new.txt", which shows the same imbalance for me. Michael Am 02.06.2018 um 13:40 schrieb Mark Adams: > > > On Fri, Jun 1, 2018 at 11:20 PM, Junchao Zhang > wrote: > > Hi,Michael, > ? You can add -log_sync besides -log_view, which adds barriers to > certain events but measures barrier time separately from the > events. I find this option makes it easier to interpret log_view > output. > > > That is great (good to know). > > This should give us a better idea if your large VecScatter costs are > from slow communication or if it catching some sort of load imbalance. > > > --Junchao Zhang > > On Wed, May 30, 2018 at 3:27 AM, Michael Becker > > wrote: > > Barry: On its way. Could take a couple days again. > > Junchao: I unfortunately don't have access to a cluster with a > faster network. This one has a mixed 4X QDR-FDR InfiniBand 2:1 > blocking fat-tree network, which I realize causes parallel > slowdown if the nodes are not connected to the same switch. > Each node has 24 processors (2x12/socket) and four NUMA > domains (two for each socket). > The ranks are usually not distributed perfectly even, i.e. for > 125 processes, of the six required nodes, five would use 21 > cores and one 20. > Would using another CPU type make a difference > communication-wise? I could switch to faster ones (on the same > network), but I always assumed this would only improve > performance of the stuff that is unrelated to communication. > > Michael > > > >> The log files have something like "Average time for zero size >> MPI_Send(): 1.84231e-05". It looks you ran on a cluster with >> a very slow network. A typical machine should give less than >> 1/10 of the latency you have. An easy way to try is just >> running the code on a machine with a faster network and see >> what happens. >> >> Also, how many cores & numa domains does a compute node have? >> I could not figure out how you distributed the 125 MPI ranks >> evenly. >> >> --Junchao Zhang >> >> On Tue, May 29, 2018 at 6:18 AM, Michael Becker >> > > wrote: >> >> Hello again, >> >> here are the updated log_view files for 125 and 1000 >> processors. I ran both problems twice, the first time >> with all processors per node allocated ("-1.txt"), the >> second with only half on twice the number of nodes >> ("-2.txt"). >> >> >>>> On May 24, 2018, at 12:24 AM, Michael Becker >>>> wrote: >>>> >>>> I noticed that for every individual KSP iteration, six vector objects are created and destroyed (with CG, more with e.g. GMRES). >>> Hmm, it is certainly not intended at vectors be created and destroyed within each KSPSolve() could you please point us to the code that makes you think they are being created and destroyed? We create all the work vectors at KSPSetUp() and destroy them in KSPReset() not during the solve. Not that this would be a measurable distance. >> >> I mean this, right in the log_view output: >> >>> Memory usage is given in bytes: >>> >>> Object Type Creations Destructions Memory Descendants' Mem. >>> Reports information only for process 0. >>> >>> --- Event Stage 0: Main Stage >>> >>> ... >>> >>> --- Event Stage 1: First Solve >>> >>> ... >>> >>> --- Event Stage 2: Remaining Solves >>> >>> Vector 23904 23904 1295501184 0. >> I logged the exact number of KSP iterations over the 999 >> timesteps and its exactly 23904/6 = 3984. >> >> Michael >> >> >> >> Am 24.05.2018 um 19:50 schrieb Smith, Barry F.: >>> Please send the log file for 1000 with cg as the solver. >>> >>> You should make a bar chart of each event for the two cases to see which ones are taking more time and which are taking less (we cannot tell with the two logs you sent us since they are for different solvers.) >>> >>> >>> >>>> On May 24, 2018, at 12:24 AM, Michael Becker >>>> wrote: >>>> >>>> I noticed that for every individual KSP iteration, six vector objects are created and destroyed (with CG, more with e.g. GMRES). >>> Hmm, it is certainly not intended at vectors be created and destroyed within each KSPSolve() could you please point us to the code that makes you think they are being created and destroyed? We create all the work vectors at KSPSetUp() and destroy them in KSPReset() not during the solve. Not that this would be a measurable distance. >>> >>> >>> >>>> This seems kind of wasteful, is this supposed to be like this? Is this even the reason for my problems? Apart from that, everything seems quite normal to me (but I'm not the expert here). >>>> >>>> >>>> Thanks in advance. >>>> >>>> Michael >>>> >>>> >>>> >>>> >> >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- /home/ritsat/beckerm/petsc_ws2/ws_test on a arch-linux-amd-opt named node1-051 with 125 processors, by beckerm Mon Jun 4 16:37:26 2018 Using Petsc Development GIT revision: v3.9.2-503-g9e88a8b GIT Date: 2018-05-24 08:01:24 -0500 Max Max/Min Avg Total Time (sec): 2.249e+03 1.00000 2.249e+03 Objects: 4.205e+05 1.00000 4.205e+05 Flop: 3.694e+11 1.15838 3.497e+11 4.372e+13 Flop/sec: 1.643e+08 1.15838 1.555e+08 1.944e+10 MPI Messages: 1.812e+07 3.38507 1.233e+07 1.542e+09 MPI Message Lengths: 2.261e+10 2.20109 1.418e+03 2.186e+12 MPI Reductions: 3.706e+05 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flop and VecAXPY() for complex vectors of length N --> 8N flop Summary of Stages: ----- Time ------ ----- Flop ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 1.4622e-01 0.0% 0.0000e+00 0.0% 1.200e+03 0.0% 1.802e+03 0.0% 1.700e+01 0.0% 1: First Solve: 2.9678e+00 0.1% 5.6491e+09 0.0% 4.212e+05 0.0% 3.421e+03 0.1% 5.660e+02 0.2% 2: Remaining Solves: 2.2459e+03 99.9% 4.3710e+13 100.0% 1.541e+09 100.0% 1.417e+03 99.9% 3.700e+05 99.8% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flop: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flop in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flop --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage VecSet 2 1.0 1.3185e-04 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 --- Event Stage 1: First Solve BuildTwoSided 12 1.0 9.4547e-03 1.6 0.00e+00 0.0 8.8e+03 4.0e+00 0.0e+00 0 0 0 0 0 0 0 2 0 0 0 BuildTwoSidedF 30 1.0 2.2989e-01 3.1 0.00e+00 0.0 7.1e+03 1.0e+04 0.0e+00 0 0 0 0 0 5 0 2 5 0 0 KSPSetUp 9 1.0 5.4758e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.8e+01 0 0 0 0 0 0 0 0 0 3 0 KSPSolve 1 1.0 2.9672e+00 1.0 4.82e+07 1.2 4.2e+05 3.4e+03 5.7e+02 0 0 0 0 0 100100100100100 1904 VecTDot 14 1.0 1.8083e-02 5.9 7.56e+05 1.0 0.0e+00 0.0e+00 1.4e+01 0 0 0 0 0 0 2 0 0 2 5226 VecNormBarrier 9 1.0 2.1014e-03 5.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNorm 9 1.0 9.9111e-04 1.2 4.86e+05 1.0 0.0e+00 0.0e+00 9.0e+00 0 0 0 0 0 0 1 0 0 2 61295 VecScale 42 1.0 4.4274e-04 3.0 9.47e+04 2.2 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 20631 VecCopy 1 1.0 1.2612e-04 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 178 1.0 1.3885e-03 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 14 1.0 2.0151e-03 2.0 7.56e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 2 0 0 0 46896 VecAYPX 49 1.0 2.4726e-03 2.1 6.46e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 32264 VecAssemblyBegin 2 1.0 2.3842e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd 2 1.0 2.1458e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBarrie 178 1.0 9.6261e-02 3.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 VecScatterBegin 178 1.0 1.4301e-02 4.6 0.00e+00 0.0 1.5e+05 1.4e+03 0.0e+00 0 0 0 0 0 0 0 37 15 0 0 VecScatterEnd 178 1.0 3.4915e-02 4.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 MatMult 50 1.0 8.9105e-02 1.5 1.05e+07 1.1 5.1e+04 2.1e+03 0.0e+00 0 0 0 0 0 2 22 12 7 0 13981 MatMultAdd 42 1.0 3.3666e-02 1.7 2.40e+06 1.3 2.8e+04 6.7e+02 0.0e+00 0 0 0 0 0 1 5 7 1 0 8183 MatMultTranspose 42 1.0 2.7552e-02 1.5 2.40e+06 1.3 2.8e+04 6.7e+02 0.0e+00 0 0 0 0 0 1 5 7 1 0 10000 MatSolve 7 0.0 3.0994e-05 0.0 8.40e+02 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 27 MatSOR 84 1.0 1.2700e-01 1.7 1.90e+07 1.2 4.7e+04 1.6e+03 1.4e+01 0 0 0 0 0 4 40 11 5 2 17715 MatLUFactorSym 1 1.0 9.7990e-05 2.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatLUFactorNum 1 1.0 1.5020e-05 5.2 3.14e+02 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 21 MatResidual 42 1.0 7.5414e-02 1.7 7.97e+06 1.2 4.7e+04 1.6e+03 0.0e+00 0 0 0 0 0 2 17 11 5 0 12369 MatAssemblyBegin 94 1.0 2.3261e-01 2.8 0.00e+00 0.0 7.1e+03 1.0e+04 0.0e+00 0 0 0 0 0 5 0 2 5 0 0 MatAssemblyEnd 94 1.0 2.2487e-01 1.1 0.00e+00 0.0 6.3e+04 2.1e+02 2.3e+02 0 0 0 0 0 7 0 15 1 41 0 MatGetRow 3100250 1.2 4.5413e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 14 0 0 0 0 0 MatGetRowIJ 1 0.0 1.0014e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatCreateSubMats 6 1.0 5.7464e-01 1.5 0.00e+00 0.0 5.5e+04 1.8e+04 1.2e+01 0 0 0 0 0 16 0 13 67 2 0 MatCreateSubMat 4 1.0 9.9725e-02 1.0 0.00e+00 0.0 2.8e+03 2.8e+02 6.4e+01 0 0 0 0 0 3 0 1 0 11 0 MatGetOrdering 1 0.0 1.5283e-04 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatIncreaseOvrlp 6 1.0 7.9965e-02 1.3 0.00e+00 0.0 2.7e+04 1.0e+03 1.2e+01 0 0 0 0 0 2 0 6 2 2 0 MatCoarsen 6 1.0 2.0824e-02 1.1 0.00e+00 0.0 5.4e+04 6.0e+02 3.4e+01 0 0 0 0 0 1 0 13 2 6 0 MatZeroEntries 6 1.0 3.5489e-03 5.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatPtAP 6 1.0 3.1160e-01 1.0 1.13e+07 1.3 6.4e+04 2.7e+03 9.2e+01 0 0 0 0 0 10 23 15 12 16 4085 MatPtAPSymbolic 6 1.0 1.5794e-01 1.0 0.00e+00 0.0 3.4e+04 2.7e+03 4.2e+01 0 0 0 0 0 5 0 8 6 7 0 MatPtAPNumeric 6 1.0 1.4963e-01 1.0 1.13e+07 1.3 2.9e+04 2.6e+03 4.8e+01 0 0 0 0 0 5 23 7 5 8 8507 MatGetLocalMat 6 1.0 4.6241e-03 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetBrAoCol 6 1.0 1.0492e-02 1.6 0.00e+00 0.0 2.0e+04 3.6e+03 0.0e+00 0 0 0 0 0 0 0 5 5 0 0 SFSetGraph 12 1.0 1.5020e-05 3.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SFSetUp 12 1.0 1.6023e-02 1.1 0.00e+00 0.0 2.6e+04 6.3e+02 0.0e+00 0 0 0 0 0 1 0 6 1 0 0 SFBcastBegin 46 1.0 2.0034e-03 2.2 0.00e+00 0.0 5.5e+04 7.0e+02 0.0e+00 0 0 0 0 0 0 0 13 3 0 0 SFBcastEnd 46 1.0 8.9741e-03 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 GAMG: createProl 6 1.0 2.2056e+00 1.0 0.00e+00 0.0 2.0e+05 5.3e+03 2.9e+02 0 0 0 0 0 74 0 47 73 51 0 GAMG: partLevel 6 1.0 4.1617e-01 1.0 1.13e+07 1.3 6.6e+04 2.6e+03 1.9e+02 0 0 0 0 0 14 23 16 12 34 3059 repartition 2 1.0 1.3080e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01 0 0 0 0 0 0 0 0 0 2 0 Invert-Sort 2 1.0 1.3320e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 0 0 0 0 0 1 0 Move A 2 1.0 2.3206e-02 1.0 0.00e+00 0.0 1.4e+03 5.4e+02 3.4e+01 0 0 0 0 0 1 0 0 0 6 0 Move P 2 1.0 7.7642e-02 1.0 0.00e+00 0.0 1.4e+03 1.3e+01 3.4e+01 0 0 0 0 0 3 0 0 0 6 0 PCSetUp 2 1.0 2.6313e+00 1.0 1.13e+07 1.3 2.7e+05 4.6e+03 5.1e+02 0 0 0 0 0 89 23 63 85 91 484 PCSetUpOnBlocks 7 1.0 4.0460e-04 1.9 3.14e+02 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1 PCApply 7 1.0 2.2264e-01 1.1 3.18e+07 1.2 1.5e+05 1.2e+03 1.4e+01 0 0 0 0 0 7 66 35 13 2 16770 --- Event Stage 2: Remaining Solves KSPSolve 10000 1.0 2.2427e+03 1.0 3.69e+11 1.2 1.5e+09 1.4e+03 3.7e+05100100100100100 100100100100100 19490 VecTDot 140000 1.0 1.8447e+02 6.4 7.56e+09 1.0 0.0e+00 0.0e+00 1.4e+05 3 2 0 0 38 3 2 0 0 38 5123 VecNormBarrier 90000 1.0 1.9840e+01 5.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNorm 90000 1.0 8.2702e+00 1.2 4.86e+09 1.0 0.0e+00 0.0e+00 9.0e+04 0 1 0 0 24 0 1 0 0 24 73456 VecScale 420000 1.0 2.1495e+00 2.2 9.47e+08 2.2 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 42496 VecCopy 10000 1.0 1.2405e+00 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 1470000 1.0 1.3027e+01 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 140000 1.0 1.8036e+01 1.8 7.56e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 52397 VecAYPX 490000 1.0 2.4474e+01 2.2 6.46e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 32596 VecScatterBarrie 1760000 1.0 7.9999e+02 6.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 15 0 0 0 0 15 0 0 0 0 0 VecScatterBegin 1760000 1.0 1.1154e+02 3.9 0.00e+00 0.0 1.5e+09 1.4e+03 0.0e+00 3 0100100 0 3 0100100 0 0 VecScatterEnd 1760000 1.0 3.2220e+02 4.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 8 0 0 0 0 8 0 0 0 0 0 MatMult 500000 1.0 7.7645e+02 1.6 1.05e+11 1.1 5.1e+08 2.1e+03 0.0e+00 24 28 33 49 0 24 29 33 49 0 16044 MatMultAdd 420000 1.0 3.0354e+02 1.8 2.40e+10 1.3 2.8e+08 6.7e+02 0.0e+00 10 6 18 9 0 10 6 18 9 0 9077 MatMultTranspose 420000 1.0 2.5492e+02 1.7 2.40e+10 1.3 2.8e+08 6.7e+02 0.0e+00 8 6 18 9 0 8 6 18 9 0 10807 MatSolve 70000 0.0 2.8931e-01 0.0 8.40e+06 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 29 MatSOR 840000 1.0 1.2249e+03 1.7 1.90e+11 1.2 4.7e+08 1.6e+03 1.4e+05 49 51 30 33 38 49 51 30 33 38 18326 MatResidual 420000 1.0 6.4252e+02 1.8 7.97e+10 1.2 4.7e+08 1.6e+03 0.0e+00 19 21 30 33 0 19 21 30 33 0 14518 PCSetUpOnBlocks 70000 1.0 9.7365e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCApply 70000 1.0 2.0260e+03 1.1 3.18e+11 1.2 1.5e+09 1.2e+03 1.4e+05 89 85 97 84 38 89 85 97 84 38 18404 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Krylov Solver 1 9 11424 0. DMKSP interface 1 1 656 0. Vector 4 52 2371888 0. Matrix 0 72 14160468 0. Distributed Mesh 1 1 5248 0. Index Set 2 12 133928 0. IS L to G Mapping 1 1 131728 0. Star Forest Graph 2 2 1728 0. Discrete System 1 1 932 0. Vec Scatter 1 14 233696 0. Preconditioner 1 9 9676 0. Viewer 1 0 0 0. --- Event Stage 1: First Solve Krylov Solver 8 0 0 0. Vector 158 110 3181312 0. Matrix 140 68 21757144 0. Matrix Coarsen 6 6 3816 0. Index Set 110 100 543716 0. Star Forest Graph 12 12 10368 0. Vec Scatter 31 18 22752 0. Preconditioner 8 0 0 0. --- Event Stage 2: Remaining Solves Vector 420000 420000 22766800000 0. ======================================================================================================================== Average time to get PetscTime(): 9.53674e-08 Average time for MPI_Barrier(): 1.80244e-05 Average time for zero size MPI_Send(): 1.40877e-05 #PETSc Option Table entries: -gamg_est_ksp_type cg -iterations 10000 -ksp_norm_type unpreconditioned -ksp_rtol 1E-6 -ksp_type cg -log_sync -log_view -mesh_size 1E-4 -mg_levels_esteig_ksp_max_it 10 -mg_levels_esteig_ksp_type cg -mg_levels_ksp_max_it 1 -mg_levels_ksp_norm_type none -mg_levels_ksp_type richardson -mg_levels_pc_sor_its 1 -mg_levels_pc_type sor -nodes_per_proc 30 -pc_gamg_type classical -pc_type gamg #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: --known-level1-dcache-size=65536 --known-level1-dcache-linesize=64 --known-level1-dcache-assoc=2 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-memcmp-ok=1 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-has-attribute-aligned=1 PETSC_ARCH=arch-linux-amd-opt --download-f2cblaslapack --with-mpi-dir=/cm/shared/apps/mvapich2/intel-17.0.1/2.0 --download-hypre --download-ml --with-fc=0 --with-debugging=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 --with-batch --with-x --known-mpi-shared-libraries=1 --known-64-bit-blas-indices=4 ----------------------------------------- Libraries compiled on 2018-05-25 07:05:14 on node1-001 Machine characteristics: Linux-2.6.32-696.18.7.el6.x86_64-x86_64-with-redhat-6.6-Carbon Using PETSc directory: /home/ritsat/beckerm/petsc Using PETSc arch: arch-linux-amd-opt ----------------------------------------- Using C compiler: /cm/shared/apps/mvapich2/intel-17.0.1/2.0/bin/mpicc -fPIC -wd1572 -O3 ----------------------------------------- Using include paths: -I/home/ritsat/beckerm/petsc/include -I/home/ritsat/beckerm/petsc/arch-linux-amd-opt/include -I/cm/shared/apps/mvapich2/intel-17.0.1/2.0/include ----------------------------------------- Using C linker: /cm/shared/apps/mvapich2/intel-17.0.1/2.0/bin/mpicc Using libraries: -Wl,-rpath,/home/ritsat/beckerm/petsc/arch-linux-amd-opt/lib -L/home/ritsat/beckerm/petsc/arch-linux-amd-opt/lib -lpetsc -Wl,-rpath,/home/ritsat/beckerm/petsc/arch-linux-amd-opt/lib -L/home/ritsat/beckerm/petsc/arch-linux-amd-opt/lib -lHYPRE -lml -lf2clapack -lf2cblas -lX11 -ldl ----------------------------------------- -------------- next part -------------- ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- /home/ritsat/beckerm/ppp_test/plasmapic on a arch-linux-amd-opt named node4-083 with 125 processors, by beckerm Wed May 30 16:24:18 2018 Using Petsc Development GIT revision: v3.9.2-503-g9e88a8b GIT Date: 2018-05-24 08:01:24 -0500 Max Max/Min Avg Total Time (sec): 2.497e+02 1.00000 2.497e+02 Objects: 2.438e+04 1.00004 2.438e+04 Flop: 2.125e+10 1.27708 1.963e+10 2.454e+12 Flop/sec: 8.510e+07 1.27708 7.862e+07 9.828e+09 MPI Messages: 1.042e+06 3.36140 7.129e+05 8.911e+07 MPI Message Lengths: 1.344e+09 2.32209 1.439e+03 1.282e+11 MPI Reductions: 2.250e+04 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flop and VecAXPY() for complex vectors of length N --> 8N flop Summary of Stages: ----- Time ------ ----- Flop ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 6.4457e+00 2.6% 0.0000e+00 0.0% 3.000e+03 0.0% 3.178e+03 0.0% 1.700e+01 0.1% 1: First Solve: 2.4843e+00 1.0% 3.6885e+09 0.2% 3.549e+05 0.4% 3.736e+03 1.0% 5.500e+02 2.4% 2: Remaining Solves: 2.4077e+02 96.4% 2.4504e+12 99.8% 8.875e+07 99.6% 1.430e+03 99.0% 2.192e+04 97.4% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flop: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flop in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flop --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage VecSet 3 1.0 4.7421e-04 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 --- Event Stage 1: First Solve BuildTwoSided 12 1.0 6.9022e-03 1.9 0.00e+00 0.0 8.8e+03 4.0e+00 0.0e+00 0 0 0 0 0 0 0 2 0 0 0 BuildTwoSidedF 30 1.0 2.8585e-01 3.6 0.00e+00 0.0 7.1e+03 1.0e+04 0.0e+00 0 0 0 0 0 6 0 2 5 0 0 KSPSetUp 9 1.0 3.2625e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.8e+01 0 0 0 0 0 0 0 0 0 3 0 KSPSolve 1 1.0 2.4871e+00 1.0 3.26e+07 1.4 3.5e+05 3.7e+03 5.5e+02 1 0 0 1 2 100100100100100 1483 VecTDot 8 1.0 6.3627e-03 3.7 4.32e+05 1.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 0 0 1 0 0 1 8487 VecNorm 6 1.0 1.3714e-03 2.8 3.24e+05 1.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 0 0 1 0 0 1 29532 VecScale 24 1.0 1.4067e-04 2.4 5.43e+04 2.4 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 36596 VecCopy 1 1.0 1.1802e-04 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 115 1.0 7.8726e-04 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 8 1.0 1.1842e-03 1.4 4.32e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 45599 VecAYPX 28 1.0 1.3180e-03 1.6 3.58e+05 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 33275 VecAssemblyBegin 2 1.0 2.1458e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd 2 1.0 2.1458e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 103 1.0 6.1374e-03 2.8 0.00e+00 0.0 8.9e+04 1.4e+03 0.0e+00 0 0 0 0 0 0 0 25 9 0 0 VecScatterEnd 103 1.0 3.9134e-02 3.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 MatMult 29 1.0 3.4050e-02 1.4 6.14e+06 1.2 3.0e+04 2.1e+03 0.0e+00 0 0 0 0 0 1 19 8 5 0 21052 MatMultAdd 24 1.0 1.4260e-02 2.5 1.37e+06 1.6 1.6e+04 6.5e+02 0.0e+00 0 0 0 0 0 0 4 5 1 0 10595 MatMultTranspose 24 1.0 1.4215e-02 2.7 1.37e+06 1.6 1.6e+04 6.5e+02 0.0e+00 0 0 0 0 0 0 4 5 1 0 10629 MatSolve 4 0.0 1.9312e-05 0.0 2.64e+02 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 14 MatSOR 48 1.0 7.1831e-02 1.3 1.09e+07 1.3 2.7e+04 1.5e+03 8.0e+00 0 0 0 0 0 3 34 8 3 1 17455 MatLUFactorSym 1 1.0 5.3883e-05 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatLUFactorNum 1 1.0 1.4067e-05 7.4 1.29e+02 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 9 MatResidual 24 1.0 2.7424e-02 1.7 4.55e+06 1.3 2.7e+04 1.5e+03 0.0e+00 0 0 0 0 0 1 14 8 3 0 18926 MatAssemblyBegin 94 1.0 2.8823e-01 3.2 0.00e+00 0.0 7.1e+03 1.0e+04 0.0e+00 0 0 0 0 0 7 0 2 5 0 0 MatAssemblyEnd 94 1.0 8.1276e-02 1.1 0.00e+00 0.0 6.3e+04 2.1e+02 2.3e+02 0 0 0 0 1 3 0 18 1 42 0 MatGetRow 3102093 1.3 4.5239e-01 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 16 0 0 0 0 0 MatGetRowIJ 1 0.0 5.9605e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatCreateSubMats 6 1.0 4.4482e-01 2.4 0.00e+00 0.0 5.5e+04 1.7e+04 1.2e+01 0 0 0 1 0 13 0 15 71 2 0 MatCreateSubMat 4 1.0 6.8214e-03 1.0 0.00e+00 0.0 2.9e+03 2.7e+02 6.4e+01 0 0 0 0 0 0 0 1 0 12 0 MatGetOrdering 1 0.0 1.1611e-04 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatIncreaseOvrlp 6 1.0 5.6694e-02 1.2 0.00e+00 0.0 2.7e+04 1.0e+03 1.2e+01 0 0 0 0 0 2 0 8 2 2 0 MatCoarsen 6 1.0 1.6149e-02 1.0 0.00e+00 0.0 5.3e+04 5.8e+02 3.3e+01 0 0 0 0 0 1 0 15 2 6 0 MatZeroEntries 6 1.0 3.4871e-03 4.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatPtAP 6 1.0 2.5855e-01 1.0 1.13e+07 1.6 6.3e+04 2.6e+03 9.2e+01 0 0 0 0 0 10 33 18 13 17 4731 MatPtAPSymbolic 6 1.0 1.4759e-01 1.0 0.00e+00 0.0 3.4e+04 2.7e+03 4.2e+01 0 0 0 0 0 6 0 10 7 8 0 MatPtAPNumeric 6 1.0 1.1022e-01 1.0 1.13e+07 1.6 2.9e+04 2.6e+03 4.8e+01 0 0 0 0 0 4 33 8 6 9 11099 MatGetLocalMat 6 1.0 4.5311e-03 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetBrAoCol 6 1.0 8.2998e-03 1.7 0.00e+00 0.0 2.0e+04 3.5e+03 0.0e+00 0 0 0 0 0 0 0 6 5 0 0 SFSetGraph 12 1.0 1.3113e-05 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SFSetUp 12 1.0 1.2064e-02 1.1 0.00e+00 0.0 2.6e+04 6.2e+02 0.0e+00 0 0 0 0 0 0 0 7 1 0 0 SFBcastBegin 45 1.0 2.0254e-03 2.3 0.00e+00 0.0 5.4e+04 6.9e+02 0.0e+00 0 0 0 0 0 0 0 15 3 0 0 SFBcastEnd 45 1.0 5.0657e-03 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 GAMG: createProl 6 1.0 2.0193e+00 1.0 0.00e+00 0.0 2.0e+05 5.2e+03 2.8e+02 1 0 0 1 1 81 0 56 78 52 0 GAMG: partLevel 6 1.0 2.6818e-01 1.0 1.13e+07 1.6 6.6e+04 2.5e+03 1.9e+02 0 0 0 0 1 11 33 19 13 35 4562 repartition 2 1.0 6.9189e-04 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01 0 0 0 0 0 0 0 0 0 2 0 Invert-Sort 2 1.0 6.0797e-04 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 0 0 0 0 0 1 0 Move A 2 1.0 4.2229e-03 1.1 0.00e+00 0.0 1.4e+03 5.3e+02 3.4e+01 0 0 0 0 0 0 0 0 0 6 0 Move P 2 1.0 3.6728e-03 1.1 0.00e+00 0.0 1.4e+03 1.3e+01 3.4e+01 0 0 0 0 0 0 0 0 0 6 0 PCSetUp 2 1.0 2.2978e+00 1.0 1.13e+07 1.6 2.7e+05 4.5e+03 5.1e+02 1 0 0 1 2 92 33 75 90 93 532 PCSetUpOnBlocks 4 1.0 2.5201e-04 1.4 1.29e+02 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1 PCApply 4 1.0 1.1062e-01 1.0 1.82e+07 1.3 8.6e+04 1.2e+03 8.0e+00 0 0 0 0 0 4 56 24 8 1 18759 --- Event Stage 2: Remaining Solves KSPSolve 999 1.0 1.2099e+02 1.0 2.12e+10 1.3 8.8e+07 1.4e+03 2.2e+04 47100 99 97 97 49100 99 98100 20253 VecTDot 7968 1.0 6.5506e+00 4.0 4.30e+08 1.0 0.0e+00 0.0e+00 8.0e+03 1 2 0 0 35 1 2 0 0 36 8210 VecNorm 5982 1.0 5.8032e+0010.2 3.23e+08 1.0 0.0e+00 0.0e+00 6.0e+03 1 2 0 0 27 1 2 0 0 27 6958 VecScale 23904 1.0 1.1292e-01 2.2 5.40e+07 2.4 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 45407 VecCopy 999 1.0 1.1493e-01 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 83664 1.0 6.9683e-01 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 7968 1.0 1.1667e+00 1.4 4.30e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 46100 VecAYPX 27888 1.0 1.2901e+00 1.6 3.56e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 33851 VecScatterBegin 100599 1.0 5.9735e+00 2.8 0.00e+00 0.0 8.8e+07 1.4e+03 0.0e+00 2 0 99 97 0 2 0 99 98 0 0 VecScatterEnd 100599 1.0 3.5510e+01 4.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 7 0 0 0 0 8 0 0 0 0 0 MatMult 28887 1.0 3.4026e+01 1.5 6.12e+09 1.2 3.0e+07 2.1e+03 0.0e+00 11 29 33 49 0 11 29 33 49 0 20986 MatMultAdd 23904 1.0 1.1303e+01 2.1 1.37e+09 1.6 1.6e+07 6.5e+02 0.0e+00 3 6 18 8 0 3 6 18 8 0 13314 MatMultTranspose 23904 1.0 1.3880e+01 2.6 1.37e+09 1.6 1.6e+07 6.5e+02 0.0e+00 4 6 18 8 0 4 6 18 8 0 10842 MatSolve 3984 0.0 1.6525e-02 0.0 2.63e+05 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 16 MatSOR 47808 1.0 6.6732e+01 1.3 1.08e+10 1.3 2.7e+07 1.5e+03 8.0e+03 25 51 30 32 35 26 51 30 32 36 18638 MatResidual 23904 1.0 2.6673e+01 1.7 4.54e+09 1.3 2.7e+07 1.5e+03 0.0e+00 8 21 30 32 0 8 21 30 32 0 19381 PCSetUpOnBlocks 3984 1.0 5.2946e-03 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCApply 3984 1.0 1.0349e+02 1.0 1.81e+10 1.3 8.5e+07 1.2e+03 8.0e+03 41 84 96 80 35 42 84 96 81 36 19921 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Krylov Solver 1 9 11424 0. DMKSP interface 1 0 0 0. Vector 5 52 2371496 0. Matrix 0 72 14138216 0. Distributed Mesh 1 0 0 0. Index Set 2 12 133768 0. IS L to G Mapping 1 0 0 0. Star Forest Graph 2 0 0 0. Discrete System 1 0 0 0. Vec Scatter 1 13 16432 0. Preconditioner 1 9 9676 0. Viewer 1 0 0 0. --- Event Stage 1: First Solve Krylov Solver 8 0 0 0. Vector 140 92 2204792 0. Matrix 140 68 21738552 0. Matrix Coarsen 6 6 3816 0. Index Set 110 100 543240 0. Star Forest Graph 12 12 10368 0. Vec Scatter 31 18 22752 0. Preconditioner 8 0 0 0. --- Event Stage 2: Remaining Solves Vector 23904 23904 1295501184 0. ======================================================================================================================== Average time to get PetscTime(): 9.53674e-08 Average time for MPI_Barrier(): 1.81675e-05 Average time for zero size MPI_Send(): 1.45779e-05 #PETSc Option Table entries: -gamg_est_ksp_type cg -ksp_norm_type unpreconditioned -ksp_type cg -log_view -mg_levels_esteig_ksp_max_it 10 -mg_levels_esteig_ksp_type cg -mg_levels_ksp_max_it 1 -mg_levels_ksp_norm_type none -mg_levels_ksp_type richardson -mg_levels_pc_sor_its 1 -mg_levels_pc_type sor -pc_gamg_type classical -pc_type gamg #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: --known-level1-dcache-size=65536 --known-level1-dcache-linesize=64 --known-level1-dcache-assoc=2 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-memcmp-ok=1 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-has-attribute-aligned=1 PETSC_ARCH=arch-linux-amd-opt --download-f2cblaslapack --with-mpi-dir=/cm/shared/apps/mvapich2/intel-17.0.1/2.0 --download-hypre --download-ml --with-fc=0 --with-debugging=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 --with-batch --with-x --known-mpi-shared-libraries=1 --known-64-bit-blas-indices=4 ----------------------------------------- Libraries compiled on 2018-05-25 07:05:14 on node1-001 Machine characteristics: Linux-2.6.32-696.18.7.el6.x86_64-x86_64-with-redhat-6.6-Carbon Using PETSc directory: /home/ritsat/beckerm/petsc Using PETSc arch: arch-linux-amd-opt ----------------------------------------- Using C compiler: /cm/shared/apps/mvapich2/intel-17.0.1/2.0/bin/mpicc -fPIC -wd1572 -O3 ----------------------------------------- Using include paths: -I/home/ritsat/beckerm/petsc/include -I/home/ritsat/beckerm/petsc/arch-linux-amd-opt/include -I/cm/shared/apps/mvapich2/intel-17.0.1/2.0/include ----------------------------------------- Using C linker: /cm/shared/apps/mvapich2/intel-17.0.1/2.0/bin/mpicc Using libraries: -Wl,-rpath,/home/ritsat/beckerm/petsc/arch-linux-amd-opt/lib -L/home/ritsat/beckerm/petsc/arch-linux-amd-opt/lib -lpetsc -Wl,-rpath,/home/ritsat/beckerm/petsc/arch-linux-amd-opt/lib -L/home/ritsat/beckerm/petsc/arch-linux-amd-opt/lib -lHYPRE -lml -lf2clapack -lf2cblas -lX11 -ldl ----------------------------------------- -------------- next part -------------- ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- /home/ritsat/beckerm/ppp_test/plasmapic on a arch-linux-amd-opt named node1-013 with 1000 processors, by beckerm Wed May 30 19:06:34 2018 Using Petsc Development GIT revision: v3.9.2-503-g9e88a8b GIT Date: 2018-05-24 08:01:24 -0500 Max Max/Min Avg Total Time (sec): 2.892e+02 1.00001 2.892e+02 Objects: 2.440e+04 1.00004 2.440e+04 Flop: 2.124e+10 1.27708 2.041e+10 2.041e+13 Flop/sec: 7.342e+07 1.27708 7.057e+07 7.057e+10 MPI Messages: 1.238e+06 3.99536 8.489e+05 8.489e+08 MPI Message Lengths: 1.343e+09 2.32238 1.393e+03 1.183e+12 MPI Reductions: 2.256e+04 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flop and VecAXPY() for complex vectors of length N --> 8N flop Summary of Stages: ----- Time ------ ----- Flop ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 2.7802e+01 9.6% 0.0000e+00 0.0% 2.700e+04 0.0% 3.178e+03 0.0% 1.700e+01 0.1% 1: First Solve: 3.3072e+00 1.1% 3.0885e+10 0.2% 3.675e+06 0.4% 3.508e+03 1.1% 6.220e+02 2.8% 2: Remaining Solves: 2.5814e+02 89.2% 2.0380e+13 99.8% 8.452e+08 99.6% 1.384e+03 98.9% 2.191e+04 97.1% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flop: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flop in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flop --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage VecSet 3 1.0 5.7411e-04 3.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 --- Event Stage 1: First Solve BuildTwoSided 12 1.0 1.3082e-02 1.5 0.00e+00 0.0 8.9e+04 4.0e+00 0.0e+00 0 0 0 0 0 0 0 2 0 0 0 BuildTwoSidedF 30 1.0 4.3104e-01 2.7 0.00e+00 0.0 6.5e+04 1.0e+04 0.0e+00 0 0 0 0 0 8 0 2 5 0 0 KSPSetUp 9 1.0 5.3465e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.8e+01 0 0 0 0 0 0 0 0 0 3 0 KSPSolve 1 1.0 3.3113e+00 1.0 3.25e+07 1.4 3.7e+06 3.5e+03 6.2e+02 1 0 0 1 3 100100100100100 9327 VecTDot 8 1.0 6.8474e-03 4.2 4.32e+05 1.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 0 0 1 0 0 1 63089 VecNorm 6 1.0 1.6654e-03 2.8 3.24e+05 1.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 0 0 1 0 0 1 194553 VecScale 24 1.0 1.3828e-04 2.4 5.43e+04 2.4 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 344362 VecCopy 1 1.0 1.4305e-04 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 124 1.0 8.4376e-04 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 8 1.0 1.2994e-03 1.6 4.32e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 332466 VecAYPX 28 1.0 1.5504e-03 2.0 3.58e+05 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 228452 VecAssemblyBegin 3 1.0 3.0994e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd 3 1.0 3.0994e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 108 1.0 6.7146e-03 2.8 0.00e+00 0.0 8.4e+05 1.4e+03 0.0e+00 0 0 0 0 0 0 0 23 9 0 0 VecScatterEnd 108 1.0 4.8888e-02 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 MatMult 29 1.0 3.6063e-02 1.5 6.14e+06 1.2 2.8e+05 2.0e+03 0.0e+00 0 0 0 0 0 1 19 8 4 0 164544 MatMultAdd 24 1.0 2.2712e-02 3.2 1.37e+06 1.6 1.5e+05 6.5e+02 0.0e+00 0 0 0 0 0 1 4 4 1 0 56747 MatMultTranspose 24 1.0 1.4198e-02 2.3 1.37e+06 1.6 1.5e+05 6.5e+02 0.0e+00 0 0 0 0 0 0 4 4 1 0 90779 MatSolve 4 0.0 4.7922e-05 0.0 1.10e+04 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 229 MatSOR 48 1.0 7.6652e-02 1.3 1.09e+07 1.3 2.6e+05 1.5e+03 8.0e+00 0 0 0 0 0 2 34 7 3 1 136439 MatLUFactorSym 1 1.0 9.4891e-05 3.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatLUFactorNum 1 1.0 6.6042e-0534.6 3.29e+04 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 498 MatResidual 24 1.0 2.7970e-02 1.7 4.55e+06 1.3 2.6e+05 1.5e+03 0.0e+00 0 0 0 0 0 1 14 7 3 0 155507 MatAssemblyBegin 102 1.0 4.3351e-01 2.5 0.00e+00 0.0 6.5e+04 1.0e+04 0.0e+00 0 0 0 0 0 8 0 2 5 0 0 MatAssemblyEnd 102 1.0 1.1318e-01 1.1 0.00e+00 0.0 6.2e+05 2.0e+02 2.5e+02 0 0 0 0 1 3 0 17 1 40 0 MatGetRow 3102093 1.3 5.0438e-01 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 12 0 0 0 0 0 MatGetRowIJ 1 0.0 1.5974e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatCreateSubMats 6 1.0 4.6014e-01 2.3 0.00e+00 0.0 5.7e+05 1.6e+04 1.2e+01 0 0 0 1 0 10 0 15 72 2 0 MatCreateSubMat 6 1.0 1.9575e-02 1.0 0.00e+00 0.0 2.2e+04 3.3e+02 9.4e+01 0 0 0 0 0 1 0 1 0 15 0 MatGetOrdering 1 0.0 1.4710e-04 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatIncreaseOvrlp 6 1.0 1.1349e-01 1.1 0.00e+00 0.0 2.6e+05 9.9e+02 1.2e+01 0 0 0 0 0 3 0 7 2 2 0 MatCoarsen 6 1.0 3.4475e-02 1.1 0.00e+00 0.0 7.1e+05 4.4e+02 5.6e+01 0 0 0 0 0 1 0 19 2 9 0 MatZeroEntries 6 1.0 3.4661e-03 4.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatPtAP 6 1.0 3.8021e-01 1.1 1.11e+07 1.6 6.3e+05 2.5e+03 9.2e+01 0 0 0 0 0 11 34 17 12 15 27282 MatPtAPSymbolic 6 1.0 2.1572e-01 1.0 0.00e+00 0.0 3.2e+05 2.7e+03 4.2e+01 0 0 0 0 0 7 0 9 7 7 0 MatPtAPNumeric 6 1.0 1.4471e-01 1.0 1.11e+07 1.6 3.0e+05 2.3e+03 4.8e+01 0 0 0 0 0 4 34 8 6 8 71679 MatGetLocalMat 6 1.0 4.7863e-03 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetBrAoCol 6 1.0 1.9191e-02 3.1 0.00e+00 0.0 1.9e+05 3.4e+03 0.0e+00 0 0 0 0 0 0 0 5 5 0 0 SFSetGraph 12 1.0 2.3127e-05 6.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SFSetUp 12 1.0 1.9232e-02 1.1 0.00e+00 0.0 2.7e+05 5.8e+02 0.0e+00 0 0 0 0 0 1 0 7 1 0 0 SFBcastBegin 68 1.0 2.7168e-03 2.7 0.00e+00 0.0 7.2e+05 5.1e+02 0.0e+00 0 0 0 0 0 0 0 20 3 0 0 SFBcastEnd 68 1.0 1.5905e-02 3.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 GAMG: createProl 6 1.0 2.4556e+00 1.0 0.00e+00 0.0 2.2e+06 4.7e+03 3.1e+02 1 0 0 1 1 74 0 59 79 50 0 GAMG: partLevel 6 1.0 4.0791e-01 1.1 1.11e+07 1.6 6.5e+05 2.4e+03 2.4e+02 0 0 0 0 1 12 34 18 12 39 25428 repartition 3 1.0 2.3220e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.8e+01 0 0 0 0 0 0 0 0 0 3 0 Invert-Sort 3 1.0 2.0869e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01 0 0 0 0 0 0 0 0 0 2 0 Move A 3 1.0 1.2217e-02 1.1 0.00e+00 0.0 9.5e+03 7.4e+02 5.0e+01 0 0 0 0 0 0 0 0 0 8 0 Move P 3 1.0 9.7649e-03 1.1 0.00e+00 0.0 1.3e+04 1.3e+01 5.0e+01 0 0 0 0 0 0 0 0 0 8 0 PCSetUp 2 1.0 3.0642e+00 1.0 1.11e+07 1.6 2.8e+06 4.2e+03 5.8e+02 1 0 0 1 3 93 34 77 91 94 3385 PCSetUpOnBlocks 4 1.0 3.9506e-04 2.9 3.29e+04 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 83 PCApply 4 1.0 1.2272e-01 1.0 1.82e+07 1.3 8.2e+05 1.2e+03 8.0e+00 0 0 0 0 0 4 56 22 7 1 141671 --- Event Stage 2: Remaining Solves KSPSolve 999 1.0 1.3648e+02 1.0 2.12e+10 1.3 8.4e+08 1.4e+03 2.2e+04 46100 99 97 97 51100 99 98100 149327 VecTDot 7964 1.0 7.1750e+00 3.4 4.30e+08 1.0 0.0e+00 0.0e+00 8.0e+03 1 2 0 0 35 1 2 0 0 36 59937 VecNorm 5980 1.0 6.0258e+00 4.9 3.23e+08 1.0 0.0e+00 0.0e+00 6.0e+03 1 2 0 0 27 1 2 0 0 27 53589 VecScale 23892 1.0 1.2695e-01 2.4 5.40e+07 2.4 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 373403 VecCopy 999 1.0 1.3586e-01 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 83622 1.0 7.4253e-01 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 7964 1.0 1.2791e+00 1.6 4.30e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 336221 VecAYPX 27874 1.0 1.4993e+00 2.0 3.56e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 235111 VecScatterBegin 100549 1.0 6.4721e+00 2.9 0.00e+00 0.0 8.4e+08 1.4e+03 0.0e+00 2 0 99 97 0 2 0 99 98 0 0 VecScatterEnd 100549 1.0 4.9949e+01 2.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 8 0 0 0 0 9 0 0 0 0 0 MatMult 28873 1.0 3.3321e+01 1.4 6.11e+09 1.2 2.8e+08 2.0e+03 0.0e+00 9 29 33 48 0 10 29 34 49 0 177317 MatMultAdd 23892 1.0 1.9948e+01 2.7 1.37e+09 1.6 1.5e+08 6.5e+02 0.0e+00 6 6 18 8 0 7 6 18 8 0 64321 MatMultTranspose 23892 1.0 1.3810e+01 2.1 1.37e+09 1.6 1.5e+08 6.5e+02 0.0e+00 3 6 18 8 0 3 6 18 8 0 92909 MatSolve 3982 0.0 4.4663e-02 0.0 1.09e+07 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 245 MatSOR 47784 1.0 7.1050e+01 1.3 1.08e+10 1.3 2.6e+08 1.5e+03 8.0e+03 23 51 30 32 35 26 51 30 32 36 145956 MatResidual 23892 1.0 2.4668e+01 1.5 4.53e+09 1.3 2.6e+08 1.5e+03 0.0e+00 7 21 30 32 0 7 21 30 32 0 175528 PCSetUpOnBlocks 3982 1.0 5.2893e-03 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCApply 3982 1.0 1.1642e+02 1.0 1.81e+10 1.3 8.1e+08 1.2e+03 8.0e+03 40 85 96 81 35 45 85 96 82 36 148313 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Krylov Solver 1 9 11424 0. DMKSP interface 1 0 0 0. Vector 5 52 2382208 0. Matrix 0 65 14780672 0. Distributed Mesh 1 0 0 0. Index Set 2 18 171852 0. IS L to G Mapping 1 0 0 0. Star Forest Graph 2 0 0 0. Discrete System 1 0 0 0. Vec Scatter 1 13 16432 0. Preconditioner 1 9 9676 0. Viewer 1 0 0 0. --- Event Stage 1: First Solve Krylov Solver 8 0 0 0. Vector 152 104 2238504 0. Matrix 148 83 22951356 0. Matrix Coarsen 6 6 3816 0. Index Set 128 112 590828 0. Star Forest Graph 12 12 10368 0. Vec Scatter 34 21 26544 0. Preconditioner 8 0 0 0. --- Event Stage 2: Remaining Solves Vector 23892 23892 1302241424 0. ======================================================================================================================== Average time to get PetscTime(): 9.53674e-08 Average time for MPI_Barrier(): 3.45707e-05 Average time for zero size MPI_Send(): 1.60329e-05 #PETSc Option Table entries: -gamg_est_ksp_type cg -ksp_norm_type unpreconditioned -ksp_type cg -log_view -mg_levels_esteig_ksp_max_it 10 -mg_levels_esteig_ksp_type cg -mg_levels_ksp_max_it 1 -mg_levels_ksp_norm_type none -mg_levels_ksp_type richardson -mg_levels_pc_sor_its 1 -mg_levels_pc_type sor -pc_gamg_type classical -pc_type gamg #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: --known-level1-dcache-size=65536 --known-level1-dcache-linesize=64 --known-level1-dcache-assoc=2 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-memcmp-ok=1 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-has-attribute-aligned=1 PETSC_ARCH=arch-linux-amd-opt --download-f2cblaslapack --with-mpi-dir=/cm/shared/apps/mvapich2/intel-17.0.1/2.0 --download-hypre --download-ml --with-fc=0 --with-debugging=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 --with-batch --with-x --known-mpi-shared-libraries=1 --known-64-bit-blas-indices=4 ----------------------------------------- Libraries compiled on 2018-05-25 07:05:14 on node1-001 Machine characteristics: Linux-2.6.32-696.18.7.el6.x86_64-x86_64-with-redhat-6.6-Carbon Using PETSc directory: /home/ritsat/beckerm/petsc Using PETSc arch: arch-linux-amd-opt ----------------------------------------- Using C compiler: /cm/shared/apps/mvapich2/intel-17.0.1/2.0/bin/mpicc -fPIC -wd1572 -O3 ----------------------------------------- Using include paths: -I/home/ritsat/beckerm/petsc/include -I/home/ritsat/beckerm/petsc/arch-linux-amd-opt/include -I/cm/shared/apps/mvapich2/intel-17.0.1/2.0/include ----------------------------------------- Using C linker: /cm/shared/apps/mvapich2/intel-17.0.1/2.0/bin/mpicc Using libraries: -Wl,-rpath,/home/ritsat/beckerm/petsc/arch-linux-amd-opt/lib -L/home/ritsat/beckerm/petsc/arch-linux-amd-opt/lib -lpetsc -Wl,-rpath,/home/ritsat/beckerm/petsc/arch-linux-amd-opt/lib -L/home/ritsat/beckerm/petsc/arch-linux-amd-opt/lib -lHYPRE -lml -lf2clapack -lf2cblas -lX11 -ldl ----------------------------------------- -------------- next part -------------- A non-text attachment was scrubbed... Name: wstest.zip Type: application/zip Size: 5833 bytes Desc: not available URL: From jczhang at mcs.anl.gov Mon Jun 4 14:56:10 2018 From: jczhang at mcs.anl.gov (Junchao Zhang) Date: Mon, 4 Jun 2018 14:56:10 -0500 Subject: [petsc-users] Poor weak scaling when solving successive linearsystems In-Reply-To: References: <6d9b6488-3957-0d74-148a-323c78d42f6a@physik.uni-giessen.de> <079D8B4D-C57B-4054-B9BD-05644E52C3A4@anl.gov> Message-ID: Miachael, I can compile and run you test. I am now profiling it. Thanks. --Junchao Zhang On Mon, Jun 4, 2018 at 11:59 AM, Michael Becker < Michael.Becker at physik.uni-giessen.de> wrote: > Hello again, > this took me longer than I anticipated, but here we go. > I did reruns of the cases where only half the processes per node were used > (without -log_sync): > > 125 procs,1st 125 procs,2nd 1000 > procs,1st 1000 procs,2nd > Max Ratio Max Ratio Max > Ratio Max Ratio > KSPSolve 1.203E+02 1.0 1.210E+02 1.0 > 1.399E+02 1.1 1.365E+02 1.0 > VecTDot 6.376E+00 3.7 6.551E+00 4.0 > 7.885E+00 2.9 7.175E+00 3.4 > VecNorm 4.579E+00 7.1 5.803E+00 10.2 > 8.534E+00 6.9 6.026E+00 4.9 > VecScale 1.070E-01 2.1 1.129E-01 2.2 > 1.301E-01 2.5 1.270E-01 2.4 > VecCopy 1.123E-01 1.3 1.149E-01 1.3 > 1.301E-01 1.6 1.359E-01 1.6 > VecSet 7.063E-01 1.7 6.968E-01 1.7 > 7.432E-01 1.8 7.425E-01 1.8 > VecAXPY 1.166E+00 1.4 1.167E+00 1.4 > 1.221E+00 1.5 1.279E+00 1.6 > VecAYPX 1.317E+00 1.6 1.290E+00 1.6 > 1.536E+00 1.9 1.499E+00 2.0 > VecScatterBegin 6.142E+00 3.2 5.974E+00 2.8 > 6.448E+00 3.0 6.472E+00 2.9 > VecScatterEnd 3.606E+01 4.2 3.551E+01 4.0 > 5.244E+01 2.7 4.995E+01 2.7 > MatMult 3.561E+01 1.6 3.403E+01 1.5 > 3.435E+01 1.4 3.332E+01 1.4 > MatMultAdd 1.124E+01 2.0 1.130E+01 2.1 > 2.093E+01 2.9 1.995E+01 2.7 > MatMultTranspose 1.372E+01 2.5 1.388E+01 2.6 > 1.477E+01 2.2 1.381E+01 2.1 > MatSolve 1.949E-02 0.0 1.653E-02 0.0 > 4.789E-02 0.0 4.466E-02 0.0 > MatSOR 6.610E+01 1.3 6.673E+01 1.3 > 7.111E+01 1.3 7.105E+01 1.3 > MatResidual 2.647E+01 1.7 2.667E+01 1.7 > 2.446E+01 1.4 2.467E+01 1.5 > PCSetUpOnBlocks 5.266E-03 1.4 5.295E-03 1.4 > 5.427E-03 1.5 5.289E-03 1.4 > PCApply 1.031E+02 1.0 1.035E+02 1.0 > 1.180E+02 1.0 1.164E+02 1.0 > > I also slimmed down my code and basically wrote a simple weak scaling test > (source files attached) so you can profile it yourself. I appreciate the > offer Junchao, thank you. > You can adjust the system size per processor at runtime via > "-nodes_per_proc 30" and the number of repeated calls to the function > containing KSPsolve() via "-iterations 1000". The physical problem is > simply calculating the electric potential from a homogeneous charge > distribution, done multiple times to accumulate time in KSPsolve(). > A job would be started using something like > > mpirun -n 125 ~/petsc_ws/ws_test -nodes_per_proc 30 -mesh_size 1E-4 > -iterations 1000 \\ > -ksp_rtol 1E-6 \ > -log_view -log_sync\ > -pc_type gamg -pc_gamg_type classical\ > -ksp_type cg \ > -ksp_norm_type unpreconditioned \ > -mg_levels_ksp_type richardson \ > -mg_levels_ksp_norm_type none \ > -mg_levels_pc_type sor \ > -mg_levels_ksp_max_it 1 \ > -mg_levels_pc_sor_its 1 \ > -mg_levels_esteig_ksp_type cg \ > -mg_levels_esteig_ksp_max_it 10 \ > -gamg_est_ksp_type cg > > , ideally started on a cube number of processes for a cubical process grid. > Using 125 processes and 10.000 iterations I get the output in > "log_view_125_new.txt", which shows the same imbalance for me. > > Michael > > > Am 02.06.2018 um 13:40 schrieb Mark Adams: > > > > On Fri, Jun 1, 2018 at 11:20 PM, Junchao Zhang > wrote: > >> Hi,Michael, >> You can add -log_sync besides -log_view, which adds barriers to certain >> events but measures barrier time separately from the events. I find this >> option makes it easier to interpret log_view output. >> > > That is great (good to know). > > This should give us a better idea if your large VecScatter costs are from > slow communication or if it catching some sort of load imbalance. > > >> >> --Junchao Zhang >> >> On Wed, May 30, 2018 at 3:27 AM, Michael Becker < >> Michael.Becker at physik.uni-giessen.de> wrote: >> >>> Barry: On its way. Could take a couple days again. >>> >>> Junchao: I unfortunately don't have access to a cluster with a faster >>> network. This one has a mixed 4X QDR-FDR InfiniBand 2:1 blocking fat-tree >>> network, which I realize causes parallel slowdown if the nodes are not >>> connected to the same switch. Each node has 24 processors (2x12/socket) and >>> four NUMA domains (two for each socket). >>> The ranks are usually not distributed perfectly even, i.e. for 125 >>> processes, of the six required nodes, five would use 21 cores and one 20. >>> Would using another CPU type make a difference communication-wise? I >>> could switch to faster ones (on the same network), but I always assumed >>> this would only improve performance of the stuff that is unrelated to >>> communication. >>> >>> Michael >>> >>> >>> >>> The log files have something like "Average time for zero size >>> MPI_Send(): 1.84231e-05". It looks you ran on a cluster with a very slow >>> network. A typical machine should give less than 1/10 of the latency you >>> have. An easy way to try is just running the code on a machine with a >>> faster network and see what happens. >>> >>> Also, how many cores & numa domains does a compute node have? I could >>> not figure out how you distributed the 125 MPI ranks evenly. >>> >>> --Junchao Zhang >>> >>> On Tue, May 29, 2018 at 6:18 AM, Michael Becker < >>> Michael.Becker at physik.uni-giessen.de> wrote: >>> >>>> Hello again, >>>> >>>> here are the updated log_view files for 125 and 1000 processors. I ran >>>> both problems twice, the first time with all processors per node allocated >>>> ("-1.txt"), the second with only half on twice the number of nodes >>>> ("-2.txt"). >>>> >>>> On May 24, 2018, at 12:24 AM, Michael Becker wrote: >>>> >>>> I noticed that for every individual KSP iteration, six vector objects are created and destroyed (with CG, more with e.g. GMRES). >>>> >>>> Hmm, it is certainly not intended at vectors be created and destroyed within each KSPSolve() could you please point us to the code that makes you think they are being created and destroyed? We create all the work vectors at KSPSetUp() and destroy them in KSPReset() not during the solve. Not that this would be a measurable distance. >>>> >>>> >>>> I mean this, right in the log_view output: >>>> >>>> Memory usage is given in bytes: >>>> >>>> Object Type Creations Destructions Memory Descendants' Mem. >>>> Reports information only for process 0. >>>> >>>> --- Event Stage 0: Main Stage >>>> >>>> ... >>>> >>>> --- Event Stage 1: First Solve >>>> >>>> ... >>>> >>>> --- Event Stage 2: Remaining Solves >>>> >>>> Vector 23904 23904 1295501184 0. >>>> >>>> I logged the exact number of KSP iterations over the 999 timesteps and >>>> its exactly 23904/6 = 3984. >>>> >>>> Michael >>>> >>>> >>>> >>>> Am 24.05.2018 um 19:50 schrieb Smith, Barry F.: >>>> >>>> Please send the log file for 1000 with cg as the solver. >>>> >>>> You should make a bar chart of each event for the two cases to see which ones are taking more time and which are taking less (we cannot tell with the two logs you sent us since they are for different solvers.) >>>> >>>> >>>> >>>> >>>> On May 24, 2018, at 12:24 AM, Michael Becker wrote: >>>> >>>> I noticed that for every individual KSP iteration, six vector objects are created and destroyed (with CG, more with e.g. GMRES). >>>> >>>> Hmm, it is certainly not intended at vectors be created and destroyed within each KSPSolve() could you please point us to the code that makes you think they are being created and destroyed? We create all the work vectors at KSPSetUp() and destroy them in KSPReset() not during the solve. Not that this would be a measurable distance. >>>> >>>> >>>> >>>> >>>> This seems kind of wasteful, is this supposed to be like this? Is this even the reason for my problems? Apart from that, everything seems quite normal to me (but I'm not the expert here). >>>> >>>> >>>> Thanks in advance. >>>> >>>> Michael >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>> >>> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fande.kong at inl.gov Mon Jun 4 15:27:53 2018 From: fande.kong at inl.gov (Kong, Fande) Date: Mon, 4 Jun 2018 14:27:53 -0600 Subject: [petsc-users] "snes_type test" is gone? Message-ID: Hi PETSc Team, I was wondering if "snes_type test" has been gone? Quite a few MOOSE users use this option to test their Jacobian matrices. If it is gone, any reason? Fande, -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexlindsay239 at gmail.com Mon Jun 4 16:21:06 2018 From: alexlindsay239 at gmail.com (Alexander Lindsay) Date: Mon, 4 Jun 2018 15:21:06 -0600 Subject: [petsc-users] "snes_type test" is gone? In-Reply-To: References: Message-ID: Looks like `-snes_test_jacobian` and `-snes_test_jacobian_view` are the options to use... On Mon, Jun 4, 2018 at 2:27 PM, Kong, Fande wrote: > Hi PETSc Team, > > I was wondering if "snes_type test" has been gone? Quite a few MOOSE users > use this option to test their Jacobian matrices. > > If it is gone, any reason? > > Fande, > -------------- next part -------------- An HTML attachment was scrubbed... URL: From overholt at capesim.com Mon Jun 4 16:32:19 2018 From: overholt at capesim.com (Matthew Overholt) Date: Mon, 4 Jun 2018 17:32:19 -0400 Subject: [petsc-users] MKL Pardiso Solver Execution Step control Message-ID: Hello, I am using KSP in KSPPREONLY mode to do a direct solve on an A*x = b system, with solver algorithms MUMPS, CPardiso and Pardiso. For Pardiso, is it possible to control the solver execution step (denoted "phase" in Intel's docs)? I would like to be able to control when it refactors as one can when calling it directly. https://software.intel.com/en-us/mkl-developer-reference-c-pardiso If so, please give me the details, since I can't see how to do it from the MATSOLVERMKL_PARDISO docs page (which is clearly listing the iparm[] flags). Thank you. Matt Overholt CapeSym, Inc. (508) 653-7100 x204 overholt at capesim.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From hongzhang at anl.gov Mon Jun 4 16:59:15 2018 From: hongzhang at anl.gov (Zhang, Hong) Date: Mon, 4 Jun 2018 21:59:15 +0000 Subject: [petsc-users] "snes_type test" is gone? In-Reply-To: References: Message-ID: <9F7A97FC-EBC7-4F20-ADA6-4D03B145DFB0@anl.gov> -snes_type has been removed. We can just use -snes_test_jacobian instead. Note that the test is done every time the Jacobian is computed. Hong (Mr.) > On Jun 4, 2018, at 3:27 PM, Kong, Fande wrote: > > Hi PETSc Team, > > I was wondering if "snes_type test" has been gone? Quite a few MOOSE users use this option to test their Jacobian matrices. > > If it is gone, any reason? > > Fande, From hongzhang at anl.gov Mon Jun 4 17:01:08 2018 From: hongzhang at anl.gov (Zhang, Hong) Date: Mon, 4 Jun 2018 22:01:08 +0000 Subject: [petsc-users] "snes_type test" is gone? In-Reply-To: <9F7A97FC-EBC7-4F20-ADA6-4D03B145DFB0@anl.gov> References: <9F7A97FC-EBC7-4F20-ADA6-4D03B145DFB0@anl.gov> Message-ID: > On Jun 4, 2018, at 4:59 PM, Zhang, Hong wrote: > > -snes_type has been removed. We can just use -snes_test_jacobian instead. Note that the test is done every time the Jacobian is computed. It was meant to be "-snes_type test". > Hong (Mr.) > >> On Jun 4, 2018, at 3:27 PM, Kong, Fande wrote: >> >> Hi PETSc Team, >> >> I was wondering if "snes_type test" has been gone? Quite a few MOOSE users use this option to test their Jacobian matrices. >> >> If it is gone, any reason? >> >> Fande, > From fande.kong at inl.gov Mon Jun 4 17:26:54 2018 From: fande.kong at inl.gov (Kong, Fande) Date: Mon, 4 Jun 2018 16:26:54 -0600 Subject: [petsc-users] "snes_type test" is gone? In-Reply-To: References: <9F7A97FC-EBC7-4F20-ADA6-4D03B145DFB0@anl.gov> Message-ID: Thanks, Hong, I see. It is better if "-snes_type test" did not exist at the first place. Fande, On Mon, Jun 4, 2018 at 4:01 PM, Zhang, Hong wrote: > > > > On Jun 4, 2018, at 4:59 PM, Zhang, Hong wrote: > > > > -snes_type has been removed. We can just use -snes_test_jacobian > instead. Note that the test is done every time the Jacobian is computed. > > It was meant to be "-snes_type test". > > > Hong (Mr.) > > > >> On Jun 4, 2018, at 3:27 PM, Kong, Fande wrote: > >> > >> Hi PETSc Team, > >> > >> I was wondering if "snes_type test" has been gone? Quite a few MOOSE > users use this option to test their Jacobian matrices. > >> > >> If it is gone, any reason? > >> > >> Fande, > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jun 5 04:03:25 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 5 Jun 2018 09:03:25 +0000 Subject: [petsc-users] MKL Pardiso Solver Execution Step control In-Reply-To: References: Message-ID: <48CE4D2E-D6DB-43E8-A57A-7D4DC801D2AE@anl.gov> > On Jun 4, 2018, at 10:32 PM, Matthew Overholt wrote: > > Hello, > > I am using KSP in KSPPREONLY mode to do a direct solve on an A*x = b system, with solver algorithms MUMPS, CPardiso and Pardiso. For Pardiso, is it possible to control the solver execution step (denoted "phase" in Intel's docs)? I would like to be able to control when it refactors as one can when calling it directly. Matt, This is handled automatically by PETSc. If the matrix entries change than PETSc will automatically call the correct code to perform a new numerical factorization; if the matrix entries are not changed then it will not refactor the matrix. I am not sure if it makes sense for a user to be setting different phase values since it is handled automatically. Could you explain your exact use case where you would like to control the phase variable directly? Barry > > https://software.intel.com/en-us/mkl-developer-reference-c-pardiso > > If so, please give me the details, since I can't see how to do it from the MATSOLVERMKL_PARDISO docs page (which is clearly listing the iparm[] flags). > > Thank you. > > Matt Overholt > CapeSym, Inc. > (508) 653-7100 x204 > overholt at capesim.com > From overholt at capesim.com Tue Jun 5 09:04:18 2018 From: overholt at capesim.com (Matthew Overholt) Date: Tue, 5 Jun 2018 10:04:18 -0400 Subject: [petsc-users] MKL Pardiso Solver Execution Step control In-Reply-To: <48CE4D2E-D6DB-43E8-A57A-7D4DC801D2AE@anl.gov> References: <48CE4D2E-D6DB-43E8-A57A-7D4DC801D2AE@anl.gov> Message-ID: Barry, What I should have said was that I wanted to control when it does the "Analysis" (phase 11), not Factoring (phase 22). We are solving the heat conduction equation, which makes for a nearly linear system; the non-linearity only enters through the material properties dependence on temperature, which is usually weak. We use the direct approach with several fixed point iterations to get the solution. We have found (in our non-PETSc code) that a lot of time can be saved, particularly when unsteady, by judiciously choosing when to (re)do the Analysis. Typically we do the Analysis on output time steps when unsteady, and just for the first fixed point iteration when steady. Matt... On Tue, Jun 5, 2018 at 5:03 AM, Smith, Barry F. wrote: > > > > On Jun 4, 2018, at 10:32 PM, Matthew Overholt > wrote: > > > > Hello, > > > > I am using KSP in KSPPREONLY mode to do a direct solve on an A*x = b > system, with solver algorithms MUMPS, CPardiso and Pardiso. For Pardiso, > is it possible to control the solver execution step (denoted "phase" in > Intel's docs)? I would like to be able to control when it refactors as one > can when calling it directly. > > Matt, > > This is handled automatically by PETSc. If the matrix entries change > than PETSc will automatically call the correct code to perform a new > numerical factorization; if the matrix entries are not changed then it will > not refactor the matrix. I am not sure if it makes sense for a user to be > setting different phase values since it is handled automatically. Could you > explain your exact use case where you would like to control the phase > variable directly? > > Barry > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jun 5 09:37:42 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 5 Jun 2018 14:37:42 +0000 Subject: [petsc-users] MKL Pardiso Solver Execution Step control In-Reply-To: References: <48CE4D2E-D6DB-43E8-A57A-7D4DC801D2AE@anl.gov> Message-ID: <4012106D-EBF0-4552-888F-7D3625536A31@mcs.anl.gov> Are you saying that you want to use Newton's method (or a Picard iteration) but don't want to refactor the matrix every time the matrices numerical values change? Are you using SNES or using KSP? If you are using KSP then you can call KSPSetReusePreconditioner() to prevent KSP from performing a new numerical factorization even when the matrix changes. This allows you to control exactly when the new numerical factorizations are done (or not done). If you are using SNES then you can call SNESSetLagPreconditioner() to control when new factorizations are done. You never need to monkey directly with calls to MKL Pardiso Is this what you need or do you need something else? Barry > On Jun 5, 2018, at 3:04 PM, Matthew Overholt wrote: > > Barry, > > What I should have said was that I wanted to control when it does the "Analysis" (phase 11), not Factoring (phase 22). We are solving the heat conduction equation, which makes for a nearly linear system; the non-linearity only enters through the material properties dependence on temperature, which is usually weak. We use the direct approach with several fixed point iterations to get the solution. We have found (in our non-PETSc code) that a lot of time can be saved, particularly when unsteady, by judiciously choosing when to (re)do the Analysis. Typically we do the Analysis on output time steps when unsteady, and just for the first fixed point iteration when steady. > > Matt... > > On Tue, Jun 5, 2018 at 5:03 AM, Smith, Barry F. wrote: > > > > On Jun 4, 2018, at 10:32 PM, Matthew Overholt wrote: > > > > Hello, > > > > I am using KSP in KSPPREONLY mode to do a direct solve on an A*x = b system, with solver algorithms MUMPS, CPardiso and Pardiso. For Pardiso, is it possible to control the solver execution step (denoted "phase" in Intel's docs)? I would like to be able to control when it refactors as one can when calling it directly. > > Matt, > > This is handled automatically by PETSc. If the matrix entries change than PETSc will automatically call the correct code to perform a new numerical factorization; if the matrix entries are not changed then it will not refactor the matrix. I am not sure if it makes sense for a user to be setting different phase values since it is handled automatically. Could you explain your exact use case where you would like to control the phase variable directly? > > Barry > From knepley at gmail.com Tue Jun 5 09:40:53 2018 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 5 Jun 2018 10:40:53 -0400 Subject: [petsc-users] MKL Pardiso Solver Execution Step control In-Reply-To: References: <48CE4D2E-D6DB-43E8-A57A-7D4DC801D2AE@anl.gov> Message-ID: On Tue, Jun 5, 2018 at 10:04 AM, Matthew Overholt wrote: > Barry, > > What I should have said was that I wanted to control when it does the > "Analysis" (phase 11), not Factoring (phase 22). > If you only change values in the matrix, but not the structure of the matrix, PETSc will only repeat the numerical factorization, not the symbolic factorization, which I think is what you want. Matt > We are solving the heat conduction equation, which makes for a nearly > linear system; the non-linearity only enters through the material > properties dependence on temperature, which is usually weak. We use the > direct approach with several fixed point iterations to get the solution. > We have found (in our non-PETSc code) that a lot of time can be saved, > particularly when unsteady, by judiciously choosing when to (re)do the > Analysis. Typically we do the Analysis on output time steps when unsteady, > and just for the first fixed point iteration when steady. > > Matt... > > On Tue, Jun 5, 2018 at 5:03 AM, Smith, Barry F. > wrote: > >> >> >> > On Jun 4, 2018, at 10:32 PM, Matthew Overholt >> wrote: >> > >> > Hello, >> > >> > I am using KSP in KSPPREONLY mode to do a direct solve on an A*x = b >> system, with solver algorithms MUMPS, CPardiso and Pardiso. For Pardiso, >> is it possible to control the solver execution step (denoted "phase" in >> Intel's docs)? I would like to be able to control when it refactors as one >> can when calling it directly. >> >> Matt, >> >> This is handled automatically by PETSc. If the matrix entries change >> than PETSc will automatically call the correct code to perform a new >> numerical factorization; if the matrix entries are not changed then it will >> not refactor the matrix. I am not sure if it makes sense for a user to be >> setting different phase values since it is handled automatically. Could you >> explain your exact use case where you would like to control the phase >> variable directly? >> >> Barry >> >> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From overholt at capesim.com Tue Jun 5 10:08:32 2018 From: overholt at capesim.com (Matthew Overholt) Date: Tue, 5 Jun 2018 11:08:32 -0400 Subject: [petsc-users] MKL Pardiso Solver Execution Step control In-Reply-To: References: <48CE4D2E-D6DB-43E8-A57A-7D4DC801D2AE@anl.gov> Message-ID: Yes to Matthew - not repeating Phase 1: Fill-reduction analysis and symbolic factorization . Numerical factoring is required because the matrix values have changed (although only slightly) as well as the RHS. We are using KSPPREONLY with Picard iterations. Thank you, Barry, for the KSP*Preconditioner calls I wasn't aware of. It sounds like PETSc and/or Pardiso automatically determines whether Phase 1 is required or not, but if we want to force a new Phase 1 (such as in a long unsteady run), then I suppose we could just destroy KSP and remake it. Matt... Matt Overholt CapeSym, Inc. (508) 653-7100 x204 overholt at capesim.com On Tue, Jun 5, 2018 at 10:40 AM, Matthew Knepley wrote: > On Tue, Jun 5, 2018 at 10:04 AM, Matthew Overholt > wrote: > >> Barry, >> >> What I should have said was that I wanted to control when it does the >> "Analysis" (phase 11), not Factoring (phase 22). >> > > If you only change values in the matrix, but not the structure of the > matrix, PETSc will only repeat the numerical factorization, > not the symbolic factorization, which I think is what you want. > > Matt > > >> We are solving the heat conduction equation, which makes for a nearly >> linear system; the non-linearity only enters through the material >> properties dependence on temperature, which is usually weak. We use the >> direct approach with several fixed point iterations to get the solution. >> We have found (in our non-PETSc code) that a lot of time can be saved, >> particularly when unsteady, by judiciously choosing when to (re)do the >> Analysis. Typically we do the Analysis on output time steps when unsteady, >> and just for the first fixed point iteration when steady. >> >> Matt... >> >> On Tue, Jun 5, 2018 at 5:03 AM, Smith, Barry F. >> wrote: >> >>> >>> >>> > On Jun 4, 2018, at 10:32 PM, Matthew Overholt >>> wrote: >>> > >>> > Hello, >>> > >>> > I am using KSP in KSPPREONLY mode to do a direct solve on an A*x = b >>> system, with solver algorithms MUMPS, CPardiso and Pardiso. For Pardiso, >>> is it possible to control the solver execution step (denoted "phase" in >>> Intel's docs)? I would like to be able to control when it refactors as one >>> can when calling it directly. >>> >>> Matt, >>> >>> This is handled automatically by PETSc. If the matrix entries >>> change than PETSc will automatically call the correct code to perform a new >>> numerical factorization; if the matrix entries are not changed then it will >>> not refactor the matrix. I am not sure if it makes sense for a user to be >>> setting different phase values since it is handled automatically. Could you >>> explain your exact use case where you would like to control the phase >>> variable directly? >>> >>> Barry >>> >>> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jun 5 10:29:57 2018 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 5 Jun 2018 11:29:57 -0400 Subject: [petsc-users] MKL Pardiso Solver Execution Step control In-Reply-To: References: <48CE4D2E-D6DB-43E8-A57A-7D4DC801D2AE@anl.gov> Message-ID: On Tue, Jun 5, 2018 at 11:08 AM, Matthew Overholt wrote: > Yes to Matthew - not repeating Phase 1: Fill-reduction analysis and > symbolic factorization . Numerical factoring is required because the > matrix values have changed (although only slightly) as well as the RHS. > > We are using KSPPREONLY with Picard iterations. Thank you, Barry, for the > KSP*Preconditioner calls I wasn't aware of. > > It sounds like PETSc and/or Pardiso automatically determines whether Phase > 1 is required or not, but if we want to force a new Phase 1 (such as in a > long unsteady run), then I suppose we could just destroy KSP and remake it. > Hmm, we have PetscErrorCode MatGetNonzeroState(Mat mat,PetscObjectState *state). We probably should add MatIncrementNonzeroState(). Matt > Matt... > > Matt Overholt > CapeSym, Inc. > (508) 653-7100 x204 > overholt at capesim.com > > On Tue, Jun 5, 2018 at 10:40 AM, Matthew Knepley > wrote: > >> On Tue, Jun 5, 2018 at 10:04 AM, Matthew Overholt >> wrote: >> >>> Barry, >>> >>> What I should have said was that I wanted to control when it does the >>> "Analysis" (phase 11), not Factoring (phase 22). >>> >> >> If you only change values in the matrix, but not the structure of the >> matrix, PETSc will only repeat the numerical factorization, >> not the symbolic factorization, which I think is what you want. >> >> Matt >> >> >>> We are solving the heat conduction equation, which makes for a nearly >>> linear system; the non-linearity only enters through the material >>> properties dependence on temperature, which is usually weak. We use the >>> direct approach with several fixed point iterations to get the solution. >>> We have found (in our non-PETSc code) that a lot of time can be saved, >>> particularly when unsteady, by judiciously choosing when to (re)do the >>> Analysis. Typically we do the Analysis on output time steps when unsteady, >>> and just for the first fixed point iteration when steady. >>> >>> Matt... >>> >>> On Tue, Jun 5, 2018 at 5:03 AM, Smith, Barry F. >>> wrote: >>> >>>> >>>> >>>> > On Jun 4, 2018, at 10:32 PM, Matthew Overholt >>>> wrote: >>>> > >>>> > Hello, >>>> > >>>> > I am using KSP in KSPPREONLY mode to do a direct solve on an A*x = b >>>> system, with solver algorithms MUMPS, CPardiso and Pardiso. For Pardiso, >>>> is it possible to control the solver execution step (denoted "phase" in >>>> Intel's docs)? I would like to be able to control when it refactors as one >>>> can when calling it directly. >>>> >>>> Matt, >>>> >>>> This is handled automatically by PETSc. If the matrix entries >>>> change than PETSc will automatically call the correct code to perform a new >>>> numerical factorization; if the matrix entries are not changed then it will >>>> not refactor the matrix. I am not sure if it makes sense for a user to be >>>> setting different phase values since it is handled automatically. Could you >>>> explain your exact use case where you would like to control the phase >>>> variable directly? >>>> >>>> Barry >>>> >>>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jun 5 12:48:17 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 5 Jun 2018 17:48:17 +0000 Subject: [petsc-users] MKL Pardiso Solver Execution Step control In-Reply-To: References: <48CE4D2E-D6DB-43E8-A57A-7D4DC801D2AE@anl.gov> Message-ID: <7F8C246E-B6AC-4165-B9F3-4E31976E537D@mcs.anl.gov> > On Jun 5, 2018, at 4:08 PM, Matthew Overholt wrote: > > Yes to Matthew - not repeating Phase 1: Fill-reduction analysis and symbolic factorization . Numerical factoring is required because the matrix values have changed (although only slightly) as well as the RHS. > > We are using KSPPREONLY with Picard iterations. Thank you, Barry, for the KSP*Preconditioner calls I wasn't aware of. > > It sounds like PETSc and/or Pardiso automatically determines whether Phase 1 is required or not, but if we want to force a new Phase 1 (such as in a long unsteady run), then I suppose we could just destroy KSP and remake it. I'm confused. Yes PETSc will do a new analysis if the nonzero structure changes otherwise it will never do a new analysis. Is your nonzero structure changing? If you nonzero structure is not changing you have to do nothing. If the nonzero structure does change than the code needs to do a new analysis because the old factors (for the numerical part) cannot work with the new nonzero locations of the matrix. I really don't see why you have to do anything? Barry > > Matt... > > Matt Overholt > CapeSym, Inc. > (508) 653-7100 x204 > overholt at capesim.com > > On Tue, Jun 5, 2018 at 10:40 AM, Matthew Knepley wrote: > On Tue, Jun 5, 2018 at 10:04 AM, Matthew Overholt wrote: > Barry, > > What I should have said was that I wanted to control when it does the "Analysis" (phase 11), not Factoring (phase 22). > > If you only change values in the matrix, but not the structure of the matrix, PETSc will only repeat the numerical factorization, > not the symbolic factorization, which I think is what you want. > > Matt > > We are solving the heat conduction equation, which makes for a nearly linear system; the non-linearity only enters through the material properties dependence on temperature, which is usually weak. We use the direct approach with several fixed point iterations to get the solution. We have found (in our non-PETSc code) that a lot of time can be saved, particularly when unsteady, by judiciously choosing when to (re)do the Analysis. Typically we do the Analysis on output time steps when unsteady, and just for the first fixed point iteration when steady. > > Matt... > > On Tue, Jun 5, 2018 at 5:03 AM, Smith, Barry F. wrote: > > > > On Jun 4, 2018, at 10:32 PM, Matthew Overholt wrote: > > > > Hello, > > > > I am using KSP in KSPPREONLY mode to do a direct solve on an A*x = b system, with solver algorithms MUMPS, CPardiso and Pardiso. For Pardiso, is it possible to control the solver execution step (denoted "phase" in Intel's docs)? I would like to be able to control when it refactors as one can when calling it directly. > > Matt, > > This is handled automatically by PETSc. If the matrix entries change than PETSc will automatically call the correct code to perform a new numerical factorization; if the matrix entries are not changed then it will not refactor the matrix. I am not sure if it makes sense for a user to be setting different phase values since it is handled automatically. Could you explain your exact use case where you would like to control the phase variable directly? > > Barry > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > From mcygorek at uottawa.ca Tue Jun 5 13:00:15 2018 From: mcygorek at uottawa.ca (Moritz Cygorek) Date: Tue, 5 Jun 2018 18:00:15 +0000 Subject: [petsc-users] Documentation for different parallelization options Message-ID: Hi everyone, I'm looking for a document/tutorial/howto that describes the different options to compile PETSc with parallelization. My problem is the following: I'm trying to solve a large sparse eigenvalue problem using the Krylov-Schur method implemented in SLEPc When I install SLEPc/PETSc on my Ubuntu laptop via apt-get, everything works smoothly and parallelization works automatically. I see this by the fact that the CPU-load of the process (only one process, not using mpiexec) is close to 400% according to "top" Therefore, it seems that OpenMP is used. I have access to better computers and I would like to install SLEPc/PETSc there, but I have to configure it manually. I have tried different options, none of the satisfactory: When I compile PETSc with the --with-openmp flag, I see that the program never runs with cpu load above 100%. I use the same command to call the program as on my laptop where everything works. So it seems that openmp is somehow not activated. An old mailing list entry says that I am supposed to configure PETSc using --with-threadcomm --with-openmp, which I did, but it also didn't help. However that entry was from 2014 and I found in the list of changes for PETSc in version 3.6: "Removed all threadcomm support including --with-pthreadclasses and --with-openmpclasses configure arguments" Does that mean that openmp is no longer supported in newer versions? Given my resources, I would prefer OpenMP over MPI. Nevertheless, I then spent some time to go full MPI without openmp and to split up the sparse matrix across several processes. When I start the program using mpiexec, I see indeed that multiple processes are started, but even when I use 12 processes, the computation time is about the same as with only 1 process. Is there anything I have to tell the EPS solver to activate parallelization? So, all in all, I can't get to run anything faster on a large multi-core computer than on my old crappy laptop. I have no idea how to start debugging and assessing the performance and the documentation on this issue on the website is not very verbose. Can you give me a few hints? Regards, Moritz From overholt at capesim.com Tue Jun 5 13:36:10 2018 From: overholt at capesim.com (Matthew Overholt) Date: Tue, 5 Jun 2018 14:36:10 -0400 Subject: [petsc-users] MKL Pardiso Solver Execution Step control In-Reply-To: <7F8C246E-B6AC-4165-B9F3-4E31976E537D@mcs.anl.gov> References: <48CE4D2E-D6DB-43E8-A57A-7D4DC801D2AE@anl.gov> <7F8C246E-B6AC-4165-B9F3-4E31976E537D@mcs.anl.gov> Message-ID: On Tue, Jun 5, 2018 at 1:48 PM, Smith, Barry F. wrote: > > > > On Jun 5, 2018, at 4:08 PM, Matthew Overholt > wrote: > > > > Yes to Matthew - not repeating Phase 1: Fill-reduction analysis and > symbolic factorization . Numerical factoring is required because the > matrix values have changed (although only slightly) as well as the RHS. > > > > We are using KSPPREONLY with Picard iterations. Thank you, Barry, for > the KSP*Preconditioner calls I wasn't aware of. > > > > It sounds like PETSc and/or Pardiso automatically determines whether > Phase 1 is required or not, but if we want to force a new Phase 1 (such as > in a long unsteady run), then I suppose we could just destroy KSP and > remake it. > > I'm confused. Yes PETSc will do a new analysis if the nonzero structure > changes otherwise it will never do a new analysis. > > Is your nonzero structure changing? > > If you nonzero structure is not changing you have to do nothing. > > If the nonzero structure does change than the code needs to do a new > analysis because the old factors (for the numerical part) cannot work with > the new nonzero locations of the matrix. > > I really don't see why you have to do anything? > > Barry > > No, the nonzero structure never changes. Redoing the analysis is not required, in the sense that a solution can still be obtained without it; however, for long unsteady runs (with heat sources turning on and off) the matrix values change so much that, without redoing the analysis, more iterations are required to converge at each time step, resulting in a doubling of the total solve time compared to when the analysis is periodically redone. I don't know exactly what Pardiso is doing "under the hood" but that has been our experience. Matt... -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Tue Jun 5 16:43:37 2018 From: jroman at dsic.upv.es (Jose E. Roman) Date: Tue, 5 Jun 2018 22:43:37 +0100 Subject: [petsc-users] Documentation for different parallelization options In-Reply-To: References: Message-ID: For multi-threaded parallelism you have to use a multi-threaded BLAS such as MKL or OpenBLAS: $ ./configure --with-blaslapack-dir=$MKLROOT or $ ./configure --download-openblas For MPI parallelism, if you are solving linear systems within EPS you most probably need PETSc be configured with a parallel linear solver such as MUMPS, see section 3.4.1 of SLEPc's user manual. Jose > El 5 jun 2018, a las 19:00, Moritz Cygorek escribi?: > > Hi everyone, > > I'm looking for a document/tutorial/howto that describes the different options to compile PETSc with parallelization. > > My problem is the following: > I'm trying to solve a large sparse eigenvalue problem using the Krylov-Schur method implemented in SLEPc > When I install SLEPc/PETSc on my Ubuntu laptop via apt-get, everything works smoothly and parallelization works automatically. > I see this by the fact that the CPU-load of the process (only one process, not using mpiexec) is close to 400% according to "top" > Therefore, it seems that OpenMP is used. > > I have access to better computers and I would like to install SLEPc/PETSc there, but I have to configure it manually. > I have tried different options, none of the satisfactory: > > When I compile PETSc with the --with-openmp flag, I see that the program never runs with cpu load above 100%. > I use the same command to call the program as on my laptop where everything works. So it seems that openmp is somehow not activated. > An old mailing list entry says that I am supposed to configure PETSc using --with-threadcomm --with-openmp, which I did, but it also didn't help. > However that entry was from 2014 and I found in the list of changes for PETSc in version 3.6: > "Removed all threadcomm support including --with-pthreadclasses and --with-openmpclasses configure arguments" > > Does that mean that openmp is no longer supported in newer versions? > > > Given my resources, I would prefer OpenMP over MPI. Nevertheless, I then spent some time to go full MPI without openmp and to split up the sparse matrix across several processes. When I start the program using mpiexec, > I see indeed that multiple processes are started, but even when I use 12 processes, the computation time is about the same as with only 1 process. > Is there anything I have to tell the EPS solver to activate parallelization? > > > So, all in all, I can't get to run anything faster on a large multi-core computer than on my old crappy laptop. > > > I have no idea how to start debugging and assessing the performance and the documentation on this issue on the website is not very verbose. > Can you give me a few hints? > > Regards, > Moritz > > > From aamor at pa.uc3m.es Tue Jun 5 17:46:22 2018 From: aamor at pa.uc3m.es (=?UTF-8?B?QWRyacOhbiBBbW9y?=) Date: Wed, 6 Jun 2018 00:46:22 +0200 Subject: [petsc-users] Solving reduced system of equations with global preconditioner Message-ID: Hi all, I think this is a basic question that I am pretty sure that it can be done in PETSC but I don't know the way, so sorry for wasting your time if it's too basic! I have a big system of equations (say 1M of unknowns) which I divide into 25 blocks and I use a block-Jacobi preconditioner to get the solution from KSPBICG. However, I know that I can build a reduced matrix with the nonzeros of the big preconditioned matrix, solve this reduced problem, and then recover the solution for the original big system with the factorized matrices from block Jacobi. How can I do this with PETSC? Since using KSPComputeExplicitOperator and then copy blocks of this matrix to build the reduced surface problem in PETSC is definitely not an option. I was doing this by hand using MUMPS to build the reduced system of equations, then using PETSC without preconditioner and then, recover the full solution. However, the burden in time and memory is not negligible so I wanted to explore a different way with PETSC (using maybe some ILU). I think that the performance in memory is going to be similar but I expect a better performance in time. Thanks a lot! Adrian. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hsahasra at purdue.edu Tue Jun 5 19:42:07 2018 From: hsahasra at purdue.edu (Harshad Sahasrabudhe) Date: Tue, 5 Jun 2018 20:42:07 -0400 Subject: [petsc-users] Find an orthogonal set of vectors Message-ID: Hello, I have a set of m n-dimensional vectors and I need to find orthogonal vectors in this subspace. This can be done using the Gram-Schmidt process. Is there a way to set up the KSP to do this directly in PETSc? Thanks, Harshad -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Wed Jun 6 00:33:02 2018 From: jed at jedbrown.org (Jed Brown) Date: Tue, 05 Jun 2018 23:33:02 -0600 Subject: [petsc-users] Find an orthogonal set of vectors In-Reply-To: References: Message-ID: <8736y0bhtd.fsf@jedbrown.org> Consider BVOrthogonalize in SLEPc. It can also be more stable than Gram-Schmidt. Harshad Sahasrabudhe writes: > Hello, > > I have a set of m n-dimensional vectors and I need to find orthogonal > vectors in this subspace. This can be done using the Gram-Schmidt process. > Is there a way to set up the KSP to do this directly in PETSc? > > Thanks, > Harshad From bsmith at mcs.anl.gov Wed Jun 6 05:30:21 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 6 Jun 2018 10:30:21 +0000 Subject: [petsc-users] MKL Pardiso Solver Execution Step control In-Reply-To: References: <48CE4D2E-D6DB-43E8-A57A-7D4DC801D2AE@anl.gov> <7F8C246E-B6AC-4165-B9F3-4E31976E537D@mcs.anl.gov> Message-ID: <1D566E33-CF52-471C-8666-D0E0FEFD93B0@mcs.anl.gov> > On Jun 5, 2018, at 7:36 PM, Matthew Overholt wrote: > > > On Tue, Jun 5, 2018 at 1:48 PM, Smith, Barry F. wrote: > > > > On Jun 5, 2018, at 4:08 PM, Matthew Overholt wrote: > > > > Yes to Matthew - not repeating Phase 1: Fill-reduction analysis and symbolic factorization . Numerical factoring is required because the matrix values have changed (although only slightly) as well as the RHS. > > > > We are using KSPPREONLY with Picard iterations. Thank you, Barry, for the KSP*Preconditioner calls I wasn't aware of. > > > > It sounds like PETSc and/or Pardiso automatically determines whether Phase 1 is required or not, but if we want to force a new Phase 1 (such as in a long unsteady run), then I suppose we could just destroy KSP and remake it. > > I'm confused. Yes PETSc will do a new analysis if the nonzero structure changes otherwise it will never do a new analysis. > > Is your nonzero structure changing? > > If you nonzero structure is not changing you have to do nothing. > > If the nonzero structure does change than the code needs to do a new analysis because the old factors (for the numerical part) cannot work with the new nonzero locations of the matrix. > > I really don't see why you have to do anything? > > Barry > > No, the nonzero structure never changes. Redoing the analysis is not required, in the sense that a solution can still be obtained without it; however, for long unsteady runs (with heat sources turning on and off) the matrix values change so much that, without redoing the analysis, more iterations are required to converge at each time step, resulting in a doubling of the total solve time compared to when the analysis is periodically redone. I don't know exactly what Pardiso is doing "under the hood" but that has been our experience. Strange; I've never heard of this before, anyways, You could trick it by "saying" that the nonzero structure has changed even if it has not. The matrix struct has a field A->nonzerostate you could simply increase this value by 1 (include #include to get access to the field). Any time you change this value will trigger a new analysis stage. > > Matt... From knepley at gmail.com Wed Jun 6 05:50:33 2018 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 6 Jun 2018 06:50:33 -0400 Subject: [petsc-users] Solving reduced system of equations with global preconditioner In-Reply-To: References: Message-ID: On Tue, Jun 5, 2018 at 6:46 PM, Adri?n Amor wrote: > Hi all, > > I think this is a basic question that I am pretty sure that it can be done > in PETSC but I don't know the way, so sorry for wasting your time if it's > too basic! > > I have a big system of equations (say 1M of unknowns) which I divide into > 25 blocks and I use a block-Jacobi preconditioner to get the solution from > KSPBICG. However, I know that I can build a reduced matrix with the > nonzeros of the big preconditioned matrix, solve this reduced problem, and > then recover the solution for the original big system with the factorized > matrices from block Jacobi. How can I do this with PETSC? Since using > KSPComputeExplicitOperator and then copy blocks of this matrix to build the > reduced surface problem in PETSC is definitely not an option. > Its a little hard to understand what algorithm you are describing. Do you mean something like https://en.wikipedia.org/wiki/SPIKE_algorithm We do not have that. We can do additive/multiplicative iterations on these blocks using PCFIELDSPLIT (however the block operations would be serialized). If you want something direct like this, have you tried just applying SuperLU_dist or MUMPS? Thanks, Matt > I was doing this by hand using MUMPS to build the reduced system of > equations, then using PETSC without preconditioner and then, recover the > full solution. However, the burden in time and memory is not negligible so > I wanted to explore a different way with PETSC (using maybe some ILU). I > think that the performance in memory is going to be similar but I expect a > better performance in time. > > Thanks a lot! > > Adrian. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From aamor at pa.uc3m.es Wed Jun 6 08:13:48 2018 From: aamor at pa.uc3m.es (=?UTF-8?B?QWRyacOhbiBBbW9y?=) Date: Wed, 6 Jun 2018 15:13:48 +0200 Subject: [petsc-users] Solving reduced system of equations with global preconditioner In-Reply-To: References: Message-ID: Hi Matthew, Thanks for letting me notice that algorithm I didn?t know, but I think it is not exactly the same. The two-step procedure that I want to implement within PETSC is the one shown in this article ( https://epubs.siam.org/doi/abs/10.1137/100817978) and I show you the reduced matrix for your convenience: [image: image.png] A_1, A_2 are the blocks of the big original problem used for PCBJACOBI while R_1 and R_2 are some boolean matrices that select "nonzeros" of the off-diagonal blocks to build the reduced problem. As I told you, the algorithm that I am using is to compute the inverse of the A_1 with MUMPS and then solving that matrix with an iterative solver within PETSC, but I was wondering if I can merge everything inside PETSC doing the precondition PCBJACOBI of the big original problem and then indicating to PETSC where the reduced problem is. I am so sorry if I explained myself wrong! Thanks a lot for your quick answer, Adrian. El mi?., 6 jun. 2018 a las 12:50, Matthew Knepley () escribi?: > On Tue, Jun 5, 2018 at 6:46 PM, Adri?n Amor wrote: > >> Hi all, >> >> I think this is a basic question that I am pretty sure that it can be >> done in PETSC but I don't know the way, so sorry for wasting your time if >> it's too basic! >> >> I have a big system of equations (say 1M of unknowns) which I divide into >> 25 blocks and I use a block-Jacobi preconditioner to get the solution from >> KSPBICG. However, I know that I can build a reduced matrix with the >> nonzeros of the big preconditioned matrix, solve this reduced problem, and >> then recover the solution for the original big system with the factorized >> matrices from block Jacobi. How can I do this with PETSC? Since using >> KSPComputeExplicitOperator and then copy blocks of this matrix to build the >> reduced surface problem in PETSC is definitely not an option. >> > > Its a little hard to understand what algorithm you are describing. Do you > mean something like > > https://en.wikipedia.org/wiki/SPIKE_algorithm > > We do not have that. We can do additive/multiplicative iterations on these > blocks using PCFIELDSPLIT (however the block operations would be > serialized). > If you want something direct like this, have you tried just applying > SuperLU_dist or MUMPS? > > Thanks, > > Matt > > >> I was doing this by hand using MUMPS to build the reduced system of >> equations, then using PETSC without preconditioner and then, recover the >> full solution. However, the burden in time and memory is not negligible so >> I wanted to explore a different way with PETSC (using maybe some ILU). I >> think that the performance in memory is going to be similar but I expect a >> better performance in time. >> >> Thanks a lot! >> >> Adrian. >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 30739 bytes Desc: not available URL: From mcygorek at uottawa.ca Wed Jun 6 13:20:15 2018 From: mcygorek at uottawa.ca (Moritz Cygorek) Date: Wed, 6 Jun 2018 18:20:15 +0000 Subject: [petsc-users] Documentation for different parallelization options In-Reply-To: References: , Message-ID: <0fe9c0dcb8284a4997aa42ee47859214@uottawa.ca> Thank you very much for your response. I have tested the --download-openblas option and it did not do what I expected. The total cpu-usage only moved to something like 105%, so it did not significantly make use of parallelization. I did not test MKL yet, because I'll first have to install it. However, compiling PETSc with MUMPS works very well and significantly speeds up the caluclation for my full-MPI code. I will have to do some more testing, but MPI with MUMPS support seems to be the way to go for me. Thanks again, Moritz ________________________________ From: Jose E. Roman Sent: Tuesday, June 5, 2018 5:43:37 PM To: Moritz Cygorek Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Documentation for different parallelization options For multi-threaded parallelism you have to use a multi-threaded BLAS such as MKL or OpenBLAS: $ ./configure --with-blaslapack-dir=$MKLROOT or $ ./configure --download-openblas For MPI parallelism, if you are solving linear systems within EPS you most probably need PETSc be configured with a parallel linear solver such as MUMPS, see section 3.4.1 of SLEPc's user manual. Jose > El 5 jun 2018, a las 19:00, Moritz Cygorek escribi?: > > Hi everyone, > > I'm looking for a document/tutorial/howto that describes the different options to compile PETSc with parallelization. > > My problem is the following: > I'm trying to solve a large sparse eigenvalue problem using the Krylov-Schur method implemented in SLEPc > When I install SLEPc/PETSc on my Ubuntu laptop via apt-get, everything works smoothly and parallelization works automatically. > I see this by the fact that the CPU-load of the process (only one process, not using mpiexec) is close to 400% according to "top" > Therefore, it seems that OpenMP is used. > > I have access to better computers and I would like to install SLEPc/PETSc there, but I have to configure it manually. > I have tried different options, none of the satisfactory: > > When I compile PETSc with the --with-openmp flag, I see that the program never runs with cpu load above 100%. > I use the same command to call the program as on my laptop where everything works. So it seems that openmp is somehow not activated. > An old mailing list entry says that I am supposed to configure PETSc using --with-threadcomm --with-openmp, which I did, but it also didn't help. > However that entry was from 2014 and I found in the list of changes for PETSc in version 3.6: > "Removed all threadcomm support including --with-pthreadclasses and --with-openmpclasses configure arguments" > > Does that mean that openmp is no longer supported in newer versions? > > > Given my resources, I would prefer OpenMP over MPI. Nevertheless, I then spent some time to go full MPI without openmp and to split up the sparse matrix across several processes. When I start the program using mpiexec, > I see indeed that multiple processes are started, but even when I use 12 processes, the computation time is about the same as with only 1 process. > Is there anything I have to tell the EPS solver to activate parallelization? > > > So, all in all, I can't get to run anything faster on a large multi-core computer than on my old crappy laptop. > > > I have no idea how to start debugging and assessing the performance and the documentation on this issue on the website is not very verbose. > Can you give me a few hints? > > Regards, > Moritz > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Jun 6 13:46:15 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 6 Jun 2018 18:46:15 +0000 Subject: [petsc-users] Documentation for different parallelization options In-Reply-To: <0fe9c0dcb8284a4997aa42ee47859214@uottawa.ca> References: <0fe9c0dcb8284a4997aa42ee47859214@uottawa.ca> Message-ID: <5FD594C6-8E3D-4F0E-9BDD-D99477602D76@anl.gov> From https://github.com/xianyi/OpenBLAS Setting the number of threads using environment variables Environment variables are used to specify a maximum number of threads. For example, export OPENBLAS_NUM_THREADS=4 export GOTO_NUM_THREADS=4 export OMP_NUM_THREADS=4 The priorities are OPENBLAS_NUM_THREADS > GOTO_NUM_THREADS > OMP_NUM_THREADS. If you compile this library with USE_OPENMP=1, you should set the OMP_NUM_THREADS environment variable; OpenBLAS ignores OPENBLAS_NUM_THREADS and GOTO_NUM_THREADS when compiled with USE_OPENMP=1. > On Jun 6, 2018, at 7:20 PM, Moritz Cygorek wrote: > > Thank you very much for your response. > > I have tested the --download-openblas option and it did not do what I expected. > The total cpu-usage only moved to something like 105%, so it did not significantly make use of parallelization. > I did not test MKL yet, because I'll first have to install it. > > However, compiling PETSc with MUMPS works very well and significantly speeds up the caluclation for my full-MPI code. > > I will have to do some more testing, but MPI with MUMPS support seems to be the way to go for me. > > Thanks again, > Moritz > > > From: Jose E. Roman > Sent: Tuesday, June 5, 2018 5:43:37 PM > To: Moritz Cygorek > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Documentation for different parallelization options > > For multi-threaded parallelism you have to use a multi-threaded BLAS such as MKL or OpenBLAS: > $ ./configure --with-blaslapack-dir=$MKLROOT > or > $ ./configure --download-openblas > > For MPI parallelism, if you are solving linear systems within EPS you most probably need PETSc be configured with a parallel linear solver such as MUMPS, see section 3.4.1 of SLEPc's user manual. > > Jose > > > > El 5 jun 2018, a las 19:00, Moritz Cygorek escribi?: > > > > Hi everyone, > > > > I'm looking for a document/tutorial/howto that describes the different options to compile PETSc with parallelization. > > > > My problem is the following: > > I'm trying to solve a large sparse eigenvalue problem using the Krylov-Schur method implemented in SLEPc > > When I install SLEPc/PETSc on my Ubuntu laptop via apt-get, everything works smoothly and parallelization works automatically. > > I see this by the fact that the CPU-load of the process (only one process, not using mpiexec) is close to 400% according to "top" > > Therefore, it seems that OpenMP is used. > > > > I have access to better computers and I would like to install SLEPc/PETSc there, but I have to configure it manually. > > I have tried different options, none of the satisfactory: > > > > When I compile PETSc with the --with-openmp flag, I see that the program never runs with cpu load above 100%. > > I use the same command to call the program as on my laptop where everything works. So it seems that openmp is somehow not activated. > > An old mailing list entry says that I am supposed to configure PETSc using --with-threadcomm --with-openmp, which I did, but it also didn't help. > > However that entry was from 2014 and I found in the list of changes for PETSc in version 3.6: > > "Removed all threadcomm support including --with-pthreadclasses and --with-openmpclasses configure arguments" > > > > Does that mean that openmp is no longer supported in newer versions? > > > > > > Given my resources, I would prefer OpenMP over MPI. Nevertheless, I then spent some time to go full MPI without openmp and to split up the sparse matrix across several processes. When I start the program using mpiexec, > > I see indeed that multiple processes are started, but even when I use 12 processes, the computation time is about the same as with only 1 process. > > Is there anything I have to tell the EPS solver to activate parallelization? > > > > > > So, all in all, I can't get to run anything faster on a large multi-core computer than on my old crappy laptop. > > > > > > I have no idea how to start debugging and assessing the performance and the documentation on this issue on the website is not very verbose. > > Can you give me a few hints? > > > > Regards, > > Moritz > > > > > > From overholt at capesim.com Wed Jun 6 14:57:22 2018 From: overholt at capesim.com (Matthew Overholt) Date: Wed, 6 Jun 2018 15:57:22 -0400 Subject: [petsc-users] MKL Pardiso Solver Execution Step control In-Reply-To: <1D566E33-CF52-471C-8666-D0E0FEFD93B0@mcs.anl.gov> References: <48CE4D2E-D6DB-43E8-A57A-7D4DC801D2AE@anl.gov> <7F8C246E-B6AC-4165-B9F3-4E31976E537D@mcs.anl.gov> <1D566E33-CF52-471C-8666-D0E0FEFD93B0@mcs.anl.gov> Message-ID: On Wed, Jun 6, 2018 at 6:30 AM, Smith, Barry F. wrote: > > > > > On Jun 5, 2018, at 7:36 PM, Matthew Overholt > wrote: > > > > > > On Tue, Jun 5, 2018 at 1:48 PM, Smith, Barry F. > wrote: > > > > > > > On Jun 5, 2018, at 4:08 PM, Matthew Overholt > wrote: > > > > > > Yes to Matthew - not repeating Phase 1: Fill-reduction analysis and > symbolic factorization . Numerical factoring is required because the > matrix values have changed (although only slightly) as well as the RHS. > > > > > > We are using KSPPREONLY with Picard iterations. Thank you, Barry, for > the KSP*Preconditioner calls I wasn't aware of. > > > > > > It sounds like PETSc and/or Pardiso automatically determines whether > Phase 1 is required or not, but if we want to force a new Phase 1 (such as > in a long unsteady run), then I suppose we could just destroy KSP and > remake it. > > > > I'm confused. Yes PETSc will do a new analysis if the nonzero > structure changes otherwise it will never do a new analysis. > > > > Is your nonzero structure changing? > > > > If you nonzero structure is not changing you have to do nothing. > > > > If the nonzero structure does change than the code needs to do a new > analysis because the old factors (for the numerical part) cannot work with > the new nonzero locations of the matrix. > > > > I really don't see why you have to do anything? > > > > Barry > > > > No, the nonzero structure never changes. Redoing the analysis is not > required, in the sense that a solution can still be obtained without it; > however, for long unsteady runs (with heat sources turning on and off) the > matrix values change so much that, without redoing the analysis, more > iterations are required to converge at each time step, resulting in a > doubling of the total solve time compared to when the analysis is > periodically redone. I don't know exactly what Pardiso is doing "under the > hood" but that has been our experience. > > Strange; I've never heard of this before, anyways, > > You could trick it by "saying" that the nonzero structure has changed > even if it has not. The matrix struct has a field A->nonzerostate you > could simply increase this value by 1 (include #include > to get access to the field). Any time you change > this value will trigger a new analysis stage. > > Thanks, I'll give this a try. Matt... -------------- next part -------------- An HTML attachment was scrubbed... URL: From mbuerkle at web.de Wed Jun 6 21:55:14 2018 From: mbuerkle at web.de (Marius Buerkle) Date: Thu, 7 Jun 2018 04:55:14 +0200 Subject: [petsc-users] MatGetOwnershipIS in fortran Message-ID: Hi ! ? I try to use MatGetOwnershipIS in Fortran but i get an undefined reference to `matgetownershipis_' when linking my program. Is this available for Fortran ? From fdkong.jd at gmail.com Wed Jun 6 22:34:45 2018 From: fdkong.jd at gmail.com (Fande Kong) Date: Wed, 6 Jun 2018 21:34:45 -0600 Subject: [petsc-users] Fwd: Installing PETSC manually. In-Reply-To: References: Message-ID: Hi Satish, A MOOSE user has troubles to build Metis that is "downloaded" from a local directory. Do you have any idea? Vi, Could you share "configure.log" with PETSc team? Thanks, Fande, ---------- Forwarded message ---------- From: Vi Ha Date: Wed, Jun 6, 2018 at 11:00 AM Subject: Re: Installing PETSC manually. To: moose-users Hi Jason, Thanks for the reply. I am having the same exact error as you linked. Here's the full error message from configure.log: ============================================================ =================== Trying to download file:///home/vi/Documents/v- install-moose/files/petsc-config-backup/v5.1.0-p4.tar.gz for METIS ============================== ================================================= Downloading file:///home/vi/Documents/v- install-moose/files/petsc-config-backup/v5.1.0-p4.tar.gz to /home/vi/Documents/moose/packages/projects/src/petsc-3. 7.6/arch-linux2-c-opt/externalpackages/_d_v5.1.0-p4.tar.gz Extracting /home/vi/Documents/moose/ packages/projects/src/petsc-3.7.6/arch-linux2-c-opt/ externalpackages/_d_v5.1.0-p4.tar.gz Executing: cd /home/vi/Documents/moose/packages/projects/src/petsc-3. 7.6/arch-linux2-c-opt/externalpackages; chmod -R a+r petsc-pkg-metis-0adf3ea7785d;find petsc-pkg-metis-0adf3ea7785d -type d -name "*" -exec chmod a+rx {} \; Looking for METIS at git.metis, hg.metis or a directory starting with ['metis'] Could not locate an existing copy of METIS: ['fblaslapack-3.4.2', 'petsc-pkg-metis-0adf3ea7785d', 'hypre-2.11.1', 'scalapack-2.0.2', 'pkg-metis'] ERROR: Failed to download METIS And then the on screen error becomes: Unable to download METIS Failed to download METIS Same as previous post. It "downloads" the file but it cannot detect it for some reason. On Tuesday, June 5, 2018 at 3:08:17 PM UTC-4, jason.miller wrote: > > We ran into an issue like this recently. I believe this thread will help > you: MOOSE-Users: Error while installing PETSc > > > On Tue, Jun 5, 2018 at 1:34 PM, Vi Ha wrote: > >> I am trying to install petsc using these instructions: >> http://mooseframework.org/wiki/BasicManualInstallation/Linux/ >> on a linux system that isn't allowing me to download things directly. >> >> I am trying to install the dependent packages such as hypre, metis etc. >> does anyone know if there are instructions on how to download and install >> them manually? I know there are parameters in PETSC's configure file that >> allow me to do so but I don't know what they are. >> >> Thank you. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "moose-users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to moose-users... at googlegroups.com. >> Visit this group at https://groups.google.com/group/moose-users >> >> . >> To view this discussion on the web visit https://groups.google.com/d/ms >> gid/moose-users/ae2cfd8e-ce5c-4525-990c-cdd524814c8c%40googlegroups.com >> >> . >> For more options, visit https://groups.google.com/d/optout >> >> . >> > > -- You received this message because you are subscribed to the Google Groups "moose-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe at googlegroups.com. Visit this group at https://groups.google.com/group/moose-users. To view this discussion on the web visit https://groups.google.com/d/ msgid/moose-users/f2d4a0f1-3b81-4fe8-9acf-b1814ee6a290%40googlegroups.com . For more options, visit https://groups.google.com/d/optout. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jczhang at mcs.anl.gov Wed Jun 6 23:33:32 2018 From: jczhang at mcs.anl.gov (Junchao Zhang) Date: Wed, 6 Jun 2018 23:33:32 -0500 Subject: [petsc-users] Fwd: Installing PETSC manually. In-Reply-To: References: Message-ID: Make sure unzipping v5.1.0-p4.tar.gz generates a directory name started with metis but not METIS. --Junchao Zhang On Wed, Jun 6, 2018 at 10:34 PM, Fande Kong wrote: > Hi Satish, > > A MOOSE user has troubles to build Metis that is "downloaded" from a local > directory. Do you have any idea? > > Vi, > > Could you share "configure.log" with PETSc team? > > > Thanks, > > Fande, > > ---------- Forwarded message ---------- > From: Vi Ha > Date: Wed, Jun 6, 2018 at 11:00 AM > Subject: Re: Installing PETSC manually. > To: moose-users > > > Hi Jason, > > Thanks for the reply. > I am having the same exact error as you linked. Here's the full error > message from configure.log: > > ============================================================ > =================== > > Trying to download file:///home/vi/Documents/v-in > stall-moose/files/petsc-config-backup/v5.1.0-p4.tar.gz for METIS > > ============================== > ================================================= > > > > Downloading file:///home/vi/Documents/v-in > stall-moose/files/petsc-config-backup/v5.1.0-p4.tar.gz to > /home/vi/Documents/moose/packages/projects/src/petsc-3.7.6/ > arch-linux2-c-opt/externalpackages/_d_v5.1.0-p4.tar.gz > > Extracting /home/vi/Documents/moose/packa > ges/projects/src/petsc-3.7.6/arch-linux2-c-opt/externalpack > ages/_d_v5.1.0-p4.tar.gz > > Executing: cd /home/vi/Documents/moose/packages/projects/src/petsc-3.7.6/ > arch-linux2-c-opt/externalpackages; chmod -R a+r > petsc-pkg-metis-0adf3ea7785d;find petsc-pkg-metis-0adf3ea7785d -type d > -name "*" -exec chmod a+rx {} \; > > Looking for METIS at git.metis, hg.metis or a directory > starting with ['metis'] > > Could not locate an existing copy of METIS: > > ['fblaslapack-3.4.2', 'petsc-pkg-metis-0adf3ea7785d', > 'hypre-2.11.1', 'scalapack-2.0.2', 'pkg-metis'] > > ERROR: Failed to download METIS > > And then the on screen error becomes: > > Unable to download METIS > Failed to download METIS > > Same as previous post. It "downloads" the file but it cannot detect it for > some reason. > > > > On Tuesday, June 5, 2018 at 3:08:17 PM UTC-4, jason.miller wrote: >> >> We ran into an issue like this recently. I believe this thread will help >> you: MOOSE-Users: Error while installing PETSc >> >> >> On Tue, Jun 5, 2018 at 1:34 PM, Vi Ha wrote: >> >>> I am trying to install petsc using these instructions: >>> http://mooseframework.org/wiki/BasicManualInstallation/Linux/ >>> on a linux system that isn't allowing me to download things directly. >>> >>> I am trying to install the dependent packages such as hypre, metis etc. >>> does anyone know if there are instructions on how to download and install >>> them manually? I know there are parameters in PETSC's configure file that >>> allow me to do so but I don't know what they are. >>> >>> Thank you. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "moose-users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to moose-users... at googlegroups.com. >>> Visit this group at https://groups.google.com/group/moose-users >>> >>> . >>> To view this discussion on the web visit https://groups.google.com/d/ms >>> gid/moose-users/ae2cfd8e-ce5c-4525-990c-cdd524814c8c%40googlegroups.com >>> >>> . >>> For more options, visit https://groups.google.com/d/optout >>> >>> . >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "moose-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to moose-users+unsubscribe at googlegroups.com. > Visit this group at https://groups.google.com/group/moose-users. > To view this discussion on the web visit https://groups.google.com/d/ms > gid/moose-users/f2d4a0f1-3b81-4fe8-9acf-b1814ee6a290%40googlegroups.com > > . > > For more options, visit https://groups.google.com/d/optout. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thronesf at gmail.com Thu Jun 7 02:02:18 2018 From: thronesf at gmail.com (Sharp Stone) Date: Thu, 7 Jun 2018 03:02:18 -0400 Subject: [petsc-users] Petsc Installment on Cray Message-ID: ?? Dear all, I try to install petsc on cray. But the mpi seems always not working. Below is the error message. The configure log is also attached. Please help. Thank you in advance! /mnt/lustre/lus0/home/IAM851/jhk1009/local/petsc-3.7.6/src/ sys/f90-mod/petscsysmod.F:6.11: use mpi 1 Fatal Error: Parse error when checking module version for file 'mpi.mod' opened at (1) gmake[2]: *** [arch-linux2-c-debug/obj/src/sys/f90-mod/petscsysmod.o] Error 1 gmake[2]: *** Waiting for unfinished jobs.... gmake[2]: Leaving directory `/mnt/lustre/lus0/home/IAM851/ jhk1009/local/petsc-3.7.6' gmake[1]: *** [gnumake] Error 2 gmake[1]: Leaving directory `/mnt/lustre/lus0/home/IAM851/ jhk1009/local/petsc-3.7.6' **************************ERROR************************************* Error during compile, check arch-linux2-c-debug/lib/petsc/conf/make.log Send it and arch-linux2-c-debug/lib/petsc/conf/configure.log to petsc-maint at mcs.anl.gov ******************************************************************** make: *** [all] Error 1 -- Best regards, Feng -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 1591094 bytes Desc: not available URL: From fdkong.jd at gmail.com Wed Jun 6 22:34:45 2018 From: fdkong.jd at gmail.com (Fande Kong) Date: Wed, 6 Jun 2018 21:34:45 -0600 Subject: [petsc-users] Fwd: Installing PETSC manually. In-Reply-To: References: Message-ID: Hi Satish, A MOOSE user has troubles to build Metis that is "downloaded" from a local directory. Do you have any idea? Vi, Could you share "configure.log" with PETSc team? Thanks, Fande, ---------- Forwarded message ---------- From: Vi Ha Date: Wed, Jun 6, 2018 at 11:00 AM Subject: Re: Installing PETSC manually. To: moose-users Hi Jason, Thanks for the reply. I am having the same exact error as you linked. Here's the full error message from configure.log: ============================================================ =================== Trying to download file:///home/vi/Documents/v- install-moose/files/petsc-config-backup/v5.1.0-p4.tar.gz for METIS ============================== ================================================= Downloading file:///home/vi/Documents/v- install-moose/files/petsc-config-backup/v5.1.0-p4.tar.gz to /home/vi/Documents/moose/packages/projects/src/petsc-3. 7.6/arch-linux2-c-opt/externalpackages/_d_v5.1.0-p4.tar.gz Extracting /home/vi/Documents/moose/ packages/projects/src/petsc-3.7.6/arch-linux2-c-opt/ externalpackages/_d_v5.1.0-p4.tar.gz Executing: cd /home/vi/Documents/moose/packages/projects/src/petsc-3. 7.6/arch-linux2-c-opt/externalpackages; chmod -R a+r petsc-pkg-metis-0adf3ea7785d;find petsc-pkg-metis-0adf3ea7785d -type d -name "*" -exec chmod a+rx {} \; Looking for METIS at git.metis, hg.metis or a directory starting with ['metis'] Could not locate an existing copy of METIS: ['fblaslapack-3.4.2', 'petsc-pkg-metis-0adf3ea7785d', 'hypre-2.11.1', 'scalapack-2.0.2', 'pkg-metis'] ERROR: Failed to download METIS And then the on screen error becomes: Unable to download METIS Failed to download METIS Same as previous post. It "downloads" the file but it cannot detect it for some reason. On Tuesday, June 5, 2018 at 3:08:17 PM UTC-4, jason.miller wrote: > > We ran into an issue like this recently. I believe this thread will help > you: MOOSE-Users: Error while installing PETSc > > > On Tue, Jun 5, 2018 at 1:34 PM, Vi Ha wrote: > >> I am trying to install petsc using these instructions: >> http://mooseframework.org/wiki/BasicManualInstallation/Linux/ >> on a linux system that isn't allowing me to download things directly. >> >> I am trying to install the dependent packages such as hypre, metis etc. >> does anyone know if there are instructions on how to download and install >> them manually? I know there are parameters in PETSC's configure file that >> allow me to do so but I don't know what they are. >> >> Thank you. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "moose-users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to moose-users... at googlegroups.com. >> Visit this group at https://groups.google.com/group/moose-users >> >> . >> To view this discussion on the web visit https://groups.google.com/d/ms >> gid/moose-users/ae2cfd8e-ce5c-4525-990c-cdd524814c8c%40googlegroups.com >> >> . >> For more options, visit https://groups.google.com/d/optout >> >> . >> > > -- You received this message because you are subscribed to the Google Groups "moose-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe at googlegroups.com. Visit this group at https://groups.google.com/group/moose-users. To view this discussion on the web visit https://groups.google.com/d/ msgid/moose-users/f2d4a0f1-3b81-4fe8-9acf-b1814ee6a290%40googlegroups.com . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "moose-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe at googlegroups.com. Visit this group at https://groups.google.com/group/moose-users. To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAN5Wd-JMXS6JErFfbSr%3DKKTSVArXVx1wLDL8kOHKTT4g4%2BNTaA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Jun 7 10:46:42 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 7 Jun 2018 15:46:42 +0000 Subject: [petsc-users] Fwd: Installing PETSC manually. In-Reply-To: References: Message-ID: <8BA93AC2-F33E-4017-8482-4EB90DC0C16A@anl.gov> The tarball creates a file petsc-pkg-metis-0adf3ea7785d which we do not check for. Looks like maint checks for the directory named petsc-pkg-metis Exactly how did you obtain the metis tarball? I will check this if it is not resolved later today. Flying back from the PETSc meeting now so cannot utilize the web. Barry On Jun 7, 2018, at 4:34 AM, Fande Kong wrote: Hi Satish, A MOOSE user has troubles to build Metis that is "downloaded" from a local directory. Do you have any idea? Vi, Could you share "configure.log" with PETSc team? Thanks, Fande, ---------- Forwarded message ---------- From: Vi Ha Date: Wed, Jun 6, 2018 at 11:00 AM Subject: Re: Installing PETSC manually. To: moose-users Hi Jason, Thanks for the reply. I am having the same exact error as you linked. Here's the full error message from configure.log: =============================================================================== Trying to download file:///home/vi/Documents/v-install-moose/files/petsc-config-backup/v5.1.0-p4.tar.gz for METIS =============================================================================== Downloading file:///home/vi/Documents/v-install-moose/files/petsc-config-backup/v5.1.0-p4.tar.gz to /home/vi/Documents/moose/packages/projects/src/petsc-3.7.6/arch-linux2-c-opt/externalpackages/_d_v5.1.0-p4.tar.gz Extracting /home/vi/Documents/moose/packages/projects/src/petsc-3.7.6/arch-linux2-c-opt/externalpackages/_d_v5.1.0-p4.tar.gz Executing: cd /home/vi/Documents/moose/packages/projects/src/petsc-3.7.6/arch-linux2-c-opt/externalpackages; chmod -R a+r petsc-pkg-metis-0adf3ea7785d;find petsc-pkg-metis-0adf3ea7785d -type d -name "*" -exec chmod a+rx {} \; Looking for METIS at git.metis, hg.metis or a directory starting with ['metis'] Could not locate an existing copy of METIS: ['fblaslapack-3.4.2', 'petsc-pkg-metis-0adf3ea7785d', 'hypre-2.11.1', 'scalapack-2.0.2', 'pkg-metis'] ERROR: Failed to download METIS And then the on screen error becomes: Unable to download METIS Failed to download METIS Same as previous post. It "downloads" the file but it cannot detect it for some reason. On Tuesday, June 5, 2018 at 3:08:17 PM UTC-4, jason.miller wrote: We ran into an issue like this recently. I believe this thread will help you: MOOSE-Users: Error while installing PETSc On Tue, Jun 5, 2018 at 1:34 PM, Vi Ha wrote: I am trying to install petsc using these instructions: http://mooseframework.org/wiki/BasicManualInstallation/Linux/ on a linux system that isn't allowing me to download things directly. I am trying to install the dependent packages such as hypre, metis etc. does anyone know if there are instructions on how to download and install them manually? I know there are parameters in PETSC's configure file that allow me to do so but I don't know what they are. Thank you. -- You received this message because you are subscribed to the Google Groups "moose-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to moose-users... at googlegroups.com. Visit this group at https://groups.google.com/group/moose-users. To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/ae2cfd8e-ce5c-4525-990c-cdd524814c8c%40googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "moose-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe at googlegroups.com. Visit this group at https://groups.google.com/group/moose-users. To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/f2d4a0f1-3b81-4fe8-9acf-b1814ee6a290%40googlegroups.com. For more options, visit https://groups.google.com/d/optout. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Jun 7 10:46:50 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 7 Jun 2018 15:46:50 +0000 Subject: [petsc-users] MatGetOwnershipIS in fortran In-Reply-To: References: Message-ID: Looks like the Fortran interface is missing. I will add it shortly. Barry > On Jun 7, 2018, at 3:55 AM, Marius Buerkle wrote: > > > Hi ! > > I try to use MatGetOwnershipIS in Fortran but i get an undefined reference to `matgetownershipis_' when linking my program. Is this available for Fortran ? From bsmith at mcs.anl.gov Thu Jun 7 10:46:58 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 7 Jun 2018 15:46:58 +0000 Subject: [petsc-users] Fwd: Installing PETSC manually. In-Reply-To: References: Message-ID: <7FABB13A-19CA-46E6-A51D-02BB170A91DB@anl.gov> The tarball creates a file petsc-pkg-metis-0adf3ea7785d which we do not check for. Looks like maint checks for the directory named petsc-pkg-metis Exactly how did you obtain the metis tarball? I will check this if it is not resolved later today. Flying back from the PETSc meeting now so cannot utilize the web. Barry > On Jun 7, 2018, at 4:34 AM, Fande Kong wrote: > > Hi Satish, > > A MOOSE user has troubles to build Metis that is "downloaded" from a local directory. Do you have any idea? > > Vi, > > Could you share "configure.log" with PETSc team? > > > Thanks, > > Fande, > > ---------- Forwarded message ---------- > From: Vi Ha > Date: Wed, Jun 6, 2018 at 11:00 AM > Subject: Re: Installing PETSC manually. > To: moose-users > > > Hi Jason, > > Thanks for the reply. > I am having the same exact error as you linked. Here's the full error message from configure.log: > > =============================================================================== > Trying to download file:///home/vi/Documents/v-install-moose/files/petsc-config-backup/v5.1.0-p4.tar.gz for METIS > =============================================================================== > > Downloading file:///home/vi/Documents/v-install-moose/files/petsc-config-backup/v5.1.0-p4.tar.gz to /home/vi/Documents/moose/packages/projects/src/petsc-3.7.6/arch-linux2-c-opt/externalpackages/_d_v5.1.0-p4.tar.gz > Extracting /home/vi/Documents/moose/packages/projects/src/petsc-3.7.6/arch-linux2-c-opt/externalpackages/_d_v5.1.0-p4.tar.gz > Executing: cd /home/vi/Documents/moose/packages/projects/src/petsc-3.7.6/arch-linux2-c-opt/externalpackages; chmod -R a+r petsc-pkg-metis-0adf3ea7785d;find petsc-pkg-metis-0adf3ea7785d -type d -name "*" -exec chmod a+rx {} \; > Looking for METIS at git.metis, hg.metis or a directory starting with ['metis'] > Could not locate an existing copy of METIS: > ['fblaslapack-3.4.2', 'petsc-pkg-metis-0adf3ea7785d', 'hypre-2.11.1', 'scalapack-2.0.2', 'pkg-metis'] > ERROR: Failed to download METIS > > And then the on screen error becomes: > > Unable to download METIS > Failed to download METIS > > Same as previous post. It "downloads" the file but it cannot detect it for some reason. > > > > On Tuesday, June 5, 2018 at 3:08:17 PM UTC-4, jason.miller wrote: > We ran into an issue like this recently. I believe this thread will help you: MOOSE-Users: Error while installing PETSc > > On Tue, Jun 5, 2018 at 1:34 PM, Vi Ha wrote: > I am trying to install petsc using these instructions: http://mooseframework.org/wiki/BasicManualInstallation/Linux/ > on a linux system that isn't allowing me to download things directly. > > I am trying to install the dependent packages such as hypre, metis etc. does anyone know if there are instructions on how to download and install them manually? I know there are parameters in PETSC's configure file that allow me to do so but I don't know what they are. > > Thank you. > > -- > You received this message because you are subscribed to the Google Groups "moose-users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to moose-users... at googlegroups.com. > Visit this group at https://groups.google.com/group/moose-users. > To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/ae2cfd8e-ce5c-4525-990c-cdd524814c8c%40googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to the Google Groups "moose-users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe at googlegroups.com. > Visit this group at https://groups.google.com/group/moose-users. > To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/f2d4a0f1-3b81-4fe8-9acf-b1814ee6a290%40googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > From bsmith at mcs.anl.gov Thu Jun 7 10:56:11 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 7 Jun 2018 15:56:11 +0000 Subject: [petsc-users] Petsc Installment on Cray In-Reply-To: References: Message-ID: <359135CE-B497-497B-A981-0CEC2C9817AB@anl.gov> Please send make.log Barry > On Jun 7, 2018, at 8:02 AM, Sharp Stone wrote: > > ?? > Dear all, > > I try to install petsc on cray. But the mpi seems always not working. Below is the error message. The configure log is also attached. Please help. Thank you in advance! > > > > /mnt/lustre/lus0/home/IAM851/jhk1009/local/petsc-3.7.6/src/sys/f90-mod/petscsysmod.F:6.11: > > use mpi > 1 > Fatal Error: Parse error when checking module version for file 'mpi.mod' opened at (1) > gmake[2]: *** [arch-linux2-c-debug/obj/src/sys/f90-mod/petscsysmod.o] Error 1 > gmake[2]: *** Waiting for unfinished jobs.... > gmake[2]: Leaving directory `/mnt/lustre/lus0/home/IAM851/jhk1009/local/petsc-3.7.6' > gmake[1]: *** [gnumake] Error 2 > gmake[1]: Leaving directory `/mnt/lustre/lus0/home/IAM851/jhk1009/local/petsc-3.7.6' > **************************ERROR************************************* > Error during compile, check arch-linux2-c-debug/lib/petsc/conf/make.log > Send it and arch-linux2-c-debug/lib/petsc/conf/configure.log to petsc-maint at mcs.anl.gov > ******************************************************************** > make: *** [all] Error 1 > > > -- > Best regards, > > Feng > From bsmith at mcs.anl.gov Thu Jun 7 11:13:25 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 7 Jun 2018 16:13:25 +0000 Subject: [petsc-users] MatGetOwnershipIS in fortran In-Reply-To: References: Message-ID: <738E147D-5C12-4B68-B28B-279E828ED502@anl.gov> Here is the patch; if it works for you it will go into the maint and master branches Barry > On Jun 7, 2018, at 3:55 AM, Marius Buerkle wrote: > > > Hi ! > > I try to use MatGetOwnershipIS in Fortran but i get an undefined reference to `matgetownershipis_' when linking my program. Is this available for Fortran ? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: matgetownershipis-fortran.patch Type: application/octet-stream Size: 1506 bytes Desc: matgetownershipis-fortran.patch URL: From Anthony.J.Ruth.12 at nd.edu Thu Jun 7 12:37:00 2018 From: Anthony.J.Ruth.12 at nd.edu (Anthony Ruth) Date: Thu, 7 Jun 2018 13:37:00 -0400 Subject: [petsc-users] tmat is not C interoperable Message-ID: Hello, I am trying to call a function in a C library from fortran. The fortran code should pass a Mat object to the C code. I defined the interface within the fortran code like this: #include interface subroutine MY_FUNC(A_Mat) bind ( C, name = "MY_FUNC" ) use petscmat Mat, intent(in) :: A_Mat end subroutine MY_FUNC end interface When I try to compile the code it produces this error: Variable ?a_mat? at (1) is a dummy argument to the BIND(C) procedure ?MY_FUNC? but is not C interoperable because derived type ?tmat? is not C interoperable My interpretation of the error is that there is some mismatch between the Mat data type defined in fortran and the Mat data type defined in C, so that I cannot pass one to the other. Is this true? Is there a compatible way to pass the Mat? Are the C and fortran implementations actually properly aligned and interchangeable so that all that is needed is something to tell the compiler to try anyway? I tried changing the data type of A_Mat to Integer Pointer (void *), but ran into a syntax error when attempting that. regards, Anthony Ruth NASA Space Technology Research Fellow Condensed Matter Theory University of Notre Dame -------------- next part -------------- An HTML attachment was scrubbed... URL: From olk548 at mail.usask.ca Thu Jun 7 15:46:47 2018 From: olk548 at mail.usask.ca (Oleksandr Koshkarov) Date: Thu, 7 Jun 2018 14:46:47 -0600 Subject: [petsc-users] Problem with options and PetscFOpen/fopen Message-ID: <86bc8742-59a4-11d0-6f57-010bcace4b3d@mail.usask.ca> Hello all, I am trying to read options using petsc capabilities and it is working fine. However, if I open file to read with fopen/PetscFOpen some of options stopped being read. I think It may be a bug or I am doing something very stupid and cannot see it. Here is the snippets of what I do: PetscBool flag; int path_str_size = 100; char str[path_str_size]; PetscOptionsGetString(NULL,NULL,"--config",str,path_str_size,&flag); if (flag) ? PetscOptionsInsert(NULL,argc,args,str); else ? try_again("Please specify input file: \"--config config_path\"\n"); // Loading options: PetscOptionsGetReal(NULL,NULL,"--dt",&(env->dt),&flag); if (!flag) try_again("Please specify dt: \"--dt value\"\n"); .... PetscOptionsGetInt(NULL,NULL,"--Ns",&(env->Ns),&flag); if (!flag) try_again("Please specify Ns: \"--Ns value\"\n"); for (int s; sNs; s++) { ? sprintf(str,"--mass%d",s); PetscOptionsGetReal(NULL,NULL,str,&(env->species[s].mass),&flag); ? if (!flag) try_again("Please specify all masses: \"--mass value\" (N starts from 0)\n"); ? ... } PetscOptionsGetString(NULL,NULL,"--grid",str,path_str_size,&flag); if (!flag) try_again("Please specify grid file \"--grid path\"\n"); // This code alone works fine without a trouble! // but if I add next three lines, options which are in the loop stopped being read (i.e., --mass0, --mass1, etc) (I am even getting a warning from petsc that they are not being used // while other options (i.e., --dt, --Ns) are alright FILE *fp; PetscFOpen(PETSC_COMM_SELF,str,"r",&fp); // if I use fopen or different variable (not str) - the same effect. PetscFClose(PETSC_COMM_SELF,fp); It seems it is possible to avoid this problem if you wrap 3 last lines in the separate function. p.s. Am I doing something stupid? Best regards, Alex. From bsmith at mcs.anl.gov Thu Jun 7 18:54:36 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 7 Jun 2018 23:54:36 +0000 Subject: [petsc-users] tmat is not C interoperable In-Reply-To: References: Message-ID: <0DB5E3BC-A18B-4F0F-8659-16FAD82CCD67@anl.gov> Please find attached an example that uses the bind C and passes a PETSc object through. Note that you cannot pass directly a Fortran derived type to C. Barry > On Jun 7, 2018, at 12:37 PM, Anthony Ruth wrote: > > Hello, > > I am trying to call a function in a C library from fortran. The fortran code should pass a Mat object to the C code. I defined the interface within the fortran code like this: > > #include > > interface > subroutine MY_FUNC(A_Mat) bind ( C, name = "MY_FUNC" ) > use petscmat > Mat, intent(in) :: A_Mat > end subroutine MY_FUNC > end interface > > When I try to compile the code it produces this error: > > Variable ?a_mat? at (1) is a dummy argument to the BIND(C) procedure ?MY_FUNC? but is not C interoperable because derived type ?tmat? is not C interoperable > > My interpretation of the error is that there is some mismatch between the Mat data type defined in fortran and the Mat data type defined in C, so that I cannot pass one to the other. Is this true? Is there a compatible way to pass the Mat? Are the C and fortran implementations actually properly aligned and interchangeable so that all that is needed is something to tell the compiler to try anyway? I tried changing the data type of A_Mat to Integer Pointer (void *), but ran into a syntax error when attempting that. > > regards, > Anthony Ruth > NASA Space Technology Research Fellow > Condensed Matter Theory > University of Notre Dame -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex43.c Type: application/octet-stream Size: 211 bytes Desc: ex43.c URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex43f.F90 Type: application/octet-stream Size: 1101 bytes Desc: ex43f.F90 URL: From mbuerkle at web.de Thu Jun 7 22:08:43 2018 From: mbuerkle at web.de (Marius Buerkle) Date: Fri, 8 Jun 2018 05:08:43 +0200 Subject: [petsc-users] MatGetOwnershipIS in fortran In-Reply-To: <738E147D-5C12-4B68-B28B-279E828ED502@anl.gov> References: <738E147D-5C12-4B68-B28B-279E828ED502@anl.gov> Message-ID: Thanks it worked. > > > Here is the patch; if it works for you it will go into the maint and master branches > > Barry > > > > On Jun 7, 2018, at 3:55 AM, Marius Buerkle wrote: > > > > > > Hi ! > > > > I try to use MatGetOwnershipIS in Fortran but i get an undefined reference to `matgetownershipis_' when linking my program. Is this available for Fortran ? > > From matteo.semplice at unito.it Fri Jun 8 04:10:07 2018 From: matteo.semplice at unito.it (Matteo Semplice) Date: Fri, 8 Jun 2018 11:10:07 +0200 Subject: [petsc-users] removing cells from existing DMPLEX Message-ID: Hi. We are developing a ghost-fluid type scheme, so we start creating a (cartesian) grid and then "mark" some cells in which the evolution actually takes place and solve a PDE only there. We currently handle the grid via a DMPLEX, we through a Label that marks the fluid cells we define a trivial operator for the non-fluid part of the grid, but of course the dofs attached to those cells and faces are still there in the Vec's and in the output. We know that (sometimes large) portions of the original grid will never be touched during the evolution, so we would like to remove those from the DMPLEX before partitioning it. Is there a simple way to do it? Ideally automagically via a Label or an IndexSet... The second best option would be to give less weight to the non-fluid portion of the DM when partitioning the DMPLEX: is there an example showing the use of weights in a Plex partitioner? Thanks in advance for your time. Best, ?? Matteo From olk548 at mail.usask.ca Fri Jun 8 06:44:27 2018 From: olk548 at mail.usask.ca (Oleksandr Koshkarov) Date: Fri, 8 Jun 2018 05:44:27 -0600 Subject: [petsc-users] Problem with options and PetscFOpen/fopen In-Reply-To: <86bc8742-59a4-11d0-6f57-010bcace4b3d@mail.usask.ca> References: <86bc8742-59a4-11d0-6f57-010bcace4b3d@mail.usask.ca> Message-ID: <80dfff99-5273-d0de-64d8-af3ea0ed82b1@mail.usask.ca> Never mind! Sorry I found my stupid mistake - it is not related to PETSc :) Sorry, Alex. On 06/07/2018 02:46 PM, Oleksandr Koshkarov wrote: > Hello all, > > I am trying to read options using petsc capabilities and it is working > fine. > > However, if I open file to read with fopen/PetscFOpen some of options > stopped being read. > > I think It may be a bug or I am doing something very stupid and cannot > see it. > > Here is the snippets of what I do: > > PetscBool flag; > int path_str_size = 100; > char str[path_str_size]; > PetscOptionsGetString(NULL,NULL,"--config",str,path_str_size,&flag); > if (flag) > ? PetscOptionsInsert(NULL,argc,args,str); > else > ? try_again("Please specify input file: \"--config config_path\"\n"); > > // Loading options: > PetscOptionsGetReal(NULL,NULL,"--dt",&(env->dt),&flag); > if (!flag) try_again("Please specify dt: \"--dt value\"\n"); > > .... > > PetscOptionsGetInt(NULL,NULL,"--Ns",&(env->Ns),&flag); > if (!flag) try_again("Please specify Ns: \"--Ns value\"\n"); > > for (int s; sNs; s++) { > > ? sprintf(str,"--mass%d",s); > PetscOptionsGetReal(NULL,NULL,str,&(env->species[s].mass),&flag); > ? if (!flag) try_again("Please specify all masses: \"--mass value\" > (N starts from 0)\n"); > > ? ... > } > > > PetscOptionsGetString(NULL,NULL,"--grid",str,path_str_size,&flag); > if (!flag) try_again("Please specify grid file \"--grid path\"\n"); > > > // This code alone works fine without a trouble! > // but if I add next three lines, options which are in the loop > stopped being read (i.e., --mass0, --mass1, etc) (I am even getting a > warning from petsc that they are not being used > // while other options (i.e., --dt, --Ns) are alright > FILE *fp; > PetscFOpen(PETSC_COMM_SELF,str,"r",&fp); // if I use fopen or > different variable (not str) - the same effect. > PetscFClose(PETSC_COMM_SELF,fp); > > It seems it is possible to avoid this problem if you wrap 3 last lines > in the separate function. > > p.s. Am I doing something stupid? > > Best regards, > Alex. > > From valerio.barnabei at gmail.com Fri Jun 8 07:45:02 2018 From: valerio.barnabei at gmail.com (Valerio Barnabei) Date: Fri, 8 Jun 2018 14:45:02 +0200 Subject: [petsc-users] DMPlex CGNS import Message-ID: Hello, I'm trying to import a simple CGNS file representing a simple 3D unstructured box, but i keep getting the same error: [0]PETSC ERROR: Error in external library > [0]PETSC ERROR: CGNS file must have a single section, not 2 > I'm using the following script: > #include > #include > > > int main(int argc, char **argv){ > > char MeshFileName[2048]; > MPI_Comm comm; > DM dm; > PetscErrorCode ierr; > PetscBool flg; > > ierr = PetscInitialize(&argc, &argv, NULL,NULL);if (ierr) return ierr; > ierr = > PetscOptionsGetString(NULL,NULL,"-file",MeshFileName,sizeof(MeshFileName),&flg);CHKERRQ(ierr); > comm= PETSC_COMM_WORLD; > if(!flg){ > SETERRQ(comm,PETSC_ERR_ARG_NULL,"please specify a fine mesh > file \n"); > }; > > ierr = DMPlexCreateCGNSFromFile(comm, MeshFileName, PETSC_FALSE, > &dm);CHKERRQ(ierr); > > ierr = DMDestroy(&dm);CHKERRQ(ierr); > ierr = PetscFinalize();CHKERRQ(ierr); > > return ierr; > } > I'm not using a multiblock mesh, i just created a single unstructured block volume in pointwise, exported as adf (using hdf i got a crash during file opening so i switched). I tried with different exportation options (node to node, cell to cell, face to face) but nothing seems to work. I got the same identical error also downloading files from official CGNS site: http://cgns.sourceforge.net/CGNSFiles.html It's obvious there's something I'm missing. -Where is my mistake? -Is there any notable limitation in the importation of CGNS file? (e.g. multiblock files can't be imported as dmplex objects) Thanks in advance for your kind help. Valerio -------------- next part -------------- An HTML attachment was scrubbed... URL: From wesley.bland at intel.com Fri Jun 8 10:58:23 2018 From: wesley.bland at intel.com (Bland, Wesley) Date: Fri, 8 Jun 2018 15:58:23 +0000 Subject: [petsc-users] MPI Error Handling Inside PETSc Message-ID: Hi PETSc folks, In the MPI Forum, we're getting close to adopting a proposal to change the default communicator where errors are raised when they don't have anywhere else to go (think something like MPI_ALLOC_MEM, which doesn't have a communicator). Instead of getting those errors on the error handler for MPI_COMM_WORLD, it would move to MPI_COMM_SELF to allow more local error handling. Rather than having an error for passing an invalid argument potentially cause all processes to trigger the error handler, only the local process would see it. This doesn't impact normal error handling, such as if an MPI_RECV fails for some reason. That would trigger the error handler attached to the communicator in the receive. However, this is potentially a backward incompatible change due to the fact that people might be changing the error handler of MPI_COMM_WORLD, but not MPI_COMM_SELF. The details can be found on this GitHub issue and in this PDF (search for ticket3). I see that in PETSc, you guys do some basic error handling by changing the default error handler to MPI_ERRORS_RETURN on PETSC_COMM_WORLD, which may or may not be equal to MPI_COMM_WORLD. So in this case, I believe that in order to get the same error handling in all possible (though unlikely) cases, you would also need to set the same error handler on MPI_COMM_SELF. Probably something along the lines of: MPI_Comm_set_errhandler(MPI_COMM_SELF, MPI_ERRORS_RETURN); Or, if you want to preserve the user error handler: MPI_Comm_get_errhandler(MPI_COMM_SELF, &orig_errhandler); if (orig_errhandler != MPI_ERRORS_ARE_FATAL) { /* Create custom error handler to deal with internal * PETSc errors and then call the user's error handler */ } Before we vote this in next week, we wanted to reach out to some users to see if you have strong opinions about this. Despite the fact that this will have some impact on users, we think this is the right way to go to improve error management in the MPI Standard (there are other efforts going on if you're interested). Thanks, Wesley Bland -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Jun 8 11:28:08 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 8 Jun 2018 16:28:08 +0000 Subject: [petsc-users] MPI Error Handling Inside PETSc In-Reply-To: References: Message-ID: Wesley, Thanks for the email; I don't see this as having any effect on our usage of MPI (aside from the small addition). Barry > On Jun 8, 2018, at 10:58 AM, Bland, Wesley wrote: > > Hi PETSc folks, > > In the MPI Forum, we're getting close to adopting a proposal to change the default communicator where errors are raised when they don't have anywhere else to go (think something like MPI_ALLOC_MEM, which doesn't have a communicator). Instead of getting those errors on the error handler for MPI_COMM_WORLD, it would move to MPI_COMM_SELF to allow more local error handling. Rather than having an error for passing an invalid argument potentially cause all processes to trigger the error handler, only the local process would see it. > > This doesn't impact normal error handling, such as if an MPI_RECV fails for some reason. That would trigger the error handler attached to the communicator in the receive. However, this is potentially a backward incompatible change due to the fact that people might be changing the error handler of MPI_COMM_WORLD, but not MPI_COMM_SELF. > > The details can be found on this GitHub issue and in this PDF (search for ticket3). > > I see that in PETSc, you guys do some basic error handling by changing the default error handler to MPI_ERRORS_RETURN on PETSC_COMM_WORLD, which may or may not be equal to MPI_COMM_WORLD. So in this case, I believe that in order to get the same error handling in all possible (though unlikely) cases, you would also need to set the same error handler on MPI_COMM_SELF. Probably something along the lines of: > > MPI_Comm_set_errhandler(MPI_COMM_SELF, MPI_ERRORS_RETURN); > > Or, if you want to preserve the user error handler: > > MPI_Comm_get_errhandler(MPI_COMM_SELF, &orig_errhandler); > if (orig_errhandler != MPI_ERRORS_ARE_FATAL) { > /* Create custom error handler to deal with internal > * PETSc errors and then call the user's error handler */ > } > > Before we vote this in next week, we wanted to reach out to some users to see if you have strong opinions about this. Despite the fact that this will have some impact on users, we think this is the right way to go to improve error management in the MPI Standard (there are other efforts going on if you're interested). > > Thanks, > Wesley Bland From balay at mcs.anl.gov Fri Jun 8 12:15:29 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 8 Jun 2018 12:15:29 -0500 Subject: [petsc-users] Fwd: Installing PETSC manually. In-Reply-To: <7FABB13A-19CA-46E6-A51D-02BB170A91DB@anl.gov> References: <7FABB13A-19CA-46E6-A51D-02BB170A91DB@anl.gov> Message-ID: >>>>> Extracting /home/vi/Documents/moose/packages/projects/src/petsc-3.7.6/arch-linux2-c-opt/externalpackages/_d_v5.1.0-p4.tar.gz <<<<<< Check for 'petsc-pkg-metis' is correct - when the dir is 'petsc-pkg-metis-0adf3ea7785d'. Looks like this issue was introduced in : v3.7.6 - and fixed in v3.7.7 https://bitbucket.org/petsc/petsc/commits/6d90219d8fa4f8b2fd242cc6ff9008358d87c26f https://bitbucket.org/petsc/petsc/commits/d6a7209b814d57568c8cb79bddb09b20cc814191 Generally one should use the latest patch release traball for any given (major/minor) release. http://www.mcs.anl.gov/petsc/download/ i.e petsc-3.7.7.tar.gz for petsc-3.7 release. Satish On Thu, 7 Jun 2018, Smith, Barry F. wrote: > > The tarball creates a file petsc-pkg-metis-0adf3ea7785d which we do not check for. > > Looks like maint checks for the directory named petsc-pkg-metis > > Exactly how did you obtain the metis tarball? > > I will check this if it is not resolved later today. Flying back from the PETSc meeting now so cannot utilize the web. > > Barry > > > > > > On Jun 7, 2018, at 4:34 AM, Fande Kong wrote: > > > > Hi Satish, > > > > A MOOSE user has troubles to build Metis that is "downloaded" from a local directory. Do you have any idea? > > > > Vi, > > > > Could you share "configure.log" with PETSc team? > > > > > > Thanks, > > > > Fande, > > > > ---------- Forwarded message ---------- > > From: Vi Ha > > Date: Wed, Jun 6, 2018 at 11:00 AM > > Subject: Re: Installing PETSC manually. > > To: moose-users > > > > > > Hi Jason, > > > > Thanks for the reply. > > I am having the same exact error as you linked. Here's the full error message from configure.log: > > > > =============================================================================== > > Trying to download file:///home/vi/Documents/v-install-moose/files/petsc-config-backup/v5.1.0-p4.tar.gz for METIS > > =============================================================================== > > > > Downloading file:///home/vi/Documents/v-install-moose/files/petsc-config-backup/v5.1.0-p4.tar.gz to /home/vi/Documents/moose/packages/projects/src/petsc-3.7.6/arch-linux2-c-opt/externalpackages/_d_v5.1.0-p4.tar.gz > > Extracting /home/vi/Documents/moose/packages/projects/src/petsc-3.7.6/arch-linux2-c-opt/externalpackages/_d_v5.1.0-p4.tar.gz > > Executing: cd /home/vi/Documents/moose/packages/projects/src/petsc-3.7.6/arch-linux2-c-opt/externalpackages; chmod -R a+r petsc-pkg-metis-0adf3ea7785d;find petsc-pkg-metis-0adf3ea7785d -type d -name "*" -exec chmod a+rx {} \; > > Looking for METIS at git.metis, hg.metis or a directory starting with ['metis'] > > Could not locate an existing copy of METIS: > > ['fblaslapack-3.4.2', 'petsc-pkg-metis-0adf3ea7785d', 'hypre-2.11.1', 'scalapack-2.0.2', 'pkg-metis'] > > ERROR: Failed to download METIS > > > > And then the on screen error becomes: > > > > Unable to download METIS > > Failed to download METIS > > > > Same as previous post. It "downloads" the file but it cannot detect it for some reason. > > > > > > > > On Tuesday, June 5, 2018 at 3:08:17 PM UTC-4, jason.miller wrote: > > We ran into an issue like this recently. I believe this thread will help you: MOOSE-Users: Error while installing PETSc > > > > On Tue, Jun 5, 2018 at 1:34 PM, Vi Ha wrote: > > I am trying to install petsc using these instructions: http://mooseframework.org/wiki/BasicManualInstallation/Linux/ > > on a linux system that isn't allowing me to download things directly. > > > > I am trying to install the dependent packages such as hypre, metis etc. does anyone know if there are instructions on how to download and install them manually? I know there are parameters in PETSC's configure file that allow me to do so but I don't know what they are. > > > > Thank you. > > > > -- > > You received this message because you are subscribed to the Google Groups "moose-users" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to moose-users... at googlegroups.com. > > Visit this group at https://groups.google.com/group/moose-users. > > To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/ae2cfd8e-ce5c-4525-990c-cdd524814c8c%40googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > > > > > -- > > You received this message because you are subscribed to the Google Groups "moose-users" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe at googlegroups.com. > > Visit this group at https://groups.google.com/group/moose-users. > > To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/f2d4a0f1-3b81-4fe8-9acf-b1814ee6a290%40googlegroups.com. > > > > For more options, visit https://groups.google.com/d/optout. > > > From balay at mcs.anl.gov Fri Jun 8 12:24:49 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 8 Jun 2018 12:24:49 -0500 Subject: [petsc-users] Petsc Installment on Cray In-Reply-To: <359135CE-B497-497B-A981-0CEC2C9817AB@anl.gov> References: <359135CE-B497-497B-A981-0CEC2C9817AB@anl.gov> Message-ID: 1. confiure.log for me looks corrupted [don't know if my mailer was responsible for this corruption - or if it was previously corrupted. 2. Suggesting doing 'rm -rf arch-linux2-c-debug' - and redoing the build - and see if the problem perisists. If so - one Workarround for this is to remove lines related to 'PETSC_HAVE_MPI_F90MODULE' flag from PETSC_ARCH/include/petscconf.h - and then recompile the libraries [i.e 'make all' - without running configure again] Subsequent releases of petsc should have fixes for improved PETSC_HAVE_MPI_F90MODULE test. [However I don't know why this failure would occur with cray-mpi] Satish On Thu, 7 Jun 2018, Smith, Barry F. wrote: > > Please send make.log > > Barry > > > > On Jun 7, 2018, at 8:02 AM, Sharp Stone wrote: > > > > ?? > > Dear all, > > > > I try to install petsc on cray. But the mpi seems always not working. Below is the error message. The configure log is also attached. Please help. Thank you in advance! > > > > > > > > /mnt/lustre/lus0/home/IAM851/jhk1009/local/petsc-3.7.6/src/sys/f90-mod/petscsysmod.F:6.11: > > > > use mpi > > 1 > > Fatal Error: Parse error when checking module version for file 'mpi.mod' opened at (1) > > gmake[2]: *** [arch-linux2-c-debug/obj/src/sys/f90-mod/petscsysmod.o] Error 1 > > gmake[2]: *** Waiting for unfinished jobs.... > > gmake[2]: Leaving directory `/mnt/lustre/lus0/home/IAM851/jhk1009/local/petsc-3.7.6' > > gmake[1]: *** [gnumake] Error 2 > > gmake[1]: Leaving directory `/mnt/lustre/lus0/home/IAM851/jhk1009/local/petsc-3.7.6' > > **************************ERROR************************************* > > Error during compile, check arch-linux2-c-debug/lib/petsc/conf/make.log > > Send it and arch-linux2-c-debug/lib/petsc/conf/configure.log to petsc-maint at mcs.anl.gov > > ******************************************************************** > > make: *** [all] Error 1 > > > > > > -- > > Best regards, > > > > Feng > > > > From shash_91 at hotmail.com Fri Jun 8 13:42:27 2018 From: shash_91 at hotmail.com (Shashwat Sharma) Date: Fri, 8 Jun 2018 18:42:27 +0000 Subject: [petsc-users] Installation of prerequisite libraries Message-ID: Hello, I'm using a Petsc installation along with the external package superLU_dist. As per the instructions on the online documentation, I am letting Petsc take care of the download and installation at the configure stage. SuperLU requires that I also download and install parMeTiS and PTScotch, which I can readily do the same way. However, those packages in turn require certain other packages to have been preinstalled, namely GNU bison and flex. The issue is that I have to make sure bison and flex are pre-installed globally (with sudo rights) before I can properly configure Petsc. However, I would like to remove this dependency on sudo rights and be able to point Petsc to a local installation of bison and flex (or even better, have Petsc download and install them). Is this possible? I didn't see these packages listed on https://www.mcs.anl.gov/petsc/miscellaneous/external.html, so I'm not sure if Petsc can automatically download them. Is there a way to point Petsc to a local installation of bison and flex? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jun 8 13:43:06 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 8 Jun 2018 14:43:06 -0400 Subject: [petsc-users] removing cells from existing DMPLEX In-Reply-To: References: Message-ID: On Fri, Jun 8, 2018 at 5:10 AM, Matteo Semplice wrote: > Hi. > > We are developing a ghost-fluid type scheme, so we start creating a > (cartesian) grid and then "mark" some cells in which the evolution actually > takes place and solve a PDE only there. > > We currently handle the grid via a DMPLEX, we through a Label that marks > the fluid cells we define a trivial operator for the non-fluid part of the > grid, but of course the dofs attached to those cells and faces are still > there in the Vec's and in the output. > > We know that (sometimes large) portions of the original grid will never be > touched during the evolution, so we would like to remove those from the > DMPLEX before partitioning it. Is there a simple way to do it? Ideally > automagically via a Label or an IndexSet... > Can this do what you want? http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMPLEX/DMPlexFilter.html If so, I would caution you that no one uses this right now, so it might forget to do something (say edit out all those things from existing labels). I will fix anything that gives you a problem. Thanks, Matt > The second best option would be to give less weight to the non-fluid > portion of the DM when partitioning the DMPLEX: is there an example showing > the use of weights in a Plex partitioner? > > Thanks in advance for your time. > > Best, > > Matteo > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Jun 8 13:47:22 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 8 Jun 2018 18:47:22 +0000 Subject: [petsc-users] Installation of prerequisite libraries In-Reply-To: References: Message-ID: <4C26B7E1-24BD-4FD1-BA5B-BEC489335BB6@anl.gov> Sorry. No PETSc can't install bison and flex and we won't add that capability bison and flex are so standard that any standard distribution should either contain them or have an easy way to install them. Perhaps you are working on a machine without bison and flex and you do not have sudo access. I suggest you email/talk to the sys admin for this machine and ask them to install these packages. Barry > On Jun 8, 2018, at 1:42 PM, Shashwat Sharma wrote: > > Hello, > > I'm using a Petsc installation along with the external package superLU_dist. As per the instructions on the online documentation, I am letting Petsc take care of the download and installation at the configure stage. > > SuperLU requires that I also download and install parMeTiS and PTScotch, which I can readily do the same way. However, those packages in turn require certain other packages to have been preinstalled, namely GNU bison and flex. > > The issue is that I have to make sure bison and flex are pre-installed globally (with sudo rights) before I can properly configure Petsc. However, I would like to remove this dependency on sudo rights and be able to point Petsc to a local installation of bison and flex (or even better, have Petsc download and install them). Is this possible? I didn't see these packages listed on https://www.mcs.anl.gov/petsc/miscellaneous/external.html, so I'm not sure if Petsc can automatically download them. > > Is there a way to point Petsc to a local installation of bison and flex? > > Thanks! > > > From knepley at gmail.com Fri Jun 8 13:49:52 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 8 Jun 2018 14:49:52 -0400 Subject: [petsc-users] DMPlex CGNS import In-Reply-To: References: Message-ID: On Fri, Jun 8, 2018 at 8:45 AM, Valerio Barnabei wrote: > Hello, > I'm trying to import a simple CGNS file representing a simple 3D > unstructured box, but i keep getting the same error: > > [0]PETSC ERROR: Error in external library >> [0]PETSC ERROR: CGNS file must have a single section, not 2 >> > CGNS support stinks because I never used it and no one else has requested it. Also the format is not workable in parallel. Here is the code https://bitbucket.org/petsc/petsc/src/58edc67f565b6826877b6037e2237b6b17930a07/src/dm/impls/plex/plexcgns.c#lines-149 You can see I am just asking CGNS how many sections it has in the the zone. I do not really know what a section is, so I am limiting it to one section. The files you have must have multiple sections for reason that are not clear to me. Here are some meshes I can read: https://bitbucket.org/petsc/petsc/src/master/share/petsc/datafiles/meshes/ Thanks, Matt I'm using the following script: > > >> #include >> #include >> >> >> int main(int argc, char **argv){ >> >> char MeshFileName[2048]; >> MPI_Comm comm; >> DM dm; >> PetscErrorCode ierr; >> PetscBool flg; >> >> ierr = PetscInitialize(&argc, &argv, NULL,NULL);if (ierr) return ierr; >> ierr = PetscOptionsGetString(NULL,NULL,"-file",MeshFileName, >> sizeof(MeshFileName),&flg);CHKERRQ(ierr); >> comm= PETSC_COMM_WORLD; >> if(!flg){ >> SETERRQ(comm,PETSC_ERR_ARG_NULL,"please specify a fine >> mesh file \n"); >> }; >> >> ierr = DMPlexCreateCGNSFromFile(comm, MeshFileName, PETSC_FALSE, >> &dm);CHKERRQ(ierr); >> >> ierr = DMDestroy(&dm);CHKERRQ(ierr); >> ierr = PetscFinalize();CHKERRQ(ierr); >> >> return ierr; >> } >> > > > I'm not using a multiblock mesh, i just created a single unstructured > block volume in pointwise, exported as adf (using hdf i got a crash during > file opening so i switched). I tried with different exportation options > (node to node, cell to cell, face to face) but nothing seems to work. > I got the same identical error also downloading files from official CGNS > site: > http://cgns.sourceforge.net/CGNSFiles.html > It's obvious there's something I'm missing. > > -Where is my mistake? > -Is there any notable limitation in the importation of CGNS file? (e.g. > multiblock files can't be imported as dmplex objects) > > Thanks in advance for your kind help. > > Valerio > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From shash_91 at hotmail.com Fri Jun 8 13:50:18 2018 From: shash_91 at hotmail.com (Shashwat Sharma) Date: Fri, 8 Jun 2018 18:50:18 +0000 Subject: [petsc-users] Installation of prerequisite libraries In-Reply-To: <4C26B7E1-24BD-4FD1-BA5B-BEC489335BB6@anl.gov> References: <4C26B7E1-24BD-4FD1-BA5B-BEC489335BB6@anl.gov> Message-ID: Alright, thanks for the quick response. On Fri, Jun 8, 2018 at 2:47 PM Smith, Barry F. > wrote: Sorry. No PETSc can't install bison and flex and we won't add that capability bison and flex are so standard that any standard distribution should either contain them or have an easy way to install them. Perhaps you are working on a machine without bison and flex and you do not have sudo access. I suggest you email/talk to the sys admin for this machine and ask them to install these packages. Barry > On Jun 8, 2018, at 1:42 PM, Shashwat Sharma > wrote: > > Hello, > > I'm using a Petsc installation along with the external package superLU_dist. As per the instructions on the online documentation, I am letting Petsc take care of the download and installation at the configure stage. > > SuperLU requires that I also download and install parMeTiS and PTScotch, which I can readily do the same way. However, those packages in turn require certain other packages to have been preinstalled, namely GNU bison and flex. > > The issue is that I have to make sure bison and flex are pre-installed globally (with sudo rights) before I can properly configure Petsc. However, I would like to remove this dependency on sudo rights and be able to point Petsc to a local installation of bison and flex (or even better, have Petsc download and install them). Is this possible? I didn't see these packages listed on https://www.mcs.anl.gov/petsc/miscellaneous/external.html, so I'm not sure if Petsc can automatically download them. > > Is there a way to point Petsc to a local installation of bison and flex? > > Thanks! > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From valerio.barnabei at gmail.com Fri Jun 8 14:01:44 2018 From: valerio.barnabei at gmail.com (Valerio Barnabei) Date: Fri, 8 Jun 2018 21:01:44 +0200 Subject: [petsc-users] DMPlex CGNS import In-Reply-To: References: Message-ID: Thank you Matt for your advice, i will let you know if something pops out from the mist. Il 08 giu 2018 20:49, "Matthew Knepley" ha scritto: On Fri, Jun 8, 2018 at 8:45 AM, Valerio Barnabei wrote: > Hello, > I'm trying to import a simple CGNS file representing a simple 3D > unstructured box, but i keep getting the same error: > > [0]PETSC ERROR: Error in external library >> [0]PETSC ERROR: CGNS file must have a single section, not 2 >> > CGNS support stinks because I never used it and no one else has requested it. Also the format is not workable in parallel. Here is the code https://bitbucket.org/petsc/petsc/src/58edc67f565b6826877b6037e2237b6b17930a07/src/dm/impls/plex/plexcgns.c#lines-149 You can see I am just asking CGNS how many sections it has in the the zone. I do not really know what a section is, so I am limiting it to one section. The files you have must have multiple sections for reason that are not clear to me. Here are some meshes I can read: https://bitbucket.org/petsc/petsc/src/master/share/petsc/datafiles/meshes/ Thanks, Matt I'm using the following script: > > >> #include >> #include >> >> >> int main(int argc, char **argv){ >> >> char MeshFileName[2048]; >> MPI_Comm comm; >> DM dm; >> PetscErrorCode ierr; >> PetscBool flg; >> >> ierr = PetscInitialize(&argc, &argv, NULL,NULL);if (ierr) return ierr; >> ierr = >> PetscOptionsGetString(NULL,NULL,"-file",MeshFileName,sizeof(MeshFileName),&flg);CHKERRQ(ierr); >> comm= PETSC_COMM_WORLD; >> if(!flg){ >> SETERRQ(comm,PETSC_ERR_ARG_NULL,"please specify a fine >> mesh file \n"); >> }; >> >> ierr = DMPlexCreateCGNSFromFile(comm, MeshFileName, PETSC_FALSE, >> &dm);CHKERRQ(ierr); >> >> ierr = DMDestroy(&dm);CHKERRQ(ierr); >> ierr = PetscFinalize();CHKERRQ(ierr); >> >> return ierr; >> } >> > > > I'm not using a multiblock mesh, i just created a single unstructured > block volume in pointwise, exported as adf (using hdf i got a crash during > file opening so i switched). I tried with different exportation options > (node to node, cell to cell, face to face) but nothing seems to work. > I got the same identical error also downloading files from official CGNS > site: > http://cgns.sourceforge.net/CGNSFiles.html > It's obvious there's something I'm missing. > > -Where is my mistake? > -Is there any notable limitation in the importation of CGNS file? (e.g. > multiblock files can't be imported as dmplex objects) > > Thanks in advance for your kind help. > > Valerio > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From matteo.semplice at unito.it Sat Jun 9 10:39:11 2018 From: matteo.semplice at unito.it (Matteo Semplice) Date: Sat, 9 Jun 2018 17:39:11 +0200 Subject: [petsc-users] removing cells from existing DMPLEX In-Reply-To: References: Message-ID: <45fab488-2d41-cf8f-b971-0a88864875a2@unito.it> On 08/06/2018 20:43, Matthew Knepley wrote: > On Fri, Jun 8, 2018 at 5:10 AM, Matteo Semplice > > wrote: > > Hi. > > We are developing a ghost-fluid type scheme, so we start creating > a (cartesian) grid and then "mark" some cells in which the > evolution actually takes place and solve a PDE only there. > > We currently handle the grid via a DMPLEX, we through a Label that > marks the fluid cells we define a trivial operator for the > non-fluid part of the grid, but of course the dofs attached to > those cells and faces are still there in the Vec's and in the output. > > We know that (sometimes large) portions of the original grid will > never be touched during the evolution, so we would like to remove > those from the DMPLEX before partitioning it. Is there a simple > way to do it? Ideally automagically via a Label or an IndexSet... > > > Can this do what you want? > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMPLEX/DMPlexFilter.html > > If so, I would caution you that no one uses this right now, so it > might forget to do something (say edit out all those > things from existing labels). I will fix anything that gives you a > problem. > > ? Thanks, > > ? ? Matt It looks like it is exactly what I was looking for. I'll give it a go. Thanks! Matteo -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.becker at physik.uni-giessen.de Tue Jun 12 00:42:46 2018 From: michael.becker at physik.uni-giessen.de (Michael Becker) Date: Tue, 12 Jun 2018 07:42:46 +0200 Subject: [petsc-users] Poor weak scaling when solving successive linearsystems In-Reply-To: References: <6d9b6488-3957-0d74-148a-323c78d42f6a@physik.uni-giessen.de> <079D8B4D-C57B-4054-B9BD-05644E52C3A4@anl.gov> Message-ID: Hello, any new insights yet? Michael Am 04.06.2018 um 21:56 schrieb Junchao Zhang: > Miachael,? I can compile and run you test.? I am now profiling it. Thanks. > > --Junchao Zhang > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jczhang at mcs.anl.gov Tue Jun 12 15:07:46 2018 From: jczhang at mcs.anl.gov (Junchao Zhang) Date: Tue, 12 Jun 2018 15:07:46 -0500 Subject: [petsc-users] Poor weak scaling when solving successive linearsystems In-Reply-To: References: <6d9b6488-3957-0d74-148a-323c78d42f6a@physik.uni-giessen.de> <079D8B4D-C57B-4054-B9BD-05644E52C3A4@anl.gov> Message-ID: Hello, Michael, Sorry for the delay. I am actively doing experiments with your example code. I tested it on a cluster with 36 cores/node. To distribute MPI ranks evenly among nodes, I used 216 and 1728 ranks instead of 125, 1000. So far I have these findings: 1) It is not a strict weak scaling test since with 1728 ranks it needs more KPS iterations, and more calls to MatSOR etc functions. 2) If I use half cores per node but double the nodes (keep MPI ranks the same), the performance is 60~70% better. It implies memory bandwidth plays an important role in performance. 3) I find you define the outermost two layers of nodes of the grid as boundary. Boundary processors have less nonzeros than interior processors. It is a source of load imbalance. At coarser grids, it gets worse. But I need to confirm this caused the poor scaling and big vecscatter delays in the experiment. Thanks. --Junchao Zhang On Tue, Jun 12, 2018 at 12:42 AM, Michael Becker < michael.becker at physik.uni-giessen.de> wrote: > Hello, > > any new insights yet? > > Michael > > > > Am 04.06.2018 um 21:56 schrieb Junchao Zhang: > > Miachael, I can compile and run you test. I am now profiling it. Thanks. > > --Junchao Zhang > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cpraveen at gmail.com Wed Jun 13 00:32:01 2018 From: cpraveen at gmail.com (Praveen C) Date: Wed, 13 Jun 2018 11:02:01 +0530 Subject: [petsc-users] residual norm with TSPSEUDO Message-ID: Dear all How can I access the current residual norm inside my TSMonitor function when using TSPSEUDO ? Thanks praveen From bsmith at mcs.anl.gov Wed Jun 13 09:59:00 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 13 Jun 2018 14:59:00 +0000 Subject: [petsc-users] residual norm with TSPSEUDO In-Reply-To: References: Message-ID: Take a look at TSPseudoMonitorDefault() in src/ts/impls/pseudo/posindep.c The problem is that you need to expose typedef struct { Vec update; /* work vector where new solution is formed */ Vec func; /* work vector where F(t[i],u[i]) is stored */ Vec xdot; /* work vector for time derivative of state */ /* information used for Pseudo-timestepping */ PetscErrorCode (*dt)(TS,PetscReal*,void*); /* compute next timestep, and related context */ void *dtctx; PetscErrorCode (*verify)(TS,Vec,void*,PetscReal*,PetscBool*); /* verify previous timestep and related context */ void *verifyctx; PetscReal fnorm_initial,fnorm; /* original and current norm of F(u) */ PetscReal fnorm_previous; PetscReal dt_initial; /* initial time-step */ PetscReal dt_increment; /* scaling that dt is incremented each time-step */ PetscReal dt_max; /* maximum time step */ PetscBool increment_dt_from_initial_dt; PetscReal fatol,frtol; } TS_Pseudo; inside your application code (that is make a copy of this struct definition and put it in your code so you can access the fields). Barry > On Jun 13, 2018, at 12:32 AM, Praveen C wrote: > > Dear all > > How can I access the current residual norm inside my TSMonitor function when using TSPSEUDO ? > > Thanks > praveen From cpraveen at gmail.com Wed Jun 13 10:54:05 2018 From: cpraveen at gmail.com (Praveen C) Date: Wed, 13 Jun 2018 21:24:05 +0530 Subject: [petsc-users] residual norm with TSPSEUDO In-Reply-To: References: Message-ID: <9CBB17E1-05DA-41DE-B414-9EE6ABADBEF1@gmail.com> Hello Barry I am working in Fortran :-( I can see the residual if I use -info but it prints too much information. Is it possible to reduce the amount of log messages printed to screen, some sort of log-level ??? Thanks praveen > On 13-Jun-2018, at 8:29 PM, Smith, Barry F. wrote: > > Take a look at TSPseudoMonitorDefault() in src/ts/impls/pseudo/posindep.c > > The problem is that you need to expose > > typedef struct { > Vec update; /* work vector where new solution is formed */ > Vec func; /* work vector where F(t[i],u[i]) is stored */ > Vec xdot; /* work vector for time derivative of state */ > > /* information used for Pseudo-timestepping */ > > PetscErrorCode (*dt)(TS,PetscReal*,void*); /* compute next timestep, and related context */ > void *dtctx; > PetscErrorCode (*verify)(TS,Vec,void*,PetscReal*,PetscBool*); /* verify previous timestep and related context */ > void *verifyctx; > > PetscReal fnorm_initial,fnorm; /* original and current norm of F(u) */ > PetscReal fnorm_previous; > > PetscReal dt_initial; /* initial time-step */ > PetscReal dt_increment; /* scaling that dt is incremented each time-step */ > PetscReal dt_max; /* maximum time step */ > PetscBool increment_dt_from_initial_dt; > PetscReal fatol,frtol; > } TS_Pseudo; > > inside your application code (that is make a copy of this struct definition and put it in your code so you can access the fields). > > Barry -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Jun 13 11:18:43 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 13 Jun 2018 16:18:43 +0000 Subject: [petsc-users] residual norm with TSPSEUDO In-Reply-To: <9CBB17E1-05DA-41DE-B414-9EE6ABADBEF1@gmail.com> References: <9CBB17E1-05DA-41DE-B414-9EE6ABADBEF1@gmail.com> Message-ID: <2434A7BE-78CD-47CB-A9D7-8BADE05066C2@mcs.anl.gov> -ts_monitor_pseudo ? > On Jun 13, 2018, at 10:54 AM, Praveen C wrote: > > Hello Barry > > I am working in Fortran :-( > > I can see the residual if I use -info but it prints too much information. Is it possible to reduce the amount of log messages printed to screen, some sort of log-level ??? | grep -i "String you want" ? > > Thanks > praveen > >> On 13-Jun-2018, at 8:29 PM, Smith, Barry F. wrote: >> >> Take a look at TSPseudoMonitorDefault() in src/ts/impls/pseudo/posindep.c >> >> The problem is that you need to expose >> >> typedef struct { >> Vec update; /* work vector where new solution is formed */ >> Vec func; /* work vector where F(t[i],u[i]) is stored */ >> Vec xdot; /* work vector for time derivative of state */ >> >> /* information used for Pseudo-timestepping */ >> >> PetscErrorCode (*dt)(TS,PetscReal*,void*); /* compute next timestep, and related context */ >> void *dtctx; >> PetscErrorCode (*verify)(TS,Vec,void*,PetscReal*,PetscBool*); /* verify previous timestep and related context */ >> void *verifyctx; >> >> PetscReal fnorm_initial,fnorm; /* original and current norm of F(u) */ >> PetscReal fnorm_previous; >> >> PetscReal dt_initial; /* initial time-step */ >> PetscReal dt_increment; /* scaling that dt is incremented each time-step */ >> PetscReal dt_max; /* maximum time step */ >> PetscBool increment_dt_from_initial_dt; >> PetscReal fatol,frtol; >> } TS_Pseudo; >> >> inside your application code (that is make a copy of this struct definition and put it in your code so you can access the fields). >> >> Barry > From t.appel17 at imperial.ac.uk Wed Jun 13 12:31:59 2018 From: t.appel17 at imperial.ac.uk (Thibaut Appel) Date: Wed, 13 Jun 2018 18:31:59 +0100 Subject: [petsc-users] Debugger fails because of nanosleep.c Message-ID: <5f1bd372-245b-eacd-bfd0-da5210af4e0c@imperial.ac.uk> Good afternoon, I have a "Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range" error on my application code. After changing from 1 to 0 in /proc/sys/kernel/yama/ptrace_scope, I tried to start my code with -start_in_debugger and -fp_trap but in the x-terminal that pops up, I have 0x00007f06f19f69a4 in __GI___nanosleep (requested_time=0x7ffc064c15a0, ??? remaining=0x7ffc064c15a0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28 28????? ../sysdeps/unix/sysv/linux/nanosleep.c: No such file or directory. popping up. How can I solve this? Running Ubuntu 18.04 LTS with petsc 3.9.2, my code is compiled with the following flags -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g? -Og -fimplicit-none -std=f2018 -pedantic -fmodule-private -fcheck=all -fbacktrace -ffpe-trap=invalid,zero,overflow PETSc is configured with --with-cc=gcc-8 --with-cxx=g++-8 --with-fc=gfortran-8 --with-scalar-type=complex --with-precision=double --with-debugging=1 --with-fortran-kernels=1 --with-x=1 --download-mpich --download-fblaslapack --download-scalapack --download-metis --download-parmetis --download-ptscotch --download-mumps --with-debugger=gdb My xTerm version seems to be 330. Thank you for any useful piece of information, Thibaut From jed at jedbrown.org Wed Jun 13 12:41:11 2018 From: jed at jedbrown.org (Jed Brown) Date: Wed, 13 Jun 2018 11:41:11 -0600 Subject: [petsc-users] Debugger fails because of nanosleep.c In-Reply-To: <5f1bd372-245b-eacd-bfd0-da5210af4e0c@imperial.ac.uk> References: <5f1bd372-245b-eacd-bfd0-da5210af4e0c@imperial.ac.uk> Message-ID: <871sdamvns.fsf@jedbrown.org> Thibaut Appel writes: > Good afternoon, > > I have a "Caught signal number 11 SEGV: Segmentation Violation, probably > memory access out of range" error on my application code. > After changing from 1 to 0 in /proc/sys/kernel/yama/ptrace_scope, I > tried to start my code with -start_in_debugger and -fp_trap but in the > x-terminal that pops up, I have > > 0x00007f06f19f69a4 in __GI___nanosleep (requested_time=0x7ffc064c15a0, > ??? remaining=0x7ffc064c15a0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28 > 28????? ../sysdeps/unix/sysv/linux/nanosleep.c: No such file or directory. > > popping up. How can I solve this? Please include backtrace output (type "bt" at the gdb prompt) when writing. You may just need to "continue". > Running Ubuntu 18.04 LTS with petsc 3.9.2, my code is compiled with the > following flags > > -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g? -Og > -fimplicit-none -std=f2018 -pedantic -fmodule-private -fcheck=all > -fbacktrace -ffpe-trap=invalid,zero,overflow > > PETSc is configured with --with-cc=gcc-8 --with-cxx=g++-8 > --with-fc=gfortran-8 --with-scalar-type=complex --with-precision=double > --with-debugging=1 --with-fortran-kernels=1 --with-x=1 --download-mpich > --download-fblaslapack --download-scalapack --download-metis > --download-parmetis --download-ptscotch --download-mumps --with-debugger=gdb > > My xTerm version seems to be 330. > > Thank you for any useful piece of information, > > Thibaut From epscodes at gmail.com Wed Jun 13 15:29:41 2018 From: epscodes at gmail.com (Xiangdong) Date: Wed, 13 Jun 2018 16:29:41 -0400 Subject: [petsc-users] reuse coarsening setup in hypre Message-ID: Hello everyone, If I solve a few matrices with same sparsity pattern by using hypre_boomeramg pc, can I reuse the coarsening setup information? Thank you. Best, Xiangdong -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Jun 13 15:46:56 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 13 Jun 2018 20:46:56 +0000 Subject: [petsc-users] reuse coarsening setup in hypre In-Reply-To: References: Message-ID: <1789B763-C038-469A-99C6-990B3C160CF4@anl.gov> > On Jun 13, 2018, at 3:29 PM, Xiangdong wrote: > > Hello everyone, > > If I solve a few matrices with same sparsity pattern by using hypre_boomeramg pc, can I reuse the coarsening setup information? I don't think so. The coarsening process etc depends not just on the nonzero pattern but also depends on the numerical values so changing the numerical values can change the coarsening process. Barry > > Thank you. > > Best, > Xiangdong From qicangsh at umich.edu Thu Jun 14 13:17:38 2018 From: qicangsh at umich.edu (Qicang Shen) Date: Thu, 14 Jun 2018 14:17:38 -0400 Subject: [petsc-users] Whether Petsc will deleted zero entries during the solve Message-ID: Hi Guys, I'm now confronting a problem. I'm using PETSC to construct a SPARSE Matrix. And I'm sure that the matrix has been allocated correctly using *MatMPIAIJSetPreallocation *with the upper limit of the size. The code works well when I just solve the system once. However, after the system been solved. And I want to use the original non-zero structure, but change the elements inside the matrix. The petsc will show the error message as: [14]PETSC ERROR: Argument out of range [14]PETSC ERROR: New nonzero at (58,56) caused a malloc Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check My current the solution is destroy the system and reallocate with the same size. But I believe there should be more efficient way, i.e. just use the original structure. I would like search for help to confirm that whether PETSC will compress/delete the zero entries during the MatAssemblyBegin/MatAssemblyEnd/KSPSolve, or other possible places. And how to avoid deleting zero elements during the process? Thanks very much. Qicang SHEN PhD Candidate Nuclear Engineering and Radiological Sciences, University of Michigan, Ann Arbor Email: qicangsh at umich.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Jun 14 14:08:04 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 14 Jun 2018 19:08:04 +0000 Subject: [petsc-users] Whether Petsc will deleted zero entries during the solve In-Reply-To: References: Message-ID: <5BBAE171-0B37-4E3A-9CF8-B164A97BA4C5@anl.gov> I am guessing your matrix has an "envelope" of nonzero values but the first time you fill the matrix you do not fill up the entire envelope? Hence internally within the matrix we squeeze out those nonexistent locations so next time you fill the matrix the new location you need is not available. There is really only one way to deal with this. Initially build the entire envelope of values (putting zeros in certain locations is fine they won't get squuzed out) then for future calls the locations are already there and so you will have no problems with new nonzero. On the other hand if the envelope is much much larger than any particular set of nonzero locations then it is better to create a new matrix each time because all the solves with the first approach treats all the matrix entries in the envelope as nonzero (even if they happen to be zero). Barry > On Jun 14, 2018, at 1:17 PM, Qicang Shen wrote: > > Hi Guys, > > I'm now confronting a problem. > > I'm using PETSC to construct a SPARSE Matrix. And I'm sure that the matrix has been allocated correctly using MatMPIAIJSetPreallocation with the upper limit of the size. > > The code works well when I just solve the system once. > > However, after the system been solved. And I want to use the original non-zero structure, but change the elements inside the matrix. The petsc will show the error message as: > > [14]PETSC ERROR: Argument out of range > [14]PETSC ERROR: New nonzero at (58,56) caused a malloc > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check > > My current the solution is destroy the system and reallocate with the same size. But I believe there should be more efficient way, i.e. just use the original structure. > > I would like search for help to confirm that whether PETSC will compress/delete the zero entries during the MatAssemblyBegin/MatAssemblyEnd/KSPSolve, or other possible places. > > And how to avoid deleting zero elements during the process? > > Thanks very much. > > > > > Qicang SHEN > PhD Candidate > Nuclear Engineering and Radiological Sciences, University of Michigan, Ann Arbor > Email: qicangsh at umich.edu From fdkong.jd at gmail.com Thu Jun 14 14:33:38 2018 From: fdkong.jd at gmail.com (Fande Kong) Date: Thu, 14 Jun 2018 13:33:38 -0600 Subject: [petsc-users] Whether Petsc will deleted zero entries during the solve In-Reply-To: <5BBAE171-0B37-4E3A-9CF8-B164A97BA4C5@anl.gov> References: <5BBAE171-0B37-4E3A-9CF8-B164A97BA4C5@anl.gov> Message-ID: This may help in this situation. http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatResetPreallocation.html Fande, On Thu, Jun 14, 2018 at 1:08 PM, Smith, Barry F. wrote: > > I am guessing your matrix has an "envelope" of nonzero values but the > first time you fill the matrix you do not fill up the entire envelope? > Hence internally within the matrix we squeeze out those nonexistent > locations so next time you fill the matrix the new location you need is not > available. > > There is really only one way to deal with this. Initially build the > entire envelope of values (putting zeros in certain locations is fine they > won't get squuzed out) then for future calls the locations are already > there and so you will have no problems with new nonzero. > > On the other hand if the envelope is much much larger than any > particular set of nonzero locations then it is better to create a new > matrix each time because all the solves with the first approach treats all > the matrix entries in the envelope as nonzero (even if they happen to be > zero). > > Barry > > > > On Jun 14, 2018, at 1:17 PM, Qicang Shen wrote: > > > > Hi Guys, > > > > I'm now confronting a problem. > > > > I'm using PETSC to construct a SPARSE Matrix. And I'm sure that the > matrix has been allocated correctly using MatMPIAIJSetPreallocation with > the upper limit of the size. > > > > The code works well when I just solve the system once. > > > > However, after the system been solved. And I want to use the original > non-zero structure, but change the elements inside the matrix. The petsc > will show the error message as: > > > > [14]PETSC ERROR: Argument out of range > > [14]PETSC ERROR: New nonzero at (58,56) caused a malloc > > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to > turn off this check > > > > My current the solution is destroy the system and reallocate with the > same size. But I believe there should be more efficient way, i.e. just use > the original structure. > > > > I would like search for help to confirm that whether PETSC will > compress/delete the zero entries during the MatAssemblyBegin/MatAssemblyEnd/KSPSolve, > or other possible places. > > > > And how to avoid deleting zero elements during the process? > > > > Thanks very much. > > > > > > > > > > Qicang SHEN > > PhD Candidate > > Nuclear Engineering and Radiological Sciences, University of Michigan, > Ann Arbor > > Email: qicangsh at umich.edu > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.becker at physik.uni-giessen.de Fri Jun 15 02:26:57 2018 From: michael.becker at physik.uni-giessen.de (Michael Becker) Date: Fri, 15 Jun 2018 09:26:57 +0200 Subject: [petsc-users] Poor weak scaling when solving successive linearsystems In-Reply-To: References: <6d9b6488-3957-0d74-148a-323c78d42f6a@physik.uni-giessen.de> <079D8B4D-C57B-4054-B9BD-05644E52C3A4@anl.gov> Message-ID: <7e406660-96b7-9c15-ed26-ebc348804151@physik.uni-giessen.de> Hello, thanks again for your efforts. > Boundary processors have less nonzeros than interior processors. Doesn't that mean that boundary processors have less work to do than the others? Or does this affect the size of the coarse grids? I know that defining the outermost nodes as boundary is not the most efficient way in this particular case (using Dirichlet boundary conditions on a smaller grid would do the same), but I need the solver to be able to handle arbitrarily shaped boundaries inside the domain, e.g. to calculate the potential inside a spherical capacitor (constant potential on the boundaries, charge distribution inside). Is there a better way to do that? Michael Am 12.06.2018 um 22:07 schrieb Junchao Zhang: > Hello, Michael, > ? Sorry for the delay. I am actively doing experiments with your > example code. I tested it on a cluster with 36 cores/node. To > distribute MPI ranks evenly among nodes, I used 216? and 1728 ranks > instead of 125, 1000.? So far I have these findings: > ?1) It is not a strict weak scaling test since with 1728 ranks it > needs more KPS iterations, and more calls to MatSOR etc functions. > ?2) If I use half cores per node but double the nodes (keep MPI ranks > the same), the performance is 60~70% better. It implies memory > bandwidth plays an important role in performance. > ?3) I find you define the outermost two layers of nodes of the grid as > boundary. Boundary processors have less nonzeros than interior > processors. It is a source of load imbalance. At coarser grids, it > gets worse. But I need to confirm this caused the poor scaling and big > vecscatter delays in the experiment. > > ?Thanks. > > > --Junchao Zhang > > On Tue, Jun 12, 2018 at 12:42 AM, Michael Becker > > wrote: > > Hello, > > any new insights yet? > > Michael > > > > > Am 04.06.2018 um 21:56 schrieb Junchao Zhang: >> Miachael,? I can compile and run you test.? I am now profiling >> it. Thanks. >> >> --Junchao Zhang >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jczhang at mcs.anl.gov Fri Jun 15 12:38:57 2018 From: jczhang at mcs.anl.gov (Junchao Zhang) Date: Fri, 15 Jun 2018 12:38:57 -0500 Subject: [petsc-users] Poor weak scaling when solving successive linearsystems In-Reply-To: <7e406660-96b7-9c15-ed26-ebc348804151@physik.uni-giessen.de> References: <6d9b6488-3957-0d74-148a-323c78d42f6a@physik.uni-giessen.de> <079D8B4D-C57B-4054-B9BD-05644E52C3A4@anl.gov> <7e406660-96b7-9c15-ed26-ebc348804151@physik.uni-giessen.de> Message-ID: I tried periodic boundary conditions but found load-imbalance still existed. So the boundary may not be a big issue. I am debugging the code to see why. Thanks. --Junchao Zhang On Fri, Jun 15, 2018 at 2:26 AM, Michael Becker < michael.becker at physik.uni-giessen.de> wrote: > Hello, > > thanks again for your efforts. > > Boundary processors have less nonzeros than interior processors. > > Doesn't that mean that boundary processors have less work to do than the > others? Or does this affect the size of the coarse grids? > > I know that defining the outermost nodes as boundary is not the most > efficient way in this particular case (using Dirichlet boundary conditions > on a smaller grid would do the same), but I need the solver to be able to > handle arbitrarily shaped boundaries inside the domain, e.g. to calculate > the potential inside a spherical capacitor (constant potential on the > boundaries, charge distribution inside). Is there a better way to do that? > > Michael > > > > > Am 12.06.2018 um 22:07 schrieb Junchao Zhang: > > Hello, Michael, > Sorry for the delay. I am actively doing experiments with your example > code. I tested it on a cluster with 36 cores/node. To distribute MPI ranks > evenly among nodes, I used 216 and 1728 ranks instead of 125, 1000. So > far I have these findings: > 1) It is not a strict weak scaling test since with 1728 ranks it needs > more KPS iterations, and more calls to MatSOR etc functions. > 2) If I use half cores per node but double the nodes (keep MPI ranks the > same), the performance is 60~70% better. It implies memory bandwidth plays > an important role in performance. > 3) I find you define the outermost two layers of nodes of the grid as > boundary. Boundary processors have less nonzeros than interior processors. > It is a source of load imbalance. At coarser grids, it gets worse. But I > need to confirm this caused the poor scaling and big vecscatter delays in > the experiment. > > Thanks. > > > --Junchao Zhang > > On Tue, Jun 12, 2018 at 12:42 AM, Michael Becker < > michael.becker at physik.uni-giessen.de> wrote: > >> Hello, >> >> any new insights yet? >> >> Michael >> >> >> >> Am 04.06.2018 um 21:56 schrieb Junchao Zhang: >> >> Miachael, I can compile and run you test. I am now profiling it. Thanks. >> >> --Junchao Zhang >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stormweiner at berkeley.edu Fri Jun 15 17:30:47 2018 From: stormweiner at berkeley.edu (Storm Weiner) Date: Fri, 15 Jun 2018 15:30:47 -0700 Subject: [petsc-users] detailed documentation for benchmarkExample.py Message-ID: Hey there, I'm trying to use the benchmark example script but I don't understand the usage. For reference, I am talking about the script located at *petsc/src/benchmarks/benchmarkExample.py* I execute using the suggested example parameters at the bottom of the script *>> ./src/benchmarks/benchmarkExample.py --events IntegBatchCPU IntegBatchGPU IntegGPUOnly --num 52 DMComplex --refine 0.0625 0.00625 0.000625 0.0000625 --blockExp 4 --order=1 CPU='dm_view show_residual=0 compute_function batch' GPU='dm_view show_residual=0 compute_function batch gpu gpu_batches=8'* *Namespace(batch=False, comp=4, daemon=False, events=['VecMDot', 'VecMAXPY', 'KSPGMRESOrthog', 'MatMult', 'VecCUSPCopyTo', 'VecCUSPCopyFrom', 'MatCUSPCopyTo'], gpulang='OpenCL', library='SNES', module='summary', num=50, runs=["CPU='pc_type=none", 'mat_no_inode', 'dm_vec_type=seq', "dm_mat_type=seqaij'", "GPU='pc_type=none", 'mat_no_inode', 'dm_vec_type=seqcusp', 'dm_mat_type=seqaijcusp', "cusp_synchronize'"], size=['10', '20', '50', '100'], stage='Main_Stage')Executing: /g/g19/weiner6/petsc/arch-mvapich-fftw3/lib/snes/ex50 -preload off -log_summary summary.dat -log_summary_python summary.py -'pc_type none -da_grid_x 10 -da_grid_y 10/bin/sh: -c: line 0: unexpected EOF while looking for matching `''/bin/sh: -c: line 1: syntax error: unexpected end of fileTraceback (most recent call last): File "./benchmarkExample.py", line 453, in run_DMDA(ex, name, opts, args, sizes, times, events, log=log) File "./benchmarkExample.py", line 356, in run_DMDA processSummary('summary', args.stage, args.events, times[name], events[name]) File "./benchmarkExample.py", line 123, in processSummary m = __import__(moduleName)ImportError: No module named summary* There are two issues here. 1) the script attempts to run *petsc/arch-mvapich-fftw3/lib/snes/ex50* when it should attempt to run *petsc/src//examples/tutorials/ex* as indicated in the help menu. In fact, there is no *petsc//lib/snes* directory. 2) The *"No module named summary"* error causes the python script to crash. What is going on here? How should I be using this script? Is there any documentation to describe the usage? I find the help menu to be a bit too sparse and the source code too dense to follow closely. More specifically, I want to use this profiling script for more than just the standard petsc examples. If I understand the source correctly, the simplest way to accomplish that is to create a new directory tree *petsc/src/dummy_lib/examples/tutorials *and populate it with *ex *compiled executables. I would then access these by running benchmarkExample.py with the *--library dummy_lib* argument. Is this a reasonable way to benchmark arbitrary petsc programs? Thanks for your time, Storm -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jun 15 18:11:16 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 15 Jun 2018 19:11:16 -0400 Subject: [petsc-users] detailed documentation for benchmarkExample.py In-Reply-To: References: Message-ID: On Fri, Jun 15, 2018 at 6:30 PM Storm Weiner wrote: > Hey there, > > I'm trying to use the benchmark example script but I don't understand the > usage. > I have not updated the script. We now use -log_view, and so the option would be -log_view :summary.py:ascii_info_detail Eventually, I will update it. We are working on it now. Thanks, Matt > For reference, I am talking about the script located at > *petsc/src/benchmarks/benchmarkExample.py* > > I execute using the suggested example parameters at the bottom of the > script > > > *>> ./src/benchmarks/benchmarkExample.py --events IntegBatchCPU > IntegBatchGPU IntegGPUOnly --num 52 DMComplex --refine 0.0625 0.00625 > 0.000625 0.0000625 --blockExp 4 --order=1 CPU='dm_view show_residual=0 > compute_function batch' GPU='dm_view show_residual=0 compute_function batch > gpu gpu_batches=8'* > > > > > > > > > > > > > > > > > > > *Namespace(batch=False, comp=4, daemon=False, events=['VecMDot', > 'VecMAXPY', 'KSPGMRESOrthog', 'MatMult', 'VecCUSPCopyTo', > 'VecCUSPCopyFrom', 'MatCUSPCopyTo'], gpulang='OpenCL', library='SNES', > module='summary', num=50, runs=["CPU='pc_type=none", 'mat_no_inode', > 'dm_vec_type=seq', "dm_mat_type=seqaij'", "GPU='pc_type=none", > 'mat_no_inode', 'dm_vec_type=seqcusp', 'dm_mat_type=seqaijcusp', > "cusp_synchronize'"], size=['10', '20', '50', '100'], > stage='Main_Stage')Executing: > /g/g19/weiner6/petsc/arch-mvapich-fftw3/lib/snes/ex50 -preload off > -log_summary summary.dat -log_summary_python summary.py -'pc_type none > -da_grid_x 10 -da_grid_y 10/bin/sh: -c: line 0: unexpected EOF while > looking for matching `''/bin/sh: -c: line 1: syntax error: unexpected end > of fileTraceback (most recent call last): File "./benchmarkExample.py", > line 453, in run_DMDA(ex, name, opts, args, sizes, times, > events, log=log) File "./benchmarkExample.py", line 356, in run_DMDA > processSummary('summary', args.stage, args.events, times[name], > events[name]) File "./benchmarkExample.py", line 123, in processSummary > m = __import__(moduleName)ImportError: No module named summary* > > > > There are two issues here. > > 1) the script attempts to run *petsc/arch-mvapich-fftw3/lib/snes/ex50* > when it should attempt to run > *petsc/src//examples/tutorials/ex* as indicated in the help > menu. In fact, there is no *petsc//lib/snes* directory. > > 2) The *"No module named summary"* error causes the python script to > crash. > > What is going on here? How should I be using this script? Is there any > documentation to describe the usage? I find the help menu to be a bit too > sparse and the source code too dense to follow closely. > > > More specifically, I want to use this profiling script for more than just > the standard petsc examples. If I understand the source correctly, the > simplest way to accomplish that is to create a new directory tree *petsc/src/dummy_lib/examples/tutorials > *and populate it with *ex *compiled executables. I would then > access these by running benchmarkExample.py with the *--library dummy_lib* > argument. Is this a reasonable way to benchmark arbitrary petsc programs? > > > Thanks for your time, > Storm > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From nahmad16 at ku.edu.tr Mon Jun 18 09:10:11 2018 From: nahmad16 at ku.edu.tr (Najeeb Ahmad) Date: Mon, 18 Jun 2018 19:10:11 +0500 Subject: [petsc-users] Accessing non-local values of a vector Message-ID: Hi all, I am a novice user and I am trying to write my first project in PETSc. In my code, I need to get vector values that may be residing on other processes. I noticed that VecGetValues can get values on the same processor. For instance, I need a calculation like this: temp = (dy(i) + dy(i + xmax)); /// xmax is length along x - direction where dy(i + xmax) may be on another processes. What is the best way to accomplish this? Thanks -- *Najeeb Ahmad* *Research and Teaching Assistant* *PARallel and MultiCORE Computing Laboratory (ParCoreLab) * *Computer Science and Engineering* *Ko? University, Istanbul, Turkey* -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Jun 18 09:50:34 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 18 Jun 2018 20:20:34 +0530 Subject: [petsc-users] Accessing non-local values of a vector In-Reply-To: References: Message-ID: You would have to use VecScatter. http://www.mcs.anl.gov/petsc/documentation/faq.html#mpi-vec-access Satish On Mon, 18 Jun 2018, Najeeb Ahmad wrote: > Hi all, > > I am a novice user and I am trying to write my first project in PETSc. In > my code, I need to get vector values that may be residing on other > processes. I noticed that VecGetValues can get values on the same > processor. For instance, I need a calculation like this: > > temp = (dy(i) + dy(i + xmax)); /// xmax is length along x - > direction > > where dy(i + xmax) may be on another processes. What is the best way to > accomplish this? > > Thanks > > From nahmad16 at ku.edu.tr Mon Jun 18 10:05:54 2018 From: nahmad16 at ku.edu.tr (Najeeb Ahmad) Date: Mon, 18 Jun 2018 20:05:54 +0500 Subject: [petsc-users] Accessing non-local values of a vector In-Reply-To: References: Message-ID: Thank you Satish. This is very helpful. Regards, On Mon, Jun 18, 2018 at 7:51 PM Satish Balay wrote: > You would have to use VecScatter. > > http://www.mcs.anl.gov/petsc/documentation/faq.html#mpi-vec-access > > Satish > > On Mon, 18 Jun 2018, Najeeb Ahmad wrote: > > > Hi all, > > > > I am a novice user and I am trying to write my first project in PETSc. In > > my code, I need to get vector values that may be residing on other > > processes. I noticed that VecGetValues can get values on the same > > processor. For instance, I need a calculation like this: > > > > temp = (dy(i) + dy(i + xmax)); /// xmax is length along x - > > direction > > > > where dy(i + xmax) may be on another processes. What is the best way to > > accomplish this? > > > > Thanks > > > > > > -- *Najeeb Ahmad* *Research and Teaching Assistant* *PARallel and MultiCORE Computing Laboratory (ParCoreLab) * *Computer Science and Engineering* *Ko? University, Istanbul, Turkey* -------------- next part -------------- An HTML attachment was scrubbed... URL: From gaetank at gmail.com Tue Jun 19 12:36:28 2018 From: gaetank at gmail.com (Gaetan Kenway) Date: Tue, 19 Jun 2018 10:36:28 -0700 Subject: [petsc-users] MatMFFDSetBase Message-ID: Seems like the MatMFFDSetBase is now broken in petsc 3.9 In 3.7 I used: call MatMFFDSetBase(dRdW, wVec, PETSC_NULL_OBJECT, ierr) and the most logical translation is now call MatMFFDSetBase(dRdW, wVec, PETSC_NULL_VEC, ierr) Unfortunately this just segfaults: #0 VecAXPY (y=y at entry=0x10a3750, alpha=alpha at entry=-1, x=x at entry=0xffffffffffffffff) at /nobackup/gkenway/packages/petsc-3.9.2/src/vec/vec/interface/rvector.c:601 #1 0x00007fffb697faae in MatMult_MFFD (mat=0x291a150, a=0x10b25c0, y=0x10a3750) at /nobackup/gkenway/packages/petsc-3.9.2/src/mat/impls/mffd/mffd.c:368 #2 0x00007fffb6847af2 in MatMult (mat=0x291a150, x=x at entry=0x10b25c0, y=y at entry=0x10a3750) at /nobackup/gkenway/packages/petsc-3.9.2/src/mat/interface/matrix.c:2305 #3 0x00007fffb6e982e7 in PCApplyBAorAB (pc=0xe64d00, side=PC_RIGHT, x=0x10aaee0, y=y at entry=0x10a3750, work=0x10b25c0) at /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/pc/interface/precon.c:680 #4 0x00007fffb6f02b84 in KSP_PCApplyBAorAB (w=, y=0x10a3750, x=, ksp=0xf135e0) at /u/wk/gkenway/scratch/packages/petsc-3.9.2/include/petsc/private/kspimpl.h:304 #5 KSPGMRESCycle (itcount=itcount at entry=0x7fffffffb73c, ksp=ksp at entry=0xf135e0) at /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/ksp/impls/gmres/gmres.c:152 #6 0x00007fffb6f035c2 in KSPSolve_GMRES (ksp=0xf135e0) at /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/ksp/impls/gmres/gmres.c:234 #7 0x00007fffb6f35050 in KSPSolve (ksp=0xf135e0, b=0xf7ced0, x=0xf77be0) at /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/ksp/interface/itfunc.c:669 #8 0x00007fffb6f482ee in kspsolve_ (ksp=ksp at entry=0x7fffb461bae8 <__anksolver_MOD_ank_ksp>, b=, b at entry=0x7fffb461b9c8 <__anksolver_MOD_rvec>, x=, x at entry=0x7fffb461ba18 <__anksolver_MOD_deltaw>, ierr=ierr at entry=0x7fffffffb938) at /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/ksp/interface/ftn-custom/zitfuncf.c:54 #9 0x00007fffb3d7e697 in anksolver::ankstep (firstcall=) at ../NKSolver/NKSolvers.F90:2891 #10 0x00007fffb3ebbbb9 in solvers::solvestate () at ../solver/solvers.F90:1114 #11 0x00007fffb3ebe285 in solvers::solver () at ../solver/solvers.F90:69 #12 0x00007fffb3cfca17 in f2py_rout_libadflow_solvers_solver (capi_self=, capi_args=, capi_keywds=, f2py_func=0x7fffb3ebe0e0 ) at libadflowmodule.c:6710 #13 0x00007fffb3cf85b3 in fortran_call (fp=, arg=, kw=) at /u/wk/gkenway/.local/lib/python2.7/site-packages/numpy/f2py/src/fortranobject.c:404 #14 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 #15 0x00007ffff7aef236 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 #16 0x00007ffff7af603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 #17 0x00007ffff7a7f978 in function_call () from /lib64/libpython2.7.so.1.0 #18 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 #19 0x00007ffff7a69a55 in instancemethod_call () from /lib64/libpython2.7.so.1.0 #20 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 #21 0x00007ffff7ab1e27 in slot_tp_call () from /lib64/libpython2.7.so.1.0 #22 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 #23 0x00007ffff7aef236 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 #24 0x00007ffff7af603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 #25 0x00007ffff7af6142 in PyEval_EvalCode () from /lib64/libpython2.7.so.1.0 #26 0x00007ffff7b0f57f in run_mod () from /lib64/libpython2.7.so.1.0 #27 0x00007ffff7b1073e in PyRun_FileExFlags () from /lib64/libpython2.7.so.1.0 #28 0x00007ffff7b119c9 in PyRun_SimpleFileExFlags () from /lib64/libpython2.7.so.1.0 #29 0x00007ffff7b22b7f in Py_Main () from /lib64/libpython2.7.so.1.0 #30 0x00007ffff6d3f3d5 in __libc_start_main () from /lib64/libc.so.6 #31 0x000000000040066e in _start () Thanks, Gaetan -------------- next part -------------- An HTML attachment was scrubbed... URL: From gaetank at gmail.com Tue Jun 19 13:03:12 2018 From: gaetank at gmail.com (Gaetan Kenway) Date: Tue, 19 Jun 2018 11:03:12 -0700 Subject: [petsc-users] MatMFFDSetBase In-Reply-To: References: Message-ID: The issue is that in matffd.c passing PETSC_NULL_VEC for F is treated as true in : if (F) { if (ctx->current_f_allocated) {ierr = VecDestroy(&ctx->current_f);CHKERRQ(ierr);} ctx->current_f = F; ctx->current_f_allocated = PETSC_FALSE; printf("F is given\n"); } else if (!ctx->current_f_allocated) { ierr = MatCreateVecs(J,NULL,&ctx->current_f);CHKERRQ(ierr); printf("F create\n"); ctx->current_f_allocated = PETSC_TRUE; } so the ctx->current_f vector is never created as it should be. How do you *actually* pass a "null" since petsc_null_object is gone? Thanks, Gaetan On Tue, Jun 19, 2018 at 10:36 AM Gaetan Kenway wrote: > Seems like the MatMFFDSetBase is now broken in petsc 3.9 > > In 3.7 I used: > > call MatMFFDSetBase(dRdW, wVec, PETSC_NULL_OBJECT, ierr) > > and the most logical translation is now > > call MatMFFDSetBase(dRdW, wVec, PETSC_NULL_VEC, ierr) > > Unfortunately this just segfaults: > #0 VecAXPY (y=y at entry=0x10a3750, alpha=alpha at entry=-1, x=x at entry=0xffffffffffffffff) > at > /nobackup/gkenway/packages/petsc-3.9.2/src/vec/vec/interface/rvector.c:601 > #1 0x00007fffb697faae in MatMult_MFFD (mat=0x291a150, a=0x10b25c0, > y=0x10a3750) at > /nobackup/gkenway/packages/petsc-3.9.2/src/mat/impls/mffd/mffd.c:368 > #2 0x00007fffb6847af2 in MatMult (mat=0x291a150, x=x at entry=0x10b25c0, > y=y at entry=0x10a3750) at > /nobackup/gkenway/packages/petsc-3.9.2/src/mat/interface/matrix.c:2305 > #3 0x00007fffb6e982e7 in PCApplyBAorAB (pc=0xe64d00, side=PC_RIGHT, > x=0x10aaee0, y=y at entry=0x10a3750, work=0x10b25c0) at > /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/pc/interface/precon.c:680 > #4 0x00007fffb6f02b84 in KSP_PCApplyBAorAB (w=, > y=0x10a3750, x=, ksp=0xf135e0) at > /u/wk/gkenway/scratch/packages/petsc-3.9.2/include/petsc/private/kspimpl.h:304 > #5 KSPGMRESCycle (itcount=itcount at entry=0x7fffffffb73c, ksp=ksp at entry=0xf135e0) > at > /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/ksp/impls/gmres/gmres.c:152 > #6 0x00007fffb6f035c2 in KSPSolve_GMRES (ksp=0xf135e0) at > /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/ksp/impls/gmres/gmres.c:234 > #7 0x00007fffb6f35050 in KSPSolve (ksp=0xf135e0, b=0xf7ced0, x=0xf77be0) > at /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/ksp/interface/itfunc.c:669 > #8 0x00007fffb6f482ee in kspsolve_ (ksp=ksp at entry=0x7fffb461bae8 > <__anksolver_MOD_ank_ksp>, b=, b at entry=0x7fffb461b9c8 > <__anksolver_MOD_rvec>, x=, > x at entry=0x7fffb461ba18 <__anksolver_MOD_deltaw>, ierr=ierr at entry=0x7fffffffb938) > at > /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/ksp/interface/ftn-custom/zitfuncf.c:54 > #9 0x00007fffb3d7e697 in anksolver::ankstep (firstcall=) > at ../NKSolver/NKSolvers.F90:2891 > #10 0x00007fffb3ebbbb9 in solvers::solvestate () at > ../solver/solvers.F90:1114 > #11 0x00007fffb3ebe285 in solvers::solver () at ../solver/solvers.F90:69 > #12 0x00007fffb3cfca17 in f2py_rout_libadflow_solvers_solver > (capi_self=, capi_args=, > capi_keywds=, f2py_func=0x7fffb3ebe0e0 ) > at libadflowmodule.c:6710 > #13 0x00007fffb3cf85b3 in fortran_call (fp=, arg= out>, kw=) at > /u/wk/gkenway/.local/lib/python2.7/site-packages/numpy/f2py/src/fortranobject.c:404 > #14 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 > #15 0x00007ffff7aef236 in PyEval_EvalFrameEx () from > /lib64/libpython2.7.so.1.0 > #16 0x00007ffff7af603d in PyEval_EvalCodeEx () from > /lib64/libpython2.7.so.1.0 > #17 0x00007ffff7a7f978 in function_call () from /lib64/libpython2.7.so.1.0 > #18 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 > #19 0x00007ffff7a69a55 in instancemethod_call () from > /lib64/libpython2.7.so.1.0 > #20 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 > #21 0x00007ffff7ab1e27 in slot_tp_call () from /lib64/libpython2.7.so.1.0 > #22 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 > #23 0x00007ffff7aef236 in PyEval_EvalFrameEx () from > /lib64/libpython2.7.so.1.0 > #24 0x00007ffff7af603d in PyEval_EvalCodeEx () from > /lib64/libpython2.7.so.1.0 > #25 0x00007ffff7af6142 in PyEval_EvalCode () from > /lib64/libpython2.7.so.1.0 > #26 0x00007ffff7b0f57f in run_mod () from /lib64/libpython2.7.so.1.0 > #27 0x00007ffff7b1073e in PyRun_FileExFlags () from > /lib64/libpython2.7.so.1.0 > #28 0x00007ffff7b119c9 in PyRun_SimpleFileExFlags () from > /lib64/libpython2.7.so.1.0 > #29 0x00007ffff7b22b7f in Py_Main () from /lib64/libpython2.7.so.1.0 > #30 0x00007ffff6d3f3d5 in __libc_start_main () from /lib64/libc.so.6 > #31 0x000000000040066e in _start () > > Thanks, > > Gaetan > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jun 19 13:24:25 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 19 Jun 2018 18:24:25 +0000 Subject: [petsc-users] MatMFFDSetBase In-Reply-To: References: Message-ID: The code to handle the null vector got mistakenly removed. Please find attached a patch that should resolve the problem. Please let us know if it works, Thanks Barry > On Jun 19, 2018, at 12:36 PM, Gaetan Kenway wrote: > > Seems like the MatMFFDSetBase is now broken in petsc 3.9 > > In 3.7 I used: > > call MatMFFDSetBase(dRdW, wVec, PETSC_NULL_OBJECT, ierr) > > and the most logical translation is now > > call MatMFFDSetBase(dRdW, wVec, PETSC_NULL_VEC, ierr) > > Unfortunately this just segfaults: > #0 VecAXPY (y=y at entry=0x10a3750, alpha=alpha at entry=-1, x=x at entry=0xffffffffffffffff) at /nobackup/gkenway/packages/petsc-3.9.2/src/vec/vec/interface/rvector.c:601 > #1 0x00007fffb697faae in MatMult_MFFD (mat=0x291a150, a=0x10b25c0, y=0x10a3750) at /nobackup/gkenway/packages/petsc-3.9.2/src/mat/impls/mffd/mffd.c:368 > #2 0x00007fffb6847af2 in MatMult (mat=0x291a150, x=x at entry=0x10b25c0, y=y at entry=0x10a3750) at /nobackup/gkenway/packages/petsc-3.9.2/src/mat/interface/matrix.c:2305 > #3 0x00007fffb6e982e7 in PCApplyBAorAB (pc=0xe64d00, side=PC_RIGHT, x=0x10aaee0, y=y at entry=0x10a3750, work=0x10b25c0) at /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/pc/interface/precon.c:680 > #4 0x00007fffb6f02b84 in KSP_PCApplyBAorAB (w=, y=0x10a3750, x=, ksp=0xf135e0) at /u/wk/gkenway/scratch/packages/petsc-3.9.2/include/petsc/private/kspimpl.h:304 > #5 KSPGMRESCycle (itcount=itcount at entry=0x7fffffffb73c, ksp=ksp at entry=0xf135e0) at /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/ksp/impls/gmres/gmres.c:152 > #6 0x00007fffb6f035c2 in KSPSolve_GMRES (ksp=0xf135e0) at /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/ksp/impls/gmres/gmres.c:234 > #7 0x00007fffb6f35050 in KSPSolve (ksp=0xf135e0, b=0xf7ced0, x=0xf77be0) at /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/ksp/interface/itfunc.c:669 > #8 0x00007fffb6f482ee in kspsolve_ (ksp=ksp at entry=0x7fffb461bae8 <__anksolver_MOD_ank_ksp>, b=, b at entry=0x7fffb461b9c8 <__anksolver_MOD_rvec>, x=, > x at entry=0x7fffb461ba18 <__anksolver_MOD_deltaw>, ierr=ierr at entry=0x7fffffffb938) at /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/ksp/interface/ftn-custom/zitfuncf.c:54 > #9 0x00007fffb3d7e697 in anksolver::ankstep (firstcall=) at ../NKSolver/NKSolvers.F90:2891 > #10 0x00007fffb3ebbbb9 in solvers::solvestate () at ../solver/solvers.F90:1114 > #11 0x00007fffb3ebe285 in solvers::solver () at ../solver/solvers.F90:69 > #12 0x00007fffb3cfca17 in f2py_rout_libadflow_solvers_solver (capi_self=, capi_args=, capi_keywds=, f2py_func=0x7fffb3ebe0e0 ) > at libadflowmodule.c:6710 > #13 0x00007fffb3cf85b3 in fortran_call (fp=, arg=, kw=) at /u/wk/gkenway/.local/lib/python2.7/site-packages/numpy/f2py/src/fortranobject.c:404 > #14 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 > #15 0x00007ffff7aef236 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 > #16 0x00007ffff7af603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 > #17 0x00007ffff7a7f978 in function_call () from /lib64/libpython2.7.so.1.0 > #18 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 > #19 0x00007ffff7a69a55 in instancemethod_call () from /lib64/libpython2.7.so.1.0 > #20 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 > #21 0x00007ffff7ab1e27 in slot_tp_call () from /lib64/libpython2.7.so.1.0 > #22 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 > #23 0x00007ffff7aef236 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 > #24 0x00007ffff7af603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 > #25 0x00007ffff7af6142 in PyEval_EvalCode () from /lib64/libpython2.7.so.1.0 > #26 0x00007ffff7b0f57f in run_mod () from /lib64/libpython2.7.so.1.0 > #27 0x00007ffff7b1073e in PyRun_FileExFlags () from /lib64/libpython2.7.so.1.0 > #28 0x00007ffff7b119c9 in PyRun_SimpleFileExFlags () from /lib64/libpython2.7.so.1.0 > #29 0x00007ffff7b22b7f in Py_Main () from /lib64/libpython2.7.so.1.0 > #30 0x00007ffff6d3f3d5 in __libc_start_main () from /lib64/libc.so.6 > #31 0x000000000040066e in _start () > > Thanks, > > Gaetan > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: matmffdsetbase-fortran-null.patch Type: application/octet-stream Size: 1552 bytes Desc: matmffdsetbase-fortran-null.patch URL: From gaetank at gmail.com Tue Jun 19 15:28:28 2018 From: gaetank at gmail.com (Gaetan Kenway) Date: Tue, 19 Jun 2018 13:28:28 -0700 Subject: [petsc-users] MatMFFDSetBase In-Reply-To: References: Message-ID: I tried the patch. It initially didn't compile because of a conflict with the corresponding ftn-auto file. When i commented out the stuff there it compiled, but I got a segfault here: at /nobackup/gkenway/packages/petsc-3.9.2/src/mat/impls/mffd/ftn-custom/zmffdf.c:52 52 *ierr = MatMFFDSetBase(*mat,*u,*f); (gdb) bt #0 matmffdsetbase_ (mat=mat at entry=0x7fffb461ba10 <__anksolver_MOD_drdw>, u=u at entry=0x7fffb461b9a0 <__anksolver_MOD_wvec>, f=0x0, f at entry=0x7fffffffb970, ierr=ierr at entry=0x7fffffffb938) at /nobackup/gkenway/packages/petsc-3.9.2/src/mat/impls/mffd/ftn-custom/zmffdf.c:52 #1 0x00007fffb3d7e871 in anksolver::ankstep (firstcall=) at ../NKSolver/NKSolvers.F90:2891 #2 0x00007fffb3ebbde9 in solvers::solvestate () at ../solver/solvers.F90:1114 #3 0x00007fffb3ebe4b5 in solvers::solver () at ../solver/solvers.F90:69 #4 0x00007fffb3cfcaa7 in f2py_rout_libadflow_solvers_solver (capi_self=, capi_args=, capi_keywds=, f2py_func=0x7fffb3ebe310 ) at libadflowmodule.c:6710 #5 0x00007fffb3cf8643 in fortran_call (fp=, arg=, kw=) at /u/wk/gkenway/.local/lib/python2.7/site-packages/numpy/f2py/src/fortranobject.c:404 #6 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 #7 0x00007ffff7aef236 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 #8 0x00007ffff7af603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 #9 0x00007ffff7a7f978 in function_call () from /lib64/libpython2.7.so.1.0 #10 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 #11 0x00007ffff7a69a55 in instancemethod_call () from /lib64/libpython2.7.so.1.0 #12 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 #13 0x00007ffff7ab1e27 in slot_tp_call () from /lib64/libpython2.7.so.1.0 #14 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 #15 0x00007ffff7aef236 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 #16 0x00007ffff7af603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 #17 0x00007ffff7af6142 in PyEval_EvalCode () from /lib64/libpython2.7.so.1.0 #18 0x00007ffff7b0f57f in run_mod () from /lib64/libpython2.7.so.1.0 #19 0x00007ffff7b1073e in PyRun_FileExFlags () from /lib64/libpython2.7.so.1.0 #20 0x00007ffff7b119c9 in PyRun_SimpleFileExFlags () from /lib64/libpython2.7.so.1.0 #21 0x00007ffff7b22b7f in Py_Main () from /lib64/libpython2.7.so.1.0 #22 0x00007ffff6d3f3d5 in __libc_start_main () from /lib64/libc.so.6 #23 0x000000000040066e in _start () which was the new code that got added. So I never got the patch to work correctly. However, I circumvented the problem by making my own vector and calling the matfd-function, and then passing in the vector. This is essentially the same thing and I means I don't need an if-def for older versions. Gaetan On Tue, Jun 19, 2018 at 11:24 AM Smith, Barry F. wrote: > > The code to handle the null vector got mistakenly removed. Please find > attached a patch that should resolve the problem. > > > > Please let us know if it works, > > Thanks > > Barry > > > > On Jun 19, 2018, at 12:36 PM, Gaetan Kenway wrote: > > > > Seems like the MatMFFDSetBase is now broken in petsc 3.9 > > > > In 3.7 I used: > > > > call MatMFFDSetBase(dRdW, wVec, PETSC_NULL_OBJECT, ierr) > > > > and the most logical translation is now > > > > call MatMFFDSetBase(dRdW, wVec, PETSC_NULL_VEC, ierr) > > > > Unfortunately this just segfaults: > > #0 VecAXPY (y=y at entry=0x10a3750, alpha=alpha at entry=-1, x=x at entry=0xffffffffffffffff) > at > /nobackup/gkenway/packages/petsc-3.9.2/src/vec/vec/interface/rvector.c:601 > > #1 0x00007fffb697faae in MatMult_MFFD (mat=0x291a150, a=0x10b25c0, > y=0x10a3750) at > /nobackup/gkenway/packages/petsc-3.9.2/src/mat/impls/mffd/mffd.c:368 > > #2 0x00007fffb6847af2 in MatMult (mat=0x291a150, x=x at entry=0x10b25c0, > y=y at entry=0x10a3750) at > /nobackup/gkenway/packages/petsc-3.9.2/src/mat/interface/matrix.c:2305 > > #3 0x00007fffb6e982e7 in PCApplyBAorAB (pc=0xe64d00, side=PC_RIGHT, > x=0x10aaee0, y=y at entry=0x10a3750, work=0x10b25c0) at > /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/pc/interface/precon.c:680 > > #4 0x00007fffb6f02b84 in KSP_PCApplyBAorAB (w=, > y=0x10a3750, x=, ksp=0xf135e0) at > /u/wk/gkenway/scratch/packages/petsc-3.9.2/include/petsc/private/kspimpl.h:304 > > #5 KSPGMRESCycle (itcount=itcount at entry=0x7fffffffb73c, ksp=ksp at entry=0xf135e0) > at > /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/ksp/impls/gmres/gmres.c:152 > > #6 0x00007fffb6f035c2 in KSPSolve_GMRES (ksp=0xf135e0) at > /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/ksp/impls/gmres/gmres.c:234 > > #7 0x00007fffb6f35050 in KSPSolve (ksp=0xf135e0, b=0xf7ced0, > x=0xf77be0) at > /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/ksp/interface/itfunc.c:669 > > #8 0x00007fffb6f482ee in kspsolve_ (ksp=ksp at entry=0x7fffb461bae8 > <__anksolver_MOD_ank_ksp>, b=, b at entry=0x7fffb461b9c8 > <__anksolver_MOD_rvec>, x=, > > x at entry=0x7fffb461ba18 <__anksolver_MOD_deltaw>, ierr=ierr at entry=0x7fffffffb938) > at > /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/ksp/interface/ftn-custom/zitfuncf.c:54 > > #9 0x00007fffb3d7e697 in anksolver::ankstep (firstcall=) > at ../NKSolver/NKSolvers.F90:2891 > > #10 0x00007fffb3ebbbb9 in solvers::solvestate () at > ../solver/solvers.F90:1114 > > #11 0x00007fffb3ebe285 in solvers::solver () at ../solver/solvers.F90:69 > > #12 0x00007fffb3cfca17 in f2py_rout_libadflow_solvers_solver > (capi_self=, capi_args=, > capi_keywds=, f2py_func=0x7fffb3ebe0e0 ) > > at libadflowmodule.c:6710 > > #13 0x00007fffb3cf85b3 in fortran_call (fp=, > arg=, kw=) at > /u/wk/gkenway/.local/lib/python2.7/site-packages/numpy/f2py/src/fortranobject.c:404 > > #14 0x00007ffff7a5aa63 in PyObject_Call () from > /lib64/libpython2.7.so.1.0 > > #15 0x00007ffff7aef236 in PyEval_EvalFrameEx () from > /lib64/libpython2.7.so.1.0 > > #16 0x00007ffff7af603d in PyEval_EvalCodeEx () from > /lib64/libpython2.7.so.1.0 > > #17 0x00007ffff7a7f978 in function_call () from > /lib64/libpython2.7.so.1.0 > > #18 0x00007ffff7a5aa63 in PyObject_Call () from > /lib64/libpython2.7.so.1.0 > > #19 0x00007ffff7a69a55 in instancemethod_call () from > /lib64/libpython2.7.so.1.0 > > #20 0x00007ffff7a5aa63 in PyObject_Call () from > /lib64/libpython2.7.so.1.0 > > #21 0x00007ffff7ab1e27 in slot_tp_call () from /lib64/libpython2.7.so.1.0 > > #22 0x00007ffff7a5aa63 in PyObject_Call () from > /lib64/libpython2.7.so.1.0 > > #23 0x00007ffff7aef236 in PyEval_EvalFrameEx () from > /lib64/libpython2.7.so.1.0 > > #24 0x00007ffff7af603d in PyEval_EvalCodeEx () from > /lib64/libpython2.7.so.1.0 > > #25 0x00007ffff7af6142 in PyEval_EvalCode () from > /lib64/libpython2.7.so.1.0 > > #26 0x00007ffff7b0f57f in run_mod () from /lib64/libpython2.7.so.1.0 > > #27 0x00007ffff7b1073e in PyRun_FileExFlags () from > /lib64/libpython2.7.so.1.0 > > #28 0x00007ffff7b119c9 in PyRun_SimpleFileExFlags () from > /lib64/libpython2.7.so.1.0 > > #29 0x00007ffff7b22b7f in Py_Main () from /lib64/libpython2.7.so.1.0 > > #30 0x00007ffff6d3f3d5 in __libc_start_main () from /lib64/libc.so.6 > > #31 0x000000000040066e in _start () > > > > Thanks, > > > > Gaetan > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jun 19 16:55:20 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 19 Jun 2018 21:55:20 +0000 Subject: [petsc-users] MatMFFDSetBase In-Reply-To: References: Message-ID: Could you please try changing it to CHKFORTRANNULLOBJECTDEREFERENCE; recompile and run again Barry > On Jun 19, 2018, at 3:28 PM, Gaetan Kenway wrote: > > I tried the patch. It initially didn't compile because of a conflict with the corresponding ftn-auto file. When i commented out the stuff there it compiled, but I got a segfault here: > > at /nobackup/gkenway/packages/petsc-3.9.2/src/mat/impls/mffd/ftn-custom/zmffdf.c:52 > 52 *ierr = MatMFFDSetBase(*mat,*u,*f); > (gdb) bt > #0 matmffdsetbase_ (mat=mat at entry=0x7fffb461ba10 <__anksolver_MOD_drdw>, u=u at entry=0x7fffb461b9a0 <__anksolver_MOD_wvec>, f=0x0, f at entry=0x7fffffffb970, ierr=ierr at entry=0x7fffffffb938) > at /nobackup/gkenway/packages/petsc-3.9.2/src/mat/impls/mffd/ftn-custom/zmffdf.c:52 > #1 0x00007fffb3d7e871 in anksolver::ankstep (firstcall=) at ../NKSolver/NKSolvers.F90:2891 > #2 0x00007fffb3ebbde9 in solvers::solvestate () at ../solver/solvers.F90:1114 > #3 0x00007fffb3ebe4b5 in solvers::solver () at ../solver/solvers.F90:69 > #4 0x00007fffb3cfcaa7 in f2py_rout_libadflow_solvers_solver (capi_self=, capi_args=, capi_keywds=, f2py_func=0x7fffb3ebe310 ) > at libadflowmodule.c:6710 > #5 0x00007fffb3cf8643 in fortran_call (fp=, arg=, kw=) at /u/wk/gkenway/.local/lib/python2.7/site-packages/numpy/f2py/src/fortranobject.c:404 > #6 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 > #7 0x00007ffff7aef236 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 > #8 0x00007ffff7af603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 > #9 0x00007ffff7a7f978 in function_call () from /lib64/libpython2.7.so.1.0 > #10 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 > #11 0x00007ffff7a69a55 in instancemethod_call () from /lib64/libpython2.7.so.1.0 > #12 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 > #13 0x00007ffff7ab1e27 in slot_tp_call () from /lib64/libpython2.7.so.1.0 > #14 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 > #15 0x00007ffff7aef236 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 > #16 0x00007ffff7af603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 > #17 0x00007ffff7af6142 in PyEval_EvalCode () from /lib64/libpython2.7.so.1.0 > #18 0x00007ffff7b0f57f in run_mod () from /lib64/libpython2.7.so.1.0 > #19 0x00007ffff7b1073e in PyRun_FileExFlags () from /lib64/libpython2.7.so.1.0 > #20 0x00007ffff7b119c9 in PyRun_SimpleFileExFlags () from /lib64/libpython2.7.so.1.0 > #21 0x00007ffff7b22b7f in Py_Main () from /lib64/libpython2.7.so.1.0 > #22 0x00007ffff6d3f3d5 in __libc_start_main () from /lib64/libc.so.6 > #23 0x000000000040066e in _start () > > which was the new code that got added. So I never got the patch to work correctly. > > However, I circumvented the problem by making my own vector and calling the matfd-function, and then passing in the vector. This is essentially the same thing and I means I don't need an if-def for older versions. > > Gaetan > > > > On Tue, Jun 19, 2018 at 11:24 AM Smith, Barry F. wrote: > > The code to handle the null vector got mistakenly removed. Please find attached a patch that should resolve the problem. > > > > Please let us know if it works, > > Thanks > > Barry > > > > On Jun 19, 2018, at 12:36 PM, Gaetan Kenway wrote: > > > > Seems like the MatMFFDSetBase is now broken in petsc 3.9 > > > > In 3.7 I used: > > > > call MatMFFDSetBase(dRdW, wVec, PETSC_NULL_OBJECT, ierr) > > > > and the most logical translation is now > > > > call MatMFFDSetBase(dRdW, wVec, PETSC_NULL_VEC, ierr) > > > > Unfortunately this just segfaults: > > #0 VecAXPY (y=y at entry=0x10a3750, alpha=alpha at entry=-1, x=x at entry=0xffffffffffffffff) at /nobackup/gkenway/packages/petsc-3.9.2/src/vec/vec/interface/rvector.c:601 > > #1 0x00007fffb697faae in MatMult_MFFD (mat=0x291a150, a=0x10b25c0, y=0x10a3750) at /nobackup/gkenway/packages/petsc-3.9.2/src/mat/impls/mffd/mffd.c:368 > > #2 0x00007fffb6847af2 in MatMult (mat=0x291a150, x=x at entry=0x10b25c0, y=y at entry=0x10a3750) at /nobackup/gkenway/packages/petsc-3.9.2/src/mat/interface/matrix.c:2305 > > #3 0x00007fffb6e982e7 in PCApplyBAorAB (pc=0xe64d00, side=PC_RIGHT, x=0x10aaee0, y=y at entry=0x10a3750, work=0x10b25c0) at /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/pc/interface/precon.c:680 > > #4 0x00007fffb6f02b84 in KSP_PCApplyBAorAB (w=, y=0x10a3750, x=, ksp=0xf135e0) at /u/wk/gkenway/scratch/packages/petsc-3.9.2/include/petsc/private/kspimpl.h:304 > > #5 KSPGMRESCycle (itcount=itcount at entry=0x7fffffffb73c, ksp=ksp at entry=0xf135e0) at /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/ksp/impls/gmres/gmres.c:152 > > #6 0x00007fffb6f035c2 in KSPSolve_GMRES (ksp=0xf135e0) at /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/ksp/impls/gmres/gmres.c:234 > > #7 0x00007fffb6f35050 in KSPSolve (ksp=0xf135e0, b=0xf7ced0, x=0xf77be0) at /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/ksp/interface/itfunc.c:669 > > #8 0x00007fffb6f482ee in kspsolve_ (ksp=ksp at entry=0x7fffb461bae8 <__anksolver_MOD_ank_ksp>, b=, b at entry=0x7fffb461b9c8 <__anksolver_MOD_rvec>, x=, > > x at entry=0x7fffb461ba18 <__anksolver_MOD_deltaw>, ierr=ierr at entry=0x7fffffffb938) at /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/ksp/interface/ftn-custom/zitfuncf.c:54 > > #9 0x00007fffb3d7e697 in anksolver::ankstep (firstcall=) at ../NKSolver/NKSolvers.F90:2891 > > #10 0x00007fffb3ebbbb9 in solvers::solvestate () at ../solver/solvers.F90:1114 > > #11 0x00007fffb3ebe285 in solvers::solver () at ../solver/solvers.F90:69 > > #12 0x00007fffb3cfca17 in f2py_rout_libadflow_solvers_solver (capi_self=, capi_args=, capi_keywds=, f2py_func=0x7fffb3ebe0e0 ) > > at libadflowmodule.c:6710 > > #13 0x00007fffb3cf85b3 in fortran_call (fp=, arg=, kw=) at /u/wk/gkenway/.local/lib/python2.7/site-packages/numpy/f2py/src/fortranobject.c:404 > > #14 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 > > #15 0x00007ffff7aef236 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 > > #16 0x00007ffff7af603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 > > #17 0x00007ffff7a7f978 in function_call () from /lib64/libpython2.7.so.1.0 > > #18 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 > > #19 0x00007ffff7a69a55 in instancemethod_call () from /lib64/libpython2.7.so.1.0 > > #20 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 > > #21 0x00007ffff7ab1e27 in slot_tp_call () from /lib64/libpython2.7.so.1.0 > > #22 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 > > #23 0x00007ffff7aef236 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 > > #24 0x00007ffff7af603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 > > #25 0x00007ffff7af6142 in PyEval_EvalCode () from /lib64/libpython2.7.so.1.0 > > #26 0x00007ffff7b0f57f in run_mod () from /lib64/libpython2.7.so.1.0 > > #27 0x00007ffff7b1073e in PyRun_FileExFlags () from /lib64/libpython2.7.so.1.0 > > #28 0x00007ffff7b119c9 in PyRun_SimpleFileExFlags () from /lib64/libpython2.7.so.1.0 > > #29 0x00007ffff7b22b7f in Py_Main () from /lib64/libpython2.7.so.1.0 > > #30 0x00007ffff6d3f3d5 in __libc_start_main () from /lib64/libc.so.6 > > #31 0x000000000040066e in _start () > > > > Thanks, > > > > Gaetan > > > > > > > > > From nahmad16 at ku.edu.tr Wed Jun 20 03:02:51 2018 From: nahmad16 at ku.edu.tr (Najeeb Ahmad) Date: Wed, 20 Jun 2018 13:02:51 +0500 Subject: [petsc-users] f2cblashlapack download failure Message-ID: Hi, I am trying to configure PETSc with --download-f2cblaslapack but the library couldn't be downloaded. I also tried to access the URL manually but it is not accessible. Is the download server under maintenance? The URLs tried by PETSc for the download are: http://ftp.mcs.anl.gov/pub/petsc/externalpackages/f2cblaslapack-3.4.2.q3.tar.gz and ftp://ftp.mcs.anl.gov/pub/petsc/externalpackages/f2cblaslapack-3.4.2.q3.tar.gz Thanks for your assistance. -- *Najeeb Ahmad* *Research and Teaching Assistant* *PARallel and MultiCORE Computing Laboratory (ParCoreLab) * *Computer Science and Engineering* *Ko? University, Istanbul, Turkey* -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Jun 20 03:11:43 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 20 Jun 2018 13:41:43 +0530 Subject: [petsc-users] f2cblashlapack download failure In-Reply-To: References: Message-ID: Yes - ftp.mcs.anl.gov is down - and likely to be down for many hours. Sorry for the inconvinence. Unless quad precision is needed - system blas (on most machines) should work. Satish On Wed, 20 Jun 2018, Najeeb Ahmad wrote: > Hi, > > I am trying to configure PETSc with --download-f2cblaslapack but the > library couldn't be downloaded. I also tried to access the URL manually but > it is not accessible. Is the download server under maintenance? The URLs > tried by PETSc for the download are: > > http://ftp.mcs.anl.gov/pub/petsc/externalpackages/f2cblaslapack-3.4.2.q3.tar.gz > > and > > ftp://ftp.mcs.anl.gov/pub/petsc/externalpackages/f2cblaslapack-3.4.2.q3.tar.gz > > Thanks for your assistance. > > From nahmad16 at ku.edu.tr Wed Jun 20 03:20:05 2018 From: nahmad16 at ku.edu.tr (Najeeb Ahmad) Date: Wed, 20 Jun 2018 13:20:05 +0500 Subject: [petsc-users] f2cblashlapack download failure In-Reply-To: References: Message-ID: Thanks for your prompt response Satish. I was getting difference in results precision between PETSc and MATLAB and therefore wanted to try if PetscReal precision is an issue. Just noted that KSP is converging with Residual norm of 9.283109e-02 although I have specified -ksp_atol 1E-10 in my command options. Here is the solver setup code: ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr); ierr = KSPSetOperators(ksp,A,A);CHKERRQ(ierr); ierr = KSPSetType(ksp, KSPGMRES);CHKERRQ(ierr); ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); ierr = PCSetType(pc, PCJACOBI); ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr); ierr = KSPSetUp(ksp);CHKERRQ(ierr); Thanks for your assistance. On Wed, Jun 20, 2018 at 1:13 PM Satish Balay wrote: > Yes - ftp.mcs.anl.gov is down - and likely to be down for many hours. > > Sorry for the inconvinence. > > Unless quad precision is needed - system blas (on most machines) should > work. > > Satish > > > On Wed, 20 Jun 2018, Najeeb Ahmad wrote: > > > Hi, > > > > I am trying to configure PETSc with --download-f2cblaslapack but the > > library couldn't be downloaded. I also tried to access the URL manually > but > > it is not accessible. Is the download server under maintenance? The URLs > > tried by PETSc for the download are: > > > > > http://ftp.mcs.anl.gov/pub/petsc/externalpackages/f2cblaslapack-3.4.2.q3.tar.gz > > > > and > > > > > ftp://ftp.mcs.anl.gov/pub/petsc/externalpackages/f2cblaslapack-3.4.2.q3.tar.gz > > > > Thanks for your assistance. > > > > > > -- *Najeeb Ahmad* *Research and Teaching Assistant* *PARallel and MultiCORE Computing Laboratory (ParCoreLab) * *Computer Science and Engineering* *Ko? University, Istanbul, Turkey* -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Jun 20 03:56:01 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 20 Jun 2018 14:26:01 +0530 Subject: [petsc-users] f2cblashlapack download failure In-Reply-To: References: Message-ID: Hm - matlab [64bit] uses ILP64 mkl. You can try using the same ILP64 mkl from matlab - and compare. for eg: ./configure PETSC_ARCH=arch-test --with-blaslapack-lib=/soft/com/packages/MATLAB/R2016abin/glnxa64/mkl.so -known-64-bit-blas-indices=1 Or use openblas: ./configure --download-openblas --download-openblas-64-bit-blas-indices=1 Satish On Wed, 20 Jun 2018, Najeeb Ahmad wrote: > Thanks for your prompt response Satish. > > I was getting difference in results precision between PETSc and MATLAB and > therefore wanted to try if PetscReal precision is an issue. Just noted that > KSP is converging with Residual norm of 9.283109e-02 although I have > specified -ksp_atol 1E-10 in my command options. Here is the solver setup > code: > > ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr); > ierr = KSPSetOperators(ksp,A,A);CHKERRQ(ierr); > ierr = KSPSetType(ksp, KSPGMRES);CHKERRQ(ierr); > ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); > ierr = PCSetType(pc, PCJACOBI); > ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr); > ierr = KSPSetUp(ksp);CHKERRQ(ierr); > > Thanks for your assistance. > > On Wed, Jun 20, 2018 at 1:13 PM Satish Balay wrote: > > > Yes - ftp.mcs.anl.gov is down - and likely to be down for many hours. > > > > Sorry for the inconvinence. > > > > Unless quad precision is needed - system blas (on most machines) should > > work. > > > > Satish > > > > > > On Wed, 20 Jun 2018, Najeeb Ahmad wrote: > > > > > Hi, > > > > > > I am trying to configure PETSc with --download-f2cblaslapack but the > > > library couldn't be downloaded. I also tried to access the URL manually > > but > > > it is not accessible. Is the download server under maintenance? The URLs > > > tried by PETSc for the download are: > > > > > > > > http://ftp.mcs.anl.gov/pub/petsc/externalpackages/f2cblaslapack-3.4.2.q3.tar.gz > > > > > > and > > > > > > > > ftp://ftp.mcs.anl.gov/pub/petsc/externalpackages/f2cblaslapack-3.4.2.q3.tar.gz > > > > > > Thanks for your assistance. > > > > > > > > > > > > From nahmad16 at ku.edu.tr Wed Jun 20 04:36:10 2018 From: nahmad16 at ku.edu.tr (Najeeb Ahmad) Date: Wed, 20 Jun 2018 14:36:10 +0500 Subject: [petsc-users] f2cblashlapack download failure In-Reply-To: References: Message-ID: Thanks a lot Satish. I will try it out. On Wed, Jun 20, 2018 at 1:56 PM Satish Balay wrote: > Hm - matlab [64bit] uses ILP64 mkl. > > You can try using the same ILP64 mkl from matlab - and compare. > > for eg: > > ./configure PETSC_ARCH=arch-test > --with-blaslapack-lib=/soft/com/packages/MATLAB/R2016abin/glnxa64/mkl.so > -known-64-bit-blas-indices=1 > > Or use openblas: > > ./configure --download-openblas --download-openblas-64-bit-blas-indices=1 > > Satish > > On Wed, 20 Jun 2018, Najeeb Ahmad wrote: > > > Thanks for your prompt response Satish. > > > > I was getting difference in results precision between PETSc and MATLAB > and > > therefore wanted to try if PetscReal precision is an issue. Just noted > that > > KSP is converging with Residual norm of 9.283109e-02 although I have > > specified -ksp_atol 1E-10 in my command options. Here is the solver setup > > code: > > > > ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr); > > ierr = KSPSetOperators(ksp,A,A);CHKERRQ(ierr); > > ierr = KSPSetType(ksp, KSPGMRES);CHKERRQ(ierr); > > ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); > > ierr = PCSetType(pc, PCJACOBI); > > ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr); > > ierr = KSPSetUp(ksp);CHKERRQ(ierr); > > > > Thanks for your assistance. > > > > On Wed, Jun 20, 2018 at 1:13 PM Satish Balay wrote: > > > > > Yes - ftp.mcs.anl.gov is down - and likely to be down for many hours. > > > > > > Sorry for the inconvinence. > > > > > > Unless quad precision is needed - system blas (on most machines) should > > > work. > > > > > > Satish > > > > > > > > > On Wed, 20 Jun 2018, Najeeb Ahmad wrote: > > > > > > > Hi, > > > > > > > > I am trying to configure PETSc with --download-f2cblaslapack but the > > > > library couldn't be downloaded. I also tried to access the URL > manually > > > but > > > > it is not accessible. Is the download server under maintenance? The > URLs > > > > tried by PETSc for the download are: > > > > > > > > > > > > http://ftp.mcs.anl.gov/pub/petsc/externalpackages/f2cblaslapack-3.4.2.q3.tar.gz > > > > > > > > and > > > > > > > > > > > > ftp://ftp.mcs.anl.gov/pub/petsc/externalpackages/f2cblaslapack-3.4.2.q3.tar.gz > > > > > > > > Thanks for your assistance. > > > > > > > > > > > > > > > > > > > > -- *Najeeb Ahmad* *Research and Teaching Assistant* *PARallel and MultiCORE Computing Laboratory (ParCoreLab) * *Computer Science and Engineering* *Ko? University, Istanbul, Turkey* -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Wed Jun 20 10:02:54 2018 From: hzhang at mcs.anl.gov (Hong) Date: Wed, 20 Jun 2018 10:02:54 -0500 Subject: [petsc-users] MAT_NEW_NONZERO_LOCATIONS working? In-Reply-To: References: <1153922A-8876-492B-8AD9-F41DD718FD21@anl.gov> <58D5E959-0E3C-4EF2-8270-882D1EA6EA38@mcs.anl.gov> Message-ID: Marius: I added support for parallel sparse RHS (in host) for MatMatSolve_MUMPS() https://bitbucket.org/petsc/petsc/commits/2b691707dd0cf456c808def006e14b6f56b364b6 It is in the branch hzhang/mumps-spRHS. You may test it. I'll further cleanup the routine, test it, then merge it to petsc. Hong On Mon, Jun 4, 2018 at 9:54 AM, Hong wrote: > > > On Mon, Jun 4, 2018 at 1:03 PM, Jean-Yves LExcellent < > Jean-Yves.L.Excellent at ens-lyon.fr> wrote: > >> >> Thanks for the details of your needs. >> >> For the first application, the sparse RHS feature with distributed >> solution should effectively be fine. >> > I'll add parallel support of this feature in petsc after I'm back the ANL > after June 14th. > Hong > >> >> For the second one, a future distributed RHS feature (not currently >> available in MUMPS) might help if the centralized sparse RHS is too >> memory consuming, depending on the size of B1. >> >> Regards, >> Jean-Yves, for the MUMPS developers >> >> >> I want to invert a rather large sparse matrix for this using a sparse >> rhs with centralized input would be ok as long as the solution is >> distributed. >> >> and the second application I have in mind is solving a system of the from >> AX=B where A and B are sparse and B is given by a block matrix of the form >> B=[B1 0, 0 0] where B1 is dense but the dimension is (much) smaller than >> that of the whole matrix B. >> >> Marius: >> Current PETSc interface supports sequential sparse multiple right-hand >> side, but not distributed. >> It turns out that mumps does not support distributed sparse multiple >> right-hand sides at >> the moment (see attached email). >> >> Jean-Yves invites you to communicate with him directly. >> Let me know what we can help on this matter, >> e.g., add support for parallel implementation of sparse multiple >> right-hand side with >> centralized rhs input? >> >> Hong >> >> ---------------------- >> Jean-Yves LExcellent >> 5:14 AM (3 hours ago) >> >> to Hong, mumps-dev >> >> >> >> >> Hello, >> >> We do not support distributed sparse multiple right-hand sides at >> the moment. From the feedback we have from applications, the >> right-hand sides are often very sparse, and having them distributed >> did not seem critical. >> >> Since we are specifying a distributed right-hand sides feature at the >> moment, could you let us know in more detail the need regarding >> distributed sparse right-hand side (e.g., do all columns have the same >> nonzero structure in that case) or put us in contact with the user who >> needs this? >> >> Thanks, >> Jean-Yves and Patrick >> >>> >>> Thanks a lot guys, very helpful. >>> >>> >>> >>> >>> I see MUMPS http://mumps.enseeiht.fr/ >>> >>> Sparse multiple right-hand side, distributed solution; Exploitation of >>> sparsity in the right-hand sidesPETSc interface computes mumps distributed >>> solution as default (this is not new) (ICNTL(21) = 1) >>> >>> I will add support for Sparse multiple right-hand side. >>> >>> Hong >>> >>> On Thu, May 31, 2018 at 11:25 AM, Smith, Barry F. >> [mailto:bsmith at mcs.anl.gov]> wrote: >>> Hong, >>> >>> Can you see about adding support for distributed right hand side? >>> >>> Thanks >>> >>> Barry >>> >>> > On May 31, 2018, at 2:37 AM, Marius Buerkle >> mbuerkle at web.de]> wrote: >>> > >>> > The fix for MAT_NEW_NONZERO_LOCATIONS, thanks again. >>> > >>> > I have yet another question, sorry. The recent version of MUMPS >>> supports distributed and sparse RHS is there any chance that this will be >>> supported in PETSc in the near future? >>> > >>> > >>> > >>> > >>> >> On May 30, 2018, at 6:55 PM, Marius Buerkle >> mbuerkle at web.de]> wrote: >>> >> >>> >> Thanks for the quick fix, I will test it and report back. >>> >> I have another maybe related question, if MAT_NEW_NONZERO_LOCATIONS >>> is true and let's say 1 new nonzero position is created it does not >>> allocated 1 but several new nonzeros but only use 1. >>> > >>> > Correct >>> > >>> >> I think that is normal, right? >>> > >>> > Yes >>> > >>> >> But, at least as far as I understand the manual, a subsequent call of >>> mat assemble with >>> >> MAT_FINAL_ASSEMBLY should compress out the unused allocations and >>> release the memory, is this correct? >>> > >>> > It "compresses it out" (by shifting all the nonzero entries to the >>> beginning of the internal i, j, and a arrays), but does NOT release any >>> memory. Since the values are stored in one big contiguous array (obtained >>> with a single malloc) it cannot just free part of the array, so the extra >>> locations just sit harmlessly at the end if the array unused. >>> > >>> >> If so, this did not work for me, even after doing >>> >> MAT_FINAL_ASSEMBLY the unused nonzero allocations remain. Is this >>> normal? >>> > >>> > Yes, >>> > >>> > Barry >>> > >>> >> >>> >>> >>> >>> Fixed in the branch barry/fix-mat-new-nonzero-locations/maint >>> >>> >>> >>> Once this passes testing it will go into the maint branch and then >>> the next patch release but you can use it now in the branch >>> barry/fix-mat-new-nonzero-locations/maint >>> >>> >>> >>> Thanks for the report and reproducible example >>> >>> >>> >>> Barry >>> >>> >>> >>> >>> >>>> On May 29, 2018, at 7:51 PM, Marius Buerkle >> [mailto:mbuerkle at web.de]> wrote: >>> >>>> >>> >>>> Sure, I made a small reproducer, it is Fortran though I hope that >>> is ok. If MAT_NEW_NONZERO_LOCATIONS is set to false I get an error, if it >>> is set to true the new nonzero element is inserted, if >>> MAT_NEW_NONZERO_LOCATIONS is false and either MAT_NEW_NONZERO_LOCATION_ERR >>> or MAT_NEW_NONZERO_ALLOCATION_ERR is set to false afterwards then the new >>> nonzero is also created without an error, but if MAT_NEW_NONZERO_LOCATIONS >>> is set to false after MAT_NEW_NONZERO_LOCATION_ERR/MAT_NEW_NONZERO_ALLOCATION_ERR >>> have been set to false I get an error again. >>> >>>> >>> >>>> >>> >>>> program newnonzero >>> >>>> #include >>> >>>> use petscmat >>> >>>> implicit none >>> >>>> >>> >>>> Mat :: A >>> >>>> PetscInt :: dnnz,onnz,n,m,idxm(1),idxn(1),nl1,nl2 >>> >>>> PetscScalar :: v(1) >>> >>>> PetscReal :: info(MAT_INFO_SIZE) >>> >>>> PetscErrorCode :: ierr >>> >>>> >>> >>>> integer :: nproc,iproc,i >>> >>>> >>> >>>> call PetscInitialize(PETSC_NULL_CHARACTER,ierr) >>> >>>> >>> >>>> call MPI_COMM_SIZE(PETSC_COMM_WORLD, nproc,ierr) >>> >>>> >>> >>>> call MPI_Comm_rank( PETSC_COMM_WORLD, iproc, ierr ) >>> >>>> >>> >>>> n=3 >>> >>>> m=n >>> >>>> call MatCreateAIJ(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,n,m, >>> 1,PETSC_NULL_INTEGER,0,PETSC_NULL_INTEGER,A,ierr) >>> >>>> >>> >>>> >>> >>>> call MatGetOwnershipRange(A,nl1,nl2,ierr) >>> >>>> do i=nl1,nl2-1 >>> >>>> idxn(1)=i >>> >>>> idxm(1)=i >>> >>>> v(1)=1d0 >>> >>>> call MatSetValues(A,1,idxn,1,idxm, v,INSERT_VALUES,ierr) >>> >>>> end do >>> >>>> call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) >>> >>>> call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) >>> >>>> >>> >>>> call MatSetOption(A,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE,ierr) >>> >>>> !~ call MatSetOption(A,MAT_NEW_NONZERO_LOCATION_ERR,PETSC_FALSE, >>> ierr) >>> >>>> !~ call MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR >>> ,PETSC_FALSE,ierr) >>> >>>> !~ call MatSetOption(A,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE,ierr) >>> >>>> >>> >>>> >>> >>>> idxn(1)=0 >>> >>>> idxm(1)=n-1 >>> >>>> if ((idxn(1).ge.nl1).and.(idxn(1).le.nl2-1)) then >>> >>>> v(1)=2d0 >>> >>>> call MatSetValues(A,1,idxn,1,idxm, v,INSERT_VALUES,ierr) >>> >>>> end if >>> >>>> call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) >>> >>>> call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) >>> >>>> >>> >>>> if ((idxn(1).ge.nl1).and.(idxn(1).le.nl2-1)) then >>> >>>> v(1)=2d0 >>> >>>> call MatGetValues(A,1,idxn,1,idxm, v,ierr) >>> >>>> write(6,*) v >>> >>>> end if >>> >>>> >>> >>>> call PetscFinalize(ierr) >>> >>>> >>> >>>> end program newnonzero >>> >>>> >>> >>>> >>> >>>> >>> >>>> $ mpiexec.hydra -n 3 ./a.out >>> >>>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> >>>> [0]PETSC ERROR: Argument out of range >>> >>>> [0]PETSC ERROR: Inserting a new nonzero at global row/column (0, 2) >>> into matrix >>> >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>> ocumentation/faq.html[http://www.mcs.anl.gov/petsc/documenta >>> tion/faq.html] for trouble shooting. >>> >>>> [0]PETSC ERROR: Petsc Release Version 3.9.2, May, 20, 2018 >>> >>>> [0]PETSC ERROR: ./a.out on a named tono-hpc1 by marius Wed May 30 >>> 09:42:40 2018 >>> >>>> [0]PETSC ERROR: Configure options --prefix=/home/marius/prog/petsc/3.9.2 >>> --download-elemental=yes --download-metis=yes --download-parmetis=yes >>> --download-mumps=yes --with-scalapack-lib="/home/ma >>> rius/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/ >>> intel64/libmkl_scalapack_lp64.a -Wl,--start-group >>> /home/marius/intel/compilers_and_libraries_2018.2.199/linux/ >>> mkl/lib/intel64/libmkl_intel_lp64.a /home/marius/intel/compilers_a >>> nd_libraries_2018.2.199/linux/mkl/lib/intel64/libmkl_sequential.a >>> /home/marius/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64/libmkl_core.a >>> /home/marius/intel/compilers_and_libraries_2018.2.199/linux/ >>> mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.a -Wl,--end-group -lpthread >>> -lm -ldl" --FC=mpiifort --CC=mpicc --CXX=mpicxx --with-scalar-type=complex >>> --with-mpi-dir= --with-blaslapack-lib="/home/m >>> arius/intel/compilers_and_libraries_2018.2.199/linux/mkl/ >>> lib/intel64/libmkl_scalapack_lp64.a -Wl,--start-group >>> /home/marius/intel/compilers_and_libraries_2018.2.199/linux/ >>> mkl/lib/intel64/libmkl_intel_lp64.a /home/marius/intel/compilers_a >>> nd_libraries_2018.2.199/linux/mkl/lib/intel64/libmkl_sequential.a >>> /home/marius/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64/libmkl_core.a >>> /home/marius/intel/compilers_and_libraries_2018.2.199/linux/ >>> mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.a -Wl,--end-group -lpthread >>> -lm -ldl" --with-cxx-dialect=C++11 --download-superlu_dist=yes >>> --download-ptscotch=yes --with-x --with-debugging=1 --download-superlu=yes >>> --with-mkl_cpardiso=1 --with-mkl_pardiso=1 --with-scalapack=1 >>> >>>> [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 607 in >>> /home/marius/prog/petsc/petsc-3.9.2/src/mat/impls/aij/mpi/mpiaij.c >>> >>>> [0]PETSC ERROR: #2 MatSetValues() line 1312 in >>> /home/marius/prog/petsc/petsc-3.9.2/src/mat/interface/matrix.c >>> >>>> (0.000000000000000E+000,0.000000000000000E+000) >>> >>>> >>> >>>> >>> >>>> >>> >>>> Please send complete error message; type of matrix used etc. >>> Ideally code that demonstrates the problem. >>> >>>> >>> >>>> Barry >>> >>>> >>> >>>> >>> >>>>> On May 29, 2018, at 3:31 AM, Marius Buerkle >> [mailto:mbuerkle at web.de]> wrote: >>> >>>>> >>> >>>>> >>> >>>>> Hi, >>> >>>>> >>> >>>>> I tried to set MAT_NEW_NONZERO_LOCATIONS to false, as far as I >>> understood MatSetValues should simply ignore entries which would give rise >>> to new nonzero values not creating a new entry and not cause an error, but >>> I get "[1]PETSC ERROR: Inserting a new nonzero at global row/column". Is >>> this option supposed to work or not? >>> >>>> >>> >>> >>> >>> >>> > >>> >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Jun 20 10:44:17 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 20 Jun 2018 15:44:17 +0000 Subject: [petsc-users] f2cblashlapack download failure In-Reply-To: References: Message-ID: <24883272-6025-421A-A417-03C0DE94D6FB@anl.gov> Also try running with -ksp_converged_reason to see why the solver is returning earlier than you expect. Barry > On Jun 20, 2018, at 3:02 AM, Najeeb Ahmad wrote: > > Hi, > > I am trying to configure PETSc with --download-f2cblaslapack but the library couldn't be downloaded. I also tried to access the URL manually but it is not accessible. Is the download server under maintenance? The URLs tried by PETSc for the download are: > > http://ftp.mcs.anl.gov/pub/petsc/externalpackages/f2cblaslapack-3.4.2.q3.tar.gz > > and > > ftp://ftp.mcs.anl.gov/pub/petsc/externalpackages/f2cblaslapack-3.4.2.q3.tar.gz > > Thanks for your assistance. > > -- > Najeeb Ahmad > > Research and Teaching Assistant > PARallel and MultiCORE Computing Laboratory (ParCoreLab) > Computer Science and Engineering > Ko? University, Istanbul, Turkey > From nahmad16 at ku.edu.tr Wed Jun 20 12:11:31 2018 From: nahmad16 at ku.edu.tr (Najeeb Ahmad) Date: Wed, 20 Jun 2018 22:11:31 +0500 Subject: [petsc-users] f2cblashlapack download failure In-Reply-To: <24883272-6025-421A-A417-03C0DE94D6FB@anl.gov> References: <24883272-6025-421A-A417-03C0DE94D6FB@anl.gov> Message-ID: Thank you Barry for the useful suggestion. I will try it out. On Wed, Jun 20, 2018 at 8:44 PM Smith, Barry F. wrote: > > Also try running with -ksp_converged_reason to see why the solver is > returning earlier than you expect. > > Barry > > > > On Jun 20, 2018, at 3:02 AM, Najeeb Ahmad wrote: > > > > Hi, > > > > I am trying to configure PETSc with --download-f2cblaslapack but the > library couldn't be downloaded. I also tried to access the URL manually but > it is not accessible. Is the download server under maintenance? The URLs > tried by PETSc for the download are: > > > > > http://ftp.mcs.anl.gov/pub/petsc/externalpackages/f2cblaslapack-3.4.2.q3.tar.gz > > > > and > > > > > ftp://ftp.mcs.anl.gov/pub/petsc/externalpackages/f2cblaslapack-3.4.2.q3.tar.gz > > > > Thanks for your assistance. > > > > -- > > Najeeb Ahmad > > > > Research and Teaching Assistant > > PARallel and MultiCORE Computing Laboratory (ParCoreLab) > > Computer Science and Engineering > > Ko? University, Istanbul, Turkey > > > > -- *Najeeb Ahmad* *Research and Teaching Assistant* *PARallel and MultiCORE Computing Laboratory (ParCoreLab) * *Computer Science and Engineering* *Ko? University, Istanbul, Turkey* -------------- next part -------------- An HTML attachment was scrubbed... URL: From mpovolot at purdue.edu Wed Jun 20 12:40:30 2018 From: mpovolot at purdue.edu (Michael Povolotskyi) Date: Wed, 20 Jun 2018 13:40:30 -0400 Subject: [petsc-users] question about the most effective implementation Message-ID: <53ac078b-5399-63d1-5a83-97f5adbb4ffe@purdue.edu> Dear Petsc team, what would be the most efficient way to compute the following: V += A*B*C, where V is a dense square matrix, B is a square diagonal matrix, A and C are dense rectangular matrices? I would like to benefit from the fact that B is diagonal. thank you, Michael. From bsmith at mcs.anl.gov Wed Jun 20 13:02:50 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 20 Jun 2018 18:02:50 +0000 Subject: [petsc-users] question about the most effective implementation In-Reply-To: <53ac078b-5399-63d1-5a83-97f5adbb4ffe@purdue.edu> References: <53ac078b-5399-63d1-5a83-97f5adbb4ffe@purdue.edu> Message-ID: Just store B as AIJ. The huge amount of work and communication for the dense matrix matrix product will totally swamp out the little extra overhead of using AIJ instead of writing custom code for B*C. Barry > On Jun 20, 2018, at 12:40 PM, Michael Povolotskyi wrote: > > Dear Petsc team, > > what would be the most efficient way to compute the following: > > V += A*B*C, > > where V is a dense square matrix, > > B is a square diagonal matrix, A and C are dense rectangular matrices? > > I would like to benefit from the fact that B is diagonal. > > thank you, > > Michael. > From jed at jedbrown.org Wed Jun 20 13:06:05 2018 From: jed at jedbrown.org (Jed Brown) Date: Wed, 20 Jun 2018 12:06:05 -0600 Subject: [petsc-users] question about the most effective implementation In-Reply-To: <53ac078b-5399-63d1-5a83-97f5adbb4ffe@purdue.edu> References: <53ac078b-5399-63d1-5a83-97f5adbb4ffe@purdue.edu> Message-ID: <87fu1h8h9u.fsf@jedbrown.org> Michael Povolotskyi writes: > Dear Petsc team, > > what would be the most efficient way to compute the following: > > V += A*B*C, > > where V is a dense square matrix, > > B is a square diagonal matrix, A and C are dense rectangular matrices? > > I would like to benefit from the fact that B is diagonal. MatDiagonalScale. No matter what you do, the cost will be dominated by the dense product. From zakaryah at gmail.com Wed Jun 20 13:35:12 2018 From: zakaryah at gmail.com (zakaryah) Date: Wed, 20 Jun 2018 14:35:12 -0400 Subject: [petsc-users] f2cblashlapack download failure In-Reply-To: References: <24883272-6025-421A-A417-03C0DE94D6FB@anl.gov> Message-ID: Is there an expected time for the ftp site to be back up? On Wed, Jun 20, 2018 at 1:11 PM, Najeeb Ahmad wrote: > Thank you Barry for the useful suggestion. I will try it out. > > On Wed, Jun 20, 2018 at 8:44 PM Smith, Barry F. > wrote: > >> >> Also try running with -ksp_converged_reason to see why the solver is >> returning earlier than you expect. >> >> Barry >> >> >> > On Jun 20, 2018, at 3:02 AM, Najeeb Ahmad wrote: >> > >> > Hi, >> > >> > I am trying to configure PETSc with --download-f2cblaslapack but the >> library couldn't be downloaded. I also tried to access the URL manually but >> it is not accessible. Is the download server under maintenance? The URLs >> tried by PETSc for the download are: >> > >> > http://ftp.mcs.anl.gov/pub/petsc/externalpackages/ >> f2cblaslapack-3.4.2.q3.tar.gz >> > >> > and >> > >> > ftp://ftp.mcs.anl.gov/pub/petsc/externalpackages/ >> f2cblaslapack-3.4.2.q3.tar.gz >> > >> > Thanks for your assistance. >> > >> > -- >> > Najeeb Ahmad >> > >> > Research and Teaching Assistant >> > PARallel and MultiCORE Computing Laboratory (ParCoreLab) >> > Computer Science and Engineering >> > Ko? University, Istanbul, Turkey >> > >> >> > > -- > *Najeeb Ahmad* > > > *Research and Teaching Assistant* > *PARallel and MultiCORE Computing Laboratory (ParCoreLab) * > > *Computer Science and Engineering* > *Ko? University, Istanbul, Turkey* > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Jun 20 13:42:47 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 21 Jun 2018 00:12:47 +0530 Subject: [petsc-users] f2cblashlapack download failure In-Reply-To: References: <24883272-6025-421A-A417-03C0DE94D6FB@anl.gov> Message-ID: The notice regarding the server says: "ETA is COB 6/20/18" This would translate to 5PM CST [but this is just an estimate as the downtime is unscheduled].. Satish On Wed, 20 Jun 2018, zakaryah wrote: > Is there an expected time for the ftp site to be back up? From zakaryah at gmail.com Wed Jun 20 13:45:17 2018 From: zakaryah at gmail.com (zakaryah) Date: Wed, 20 Jun 2018 14:45:17 -0400 Subject: [petsc-users] f2cblashlapack download failure In-Reply-To: References: <24883272-6025-421A-A417-03C0DE94D6FB@anl.gov> Message-ID: Thanks Satish! On Wed, Jun 20, 2018 at 2:42 PM, Satish Balay wrote: > The notice regarding the server says: "ETA is COB 6/20/18" > > This would translate to 5PM CST [but this is just an estimate as the > downtime is unscheduled].. > > Satish > > On Wed, 20 Jun 2018, zakaryah wrote: > > > Is there an expected time for the ftp site to be back up? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mpovolot at purdue.edu Wed Jun 20 14:07:27 2018 From: mpovolot at purdue.edu (Michael Povolotskyi) Date: Wed, 20 Jun 2018 15:07:27 -0400 Subject: [petsc-users] question about the most effective implementation In-Reply-To: <87fu1h8h9u.fsf@jedbrown.org> References: <53ac078b-5399-63d1-5a83-97f5adbb4ffe@purdue.edu> <87fu1h8h9u.fsf@jedbrown.org> Message-ID: Thank you Barry and Jed, I think I'll go with MatDiagonalScale. Michael. On 06/20/2018 02:06 PM, Jed Brown wrote: > Michael Povolotskyi writes: > >> Dear Petsc team, >> >> what would be the most efficient way to compute the following: >> >> V += A*B*C, >> >> where V is a dense square matrix, >> >> B is a square diagonal matrix, A and C are dense rectangular matrices? >> >> I would like to benefit from the fact that B is diagonal. > MatDiagonalScale. No matter what you do, the cost will be dominated by the dense product. From gaetank at gmail.com Wed Jun 20 14:11:57 2018 From: gaetank at gmail.com (Gaetan Kenway) Date: Wed, 20 Jun 2018 12:11:57 -0700 Subject: [petsc-users] MatMFFDSetBase In-Reply-To: References: Message-ID: Hi Barry When I changed it to CHKFORTRANNULLOBJECTDEREFERENCE and commented out the duplicated stuff in the auto wrapper it worked just fine. Thanks, Gaetan On Tue, Jun 19, 2018 at 2:55 PM Smith, Barry F. wrote: > > Could you please try changing it to CHKFORTRANNULLOBJECTDEREFERENCE; > recompile and run again > > Barry > > > > On Jun 19, 2018, at 3:28 PM, Gaetan Kenway wrote: > > > > I tried the patch. It initially didn't compile because of a conflict > with the corresponding ftn-auto file. When i commented out the stuff there > it compiled, but I got a segfault here: > > > > at > /nobackup/gkenway/packages/petsc-3.9.2/src/mat/impls/mffd/ftn-custom/zmffdf.c:52 > > 52 *ierr = MatMFFDSetBase(*mat,*u,*f); > > (gdb) bt > > #0 matmffdsetbase_ (mat=mat at entry=0x7fffb461ba10 > <__anksolver_MOD_drdw>, u=u at entry=0x7fffb461b9a0 <__anksolver_MOD_wvec>, > f=0x0, f at entry=0x7fffffffb970, ierr=ierr at entry=0x7fffffffb938) > > at > /nobackup/gkenway/packages/petsc-3.9.2/src/mat/impls/mffd/ftn-custom/zmffdf.c:52 > > #1 0x00007fffb3d7e871 in anksolver::ankstep (firstcall=) > at ../NKSolver/NKSolvers.F90:2891 > > #2 0x00007fffb3ebbde9 in solvers::solvestate () at > ../solver/solvers.F90:1114 > > #3 0x00007fffb3ebe4b5 in solvers::solver () at ../solver/solvers.F90:69 > > #4 0x00007fffb3cfcaa7 in f2py_rout_libadflow_solvers_solver > (capi_self=, capi_args=, > capi_keywds=, f2py_func=0x7fffb3ebe310 ) > > at libadflowmodule.c:6710 > > #5 0x00007fffb3cf8643 in fortran_call (fp=, > arg=, kw=) at > /u/wk/gkenway/.local/lib/python2.7/site-packages/numpy/f2py/src/fortranobject.c:404 > > #6 0x00007ffff7a5aa63 in PyObject_Call () from > /lib64/libpython2.7.so.1.0 > > #7 0x00007ffff7aef236 in PyEval_EvalFrameEx () from > /lib64/libpython2.7.so.1.0 > > #8 0x00007ffff7af603d in PyEval_EvalCodeEx () from > /lib64/libpython2.7.so.1.0 > > #9 0x00007ffff7a7f978 in function_call () from > /lib64/libpython2.7.so.1.0 > > #10 0x00007ffff7a5aa63 in PyObject_Call () from > /lib64/libpython2.7.so.1.0 > > #11 0x00007ffff7a69a55 in instancemethod_call () from > /lib64/libpython2.7.so.1.0 > > #12 0x00007ffff7a5aa63 in PyObject_Call () from > /lib64/libpython2.7.so.1.0 > > #13 0x00007ffff7ab1e27 in slot_tp_call () from /lib64/libpython2.7.so.1.0 > > #14 0x00007ffff7a5aa63 in PyObject_Call () from > /lib64/libpython2.7.so.1.0 > > #15 0x00007ffff7aef236 in PyEval_EvalFrameEx () from > /lib64/libpython2.7.so.1.0 > > #16 0x00007ffff7af603d in PyEval_EvalCodeEx () from > /lib64/libpython2.7.so.1.0 > > #17 0x00007ffff7af6142 in PyEval_EvalCode () from > /lib64/libpython2.7.so.1.0 > > #18 0x00007ffff7b0f57f in run_mod () from /lib64/libpython2.7.so.1.0 > > #19 0x00007ffff7b1073e in PyRun_FileExFlags () from > /lib64/libpython2.7.so.1.0 > > #20 0x00007ffff7b119c9 in PyRun_SimpleFileExFlags () from > /lib64/libpython2.7.so.1.0 > > #21 0x00007ffff7b22b7f in Py_Main () from /lib64/libpython2.7.so.1.0 > > #22 0x00007ffff6d3f3d5 in __libc_start_main () from /lib64/libc.so.6 > > #23 0x000000000040066e in _start () > > > > which was the new code that got added. So I never got the patch to work > correctly. > > > > However, I circumvented the problem by making my own vector and calling > the matfd-function, and then passing in the vector. This is essentially the > same thing and I means I don't need an if-def for older versions. > > > > Gaetan > > > > > > > > On Tue, Jun 19, 2018 at 11:24 AM Smith, Barry F. > wrote: > > > > The code to handle the null vector got mistakenly removed. Please > find attached a patch that should resolve the problem. > > > > > > > > Please let us know if it works, > > > > Thanks > > > > Barry > > > > > > > On Jun 19, 2018, at 12:36 PM, Gaetan Kenway wrote: > > > > > > Seems like the MatMFFDSetBase is now broken in petsc 3.9 > > > > > > In 3.7 I used: > > > > > > call MatMFFDSetBase(dRdW, wVec, PETSC_NULL_OBJECT, ierr) > > > > > > and the most logical translation is now > > > > > > call MatMFFDSetBase(dRdW, wVec, PETSC_NULL_VEC, ierr) > > > > > > Unfortunately this just segfaults: > > > #0 VecAXPY (y=y at entry=0x10a3750, alpha=alpha at entry=-1, x=x at entry=0xffffffffffffffff) > at > /nobackup/gkenway/packages/petsc-3.9.2/src/vec/vec/interface/rvector.c:601 > > > #1 0x00007fffb697faae in MatMult_MFFD (mat=0x291a150, a=0x10b25c0, > y=0x10a3750) at > /nobackup/gkenway/packages/petsc-3.9.2/src/mat/impls/mffd/mffd.c:368 > > > #2 0x00007fffb6847af2 in MatMult (mat=0x291a150, x=x at entry=0x10b25c0, > y=y at entry=0x10a3750) at > /nobackup/gkenway/packages/petsc-3.9.2/src/mat/interface/matrix.c:2305 > > > #3 0x00007fffb6e982e7 in PCApplyBAorAB (pc=0xe64d00, side=PC_RIGHT, > x=0x10aaee0, y=y at entry=0x10a3750, work=0x10b25c0) at > /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/pc/interface/precon.c:680 > > > #4 0x00007fffb6f02b84 in KSP_PCApplyBAorAB (w=, > y=0x10a3750, x=, ksp=0xf135e0) at > /u/wk/gkenway/scratch/packages/petsc-3.9.2/include/petsc/private/kspimpl.h:304 > > > #5 KSPGMRESCycle (itcount=itcount at entry=0x7fffffffb73c, ksp=ksp at entry=0xf135e0) > at > /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/ksp/impls/gmres/gmres.c:152 > > > #6 0x00007fffb6f035c2 in KSPSolve_GMRES (ksp=0xf135e0) at > /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/ksp/impls/gmres/gmres.c:234 > > > #7 0x00007fffb6f35050 in KSPSolve (ksp=0xf135e0, b=0xf7ced0, > x=0xf77be0) at > /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/ksp/interface/itfunc.c:669 > > > #8 0x00007fffb6f482ee in kspsolve_ (ksp=ksp at entry=0x7fffb461bae8 > <__anksolver_MOD_ank_ksp>, b=, b at entry=0x7fffb461b9c8 > <__anksolver_MOD_rvec>, x=, > > > x at entry=0x7fffb461ba18 <__anksolver_MOD_deltaw>, ierr=ierr at entry=0x7fffffffb938) > at > /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/ksp/interface/ftn-custom/zitfuncf.c:54 > > > #9 0x00007fffb3d7e697 in anksolver::ankstep (firstcall= out>) at ../NKSolver/NKSolvers.F90:2891 > > > #10 0x00007fffb3ebbbb9 in solvers::solvestate () at > ../solver/solvers.F90:1114 > > > #11 0x00007fffb3ebe285 in solvers::solver () at > ../solver/solvers.F90:69 > > > #12 0x00007fffb3cfca17 in f2py_rout_libadflow_solvers_solver > (capi_self=, capi_args=, > capi_keywds=, f2py_func=0x7fffb3ebe0e0 ) > > > at libadflowmodule.c:6710 > > > #13 0x00007fffb3cf85b3 in fortran_call (fp=, > arg=, kw=) at > /u/wk/gkenway/.local/lib/python2.7/site-packages/numpy/f2py/src/fortranobject.c:404 > > > #14 0x00007ffff7a5aa63 in PyObject_Call () from > /lib64/libpython2.7.so.1.0 > > > #15 0x00007ffff7aef236 in PyEval_EvalFrameEx () from > /lib64/libpython2.7.so.1.0 > > > #16 0x00007ffff7af603d in PyEval_EvalCodeEx () from > /lib64/libpython2.7.so.1.0 > > > #17 0x00007ffff7a7f978 in function_call () from > /lib64/libpython2.7.so.1.0 > > > #18 0x00007ffff7a5aa63 in PyObject_Call () from > /lib64/libpython2.7.so.1.0 > > > #19 0x00007ffff7a69a55 in instancemethod_call () from > /lib64/libpython2.7.so.1.0 > > > #20 0x00007ffff7a5aa63 in PyObject_Call () from > /lib64/libpython2.7.so.1.0 > > > #21 0x00007ffff7ab1e27 in slot_tp_call () from > /lib64/libpython2.7.so.1.0 > > > #22 0x00007ffff7a5aa63 in PyObject_Call () from > /lib64/libpython2.7.so.1.0 > > > #23 0x00007ffff7aef236 in PyEval_EvalFrameEx () from > /lib64/libpython2.7.so.1.0 > > > #24 0x00007ffff7af603d in PyEval_EvalCodeEx () from > /lib64/libpython2.7.so.1.0 > > > #25 0x00007ffff7af6142 in PyEval_EvalCode () from > /lib64/libpython2.7.so.1.0 > > > #26 0x00007ffff7b0f57f in run_mod () from /lib64/libpython2.7.so.1.0 > > > #27 0x00007ffff7b1073e in PyRun_FileExFlags () from > /lib64/libpython2.7.so.1.0 > > > #28 0x00007ffff7b119c9 in PyRun_SimpleFileExFlags () from > /lib64/libpython2.7.so.1.0 > > > #29 0x00007ffff7b22b7f in Py_Main () from /lib64/libpython2.7.so.1.0 > > > #30 0x00007ffff6d3f3d5 in __libc_start_main () from /lib64/libc.so.6 > > > #31 0x000000000040066e in _start () > > > > > > Thanks, > > > > > > Gaetan > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Jun 20 21:00:36 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 21 Jun 2018 07:30:36 +0530 Subject: [petsc-users] f2cblashlapack download failure In-Reply-To: References: <24883272-6025-421A-A417-03C0DE94D6FB@anl.gov> Message-ID: All - an update: ftp.mcs.anl.gov is now back online. Satish On Thu, 21 Jun 2018, Satish Balay wrote: > The notice regarding the server says: "ETA is COB 6/20/18" > > This would translate to 5PM CST [but this is just an estimate as the downtime is unscheduled].. > > Satish > > On Wed, 20 Jun 2018, zakaryah wrote: > > > Is there an expected time for the ftp site to be back up? > From bsmith at mcs.anl.gov Wed Jun 20 21:34:01 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 21 Jun 2018 02:34:01 +0000 Subject: [petsc-users] MatMFFDSetBase In-Reply-To: References: Message-ID: Thanks for the update. This fix will be in the next patch release. Barry > On Jun 20, 2018, at 2:11 PM, Gaetan Kenway wrote: > > Hi Barry > > When I changed it to CHKFORTRANNULLOBJECTDEREFERENCE and commented out the duplicated stuff in the auto wrapper it worked just fine. > > Thanks, > Gaetan > > On Tue, Jun 19, 2018 at 2:55 PM Smith, Barry F. wrote: > > Could you please try changing it to CHKFORTRANNULLOBJECTDEREFERENCE; recompile and run again > > Barry > > > > On Jun 19, 2018, at 3:28 PM, Gaetan Kenway wrote: > > > > I tried the patch. It initially didn't compile because of a conflict with the corresponding ftn-auto file. When i commented out the stuff there it compiled, but I got a segfault here: > > > > at /nobackup/gkenway/packages/petsc-3.9.2/src/mat/impls/mffd/ftn-custom/zmffdf.c:52 > > 52 *ierr = MatMFFDSetBase(*mat,*u,*f); > > (gdb) bt > > #0 matmffdsetbase_ (mat=mat at entry=0x7fffb461ba10 <__anksolver_MOD_drdw>, u=u at entry=0x7fffb461b9a0 <__anksolver_MOD_wvec>, f=0x0, f at entry=0x7fffffffb970, ierr=ierr at entry=0x7fffffffb938) > > at /nobackup/gkenway/packages/petsc-3.9.2/src/mat/impls/mffd/ftn-custom/zmffdf.c:52 > > #1 0x00007fffb3d7e871 in anksolver::ankstep (firstcall=) at ../NKSolver/NKSolvers.F90:2891 > > #2 0x00007fffb3ebbde9 in solvers::solvestate () at ../solver/solvers.F90:1114 > > #3 0x00007fffb3ebe4b5 in solvers::solver () at ../solver/solvers.F90:69 > > #4 0x00007fffb3cfcaa7 in f2py_rout_libadflow_solvers_solver (capi_self=, capi_args=, capi_keywds=, f2py_func=0x7fffb3ebe310 ) > > at libadflowmodule.c:6710 > > #5 0x00007fffb3cf8643 in fortran_call (fp=, arg=, kw=) at /u/wk/gkenway/.local/lib/python2.7/site-packages/numpy/f2py/src/fortranobject.c:404 > > #6 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 > > #7 0x00007ffff7aef236 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 > > #8 0x00007ffff7af603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 > > #9 0x00007ffff7a7f978 in function_call () from /lib64/libpython2.7.so.1.0 > > #10 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 > > #11 0x00007ffff7a69a55 in instancemethod_call () from /lib64/libpython2.7.so.1.0 > > #12 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 > > #13 0x00007ffff7ab1e27 in slot_tp_call () from /lib64/libpython2.7.so.1.0 > > #14 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 > > #15 0x00007ffff7aef236 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 > > #16 0x00007ffff7af603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 > > #17 0x00007ffff7af6142 in PyEval_EvalCode () from /lib64/libpython2.7.so.1.0 > > #18 0x00007ffff7b0f57f in run_mod () from /lib64/libpython2.7.so.1.0 > > #19 0x00007ffff7b1073e in PyRun_FileExFlags () from /lib64/libpython2.7.so.1.0 > > #20 0x00007ffff7b119c9 in PyRun_SimpleFileExFlags () from /lib64/libpython2.7.so.1.0 > > #21 0x00007ffff7b22b7f in Py_Main () from /lib64/libpython2.7.so.1.0 > > #22 0x00007ffff6d3f3d5 in __libc_start_main () from /lib64/libc.so.6 > > #23 0x000000000040066e in _start () > > > > which was the new code that got added. So I never got the patch to work correctly. > > > > However, I circumvented the problem by making my own vector and calling the matfd-function, and then passing in the vector. This is essentially the same thing and I means I don't need an if-def for older versions. > > > > Gaetan > > > > > > > > On Tue, Jun 19, 2018 at 11:24 AM Smith, Barry F. wrote: > > > > The code to handle the null vector got mistakenly removed. Please find attached a patch that should resolve the problem. > > > > > > > > Please let us know if it works, > > > > Thanks > > > > Barry > > > > > > > On Jun 19, 2018, at 12:36 PM, Gaetan Kenway wrote: > > > > > > Seems like the MatMFFDSetBase is now broken in petsc 3.9 > > > > > > In 3.7 I used: > > > > > > call MatMFFDSetBase(dRdW, wVec, PETSC_NULL_OBJECT, ierr) > > > > > > and the most logical translation is now > > > > > > call MatMFFDSetBase(dRdW, wVec, PETSC_NULL_VEC, ierr) > > > > > > Unfortunately this just segfaults: > > > #0 VecAXPY (y=y at entry=0x10a3750, alpha=alpha at entry=-1, x=x at entry=0xffffffffffffffff) at /nobackup/gkenway/packages/petsc-3.9.2/src/vec/vec/interface/rvector.c:601 > > > #1 0x00007fffb697faae in MatMult_MFFD (mat=0x291a150, a=0x10b25c0, y=0x10a3750) at /nobackup/gkenway/packages/petsc-3.9.2/src/mat/impls/mffd/mffd.c:368 > > > #2 0x00007fffb6847af2 in MatMult (mat=0x291a150, x=x at entry=0x10b25c0, y=y at entry=0x10a3750) at /nobackup/gkenway/packages/petsc-3.9.2/src/mat/interface/matrix.c:2305 > > > #3 0x00007fffb6e982e7 in PCApplyBAorAB (pc=0xe64d00, side=PC_RIGHT, x=0x10aaee0, y=y at entry=0x10a3750, work=0x10b25c0) at /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/pc/interface/precon.c:680 > > > #4 0x00007fffb6f02b84 in KSP_PCApplyBAorAB (w=, y=0x10a3750, x=, ksp=0xf135e0) at /u/wk/gkenway/scratch/packages/petsc-3.9.2/include/petsc/private/kspimpl.h:304 > > > #5 KSPGMRESCycle (itcount=itcount at entry=0x7fffffffb73c, ksp=ksp at entry=0xf135e0) at /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/ksp/impls/gmres/gmres.c:152 > > > #6 0x00007fffb6f035c2 in KSPSolve_GMRES (ksp=0xf135e0) at /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/ksp/impls/gmres/gmres.c:234 > > > #7 0x00007fffb6f35050 in KSPSolve (ksp=0xf135e0, b=0xf7ced0, x=0xf77be0) at /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/ksp/interface/itfunc.c:669 > > > #8 0x00007fffb6f482ee in kspsolve_ (ksp=ksp at entry=0x7fffb461bae8 <__anksolver_MOD_ank_ksp>, b=, b at entry=0x7fffb461b9c8 <__anksolver_MOD_rvec>, x=, > > > x at entry=0x7fffb461ba18 <__anksolver_MOD_deltaw>, ierr=ierr at entry=0x7fffffffb938) at /nobackup/gkenway/packages/petsc-3.9.2/src/ksp/ksp/interface/ftn-custom/zitfuncf.c:54 > > > #9 0x00007fffb3d7e697 in anksolver::ankstep (firstcall=) at ../NKSolver/NKSolvers.F90:2891 > > > #10 0x00007fffb3ebbbb9 in solvers::solvestate () at ../solver/solvers.F90:1114 > > > #11 0x00007fffb3ebe285 in solvers::solver () at ../solver/solvers.F90:69 > > > #12 0x00007fffb3cfca17 in f2py_rout_libadflow_solvers_solver (capi_self=, capi_args=, capi_keywds=, f2py_func=0x7fffb3ebe0e0 ) > > > at libadflowmodule.c:6710 > > > #13 0x00007fffb3cf85b3 in fortran_call (fp=, arg=, kw=) at /u/wk/gkenway/.local/lib/python2.7/site-packages/numpy/f2py/src/fortranobject.c:404 > > > #14 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 > > > #15 0x00007ffff7aef236 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 > > > #16 0x00007ffff7af603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 > > > #17 0x00007ffff7a7f978 in function_call () from /lib64/libpython2.7.so.1.0 > > > #18 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 > > > #19 0x00007ffff7a69a55 in instancemethod_call () from /lib64/libpython2.7.so.1.0 > > > #20 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 > > > #21 0x00007ffff7ab1e27 in slot_tp_call () from /lib64/libpython2.7.so.1.0 > > > #22 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 > > > #23 0x00007ffff7aef236 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 > > > #24 0x00007ffff7af603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 > > > #25 0x00007ffff7af6142 in PyEval_EvalCode () from /lib64/libpython2.7.so.1.0 > > > #26 0x00007ffff7b0f57f in run_mod () from /lib64/libpython2.7.so.1.0 > > > #27 0x00007ffff7b1073e in PyRun_FileExFlags () from /lib64/libpython2.7.so.1.0 > > > #28 0x00007ffff7b119c9 in PyRun_SimpleFileExFlags () from /lib64/libpython2.7.so.1.0 > > > #29 0x00007ffff7b22b7f in Py_Main () from /lib64/libpython2.7.so.1.0 > > > #30 0x00007ffff6d3f3d5 in __libc_start_main () from /lib64/libc.so.6 > > > #31 0x000000000040066e in _start () > > > > > > Thanks, > > > > > > Gaetan > > > > > > > > > > > > > > > From nishantnangia329 at gmail.com Fri Jun 22 15:33:11 2018 From: nishantnangia329 at gmail.com (Nishant Nangia) Date: Fri, 22 Jun 2018 15:33:11 -0500 Subject: [petsc-users] Preconditioned residuals for FGMRES Message-ID: Hi, I am solving a saddle point system using a shell preconditioner (which itself uses Krylov solvers, hence the use of FGMRES). I had added the option to re-scale parts of the saddle point system to minimize loss of floating point precision for cases where there are varying orders of magnitude in the system/unknowns. I wanted to show that re-scaling can alleviate large differences between the preconditioned and unpreconditioned residual norms. However, I notice that FGMRES only supports right preconditioning, meaning the preconditioned residual is never formed/used (I think). Is there any way to form the preconditioned norm for FGMRES, or does it just not make sense in the context of right-preconditioned iterative solvers? Is there any way to show that the re-scaling is improving the solver convergence (i.e. showing that it ensures that the true and relative residual are close to each other)? *Nishant Nangia* Northwestern University Ph.D. Candidate | Engineering Sciences and Applied Mathematics Tech L386 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Jun 22 15:41:15 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 22 Jun 2018 20:41:15 +0000 Subject: [petsc-users] Preconditioned residuals for FGMRES In-Reply-To: References: Message-ID: <8BE0302F-6D4A-4A99-9137-3A092E63893F@anl.gov> > On Jun 22, 2018, at 3:33 PM, Nishant Nangia wrote: > > Hi, > > I am solving a saddle point system using a shell preconditioner (which itself uses Krylov solvers, hence the use of FGMRES). I had added the option to re-scale parts of the saddle point system to minimize loss of floating point precision for cases where there are varying orders of magnitude in the system/unknowns. > > I wanted to show that re-scaling can alleviate large differences between the preconditioned and unpreconditioned residual norms. However, I notice that FGMRES only supports right preconditioning, meaning the preconditioned residual is never formed/used (I think). Yes, FGMRES doesn't make sense with left preconditioning. > > Is there any way to form the preconditioned norm for FGMRES, or does it just not make sense in the context of right-preconditioned iterative solvers? It doesn't make sense for right preconditioning. > Is there any way to show that the re-scaling is improving the solver convergence (i.e. showing that it ensures that the true and relative residual are close to each other)? I don't think so. I think the only thing you could hope to show is that the rescaling requires fewer FGMRES iterations. Barry > > Nishant Nangia > Northwestern University > Ph.D. Candidate | Engineering Sciences and Applied Mathematics > Tech L386 From jed at jedbrown.org Fri Jun 22 15:46:32 2018 From: jed at jedbrown.org (Jed Brown) Date: Fri, 22 Jun 2018 14:46:32 -0600 Subject: [petsc-users] Preconditioned residuals for FGMRES In-Reply-To: References: Message-ID: <87in6a35xz.fsf@jedbrown.org> It's a very different algorithm, but FBCGS is a flexible method that supports preconditioned norm (though I don't how meaningful it is since the preconditioner is changing). Nishant Nangia writes: > Hi, > > I am solving a saddle point system using a shell preconditioner (which > itself uses Krylov solvers, hence the use of FGMRES). I had added the > option to re-scale parts of the saddle point system to minimize loss of > floating point precision for cases where there are varying orders of > magnitude in the system/unknowns. > > I wanted to show that re-scaling can alleviate large differences between > the preconditioned and unpreconditioned residual norms. However, I notice > that FGMRES only supports right preconditioning, meaning the preconditioned > residual is never formed/used (I think). > > Is there any way to form the preconditioned norm for FGMRES, or does it > just not make sense in the context of right-preconditioned iterative > solvers? Is there any way to show that the re-scaling is improving the > solver convergence (i.e. showing that it ensures that the true and relative > residual are close to each other)? > > *Nishant Nangia* > Northwestern University > Ph.D. Candidate | Engineering Sciences and Applied Mathematics > Tech L386 From griffith at cims.nyu.edu Fri Jun 22 15:47:42 2018 From: griffith at cims.nyu.edu (Boyce Griffith) Date: Fri, 22 Jun 2018 16:47:42 -0400 Subject: [petsc-users] Preconditioned residuals for FGMRES In-Reply-To: References: Message-ID: Can you set up the preconditioner so that you can just use GMRES? > On Jun 22, 2018, at 4:33 PM, Nishant Nangia wrote: > > Hi, > > I am solving a saddle point system using a shell preconditioner (which itself uses Krylov solvers, hence the use of FGMRES). I had added the option to re-scale parts of the saddle point system to minimize loss of floating point precision for cases where there are varying orders of magnitude in the system/unknowns. > > I wanted to show that re-scaling can alleviate large differences between the preconditioned and unpreconditioned residual norms. However, I notice that FGMRES only supports right preconditioning, meaning the preconditioned residual is never formed/used (I think). > > Is there any way to form the preconditioned norm for FGMRES, or does it just not make sense in the context of right-preconditioned iterative solvers? Is there any way to show that the re-scaling is improving the solver convergence (i.e. showing that it ensures that the true and relative residual are close to each other)? > > Nishant Nangia > Northwestern University > Ph.D. Candidate | Engineering Sciences and Applied Mathematics > Tech L386 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jun 22 15:49:18 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 22 Jun 2018 13:49:18 -0700 Subject: [petsc-users] Preconditioned residuals for FGMRES In-Reply-To: References: Message-ID: On Fri, Jun 22, 2018 at 1:47 PM Boyce Griffith wrote: > Can you set up the preconditioner so that you can just use GMRES? > So I think what Boyce is saying is, can't you fix the number of iterates in the inner Krylov solvers so that it becomes a linear operator and you can use GMRES? Thanks, Matt > On Jun 22, 2018, at 4:33 PM, Nishant Nangia > wrote: > > Hi, > > I am solving a saddle point system using a shell preconditioner (which > itself uses Krylov solvers, hence the use of FGMRES). I had added the > option to re-scale parts of the saddle point system to minimize loss of > floating point precision for cases where there are varying orders of > magnitude in the system/unknowns. > > I wanted to show that re-scaling can alleviate large differences between > the preconditioned and unpreconditioned residual norms. However, I notice > that FGMRES only supports right preconditioning, meaning the preconditioned > residual is never formed/used (I think). > > Is there any way to form the preconditioned norm for FGMRES, or does it > just not make sense in the context of right-preconditioned iterative > solvers? Is there any way to show that the re-scaling is improving the > solver convergence (i.e. showing that it ensures that the true and relative > residual are close to each other)? > > *Nishant Nangia* > Northwestern University > Ph.D. Candidate | Engineering Sciences and Applied Mathematics > Tech L386 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From apbhalla at lbl.gov Fri Jun 22 15:51:23 2018 From: apbhalla at lbl.gov (Amneet Pal Bhalla) Date: Fri, 22 Jun 2018 13:51:23 -0700 Subject: [petsc-users] Preconditioned residuals for FGMRES In-Reply-To: References: Message-ID: <26DA9F86-20C1-4000-8794-158CC0F34CAB@lbl.gov> Like using the Krylov preconditioner as a Richardson KSP solver with max_its = N? > On Jun 22, 2018, at 1:49 PM, Matthew Knepley wrote: > > On Fri, Jun 22, 2018 at 1:47 PM Boyce Griffith > wrote: > Can you set up the preconditioner so that you can just use GMRES? > > So I think what Boyce is saying is, can't you fix the number of iterates in the inner Krylov solvers so that it becomes a linear > operator and you can use GMRES? > > Thanks, > > Matt > > On Jun 22, 2018, at 4:33 PM, Nishant Nangia > wrote: > >> Hi, >> >> I am solving a saddle point system using a shell preconditioner (which itself uses Krylov solvers, hence the use of FGMRES). I had added the option to re-scale parts of the saddle point system to minimize loss of floating point precision for cases where there are varying orders of magnitude in the system/unknowns. >> >> I wanted to show that re-scaling can alleviate large differences between the preconditioned and unpreconditioned residual norms. However, I notice that FGMRES only supports right preconditioning, meaning the preconditioned residual is never formed/used (I think). >> >> Is there any way to form the preconditioned norm for FGMRES, or does it just not make sense in the context of right-preconditioned iterative solvers? Is there any way to show that the re-scaling is improving the solver convergence (i.e. showing that it ensures that the true and relative residual are close to each other)? >> >> Nishant Nangia >> Northwestern University >> Ph.D. Candidate | Engineering Sciences and Applied Mathematics >> Tech L386 > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From griffith at cims.nyu.edu Fri Jun 22 15:52:30 2018 From: griffith at cims.nyu.edu (Boyce Griffith) Date: Fri, 22 Jun 2018 16:52:30 -0400 Subject: [petsc-users] Preconditioned residuals for FGMRES In-Reply-To: References: Message-ID: > On Jun 22, 2018, at 4:49 PM, Matthew Knepley wrote: > >> On Fri, Jun 22, 2018 at 1:47 PM Boyce Griffith wrote: >> Can you set up the preconditioner so that you can just use GMRES? > > So I think what Boyce is saying is, can't you fix the number of iterates in the inner Krylov solvers so that it becomes a linear > operator and you can use GMRES? Yes that is it. I think we default to FGMRES for robustness, because the preconditioner can easily be configured to be nonlinear, but unless you have made some big changes to the algorithm that I think you are using, I think it should be possible to set it up as a stationary linear operator. > Thanks, > > Matt > >>> On Jun 22, 2018, at 4:33 PM, Nishant Nangia wrote: >>> >>> Hi, >>> >>> I am solving a saddle point system using a shell preconditioner (which itself uses Krylov solvers, hence the use of FGMRES). I had added the option to re-scale parts of the saddle point system to minimize loss of floating point precision for cases where there are varying orders of magnitude in the system/unknowns. >>> >>> I wanted to show that re-scaling can alleviate large differences between the preconditioned and unpreconditioned residual norms. However, I notice that FGMRES only supports right preconditioning, meaning the preconditioned residual is never formed/used (I think). >>> >>> Is there any way to form the preconditioned norm for FGMRES, or does it just not make sense in the context of right-preconditioned iterative solvers? Is there any way to show that the re-scaling is improving the solver convergence (i.e. showing that it ensures that the true and relative residual are close to each other)? >>> >>> Nishant Nangia >>> Northwestern University >>> Ph.D. Candidate | Engineering Sciences and Applied Mathematics >>> Tech L386 > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Jun 22 16:02:13 2018 From: jed at jedbrown.org (Jed Brown) Date: Fri, 22 Jun 2018 15:02:13 -0600 Subject: [petsc-users] Preconditioned residuals for FGMRES In-Reply-To: References: Message-ID: <87efgy357u.fsf@jedbrown.org> Matthew Knepley writes: > On Fri, Jun 22, 2018 at 1:47 PM Boyce Griffith > wrote: > >> Can you set up the preconditioner so that you can just use GMRES? >> > > So I think what Boyce is saying is, can't you fix the number of iterates in > the inner Krylov solvers so that it becomes a linear > operator and you can use GMRES? A fixed number of Krylov iterations is not a linear operation. A fixed number of Chebyshev iterations (or other method that applies a constant polynomial) is linear. From nahmad16 at ku.edu.tr Sat Jun 23 06:36:59 2018 From: nahmad16 at ku.edu.tr (Najeeb Ahmad) Date: Sat, 23 Jun 2018 16:36:59 +0500 Subject: [petsc-users] PETSc Solver: Large iteration count for first solution Message-ID: Hi, I am trying to solve a 64008000 x 64008000 system using PETSc. The problem I am facing is that I am getting a very large iteration count for the first solution (order of thousands). For the subsequent soluitons, the iteration count is less. For instance, here is the iteration count for the first solution when my code is run on 32 processes. ... 2772 KSP Residual norm 1.198554448053e-07 2773 KSP Residual norm 1.190214006391e-07 2774 KSP Residual norm 1.181931531360e-07 2775 KSP Residual norm 1.173706620648e-07 2776 KSP Residual norm 1.165538874733e-07 2777 KSP Residual norm 1.157427896857e-07 2778 KSP Residual norm 1.149373293013e-07 2779 KSP Residual norm 1.141374671925e-07 2780 KSP Residual norm 1.133431645025e-07 Linear solve converged due to CONVERGED_RTOL iterations 2780 For the subsequent solutions, the iteration count is as follows: ... 116 KSP Residual norm 2.657138978992e-04 117 KSP Residual norm 2.767485246885e-04 118 KSP Residual norm 2.071574619161e-04 119 KSP Residual norm 2.273562103015e-04 120 KSP Residual norm 1.693847273193e-04 121 KSP Residual norm 1.803833437167e-04 122 KSP Residual norm 1.443434809819e-04 123 KSP Residual norm 1.390878278231e-04 124 KSP Residual norm 1.247521972621e-04 125 KSP Residual norm 1.059224678494e-04 Linear solve converged due to CONVERGED_RTOL iterations 125 The solver setup for my code is as follows: ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr); ierr = KSPSetOperators(ksp,A,A);CHKERRQ(ierr); ierr = KSPSetType(ksp, KSPCG);CHKERRQ(ierr); ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); ierr = PCSetType(pc, PCJACOBI); ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr); ierr = KSPSetUp(ksp);CHKERRQ(ierr); and the command line for running the code is: mpiexec -n 32 ./main -mat_type mpiaij -ksp_atol 1E-7 -ksp_rtol 1E-7 -ksp_converged_reason -ksp_monitor Any help is highly appreciated. -- *Najeeb Ahmad* *Research and Teaching Assistant* *PARallel and MultiCORE Computing Laboratory (ParCoreLab) * *Computer Science and Engineering* *Ko? University, Istanbul, Turkey* -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Jun 23 09:09:19 2018 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 23 Jun 2018 07:09:19 -0700 Subject: [petsc-users] PETSc Solver: Large iteration count for first solution In-Reply-To: References: Message-ID: On Sat, Jun 23, 2018 at 4:37 AM Najeeb Ahmad wrote: > Hi, > > I am trying to solve a 64008000 x 64008000 system using PETSc. The > problem I am facing is that I am getting a very large iteration count for > the first solution (order of thousands). For the subsequent soluitons, the > iteration count is less. For instance, here is the iteration count for the > first solution when my code is run on 32 processes. > > ... > 2772 KSP Residual norm 1.198554448053e-07 > 2773 KSP Residual norm 1.190214006391e-07 > 2774 KSP Residual norm 1.181931531360e-07 > 2775 KSP Residual norm 1.173706620648e-07 > 2776 KSP Residual norm 1.165538874733e-07 > 2777 KSP Residual norm 1.157427896857e-07 > 2778 KSP Residual norm 1.149373293013e-07 > 2779 KSP Residual norm 1.141374671925e-07 > 2780 KSP Residual norm 1.133431645025e-07 > Linear solve converged due to CONVERGED_RTOL iterations 2780 > > For the subsequent solutions, the iteration count is as follows: > ... > 116 KSP Residual norm 2.657138978992e-04 > 117 KSP Residual norm 2.767485246885e-04 > 118 KSP Residual norm 2.071574619161e-04 > 119 KSP Residual norm 2.273562103015e-04 > 120 KSP Residual norm 1.693847273193e-04 > 121 KSP Residual norm 1.803833437167e-04 > 122 KSP Residual norm 1.443434809819e-04 > 123 KSP Residual norm 1.390878278231e-04 > 124 KSP Residual norm 1.247521972621e-04 > 125 KSP Residual norm 1.059224678494e-04 > Linear solve converged due to CONVERGED_RTOL iterations 125 > > The solver setup for my code is as follows: > > ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr); > ierr = KSPSetOperators(ksp,A,A);CHKERRQ(ierr); > ierr = KSPSetType(ksp, KSPCG);CHKERRQ(ierr); > ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); > ierr = PCSetType(pc, PCJACOBI); > ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr); > ierr = KSPSetUp(ksp);CHKERRQ(ierr); > > and the command line for running the code is: > mpiexec -n 32 ./main -mat_type mpiaij -ksp_atol 1E-7 -ksp_rtol 1E-7 > -ksp_converged_reason -ksp_monitor > What equation are you trying to solve. CG/Jacobi is a scalable solver for the mass matrix (identity), but not much else. Thanks, Matt > Any help is highly appreciated. > > > -- > *Najeeb Ahmad* > > > *Research and Teaching Assistant* > *PARallel and MultiCORE Computing Laboratory (ParCoreLab) * > > *Computer Science and Engineering* > *Ko? University, Istanbul, Turkey* > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ellen.price at cfa.harvard.edu Sat Jun 23 19:51:52 2018 From: ellen.price at cfa.harvard.edu (Ellen M. Price) Date: Sat, 23 Jun 2018 20:51:52 -0400 Subject: [petsc-users] Errors encountered when linking against a successful build including MUMPS Message-ID: <27212c88-ce09-18aa-ba28-6a5e74ecf9dc@cfa.harvard.edu> I just successfully (re)-compiled petsc-3.9.2 to use my MUMPS install. The PETSc build process completed with no errors and I installed my new shared libs. However, when I tried linking against the new libraries, I get the following errors: /home/eprice/software/petsc-opt/lib/libpetsc.so: undefined reference to `MatMumpsSetIcntl' /home/eprice/software/petsc-opt/lib/libpetsc.so: undefined reference to `MatPartitioningParmetisSetRepartition' /home/eprice/software/petsc-opt/lib/libpetsc.so: undefined reference to `MatSolverTypeRegister_MUMPS' /home/eprice/software/petsc-opt/lib/libpetsc.so: undefined reference to `MatPartitioningCreate_Parmetis' /home/eprice/software/petsc-opt/lib/libpetsc.so: undefined reference to `MatPartitioningCreate_PTScotch' collect2: error: ld returned 1 exit status This is confusing, because I can't see any reason these functions shouldn't be included in the final build. My configure command follows: ./configure --prefix=/home/eprice/software/petsc-opt --with-hdf5=1 --with-hdf5-dir=/home/eprice/software/hdf5-parallel --with-mpe=1 --with-mpe-dir=/home/eprice/software/mpe --with-debugging=0 LDFLAGS='-pthread -lz' COPTFLAGS='-O3 -march=native -mtune=native' CXXOPTFLAGS='-O3 -march=native -mtune=native' FOPTFLAGS='-O3 -march=native -mtune=native' --with-mpi=1 --with-mpi-dir=/home/eprice/software/mpich --with-mumps=1 --with-mumps-dir=/home/eprice/software/mumps --with-parmetis=1 --with-parmetis-dir=/home/eprice/software/parmetis --with-metis=1 --with-metis-dir=/home/eprice/software/parmetis --with-ptscotch=1 --with-ptscotch-dir=/home/eprice/software/scotch --with-scalapack=1 --with-scalapack-dir=/home/eprice/software/scalapack Any help is greatly appreciated. I tried picking through the build system to find a problem, but it's difficult to understand without docs. Ellen From bsmith at mcs.anl.gov Sat Jun 23 19:57:01 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Sun, 24 Jun 2018 00:57:01 +0000 Subject: [petsc-users] Errors encountered when linking against a successful build including MUMPS In-Reply-To: <27212c88-ce09-18aa-ba28-6a5e74ecf9dc@cfa.harvard.edu> References: <27212c88-ce09-18aa-ba28-6a5e74ecf9dc@cfa.harvard.edu> Message-ID: <4B398F9D-3487-40BC-A840-CA7C491A4ACE@anl.gov> Ellen, Send configure.log and make.log to petsc-maint at mcs.anl.gov Barry On Jun 23, 2018, at 7:51 PM, Ellen M. Price wrote: > > I just successfully (re)-compiled petsc-3.9.2 to use my MUMPS install. > The PETSc build process completed with no errors and I installed my new > shared libs. However, when I tried linking against the new libraries, I > get the following errors: > > /home/eprice/software/petsc-opt/lib/libpetsc.so: undefined reference to > `MatMumpsSetIcntl' > /home/eprice/software/petsc-opt/lib/libpetsc.so: undefined reference to > `MatPartitioningParmetisSetRepartition' > /home/eprice/software/petsc-opt/lib/libpetsc.so: undefined reference to > `MatSolverTypeRegister_MUMPS' > /home/eprice/software/petsc-opt/lib/libpetsc.so: undefined reference to > `MatPartitioningCreate_Parmetis' > /home/eprice/software/petsc-opt/lib/libpetsc.so: undefined reference to > `MatPartitioningCreate_PTScotch' > collect2: error: ld returned 1 exit status > > This is confusing, because I can't see any reason these functions > shouldn't be included in the final build. My configure command follows: > > ./configure --prefix=/home/eprice/software/petsc-opt --with-hdf5=1 > --with-hdf5-dir=/home/eprice/software/hdf5-parallel --with-mpe=1 > --with-mpe-dir=/home/eprice/software/mpe --with-debugging=0 > LDFLAGS='-pthread -lz' COPTFLAGS='-O3 -march=native -mtune=native' > CXXOPTFLAGS='-O3 -march=native -mtune=native' FOPTFLAGS='-O3 > -march=native -mtune=native' --with-mpi=1 > --with-mpi-dir=/home/eprice/software/mpich --with-mumps=1 > --with-mumps-dir=/home/eprice/software/mumps --with-parmetis=1 > --with-parmetis-dir=/home/eprice/software/parmetis --with-metis=1 > --with-metis-dir=/home/eprice/software/parmetis --with-ptscotch=1 > --with-ptscotch-dir=/home/eprice/software/scotch --with-scalapack=1 > --with-scalapack-dir=/home/eprice/software/scalapack > > Any help is greatly appreciated. I tried picking through the build > system to find a problem, but it's difficult to understand without docs. > > Ellen From leoni.massimiliano1 at gmail.com Sun Jun 24 05:26:40 2018 From: leoni.massimiliano1 at gmail.com (Massimiliano Leoni) Date: Sun, 24 Jun 2018 12:26:40 +0200 Subject: [petsc-users] PETSc Solver: Large iteration count for first solution In-Reply-To: References: Message-ID: <2027993.k9ILWqt9yH@debianxps> Hi, are the various solves a time stepping procedure? Could it be that it is re-using the solution at a previous iteration as an initial guess for the subsequent ones, and for this reason it converges sooner? Massimiliano On sabato 23 giugno 2018 13:36:59 CEST Najeeb Ahmad wrote: Hi, I am trying to solve a 64008000 x 64008000 system using PETSc. The problem I am facing is that I am getting a very large iteration count for the first solution (order of thousands). For the subsequent soluitons, the iteration count is less. For instance, here is the iteration count for the first solution when my code is run on 32 processes. ... 2772 KSP Residual norm 1.198554448053e-07 2773 KSP Residual norm 1.190214006391e-07 2774 KSP Residual norm 1.181931531360e-07 2775 KSP Residual norm 1.173706620648e-07 2776 KSP Residual norm 1.165538874733e-07 2777 KSP Residual norm 1.157427896857e-07 2778 KSP Residual norm 1.149373293013e-07 2779 KSP Residual norm 1.141374671925e-07 2780 KSP Residual norm 1.133431645025e-07 Linear solve converged due to CONVERGED_RTOL iterations 2780 For the subsequent solutions, the iteration count is as follows: ... 116 KSP Residual norm 2.657138978992e-04 117 KSP Residual norm 2.767485246885e-04 118 KSP Residual norm 2.071574619161e-04 119 KSP Residual norm 2.273562103015e-04 120 KSP Residual norm 1.693847273193e-04 121 KSP Residual norm 1.803833437167e-04 122 KSP Residual norm 1.443434809819e-04 123 KSP Residual norm 1.390878278231e-04 124 KSP Residual norm 1.247521972621e-04 125 KSP Residual norm 1.059224678494e-04 Linear solve converged due to CONVERGED_RTOL iterations 125 The solver setup for my code is as follows: ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr); ierr = KSPSetOperators(ksp,A,A);CHKERRQ(ierr); ierr = KSPSetType(ksp, KSPCG);CHKERRQ(ierr); ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); ierr = PCSetType(pc, PCJACOBI); ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr); ierr = KSPSetUp(ksp);CHKERRQ(ierr); and the command line for running the code is: mpiexec -n 32 ./main -mat_type mpiaij -ksp_atol 1E-7 -ksp_rtol 1E-7 -ksp_converged_reason -ksp_monitor Any help is highly appreciated. /*Najeeb Ahmad*/ *Research and Teaching Assistant PARallel and MultiCORE Computing Laboratory (ParCoreLab) * *Computer Science and Engineering* *Ko? University, Istanbul, Turkey* -------------- next part -------------- An HTML attachment was scrubbed... URL: From nahmad16 at ku.edu.tr Sun Jun 24 05:36:54 2018 From: nahmad16 at ku.edu.tr (Najeeb Ahmad) Date: Sun, 24 Jun 2018 15:36:54 +0500 Subject: [petsc-users] PETSc Solver: Large iteration count for first solution In-Reply-To: <2027993.k9ILWqt9yH@debianxps> References: <2027993.k9ILWqt9yH@debianxps> Message-ID: Hi Massimiliano, Thanks for your feedback. Indeed i am using previous solution for the subsequent ones and probably that is the reason for the faster convergence for the subsequent iterations. As Matt suggested, i need to improve my preconditioning to make first iteration converge faster. Thanks On Sun, Jun 24, 2018 at 3:26 PM Massimiliano Leoni < leoni.massimiliano1 at gmail.com> wrote: > Hi, are the various solves a time stepping procedure? Could it be that it > is re-using the solution at a previous iteration as an initial guess for > the subsequent ones, and for this reason it converges sooner? > > > > Massimiliano > > > > On sabato 23 giugno 2018 13:36:59 CEST Najeeb Ahmad wrote: > > Hi, > > > I am trying to solve a 64008000 x 64008000 system using PETSc. The > problem I am facing is that I am getting a very large iteration count for > the first solution (order of thousands). For the subsequent soluitons, the > iteration count is less. For instance, here is the iteration count for the > first solution when my code is run on 32 processes. > > > ... > > 2772 KSP Residual norm 1.198554448053e-07 > > 2773 KSP Residual norm 1.190214006391e-07 > > 2774 KSP Residual norm 1.181931531360e-07 > > 2775 KSP Residual norm 1.173706620648e-07 > > 2776 KSP Residual norm 1.165538874733e-07 > > 2777 KSP Residual norm 1.157427896857e-07 > > 2778 KSP Residual norm 1.149373293013e-07 > > 2779 KSP Residual norm 1.141374671925e-07 > > 2780 KSP Residual norm 1.133431645025e-07 > > Linear solve converged due to CONVERGED_RTOL iterations 2780 > > > For the subsequent solutions, the iteration count is as follows: > > ... > > 116 KSP Residual norm 2.657138978992e-04 > > 117 KSP Residual norm 2.767485246885e-04 > > 118 KSP Residual norm 2.071574619161e-04 > > 119 KSP Residual norm 2.273562103015e-04 > > 120 KSP Residual norm 1.693847273193e-04 > > 121 KSP Residual norm 1.803833437167e-04 > > 122 KSP Residual norm 1.443434809819e-04 > > 123 KSP Residual norm 1.390878278231e-04 > > 124 KSP Residual norm 1.247521972621e-04 > > 125 KSP Residual norm 1.059224678494e-04 > > Linear solve converged due to CONVERGED_RTOL iterations 125 > > > The solver setup for my code is as follows: > > > ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr); > > ierr = KSPSetOperators(ksp,A,A);CHKERRQ(ierr); > > ierr = KSPSetType(ksp, KSPCG);CHKERRQ(ierr); > > ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); > > ierr = PCSetType(pc, PCJACOBI); > > ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr); > > ierr = KSPSetUp(ksp);CHKERRQ(ierr); > > > and the command line for running the code is: > > mpiexec -n 32 ./main -mat_type mpiaij -ksp_atol 1E-7 -ksp_rtol 1E-7 > -ksp_converged_reason -ksp_monitor > > > Any help is highly appreciated. > > > > -- > > Najeeb Ahmad > > > Research and Teaching Assistant > > PARallel and MultiCORE Computing Laboratory (ParCoreLab) > > Computer Science and Engineering > > > Ko? University, Istanbul, Turkey > > > > > -- *Najeeb Ahmad* *Research and Teaching Assistant* *PARallel and MultiCORE Computing Laboratory (ParCoreLab) * *Computer Science and Engineering* *Ko? University, Istanbul, Turkey* -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Sun Jun 24 10:45:31 2018 From: mfadams at lbl.gov (Mark Adams) Date: Sun, 24 Jun 2018 11:45:31 -0400 Subject: [petsc-users] PETSc Solver: Large iteration count for first solution In-Reply-To: References: <2027993.k9ILWqt9yH@debianxps> Message-ID: Note, you converged because of the relative tolerance so initial guess does not really matter except that the initial residual on the second time step may have more high frequency content then your first initial residual. On Sun, Jun 24, 2018 at 6:37 AM Najeeb Ahmad wrote: > Hi Massimiliano, > > Thanks for your feedback. Indeed i am using previous solution for the > subsequent ones and probably that is the reason for the faster convergence > for the subsequent iterations. As Matt suggested, i need to improve my > preconditioning to make first iteration converge faster. > > Thanks > > On Sun, Jun 24, 2018 at 3:26 PM Massimiliano Leoni < > leoni.massimiliano1 at gmail.com> wrote: > >> Hi, are the various solves a time stepping procedure? Could it be that it >> is re-using the solution at a previous iteration as an initial guess for >> the subsequent ones, and for this reason it converges sooner? >> >> >> >> Massimiliano >> >> >> >> On sabato 23 giugno 2018 13:36:59 CEST Najeeb Ahmad wrote: >> >> Hi, >> >> >> I am trying to solve a 64008000 x 64008000 system using PETSc. The >> problem I am facing is that I am getting a very large iteration count for >> the first solution (order of thousands). For the subsequent soluitons, the >> iteration count is less. For instance, here is the iteration count for the >> first solution when my code is run on 32 processes. >> >> >> ... >> >> 2772 KSP Residual norm 1.198554448053e-07 >> >> 2773 KSP Residual norm 1.190214006391e-07 >> >> 2774 KSP Residual norm 1.181931531360e-07 >> >> 2775 KSP Residual norm 1.173706620648e-07 >> >> 2776 KSP Residual norm 1.165538874733e-07 >> >> 2777 KSP Residual norm 1.157427896857e-07 >> >> 2778 KSP Residual norm 1.149373293013e-07 >> >> 2779 KSP Residual norm 1.141374671925e-07 >> >> 2780 KSP Residual norm 1.133431645025e-07 >> >> Linear solve converged due to CONVERGED_RTOL iterations 2780 >> >> >> For the subsequent solutions, the iteration count is as follows: >> >> ... >> >> 116 KSP Residual norm 2.657138978992e-04 >> >> 117 KSP Residual norm 2.767485246885e-04 >> >> 118 KSP Residual norm 2.071574619161e-04 >> >> 119 KSP Residual norm 2.273562103015e-04 >> >> 120 KSP Residual norm 1.693847273193e-04 >> >> 121 KSP Residual norm 1.803833437167e-04 >> >> 122 KSP Residual norm 1.443434809819e-04 >> >> 123 KSP Residual norm 1.390878278231e-04 >> >> 124 KSP Residual norm 1.247521972621e-04 >> >> 125 KSP Residual norm 1.059224678494e-04 >> >> Linear solve converged due to CONVERGED_RTOL iterations 125 >> >> >> The solver setup for my code is as follows: >> >> >> ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr); >> >> ierr = KSPSetOperators(ksp,A,A);CHKERRQ(ierr); >> >> ierr = KSPSetType(ksp, KSPCG);CHKERRQ(ierr); >> >> ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); >> >> ierr = PCSetType(pc, PCJACOBI); >> >> ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr); >> >> ierr = KSPSetUp(ksp);CHKERRQ(ierr); >> >> >> and the command line for running the code is: >> >> mpiexec -n 32 ./main -mat_type mpiaij -ksp_atol 1E-7 -ksp_rtol 1E-7 >> -ksp_converged_reason -ksp_monitor >> >> >> Any help is highly appreciated. >> >> >> >> -- >> >> Najeeb Ahmad >> >> >> Research and Teaching Assistant >> >> PARallel and MultiCORE Computing Laboratory (ParCoreLab) >> >> Computer Science and Engineering >> >> >> Ko? University, Istanbul, Turkey >> >> >> >> >> > > -- > *Najeeb Ahmad* > > > *Research and Teaching Assistant* > *PARallel and MultiCORE Computing Laboratory (ParCoreLab) * > > *Computer Science and Engineering* > *Ko? University, Istanbul, Turkey* > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nahmad16 at ku.edu.tr Sun Jun 24 14:26:47 2018 From: nahmad16 at ku.edu.tr (Najeeb Ahmad) Date: Mon, 25 Jun 2018 00:26:47 +0500 Subject: [petsc-users] PETSc Solver: Large iteration count for first solution In-Reply-To: References: <2027993.k9ILWqt9yH@debianxps> Message-ID: Very good point. Thank you for the clarification. On Sun, Jun 24, 2018 at 8:46 PM Mark Adams wrote: > Note, you converged because of the relative tolerance so initial guess > does not really matter except that the initial residual on the second time > step may have more high frequency content then your first initial residual. > > On Sun, Jun 24, 2018 at 6:37 AM Najeeb Ahmad wrote: > >> Hi Massimiliano, >> >> Thanks for your feedback. Indeed i am using previous solution for the >> subsequent ones and probably that is the reason for the faster convergence >> for the subsequent iterations. As Matt suggested, i need to improve my >> preconditioning to make first iteration converge faster. >> >> Thanks >> >> On Sun, Jun 24, 2018 at 3:26 PM Massimiliano Leoni < >> leoni.massimiliano1 at gmail.com> wrote: >> >>> Hi, are the various solves a time stepping procedure? Could it be that >>> it is re-using the solution at a previous iteration as an initial guess for >>> the subsequent ones, and for this reason it converges sooner? >>> >>> >>> >>> Massimiliano >>> >>> >>> >>> On sabato 23 giugno 2018 13:36:59 CEST Najeeb Ahmad wrote: >>> >>> Hi, >>> >>> >>> I am trying to solve a 64008000 x 64008000 system using PETSc. The >>> problem I am facing is that I am getting a very large iteration count for >>> the first solution (order of thousands). For the subsequent soluitons, the >>> iteration count is less. For instance, here is the iteration count for the >>> first solution when my code is run on 32 processes. >>> >>> >>> ... >>> >>> 2772 KSP Residual norm 1.198554448053e-07 >>> >>> 2773 KSP Residual norm 1.190214006391e-07 >>> >>> 2774 KSP Residual norm 1.181931531360e-07 >>> >>> 2775 KSP Residual norm 1.173706620648e-07 >>> >>> 2776 KSP Residual norm 1.165538874733e-07 >>> >>> 2777 KSP Residual norm 1.157427896857e-07 >>> >>> 2778 KSP Residual norm 1.149373293013e-07 >>> >>> 2779 KSP Residual norm 1.141374671925e-07 >>> >>> 2780 KSP Residual norm 1.133431645025e-07 >>> >>> Linear solve converged due to CONVERGED_RTOL iterations 2780 >>> >>> >>> For the subsequent solutions, the iteration count is as follows: >>> >>> ... >>> >>> 116 KSP Residual norm 2.657138978992e-04 >>> >>> 117 KSP Residual norm 2.767485246885e-04 >>> >>> 118 KSP Residual norm 2.071574619161e-04 >>> >>> 119 KSP Residual norm 2.273562103015e-04 >>> >>> 120 KSP Residual norm 1.693847273193e-04 >>> >>> 121 KSP Residual norm 1.803833437167e-04 >>> >>> 122 KSP Residual norm 1.443434809819e-04 >>> >>> 123 KSP Residual norm 1.390878278231e-04 >>> >>> 124 KSP Residual norm 1.247521972621e-04 >>> >>> 125 KSP Residual norm 1.059224678494e-04 >>> >>> Linear solve converged due to CONVERGED_RTOL iterations 125 >>> >>> >>> The solver setup for my code is as follows: >>> >>> >>> ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr); >>> >>> ierr = KSPSetOperators(ksp,A,A);CHKERRQ(ierr); >>> >>> ierr = KSPSetType(ksp, KSPCG);CHKERRQ(ierr); >>> >>> ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); >>> >>> ierr = PCSetType(pc, PCJACOBI); >>> >>> ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr); >>> >>> ierr = KSPSetUp(ksp);CHKERRQ(ierr); >>> >>> >>> and the command line for running the code is: >>> >>> mpiexec -n 32 ./main -mat_type mpiaij -ksp_atol 1E-7 -ksp_rtol 1E-7 >>> -ksp_converged_reason -ksp_monitor >>> >>> >>> Any help is highly appreciated. >>> >>> >>> >>> -- >>> >>> Najeeb Ahmad >>> >>> >>> Research and Teaching Assistant >>> >>> PARallel and MultiCORE Computing Laboratory (ParCoreLab) >>> >>> Computer Science and Engineering >>> >>> >>> Ko? University, Istanbul, Turkey >>> >>> >>> >>> >>> >> >> -- >> *Najeeb Ahmad* >> >> >> *Research and Teaching Assistant* >> *PARallel and MultiCORE Computing Laboratory (ParCoreLab) * >> >> *Computer Science and Engineering* >> *Ko? University, Istanbul, Turkey* >> >> -- *Najeeb Ahmad* *Research and Teaching Assistant* *PARallel and MultiCORE Computing Laboratory (ParCoreLab) * *Computer Science and Engineering* *Ko? University, Istanbul, Turkey* -------------- next part -------------- An HTML attachment was scrubbed... URL: From mbuerkle at web.de Sun Jun 24 23:54:07 2018 From: mbuerkle at web.de (Marius Buerkle) Date: Mon, 25 Jun 2018 06:54:07 +0200 Subject: [petsc-users] MAT_NEW_NONZERO_LOCATIONS working? In-Reply-To: References: <1153922A-8876-492B-8AD9-F41DD718FD21@anl.gov> <58D5E959-0E3C-4EF2-8270-882D1EA6EA38@mcs.anl.gov> Message-ID: An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Mon Jun 25 09:25:21 2018 From: hzhang at mcs.anl.gov (Hong) Date: Mon, 25 Jun 2018 09:25:21 -0500 Subject: [petsc-users] MAT_NEW_NONZERO_LOCATIONS working? In-Reply-To: References: <1153922A-8876-492B-8AD9-F41DD718FD21@anl.gov> <58D5E959-0E3C-4EF2-8270-882D1EA6EA38@mcs.anl.gov> Message-ID: Marius: > Thanks a lot, I did not test it thoroughly but it seems to work well and > really helpful. I have one question, is it necessary to do the MatTranspose > step as in ex214.c, I thought this is handled internally by petsc? > MUMPS requires sparse compressed COLUMN format in the host for rhs matrix, which is not supported by PETSc. MatTranspose() is expensive for large size matrices. We added a new petsc function MatMatTransposeSolve(): Solves A X = B^T, A: factored matrix (see petsc/src/mat/examples/tests/ex214c) With this function, user does not need to call MatTranspose, but needs to create B^T in AIJ matrix with all rows and columns stored in processor[0]. Let us know if any of you have better idea to handle it. The easiest way for us is MUMPS supports sparse compressed row format for rhs matrix. The branch hzhang/mumps-spRHS will be merged to petsc-master soon. Hong > > > Marius: > I added support for parallel sparse RHS (in host) for MatMatSolve_MUMPS() > https://bitbucket.org/petsc/petsc/commits/2b691707dd0cf456c808def006e14b > 6f56b364b6 > > It is in the branch hzhang/mumps-spRHS. > You may test it. > > I'll further cleanup the routine, test it, then merge it to petsc. > Hong > > On Mon, Jun 4, 2018 at 9:54 AM, Hong wrote: >> >> >> >> On Mon, Jun 4, 2018 at 1:03 PM, Jean-Yves LExcellent < >> Jean-Yves.L.Excellent at ens-lyon.fr> wrote: >>> >>> >>> Thanks for the details of your needs. >>> >>> For the first application, the sparse RHS feature with distributed >>> solution should effectively be fine. >>> >> I'll add parallel support of this feature in petsc after I'm back the ANL >> after June 14th. >> Hong >> >>> >>> For the second one, a future distributed RHS feature (not currently >>> available in MUMPS) might help if the centralized sparse RHS is too >>> memory consuming, depending on the size of B1. >>> >>> Regards, >>> Jean-Yves, for the MUMPS developers >>> >>> >>> >>> I want to invert a rather large sparse matrix for this using a sparse >>> rhs with centralized input would be ok as long as the solution is >>> distributed. >>> >>> and the second application I have in mind is solving a system of the >>> from AX=B where A and B are sparse and B is given by a block matrix of the >>> form B=[B1 0, 0 0] where B1 is dense but the dimension is (much) smaller >>> than that of the whole matrix B. >>> >>> Marius: >>> Current PETSc interface supports sequential sparse multiple right-hand >>> side, but not distributed. >>> It turns out that mumps does not support distributed sparse multiple >>> right-hand sides at >>> the moment (see attached email). >>> >>> Jean-Yves invites you to communicate with him directly. >>> Let me know what we can help on this matter, >>> e.g., add support for parallel implementation of sparse multiple >>> right-hand side with >>> centralized rhs input? >>> >>> Hong >>> >>> ---------------------- >>> Jean-Yves LExcellent >>> 5:14 AM (3 hours ago) >>> >>> to Hong, mumps-dev >>> >>> >>> >>> >>> Hello, >>> >>> We do not support distributed sparse multiple right-hand sides at >>> the moment. From the feedback we have from applications, the >>> right-hand sides are often very sparse, and having them distributed >>> did not seem critical. >>> >>> Since we are specifying a distributed right-hand sides feature at the >>> moment, could you let us know in more detail the need regarding >>> distributed sparse right-hand side (e.g., do all columns have the same >>> nonzero structure in that case) or put us in contact with the user who >>> needs this? >>> >>> Thanks, >>> Jean-Yves and Patrick >>> >>>> >>>> Thanks a lot guys, very helpful. >>>> >>>> >>>> >>>> >>>> I see MUMPS http://mumps.enseeiht.fr/ >>>> >>>> Sparse multiple right-hand side, distributed solution; Exploitation of >>>> sparsity in the right-hand sidesPETSc interface computes mumps distributed >>>> solution as default (this is not new) (ICNTL(21) = 1) >>>> >>>> I will add support for Sparse multiple right-hand side. >>>> >>>> Hong >>>> >>>> On Thu, May 31, 2018 at 11:25 AM, Smith, Barry F. >>> [mailto:bsmith at mcs.anl.gov]> wrote: >>>> Hong, >>>> >>>> Can you see about adding support for distributed right hand side? >>>> >>>> Thanks >>>> >>>> Barry >>>> >>>> > On May 31, 2018, at 2:37 AM, Marius Buerkle >>> mbuerkle at web.de]> wrote: >>>> > >>>> > The fix for MAT_NEW_NONZERO_LOCATIONS, thanks again. >>>> > >>>> > I have yet another question, sorry. The recent version of MUMPS >>>> supports distributed and sparse RHS is there any chance that this will be >>>> supported in PETSc in the near future? >>>> > >>>> > >>>> > >>>> > >>>> >> On May 30, 2018, at 6:55 PM, Marius Buerkle >>> mbuerkle at web.de]> wrote: >>>> >> >>>> >> Thanks for the quick fix, I will test it and report back. >>>> >> I have another maybe related question, if MAT_NEW_NONZERO_LOCATIONS >>>> is true and let's say 1 new nonzero position is created it does not >>>> allocated 1 but several new nonzeros but only use 1. >>>> > >>>> > Correct >>>> > >>>> >> I think that is normal, right? >>>> > >>>> > Yes >>>> > >>>> >> But, at least as far as I understand the manual, a subsequent call >>>> of mat assemble with >>>> >> MAT_FINAL_ASSEMBLY should compress out the unused allocations and >>>> release the memory, is this correct? >>>> > >>>> > It "compresses it out" (by shifting all the nonzero entries to the >>>> beginning of the internal i, j, and a arrays), but does NOT release any >>>> memory. Since the values are stored in one big contiguous array (obtained >>>> with a single malloc) it cannot just free part of the array, so the extra >>>> locations just sit harmlessly at the end if the array unused. >>>> > >>>> >> If so, this did not work for me, even after doing >>>> >> MAT_FINAL_ASSEMBLY the unused nonzero allocations remain. Is this >>>> normal? >>>> > >>>> > Yes, >>>> > >>>> > Barry >>>> > >>>> >> >>>> >>> >>>> >>> Fixed in the branch barry/fix-mat-new-nonzero-locations/maint >>>> >>> >>>> >>> Once this passes testing it will go into the maint branch and then >>>> the next patch release but you can use it now in the branch >>>> barry/fix-mat-new-nonzero-locations/maint >>>> >>> >>>> >>> Thanks for the report and reproducible example >>>> >>> >>>> >>> Barry >>>> >>> >>>> >>> >>>> >>>> On May 29, 2018, at 7:51 PM, Marius Buerkle >>> [mailto:mbuerkle at web.de]> wrote: >>>> >>>> >>>> >>>> Sure, I made a small reproducer, it is Fortran though I hope that >>>> is ok. If MAT_NEW_NONZERO_LOCATIONS is set to false I get an error, if it >>>> is set to true the new nonzero element is inserted, if >>>> MAT_NEW_NONZERO_LOCATIONS is false and either MAT_NEW_NONZERO_LOCATION_ERR >>>> or MAT_NEW_NONZERO_ALLOCATION_ERR is set to false afterwards then the new >>>> nonzero is also created without an error, but if MAT_NEW_NONZERO_LOCATIONS >>>> is set to false after MAT_NEW_NONZERO_LOCATION_ERR/MAT_NEW_NONZERO_ALLOCATION_ERR >>>> have been set to false I get an error again. >>>> >>>> >>>> >>>> >>>> >>>> program newnonzero >>>> >>>> #include >>>> >>>> use petscmat >>>> >>>> implicit none >>>> >>>> >>>> >>>> Mat :: A >>>> >>>> PetscInt :: dnnz,onnz,n,m,idxm(1),idxn(1),nl1,nl2 >>>> >>>> PetscScalar :: v(1) >>>> >>>> PetscReal :: info(MAT_INFO_SIZE) >>>> >>>> PetscErrorCode :: ierr >>>> >>>> >>>> >>>> integer :: nproc,iproc,i >>>> >>>> >>>> >>>> call PetscInitialize(PETSC_NULL_CHARACTER,ierr) >>>> >>>> >>>> >>>> call MPI_COMM_SIZE(PETSC_COMM_WORLD, nproc,ierr) >>>> >>>> >>>> >>>> call MPI_Comm_rank( PETSC_COMM_WORLD, iproc, ierr ) >>>> >>>> >>>> >>>> n=3 >>>> >>>> m=n >>>> >>>> call MatCreateAIJ(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,n,m, >>>> 1,PETSC_NULL_INTEGER,0,PETSC_NULL_INTEGER,A,ierr) >>>> >>>> >>>> >>>> >>>> >>>> call MatGetOwnershipRange(A,nl1,nl2,ierr) >>>> >>>> do i=nl1,nl2-1 >>>> >>>> idxn(1)=i >>>> >>>> idxm(1)=i >>>> >>>> v(1)=1d0 >>>> >>>> call MatSetValues(A,1,idxn,1,idxm, v,INSERT_VALUES,ierr) >>>> >>>> end do >>>> >>>> call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) >>>> >>>> call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) >>>> >>>> >>>> >>>> call MatSetOption(A,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE,ierr) >>>> >>>> !~ call MatSetOption(A,MAT_NEW_NONZERO_LOCATION_ERR,PETSC_ >>>> FALSE,ierr) >>>> >>>> !~ call MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR >>>> ,PETSC_FALSE,ierr) >>>> >>>> !~ call MatSetOption(A,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE,ierr) >>>> >>>> >>>> >>>> >>>> >>>> idxn(1)=0 >>>> >>>> idxm(1)=n-1 >>>> >>>> if ((idxn(1).ge.nl1).and.(idxn(1).le.nl2-1)) then >>>> >>>> v(1)=2d0 >>>> >>>> call MatSetValues(A,1,idxn,1,idxm, v,INSERT_VALUES,ierr) >>>> >>>> end if >>>> >>>> call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) >>>> >>>> call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) >>>> >>>> >>>> >>>> if ((idxn(1).ge.nl1).and.(idxn(1).le.nl2-1)) then >>>> >>>> v(1)=2d0 >>>> >>>> call MatGetValues(A,1,idxn,1,idxm, v,ierr) >>>> >>>> write(6,*) v >>>> >>>> end if >>>> >>>> >>>> >>>> call PetscFinalize(ierr) >>>> >>>> >>>> >>>> end program newnonzero >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> $ mpiexec.hydra -n 3 ./a.out >>>> >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> >>>> [0]PETSC ERROR: Argument out of range >>>> >>>> [0]PETSC ERROR: Inserting a new nonzero at global row/column (0, >>>> 2) into matrix >>>> >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/ >>>> documentation/faq.html[http://www.mcs.anl.gov/petsc/ >>>> documentation/faq.html] for trouble shooting. >>>> >>>> [0]PETSC ERROR: Petsc Release Version 3.9.2, May, 20, 2018 >>>> >>>> [0]PETSC ERROR: ./a.out on a named tono-hpc1 by marius Wed May 30 >>>> 09:42:40 2018 >>>> >>>> [0]PETSC ERROR: Configure options --prefix=/home/marius/prog/petsc/3.9.2 >>>> --download-elemental=yes --download-metis=yes --download-parmetis=yes >>>> --download-mumps=yes --with-scalapack-lib="/home/ >>>> marius/intel/compilers_and_libraries_2018.2.199/linux/ >>>> mkl/lib/intel64/libmkl_scalapack_lp64.a -Wl,--start-group >>>> /home/marius/intel/compilers_and_libraries_2018.2.199/ >>>> linux/mkl/lib/intel64/libmkl_intel_lp64.a /home/marius/intel/compilers_ >>>> and_libraries_2018.2.199/linux/mkl/lib/intel64/libmkl_sequential.a >>>> /home/marius/intel/compilers_and_libraries_2018.2.199/ >>>> linux/mkl/lib/intel64/libmkl_core.a /home/marius/intel/compilers_ >>>> and_libraries_2018.2.199/linux/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.a >>>> -Wl,--end-group -lpthread -lm -ldl" --FC=mpiifort --CC=mpicc --CXX=mpicxx >>>> --with-scalar-type=complex --with-mpi-dir= --with-blaslapack-lib="/home/ >>>> marius/intel/compilers_and_libraries_2018.2.199/linux/ >>>> mkl/lib/intel64/libmkl_scalapack_lp64.a -Wl,--start-group >>>> /home/marius/intel/compilers_and_libraries_2018.2.199/ >>>> linux/mkl/lib/intel64/libmkl_intel_lp64.a /home/marius/intel/compilers_ >>>> and_libraries_2018.2.199/linux/mkl/lib/intel64/libmkl_sequential.a >>>> /home/marius/intel/compilers_and_libraries_2018.2.199/ >>>> linux/mkl/lib/intel64/libmkl_core.a /home/marius/intel/compilers_ >>>> and_libraries_2018.2.199/linux/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.a >>>> -Wl,--end-group -lpthread -lm -ldl" --with-cxx-dialect=C++11 >>>> --download-superlu_dist=yes --download-ptscotch=yes --with-x >>>> --with-debugging=1 --download-superlu=yes --with-mkl_cpardiso=1 >>>> --with-mkl_pardiso=1 --with-scalapack=1 >>>> >>>> [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 607 in >>>> /home/marius/prog/petsc/petsc-3.9.2/src/mat/impls/aij/mpi/mpiaij.c >>>> >>>> [0]PETSC ERROR: #2 MatSetValues() line 1312 in >>>> /home/marius/prog/petsc/petsc-3.9.2/src/mat/interface/matrix.c >>>> >>>> (0.000000000000000E+000,0.000000000000000E+000) >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> Please send complete error message; type of matrix used etc. >>>> Ideally code that demonstrates the problem. >>>> >>>> >>>> >>>> Barry >>>> >>>> >>>> >>>> >>>> >>>>> On May 29, 2018, at 3:31 AM, Marius Buerkle >>> [mailto:mbuerkle at web.de]> wrote: >>>> >>>>> >>>> >>>>> >>>> >>>>> Hi, >>>> >>>>> >>>> >>>>> I tried to set MAT_NEW_NONZERO_LOCATIONS to false, as far as I >>>> understood MatSetValues should simply ignore entries which would give rise >>>> to new nonzero values not creating a new entry and not cause an error, but >>>> I get "[1]PETSC ERROR: Inserting a new nonzero at global row/column". Is >>>> this option supposed to work or not? >>>> >>>> >>>> >>> >>>> >>> >>>> > >>>> >>>> >>> >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From zakaryah at gmail.com Mon Jun 25 15:55:27 2018 From: zakaryah at gmail.com (zakaryah) Date: Mon, 25 Jun 2018 16:55:27 -0400 Subject: [petsc-users] Reusing pseudo time-stepper Message-ID: I'm interested in using a time-stepper to solve problems arising from multiple data sets (external forces). Currently, I do the following: TSCreate TSSetProblemType TSSetSolution TSSetDM TSSetRHSFunction TSSetRHSJacobian TSSetType TSSetMaxSteps TSSetMaxTime TSSetExactFinalTime TSSetMaxStepRejections then, inside the loop over different data sets: TSSetTimeStep TSPseudoSetTimeStepIncrement TSSetTime (to zero) TSSetStepNumber (to zero) TSSetUp TSSolve This seems to be working, but the step increment behaves oddly on every loop iteration after the first. Regardless of what I set the time step to, the first TSSolve iteration makes a step which is about twelve orders of magnitude smaller than the specified dt, and from that point on, the controller multiplies every step size by the specified pseudo time step increment (1.5), up until the very end, at which point the multiplier becomes very large as expected near convergence. If I create, set up, and destroy the TS in each iteration of the loop, the behavior is as expected, i.e. the first step is almost always by the specified dt. Is there additional information inside the TS that I need to reset? Is it a bad idea to reuse a TS? A couple tangential questions - I don't seem to be getting any output from -ts_adapt_monitor, and calling TSSetMaxStepRejections(ts,(PetscInt)-1) seems to set the maximum number of accepted step rejections to zero, rather than to unlimited, as specified by the documentation. Thanks for any help you can provide! -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Jun 25 16:12:54 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Mon, 25 Jun 2018 21:12:54 +0000 Subject: [petsc-users] Reusing pseudo time-stepper In-Reply-To: References: Message-ID: <5C940A2E-A47A-406F-B6A3-11569F94F5E3@anl.gov> It is intended to be reusable as you are attempting. You will have to either (1) run in the debugger with some break points to see why the initial time is always forced to be so small (2) send us the code so we can reproduce the problem. Barry > On Jun 25, 2018, at 3:55 PM, zakaryah wrote: > > I'm interested in using a time-stepper to solve problems arising from multiple data sets (external forces). Currently, I do the following: > > TSCreate > TSSetProblemType > TSSetSolution > TSSetDM > TSSetRHSFunction > TSSetRHSJacobian > TSSetType > TSSetMaxSteps > TSSetMaxTime > TSSetExactFinalTime > TSSetMaxStepRejections > > then, inside the loop over different data sets: > > TSSetTimeStep > TSPseudoSetTimeStepIncrement > TSSetTime (to zero) > TSSetStepNumber (to zero) > TSSetUp > TSSolve > > This seems to be working, but the step increment behaves oddly on every loop iteration after the first. Regardless of what I set the time step to, the first TSSolve iteration makes a step which is about twelve orders of magnitude smaller than the specified dt, and from that point on, the controller multiplies every step size by the specified pseudo time step increment (1.5), up until the very end, at which point the multiplier becomes very large as expected near convergence. > > If I create, set up, and destroy the TS in each iteration of the loop, the behavior is as expected, i.e. the first step is almost always by the specified dt. Is there additional information inside the TS that I need to reset? Is it a bad idea to reuse a TS? > > A couple tangential questions - I don't seem to be getting any output from -ts_adapt_monitor, and calling TSSetMaxStepRejections(ts,(PetscInt)-1) seems to set the maximum number of accepted step rejections to zero, rather than to unlimited, as specified by the documentation. > > Thanks for any help you can provide! From david.knezevic at akselos.com Wed Jun 27 13:36:41 2018 From: david.knezevic at akselos.com (David Knezevic) Date: Wed, 27 Jun 2018 14:36:41 -0400 Subject: [petsc-users] Replicating a hang with MUMPS Message-ID: I ran into a case where using MUMPS (called via "-ksp_type preonly -pc_type lu -pc_factor_mat_solver_package mumps") for a particular solve hangs indefinitely with 24 MPI processes (but it works fine with other numbers of processes). The stack trace when killing the job is below, in case that gives any clue as to what is wrong. I'm trying to replicate this with a simple test case. I wrote out the matrix and right-hand side to disk using MatView and VecView, and then I modified ksp ex10 to read in these files and solve with 24 cores. However, that did not replicate the error, so I think I also need to make sure that I use the same number of rows per process in the test case as in the case that hung. As a result I'm wondering if there is a way to modify the parallel layout of the matrix and vector after I read them in? Also, if there are any other suggestions about reproducing or debugging this issue, please let me know! Best, David -------------------------------- #0 0x00007fb12bf0e74d in poll () at ../sysdeps/unix/syscall-template.S:84 #1 0x00007fb126262e58 in ?? () from /usr/lib/libopen-pal.so.13 #2 0x00007fb1262596fb in opal_libevent2021_event_base_loop () from /usr/lib/libopen-pal.so.13 #3 0x00007fb126223238 in opal_progress () from /usr/lib/libopen-pal.so.13 #4 0x00007fb12cef53db in ompi_request_default_test () from /usr/lib/libmpi.so.12 #5 0x00007fb12cf21d61 in PMPI_Test () from /usr/lib/libmpi.so.12 #6 0x00007fb127a5b939 in pmpi_test__ () from /usr/lib/libmpi_mpifh.so.12 #7 0x00007fb132888d87 in dmumps_try_recvtreat (comm_load=8, ass_irecv=40, blocking=.FALSE., set_irecv=.TRUE., message_received=.FALSE., msgsou=-1, msgtag=-1, status=..., bufr=..., lbufr=401408, lbufr_bytes=1605629, procnode_steps=..., posfac=410095, iwpos=3151, iwposcb=30557, iptrlu=1536548, lrlu=1126454, lrlus=2864100, n=30675, iw=..., liw=39935, a=..., la=3367108, ptrist=..., ptlust=..., ptrfac=..., ptrast=..., step=..., pimaster=..., pamaster=..., nstk_s=..., comp=0, iflag=0, ierror=0, comm=7, nbprocfils=..., ipool=..., lpool=48, leaf=2, nbfin=90, myid=33, slavef=90, root=..., opassw=353031, opeliw=700399235, itloc=..., rhs_mumps=..., fils=..., ptrarw=..., ptraiw=..., intarr=..., dblarr=..., icntl=..., keep=..., keep8=..., dkeep=..., nd=..., frere=..., lptrar=30675, nelt=1, frtptr=..., frtelt=..., istep_to_iniv2=..., tab_pos_in_pere=..., stack_right_authorized=.TRUE., lrgroups=...) at dfac_process_message.F:646 #8 0x00007fb1328cfcd1 in dmumps_fac_par_m::dmumps_fac_par (n=30675, iw=..., liw=39935, a=..., la=3367108, nstk_steps=..., nbprocfils=..., nd=..., fils=..., step=..., frere=..., dad=..., cand=..., istep_to_iniv2=..., tab_pos_in_pere=..., maxfrt=0, ntotpv=0, nmaxnpiv=150, ptrist=..., ptrast=..., pimaster=..., pamaster=..., ptrarw=..., ptraiw=..., itloc=..., rhs_mumps=..., ipool=..., lpool=48, rinfo=..., posfac=410095, iwpos=3151, lrlu=1126454, iptrlu=1536548, lrlus=2864100, leaf=2, nbroot=1, nbrtot=90, uu=0.01, icntl=..., ptlust=..., ptrfac=..., nsteps=1, info=..., keep=..., keep8=..., procnode_steps=..., slavef=90, myid=33, comm_nodes=7, myid_nodes=33, bufr=..., lbufr=401408, lbufr_bytes=1605629, intarr=..., dblarr=..., root=..., perm=..., nelt=1, frtptr=..., frtelt=..., lptrar=30675, comm_load=8, ass_irecv=40, seuil=0, seuil_ldlt_niv2=0, mem_distrib=..., ne=..., dkeep=..., pivnul_list=..., lpn_list=1, lrgroups=...) at dfac_par_m.F:207 #9 0x00007fb13287f875 in dmumps_fac_b (n=30675, nsteps=1, a=..., la=3367108, iw=..., liw=39935, sym_perm=..., na=..., lna=47, ne_steps=..., nfsiz=..., fils=..., step=..., frere=..., dad=..., cand=..., istep_to_iniv2=..., tab_pos_in_pere=..., ptrar=..., ldptrar=30675, ptrist=..., ptlust_s=..., ptrfac=..., iw1=..., iw2=..., itloc=..., rhs_mumps=..., pool=..., lpool=48, cntl1=0.01, icntl=..., info=..., rinfo=..., keep=..., keep8=..., procnode_steps=..., slavef=90, comm_nodes=7, myid=33, myid_nodes=33, bufr=..., lbufr=401408, lbufr_bytes=1605629, intarr=..., dblarr=..., root=..., nelt=1, frtptr=..., frtelt=..., comm_load=8, ass_irecv=40, seuil=0, seuil_ldlt_niv2=0, mem_distrib=..., dkeep=..., pivnul_list=..., lpn_list=1, lrgroups=...) at dfac_b.F:167 #10 0x00007fb1328419ed in dmumps_fac_driver (id=) at dfac_driver.F:2291 #11 0x00007fb1327ff6dc in dmumps (id=) at dmumps_driver.F:1686 #12 0x00007fb1327faf0a in dmumps_f77 (job=2, sym=0, par=1, comm_f77=5, n=30675, icntl=..., cntl=..., keep=..., dkeep=..., keep8=..., nz=0, nnz=0, irn=..., irnhere=0, jcn=..., jcnhere=0, a=..., ahere=0, nz_loc=622296, nnz_loc=0, irn_loc=..., irn_lochere=1, jcn_loc=..., jcn_lochere=1, a_loc=..., a_lochere=1, nelt=0, eltptr=..., eltptrhere=0, eltvar=..., eltvarhere=0, a_elt=..., a_elthere=0, perm_in=..., perm_inhere=0, rhs=..., rhshere=0, redrhs=..., redrhshere=0, info=..., rinfo=..., infog=..., rinfog=..., deficiency=0, lwk_user=0, size_schur=0, listvar_schur=..., listvar_schurhere=0, schur=..., schurhere=0, wk_user=..., wk_userhere=0, colsca=..., colscahere=0, rowsca=..., rowscahere=0, instance_number=1, nrhs=1, lrhs=0, lredrhs=0, rhs_sparse=..., rhs_sparsehere=0, sol_loc=..., sol_lochere=0, irhs_sparse=..., irhs_sparsehere=0, irhs_ptr=..., irhs_ptrhere=0, isol_loc=..., isol_lochere=0, nz_rhs=0, lsol_loc=0, schur_mloc=0, schur_nloc=0, schur_lld=0, mblock=0, nblock=0, nprow=0, npcol=0, ooc_tmpdir=..., ooc_prefix=..., write_problem=..., tmpdirlen=20, prefixlen=20, write_problemlen=20) at dmumps_f77.F:267 #13 0x00007fb1327f9cfa in dmumps_c (mumps_par=mumps_par at entry=0x12bd9660) at mumps_c.c:417 #14 0x00007fb1321a23fc in MatFactorNumeric_MUMPS (F=0x12bd8b60, A=0x26bd890, info=) at /home/buildslave/software/petsc-src/src/mat/impls/aij/mpi/mumps/mumps.c:1073 #15 0x00007fb131ec6ea7 in MatLUFactorNumeric (fact=0x12bd8b60, mat=0x26bd890, info=info at entry=0xc2a66f8) at /home/buildslave/software/petsc-src/src/mat/interface/matrix.c:3025 #16 0x00007fb1325040d6 in PCSetUp_LU (pc=0xc2a6380) at /home/buildslave/software/petsc-src/src/ksp/pc/impls/factor/lu/lu.c:131 #17 0x00007fb13259903e in PCSetUp (pc=0xc2a6380) at /home/buildslave/software/petsc-src/src/ksp/pc/interface/precon.c:923 #18 0x00007fb13263e53f in KSPSetUp (ksp=ksp at entry=0x12b28c70) at /home/buildslave/software/petsc-src/src/ksp/ksp/interface/itfunc.c:381 #19 0x00007fb13263ed36 in KSPSolve (ksp=0x12b28c70, b=0xad77d50, x=0xad801c0) at /home/buildslave/software/petsc-src/src/ksp/ksp/interface/itfunc.c:612 #20 0x00007fb12db5dfc2 in libMesh::PetscLinearSolver::solve(libMesh::SparseMatrix&, libMesh::SparseMatrix&, libMesh::NumericVector&, libMesh::NumericVector&, double, unsigned int) () from /mnt/fileserver/akselos-4.2.x/scrbe/build/bin/../../third_party/opt_real/libmesh_opt.so.0 #21 0x00007fb1338d0c06 in libMesh::PetscLinearSolver::solve(libMesh::SparseMatrix&, libMesh::NumericVector&, libMesh::NumericVector&, double, unsigned int) () from /mnt/fileserver/akselos-4.2.x/scrbe/build/bin/../lib/libscrbe-opt_real.so #22 0x00007fb1335e8abd in std::pair SolveHelper::try_linear_solve >(libMesh::LinearSolver&, libMesh::SolverConfiguration&, libMesh::SparseMatrix&, libMesh::NumericVector&, libMesh::NumericVector&) () from /mnt/fileserver/akselos-4.2.x/scrbe/build/bin/../lib/libscrbe-opt_real.so #23 0x00007fb133a70206 in -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Wed Jun 27 13:47:23 2018 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Wed, 27 Jun 2018 21:47:23 +0300 Subject: [petsc-users] Replicating a hang with MUMPS In-Reply-To: References: Message-ID: Forwarding to the list Il Mer 27 Giu 2018, 21:46 Stefano Zampini ha scritto: > You can call MatSetSizes before MatLoad > > Il Mer 27 Giu 2018, 21:36 David Knezevic ha > scritto: > >> I ran into a case where using MUMPS (called via "-ksp_type preonly >> -pc_type lu -pc_factor_mat_solver_package mumps") for a particular solve >> hangs indefinitely with 24 MPI processes (but it works fine with other >> numbers of processes). The stack trace when killing the job is below, in >> case that gives any clue as to what is wrong. >> >> I'm trying to replicate this with a simple test case. I wrote out the >> matrix and right-hand side to disk using MatView and VecView, and then I >> modified ksp ex10 to read in these files and solve with 24 cores. However, >> that did not replicate the error, so I think I also need to make sure that >> I use the same number of rows per process in the test case as in the case >> that hung. As a result I'm wondering if there is a way to modify the >> parallel layout of the matrix and vector after I read them in? >> >> Also, if there are any other suggestions about reproducing or debugging >> this issue, please let me know! >> >> Best, >> David >> >> -------------------------------- >> >> #0 0x00007fb12bf0e74d in poll () at ../sysdeps/unix/syscall-template.S:84 >> #1 0x00007fb126262e58 in ?? () from /usr/lib/libopen-pal.so.13 >> #2 0x00007fb1262596fb in opal_libevent2021_event_base_loop () from >> /usr/lib/libopen-pal.so.13 >> #3 0x00007fb126223238 in opal_progress () from /usr/lib/libopen-pal.so.13 >> #4 0x00007fb12cef53db in ompi_request_default_test () from >> /usr/lib/libmpi.so.12 >> #5 0x00007fb12cf21d61 in PMPI_Test () from /usr/lib/libmpi.so.12 >> #6 0x00007fb127a5b939 in pmpi_test__ () from /usr/lib/libmpi_mpifh.so.12 >> #7 0x00007fb132888d87 in dmumps_try_recvtreat (comm_load=8, >> ass_irecv=40, blocking=.FALSE., set_irecv=.TRUE., message_received=.FALSE., >> msgsou=-1, msgtag=-1, status=..., bufr=..., lbufr=401408, >> lbufr_bytes=1605629, procnode_steps=..., posfac=410095, iwpos=3151, >> iwposcb=30557, >> iptrlu=1536548, lrlu=1126454, lrlus=2864100, n=30675, iw=..., >> liw=39935, a=..., la=3367108, ptrist=..., ptlust=..., ptrfac=..., >> ptrast=..., step=..., pimaster=..., pamaster=..., nstk_s=..., comp=0, >> iflag=0, ierror=0, comm=7, nbprocfils=..., ipool=..., lpool=48, leaf=2, >> nbfin=90, myid=33, slavef=90, root=..., opassw=353031, >> opeliw=700399235, itloc=..., rhs_mumps=..., fils=..., ptrarw=..., >> ptraiw=..., intarr=..., dblarr=..., icntl=..., keep=..., keep8=..., >> dkeep=..., nd=..., frere=..., lptrar=30675, nelt=1, frtptr=..., frtelt=..., >> istep_to_iniv2=..., tab_pos_in_pere=..., >> stack_right_authorized=.TRUE., lrgroups=...) at dfac_process_message.F:646 >> #8 0x00007fb1328cfcd1 in dmumps_fac_par_m::dmumps_fac_par (n=30675, >> iw=..., liw=39935, a=..., la=3367108, nstk_steps=..., nbprocfils=..., >> nd=..., fils=..., step=..., frere=..., dad=..., cand=..., >> istep_to_iniv2=..., tab_pos_in_pere=..., maxfrt=0, ntotpv=0, nmaxnpiv=150, >> ptrist=..., ptrast=..., pimaster=..., pamaster=..., ptrarw=..., >> ptraiw=..., itloc=..., rhs_mumps=..., ipool=..., lpool=48, rinfo=..., >> posfac=410095, iwpos=3151, lrlu=1126454, iptrlu=1536548, lrlus=2864100, >> leaf=2, nbroot=1, nbrtot=90, uu=0.01, icntl=..., ptlust=..., ptrfac=..., >> nsteps=1, info=..., keep=..., keep8=..., procnode_steps=..., >> slavef=90, myid=33, comm_nodes=7, myid_nodes=33, bufr=..., lbufr=401408, >> lbufr_bytes=1605629, intarr=..., dblarr=..., root=..., perm=..., nelt=1, >> frtptr=..., frtelt=..., lptrar=30675, comm_load=8, ass_irecv=40, >> seuil=0, seuil_ldlt_niv2=0, mem_distrib=..., ne=..., dkeep=..., >> pivnul_list=..., lpn_list=1, lrgroups=...) at dfac_par_m.F:207 >> #9 0x00007fb13287f875 in dmumps_fac_b (n=30675, nsteps=1, a=..., >> la=3367108, iw=..., liw=39935, sym_perm=..., na=..., lna=47, ne_steps=..., >> nfsiz=..., fils=..., step=..., frere=..., dad=..., cand=..., >> istep_to_iniv2=..., tab_pos_in_pere=..., ptrar=..., ldptrar=30675, >> ptrist=..., >> ptlust_s=..., ptrfac=..., iw1=..., iw2=..., itloc=..., rhs_mumps=..., >> pool=..., lpool=48, cntl1=0.01, icntl=..., info=..., rinfo=..., keep=..., >> keep8=..., procnode_steps=..., slavef=90, comm_nodes=7, myid=33, >> myid_nodes=33, bufr=..., lbufr=401408, lbufr_bytes=1605629, >> intarr=..., dblarr=..., root=..., nelt=1, frtptr=..., frtelt=..., >> comm_load=8, ass_irecv=40, seuil=0, seuil_ldlt_niv2=0, mem_distrib=..., >> dkeep=..., pivnul_list=..., lpn_list=1, lrgroups=...) at dfac_b.F:167 >> #10 0x00007fb1328419ed in dmumps_fac_driver (id=> value requires 600640 bytes, which is more than max-value-size>) at >> dfac_driver.F:2291 >> #11 0x00007fb1327ff6dc in dmumps (id=> requires 600640 bytes, which is more than max-value-size>) at >> dmumps_driver.F:1686 >> #12 0x00007fb1327faf0a in dmumps_f77 (job=2, sym=0, par=1, comm_f77=5, >> n=30675, icntl=..., cntl=..., keep=..., dkeep=..., keep8=..., nz=0, nnz=0, >> irn=..., irnhere=0, jcn=..., jcnhere=0, a=..., ahere=0, nz_loc=622296, >> nnz_loc=0, irn_loc=..., irn_lochere=1, jcn_loc=..., >> jcn_lochere=1, a_loc=..., a_lochere=1, nelt=0, eltptr=..., >> eltptrhere=0, eltvar=..., eltvarhere=0, a_elt=..., a_elthere=0, >> perm_in=..., perm_inhere=0, rhs=..., rhshere=0, redrhs=..., redrhshere=0, >> info=..., rinfo=..., infog=..., rinfog=..., deficiency=0, lwk_user=0, >> size_schur=0, listvar_schur=..., listvar_schurhere=0, schur=..., >> schurhere=0, wk_user=..., wk_userhere=0, colsca=..., colscahere=0, >> rowsca=..., rowscahere=0, instance_number=1, nrhs=1, lrhs=0, lredrhs=0, >> rhs_sparse=..., rhs_sparsehere=0, sol_loc=..., sol_lochere=0, >> irhs_sparse=..., irhs_sparsehere=0, irhs_ptr=..., irhs_ptrhere=0, >> isol_loc=..., isol_lochere=0, nz_rhs=0, lsol_loc=0, schur_mloc=0, >> schur_nloc=0, schur_lld=0, mblock=0, nblock=0, nprow=0, npcol=0, >> ooc_tmpdir=..., ooc_prefix=..., write_problem=..., tmpdirlen=20, >> prefixlen=20, >> write_problemlen=20) at dmumps_f77.F:267 >> #13 0x00007fb1327f9cfa in dmumps_c (mumps_par=mumps_par at entry=0x12bd9660) >> at mumps_c.c:417 >> #14 0x00007fb1321a23fc in MatFactorNumeric_MUMPS (F=0x12bd8b60, >> A=0x26bd890, info=) at >> /home/buildslave/software/petsc-src/src/mat/impls/aij/mpi/mumps/mumps.c:1073 >> #15 0x00007fb131ec6ea7 in MatLUFactorNumeric (fact=0x12bd8b60, >> mat=0x26bd890, info=info at entry=0xc2a66f8) at >> /home/buildslave/software/petsc-src/src/mat/interface/matrix.c:3025 >> #16 0x00007fb1325040d6 in PCSetUp_LU (pc=0xc2a6380) at >> /home/buildslave/software/petsc-src/src/ksp/pc/impls/factor/lu/lu.c:131 >> #17 0x00007fb13259903e in PCSetUp (pc=0xc2a6380) at >> /home/buildslave/software/petsc-src/src/ksp/pc/interface/precon.c:923 >> #18 0x00007fb13263e53f in KSPSetUp (ksp=ksp at entry=0x12b28c70) at >> /home/buildslave/software/petsc-src/src/ksp/ksp/interface/itfunc.c:381 >> #19 0x00007fb13263ed36 in KSPSolve (ksp=0x12b28c70, b=0xad77d50, >> x=0xad801c0) at >> /home/buildslave/software/petsc-src/src/ksp/ksp/interface/itfunc.c:612 >> #20 0x00007fb12db5dfc2 in >> libMesh::PetscLinearSolver::solve(libMesh::SparseMatrix&, >> libMesh::SparseMatrix&, libMesh::NumericVector&, >> libMesh::NumericVector&, double, unsigned int) () >> from >> /mnt/fileserver/akselos-4.2.x/scrbe/build/bin/../../third_party/opt_real/libmesh_opt.so.0 >> #21 0x00007fb1338d0c06 in >> libMesh::PetscLinearSolver::solve(libMesh::SparseMatrix&, >> libMesh::NumericVector&, libMesh::NumericVector&, double, >> unsigned int) () from >> /mnt/fileserver/akselos-4.2.x/scrbe/build/bin/../lib/libscrbe-opt_real.so >> #22 0x00007fb1335e8abd in std::pair >> SolveHelper::try_linear_solve >> >(libMesh::LinearSolver&, libMesh::SolverConfiguration&, >> libMesh::SparseMatrix&, libMesh::NumericVector&, >> libMesh::NumericVector&) () >> from >> /mnt/fileserver/akselos-4.2.x/scrbe/build/bin/../lib/libscrbe-opt_real.so >> #23 0x00007fb133a70206 in >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Jun 27 14:12:06 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 27 Jun 2018 19:12:06 +0000 Subject: [petsc-users] Replicating a hang with MUMPS In-Reply-To: References: Message-ID: <6D44658D-3A93-4B34-83FF-7A90A9F4AC1D@anl.gov> David, This is ugly but should work. BEFORE reading in the matrix and right hand side set the LOCAL sizes for the matrix and vector. This way you can control exactly which rows go on which process. Note you will have to have your own mechanism to know what the local sizes should be (for example have the original program print out the sizes and just cut and paste them into your copy of ex10.c) PETSc doesn't provide an automatic way to do this (nor should it). Barry > On Jun 27, 2018, at 1:36 PM, David Knezevic wrote: > > I ran into a case where using MUMPS (called via "-ksp_type preonly -pc_type lu -pc_factor_mat_solver_package mumps") for a particular solve hangs indefinitely with 24 MPI processes (but it works fine with other numbers of processes). The stack trace when killing the job is below, in case that gives any clue as to what is wrong. > > I'm trying to replicate this with a simple test case. I wrote out the matrix and right-hand side to disk using MatView and VecView, and then I modified ksp ex10 to read in these files and solve with 24 cores. However, that did not replicate the error, so I think I also need to make sure that I use the same number of rows per process in the test case as in the case that hung. As a result I'm wondering if there is a way to modify the parallel layout of the matrix and vector after I read them in? > > Also, if there are any other suggestions about reproducing or debugging this issue, please let me know! > > Best, > David > > -------------------------------- > > #0 0x00007fb12bf0e74d in poll () at ../sysdeps/unix/syscall-template.S:84 > #1 0x00007fb126262e58 in ?? () from /usr/lib/libopen-pal.so.13 > #2 0x00007fb1262596fb in opal_libevent2021_event_base_loop () from /usr/lib/libopen-pal.so.13 > #3 0x00007fb126223238 in opal_progress () from /usr/lib/libopen-pal.so.13 > #4 0x00007fb12cef53db in ompi_request_default_test () from /usr/lib/libmpi.so.12 > #5 0x00007fb12cf21d61 in PMPI_Test () from /usr/lib/libmpi.so.12 > #6 0x00007fb127a5b939 in pmpi_test__ () from /usr/lib/libmpi_mpifh.so.12 > #7 0x00007fb132888d87 in dmumps_try_recvtreat (comm_load=8, ass_irecv=40, blocking=.FALSE., set_irecv=.TRUE., message_received=.FALSE., msgsou=-1, msgtag=-1, status=..., bufr=..., lbufr=401408, lbufr_bytes=1605629, procnode_steps=..., posfac=410095, iwpos=3151, iwposcb=30557, > iptrlu=1536548, lrlu=1126454, lrlus=2864100, n=30675, iw=..., liw=39935, a=..., la=3367108, ptrist=..., ptlust=..., ptrfac=..., ptrast=..., step=..., pimaster=..., pamaster=..., nstk_s=..., comp=0, iflag=0, ierror=0, comm=7, nbprocfils=..., ipool=..., lpool=48, leaf=2, > nbfin=90, myid=33, slavef=90, root=..., opassw=353031, opeliw=700399235, itloc=..., rhs_mumps=..., fils=..., ptrarw=..., ptraiw=..., intarr=..., dblarr=..., icntl=..., keep=..., keep8=..., dkeep=..., nd=..., frere=..., lptrar=30675, nelt=1, frtptr=..., frtelt=..., > istep_to_iniv2=..., tab_pos_in_pere=..., stack_right_authorized=.TRUE., lrgroups=...) at dfac_process_message.F:646 > #8 0x00007fb1328cfcd1 in dmumps_fac_par_m::dmumps_fac_par (n=30675, iw=..., liw=39935, a=..., la=3367108, nstk_steps=..., nbprocfils=..., nd=..., fils=..., step=..., frere=..., dad=..., cand=..., istep_to_iniv2=..., tab_pos_in_pere=..., maxfrt=0, ntotpv=0, nmaxnpiv=150, > ptrist=..., ptrast=..., pimaster=..., pamaster=..., ptrarw=..., ptraiw=..., itloc=..., rhs_mumps=..., ipool=..., lpool=48, rinfo=..., posfac=410095, iwpos=3151, lrlu=1126454, iptrlu=1536548, lrlus=2864100, leaf=2, nbroot=1, nbrtot=90, uu=0.01, icntl=..., ptlust=..., ptrfac=..., > nsteps=1, info=..., keep=..., keep8=..., procnode_steps=..., slavef=90, myid=33, comm_nodes=7, myid_nodes=33, bufr=..., lbufr=401408, lbufr_bytes=1605629, intarr=..., dblarr=..., root=..., perm=..., nelt=1, frtptr=..., frtelt=..., lptrar=30675, comm_load=8, ass_irecv=40, > seuil=0, seuil_ldlt_niv2=0, mem_distrib=..., ne=..., dkeep=..., pivnul_list=..., lpn_list=1, lrgroups=...) at dfac_par_m.F:207 > #9 0x00007fb13287f875 in dmumps_fac_b (n=30675, nsteps=1, a=..., la=3367108, iw=..., liw=39935, sym_perm=..., na=..., lna=47, ne_steps=..., nfsiz=..., fils=..., step=..., frere=..., dad=..., cand=..., istep_to_iniv2=..., tab_pos_in_pere=..., ptrar=..., ldptrar=30675, ptrist=..., > ptlust_s=..., ptrfac=..., iw1=..., iw2=..., itloc=..., rhs_mumps=..., pool=..., lpool=48, cntl1=0.01, icntl=..., info=..., rinfo=..., keep=..., keep8=..., procnode_steps=..., slavef=90, comm_nodes=7, myid=33, myid_nodes=33, bufr=..., lbufr=401408, lbufr_bytes=1605629, > intarr=..., dblarr=..., root=..., nelt=1, frtptr=..., frtelt=..., comm_load=8, ass_irecv=40, seuil=0, seuil_ldlt_niv2=0, mem_distrib=..., dkeep=..., pivnul_list=..., lpn_list=1, lrgroups=...) at dfac_b.F:167 > #10 0x00007fb1328419ed in dmumps_fac_driver (id=) at dfac_driver.F:2291 > #11 0x00007fb1327ff6dc in dmumps (id=) at dmumps_driver.F:1686 > #12 0x00007fb1327faf0a in dmumps_f77 (job=2, sym=0, par=1, comm_f77=5, n=30675, icntl=..., cntl=..., keep=..., dkeep=..., keep8=..., nz=0, nnz=0, irn=..., irnhere=0, jcn=..., jcnhere=0, a=..., ahere=0, nz_loc=622296, nnz_loc=0, irn_loc=..., irn_lochere=1, jcn_loc=..., > jcn_lochere=1, a_loc=..., a_lochere=1, nelt=0, eltptr=..., eltptrhere=0, eltvar=..., eltvarhere=0, a_elt=..., a_elthere=0, perm_in=..., perm_inhere=0, rhs=..., rhshere=0, redrhs=..., redrhshere=0, info=..., rinfo=..., infog=..., rinfog=..., deficiency=0, lwk_user=0, > size_schur=0, listvar_schur=..., listvar_schurhere=0, schur=..., schurhere=0, wk_user=..., wk_userhere=0, colsca=..., colscahere=0, rowsca=..., rowscahere=0, instance_number=1, nrhs=1, lrhs=0, lredrhs=0, rhs_sparse=..., rhs_sparsehere=0, sol_loc=..., sol_lochere=0, > irhs_sparse=..., irhs_sparsehere=0, irhs_ptr=..., irhs_ptrhere=0, isol_loc=..., isol_lochere=0, nz_rhs=0, lsol_loc=0, schur_mloc=0, schur_nloc=0, schur_lld=0, mblock=0, nblock=0, nprow=0, npcol=0, ooc_tmpdir=..., ooc_prefix=..., write_problem=..., tmpdirlen=20, prefixlen=20, > write_problemlen=20) at dmumps_f77.F:267 > #13 0x00007fb1327f9cfa in dmumps_c (mumps_par=mumps_par at entry=0x12bd9660) at mumps_c.c:417 > #14 0x00007fb1321a23fc in MatFactorNumeric_MUMPS (F=0x12bd8b60, A=0x26bd890, info=) at /home/buildslave/software/petsc-src/src/mat/impls/aij/mpi/mumps/mumps.c:1073 > #15 0x00007fb131ec6ea7 in MatLUFactorNumeric (fact=0x12bd8b60, mat=0x26bd890, info=info at entry=0xc2a66f8) at /home/buildslave/software/petsc-src/src/mat/interface/matrix.c:3025 > #16 0x00007fb1325040d6 in PCSetUp_LU (pc=0xc2a6380) at /home/buildslave/software/petsc-src/src/ksp/pc/impls/factor/lu/lu.c:131 > #17 0x00007fb13259903e in PCSetUp (pc=0xc2a6380) at /home/buildslave/software/petsc-src/src/ksp/pc/interface/precon.c:923 > #18 0x00007fb13263e53f in KSPSetUp (ksp=ksp at entry=0x12b28c70) at /home/buildslave/software/petsc-src/src/ksp/ksp/interface/itfunc.c:381 > #19 0x00007fb13263ed36 in KSPSolve (ksp=0x12b28c70, b=0xad77d50, x=0xad801c0) at /home/buildslave/software/petsc-src/src/ksp/ksp/interface/itfunc.c:612 > #20 0x00007fb12db5dfc2 in libMesh::PetscLinearSolver::solve(libMesh::SparseMatrix&, libMesh::SparseMatrix&, libMesh::NumericVector&, libMesh::NumericVector&, double, unsigned int) () > from /mnt/fileserver/akselos-4.2.x/scrbe/build/bin/../../third_party/opt_real/libmesh_opt.so.0 > #21 0x00007fb1338d0c06 in libMesh::PetscLinearSolver::solve(libMesh::SparseMatrix&, libMesh::NumericVector&, libMesh::NumericVector&, double, unsigned int) () from /mnt/fileserver/akselos-4.2.x/scrbe/build/bin/../lib/libscrbe-opt_real.so > #22 0x00007fb1335e8abd in std::pair SolveHelper::try_linear_solve >(libMesh::LinearSolver&, libMesh::SolverConfiguration&, libMesh::SparseMatrix&, libMesh::NumericVector&, libMesh::NumericVector&) () > from /mnt/fileserver/akselos-4.2.x/scrbe/build/bin/../lib/libscrbe-opt_real.so > #23 0x00007fb133a70206 in From david.knezevic at akselos.com Wed Jun 27 14:16:54 2018 From: david.knezevic at akselos.com (David Knezevic) Date: Wed, 27 Jun 2018 15:16:54 -0400 Subject: [petsc-users] Replicating a hang with MUMPS In-Reply-To: <6D44658D-3A93-4B34-83FF-7A90A9F4AC1D@anl.gov> References: <6D44658D-3A93-4B34-83FF-7A90A9F4AC1D@anl.gov> Message-ID: On Wed, Jun 27, 2018 at 3:12 PM, Smith, Barry F. wrote: > > David, > > This is ugly but should work. BEFORE reading in the matrix and right > hand side set the LOCAL sizes for the matrix and vector. This way you can > control exactly which rows go on which process. Note you will have to have > your own mechanism to know what the local sizes should be (for example have > the original program print out the sizes and just cut and paste them into > your copy of ex10.c) PETSc doesn't provide an automatic way to do this (nor > should it). > > Barry > Thanks Barry and Stefano, the approach you both suggested (calling MatSetSizes and VecSetSizes before reading in) was what I was looking for. Best, David > On Jun 27, 2018, at 1:36 PM, David Knezevic > wrote: > > > > I ran into a case where using MUMPS (called via "-ksp_type preonly > -pc_type lu -pc_factor_mat_solver_package mumps") for a particular solve > hangs indefinitely with 24 MPI processes (but it works fine with other > numbers of processes). The stack trace when killing the job is below, in > case that gives any clue as to what is wrong. > > > > I'm trying to replicate this with a simple test case. I wrote out the > matrix and right-hand side to disk using MatView and VecView, and then I > modified ksp ex10 to read in these files and solve with 24 cores. However, > that did not replicate the error, so I think I also need to make sure that > I use the same number of rows per process in the test case as in the case > that hung. As a result I'm wondering if there is a way to modify the > parallel layout of the matrix and vector after I read them in? > > > > Also, if there are any other suggestions about reproducing or debugging > this issue, please let me know! > > > > Best, > > David > > > > -------------------------------- > > > > #0 0x00007fb12bf0e74d in poll () at ../sysdeps/unix/syscall-templa > te.S:84 > > #1 0x00007fb126262e58 in ?? () from /usr/lib/libopen-pal.so.13 > > #2 0x00007fb1262596fb in opal_libevent2021_event_base_loop () from > /usr/lib/libopen-pal.so.13 > > #3 0x00007fb126223238 in opal_progress () from > /usr/lib/libopen-pal.so.13 > > #4 0x00007fb12cef53db in ompi_request_default_test () from > /usr/lib/libmpi.so.12 > > #5 0x00007fb12cf21d61 in PMPI_Test () from /usr/lib/libmpi.so.12 > > #6 0x00007fb127a5b939 in pmpi_test__ () from /usr/lib/libmpi_mpifh.so.12 > > #7 0x00007fb132888d87 in dmumps_try_recvtreat (comm_load=8, > ass_irecv=40, blocking=.FALSE., set_irecv=.TRUE., message_received=.FALSE., > msgsou=-1, msgtag=-1, status=..., bufr=..., lbufr=401408, > lbufr_bytes=1605629, procnode_steps=..., posfac=410095, iwpos=3151, > iwposcb=30557, > > iptrlu=1536548, lrlu=1126454, lrlus=2864100, n=30675, iw=..., > liw=39935, a=..., la=3367108, ptrist=..., ptlust=..., ptrfac=..., > ptrast=..., step=..., pimaster=..., pamaster=..., nstk_s=..., comp=0, > iflag=0, ierror=0, comm=7, nbprocfils=..., ipool=..., lpool=48, leaf=2, > > nbfin=90, myid=33, slavef=90, root=..., opassw=353031, > opeliw=700399235, itloc=..., rhs_mumps=..., fils=..., ptrarw=..., > ptraiw=..., intarr=..., dblarr=..., icntl=..., keep=..., keep8=..., > dkeep=..., nd=..., frere=..., lptrar=30675, nelt=1, frtptr=..., frtelt=..., > > istep_to_iniv2=..., tab_pos_in_pere=..., > stack_right_authorized=.TRUE., lrgroups=...) at dfac_process_message.F:646 > > #8 0x00007fb1328cfcd1 in dmumps_fac_par_m::dmumps_fac_par (n=30675, > iw=..., liw=39935, a=..., la=3367108, nstk_steps=..., nbprocfils=..., > nd=..., fils=..., step=..., frere=..., dad=..., cand=..., > istep_to_iniv2=..., tab_pos_in_pere=..., maxfrt=0, ntotpv=0, nmaxnpiv=150, > > ptrist=..., ptrast=..., pimaster=..., pamaster=..., ptrarw=..., > ptraiw=..., itloc=..., rhs_mumps=..., ipool=..., lpool=48, rinfo=..., > posfac=410095, iwpos=3151, lrlu=1126454, iptrlu=1536548, lrlus=2864100, > leaf=2, nbroot=1, nbrtot=90, uu=0.01, icntl=..., ptlust=..., ptrfac=..., > > nsteps=1, info=..., keep=..., keep8=..., procnode_steps=..., > slavef=90, myid=33, comm_nodes=7, myid_nodes=33, bufr=..., lbufr=401408, > lbufr_bytes=1605629, intarr=..., dblarr=..., root=..., perm=..., nelt=1, > frtptr=..., frtelt=..., lptrar=30675, comm_load=8, ass_irecv=40, > > seuil=0, seuil_ldlt_niv2=0, mem_distrib=..., ne=..., dkeep=..., > pivnul_list=..., lpn_list=1, lrgroups=...) at dfac_par_m.F:207 > > #9 0x00007fb13287f875 in dmumps_fac_b (n=30675, nsteps=1, a=..., > la=3367108, iw=..., liw=39935, sym_perm=..., na=..., lna=47, ne_steps=..., > nfsiz=..., fils=..., step=..., frere=..., dad=..., cand=..., > istep_to_iniv2=..., tab_pos_in_pere=..., ptrar=..., ldptrar=30675, > ptrist=..., > > ptlust_s=..., ptrfac=..., iw1=..., iw2=..., itloc=..., > rhs_mumps=..., pool=..., lpool=48, cntl1=0.01, icntl=..., info=..., > rinfo=..., keep=..., keep8=..., procnode_steps=..., slavef=90, > comm_nodes=7, myid=33, myid_nodes=33, bufr=..., lbufr=401408, > lbufr_bytes=1605629, > > intarr=..., dblarr=..., root=..., nelt=1, frtptr=..., frtelt=..., > comm_load=8, ass_irecv=40, seuil=0, seuil_ldlt_niv2=0, mem_distrib=..., > dkeep=..., pivnul_list=..., lpn_list=1, lrgroups=...) at dfac_b.F:167 > > #10 0x00007fb1328419ed in dmumps_fac_driver (id= value requires 600640 bytes, which is more than max-value-size>) at > dfac_driver.F:2291 > > #11 0x00007fb1327ff6dc in dmumps (id= requires 600640 bytes, which is more than max-value-size>) at > dmumps_driver.F:1686 > > #12 0x00007fb1327faf0a in dmumps_f77 (job=2, sym=0, par=1, comm_f77=5, > n=30675, icntl=..., cntl=..., keep=..., dkeep=..., keep8=..., nz=0, nnz=0, > irn=..., irnhere=0, jcn=..., jcnhere=0, a=..., ahere=0, nz_loc=622296, > nnz_loc=0, irn_loc=..., irn_lochere=1, jcn_loc=..., > > jcn_lochere=1, a_loc=..., a_lochere=1, nelt=0, eltptr=..., > eltptrhere=0, eltvar=..., eltvarhere=0, a_elt=..., a_elthere=0, > perm_in=..., perm_inhere=0, rhs=..., rhshere=0, redrhs=..., redrhshere=0, > info=..., rinfo=..., infog=..., rinfog=..., deficiency=0, lwk_user=0, > > size_schur=0, listvar_schur=..., listvar_schurhere=0, schur=..., > schurhere=0, wk_user=..., wk_userhere=0, colsca=..., colscahere=0, > rowsca=..., rowscahere=0, instance_number=1, nrhs=1, lrhs=0, lredrhs=0, > rhs_sparse=..., rhs_sparsehere=0, sol_loc=..., sol_lochere=0, > > irhs_sparse=..., irhs_sparsehere=0, irhs_ptr=..., irhs_ptrhere=0, > isol_loc=..., isol_lochere=0, nz_rhs=0, lsol_loc=0, schur_mloc=0, > schur_nloc=0, schur_lld=0, mblock=0, nblock=0, nprow=0, npcol=0, > ooc_tmpdir=..., ooc_prefix=..., write_problem=..., tmpdirlen=20, > prefixlen=20, > > write_problemlen=20) at dmumps_f77.F:267 > > #13 0x00007fb1327f9cfa in dmumps_c (mumps_par=mumps_par at entry=0x12bd9660) > at mumps_c.c:417 > > #14 0x00007fb1321a23fc in MatFactorNumeric_MUMPS (F=0x12bd8b60, > A=0x26bd890, info=) at /home/buildslave/software/pets > c-src/src/mat/impls/aij/mpi/mumps/mumps.c:1073 > > #15 0x00007fb131ec6ea7 in MatLUFactorNumeric (fact=0x12bd8b60, > mat=0x26bd890, info=info at entry=0xc2a66f8) at > /home/buildslave/software/petsc-src/src/mat/interface/matrix.c:3025 > > #16 0x00007fb1325040d6 in PCSetUp_LU (pc=0xc2a6380) at > /home/buildslave/software/petsc-src/src/ksp/pc/impls/factor/lu/lu.c:131 > > #17 0x00007fb13259903e in PCSetUp (pc=0xc2a6380) at > /home/buildslave/software/petsc-src/src/ksp/pc/interface/precon.c:923 > > #18 0x00007fb13263e53f in KSPSetUp (ksp=ksp at entry=0x12b28c70) at > /home/buildslave/software/petsc-src/src/ksp/ksp/interface/itfunc.c:381 > > #19 0x00007fb13263ed36 in KSPSolve (ksp=0x12b28c70, b=0xad77d50, > x=0xad801c0) at /home/buildslave/software/pets > c-src/src/ksp/ksp/interface/itfunc.c:612 > > #20 0x00007fb12db5dfc2 in libMesh::PetscLinearSolver ble>::solve(libMesh::SparseMatrix&, libMesh::SparseMatrix&, > libMesh::NumericVector&, libMesh::NumericVector&, double, > unsigned int) () > > from /mnt/fileserver/akselos-4.2.x/scrbe/build/bin/../../third_pa > rty/opt_real/libmesh_opt.so.0 > > #21 0x00007fb1338d0c06 in libMesh::PetscLinearSolver ble>::solve(libMesh::SparseMatrix&, libMesh::NumericVector&, > libMesh::NumericVector&, double, unsigned int) () from > /mnt/fileserver/akselos-4.2.x/scrbe/build/bin/../lib/libscrbe-opt_real.so > > #22 0x00007fb1335e8abd in std::pair > SolveHelper::try_linear_solve > >(libMesh::LinearSolver&, libMesh::SolverConfiguration&, > libMesh::SparseMatrix&, libMesh::NumericVector&, > libMesh::NumericVector&) () > > from /mnt/fileserver/akselos-4.2.x/scrbe/build/bin/../lib/libscrb > e-opt_real.so > > #23 0x00007fb133a70206 in > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gregory.meyer at gmail.com Thu Jun 28 15:53:29 2018 From: gregory.meyer at gmail.com (Greg Meyer) Date: Thu, 28 Jun 2018 13:53:29 -0700 Subject: [petsc-users] Solve for all repeated eigenvalues Message-ID: Hi all, I'm using SLEPc to solve for some eigenvalues. If I ask for the eigenvalues with the leftmost real part (using SMALLEST_REAL), and there are repeated eigenvalues, it appears that the solver does not always find all of the repeated values before jumping up to the next unique eigenvalue. For example if the lowest eigenvalues of my matrix are [-12, -10, -10, -10, -8], and I request 4 eigenvalues, I may get [-12, -10, -10, -8]. I'm using the default solver. Is there any way, or any solver, that ensures that no eigenvalues are skipped? Thanks in advance, Greg -------------- next part -------------- An HTML attachment was scrubbed... URL: From gregory.meyer at gmail.com Thu Jun 28 15:58:53 2018 From: gregory.meyer at gmail.com (Greg Meyer) Date: Thu, 28 Jun 2018 13:58:53 -0700 Subject: [petsc-users] Solve for all repeated eigenvalues In-Reply-To: References: Message-ID: (by the way this is a standard Hermitian eigenvalue problem) On Thu, Jun 28, 2018 at 1:53 PM, Greg Meyer wrote: > Hi all, > > I'm using SLEPc to solve for some eigenvalues. If I ask for the > eigenvalues with the leftmost real part (using SMALLEST_REAL), and there > are repeated eigenvalues, it appears that the solver does not always find > all of the repeated values before jumping up to the next unique eigenvalue. > For example if the lowest eigenvalues of my matrix are [-12, -10, -10, -10, > -8], and I request 4 eigenvalues, I may get [-12, -10, -10, -8]. I'm using > the default solver. Is there any way, or any solver, that ensures that no > eigenvalues are skipped? > > Thanks in advance, > Greg > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Thu Jun 28 16:19:56 2018 From: hzhang at mcs.anl.gov (Hong) Date: Thu, 28 Jun 2018 16:19:56 -0500 Subject: [petsc-users] Solve for all repeated eigenvalues In-Reply-To: References: Message-ID: Greg: Find out what default solver, convergence tolerance are used. For duplicate/clustered eigenvalues, you need strong eigensolver. Hong > (by the way this is a standard Hermitian eigenvalue problem) > > On Thu, Jun 28, 2018 at 1:53 PM, Greg Meyer > wrote: > >> Hi all, >> >> I'm using SLEPc to solve for some eigenvalues. If I ask for the >> eigenvalues with the leftmost real part (using SMALLEST_REAL), and there >> are repeated eigenvalues, it appears that the solver does not always find >> all of the repeated values before jumping up to the next unique eigenvalue. >> For example if the lowest eigenvalues of my matrix are [-12, -10, -10, -10, >> -8], and I request 4 eigenvalues, I may get [-12, -10, -10, -8]. I'm using >> the default solver. Is there any way, or any solver, that ensures that no >> eigenvalues are skipped? >> >> Thanks in advance, >> Greg >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Thu Jun 28 16:21:50 2018 From: jroman at dsic.upv.es (Jose E. Roman) Date: Thu, 28 Jun 2018 23:21:50 +0200 Subject: [petsc-users] Solve for all repeated eigenvalues In-Reply-To: References: Message-ID: This is a known issue of Krylov solvers. Sometimes you can workaround it by increasing the number of eigenvalues to compute (nev, for instance request twice as much as you need) and maybe reduce the mpd parameter to force more restarts. This approach is quite wasteful and still there is no guarantee that you get the full multiplicities right. To avoid the issue completely, use the lobpcg solver. However, SLEPc's implementation of lobpcg is not very well tuned, I have to take some time to improve it. You can also try the BLOPEX interface (configure --download-blopex in SLEPc). Jose > El 28 jun 2018, a las 22:53, Greg Meyer escribi?: > > Hi all, > > I'm using SLEPc to solve for some eigenvalues. If I ask for the eigenvalues with the leftmost real part (using SMALLEST_REAL), and there are repeated eigenvalues, it appears that the solver does not always find all of the repeated values before jumping up to the next unique eigenvalue. For example if the lowest eigenvalues of my matrix are [-12, -10, -10, -10, -8], and I request 4 eigenvalues, I may get [-12, -10, -10, -8]. I'm using the default solver. Is there any way, or any solver, that ensures that no eigenvalues are skipped? > > Thanks in advance, > Greg From vinh.phamxuan2 at mail.dcu.ie Thu Jun 28 16:51:46 2018 From: vinh.phamxuan2 at mail.dcu.ie (Vinh Pham-Xuan) Date: Thu, 28 Jun 2018 23:51:46 +0200 Subject: [petsc-users] Solve for all repeated eigenvalues In-Reply-To: References: Message-ID: You may try the contour integral method (CISS). Obviously, it is computationally more expensive. I used the contour integral method on MATLAB and it worked quite well. I expected the same result with SLEPc. On Thu, 28 Jun 2018 at 23:22, Jose E. Roman wrote: > This is a known issue of Krylov solvers. Sometimes you can workaround it > by increasing the number of eigenvalues to compute (nev, for instance > request twice as much as you need) and maybe reduce the mpd parameter to > force more restarts. This approach is quite wasteful and still there is no > guarantee that you get the full multiplicities right. > > To avoid the issue completely, use the lobpcg solver. However, SLEPc's > implementation of lobpcg is not very well tuned, I have to take some time > to improve it. You can also try the BLOPEX interface (configure > --download-blopex in SLEPc). > > Jose > > > > El 28 jun 2018, a las 22:53, Greg Meyer > escribi?: > > > > Hi all, > > > > I'm using SLEPc to solve for some eigenvalues. If I ask for the > eigenvalues with the leftmost real part (using SMALLEST_REAL), and there > are repeated eigenvalues, it appears that the solver does not always find > all of the repeated values before jumping up to the next unique eigenvalue. > For example if the lowest eigenvalues of my matrix are [-12, -10, -10, -10, > -8], and I request 4 eigenvalues, I may get [-12, -10, -10, -8]. I'm using > the default solver. Is there any way, or any solver, that ensures that no > eigenvalues are skipped? > > > > Thanks in advance, > > Greg > > -- Vinh Pham-Xuan, Ph.D School of Electronic Engineering Dublin City University Ireland -- __ S?anadh R?omhphoist/_ Email?Disclaimer__ ** T? an r?omhphost seo agus aon chomhad a sheoltar leis faoi r?n agus is lena ?s?id ag an seola? agus sin amh?in ?.?Is f?idir tuilleadh a l?amh anseo. ? * _ This e-mail and any files transmitted with it are confidential and are intended solely for use by the addressee.?Read more here. ?_ *_ -------------- next part -------------- An HTML attachment was scrubbed... URL: From skavou1 at lsu.edu Thu Jun 28 17:11:55 2018 From: skavou1 at lsu.edu (Sepideh Kavousi) Date: Thu, 28 Jun 2018 22:11:55 +0000 Subject: [petsc-users] ghoat point in DMDATSSetIFunctionLocal Message-ID: Hello, I am solving Cahn-Hillard equation where I have Laplacian in my equation. In order to discretize the Laplacian, I can discretize with 5-point or 9-point stencil. 5 point stencile: laplacian=(aY[j][i+1].p+aY[j][i-1].p-2.0*aY[j][i].p)/(hx2))+((aY[j+1][i].p+aY[j-1][i].p-2.0*aY[j][i].p)/hy2) 9-point stencil:(4.0*(aY[j][i+1].p+aY[j][i-1].p+aY[j+1][i].p+aY[j-1][i].p)+1.0*(aY[j+1][i+1].p+aY[j+1][i-1].p+aY[j-1][i+1].p+aY[j-1][i-1].p)-20.0*aY[j][i].p)/(6.0*hx2); I use DMDATSSetIFunctionLocal. In the function definition, if I use 5 point stencil, everything works fine, if I use a 9-point stencil, at the points which are in common between processors the answers behave strange and wrong. while for 9-point stencil, I use single processor everything works fine. Assume I have 4 processors and the domain that belongs to each processor is: (0:nx , 0:ny) process one (nx+1:2nx , 0:ny) process two (0:nx , ny+1:2*ny) process three (nx+1:2nx , ny+1:2*n) process four I guess the problem is when processor one tries to access point (nx+1,ny+1). I did not find any data if the definition of local vector includes the point (nx+1,ny+1) for this processor or not. I would like to see how should I solve this issue. Regards, Sepideh -------------- next part -------------- An HTML attachment was scrubbed... URL: From keceli at gmail.com Thu Jun 28 17:15:36 2018 From: keceli at gmail.com (=?UTF-8?Q?murat_ke=C3=A7eli?=) Date: Thu, 28 Jun 2018 17:15:36 -0500 Subject: [petsc-users] Solve for all repeated eigenvalues In-Reply-To: References: Message-ID: How about spectrum-slicing (i.e. -eps_interval -13,-7 -st_type sinvert -st_ksp_type preonly -st_pc_type cholesky)? I guess you'd need real symmetric matrices though. On Thu, Jun 28, 2018 at 4:51 PM, Vinh Pham-Xuan wrote: > You may try the contour integral method (CISS). Obviously, it is > computationally more expensive. I used the contour integral method on MATLAB > and it worked quite well. I expected the same result with SLEPc. > > On Thu, 28 Jun 2018 at 23:22, Jose E. Roman wrote: >> >> This is a known issue of Krylov solvers. Sometimes you can workaround it >> by increasing the number of eigenvalues to compute (nev, for instance >> request twice as much as you need) and maybe reduce the mpd parameter to >> force more restarts. This approach is quite wasteful and still there is no >> guarantee that you get the full multiplicities right. >> >> To avoid the issue completely, use the lobpcg solver. However, SLEPc's >> implementation of lobpcg is not very well tuned, I have to take some time to >> improve it. You can also try the BLOPEX interface (configure >> --download-blopex in SLEPc). >> >> Jose >> >> >> > El 28 jun 2018, a las 22:53, Greg Meyer >> > escribi?: >> > >> > Hi all, >> > >> > I'm using SLEPc to solve for some eigenvalues. If I ask for the >> > eigenvalues with the leftmost real part (using SMALLEST_REAL), and there are >> > repeated eigenvalues, it appears that the solver does not always find all of >> > the repeated values before jumping up to the next unique eigenvalue. For >> > example if the lowest eigenvalues of my matrix are [-12, -10, -10, -10, -8], >> > and I request 4 eigenvalues, I may get [-12, -10, -10, -8]. I'm using the >> > default solver. Is there any way, or any solver, that ensures that no >> > eigenvalues are skipped? >> > >> > Thanks in advance, >> > Greg >> > > > -- > Vinh Pham-Xuan, Ph.D > School of Electronic Engineering > Dublin City University > Ireland > > S?anadh R?omhphoist/ > > Email Disclaimer > > T? an r?omhphost seo agus aon chomhad a sheoltar leis faoi r?n agus is lena > ?s?id ag an seola? agus sin amh?in ?. Is f?idir tuilleadh a l?amh anseo. > > This e-mail and any files transmitted with it are confidential and are > intended solely for use by the addressee. Read more here. From vinh.phamxuan2 at mail.dcu.ie Thu Jun 28 17:30:11 2018 From: vinh.phamxuan2 at mail.dcu.ie (Vinh Pham-Xuan) Date: Fri, 29 Jun 2018 00:30:11 +0200 Subject: [petsc-users] Solve for all repeated eigenvalues In-Reply-To: References: Message-ID: I do not know about -st_type but the other options are ok (I guess). With these options, you are looking for eigenvalues on the real axis between [-13, -7]. Each linear system involved in the contour integral method is solved by using cholesky factorization. CISS in SLEPC can also find complex-valued eigenvalues which are inside a rectangle or ellipse. Therefore, the matrix is not restricted to real and symmetric. On Fri, 29 Jun 2018 at 00:15, murat ke?eli wrote: > How about spectrum-slicing (i.e. -eps_interval -13,-7 -st_type sinvert > -st_ksp_type preonly -st_pc_type cholesky)? I guess you'd need real > symmetric matrices though. > > On Thu, Jun 28, 2018 at 4:51 PM, Vinh Pham-Xuan > wrote: > > You may try the contour integral method (CISS). Obviously, it is > > computationally more expensive. I used the contour integral method on > MATLAB > > and it worked quite well. I expected the same result with SLEPc. > > > > On Thu, 28 Jun 2018 at 23:22, Jose E. Roman wrote: > >> > >> This is a known issue of Krylov solvers. Sometimes you can workaround it > >> by increasing the number of eigenvalues to compute (nev, for instance > >> request twice as much as you need) and maybe reduce the mpd parameter to > >> force more restarts. This approach is quite wasteful and still there is > no > >> guarantee that you get the full multiplicities right. > >> > >> To avoid the issue completely, use the lobpcg solver. However, SLEPc's > >> implementation of lobpcg is not very well tuned, I have to take some > time to > >> improve it. You can also try the BLOPEX interface (configure > >> --download-blopex in SLEPc). > >> > >> Jose > >> > >> > >> > El 28 jun 2018, a las 22:53, Greg Meyer > >> > escribi?: > >> > > >> > Hi all, > >> > > >> > I'm using SLEPc to solve for some eigenvalues. If I ask for the > >> > eigenvalues with the leftmost real part (using SMALLEST_REAL), and > there are > >> > repeated eigenvalues, it appears that the solver does not always find > all of > >> > the repeated values before jumping up to the next unique eigenvalue. > For > >> > example if the lowest eigenvalues of my matrix are [-12, -10, -10, > -10, -8], > >> > and I request 4 eigenvalues, I may get [-12, -10, -10, -8]. I'm using > the > >> > default solver. Is there any way, or any solver, that ensures that no > >> > eigenvalues are skipped? > >> > > >> > Thanks in advance, > >> > Greg > >> > > > > > > -- > > Vinh Pham-Xuan, Ph.D > > School of Electronic Engineering > > Dublin City University > > Ireland > > > > S?anadh R?omhphoist/ > > > > Email Disclaimer > > > > T? an r?omhphost seo agus aon chomhad a sheoltar leis faoi r?n agus is > lena > > ?s?id ag an seola? agus sin amh?in ?. Is f?idir tuilleadh a l?amh anseo. > > > > This e-mail and any files transmitted with it are confidential and are > > intended solely for use by the addressee. Read more here. > -- Vinh Pham-Xuan, Ph.D School of Electronic Engineering Dublin City University Ireland -- __ S?anadh R?omhphoist/_ Email?Disclaimer__ ** T? an r?omhphost seo agus aon chomhad a sheoltar leis faoi r?n agus is lena ?s?id ag an seola? agus sin amh?in ?.?Is f?idir tuilleadh a l?amh anseo. ? * _ This e-mail and any files transmitted with it are confidential and are intended solely for use by the addressee.?Read more here. ?_ *_ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Jun 28 18:50:44 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 28 Jun 2018 23:50:44 +0000 Subject: [petsc-users] ghoat point in DMDATSSetIFunctionLocal In-Reply-To: References: Message-ID: <75BD0A70-444E-4F83-B8F8-B0E2681DB349@anl.gov> When you create the DMDA there is a "stencil" argument that can be star or box. For the 9 pt stencil you need to specify box. Barry > On Jun 28, 2018, at 5:11 PM, Sepideh Kavousi wrote: > > Hello, > I am solving Cahn-Hillard equation where I have Laplacian in my equation. In order to discretize the Laplacian, I can discretize with 5-point or 9-point stencil. > 5 point stencile: laplacian=(aY[j][i+1].p+aY[j][i-1].p-2.0*aY[j][i].p)/(hx2))+((aY[j+1][i].p+aY[j-1][i].p-2.0*aY[j][i].p)/hy2) > > 9-point stencil:(4.0*(aY[j][i+1].p+aY[j][i-1].p+aY[j+1][i].p+aY[j-1][i].p)+1.0*(aY[j+1][i+1].p+aY[j+1][i-1].p+aY[j-1][i+1].p+aY[j-1][i-1].p)-20.0*aY[j][i].p)/(6.0*hx2); > > > I use DMDATSSetIFunctionLocal. In the function definition, if I use 5 point stencil, everything works fine, if I use a 9-point stencil, at the points which are in common between processors the answers behave strange and wrong. while for 9-point stencil, I use single processor everything works fine. > Assume I have 4 processors and the domain that belongs to each processor is: > (0:nx , 0:ny) process one > (nx+1:2nx , 0:ny) process two > (0:nx , ny+1:2*ny) process three > (nx+1:2nx , ny+1:2*n) process four > > I guess the problem is when processor one tries to access point (nx+1,ny+1). I did not find any data if the definition of local vector includes the point (nx+1,ny+1) for this processor or not. I would like to see how should I solve this issue. > > Regards, > Sepideh From lei.wang at durham.ac.uk Fri Jun 29 06:07:10 2018 From: lei.wang at durham.ac.uk (WANG, LEI) Date: Fri, 29 Jun 2018 11:07:10 +0000 Subject: [petsc-users] how to exit in SNES iteration Message-ID: Dear all I am writing a code for nonlinear mechanical problem with SNES. The code often stops inside of the iterations when computing the Jacobian or Function. I would like to force the code exit once some abnormal value appears, but do not know what function to use? I have tried to set the residual norm to a large value, iteration number to a large value, but seems those not work. What should I write in 'if(abnormal) ???' for exiting SNES iteration in the functions FormJacobian or FormFunction? Cheers Lei -------------- next part -------------- An HTML attachment was scrubbed... URL: From Hassan.Raiesi at aero.bombardier.com Fri Jun 29 09:48:58 2018 From: Hassan.Raiesi at aero.bombardier.com (Hassan Raiesi) Date: Fri, 29 Jun 2018 14:48:58 +0000 Subject: [petsc-users] dmplex confusion in refereeing to stratum values (are they height or depth) Message-ID: Hi, I'm trying to understand the dmplex interface, and hope somebody could clarify what are the values of stratum used, i.e in DMLabel, and many other DMplex functions. Is it refereeing to the stratum height or the stratum depth by "value", or none of them, for example in DMPlexSetLabelValue, the value refers to depth or height, same is for several other places, Thank you -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Jun 29 10:24:51 2018 From: jed at jedbrown.org (Jed Brown) Date: Fri, 29 Jun 2018 09:24:51 -0600 Subject: [petsc-users] how to exit in SNES iteration In-Reply-To: References: Message-ID: <87fu1539a4.fsf@jedbrown.org> See SNESSetFunctionDomainError() "WANG, LEI" writes: > Dear all > > I am writing a code for nonlinear mechanical problem with SNES. The code often stops inside of the iterations when computing the Jacobian or Function. I would like to force the code exit once some abnormal value appears, but do not know what function to use? I have tried to set the residual norm to a large value, iteration number to a large value, but seems those not work. What should I write in 'if(abnormal) ???' for exiting SNES iteration in the functions FormJacobian or FormFunction? > > Cheers > Lei From lei.wang at durham.ac.uk Fri Jun 29 10:49:03 2018 From: lei.wang at durham.ac.uk (WANG, LEI) Date: Fri, 29 Jun 2018 15:49:03 +0000 Subject: [petsc-users] how to exit in SNES iteration In-Reply-To: <87fu1539a4.fsf@jedbrown.org> References: <87fu1539a4.fsf@jedbrown.org> Message-ID: This is just what I need. Thanks, Jed. Cheers Lei -----Original Message----- From: Jed Brown [mailto:jed at jedbrown.org] Sent: 29 June 2018 16:25 To: WANG, LEI; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] how to exit in SNES iteration See SNESSetFunctionDomainError() "WANG, LEI" writes: > Dear all > > I am writing a code for nonlinear mechanical problem with SNES. The code often stops inside of the iterations when computing the Jacobian or Function. I would like to force the code exit once some abnormal value appears, but do not know what function to use? I have tried to set the residual norm to a large value, iteration number to a large value, but seems those not work. What should I write in 'if(abnormal) ???' for exiting SNES iteration in the functions FormJacobian or FormFunction? > > Cheers > Lei From alexlindsay239 at gmail.com Fri Jun 29 17:06:17 2018 From: alexlindsay239 at gmail.com (Alexander Lindsay) Date: Fri, 29 Jun 2018 16:06:17 -0600 Subject: [petsc-users] "snes_type test" is gone? In-Reply-To: References: <9F7A97FC-EBC7-4F20-ADA6-4D03B145DFB0@anl.gov> Message-ID: Is there a way to mimic the behavior of `-snes_type test` with new PETSc? E.g. don't attempt to perform any solves? They're stupid but we have a bunch of tests in MOOSE that are set-up only to test the Jacobian, and any attempts to actually solve the system are disastrous. I can hack around this by doing things like setting the convergence reason to `SNES_CONVERGED_ITS` and similarly for ksp, making sure I use `-pc_type gamg` since there are missing Jacobian diagonal entries, etc. but this doesn't seem ideal. Alex On Mon, Jun 4, 2018 at 4:26 PM, Kong, Fande wrote: > Thanks, Hong, > > I see. It is better if "-snes_type test" did not exist at the first > place. > > > Fande, > > On Mon, Jun 4, 2018 at 4:01 PM, Zhang, Hong wrote: > >> >> >> > On Jun 4, 2018, at 4:59 PM, Zhang, Hong wrote: >> > >> > -snes_type has been removed. We can just use -snes_test_jacobian >> instead. Note that the test is done every time the Jacobian is computed. >> >> It was meant to be "-snes_type test". >> >> > Hong (Mr.) >> > >> >> On Jun 4, 2018, at 3:27 PM, Kong, Fande wrote: >> >> >> >> Hi PETSc Team, >> >> >> >> I was wondering if "snes_type test" has been gone? Quite a few MOOSE >> users use this option to test their Jacobian matrices. >> >> >> >> If it is gone, any reason? >> >> >> >> Fande, >> > >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Jun 29 17:16:51 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 29 Jun 2018 22:16:51 +0000 Subject: [petsc-users] "snes_type test" is gone? In-Reply-To: References: <9F7A97FC-EBC7-4F20-ADA6-4D03B145DFB0@anl.gov> Message-ID: <3C655311-8596-4AE9-82AD-2441573303A3@anl.gov> I think you need to write your own tester, possibly by taking code from the previous release of PETSc where we supported -snes_type test. Barry Note that the new code does a much more comprehensive testing of the Jacobian. > On Jun 29, 2018, at 5:06 PM, Alexander Lindsay wrote: > > Is there a way to mimic the behavior of `-snes_type test` with new PETSc? E.g. don't attempt to perform any solves? They're stupid but we have a bunch of tests in MOOSE that are set-up only to test the Jacobian, and any attempts to actually solve the system are disastrous. I can hack around this by doing things like setting the convergence reason to `SNES_CONVERGED_ITS` and similarly for ksp, making sure I use `-pc_type gamg` since there are missing Jacobian diagonal entries, etc. but this doesn't seem ideal. > > Alex > > On Mon, Jun 4, 2018 at 4:26 PM, Kong, Fande wrote: > Thanks, Hong, > > I see. It is better if "-snes_type test" did not exist at the first place. > > > Fande, > > On Mon, Jun 4, 2018 at 4:01 PM, Zhang, Hong wrote: > > > > On Jun 4, 2018, at 4:59 PM, Zhang, Hong wrote: > > > > -snes_type has been removed. We can just use -snes_test_jacobian instead. Note that the test is done every time the Jacobian is computed. > > It was meant to be "-snes_type test". > > > Hong (Mr.) > > > >> On Jun 4, 2018, at 3:27 PM, Kong, Fande wrote: > >> > >> Hi PETSc Team, > >> > >> I was wondering if "snes_type test" has been gone? Quite a few MOOSE users use this option to test their Jacobian matrices. > >> > >> If it is gone, any reason? > >> > >> Fande, > > > > > From dalcinl at gmail.com Sat Jun 30 05:28:48 2018 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sat, 30 Jun 2018 13:28:48 +0300 Subject: [petsc-users] "snes_type test" is gone? In-Reply-To: References: <9F7A97FC-EBC7-4F20-ADA6-4D03B145DFB0@anl.gov> Message-ID: ./example.exe -snes_test_jacobian -snes_type ksponly -ksp_type preonly -pc_type none -snes_convergence_test skip On Sat, 30 Jun 2018 at 01:06, Alexander Lindsay wrote: > > Is there a way to mimic the behavior of `-snes_type test` with new PETSc? E.g. don't attempt to perform any solves? They're stupid but we have a bunch of tests in MOOSE that are set-up only to test the Jacobian, and any attempts to actually solve the system are disastrous. I can hack around this by doing things like setting the convergence reason to `SNES_CONVERGED_ITS` and similarly for ksp, making sure I use `-pc_type gamg` since there are missing Jacobian diagonal entries, etc. but this doesn't seem ideal. > > Alex > > On Mon, Jun 4, 2018 at 4:26 PM, Kong, Fande wrote: >> >> Thanks, Hong, >> >> I see. It is better if "-snes_type test" did not exist at the first place. >> >> >> Fande, >> >> On Mon, Jun 4, 2018 at 4:01 PM, Zhang, Hong wrote: >>> >>> >>> >>> > On Jun 4, 2018, at 4:59 PM, Zhang, Hong wrote: >>> > >>> > -snes_type has been removed. We can just use -snes_test_jacobian instead. Note that the test is done every time the Jacobian is computed. >>> >>> It was meant to be "-snes_type test". >>> >>> > Hong (Mr.) >>> > >>> >> On Jun 4, 2018, at 3:27 PM, Kong, Fande wrote: >>> >> >>> >> Hi PETSc Team, >>> >> >>> >> I was wondering if "snes_type test" has been gone? Quite a few MOOSE users use this option to test their Jacobian matrices. >>> >> >>> >> If it is gone, any reason? >>> >> >>> >> Fande, >>> > >>> >> > -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 0109 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 From knepley at gmail.com Sat Jun 30 07:14:59 2018 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 30 Jun 2018 07:14:59 -0500 Subject: [petsc-users] dmplex confusion in refereeing to stratum values (are they height or depth) In-Reply-To: References: Message-ID: On Fri, Jun 29, 2018 at 9:49 AM Hassan Raiesi < Hassan.Raiesi at aero.bombardier.com> wrote: > Hi, > > > > I?m trying to understand the dmplex interface, and hope somebody could > clarify what are the values of stratum used, i.e in DMLabel, and many other > DMplex functions. > > Is it refereeing to the stratum height or the stratum depth by ?value?, or > none of them, for example in > > DMPlexSetLabelValue, the value refers to depth or height, same is for > several other places, > DMLabel maps an integer to a set of mesh points. These values can be anything. A stratum is the set of points with the same value. Some specific labels attach meaning to the values. For example, the depth label inside a Plex marks the depth of each mesh point, so that we can support DMPlexGetDepthStratum() DMPlexGetHeightStratum() Thanks, Matt Thank you > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: