From paololampitella at hotmail.com Wed Feb 1 09:21:46 2017 From: paololampitella at hotmail.com (Paolo Lampitella) Date: Wed, 1 Feb 2017 15:21:46 +0000 Subject: [petsc-users] Best workflow for different systems with different block sizes Message-ID: Dear PETSc users and developers, I successfully, and very satisfactorily, use PETSc for the linear algebra part of a coupled, preconditioned, density-based, unstructured, cell-centered, finite volume CFD code. In short, the method is the one presented in the paper: J.M. Weiss, J.P. Maruszewski, W.A. Smith: Implicit Solution of Preconditioned Navier-Stokes Equations Using Algebraic Multigrid http://dx.doi.org/10.2514/2.689 except for the AMG part as, at the moment, I use GMRES + point Jacobi trough PETSc (this will probably be the subject for another post). However, I also have a very simple, internally coded, symmetric block Gauss-Seidel solver. The code, written in Fortran with MPI, manages all the aspects of the solver, including outer iterations, with PETSc just handling the resolution of the linear systems at each outer iteration. In particular, note that, for certain combination of models, the solver can end up having different systems of equations to solve, in sequence, at each outer iteration. For example, I might have: - Main equations (with block size = 5 + n species) - Turbulence equations (with block size = number of equations implied by the turbulence model) - Additional Transported Scalar Equations (with block size = number of required scalars) - Etc. The way I manage the workflow with the internal GS solver is such that, for each block of equations, at each outer iteration, I do the following: - allocate matrix, solution and rhs - fill the matrix and rhs - solve the system - update the independent variables (system solution is in delta form) - deallocate matrix, solution and rhs so that the allocated memory is kept as low as possible at any given time. However, for legacy reasons now obsolete, the PETSc workflow used in the code is different as all the required matrices and rhs are instead created in advance with the routine petsc_create.f90 in attachment. Then iterations start, and at each iteration, each system is solved with the routine petsc_solve.f90, also in attachment (both are included in a dedicated petsc module). At the end of the iterations, before the finalization, a petsc_destroy subroutine (not attached) is obviously also called for each matrix/rhs allocated. So, in conclusion, I keep in memory all the matrices for the whole time (the matrix structure, of course, doesn't change with the iterations). My first question then is: 1) Is this approach recommended? Wouldn't, instead, calling my petsc_create inside my petsc_solve be a better option in my case? In this case I could avoid storing any petsc matrix or rhs outside the petsc_solve routine. Would the overhead implied by the routines called in my petsc_create be sustainable if that subroutine is called at every outer iteration for every system? Also, note that the way I fill the RHS and the matrix of the systems for PETSc are different. For the RHS I always allocate mine in the code, which is then copied in the petsc one in the petsc_solve routine. For the matrix, instead, I directly fill the petsc one outside the subroutine, which is then passed already filled to petsc_solve. So, the second question is: 2) Independently from the approach at question (1), which method do you suggest to adopt? Storing directly the rhs and the matrix in the petsc ones Or copying them at the solve time? Is there a mainstream approach in such cases? I don't know if this has relevance but, consider that, in my case, every process always writes only its own matrix and rhs entries. Unfortunately, at the moment, I have no access to a system where a straightforward test would give a clear answer to this. Still, I guess, the matter is more conceptual than practical. Thank you and sorry for the long mail -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: petsc_solve.f90 Type: application/octet-stream Size: 1918 bytes Desc: petsc_solve.f90 URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: petsc_create.f90 Type: application/octet-stream Size: 1489 bytes Desc: petsc_create.f90 URL: From bsmith at mcs.anl.gov Wed Feb 1 10:52:41 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 1 Feb 2017 10:52:41 -0600 Subject: [petsc-users] Best workflow for different systems with different block sizes In-Reply-To: References: Message-ID: <1CD93494-BC83-402A-95DD-D226E27841BA@mcs.anl.gov> Paolo, If the various matrices keep the same nonzero structure throughout the simulation there is a slight performance gain to reusing them vs creating and destroying them in each outer iteration. Likely not more than say 10% of the run time if the linear solves are relatively difficult (i.e. most of the time is spent in the linear solves) but it could be a bit more maybe 20% if the linear solves are not dominating the time. I am not basing this on any quantitive information I have. So it really comes down to if you want to run larger problems that require reusing the memory. Regarding creating the right hand side in your memory and copying to PETSc. Creating the right hand side directly in PETSc vectors (with VecGetArrayF90()) will be a little faster and require slightly less memory so is a "good thing to do" but I suspect you will barely be able to measure the difference. Barry > On Feb 1, 2017, at 9:21 AM, Paolo Lampitella wrote: > > Dear PETSc users and developers, > > I successfully, and very satisfactorily, use PETSc for the linear algebra part of a coupled, preconditioned, density-based, > unstructured, cell-centered, finite volume CFD code. In short, the method is the one presented in the paper: > > J.M. Weiss, J.P. Maruszewski, W.A. Smith: Implicit Solution of Preconditioned Navier-Stokes Equations Using Algebraic Multigrid > http://dx.doi.org/10.2514/2.689 > > except for the AMG part as, at the moment, I use GMRES + point Jacobi trough PETSc (this will probably be the subject for another post). > However, I also have a very simple, internally coded, symmetric block Gauss-Seidel solver. > > The code, written in Fortran with MPI, manages all the aspects of the solver, including outer iterations, with PETSc just handling the > resolution of the linear systems at each outer iteration. In particular, note that, for certain combination of models, the solver > can end up having different systems of equations to solve, in sequence, at each outer iteration. For example, I might have: > > - Main equations (with block size = 5 + n species) > - Turbulence equations (with block size = number of equations implied by the turbulence model) > - Additional Transported Scalar Equations (with block size = number of required scalars) > - Etc. > > The way I manage the workflow with the internal GS solver is such that, for each block of equations, at each outer iteration, I do the following: > > - allocate matrix, solution and rhs > - fill the matrix and rhs > - solve the system > - update the independent variables (system solution is in delta form) > - deallocate matrix, solution and rhs > > so that the allocated memory is kept as low as possible at any given time. However, for legacy reasons now obsolete, the PETSc workflow used in the code > is different as all the required matrices and rhs are instead created in advance with the routine petsc_create.f90 in attachment. Then iterations start, > and at each iteration, each system is solved with the routine petsc_solve.f90, also in attachment (both are included in a dedicated petsc module). > At the end of the iterations, before the finalization, a petsc_destroy subroutine (not attached) is obviously also called for each matrix/rhs allocated. > So, in conclusion, I keep in memory all the matrices for the whole time (the matrix structure, of course, doesn?t change with the iterations). > My first question then is: > > 1) Is this approach recommended? Wouldn?t, instead, calling my petsc_create inside my petsc_solve be a better option in my case? > In this case I could avoid storing any petsc matrix or rhs outside the petsc_solve routine. > Would the overhead implied by the routines called in my petsc_create be sustainable if that subroutine is called at every outer iteration for every system? > > Also, note that the way I fill the RHS and the matrix of the systems for PETSc are different. For the RHS I always allocate mine in the code, which is then > copied in the petsc one in the petsc_solve routine. For the matrix, instead, I directly fill the petsc one outside the subroutine, > which is then passed already filled to petsc_solve. So, the second question is: > > 2) Independently from the approach at question (1), which method do you suggest to adopt? Storing directly the rhs and the matrix in the petsc ones > Or copying them at the solve time? Is there a mainstream approach in such cases? > > I don?t know if this has relevance but, consider that, in my case, every process always writes only its own matrix and rhs entries. > > Unfortunately, at the moment, I have no access to a system where a straightforward test would give a clear answer to this. Still, I guess, the matter > is more conceptual than practical. > > Thank you and sorry for the long mail > From cpraveen at gmail.com Wed Feb 1 23:46:42 2017 From: cpraveen at gmail.com (Praveen C) Date: Thu, 2 Feb 2017 11:16:42 +0530 Subject: [petsc-users] Usage of VecCreateMPIWithArray in fortran Message-ID: Dear all I am creating a vector in fortran like this PetscReal,allocatable :: res(:,:) Vec :: v_res allocate( res(nvar,nvl_nvg) ) call VecCreateMPIWithArray(PETSC_COMM_WORLD, nvar, nvar*nvl, & PETSC_DECIDE, res(1,1), v_res, & ierr); CHKERRQ(ierr) Note that my array "res" is larger than needed by the vector v_res. Is this ok to do this ? I use the extra "res" array for dummy ghost values, but I dont need v_res to be ghosted. Thanks praveen -------------- next part -------------- An HTML attachment was scrubbed... URL: From paololampitella at hotmail.com Thu Feb 2 02:06:40 2017 From: paololampitella at hotmail.com (Paolo Lampitella) Date: Thu, 2 Feb 2017 08:06:40 +0000 Subject: [petsc-users] R: Best workflow for different systems with different block sizes In-Reply-To: <1CD93494-BC83-402A-95DD-D226E27841BA@mcs.anl.gov> References: <1CD93494-BC83-402A-95DD-D226E27841BA@mcs.anl.gov> Message-ID: Barry, thank you very much. Hopefully, I will soon be able to come back with some real numbers on some real machine, to compare the two approaches. Paolo -----Messaggio originale----- Da: Barry Smith [mailto:bsmith at mcs.anl.gov] Inviato: mercoled? 1 febbraio 2017 17:53 A: Paolo Lampitella Cc: petsc-users at mcs.anl.gov Oggetto: Re: [petsc-users] Best workflow for different systems with different block sizes Paolo, If the various matrices keep the same nonzero structure throughout the simulation there is a slight performance gain to reusing them vs creating and destroying them in each outer iteration. Likely not more than say 10% of the run time if the linear solves are relatively difficult (i.e. most of the time is spent in the linear solves) but it could be a bit more maybe 20% if the linear solves are not dominating the time. I am not basing this on any quantitive information I have. So it really comes down to if you want to run larger problems that require reusing the memory. Regarding creating the right hand side in your memory and copying to PETSc. Creating the right hand side directly in PETSc vectors (with VecGetArrayF90()) will be a little faster and require slightly less memory so is a "good thing to do" but I suspect you will barely be able to measure the difference. Barry > On Feb 1, 2017, at 9:21 AM, Paolo Lampitella wrote: > > Dear PETSc users and developers, > > I successfully, and very satisfactorily, use PETSc for the linear > algebra part of a coupled, preconditioned, density-based, unstructured, cell-centered, finite volume CFD code. In short, the method is the one presented in the paper: > > J.M. Weiss, J.P. Maruszewski, W.A. Smith: Implicit Solution of > Preconditioned Navier-Stokes Equations Using Algebraic Multigrid > http://dx.doi.org/10.2514/2.689 > > except for the AMG part as, at the moment, I use GMRES + point Jacobi trough PETSc (this will probably be the subject for another post). > However, I also have a very simple, internally coded, symmetric block Gauss-Seidel solver. > > The code, written in Fortran with MPI, manages all the aspects of the > solver, including outer iterations, with PETSc just handling the > resolution of the linear systems at each outer iteration. In particular, note that, for certain combination of models, the solver can end up having different systems of equations to solve, in sequence, at each outer iteration. For example, I might have: > > - Main equations (with block size = 5 + n species) > - Turbulence equations (with block size = number of equations implied by the turbulence model) > - Additional Transported Scalar Equations (with block size = number of required scalars) > - Etc. > > The way I manage the workflow with the internal GS solver is such that, for each block of equations, at each outer iteration, I do the following: > > - allocate matrix, solution and rhs > - fill the matrix and rhs > - solve the system > - update the independent variables (system solution is in delta form) > - deallocate matrix, solution and rhs > > so that the allocated memory is kept as low as possible at any given > time. However, for legacy reasons now obsolete, the PETSc workflow > used in the code is different as all the required matrices and rhs are instead created in advance with the routine petsc_create.f90 in attachment. Then iterations start, and at each iteration, each system is solved with the routine petsc_solve.f90, also in attachment (both are included in a dedicated petsc module). > At the end of the iterations, before the finalization, a petsc_destroy subroutine (not attached) is obviously also called for each matrix/rhs allocated. > So, in conclusion, I keep in memory all the matrices for the whole time (the matrix structure, of course, doesn?t change with the iterations). > My first question then is: > > 1) Is this approach recommended? Wouldn?t, instead, calling my petsc_create inside my petsc_solve be a better option in my case? > In this case I could avoid storing any petsc matrix or rhs outside the petsc_solve routine. > Would the overhead implied by the routines called in my petsc_create be sustainable if that subroutine is called at every outer iteration for every system? > > Also, note that the way I fill the RHS and the matrix of the systems > for PETSc are different. For the RHS I always allocate mine in the > code, which is then copied in the petsc one in the petsc_solve routine. For the matrix, instead, I directly fill the petsc one outside the subroutine, which is then passed already filled to petsc_solve. So, the second question is: > > 2) Independently from the approach at question (1), which method do > you suggest to adopt? Storing directly the rhs and the matrix in the petsc ones Or copying them at the solve time? Is there a mainstream approach in such cases? > > I don?t know if this has relevance but, consider that, in my case, every process always writes only its own matrix and rhs entries. > > Unfortunately, at the moment, I have no access to a system where a > straightforward test would give a clear answer to this. Still, I guess, the matter is more conceptual than practical. > > Thank you and sorry for the long mail > From knepley at gmail.com Thu Feb 2 06:16:05 2017 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 2 Feb 2017 06:16:05 -0600 Subject: [petsc-users] Usage of VecCreateMPIWithArray in fortran In-Reply-To: References: Message-ID: On Wed, Feb 1, 2017 at 11:46 PM, Praveen C wrote: > Dear all > I am creating a vector in fortran like this > > PetscReal,allocatable :: res(:,:) > > Vec :: v_res > > allocate( res(nvar,nvl_nvg) ) > > call VecCreateMPIWithArray(PETSC_COMM_WORLD, nvar, nvar*nvl, & > > PETSC_DECIDE, res(1,1), v_res, & > > ierr); CHKERRQ(ierr) > > > Note that my array "res" is larger than needed by the vector v_res. > > Is this ok to do this ? > Yes Matt > I use the extra "res" array for dummy ghost values, but I dont need v_res > to be ghosted. > > Thanks > praveen > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From sonia.pozzi at usi.ch Thu Feb 2 08:55:22 2017 From: sonia.pozzi at usi.ch (Sonia Pozzi) Date: Thu, 2 Feb 2017 15:55:22 +0100 Subject: [petsc-users] ASM with matrix-free method Message-ID: <178AEC46-E850-448E-9004-1E6318F4CF9C@usi.ch> Dear Barry, At the moment I?m dealing with the following problem. Suppose I want to solve Sx=b, where is a product of matrices. In particular inside it contains also an inverse of a matrix. To compute the action of S I have created a shell matrix, but know I would like to apply the ASM as preconditioner. Citing your words as answer to a previous similar question: Thu Jan 13 13:14:47 CST 2011 Is it possible use ASM and/or FieldSplit with a matrix-free method? "Yes, BUT the algorithms are coded around MatGetSubMatrix() and or MatGetSubMatrices() so to do matrix free you need to have code that applies "part" of the operator at a time (that is you cannot just have a matrix vector product that applies the entire operator to the entire vector. Once you have the ability to apply "part" of the operator at a time you need to code up a MATSHELL that responds appropriately to MatGetSubMatrix() and or MatGetSubMatrices() and returns new matrix-free shell matrices that apply only "their" part of the operator. This is non-trivial for many people but possible." Could you be so kind to explain me better how, or do you have some code that is doing something similar? Regards, Sonia -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Feb 2 09:02:40 2017 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 2 Feb 2017 09:02:40 -0600 Subject: [petsc-users] ASM with matrix-free method In-Reply-To: <178AEC46-E850-448E-9004-1E6318F4CF9C@usi.ch> References: <178AEC46-E850-448E-9004-1E6318F4CF9C@usi.ch> Message-ID: On Thu, Feb 2, 2017 at 8:55 AM, Sonia Pozzi wrote: > Dear Barry, > > At the moment I?m dealing with the following problem. > > Suppose I want to solve Sx=b, where is a product of matrices. In > particular inside it contains also an inverse of a matrix. > To compute the action of S I have created a shell matrix, but know I would > like to apply the ASM as preconditioner. > Are you using a Schur complement? Have you looked at http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/MatCreateSchurComplement.html > Citing your words as answer to a previous similar question: > > *Thu Jan 13 13:14:47 CST 2011* > *Is it possible use ASM and/or FieldSplit with a matrix-free method?* > > "Yes, BUT the algorithms are coded around MatGetSubMatrix() and or MatGetSubMatrices() so to do matrix free you need to have code that applies "part" of the operator at a time (that is you cannot just have a matrix vector product that applies the entire operator to the entire vector. Once you have the ability to apply "part" of the operator at a time you need to code up a MATSHELL that responds appropriately to MatGetSubMatrix() and or MatGetSubMatrices() and returns new matrix-free shell matrices that apply only "their" part of the operator. This is non-trivial for many people but possible." > > > Could you be so kind to explain me better how, or do you have some code > that is doing something similar? > > ASM says "I will take a block of your operator and invert it", but we do not know how to take a block of your MATSHELL because it only provides the action on a vector. You could a) Also provide MatGetSubmatrix() or MAtGetSubmatrices() for your MATSHELL as Barry says b) Make a matrix M that is an approximation to your MATSHELL, perhaps using the block diagonal portions of the inverse so that you can explicitly give the values. Then pass M in as the preconditioner matrix for the solver. c) Use Chebyshev or some matrix-free preconditioner instead of ASM Thanks, Matt > Regards, > > Sonia > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Thu Feb 2 09:20:09 2017 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Thu, 2 Feb 2017 18:20:09 +0300 Subject: [petsc-users] ASM with matrix-free method In-Reply-To: References: <178AEC46-E850-448E-9004-1E6318F4CF9C@usi.ch> Message-ID: <5949A2AB-FA2E-4032-B5B2-E3EED0A5A47B@gmail.com> > On Feb 2, 2017, at 6:02 PM, Matthew Knepley wrote: > > On Thu, Feb 2, 2017 at 8:55 AM, Sonia Pozzi > wrote: > Dear Barry, > > At the moment I?m dealing with the following problem. > > Suppose I want to solve Sx=b, where is a product of matrices. In particular inside it contains also an inverse of a matrix. > To compute the action of S I have created a shell matrix, but know I would like to apply the ASM as preconditioner. > > Are you using a Schur complement? Have you looked at > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/MatCreateSchurComplement.html Which kind of Schur complement is it? Why do you want to use ASM? If it comes from non-overlapping domain decomposition, then you could try PCNN (though it has been not maintained for a while) or PCBDDC. > > Citing your words as answer to a previous similar question: > > Thu Jan 13 13:14:47 CST 2011 > Is it possible use ASM and/or FieldSplit with a matrix-free method? > "Yes, BUT the algorithms are coded around MatGetSubMatrix() and or MatGetSubMatrices() so to do matrix free you need to have code that applies "part" of the operator at a time (that is you cannot just have a matrix vector product that applies the entire operator to the entire vector. Once you have the ability to apply "part" of the operator at a time you need to code up a MATSHELL that responds appropriately to MatGetSubMatrix() and or MatGetSubMatrices() and returns new matrix-free shell matrices that apply only "their" part of the operator. This is non-trivial for many people but possible." > > Could you be so kind to explain me better how, or do you have some code that is doing something similar? > ASM says "I will take a block of your operator and invert it", but we do not know how to take a block of your MATSHELL > because it only provides the action on a vector. You could > > a) Also provide MatGetSubmatrix() or MAtGetSubmatrices() for your MATSHELL as Barry says > > b) Make a matrix M that is an approximation to your MATSHELL, perhaps using the block diagonal portions of the inverse > so that you can explicitly give the values. Then pass M in as the preconditioner matrix for the solver. > > c) Use Chebyshev or some matrix-free preconditioner instead of ASM > > Thanks, > > Matt > Regards, > > Sonia > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From aherrema at iastate.edu Thu Feb 2 13:25:57 2017 From: aherrema at iastate.edu (Austin Herrema) Date: Thu, 2 Feb 2017 13:25:57 -0600 Subject: [petsc-users] Segmentation violation issue when using MatSetVals in Fortran Message-ID: Hello all, I'm having a strange issue when using the MatSetVals function in Fortran. The code is quite large, but relevant snippets of what I'm trying to do are below. ! PETSc variable declaration Mat LHS PetscErrorCode ier PetscInt idx1(nsd*nsd*nshl*nshl), idx2(nsd*nsd*nshl*nshl) PetscScalar vals(nsd*nsd*nshl*nshl) ! Code for filling in index and value arrays... I've verified that there are 2304 elements in each array. ! Fill in values call MatSetValues(LHS, nvals, idx1, nvals, idx2, vals, ADD_VALUES, ier) Upon executing this, I get a general segmentation violation error. Running in Valgrind I get: ==87676== Invalid read of size 8 ==87676== at 0x1008D1F45: MatSetValues_SeqAIJ (aij.c:461) ==87676== by 0x1004C523C: MatSetValues (matrix.c:1190) ==87676== by 0x10053F1A5: matsetvalues_ (matrixf.c:696) ==87676== by 0x10005F47D: petsc_fillmat_ (petsc_fillMat.F90:78) ==87676== by 0x10005E5C9: intelmass_shell_ (shell_IntElmAss.F90:330) ==87676== by 0x100057259: solflowgmres_ (solflow.F90:125) ==87676== by 0x100051051: MAIN__ (driver.F90:140) ==87676== by 0x1000545C4: main (driver.F90:7) ==87676== Address 0x10a4dac60 is 0 bytes after a block of size 18,432 alloc'd ==87676== at 0x100088681: malloc (in /usr/local/Cellar/valgrind/3.12.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so) ==87676== by 0x10005EF47: petsc_fillmat_ (petsc_fillMat.F90:23) ==87676== by 0x10005E5C9: intelmass_shell_ (shell_IntElmAss.F90:330) ==87676== by 0x100057259: solflowgmres_ (solflow.F90:125) ==87676== by 0x100051051: MAIN__ (driver.F90:140) ==87676== by 0x1000545C4: main (driver.F90:7) ==87676== ==87676== Conditional jump or move depends on uninitialised value(s) ==87676== at 0x1008D1F57: MatSetValues_SeqAIJ (aij.c:463) ==87676== by 0x1004C523C: MatSetValues (matrix.c:1190) ==87676== by 0x10053F1A5: matsetvalues_ (matrixf.c:696) ==87676== by 0x10005F47D: petsc_fillmat_ (petsc_fillMat.F90:78) ==87676== by 0x10005E5C9: intelmass_shell_ (shell_IntElmAss.F90:330) ==87676== by 0x100057259: solflowgmres_ (solflow.F90:125) ==87676== by 0x100051051: MAIN__ (driver.F90:140) ==87676== by 0x1000545C4: main (driver.F90:7) ==87676== ==87676== Conditional jump or move depends on uninitialised value(s) ==87676== at 0x1008D1F62: MatSetValues_SeqAIJ (aij.c:463) ==87676== by 0x1004C523C: MatSetValues (matrix.c:1190) ==87676== by 0x10053F1A5: matsetvalues_ (matrixf.c:696) ==87676== by 0x10005F47D: petsc_fillmat_ (petsc_fillMat.F90:78) ==87676== by 0x10005E5C9: intelmass_shell_ (shell_IntElmAss.F90:330) ==87676== by 0x100057259: solflowgmres_ (solflow.F90:125) ==87676== by 0x100051051: MAIN__ (driver.F90:140) ==87676== by 0x1000545C4: main (driver.F90:7) ==87676== I'm not terribly experienced with Valgrind but I can't find anything too helpful in that output. It shows the traceback but I still cannot understand *why *the error is being produced. Strangely, if I try to set the matrix values in a few smaller chunks, i.e. call MatSetValues(LHS, 500, idx1(1:500), 500, idx2(1:500), vals(1:500), ADD_VALUES, ier) call MatSetValues(LHS, 500, idx1(501:1000), 500, idx2(501:1000), vals(501:1000), ADD_VALUES, ier) call MatSetValues(LHS, 500, idx1(1001:1500), 500, idx2(1001:1500), vals(1001:1500), ADD_VALUES, ier) call MatSetValues(LHS, 500, idx1(1501:2000), 500, idx2(1501:2000), vals(1501:2000), ADD_VALUES, ier) call MatSetValues(LHS, 304, idx1(2001:2304), 304, idx2(2001:2304), vals(2001:2304), ADD_VALUES, ier) Then everything works with no errors. Is there some kind of limitation to how many values you can insert into a matrix at once? Would strike me as strange. I can assign elements one at a time but reducing the number of calls to MatSetVals by a factor of 2000 would be fantastic. I would be happy to provide more details of the matrix setup if necessary. Thank you for any help! Austin -- *Austin Herrema* PhD Student | Graduate Research Assistant | Iowa State University Wind Energy Science, Engineering, and Policy | Mechanical Engineering -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Thu Feb 2 13:31:51 2017 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Thu, 2 Feb 2017 22:31:51 +0300 Subject: [petsc-users] Segmentation violation issue when using MatSetVals in Fortran In-Reply-To: References: Message-ID: <2EA23808-64E1-474D-ADE1-AB5EA5BF6FC5@gmail.com> > On Feb 2, 2017, at 10:25 PM, Austin Herrema wrote: > > Hello all, > > I'm having a strange issue when using the MatSetVals function in Fortran. The code is quite large, but relevant snippets of what I'm trying to do are below. > > ! PETSc variable declaration > Mat LHS > PetscErrorCode ier > PetscInt idx1(nsd*nsd*nshl*nshl), idx2(nsd*nsd*nshl*nshl) > PetscScalar vals(nsd*nsd*nshl*nshl) > > ! Code for filling in index and value arrays... I've verified that there are 2304 elements in each array. > > ! Fill in values > call MatSetValues(LHS, nvals, idx1, nvals, idx2, vals, ADD_VALUES, ier) > > Upon executing this, I get a general segmentation violation error. Running in Valgrind I get: > > ==87676== Invalid read of size 8 > ==87676== at 0x1008D1F45: MatSetValues_SeqAIJ (aij.c:461) > ==87676== by 0x1004C523C: MatSetValues (matrix.c:1190) > ==87676== by 0x10053F1A5: matsetvalues_ (matrixf.c:696) > ==87676== by 0x10005F47D: petsc_fillmat_ (petsc_fillMat.F90:78) > ==87676== by 0x10005E5C9: intelmass_shell_ (shell_IntElmAss.F90:330) > ==87676== by 0x100057259: solflowgmres_ (solflow.F90:125) > ==87676== by 0x100051051: MAIN__ (driver.F90:140) > ==87676== by 0x1000545C4: main (driver.F90:7) > ==87676== Address 0x10a4dac60 is 0 bytes after a block of size 18,432 alloc'd > ==87676== at 0x100088681: malloc (in /usr/local/Cellar/valgrind/3.12.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so) > ==87676== by 0x10005EF47: petsc_fillmat_ (petsc_fillMat.F90:23) > ==87676== by 0x10005E5C9: intelmass_shell_ (shell_IntElmAss.F90:330) > ==87676== by 0x100057259: solflowgmres_ (solflow.F90:125) > ==87676== by 0x100051051: MAIN__ (driver.F90:140) > ==87676== by 0x1000545C4: main (driver.F90:7) > ==87676== > ==87676== Conditional jump or move depends on uninitialised value(s) > ==87676== at 0x1008D1F57: MatSetValues_SeqAIJ (aij.c:463) > ==87676== by 0x1004C523C: MatSetValues (matrix.c:1190) > ==87676== by 0x10053F1A5: matsetvalues_ (matrixf.c:696) > ==87676== by 0x10005F47D: petsc_fillmat_ (petsc_fillMat.F90:78) > ==87676== by 0x10005E5C9: intelmass_shell_ (shell_IntElmAss.F90:330) > ==87676== by 0x100057259: solflowgmres_ (solflow.F90:125) > ==87676== by 0x100051051: MAIN__ (driver.F90:140) > ==87676== by 0x1000545C4: main (driver.F90:7) > ==87676== > ==87676== Conditional jump or move depends on uninitialised value(s) > ==87676== at 0x1008D1F62: MatSetValues_SeqAIJ (aij.c:463) > ==87676== by 0x1004C523C: MatSetValues (matrix.c:1190) > ==87676== by 0x10053F1A5: matsetvalues_ (matrixf.c:696) > ==87676== by 0x10005F47D: petsc_fillmat_ (petsc_fillMat.F90:78) > ==87676== by 0x10005E5C9: intelmass_shell_ (shell_IntElmAss.F90:330) > ==87676== by 0x100057259: solflowgmres_ (solflow.F90:125) > ==87676== by 0x100051051: MAIN__ (driver.F90:140) > ==87676== by 0x1000545C4: main (driver.F90:7) > ==87676== > > I'm not terribly experienced with Valgrind but I can't find anything too helpful in that output. It shows the traceback but I still cannot understand why the error is being produced. Strangely, if I try to set the matrix values in a few smaller chunks, i.e. > > call MatSetValues(LHS, 500, idx1(1:500), 500, idx2(1:500), vals(1:500), ADD_VALUES, ier) > call MatSetValues(LHS, 500, idx1(501:1000), 500, idx2(501:1000), vals(501:1000), ADD_VALUES, ier) > call MatSetValues(LHS, 500, idx1(1001:1500), 500, idx2(1001:1500), vals(1001:1500), ADD_VALUES, ier) > call MatSetValues(LHS, 500, idx1(1501:2000), 500, idx2(1501:2000), vals(1501:2000), ADD_VALUES, ier) > call MatSetValues(LHS, 304, idx1(2001:2304), 304, idx2(2001:2304), vals(2001:2304), ADD_VALUES, ier) > vals is supposed to be a 2D array of size nvals*nvals; it seems to me that the size of your vals is nvals (supposedly, nsd*nsd*nshl*nshl) > Then everything works with no errors. Is there some kind of limitation to how many values you can insert into a matrix at once? Would strike me as strange. I can assign elements one at a time but reducing the number of calls to MatSetVals by a factor of 2000 would be fantastic. I would be happy to provide more details of the matrix setup if necessary. > > Thank you for any help! > Austin > > > -- > Austin Herrema > PhD Student | Graduate Research Assistant | Iowa State University > Wind Energy Science, Engineering, and Policy | Mechanical Engineering -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Feb 2 13:32:15 2017 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 2 Feb 2017 13:32:15 -0600 Subject: [petsc-users] Segmentation violation issue when using MatSetVals in Fortran In-Reply-To: References: Message-ID: On Thu, Feb 2, 2017 at 1:25 PM, Austin Herrema wrote: > Hello all, > > I'm having a strange issue when using the MatSetVals function in Fortran. > The code is quite large, but relevant snippets of what I'm trying to do are > below. > > ! PETSc variable declaration > Mat LHS > PetscErrorCode ier > PetscInt idx1(nsd*nsd*nshl*nshl), idx2(nsd*nsd*nshl*nshl) > PetscScalar vals(nsd*nsd*nshl*nshl) > vals should be of size nidx1 * nidx2. Matt > > ! Code for filling in index and value arrays... I've verified that > there are 2304 elements in each array. > > ! Fill in values > call MatSetValues(LHS, nvals, idx1, nvals, idx2, vals, ADD_VALUES, ier) > > Upon executing this, I get a general segmentation violation error. Running > in Valgrind I get: > > ==87676== Invalid read of size 8 > ==87676== at 0x1008D1F45: MatSetValues_SeqAIJ (aij.c:461) > ==87676== by 0x1004C523C: MatSetValues (matrix.c:1190) > ==87676== by 0x10053F1A5: matsetvalues_ (matrixf.c:696) > ==87676== by 0x10005F47D: petsc_fillmat_ (petsc_fillMat.F90:78) > ==87676== by 0x10005E5C9: intelmass_shell_ (shell_IntElmAss.F90:330) > ==87676== by 0x100057259: solflowgmres_ (solflow.F90:125) > ==87676== by 0x100051051: MAIN__ (driver.F90:140) > ==87676== by 0x1000545C4: main (driver.F90:7) > ==87676== Address 0x10a4dac60 is 0 bytes after a block of size 18,432 > alloc'd > ==87676== at 0x100088681: malloc (in /usr/local/Cellar/valgrind/3. > 12.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so) > ==87676== by 0x10005EF47: petsc_fillmat_ (petsc_fillMat.F90:23) > ==87676== by 0x10005E5C9: intelmass_shell_ (shell_IntElmAss.F90:330) > ==87676== by 0x100057259: solflowgmres_ (solflow.F90:125) > ==87676== by 0x100051051: MAIN__ (driver.F90:140) > ==87676== by 0x1000545C4: main (driver.F90:7) > ==87676== > ==87676== Conditional jump or move depends on uninitialised value(s) > ==87676== at 0x1008D1F57: MatSetValues_SeqAIJ (aij.c:463) > ==87676== by 0x1004C523C: MatSetValues (matrix.c:1190) > ==87676== by 0x10053F1A5: matsetvalues_ (matrixf.c:696) > ==87676== by 0x10005F47D: petsc_fillmat_ (petsc_fillMat.F90:78) > ==87676== by 0x10005E5C9: intelmass_shell_ (shell_IntElmAss.F90:330) > ==87676== by 0x100057259: solflowgmres_ (solflow.F90:125) > ==87676== by 0x100051051: MAIN__ (driver.F90:140) > ==87676== by 0x1000545C4: main (driver.F90:7) > ==87676== > ==87676== Conditional jump or move depends on uninitialised value(s) > ==87676== at 0x1008D1F62: MatSetValues_SeqAIJ (aij.c:463) > ==87676== by 0x1004C523C: MatSetValues (matrix.c:1190) > ==87676== by 0x10053F1A5: matsetvalues_ (matrixf.c:696) > ==87676== by 0x10005F47D: petsc_fillmat_ (petsc_fillMat.F90:78) > ==87676== by 0x10005E5C9: intelmass_shell_ (shell_IntElmAss.F90:330) > ==87676== by 0x100057259: solflowgmres_ (solflow.F90:125) > ==87676== by 0x100051051: MAIN__ (driver.F90:140) > ==87676== by 0x1000545C4: main (driver.F90:7) > ==87676== > > I'm not terribly experienced with Valgrind but I can't find anything too > helpful in that output. It shows the traceback but I still cannot > understand *why *the error is being produced. Strangely, if I try to set > the matrix values in a few smaller chunks, i.e. > > call MatSetValues(LHS, 500, idx1(1:500), 500, idx2(1:500), vals(1:500), > ADD_VALUES, ier) > call MatSetValues(LHS, 500, idx1(501:1000), 500, idx2(501:1000), > vals(501:1000), ADD_VALUES, ier) > call MatSetValues(LHS, 500, idx1(1001:1500), 500, idx2(1001:1500), > vals(1001:1500), ADD_VALUES, ier) > call MatSetValues(LHS, 500, idx1(1501:2000), 500, idx2(1501:2000), > vals(1501:2000), ADD_VALUES, ier) > call MatSetValues(LHS, 304, idx1(2001:2304), 304, idx2(2001:2304), > vals(2001:2304), ADD_VALUES, ier) > > Then everything works with no errors. Is there some kind of limitation to > how many values you can insert into a matrix at once? Would strike me as > strange. I can assign elements one at a time but reducing the number of > calls to MatSetVals by a factor of 2000 would be fantastic. I would be > happy to provide more details of the matrix setup if necessary. > > Thank you for any help! > Austin > > > -- > *Austin Herrema* > PhD Student | Graduate Research Assistant | Iowa State University > Wind Energy Science, Engineering, and Policy | Mechanical Engineering > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From aherrema at iastate.edu Thu Feb 2 13:49:21 2017 From: aherrema at iastate.edu (Austin Herrema) Date: Thu, 2 Feb 2017 13:49:21 -0600 Subject: [petsc-users] Segmentation violation issue when using MatSetVals in Fortran In-Reply-To: References: Message-ID: Ah. Yes that definitely explains it. Weird that it seemed to work in some cases but I suppose the output would be total garbage. Anyways. Thank you both. Austin On Thu, Feb 2, 2017 at 1:32 PM, Matthew Knepley wrote: > On Thu, Feb 2, 2017 at 1:25 PM, Austin Herrema > wrote: > >> Hello all, >> >> I'm having a strange issue when using the MatSetVals function in Fortran. >> The code is quite large, but relevant snippets of what I'm trying to do are >> below. >> >> ! PETSc variable declaration >> Mat LHS >> PetscErrorCode ier >> PetscInt idx1(nsd*nsd*nshl*nshl), idx2(nsd*nsd*nshl*nshl) >> PetscScalar vals(nsd*nsd*nshl*nshl) >> > > vals should be of size nidx1 * nidx2. > > Matt > > >> >> ! Code for filling in index and value arrays... I've verified that >> there are 2304 elements in each array. >> >> ! Fill in values >> call MatSetValues(LHS, nvals, idx1, nvals, idx2, vals, ADD_VALUES, ier) >> >> Upon executing this, I get a general segmentation violation error. >> Running in Valgrind I get: >> >> ==87676== Invalid read of size 8 >> ==87676== at 0x1008D1F45: MatSetValues_SeqAIJ (aij.c:461) >> ==87676== by 0x1004C523C: MatSetValues (matrix.c:1190) >> ==87676== by 0x10053F1A5: matsetvalues_ (matrixf.c:696) >> ==87676== by 0x10005F47D: petsc_fillmat_ (petsc_fillMat.F90:78) >> ==87676== by 0x10005E5C9: intelmass_shell_ (shell_IntElmAss.F90:330) >> ==87676== by 0x100057259: solflowgmres_ (solflow.F90:125) >> ==87676== by 0x100051051: MAIN__ (driver.F90:140) >> ==87676== by 0x1000545C4: main (driver.F90:7) >> ==87676== Address 0x10a4dac60 is 0 bytes after a block of size 18,432 >> alloc'd >> ==87676== at 0x100088681: malloc (in /usr/local/Cellar/valgrind/3.1 >> 2.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so) >> ==87676== by 0x10005EF47: petsc_fillmat_ (petsc_fillMat.F90:23) >> ==87676== by 0x10005E5C9: intelmass_shell_ (shell_IntElmAss.F90:330) >> ==87676== by 0x100057259: solflowgmres_ (solflow.F90:125) >> ==87676== by 0x100051051: MAIN__ (driver.F90:140) >> ==87676== by 0x1000545C4: main (driver.F90:7) >> ==87676== >> ==87676== Conditional jump or move depends on uninitialised value(s) >> ==87676== at 0x1008D1F57: MatSetValues_SeqAIJ (aij.c:463) >> ==87676== by 0x1004C523C: MatSetValues (matrix.c:1190) >> ==87676== by 0x10053F1A5: matsetvalues_ (matrixf.c:696) >> ==87676== by 0x10005F47D: petsc_fillmat_ (petsc_fillMat.F90:78) >> ==87676== by 0x10005E5C9: intelmass_shell_ (shell_IntElmAss.F90:330) >> ==87676== by 0x100057259: solflowgmres_ (solflow.F90:125) >> ==87676== by 0x100051051: MAIN__ (driver.F90:140) >> ==87676== by 0x1000545C4: main (driver.F90:7) >> ==87676== >> ==87676== Conditional jump or move depends on uninitialised value(s) >> ==87676== at 0x1008D1F62: MatSetValues_SeqAIJ (aij.c:463) >> ==87676== by 0x1004C523C: MatSetValues (matrix.c:1190) >> ==87676== by 0x10053F1A5: matsetvalues_ (matrixf.c:696) >> ==87676== by 0x10005F47D: petsc_fillmat_ (petsc_fillMat.F90:78) >> ==87676== by 0x10005E5C9: intelmass_shell_ (shell_IntElmAss.F90:330) >> ==87676== by 0x100057259: solflowgmres_ (solflow.F90:125) >> ==87676== by 0x100051051: MAIN__ (driver.F90:140) >> ==87676== by 0x1000545C4: main (driver.F90:7) >> ==87676== >> >> I'm not terribly experienced with Valgrind but I can't find anything too >> helpful in that output. It shows the traceback but I still cannot >> understand *why *the error is being produced. Strangely, if I try to set >> the matrix values in a few smaller chunks, i.e. >> >> call MatSetValues(LHS, 500, idx1(1:500), 500, idx2(1:500), vals(1:500), >> ADD_VALUES, ier) >> call MatSetValues(LHS, 500, idx1(501:1000), 500, idx2(501:1000), >> vals(501:1000), ADD_VALUES, ier) >> call MatSetValues(LHS, 500, idx1(1001:1500), 500, idx2(1001:1500), >> vals(1001:1500), ADD_VALUES, ier) >> call MatSetValues(LHS, 500, idx1(1501:2000), 500, idx2(1501:2000), >> vals(1501:2000), ADD_VALUES, ier) >> call MatSetValues(LHS, 304, idx1(2001:2304), 304, idx2(2001:2304), >> vals(2001:2304), ADD_VALUES, ier) >> >> Then everything works with no errors. Is there some kind of limitation to >> how many values you can insert into a matrix at once? Would strike me as >> strange. I can assign elements one at a time but reducing the number of >> calls to MatSetVals by a factor of 2000 would be fantastic. I would be >> happy to provide more details of the matrix setup if necessary. >> >> Thank you for any help! >> Austin >> >> >> -- >> *Austin Herrema* >> PhD Student | Graduate Research Assistant | Iowa State University >> Wind Energy Science, Engineering, and Policy | Mechanical Engineering >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- *Austin Herrema* PhD Student | Graduate Research Assistant | Iowa State University Wind Energy Science, Engineering, and Policy | Mechanical Engineering -------------- next part -------------- An HTML attachment was scrubbed... URL: From gideon.simpson at gmail.com Fri Feb 3 19:14:59 2017 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Fri, 3 Feb 2017 20:14:59 -0500 Subject: [petsc-users] projection methods in TS Message-ID: I?m interested in implementing a projection method for an ODE of the form: y? = f(y), such that g(y) = 0 for all time (i.e., g is conserved). Note that in a projection method, a standard time step is made to produce y* from y_{n}, and then this is corrected to obtain y_{n+1} satisfying g(y) = 0. There were two ways I was thinking of doing this, and I was hoping to get some input: Idea 1: Manually loop through using taking a time step and then implementing the projection routine. I see that there is a TSStep command, but this doesn?t seem to be much documentation on how to use it in this scenario. Does anyone have any guidance? Idea 2: Is there some analog to TSMonitor that allows me to modify the solution after each time step, instead of just allowing for some computation of a statistic? From bsmith at mcs.anl.gov Fri Feb 3 20:40:39 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 3 Feb 2017 20:40:39 -0600 Subject: [petsc-users] projection methods in TS In-Reply-To: References: Message-ID: <2E0519CB-7CA7-41A4-9E67-C973BE33F35C@mcs.anl.gov> TSSetPostStep(); in your function use TSGetSolution() to get the current solution. Please let us know how it works out Barry > On Feb 3, 2017, at 7:14 PM, Gideon Simpson wrote: > > I?m interested in implementing a projection method for an ODE of the form: > > y? = f(y), > > such that g(y) = 0 for all time (i.e., g is conserved). Note that in a projection method, a standard time step is made to produce y* from y_{n}, and then this is corrected to obtain y_{n+1} satisfying g(y) = 0. > > There were two ways I was thinking of doing this, and I was hoping to get some input: > > Idea 1: Manually loop through using taking a time step and then implementing the projection routine. I see that there is a TSStep command, but this doesn?t seem to be much documentation on how to use it in this scenario. Does anyone have any guidance? > > Idea 2: Is there some analog to TSMonitor that allows me to modify the solution after each time step, instead of just allowing for some computation of a statistic? > > From knepley at gmail.com Fri Feb 3 22:47:09 2017 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 3 Feb 2017 22:47:09 -0600 Subject: [petsc-users] projection methods in TS In-Reply-To: <2E0519CB-7CA7-41A4-9E67-C973BE33F35C@mcs.anl.gov> References: <2E0519CB-7CA7-41A4-9E67-C973BE33F35C@mcs.anl.gov> Message-ID: That is one answer. Another one is that this particular system is a DAE and we have methods for that. Matt On Fri, Feb 3, 2017 at 8:40 PM, Barry Smith wrote: > > TSSetPostStep(); in your function use TSGetSolution() to get the current > solution. > > Please let us know how it works out > > Barry > > > > > On Feb 3, 2017, at 7:14 PM, Gideon Simpson > wrote: > > > > I?m interested in implementing a projection method for an ODE of the > form: > > > > y? = f(y), > > > > such that g(y) = 0 for all time (i.e., g is conserved). Note that in a > projection method, a standard time step is made to produce y* from y_{n}, > and then this is corrected to obtain y_{n+1} satisfying g(y) = 0. > > > > There were two ways I was thinking of doing this, and I was hoping to > get some input: > > > > Idea 1: Manually loop through using taking a time step and then > implementing the projection routine. I see that there is a TSStep command, > but this doesn?t seem to be much documentation on how to use it in this > scenario. Does anyone have any guidance? > > > > Idea 2: Is there some analog to TSMonitor that allows me to modify the > solution after each time step, instead of just allowing for some > computation of a statistic? > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From gideon.simpson at gmail.com Sat Feb 4 09:47:58 2017 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Sat, 4 Feb 2017 10:47:58 -0500 Subject: [petsc-users] projection methods in TS In-Reply-To: References: <2E0519CB-7CA7-41A4-9E67-C973BE33F35C@mcs.anl.gov> Message-ID: Would setting it up as a DAE in petsc be algorithmically euivalent to a projected method (i.e., step of standard RK followed by nonlinear projection)? -gideon > On Feb 3, 2017, at 11:47 PM, Matthew Knepley wrote: > > That is one answer. Another one is that this particular system is a DAE and we have methods for that. > > Matt > > On Fri, Feb 3, 2017 at 8:40 PM, Barry Smith > wrote: > > TSSetPostStep(); in your function use TSGetSolution() to get the current solution. > > Please let us know how it works out > > Barry > > > > > On Feb 3, 2017, at 7:14 PM, Gideon Simpson > wrote: > > > > I?m interested in implementing a projection method for an ODE of the form: > > > > y? = f(y), > > > > such that g(y) = 0 for all time (i.e., g is conserved). Note that in a projection method, a standard time step is made to produce y* from y_{n}, and then this is corrected to obtain y_{n+1} satisfying g(y) = 0. > > > > There were two ways I was thinking of doing this, and I was hoping to get some input: > > > > Idea 1: Manually loop through using taking a time step and then implementing the projection routine. I see that there is a TSStep command, but this doesn?t seem to be much documentation on how to use it in this scenario. Does anyone have any guidance? > > > > Idea 2: Is there some analog to TSMonitor that allows me to modify the solution after each time step, instead of just allowing for some computation of a statistic? > > > > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Feb 4 11:00:39 2017 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 4 Feb 2017 11:00:39 -0600 Subject: [petsc-users] projection methods in TS In-Reply-To: References: <2E0519CB-7CA7-41A4-9E67-C973BE33F35C@mcs.anl.gov> Message-ID: On Sat, Feb 4, 2017 at 9:47 AM, Gideon Simpson wrote: > Would setting it up as a DAE in petsc be algorithmically euivalent to a > projected method (i.e., step of standard RK followed by nonlinear > projection)? > I am not sure, as I do not understand those solvers. However, I wrote my own solver that does exactly that MIMEX. Matt > -gideon > > On Feb 3, 2017, at 11:47 PM, Matthew Knepley wrote: > > That is one answer. Another one is that this particular system is a DAE > and we have methods for that. > > Matt > > On Fri, Feb 3, 2017 at 8:40 PM, Barry Smith wrote: > >> >> TSSetPostStep(); in your function use TSGetSolution() to get the current >> solution. >> >> Please let us know how it works out >> >> Barry >> >> >> >> > On Feb 3, 2017, at 7:14 PM, Gideon Simpson >> wrote: >> > >> > I?m interested in implementing a projection method for an ODE of the >> form: >> > >> > y? = f(y), >> > >> > such that g(y) = 0 for all time (i.e., g is conserved). Note that in a >> projection method, a standard time step is made to produce y* from y_{n}, >> and then this is corrected to obtain y_{n+1} satisfying g(y) = 0. >> > >> > There were two ways I was thinking of doing this, and I was hoping to >> get some input: >> > >> > Idea 1: Manually loop through using taking a time step and then >> implementing the projection routine. I see that there is a TSStep command, >> but this doesn?t seem to be much documentation on how to use it in this >> scenario. Does anyone have any guidance? >> > >> > Idea 2: Is there some analog to TSMonitor that allows me to modify the >> solution after each time step, instead of just allowing for some >> computation of a statistic? >> > >> > >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mvalera at mail.sdsu.edu Sat Feb 4 18:02:52 2017 From: mvalera at mail.sdsu.edu (Manuel Valera) Date: Sat, 4 Feb 2017 16:02:52 -0800 Subject: [petsc-users] DMCreateGlobalVector in fortran. Message-ID: Hello devs, Ive been creating the framework to distribute the arrays from my model using the global/local vectors and the DMDAs, for this i created a small subroutine that is responsible for making the necessary calls: SUBROUTINE DAs(da,globalv,localv) ! use PetscObjects, only :: ierr ! Umbrella program to update and communicate the arrays in a ! distributed fashion using the DMDA objects from PETSc. ! Manuel Valera 1/20/17 ! Arguments: ! da = DMDA array either 1d,2d or 3d, already created and setup ! globalv = global vector to be operated into ! localv = local chunk each processor works in. Vec,intent(inout) :: globalv,localv DM,intent(inout) :: da call DMCreateGlobalVector(da,globalv,ierr) call DMCreateLocalVector(da,localv,ierr) call DMGlobalToLocalBegin(da,globalv,INSERT_VALUES,localv,ierr) call DMGlobalToLocalEnd(da,globalv,INSERT_VALUES,localv,ierr) END SUBROUTINE DAs I call this from another program where i have declared a vector as: #include "petsc/finclude/petscdm.h"#include "petsc/finclude/petscdmda.h"use petscuse petscdmuse petscdmda IMPLICIT NONE Vec :: xpdmda,yp,zp,xcp,ycp,zcp DM :: da,dac,dau,dav,daw CONTAINSSUBROUTINE InitGridPIMPLICIT NONE!PetscInt :: ierr! CREATE DMDA:call CDA3(da,IMax-1,JMax-1,KMax-1)!xpdmda,yp,zp have the same dimensions than x,y,z!xpl,ypl,zpl are the local versions of xp,yp,zp!We create the global and local vectors and communicate their values:call DAs(da,xpdmda,xpl,x) And at this call it crashes with the following message: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Null argument, when expecting valid pointer [0]PETSC ERROR: Null Object: Parameter # 2 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.7.5-2949-gc3c607c GIT Date: 2017-01-20 17:34:16 -0600 [0]PETSC ERROR: ./ucmsLEP on a arch-linux2-c-debug named ocean by valera Sat Feb 4 15:53:53 2017 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-hdf5 --download-netcdf --download-hypre --download-metis --download-parmetis --download-trillinos --with-debugging=1 [0]PETSC ERROR: #3 VecSetLocalToGlobalMapping() line 79 in /usr/dataC/home/valera/petsc/src/vec/vec/interface/vector.c [0]PETSC ERROR: #4 DMCreateGlobalVector_DA() line 41 in /usr/dataC/home/valera/petsc/src/dm/impls/da/dadist.c [0]PETSC ERROR: #5 DMCreateGlobalVector() line 840 in /usr/dataC/home/valera/petsc/src/dm/interface/dm.c So my question is, do i have to setup the vectors in any additional way? i followed the fortran DMDA examples closely and i didnt notice anything extra, Thanks, Manuel -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Feb 4 18:18:46 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 4 Feb 2017 18:18:46 -0600 Subject: [petsc-users] DMCreateGlobalVector in fortran. In-Reply-To: References: Message-ID: See: http://www.mcs.anl.gov/petsc/documentation/changes/dev.html search for DMDACreate and update your code. Barry > On Feb 4, 2017, at 6:02 PM, Manuel Valera wrote: > > Hello devs, > > Ive been creating the framework to distribute the arrays from my model using the global/local vectors and the DMDAs, for this i created a small subroutine that is responsible for making the necessary calls: > > SUBROUTINE DAs(da,globalv,localv) > > ! use PetscObjects, only :: ierr > > > ! Umbrella program to update and communicate the arrays in a > ! distributed fashion using the DMDA objects from PETSc. > ! Manuel Valera 1/20/17 > > ! Arguments: > ! da = DMDA array either 1d,2d or 3d, already created and setup > ! globalv = global vector to be operated into > ! localv = local chunk each processor works in. > > > > Vec,intent(inout) :: globalv,localv > > > > DM,intent(inout) :: da > > call DMCreateGlobalVector(da,globalv,ierr) > call DMCreateLocalVector(da,localv,ierr) > > call DMGlobalToLocalBegin(da,globalv,INSERT_VALUES,localv,ierr) > call DMGlobalToLocalEnd(da,globalv,INSERT_VALUES,localv,ierr) > > > > END SUBROUTINE DAs > > > I call this from another program where i have declared a vector as: > > #include "petsc/finclude/petscdm.h" > #include "petsc/finclude/petscdmda.h" > > use petsc > use petscdm > use petscdmda > > IMPLICIT NONE > > Vec :: xpdmda,yp,zp,xcp,ycp,zcp > DM :: da,dac,dau,dav,daw > > CONTAINS > SUBROUTINE InitGridP > > IMPLICIT NONE > !PetscInt :: ierr > > ! CREATE DMDA: > > call CDA3(da,IMax-1,JMax-1,KMax-1) > > !xpdmda,yp,zp have the same dimensions than x,y,z > !xpl,ypl,zpl are the local versions of xp,yp,zp > > !We create the global and local vectors and communicate their values: > call DAs(da,xpdmda,xpl,x) > > And at this call it crashes with the following message: > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Null argument, when expecting valid pointer > [0]PETSC ERROR: Null Object: Parameter # 2 > [0]PETSC ERROR: See > http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.7.5-2949-gc3c607c GIT Date: 2017-01-20 17:34:16 -0600 > [0]PETSC ERROR: ./ucmsLEP on a arch-linux2-c-debug named ocean by valera Sat Feb 4 15:53:53 2017 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-hdf5 --download-netcdf --download-hypre --download-metis --download-parmetis --download-trillinos --with-debugging=1 > [0]PETSC ERROR: #3 VecSetLocalToGlobalMapping() line 79 in /usr/dataC/home/valera/petsc/src/vec/vec/interface/vector.c > [0]PETSC ERROR: #4 DMCreateGlobalVector_DA() line 41 in /usr/dataC/home/valera/petsc/src/dm/impls/da/dadist.c > [0]PETSC ERROR: #5 DMCreateGlobalVector() line 840 in /usr/dataC/home/valera/petsc/src/dm/interface/dm.c > > > > So my question is, do i have to setup the vectors in any additional way? i followed the fortran DMDA examples closely and i didnt notice anything extra, > > Thanks, > > Manuel From hongzhang at anl.gov Sat Feb 4 19:24:54 2017 From: hongzhang at anl.gov (Zhang, Hong) Date: Sun, 5 Feb 2017 01:24:54 +0000 Subject: [petsc-users] projection methods in TS In-Reply-To: References: <2E0519CB-7CA7-41A4-9E67-C973BE33F35C@mcs.anl.gov> Message-ID: <53402E0B-DEE9-4DE4-84AB-D8A33D94C566@anl.gov> Can you elaborate a bit more on your problem? If your problem is an index-1 DAE, there is no need to use a projection method, and it is perfectly fine to set it up as a DAE in PETSc. For high-index DAEs, you may have to use TSSetPostStep() to implement your own projection algorithm. If you happen to have a Hamiltonian system to solve, I have a symplectic solver in my own branch that you can use directly. Hong (Mr.) On Feb 4, 2017, at 9:47 AM, Gideon Simpson > wrote: Would setting it up as a DAE in petsc be algorithmically euivalent to a projected method (i.e., step of standard RK followed by nonlinear projection)? -gideon On Feb 3, 2017, at 11:47 PM, Matthew Knepley > wrote: That is one answer. Another one is that this particular system is a DAE and we have methods for that. Matt On Fri, Feb 3, 2017 at 8:40 PM, Barry Smith > wrote: TSSetPostStep(); in your function use TSGetSolution() to get the current solution. Please let us know how it works out Barry > On Feb 3, 2017, at 7:14 PM, Gideon Simpson > wrote: > > I?m interested in implementing a projection method for an ODE of the form: > > y? = f(y), > > such that g(y) = 0 for all time (i.e., g is conserved). Note that in a projection method, a standard time step is made to produce y* from y_{n}, and then this is corrected to obtain y_{n+1} satisfying g(y) = 0. > > There were two ways I was thinking of doing this, and I was hoping to get some input: > > Idea 1: Manually loop through using taking a time step and then implementing the projection routine. I see that there is a TSStep command, but this doesn?t seem to be much documentation on how to use it in this scenario. Does anyone have any guidance? > > Idea 2: Is there some analog to TSMonitor that allows me to modify the solution after each time step, instead of just allowing for some computation of a statistic? > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mvalera at mail.sdsu.edu Sat Feb 4 19:30:21 2017 From: mvalera at mail.sdsu.edu (Manuel Valera) Date: Sat, 4 Feb 2017 17:30:21 -0800 Subject: [petsc-users] DMCreateGlobalVector in fortran. In-Reply-To: References: Message-ID: Thanks, that helped a lot, Now im a little confused with my next error, in my serial code i have several arrays with dimensions declared as: u0(-1:IMax+2,-1:JMax+1,-1:KMax+1) and i create my DMDA for this array as: call CDA3(daub,IMax+4,JMax+3,KMax+3)call DAs(daub,u0p,u0pl,u0) With the subroutines as: SUBROUTINE CDA3(da3d,dim3dx,dim3dy,dim3dz) DM, intent(inout) :: da3d PetscInt, intent(in) :: dim3dx,dim3dy,dim3dz DMBoundaryType :: bx,by,bz bx = DM_BOUNDARY_NONE by = DM_BOUNDARY_NONE bz = DM_BOUNDARY_NONE call DMDACreate3d(PETSC_COMM_WORLD,bx,by,bz,DMDA_STENCIL_BOX,dim3dx,dim3dy,dim3dz,PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da3d,ierr) call DMSetFromOptions(da3d,ierr) call DMSetUp(da3d,ierr) END SUBROUTINE CDA3 SUBROUTINE DAs(da,globalv,localv,array) ! use PetscObjects, only :: ierr ! Umbrella program to update and communicate the arrays in a ! distributed fashion using the DMDA objects from PETSc. ! Manuel Valera 1/20/17 ! Arguments: ! da = DMDA array either 1d,2d or 3d, already created and setup ! globalv = global vector to be operated into ! localv = local chunk each processor works in. ! array = the array to be petscified. ONLY 3D ARRAYS rn. ! just in case. Vec,intent(inout) :: globalv,localv PetscScalar,dimension(:,:,:) :: array DM,intent(inout) :: da call DMCreateGlobalVector(da,globalv,ierr) call DMCreateLocalVector(da,localv,ierr) call DMGlobalToLocalBegin(da,globalv,INSERT_VALUES,localv,ierr) call DMGlobalToLocalEnd(da,globalv,INSERT_VALUES,localv,ierr) END SUBROUTINE DAs Which i later open into an array to operate into it, but when i try to access u0(:,j-2,:) it gives me the following error: Fortran runtime error: Index '-1' of dimension 2 of array 'u0' below lower bound of 0 So i'm guessing the DMDA is creating the arrays with dimensions starting at 0 instead of -1 as it should, right? what can be done in this case? Thanks, Manuel On Sat, Feb 4, 2017 at 4:18 PM, Barry Smith wrote: > > See: http://www.mcs.anl.gov/petsc/documentation/changes/dev.html > search for DMDACreate and update your code. > > Barry > > > > On Feb 4, 2017, at 6:02 PM, Manuel Valera wrote: > > > > Hello devs, > > > > Ive been creating the framework to distribute the arrays from my model > using the global/local vectors and the DMDAs, for this i created a small > subroutine that is responsible for making the necessary calls: > > > > SUBROUTINE DAs(da,globalv,localv) > > > > ! use PetscObjects, only :: ierr > > > > > > ! Umbrella program to update and communicate the arrays in a > > ! distributed fashion using the DMDA objects from PETSc. > > ! Manuel Valera 1/20/17 > > > > ! Arguments: > > ! da = DMDA array either 1d,2d or 3d, already created and setup > > ! globalv = global vector to be operated into > > ! localv = local chunk each processor works in. > > > > > > > > Vec,intent(inout) :: globalv,localv > > > > > > > > DM,intent(inout) :: da > > > > call DMCreateGlobalVector(da,globalv,ierr) > > call DMCreateLocalVector(da,localv,ierr) > > > > call DMGlobalToLocalBegin(da,globalv,INSERT_VALUES,localv,ierr) > > call DMGlobalToLocalEnd(da,globalv,INSERT_VALUES,localv,ierr) > > > > > > > > END SUBROUTINE DAs > > > > > > I call this from another program where i have declared a vector as: > > > > #include "petsc/finclude/petscdm.h" > > #include "petsc/finclude/petscdmda.h" > > > > use petsc > > use petscdm > > use petscdmda > > > > IMPLICIT NONE > > > > Vec :: xpdmda,yp,zp,xcp,ycp,zcp > > DM :: da,dac,dau,dav,daw > > > > CONTAINS > > SUBROUTINE InitGridP > > > > IMPLICIT NONE > > !PetscInt :: ierr > > > > ! CREATE DMDA: > > > > call CDA3(da,IMax-1,JMax-1,KMax-1) > > > > !xpdmda,yp,zp have the same dimensions than x,y,z > > !xpl,ypl,zpl are the local versions of xp,yp,zp > > > > !We create the global and local vectors and communicate their values: > > call DAs(da,xpdmda,xpl,x) > > > > And at this call it crashes with the following message: > > > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [0]PETSC ERROR: Null argument, when expecting valid pointer > > [0]PETSC ERROR: Null Object: Parameter # 2 > > [0]PETSC ERROR: See > > http://www.mcs.anl.gov/petsc/documentation/faq.html > > for trouble shooting. > > [0]PETSC ERROR: Petsc Development GIT revision: v3.7.5-2949-gc3c607c > GIT Date: 2017-01-20 17:34:16 -0600 > > [0]PETSC ERROR: ./ucmsLEP on a arch-linux2-c-debug named ocean by valera > Sat Feb 4 15:53:53 2017 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-fblaslapack --download-mpich --download-hdf5 > --download-netcdf --download-hypre --download-metis --download-parmetis > --download-trillinos --with-debugging=1 > > [0]PETSC ERROR: #3 VecSetLocalToGlobalMapping() line 79 in > /usr/dataC/home/valera/petsc/src/vec/vec/interface/vector.c > > [0]PETSC ERROR: #4 DMCreateGlobalVector_DA() line 41 in > /usr/dataC/home/valera/petsc/src/dm/impls/da/dadist.c > > [0]PETSC ERROR: #5 DMCreateGlobalVector() line 840 in > /usr/dataC/home/valera/petsc/src/dm/interface/dm.c > > > > > > > > So my question is, do i have to setup the vectors in any additional way? > i followed the fortran DMDA examples closely and i didnt notice anything > extra, > > > > Thanks, > > > > Manuel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Feb 4 19:34:22 2017 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 4 Feb 2017 19:34:22 -0600 Subject: [petsc-users] DMCreateGlobalVector in fortran. In-Reply-To: References: Message-ID: On Sat, Feb 4, 2017 at 7:30 PM, Manuel Valera wrote: > Thanks, that helped a lot, > > Now im a little confused with my next error, in my serial code i have > several arrays with dimensions declared as: > > u0(-1:IMax+2,-1:JMax+1,-1:KMax+1) > > > and i create my DMDA for this array as: > > > call CDA3(daub,IMax+4,JMax+3,KMax+3)call DAs(daub,u0p,u0pl,u0) > > > With the subroutines as: > > > SUBROUTINE CDA3(da3d,dim3dx,dim3dy,dim3dz) DM, intent(inout) :: da3d PetscInt, intent(in) :: dim3dx,dim3dy,dim3dz DMBoundaryType :: bx,by,bz bx = DM_BOUNDARY_NONE by = DM_BOUNDARY_NONE bz = DM_BOUNDARY_NONE call DMDACreate3d(PETSC_COMM_WORLD,bx,by,bz,DMDA_STENCIL_BOX,dim3dx,dim3dy,dim3dz,PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da3d,ierr) call DMSetFromOptions(da3d,ierr) call DMSetUp(da3d,ierr) END SUBROUTINE CDA3 SUBROUTINE DAs(da,globalv,localv,array) ! use PetscObjects, only :: ierr ! Umbrella program to update and communicate the arrays in a ! distributed fashion using the DMDA objects from PETSc. ! Manuel Valera 1/20/17 ! Arguments: ! da = DMDA array either 1d,2d or 3d, already created and setup ! globalv = global vector to be operated into ! localv = local chunk each processor works in. ! array = the array to be petscified. ONLY 3D ARRAYS rn. ! just in case. Vec,intent(inout) :: globalv,localv PetscScalar,dimension(:,:,:) :: array DM,intent(inout) :: da call DMCreateGlobalVector(da,globalv,ierr) call DMCreateLocalVector(da,localv,ierr) call DMGlobalToLocalBegin(da,globalv,INSERT_VALUES,localv,ierr) call DMGlobalToLocalEnd(da,globalv,INSERT_VALUES,localv,ierr) END SUBROUTINE DAs > > > Which i later open into an array to operate into it, but when i try to access u0(:,j-2,:) it gives me the following error: > > > Fortran runtime error: Index '-1' of dimension 2 of array 'u0' below lower bound of 0 > > > So i'm guessing the DMDA is creating the arrays with dimensions starting at 0 instead of -1 as it should, right? what can be done in this case? > > 1) The DMDA only takes sizes, not index bounds, in the constructor. 2) Its not the DMDA you must be talking about, but the array you get from DMDAVecGetArrayF90() or VecGetArrayF90() 3) The VecGetArrayF90() always start from 0 since it is indexed in local indices 4) The DMDVecGetArrayF90() arrays are indexed by global indices for the patch your process owns (including ghosts if its a local vector) Thanks, Matt > Thanks, > Manuel > > > On Sat, Feb 4, 2017 at 4:18 PM, Barry Smith wrote: > >> >> See: http://www.mcs.anl.gov/petsc/documentation/changes/dev.html >> search for DMDACreate and update your code. >> >> Barry >> >> >> > On Feb 4, 2017, at 6:02 PM, Manuel Valera >> wrote: >> > >> > Hello devs, >> > >> > Ive been creating the framework to distribute the arrays from my model >> using the global/local vectors and the DMDAs, for this i created a small >> subroutine that is responsible for making the necessary calls: >> > >> > SUBROUTINE DAs(da,globalv,localv) >> > >> > ! use PetscObjects, only :: ierr >> > >> > >> > ! Umbrella program to update and communicate the arrays in a >> > ! distributed fashion using the DMDA objects from PETSc. >> > ! Manuel Valera 1/20/17 >> > >> > ! Arguments: >> > ! da = DMDA array either 1d,2d or 3d, already created and setup >> > ! globalv = global vector to be operated into >> > ! localv = local chunk each processor works in. >> > >> > >> > >> > Vec,intent(inout) :: globalv,localv >> > >> > >> > >> > DM,intent(inout) :: da >> > >> > call DMCreateGlobalVector(da,globalv,ierr) >> > call DMCreateLocalVector(da,localv,ierr) >> > >> > call DMGlobalToLocalBegin(da,globalv,INSERT_VALUES,localv,ierr) >> > call DMGlobalToLocalEnd(da,globalv,INSERT_VALUES,localv,ierr) >> > >> > >> > >> > END SUBROUTINE DAs >> > >> > >> > I call this from another program where i have declared a vector as: >> > >> > #include "petsc/finclude/petscdm.h" >> > #include "petsc/finclude/petscdmda.h" >> > >> > use petsc >> > use petscdm >> > use petscdmda >> > >> > IMPLICIT NONE >> > >> > Vec :: xpdmda,yp,zp,xcp,ycp,zcp >> > DM :: da,dac,dau,dav,daw >> > >> > CONTAINS >> > SUBROUTINE InitGridP >> > >> > IMPLICIT NONE >> > !PetscInt :: ierr >> > >> > ! CREATE DMDA: >> > >> > call CDA3(da,IMax-1,JMax-1,KMax-1) >> > >> > !xpdmda,yp,zp have the same dimensions than x,y,z >> > !xpl,ypl,zpl are the local versions of xp,yp,zp >> > >> > !We create the global and local vectors and communicate their values: >> > call DAs(da,xpdmda,xpl,x) >> > >> > And at this call it crashes with the following message: >> > >> > [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> > [0]PETSC ERROR: Null argument, when expecting valid pointer >> > [0]PETSC ERROR: Null Object: Parameter # 2 >> > [0]PETSC ERROR: See >> > http://www.mcs.anl.gov/petsc/documentation/faq.html >> > for trouble shooting. >> > [0]PETSC ERROR: Petsc Development GIT revision: v3.7.5-2949-gc3c607c >> GIT Date: 2017-01-20 17:34:16 -0600 >> > [0]PETSC ERROR: ./ucmsLEP on a arch-linux2-c-debug named ocean by >> valera Sat Feb 4 15:53:53 2017 >> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >> --with-fc=gfortran --download-fblaslapack --download-mpich --download-hdf5 >> --download-netcdf --download-hypre --download-metis --download-parmetis >> --download-trillinos --with-debugging=1 >> > [0]PETSC ERROR: #3 VecSetLocalToGlobalMapping() line 79 in >> /usr/dataC/home/valera/petsc/src/vec/vec/interface/vector.c >> > [0]PETSC ERROR: #4 DMCreateGlobalVector_DA() line 41 in >> /usr/dataC/home/valera/petsc/src/dm/impls/da/dadist.c >> > [0]PETSC ERROR: #5 DMCreateGlobalVector() line 840 in >> /usr/dataC/home/valera/petsc/src/dm/interface/dm.c >> > >> > >> > >> > So my question is, do i have to setup the vectors in any additional >> way? i followed the fortran DMDA examples closely and i didnt notice >> anything extra, >> > >> > Thanks, >> > >> > Manuel >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Feb 4 19:35:21 2017 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 4 Feb 2017 19:35:21 -0600 Subject: [petsc-users] projection methods in TS In-Reply-To: <53402E0B-DEE9-4DE4-84AB-D8A33D94C566@anl.gov> References: <2E0519CB-7CA7-41A4-9E67-C973BE33F35C@mcs.anl.gov> <53402E0B-DEE9-4DE4-84AB-D8A33D94C566@anl.gov> Message-ID: On Sat, Feb 4, 2017 at 7:24 PM, Zhang, Hong wrote: > Can you elaborate a bit more on your problem? > > If your problem is an index-1 DAE, there is no need to use a projection > method, and it is perfectly fine to set it up as a DAE in PETSc. For > high-index DAEs, you may have to use TSSetPostStep() to implement your own > projection algorithm. > Please define index. Matt > If you happen to have a Hamiltonian system to solve, I have a symplectic > solver in my own branch that you can use directly. > > Hong (Mr.) > > On Feb 4, 2017, at 9:47 AM, Gideon Simpson > wrote: > > Would setting it up as a DAE in petsc be algorithmically euivalent to a > projected method (i.e., step of standard RK followed by nonlinear > projection)? > > -gideon > > On Feb 3, 2017, at 11:47 PM, Matthew Knepley wrote: > > That is one answer. Another one is that this particular system is a DAE > and we have methods for that. > > Matt > > On Fri, Feb 3, 2017 at 8:40 PM, Barry Smith wrote: > >> >> TSSetPostStep(); in your function use TSGetSolution() to get the current >> solution. >> >> Please let us know how it works out >> >> Barry >> >> >> >> > On Feb 3, 2017, at 7:14 PM, Gideon Simpson >> wrote: >> > >> > I?m interested in implementing a projection method for an ODE of the >> form: >> > >> > y? = f(y), >> > >> > such that g(y) = 0 for all time (i.e., g is conserved). Note that in a >> projection method, a standard time step is made to produce y* from y_{n}, >> and then this is corrected to obtain y_{n+1} satisfying g(y) = 0. >> > >> > There were two ways I was thinking of doing this, and I was hoping to >> get some input: >> > >> > Idea 1: Manually loop through using taking a time step and then >> implementing the projection routine. I see that there is a TSStep command, >> but this doesn?t seem to be much documentation on how to use it in this >> scenario. Does anyone have any guidance? >> > >> > Idea 2: Is there some analog to TSMonitor that allows me to modify the >> solution after each time step, instead of just allowing for some >> computation of a statistic? >> > >> > >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Sat Feb 4 19:44:40 2017 From: jed at jedbrown.org (Jed Brown) Date: Sat, 04 Feb 2017 18:44:40 -0700 Subject: [petsc-users] projection methods in TS In-Reply-To: References: <2E0519CB-7CA7-41A4-9E67-C973BE33F35C@mcs.anl.gov> <53402E0B-DEE9-4DE4-84AB-D8A33D94C566@anl.gov> Message-ID: <871svdif7b.fsf@jedbrown.org> Matthew Knepley writes: > On Sat, Feb 4, 2017 at 7:24 PM, Zhang, Hong wrote: > >> Can you elaborate a bit more on your problem? >> >> If your problem is an index-1 DAE, there is no need to use a projection >> method, and it is perfectly fine to set it up as a DAE in PETSc. For >> high-index DAEs, you may have to use TSSetPostStep() to implement your own >> projection algorithm. >> > > Please define index. Think of it as a measure of singularity of the "mass matrix". Higher index DAE have more complicated constraints on compatibility of initial conditions. It's covered in any book or paper on DAEs. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From bsmith at mcs.anl.gov Sat Feb 4 19:46:00 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 4 Feb 2017 19:46:00 -0600 Subject: [petsc-users] DMCreateGlobalVector in fortran. In-Reply-To: References: Message-ID: For DMDAVecGetArrayF90 you need to declare the "array" arguments as Fortran pointers you don't declare them like > u0(-1:IMax+2,-1:JMax+1,-1:KMax+1) > On Feb 4, 2017, at 7:34 PM, Matthew Knepley wrote: > > On Sat, Feb 4, 2017 at 7:30 PM, Manuel Valera wrote: > Thanks, that helped a lot, > > Now im a little confused with my next error, in my serial code i have several arrays with dimensions declared as: > > u0(-1:IMax+2,-1:JMax+1,-1:KMax+1) > > and i create my DMDA for this array as: > > call CDA3(daub,IMax+4,JMax+3,KMax+3) > call DAs(daub,u0p,u0pl,u0) > > With the subroutines as: > > SUBROUTINE CDA3(da3d,dim3dx,dim3dy,dim3dz) > > > > DM, intent(inout) :: da3d > PetscInt, intent(in) :: dim3dx,dim3dy,dim3dz > DMBoundaryType :: bx,by,bz > > > bx = DM_BOUNDARY_NONE > by = DM_BOUNDARY_NONE > bz = DM_BOUNDARY_NONE > > call DMDACreate3d(PETSC_COMM_WORLD,bx,by,bz,DMDA_STENCIL_BOX,dim3dx,dim3dy,dim3dz,PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da3d,ierr) > > call DMSetFromOptions(da3d,ierr) > call DMSetUp(da3d,ierr) > > > END SUBROUTINE CDA3 > > SUBROUTINE DAs(da,globalv,localv,array) > > ! use PetscObjects, only :: ierr > > > ! Umbrella program to update and communicate the arrays in a > ! distributed fashion using the DMDA objects from PETSc. > ! Manuel Valera 1/20/17 > > ! Arguments: > ! da = DMDA array either 1d,2d or 3d, already created and setup > ! globalv = global vector to be operated into > ! localv = local chunk each processor works in. > ! array = the array to be petscified. ONLY 3D ARRAYS rn. > ! just in case. > > Vec,intent(inout) :: globalv,localv > > > PetscScalar,dimension(:,:,:) :: array > DM,intent(inout) :: da > > > > > > > call DMCreateGlobalVector(da,globalv,ierr) > call DMCreateLocalVector(da,localv,ierr) > > call DMGlobalToLocalBegin(da,globalv,INSERT_VALUES,localv,ierr) > call DMGlobalToLocalEnd(da,globalv,INSERT_VALUES,localv,ierr) > > > > END SUBROUTINE DAs > > Which i later open into an array to operate into it, but when i try to access u0(:,j-2,:) it gives me the following error: > > Fortran runtime error: Index '-1' of dimension 2 of array 'u0' below lower bound of 0 > > So i'm guessing the DMDA is creating the arrays with dimensions starting at 0 instead of -1 as it should, right? what can be done in this case? > > 1) The DMDA only takes sizes, not index bounds, in the constructor. > > 2) Its not the DMDA you must be talking about, but the array you get from DMDAVecGetArrayF90() or VecGetArrayF90() > > 3) The VecGetArrayF90() always start from 0 since it is indexed in local indices > > 4) The DMDVecGetArrayF90() arrays are indexed by global indices for the patch your process owns (including ghosts if its a local vector) > > Thanks, > > Matt > > Thanks, > Manuel > > On Sat, Feb 4, 2017 at 4:18 PM, Barry Smith wrote: > > See: http://www.mcs.anl.gov/petsc/documentation/changes/dev.html search for DMDACreate and update your code. > > Barry > > > > On Feb 4, 2017, at 6:02 PM, Manuel Valera wrote: > > > > Hello devs, > > > > Ive been creating the framework to distribute the arrays from my model using the global/local vectors and the DMDAs, for this i created a small subroutine that is responsible for making the necessary calls: > > > > SUBROUTINE DAs(da,globalv,localv) > > > > ! use PetscObjects, only :: ierr > > > > > > ! Umbrella program to update and communicate the arrays in a > > ! distributed fashion using the DMDA objects from PETSc. > > ! Manuel Valera 1/20/17 > > > > ! Arguments: > > ! da = DMDA array either 1d,2d or 3d, already created and setup > > ! globalv = global vector to be operated into > > ! localv = local chunk each processor works in. > > > > > > > > Vec,intent(inout) :: globalv,localv > > > > > > > > DM,intent(inout) :: da > > > > call DMCreateGlobalVector(da,globalv,ierr) > > call DMCreateLocalVector(da,localv,ierr) > > > > call DMGlobalToLocalBegin(da,globalv,INSERT_VALUES,localv,ierr) > > call DMGlobalToLocalEnd(da,globalv,INSERT_VALUES,localv,ierr) > > > > > > > > END SUBROUTINE DAs > > > > > > I call this from another program where i have declared a vector as: > > > > #include "petsc/finclude/petscdm.h" > > #include "petsc/finclude/petscdmda.h" > > > > use petsc > > use petscdm > > use petscdmda > > > > IMPLICIT NONE > > > > Vec :: xpdmda,yp,zp,xcp,ycp,zcp > > DM :: da,dac,dau,dav,daw > > > > CONTAINS > > SUBROUTINE InitGridP > > > > IMPLICIT NONE > > !PetscInt :: ierr > > > > ! CREATE DMDA: > > > > call CDA3(da,IMax-1,JMax-1,KMax-1) > > > > !xpdmda,yp,zp have the same dimensions than x,y,z > > !xpl,ypl,zpl are the local versions of xp,yp,zp > > > > !We create the global and local vectors and communicate their values: > > call DAs(da,xpdmda,xpl,x) > > > > And at this call it crashes with the following message: > > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [0]PETSC ERROR: Null argument, when expecting valid pointer > > [0]PETSC ERROR: Null Object: Parameter # 2 > > [0]PETSC ERROR: See > > http://www.mcs.anl.gov/petsc/documentation/faq.html > > for trouble shooting. > > [0]PETSC ERROR: Petsc Development GIT revision: v3.7.5-2949-gc3c607c GIT Date: 2017-01-20 17:34:16 -0600 > > [0]PETSC ERROR: ./ucmsLEP on a arch-linux2-c-debug named ocean by valera Sat Feb 4 15:53:53 2017 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-hdf5 --download-netcdf --download-hypre --download-metis --download-parmetis --download-trillinos --with-debugging=1 > > [0]PETSC ERROR: #3 VecSetLocalToGlobalMapping() line 79 in /usr/dataC/home/valera/petsc/src/vec/vec/interface/vector.c > > [0]PETSC ERROR: #4 DMCreateGlobalVector_DA() line 41 in /usr/dataC/home/valera/petsc/src/dm/impls/da/dadist.c > > [0]PETSC ERROR: #5 DMCreateGlobalVector() line 840 in /usr/dataC/home/valera/petsc/src/dm/interface/dm.c > > > > > > > > So my question is, do i have to setup the vectors in any additional way? i followed the fortran DMDA examples closely and i didnt notice anything extra, > > > > Thanks, > > > > Manuel > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From knepley at gmail.com Sat Feb 4 19:47:41 2017 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 4 Feb 2017 19:47:41 -0600 Subject: [petsc-users] projection methods in TS In-Reply-To: <871svdif7b.fsf@jedbrown.org> References: <2E0519CB-7CA7-41A4-9E67-C973BE33F35C@mcs.anl.gov> <53402E0B-DEE9-4DE4-84AB-D8A33D94C566@anl.gov> <871svdif7b.fsf@jedbrown.org> Message-ID: On Sat, Feb 4, 2017 at 7:44 PM, Jed Brown wrote: > Matthew Knepley writes: > > > On Sat, Feb 4, 2017 at 7:24 PM, Zhang, Hong wrote: > > > >> Can you elaborate a bit more on your problem? > >> > >> If your problem is an index-1 DAE, there is no need to use a projection > >> method, and it is perfectly fine to set it up as a DAE in PETSc. For > >> high-index DAEs, you may have to use TSSetPostStep() to implement your > own > >> projection algorithm. > >> > > > > Please define index. > > Think of it as a measure of singularity of the "mass matrix". Higher > index DAE have more complicated constraints on compatibility of initial > conditions. It's covered in any book or paper on DAEs. > Both your explanation and Hong's use of the term do not help Gideon (or me) know whether he has an index-1 DAE. There has to be some simple form you can write down so that we can tell. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Feb 4 19:48:05 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 4 Feb 2017 19:48:05 -0600 Subject: [petsc-users] projection methods in TS In-Reply-To: References: <2E0519CB-7CA7-41A4-9E67-C973BE33F35C@mcs.anl.gov> <53402E0B-DEE9-4DE4-84AB-D8A33D94C566@anl.gov> Message-ID: <71CA835F-82D0-4965-9048-A869C6FE8AC4@mcs.anl.gov> > On Feb 4, 2017, at 7:35 PM, Matthew Knepley wrote: > > On Sat, Feb 4, 2017 at 7:24 PM, Zhang, Hong wrote: > Can you elaborate a bit more on your problem? > > If your problem is an index-1 DAE, there is no need to use a projection method, and it is perfectly fine to set it up as a DAE in PETSc. For high-index DAEs, you may have to use TSSetPostStep() to implement your own projection algorithm. > > Please define index. Please read one of Petzold's elementary books :-) > > Matt > > If you happen to have a Hamiltonian system to solve, I have a symplectic solver in my own branch that you can use directly. > > Hong (Mr.) > >> On Feb 4, 2017, at 9:47 AM, Gideon Simpson wrote: >> >> Would setting it up as a DAE in petsc be algorithmically euivalent to a projected method (i.e., step of standard RK followed by nonlinear projection)? >> >> -gideon >> >>> On Feb 3, 2017, at 11:47 PM, Matthew Knepley wrote: >>> >>> That is one answer. Another one is that this particular system is a DAE and we have methods for that. >>> >>> Matt >>> >>> On Fri, Feb 3, 2017 at 8:40 PM, Barry Smith wrote: >>> >>> TSSetPostStep(); in your function use TSGetSolution() to get the current solution. >>> >>> Please let us know how it works out >>> >>> Barry >>> >>> >>> >>> > On Feb 3, 2017, at 7:14 PM, Gideon Simpson wrote: >>> > >>> > I?m interested in implementing a projection method for an ODE of the form: >>> > >>> > y? = f(y), >>> > >>> > such that g(y) = 0 for all time (i.e., g is conserved). Note that in a projection method, a standard time step is made to produce y* from y_{n}, and then this is corrected to obtain y_{n+1} satisfying g(y) = 0. >>> > >>> > There were two ways I was thinking of doing this, and I was hoping to get some input: >>> > >>> > Idea 1: Manually loop through using taking a time step and then implementing the projection routine. I see that there is a TSStep command, but this doesn?t seem to be much documentation on how to use it in this scenario. Does anyone have any guidance? >>> > >>> > Idea 2: Is there some analog to TSMonitor that allows me to modify the solution after each time step, instead of just allowing for some computation of a statistic? >>> > >>> > >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >> > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From bsmith at mcs.anl.gov Sat Feb 4 19:49:11 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 4 Feb 2017 19:49:11 -0600 Subject: [petsc-users] projection methods in TS In-Reply-To: <53402E0B-DEE9-4DE4-84AB-D8A33D94C566@anl.gov> References: <2E0519CB-7CA7-41A4-9E67-C973BE33F35C@mcs.anl.gov> <53402E0B-DEE9-4DE4-84AB-D8A33D94C566@anl.gov> Message-ID: > On Feb 4, 2017, at 7:24 PM, Zhang, Hong wrote: > > Can you elaborate a bit more on your problem? > > If your problem is an index-1 DAE, there is no need to use a projection method, and it is perfectly fine to set it up as a DAE in PETSc. For high-index DAEs, you may have to use TSSetPostStep() to implement your own projection algorithm. > > If you happen to have a Hamiltonian system to solve, I have a symplectic solver in my own branch that you can use directly. Why has this not been moved into master? Feature branches that linger outside of master end up being a pain. > > Hong (Mr.) > >> On Feb 4, 2017, at 9:47 AM, Gideon Simpson wrote: >> >> Would setting it up as a DAE in petsc be algorithmically euivalent to a projected method (i.e., step of standard RK followed by nonlinear projection)? >> >> -gideon >> >>> On Feb 3, 2017, at 11:47 PM, Matthew Knepley wrote: >>> >>> That is one answer. Another one is that this particular system is a DAE and we have methods for that. >>> >>> Matt >>> >>> On Fri, Feb 3, 2017 at 8:40 PM, Barry Smith wrote: >>> >>> TSSetPostStep(); in your function use TSGetSolution() to get the current solution. >>> >>> Please let us know how it works out >>> >>> Barry >>> >>> >>> >>> > On Feb 3, 2017, at 7:14 PM, Gideon Simpson wrote: >>> > >>> > I?m interested in implementing a projection method for an ODE of the form: >>> > >>> > y? = f(y), >>> > >>> > such that g(y) = 0 for all time (i.e., g is conserved). Note that in a projection method, a standard time step is made to produce y* from y_{n}, and then this is corrected to obtain y_{n+1} satisfying g(y) = 0. >>> > >>> > There were two ways I was thinking of doing this, and I was hoping to get some input: >>> > >>> > Idea 1: Manually loop through using taking a time step and then implementing the projection routine. I see that there is a TSStep command, but this doesn?t seem to be much documentation on how to use it in this scenario. Does anyone have any guidance? >>> > >>> > Idea 2: Is there some analog to TSMonitor that allows me to modify the solution after each time step, instead of just allowing for some computation of a statistic? >>> > >>> > >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >> > From hongzhang at anl.gov Sat Feb 4 19:49:15 2017 From: hongzhang at anl.gov (Zhang, Hong) Date: Sun, 5 Feb 2017 01:49:15 +0000 Subject: [petsc-users] projection methods in TS In-Reply-To: References: <2E0519CB-7CA7-41A4-9E67-C973BE33F35C@mcs.anl.gov> <53402E0B-DEE9-4DE4-84AB-D8A33D94C566@anl.gov> Message-ID: <8CE3BB7F-4D87-446C-A691-064FAE73BF0A@anl.gov> On Feb 4, 2017, at 7:35 PM, Matthew Knepley > wrote: On Sat, Feb 4, 2017 at 7:24 PM, Zhang, Hong > wrote: Can you elaborate a bit more on your problem? If your problem is an index-1 DAE, there is no need to use a projection method, and it is perfectly fine to set it up as a DAE in PETSc. For high-index DAEs, you may have to use TSSetPostStep() to implement your own projection algorithm. Please define index. http://www.scholarpedia.org/article/Differential-algebraic_equations#Index Hong Matt If you happen to have a Hamiltonian system to solve, I have a symplectic solver in my own branch that you can use directly. Hong (Mr.) On Feb 4, 2017, at 9:47 AM, Gideon Simpson > wrote: Would setting it up as a DAE in petsc be algorithmically euivalent to a projected method (i.e., step of standard RK followed by nonlinear projection)? -gideon On Feb 3, 2017, at 11:47 PM, Matthew Knepley > wrote: That is one answer. Another one is that this particular system is a DAE and we have methods for that. Matt On Fri, Feb 3, 2017 at 8:40 PM, Barry Smith > wrote: TSSetPostStep(); in your function use TSGetSolution() to get the current solution. Please let us know how it works out Barry > On Feb 3, 2017, at 7:14 PM, Gideon Simpson > wrote: > > I?m interested in implementing a projection method for an ODE of the form: > > y? = f(y), > > such that g(y) = 0 for all time (i.e., g is conserved). Note that in a projection method, a standard time step is made to produce y* from y_{n}, and then this is corrected to obtain y_{n+1} satisfying g(y) = 0. > > There were two ways I was thinking of doing this, and I was hoping to get some input: > > Idea 1: Manually loop through using taking a time step and then implementing the projection routine. I see that there is a TSStep command, but this doesn?t seem to be much documentation on how to use it in this scenario. Does anyone have any guidance? > > Idea 2: Is there some analog to TSMonitor that allows me to modify the solution after each time step, instead of just allowing for some computation of a statistic? > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hongzhang at anl.gov Sat Feb 4 19:56:41 2017 From: hongzhang at anl.gov (Zhang, Hong) Date: Sun, 5 Feb 2017 01:56:41 +0000 Subject: [petsc-users] projection methods in TS In-Reply-To: References: <2E0519CB-7CA7-41A4-9E67-C973BE33F35C@mcs.anl.gov> <53402E0B-DEE9-4DE4-84AB-D8A33D94C566@anl.gov> <871svdif7b.fsf@jedbrown.org> Message-ID: On Feb 4, 2017, at 7:47 PM, Matthew Knepley > wrote: On Sat, Feb 4, 2017 at 7:44 PM, Jed Brown > wrote: Matthew Knepley > writes: > On Sat, Feb 4, 2017 at 7:24 PM, Zhang, Hong > wrote: > >> Can you elaborate a bit more on your problem? >> >> If your problem is an index-1 DAE, there is no need to use a projection >> method, and it is perfectly fine to set it up as a DAE in PETSc. For >> high-index DAEs, you may have to use TSSetPostStep() to implement your own >> projection algorithm. >> > > Please define index. Think of it as a measure of singularity of the "mass matrix". Higher index DAE have more complicated constraints on compatibility of initial conditions. It's covered in any book or paper on DAEs. Both your explanation and Hong's use of the term do not help Gideon (or me) know whether he has an index-1 DAE. There has to be some simple form you can write down so that we can tell. This is why we need to learn more about Gideon's problem. It is easy to determine the index if he can write down his problem in a simple form. But it is not that easy the other way round. Hong Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Feb 4 19:58:59 2017 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 4 Feb 2017 19:58:59 -0600 Subject: [petsc-users] projection methods in TS In-Reply-To: References: <2E0519CB-7CA7-41A4-9E67-C973BE33F35C@mcs.anl.gov> <53402E0B-DEE9-4DE4-84AB-D8A33D94C566@anl.gov> <871svdif7b.fsf@jedbrown.org> Message-ID: On Sat, Feb 4, 2017 at 7:56 PM, Zhang, Hong wrote: > On Feb 4, 2017, at 7:47 PM, Matthew Knepley wrote: > > On Sat, Feb 4, 2017 at 7:44 PM, Jed Brown wrote: > >> Matthew Knepley writes: >> >> > On Sat, Feb 4, 2017 at 7:24 PM, Zhang, Hong wrote: >> > >> >> Can you elaborate a bit more on your problem? >> >> >> >> If your problem is an index-1 DAE, there is no need to use a projection >> >> method, and it is perfectly fine to set it up as a DAE in PETSc. For >> >> high-index DAEs, you may have to use TSSetPostStep() to implement your >> own >> >> projection algorithm. >> >> >> > >> > Please define index. >> >> Think of it as a measure of singularity of the "mass matrix". Higher >> index DAE have more complicated constraints on compatibility of initial >> conditions. It's covered in any book or paper on DAEs. >> > > Both your explanation and Hong's use of the term do not help Gideon (or > me) know whether he has an index-1 DAE. There has > to be some simple form you can write down so that we can tell. > > > This is why we need to learn more about Gideon's problem. It is easy to > determine the index if he can write down his problem in a simple form. But > it is not that easy the other way round. > He says y' = f(y) 0 = g(y) which appears to me to be a Hessenberg index-2 DAE. Is that correct? Matt Hong > > > Matt > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hongzhang at anl.gov Sat Feb 4 20:27:39 2017 From: hongzhang at anl.gov (Zhang, Hong) Date: Sun, 5 Feb 2017 02:27:39 +0000 Subject: [petsc-users] projection methods in TS In-Reply-To: References: <2E0519CB-7CA7-41A4-9E67-C973BE33F35C@mcs.anl.gov> <53402E0B-DEE9-4DE4-84AB-D8A33D94C566@anl.gov> <871svdif7b.fsf@jedbrown.org> Message-ID: <208B4205-4A26-4D82-843A-4E6C5931F63C@anl.gov> On Feb 4, 2017, at 7:58 PM, Matthew Knepley > wrote: On Sat, Feb 4, 2017 at 7:56 PM, Zhang, Hong > wrote: On Feb 4, 2017, at 7:47 PM, Matthew Knepley > wrote: On Sat, Feb 4, 2017 at 7:44 PM, Jed Brown > wrote: Matthew Knepley > writes: > On Sat, Feb 4, 2017 at 7:24 PM, Zhang, Hong > wrote: > >> Can you elaborate a bit more on your problem? >> >> If your problem is an index-1 DAE, there is no need to use a projection >> method, and it is perfectly fine to set it up as a DAE in PETSc. For >> high-index DAEs, you may have to use TSSetPostStep() to implement your own >> projection algorithm. >> > > Please define index. Think of it as a measure of singularity of the "mass matrix". Higher index DAE have more complicated constraints on compatibility of initial conditions. It's covered in any book or paper on DAEs. Both your explanation and Hong's use of the term do not help Gideon (or me) know whether he has an index-1 DAE. There has to be some simple form you can write down so that we can tell. This is why we need to learn more about Gideon's problem. It is easy to determine the index if he can write down his problem in a simple form. But it is not that easy the other way round. He says y' = f(y) 0 = g(y) which appears to me to be a Hessenberg index-2 DAE. Is that correct? This is not a DAE form because neither f() nor g() contains the algebraic variable. Hong Matt Hong Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Sat Feb 4 20:30:39 2017 From: jed at jedbrown.org (Jed Brown) Date: Sat, 04 Feb 2017 19:30:39 -0700 Subject: [petsc-users] projection methods in TS In-Reply-To: References: <2E0519CB-7CA7-41A4-9E67-C973BE33F35C@mcs.anl.gov> <53402E0B-DEE9-4DE4-84AB-D8A33D94C566@anl.gov> <871svdif7b.fsf@jedbrown.org> Message-ID: <87y3xlgyi8.fsf@jedbrown.org> Matthew Knepley writes: > He says > > y' = f(y) > > 0 = g(y) > > which appears to me to be a Hessenberg index-2 DAE. Is that correct? The equations above are obviously overdetermined as stated. Perhaps there are more variables? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From emconsta at mcs.anl.gov Sat Feb 4 21:26:24 2017 From: emconsta at mcs.anl.gov (Emil Constantinescu) Date: Sat, 4 Feb 2017 21:26:24 -0600 Subject: [petsc-users] projection methods in TS In-Reply-To: References: <2E0519CB-7CA7-41A4-9E67-C973BE33F35C@mcs.anl.gov> Message-ID: <57c95344-7062-e23a-d2e9-11c04bf18975@mcs.anl.gov> On 2/4/17 11:00 AM, Matthew Knepley wrote: > On Sat, Feb 4, 2017 at 9:47 AM, Gideon Simpson > wrote: > > Would setting it up as a DAE in petsc be algorithmically euivalent > to a projected method (i.e., step of standard RK followed by > nonlinear projection)? Not quite, if you set it as a DAE, then both the time stepping part and the algebraic constraint are solved at the same time (you are going to use an implicit time stepping method. So for backward Euler and DAE, PETSc solves something like: y_{n+1} - y{n} - dt* f(y_{n+1}) = 0 g(y_{n+1}) = 0 By projection (assuming index 1 or 2): you are solving y* - y{n} - dt* f(y*) = 0 then initialize SNES to y* and solve: g(y_{n+1}) = 0 In some cases g(y_{n+1}) may not have a value (you may not land on the manifold that is defined by the conservation property). Whereas, if you define it as a DAE in almost all cases (there are crazy exceptions) you will. Emil > > I am not sure, as I do not understand those solvers. However, I wrote my > own solver that does exactly that MIMEX. > > Matt > > > -gideon > >> On Feb 3, 2017, at 11:47 PM, Matthew Knepley > > wrote: >> >> That is one answer. Another one is that this particular system is >> a DAE and we have methods for that. >> >> Matt >> >> On Fri, Feb 3, 2017 at 8:40 PM, Barry Smith > > wrote: >> >> >> TSSetPostStep(); in your function use TSGetSolution() to get >> the current solution. >> >> Please let us know how it works out >> >> Barry >> >> >> >> > On Feb 3, 2017, at 7:14 PM, Gideon Simpson >> > >> wrote: >> > >> > I?m interested in implementing a projection method for an >> ODE of the form: >> > >> > y? = f(y), >> > >> > such that g(y) = 0 for all time (i.e., g is conserved). >> Note that in a projection method, a standard time step is made >> to produce y* from y_{n}, and then this is corrected to obtain >> y_{n+1} satisfying g(y) = 0. >> > >> > There were two ways I was thinking of doing this, and I was >> hoping to get some input: >> > >> > Idea 1: Manually loop through using taking a time step and >> then implementing the projection routine. I see that there is >> a TSStep command, but this doesn?t seem to be much >> documentation on how to use it in this scenario. Does anyone >> have any guidance? >> > >> > Idea 2: Is there some analog to TSMonitor that allows me to >> modify the solution after each time step, instead of just >> allowing for some computation of a statistic? >> > >> > >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener From bsmith at mcs.anl.gov Sat Feb 4 21:37:19 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 4 Feb 2017 21:37:19 -0600 Subject: [petsc-users] projection methods in TS In-Reply-To: <57c95344-7062-e23a-d2e9-11c04bf18975@mcs.anl.gov> References: <2E0519CB-7CA7-41A4-9E67-C973BE33F35C@mcs.anl.gov> <57c95344-7062-e23a-d2e9-11c04bf18975@mcs.anl.gov> Message-ID: <8A5E3100-C5E6-4D08-BD26-139C2B161E8F@mcs.anl.gov> Emil, Information about "projection methods" vs solving as a DAE should be documented in the users manual, manual pages, or FAQ.... > On Feb 4, 2017, at 9:26 PM, Emil Constantinescu wrote: > > > > On 2/4/17 11:00 AM, Matthew Knepley wrote: >> On Sat, Feb 4, 2017 at 9:47 AM, Gideon Simpson > > wrote: >> >> Would setting it up as a DAE in petsc be algorithmically euivalent >> to a projected method (i.e., step of standard RK followed by >> nonlinear projection)? > > Not quite, if you set it as a DAE, then both the time stepping part and the algebraic constraint are solved at the same time (you are going to use an implicit time stepping method. > > So for backward Euler and DAE, PETSc solves something like: > > y_{n+1} - y{n} - dt* f(y_{n+1}) = 0 > g(y_{n+1}) = 0 > > By projection (assuming index 1 or 2): you are solving > > y* - y{n} - dt* f(y*) = 0 > > then initialize SNES to y* and solve: > > g(y_{n+1}) = 0 > > In some cases g(y_{n+1}) may not have a value (you may not land on the manifold that is defined by the conservation property). Whereas, if you define it as a DAE in almost all cases (there are crazy exceptions) you will. > > > Emil > >> >> I am not sure, as I do not understand those solvers. However, I wrote my >> own solver that does exactly that MIMEX. >> >> Matt >> >> >> -gideon >> >>> On Feb 3, 2017, at 11:47 PM, Matthew Knepley >> > wrote: >>> >>> That is one answer. Another one is that this particular system is >>> a DAE and we have methods for that. >>> >>> Matt >>> >>> On Fri, Feb 3, 2017 at 8:40 PM, Barry Smith >> > wrote: >>> >>> >>> TSSetPostStep(); in your function use TSGetSolution() to get >>> the current solution. >>> >>> Please let us know how it works out >>> >>> Barry >>> >>> >>> >>> > On Feb 3, 2017, at 7:14 PM, Gideon Simpson >>> > >>> wrote: >>> > >>> > I?m interested in implementing a projection method for an >>> ODE of the form: >>> > >>> > y? = f(y), >>> > >>> > such that g(y) = 0 for all time (i.e., g is conserved). >>> Note that in a projection method, a standard time step is made >>> to produce y* from y_{n}, and then this is corrected to obtain >>> y_{n+1} satisfying g(y) = 0. >>> > >>> > There were two ways I was thinking of doing this, and I was >>> hoping to get some input: >>> > >>> > Idea 1: Manually loop through using taking a time step and >>> then implementing the projection routine. I see that there is >>> a TSStep command, but this doesn?t seem to be much >>> documentation on how to use it in this scenario. Does anyone >>> have any guidance? >>> > >>> > Idea 2: Is there some analog to TSMonitor that allows me to >>> modify the solution after each time step, instead of just >>> allowing for some computation of a statistic? >>> > >>> > >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to >>> which their experiments lead. >>> -- Norbert Wiener >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which >> their experiments lead. >> -- Norbert Wiener From mvalera at mail.sdsu.edu Sat Feb 4 22:19:49 2017 From: mvalera at mail.sdsu.edu (Manuel Valera) Date: Sat, 4 Feb 2017 20:19:49 -0800 Subject: [petsc-users] DMCreateGlobalVector in fortran. In-Reply-To: References: Message-ID: Ok, let me try to clarify a little bit, I have a u0 3d array with the mentioned indices (starting from -1), im creating a dmda with the sizes in the constructor, no problem with that. Next, im trying to distribute this u0 array into global and local vectors, apparently i can do that since no error comes out. Now, when i get the array back with VecGetArrayF90(), the indices start now from 0, is that right? so the whole array have been shifted in its indices. This changes things for me because my idea was to use the previous serial code to operate on the arrays, but this is written with indices starting from -1. There are dozens of arrays with the same structure, but they also operate with other arrays with indices starting from 1, at the same time, usually in triple nested loops which code the stencil used within them, to update and correct the velocities after solving for pressure, per example. So it would be much easier to shift the indices of these arrays by one, rather than changing all the code to match the new indices starting from zero, Is there a way to do this? Thanks, On Sat, Feb 4, 2017 at 5:46 PM, Barry Smith wrote: > > For DMDAVecGetArrayF90 you need to declare the "array" arguments as > Fortran pointers you don't declare them like > > > u0(-1:IMax+2,-1:JMax+1,-1:KMax+1) > > > > > On Feb 4, 2017, at 7:34 PM, Matthew Knepley wrote: > > > > On Sat, Feb 4, 2017 at 7:30 PM, Manuel Valera > wrote: > > Thanks, that helped a lot, > > > > Now im a little confused with my next error, in my serial code i have > several arrays with dimensions declared as: > > > > u0(-1:IMax+2,-1:JMax+1,-1:KMax+1) > > > > and i create my DMDA for this array as: > > > > call CDA3(daub,IMax+4,JMax+3,KMax+3) > > call DAs(daub,u0p,u0pl,u0) > > > > With the subroutines as: > > > > SUBROUTINE CDA3(da3d,dim3dx,dim3dy,dim3dz) > > > > > > > > DM, intent(inout) :: da3d > > PetscInt, intent(in) :: dim3dx,dim3dy,dim3dz > > DMBoundaryType :: bx,by,bz > > > > > > bx = DM_BOUNDARY_NONE > > by = DM_BOUNDARY_NONE > > bz = DM_BOUNDARY_NONE > > > > call DMDACreate3d(PETSC_COMM_WORLD,bx,by,bz,DMDA_STENCIL_BOX, > dim3dx,dim3dy,dim3dz,PETSC_DECIDE,PETSC_DECIDE,PETSC_ > DECIDE,1,1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_ > INTEGER,da3d,ierr) > > > > call DMSetFromOptions(da3d,ierr) > > call DMSetUp(da3d,ierr) > > > > > > END SUBROUTINE CDA3 > > > > SUBROUTINE DAs(da,globalv,localv,array) > > > > ! use PetscObjects, only :: ierr > > > > > > ! Umbrella program to update and communicate the arrays in a > > ! distributed fashion using the DMDA objects from PETSc. > > ! Manuel Valera 1/20/17 > > > > ! Arguments: > > ! da = DMDA array either 1d,2d or 3d, already created and setup > > ! globalv = global vector to be operated into > > ! localv = local chunk each processor works in. > > ! array = the array to be petscified. ONLY 3D ARRAYS rn. > > ! just in case. > > > > Vec,intent(inout) :: globalv,localv > > > > > > PetscScalar,dimension(:,:,:) :: array > > DM,intent(inout) :: da > > > > > > > > > > > > > > call DMCreateGlobalVector(da,globalv,ierr) > > call DMCreateLocalVector(da,localv,ierr) > > > > call DMGlobalToLocalBegin(da,globalv,INSERT_VALUES,localv,ierr) > > call DMGlobalToLocalEnd(da,globalv,INSERT_VALUES,localv,ierr) > > > > > > > > END SUBROUTINE DAs > > > > Which i later open into an array to operate into it, but when i try to > access u0(:,j-2,:) it gives me the following error: > > > > Fortran runtime error: Index '-1' of dimension 2 of array 'u0' below > lower bound of 0 > > > > So i'm guessing the DMDA is creating the arrays with dimensions starting > at 0 instead of -1 as it should, right? what can be done in this case? > > > > 1) The DMDA only takes sizes, not index bounds, in the constructor. > > > > 2) Its not the DMDA you must be talking about, but the array you get > from DMDAVecGetArrayF90() or VecGetArrayF90() > > > > 3) The VecGetArrayF90() always start from 0 since it is indexed in local > indices > > > > 4) The DMDVecGetArrayF90() arrays are indexed by global indices for the > patch your process owns (including ghosts if its a local vector) > > > > Thanks, > > > > Matt > > > > Thanks, > > Manuel > > > > On Sat, Feb 4, 2017 at 4:18 PM, Barry Smith wrote: > > > > See: http://www.mcs.anl.gov/petsc/documentation/changes/dev.html > search for DMDACreate and update your code. > > > > Barry > > > > > > > On Feb 4, 2017, at 6:02 PM, Manuel Valera > wrote: > > > > > > Hello devs, > > > > > > Ive been creating the framework to distribute the arrays from my model > using the global/local vectors and the DMDAs, for this i created a small > subroutine that is responsible for making the necessary calls: > > > > > > SUBROUTINE DAs(da,globalv,localv) > > > > > > ! use PetscObjects, only :: ierr > > > > > > > > > ! Umbrella program to update and communicate the arrays in a > > > ! distributed fashion using the DMDA objects from PETSc. > > > ! Manuel Valera 1/20/17 > > > > > > ! Arguments: > > > ! da = DMDA array either 1d,2d or 3d, already created and setup > > > ! globalv = global vector to be operated into > > > ! localv = local chunk each processor works in. > > > > > > > > > > > > Vec,intent(inout) :: globalv,localv > > > > > > > > > > > > DM,intent(inout) :: da > > > > > > call DMCreateGlobalVector(da,globalv,ierr) > > > call DMCreateLocalVector(da,localv,ierr) > > > > > > call DMGlobalToLocalBegin(da,globalv,INSERT_VALUES,localv, > ierr) > > > call DMGlobalToLocalEnd(da,globalv,INSERT_VALUES,localv,ierr) > > > > > > > > > > > > END SUBROUTINE DAs > > > > > > > > > I call this from another program where i have declared a vector as: > > > > > > #include "petsc/finclude/petscdm.h" > > > #include "petsc/finclude/petscdmda.h" > > > > > > use petsc > > > use petscdm > > > use petscdmda > > > > > > IMPLICIT NONE > > > > > > Vec :: xpdmda,yp,zp,xcp,ycp,zcp > > > DM :: da,dac,dau,dav,daw > > > > > > CONTAINS > > > SUBROUTINE InitGridP > > > > > > IMPLICIT NONE > > > !PetscInt :: ierr > > > > > > ! CREATE DMDA: > > > > > > call CDA3(da,IMax-1,JMax-1,KMax-1) > > > > > > !xpdmda,yp,zp have the same dimensions than x,y,z > > > !xpl,ypl,zpl are the local versions of xp,yp,zp > > > > > > !We create the global and local vectors and communicate their values: > > > call DAs(da,xpdmda,xpl,x) > > > > > > And at this call it crashes with the following message: > > > > > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > > [0]PETSC ERROR: Null argument, when expecting valid pointer > > > [0]PETSC ERROR: Null Object: Parameter # 2 > > > [0]PETSC ERROR: See > > > http://www.mcs.anl.gov/petsc/documentation/faq.html > > > for trouble shooting. > > > [0]PETSC ERROR: Petsc Development GIT revision: v3.7.5-2949-gc3c607c > GIT Date: 2017-01-20 17:34:16 -0600 > > > [0]PETSC ERROR: ./ucmsLEP on a arch-linux2-c-debug named ocean by > valera Sat Feb 4 15:53:53 2017 > > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-fblaslapack --download-mpich --download-hdf5 > --download-netcdf --download-hypre --download-metis --download-parmetis > --download-trillinos --with-debugging=1 > > > [0]PETSC ERROR: #3 VecSetLocalToGlobalMapping() line 79 in > /usr/dataC/home/valera/petsc/src/vec/vec/interface/vector.c > > > [0]PETSC ERROR: #4 DMCreateGlobalVector_DA() line 41 in > /usr/dataC/home/valera/petsc/src/dm/impls/da/dadist.c > > > [0]PETSC ERROR: #5 DMCreateGlobalVector() line 840 in > /usr/dataC/home/valera/petsc/src/dm/interface/dm.c > > > > > > > > > > > > So my question is, do i have to setup the vectors in any additional > way? i followed the fortran DMDA examples closely and i didnt notice > anything extra, > > > > > > Thanks, > > > > > > Manuel > > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Sat Feb 4 22:37:01 2017 From: jed at jedbrown.org (Jed Brown) Date: Sat, 04 Feb 2017 21:37:01 -0700 Subject: [petsc-users] DMCreateGlobalVector in fortran. In-Reply-To: References: Message-ID: <87tw89gsnm.fsf@jedbrown.org> Manuel Valera writes: > Ok, let me try to clarify a little bit, > > I have a u0 3d array with the mentioned indices (starting from -1), im > creating a dmda with the sizes in the constructor, no problem with that. > > Next, im trying to distribute this u0 array into global and local vectors, > apparently i can do that since no error comes out. > > Now, when i get the array back with VecGetArrayF90(), the indices start now > from 0, is that right? so the whole array have been shifted in its indices. See FormFunction and FormFunctionLocal. https://www.mcs.anl.gov/petsc/petsc-dev/src/snes/examples/tutorials/ex5f90.F90.html Of course you can just define FormFunctionLocal to use an index range that is shifted from PetscScalar x(user%gxs:user%gxe,user%gys:user%gye) PetscScalar f(user%xs:user%xe,user%ys:user%ye) (this is just Fortran array passing; nothing to do with PETSc). > This changes things for me because my idea was to use the previous serial > code to operate on the arrays, but this is written with indices starting > from -1. There are dozens of arrays with the same structure, but they also > operate with other arrays with indices starting from 1, at the same time, > usually in triple nested loops which code the stencil used within them, to > update and correct the velocities after solving for pressure, per example. > > So it would be much easier to shift the indices of these arrays by one, > rather than changing all the code to match the new indices starting from > zero, Maybe, but note that you need to enforce boundary conditions and different processes will have a different range so preserving the index bounds might cause more confusion than it's worth. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From bsmith at mcs.anl.gov Sun Feb 5 12:13:26 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 5 Feb 2017 12:13:26 -0600 Subject: [petsc-users] DMCreateGlobalVector in fortran. In-Reply-To: References: Message-ID: <17943611-113B-48CF-8C88-F818494225A4@mcs.anl.gov> Are your -1 index locations used for "boundary conditions", that is extra variables that are used to update the other variables but don't get updated themselves? If so you should look at using a DMBoundaryType of DM_BOUNDARY_GHOSTED, this will result in the arrays returned from DMDAVecGetArrayF90() starting with -1. But I urge you to start with a simple example in order to understand how DMDA and its vectors work. I think you may misunderstand it. For example: > call DMCreateGlobalVector(da,globalv,ierr) > > call DMCreateLocalVector(da,localv,ierr) > > > > call DMGlobalToLocalBegin(da,globalv,INSERT_VALUES,localv,ierr) > > call DMGlobalToLocalEnd(da,globalv,INSERT_VALUES,localv,ierr) > > You create a global and local array which have all zero entries (PETSc vectors by default have all zeros) and then you scatter the zero entries from the global into the local. This scatter serves no purpose, unless you are just checking that it does not crash. Barry > On Feb 4, 2017, at 10:19 PM, Manuel Valera wrote: > > Ok, let me try to clarify a little bit, > > I have a u0 3d array with the mentioned indices (starting from -1), im creating a dmda with the sizes in the constructor, no problem with that. > > Next, im trying to distribute this u0 array into global and local vectors, apparently i can do that since no error comes out. > > Now, when i get the array back with VecGetArrayF90(), the indices start now from 0, is that right? so the whole array have been shifted in its indices. > > This changes things for me because my idea was to use the previous serial code to operate on the arrays, but this is written with indices starting from -1. There are dozens of arrays with the same structure, but they also operate with other arrays with indices starting from 1, at the same time, usually in triple nested loops which code the stencil used within them, to update and correct the velocities after solving for pressure, per example. > > So it would be much easier to shift the indices of these arrays by one, rather than changing all the code to match the new indices starting from zero, > > Is there a way to do this? > > Thanks, > > > > > On Sat, Feb 4, 2017 at 5:46 PM, Barry Smith wrote: > > For DMDAVecGetArrayF90 you need to declare the "array" arguments as Fortran pointers you don't declare them like > > > u0(-1:IMax+2,-1:JMax+1,-1:KMax+1) > > > > > On Feb 4, 2017, at 7:34 PM, Matthew Knepley wrote: > > > > On Sat, Feb 4, 2017 at 7:30 PM, Manuel Valera wrote: > > Thanks, that helped a lot, > > > > Now im a little confused with my next error, in my serial code i have several arrays with dimensions declared as: > > > > u0(-1:IMax+2,-1:JMax+1,-1:KMax+1) > > > > and i create my DMDA for this array as: > > > > call CDA3(daub,IMax+4,JMax+3,KMax+3) > > call DAs(daub,u0p,u0pl,u0) > > > > With the subroutines as: > > > > SUBROUTINE CDA3(da3d,dim3dx,dim3dy,dim3dz) > > > > > > > > DM, intent(inout) :: da3d > > PetscInt, intent(in) :: dim3dx,dim3dy,dim3dz > > DMBoundaryType :: bx,by,bz > > > > > > bx = DM_BOUNDARY_NONE > > by = DM_BOUNDARY_NONE > > bz = DM_BOUNDARY_NONE > > > > call DMDACreate3d(PETSC_COMM_WORLD,bx,by,bz,DMDA_STENCIL_BOX,dim3dx,dim3dy,dim3dz,PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da3d,ierr) > > > > call DMSetFromOptions(da3d,ierr) > > call DMSetUp(da3d,ierr) > > > > > > END SUBROUTINE CDA3 > > > > SUBROUTINE DAs(da,globalv,localv,array) > > > > ! use PetscObjects, only :: ierr > > > > > > ! Umbrella program to update and communicate the arrays in a > > ! distributed fashion using the DMDA objects from PETSc. > > ! Manuel Valera 1/20/17 > > > > ! Arguments: > > ! da = DMDA array either 1d,2d or 3d, already created and setup > > ! globalv = global vector to be operated into > > ! localv = local chunk each processor works in. > > ! array = the array to be petscified. ONLY 3D ARRAYS rn. > > ! just in case. > > > > Vec,intent(inout) :: globalv,localv > > > > > > PetscScalar,dimension(:,:,:) :: array > > DM,intent(inout) :: da > > > > > > > > > > > > > > call DMCreateGlobalVector(da,globalv,ierr) > > call DMCreateLocalVector(da,localv,ierr) > > > > call DMGlobalToLocalBegin(da,globalv,INSERT_VALUES,localv,ierr) > > call DMGlobalToLocalEnd(da,globalv,INSERT_VALUES,localv,ierr) > > > > > > > > END SUBROUTINE DAs > > > > Which i later open into an array to operate into it, but when i try to access u0(:,j-2,:) it gives me the following error: > > > > Fortran runtime error: Index '-1' of dimension 2 of array 'u0' below lower bound of 0 > > > > So i'm guessing the DMDA is creating the arrays with dimensions starting at 0 instead of -1 as it should, right? what can be done in this case? > > > > 1) The DMDA only takes sizes, not index bounds, in the constructor. > > > > 2) Its not the DMDA you must be talking about, but the array you get from DMDAVecGetArrayF90() or VecGetArrayF90() > > > > 3) The VecGetArrayF90() always start from 0 since it is indexed in local indices > > > > 4) The DMDVecGetArrayF90() arrays are indexed by global indices for the patch your process owns (including ghosts if its a local vector) > > > > Thanks, > > > > Matt > > > > Thanks, > > Manuel > > > > On Sat, Feb 4, 2017 at 4:18 PM, Barry Smith wrote: > > > > See: http://www.mcs.anl.gov/petsc/documentation/changes/dev.html search for DMDACreate and update your code. > > > > Barry > > > > > > > On Feb 4, 2017, at 6:02 PM, Manuel Valera wrote: > > > > > > Hello devs, > > > > > > Ive been creating the framework to distribute the arrays from my model using the global/local vectors and the DMDAs, for this i created a small subroutine that is responsible for making the necessary calls: > > > > > > SUBROUTINE DAs(da,globalv,localv) > > > > > > ! use PetscObjects, only :: ierr > > > > > > > > > ! Umbrella program to update and communicate the arrays in a > > > ! distributed fashion using the DMDA objects from PETSc. > > > ! Manuel Valera 1/20/17 > > > > > > ! Arguments: > > > ! da = DMDA array either 1d,2d or 3d, already created and setup > > > ! globalv = global vector to be operated into > > > ! localv = local chunk each processor works in. > > > > > > > > > > > > Vec,intent(inout) :: globalv,localv > > > > > > > > > > > > DM,intent(inout) :: da > > > > > > call DMCreateGlobalVector(da,globalv,ierr) > > > call DMCreateLocalVector(da,localv,ierr) > > > > > > call DMGlobalToLocalBegin(da,globalv,INSERT_VALUES,localv,ierr) > > > call DMGlobalToLocalEnd(da,globalv,INSERT_VALUES,localv,ierr) > > > > > > > > > > > > END SUBROUTINE DAs > > > > > > > > > I call this from another program where i have declared a vector as: > > > > > > #include "petsc/finclude/petscdm.h" > > > #include "petsc/finclude/petscdmda.h" > > > > > > use petsc > > > use petscdm > > > use petscdmda > > > > > > IMPLICIT NONE > > > > > > Vec :: xpdmda,yp,zp,xcp,ycp,zcp > > > DM :: da,dac,dau,dav,daw > > > > > > CONTAINS > > > SUBROUTINE InitGridP > > > > > > IMPLICIT NONE > > > !PetscInt :: ierr > > > > > > ! CREATE DMDA: > > > > > > call CDA3(da,IMax-1,JMax-1,KMax-1) > > > > > > !xpdmda,yp,zp have the same dimensions than x,y,z > > > !xpl,ypl,zpl are the local versions of xp,yp,zp > > > > > > !We create the global and local vectors and communicate their values: > > > call DAs(da,xpdmda,xpl,x) > > > > > > And at this call it crashes with the following message: > > > > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > > [0]PETSC ERROR: Null argument, when expecting valid pointer > > > [0]PETSC ERROR: Null Object: Parameter # 2 > > > [0]PETSC ERROR: See > > > http://www.mcs.anl.gov/petsc/documentation/faq.html > > > for trouble shooting. > > > [0]PETSC ERROR: Petsc Development GIT revision: v3.7.5-2949-gc3c607c GIT Date: 2017-01-20 17:34:16 -0600 > > > [0]PETSC ERROR: ./ucmsLEP on a arch-linux2-c-debug named ocean by valera Sat Feb 4 15:53:53 2017 > > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-hdf5 --download-netcdf --download-hypre --download-metis --download-parmetis --download-trillinos --with-debugging=1 > > > [0]PETSC ERROR: #3 VecSetLocalToGlobalMapping() line 79 in /usr/dataC/home/valera/petsc/src/vec/vec/interface/vector.c > > > [0]PETSC ERROR: #4 DMCreateGlobalVector_DA() line 41 in /usr/dataC/home/valera/petsc/src/dm/impls/da/dadist.c > > > [0]PETSC ERROR: #5 DMCreateGlobalVector() line 840 in /usr/dataC/home/valera/petsc/src/dm/interface/dm.c > > > > > > > > > > > > So my question is, do i have to setup the vectors in any additional way? i followed the fortran DMDA examples closely and i didnt notice anything extra, > > > > > > Thanks, > > > > > > Manuel > > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > From messages-noreply at linkedin.com Sun Feb 5 23:31:57 2017 From: messages-noreply at linkedin.com (Soundes MARZOUGUI) Date: Mon, 6 Feb 2017 05:31:57 +0000 (UTC) Subject: [petsc-users] =?utf-8?q?Invitation_=C3=A0_se_connecter_sur_Linked?= =?utf-8?q?In?= Message-ID: <1459765179.1838760.1486359117445.JavaMail.app@lor1-app3015.prod.linkedin.com> Bonjour, J?aimerais vous inviter ? rejoindre mon r?seau professionnel sur LinkedIn. Soundes Veuillez confirmer que vous connaissez Soundes: https://www.linkedin.com/comm/start/accept-invitation?sharedKey=O7NzTLKl&invitationId=6234241840196001815&trk=eml-guest-invite-cta&trkEmail=eml-invite_guest-null-2-null-null-35olqj%7Eiytnu92j%7En9 Vous avez re?u une invitation ? rejoindre un r?seau. LinkedIn utilise votre adresse e-mail pour faire des suggestions ? ses utilisateurs dans des fonctionnalit?s telles que Les connaissez-vous?? Se d?sabonner ici?: https://www.linkedin.com/e/v2?e=35olqj-iytnu92j-n9&a=nsettings-loid-email-unsubscribe-router&midToken=AQFmrFVhGEd36Q&ek=invite_guest&loid=AQF5AGH86mcbXwAAAVoR6D5IP5ENJCGQZk_-LwJU5bxYAETwFHda3VGweR3tdOJTewQZ0MVCePQdXlG_--bapdEJ_5-Dp2e9VdoUxK43zQ&eid=35olqj-iytnu92j-n9 Cet e-mail est destin? ? petsc-users at mcs.anl.gov. If you need assistance or have questions, please contact LinkedIn Customer Service: https://www.linkedin.com/e/v2?e=35olqj-iytnu92j-n9&a=customerServiceUrl&ek=invite_guest ? 2017 LinkedIn Corporation, 1?000 West Maude Avenue, Sunnyvale, CA 94085, USA. LinkedIn et le logo de LinkedIn sont des marques d?pos?es de LinkedIn. -------------- next part -------------- An HTML attachment was scrubbed... URL: From michel.kern at inria.fr Tue Feb 7 02:49:34 2017 From: michel.kern at inria.fr (Michel Kern) Date: Tue, 7 Feb 2017 09:49:34 +0100 (CET) Subject: [petsc-users] Replacement for Euclid ? In-Reply-To: <991142340.22990867.1486457174180.JavaMail.zimbra@inria.fr> Message-ID: <472771747.22991921.1486457374460.JavaMail.zimbra@inria.fr> I have a code that uses euclid for ILU preconditioning (as part of a CPR-AMG preconditioner for a mulitphase flow solver). As noted in the release notes, euclid is no longer supported starting from 3.6. What is the recommended replacement ? Is block Jacobi plus a local solver an alternative ? Thanks Michel -- Michel Kern Centre de Recherche Inria de Paris, Maison de la Simulation 2 rue Simone Iff, bureau A413, Digiteo Labs, b?t 565, CEA Saclay 75012 Paris 91191 Gif-sur-Yvette cedex tel. : +33 (0)1 80 49 42 36, tel. : +33 (0)1 69 08 58 32 -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas.drinkt.thee at gmail.com Tue Feb 7 05:12:08 2017 From: lukas.drinkt.thee at gmail.com (Lukas van de Wiel) Date: Tue, 7 Feb 2017 12:12:08 +0100 Subject: [petsc-users] missing types in petscsysdef.h Message-ID: Dear fellow PETSc users, I have been using PETSc for several years now, but I notice that after every upgrade I have to make changes to include/finclude/petscsysdef.h to get my Fortran code to work with it (or, in 3.7.5 include/petsc/finclude/petscsysdef.h) I have been running 3.4.2 for a few years, and PetscReal and PetscScalar were never defined, apparently because no PETSC_USE_REAL_[TYPE] was never defined. In 3.7.5 I have the same problem, and it also affects PetscFortranComplex which also depends on PETSC_USE_REAL_[TYPE]. However, my configure.log nicely shows: #ifndef PETSC_USE_REAL_DOUBLE #define PETSC_USE_REAL_DOUBLE 1 #endif It seems that changing petscsysdef.h is not the proper solution. Is there a more elegant way for me to fix this? Thanks for your expertise, Lukas From knepley at gmail.com Tue Feb 7 05:17:22 2017 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 7 Feb 2017 05:17:22 -0600 Subject: [petsc-users] Replacement for Euclid ? In-Reply-To: <472771747.22991921.1486457374460.JavaMail.zimbra@inria.fr> References: <991142340.22990867.1486457174180.JavaMail.zimbra@inria.fr> <472771747.22991921.1486457374460.JavaMail.zimbra@inria.fr> Message-ID: On Tue, Feb 7, 2017 at 2:49 AM, Michel Kern wrote: > I have a code that uses euclid for ILU preconditioning (as part of a > CPR-AMG preconditioner for a mulitphase flow solver). As noted in the > release notes, euclid is no longer supported starting from 3.6. What is the > recommended replacement ? Is block Jacobi plus a local solver an > alternative ? > There is pilut from Hypre. How big is the problem? Did you try sparse direct on it (SuperLU, MUMPS)? Thanks, Matt > Thanks > Michel > > -- > Michel Kern > Centre de Recherche Inria de Paris, Maison de la Simulation > 2 rue Simone Iff, bureau A413, Digiteo Labs, b?t 565, CEA > Saclay > 75012 Paris 91191 Gif-sur-Yvette > cedex > tel. : +33 (0)1 80 49 42 36 <+33%201%2080%2049%2042%2036>, > tel. : +33 (0)1 69 08 58 32 <+33%201%2069%2008%2058%2032> > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Feb 7 05:20:49 2017 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 7 Feb 2017 05:20:49 -0600 Subject: [petsc-users] missing types in petscsysdef.h In-Reply-To: References: Message-ID: On Tue, Feb 7, 2017 at 5:12 AM, Lukas van de Wiel < lukas.drinkt.thee at gmail.com> wrote: > Dear fellow PETSc users, > > I have been using PETSc for several years now, but I notice that after > every upgrade I have to make changes to include/finclude/petscsysdef.h > to get my Fortran code to work with it (or, in 3.7.5 > include/petsc/finclude/petscsysdef.h) > We will definitely fix this. However, the entire Fortran interface has been rewritten to make it simpler, and allow using all the features os F90. Please see the bottom of http://www.mcs.anl.gov/petsc/documentation/changes/dev.html Would you be able to try out the master branch of PETSc development (instructions here http://www.mcs.anl.gov/petsc/developers/index.html)? This would let us know if the bug is still there. Thanks, Matt > I have been running 3.4.2 for a few years, and PetscReal and > PetscScalar were never defined, apparently because no > PETSC_USE_REAL_[TYPE] was never defined. > > In 3.7.5 I have the same problem, and it also affects > PetscFortranComplex which also depends on PETSC_USE_REAL_[TYPE]. > > However, my configure.log nicely shows: > > #ifndef PETSC_USE_REAL_DOUBLE > #define PETSC_USE_REAL_DOUBLE 1 > #endif > > It seems that changing petscsysdef.h is not the proper solution. Is > there a more elegant way for me to fix this? > > Thanks for your expertise, > Lukas > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas.drinkt.thee at gmail.com Tue Feb 7 06:09:10 2017 From: lukas.drinkt.thee at gmail.com (Lukas van de Wiel) Date: Tue, 7 Feb 2017 13:09:10 +0100 Subject: [petsc-users] missing types in petscsysdef.h In-Reply-To: References: Message-ID: Hi Matt, thank you for your fast reply. I have tried this with the dev version. The structure of the include files is a lot more transparent, now! Thanks! However, PetscReal is still not defined. I can use my old method to change the third to last line from petscsys.h block #if defined (PETSC_USE_REAL_SINGLE) #define PetscReal PetscFortranFloat #elif defined(PETSC_USE_REAL___FLOAT128) #define PetscReal PetscFortranLongDouble #elif defined(PETSC_USE_REAL_DOUBLE) #define PetscReal PetscFortranDouble #endif to #if defined (PETSC_USE_REAL_SINGLE) #define PetscReal PetscFortranFloat #elif defined(PETSC_USE_REAL___FLOAT128) #define PetscReal PetscFortranLongDouble #else #define PetscReal PetscFortranDouble #endif To make sure it PetscReal is defined as double, even wthen there is no PETSC_USE_REAL_[TYPE] defined, and similar for PetscScalar, but that is ugly. :-\ Cheers Lukas On 2/7/17, Matthew Knepley wrote: > On Tue, Feb 7, 2017 at 5:12 AM, Lukas van de Wiel < > lukas.drinkt.thee at gmail.com> wrote: > >> Dear fellow PETSc users, >> >> I have been using PETSc for several years now, but I notice that after >> every upgrade I have to make changes to include/finclude/petscsysdef.h >> to get my Fortran code to work with it (or, in 3.7.5 >> include/petsc/finclude/petscsysdef.h) >> > > We will definitely fix this. However, the entire Fortran interface has been > rewritten to make it simpler, > and allow using all the features os F90. Please see the bottom of > http://www.mcs.anl.gov/petsc/documentation/changes/dev.html > > Would you be able to try out the master branch of PETSc development > (instructions here http://www.mcs.anl.gov/petsc/developers/index.html)? > This would let us know if the bug is still there. > > Thanks, > > Matt > > >> I have been running 3.4.2 for a few years, and PetscReal and >> PetscScalar were never defined, apparently because no >> PETSC_USE_REAL_[TYPE] was never defined. >> >> In 3.7.5 I have the same problem, and it also affects >> PetscFortranComplex which also depends on PETSC_USE_REAL_[TYPE]. >> >> However, my configure.log nicely shows: >> >> #ifndef PETSC_USE_REAL_DOUBLE >> #define PETSC_USE_REAL_DOUBLE 1 >> #endif >> >> It seems that changing petscsysdef.h is not the proper solution. Is >> there a more elegant way for me to fix this? >> >> Thanks for your expertise, >> Lukas >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > From knepley at gmail.com Tue Feb 7 06:59:28 2017 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 7 Feb 2017 06:59:28 -0600 Subject: [petsc-users] missing types in petscsysdef.h In-Reply-To: References: Message-ID: On Tue, Feb 7, 2017 at 6:09 AM, Lukas van de Wiel < lukas.drinkt.thee at gmail.com> wrote: > Hi Matt, > > thank you for your fast reply. > > I have tried this with the dev version. > The structure of the include files is a lot more transparent, now! Thanks! > > However, PetscReal is still not defined. > I think you need use petscsys Thanks, Matt > I can use my old method to change the third to last line from petscsys.h > block > > #if defined (PETSC_USE_REAL_SINGLE) > #define PetscReal PetscFortranFloat > #elif defined(PETSC_USE_REAL___FLOAT128) > #define PetscReal PetscFortranLongDouble > #elif defined(PETSC_USE_REAL_DOUBLE) > #define PetscReal PetscFortranDouble > #endif > > to > > #if defined (PETSC_USE_REAL_SINGLE) > #define PetscReal PetscFortranFloat > #elif defined(PETSC_USE_REAL___FLOAT128) > #define PetscReal PetscFortranLongDouble > #else > #define PetscReal PetscFortranDouble > #endif > > To make sure it PetscReal is defined as double, even wthen there is no > PETSC_USE_REAL_[TYPE] defined, and similar for PetscScalar, but that > is ugly. :-\ > > Cheers > Lukas > > On 2/7/17, Matthew Knepley wrote: > > On Tue, Feb 7, 2017 at 5:12 AM, Lukas van de Wiel < > > lukas.drinkt.thee at gmail.com> wrote: > > > >> Dear fellow PETSc users, > >> > >> I have been using PETSc for several years now, but I notice that after > >> every upgrade I have to make changes to include/finclude/petscsysdef.h > >> to get my Fortran code to work with it (or, in 3.7.5 > >> include/petsc/finclude/petscsysdef.h) > >> > > > > We will definitely fix this. However, the entire Fortran interface has > been > > rewritten to make it simpler, > > and allow using all the features os F90. Please see the bottom of > > http://www.mcs.anl.gov/petsc/documentation/changes/dev.html > > > > Would you be able to try out the master branch of PETSc development > > (instructions here http://www.mcs.anl.gov/petsc/developers/index.html)? > > This would let us know if the bug is still there. > > > > Thanks, > > > > Matt > > > > > >> I have been running 3.4.2 for a few years, and PetscReal and > >> PetscScalar were never defined, apparently because no > >> PETSC_USE_REAL_[TYPE] was never defined. > >> > >> In 3.7.5 I have the same problem, and it also affects > >> PetscFortranComplex which also depends on PETSC_USE_REAL_[TYPE]. > >> > >> However, my configure.log nicely shows: > >> > >> #ifndef PETSC_USE_REAL_DOUBLE > >> #define PETSC_USE_REAL_DOUBLE 1 > >> #endif > >> > >> It seems that changing petscsysdef.h is not the proper solution. Is > >> there a more elegant way for me to fix this? > >> > >> Thanks for your expertise, > >> Lukas > >> > > > > > > > > -- > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which > their > > experiments lead. > > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From michel.kern at inria.fr Tue Feb 7 07:53:26 2017 From: michel.kern at inria.fr (Michel Kern) Date: Tue, 7 Feb 2017 14:53:26 +0100 (CET) Subject: [petsc-users] Replacement for Euclid ? In-Reply-To: References: <991142340.22990867.1486457174180.JavaMail.zimbra@inria.fr> <472771747.22991921.1486457374460.JavaMail.zimbra@inria.fr> Message-ID: <1406626865.23285042.1486475606683.JavaMail.zimbra@inria.fr> Thanks Mat, according tot he hypre documentation, pilut is "no longer supported by the hypre team", as less efficient than Euclid, so I wouldn't consider an option. I wsa looking for a "natural" upgrade path. The problems we've looked at for the moment can be up to a few millions cells (that's the size of the pressure block, the saturation block is larger, but should be easier to solve). We're aiming for 10^6 cells, and 10^3 cores. Yes, direct methods deserve a second look ! Michel ----- Mail original ----- > On Tue, Feb 7, 2017 at 2:49 AM, Michel Kern < michel.kern at inria.fr > wrote: > > I have a code that uses euclid for ILU preconditioning (as part of a > > CPR-AMG > > preconditioner for a mulitphase flow solver). As noted in the release > > notes, > > euclid is no longer supported starting from 3.6. What is the recommended > > replacement ? Is block Jacobi plus a local solver an alternative ? > > There is pilut from Hypre. > How big is the problem? Did you try sparse direct on it (SuperLU, MUMPS)? > Thanks, > Matt > > Thanks > > > Michel > > > -- > > > Michel Kern > > > Centre de Recherche Inria de Paris, Maison de la Simulation > > > 2 rue Simone Iff, bureau A413, Digiteo Labs, b?t 565, CEA Saclay > > > 75012 Paris 91191 Gif-sur-Yvette cedex > > > tel. : +33 (0)1 80 49 42 36 , tel. : +33 (0)1 69 08 58 32 > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener -- Michel Kern Centre de recherche Inria de Paris, Maison de la Simulation, 2 rue Simone Iff, bureau A413, Digiteo Labs, b?t. 565, CEA Saclay 75012 Paris 91191 Gif-sur-Yvette Cedex tel. : +33 (0)1 80 49 42 36, tel. : +33 (0)1 69 08 58 32 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Feb 7 08:00:29 2017 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 7 Feb 2017 08:00:29 -0600 Subject: [petsc-users] Replacement for Euclid ? In-Reply-To: <1406626865.23285042.1486475606683.JavaMail.zimbra@inria.fr> References: <991142340.22990867.1486457174180.JavaMail.zimbra@inria.fr> <472771747.22991921.1486457374460.JavaMail.zimbra@inria.fr> <1406626865.23285042.1486475606683.JavaMail.zimbra@inria.fr> Message-ID: On Tue, Feb 7, 2017 at 7:53 AM, Michel Kern wrote: > Thanks Mat, > > according tot he hypre documentation, pilut is "no longer supported by the > hypre team", as less efficient than Euclid, so I wouldn't consider an > option. I wsa looking for a "natural" upgrade path. > I don't know of another ILU. I have never understood the use of ILU. It is a fragile, not very scalable algorithm which is soundly beaten in every arena where we understand the problem. I also do not understand why it is used as part of CPR. I can't believe that is the best way to solve the problem. But maybe I do not understand. Thanks, Matt > The problems we've looked at for the moment can be up to a few millions > cells (that's the size of the pressure block, the saturation block is > larger, but should be easier to solve). We're aiming for 10^6 cells, and > 10^3 cores. > Yes, direct methods deserve a second look ! > > Michel > > ------------------------------ > > On Tue, Feb 7, 2017 at 2:49 AM, Michel Kern wrote: > >> I have a code that uses euclid for ILU preconditioning (as part of a >> CPR-AMG preconditioner for a mulitphase flow solver). As noted in the >> release notes, euclid is no longer supported starting from 3.6. What is the >> recommended replacement ? Is block Jacobi plus a local solver an >> alternative ? >> > > There is pilut from Hypre. > > How big is the problem? Did you try sparse direct on it (SuperLU, MUMPS)? > > Thanks, > > Matt > > >> Thanks >> Michel >> >> -- >> Michel Kern >> Centre de Recherche Inria de Paris, Maison de la Simulation >> 2 rue Simone Iff, bureau A413, Digiteo Labs, b?t 565, CEA >> Saclay >> 75012 Paris 91191 Gif-sur-Yvette >> cedex >> tel. : +33 (0)1 80 49 42 36 <+33%201%2080%2049%2042%2036>, >> tel. : +33 (0)1 69 08 58 32 <+33%201%2069%2008%2058%2032> >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > > -- > Michel Kern > Centre de recherche Inria de Paris, Maison de la Simulation, > 2 rue Simone Iff, bureau A413, Digiteo Labs, b?t. 565, CEA Saclay > 75012 Paris 91191 Gif-sur-Yvette Cedex > tel. : +33 (0)1 80 49 42 36 <+33%201%2080%2049%2042%2036>, tel. : +33 > (0)1 69 08 58 32 <+33%201%2069%2008%2058%2032> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mailinglists at xgm.de Tue Feb 7 08:06:54 2017 From: mailinglists at xgm.de (Florian Lindner) Date: Tue, 7 Feb 2017 15:06:54 +0100 Subject: [petsc-users] Where to restrict MPI Communicator Message-ID: <04b40fa3-afda-82e0-f71f-8157f7c59530@xgm.de> Hello, so far we have happily set PETSC_COMM_WORLD anywhere in our code and we were not having trouble with that. However http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PETSC_COMM_WORLD.html states that: "Notes: By default PETSC_COMM_WORLD and MPI_COMM_WORLD are identical unless you wish to run PETSc on ONLY a subset of MPI_COMM_WORLD. In that case create your new (smaller) communicator, call it, say comm, and set PETSC_COMM_WORLD = comm BEFORE calling PetscInitialize(), but after MPI_Init() has been called." So, you should never modify PETSC_COMM_WORLD after PetscInitialize has been called. Is that warning still in effect? What other ways are to set the MPI comm used by PETSc. There is an comm argument on most (all?) functions like VecCreate, PetscViewer*Open, PetscRandomCreate, MatCreate). Is it sufficient to use that argument to supply an alternate communicator? Thanks, Florian From knepley at gmail.com Tue Feb 7 08:14:27 2017 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 7 Feb 2017 08:14:27 -0600 Subject: [petsc-users] Where to restrict MPI Communicator In-Reply-To: <04b40fa3-afda-82e0-f71f-8157f7c59530@xgm.de> References: <04b40fa3-afda-82e0-f71f-8157f7c59530@xgm.de> Message-ID: On Tue, Feb 7, 2017 at 8:06 AM, Florian Lindner wrote: > Hello, > > so far we have happily set PETSC_COMM_WORLD anywhere in our code and we > were not having trouble with that. > > However http://www.mcs.anl.gov/petsc/petsc-current/docs/ > manualpages/Sys/PETSC_COMM_WORLD.html states that: > > "Notes: By default PETSC_COMM_WORLD and MPI_COMM_WORLD are identical > unless you wish to run PETSc on ONLY a subset of > MPI_COMM_WORLD. In that case create your new (smaller) communicator, call > it, say comm, and set PETSC_COMM_WORLD = comm > BEFORE calling PetscInitialize(), but after MPI_Init() has been called." > > So, you should never modify PETSC_COMM_WORLD after PetscInitialize has > been called. Is that warning still in effect? > Yes. > What other ways are to set the MPI comm used by PETSc. There is an comm > argument on most (all?) functions like > VecCreate, PetscViewer*Open, PetscRandomCreate, MatCreate). Is it > sufficient to use that argument to supply an alternate > communicator? > Yes. Thanks, Matt > Thanks, > Florian > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Feb 7 09:01:53 2017 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 7 Feb 2017 09:01:53 -0600 Subject: [petsc-users] missing types in petscsysdef.h In-Reply-To: References: Message-ID: PETSC_USE_REAL_DOUBLE should be defined in petscconf.h You did mention earlier that you found it in configure.log. Please verify if this exists in petscconf.h for your install. And how are you compiling your code? Not using petsc makefiles? Perhaps your makefiles are picking up the wrong petscconf.h? Please check the -I flags that you are passing to the compiler. And verify by checking the contents of preprocessed sourcefile. >>>>>>>>>> balay at asterix /home/balay/petsc/src/ksp/ksp/examples/tutorials (next=) $ make ex2f.o FFLAGS=-E mpif90 -c -fPIC -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g -E -I/home/balay/petsc/include -I/home/balay/petsc/arch-linux2-c-debug/include -o ex2f.o ex2f.F balay at asterix /home/balay/petsc/src/ksp/ksp/examples/tutorials (next=) $ grep petscconf.h ex2f.o # 1 "/home/balay/petsc/arch-linux2-c-debug/include/petscconf.h" 1 balay at asterix /home/balay/petsc/src/ksp/ksp/examples/tutorials (next=) $ <<<<<<<<<< For my build - its using the correct petscconf.h You can use the following commands with a petsc makefile - to get buildinfo that you should use in your makefile balay at asterix /home/balay/petsc (next=) $ make getincludedirs -I/home/balay/petsc/include -I/home/balay/petsc/arch-linux2-c-debug/include balay at asterix /home/balay/petsc (next=) $ make getlinklibs -Wl,-rpath,/home/balay/petsc/arch-linux2-c-debug/lib -Wl,-rpath,/home/balay/petsc/arch-linux2-c-debug/lib -L/home/balay/petsc/arch-linux2-c-debug/lib -lpetsc -llapack -lblas -lX11 -lpthread -lm -Wl,-rpath,/home/balay/soft/mpich-3.1.4/lib -L/home/balay/soft/mpich-3.1.4/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/6.3.1 -L/usr/lib/gcc/x86_64-redhat-linux/6.3.1 -lmpifort -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpicxx -lstdc++ -Wl,-rpath,/home/balay/soft/mpich-3.1.4/lib -L/home/balay/soft/mpich-3.1.4/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/6.3.1 -L/usr/lib/gcc/x86_64-redhat-linux/6.3.1 -ldl -Wl,-rpath,/home/balay/soft/mpich-3.1.4/lib -lmpi -lgcc_s -ldl Satish On Tue, 7 Feb 2017, Lukas van de Wiel wrote: > Hi Matt, > > thank you for your fast reply. > > I have tried this with the dev version. > The structure of the include files is a lot more transparent, now! Thanks! > > However, PetscReal is still not defined. > > > I can use my old method to change the third to last line from petscsys.h block > > #if defined (PETSC_USE_REAL_SINGLE) > #define PetscReal PetscFortranFloat > #elif defined(PETSC_USE_REAL___FLOAT128) > #define PetscReal PetscFortranLongDouble > #elif defined(PETSC_USE_REAL_DOUBLE) > #define PetscReal PetscFortranDouble > #endif > > to > > #if defined (PETSC_USE_REAL_SINGLE) > #define PetscReal PetscFortranFloat > #elif defined(PETSC_USE_REAL___FLOAT128) > #define PetscReal PetscFortranLongDouble > #else > #define PetscReal PetscFortranDouble > #endif > > To make sure it PetscReal is defined as double, even wthen there is no > PETSC_USE_REAL_[TYPE] defined, and similar for PetscScalar, but that > is ugly. :-\ > > Cheers > Lukas > > On 2/7/17, Matthew Knepley wrote: > > On Tue, Feb 7, 2017 at 5:12 AM, Lukas van de Wiel < > > lukas.drinkt.thee at gmail.com> wrote: > > > >> Dear fellow PETSc users, > >> > >> I have been using PETSc for several years now, but I notice that after > >> every upgrade I have to make changes to include/finclude/petscsysdef.h > >> to get my Fortran code to work with it (or, in 3.7.5 > >> include/petsc/finclude/petscsysdef.h) > >> > > > > We will definitely fix this. However, the entire Fortran interface has been > > rewritten to make it simpler, > > and allow using all the features os F90. Please see the bottom of > > http://www.mcs.anl.gov/petsc/documentation/changes/dev.html > > > > Would you be able to try out the master branch of PETSc development > > (instructions here http://www.mcs.anl.gov/petsc/developers/index.html)? > > This would let us know if the bug is still there. > > > > Thanks, > > > > Matt > > > > > >> I have been running 3.4.2 for a few years, and PetscReal and > >> PetscScalar were never defined, apparently because no > >> PETSC_USE_REAL_[TYPE] was never defined. > >> > >> In 3.7.5 I have the same problem, and it also affects > >> PetscFortranComplex which also depends on PETSC_USE_REAL_[TYPE]. > >> > >> However, my configure.log nicely shows: > >> > >> #ifndef PETSC_USE_REAL_DOUBLE > >> #define PETSC_USE_REAL_DOUBLE 1 > >> #endif > >> > >> It seems that changing petscsysdef.h is not the proper solution. Is > >> there a more elegant way for me to fix this? > >> > >> Thanks for your expertise, > >> Lukas > >> > > > > > > > > -- > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which their > > experiments lead. > > -- Norbert Wiener > > > From bsmith at mcs.anl.gov Tue Feb 7 13:02:14 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 7 Feb 2017 13:02:14 -0600 Subject: [petsc-users] Replacement for Euclid ? In-Reply-To: <472771747.22991921.1486457374460.JavaMail.zimbra@inria.fr> References: <472771747.22991921.1486457374460.JavaMail.zimbra@inria.fr> Message-ID: > On Feb 7, 2017, at 2:49 AM, Michel Kern wrote: > > I have a code that uses euclid for ILU preconditioning (as part of a CPR-AMG preconditioner for a mulitphase flow solver). As noted in the release notes, euclid is no longer supported starting from 3.6. What is the recommended replacement ? Is block Jacobi plus a local solver an alternative ? I would use block Jacobi with Gauss-Seidel as the local solver first. Does that not work due to zero pivots? Then you can use PCFIELDSPLIT with the option -pc_fieldsplit_detect_saddle_point or you can use block Jacobi with ILU(0) as the local solver. There are many combinations to try. Barry > > Thanks > Michel > > -- > Michel Kern > Centre de Recherche Inria de Paris, Maison de la Simulation > 2 rue Simone Iff, bureau A413, Digiteo Labs, b?t 565, CEA Saclay > 75012 Paris 91191 Gif-sur-Yvette cedex > tel. : +33 (0)1 80 49 42 36, tel. : +33 (0)1 69 08 58 32 > > From abdullahasivas at gmail.com Tue Feb 7 14:02:37 2017 From: abdullahasivas at gmail.com (Abdullah Ali Sivas) Date: Tue, 7 Feb 2017 15:02:37 -0500 Subject: [petsc-users] Replacement for Euclid ? Message-ID: Hi Michel, I am not entirely sure if there is a PETSc version, but there is a preconditioner called DS factorization (details in High-Performance Scientific Computing: Algorithms and Applications). It shares few properties of Euclid (like being quite scalable). If you have to use a factorization based preconditioner, you can try that. I hope it helps, Best wishes, Abdullah Ali Sivas From davydden at gmail.com Tue Feb 7 15:51:46 2017 From: davydden at gmail.com (Denis Davydov) Date: Tue, 7 Feb 2017 22:51:46 +0100 Subject: [petsc-users] pc gamg did not converge in sinv where it used to Message-ID: <30A0F820-09ED-46E8-A4CD-3A8100D94C04@gmail.com> Dear all, I rerun calculations (unit tests) which used to work with slightly older versions of PETSc/SLEPc (a year ago, or so) and see the "KSPSolve has not converged? error for shift and invert transformation with gamg preconditioner (below). Shifted matrix is SPD, could have bad condition number but it worked before. Judging from a pre-last line in the log below, something is wrong with GAMG or its interaction with SLEPc. Regards, Denis. [0]PETSC ERROR: Configure options --prefix=/Users/davydden/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/petsc-3.7.5-e35uxj5pf7ndtedbnzvq7dot5oie3psc --with-ssl=0 --with-x=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 --with-mpi=1 --with-mpi-dir=/Users/davydden/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/openmpi-2.0.2-oos4cxprn2bislzhc3rbc3lo4dadginw --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-64-bit-indices=0 --with-blas-lapack-lib=/Users/davydden/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/openblas-0.2.19-xg7etkjyo7xjnvuojsdc2xoixesxoerh/lib/libopenblas.dylib --with-metis=1 --with-metis-dir=/Users/davydden/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/metis-5.1.0-c2muo3b6k5aea2743g34s35u2ve23yxq --with-boost=0 --with-hdf5=1 --with-hdf5-dir=/Users/davydden/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/hdf5-1.10.0-patch1-kwkdkp5mo4olqpkitojab3xg3xhl6gfz --with-hypre=1 --with-hypre-dir=/Users/davydden/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/hypre-2.11.1-dprfz6t4t24wesj74dcfdtajwm5n5qzu --with-parmetis=1 --with-parmetis-dir=/Users/davydden/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/parmetis-4.0.3-tm5dd6fqnbw6hiepajhkvmqk42xrrgos --with-mumps=1 --with-mumps-dir=/Users/davydden/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/mumps-5.0.2-4pwiz7bplhnmk2lfvg746u4pq4z5wuc3 --with-scalapack=1 --with-scalapack-dir=/Users/davydden/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/netlib-scalapack-2.0.2-gu4tjf4bjvtia3ohzjqbickedmywqslk --with-superlu_dist-include=/Users/davydden/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/superlu-dist-5.1.1-hut4gapp5v6qzvjlqahattdvr3tyynoy/include --with-superlu_dist-lib=/Users/davydden/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/superlu-dist-5.1.1-hut4gapp5v6qzvjlqahattdvr3tyynoy/lib/libsuperlu_dist.a --with-superlu_dist=1 [0]PETSC ERROR: #1 KSPSolve() line 847 in /private/var/folders/5k/sqpp24tx3ylds4fgm13pfht00000gn/T/davydden/spack-stage/spack-stage-zUUkMX/petsc-3.7.5/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #2 PCGAMGOptProlongator_AGG() line 1227 in /private/var/folders/5k/sqpp24tx3ylds4fgm13pfht00000gn/T/davydden/spack-stage/spack-stage-zUUkMX/petsc-3.7.5/src/ksp/pc/impls/gamg/agg.c [0]PETSC ERROR: #3 PCSetUp_GAMG() line 581 in /private/var/folders/5k/sqpp24tx3ylds4fgm13pfht00000gn/T/davydden/spack-stage/spack-stage-zUUkMX/petsc-3.7.5/src/ksp/pc/impls/gamg/gamg.c [0]PETSC ERROR: #4 PCSetUp() line 968 in /private/var/folders/5k/sqpp24tx3ylds4fgm13pfht00000gn/T/davydden/spack-stage/spack-stage-zUUkMX/petsc-3.7.5/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #5 KSPSetUp() line 390 in /private/var/folders/5k/sqpp24tx3ylds4fgm13pfht00000gn/T/davydden/spack-stage/spack-stage-zUUkMX/petsc-3.7.5/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #6 STSetUp_Sinvert() line 153 in /private/var/folders/5k/sqpp24tx3ylds4fgm13pfht00000gn/T/davydden/spack-stage/spack-stage-5WvuFP/slepc-3.7.3/src/sys/classes/st/impls/sinvert/sinvert.c [0]PETSC ERROR: #7 STSetUp() line 301 in /private/var/folders/5k/sqpp24tx3ylds4fgm13pfht00000gn/T/davydden/spack-stage/spack-stage-5WvuFP/slepc-3.7.3/src/sys/classes/st/interface/stsolve.c [0]PETSC ERROR: #8 EPSSetUp() line 205 in /private/var/folders/5k/sqpp24tx3ylds4fgm13pfht00000gn/T/davydden/spack-stage/spack-stage-5WvuFP/slepc-3.7.3/src/eps/interface/epssetup.c [0]PETSC ERROR: #9 EPSSolve() line 89 in /private/var/folders/5k/sqpp24tx3ylds4fgm13pfht00000gn/T/davydden/spack-stage/spack-stage-5WvuFP/slepc-3.7.3/src/eps/interface/epssolve.c ***[0]PCReset_GAMG this should not happen, cleaned up in SetUp ERROR: Uncaught exception in MPI_InitFinalize on proc 0. Skipping MPI_Finalize() to avoid a deadlock. From mfadams at lbl.gov Tue Feb 7 16:57:50 2017 From: mfadams at lbl.gov (Mark Adams) Date: Tue, 7 Feb 2017 17:57:50 -0500 Subject: [petsc-users] pc gamg did not converge in sinv where it used to In-Reply-To: <30A0F820-09ED-46E8-A4CD-3A8100D94C04@gmail.com> References: <30A0F820-09ED-46E8-A4CD-3A8100D94C04@gmail.com> Message-ID: This is failing on a test for a data cache being NULL. It is not. It is in the reset routine, which might not be used by many people. This code is pretty stable as far as I know, but maybe someone changed something in here. This is probably a false positive -- you could just comment out that line ("this should not happen, cleaned up in SetUp") in gamg.c:38 This data is freed in a few places with something like: ierr = PetscFree(pc_gamg->data);CHKERRQ(ierr); But, not set to NULL. Perhaps PetscFree should be called with the address of the pointer so that it can NULL it or explicitly zeroed out. PETSc: what is the proper way to do this? Mark On Tue, Feb 7, 2017 at 4:51 PM, Denis Davydov wrote: > Dear all, > > I rerun calculations (unit tests) which used to work with slightly older > versions of PETSc/SLEPc (a year ago, or so) and > see the "KSPSolve has not converged? error for shift and invert > transformation with gamg preconditioner (below). > Shifted matrix is SPD, could have bad condition number but it worked > before. > > Judging from a pre-last line in the log below, something is wrong with > GAMG or its interaction with SLEPc. > > Regards, > Denis. > > [0]PETSC ERROR: Configure options --prefix=/Users/davydden/spack > /opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/petsc-3. > 7.5-e35uxj5pf7ndtedbnzvq7dot5oie3psc --with-ssl=0 --with-x=0 > --download-c2html=0 --download-sowing=0 --download-hwloc=0 --with-mpi=1 > --with-mpi-dir=/Users/davydden/spack/opt/spack/darwin- > sierra-x86_64/clang-8.0.0-apple/openmpi-2.0.2-oos4cxprn2bislzhc3rbc3lo4dadginw > --with-precision=double --with-scalar-type=real --with-shared-libraries=1 > --with-debugging=0 --with-64-bit-indices=0 --with-blas-lapack-lib=/Users/ > davydden/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0- > apple/openblas-0.2.19-xg7etkjyo7xjnvuojsdc2xoixesxoerh/lib/libopenblas.dylib > --with-metis=1 --with-metis-dir=/Users/davydden/spack/opt/spack/darwin- > sierra-x86_64/clang-8.0.0-apple/metis-5.1.0-c2muo3b6k5aea2743g34s35u2ve23yxq > --with-boost=0 --with-hdf5=1 --with-hdf5-dir=/Users/davydde > n/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0- > apple/hdf5-1.10.0-patch1-kwkdkp5mo4olqpkitojab3xg3xhl6gfz --with-hypre=1 > --with-hypre-dir=/Users/davydden/spack/opt/spack/darwin- > sierra-x86_64/clang-8.0.0-apple/hypre-2.11.1-dprfz6t4t24wesj74dcfdtajwm5n5qzu > --with-parmetis=1 --with-parmetis-dir=/Users/dav > ydden/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0- > apple/parmetis-4.0.3-tm5dd6fqnbw6hiepajhkvmqk42xrrgos --with-mumps=1 > --with-mumps-dir=/Users/davydden/spack/opt/spack/darwin- > sierra-x86_64/clang-8.0.0-apple/mumps-5.0.2-4pwiz7bplhnmk2lfvg746u4pq4z5wuc3 > --with-scalapack=1 --with-scalapack-dir=/Users/da > vydden/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0- > apple/netlib-scalapack-2.0.2-gu4tjf4bjvtia3ohzjqbickedmywqslk > --with-superlu_dist-include=/Users/davydden/spack/opt/spack/ > darwin-sierra-x86_64/clang-8.0.0-apple/superlu-dist-5.1.1-h > ut4gapp5v6qzvjlqahattdvr3tyynoy/include --with-superlu_dist-lib=/Users > /davydden/spack/opt/spack/darwin-sierra-x86_64/clang-8. > 0.0-apple/superlu-dist-5.1.1-hut4gapp5v6qzvjlqahattdvr3tyynoy/lib/libsuperlu_dist.a > --with-superlu_dist=1 > [0]PETSC ERROR: #1 KSPSolve() line 847 in /private/var/folders/5k/sqpp24 > tx3ylds4fgm13pfht00000gn/T/davydden/spack-stage/spack-sta > ge-zUUkMX/petsc-3.7.5/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #2 PCGAMGOptProlongator_AGG() line 1227 in > /private/var/folders/5k/sqpp24tx3ylds4fgm13pfht00000gn/T/ > davydden/spack-stage/spack-stage-zUUkMX/petsc-3.7.5/src/ksp/ > pc/impls/gamg/agg.c > [0]PETSC ERROR: #3 PCSetUp_GAMG() line 581 in > /private/var/folders/5k/sqpp24tx3ylds4fgm13pfht00000gn/T/ > davydden/spack-stage/spack-stage-zUUkMX/petsc-3.7.5/src/ksp/ > pc/impls/gamg/gamg.c > [0]PETSC ERROR: #4 PCSetUp() line 968 in /private/var/folders/5k/sqpp24 > tx3ylds4fgm13pfht00000gn/T/davydden/spack-stage/spack-sta > ge-zUUkMX/petsc-3.7.5/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #5 KSPSetUp() line 390 in /private/var/folders/5k/sqpp24 > tx3ylds4fgm13pfht00000gn/T/davydden/spack-stage/spack-sta > ge-zUUkMX/petsc-3.7.5/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #6 STSetUp_Sinvert() line 153 in > /private/var/folders/5k/sqpp24tx3ylds4fgm13pfht00000gn/T/ > davydden/spack-stage/spack-stage-5WvuFP/slepc-3.7.3/src/sys/ > classes/st/impls/sinvert/sinvert.c > [0]PETSC ERROR: #7 STSetUp() line 301 in /private/var/folders/5k/sqpp24 > tx3ylds4fgm13pfht00000gn/T/davydden/spack-stage/spack-sta > ge-5WvuFP/slepc-3.7.3/src/sys/classes/st/interface/stsolve.c > [0]PETSC ERROR: #8 EPSSetUp() line 205 in /private/var/folders/5k/sqpp24 > tx3ylds4fgm13pfht00000gn/T/davydden/spack-stage/spack-sta > ge-5WvuFP/slepc-3.7.3/src/eps/interface/epssetup.c > [0]PETSC ERROR: #9 EPSSolve() line 89 in /private/var/folders/5k/sqpp24 > tx3ylds4fgm13pfht00000gn/T/davydden/spack-stage/spack-sta > ge-5WvuFP/slepc-3.7.3/src/eps/interface/epssolve.c > ***[0]PCReset_GAMG this should not happen, cleaned up in SetUp > ERROR: Uncaught exception in MPI_InitFinalize on proc 0. Skipping > MPI_Finalize() to avoid a deadlock. > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Feb 7 17:01:52 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 7 Feb 2017 17:01:52 -0600 Subject: [petsc-users] pc gamg did not converge in sinv where it used to In-Reply-To: References: <30A0F820-09ED-46E8-A4CD-3A8100D94C04@gmail.com> Message-ID: Mark, Look at the PetscFree() macro; it does set the pointer to null after freeing. Denis, Valgrind the code next > On Feb 7, 2017, at 4:57 PM, Mark Adams wrote: > > This is failing on a test for a data cache being NULL. It is not. It is in the reset routine, which might not be used by many people. This code is pretty stable as far as I know, but maybe someone changed something in here. This is probably a false positive -- you could just comment out that line ("this should not happen, cleaned up in SetUp") in gamg.c:38 > > This data is freed in a few places with something like: > > ierr = PetscFree(pc_gamg->data);CHKERRQ(ierr); > > But, not set to NULL. > > Perhaps PetscFree should be called with the address of the pointer so that it can NULL it or explicitly zeroed out. > > PETSc: what is the proper way to do this? > > Mark > > > On Tue, Feb 7, 2017 at 4:51 PM, Denis Davydov wrote: > Dear all, > > I rerun calculations (unit tests) which used to work with slightly older versions of PETSc/SLEPc (a year ago, or so) and > see the "KSPSolve has not converged? error for shift and invert transformation with gamg preconditioner (below). > Shifted matrix is SPD, could have bad condition number but it worked before. > > Judging from a pre-last line in the log below, something is wrong with GAMG or its interaction with SLEPc. > > Regards, > Denis. > > [0]PETSC ERROR: Configure options --prefix=/Users/davydden/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/petsc-3.7.5-e35uxj5pf7ndtedbnzvq7dot5oie3psc --with-ssl=0 --with-x=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 --with-mpi=1 --with-mpi-dir=/Users/davydden/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/openmpi-2.0.2-oos4cxprn2bislzhc3rbc3lo4dadginw --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-64-bit-indices=0 --with-blas-lapack-lib=/Users/davydden/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/openblas-0.2.19-xg7etkjyo7xjnvuojsdc2xoixesxoerh/lib/libopenblas.dylib --with-metis=1 --with-metis-dir=/Users/davydden/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/metis-5.1.0-c2muo3b6k5aea2743g34s35u2ve23yxq --with-boost=0 --with-hdf5=1 --with-hdf5-dir=/Users/davydden/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/hdf5-1.10.0-patch1-kwkdkp5mo4olqpkitojab3xg3xhl6gfz --with-hypre=1 --with-hypre-dir=/Users/davydden/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/hypre-2.11.1-dprfz6t4t24wesj74dcfdtajwm5n5qzu --with-parmetis=1 --with-parmetis-dir=/Users/davydden/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/parmetis-4.0.3-tm5dd6fqnbw6hiepajhkvmqk42xrrgos --with-mumps=1 --with-mumps-dir=/Users/davydden/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/mumps-5.0.2-4pwiz7bplhnmk2lfvg746u4pq4z5wuc3 --with-scalapack=1 --with-scalapack-dir=/Users/davydden/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/netlib-scalapack-2.0.2-gu4tjf4bjvtia3ohzjqbickedmywqslk --with-superlu_dist-include=/Users/davydden/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/superlu-dist-5.1.1-hut4gapp5v6qzvjlqahattdvr3tyynoy/include --with-superlu_dist-lib=/Users/davydden/spack/opt/spack/darwin-sierra-x86_64/clang-8.0.0-apple/superlu-dist-5.1.1-hut4gapp5v6qzvjlqahattdvr3tyynoy/lib/libsuperlu_dist.a --with-superlu_dist=1 > [0]PETSC ERROR: #1 KSPSolve() line 847 in /private/var/folders/5k/sqpp24tx3ylds4fgm13pfht00000gn/T/davydden/spack-stage/spack-stage-zUUkMX/petsc-3.7.5/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #2 PCGAMGOptProlongator_AGG() line 1227 in /private/var/folders/5k/sqpp24tx3ylds4fgm13pfht00000gn/T/davydden/spack-stage/spack-stage-zUUkMX/petsc-3.7.5/src/ksp/pc/impls/gamg/agg.c > [0]PETSC ERROR: #3 PCSetUp_GAMG() line 581 in /private/var/folders/5k/sqpp24tx3ylds4fgm13pfht00000gn/T/davydden/spack-stage/spack-stage-zUUkMX/petsc-3.7.5/src/ksp/pc/impls/gamg/gamg.c > [0]PETSC ERROR: #4 PCSetUp() line 968 in /private/var/folders/5k/sqpp24tx3ylds4fgm13pfht00000gn/T/davydden/spack-stage/spack-stage-zUUkMX/petsc-3.7.5/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #5 KSPSetUp() line 390 in /private/var/folders/5k/sqpp24tx3ylds4fgm13pfht00000gn/T/davydden/spack-stage/spack-stage-zUUkMX/petsc-3.7.5/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #6 STSetUp_Sinvert() line 153 in /private/var/folders/5k/sqpp24tx3ylds4fgm13pfht00000gn/T/davydden/spack-stage/spack-stage-5WvuFP/slepc-3.7.3/src/sys/classes/st/impls/sinvert/sinvert.c > [0]PETSC ERROR: #7 STSetUp() line 301 in /private/var/folders/5k/sqpp24tx3ylds4fgm13pfht00000gn/T/davydden/spack-stage/spack-stage-5WvuFP/slepc-3.7.3/src/sys/classes/st/interface/stsolve.c > [0]PETSC ERROR: #8 EPSSetUp() line 205 in /private/var/folders/5k/sqpp24tx3ylds4fgm13pfht00000gn/T/davydden/spack-stage/spack-stage-5WvuFP/slepc-3.7.3/src/eps/interface/epssetup.c > [0]PETSC ERROR: #9 EPSSolve() line 89 in /private/var/folders/5k/sqpp24tx3ylds4fgm13pfht00000gn/T/davydden/spack-stage/spack-stage-5WvuFP/slepc-3.7.3/src/eps/interface/epssolve.c > ***[0]PCReset_GAMG this should not happen, cleaned up in SetUp > ERROR: Uncaught exception in MPI_InitFinalize on proc 0. Skipping MPI_Finalize() to avoid a deadlock. > > > > From bhatiamanav at gmail.com Wed Feb 8 15:40:27 2017 From: bhatiamanav at gmail.com (Manav Bhatia) Date: Wed, 8 Feb 2017 15:40:27 -0600 Subject: [petsc-users] ksp solve error with nested matrix Message-ID: Hi, I have a nested matrix with 2x2 blocks. The blocks (1,1) and (2,2) are AIJ matrices and blocks (1,2) and (2,1) and shell matrices. I am calling the code with the following arguments: -pc_type fieldsplit , and get the error shown below. I see that the error is complaining about the matrix being in unassembled state, but I think I am initializing and calling assembly end routines on both the diagonal blocks. Still, there is something obvious I am missing. I would appreciate any guidance on this. Regards, Manav [1;31m[0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0;39m[0;49m[0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Not for unassembled matrix [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.7.4, Oct, 02, 2016 [0]PETSC ERROR: /Users/manav/Library/Developer/Xcode/DerivedData/MAST-crggwcqrouiyeucduvscdahjauvx/Build/Products/Debug/examples on a arch-darwin-cxx-opt named Dhcp-90-250.HPC.MsState.Edu by manav Wed Feb 8 15:28:04 2017 [0]PETSC ERROR: Configure options --prefix=/Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/../ --CC=mpicc-openmpi-clang38 --CXX=mpicxx-openmpi-clang38 --FC=mpif90-openmpi-clang38 --with-clanguage=c++ --with-fortran=0 --with-mpiexec=/opt/local/bin/mpiexec-openmpi-clang38 --with-shared-libraries=1 --with-x=1 --with-x-dir=/opt/X11 --with-debugging=0 --with-lapack-lib=/usr/lib/liblapack.dylib --with-blas-lib=/usr/lib/libblas.dylib --download-superlu=yes --download-superlu_dist=yes --download-suitesparse=yes --download-mumps=yes --download-scalapack=yes --download-parmetis=yes --download-metis=yes --download-hypre=yes --download-ml=yes [0]PETSC ERROR: #1 MatMult() line 2248 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/mat/interface/matrix.c [0]PETSC ERROR: #2 PCApplyBAorAB() line 717 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #3 KSP_PCApplyBAorAB() line 274 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/include/petsc/private/kspimpl.h [0]PETSC ERROR: #4 KSPGMRESCycle() line 156 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: #5 KSPSolve_GMRES() line 240 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: #6 KSPSolve() line 656 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #7 SNESSolve_NEWTONLS() line 230 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/snes/impls/ls/ls.c [0]PETSC ERROR: #8 SNESSolve() line 4005 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/snes/interface/snes.c -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Feb 8 15:47:17 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 8 Feb 2017 15:47:17 -0600 Subject: [petsc-users] ksp solve error with nested matrix In-Reply-To: References: Message-ID: <98DD59F5-9240-4C42-854C-F3160C021785@mcs.anl.gov> > On Feb 8, 2017, at 3:40 PM, Manav Bhatia wrote: > > Hi, > > I have a nested matrix with 2x2 blocks. The blocks (1,1) and (2,2) are AIJ matrices and blocks (1,2) and (2,1) and shell matrices. I am calling the code with the following arguments: -pc_type fieldsplit , and get the error shown below. > > I see that the error is complaining about the matrix being in unassembled state, but I think I am initializing and calling assembly end routines on both the diagonal blocks. Still, there is something obvious I am missing. It is complaining about the outer most matrix, not the blocks. Perhaps you haven't finished with setting up your nest matrix? > > I would appreciate any guidance on this. > > Regards, > Manav > > > > [1;31m[0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0;39m[0;49m[0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: Not for unassembled matrix > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.7.4, Oct, 02, 2016 > [0]PETSC ERROR: /Users/manav/Library/Developer/Xcode/DerivedData/MAST-crggwcqrouiyeucduvscdahjauvx/Build/Products/Debug/examples on a arch-darwin-cxx-opt named Dhcp-90-250.HPC.MsState.Edu by manav Wed Feb 8 15:28:04 2017 > [0]PETSC ERROR: Configure options --prefix=/Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/../ --CC=mpicc-openmpi-clang38 --CXX=mpicxx-openmpi-clang38 --FC=mpif90-openmpi-clang38 --with-clanguage=c++ --with-fortran=0 --with-mpiexec=/opt/local/bin/mpiexec-openmpi-clang38 --with-shared-libraries=1 --with-x=1 --with-x-dir=/opt/X11 --with-debugging=0 --with-lapack-lib=/usr/lib/liblapack.dylib --with-blas-lib=/usr/lib/libblas.dylib --download-superlu=yes --download-superlu_dist=yes --download-suitesparse=yes --download-mumps=yes --download-scalapack=yes --download-parmetis=yes --download-metis=yes --download-hypre=yes --download-ml=yes > [0]PETSC ERROR: #1 MatMult() line 2248 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/mat/interface/matrix.c > [0]PETSC ERROR: #2 PCApplyBAorAB() line 717 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #3 KSP_PCApplyBAorAB() line 274 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/include/petsc/private/kspimpl.h > [0]PETSC ERROR: #4 KSPGMRESCycle() line 156 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: #5 KSPSolve_GMRES() line 240 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: #6 KSPSolve() line 656 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #7 SNESSolve_NEWTONLS() line 230 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/snes/impls/ls/ls.c > [0]PETSC ERROR: #8 SNESSolve() line 4005 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/snes/interface/snes.c > From bhatiamanav at gmail.com Wed Feb 8 15:51:41 2017 From: bhatiamanav at gmail.com (Manav Bhatia) Date: Wed, 8 Feb 2017 15:51:41 -0600 Subject: [petsc-users] ksp solve error with nested matrix In-Reply-To: <98DD59F5-9240-4C42-854C-F3160C021785@mcs.anl.gov> References: <98DD59F5-9240-4C42-854C-F3160C021785@mcs.anl.gov> Message-ID: aha.. that might be it. Does that need to be called for the global matrix after each assembly of the Jacobian blocks, or just once for the whole matrix? -Manav > On Feb 8, 2017, at 3:47 PM, Barry Smith wrote: > > >> On Feb 8, 2017, at 3:40 PM, Manav Bhatia wrote: >> >> Hi, >> >> I have a nested matrix with 2x2 blocks. The blocks (1,1) and (2,2) are AIJ matrices and blocks (1,2) and (2,1) and shell matrices. I am calling the code with the following arguments: -pc_type fieldsplit , and get the error shown below. >> >> I see that the error is complaining about the matrix being in unassembled state, but I think I am initializing and calling assembly end routines on both the diagonal blocks. Still, there is something obvious I am missing. > > It is complaining about the outer most matrix, not the blocks. Perhaps you haven't finished with setting up your nest matrix? > >> >> I would appreciate any guidance on this. >> >> Regards, >> Manav >> >> >> >> [1;31m[0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [0;39m[0;49m[0]PETSC ERROR: Object is in wrong state >> [0]PETSC ERROR: Not for unassembled matrix >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.7.4, Oct, 02, 2016 >> [0]PETSC ERROR: /Users/manav/Library/Developer/Xcode/DerivedData/MAST-crggwcqrouiyeucduvscdahjauvx/Build/Products/Debug/examples on a arch-darwin-cxx-opt named Dhcp-90-250.HPC.MsState.Edu by manav Wed Feb 8 15:28:04 2017 >> [0]PETSC ERROR: Configure options --prefix=/Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/../ --CC=mpicc-openmpi-clang38 --CXX=mpicxx-openmpi-clang38 --FC=mpif90-openmpi-clang38 --with-clanguage=c++ --with-fortran=0 --with-mpiexec=/opt/local/bin/mpiexec-openmpi-clang38 --with-shared-libraries=1 --with-x=1 --with-x-dir=/opt/X11 --with-debugging=0 --with-lapack-lib=/usr/lib/liblapack.dylib --with-blas-lib=/usr/lib/libblas.dylib --download-superlu=yes --download-superlu_dist=yes --download-suitesparse=yes --download-mumps=yes --download-scalapack=yes --download-parmetis=yes --download-metis=yes --download-hypre=yes --download-ml=yes >> [0]PETSC ERROR: #1 MatMult() line 2248 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/mat/interface/matrix.c >> [0]PETSC ERROR: #2 PCApplyBAorAB() line 717 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: #3 KSP_PCApplyBAorAB() line 274 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/include/petsc/private/kspimpl.h >> [0]PETSC ERROR: #4 KSPGMRESCycle() line 156 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/ksp/ksp/impls/gmres/gmres.c >> [0]PETSC ERROR: #5 KSPSolve_GMRES() line 240 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/ksp/ksp/impls/gmres/gmres.c >> [0]PETSC ERROR: #6 KSPSolve() line 656 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: #7 SNESSolve_NEWTONLS() line 230 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/snes/impls/ls/ls.c >> [0]PETSC ERROR: #8 SNESSolve() line 4005 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/snes/interface/snes.c >> > From dave.mayhem23 at gmail.com Wed Feb 8 23:39:15 2017 From: dave.mayhem23 at gmail.com (Dave May) Date: Thu, 09 Feb 2017 05:39:15 +0000 Subject: [petsc-users] ksp solve error with nested matrix In-Reply-To: References: <98DD59F5-9240-4C42-854C-F3160C021785@mcs.anl.gov> Message-ID: Any time you modify one of the submats, you need to call assembly begin/end on that sub matrix AND on the outer matnest object. Thanks, Dave On Wed, 8 Feb 2017 at 22:51, Manav Bhatia wrote: > aha.. that might be it. > > Does that need to be called for the global matrix after each assembly of > the Jacobian blocks, or just once for the whole matrix? > > -Manav > > > On Feb 8, 2017, at 3:47 PM, Barry Smith wrote: > > > > > >> On Feb 8, 2017, at 3:40 PM, Manav Bhatia wrote: > >> > >> Hi, > >> > >> I have a nested matrix with 2x2 blocks. The blocks (1,1) and (2,2) > are AIJ matrices and blocks (1,2) and (2,1) and shell matrices. I am > calling the code with the following arguments: -pc_type fieldsplit , and > get the error shown below. > >> > >> I see that the error is complaining about the matrix being in > unassembled state, but I think I am initializing and calling assembly end > routines on both the diagonal blocks. Still, there is something obvious I > am missing. > > > > It is complaining about the outer most matrix, not the blocks. Perhaps > you haven't finished with setting up your nest matrix? > > > >> > >> I would appreciate any guidance on this. > >> > >> Regards, > >> Manav > >> > >> > >> > >> [1;31m[0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > >> [0;39m[0;49m[0]PETSC ERROR: Object is in wrong state > >> [0]PETSC ERROR: Not for unassembled matrix > >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > >> [0]PETSC ERROR: Petsc Release Version 3.7.4, Oct, 02, 2016 > >> [0]PETSC ERROR: > /Users/manav/Library/Developer/Xcode/DerivedData/MAST-crggwcqrouiyeucduvscdahjauvx/Build/Products/Debug/examples > on a arch-darwin-cxx-opt named Dhcp-90-250.HPC.MsState.Edu by manav Wed > Feb 8 15:28:04 2017 > >> [0]PETSC ERROR: Configure options > --prefix=/Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/../ > --CC=mpicc-openmpi-clang38 --CXX=mpicxx-openmpi-clang38 > --FC=mpif90-openmpi-clang38 --with-clanguage=c++ --with-fortran=0 > --with-mpiexec=/opt/local/bin/mpiexec-openmpi-clang38 > --with-shared-libraries=1 --with-x=1 --with-x-dir=/opt/X11 > --with-debugging=0 --with-lapack-lib=/usr/lib/liblapack.dylib > --with-blas-lib=/usr/lib/libblas.dylib --download-superlu=yes > --download-superlu_dist=yes --download-suitesparse=yes --download-mumps=yes > --download-scalapack=yes --download-parmetis=yes --download-metis=yes > --download-hypre=yes --download-ml=yes > >> [0]PETSC ERROR: #1 MatMult() line 2248 in > /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/mat/interface/matrix.c > >> [0]PETSC ERROR: #2 PCApplyBAorAB() line 717 in > /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/ksp/pc/interface/precon.c > >> [0]PETSC ERROR: #3 KSP_PCApplyBAorAB() line 274 in > /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/include/petsc/private/kspimpl.h > >> [0]PETSC ERROR: #4 KSPGMRESCycle() line 156 in > /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/ksp/ksp/impls/gmres/gmres.c > >> [0]PETSC ERROR: #5 KSPSolve_GMRES() line 240 in > /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/ksp/ksp/impls/gmres/gmres.c > >> [0]PETSC ERROR: #6 KSPSolve() line 656 in > /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/ksp/ksp/interface/itfunc.c > >> [0]PETSC ERROR: #7 SNESSolve_NEWTONLS() line 230 in > /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/snes/impls/ls/ls.c > >> [0]PETSC ERROR: #8 SNESSolve() line 4005 in > /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/snes/interface/snes.c > >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gideon.simpson at gmail.com Thu Feb 9 09:38:10 2017 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Thu, 9 Feb 2017 10:38:10 -0500 Subject: [petsc-users] projection methods in TS In-Reply-To: <57c95344-7062-e23a-d2e9-11c04bf18975@mcs.anl.gov> References: <2E0519CB-7CA7-41A4-9E67-C973BE33F35C@mcs.anl.gov> <57c95344-7062-e23a-d2e9-11c04bf18975@mcs.anl.gov> Message-ID: <81807603-FF09-47CB-B2E0-5E9D58FFF041@gmail.com> Ok, thanks for the information. I am looking at a Hamiltonian system, and I wanted to try out different schemes for preserving the energy. I was curious about textbook projected RK, so I think I?ll try Barry?s suggestion about TSSetPostStep. -gideon > On Feb 4, 2017, at 10:26 PM, Emil Constantinescu wrote: > > > > On 2/4/17 11:00 AM, Matthew Knepley wrote: >> On Sat, Feb 4, 2017 at 9:47 AM, Gideon Simpson > > wrote: >> >> Would setting it up as a DAE in petsc be algorithmically euivalent >> to a projected method (i.e., step of standard RK followed by >> nonlinear projection)? > > Not quite, if you set it as a DAE, then both the time stepping part and the algebraic constraint are solved at the same time (you are going to use an implicit time stepping method. > > So for backward Euler and DAE, PETSc solves something like: > > y_{n+1} - y{n} - dt* f(y_{n+1}) = 0 > g(y_{n+1}) = 0 > > By projection (assuming index 1 or 2): you are solving > > y* - y{n} - dt* f(y*) = 0 > > then initialize SNES to y* and solve: > > g(y_{n+1}) = 0 > > In some cases g(y_{n+1}) may not have a value (you may not land on the manifold that is defined by the conservation property). Whereas, if you define it as a DAE in almost all cases (there are crazy exceptions) you will. > > > Emil > >> >> I am not sure, as I do not understand those solvers. However, I wrote my >> own solver that does exactly that MIMEX. >> >> Matt >> >> >> -gideon >> >>> On Feb 3, 2017, at 11:47 PM, Matthew Knepley >> > wrote: >>> >>> That is one answer. Another one is that this particular system is >>> a DAE and we have methods for that. >>> >>> Matt >>> >>> On Fri, Feb 3, 2017 at 8:40 PM, Barry Smith >> > wrote: >>> >>> >>> TSSetPostStep(); in your function use TSGetSolution() to get >>> the current solution. >>> >>> Please let us know how it works out >>> >>> Barry >>> >>> >>> >>> > On Feb 3, 2017, at 7:14 PM, Gideon Simpson >>> > >>> wrote: >>> > >>> > I?m interested in implementing a projection method for an >>> ODE of the form: >>> > >>> > y? = f(y), >>> > >>> > such that g(y) = 0 for all time (i.e., g is conserved). >>> Note that in a projection method, a standard time step is made >>> to produce y* from y_{n}, and then this is corrected to obtain >>> y_{n+1} satisfying g(y) = 0. >>> > >>> > There were two ways I was thinking of doing this, and I was >>> hoping to get some input: >>> > >>> > Idea 1: Manually loop through using taking a time step and >>> then implementing the projection routine. I see that there is >>> a TSStep command, but this doesn?t seem to be much >>> documentation on how to use it in this scenario. Does anyone >>> have any guidance? >>> > >>> > Idea 2: Is there some analog to TSMonitor that allows me to >>> modify the solution after each time step, instead of just >>> allowing for some computation of a statistic? >>> > >>> > >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to >>> which their experiments lead. >>> -- Norbert Wiener >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which >> their experiments lead. >> -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Feb 9 11:29:48 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 9 Feb 2017 11:29:48 -0600 Subject: [petsc-users] ksp solve error with nested matrix In-Reply-To: References: <98DD59F5-9240-4C42-854C-F3160C021785@mcs.anl.gov> Message-ID: <6F2742DC-B510-4737-8FA1-4806B13358A7@mcs.anl.gov> > On Feb 8, 2017, at 11:39 PM, Dave May wrote: > > Any time you modify one of the submats, you need to call assembly begin/end on that sub matrix AND on the outer matnest object. Weird, and prone to errors it seems to me. Perhaps this needs to be rethought > > Thanks, > Dave > > > On Wed, 8 Feb 2017 at 22:51, Manav Bhatia wrote: > aha.. that might be it. > > Does that need to be called for the global matrix after each assembly of the Jacobian blocks, or just once for the whole matrix? > > -Manav > > > On Feb 8, 2017, at 3:47 PM, Barry Smith wrote: > > > > > >> On Feb 8, 2017, at 3:40 PM, Manav Bhatia wrote: > >> > >> Hi, > >> > >> I have a nested matrix with 2x2 blocks. The blocks (1,1) and (2,2) are AIJ matrices and blocks (1,2) and (2,1) and shell matrices. I am calling the code with the following arguments: -pc_type fieldsplit , and get the error shown below. > >> > >> I see that the error is complaining about the matrix being in unassembled state, but I think I am initializing and calling assembly end routines on both the diagonal blocks. Still, there is something obvious I am missing. > > > > It is complaining about the outer most matrix, not the blocks. Perhaps you haven't finished with setting up your nest matrix? > > > >> > >> I would appreciate any guidance on this. > >> > >> Regards, > >> Manav > >> > >> > >> > >> [1;31m[0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > >> [0;39m[0;49m[0]PETSC ERROR: Object is in wrong state > >> [0]PETSC ERROR: Not for unassembled matrix > >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > >> [0]PETSC ERROR: Petsc Release Version 3.7.4, Oct, 02, 2016 > >> [0]PETSC ERROR: /Users/manav/Library/Developer/Xcode/DerivedData/MAST-crggwcqrouiyeucduvscdahjauvx/Build/Products/Debug/examples on a arch-darwin-cxx-opt named Dhcp-90-250.HPC.MsState.Edu by manav Wed Feb 8 15:28:04 2017 > >> [0]PETSC ERROR: Configure options --prefix=/Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/../ --CC=mpicc-openmpi-clang38 --CXX=mpicxx-openmpi-clang38 --FC=mpif90-openmpi-clang38 --with-clanguage=c++ --with-fortran=0 --with-mpiexec=/opt/local/bin/mpiexec-openmpi-clang38 --with-shared-libraries=1 --with-x=1 --with-x-dir=/opt/X11 --with-debugging=0 --with-lapack-lib=/usr/lib/liblapack.dylib --with-blas-lib=/usr/lib/libblas.dylib --download-superlu=yes --download-superlu_dist=yes --download-suitesparse=yes --download-mumps=yes --download-scalapack=yes --download-parmetis=yes --download-metis=yes --download-hypre=yes --download-ml=yes > >> [0]PETSC ERROR: #1 MatMult() line 2248 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/mat/interface/matrix.c > >> [0]PETSC ERROR: #2 PCApplyBAorAB() line 717 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/ksp/pc/interface/precon.c > >> [0]PETSC ERROR: #3 KSP_PCApplyBAorAB() line 274 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/include/petsc/private/kspimpl.h > >> [0]PETSC ERROR: #4 KSPGMRESCycle() line 156 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/ksp/ksp/impls/gmres/gmres.c > >> [0]PETSC ERROR: #5 KSPSolve_GMRES() line 240 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/ksp/ksp/impls/gmres/gmres.c > >> [0]PETSC ERROR: #6 KSPSolve() line 656 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/ksp/ksp/interface/itfunc.c > >> [0]PETSC ERROR: #7 SNESSolve_NEWTONLS() line 230 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/snes/impls/ls/ls.c > >> [0]PETSC ERROR: #8 SNESSolve() line 4005 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/snes/interface/snes.c > >> > > > From lawrence.mitchell at imperial.ac.uk Thu Feb 9 11:33:04 2017 From: lawrence.mitchell at imperial.ac.uk (Lawrence Mitchell) Date: Thu, 9 Feb 2017 17:33:04 +0000 Subject: [petsc-users] ksp solve error with nested matrix In-Reply-To: <6F2742DC-B510-4737-8FA1-4806B13358A7@mcs.anl.gov> References: <98DD59F5-9240-4C42-854C-F3160C021785@mcs.anl.gov> <6F2742DC-B510-4737-8FA1-4806B13358A7@mcs.anl.gov> Message-ID: <6b981310-f6bf-31aa-9b13-4828f1db718b@imperial.ac.uk> On 09/02/17 17:29, Barry Smith wrote: > >> On Feb 8, 2017, at 11:39 PM, Dave May wrote: >> >> Any time you modify one of the submats, you need to call assembly begin/end on that sub matrix AND on the outer matnest object. > > Weird, and prone to errors it seems to me. Perhaps this needs to be rethought If you just call assembly begin/end on the outer object, that takes care of calling it on the nested matrices. But that's still not ideal, I think. Lawrence -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 473 bytes Desc: OpenPGP digital signature URL: From bsmith at mcs.anl.gov Thu Feb 9 12:55:58 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 9 Feb 2017 12:55:58 -0600 Subject: [petsc-users] ksp solve error with nested matrix In-Reply-To: <6b981310-f6bf-31aa-9b13-4828f1db718b@imperial.ac.uk> References: <98DD59F5-9240-4C42-854C-F3160C021785@mcs.anl.gov> <6F2742DC-B510-4737-8FA1-4806B13358A7@mcs.anl.gov> <6b981310-f6bf-31aa-9b13-4828f1db718b@imperial.ac.uk> Message-ID: <8DBBD153-0D53-486E-82B3-19A9D476861C@mcs.anl.gov> Hmm, we might need to overload the MatAssemblyEnd() on the inner matrices to have them update the state of the outer matrix. Barry > On Feb 9, 2017, at 11:33 AM, Lawrence Mitchell wrote: > > > > On 09/02/17 17:29, Barry Smith wrote: >> >>> On Feb 8, 2017, at 11:39 PM, Dave May wrote: >>> >>> Any time you modify one of the submats, you need to call assembly begin/end on that sub matrix AND on the outer matnest object. >> >> Weird, and prone to errors it seems to me. Perhaps this needs to be rethought > > If you just call assembly begin/end on the outer object, that takes > care of calling it on the nested matrices. But that's still not > ideal, I think. > > Lawrence > From thronesf at gmail.com Thu Feb 9 20:28:05 2017 From: thronesf at gmail.com (Sharp Stone) Date: Thu, 9 Feb 2017 21:28:05 -0500 Subject: [petsc-users] petscprint problem Message-ID: Hi, I use "PetscPrintf" and "PetscFPrintf" to make outputs in my code. In Linux, it works fine. But now I run the code in Cray cluster, I found every process will make a output, which make me go crazy and confused. Any ideas? Thanks! -- Best regards, Feng -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Feb 9 20:42:23 2017 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 9 Feb 2017 20:42:23 -0600 Subject: [petsc-users] petscprint problem In-Reply-To: References: Message-ID: Who prints is controlled by what communicator you pass it. If you want 1 person to print, use PETSC_COMM_WORLD. Matt On Thu, Feb 9, 2017 at 8:28 PM, Sharp Stone wrote: > Hi, > > I use "PetscPrintf" and "PetscFPrintf" to make outputs in my code. In > Linux, it works fine. But now I run the code in Cray cluster, I found every > process will make a output, which make me go crazy and confused. Any ideas? > > Thanks! > > -- > Best regards, > > Feng > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From thronesf at gmail.com Fri Feb 10 11:51:15 2017 From: thronesf at gmail.com (Sharp Stone) Date: Fri, 10 Feb 2017 12:51:15 -0500 Subject: [petsc-users] configure PETSc on Cray Message-ID: Hi all, I noticed the instructions of Petsc with MPI. Now I'm using Cray. So the instruction: Vendor provided MPI might already be installed. IBM, SGI, Cray etc provide their own: ./config/confiure.py --with-cc=mpcc --with-fc=mpf77 When I use the command "./config/confiure.py --with-cc=mpcc --with-fc=mpf77", it gives me the error "-bash: ./config/confiure.py: No such file or directory". But the file "confiure.py" is there in the "config" folder. What should I do? Thanks very much! -- Best regards, Feng -------------- next part -------------- An HTML attachment was scrubbed... URL: From patrick.sanan at gmail.com Fri Feb 10 11:55:41 2017 From: patrick.sanan at gmail.com (Patrick Sanan) Date: Fri, 10 Feb 2017 18:55:41 +0100 Subject: [petsc-users] configure PETSc on Cray In-Reply-To: References: Message-ID: You're missing the 'g' in 'configure.py'. On Fri, Feb 10, 2017 at 6:51 PM, Sharp Stone wrote: > Hi all, > > I noticed the instructions of Petsc with MPI. Now I'm using Cray. So the > instruction: > > Vendor provided MPI might already be installed. IBM, SGI, Cray etc provide > their own: > ./config/confiure.py --with-cc=mpcc --with-fc=mpf77 > > When I use the command "./config/confiure.py --with-cc=mpcc > --with-fc=mpf77", it gives me the error "-bash: ./config/confiure.py: No > such file or directory". But the file "confiure.py" is there in the "config" > folder. > > What should I do? > > Thanks very much! > > -- > Best regards, > > Feng From balay at mcs.anl.gov Fri Feb 10 12:01:42 2017 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 10 Feb 2017 12:01:42 -0600 Subject: [petsc-users] configure PETSc on Cray In-Reply-To: References: Message-ID: check config/examples/arch-cray-xt6-pkgs-opt.py Also check if your machine already has petsc [from cray] installed module avail petsc Satish On Fri, 10 Feb 2017, Patrick Sanan wrote: > You're missing the 'g' in 'configure.py'. > > On Fri, Feb 10, 2017 at 6:51 PM, Sharp Stone wrote: > > Hi all, > > > > I noticed the instructions of Petsc with MPI. Now I'm using Cray. So the > > instruction: > > > > Vendor provided MPI might already be installed. IBM, SGI, Cray etc provide > > their own: > > ./config/confiure.py --with-cc=mpcc --with-fc=mpf77 > > > > When I use the command "./config/confiure.py --with-cc=mpcc > > --with-fc=mpf77", it gives me the error "-bash: ./config/confiure.py: No > > such file or directory". But the file "confiure.py" is there in the "config" > > folder. > > > > What should I do? > > > > Thanks very much! > > > > -- > > Best regards, > > > > Feng > From thronesf at gmail.com Fri Feb 10 12:20:03 2017 From: thronesf at gmail.com (Sharp Stone) Date: Fri, 10 Feb 2017 13:20:03 -0500 Subject: [petsc-users] configure PETSc on Cray In-Reply-To: References: Message-ID: Thanks for the replies. The PETSc is installed on the Cray I'm using, but the version is too old cray-petsc/3.3.04. I've still got two questions: 1. how can I use the installed cray-petsc? I've used module load cray-petsc, but I can't find the path of petsc, which is required in the makefile. 2. Therefore, I decided to configure the latest version of Petsc. When i use ./config/configure.py --with-cc=cc --with-cxx=CC --with-fc=ftn, it give me the errors ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- C libraries cannot directly be used from Fortran In the log file, I found these: ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- C libraries cannot directly be used from Fortran ******************************************************************************* File "./config/configure.py", line 405, in petsc_configure framework.configure(out = sys.stdout) File "/mnt/lustre/lus0/space/fs1036/local/PETSc/petsc-3.7.4/config/BuildSystem/config/framework.py", line 1090, in configure self.processChildren() File "/mnt/lustre/lus0/space/fs1036/local/PETSc/petsc-3.7.4/config/BuildSystem/config/framework.py", line 1079, in processChildren self.serialEvaluation(self.childGraph) File "/mnt/lustre/lus0/space/fs1036/local/PETSc/petsc-3.7.4/config/BuildSystem/config/framework.py", line 1060, in serialEvaluation child.configure() File "/mnt/lustre/lus0/space/fs1036/local/PETSc/petsc-3.7.4/config/BuildSystem/config/compilers.py", line 1438, in configure self.executeTest(self.checkCLibraries) File "/mnt/lustre/lus0/space/fs1036/local/PETSc/petsc-3.7.4/config/BuildSystem/config/base.py", line 126, in executeTest ret = test(*args,**kargs) File "/mnt/lustre/lus0/space/fs1036/local/PETSc/petsc-3.7.4/config/BuildSystem/config/compilers.py", line 313, in checkCLibraries raise RuntimeError('C libraries cannot directly be used from Fortran') ================================================================================ Finishing Configure Run at Fri Feb 10 13:07:10 2017 ================================================================================ Thank you again for your help! On Fri, Feb 10, 2017 at 1:01 PM, Satish Balay wrote: > check config/examples/arch-cray-xt6-pkgs-opt.py > > Also check if your machine already has petsc [from cray] installed > > module avail petsc > > Satish > > On Fri, 10 Feb 2017, Patrick Sanan wrote: > > > You're missing the 'g' in 'configure.py'. > > > > On Fri, Feb 10, 2017 at 6:51 PM, Sharp Stone wrote: > > > Hi all, > > > > > > I noticed the instructions of Petsc with MPI. Now I'm using Cray. So > the > > > instruction: > > > > > > Vendor provided MPI might already be installed. IBM, SGI, Cray etc > provide > > > their own: > > > ./config/confiure.py --with-cc=mpcc --with-fc=mpf77 > > > > > > When I use the command "./config/confiure.py --with-cc=mpcc > > > --with-fc=mpf77", it gives me the error "-bash: ./config/confiure.py: > No > > > such file or directory". But the file "confiure.py" is there in the > "config" > > > folder. > > > > > > What should I do? > > > > > > Thanks very much! > > > > > > -- > > > Best regards, > > > > > > Feng > > > > -- Best regards, Feng -------------- next part -------------- An HTML attachment was scrubbed... URL: From patrick.sanan at gmail.com Fri Feb 10 12:38:05 2017 From: patrick.sanan at gmail.com (Patrick Sanan) Date: Fri, 10 Feb 2017 19:38:05 +0100 Subject: [petsc-users] configure PETSc on Cray In-Reply-To: References: Message-ID: On Fri, Feb 10, 2017 at 7:20 PM, Sharp Stone wrote: > Thanks for the replies. > > The PETSc is installed on the Cray I'm using, but the version is too old > cray-petsc/3.3.04. > I've still got two questions: > > 1. how can I use the installed cray-petsc? I've used module load cray-petsc, > but I can't find the path of petsc, which is required in the makefile. You can use module show cray-petsc to at least get some more hints as to what cray did, PETSc-wise. > > 2. Therefore, I decided to configure the latest version of Petsc. When i use > ./config/configure.py --with-cc=cc --with-cxx=CC --with-fc=ftn, it give me > the errors > ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > details): > ------------------------------------------------------------------------------- > C libraries cannot directly be used from Fortran > > In the log file, I found these: > ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > details): > ------------------------------------------------------------------------------- > C libraries cannot directly be used from Fortran > ******************************************************************************* > File "./config/configure.py", line 405, in petsc_configure > framework.configure(out = sys.stdout) > File > "/mnt/lustre/lus0/space/fs1036/local/PETSc/petsc-3.7.4/config/BuildSystem/config/framework.py", > line 1090, in configure > self.processChildren() > File > "/mnt/lustre/lus0/space/fs1036/local/PETSc/petsc-3.7.4/config/BuildSystem/config/framework.py", > line 1079, in processChildren > self.serialEvaluation(self.childGraph) > File > "/mnt/lustre/lus0/space/fs1036/local/PETSc/petsc-3.7.4/config/BuildSystem/config/framework.py", > line 1060, in serialEvaluation > child.configure() > File > "/mnt/lustre/lus0/space/fs1036/local/PETSc/petsc-3.7.4/config/BuildSystem/config/compilers.py", > line 1438, in configure > self.executeTest(self.checkCLibraries) > File > "/mnt/lustre/lus0/space/fs1036/local/PETSc/petsc-3.7.4/config/BuildSystem/config/base.py", > line 126, in executeTest > ret = test(*args,**kargs) > File > "/mnt/lustre/lus0/space/fs1036/local/PETSc/petsc-3.7.4/config/BuildSystem/config/compilers.py", > line 313, in checkCLibraries > raise RuntimeError('C libraries cannot directly be used from Fortran') > ================================================================================ > Finishing Configure Run at Fri Feb 10 13:07:10 2017 > ================================================================================ > > > Thank you again for your help! > > On Fri, Feb 10, 2017 at 1:01 PM, Satish Balay wrote: >> >> check config/examples/arch-cray-xt6-pkgs-opt.py >> >> Also check if your machine already has petsc [from cray] installed >> >> module avail petsc >> >> Satish >> >> On Fri, 10 Feb 2017, Patrick Sanan wrote: >> >> > You're missing the 'g' in 'configure.py'. >> > >> > On Fri, Feb 10, 2017 at 6:51 PM, Sharp Stone wrote: >> > > Hi all, >> > > >> > > I noticed the instructions of Petsc with MPI. Now I'm using Cray. So >> > > the >> > > instruction: >> > > >> > > Vendor provided MPI might already be installed. IBM, SGI, Cray etc >> > > provide >> > > their own: >> > > ./config/confiure.py --with-cc=mpcc --with-fc=mpf77 >> > > >> > > When I use the command "./config/confiure.py --with-cc=mpcc >> > > --with-fc=mpf77", it gives me the error "-bash: ./config/confiure.py: >> > > No >> > > such file or directory". But the file "confiure.py" is there in the >> > > "config" >> > > folder. >> > > >> > > What should I do? >> > > >> > > Thanks very much! >> > > >> > > -- >> > > Best regards, >> > > >> > > Feng >> > >> > > > > -- > Best regards, > > Feng From balay at mcs.anl.gov Fri Feb 10 13:01:54 2017 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 10 Feb 2017 13:01:54 -0600 Subject: [petsc-users] configure PETSc on Cray In-Reply-To: References: Message-ID: On Fri, 10 Feb 2017, Patrick Sanan wrote: > On Fri, Feb 10, 2017 at 7:20 PM, Sharp Stone wrote: > > Thanks for the replies. > > > > The PETSc is installed on the Cray I'm using, but the version is too old > > cray-petsc/3.3.04. > > I've still got two questions: > > > > 1. how can I use the installed cray-petsc? I've used module load cray-petsc, > > but I can't find the path of petsc, which is required in the makefile. > You can use > > module show cray-petsc > > to at least get some more hints as to what cray did, PETSc-wise. Ok this is old version - and you might want to avoid it. The way to use it would be - without petsc makefiles. i.e you would compile with: cc code.c > > > > 2. Therefore, I decided to configure the latest version of Petsc. When i use > > ./config/configure.py --with-cc=cc --with-cxx=CC --with-fc=ftn, it give me > > the errors send configure.log What type of cray machine is this? Also what compilers are you using? send module list Satish > > ******************************************************************************* > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > > details): > > ------------------------------------------------------------------------------- > > C libraries cannot directly be used from Fortran > > > > In the log file, I found these: > > ******************************************************************************* > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > > details): > > ------------------------------------------------------------------------------- > > C libraries cannot directly be used from Fortran > > ******************************************************************************* > > File "./config/configure.py", line 405, in petsc_configure > > framework.configure(out = sys.stdout) > > File > > "/mnt/lustre/lus0/space/fs1036/local/PETSc/petsc-3.7.4/config/BuildSystem/config/framework.py", > > line 1090, in configure > > self.processChildren() > > File > > "/mnt/lustre/lus0/space/fs1036/local/PETSc/petsc-3.7.4/config/BuildSystem/config/framework.py", > > line 1079, in processChildren > > self.serialEvaluation(self.childGraph) > > File > > "/mnt/lustre/lus0/space/fs1036/local/PETSc/petsc-3.7.4/config/BuildSystem/config/framework.py", > > line 1060, in serialEvaluation > > child.configure() > > File > > "/mnt/lustre/lus0/space/fs1036/local/PETSc/petsc-3.7.4/config/BuildSystem/config/compilers.py", > > line 1438, in configure > > self.executeTest(self.checkCLibraries) > > File > > "/mnt/lustre/lus0/space/fs1036/local/PETSc/petsc-3.7.4/config/BuildSystem/config/base.py", > > line 126, in executeTest > > ret = test(*args,**kargs) > > File > > "/mnt/lustre/lus0/space/fs1036/local/PETSc/petsc-3.7.4/config/BuildSystem/config/compilers.py", > > line 313, in checkCLibraries > > raise RuntimeError('C libraries cannot directly be used from Fortran') > > ================================================================================ > > Finishing Configure Run at Fri Feb 10 13:07:10 2017 > > ================================================================================ > > > > > > Thank you again for your help! > > > > On Fri, Feb 10, 2017 at 1:01 PM, Satish Balay wrote: > >> > >> check config/examples/arch-cray-xt6-pkgs-opt.py > >> > >> Also check if your machine already has petsc [from cray] installed > >> > >> module avail petsc > >> > >> Satish > >> > >> On Fri, 10 Feb 2017, Patrick Sanan wrote: > >> > >> > You're missing the 'g' in 'configure.py'. > >> > > >> > On Fri, Feb 10, 2017 at 6:51 PM, Sharp Stone wrote: > >> > > Hi all, > >> > > > >> > > I noticed the instructions of Petsc with MPI. Now I'm using Cray. So > >> > > the > >> > > instruction: > >> > > > >> > > Vendor provided MPI might already be installed. IBM, SGI, Cray etc > >> > > provide > >> > > their own: > >> > > ./config/confiure.py --with-cc=mpcc --with-fc=mpf77 > >> > > > >> > > When I use the command "./config/confiure.py --with-cc=mpcc > >> > > --with-fc=mpf77", it gives me the error "-bash: ./config/confiure.py: > >> > > No > >> > > such file or directory". But the file "confiure.py" is there in the > >> > > "config" > >> > > folder. > >> > > > >> > > What should I do? > >> > > > >> > > Thanks very much! > >> > > > >> > > -- > >> > > Best regards, > >> > > > >> > > Feng > >> > > >> > > > > > > > > -- > > Best regards, > > > > Feng > From thronesf at gmail.com Fri Feb 10 13:31:09 2017 From: thronesf at gmail.com (Sharp Stone) Date: Fri, 10 Feb 2017 14:31:09 -0500 Subject: [petsc-users] configure PETSc on Cray In-Reply-To: References: Message-ID: Thank you all for the replies. I'm using Cray XE6M-200, and compilers of gcc/g++/gfortran (by default). The module list is: Currently Loaded Modulefiles: 1) modules/3.2.6.7 2) nodestat/2.2-1.0402.39779.3.4.gem 3) sdb/1.0-1.0402.41597.10.13.gem 4) MySQL/5.0.64-1.0000.5053.22.1 5) lustre-cray_gem_s/1.8.6_2.6.32.59_0.7.1_1.0402.6998.7.1-1.0402.41312.15.19 6) udreg/2.3.2-1.0402.6376.3.1.gem 7) ugni/5.0-1.0402.6488.5.7.gem 8) gni-headers/2.1-1.0402.6487.6.4.gem 9) dmapp/4.0.1-1.0402.6266.5.11.gem 10) xpmem/0.1-2.0402.39965.2.5.gem 11) hss-llm/7.0.0 12) Base-opts/1.0.2-1.0402.38884.1.2.gem 13) craype-network-gemini 14) xt-asyncpe/5.19 15) cce/8.1.7 16) cray-libsci/12.0.01 17) pmi/4.0.1-1.0000.9421.73.3.gem 18) rca/1.0.0-2.0402.39780.3.94.gem 19) atp/1.6.2 20) PrgEnv-cray/4.2.15 21) pbs/11.3.1.122650 22) cray-mpich2/5.6.4 23) cray-hdf5-parallel/1.8.9 Thanks in advance! On Fri, Feb 10, 2017 at 2:01 PM, Satish Balay wrote: > On Fri, 10 Feb 2017, Patrick Sanan wrote: > > > On Fri, Feb 10, 2017 at 7:20 PM, Sharp Stone wrote: > > > Thanks for the replies. > > > > > > The PETSc is installed on the Cray I'm using, but the version is too > old > > > cray-petsc/3.3.04. > > > I've still got two questions: > > > > > > 1. how can I use the installed cray-petsc? I've used module load > cray-petsc, > > > but I can't find the path of petsc, which is required in the makefile. > > You can use > > > > module show cray-petsc > > > > to at least get some more hints as to what cray did, PETSc-wise. > > Ok this is old version - and you might want to avoid it. The way to > use it would be - without petsc makefiles. i.e you would compile with: > > cc code.c > > > > > > > > 2. Therefore, I decided to configure the latest version of Petsc. When > i use > > > ./config/configure.py --with-cc=cc --with-cxx=CC --with-fc=ftn, it > give me > > > the errors > > send configure.log > > What type of cray machine is this? Also what compilers are you using? send > module list > > Satish > > > > > ************************************************************ > ******************* > > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > for > > > details): > > > ------------------------------------------------------------ > ------------------- > > > C libraries cannot directly be used from Fortran > > > > > > In the log file, I found these: > > > ************************************************************ > ******************* > > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > for > > > details): > > > ------------------------------------------------------------ > ------------------- > > > C libraries cannot directly be used from Fortran > > > ************************************************************ > ******************* > > > File "./config/configure.py", line 405, in petsc_configure > > > framework.configure(out = sys.stdout) > > > File > > > "/mnt/lustre/lus0/space/fs1036/local/PETSc/petsc-3.7. > 4/config/BuildSystem/config/framework.py", > > > line 1090, in configure > > > self.processChildren() > > > File > > > "/mnt/lustre/lus0/space/fs1036/local/PETSc/petsc-3.7. > 4/config/BuildSystem/config/framework.py", > > > line 1079, in processChildren > > > self.serialEvaluation(self.childGraph) > > > File > > > "/mnt/lustre/lus0/space/fs1036/local/PETSc/petsc-3.7. > 4/config/BuildSystem/config/framework.py", > > > line 1060, in serialEvaluation > > > child.configure() > > > File > > > "/mnt/lustre/lus0/space/fs1036/local/PETSc/petsc-3.7. > 4/config/BuildSystem/config/compilers.py", > > > line 1438, in configure > > > self.executeTest(self.checkCLibraries) > > > File > > > "/mnt/lustre/lus0/space/fs1036/local/PETSc/petsc-3.7. > 4/config/BuildSystem/config/base.py", > > > line 126, in executeTest > > > ret = test(*args,**kargs) > > > File > > > "/mnt/lustre/lus0/space/fs1036/local/PETSc/petsc-3.7. > 4/config/BuildSystem/config/compilers.py", > > > line 313, in checkCLibraries > > > raise RuntimeError('C libraries cannot directly be used from > Fortran') > > > ============================================================ > ==================== > > > Finishing Configure Run at Fri Feb 10 13:07:10 2017 > > > ============================================================ > ==================== > > > > > > > > > Thank you again for your help! > > > > > > On Fri, Feb 10, 2017 at 1:01 PM, Satish Balay > wrote: > > >> > > >> check config/examples/arch-cray-xt6-pkgs-opt.py > > >> > > >> Also check if your machine already has petsc [from cray] installed > > >> > > >> module avail petsc > > >> > > >> Satish > > >> > > >> On Fri, 10 Feb 2017, Patrick Sanan wrote: > > >> > > >> > You're missing the 'g' in 'configure.py'. > > >> > > > >> > On Fri, Feb 10, 2017 at 6:51 PM, Sharp Stone > wrote: > > >> > > Hi all, > > >> > > > > >> > > I noticed the instructions of Petsc with MPI. Now I'm using Cray. > So > > >> > > the > > >> > > instruction: > > >> > > > > >> > > Vendor provided MPI might already be installed. IBM, SGI, Cray etc > > >> > > provide > > >> > > their own: > > >> > > ./config/confiure.py --with-cc=mpcc --with-fc=mpf77 > > >> > > > > >> > > When I use the command "./config/confiure.py --with-cc=mpcc > > >> > > --with-fc=mpf77", it gives me the error "-bash: > ./config/confiure.py: > > >> > > No > > >> > > such file or directory". But the file "confiure.py" is there in > the > > >> > > "config" > > >> > > folder. > > >> > > > > >> > > What should I do? > > >> > > > > >> > > Thanks very much! > > >> > > > > >> > > -- > > >> > > Best regards, > > >> > > > > >> > > Feng > > >> > > > >> > > > > > > > > > > > > -- > > > Best regards, > > > > > > Feng > > > > -- Best regards, Feng -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 299104 bytes Desc: not available URL: From balay at mcs.anl.gov Fri Feb 10 13:39:06 2017 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 10 Feb 2017 13:39:06 -0600 Subject: [petsc-users] configure PETSc on Cray In-Reply-To: References: Message-ID: Try: ./configure --with-cc=cc --with-cxx=CC --with-fc=ftn --with-debugging=0 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 Check config/examples/arch-cray-xt6-pkgs-opt.py Satish On Fri, 10 Feb 2017, Sharp Stone wrote: > Thank you all for the replies. > I'm using Cray XE6M-200, and compilers of gcc/g++/gfortran (by default). > The module list is: > Currently Loaded Modulefiles: > 1) modules/3.2.6.7 > 2) nodestat/2.2-1.0402.39779.3.4.gem > 3) sdb/1.0-1.0402.41597.10.13.gem > 4) MySQL/5.0.64-1.0000.5053.22.1 > 5) > lustre-cray_gem_s/1.8.6_2.6.32.59_0.7.1_1.0402.6998.7.1-1.0402.41312.15.19 > 6) udreg/2.3.2-1.0402.6376.3.1.gem > 7) ugni/5.0-1.0402.6488.5.7.gem > 8) gni-headers/2.1-1.0402.6487.6.4.gem > 9) dmapp/4.0.1-1.0402.6266.5.11.gem > 10) xpmem/0.1-2.0402.39965.2.5.gem > 11) hss-llm/7.0.0 > 12) Base-opts/1.0.2-1.0402.38884.1.2.gem > 13) craype-network-gemini > 14) xt-asyncpe/5.19 > 15) cce/8.1.7 > 16) cray-libsci/12.0.01 > 17) pmi/4.0.1-1.0000.9421.73.3.gem > 18) rca/1.0.0-2.0402.39780.3.94.gem > 19) atp/1.6.2 > 20) PrgEnv-cray/4.2.15 > 21) pbs/11.3.1.122650 > 22) cray-mpich2/5.6.4 > 23) cray-hdf5-parallel/1.8.9 > > Thanks in advance! > > On Fri, Feb 10, 2017 at 2:01 PM, Satish Balay wrote: > > > On Fri, 10 Feb 2017, Patrick Sanan wrote: > > > > > On Fri, Feb 10, 2017 at 7:20 PM, Sharp Stone wrote: > > > > Thanks for the replies. > > > > > > > > The PETSc is installed on the Cray I'm using, but the version is too > > old > > > > cray-petsc/3.3.04. > > > > I've still got two questions: > > > > > > > > 1. how can I use the installed cray-petsc? I've used module load > > cray-petsc, > > > > but I can't find the path of petsc, which is required in the makefile. > > > You can use > > > > > > module show cray-petsc > > > > > > to at least get some more hints as to what cray did, PETSc-wise. > > > > Ok this is old version - and you might want to avoid it. The way to > > use it would be - without petsc makefiles. i.e you would compile with: > > > > cc code.c > > > > > > > > > > > > 2. Therefore, I decided to configure the latest version of Petsc. When > > i use > > > > ./config/configure.py --with-cc=cc --with-cxx=CC --with-fc=ftn, it > > give me > > > > the errors > > > > send configure.log > > > > What type of cray machine is this? Also what compilers are you using? send > > module list > > > > Satish > > > > > > > > ************************************************************ > > ******************* > > > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > > for > > > > details): > > > > ------------------------------------------------------------ > > ------------------- > > > > C libraries cannot directly be used from Fortran > > > > > > > > In the log file, I found these: > > > > ************************************************************ > > ******************* > > > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > > for > > > > details): > > > > ------------------------------------------------------------ > > ------------------- > > > > C libraries cannot directly be used from Fortran > > > > ************************************************************ > > ******************* > > > > File "./config/configure.py", line 405, in petsc_configure > > > > framework.configure(out = sys.stdout) > > > > File > > > > "/mnt/lustre/lus0/space/fs1036/local/PETSc/petsc-3.7. > > 4/config/BuildSystem/config/framework.py", > > > > line 1090, in configure > > > > self.processChildren() > > > > File > > > > "/mnt/lustre/lus0/space/fs1036/local/PETSc/petsc-3.7. > > 4/config/BuildSystem/config/framework.py", > > > > line 1079, in processChildren > > > > self.serialEvaluation(self.childGraph) > > > > File > > > > "/mnt/lustre/lus0/space/fs1036/local/PETSc/petsc-3.7. > > 4/config/BuildSystem/config/framework.py", > > > > line 1060, in serialEvaluation > > > > child.configure() > > > > File > > > > "/mnt/lustre/lus0/space/fs1036/local/PETSc/petsc-3.7. > > 4/config/BuildSystem/config/compilers.py", > > > > line 1438, in configure > > > > self.executeTest(self.checkCLibraries) > > > > File > > > > "/mnt/lustre/lus0/space/fs1036/local/PETSc/petsc-3.7. > > 4/config/BuildSystem/config/base.py", > > > > line 126, in executeTest > > > > ret = test(*args,**kargs) > > > > File > > > > "/mnt/lustre/lus0/space/fs1036/local/PETSc/petsc-3.7. > > 4/config/BuildSystem/config/compilers.py", > > > > line 313, in checkCLibraries > > > > raise RuntimeError('C libraries cannot directly be used from > > Fortran') > > > > ============================================================ > > ==================== > > > > Finishing Configure Run at Fri Feb 10 13:07:10 2017 > > > > ============================================================ > > ==================== > > > > > > > > > > > > Thank you again for your help! > > > > > > > > On Fri, Feb 10, 2017 at 1:01 PM, Satish Balay > > wrote: > > > >> > > > >> check config/examples/arch-cray-xt6-pkgs-opt.py > > > >> > > > >> Also check if your machine already has petsc [from cray] installed > > > >> > > > >> module avail petsc > > > >> > > > >> Satish > > > >> > > > >> On Fri, 10 Feb 2017, Patrick Sanan wrote: > > > >> > > > >> > You're missing the 'g' in 'configure.py'. > > > >> > > > > >> > On Fri, Feb 10, 2017 at 6:51 PM, Sharp Stone > > wrote: > > > >> > > Hi all, > > > >> > > > > > >> > > I noticed the instructions of Petsc with MPI. Now I'm using Cray. > > So > > > >> > > the > > > >> > > instruction: > > > >> > > > > > >> > > Vendor provided MPI might already be installed. IBM, SGI, Cray etc > > > >> > > provide > > > >> > > their own: > > > >> > > ./config/confiure.py --with-cc=mpcc --with-fc=mpf77 > > > >> > > > > > >> > > When I use the command "./config/confiure.py --with-cc=mpcc > > > >> > > --with-fc=mpf77", it gives me the error "-bash: > > ./config/confiure.py: > > > >> > > No > > > >> > > such file or directory". But the file "confiure.py" is there in > > the > > > >> > > "config" > > > >> > > folder. > > > >> > > > > > >> > > What should I do? > > > >> > > > > > >> > > Thanks very much! > > > >> > > > > > >> > > -- > > > >> > > Best regards, > > > >> > > > > > >> > > Feng > > > >> > > > > >> > > > > > > > > > > > > > > > > -- > > > > Best regards, > > > > > > > > Feng > > > > > > > > > > From bhatiamanav at gmail.com Sat Feb 11 11:27:19 2017 From: bhatiamanav at gmail.com (Manav Bhatia) Date: Sat, 11 Feb 2017 11:27:19 -0600 Subject: [petsc-users] ksp solve error with nested matrix In-Reply-To: <6F2742DC-B510-4737-8FA1-4806B13358A7@mcs.anl.gov> References: <98DD59F5-9240-4C42-854C-F3160C021785@mcs.anl.gov> <6F2742DC-B510-4737-8FA1-4806B13358A7@mcs.anl.gov> Message-ID: <5C0890A4-5068-4433-A173-658B4F7A0C42@gmail.com> I am now getting this error, which is complaining for an operation for the shell matrix. I have only specified the following operation for the matrix: ?MATOP_MULT?. I am calling the solver with -pc_type fieldsplit. What can I do to move forward with the solution? Thanks, Manav [1;31m[0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0;39m[0;49m[0]PETSC ERROR: No support for this operation for this object type [0]PETSC ERROR: Mat type shell [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.7.4, Oct, 02, 2016 [0]PETSC ERROR: /Users/manav/Library/Developer/Xcode/DerivedData/MAST-crggwcqrouiyeucduvscdahjauvx/Build/Products/Debug/examples on a arch-darwin-cxx-opt named Blackbird.local by manav Sat Feb 11 11:17:44 2017 [0]PETSC ERROR: Configure options --prefix=/Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/../ --CC=mpicc-openmpi-clang38 --CXX=mpicxx-openmpi-clang38 --FC=mpif90-openmpi-clang38 --with-clanguage=c++ --with-fortran=0 --with-mpiexec=/opt/local/bin/mpiexec-openmpi-clang38 --with-shared-libraries=1 --with-x=1 --with-x-dir=/opt/X11 --with-debugging=0 --with-lapack-lib=/usr/lib/liblapack.dylib --with-blas-lib=/usr/lib/libblas.dylib --download-superlu=yes --download-superlu_dist=yes --download-suitesparse=yes --download-mumps=yes --download-scalapack=yes --download-parmetis=yes --download-metis=yes --download-hypre=yes --download-ml=yes [0]PETSC ERROR: #1 MatMultTransposeAdd() line 2486 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/mat/interface/matrix.c [0]PETSC ERROR: #2 MatMultTranspose_Nest() line 109 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/mat/impls/nest/matnest.c [0]PETSC ERROR: #3 MatMultTranspose() line 2320 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/mat/interface/matrix.c [0]PETSC ERROR: #4 SNESNEWTONLSCheckLocalMin_Private() line 25 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/snes/impls/ls/ls.c [0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 259 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/snes/impls/ls/ls.c [0]PETSC ERROR: #6 SNESSolve() line 4005 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/snes/interface/snes.c > On Feb 9, 2017, at 11:29 AM, Barry Smith wrote: > > >> On Feb 8, 2017, at 11:39 PM, Dave May wrote: >> >> Any time you modify one of the submats, you need to call assembly begin/end on that sub matrix AND on the outer matnest object. > > Weird, and prone to errors it seems to me. Perhaps this needs to be rethought > >> >> Thanks, >> Dave >> >> >> On Wed, 8 Feb 2017 at 22:51, Manav Bhatia wrote: >> aha.. that might be it. >> >> Does that need to be called for the global matrix after each assembly of the Jacobian blocks, or just once for the whole matrix? >> >> -Manav >> >>> On Feb 8, 2017, at 3:47 PM, Barry Smith wrote: >>> >>> >>>> On Feb 8, 2017, at 3:40 PM, Manav Bhatia wrote: >>>> >>>> Hi, >>>> >>>> I have a nested matrix with 2x2 blocks. The blocks (1,1) and (2,2) are AIJ matrices and blocks (1,2) and (2,1) and shell matrices. I am calling the code with the following arguments: -pc_type fieldsplit , and get the error shown below. >>>> >>>> I see that the error is complaining about the matrix being in unassembled state, but I think I am initializing and calling assembly end routines on both the diagonal blocks. Still, there is something obvious I am missing. >>> >>> It is complaining about the outer most matrix, not the blocks. Perhaps you haven't finished with setting up your nest matrix? >>> >>>> >>>> I would appreciate any guidance on this. >>>> >>>> Regards, >>>> Manav >>>> >>>> >>>> >>>> [1;31m[0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>> [0;39m[0;49m[0]PETSC ERROR: Object is in wrong state >>>> [0]PETSC ERROR: Not for unassembled matrix >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>>> [0]PETSC ERROR: Petsc Release Version 3.7.4, Oct, 02, 2016 >>>> [0]PETSC ERROR: /Users/manav/Library/Developer/Xcode/DerivedData/MAST-crggwcqrouiyeucduvscdahjauvx/Build/Products/Debug/examples on a arch-darwin-cxx-opt named Dhcp-90-250.HPC.MsState.Edu by manav Wed Feb 8 15:28:04 2017 >>>> [0]PETSC ERROR: Configure options --prefix=/Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/../ --CC=mpicc-openmpi-clang38 --CXX=mpicxx-openmpi-clang38 --FC=mpif90-openmpi-clang38 --with-clanguage=c++ --with-fortran=0 --with-mpiexec=/opt/local/bin/mpiexec-openmpi-clang38 --with-shared-libraries=1 --with-x=1 --with-x-dir=/opt/X11 --with-debugging=0 --with-lapack-lib=/usr/lib/liblapack.dylib --with-blas-lib=/usr/lib/libblas.dylib --download-superlu=yes --download-superlu_dist=yes --download-suitesparse=yes --download-mumps=yes --download-scalapack=yes --download-parmetis=yes --download-metis=yes --download-hypre=yes --download-ml=yes >>>> [0]PETSC ERROR: #1 MatMult() line 2248 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/mat/interface/matrix.c >>>> [0]PETSC ERROR: #2 PCApplyBAorAB() line 717 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/ksp/pc/interface/precon.c >>>> [0]PETSC ERROR: #3 KSP_PCApplyBAorAB() line 274 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/include/petsc/private/kspimpl.h >>>> [0]PETSC ERROR: #4 KSPGMRESCycle() line 156 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/ksp/ksp/impls/gmres/gmres.c >>>> [0]PETSC ERROR: #5 KSPSolve_GMRES() line 240 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/ksp/ksp/impls/gmres/gmres.c >>>> [0]PETSC ERROR: #6 KSPSolve() line 656 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/ksp/ksp/interface/itfunc.c >>>> [0]PETSC ERROR: #7 SNESSolve_NEWTONLS() line 230 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/snes/impls/ls/ls.c >>>> [0]PETSC ERROR: #8 SNESSolve() line 4005 in /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7.4/src/snes/interface/snes.c >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Feb 11 11:46:16 2017 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 11 Feb 2017 11:46:16 -0600 Subject: [petsc-users] ksp solve error with nested matrix In-Reply-To: <5C0890A4-5068-4433-A173-658B4F7A0C42@gmail.com> References: <98DD59F5-9240-4C42-854C-F3160C021785@mcs.anl.gov> <6F2742DC-B510-4737-8FA1-4806B13358A7@mcs.anl.gov> <5C0890A4-5068-4433-A173-658B4F7A0C42@gmail.com> Message-ID: On Sat, Feb 11, 2017 at 11:27 AM, Manav Bhatia wrote: > I am now getting this error, which is complaining for an operation for the > shell matrix. I have only specified the following operation for the matrix: > ?MATOP_MULT?. > > I am calling the solver with -pc_type fieldsplit. > > What can I do to move forward with the solution? > Okay, this has defeated our check. The MatNest says "yes I have a transpose operation", but that call fails when it gets to your submatrix. I think we might need to do something to push this call down in MatNest. However, right now, just remove that operation from your MatNest MatSetOperation(A, MATOP_TRANSPOSE, NULL); so that it gives the right answer. Thanks, Matt > Thanks, > Manav > > > [1;31m[0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0;39m[0;49m[0]PETSC ERROR: No support for this operation for this object > type > [0]PETSC ERROR: Mat type shell > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.7.4, Oct, 02, 2016 > [0]PETSC ERROR: /Users/manav/Library/Developer/Xcode/DerivedData/MAST- > crggwcqrouiyeucduvscdahjauvx/Build/Products/Debug/examples on a > arch-darwin-cxx-opt named Blackbird.local by manav Sat Feb 11 11:17:44 2017 > [0]PETSC ERROR: Configure options --prefix=/Users/manav/ > Documents/codes/numerical_lib/petsc/petsc-3.7.4/../ > --CC=mpicc-openmpi-clang38 --CXX=mpicxx-openmpi-clang38 > --FC=mpif90-openmpi-clang38 --with-clanguage=c++ --with-fortran=0 > --with-mpiexec=/opt/local/bin/mpiexec-openmpi-clang38 > --with-shared-libraries=1 --with-x=1 --with-x-dir=/opt/X11 > --with-debugging=0 --with-lapack-lib=/usr/lib/liblapack.dylib > --with-blas-lib=/usr/lib/libblas.dylib --download-superlu=yes > --download-superlu_dist=yes --download-suitesparse=yes --download-mumps=yes > --download-scalapack=yes --download-parmetis=yes --download-metis=yes > --download-hypre=yes --download-ml=yes > [0]PETSC ERROR: #1 MatMultTransposeAdd() line 2486 in > /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7. > 4/src/mat/interface/matrix.c > [0]PETSC ERROR: #2 MatMultTranspose_Nest() line 109 in > /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7. > 4/src/mat/impls/nest/matnest.c > [0]PETSC ERROR: #3 MatMultTranspose() line 2320 in > /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7. > 4/src/mat/interface/matrix.c > [0]PETSC ERROR: #4 SNESNEWTONLSCheckLocalMin_Private() line 25 in > /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7. > 4/src/snes/impls/ls/ls.c > [0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 259 in > /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7. > 4/src/snes/impls/ls/ls.c > [0]PETSC ERROR: #6 SNESSolve() line 4005 in /Users/manav/Documents/codes/ > numerical_lib/petsc/petsc-3.7.4/src/snes/interface/snes.c > > > > On Feb 9, 2017, at 11:29 AM, Barry Smith wrote: > > > On Feb 8, 2017, at 11:39 PM, Dave May wrote: > > Any time you modify one of the submats, you need to call assembly > begin/end on that sub matrix AND on the outer matnest object. > > > Weird, and prone to errors it seems to me. Perhaps this needs to be > rethought > > > Thanks, > Dave > > > On Wed, 8 Feb 2017 at 22:51, Manav Bhatia wrote: > aha.. that might be it. > > Does that need to be called for the global matrix after each assembly of > the Jacobian blocks, or just once for the whole matrix? > > -Manav > > On Feb 8, 2017, at 3:47 PM, Barry Smith wrote: > > > On Feb 8, 2017, at 3:40 PM, Manav Bhatia wrote: > > Hi, > > I have a nested matrix with 2x2 blocks. The blocks (1,1) and (2,2) are > AIJ matrices and blocks (1,2) and (2,1) and shell matrices. I am calling > the code with the following arguments: -pc_type fieldsplit , and get the > error shown below. > > I see that the error is complaining about the matrix being in unassembled > state, but I think I am initializing and calling assembly end routines on > both the diagonal blocks. Still, there is something obvious I am missing. > > > It is complaining about the outer most matrix, not the blocks. Perhaps you > haven't finished with setting up your nest matrix? > > > I would appreciate any guidance on this. > > Regards, > Manav > > > > [1;31m[0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0;39m[0;49m[0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: Not for unassembled matrix > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.7.4, Oct, 02, 2016 > [0]PETSC ERROR: /Users/manav/Library/Developer/Xcode/DerivedData/MAST- > crggwcqrouiyeucduvscdahjauvx/Build/Products/Debug/examples on a > arch-darwin-cxx-opt named Dhcp-90-250.HPC.MsState.Edu by manav Wed Feb 8 > 15:28:04 2017 > [0]PETSC ERROR: Configure options --prefix=/Users/manav/ > Documents/codes/numerical_lib/petsc/petsc-3.7.4/../ > --CC=mpicc-openmpi-clang38 --CXX=mpicxx-openmpi-clang38 > --FC=mpif90-openmpi-clang38 --with-clanguage=c++ --with-fortran=0 > --with-mpiexec=/opt/local/bin/mpiexec-openmpi-clang38 > --with-shared-libraries=1 --with-x=1 --with-x-dir=/opt/X11 > --with-debugging=0 --with-lapack-lib=/usr/lib/liblapack.dylib > --with-blas-lib=/usr/lib/libblas.dylib --download-superlu=yes > --download-superlu_dist=yes --download-suitesparse=yes --download-mumps=yes > --download-scalapack=yes --download-parmetis=yes --download-metis=yes > --download-hypre=yes --download-ml=yes > [0]PETSC ERROR: #1 MatMult() line 2248 in /Users/manav/Documents/codes/ > numerical_lib/petsc/petsc-3.7.4/src/mat/interface/matrix.c > [0]PETSC ERROR: #2 PCApplyBAorAB() line 717 in > /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7. > 4/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #3 KSP_PCApplyBAorAB() line 274 in > /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7. > 4/include/petsc/private/kspimpl.h > [0]PETSC ERROR: #4 KSPGMRESCycle() line 156 in > /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7. > 4/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: #5 KSPSolve_GMRES() line 240 in > /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7. > 4/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: #6 KSPSolve() line 656 in /Users/manav/Documents/codes/ > numerical_lib/petsc/petsc-3.7.4/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #7 SNESSolve_NEWTONLS() line 230 in > /Users/manav/Documents/codes/numerical_lib/petsc/petsc-3.7. > 4/src/snes/impls/ls/ls.c > [0]PETSC ERROR: #8 SNESSolve() line 4005 in /Users/manav/Documents/codes/ > numerical_lib/petsc/petsc-3.7.4/src/snes/interface/snes.c > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From gideon.simpson at gmail.com Sun Feb 12 11:30:15 2017 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Sun, 12 Feb 2017 12:30:15 -0500 Subject: [petsc-users] TSSetPostStep for projected methods Message-ID: <7D11E2F5-6711-4E2E-BDDC-F48DE07682F4@gmail.com> I?ve begun working on implementing a projected time stepping method for solving y? = f(y), where g(y(t)) is an invariant of the flow. Per a previous email exchange, it was suggested that I use TSSetPostStep to perform the projection routine, calling TSGetSolution obtain the current iterate, which will be corrected with the projection. However, I noticed two things: 1. The documentation for TSGetSolution says "This vector not changed until the solution at the next timestep has been calculated.? Does that mean that if I make a change to the solution in a PostStep function, it won?t be captured until the next step? What happens at the final time step? 2. The projection is nonlinear, requiring the use of a SNES. I had originally thought that I would create/configure/destroy the SNES within the main routine of the code, and pass this to the PostStep through a user data structure. However, the TSSetPostStep function seems to only take functions which are exclusively functions of the TS. Likewise, I would need to create a residual vector, r, for the SNESSetFunction. Is there a way to pass these data structures, so that they can be reused. or would they have to be created/destroyed within the PostStep function at each iterate? How costly would that be? -gideon -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Feb 12 15:21:15 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 12 Feb 2017 15:21:15 -0600 Subject: [petsc-users] TSSetPostStep for projected methods In-Reply-To: <7D11E2F5-6711-4E2E-BDDC-F48DE07682F4@gmail.com> References: <7D11E2F5-6711-4E2E-BDDC-F48DE07682F4@gmail.com> Message-ID: <2FF79D97-DE0F-40A9-BC4A-9C7D9D05C5FB@mcs.anl.gov> > On Feb 12, 2017, at 11:30 AM, Gideon Simpson wrote: > > I?ve begun working on implementing a projected time stepping method for solving y? = f(y), where g(y(t)) is an invariant of the flow. Per a previous email exchange, it was suggested that I use TSSetPostStep to perform the projection routine, calling TSGetSolution obtain the current iterate, which will be corrected with the projection. However, I noticed two things: > > 1. The documentation for TSGetSolution says "This vector not changed until the solution at the next timestep has been calculated.? Does that mean that if I make a change to the solution in a PostStep function, it won?t be captured until the next step? No. > What happens at the final time step? Your function should still get called. > > 2. The projection is nonlinear, requiring the use of a SNES. I had originally thought that I would create/configure/destroy the SNES within the main routine of the code, and pass this to the PostStep through a user data structure. However, the TSSetPostStep function seems to only take functions which are exclusively functions of the TS. Likewise, I would need to create a residual vector, r, for the SNESSetFunction. Is there a way to pass these data structures, so that they can be reused. or would they have to be created/destroyed within the PostStep function at each iterate? How costly would that be? After you create the SNES and the TS call PetscObjectCompose() to attach the SNES to the TS then inside your post-step you can use PetscObjectQuery() or if you want to pass in more than a SNES use PetscContainerCreate(). > > > -gideon > From ibarletta at inogs.it Mon Feb 13 05:51:58 2017 From: ibarletta at inogs.it (Barletta, Ivano) Date: Mon, 13 Feb 2017 12:51:58 +0100 Subject: [petsc-users] VecScatter betweeen Vectors with different parallel layouts Message-ID: Hello PETSc users I was wondering if you've got any suggestion for the issue mentioned in the topic. I'm solving a linear system on my unstructured grid that has its own partitioning. When I set up the linear system in PETSc I use the model partitioning whose grid points might not be evenly balanced (generally is not) over the processors. So far I use VecScatter and IS to re-order the solution vector with the application numbering. What I aim at, now, is to let PETSc to decide the linear system partitioning with PETSC_DECIDE local dimension and see if I get some gain in performances, but in this case I have to scatter between vectors that, at any rate, have different parallel layouts (one of the requirements of VecScatter is to have 2 Vecs with same layouts) Does anyone of you have ever faced a similar problem? any suggestion about how to circumvent this? Feel free to ask for further information if my explanation is not clear enough Thanks in advance Ivano -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Feb 13 07:03:40 2017 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 13 Feb 2017 07:03:40 -0600 Subject: [petsc-users] TSSetPostStep for projected methods In-Reply-To: <2FF79D97-DE0F-40A9-BC4A-9C7D9D05C5FB@mcs.anl.gov> References: <7D11E2F5-6711-4E2E-BDDC-F48DE07682F4@gmail.com> <2FF79D97-DE0F-40A9-BC4A-9C7D9D05C5FB@mcs.anl.gov> Message-ID: On Sun, Feb 12, 2017 at 3:21 PM, Barry Smith wrote: > > > On Feb 12, 2017, at 11:30 AM, Gideon Simpson > wrote: > > > > I?ve begun working on implementing a projected time stepping method for > solving y? = f(y), where g(y(t)) is an invariant of the flow. Per a > previous email exchange, it was suggested that I use TSSetPostStep to > perform the projection routine, calling TSGetSolution obtain the current > iterate, which will be corrected with the projection. However, I noticed > two things: > > > > 1. The documentation for TSGetSolution says "This vector not changed > until the solution at the next timestep has been calculated.? Does that > mean that if I make a change to the solution in a PostStep function, it > won?t be captured until the next step? > > No. > > > What happens at the final time step? > > Your function should still get called. > > > > > 2. The projection is nonlinear, requiring the use of a SNES. I had > originally thought that I would create/configure/destroy the SNES within > the main routine of the code, and pass this to the PostStep through a user > data structure. However, the TSSetPostStep function seems to only take > functions which are exclusively functions of the TS. Likewise, I would > need to create a residual vector, r, for the SNESSetFunction. Is there a > way to pass these data structures, so that they can be reused. or would > they have to be created/destroyed within the PostStep function at each > iterate? How costly would that be? > > After you create the SNES and the TS call PetscObjectCompose() to > attach the SNES to the TS then inside your post-step you can use > PetscObjectQuery() or if you want to pass in more than a SNES use > PetscContainerCreate(). > An alternative to what Barry suggests, which will definitely work, is to use the DM. DMCreateShell(comm, &dm); DMSetApplicationContext(dm, ctx); TSSetDM(ts, dm); In your function TSGetDM(ts, &dm); DMGetApplicationContext(dm, &ctx); I prefer this to the Container route. Thanks, Matt > > > > > > -gideon > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Feb 13 07:18:02 2017 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 13 Feb 2017 07:18:02 -0600 Subject: [petsc-users] VecScatter betweeen Vectors with different parallel layouts In-Reply-To: References: Message-ID: On Mon, Feb 13, 2017 at 5:51 AM, Barletta, Ivano wrote: > Hello PETSc users > > I was wondering if you've got any suggestion for the issue > mentioned in the topic. > > I'm solving a linear system on my unstructured grid that has its > own partitioning. When I set up the linear system in PETSc I > use the model partitioning whose grid points might not be > evenly balanced (generally is not) over the processors. > > So far I use VecScatter and IS to re-order the solution vector > with the application numbering. > > What I aim at, now, is to let PETSc to decide the linear system > partitioning with PETSC_DECIDE local dimension and > see if I get some gain in performances, but in > this case I have to scatter between vectors that, at any rate, have > different parallel layouts (one of the requirements of > VecScatter is to have 2 Vecs with same layouts) > That is exactly what VecScatter does. It does not require the same layout. Example: proc 0 proc1 Vec A 0 1 2 3 Vec B 0 1 2 3 You create a VecScatter A --> B with indices proc 0 proc 1 from 0 1 2 3 to 3 2 1 0 and it will reverse the vector into the new layout. Matt > Does anyone of you have ever faced a similar problem? > any suggestion about how to circumvent this? > > Feel free to ask for further information if my explanation is > not clear enough > > Thanks in advance > Ivano > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Mon Feb 13 09:22:52 2017 From: jed at jedbrown.org (Jed Brown) Date: Mon, 13 Feb 2017 16:22:52 +0100 Subject: [petsc-users] VecScatter betweeen Vectors with different parallel layouts In-Reply-To: References: Message-ID: <87d1emjesz.fsf@jedbrown.org> "Barletta, Ivano" writes: > Hello PETSc users > > I was wondering if you've got any suggestion for the issue > mentioned in the topic. > > I'm solving a linear system on my unstructured grid that has its > own partitioning. When I set up the linear system in PETSc I > use the model partitioning whose grid points might not be > evenly balanced (generally is not) over the processors. > > So far I use VecScatter and IS to re-order the solution vector > with the application numbering. Mapping between two orderings with different distributions is usually not a good strategy. Much better to balance the partitioning for your unstructured grid. > What I aim at, now, is to let PETSc to decide the linear system > partitioning with PETSC_DECIDE local dimension and > see if I get some gain in performances, but in > this case I have to scatter between vectors that, at any rate, have > different parallel layouts (one of the requirements of > VecScatter is to have 2 Vecs with same layouts) PETSC_DECIDE just creates equal vertex partitions, but it has no information with which to make the partitions have locality. Unless you have a particular ordering, like a space-filling curve, this kind of simplistic partitioning by vertex index will tend to produce parts that have very poor locality (i.e., very high edge cut). > Does anyone of you have ever faced a similar problem? > any suggestion about how to circumvent this? > > Feel free to ask for further information if my explanation is > not clear enough > > Thanks in advance > Ivano -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From gideon.simpson at gmail.com Mon Feb 13 11:54:21 2017 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Mon, 13 Feb 2017 12:54:21 -0500 Subject: [petsc-users] TSSetPostStep for projected methods In-Reply-To: References: <7D11E2F5-6711-4E2E-BDDC-F48DE07682F4@gmail.com> <2FF79D97-DE0F-40A9-BC4A-9C7D9D05C5FB@mcs.anl.gov> Message-ID: <4CA7A177-C29F-4D2F-80DA-1DCED874EBA4@gmail.com> Yup, implemented that with DM. Works as desired. Thanks. -gideon > On Feb 13, 2017, at 8:03 AM, Matthew Knepley wrote: > > On Sun, Feb 12, 2017 at 3:21 PM, Barry Smith > wrote: > > > On Feb 12, 2017, at 11:30 AM, Gideon Simpson > wrote: > > > > I?ve begun working on implementing a projected time stepping method for solving y? = f(y), where g(y(t)) is an invariant of the flow. Per a previous email exchange, it was suggested that I use TSSetPostStep to perform the projection routine, calling TSGetSolution obtain the current iterate, which will be corrected with the projection. However, I noticed two things: > > > > 1. The documentation for TSGetSolution says "This vector not changed until the solution at the next timestep has been calculated.? Does that mean that if I make a change to the solution in a PostStep function, it won?t be captured until the next step? > > No. > > > What happens at the final time step? > > Your function should still get called. > > > > > 2. The projection is nonlinear, requiring the use of a SNES. I had originally thought that I would create/configure/destroy the SNES within the main routine of the code, and pass this to the PostStep through a user data structure. However, the TSSetPostStep function seems to only take functions which are exclusively functions of the TS. Likewise, I would need to create a residual vector, r, for the SNESSetFunction. Is there a way to pass these data structures, so that they can be reused. or would they have to be created/destroyed within the PostStep function at each iterate? How costly would that be? > > After you create the SNES and the TS call PetscObjectCompose() to attach the SNES to the TS then inside your post-step you can use PetscObjectQuery() or if you want to pass in more than a SNES use PetscContainerCreate(). > > An alternative to what Barry suggests, which will definitely work, is to use the DM. > > DMCreateShell(comm, &dm); > DMSetApplicationContext(dm, ctx); > TSSetDM(ts, dm); > > In your function > > TSGetDM(ts, &dm); > DMGetApplicationContext(dm, &ctx); > > I prefer this to the Container route. > > Thanks, > > Matt > > > > > > > -gideon > > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From gideon.simpson at gmail.com Mon Feb 13 11:55:38 2017 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Mon, 13 Feb 2017 12:55:38 -0500 Subject: [petsc-users] TS routine calling order Message-ID: <5317C943-C210-46C3-9531-4DEE527AA04B@gmail.com> In terms of the TS PreStep, PostStep, Monitor, etc., what is the calling order? I?m specifically curious as to whether the Monitor routine is run before or after the PostStep, but I?d be interested to know what the whole workflow is. -gideon -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Mon Feb 13 12:20:48 2017 From: jed at jedbrown.org (Jed Brown) Date: Mon, 13 Feb 2017 11:20:48 -0700 Subject: [petsc-users] TS routine calling order In-Reply-To: <5317C943-C210-46C3-9531-4DEE527AA04B@gmail.com> References: <5317C943-C210-46C3-9531-4DEE527AA04B@gmail.com> Message-ID: <87o9y66jgf.fsf@jedbrown.org> Gideon Simpson writes: > In terms of the TS PreStep, PostStep, Monitor, etc., what is the calling order? I?m specifically curious as to whether the Monitor routine is run before or after the PostStep, but I?d be interested to know what the whole workflow is. Look at TSSolve: while (!ts->reason) { ierr = TSMonitor(ts,ts->total_steps,ts->ptime,ts->vec_sol);CHKERRQ(ierr); if (!ts->steprollback) { ierr = TSPreStep(ts);CHKERRQ(ierr); } ierr = TSStep(ts);CHKERRQ(ierr); if (ts->vec_costintegral && ts->costintegralfwd) { /* Must evaluate the cost integral before event is handled. The cost integral value can also be rolled back. */ ierr = TSForwardCostIntegral(ts);CHKERRQ(ierr); } ierr = TSPostEvaluate(ts);CHKERRQ(ierr); ierr = TSEventHandler(ts);CHKERRQ(ierr); /* The right-hand side may be changed due to event. Be careful with Any computation using the RHS information after this point. */ if (!ts->steprollback) { ierr = TSTrajectorySet(ts->trajectory,ts,ts->total_steps,ts->ptime,ts->vec_sol);CHKERRQ(ierr); ierr = TSPostStep(ts);CHKERRQ(ierr); } } ierr = TSMonitor(ts,ts->total_steps,ts->ptime,ts->vec_sol);CHKERRQ(ierr); -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From elbueler at alaska.edu Mon Feb 13 13:53:14 2017 From: elbueler at alaska.edu (Ed Bueler) Date: Mon, 13 Feb 2017 10:53:14 -0900 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? Message-ID: Dear Petsc -- This is the first of two short TS usage questions. My problem is both stiff (diffusive PDE) and constrained, so I require -snes_type vinewton{rs|ss}ls *and* I split my ODE system into IFunction and RHSFunction F(t,u,u_t) = G(t,u) where F(t,u,u_t) = u_t + f(t,u) in my case (i.e. no mass matrix needed), and the stiff part goes in f(t,u). With this arrangement TS types beuler, theta, bdf, arkimex all work quite well. However, the program runs and produces nonsense with type rk and ssp, that is, explicit methods. So my question is, how do I ask the TS (at run time) whether the chosen TS type will or will not call its SNES at each step? If SNES is not going to be used then I want to SETERRQ and stop. That is, I want to error-out if the *method* is fully explicit. Note the constraints are enforced by the SNESVI, as they should be, not ad hoc projection. Also, as a technical matter, I cannot require my iterates to be feasible inside my IFunction evaluation because that would break VINEWTONSSLS. Neither TSProblemType (mine is NONLINEAR) nor TSEquationType (mine is IMPLICIT I guess) seem to address this? My problem is indeed nonlinear and has stiff parts, but it is not a DAE and it *is* constrained. Thanks! Ed PS I'd prefer not to enumerate the existing TS types and error on the bad ones. It is not nicely-maintainable. -- Ed Bueler Dept of Math and Stat and Geophysical Institute University of Alaska Fairbanks Fairbanks, AK 99775-6660 301C Chapman and 410D Elvey 907 474-7693 and 907 474-7199 (fax 907 474-5394) -------------- next part -------------- An HTML attachment was scrubbed... URL: From elbueler at alaska.edu Mon Feb 13 14:01:28 2017 From: elbueler at alaska.edu (Ed Bueler) Date: Mon, 13 Feb 2017 11:01:28 -0900 Subject: [petsc-users] TS question 2: why not -ts_adapt_wnormtype 1 ? Message-ID: Dear Petsc -- This is the second of two short TS usage questions. I would like my TS, for MOL on a PDE based on a spatial grid (DMDA) of course, to be tolerant of crap at a few locations. Thus I set -ts_adapt_wnormtype 1 and I got a very clear error saying "Only 2-norm and infinite norm supported". I was surprised because I figured if you could implement one (vector) norm you could do any. Why? If there is a good reason, *which I expect*, is there a citation on the adaptive fundamentals that would explain this surprise? Thanks! Ed -- Ed Bueler Dept of Math and Stat and Geophysical Institute University of Alaska Fairbanks Fairbanks, AK 99775-6660 301C Chapman and 410D Elvey 907 474-7693 and 907 474-7199 (fax 907 474-5394) -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Feb 13 14:26:26 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 13 Feb 2017 14:26:26 -0600 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: References: Message-ID: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> > On Feb 13, 2017, at 1:53 PM, Ed Bueler wrote: > > Dear Petsc -- > > This is the first of two short TS usage questions. > > My problem is both stiff (diffusive PDE) and constrained, so I require > > -snes_type vinewton{rs|ss}ls > > *and* I split my ODE system into IFunction and RHSFunction > > F(t,u,u_t) = G(t,u) > > where F(t,u,u_t) = u_t + f(t,u) in my case (i.e. no mass matrix needed), and the stiff part goes in f(t,u). > > With this arrangement TS types beuler, theta, bdf, arkimex all work quite well. However, the program runs and produces nonsense with type rk and ssp, that is, explicit methods. Sounds like a bug to me. The methods should be checking if an IFunction is being provided and error out in that case. Barry > > So my question is, how do I ask the TS (at run time) whether the chosen TS type will or will not call its SNES at each step? If SNES is not going to be used then I want to SETERRQ and stop. That is, I want to error-out if the *method* is fully explicit. > > Note the constraints are enforced by the SNESVI, as they should be, not ad hoc projection. Also, as a technical matter, I cannot require my iterates to be feasible inside my IFunction evaluation because that would break VINEWTONSSLS. > > Neither TSProblemType (mine is NONLINEAR) nor TSEquationType (mine is IMPLICIT I guess) seem to address this? My problem is indeed nonlinear and has stiff parts, but it is not a DAE and it *is* constrained. > > Thanks! > > Ed > > PS I'd prefer not to enumerate the existing TS types and error on the bad ones. It is not nicely-maintainable. > > > > -- > Ed Bueler > Dept of Math and Stat and Geophysical Institute > University of Alaska Fairbanks > Fairbanks, AK 99775-6660 > 301C Chapman and 410D Elvey > 907 474-7693 and 907 474-7199 (fax 907 474-5394) From jed at jedbrown.org Mon Feb 13 14:30:51 2017 From: jed at jedbrown.org (Jed Brown) Date: Mon, 13 Feb 2017 13:30:51 -0700 Subject: [petsc-users] TS question 2: why not -ts_adapt_wnormtype 1 ? In-Reply-To: References: Message-ID: <877f4t7s04.fsf@jedbrown.org> I think it just wasn't implemented. Note that these have custom code because of the weighting vectors -- it isn't just a call to VecNorm. Patches welcome. Ed Bueler writes: > Dear Petsc -- > > This is the second of two short TS usage questions. > > I would like my TS, for MOL on a PDE based on a spatial grid (DMDA) of > course, to be tolerant of crap at a few locations. Thus I set > > -ts_adapt_wnormtype 1 > > and I got a very clear error saying "Only 2-norm and infinite norm > supported". I was surprised because I figured if you could implement one > (vector) norm you could do any. > > Why? If there is a good reason, *which I expect*, is there a citation on > the adaptive fundamentals that would explain this surprise? > > Thanks! > > Ed > > -- > Ed Bueler > Dept of Math and Stat and Geophysical Institute > University of Alaska Fairbanks > Fairbanks, AK 99775-6660 > 301C Chapman and 410D Elvey > 907 474-7693 and 907 474-7199 (fax 907 474-5394) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From elbueler at alaska.edu Mon Feb 13 15:08:02 2017 From: elbueler at alaska.edu (Ed Bueler) Date: Mon, 13 Feb 2017 12:08:02 -0900 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> Message-ID: Barry -- > Sounds like a bug to me. The methods should be checking if an > IFunction is being provided and error out in that case. I don't think it is that simple. I.e. having an IFunction and an explicit scheme is not, by itself, a bug. I think. If the user has provided IFunction F(t,u,u_t) = u_t - f(t,u) which is the convenient form for e.g. TSARKIMEX, and RHSFunction G(t,u) then I guess I assumed that explicit methods like TSRK should,for unconstrained cases, get their RHS by callback like this: g(t,u) = - F(t,u,0) + G(t,u) so that the ODE is in form u_t = g(t,u) = f(t,u) + g(t,u) I think that is the behavior I am seeing (i.e. on my problem, using PETSc master). The "nonsense" I am referring to is only from the non-enforcement of the constraint; it would be o.k. for a pure ODE. I would love to have some projection mechanism to try, for comparing explicit methods with projection to the SNESVI way (i.e. the right way), but that is asking for a lot of PETSc refactoring, I think. For now I just want to error-out if the method is not going to call the SNES at each time step. Ed On Mon, Feb 13, 2017 at 11:26 AM, Barry Smith wrote: > > > On Feb 13, 2017, at 1:53 PM, Ed Bueler wrote: > > > > Dear Petsc -- > > > > This is the first of two short TS usage questions. > > > > My problem is both stiff (diffusive PDE) and constrained, so I require > > > > -snes_type vinewton{rs|ss}ls > > > > *and* I split my ODE system into IFunction and RHSFunction > > > > F(t,u,u_t) = G(t,u) > > > > where F(t,u,u_t) = u_t + f(t,u) in my case (i.e. no mass matrix needed), > and the stiff part goes in f(t,u). > > > > With this arrangement TS types beuler, theta, bdf, arkimex all work > quite well. However, the program runs and produces nonsense with type rk > and ssp, that is, explicit methods. > > Sounds like a bug to me. The methods should be checking if an IFunction > is being provided and error out in that case. > > Barry > > > > > So my question is, how do I ask the TS (at run time) whether the chosen > TS type will or will not call its SNES at each step? If SNES is not going > to be used then I want to SETERRQ and stop. That is, I want to error-out > if the *method* is fully explicit. > > > > Note the constraints are enforced by the SNESVI, as they should be, not > ad hoc projection. Also, as a technical matter, I cannot require my > iterates to be feasible inside my IFunction evaluation because that would > break VINEWTONSSLS. > > > > Neither TSProblemType (mine is NONLINEAR) nor TSEquationType (mine is > IMPLICIT I guess) seem to address this? My problem is indeed nonlinear and > has stiff parts, but it is not a DAE and it *is* constrained. > > > > Thanks! > > > > Ed > > > > PS I'd prefer not to enumerate the existing TS types and error on the > bad ones. It is not nicely-maintainable. > > > > > > > > -- > > Ed Bueler > > Dept of Math and Stat and Geophysical Institute > > University of Alaska Fairbanks > > Fairbanks, AK 99775-6660 > > 301C Chapman and 410D Elvey > > 907 474-7693 and 907 474-7199 (fax 907 474-5394) > > -- Ed Bueler Dept of Math and Stat and Geophysical Institute University of Alaska Fairbanks Fairbanks, AK 99775-6660 301C Chapman and 410D Elvey 907 474-7693 and 907 474-7199 (fax 907 474-5394) -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Feb 13 15:59:23 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 13 Feb 2017 15:59:23 -0600 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> Message-ID: > On Feb 13, 2017, at 3:08 PM, Ed Bueler wrote: > > Barry -- > > > Sounds like a bug to me. The methods should be checking if an > > IFunction is being provided and error out in that case. > > I don't think it is that simple. I.e. having an IFunction and an explicit scheme is not, by itself, a bug. I think. > > If the user has provided IFunction > F(t,u,u_t) = u_t - f(t,u) > which is the convenient form for e.g. TSARKIMEX, > and RHSFunction > G(t,u) > then I guess I assumed that explicit methods like TSRK should,for unconstrained cases, get their RHS by callback like this: > g(t,u) = - F(t,u,0) + G(t,u) > so that the ODE is in form > u_t = g(t,u) = f(t,u) + g(t,u) I guess in theory TSRK could do this but looking at the code static PetscErrorCode TSStep_RK(TS ts) { TS_RK *rk = (TS_RK*)ts->data; RKTableau tab = rk->tableau; const PetscInt s = tab->s; const PetscReal *A = tab->A,*c = tab->c; PetscScalar *w = rk->work; Vec *Y = rk->Y,*YdotRHS = rk->YdotRHS; TSAdapt adapt; PetscInt i,j; PetscInt rejections = 0; PetscBool stageok,accept = PETSC_TRUE; PetscReal next_time_step = ts->time_step; PetscErrorCode ierr; PetscFunctionBegin; rk->status = TS_STEP_INCOMPLETE; while (!ts->reason && rk->status != TS_STEP_COMPLETE) { PetscReal t = ts->ptime; PetscReal h = ts->time_step; for (i=0; istage_time = t + h*c[i]; ierr = TSPreStage(ts,rk->stage_time); CHKERRQ(ierr); ierr = VecCopy(ts->vec_sol,Y[i]);CHKERRQ(ierr); for (j=0; jstage_time,i,Y); CHKERRQ(ierr); ierr = TSGetAdapt(ts,&adapt);CHKERRQ(ierr); ierr = TSAdaptCheckStage(adapt,ts,rk->stage_time,Y[i],&stageok);CHKERRQ(ierr); if (!stageok) goto reject_step; ierr = TSComputeRHSFunction(ts,t+h*c[i],Y[i],YdotRHS[i]);CHKERRQ(ierr); etscErrorCode TSComputeRHSFunction(TS ts,PetscReal t,Vec U,Vec y) { PetscErrorCode ierr; TSRHSFunction rhsfunction; TSIFunction ifunction; void *ctx; DM dm; PetscFunctionBegin; PetscValidHeaderSpecific(ts,TS_CLASSID,1); PetscValidHeaderSpecific(U,VEC_CLASSID,3); PetscValidHeaderSpecific(y,VEC_CLASSID,4); ierr = TSGetDM(ts,&dm);CHKERRQ(ierr); ierr = DMTSGetRHSFunction(dm,&rhsfunction,&ctx);CHKERRQ(ierr); ierr = DMTSGetIFunction(dm,&ifunction,NULL);CHKERRQ(ierr); if (!rhsfunction && !ifunction) SETERRQ(PetscObjectComm((PetscObject)ts),PETSC_ERR_USER,"Must call TSSetRHSFunction() and / or TSSetIFunction()"); ierr = PetscLogEventBegin(TS_FunctionEval,ts,U,y,0);CHKERRQ(ierr); if (rhsfunction) { PetscStackPush("TS user right-hand-side function"); ierr = (*rhsfunction)(ts,t,U,y,ctx);CHKERRQ(ierr); PetscStackPop; } else { ierr = VecZeroEntries(y);CHKERRQ(ierr); } ierr = PetscLogEventEnd(TS_FunctionEval,ts,U,y,0);CHKERRQ(ie it doesn't do that. It simply ignores the ifunction the user provided. We should not do that since it will produce the wrong answer. Regarding the "poor man's way" of handling the constraint with projection methods. Our intention is that you would use TSSetPostStep() to provide a function that takes the computed solution by TSRK and "projects" it as you wish. You can try this now but note you will need to pack your ifunction() (with the u_t) and your rhsfunction() into a single right hand side function*. Barry *Yes, eventually we should support the user not having to provide the function in a different form but we don't have anybody working on "useful and needed improvements to TS" we only have people working on "publishable and fundable cool new stuff" with TS. > > I think that is the behavior I am seeing (i.e. on my problem, using PETSc master). > > The "nonsense" I am referring to is only from the non-enforcement of the constraint; it would be o.k. for a pure ODE. > > I would love to have some projection mechanism to try, for comparing explicit methods with projection to the SNESVI way (i.e. the right way), but that is asking for a lot of PETSc refactoring, I think. For now I just want to error-out if the method is not going to call the SNES at each time step. > > Ed > > > > On Mon, Feb 13, 2017 at 11:26 AM, Barry Smith wrote: > > > On Feb 13, 2017, at 1:53 PM, Ed Bueler wrote: > > > > Dear Petsc -- > > > > This is the first of two short TS usage questions. > > > > My problem is both stiff (diffusive PDE) and constrained, so I require > > > > -snes_type vinewton{rs|ss}ls > > > > *and* I split my ODE system into IFunction and RHSFunction > > > > F(t,u,u_t) = G(t,u) > > > > where F(t,u,u_t) = u_t + f(t,u) in my case (i.e. no mass matrix needed), and the stiff part goes in f(t,u). > > > > With this arrangement TS types beuler, theta, bdf, arkimex all work quite well. However, the program runs and produces nonsense with type rk and ssp, that is, explicit methods. > > Sounds like a bug to me. The methods should be checking if an IFunction is being provided and error out in that case. > > Barry > > > > > So my question is, how do I ask the TS (at run time) whether the chosen TS type will or will not call its SNES at each step? If SNES is not going to be used then I want to SETERRQ and stop. That is, I want to error-out if the *method* is fully explicit. > > > > Note the constraints are enforced by the SNESVI, as they should be, not ad hoc projection. Also, as a technical matter, I cannot require my iterates to be feasible inside my IFunction evaluation because that would break VINEWTONSSLS. > > > > Neither TSProblemType (mine is NONLINEAR) nor TSEquationType (mine is IMPLICIT I guess) seem to address this? My problem is indeed nonlinear and has stiff parts, but it is not a DAE and it *is* constrained. > > > > Thanks! > > > > Ed > > > > PS I'd prefer not to enumerate the existing TS types and error on the bad ones. It is not nicely-maintainable. > > > > > > > > -- > > Ed Bueler > > Dept of Math and Stat and Geophysical Institute > > University of Alaska Fairbanks > > Fairbanks, AK 99775-6660 > > 301C Chapman and 410D Elvey > > 907 474-7693 and 907 474-7199 (fax 907 474-5394) > > > > > -- > Ed Bueler > Dept of Math and Stat and Geophysical Institute > University of Alaska Fairbanks > Fairbanks, AK 99775-6660 > 301C Chapman and 410D Elvey > 907 474-7693 and 907 474-7199 (fax 907 474-5394) From hongzhang at anl.gov Mon Feb 13 15:59:50 2017 From: hongzhang at anl.gov (Zhang, Hong) Date: Mon, 13 Feb 2017 21:59:50 +0000 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> Message-ID: <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> On Feb 13, 2017, at 3:08 PM, Ed Bueler > wrote: Barry -- > Sounds like a bug to me. The methods should be checking if an > IFunction is being provided and error out in that case. I don't think it is that simple. I.e. having an IFunction and an explicit scheme is not, by itself, a bug. I think. If the user has provided IFunction F(t,u,u_t) = u_t - f(t,u) which is the convenient form for e.g. TSARKIMEX, and RHSFunction G(t,u) then I guess I assumed that explicit methods like TSRK should,for unconstrained cases, get their RHS by callback like this: g(t,u) = - F(t,u,0) + G(t,u) so that the ODE is in form u_t = g(t,u) = f(t,u) + g(t,u) PETSc does not transform IFunction to the RHS this way, and PETSc is not supposed to do it automatically. Note that IFunction may involve a mass matrix, e.g. F(t,u,u_t) = M*u_t - f(t,u) and sometimes M is not invertible. I think that is the behavior I am seeing (i.e. on my problem, using PETSc master). Explicit methods use only RHSFunction and ignore IFunction, so in your case, if you change TS type to rk and ssp at run time, you are actually solving u_t = G(t,u). If RHSFunction is not provided, PETSc will assume the RHS is zero (u_t=0). The "nonsense" I am referring to is only from the non-enforcement of the constraint; it would be o.k. for a pure ODE. I would love to have some projection mechanism to try, for comparing explicit methods with projection to the SNESVI way (i.e. the right way), but that is asking for a lot of PETSc refactoring, I think. For now I just want to error-out if the method is not going to call the SNES at each time step. Check if TSGetSNES() returns NULL in your code? Ed On Mon, Feb 13, 2017 at 11:26 AM, Barry Smith > wrote: > On Feb 13, 2017, at 1:53 PM, Ed Bueler > wrote: > > Dear Petsc -- > > This is the first of two short TS usage questions. > > My problem is both stiff (diffusive PDE) and constrained, so I require > > -snes_type vinewton{rs|ss}ls > > *and* I split my ODE system into IFunction and RHSFunction > > F(t,u,u_t) = G(t,u) > > where F(t,u,u_t) = u_t + f(t,u) in my case (i.e. no mass matrix needed), and the stiff part goes in f(t,u). > > With this arrangement TS types beuler, theta, bdf, arkimex all work quite well. However, the program runs and produces nonsense with type rk and ssp, that is, explicit methods. Sounds like a bug to me. The methods should be checking if an IFunction is being provided and error out in that case. Barry > > So my question is, how do I ask the TS (at run time) whether the chosen TS type will or will not call its SNES at each step? If SNES is not going to be used then I want to SETERRQ and stop. That is, I want to error-out if the *method* is fully explicit. > > Note the constraints are enforced by the SNESVI, as they should be, not ad hoc projection. Also, as a technical matter, I cannot require my iterates to be feasible inside my IFunction evaluation because that would break VINEWTONSSLS. > > Neither TSProblemType (mine is NONLINEAR) nor TSEquationType (mine is IMPLICIT I guess) seem to address this? My problem is indeed nonlinear and has stiff parts, but it is not a DAE and it *is* constrained. > > Thanks! > > Ed > > PS I'd prefer not to enumerate the existing TS types and error on the bad ones. It is not nicely-maintainable. > > > > -- > Ed Bueler > Dept of Math and Stat and Geophysical Institute > University of Alaska Fairbanks > Fairbanks, AK 99775-6660 > 301C Chapman and 410D Elvey > 907 474-7693 and 907 474-7199 (fax 907 474-5394) -- Ed Bueler Dept of Math and Stat and Geophysical Institute University of Alaska Fairbanks Fairbanks, AK 99775-6660 301C Chapman and 410D Elvey 907 474-7693 and 907 474-7199 (fax 907 474-5394) -------------- next part -------------- An HTML attachment was scrubbed... URL: From fande.kong at inl.gov Mon Feb 13 16:33:15 2017 From: fande.kong at inl.gov (Kong, Fande) Date: Mon, 13 Feb 2017 15:33:15 -0700 Subject: [petsc-users] lock matrices in EPS Message-ID: Hi ALL, I am solving a generalized eigenvalue problem Ax=\lambda Bx. I want to retrieve matrices A and B from EPS, but got the following errors: [0]PETSC ERROR: No support for this operation for this object type [0]PETSC ERROR: Cannot retrieve original matrices (have been modified) [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.7.5, unknown [0]PETSC ERROR: ./ex14 on a arch-darwin-c-debug named FN604208 by kongf Mon Feb 13 15:29:33 2017 [0]PETSC ERROR: Configure options --with-clanguage=cxx --with-shared-libraries=1 --with-blaslapack=1 --with-mpi=1 --download-parmetis=1 --download-metis=1 --with-debugging=yes --with-c2html=0 --download-hypre=1 --download-superlu_dist=1 PETSC_ARCH=arch-darwin-c-debug [0]PETSC ERROR: #1 STGetOperators() line 308 in /Users/kongf/projects/slepc/src/sys/classes/st/interface/stfunc.c [0]PETSC ERROR: #2 EPSGetOperators() line 319 in /Users/kongf/projects/slepc/src/eps/interface/epssetup.c My question is how to lock the original matrices I passed to EPS. "-st_matmode copy" does not help. Fande Kong, -------------- next part -------------- An HTML attachment was scrubbed... URL: From damon at ices.utexas.edu Mon Feb 13 20:31:24 2017 From: damon at ices.utexas.edu (Damon McDougall) Date: Mon, 13 Feb 2017 20:31:24 -0600 Subject: [petsc-users] Scientific Software Days Conference, April, 2017 Message-ID: <1487039484.1569929.880110864.28012F4C@webmail.messagingengine.com> Apologies if this is not the appropriate place. I thought folks here would be interested in this. Registration is open for the 8th Annual Scientific Software Days conference. Register here: http://scisoftdays.org/ The 8th Annual Scientific Software Days Conference (SSD) targets users and developers of scientific software. The conference will be held at the University of Texas at Austin Thursday Apr 27 - Friday Apr 28, 2017 and focuses on two themes: a) sharing best practices across scientific software communities; b) sharing the latest tools and technology relevant to scientific software. This year's list of speakers is still growing. So far these speakers have confirmed: Beatrice Riviere (keynote, Rice) Mark Hoemmen (keynote, Sandia National Laboratories) Sherry Li (Lawrence Berkeley National Laboratory) Wolfgang Bangerth (Colorado State University) Dan Negrut (University of Wisconsin-Madison) Questions can be directed to ssd-organizers at googlegroups.com. Limited travel funding is available for students and early career researchers. Register online here: http://scisoftdays.org/ -- Damon McDougall http://dmcdougall.co.uk Institute for Computational Engineering Sciences 201 E. 24th St., Stop C0200 The University of Texas at Austin Austin, TX 78712-1229 From jroman at dsic.upv.es Tue Feb 14 04:09:17 2017 From: jroman at dsic.upv.es (Jose E. Roman) Date: Tue, 14 Feb 2017 11:09:17 +0100 Subject: [petsc-users] lock matrices in EPS In-Reply-To: References: Message-ID: We cannot reproduce this problem. Can you provide more details? For instance, a sequence of calls to SLEPc in your code. Are you modifying A and B after EPSSolve()? Jose > El 13 feb 2017, a las 23:33, Kong, Fande escribi?: > > Hi ALL, > > I am solving a generalized eigenvalue problem Ax=\lambda Bx. I want to retrieve matrices A and B from EPS, but got the following errors: > > [0]PETSC ERROR: No support for this operation for this object type > [0]PETSC ERROR: Cannot retrieve original matrices (have been modified) > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.7.5, unknown > [0]PETSC ERROR: ./ex14 on a arch-darwin-c-debug named FN604208 by kongf Mon Feb 13 15:29:33 2017 > [0]PETSC ERROR: Configure options --with-clanguage=cxx --with-shared-libraries=1 --with-blaslapack=1 --with-mpi=1 --download-parmetis=1 --download-metis=1 --with-debugging=yes --with-c2html=0 --download-hypre=1 --download-superlu_dist=1 PETSC_ARCH=arch-darwin-c-debug > [0]PETSC ERROR: #1 STGetOperators() line 308 in /Users/kongf/projects/slepc/src/sys/classes/st/interface/stfunc.c > [0]PETSC ERROR: #2 EPSGetOperators() line 319 in /Users/kongf/projects/slepc/src/eps/interface/epssetup.c > > > My question is how to lock the original matrices I passed to EPS. "-st_matmode copy" does not help. > > Fande Kong, > From andrewh0 at uw.edu Tue Feb 14 06:41:01 2017 From: andrewh0 at uw.edu (Andrew Ho) Date: Tue, 14 Feb 2017 04:41:01 -0800 Subject: [petsc-users] PETSc set off-diagonal matrix pre-allocation Message-ID: I have a 3x6 matrix, and I want to set it to (just as an example): 0 0 0 1 1 1 0 0 0 1 1 1 0 0 0 1 1 1 >From what I can tell, according to the documentation the MPIAIJ sparsity of this matrix is: d_nnz = [0, 0, 0] o_nnz = [3, 3, 3] However, when I do this, I get the following error: [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [0]PETSC ERROR: Argument out of range > [0]PETSC ERROR: New nonzero at (0,3) caused a malloc > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn > off this check > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.7.5, unknown > [0]PETSC ERROR: ./ex3 on a arch-linux2-c-opt Tue Feb 14 04:23:56 2017 > [0]PETSC ERROR: Configure options --with-debugging=0 --COPTFLAGS="-O3 > -march=native" --CXXOPTFLAGS="-O3 -march=native" --FOPTFLAGS="-O3 > -march=native" > [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 582 in > petsc/src/mat/impls/aij/mpi/mpiaij.c > [0]PETSC ERROR: #2 MatSetValues() line 1190 in > petsc/src/mat/interface/matrix.c > [0]PETSC ERROR: #3 main() line 36 in ex3.c > [0]PETSC ERROR: No PETSc Option Table entries > [0]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint at mcs.anl.gov---------- Here's a working test code: #include > #include > int main(int argc, char** argv) > { > PetscErrorCode err; > err = PetscInitialize(&argc, &argv, NULL, "help"); > CHKERRQ(err); > // create a sparse AIJ matrix distributed across MPI > PetscInt global_width = 6; > PetscInt global_height = 3; > Mat A; > err = MatCreate(MPI_COMM_WORLD, &A); > CHKERRQ(err); > err = MatSetType(A, MATMPIAIJ); > CHKERRQ(err); > // setup pre-allocation for matrix space > { > err = > MatSetSizes(A, global_height, PETSC_DECIDE, global_height, > global_width); > CHKERRQ(err); > PetscInt d_nnz[] = {0, 0, 0}; > PetscInt o_nnz[] = {3, 3, 3}; > err = MatMPIAIJSetPreallocation(A, 0, d_nnz, 0, o_nnz); > CHKERRQ(err); > } > err = MatSetUp(A); > CHKERRQ(err); > // set values inside the matrix > for (PetscInt row = 0; row < global_height; ++row) > { > for (PetscInt col = global_height; col < global_width; ++col) > { > err = MatSetValue(A, row, col, 1, INSERT_VALUES); > CHKERRQ(err); > } > } > err = MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); > CHKERRQ(err); > err = MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); > CHKERRQ(err); > err = MatView(A, PETSC_VIEWER_STDOUT_WORLD); > CHKERRQ(err); > // free memory > err = MatDestroy(&A); > CHKERRQ(err); > // cleanup any internal PETSc data at end of program > err = PetscFinalize(); > CHKERRQ(err); > } Am I mis-understanding what the d_nnz and o_nnz parameter are supposed to mean? -- Andrew Ho -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Feb 14 06:53:06 2017 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 14 Feb 2017 06:53:06 -0600 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> Message-ID: On Mon, Feb 13, 2017 at 3:59 PM, Zhang, Hong wrote: > > On Feb 13, 2017, at 3:08 PM, Ed Bueler wrote: > > Barry -- > > > Sounds like a bug to me. The methods should be checking if an > > IFunction is being provided and error out in that case. > > I don't think it is that simple. I.e. having an IFunction and an explicit > scheme is not, by itself, a bug. I think. > > If the user has provided IFunction > F(t,u,u_t) = u_t - f(t,u) > which is the convenient form for e.g. TSARKIMEX, > and RHSFunction > G(t,u) > then I guess I assumed that explicit methods like TSRK should,for > unconstrained cases, get their RHS by callback like this: > g(t,u) = - F(t,u,0) + G(t,u) > so that the ODE is in form > u_t = g(t,u) = f(t,u) + g(t,u) > > > PETSc does not transform IFunction to the RHS this way, and PETSc is not > supposed to do it automatically. Note that IFunction may involve a mass > matrix, e.g. F(t,u,u_t) = M*u_t - f(t,u) and sometimes M is not > invertible. > > > I think that is the behavior I am seeing (i.e. on my problem, using PETSc > master). > > > Explicit methods use only RHSFunction and ignore IFunction, so in your > case, if you change TS type to rk and ssp at run time, you are actually > solving u_t = G(t,u). If RHSFunction is not provided, PETSc will assume the > RHS is zero (u_t=0). > I do not agree with this. I thought our aim was to support users comparing explicit with implicit with semi-implicit. This breaks that model completely. Matt > > The "nonsense" I am referring to is only from the non-enforcement of the > constraint; it would be o.k. for a pure ODE. > > I would love to have some projection mechanism to try, for comparing > explicit methods with projection to the SNESVI way (i.e. the right way), > but that is asking for a lot of PETSc refactoring, I think. For now I just > want to error-out if the method is not going to call the SNES at each time > step. > > > Check if TSGetSNES() returns NULL in your code? > > Ed > > > > On Mon, Feb 13, 2017 at 11:26 AM, Barry Smith wrote: > >> >> > On Feb 13, 2017, at 1:53 PM, Ed Bueler wrote: >> > >> > Dear Petsc -- >> > >> > This is the first of two short TS usage questions. >> > >> > My problem is both stiff (diffusive PDE) and constrained, so I require >> > >> > -snes_type vinewton{rs|ss}ls >> > >> > *and* I split my ODE system into IFunction and RHSFunction >> > >> > F(t,u,u_t) = G(t,u) >> > >> > where F(t,u,u_t) = u_t + f(t,u) in my case (i.e. no mass matrix >> needed), and the stiff part goes in f(t,u). >> > >> > With this arrangement TS types beuler, theta, bdf, arkimex all work >> quite well. However, the program runs and produces nonsense with type rk >> and ssp, that is, explicit methods. >> >> Sounds like a bug to me. The methods should be checking if an >> IFunction is being provided and error out in that case. >> >> Barry >> >> > >> > So my question is, how do I ask the TS (at run time) whether the chosen >> TS type will or will not call its SNES at each step? If SNES is not going >> to be used then I want to SETERRQ and stop. That is, I want to error-out >> if the *method* is fully explicit. >> > >> > Note the constraints are enforced by the SNESVI, as they should be, not >> ad hoc projection. Also, as a technical matter, I cannot require my >> iterates to be feasible inside my IFunction evaluation because that would >> break VINEWTONSSLS. >> > >> > Neither TSProblemType (mine is NONLINEAR) nor TSEquationType (mine is >> IMPLICIT I guess) seem to address this? My problem is indeed nonlinear and >> has stiff parts, but it is not a DAE and it *is* constrained. >> > >> > Thanks! >> > >> > Ed >> > >> > PS I'd prefer not to enumerate the existing TS types and error on the >> bad ones. It is not nicely-maintainable. >> > >> > >> > >> > -- >> > Ed Bueler >> > Dept of Math and Stat and Geophysical Institute >> > University of Alaska Fairbanks >> > Fairbanks, AK 99775-6660 >> > 301C Chapman and 410D Elvey >> > 907 474-7693 and 907 474-7199 (fax 907 474-5394) >> >> > > > -- > Ed Bueler > Dept of Math and Stat and Geophysical Institute > University of Alaska Fairbanks > Fairbanks, AK 99775-6660 > 301C Chapman and 410D Elvey > 907 474-7693 <(907)%20474-7693> and 907 474-7199 <(907)%20474-7199> (fax 907 > 474-5394 <(907)%20474-5394>) > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Feb 14 07:26:34 2017 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 14 Feb 2017 07:26:34 -0600 Subject: [petsc-users] PETSc set off-diagonal matrix pre-allocation In-Reply-To: References: Message-ID: On Tue, Feb 14, 2017 at 6:41 AM, Andrew Ho wrote: > I have a 3x6 matrix, and I want to set it to (just as an example): > > 0 0 0 1 1 1 > 0 0 0 1 1 1 > 0 0 0 1 1 1 > > From what I can tell, according to the documentation the MPIAIJ sparsity > of this matrix is: > I believe this is an inconsistency in PETSc for rectangular matrices. We divide matrices into diagonal and off-diagonal parts mainly to separate communication from computation. Thus on 1 proc, we never divide them, even if the matrix is rectangular. Therefore we misinterpret your preallocation. Barry, do you think we want to try and change this? Matt > d_nnz = [0, 0, 0] > o_nnz = [3, 3, 3] > > However, when I do this, I get the following error: > > [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> >> [0]PETSC ERROR: Argument out of range >> [0]PETSC ERROR: New nonzero at (0,3) caused a malloc >> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn >> off this check >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.7.5, unknown >> [0]PETSC ERROR: ./ex3 on a arch-linux2-c-opt Tue Feb 14 04:23:56 2017 >> [0]PETSC ERROR: Configure options --with-debugging=0 --COPTFLAGS="-O3 >> -march=native" --CXXOPTFLAGS="-O3 -march=native" --FOPTFLAGS="-O3 >> -march=native" >> [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 582 in >> petsc/src/mat/impls/aij/mpi/mpiaij.c >> [0]PETSC ERROR: #2 MatSetValues() line 1190 in petsc/src/mat/interface/matrix.c >> >> [0]PETSC ERROR: #3 main() line 36 in ex3.c >> [0]PETSC ERROR: No PETSc Option Table entries >> [0]PETSC ERROR: ----------------End of Error Message -------send entire >> error message to petsc-maint at mcs.anl.gov---------- > > > Here's a working test code: > > #include >> #include >> int main(int argc, char** argv) >> { >> PetscErrorCode err; >> err = PetscInitialize(&argc, &argv, NULL, "help"); >> CHKERRQ(err); >> // create a sparse AIJ matrix distributed across MPI >> PetscInt global_width = 6; >> PetscInt global_height = 3; >> Mat A; >> err = MatCreate(MPI_COMM_WORLD, &A); >> CHKERRQ(err); >> err = MatSetType(A, MATMPIAIJ); >> CHKERRQ(err); >> // setup pre-allocation for matrix space >> { >> err = >> MatSetSizes(A, global_height, PETSC_DECIDE, global_height, >> global_width); >> CHKERRQ(err); >> PetscInt d_nnz[] = {0, 0, 0}; >> PetscInt o_nnz[] = {3, 3, 3}; >> err = MatMPIAIJSetPreallocation(A, 0, d_nnz, 0, o_nnz); >> CHKERRQ(err); >> } >> err = MatSetUp(A); >> CHKERRQ(err); >> // set values inside the matrix >> for (PetscInt row = 0; row < global_height; ++row) >> { >> for (PetscInt col = global_height; col < global_width; ++col) >> { >> err = MatSetValue(A, row, col, 1, INSERT_VALUES); >> CHKERRQ(err); >> } >> } >> err = MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); >> CHKERRQ(err); >> err = MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); >> CHKERRQ(err); >> err = MatView(A, PETSC_VIEWER_STDOUT_WORLD); >> CHKERRQ(err); >> // free memory >> err = MatDestroy(&A); >> CHKERRQ(err); >> // cleanup any internal PETSc data at end of program >> err = PetscFinalize(); >> CHKERRQ(err); >> } > > > Am I mis-understanding what the d_nnz and o_nnz parameter are supposed to > mean? > -- > Andrew Ho > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From emconsta at mcs.anl.gov Tue Feb 14 08:55:07 2017 From: emconsta at mcs.anl.gov (Emil Constantinescu) Date: Tue, 14 Feb 2017 08:55:07 -0600 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> Message-ID: On 2/14/17 6:53 AM, Matthew Knepley wrote: > > Explicit methods use only RHSFunction and ignore IFunction, so in > your case, if you change TS type to rk and ssp at run time, you are > actually solving u_t = G(t,u). If RHSFunction is not provided, PETSc > will assume the RHS is zero (u_t=0). > > > I do not agree with this. I thought our aim was to support users > comparing explicit with implicit with semi-implicit. This breaks that > model completely. Matt, all RHSFunction can be turned into IFunction but not the other way around. Explicit TS solvers cannot deal with all forms of IFunction and providing support for classifying them may be burdensome and difficult to scale. I think it's more lucrative to improve the documentation. Emil From knepley at gmail.com Tue Feb 14 09:00:17 2017 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 14 Feb 2017 09:00:17 -0600 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> Message-ID: On Tue, Feb 14, 2017 at 8:55 AM, Emil Constantinescu wrote: > On 2/14/17 6:53 AM, Matthew Knepley wrote: > >> >> Explicit methods use only RHSFunction and ignore IFunction, so in >> your case, if you change TS type to rk and ssp at run time, you are >> actually solving u_t = G(t,u). If RHSFunction is not provided, PETSc >> will assume the RHS is zero (u_t=0). >> >> >> I do not agree with this. I thought our aim was to support users >> comparing explicit with implicit with semi-implicit. This breaks that >> model completely. >> > > Matt, all RHSFunction can be turned into IFunction but not the other way > around. Explicit TS solvers cannot deal with all forms of IFunction and > providing support for classifying them may be burdensome and difficult to > scale. I think it's more lucrative to improve the documentation. I see that is it not one-to-one, but its a pattern I think a lot of people will want, and one of the most common things people ask is "when does implicit beat explicit?". We should think about ways to make this easier. Matt > > Emil > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue Feb 14 10:38:06 2017 From: jed at jedbrown.org (Jed Brown) Date: Tue, 14 Feb 2017 09:38:06 -0700 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> Message-ID: <874lzw6841.fsf@jedbrown.org> Matthew Knepley writes: > I see that is it not one-to-one, but its a pattern I think a lot of people > will want, and one of the most common > things people ask is "when does implicit beat explicit?". We should think > about ways to make this easier. Note that one of the most common splittings is u_t - L u = F(u) - L u The user can have their own option to split or not split, but if we ask them to break out a bunch of terms, it leads to rather unnatural terms (subtracting off components that are added back somewhere else). -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From bsmith at mcs.anl.gov Tue Feb 14 11:36:30 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 14 Feb 2017 11:36:30 -0600 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> Message-ID: <60D7CDCC-E9D5-472A-9175-64C56DF8B34F@mcs.anl.gov> > On Feb 14, 2017, at 6:53 AM, Matthew Knepley wrote: > > On Mon, Feb 13, 2017 at 3:59 PM, Zhang, Hong wrote: > >> On Feb 13, 2017, at 3:08 PM, Ed Bueler wrote: >> >> Barry -- >> >> > Sounds like a bug to me. The methods should be checking if an >> > IFunction is being provided and error out in that case. >> >> I don't think it is that simple. I.e. having an IFunction and an explicit scheme is not, by itself, a bug. I think. >> >> If the user has provided IFunction >> F(t,u,u_t) = u_t - f(t,u) >> which is the convenient form for e.g. TSARKIMEX, >> and RHSFunction >> G(t,u) >> then I guess I assumed that explicit methods like TSRK should,for unconstrained cases, get their RHS by callback like this: >> g(t,u) = - F(t,u,0) + G(t,u) >> so that the ODE is in form >> u_t = g(t,u) = f(t,u) + g(t,u) >> > > PETSc does not transform IFunction to the RHS this way, and PETSc is not supposed to do it automatically. Note that IFunction may involve a mass matrix, e.g. F(t,u,u_t) = M*u_t - f(t,u) and sometimes M is not invertible. > > >> I think that is the behavior I am seeing (i.e. on my problem, using PETSc master). > > Explicit methods use only RHSFunction and ignore IFunction, so in your case, if you change TS type to rk and ssp at run time, you are actually solving u_t = G(t,u). If RHSFunction is not provided, PETSc will assume the RHS is zero (u_t=0). > > I do not agree with this. I thought our aim was to support users comparing explicit with implicit with semi-implicit. This breaks that model completely. Hence my suggestion to have TSSetLeftHandSideFunction() (or Jed's suggestion to have multiple Right Hand side functions) this will allow comparison of implicit, explicit, imex without recompiling (which we don't have currently) for the case with no mass matrix. With a mass matrix, unless we use mass lumping and have a TSSetMassMatrix() you cannot switch to full explicit, but that is due to mathematics, not the interface. Barry > > Matt > >> >> The "nonsense" I am referring to is only from the non-enforcement of the constraint; it would be o.k. for a pure ODE. >> >> I would love to have some projection mechanism to try, for comparing explicit methods with projection to the SNESVI way (i.e. the right way), but that is asking for a lot of PETSc refactoring, I think. For now I just want to error-out if the method is not going to call the SNES at each time step. > > Check if TSGetSNES() returns NULL in your code? > >> Ed >> >> >> >> On Mon, Feb 13, 2017 at 11:26 AM, Barry Smith wrote: >> >> > On Feb 13, 2017, at 1:53 PM, Ed Bueler wrote: >> > >> > Dear Petsc -- >> > >> > This is the first of two short TS usage questions. >> > >> > My problem is both stiff (diffusive PDE) and constrained, so I require >> > >> > -snes_type vinewton{rs|ss}ls >> > >> > *and* I split my ODE system into IFunction and RHSFunction >> > >> > F(t,u,u_t) = G(t,u) >> > >> > where F(t,u,u_t) = u_t + f(t,u) in my case (i.e. no mass matrix needed), and the stiff part goes in f(t,u). >> > >> > With this arrangement TS types beuler, theta, bdf, arkimex all work quite well. However, the program runs and produces nonsense with type rk and ssp, that is, explicit methods. >> >> Sounds like a bug to me. The methods should be checking if an IFunction is being provided and error out in that case. >> >> Barry >> >> > >> > So my question is, how do I ask the TS (at run time) whether the chosen TS type will or will not call its SNES at each step? If SNES is not going to be used then I want to SETERRQ and stop. That is, I want to error-out if the *method* is fully explicit. >> > >> > Note the constraints are enforced by the SNESVI, as they should be, not ad hoc projection. Also, as a technical matter, I cannot require my iterates to be feasible inside my IFunction evaluation because that would break VINEWTONSSLS. >> > >> > Neither TSProblemType (mine is NONLINEAR) nor TSEquationType (mine is IMPLICIT I guess) seem to address this? My problem is indeed nonlinear and has stiff parts, but it is not a DAE and it *is* constrained. >> > >> > Thanks! >> > >> > Ed >> > >> > PS I'd prefer not to enumerate the existing TS types and error on the bad ones. It is not nicely-maintainable. >> > >> > >> > >> > -- >> > Ed Bueler >> > Dept of Math and Stat and Geophysical Institute >> > University of Alaska Fairbanks >> > Fairbanks, AK 99775-6660 >> > 301C Chapman and 410D Elvey >> > 907 474-7693 and 907 474-7199 (fax 907 474-5394) >> >> >> >> >> -- >> Ed Bueler >> Dept of Math and Stat and Geophysical Institute >> University of Alaska Fairbanks >> Fairbanks, AK 99775-6660 >> 301C Chapman and 410D Elvey >> 907 474-7693 and 907 474-7199 (fax 907 474-5394) > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From jed at jedbrown.org Tue Feb 14 11:55:10 2017 From: jed at jedbrown.org (Jed Brown) Date: Tue, 14 Feb 2017 10:55:10 -0700 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: <60D7CDCC-E9D5-472A-9175-64C56DF8B34F@mcs.anl.gov> References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> <60D7CDCC-E9D5-472A-9175-64C56DF8B34F@mcs.anl.gov> Message-ID: <87vasc4pz5.fsf@jedbrown.org> Barry Smith writes: > Hence my suggestion to have TSSetLeftHandSideFunction() (or Jed's suggestion to have multiple Right Hand side functions) this will allow comparison of implicit, explicit, imex without recompiling (which we don't have currently) for the case with no mass matrix. With a mass matrix, unless we use mass lumping and have a TSSetMassMatrix() you cannot switch to full explicit, but that is due to mathematics, not the interface. Who in reality has a problem that looks like u_t = f(u) + g(u) + h(u) where it may make sense to move some of this to the left (implicit)? I'm concerned that this isn't natural for many applications so adding an interface would be solving a problem that doesn't really exist outside perhaps a few academic examples. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From bernardomartinsrocha at gmail.com Tue Feb 14 12:38:21 2017 From: bernardomartinsrocha at gmail.com (Bernardo Rocha) Date: Tue, 14 Feb 2017 16:38:21 -0200 Subject: [petsc-users] PC HYPRE BoomerAMG options for nodal Message-ID: Dear Petsc users, I would like to use BoomerAMG preconditioner for a system of PDEs, but we are interested in setting some of BoomerAMG options to use the nodal systems coarsening. >From the HYPRE/BoomerAMG manual, I found out that I have to use the following functions: HYPRE_BoomerAMGSetNodal HYPRE_BoomerAMGSetNumFunctions HYPRE_BoomerAMGSetDofFunc Looking at the source code (src/ksp/pc/impls/hypre/hypre.c) I've found the call to the first one, but not for the other two, where I must specify the ordering of my dofs in my problem. I would appreciate any help on this. Best regards, Bernardo -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Feb 14 12:53:56 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 14 Feb 2017 12:53:56 -0600 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: <87vasc4pz5.fsf@jedbrown.org> References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> <60D7CDCC-E9D5-472A-9175-64C56DF8B34F@mcs.anl.gov> <87vasc4pz5.fsf@jedbrown.org> Message-ID: <6D880F3B-F880-4AFD-975D-5A8A4F4AA8A9@mcs.anl.gov> I think > u_t = f(u) + g(u) where one wishes to switch from fully implicit to imex to fully explicit (with a likely small time-step constraint) is common enough that we should support it (we don't currently). The reason is that users may not know which form they should use and the easier we make it to try all forms the better it is for users. You, as an expert, might be able to say before hand based on the problem what specific form should be used but not everyone is an expert so it is better to make it painless to try everything without reorganizing your code. In the same way we try to support as many preconditioners etc without needed to change code and recompile. Barry > On Feb 14, 2017, at 11:55 AM, Jed Brown wrote: > > Barry Smith writes: >> Hence my suggestion to have TSSetLeftHandSideFunction() (or Jed's suggestion to have multiple Right Hand side functions) this will allow comparison of implicit, explicit, imex without recompiling (which we don't have currently) for the case with no mass matrix. With a mass matrix, unless we use mass lumping and have a TSSetMassMatrix() you cannot switch to full explicit, but that is due to mathematics, not the interface. > > Who in reality has a problem that looks like > > u_t = f(u) + g(u) + h(u) > > where it may make sense to move some of this to the left (implicit)? > I'm concerned that this isn't natural for many applications so adding an > interface would be solving a problem that doesn't really exist outside > perhaps a few academic examples. From bsmith at mcs.anl.gov Tue Feb 14 13:06:31 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 14 Feb 2017 13:06:31 -0600 Subject: [petsc-users] PC HYPRE BoomerAMG options for nodal In-Reply-To: References: Message-ID: <4032E7BA-A557-4491-B4E1-71F42E13DBA6@mcs.anl.gov> > On Feb 14, 2017, at 12:38 PM, Bernardo Rocha wrote: > > Dear Petsc users, > > I would like to use BoomerAMG preconditioner for a system of PDEs, but we are interested in setting some of BoomerAMG options to use the nodal systems coarsening. > > From the HYPRE/BoomerAMG manual, I found out that I have to use the following functions: > HYPRE_BoomerAMGSetNodal ierr = PetscOptionsInt("-pc_hypre_boomeramg_nodal_coarsen","Use a nodal based coarsening 1-6","HYPRE_BoomerAMGSetNodal",jac->nodal_coarsening, &jac->nodal_coarsening,&flg);CHKERRQ(ierr); if (flg) { PetscStackCallStandard(HYPRE_BoomerAMGSetNodal,(jac->hsolver,jac->nodal_coarsening)); } > HYPRE_BoomerAMGSetNumFunctions Set the block size of the matrix; this information is then transferred automatically to hypre. Note that you can set a block size even for AIJ matrices. > HYPRE_BoomerAMGSetDofFunc I could not find an indication of that this is suppose to be setting; it is mentioned in the users manual but I don't know what it means. It looks like it takes an integer array but I don't even know how long that array is. If you figure out what it means and can add support for it we'll take it as a pull request. barry > > Looking at the source code (src/ksp/pc/impls/hypre/hypre.c) I've found the call to the first one, but not for the other two, where I must specify the ordering of my dofs in my problem. > > I would appreciate any help on this. > > Best regards, > Bernardo From andrewh0 at uw.edu Tue Feb 14 13:06:32 2017 From: andrewh0 at uw.edu (Andrew Ho) Date: Tue, 14 Feb 2017 11:06:32 -0800 Subject: [petsc-users] PETSc set off-diagonal matrix pre-allocation In-Reply-To: References: Message-ID: The problem isn't only with 1 process, but it seems to be with all non-square matrices? For example, here I have a two process program which tries to set a 6x7 matrix to (each process owns 3 rows): 0 0 0 1 1 1 1 0 0 0 1 1 1 1 0 0 0 1 1 1 1 0 0 0 1 1 1 1 0 0 0 1 1 1 1 0 0 0 1 1 1 1 #include #include int main(int argc, char** argv) { PetscErrorCode err; err = PetscInitialize(&argc, &argv, NULL, "help"); CHKERRQ(err); int rank, size; MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &size); if(size != 2) { printf("must run with 2 processes"); MPI_Abort(MPI_COMM_WORLD, -1); } // create a sparse AIJ matrix distributed across MPI Mat A; err = MatCreate(MPI_COMM_WORLD, &A); CHKERRQ(err); err = MatSetType(A, MATMPIAIJ); CHKERRQ(err); // setup pre-allocation for matrix space { err = MatSetSizes(A, 3, PETSC_DECIDE, 6, 7); CHKERRQ(err); if(rank == 0) { PetscInt d_nnz[] = {0, 0, 0}; PetscInt o_nnz[] = {4, 4, 4}; err = MatMPIAIJSetPreallocation(A, 0, d_nnz, 0, o_nnz); CHKERRQ(err); } else { PetscInt d_nnz[] = {3, 3, 3}; PetscInt o_nnz[] = {1, 1, 1}; err = MatMPIAIJSetPreallocation(A, 0, d_nnz, 0, o_nnz); CHKERRQ(err); } } err = MatSetUp(A); CHKERRQ(err); // set values inside the matrix for (PetscInt row = 0; row < 3; ++row) { for (PetscInt col = 3; col < 7; ++col) { err = MatSetValue(A, 3 * rank + row, col, 1, INSERT_VALUES); CHKERRQ(err); } } err = MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); CHKERRQ(err); err = MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); CHKERRQ(err); err = MatView(A, PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(err); // free memory err = MatDestroy(&A); CHKERRQ(err); // cleanup any internal PETSc data at end of program err = PetscFinalize(); CHKERRQ(err); } On Tue, Feb 14, 2017 at 5:26 AM, Matthew Knepley wrote: > On Tue, Feb 14, 2017 at 6:41 AM, Andrew Ho wrote: > >> I have a 3x6 matrix, and I want to set it to (just as an example): >> >> 0 0 0 1 1 1 >> 0 0 0 1 1 1 >> 0 0 0 1 1 1 >> >> From what I can tell, according to the documentation the MPIAIJ sparsity >> of this matrix is: >> > > I believe this is an inconsistency in PETSc for rectangular matrices. We > divide matrices into diagonal and > off-diagonal parts mainly to separate communication from computation. Thus > on 1 proc, we never divide > them, even if the matrix is rectangular. Therefore we misinterpret your > preallocation. > > Barry, do you think we want to try and change this? > > Matt > > >> d_nnz = [0, 0, 0] >> o_nnz = [3, 3, 3] >> >> However, when I do this, I get the following error: >> >> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> >>> [0]PETSC ERROR: Argument out of range >>> [0]PETSC ERROR: New nonzero at (0,3) caused a malloc >>> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to >>> turn off this check >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.7.5, unknown >>> [0]PETSC ERROR: ./ex3 on a arch-linux2-c-opt Tue Feb 14 04:23:56 2017 >>> [0]PETSC ERROR: Configure options --with-debugging=0 --COPTFLAGS="-O3 >>> -march=native" --CXXOPTFLAGS="-O3 -march=native" --FOPTFLAGS="-O3 >>> -march=native" >>> [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 582 in >>> petsc/src/mat/impls/aij/mpi/mpiaij.c >>> [0]PETSC ERROR: #2 MatSetValues() line 1190 in >>> petsc/src/mat/interface/matrix.c >>> [0]PETSC ERROR: #3 main() line 36 in ex3.c >>> [0]PETSC ERROR: No PETSc Option Table entries >>> [0]PETSC ERROR: ----------------End of Error Message -------send entire >>> error message to petsc-maint at mcs.anl.gov---------- >> >> >> Here's a working test code: >> >> #include >>> #include >>> int main(int argc, char** argv) >>> { >>> PetscErrorCode err; >>> err = PetscInitialize(&argc, &argv, NULL, "help"); >>> CHKERRQ(err); >>> // create a sparse AIJ matrix distributed across MPI >>> PetscInt global_width = 6; >>> PetscInt global_height = 3; >>> Mat A; >>> err = MatCreate(MPI_COMM_WORLD, &A); >>> CHKERRQ(err); >>> err = MatSetType(A, MATMPIAIJ); >>> CHKERRQ(err); >>> // setup pre-allocation for matrix space >>> { >>> err = >>> MatSetSizes(A, global_height, PETSC_DECIDE, global_height, >>> global_width); >>> CHKERRQ(err); >>> PetscInt d_nnz[] = {0, 0, 0}; >>> PetscInt o_nnz[] = {3, 3, 3}; >>> err = MatMPIAIJSetPreallocation(A, 0, d_nnz, 0, o_nnz); >>> CHKERRQ(err); >>> } >>> err = MatSetUp(A); >>> CHKERRQ(err); >>> // set values inside the matrix >>> for (PetscInt row = 0; row < global_height; ++row) >>> { >>> for (PetscInt col = global_height; col < global_width; ++col) >>> { >>> err = MatSetValue(A, row, col, 1, INSERT_VALUES); >>> CHKERRQ(err); >>> } >>> } >>> err = MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); >>> CHKERRQ(err); >>> err = MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); >>> CHKERRQ(err); >>> err = MatView(A, PETSC_VIEWER_STDOUT_WORLD); >>> CHKERRQ(err); >>> // free memory >>> err = MatDestroy(&A); >>> CHKERRQ(err); >>> // cleanup any internal PETSc data at end of program >>> err = PetscFinalize(); >>> CHKERRQ(err); >>> } >> >> >> Am I mis-understanding what the d_nnz and o_nnz parameter are supposed to >> mean? >> -- >> Andrew Ho >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Andrew Ho -------------- next part -------------- An HTML attachment was scrubbed... URL: From hongzhang at anl.gov Tue Feb 14 14:33:03 2017 From: hongzhang at anl.gov (Zhang, Hong) Date: Tue, 14 Feb 2017 20:33:03 +0000 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: <87vasc4pz5.fsf@jedbrown.org> References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> <60D7CDCC-E9D5-472A-9175-64C56DF8B34F@mcs.anl.gov> <87vasc4pz5.fsf@jedbrown.org> Message-ID: <36FFCE9F-ECA8-4B5F-BE53-6B1AD2C7B205@anl.gov> I think many users (including me) would like to start with academic examples, e.g. u_t=f(u)+g(u), when they try to learn PETSc TS solvers. This simple form allows for easy switch between all kinds of different integration methods. Experienced users or experts who need to solve DAEs or complicate problems involving nontrivial mass matrix should be encouraged to try IFunction/IJacobian. Of course, we need better documentation to help them realize the switching is not always possible. Personally it took me a long time to get used to the IFunction business since I had been using u_t=f(u)+g(u) for a couple years before I jumped on PETSc. If we add support for this simple form, the learning curve would be less steep than it currently is. Hong (Mr.) > On Feb 14, 2017, at 11:55 AM, Jed Brown wrote: > > Barry Smith writes: >> Hence my suggestion to have TSSetLeftHandSideFunction() (or Jed's suggestion to have multiple Right Hand side functions) this will allow comparison of implicit, explicit, imex without recompiling (which we don't have currently) for the case with no mass matrix. With a mass matrix, unless we use mass lumping and have a TSSetMassMatrix() you cannot switch to full explicit, but that is due to mathematics, not the interface. > > Who in reality has a problem that looks like > > u_t = f(u) + g(u) + h(u) > > where it may make sense to move some of this to the left (implicit)? > I'm concerned that this isn't natural for many applications so adding an > interface would be solving a problem that doesn't really exist outside > perhaps a few academic examples. From gideon.simpson at gmail.com Tue Feb 14 14:46:39 2017 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Tue, 14 Feb 2017 15:46:39 -0500 Subject: [petsc-users] prefix (i.e. cumulative) sum Message-ID: Is there a clever way to handle a prefix (cumulative) sum in petsc? I think I can see how to do with using some underlying MPI, but I was wondering if there were some built in routines that could be useful. -gideon -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Feb 14 14:48:10 2017 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 14 Feb 2017 14:48:10 -0600 Subject: [petsc-users] prefix (i.e. cumulative) sum In-Reply-To: References: Message-ID: On Tue, Feb 14, 2017 at 2:46 PM, Gideon Simpson wrote: > Is there a clever way to handle a prefix (cumulative) sum in petsc? I > think I can see how to do with using some underlying MPI, but I was > wondering if there were some built in routines that could be useful. > It is one MPI call (MPI_Scan), but you could do it by assigning sizes to a PetscLayout (which is how I have done it several places). Thanks, Matt > -gideon > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From emconsta at mcs.anl.gov Tue Feb 14 14:55:42 2017 From: emconsta at mcs.anl.gov (Emil Constantinescu) Date: Tue, 14 Feb 2017 14:55:42 -0600 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: <36FFCE9F-ECA8-4B5F-BE53-6B1AD2C7B205@anl.gov> References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> <60D7CDCC-E9D5-472A-9175-64C56DF8B34F@mcs.anl.gov> <87vasc4pz5.fsf@jedbrown.org> <36FFCE9F-ECA8-4B5F-BE53-6B1AD2C7B205@anl.gov> Message-ID: <95c8d53c-5c77-34aa-68dd-3e5211d24c24@mcs.anl.gov> On 2/14/17 2:33 PM, Zhang, Hong wrote: > I think many users (including me) would like to start with academic examples, e.g. u_t=f(u)+g(u), when they try to learn PETSc TS solvers. This simple form allows for easy switch between all kinds of different integration methods. Right, but then you can just write an if statement along the lines in ex31.c: (this was before RHSJacobian and it's not intended for IMEX) TSGetType(ts,&time_scheme); if ((!strcmp(time_scheme,TSEULER)) || (!strcmp(time_scheme,TSRK)) || (!strcmp(time_scheme,TSSSP))) { /* Explicit time-integration -> specify right-hand side function ydot = f(y) */ TSSetRHSFunction(ts,NULL,RHSFunction,&ptype[0]); } else if ((!strcmp(time_scheme,TSTHETA)) ||... (!strcmp(time_scheme,TSARKIMEX))) { /* Implicit time-integration -> specify left-hand side function ydot-f(y) = 0 */ /* and its Jacobian function */ TSSetIFunction(ts,NULL,IFunction,&ptype[0]); TSSetIJacobian(ts,Jac,Jac,IJacobian,&ptype[0]); } This should not be a performance bottleneck. IFunction can be a wrapper of the RHSFunction so not too much extra coding. Emil > Experienced users or experts who need to solve DAEs or complicate problems involving nontrivial mass matrix should be encouraged to try IFunction/IJacobian. Of course, we need better documentation to help them realize the switching is not always possible. Personally it took me a long time to get used to the IFunction business since I had been using u_t=f(u)+g(u) for a couple years before I jumped on PETSc. If we add support for this simple form, the learning curve would be less steep than it currently is. > > Hong (Mr.) > >> On Feb 14, 2017, at 11:55 AM, Jed Brown wrote: >> >> Barry Smith writes: >>> Hence my suggestion to have TSSetLeftHandSideFunction() (or Jed's suggestion to have multiple Right Hand side functions) this will allow comparison of implicit, explicit, imex without recompiling (which we don't have currently) for the case with no mass matrix. With a mass matrix, unless we use mass lumping and have a TSSetMassMatrix() you cannot switch to full explicit, but that is due to mathematics, not the interface. >> >> Who in reality has a problem that looks like >> >> u_t = f(u) + g(u) + h(u) >> >> where it may make sense to move some of this to the left (implicit)? >> I'm concerned that this isn't natural for many applications so adding an >> interface would be solving a problem that doesn't really exist outside >> perhaps a few academic examples. From jed at jedbrown.org Tue Feb 14 15:00:49 2017 From: jed at jedbrown.org (Jed Brown) Date: Tue, 14 Feb 2017 14:00:49 -0700 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: <95c8d53c-5c77-34aa-68dd-3e5211d24c24@mcs.anl.gov> References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> <60D7CDCC-E9D5-472A-9175-64C56DF8B34F@mcs.anl.gov> <87vasc4pz5.fsf@jedbrown.org> <36FFCE9F-ECA8-4B5F-BE53-6B1AD2C7B205@anl.gov> <95c8d53c-5c77-34aa-68dd-3e5211d24c24@mcs.anl.gov> Message-ID: Criminy, that's disgusting. You implement options -term1 rhs -term2 lhs ... In your FormIFunction: if (term1 == LHS) { ... } and similarly in your FormRHSFunction. On February 14, 2017 1:55:42 PM MST, Emil Constantinescu wrote: >On 2/14/17 2:33 PM, Zhang, Hong wrote: >> I think many users (including me) would like to start with academic >examples, e.g. u_t=f(u)+g(u), when they try to learn PETSc TS solvers. >This simple form allows for easy switch between all kinds of different >integration methods. > >Right, but then you can just write an if statement along the lines in >ex31.c: (this was before RHSJacobian and it's not intended for IMEX) > >TSGetType(ts,&time_scheme); >if ((!strcmp(time_scheme,TSEULER)) || (!strcmp(time_scheme,TSRK)) || >(!strcmp(time_scheme,TSSSP))) { >/* Explicit time-integration -> specify right-hand side function ydot = > >f(y) */ >TSSetRHSFunction(ts,NULL,RHSFunction,&ptype[0]); >} else if ((!strcmp(time_scheme,TSTHETA)) ||... > (!strcmp(time_scheme,TSARKIMEX))) { > /* Implicit time-integration -> specify left-hand side function >ydot-f(y) = 0 */ > /* and its Jacobian function */ > TSSetIFunction(ts,NULL,IFunction,&ptype[0]); > TSSetIJacobian(ts,Jac,Jac,IJacobian,&ptype[0]); > } > >This should not be a performance bottleneck. IFunction can be a wrapper > >of the RHSFunction so not too much extra coding. > >Emil > >> Experienced users or experts who need to solve DAEs or complicate >problems involving nontrivial mass matrix should be encouraged to try >IFunction/IJacobian. Of course, we need better documentation to help >them realize the switching is not always possible. Personally it took >me a long time to get used to the IFunction business since I had been >using u_t=f(u)+g(u) for a couple years before I jumped on PETSc. If we >add support for this simple form, the learning curve would be less >steep than it currently is. >> >> Hong (Mr.) >> >>> On Feb 14, 2017, at 11:55 AM, Jed Brown wrote: >>> >>> Barry Smith writes: >>>> Hence my suggestion to have TSSetLeftHandSideFunction() (or Jed's >suggestion to have multiple Right Hand side functions) this will allow >comparison of implicit, explicit, imex without recompiling (which we >don't have currently) for the case with no mass matrix. With a mass >matrix, unless we use mass lumping and have a TSSetMassMatrix() you >cannot switch to full explicit, but that is due to mathematics, not the >interface. >>> >>> Who in reality has a problem that looks like >>> >>> u_t = f(u) + g(u) + h(u) >>> >>> where it may make sense to move some of this to the left (implicit)? >>> I'm concerned that this isn't natural for many applications so >adding an >>> interface would be solving a problem that doesn't really exist >outside >>> perhaps a few academic examples. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gideon.simpson at gmail.com Tue Feb 14 15:01:47 2017 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Tue, 14 Feb 2017 16:01:47 -0500 Subject: [petsc-users] prefix (i.e. cumulative) sum In-Reply-To: References: Message-ID: I gather I?d use something like this? VecGetArray(x, &xarray); VecGetArray(y, &yarray); MPI_Scan(array, array, nglobal, MPIU_SCALAR, MPI_SUM, PETSC_COMM_WORLD); VecRestoreArray(x, &xarray); VecRestoreArray(y, &yarray); -gideon > On Feb 14, 2017, at 3:48 PM, Matthew Knepley wrote: > > On Tue, Feb 14, 2017 at 2:46 PM, Gideon Simpson > wrote: > Is there a clever way to handle a prefix (cumulative) sum in petsc? I think I can see how to do with using some underlying MPI, but I was wondering if there were some built in routines that could be useful. > > It is one MPI call (MPI_Scan), but you could do it by assigning sizes to a PetscLayout (which is how I have done it several places). > > Thanks, > > Matt > > -gideon > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Feb 14 15:04:08 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 14 Feb 2017 15:04:08 -0600 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: <95c8d53c-5c77-34aa-68dd-3e5211d24c24@mcs.anl.gov> References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> <60D7CDCC-E9D5-472A-9175-64C56DF8B34F@mcs.anl.gov> <87vasc4pz5.fsf@jedbrown.org> <36FFCE9F-ECA8-4B5F-BE53-6B1AD2C7B205@anl.gov> <95c8d53c-5c77-34aa-68dd-3e5211d24c24@mcs.anl.gov> Message-ID: <45E4D3E5-B4DC-4BDC-984A-ED7735723E50@mcs.anl.gov> > On Feb 14, 2017, at 2:55 PM, Emil Constantinescu wrote: > > On 2/14/17 2:33 PM, Zhang, Hong wrote: >> I think many users (including me) would like to start with academic examples, e.g. u_t=f(u)+g(u), when they try to learn PETSc TS solvers. This simple form allows for easy switch between all kinds of different integration methods. > > Right, but then you can just write an if statement along the lines in ex31.c: (this was before RHSJacobian and it's not intended for IMEX) > > TSGetType(ts,&time_scheme); > if ((!strcmp(time_scheme,TSEULER)) || (!strcmp(time_scheme,TSRK)) || (!strcmp(time_scheme,TSSSP))) { > /* Explicit time-integration -> specify right-hand side function ydot = f(y) */ > TSSetRHSFunction(ts,NULL,RHSFunction,&ptype[0]); > } else if ((!strcmp(time_scheme,TSTHETA)) ||... > (!strcmp(time_scheme,TSARKIMEX))) { > /* Implicit time-integration -> specify left-hand side function ydot-f(y) = 0 */ > /* and its Jacobian function */ > TSSetIFunction(ts,NULL,IFunction,&ptype[0]); > TSSetIJacobian(ts,Jac,Jac,IJacobian,&ptype[0]); > } Multiple yucks! Switching methods should not require compiling code except when absolutely necessary and this is not a case of absolutely necessary! > > This should not be a performance bottleneck. IFunction can be a wrapper of the RHSFunction so not too much extra coding. > > Emil > >> Experienced users or experts who need to solve DAEs or complicate problems involving nontrivial mass matrix should be encouraged to try IFunction/IJacobian. Of course, we need better documentation to help them realize the switching is not always possible. Personally it took me a long time to get used to the IFunction business since I had been using u_t=f(u)+g(u) for a couple years before I jumped on PETSc. If we add support for this simple form, the learning curve would be less steep than it currently is. >> >> Hong (Mr.) >> >>> On Feb 14, 2017, at 11:55 AM, Jed Brown wrote: >>> >>> Barry Smith writes: >>>> Hence my suggestion to have TSSetLeftHandSideFunction() (or Jed's suggestion to have multiple Right Hand side functions) this will allow comparison of implicit, explicit, imex without recompiling (which we don't have currently) for the case with no mass matrix. With a mass matrix, unless we use mass lumping and have a TSSetMassMatrix() you cannot switch to full explicit, but that is due to mathematics, not the interface. >>> >>> Who in reality has a problem that looks like >>> >>> u_t = f(u) + g(u) + h(u) >>> >>> where it may make sense to move some of this to the left (implicit)? >>> I'm concerned that this isn't natural for many applications so adding an >>> interface would be solving a problem that doesn't really exist outside >>> perhaps a few academic examples. From emconsta at mcs.anl.gov Tue Feb 14 15:18:34 2017 From: emconsta at mcs.anl.gov (Emil Constantinescu) Date: Tue, 14 Feb 2017 15:18:34 -0600 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: <45E4D3E5-B4DC-4BDC-984A-ED7735723E50@mcs.anl.gov> References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> <60D7CDCC-E9D5-472A-9175-64C56DF8B34F@mcs.anl.gov> <87vasc4pz5.fsf@jedbrown.org> <36FFCE9F-ECA8-4B5F-BE53-6B1AD2C7B205@anl.gov> <95c8d53c-5c77-34aa-68dd-3e5211d24c24@mcs.anl.gov> <45E4D3E5-B4DC-4BDC-984A-ED7735723E50@mcs.anl.gov> Message-ID: <03329c72-3572-9f71-5e5b-34741f5d7573@mcs.anl.gov> On 2/14/17 3:04 PM, Barry Smith wrote: > >> On Feb 14, 2017, at 2:55 PM, Emil Constantinescu wrote: >> >> On 2/14/17 2:33 PM, Zhang, Hong wrote: >>> I think many users (including me) would like to start with academic examples, e.g. u_t=f(u)+g(u), when they try to learn PETSc TS solvers. This simple form allows for easy switch between all kinds of different integration methods. >> >> Right, but then you can just write an if statement along the lines in ex31.c: (this was before RHSJacobian and it's not intended for IMEX) >> >> TSGetType(ts,&time_scheme); >> if ((!strcmp(time_scheme,TSEULER)) || (!strcmp(time_scheme,TSRK)) || (!strcmp(time_scheme,TSSSP))) { >> /* Explicit time-integration -> specify right-hand side function ydot = f(y) */ >> TSSetRHSFunction(ts,NULL,RHSFunction,&ptype[0]); >> } else if ((!strcmp(time_scheme,TSTHETA)) ||... >> (!strcmp(time_scheme,TSARKIMEX))) { >> /* Implicit time-integration -> specify left-hand side function ydot-f(y) = 0 */ >> /* and its Jacobian function */ >> TSSetIFunction(ts,NULL,IFunction,&ptype[0]); >> TSSetIJacobian(ts,Jac,Jac,IJacobian,&ptype[0]); >> } > > Multiple yucks! Switching methods should not require compiling code except when absolutely necessary and this is not a case of absolutely necessary! Wait, no. You don't recompile. When you set the -ts_type then it defines RHSFunction (for Explicit and Implicit) or RHS Function & IFunction (for IMEX). Adding all that logic to keep track of left sides and right sides for academic examples is likely not the best development. Emil > >> >> This should not be a performance bottleneck. IFunction can be a wrapper of the RHSFunction so not too much extra coding. >> >> Emil >> >>> Experienced users or experts who need to solve DAEs or complicate problems involving nontrivial mass matrix should be encouraged to try IFunction/IJacobian. Of course, we need better documentation to help them realize the switching is not always possible. Personally it took me a long time to get used to the IFunction business since I had been using u_t=f(u)+g(u) for a couple years before I jumped on PETSc. If we add support for this simple form, the learning curve would be less steep than it currently is. >>> >>> Hong (Mr.) >>> >>>> On Feb 14, 2017, at 11:55 AM, Jed Brown wrote: >>>> >>>> Barry Smith writes: >>>>> Hence my suggestion to have TSSetLeftHandSideFunction() (or Jed's suggestion to have multiple Right Hand side functions) this will allow comparison of implicit, explicit, imex without recompiling (which we don't have currently) for the case with no mass matrix. With a mass matrix, unless we use mass lumping and have a TSSetMassMatrix() you cannot switch to full explicit, but that is due to mathematics, not the interface. >>>> >>>> Who in reality has a problem that looks like >>>> >>>> u_t = f(u) + g(u) + h(u) >>>> >>>> where it may make sense to move some of this to the left (implicit)? >>>> I'm concerned that this isn't natural for many applications so adding an >>>> interface would be solving a problem that doesn't really exist outside >>>> perhaps a few academic examples. > From bsmith at mcs.anl.gov Tue Feb 14 16:10:46 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 14 Feb 2017 16:10:46 -0600 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: <03329c72-3572-9f71-5e5b-34741f5d7573@mcs.anl.gov> References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> <60D7CDCC-E9D5-472A-9175-64C56DF8B34F@mcs.anl.gov> <87vasc4pz5.fsf@jedbrown.org> <36FFCE9F-ECA8-4B5F-BE53-6B1AD2C7B205@anl.gov> <95c8d53c-5c77-34aa-68dd-3e5211d24c24@mcs.anl.gov> <45E4D3E5-B4DC-4BDC-984A-ED7735723E50@mcs.anl.gov> <03329c72-3572-9f71-5e5b-34741f5d7573@mcs.anl.gov> Message-ID: <9C507CBA-8DC4-4669-BEE7-4560E27EB020@mcs.anl.gov> > On Feb 14, 2017, at 3:18 PM, Emil Constantinescu wrote: > > > > On 2/14/17 3:04 PM, Barry Smith wrote: >> >>> On Feb 14, 2017, at 2:55 PM, Emil Constantinescu wrote: >>> >>> On 2/14/17 2:33 PM, Zhang, Hong wrote: >>>> I think many users (including me) would like to start with academic examples, e.g. u_t=f(u)+g(u), when they try to learn PETSc TS solvers. This simple form allows for easy switch between all kinds of different integration methods. >>> >>> Right, but then you can just write an if statement along the lines in ex31.c: (this was before RHSJacobian and it's not intended for IMEX) >>> >>> TSGetType(ts,&time_scheme); >>> if ((!strcmp(time_scheme,TSEULER)) || (!strcmp(time_scheme,TSRK)) || (!strcmp(time_scheme,TSSSP))) { >>> /* Explicit time-integration -> specify right-hand side function ydot = f(y) */ >>> TSSetRHSFunction(ts,NULL,RHSFunction,&ptype[0]); >>> } else if ((!strcmp(time_scheme,TSTHETA)) ||... >>> (!strcmp(time_scheme,TSARKIMEX))) { >>> /* Implicit time-integration -> specify left-hand side function ydot-f(y) = 0 */ >>> /* and its Jacobian function */ >>> TSSetIFunction(ts,NULL,IFunction,&ptype[0]); >>> TSSetIJacobian(ts,Jac,Jac,IJacobian,&ptype[0]); >>> } >> >> Multiple yucks! Switching methods should not require compiling code except when absolutely necessary and this is not a case of absolutely necessary! > > Wait, no. You don't recompile. When you set the -ts_type then it defines RHSFunction (for Explicit and Implicit) or RHS Function & IFunction (for IMEX). Ok, you don't recompile but forcing that into user code is still disgusting. With my api the user code is >>> TSSetRHSFunction(ts,NULL,RHSFunction,&ptype[0]); >>> TSSetLHSFunction(ts,NULL,LHSFunction,&ptype[0]); >>> TSSetRHSJacobian(ts,Jac,Jac,RHSJacobian,&ptype[0]); >>> TSSetLHSJacobian(ts,Jac,Jac,LHSJacobian,&ptype[0]); and -ts_type xxx works correctly for ALL methods, implicit, explicit and imex without requiring any special command line options for different methods. > Adding all that logic to keep track of left sides and right sides for academic examples is likely not the best development. I don't think it is "just academic examples", it is all examples without a mass matrix. Once the user has decided with ts_type to use for production if it is fully implicit or explicit then they can depending on the type selected, write just a left hand side, just a right hand side for higher efficiency (less update of ghost points, fewer iterations over loops etc). With a constant mass matrix we can have TSSetMassMatrix() and then TSSetIFunction() is reserved for when it is absolutely needed. Barry > > Emil > >> >>> >>> This should not be a performance bottleneck. IFunction can be a wrapper of the RHSFunction so not too much extra coding. >>> >>> Emil >>> >>>> Experienced users or experts who need to solve DAEs or complicate problems involving nontrivial mass matrix should be encouraged to try IFunction/IJacobian. Of course, we need better documentation to help them realize the switching is not always possible. Personally it took me a long time to get used to the IFunction business since I had been using u_t=f(u)+g(u) for a couple years before I jumped on PETSc. If we add support for this simple form, the learning curve would be less steep than it currently is. >>>> >>>> Hong (Mr.) >>>> >>>>> On Feb 14, 2017, at 11:55 AM, Jed Brown wrote: >>>>> >>>>> Barry Smith writes: >>>>>> Hence my suggestion to have TSSetLeftHandSideFunction() (or Jed's suggestion to have multiple Right Hand side functions) this will allow comparison of implicit, explicit, imex without recompiling (which we don't have currently) for the case with no mass matrix. With a mass matrix, unless we use mass lumping and have a TSSetMassMatrix() you cannot switch to full explicit, but that is due to mathematics, not the interface. >>>>> >>>>> Who in reality has a problem that looks like >>>>> >>>>> u_t = f(u) + g(u) + h(u) >>>>> >>>>> where it may make sense to move some of this to the left (implicit)? >>>>> I'm concerned that this isn't natural for many applications so adding an >>>>> interface would be solving a problem that doesn't really exist outside >>>>> perhaps a few academic examples. >> From bsmith at mcs.anl.gov Tue Feb 14 16:28:59 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 14 Feb 2017 16:28:59 -0600 Subject: [petsc-users] prefix (i.e. cumulative) sum In-Reply-To: References: Message-ID: <5870C39B-CD4B-4133-AD17-444ABD0089BF@mcs.anl.gov> > On Feb 14, 2017, at 3:01 PM, Gideon Simpson wrote: > > I gather I?d use something like this? > > VecGetArray(x, &xarray); > VecGetArray(y, &yarray); > > MPI_Scan(array, array, nglobal, MPIU_SCALAR, MPI_SUM, PETSC_COMM_WORLD); > > VecRestoreArray(x, &xarray); > VecRestoreArray(y, &yarray); This definitely won't work! What do you want to do take a PETSc vector that lies across process and produce a new PETSc vector that lies across processes in the same way but each entry in the new vector contains the sum of the entries "to the left of that entry, plus the current entry" For example [1 3 ] [8 1] becomes [1 4] [12 13]? Where [] represents the values on each process? and you have two processes in this example. Barry > > > -gideon > >> On Feb 14, 2017, at 3:48 PM, Matthew Knepley wrote: >> >> On Tue, Feb 14, 2017 at 2:46 PM, Gideon Simpson wrote: >> Is there a clever way to handle a prefix (cumulative) sum in petsc? I think I can see how to do with using some underlying MPI, but I was wondering if there were some built in routines that could be useful. >> >> It is one MPI call (MPI_Scan), but you could do it by assigning sizes to a PetscLayout (which is how I have done it several places). >> >> Thanks, >> >> Matt >> >> -gideon >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener > From gideon.simpson at gmail.com Tue Feb 14 17:19:37 2017 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Tue, 14 Feb 2017 18:19:37 -0500 Subject: [petsc-users] prefix (i.e. cumulative) sum In-Reply-To: <5870C39B-CD4B-4133-AD17-444ABD0089BF@mcs.anl.gov> References: <5870C39B-CD4B-4133-AD17-444ABD0089BF@mcs.anl.gov> Message-ID: Yes, that?s the kind of thing that I want. -gideon > On Feb 14, 2017, at 5:28 PM, Barry Smith wrote: > > >> On Feb 14, 2017, at 3:01 PM, Gideon Simpson wrote: >> >> I gather I?d use something like this? >> >> VecGetArray(x, &xarray); >> VecGetArray(y, &yarray); >> >> MPI_Scan(array, array, nglobal, MPIU_SCALAR, MPI_SUM, PETSC_COMM_WORLD); >> >> VecRestoreArray(x, &xarray); >> VecRestoreArray(y, &yarray); > > This definitely won't work! What do you want to do take a PETSc vector that lies across process and produce a new PETSc vector that lies across processes in the same way but each entry in the new vector contains the sum of the entries "to the left of that entry, plus the current entry" For example > > [1 3 ] [8 1] > > becomes > > [1 4] [12 13]? > > Where [] represents the values on each process? and you have two processes in this example. > > Barry > > > >> >> >> -gideon >> >>> On Feb 14, 2017, at 3:48 PM, Matthew Knepley wrote: >>> >>> On Tue, Feb 14, 2017 at 2:46 PM, Gideon Simpson wrote: >>> Is there a clever way to handle a prefix (cumulative) sum in petsc? I think I can see how to do with using some underlying MPI, but I was wondering if there were some built in routines that could be useful. >>> >>> It is one MPI call (MPI_Scan), but you could do it by assigning sizes to a PetscLayout (which is how I have done it several places). >>> >>> Thanks, >>> >>> Matt >>> >>> -gideon >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From elbueler at alaska.edu Tue Feb 14 19:05:34 2017 From: elbueler at alaska.edu (Ed Bueler) Date: Tue, 14 Feb 2017 16:05:34 -0900 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: <87inoc4l17.fsf@jedbrown.org> References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> <60D7CDCC-E9D5-472A-9175-64C56DF8B34F@mcs.anl.gov> <87vasc4pz5.fsf@jedbrown.org> <6D880F3B-F880-4AFD-975D-5A8A4F4AA8A9@mcs.anl.gov> <87inoc4l17.fsf@jedbrown.org> Message-ID: Jed -- > Yup, this is the form I have, > > u_t - f(u) = g(u) > Are these really separate terms? As I mentioned, a very common case is > to write > u_t - L u = f(u) - L u > where L is some stiff part of the operator (linear or non). This is > common for stiff waves in fluid problems or stiff reaction terms in an > otherwise slower network, for example. In this case, the obvious thing > is for the user to provide their own run-time options for what to > evaluate in the IFunction versus RHSFunction. If everything goes in the > RHSFunction, then just evaluate f(u) rather than f(u) - L u + L u. Your question makes me think about why I am splitting the way I am. For sure *yes*, in my case they are separate terms and *no* I am not just subtracting Lu from both sides. The ice sheet thickness problem (u(t,x,y) >= 0 is thickness) is like degenerate p-bratu, right? My form is like u_t - div (u^k |grad u|^{p-2} grad u) = g(t,u) where the right side is only the elevation-dependent surface mass balance in my context. The right-hand stuff is (potentially) coupling to other models which will be time-split, and provided by other codes, so the "ultimate" coupled model will inevitably be imex in some vague sense. The stiff stuff on the left should be handled implicitly both for imex and implicit, of course. And I want to see the solver robustness effects of my imex splitting of the problem. I also want to compare actual explicit runs despite their absurd short time step. In any case, whatever my motivation, I am currently providing RHSFunction = g(t,u) IFunction = u_t - div (u^k |grad u|^{p-2} grad u) Currently ARKIMEX, THETA, BDF all seem to work well, but explicit breaks. As I understand it, Barry's PR will "fix" this by erroring-out when I try to run it explicitly. This is fine until ya'll work out a better API. > What "fragile code"? I mean I fear the prospect of a PETSc API which requires so conditionals determining which "side"' various terms appear on, depending on user code testing what method is running etc.. No such thing for now; my current code is not fragile. (Is an API based on "side" of an equation inherently suspect?) Ed On Tue, Feb 14, 2017 at 10:41 AM, Jed Brown wrote: > Ed Bueler writes: > > Yup, this is the form I have, > > u_t - f(u) = g(u) > > Are these really separate terms? As I mentioned, a very common case is > to write > > u_t - L u = f(u) - L u > > where L is some stiff part of the operator (linear or non). This is > common for stiff waves in fluid problems or stiff reaction terms in an > otherwise slower network, for example. In this case, the obvious thing > is for the user to provide their own run-time options for what to > evaluate in the IFunction versus RHSFunction. If everything goes in the > RHSFunction, then just evaluate f(u) rather than f(u) - L u + L u. > > Obviously you could easily do this in your code, you just didn't. But > it's clearly more powerful to offer your own options. Now my question > is whether it's really more natural to implement f(u) and g(u) > separately and have PETSc move the terms around. If that really saves > you work or makes something more robust, then I'm not opposed to adding > it. > > But while offering f(u), g(u) may save you a few lines of code today, it > greatly increases the incremental complexity of doing more intrusive > splitting. I'm concerned that we add this code to PETSc and then people > don't think to use such splittings because there can't be a PETSc > interface to do it. Or they waste a lot of communication and grid > traversal overhead to evaluate a bunch of terms independently instead of > together. > > > and Barry is describing my situation with ice sheets. I am trying to > > escape the stupidly-obvious small time step restriction of explicit ... > but > > only the full stack > > (ARKIMEX_or_BDF+SNESVI+a_yet_unknown_but_needs_to_be_good_ > PC_combination) > > has a chance of actually beating explicit performance-wise because of > > extreme low regularity arising from constraint/free-boundary. To tell if > > it is working I would really like to be able to switch from -ts_type rk > to > > -ts_type arkimex to -ts_type bdf without writing fragile code. > > What "fragile code"? > -- Ed Bueler Dept of Math and Stat and Geophysical Institute University of Alaska Fairbanks Fairbanks, AK 99775-6660 301C Chapman and 410D Elvey 907 474-7693 and 907 474-7199 (fax 907 474-5394) -------------- next part -------------- An HTML attachment was scrubbed... URL: From bernardomartinsrocha at gmail.com Tue Feb 14 19:17:55 2017 From: bernardomartinsrocha at gmail.com (Bernardo Rocha) Date: Tue, 14 Feb 2017 23:17:55 -0200 Subject: [petsc-users] PC HYPRE BoomerAMG options for nodal In-Reply-To: <4032E7BA-A557-4491-B4E1-71F42E13DBA6@mcs.anl.gov> References: <4032E7BA-A557-4491-B4E1-71F42E13DBA6@mcs.anl.gov> Message-ID: Thanks a lot for the reply. ?From my understanding and specifically for my problem I would have to use the AMGSetDofFunc as follows:? ?nsyseq = 3;? for (i=0; i HYPRE_BoomerAMGSetDofFunc > > I could not find an indication of that this is suppose to be setting; > it is mentioned in the users manual but I don't know what it means. It > looks like it takes an integer array but I don't even know how long that > array is. > > If you figure out what it means and can add support for it we'll take > it as a pull request. ??OK Barry, I will try to specify the block size of the matrix. But I'm not sure if the block size would be 3 or N (number of nodes). This is my ordering of the displacements: [ux1, ux2, ..., uxN, uy1, uy2,...uyN, uz1, uz2, ..., uzN] > > HYPRE_BoomerAMGSetNumFunctions > > Set the block size of the matrix; this information is then transferred > automatically to hypre. Note that you can set a block size even for AIJ > matrices. > ? ?Best regards, Bernardo? -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Feb 14 19:48:15 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 14 Feb 2017 19:48:15 -0600 Subject: [petsc-users] PC HYPRE BoomerAMG options for nodal In-Reply-To: References: <4032E7BA-A557-4491-B4E1-71F42E13DBA6@mcs.anl.gov> Message-ID: <8B1C4F5C-BC22-4476-ACAF-A0C105768E0B@mcs.anl.gov> Ahh, thanks for the explanation. What a nutso API: way to general, they should just support interlaced and non-interlaced. Anyways we strongly recommend using the interlaced version; for most calculations it gives much better locality for cache lines and cache. We always use interlaced. Any particular reason you use non-interlaced? Barry > On Feb 14, 2017, at 7:17 PM, Bernardo Rocha wrote: > > Thanks a lot for the reply. > > ?From my understanding and specifically for my problem I would have to use the AMGSetDofFunc as follows:? > > ?nsyseq = 3;? > for (i=0; i ? ?for (k=0; k { > ? ?eqnIndex[i*?N?+k] = i; > } > HYPRE_BoomerAMGSetDofFunc(preconditioner, eqnIndex);? > > For a system of 3 PDEs (e.g. linear elasticity in 3D?) the default behaviour of the nodal coarsening in BoomerAMG, considers an array that is filled with > > eqnIndex = ?0 1 2 0 1 2 0 1 2....0 1 2 > > > HYPRE_BoomerAMGSetDofFunc > > I could not find an indication of that this is suppose to be setting; it is mentioned in the users manual but I don't know what it means. It looks like it takes an integer array but I don't even know how long that array is. > > If you figure out what it means and can add support for it we'll take it as a pull request. > > ??OK Barry, I will try to specify the block size of the matrix. > But I'm not sure if the block size would be 3 or N (number of nodes). > This is my ordering of the displacements: [ux1, ux2, ..., uxN, uy1, uy2,...uyN, uz1, uz2, ..., uzN] > > > HYPRE_BoomerAMGSetNumFunctions > > Set the block size of the matrix; this information is then transferred automatically to hypre. Note that you can set a block size even for AIJ matrices. > ? > ?Best regards, > Bernardo? From bernardomartinsrocha at gmail.com Tue Feb 14 19:57:13 2017 From: bernardomartinsrocha at gmail.com (Bernardo Rocha) Date: Tue, 14 Feb 2017 23:57:13 -0200 Subject: [petsc-users] PC HYPRE BoomerAMG options for nodal In-Reply-To: <8B1C4F5C-BC22-4476-ACAF-A0C105768E0B@mcs.anl.gov> References: <4032E7BA-A557-4491-B4E1-71F42E13DBA6@mcs.anl.gov> <8B1C4F5C-BC22-4476-ACAF-A0C105768E0B@mcs.anl.gov> Message-ID: No particular reason. Let's say it was the first approach I tried in an old code. OK, so to get it working I should use the interlaced version for the ordering of my dofs. It does not require too much coding on my side. Just to know, if I wanted to insist in the non-interlaced, to keep using Hypre/BoomerAMG from PETSc I would need to write some code in PETSc for wrapping this feature (AMGSetDofFunc), right? Thanks. On Tue, Feb 14, 2017 at 11:48 PM, Barry Smith wrote: > > Ahh, thanks for the explanation. What a nutso API: way to general, they > should just support interlaced and non-interlaced. > > Anyways we strongly recommend using the interlaced version; for most > calculations it gives much better locality for cache lines and cache. We > always use interlaced. Any particular reason you use non-interlaced? > > Barry > > > On Feb 14, 2017, at 7:17 PM, Bernardo Rocha com> wrote: > > > > Thanks a lot for the reply. > > > > ?From my understanding and specifically for my problem I would have to > use the AMGSetDofFunc as follows:? > > > > ?nsyseq = 3;? > > for (i=0; i > ? ?for (k=0; k > { > > ? ?eqnIndex[i*?N?+k] = i; > > } > > HYPRE_BoomerAMGSetDofFunc(preconditioner, eqnIndex);? > > > > For a system of 3 PDEs (e.g. linear elasticity in 3D?) the default > behaviour of the nodal coarsening in BoomerAMG, considers an array that is > filled with > > > > eqnIndex = ?0 1 2 0 1 2 0 1 2....0 1 2 > > > > > HYPRE_BoomerAMGSetDofFunc > > > > I could not find an indication of that this is suppose to be setting; > it is mentioned in the users manual but I don't know what it means. It > looks like it takes an integer array but I don't even know how long that > array is. > > > > If you figure out what it means and can add support for it we'll take > it as a pull request. > > > > ??OK Barry, I will try to specify the block size of the matrix. > > But I'm not sure if the block size would be 3 or N (number of nodes). > > This is my ordering of the displacements: [ux1, ux2, ..., uxN, uy1, > uy2,...uyN, uz1, uz2, ..., uzN] > > > > > HYPRE_BoomerAMGSetNumFunctions > > > > Set the block size of the matrix; this information is then > transferred automatically to hypre. Note that you can set a block size > even for AIJ matrices. > > ? > > ?Best regards, > > Bernardo? > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Feb 14 20:05:59 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 14 Feb 2017 20:05:59 -0600 Subject: [petsc-users] PETSc set off-diagonal matrix pre-allocation In-Reply-To: References: Message-ID: 1) on a single process all the columns are in the diagonal block and none in the off diagonal block (note that the column partitioning corresponding to a partitioning of the vector in the product A *x 2) on two processes you guessed wrong how many columns PETSc would put on the first process. You guessed it would put the first three on the first process and the last three on the second process. a) it cannot do that, every column has to been owned by some process (in the vector x above) so it cannot be 3 and 3, it has to be 4 and 3 or 3 and 4. b) PETSc puts the "extra" columns on the earlier processes not the later processes. For rectangular matrices you really cannot get away with using "PETSC_DECIDE" for local columns when using preallocation -------------- next part -------------- A non-text attachment was scrubbed... Name: ex3.c Type: application/octet-stream Size: 1678 bytes Desc: not available URL: -------------- next part -------------- , since it may decide something different than what you assume. I've attached the parallel code that behaves as expected. Barry > On Feb 14, 2017, at 1:06 PM, Andrew Ho wrote: > > The problem isn't only with 1 process, but it seems to be with all non-square matrices? > > For example, here I have a two process program which tries to set a 6x7 matrix to (each process owns 3 rows): > > 0 0 0 1 1 1 1 > 0 0 0 1 1 1 1 > 0 0 0 1 1 1 1 > 0 0 0 1 1 1 1 > 0 0 0 1 1 1 1 > 0 0 0 1 1 1 1 > > #include > #include > > int main(int argc, char** argv) > { > PetscErrorCode err; > err = PetscInitialize(&argc, &argv, NULL, "help"); > CHKERRQ(err); > > int rank, size; > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > MPI_Comm_size(MPI_COMM_WORLD, &size); > if(size != 2) > { > printf("must run with 2 processes"); > MPI_Abort(MPI_COMM_WORLD, -1); > } > > // create a sparse AIJ matrix distributed across MPI > Mat A; > err = MatCreate(MPI_COMM_WORLD, &A); > CHKERRQ(err); > err = MatSetType(A, MATMPIAIJ); > CHKERRQ(err); > // setup pre-allocation for matrix space > { > err = > MatSetSizes(A, 3, PETSC_DECIDE, 6, 7); > CHKERRQ(err); > if(rank == 0) > { > PetscInt d_nnz[] = {0, 0, 0}; > PetscInt o_nnz[] = {4, 4, 4}; > err = MatMPIAIJSetPreallocation(A, 0, d_nnz, 0, o_nnz); > CHKERRQ(err); > } > else > { > PetscInt d_nnz[] = {3, 3, 3}; > PetscInt o_nnz[] = {1, 1, 1}; > err = MatMPIAIJSetPreallocation(A, 0, d_nnz, 0, o_nnz); > CHKERRQ(err); > } > } > err = MatSetUp(A); > CHKERRQ(err); > > // set values inside the matrix > for (PetscInt row = 0; row < 3; ++row) > { > for (PetscInt col = 3; col < 7; ++col) > { > err = MatSetValue(A, 3 * rank + row, col, 1, INSERT_VALUES); > CHKERRQ(err); > } > } > > err = MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); > CHKERRQ(err); > err = MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); > CHKERRQ(err); > > err = MatView(A, PETSC_VIEWER_STDOUT_WORLD); > CHKERRQ(err); > > // free memory > err = MatDestroy(&A); > CHKERRQ(err); > > // cleanup any internal PETSc data at end of program > err = PetscFinalize(); > CHKERRQ(err); > } > > > On Tue, Feb 14, 2017 at 5:26 AM, Matthew Knepley wrote: > On Tue, Feb 14, 2017 at 6:41 AM, Andrew Ho wrote: > I have a 3x6 matrix, and I want to set it to (just as an example): > > 0 0 0 1 1 1 > 0 0 0 1 1 1 > 0 0 0 1 1 1 > > From what I can tell, according to the documentation the MPIAIJ sparsity of this matrix is: > > I believe this is an inconsistency in PETSc for rectangular matrices. We divide matrices into diagonal and > off-diagonal parts mainly to separate communication from computation. Thus on 1 proc, we never divide > them, even if the matrix is rectangular. Therefore we misinterpret your preallocation. > > Barry, do you think we want to try and change this? > > Matt > > d_nnz = [0, 0, 0] > o_nnz = [3, 3, 3] > > However, when I do this, I get the following error: > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Argument out of range > [0]PETSC ERROR: New nonzero at (0,3) caused a malloc > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.7.5, unknown > [0]PETSC ERROR: ./ex3 on a arch-linux2-c-opt Tue Feb 14 04:23:56 2017 > [0]PETSC ERROR: Configure options --with-debugging=0 --COPTFLAGS="-O3 -march=native" --CXXOPTFLAGS="-O3 -march=native" --FOPTFLAGS="-O3 -march=native" > [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 582 in petsc/src/mat/impls/aij/mpi/mpiaij.c > [0]PETSC ERROR: #2 MatSetValues() line 1190 in petsc/src/mat/interface/matrix.c > [0]PETSC ERROR: #3 main() line 36 in ex3.c > [0]PETSC ERROR: No PETSc Option Table entries > [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- > > Here's a working test code: > > #include > #include > int main(int argc, char** argv) > { > PetscErrorCode err; > err = PetscInitialize(&argc, &argv, NULL, "help"); > CHKERRQ(err); > // create a sparse AIJ matrix distributed across MPI > PetscInt global_width = 6; > PetscInt global_height = 3; > Mat A; > err = MatCreate(MPI_COMM_WORLD, &A); > CHKERRQ(err); > err = MatSetType(A, MATMPIAIJ); > CHKERRQ(err); > // setup pre-allocation for matrix space > { > err = > MatSetSizes(A, global_height, PETSC_DECIDE, global_height, global_width); > CHKERRQ(err); > PetscInt d_nnz[] = {0, 0, 0}; > PetscInt o_nnz[] = {3, 3, 3}; > err = MatMPIAIJSetPreallocation(A, 0, d_nnz, 0, o_nnz); > CHKERRQ(err); > } > err = MatSetUp(A); > CHKERRQ(err); > // set values inside the matrix > for (PetscInt row = 0; row < global_height; ++row) > { > for (PetscInt col = global_height; col < global_width; ++col) > { > err = MatSetValue(A, row, col, 1, INSERT_VALUES); > CHKERRQ(err); > } > } > err = MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); > CHKERRQ(err); > err = MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); > CHKERRQ(err); > err = MatView(A, PETSC_VIEWER_STDOUT_WORLD); > CHKERRQ(err); > // free memory > err = MatDestroy(&A); > CHKERRQ(err); > // cleanup any internal PETSc data at end of program > err = PetscFinalize(); > CHKERRQ(err); > } > > Am I mis-understanding what the d_nnz and o_nnz parameter are supposed to mean? > -- > Andrew Ho > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > > > -- > Andrew Ho From bsmith at mcs.anl.gov Tue Feb 14 20:07:51 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 14 Feb 2017 20:07:51 -0600 Subject: [petsc-users] PC HYPRE BoomerAMG options for nodal In-Reply-To: References: <4032E7BA-A557-4491-B4E1-71F42E13DBA6@mcs.anl.gov> <8B1C4F5C-BC22-4476-ACAF-A0C105768E0B@mcs.anl.gov> Message-ID: <1FB1E38B-0C7F-4759-A7F1-C2365A9C7FBB@mcs.anl.gov> > On Feb 14, 2017, at 7:57 PM, Bernardo Rocha wrote: > > No particular reason. Let's say it was the first approach I tried in an old code. > > OK, so to get it working I should use the interlaced version for the ordering of my dofs. > It does not require too much coding on my side. > > Just to know, if I wanted to insist in the non-interlaced, to keep using Hypre/BoomerAMG from PETSc I would need to write some code in PETSc for wrapping this feature (AMGSetDofFunc), right? > > Thanks. > > On Tue, Feb 14, 2017 at 11:48 PM, Barry Smith wrote: > > Ahh, thanks for the explanation. What a nutso API: way to general, they should just support interlaced and non-interlaced. > > Anyways we strongly recommend using the interlaced version; for most calculations it gives much better locality for cache lines and cache. We always use interlaced. Any particular reason you use non-interlaced? Yes, you would have to add a call to HYPRE_BoomerAMGSetDofFunc() with the information as you computed. For trying things out the easily way would be to just stick it in an appropriate place in hypre.c Barry > > Barry > > > On Feb 14, 2017, at 7:17 PM, Bernardo Rocha wrote: > > > > Thanks a lot for the reply. > > > > ?From my understanding and specifically for my problem I would have to use the AMGSetDofFunc as follows:? > > > > ?nsyseq = 3;? > > for (i=0; i > ? ?for (k=0; k > { > > ? ?eqnIndex[i*?N?+k] = i; > > } > > HYPRE_BoomerAMGSetDofFunc(preconditioner, eqnIndex);? > > > > For a system of 3 PDEs (e.g. linear elasticity in 3D?) the default behaviour of the nodal coarsening in BoomerAMG, considers an array that is filled with > > > > eqnIndex = ?0 1 2 0 1 2 0 1 2....0 1 2 > > > > > HYPRE_BoomerAMGSetDofFunc > > > > I could not find an indication of that this is suppose to be setting; it is mentioned in the users manual but I don't know what it means. It looks like it takes an integer array but I don't even know how long that array is. > > > > If you figure out what it means and can add support for it we'll take it as a pull request. > > > > ??OK Barry, I will try to specify the block size of the matrix. > > But I'm not sure if the block size would be 3 or N (number of nodes). > > This is my ordering of the displacements: [ux1, ux2, ..., uxN, uy1, uy2,...uyN, uz1, uz2, ..., uzN] > > > > > HYPRE_BoomerAMGSetNumFunctions > > > > Set the block size of the matrix; this information is then transferred automatically to hypre. Note that you can set a block size even for AIJ matrices. > > ? > > ?Best regards, > > Bernardo? > > From bsmith at mcs.anl.gov Tue Feb 14 20:46:42 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 14 Feb 2017 20:46:42 -0600 Subject: [petsc-users] prefix (i.e. cumulative) sum In-Reply-To: References: <5870C39B-CD4B-4133-AD17-444ABD0089BF@mcs.anl.gov> Message-ID: <493CDDC7-ACCC-4F9A-82AF-5DBA801CDB28@mcs.anl.gov> Here you go. In 20+ years of PETSc no one has needed this before. -------------- next part -------------- A non-text attachment was scrubbed... Name: ex4.c Type: application/octet-stream Size: 2062 bytes Desc: not available URL: -------------- next part -------------- > On Feb 14, 2017, at 5:19 PM, Gideon Simpson wrote: > > Yes, that?s the kind of thing that I want. > > -gideon > >> On Feb 14, 2017, at 5:28 PM, Barry Smith wrote: >> >> >>> On Feb 14, 2017, at 3:01 PM, Gideon Simpson wrote: >>> >>> I gather I?d use something like this? >>> >>> VecGetArray(x, &xarray); >>> VecGetArray(y, &yarray); >>> >>> MPI_Scan(array, array, nglobal, MPIU_SCALAR, MPI_SUM, PETSC_COMM_WORLD); >>> >>> VecRestoreArray(x, &xarray); >>> VecRestoreArray(y, &yarray); >> >> This definitely won't work! What do you want to do take a PETSc vector that lies across process and produce a new PETSc vector that lies across processes in the same way but each entry in the new vector contains the sum of the entries "to the left of that entry, plus the current entry" For example >> >> [1 3 ] [8 1] >> >> becomes >> >> [1 4] [12 13]? >> >> Where [] represents the values on each process? and you have two processes in this example. >> >> Barry >> >> >> >>> >>> >>> -gideon >>> >>>> On Feb 14, 2017, at 3:48 PM, Matthew Knepley wrote: >>>> >>>> On Tue, Feb 14, 2017 at 2:46 PM, Gideon Simpson wrote: >>>> Is there a clever way to handle a prefix (cumulative) sum in petsc? I think I can see how to do with using some underlying MPI, but I was wondering if there were some built in routines that could be useful. >>>> >>>> It is one MPI call (MPI_Scan), but you could do it by assigning sizes to a PetscLayout (which is how I have done it several places). >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> -gideon >>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>> -- Norbert Wiener >>> >> > From emconsta at mcs.anl.gov Tue Feb 14 20:56:46 2017 From: emconsta at mcs.anl.gov (Emil Constantinescu) Date: Tue, 14 Feb 2017 20:56:46 -0600 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: <9C507CBA-8DC4-4669-BEE7-4560E27EB020@mcs.anl.gov> References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> <60D7CDCC-E9D5-472A-9175-64C56DF8B34F@mcs.anl.gov> <87vasc4pz5.fsf@jedbrown.org> <36FFCE9F-ECA8-4B5F-BE53-6B1AD2C7B205@anl.gov> <95c8d53c-5c77-34aa-68dd-3e5211d24c24@mcs.anl.gov> <45E4D3E5-B4DC-4BDC-984A-ED7735723E50@mcs.anl.gov> <03329c72-3572-9f71-5e5b-34741f5d7573@mcs.anl.gov> <9C507CBA-8DC4-4669-BEE7-4560E27EB020@mcs.anl.gov> Message-ID: On 2/14/17 4:10 PM, Barry Smith wrote: > Ok, you don't recompile but forcing that into user code is still disgusting. With my api the user code is > >>>> TSSetRHSFunction(ts,NULL,RHSFunction,&ptype[0]); >>>> TSSetLHSFunction(ts,NULL,LHSFunction,&ptype[0]); >>>> TSSetRHSJacobian(ts,Jac,Jac,RHSJacobian,&ptype[0]); >>>> TSSetLHSJacobian(ts,Jac,Jac,LHSJacobian,&ptype[0]); > and -ts_type xxx works correctly for ALL methods, implicit, explicit and imex without requiring any special command line options for different methods. Is this a viable solution? Growing the API to fix this situation will just put a burden with each new TS method after we refactor it in the current landscape. If the user experiments with different ways of splitting the solution they would have to define RHS and IF or RHS and LHS in different ways (according to the splittings they experiment with). It may look disgusting, but I don't see another way around it unless you allow for a list of operators to be defined and then the user to assign them to LHS or RHS. > >> Adding all that logic to keep track of left sides and right sides for academic examples is likely not the best development. > I don't think it is "just academic examples", it is all examples without a mass matrix. > > Once the user has decided with ts_type to use for production if it is fully implicit or explicit then they can depending on the type selected, write just a left hand side, just a right hand side for higher efficiency (less update of ghost points, fewer iterations over loops etc). > > With a constant mass matrix we can have TSSetMassMatrix() and then TSSetIFunction() is reserved for when it is absolutely needed. As much as I would disagree with growing the API at the level of defining the problem, I think TSSetMassMatrix() would let us do more things in the solvers. Also it would be useful to know if the mass matrix is singular or not for efficiency reasons. Emil > Barry > From jed at jedbrown.org Tue Feb 14 21:18:42 2017 From: jed at jedbrown.org (Jed Brown) Date: Tue, 14 Feb 2017 20:18:42 -0700 Subject: [petsc-users] prefix (i.e. cumulative) sum In-Reply-To: <493CDDC7-ACCC-4F9A-82AF-5DBA801CDB28@mcs.anl.gov> References: <5870C39B-CD4B-4133-AD17-444ABD0089BF@mcs.anl.gov> <493CDDC7-ACCC-4F9A-82AF-5DBA801CDB28@mcs.anl.gov> Message-ID: <877f4s3zvx.fsf@jedbrown.org> Barry Smith writes: > Here you go. In 20+ years of PETSc no one has needed this before. This is disgusting O(P) critical path. Do it in O(log P): int mysum=0,mystart; for (i=0; i From bsmith at mcs.anl.gov Tue Feb 14 21:20:37 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 14 Feb 2017 21:20:37 -0600 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> <60D7CDCC-E9D5-472A-9175-64C56DF8B34F@mcs.anl.gov> <87vasc4pz5.fsf@jedbrown.org> <36FFCE9F-ECA8-4B5F-BE53-6B1AD2C7B205@anl.gov> <95c8d53c-5c77-34aa-68dd-3e5211d24c24@mcs.anl.gov> <45E4D3E5-B4DC-4BDC-984A-ED7735723E50@mcs.anl.gov> <03329c72-3572-9f71-5e5b-34741f5d7573@mcs.anl.gov> <9C507CBA-8DC4-4669-BEE7-4560E27EB020@mcs.anl.gov> Message-ID: <110C81DD-DA04-48A4-82B4-F9E527FA3806@mcs.anl.gov> > On Feb 14, 2017, at 8:56 PM, Emil Constantinescu wrote: > > On 2/14/17 4:10 PM, Barry Smith wrote: >> Ok, you don't recompile but forcing that into user code is still disgusting. With my api the user code is >> >>>>> TSSetRHSFunction(ts,NULL,RHSFunction,&ptype[0]); >>>>> TSSetLHSFunction(ts,NULL,LHSFunction,&ptype[0]); >>>>> TSSetRHSJacobian(ts,Jac,Jac,RHSJacobian,&ptype[0]); >>>>> TSSetLHSJacobian(ts,Jac,Jac,LHSJacobian,&ptype[0]); >> and -ts_type xxx works correctly for ALL methods, implicit, explicit and imex without requiring any special command line options for different methods. > > Is this a viable solution? Growing the API to fix this situation will just put a burden with each new TS method after we refactor it in the current landscape. No just the opposite, the TS implementations will talk to functions who will put things together for it. So All implicit methods will call something like TSBuildImplicitFunction(), all explicit methods will call something like TSBuildExplicitFunction() and then IMEX methods will call both of these. In fact likely we can refactor to make things a little better than today. Depending on options and flagsTSBuildExplicitFunction() would build out of all the user provided functions what it needs etc. One problem with the current code is the TS methods call things with the same names as the user API. So implicit methods call TSComputeIFunction() while explicit methods call TSComputeRHSFunction(). This is wrong because implicit methods actually also use the rhs function provided by the user. The function below absolutely should not be called TSComputeIFunction()! It does not just compute IFunction() PetscErrorCode TSComputeIFunction(TS ts,PetscReal t,Vec U,Vec Udot,Vec Y,PetscBool imex) { PetscErrorCode ierr; TSIFunction ifunction; TSRHSFunction rhsfunction; void *ctx; DM dm; PetscFunctionBegin; PetscValidHeaderSpecific(ts,TS_CLASSID,1); PetscValidHeaderSpecific(U,VEC_CLASSID,3); PetscValidHeaderSpecific(Udot,VEC_CLASSID,4); PetscValidHeaderSpecific(Y,VEC_CLASSID,5); ierr = TSGetDM(ts,&dm);CHKERRQ(ierr); ierr = DMTSGetIFunction(dm,&ifunction,&ctx);CHKERRQ(ierr); ierr = DMTSGetRHSFunction(dm,&rhsfunction,NULL);CHKERRQ(ierr); if (!rhsfunction && !ifunction) SETERRQ(PetscObjectComm((PetscObject)ts),PETSC_ERR_USER,"Must call TSSetRHSFunction() and / or TSSetIFunction()"); ierr = PetscLogEventBegin(TS_FunctionEval,ts,U,Udot,Y);CHKERRQ(ierr); if (ifunction) { PetscStackPush("TS user implicit function"); ierr = (*ifunction)(ts,t,U,Udot,Y,ctx);CHKERRQ(ierr); PetscStackPop; } if (imex) { if (!ifunction) { ierr = VecCopy(Udot,Y);CHKERRQ(ierr); } } else if (rhsfunction) { if (ifunction) { Vec Frhs; ierr = TSGetRHSVec_Private(ts,&Frhs);CHKERRQ(ierr); ierr = TSComputeRHSFunction(ts,t,U,Frhs);CHKERRQ(ierr); ierr = VecAXPY(Y,-1,Frhs);CHKERRQ(ierr); } else { ierr = TSComputeRHSFunction(ts,t,U,Y);CHKERRQ(ierr); ierr = VecAYPX(Y,-1,Udot);CHKERRQ(ierr); } } ierr = PetscLogEventEnd(TS_FunctionEval,ts,U,Udot,Y);CHKERRQ(ierr); PetscFunctionReturn(0); } The current code entangles too much of the user API to the methods, this can be fixed. > If the user experiments with different ways of splitting the solution they would have to define RHS and IF or RHS and LHS in different ways (according to the splittings they experiment with). It may look disgusting, but I don't see another way around it unless you allow for a list of operators to be defined and then the user to assign them to LHS or RHS. Jed suggested having any number of "RHS" functions. I don't think we need more than 2, 1 for left hand side and 1 for right. If that ends up being not enough we can change to have any number of them. Just to be clear. I suggest three functions IFunction which defaults to u_t plus TSSetMassMatrix() which changes the default IFunction to M u_t LHS function which defaults to 0, if provided defaults to being treated implicitly by IMEX RHS function which defaults to 0, if provided defaults to being treated explicitly by IMEX Then a TSSetStiffMatrix(ts,Mat L) (horrible name) that provides u_t -Lu = g() + Lu None of the TS implementations will every directly know about what the user has provided. They will call the wrapper functions I mention above. I think Jed and Emil may be too enamored with the reductionist model of only IFunction() and RHSFunction() to see that though it encompasses everything it may not be the best user API. > >> >>> Adding all that logic to keep track of left sides and right sides for academic examples is likely not the best development. >> I don't think it is "just academic examples", it is all examples without a mass matrix. >> >> Once the user has decided with ts_type to use for production if it is fully implicit or explicit then they can depending on the type selected, write just a left hand side, just a right hand side for higher efficiency (less update of ghost points, fewer iterations over loops etc). >> >> With a constant mass matrix we can have TSSetMassMatrix() and then TSSetIFunction() is reserved for when it is absolutely needed. > > As much as I would disagree with growing the API at the level of defining the problem, I think TSSetMassMatrix() would let us do more things in the solvers. Also it would be useful to know if the mass matrix is singular or not for efficiency reasons. > > Emil > >> Barry >> From bsmith at mcs.anl.gov Tue Feb 14 21:24:25 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 14 Feb 2017 21:24:25 -0600 Subject: [petsc-users] prefix (i.e. cumulative) sum In-Reply-To: <877f4s3zvx.fsf@jedbrown.org> References: <5870C39B-CD4B-4133-AD17-444ABD0089BF@mcs.anl.gov> <493CDDC7-ACCC-4F9A-82AF-5DBA801CDB28@mcs.anl.gov> <877f4s3zvx.fsf@jedbrown.org> Message-ID: Ahh yes, a million times better (for appropriately sized P). Thanks > On Feb 14, 2017, at 9:18 PM, Jed Brown wrote: > > Barry Smith writes: > >> Here you go. In 20+ years of PETSc no one has needed this before. > > This is disgusting O(P) critical path. Do it in O(log P): > > int mysum=0,mystart; > for (i=0; i mysum += xx[i]; > } > MPI_Scan(&mysum,&mystart,1,MPI_INT,MPI_SUM,comm); > mystart -= mysum; > xx[0] += mystart; > for (i=1; i xx[i] += xx[i-1]; > } From jed at jedbrown.org Tue Feb 14 21:36:37 2017 From: jed at jedbrown.org (Jed Brown) Date: Tue, 14 Feb 2017 20:36:37 -0700 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: <110C81DD-DA04-48A4-82B4-F9E527FA3806@mcs.anl.gov> References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> <60D7CDCC-E9D5-472A-9175-64C56DF8B34F@mcs.anl.gov> <87vasc4pz5.fsf@jedbrown.org> <36FFCE9F-ECA8-4B5F-BE53-6B1AD2C7B205@anl.gov> <95c8d53c-5c77-34aa-68dd-3e5211d24c24@mcs.anl.gov> <45E4D3E5-B4DC-4BDC-984A-ED7735723E50@mcs.anl.gov> <03329c72-3572-9f71-5e5b-34741f5d7573@mcs.anl.gov> <9C507CBA-8DC4-4669-BEE7-4560E27EB020@mcs.anl.gov> <110C81DD-DA04-48A4-82B4-F9E527FA3806@mcs.anl.gov> Message-ID: <871sv03z22.fsf@jedbrown.org> Barry Smith writes: > The function below absolutely should not be called TSComputeIFunction()! It does not just compute IFunction() How is this different than SNESComputeFunction subtracting off the RHS? > The current code entangles too much of the user API to the methods, this can be fixed. I agree that the logic/naming is perhaps too overloaded. >> If the user experiments with different ways of splitting the solution they would have to define RHS and IF or RHS and LHS in different ways (according to the splittings they experiment with). It may look disgusting, but I don't see another way around it unless you allow for a list of operators to be defined and then the user to assign them to LHS or RHS. > > Jed suggested having any number of "RHS" functions. I don't think we need more than 2, 1 for left hand side and 1 for right. If that ends up being not enough we can change to have any number of them. Just to be clear. I suggest three functions > > IFunction which defaults to u_t plus TSSetMassMatrix() which changes the default IFunction to M u_t > LHS function which defaults to 0, if provided defaults to being treated implicitly by IMEX > RHS function which defaults to 0, if provided defaults to being treated explicitly by IMEX Are you trolling me? Reactive fluid-structure interaction: 1. fluid splitting to isolate the acoustic wave (stiff) 2. fluid containing everything but the acoustic wave (non-stiff) 3. fast reactions 4. slow reactions 5. structure Note that when using conservative variables, every one of these splits requires evaluation of the equation of state (reactions rate coefficients depend on Temperature, structure boundary condition requires pressure and shear stress). This doesn't even have viscoelastics, sedimentation, or a turbulence model. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From bsmith at mcs.anl.gov Tue Feb 14 21:54:29 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 14 Feb 2017 21:54:29 -0600 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: <871sv03z22.fsf@jedbrown.org> References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> <60D7CDCC-E9D5-472A-9175-64C56DF8B34F@mcs.anl.gov> <87vasc4pz5.fsf@jedbrown.org> <36FFCE9F-ECA8-4B5F-BE53-6B1AD2C7B205@anl.gov> <95c8d53c-5c77-34aa-68dd-3e5211d24c24@mcs.anl.gov> <45E4D3E5-B4DC-4BDC-984A-ED7735723E50@mcs.anl.gov> <03329c72-3572-9f71-5e5b-34741f5d7573@mcs.anl.gov> <9C507CBA-8DC4-4669-BEE7-4560E27EB020@mcs.anl.gov> <110C81DD-DA04-48A4-82B4-F9E527FA3806@mcs.anl.gov> <871sv03z22.fsf@jedbrown.org> Message-ID: > On Feb 14, 2017, at 9:36 PM, Jed Brown wrote: > > Barry Smith writes: > >> The function below absolutely should not be called TSComputeIFunction()! It does not just compute IFunction() > > How is this different than SNESComputeFunction subtracting off the RHS? > >> The current code entangles too much of the user API to the methods, this can be fixed. > > I agree that the logic/naming is perhaps too overloaded. > >>> If the user experiments with different ways of splitting the solution they would have to define RHS and IF or RHS and LHS in different ways (according to the splittings they experiment with). It may look disgusting, but I don't see another way around it unless you allow for a list of operators to be defined and then the user to assign them to LHS or RHS. >> >> Jed suggested having any number of "RHS" functions. I don't think we need more than 2, 1 for left hand side and 1 for right. If that ends up being not enough we can change to have any number of them. Just to be clear. I suggest three functions >> >> IFunction which defaults to u_t plus TSSetMassMatrix() which changes the default IFunction to M u_t >> LHS function which defaults to 0, if provided defaults to being treated implicitly by IMEX >> RHS function which defaults to 0, if provided defaults to being treated explicitly by IMEX > > Are you trolling me? Reactive fluid-structure interaction: > > 1. fluid splitting to isolate the acoustic wave (stiff) > > 2. fluid containing everything but the acoustic wave (non-stiff) > > 3. fast reactions > > 4. slow reactions > > 5. structure > Ok, great examples! Are there any commonalities that can be leveraged in a better TS API than we currently have? Since there are these "families" of potential splittings can't we do better than saying "oh, that is all up to the (often naive and inexperienced) user"? Like the old days of the "templates" book, since there are a bunch of confusing, overlapping families of Krylov methods and preconditioners we'll just write a book instead of writing code? I'm not saying that my API is the best or even good but surely we can do better than just leaving it to the users. AND, of course, since I don't care about backward compatibility :-) what we do for an API now can evolve over time as we better understand what should be represented in the API and what shouldn't? > Note that when using conservative variables, every one of these splits > requires evaluation of the equation of state (reactions rate > coefficients depend on Temperature, structure boundary condition > requires pressure and shear stress). This doesn't even have > viscoelastics, sedimentation, or a turbulence model. Ok, then where is the equation of state in TS? Since it is a universal abstraction ("every one"), according to you, why isn't it represented in TS? What else should TS have that is common to many simulations? From jed at jedbrown.org Tue Feb 14 21:56:41 2017 From: jed at jedbrown.org (Jed Brown) Date: Tue, 14 Feb 2017 20:56:41 -0700 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> <60D7CDCC-E9D5-472A-9175-64C56DF8B34F@mcs.anl.gov> <87vasc4pz5.fsf@jedbrown.org> <6D880F3B-F880-4AFD-975D-5A8A4F4AA8A9@mcs.anl.gov> <87inoc4l17.fsf@jedbrown.org> Message-ID: <87shng2jk6.fsf@jedbrown.org> Ed Bueler writes: > Your question makes me think about why I am splitting the way I am. For > sure *yes*, in my case they are separate terms and *no* I am not just > subtracting Lu from both sides. > > The ice sheet thickness problem (u(t,x,y) >= 0 is thickness) is like > degenerate p-bratu, right? My form is like > > u_t - div (u^k |grad u|^{p-2} grad u) = g(t,u) Are you also interested in the case with sliding? > where the right side is only the elevation-dependent surface mass balance > in my context. The right-hand stuff is (potentially) coupling to other > models which will be time-split, and provided by other codes, so the > "ultimate" coupled model will inevitably be imex in some vague sense. The > stiff stuff on the left should be handled implicitly both for imex and > implicit, of course. And I want to see the solver robustness effects of my > imex splitting of the problem. I also want to compare actual explicit runs > despite their absurd short time step. > > In any case, whatever my motivation, I am currently providing > > RHSFunction = g(t,u) > IFunction = u_t - div (u^k |grad u|^{p-2} grad u) > > Currently ARKIMEX, THETA, BDF all seem to work well, but explicit breaks. > As I understand it, Barry's PR will "fix" this by erroring-out when I try > to run it explicitly. This is fine until ya'll work out a better API. I would just have a PetscOptionsEList() for whether to evaluate the diffusive term on the left or right. Shouldn't be more than a few lines of code. >> What "fragile code"? > > I mean I fear the prospect of a PETSc API which requires so conditionals > determining which "side"' various terms appear on, depending on user code > testing what method is running etc.. No such thing for now; my current > code is not fragile. (Is an API based on "side" of an equation inherently > suspect?) I think it's a bit peculiar and would prefer a named basket of terms if Barry insists on adding this interface. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From bsmith at mcs.anl.gov Tue Feb 14 22:01:35 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 14 Feb 2017 22:01:35 -0600 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: <87shng2jk6.fsf@jedbrown.org> References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> <60D7CDCC-E9D5-472A-9175-64C56DF8B34F@mcs.anl.gov> <87vasc4pz5.fsf@jedbrown.org> <6D880F3B-F880-4AFD-975D-5A8A4F4AA8A9@mcs.anl.gov> <87inoc4l17.fsf@jedbrown.org> <87shng2jk6.fsf@jedbrown.org> Message-ID: <328799B8-08A7-473B-9BBE-81340D79DE52@mcs.anl.gov> > On Feb 14, 2017, at 9:56 PM, Jed Brown wrote: > > Ed Bueler writes: >> Your question makes me think about why I am splitting the way I am. For >> sure *yes*, in my case they are separate terms and *no* I am not just >> subtracting Lu from both sides. >> >> The ice sheet thickness problem (u(t,x,y) >= 0 is thickness) is like >> degenerate p-bratu, right? My form is like >> >> u_t - div (u^k |grad u|^{p-2} grad u) = g(t,u) > > Are you also interested in the case with sliding? > >> where the right side is only the elevation-dependent surface mass balance >> in my context. The right-hand stuff is (potentially) coupling to other >> models which will be time-split, and provided by other codes, so the >> "ultimate" coupled model will inevitably be imex in some vague sense. The >> stiff stuff on the left should be handled implicitly both for imex and >> implicit, of course. And I want to see the solver robustness effects of my >> imex splitting of the problem. I also want to compare actual explicit runs >> despite their absurd short time step. >> >> In any case, whatever my motivation, I am currently providing >> >> RHSFunction = g(t,u) >> IFunction = u_t - div (u^k |grad u|^{p-2} grad u) >> >> Currently ARKIMEX, THETA, BDF all seem to work well, but explicit breaks. >> As I understand it, Barry's PR will "fix" this by erroring-out when I try >> to run it explicitly. This is fine until ya'll work out a better API. > > I would just have a PetscOptionsEList() for whether to evaluate the > diffusive term on the left or right. Shouldn't be more than a few lines > of code. > >>> What "fragile code"? >> >> I mean I fear the prospect of a PETSc API which requires so conditionals >> determining which "side"' various terms appear on, depending on user code >> testing what method is running etc.. No such thing for now; my current >> code is not fragile. (Is an API based on "side" of an equation inherently >> suspect?) > > I think it's a bit peculiar and would prefer a named basket of terms if > Barry insists on adding this interface. I'm all ears. From jed at jedbrown.org Tue Feb 14 22:22:01 2017 From: jed at jedbrown.org (Jed Brown) Date: Tue, 14 Feb 2017 21:22:01 -0700 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> <60D7CDCC-E9D5-472A-9175-64C56DF8B34F@mcs.anl.gov> <87vasc4pz5.fsf@jedbrown.org> <36FFCE9F-ECA8-4B5F-BE53-6B1AD2C7B205@anl.gov> <95c8d53c-5c77-34aa-68dd-3e5211d24c24@mcs.anl.gov> <45E4D3E5-B4DC-4BDC-984A-ED7735723E50@mcs.anl.gov> <03329c72-3572-9f71-5e5b-34741f5d7573@mcs.anl.gov> <9C507CBA-8DC4-4669-BEE7-4560E27EB020@mcs.anl.gov> <110C81DD-DA04-48A4-82B4-F9E527FA3806@mcs.anl.gov> <871sv03z22.fsf@jedbrown.org> Message-ID: <87poik2idy.fsf@jedbrown.org> Barry Smith writes: >> On Feb 14, 2017, at 9:36 PM, Jed Brown wrote: >> 1. fluid splitting to isolate the acoustic wave (stiff) >> >> 2. fluid containing everything but the acoustic wave (non-stiff) >> >> 3. fast reactions >> >> 4. slow reactions >> >> 5. structure >> > > Ok, great examples! Are there any commonalities that can be leveraged > in a better TS API than we currently have? Since there are these > "families" of potential splittings can't we do better than saying > "oh, that is all up to the (often naive and inexperienced) user"? > Like the old days of the "templates" book, since there are a bunch of > confusing, overlapping families of Krylov methods and preconditioners > we'll just write a book instead of writing code? I'm not saying that > my API is the best or even good but surely we can do better than just > leaving it to the users. AND, of course, since I don't care about > backward compatibility :-) what we do for an API now can evolve over > time as we better understand what should be represented in the API > and what shouldn't? My reluctance to create an API is that I don't know of any useful universal abstractions. I'm concerned that anything we do will ultimately be more complicated, less transparent, and less efficient than sticking with the existing interface. I'm open to suggestions. >> Note that when using conservative variables, every one of these splits >> requires evaluation of the equation of state (reactions rate >> coefficients depend on Temperature, structure boundary condition >> requires pressure and shear stress). This doesn't even have >> viscoelastics, sedimentation, or a turbulence model. > > Ok, then where is the equation of state in TS? Since it is a > universal abstraction ("every one"), according to you, why isn't it > represented in TS? What else should TS have that is common to many > simulations? So in a finite volume method, you may or may not transform to characteristic or primitive variables in cells, reconstruct a piecewise linear/polynomial representation in each cell, perhaps with limiting, then evaluate it at quadrature points on faces. At the faces, you need to solve a Riemann problem which will require solving for a pressure or a special nonlinear "average". Equations of state are often either tabulated or involve extensive series expansions. Sometimes they are described by implicit functions which you have to solve numerically using a rootfinder. This is an arbitrary example that makes my point about complexity of equations of state: http://www.mrc-eng.com/Downloads/Moist%20Air%20Props%20English.pdf For finite element methods, these computations take place at quadrature points within each element and generally don't involve a reconstruction or limiting step. There are also flux-splitting finite volume methods that do not solve Riemann problems (it is still necessary to evaluate the equation of state because the flux function depends on it). A similar situation exists in models of plasticity or other highly nonlinear solids. This is a material modeling issue so I don't see how it has anything to do with TS. It is not uncommon for these material models to be the most time-consuming part of residual computation. In that situation, the incremental cost of evaluating all the terms is negligible when you have the equation of state evaluated at a quadrature point. (Geometric terms may be similar, but it's not an extreme effect for complicated problems.) So if you evaluate those terms 1-5 separately, it may cost about 5x more than evaluating them together. Do you really want to encourage people to split them? Note that if the user has a switch in their code, it can be done at the quadrature point level (or batches of quadrature points) and thus impose at most 2x overhead (solving the equation of state once in the IFunction and once in the RHSFunction). -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From hongzhang at anl.gov Tue Feb 14 22:24:24 2017 From: hongzhang at anl.gov (Zhang, Hong) Date: Wed, 15 Feb 2017 04:24:24 +0000 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: <110C81DD-DA04-48A4-82B4-F9E527FA3806@mcs.anl.gov> References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> <60D7CDCC-E9D5-472A-9175-64C56DF8B34F@mcs.anl.gov> <87vasc4pz5.fsf@jedbrown.org> <36FFCE9F-ECA8-4B5F-BE53-6B1AD2C7B205@anl.gov> <95c8d53c-5c77-34aa-68dd-3e5211d24c24@mcs.anl.gov> <45E4D3E5-B4DC-4BDC-984A-ED7735723E50@mcs.anl.gov> <03329c72-3572-9f71-5e5b-34741f5d7573@mcs.anl.gov> <9C507CBA-8DC4-4669-BEE7-4560E27EB020@mcs.anl.gov> <110C81DD-DA04-48A4-82B4-F9E527FA3806@mcs.anl.gov> Message-ID: <8F6798B8-C63D-4998-9DA5-E05693B87036@anl.gov> On Feb 14, 2017, at 9:20 PM, Barry Smith > wrote: On Feb 14, 2017, at 8:56 PM, Emil Constantinescu > wrote: On 2/14/17 4:10 PM, Barry Smith wrote: Ok, you don't recompile but forcing that into user code is still disgusting. With my api the user code is TSSetRHSFunction(ts,NULL,RHSFunction,&ptype[0]); TSSetLHSFunction(ts,NULL,LHSFunction,&ptype[0]); TSSetRHSJacobian(ts,Jac,Jac,RHSJacobian,&ptype[0]); TSSetLHSJacobian(ts,Jac,Jac,LHSJacobian,&ptype[0]); and -ts_type xxx works correctly for ALL methods, implicit, explicit and imex without requiring any special command line options for different methods. Is this a viable solution? Growing the API to fix this situation will just put a burden with each new TS method after we refactor it in the current landscape. No just the opposite, the TS implementations will talk to functions who will put things together for it. So All implicit methods will call something like TSBuildImplicitFunction(), all explicit methods will call something like TSBuildExplicitFunction() and then IMEX methods will call both of these. In fact likely we can refactor to make things a little better than today. Depending on options and flagsTSBuildExplicitFunction() would build out of all the user provided functions what it needs etc. One problem with the current code is the TS methods call things with the same names as the user API. So implicit methods call TSComputeIFunction() while explicit methods call TSComputeRHSFunction(). This is wrong because implicit methods actually also use the rhs function provided by the user. The function below absolutely should not be called TSComputeIFunction()! It does not just compute IFunction() PetscErrorCode TSComputeIFunction(TS ts,PetscReal t,Vec U,Vec Udot,Vec Y,PetscBool imex) { PetscErrorCode ierr; TSIFunction ifunction; TSRHSFunction rhsfunction; void *ctx; DM dm; PetscFunctionBegin; PetscValidHeaderSpecific(ts,TS_CLASSID,1); PetscValidHeaderSpecific(U,VEC_CLASSID,3); PetscValidHeaderSpecific(Udot,VEC_CLASSID,4); PetscValidHeaderSpecific(Y,VEC_CLASSID,5); ierr = TSGetDM(ts,&dm);CHKERRQ(ierr); ierr = DMTSGetIFunction(dm,&ifunction,&ctx);CHKERRQ(ierr); ierr = DMTSGetRHSFunction(dm,&rhsfunction,NULL);CHKERRQ(ierr); if (!rhsfunction && !ifunction) SETERRQ(PetscObjectComm((PetscObject)ts),PETSC_ERR_USER,"Must call TSSetRHSFunction() and / or TSSetIFunction()"); ierr = PetscLogEventBegin(TS_FunctionEval,ts,U,Udot,Y);CHKERRQ(ierr); if (ifunction) { PetscStackPush("TS user implicit function"); ierr = (*ifunction)(ts,t,U,Udot,Y,ctx);CHKERRQ(ierr); PetscStackPop; } if (imex) { if (!ifunction) { ierr = VecCopy(Udot,Y);CHKERRQ(ierr); } } else if (rhsfunction) { if (ifunction) { Vec Frhs; ierr = TSGetRHSVec_Private(ts,&Frhs);CHKERRQ(ierr); ierr = TSComputeRHSFunction(ts,t,U,Frhs);CHKERRQ(ierr); ierr = VecAXPY(Y,-1,Frhs);CHKERRQ(ierr); } else { ierr = TSComputeRHSFunction(ts,t,U,Y);CHKERRQ(ierr); ierr = VecAYPX(Y,-1,Udot);CHKERRQ(ierr); } } ierr = PetscLogEventEnd(TS_FunctionEval,ts,U,Udot,Y);CHKERRQ(ierr); PetscFunctionReturn(0); } The current code entangles too much of the user API to the methods, this can be fixed. If the user experiments with different ways of splitting the solution they would have to define RHS and IF or RHS and LHS in different ways (according to the splittings they experiment with). It may look disgusting, but I don't see another way around it unless you allow for a list of operators to be defined and then the user to assign them to LHS or RHS. Jed suggested having any number of "RHS" functions. I don't think we need more than 2, 1 for left hand side and 1 for right. If that ends up being not enough we can change to have any number of them. Just to be clear. I suggest three functions IFunction which defaults to u_t plus TSSetMassMatrix() which changes the default IFunction to M u_t LHS function which defaults to 0, if provided defaults to being treated implicitly by IMEX RHS function which defaults to 0, if provided defaults to being treated explicitly by IMEX It is a great idea to require IFunction to be M u_t or u_t. This allows us to eliminate the weird 'shift' parameter in the current interface. But the LHS function would be as confusing as the old IFunction. Apparently it can be moved to the right hand side. So why don't we use IFunction RHS function 1 / RHSJacobian 1 RHS function 2 / RHSJacobian 2 ... (and more if needed) Also we can add command line options to control which RHS terms should be treated explicitly/implicitly, e.g., IMEX: -ts_implicit rhs1 -ts_explicit rhs2 fully implicit: -ts_implicit rhs1,rhs2 explicit: -ts_explicit rhs1,rhs2 Hong (Mr.) Then a TSSetStiffMatrix(ts,Mat L) (horrible name) that provides u_t -Lu = g() + Lu None of the TS implementations will every directly know about what the user has provided. They will call the wrapper functions I mention above. I think Jed and Emil may be too enamored with the reductionist model of only IFunction() and RHSFunction() to see that though it encompasses everything it may not be the best user API. Adding all that logic to keep track of left sides and right sides for academic examples is likely not the best development. I don't think it is "just academic examples", it is all examples without a mass matrix. Once the user has decided with ts_type to use for production if it is fully implicit or explicit then they can depending on the type selected, write just a left hand side, just a right hand side for higher efficiency (less update of ghost points, fewer iterations over loops etc). With a constant mass matrix we can have TSSetMassMatrix() and then TSSetIFunction() is reserved for when it is absolutely needed. As much as I would disagree with growing the API at the level of defining the problem, I think TSSetMassMatrix() would let us do more things in the solvers. Also it would be useful to know if the mass matrix is singular or not for efficiency reasons. Emil Barry -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue Feb 14 22:34:28 2017 From: jed at jedbrown.org (Jed Brown) Date: Tue, 14 Feb 2017 21:34:28 -0700 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: <8F6798B8-C63D-4998-9DA5-E05693B87036@anl.gov> References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> <60D7CDCC-E9D5-472A-9175-64C56DF8B34F@mcs.anl.gov> <87vasc4pz5.fsf@jedbrown.org> <36FFCE9F-ECA8-4B5F-BE53-6B1AD2C7B205@anl.gov> <95c8d53c-5c77-34aa-68dd-3e5211d24c24@mcs.anl.gov> <45E4D3E5-B4DC-4BDC-984A-ED7735723E50@mcs.anl.gov> <03329c72-3572-9f71-5e5b-34741f5d7573@mcs.anl.gov> <9C507CBA-8DC4-4669-BEE7-4560E27EB020@mcs.anl.gov> <110C81DD-DA04-48A4-82B4-F9E527FA3806@mcs.anl.gov> <8F6798B8-C63D-4998-9DA5-E05693B87036@anl.gov> Message-ID: <87k28s2ht7.fsf@jedbrown.org> "Zhang, Hong" writes: > It is a great idea to require IFunction to be M u_t or u_t. This > allows us to eliminate the weird 'shift' parameter in the current > interface. Noooooo! That would prevent solving problems with moving meshes or with transient statements written in terms of dependent variables, like d rho(pressure, momentum, temperature) / dt = div(momentum) where (pressure, momentum, temperature) are the independent variables but we want a statement of conservation. There are important all-speed flow formulations that look like this. > But the LHS function would be as confusing as the old > IFunction. Apparently it can be moved to the right hand side. So why > don't we use > > IFunction > RHS function 1 / RHSJacobian 1 > RHS function 2 / RHSJacobian 2 > ... (and more if needed) > > Also we can add command line options to control which RHS terms should be treated explicitly/implicitly, e.g., > IMEX: -ts_implicit rhs1 -ts_explicit rhs2 > fully implicit: -ts_implicit rhs1,rhs2 > explicit: -ts_explicit rhs1,rhs2 If you want multiple RHS, then make TSPushRHSFunction(ts, "fname", ...); Put these in a linked list, add -ts_rhs_implicit foo,bar,qux Then TSComputeIFunction walks the list and evaluates anything that was named while TSComputeRHSFunction evaluates everything else. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From elbueler at alaska.edu Tue Feb 14 23:28:47 2017 From: elbueler at alaska.edu (Ed Bueler) Date: Tue, 14 Feb 2017 20:28:47 -0900 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: <87shng2jk6.fsf@jedbrown.org> References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> <60D7CDCC-E9D5-472A-9175-64C56DF8B34F@mcs.anl.gov> <87vasc4pz5.fsf@jedbrown.org> <6D880F3B-F880-4AFD-975D-5A8A4F4AA8A9@mcs.anl.gov> <87inoc4l17.fsf@jedbrown.org> <87shng2jk6.fsf@jedbrown.org> Message-ID: Jed -- >> u_t - div (u^k |grad u|^{p-2} grad u) = g(t,u) >Are you also interested in the case with sliding? You must be getting grumpy. There is this 2009 paper about sliding ... Yes I am interested in sliding but I was trying to keep things simple for you. ;-) The code in question is an under-development example in my book that I am trying to use to show off SNESVI functionality in an interesting case and yet not get too nasty. See the help string at the top of the file for what it actually does: https://github.com/bueler/p4pdes/blob/master/c/ch11/ice.c In summary, the actual flux is q_total = - Gamma H^{n+2} |grad s|^{n-1} grad s + V H and it is all handled implicitly by preference, i.e. in the current API I have IFunction = u_t - div q_total. The real interest of the problem is the failure of long-range communication of low-(spatial)-frequency information when you hit the free boundary, and these barriers can emerge anywhere in the domain. (In a mountain range you can have neighboring small glaciers completely out of phase; how should multigrid ... or any other preconditioning one would hope for ... work for all the glaciers in a mountain range? So now I'll emphasize what I said before: it is *degenerate* p-bratu. Doubly-degenerate because p>2, but the "H^{n+2}" aka "u^k" is the bad thing that makes the abrupt free boundary.) Sliding, in this regard, is the easy case, so I am throwing it in by generating an artificial velocity field; yes this is the coupling that should include the SSA in the future ... we already have an SSA solver or two that behave surprisingly well, you know, as long as the boundary conditions can be figured-out. > I would just have a PetscOptionsEList() for whether to evaluate the > diffusive term on the left or right. Shouldn't be more than a few lines > of code. So I have something like?: -ice_diffusive_side left,right O.k. What if user says -ts_type rk -ice_diffusive_side left Then I get the current brokenness: code runs and produces nonsense because RK does not call SNESVI (constraint never enforced) *and* diffusive stuff is never evaluated (because RK does not call IFunction). So maybe then I also go through possible -ts_types and check whether the option combination made sense? Or move some terms to the correct side based on what that -ts_type should want? Do I include -ts_type sundials because somebody's config might include that third-party thing (even though mine doesn't)? That's what I mean by fragile code. Such things build up! Each time you shove cool stuff in my face I have to get used to seeing ugly stuff turn into beautiful stuff as my aesthetic sense follows my growing understanding. But it takes time (learning curve) and all the time it is fragile code (i.e. for me ... because I don't appreciate all the motivations behind that part of the API ...) On the other hand ... I agree that if I am convinced this is really a good API for something I want, then indeed I will suck it up and learn it and write it. Recall my original request was for the ability to know if the TS type I have chosen will actually use the SNES I have set ... so long ago it now seems. Ed On Tue, Feb 14, 2017 at 6:56 PM, Jed Brown wrote: > Ed Bueler writes: > > Your question makes me think about why I am splitting the way I am. For > > sure *yes*, in my case they are separate terms and *no* I am not just > > subtracting Lu from both sides. > > > > The ice sheet thickness problem (u(t,x,y) >= 0 is thickness) is like > > degenerate p-bratu, right? My form is like > > > > u_t - div (u^k |grad u|^{p-2} grad u) = g(t,u) > > Are you also interested in the case with sliding? > > > where the right side is only the elevation-dependent surface mass balance > > in my context. The right-hand stuff is (potentially) coupling to other > > models which will be time-split, and provided by other codes, so the > > "ultimate" coupled model will inevitably be imex in some vague sense. > The > > stiff stuff on the left should be handled implicitly both for imex and > > implicit, of course. And I want to see the solver robustness effects of > my > > imex splitting of the problem. I also want to compare actual explicit > runs > > despite their absurd short time step. > > > > In any case, whatever my motivation, I am currently providing > > > > RHSFunction = g(t,u) > > IFunction = u_t - div (u^k |grad u|^{p-2} grad u) > > > > Currently ARKIMEX, THETA, BDF all seem to work well, but explicit breaks. > > As I understand it, Barry's PR will "fix" this by erroring-out when I try > > to run it explicitly. This is fine until ya'll work out a better API. > > I would just have a PetscOptionsEList() for whether to evaluate the > diffusive term on the left or right. Shouldn't be more than a few lines > of code. > > >> What "fragile code"? > > > > I mean I fear the prospect of a PETSc API which requires so conditionals > > determining which "side"' various terms appear on, depending on user code > > testing what method is running etc.. No such thing for now; my current > > code is not fragile. (Is an API based on "side" of an equation > inherently > > suspect?) > > I think it's a bit peculiar and would prefer a named basket of terms if > Barry insists on adding this interface. > -- Ed Bueler Dept of Math and Stat and Geophysical Institute University of Alaska Fairbanks Fairbanks, AK 99775-6660 301C Chapman and 410D Elvey 907 474-7693 and 907 474-7199 (fax 907 474-5394) -------------- next part -------------- An HTML attachment was scrubbed... URL: From elbueler at alaska.edu Tue Feb 14 23:44:22 2017 From: elbueler at alaska.edu (Ed Bueler) Date: Tue, 14 Feb 2017 20:44:22 -0900 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: <110C81DD-DA04-48A4-82B4-F9E527FA3806@mcs.anl.gov> References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> <60D7CDCC-E9D5-472A-9175-64C56DF8B34F@mcs.anl.gov> <87vasc4pz5.fsf@jedbrown.org> <36FFCE9F-ECA8-4B5F-BE53-6B1AD2C7B205@anl.gov> <95c8d53c-5c77-34aa-68dd-3e5211d24c24@mcs.anl.gov> <45E4D3E5-B4DC-4BDC-984A-ED7735723E50@mcs.anl.gov> <03329c72-3572-9f71-5e5b-34741f5d7573@mcs.anl.gov> <9C507CBA-8DC4-4669-BEE7-4560E27EB020@mcs.anl.gov> <110C81DD-DA04-48A4-82B4-F9E527FA3806@mcs.anl.gov> Message-ID: > Jed suggested having any number of "RHS" functions. I don't think we need more than 2, 1 for left hand side > and 1 for right. If that ends up being not enough we can change to have any number of them. Just to be clear. > I suggest three functions > IFunction which defaults to u_t plus TSSetMassMatrix() which changes the default IFunction to M u_t > LHS function which defaults to 0, if provided defaults to being treated implicitly by IMEX > RHS function which defaults to 0, if provided defaults to being treated explicitly by IMEX > Then a TSSetStiffMatrix(ts,Mat L) (horrible name) that provides u_t -Lu = g() + Lu At my current level of understanding I get this trifecta (quad-fecta?). In my ice sheet case u_t = f(t,u) + g(t,u), I would not set IFunction at all, or I would use a constant. The divergence of flux f(t,u) = div(q) would go into the LHSFunction and the elevation-dependent mass balance g(t,u) would go into the RHSFunction. I would not use TSSetStiffMatrix() at all. Barry's suggestion makes sense for this and the few other usage cases I have already experienced ... kind of a lame endorsement, but, on such a weak basis one casts a vote ... Ed PS I would not use TSSetStiffMatrix() because none of the linearizations/simplfications known at the completion of a TS step could "capture" the stiff part without being derived a la the Jacobian anyway. To see what I mean, note that VINEWTONRSLS has a different *dimension* at each Newton iteration. I don't know the (in-active-constraint) dimension of the solution I'll get at each time step, much less how to scale the various stiff-equation-parts for those in-active dimensions. In these problems there is a lot going on in the implicit solve, besides just handling stiffness. On Tue, Feb 14, 2017 at 6:20 PM, Barry Smith wrote: > > > On Feb 14, 2017, at 8:56 PM, Emil Constantinescu > wrote: > > > > On 2/14/17 4:10 PM, Barry Smith wrote: > >> Ok, you don't recompile but forcing that into user code is still > disgusting. With my api the user code is > >> > >>>>> TSSetRHSFunction(ts,NULL,RHSFunction,&ptype[0]); > >>>>> TSSetLHSFunction(ts,NULL,LHSFunction,&ptype[0]); > >>>>> TSSetRHSJacobian(ts,Jac,Jac,RHSJacobian,&ptype[0]); > >>>>> TSSetLHSJacobian(ts,Jac,Jac,LHSJacobian,&ptype[0]); > >> and -ts_type xxx works correctly for ALL methods, implicit, explicit > and imex without requiring any special command line options for different > methods. > > > > Is this a viable solution? Growing the API to fix this situation will > just put a burden with each new TS method after we refactor it in the > current landscape. > > No just the opposite, the TS implementations will talk to functions who > will put things together for it. So All implicit methods will call > something like TSBuildImplicitFunction(), all explicit methods will call > something like TSBuildExplicitFunction() and then IMEX methods will call > both of these. In fact likely we can refactor to make things a little > better than today. Depending on options and flagsTSBuildExplicitFunction() > would build out of all the user provided functions what it needs etc. > > One problem with the current code is the TS methods call things with the > same names as the user API. So implicit methods call TSComputeIFunction() > while explicit methods call TSComputeRHSFunction(). This is wrong because > implicit methods actually also use the rhs function provided by the user. > > The function below absolutely should not be called TSComputeIFunction()! > It does not just compute IFunction() > PetscErrorCode TSComputeIFunction(TS ts,PetscReal t,Vec U,Vec Udot,Vec > Y,PetscBool imex) > { > PetscErrorCode ierr; > TSIFunction ifunction; > TSRHSFunction rhsfunction; > void *ctx; > DM dm; > > PetscFunctionBegin; > PetscValidHeaderSpecific(ts,TS_CLASSID,1); > PetscValidHeaderSpecific(U,VEC_CLASSID,3); > PetscValidHeaderSpecific(Udot,VEC_CLASSID,4); > PetscValidHeaderSpecific(Y,VEC_CLASSID,5); > > ierr = TSGetDM(ts,&dm);CHKERRQ(ierr); > ierr = DMTSGetIFunction(dm,&ifunction,&ctx);CHKERRQ(ierr); > ierr = DMTSGetRHSFunction(dm,&rhsfunction,NULL);CHKERRQ(ierr); > > if (!rhsfunction && !ifunction) SETERRQ(PetscObjectComm(( > PetscObject)ts),PETSC_ERR_USER,"Must call TSSetRHSFunction() and / or > TSSetIFunction()"); > > ierr = PetscLogEventBegin(TS_FunctionEval,ts,U,Udot,Y);CHKERRQ(ierr); > if (ifunction) { > PetscStackPush("TS user implicit function"); > ierr = (*ifunction)(ts,t,U,Udot,Y,ctx);CHKERRQ(ierr); > PetscStackPop; > } > if (imex) { > if (!ifunction) { > ierr = VecCopy(Udot,Y);CHKERRQ(ierr); > } > } else if (rhsfunction) { > if (ifunction) { > Vec Frhs; > ierr = TSGetRHSVec_Private(ts,&Frhs);CHKERRQ(ierr); > ierr = TSComputeRHSFunction(ts,t,U,Frhs);CHKERRQ(ierr); > ierr = VecAXPY(Y,-1,Frhs);CHKERRQ(ierr); > } else { > ierr = TSComputeRHSFunction(ts,t,U,Y);CHKERRQ(ierr); > ierr = VecAYPX(Y,-1,Udot);CHKERRQ(ierr); > } > } > ierr = PetscLogEventEnd(TS_FunctionEval,ts,U,Udot,Y);CHKERRQ(ierr); > PetscFunctionReturn(0); > } > > The current code entangles too much of the user API to the methods, this > can be fixed. > > > If the user experiments with different ways of splitting the solution > they would have to define RHS and IF or RHS and LHS in different ways > (according to the splittings they experiment with). It may look disgusting, > but I don't see another way around it unless you allow for a list of > operators to be defined and then the user to assign them to LHS or RHS. > > Jed suggested having any number of "RHS" functions. I don't think we > need more than 2, 1 for left hand side and 1 for right. If that ends up > being not enough we can change to have any number of them. Just to be > clear. I suggest three functions > > IFunction which defaults to u_t plus TSSetMassMatrix() which changes > the default IFunction to M u_t > LHS function which defaults to 0, if provided defaults to being treated > implicitly by IMEX > RHS function which defaults to 0, if provided defaults to being treated > explicitly by IMEX > > Then a TSSetStiffMatrix(ts,Mat L) (horrible name) that provides u_t -Lu > = g() + Lu > > None of the TS implementations will every directly know about what the > user has provided. They will call the wrapper functions I mention above. > > I think Jed and Emil may be too enamored with the reductionist model of > only IFunction() and RHSFunction() to see that though it encompasses > everything it may not be the best user API. > > > > > > >> > >>> Adding all that logic to keep track of left sides and right sides for > academic examples is likely not the best development. > >> I don't think it is "just academic examples", it is all examples > without a mass matrix. > >> > >> Once the user has decided with ts_type to use for production if it is > fully implicit or explicit then they can depending on the type selected, > write just a left hand side, just a right hand side for higher efficiency > (less update of ghost points, fewer iterations over loops etc). > >> > >> With a constant mass matrix we can have TSSetMassMatrix() and then > TSSetIFunction() is reserved for when it is absolutely needed. > > > > As much as I would disagree with growing the API at the level of > defining the problem, I think TSSetMassMatrix() would let us do more things > in the solvers. Also it would be useful to know if the mass matrix is > singular or not for efficiency reasons. > > > > Emil > > > >> Barry > >> > > -- Ed Bueler Dept of Math and Stat and Geophysical Institute University of Alaska Fairbanks Fairbanks, AK 99775-6660 301C Chapman and 410D Elvey 907 474-7693 and 907 474-7199 (fax 907 474-5394) -------------- next part -------------- An HTML attachment was scrubbed... URL: From emconsta at mcs.anl.gov Tue Feb 14 23:59:17 2017 From: emconsta at mcs.anl.gov (Emil Constantinescu) Date: Tue, 14 Feb 2017 23:59:17 -0600 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> <60D7CDCC-E9D5-472A-9175-64C56DF8B34F@mcs.anl.gov> <87vasc4pz5.fsf@jedbrown.org> <36FFCE9F-ECA8-4B5F-BE53-6B1AD2C7B205@anl.gov> <95c8d53c-5c77-34aa-68dd-3e5211d24c24@mcs.anl.gov> <45E4D3E5-B4DC-4BDC-984A-ED7735723E50@mcs.anl.gov> <03329c72-3572-9f71-5e5b-34741f5d7573@mcs.anl.gov> <9C507CBA-8DC4-4669-BEE7-4560E27EB020@mcs.anl.gov> <110C81DD-DA04-48A4-82B4-F9E527FA3806@mcs.anl.gov> Message-ID: <87949040-ce11-c0f2-bac8-77c6f477ca17@mcs.anl.gov> We seem to be needing more than two components and ways to pack them. Allowing them to be dynamically assigned at run time would be really cool. We seem to like an API that allows us to specify M and u_t; but would also like to keep support for f(t,u,u_t)=0. How about a costly experiment: build an API as discussed below (support the above) that would map everything to the current one 1xRHS and 1xIFunction to see how it would work? In other words, is there a scenario in which we could introduce the new but also keep the old and be reasonable about it? Of concern is whether the new one can be made friendly enough. Emil On 2/14/17 11:44 PM, Ed Bueler wrote: >> Jed suggested having any number of "RHS" functions. I don't think we > need more than 2, 1 for left hand side >> and 1 for right. If that ends up being not enough we can change to > have any number of them. Just to be clear. >> I suggest three functions >> IFunction which defaults to u_t plus TSSetMassMatrix() which > changes the default IFunction to M u_t >> LHS function which defaults to 0, if provided defaults to being > treated implicitly by IMEX >> RHS function which defaults to 0, if provided defaults to being > treated explicitly by IMEX >> Then a TSSetStiffMatrix(ts,Mat L) (horrible name) that provides u_t > -Lu = g() + Lu > > At my current level of understanding I get this trifecta (quad-fecta?). > > In my ice sheet case u_t = f(t,u) + g(t,u), I would not set IFunction > at all, or I would use a constant. The divergence of flux f(t,u) = > div(q) would go into the LHSFunction and the elevation-dependent mass > balance g(t,u) would go into the RHSFunction. I would not use > TSSetStiffMatrix() at all. > > Barry's suggestion makes sense for this and the few other usage cases I > have already experienced ... kind of a lame endorsement, but, on such a > weak basis one casts a vote ... > > Ed > > > PS I would not use TSSetStiffMatrix() because none of the > linearizations/simplfications known at the completion of a TS step could > "capture" the stiff part without being derived a la the Jacobian > anyway. To see what I mean, note that VINEWTONRSLS has a different > *dimension* at each Newton iteration. I don't know the > (in-active-constraint) dimension of the solution I'll get at each time > step, much less how to scale the various stiff-equation-parts for those > in-active dimensions. In these problems there is a lot going on in the > implicit solve, besides just handling stiffness. > > > On Tue, Feb 14, 2017 at 6:20 PM, Barry Smith > wrote: > > > > On Feb 14, 2017, at 8:56 PM, Emil Constantinescu > wrote: > > > > On 2/14/17 4:10 PM, Barry Smith wrote: > >> Ok, you don't recompile but forcing that into user code is still disgusting. With my api the user code is > >> > >>>>> TSSetRHSFunction(ts,NULL,RHSFunction,&ptype[0]); > >>>>> TSSetLHSFunction(ts,NULL,LHSFunction,&ptype[0]); > >>>>> TSSetRHSJacobian(ts,Jac,Jac,RHSJacobian,&ptype[0]); > >>>>> TSSetLHSJacobian(ts,Jac,Jac,LHSJacobian,&ptype[0]); > >> and -ts_type xxx works correctly for ALL methods, implicit, explicit and imex without requiring any special command line options for different methods. > > > > Is this a viable solution? Growing the API to fix this situation will just put a burden with each new TS method after we refactor it in the current landscape. > > No just the opposite, the TS implementations will talk to > functions who will put things together for it. So All implicit > methods will call something like TSBuildImplicitFunction(), all > explicit methods will call something like TSBuildExplicitFunction() > and then IMEX methods will call both of these. In fact likely we can > refactor to make things a little better than today. Depending on > options and flagsTSBuildExplicitFunction() would build out of all > the user provided functions what it needs etc. > > One problem with the current code is the TS methods call things with > the same names as the user API. So implicit methods call > TSComputeIFunction() while explicit methods call > TSComputeRHSFunction(). This is wrong because implicit methods > actually also use the rhs function provided by the user. > > The function below absolutely should not be called > TSComputeIFunction()! It does not just compute IFunction() > PetscErrorCode TSComputeIFunction(TS ts,PetscReal t,Vec U,Vec > Udot,Vec Y,PetscBool imex) > { > PetscErrorCode ierr; > TSIFunction ifunction; > TSRHSFunction rhsfunction; > void *ctx; > DM dm; > > PetscFunctionBegin; > PetscValidHeaderSpecific(ts,TS_CLASSID,1); > PetscValidHeaderSpecific(U,VEC_CLASSID,3); > PetscValidHeaderSpecific(Udot,VEC_CLASSID,4); > PetscValidHeaderSpecific(Y,VEC_CLASSID,5); > > ierr = TSGetDM(ts,&dm);CHKERRQ(ierr); > ierr = DMTSGetIFunction(dm,&ifunction,&ctx);CHKERRQ(ierr); > ierr = DMTSGetRHSFunction(dm,&rhsfunction,NULL);CHKERRQ(ierr); > > if (!rhsfunction && !ifunction) > SETERRQ(PetscObjectComm((PetscObject)ts),PETSC_ERR_USER,"Must call > TSSetRHSFunction() and / or TSSetIFunction()"); > > ierr = PetscLogEventBegin(TS_FunctionEval,ts,U,Udot,Y);CHKERRQ(ierr); > if (ifunction) { > PetscStackPush("TS user implicit function"); > ierr = (*ifunction)(ts,t,U,Udot,Y,ctx);CHKERRQ(ierr); > PetscStackPop; > } > if (imex) { > if (!ifunction) { > ierr = VecCopy(Udot,Y);CHKERRQ(ierr); > } > } else if (rhsfunction) { > if (ifunction) { > Vec Frhs; > ierr = TSGetRHSVec_Private(ts,&Frhs);CHKERRQ(ierr); > ierr = TSComputeRHSFunction(ts,t,U,Frhs);CHKERRQ(ierr); > ierr = VecAXPY(Y,-1,Frhs);CHKERRQ(ierr); > } else { > ierr = TSComputeRHSFunction(ts,t,U,Y);CHKERRQ(ierr); > ierr = VecAYPX(Y,-1,Udot);CHKERRQ(ierr); > } > } > ierr = PetscLogEventEnd(TS_FunctionEval,ts,U,Udot,Y);CHKERRQ(ierr); > PetscFunctionReturn(0); > } > > The current code entangles too much of the user API to the methods, > this can be fixed. > > > If the user experiments with different ways of splitting the solution they would have to define RHS and IF or RHS and LHS in different ways (according to the splittings they experiment with). It may look disgusting, but I don't see another way around it unless you allow for a list of operators to be defined and then the user to assign them to LHS or RHS. > > Jed suggested having any number of "RHS" functions. I don't think > we need more than 2, 1 for left hand side and 1 for right. If that > ends up being not enough we can change to have any number of them. > Just to be clear. I suggest three functions > > IFunction which defaults to u_t plus TSSetMassMatrix() which > changes the default IFunction to M u_t > LHS function which defaults to 0, if provided defaults to being > treated implicitly by IMEX > RHS function which defaults to 0, if provided defaults to being > treated explicitly by IMEX > > Then a TSSetStiffMatrix(ts,Mat L) (horrible name) that provides > u_t -Lu = g() + Lu > > None of the TS implementations will every directly know about > what the user has provided. They will call the wrapper functions I > mention above. > > I think Jed and Emil may be too enamored with the reductionist > model of only IFunction() and RHSFunction() to see that though it > encompasses everything it may not be the best user API. > > > > > > >> > >>> Adding all that logic to keep track of left sides and right > sides for academic examples is likely not the best development. > >> I don't think it is "just academic examples", it is all examples > without a mass matrix. > >> > >> Once the user has decided with ts_type to use for production if > it is fully implicit or explicit then they can depending on the type > selected, write just a left hand side, just a right hand side for > higher efficiency (less update of ghost points, fewer iterations > over loops etc). > >> > >> With a constant mass matrix we can have TSSetMassMatrix() and > then TSSetIFunction() is reserved for when it is absolutely needed. > > > > As much as I would disagree with growing the API at the level of > defining the problem, I think TSSetMassMatrix() would let us do more > things in the solvers. Also it would be useful to know if the mass > matrix is singular or not for efficiency reasons. > > > > Emil > > > >> Barry > >> > > > > > -- > Ed Bueler > Dept of Math and Stat and Geophysical Institute > University of Alaska Fairbanks > Fairbanks, AK 99775-6660 > 301C Chapman and 410D Elvey > 907 474-7693 and 907 474-7199 (fax 907 474-5394) From lixin_chu at yahoo.com Wed Feb 15 00:40:11 2017 From: lixin_chu at yahoo.com (lixin chu) Date: Wed, 15 Feb 2017 06:40:11 +0000 (UTC) Subject: [petsc-users] Newbie question : iterative solver - algorithm and performance References: <2007645858.4961090.1487140811648.ref@mail.yahoo.com> Message-ID: <2007645858.4961090.1487140811648@mail.yahoo.com> Hello,New to PETSc, appreciate any help -? I have done some experiment with MUMPS (the direct solver for sparse matrix), and very interested to try out PETSc now. I have a large sparse symmetric matrix (3 millions x 3 millions, complex data type). Some questions I have: - I am assuming that I should select one algorithm of "Krylov methods",?which algorithm is a good option, GMRES, CG, or others ? (I am not a domain expert, but helping developing a program to test the matrix).?- do all the algorithms support distributed architecture (multiple machines, multiple cores)- are there any performance test data ? (total run time for example) ?thank you very much, LX -------------- next part -------------- An HTML attachment was scrubbed... URL: From bernardomartinsrocha at gmail.com Wed Feb 15 04:17:06 2017 From: bernardomartinsrocha at gmail.com (Bernardo Rocha) Date: Wed, 15 Feb 2017 08:17:06 -0200 Subject: [petsc-users] PC HYPRE BoomerAMG options for nodal In-Reply-To: <1FB1E38B-0C7F-4759-A7F1-C2365A9C7FBB@mcs.anl.gov> References: <4032E7BA-A557-4491-B4E1-71F42E13DBA6@mcs.anl.gov> <8B1C4F5C-BC22-4476-ACAF-A0C105768E0B@mcs.anl.gov> <1FB1E38B-0C7F-4759-A7F1-C2365A9C7FBB@mcs.anl.gov> Message-ID: Thanks a lot. I'm going to change to the interlaced version. It is easier and more efficient. Best regards. Bernardo On Wed, Feb 15, 2017 at 12:07 AM, Barry Smith wrote: > > > On Feb 14, 2017, at 7:57 PM, Bernardo Rocha com> wrote: > > > > No particular reason. Let's say it was the first approach I tried in an > old code. > > > > OK, so to get it working I should use the interlaced version for the > ordering of my dofs. > > It does not require too much coding on my side. > > > > Just to know, if I wanted to insist in the non-interlaced, to keep using > Hypre/BoomerAMG from PETSc I would need to write some code in PETSc for > wrapping this feature (AMGSetDofFunc), right? > > > > Thanks. > > > > On Tue, Feb 14, 2017 at 11:48 PM, Barry Smith > wrote: > > > > Ahh, thanks for the explanation. What a nutso API: way to general, > they should just support interlaced and non-interlaced. > > > > Anyways we strongly recommend using the interlaced version; for most > calculations it gives much better locality for cache lines and cache. We > always use interlaced. Any particular reason you use non-interlaced? > > Yes, you would have to add a call to HYPRE_BoomerAMGSetDofFunc() with > the information as you computed. For trying things out the easily way would > be to just stick it in an appropriate place in hypre.c > > Barry > > > > > > > > Barry > > > > > On Feb 14, 2017, at 7:17 PM, Bernardo Rocha < > bernardomartinsrocha at gmail.com> wrote: > > > > > > Thanks a lot for the reply. > > > > > > ?From my understanding and specifically for my problem I would have to > use the AMGSetDofFunc as follows:? > > > > > > ?nsyseq = 3;? > > > for (i=0; i > > ? ?for (k=0; k > > { > > > ? ?eqnIndex[i*?N?+k] = i; > > > } > > > HYPRE_BoomerAMGSetDofFunc(preconditioner, eqnIndex);? > > > > > > For a system of 3 PDEs (e.g. linear elasticity in 3D?) the default > behaviour of the nodal coarsening in BoomerAMG, considers an array that is > filled with > > > > > > eqnIndex = ?0 1 2 0 1 2 0 1 2....0 1 2 > > > > > > > HYPRE_BoomerAMGSetDofFunc > > > > > > I could not find an indication of that this is suppose to be > setting; it is mentioned in the users manual but I don't know what it > means. It looks like it takes an integer array but I don't even know how > long that array is. > > > > > > If you figure out what it means and can add support for it we'll > take it as a pull request. > > > > > > ??OK Barry, I will try to specify the block size of the matrix. > > > But I'm not sure if the block size would be 3 or N (number of nodes). > > > This is my ordering of the displacements: [ux1, ux2, ..., uxN, uy1, > uy2,...uyN, uz1, uz2, ..., uzN] > > > > > > > HYPRE_BoomerAMGSetNumFunctions > > > > > > Set the block size of the matrix; this information is then > transferred automatically to hypre. Note that you can set a block size > even for AIJ matrices. > > > ? > > > ?Best regards, > > > Bernardo? > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lixin_chu at yahoo.com Wed Feb 15 07:03:59 2017 From: lixin_chu at yahoo.com (lixin chu) Date: Wed, 15 Feb 2017 13:03:59 +0000 (UTC) Subject: [petsc-users] Newbie question : iterative solver - algorithm and performance In-Reply-To: <2007645858.4961090.1487140811648@mail.yahoo.com> References: <2007645858.4961090.1487140811648.ref@mail.yahoo.com> <2007645858.4961090.1487140811648@mail.yahoo.com> Message-ID: <915837361.5274745.1487163839533@mail.yahoo.com> I think the chapter 4 of the user manual more or less answers my first question already ...?rgds lixin On Wednesday, 15 February 2017, 14:40, lixin chu wrote: Hello,New to PETSc, appreciate any help -? I have done some experiment with MUMPS (the direct solver for sparse matrix), and very interested to try out PETSc now. I have a large sparse symmetric matrix (3 millions x 3 millions, complex data type). Some questions I have: - I am assuming that I should select one algorithm of "Krylov methods",?which algorithm is a good option, GMRES, CG, or others ? (I am not a domain expert, but helping developing a program to test the matrix).?- do all the algorithms support distributed architecture (multiple machines, multiple cores)- are there any performance test data ? (total run time for example) ?thank you very much, LX -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrewh0 at uw.edu Wed Feb 15 09:52:26 2017 From: andrewh0 at uw.edu (Andrew Ho) Date: Wed, 15 Feb 2017 07:52:26 -0800 Subject: [petsc-users] PETSc set off-diagonal matrix pre-allocation In-Reply-To: References: Message-ID: I don't understand how the matrices are partitioned then. The documentation online all shows each mpi rank owning entire rows, with no partitioning along columns. I've attached an example where I try to partition a 6x7 matrix into 4 ranks: rank 0 should own a 3x4 section rank 1 should own a 3x3 section rank 2 should own a 3x4 section rank 3 should own a 3x3 section However, when I run the example and print out the ownership range and ownership column range, I get: rank 0 owns rows/cols: [0,3), [0, 4) rank 1 owns rows/cols: [3,6), [4, 7) rank 2 owns rows/cols: [6,9), [7, 11) rank 3 owns rows/cols: [9,12), [11, 14) Which makes no sense since these ranges extend beyond the actual size of the matrix. On Tue, Feb 14, 2017 at 6:05 PM, Barry Smith wrote: > > 1) on a single process all the columns are in the diagonal block and > none in the off diagonal block (note that the column partitioning > corresponding to a partitioning of the vector in the product A *x > > 2) on two processes you guessed wrong how many columns PETSc would put > on the first process. You guessed it would put the first three on the first > process and the last three on the second process. > > a) it cannot do that, every column has to been owned by some process > (in the vector x above) so it cannot be 3 and 3, it has to be 4 and 3 or 3 > and 4. > > b) PETSc puts the "extra" columns on the earlier processes not the > later processes. > > For rectangular matrices you really cannot get away with using > "PETSC_DECIDE" for local columns when using preallocation > , since it may decide something different than what you assume. > > I've attached the parallel code that behaves as expected. > > Barry > > > On Feb 14, 2017, at 1:06 PM, Andrew Ho wrote: > > > > The problem isn't only with 1 process, but it seems to be with all > non-square matrices? > > > > For example, here I have a two process program which tries to set a 6x7 > matrix to (each process owns 3 rows): > > > > 0 0 0 1 1 1 1 > > 0 0 0 1 1 1 1 > > 0 0 0 1 1 1 1 > > 0 0 0 1 1 1 1 > > 0 0 0 1 1 1 1 > > 0 0 0 1 1 1 1 > > > > #include > > #include > > > > int main(int argc, char** argv) > > { > > PetscErrorCode err; > > err = PetscInitialize(&argc, &argv, NULL, "help"); > > CHKERRQ(err); > > > > int rank, size; > > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > > MPI_Comm_size(MPI_COMM_WORLD, &size); > > if(size != 2) > > { > > printf("must run with 2 processes"); > > MPI_Abort(MPI_COMM_WORLD, -1); > > } > > > > // create a sparse AIJ matrix distributed across MPI > > Mat A; > > err = MatCreate(MPI_COMM_WORLD, &A); > > CHKERRQ(err); > > err = MatSetType(A, MATMPIAIJ); > > CHKERRQ(err); > > // setup pre-allocation for matrix space > > { > > err = > > MatSetSizes(A, 3, PETSC_DECIDE, 6, 7); > > CHKERRQ(err); > > if(rank == 0) > > { > > PetscInt d_nnz[] = {0, 0, 0}; > > PetscInt o_nnz[] = {4, 4, 4}; > > err = MatMPIAIJSetPreallocation(A, 0, d_nnz, 0, o_nnz); > > CHKERRQ(err); > > } > > else > > { > > PetscInt d_nnz[] = {3, 3, 3}; > > PetscInt o_nnz[] = {1, 1, 1}; > > err = MatMPIAIJSetPreallocation(A, 0, d_nnz, 0, o_nnz); > > CHKERRQ(err); > > } > > } > > err = MatSetUp(A); > > CHKERRQ(err); > > > > // set values inside the matrix > > for (PetscInt row = 0; row < 3; ++row) > > { > > for (PetscInt col = 3; col < 7; ++col) > > { > > err = MatSetValue(A, 3 * rank + row, col, 1, INSERT_VALUES); > > CHKERRQ(err); > > } > > } > > > > err = MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); > > CHKERRQ(err); > > err = MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); > > CHKERRQ(err); > > > > err = MatView(A, PETSC_VIEWER_STDOUT_WORLD); > > CHKERRQ(err); > > > > // free memory > > err = MatDestroy(&A); > > CHKERRQ(err); > > > > // cleanup any internal PETSc data at end of program > > err = PetscFinalize(); > > CHKERRQ(err); > > } > > > > > > On Tue, Feb 14, 2017 at 5:26 AM, Matthew Knepley > wrote: > > On Tue, Feb 14, 2017 at 6:41 AM, Andrew Ho wrote: > > I have a 3x6 matrix, and I want to set it to (just as an example): > > > > 0 0 0 1 1 1 > > 0 0 0 1 1 1 > > 0 0 0 1 1 1 > > > > From what I can tell, according to the documentation the MPIAIJ sparsity > of this matrix is: > > > > I believe this is an inconsistency in PETSc for rectangular matrices. We > divide matrices into diagonal and > > off-diagonal parts mainly to separate communication from computation. > Thus on 1 proc, we never divide > > them, even if the matrix is rectangular. Therefore we misinterpret your > preallocation. > > > > Barry, do you think we want to try and change this? > > > > Matt > > > > d_nnz = [0, 0, 0] > > o_nnz = [3, 3, 3] > > > > However, when I do this, I get the following error: > > > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [0]PETSC ERROR: Argument out of range > > [0]PETSC ERROR: New nonzero at (0,3) caused a malloc > > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to > turn off this check > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.7.5, unknown > > [0]PETSC ERROR: ./ex3 on a arch-linux2-c-opt Tue Feb 14 04:23:56 2017 > > [0]PETSC ERROR: Configure options --with-debugging=0 --COPTFLAGS="-O3 > -march=native" --CXXOPTFLAGS="-O3 -march=native" --FOPTFLAGS="-O3 > -march=native" > > [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 582 in > petsc/src/mat/impls/aij/mpi/mpiaij.c > > [0]PETSC ERROR: #2 MatSetValues() line 1190 in petsc/src/mat/interface/ > matrix.c > > [0]PETSC ERROR: #3 main() line 36 in ex3.c > > [0]PETSC ERROR: No PETSc Option Table entries > > [0]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint at mcs.anl.gov---------- > > > > Here's a working test code: > > > > #include > > #include > > int main(int argc, char** argv) > > { > > PetscErrorCode err; > > err = PetscInitialize(&argc, &argv, NULL, "help"); > > CHKERRQ(err); > > // create a sparse AIJ matrix distributed across MPI > > PetscInt global_width = 6; > > PetscInt global_height = 3; > > Mat A; > > err = MatCreate(MPI_COMM_WORLD, &A); > > CHKERRQ(err); > > err = MatSetType(A, MATMPIAIJ); > > CHKERRQ(err); > > // setup pre-allocation for matrix space > > { > > err = > > MatSetSizes(A, global_height, PETSC_DECIDE, global_height, > global_width); > > CHKERRQ(err); > > PetscInt d_nnz[] = {0, 0, 0}; > > PetscInt o_nnz[] = {3, 3, 3}; > > err = MatMPIAIJSetPreallocation(A, 0, d_nnz, 0, o_nnz); > > CHKERRQ(err); > > } > > err = MatSetUp(A); > > CHKERRQ(err); > > // set values inside the matrix > > for (PetscInt row = 0; row < global_height; ++row) > > { > > for (PetscInt col = global_height; col < global_width; ++col) > > { > > err = MatSetValue(A, row, col, 1, INSERT_VALUES); > > CHKERRQ(err); > > } > > } > > err = MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); > > CHKERRQ(err); > > err = MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); > > CHKERRQ(err); > > err = MatView(A, PETSC_VIEWER_STDOUT_WORLD); > > CHKERRQ(err); > > // free memory > > err = MatDestroy(&A); > > CHKERRQ(err); > > // cleanup any internal PETSc data at end of program > > err = PetscFinalize(); > > CHKERRQ(err); > > } > > > > Am I mis-understanding what the d_nnz and o_nnz parameter are supposed > to mean? > > -- > > Andrew Ho > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > > > > > > > -- > > Andrew Ho > > > -- Andrew Ho -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex3.c Type: text/x-csrc Size: 2094 bytes Desc: not available URL: From matt.landreman at gmail.com Wed Feb 15 10:42:43 2017 From: matt.landreman at gmail.com (Matt Landreman) Date: Wed, 15 Feb 2017 11:42:43 -0500 Subject: [petsc-users] multigrid questions Message-ID: Hi, Petsc developers, A few basic questions about geometric and algebraic multigrid in petsc: Based on the output of -ksp_view, I see that for both -pc_type mg and -pc_type gamg, the smoothing on each multigrid level is implemented using a ksp with default type chebyshev, associated with a pc of default type sor. I also see in section 4.4.5 of the petsc manual that switching to -mg_levels_ksp_type richardson is sometimes recommended, and I see in the ksp examples makefile that ksp ex28.c uses -mg_levels_ksp_type gmres. 1. Does ?textbook? Gauss-Seidel smoothing correspond in petsc to -mg_levels_ksp_type richardson? 2. It says in http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPCHEBYSHEV.html that PETSc?s chebyshev iteration only works for symmetric matrices. So to confirm, for any problem with non-symmetric matrices such as convection-diffusion, setting -mg_levels_ksp_type to some non-default value like richardson or gmres is *mandatory* then? (If so, it would be helpful if this were stated in the manual.) 3. For smoothing with geometric multigrid, is there a way in petsc to do red-black Gauss Seidel? Line relaxation? Thanks, Matt Landreman -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Feb 15 11:11:04 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 15 Feb 2017 11:11:04 -0600 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: <87949040-ce11-c0f2-bac8-77c6f477ca17@mcs.anl.gov> References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> <60D7CDCC-E9D5-472A-9175-64C56DF8B34F@mcs.anl.gov> <87vasc4pz5.fsf@jedbrown.org> <36FFCE9F-ECA8-4B5F-BE53-6B1AD2C7B205@anl.gov> <95c8d53c-5c77-34aa-68dd-3e5211d24c24@mcs.anl.gov> <45E4D3E5-B4DC-4BDC-984A-ED7735723E50@mcs.anl.gov> <03329c72-3572-9f71-5e5b-34741f5d7573@mcs.anl.gov> <9C507CBA-8DC4-4669-BEE7-4560E27EB020@mcs.anl.gov> <110C81DD-DA04-48A4-82B4-F9E527FA3806@mcs.anl.gov> <87949040-ce11-c0f2-bac8-77c6f477ca17@mcs.anl.gov> Message-ID: <45664D89-F34C-4658-8039-74FD0017AA22@mcs.anl.gov> > On Feb 14, 2017, at 11:59 PM, Emil Constantinescu wrote: > > We seem to be needing more than two components and ways to pack them. Allowing them to be dynamically assigned at run time would be really cool. > > We seem to like an API that allows us to specify M and u_t; but would also like to keep support for f(t,u,u_t)=0. Absolutely! No one is saying that IFunction should be removed! > > How about a costly experiment: build an API as discussed below (support the above) that would map everything to the current one 1xRHS and 1xIFunction to see how it would work? In other words, is there a scenario in which we could introduce the new but also keep the old and be reasonable about it? Of concern is whether the new one can be made friendly enough. > > Emil > > On 2/14/17 11:44 PM, Ed Bueler wrote: >>> Jed suggested having any number of "RHS" functions. I don't think we >> need more than 2, 1 for left hand side >>> and 1 for right. If that ends up being not enough we can change to >> have any number of them. Just to be clear. >>> I suggest three functions >>> IFunction which defaults to u_t plus TSSetMassMatrix() which >> changes the default IFunction to M u_t >>> LHS function which defaults to 0, if provided defaults to being >> treated implicitly by IMEX >>> RHS function which defaults to 0, if provided defaults to being >> treated explicitly by IMEX >>> Then a TSSetStiffMatrix(ts,Mat L) (horrible name) that provides u_t >> -Lu = g() + Lu >> >> At my current level of understanding I get this trifecta (quad-fecta?). >> >> In my ice sheet case u_t = f(t,u) + g(t,u), I would not set IFunction >> at all, or I would use a constant. The divergence of flux f(t,u) = >> div(q) would go into the LHSFunction and the elevation-dependent mass >> balance g(t,u) would go into the RHSFunction. I would not use >> TSSetStiffMatrix() at all. >> >> Barry's suggestion makes sense for this and the few other usage cases I >> have already experienced ... kind of a lame endorsement, but, on such a >> weak basis one casts a vote ... >> >> Ed >> >> >> PS I would not use TSSetStiffMatrix() because none of the >> linearizations/simplfications known at the completion of a TS step could >> "capture" the stiff part without being derived a la the Jacobian >> anyway. To see what I mean, note that VINEWTONRSLS has a different >> *dimension* at each Newton iteration. I don't know the >> (in-active-constraint) dimension of the solution I'll get at each time >> step, much less how to scale the various stiff-equation-parts for those >> in-active dimensions. In these problems there is a lot going on in the >> implicit solve, besides just handling stiffness. >> >> >> On Tue, Feb 14, 2017 at 6:20 PM, Barry Smith > > wrote: >> >> >> > On Feb 14, 2017, at 8:56 PM, Emil Constantinescu > wrote: >> > >> > On 2/14/17 4:10 PM, Barry Smith wrote: >> >> Ok, you don't recompile but forcing that into user code is still disgusting. With my api the user code is >> >> >> >>>>> TSSetRHSFunction(ts,NULL,RHSFunction,&ptype[0]); >> >>>>> TSSetLHSFunction(ts,NULL,LHSFunction,&ptype[0]); >> >>>>> TSSetRHSJacobian(ts,Jac,Jac,RHSJacobian,&ptype[0]); >> >>>>> TSSetLHSJacobian(ts,Jac,Jac,LHSJacobian,&ptype[0]); >> >> and -ts_type xxx works correctly for ALL methods, implicit, explicit and imex without requiring any special command line options for different methods. >> > >> > Is this a viable solution? Growing the API to fix this situation will just put a burden with each new TS method after we refactor it in the current landscape. >> >> No just the opposite, the TS implementations will talk to >> functions who will put things together for it. So All implicit >> methods will call something like TSBuildImplicitFunction(), all >> explicit methods will call something like TSBuildExplicitFunction() >> and then IMEX methods will call both of these. In fact likely we can >> refactor to make things a little better than today. Depending on >> options and flagsTSBuildExplicitFunction() would build out of all >> the user provided functions what it needs etc. >> >> One problem with the current code is the TS methods call things with >> the same names as the user API. So implicit methods call >> TSComputeIFunction() while explicit methods call >> TSComputeRHSFunction(). This is wrong because implicit methods >> actually also use the rhs function provided by the user. >> >> The function below absolutely should not be called >> TSComputeIFunction()! It does not just compute IFunction() >> PetscErrorCode TSComputeIFunction(TS ts,PetscReal t,Vec U,Vec >> Udot,Vec Y,PetscBool imex) >> { >> PetscErrorCode ierr; >> TSIFunction ifunction; >> TSRHSFunction rhsfunction; >> void *ctx; >> DM dm; >> >> PetscFunctionBegin; >> PetscValidHeaderSpecific(ts,TS_CLASSID,1); >> PetscValidHeaderSpecific(U,VEC_CLASSID,3); >> PetscValidHeaderSpecific(Udot,VEC_CLASSID,4); >> PetscValidHeaderSpecific(Y,VEC_CLASSID,5); >> >> ierr = TSGetDM(ts,&dm);CHKERRQ(ierr); >> ierr = DMTSGetIFunction(dm,&ifunction,&ctx);CHKERRQ(ierr); >> ierr = DMTSGetRHSFunction(dm,&rhsfunction,NULL);CHKERRQ(ierr); >> >> if (!rhsfunction && !ifunction) >> SETERRQ(PetscObjectComm((PetscObject)ts),PETSC_ERR_USER,"Must call >> TSSetRHSFunction() and / or TSSetIFunction()"); >> >> ierr = PetscLogEventBegin(TS_FunctionEval,ts,U,Udot,Y);CHKERRQ(ierr); >> if (ifunction) { >> PetscStackPush("TS user implicit function"); >> ierr = (*ifunction)(ts,t,U,Udot,Y,ctx);CHKERRQ(ierr); >> PetscStackPop; >> } >> if (imex) { >> if (!ifunction) { >> ierr = VecCopy(Udot,Y);CHKERRQ(ierr); >> } >> } else if (rhsfunction) { >> if (ifunction) { >> Vec Frhs; >> ierr = TSGetRHSVec_Private(ts,&Frhs);CHKERRQ(ierr); >> ierr = TSComputeRHSFunction(ts,t,U,Frhs);CHKERRQ(ierr); >> ierr = VecAXPY(Y,-1,Frhs);CHKERRQ(ierr); >> } else { >> ierr = TSComputeRHSFunction(ts,t,U,Y);CHKERRQ(ierr); >> ierr = VecAYPX(Y,-1,Udot);CHKERRQ(ierr); >> } >> } >> ierr = PetscLogEventEnd(TS_FunctionEval,ts,U,Udot,Y);CHKERRQ(ierr); >> PetscFunctionReturn(0); >> } >> >> The current code entangles too much of the user API to the methods, >> this can be fixed. >> >> > If the user experiments with different ways of splitting the solution they would have to define RHS and IF or RHS and LHS in different ways (according to the splittings they experiment with). It may look disgusting, but I don't see another way around it unless you allow for a list of operators to be defined and then the user to assign them to LHS or RHS. >> >> Jed suggested having any number of "RHS" functions. I don't think >> we need more than 2, 1 for left hand side and 1 for right. If that >> ends up being not enough we can change to have any number of them. >> Just to be clear. I suggest three functions >> >> IFunction which defaults to u_t plus TSSetMassMatrix() which >> changes the default IFunction to M u_t >> LHS function which defaults to 0, if provided defaults to being >> treated implicitly by IMEX >> RHS function which defaults to 0, if provided defaults to being >> treated explicitly by IMEX >> >> Then a TSSetStiffMatrix(ts,Mat L) (horrible name) that provides >> u_t -Lu = g() + Lu >> >> None of the TS implementations will every directly know about >> what the user has provided. They will call the wrapper functions I >> mention above. >> >> I think Jed and Emil may be too enamored with the reductionist >> model of only IFunction() and RHSFunction() to see that though it >> encompasses everything it may not be the best user API. >> >> >> >> > >> >> >> >>> Adding all that logic to keep track of left sides and right >> sides for academic examples is likely not the best development. >> >> I don't think it is "just academic examples", it is all examples >> without a mass matrix. >> >> >> >> Once the user has decided with ts_type to use for production if >> it is fully implicit or explicit then they can depending on the type >> selected, write just a left hand side, just a right hand side for >> higher efficiency (less update of ghost points, fewer iterations >> over loops etc). >> >> >> >> With a constant mass matrix we can have TSSetMassMatrix() and >> then TSSetIFunction() is reserved for when it is absolutely needed. >> > >> > As much as I would disagree with growing the API at the level of >> defining the problem, I think TSSetMassMatrix() would let us do more >> things in the solvers. Also it would be useful to know if the mass >> matrix is singular or not for efficiency reasons. >> > >> > Emil >> > >> >> Barry >> >> >> >> >> >> >> -- >> Ed Bueler >> Dept of Math and Stat and Geophysical Institute >> University of Alaska Fairbanks >> Fairbanks, AK 99775-6660 >> 301C Chapman and 410D Elvey >> 907 474-7693 and 907 474-7199 (fax 907 474-5394) From jed at jedbrown.org Wed Feb 15 12:18:32 2017 From: jed at jedbrown.org (Jed Brown) Date: Wed, 15 Feb 2017 11:18:32 -0700 Subject: [petsc-users] multigrid questions In-Reply-To: References: Message-ID: <87fujf1fnr.fsf@jedbrown.org> Matt Landreman writes: > Hi, Petsc developers, > > A few basic questions about geometric and algebraic multigrid in petsc: > > Based on the output of -ksp_view, I see that for both -pc_type mg and > -pc_type gamg, the smoothing on each multigrid level is implemented using a > ksp with default type chebyshev, associated with a pc of default type sor. > I also see in section 4.4.5 of the petsc manual that switching to > -mg_levels_ksp_type richardson is sometimes recommended, and I see in the > ksp examples makefile that ksp ex28.c uses -mg_levels_ksp_type gmres. > > 1. Does ?textbook? Gauss-Seidel smoothing correspond in petsc to > -mg_levels_ksp_type richardson? Yeah, default Richardson/SOR is plain GS. > 2. It says in > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPCHEBYSHEV.html > that PETSc?s chebyshev iteration only works for symmetric matrices. So to > confirm, for any problem with non-symmetric matrices such as > convection-diffusion, setting -mg_levels_ksp_type to some non-default value > like richardson or gmres is *mandatory* then? (If so, it would be helpful > if this were stated in the manual.) A lot of nonsymmetric problems are "not too non-symmetric". Chebyshev(1) is actually just Richardson. High-degree Chebyshev has a tighter smoothing profile on the real axis so problems with eigenvalues far from the real axis can be unstable. The defaults were chosen because they tend to be robust on a variety of problems. > 3. For smoothing with geometric multigrid, is there a way in petsc to do > red-black Gauss Seidel? Line relaxation? Multi-color GS would need a coloring which is difficult in parallel and the performance tends to be poor when there are many colors. Block Jacobi/ILU smoothers often behave similarly to line smoothers (depends on ordering). Usually people write custom code for discretization-dependent smoothers. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From knepley at gmail.com Wed Feb 15 12:19:05 2017 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 15 Feb 2017 12:19:05 -0600 Subject: [petsc-users] multigrid questions In-Reply-To: References: Message-ID: On Wed, Feb 15, 2017 at 10:42 AM, Matt Landreman wrote: > Hi, Petsc developers, > > A few basic questions about geometric and algebraic multigrid in petsc: > > Based on the output of -ksp_view, I see that for both -pc_type mg and > -pc_type gamg, the smoothing on each multigrid level is implemented using a > ksp with default type chebyshev, associated with a pc of default type sor. > I also see in section 4.4.5 of the petsc manual that switching to > -mg_levels_ksp_type richardson is sometimes recommended, and I see in the > ksp examples makefile that ksp ex28.c uses -mg_levels_ksp_type gmres. > > 1. Does ?textbook? Gauss-Seidel smoothing correspond in petsc to > -mg_levels_ksp_type richardson? > Yes. > 2. It says in http://www.mcs.anl.gov/petsc/petsc-current/docs/ > manualpages/KSP/KSPCHEBYSHEV.html that PETSc?s chebyshev iteration only > works for symmetric matrices. So to confirm, for any problem with > non-symmetric matrices such as convection-diffusion, setting > -mg_levels_ksp_type to some non-default value like richardson or gmres is > *mandatory* then? (If so, it would be helpful if this were stated in the > manual.) > Yes, this is true. > 3. For smoothing with geometric multigrid, is there a way in petsc to do > red-black Gauss Seidel? Line relaxation? > I don't know if Barry has something special, but you could easily do it using PCFIELDSPLIT with the default splitting and Jacobi on each block. Line relaxation is similar, with different blocking and Block-Jacobi. Matt > Thanks, > Matt Landreman > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Wed Feb 15 12:37:10 2017 From: jed at jedbrown.org (Jed Brown) Date: Wed, 15 Feb 2017 11:37:10 -0700 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> <60D7CDCC-E9D5-472A-9175-64C56DF8B34F@mcs.anl.gov> <87vasc4pz5.fsf@jedbrown.org> <6D880F3B-F880-4AFD-975D-5A8A4F4AA8A9@mcs.anl.gov> <87inoc4l17.fsf@jedbrown.org> <87shng2jk6.fsf@jedbrown.org> Message-ID: <87efyz1esp.fsf@jedbrown.org> Ed Bueler writes: > Jed -- > >>> u_t - div (u^k |grad u|^{p-2} grad u) = g(t,u) >>Are you also interested in the case with sliding? > > You must be getting grumpy. There is this 2009 paper about sliding ... > Yes I am interested in sliding but I was trying to keep things simple for > you. ;-) My point was that including sliding potentially adds another stiff/algebraic term so whatever interface we choose better be able to support at least two stiff terms. > The code in question is an under-development example in my book that I am > trying to use to show off SNESVI functionality in an interesting case and > yet not get too nasty. See the help string at the top of the file for what > it actually does: > > https://github.com/bueler/p4pdes/blob/master/c/ch11/ice.c > > In summary, the actual flux is > > q_total = - Gamma H^{n+2} |grad s|^{n-1} grad s + V H Is V really a prescribed function or the solution of an equation that involves u? > and it is all handled implicitly by preference, i.e. in the current API I > have IFunction = u_t - div q_total. > > The real interest of the problem is the failure of long-range communication > of low-(spatial)-frequency information when you hit the free boundary, and > these barriers can emerge anywhere in the domain. (In a mountain range you > can have neighboring small glaciers completely out of phase; how should > multigrid ... or any other preconditioning one would hope for ... work for > all the glaciers in a mountain range? This already exists for variable coefficient problems -- the electric potential in a twisted wire pair is independent despite being nearby points in the solution of a Poisson equation. An important difference from ice sheets is that the Green's functions are highly nonlocal for the twisted wire pair, but very local for ice sheets with sticky beds. With slippery beds (ice shelves are the limiting case), the velocity Green's functions become similar to Poisson. That is the scenario where multigrid is needed. Otherwise the solutions are quite local. For the SIA thickness equation, I think it would be interesting to consider the conductivity tensor in the Newton linearization. d_u [-div (u^k |grad u|^{p-2} grad u)] * w = -div (k w^{k-1} |grad u|^{p-2} grad u) -div (u^k [|grad u|^{p-2} I + (p-2) |grad u|^{p-4} grad u \otimes grad u] grad w) The first part is transport of the perturbation w. The next is diffusion of w with an anisotropic conductivity tensor. > So now I'll emphasize what I said before: it is *degenerate* p-bratu. To my knowledge, "p-Bratu" is just something I invented to combine two nonlinearities. You don't have an e^u nonlinearity so there is no Bratu-type nonlinearity. > Doubly-degenerate because p>2, but the "H^{n+2}" aka "u^k" is the bad > thing that makes the abrupt free boundary.) Sliding, in this regard, > is the easy case, so I am throwing it in by generating an artificial > velocity field; yes this is the coupling that should include the SSA > in the future ... we already have an SSA solver or two that behave > surprisingly well, you know, as long as the boundary conditions can be > figured-out. I think sliding is the only phenomena that produces a nonlocal response. Consider an ice shelf. The thickness and slope at the grounding line instantaneously affect the seaward edge of the shelf. Grounded SIA has no such dynamics. >> I would just have a PetscOptionsEList() for whether to evaluate the >> diffusive term on the left or right. Shouldn't be more than a few lines >> of code. > > So I have something like?: > -ice_diffusive_side left,right > > O.k. What if user says > -ts_type rk -ice_diffusive_side left > Then I get the current brokenness: code runs and produces nonsense because > RK does not call SNESVI (constraint never enforced) *and* diffusive stuff > is never evaluated (because RK does not call IFunction). Yes, I think TSRK should error if IFunction has been specified. > So maybe then I also go through possible -ts_types and check whether the > option combination made sense? Or move some terms to the correct side > based on what that -ts_type should want? Do I include -ts_type sundials > because somebody's config might include that third-party thing (even though > mine doesn't)? I am not suggesting switching on TS type. I guess I felt that having dependent options was acceptable in exchange for it being easy/reliable to know which terms are being treated explicitly or implicitly. If the method takes over that choice, it makes the interface "smarter" which is not necessarily a good thing. It requires copious documentation and diagnostics so the user can tell what the hell is happening under the hood. But I see the utility. > That's what I mean by fragile code. > > Such things build up! Each time you shove cool stuff in my face I have to > get used to seeing ugly stuff turn into beautiful stuff as my aesthetic > sense follows my growing understanding. But it takes time (learning curve) > and all the time it is fragile code (i.e. for me ... because I don't > appreciate all the motivations behind that part of the API ...) > > On the other hand ... I agree that if I am convinced this is really a good > API for something I want, then indeed I will suck it up and learn it and > write it. > > Recall my original request was for the ability to know if the TS type I > have chosen will actually use the SNES I have set ... so long ago it now > seems. I think this is just how design conversations work. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From aero.aju at gmail.com Wed Feb 15 12:58:06 2017 From: aero.aju at gmail.com (Ajit Desai) Date: Wed, 15 Feb 2017 13:58:06 -0500 Subject: [petsc-users] FLOPS vs FLOPS/sec from PETSc -log_summary Message-ID: Hello everyone, I am little unclear between Flops and Flops/sec in the performance profile obtained for my solver using *-log_summary*. Usually, FLOPS stands for (Floating Point Operations Per Second) then what is Flops/sec produced by PETSc log_summary and how it is different from Flops? I have included the output for the reference below. Max Max/Min Avg Total Time (sec): 6.782e+02 1.00003 6.782e+02 Objects: 7.400e+01 1.02778 7.206e+01 Flops: 5.451e+11 1.99585 4.518e+11 3.181e+14 Flops/sec: 8.038e+08 1.99585 6.662e+08 4.690e+11 MPI Messages: 0.000e+00 0.00000 0.000e+00 0.000e+00 MPI Message Lengths: 0.000e+00 0.00000 0.000e+00 0.000e+00 MPI Reductions: 1.000e+00 1.00000 Thanks in advance. *Ajit Desai* PhD Scholar, Carleton University, Canada -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Feb 15 13:01:54 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 15 Feb 2017 13:01:54 -0600 Subject: [petsc-users] PETSc set off-diagonal matrix pre-allocation In-Reply-To: References: Message-ID: <7FCA7E9C-BACA-47FC-801A-F3F6A96044F2@mcs.anl.gov> Strange. I run your example and get an error message as I expect. $ petscmpiexec -n 4 ./ex5 [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Nonconforming object sizes [0]PETSC ERROR: Sum of local lengths 12 does not equal global length 6, my local length 3 likely a call to VecSetSizes() or MatSetSizes() is wrong. See http://www.mcs.anl.gov/petsc/documentation/faq.html#split [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.7.5-3049-ge39721325b GIT Date: 2017-02-05 20:43:45 -0600 [0]PETSC ERROR: ./ex5 on a arch-basic named visitor097-125.wl.anl-external.org by barrysmith Wed Feb 15 12:55:53 2017 [0]PETSC ERROR: Configure options --download-mpich PETSC_ARCH=arch-basic --download-hdf5 --download-triangle [0]PETSC ERROR: #1 PetscSplitOwnership() line 89 in /Users/barrysmith/Src/petsc/src/sys/utils/psplit.c [0]PETSC ERROR: #2 PetscLayoutSetUp() line 137 in /Users/barrysmith/Src/petsc/src/vec/is/utils/pmap.c [0]PETSC ERROR: #3 MatMPIAIJSetPreallocation_MPIAIJ() line 2640 in /Users/barrysmith/Src/petsc/src/mat/impls/aij/mpi/mpiaij.c [0]PETSC ERROR: #4 MatMPIAIJSetPreallocation() line 3368 in /Users/barrysmith/Src/petsc/src/mat/impls/aij/mpi/mpiaij.c [0]PETSC ERROR: #5 main() line 34 in /Users/barrysmith/Src/petsc/test-directory/ex5.c [0]PETSC ERROR: PETSc Option Table entries: [0]PETSC ERROR: -malloc_test [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Nonconforming object sizes [1]PETSC ERROR: Sum of local lengths 12 does not equal global length 6, my local length 3 Note that the input you provide is not consistent. You claim the matrix has a total of 6 rows but then you try to assign 3 rows to the first process, 3 rows to the second, 3 rows to the third, 3 rows to the fourth for a total of 12 rows. Hence it should error. Similarly the sum of the "local sizes" for columns has to sum up to exactly the total number of columns. > On Feb 15, 2017, at 9:52 AM, Andrew Ho wrote: > > I don't understand how the matrices are partitioned then. The documentation online all shows each mpi rank owning entire rows, with no partitioning along columns. > > I've attached an example where I try to partition a 6x7 matrix into 4 ranks: > > rank 0 should own a 3x4 section > rank 1 should own a 3x3 section > rank 2 should own a 3x4 section > rank 3 should own a 3x3 section > > However, when I run the example and print out the ownership range and ownership column range, I get: > > rank 0 owns rows/cols: [0,3), [0, 4) > rank 1 owns rows/cols: [3,6), [4, 7) > rank 2 owns rows/cols: [6,9), [7, 11) > rank 3 owns rows/cols: [9,12), [11, 14) > > Which makes no sense since these ranges extend beyond the actual size of the matrix. > > On Tue, Feb 14, 2017 at 6:05 PM, Barry Smith wrote: > > 1) on a single process all the columns are in the diagonal block and none in the off diagonal block (note that the column partitioning corresponding to a partitioning of the vector in the product A *x > > 2) on two processes you guessed wrong how many columns PETSc would put on the first process. You guessed it would put the first three on the first process and the last three on the second process. > > a) it cannot do that, every column has to been owned by some process (in the vector x above) so it cannot be 3 and 3, it has to be 4 and 3 or 3 and 4. > > b) PETSc puts the "extra" columns on the earlier processes not the later processes. > > For rectangular matrices you really cannot get away with using "PETSC_DECIDE" for local columns when using preallocation > , since it may decide something different than what you assume. > > I've attached the parallel code that behaves as expected. > > Barry > > > On Feb 14, 2017, at 1:06 PM, Andrew Ho wrote: > > > > The problem isn't only with 1 process, but it seems to be with all non-square matrices? > > > > For example, here I have a two process program which tries to set a 6x7 matrix to (each process owns 3 rows): > > > > 0 0 0 1 1 1 1 > > 0 0 0 1 1 1 1 > > 0 0 0 1 1 1 1 > > 0 0 0 1 1 1 1 > > 0 0 0 1 1 1 1 > > 0 0 0 1 1 1 1 > > > > #include > > #include > > > > int main(int argc, char** argv) > > { > > PetscErrorCode err; > > err = PetscInitialize(&argc, &argv, NULL, "help"); > > CHKERRQ(err); > > > > int rank, size; > > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > > MPI_Comm_size(MPI_COMM_WORLD, &size); > > if(size != 2) > > { > > printf("must run with 2 processes"); > > MPI_Abort(MPI_COMM_WORLD, -1); > > } > > > > // create a sparse AIJ matrix distributed across MPI > > Mat A; > > err = MatCreate(MPI_COMM_WORLD, &A); > > CHKERRQ(err); > > err = MatSetType(A, MATMPIAIJ); > > CHKERRQ(err); > > // setup pre-allocation for matrix space > > { > > err = > > MatSetSizes(A, 3, PETSC_DECIDE, 6, 7); > > CHKERRQ(err); > > if(rank == 0) > > { > > PetscInt d_nnz[] = {0, 0, 0}; > > PetscInt o_nnz[] = {4, 4, 4}; > > err = MatMPIAIJSetPreallocation(A, 0, d_nnz, 0, o_nnz); > > CHKERRQ(err); > > } > > else > > { > > PetscInt d_nnz[] = {3, 3, 3}; > > PetscInt o_nnz[] = {1, 1, 1}; > > err = MatMPIAIJSetPreallocation(A, 0, d_nnz, 0, o_nnz); > > CHKERRQ(err); > > } > > } > > err = MatSetUp(A); > > CHKERRQ(err); > > > > // set values inside the matrix > > for (PetscInt row = 0; row < 3; ++row) > > { > > for (PetscInt col = 3; col < 7; ++col) > > { > > err = MatSetValue(A, 3 * rank + row, col, 1, INSERT_VALUES); > > CHKERRQ(err); > > } > > } > > > > err = MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); > > CHKERRQ(err); > > err = MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); > > CHKERRQ(err); > > > > err = MatView(A, PETSC_VIEWER_STDOUT_WORLD); > > CHKERRQ(err); > > > > // free memory > > err = MatDestroy(&A); > > CHKERRQ(err); > > > > // cleanup any internal PETSc data at end of program > > err = PetscFinalize(); > > CHKERRQ(err); > > } > > > > > > On Tue, Feb 14, 2017 at 5:26 AM, Matthew Knepley wrote: > > On Tue, Feb 14, 2017 at 6:41 AM, Andrew Ho wrote: > > I have a 3x6 matrix, and I want to set it to (just as an example): > > > > 0 0 0 1 1 1 > > 0 0 0 1 1 1 > > 0 0 0 1 1 1 > > > > From what I can tell, according to the documentation the MPIAIJ sparsity of this matrix is: > > > > I believe this is an inconsistency in PETSc for rectangular matrices. We divide matrices into diagonal and > > off-diagonal parts mainly to separate communication from computation. Thus on 1 proc, we never divide > > them, even if the matrix is rectangular. Therefore we misinterpret your preallocation. > > > > Barry, do you think we want to try and change this? > > > > Matt > > > > d_nnz = [0, 0, 0] > > o_nnz = [3, 3, 3] > > > > However, when I do this, I get the following error: > > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [0]PETSC ERROR: Argument out of range > > [0]PETSC ERROR: New nonzero at (0,3) caused a malloc > > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.7.5, unknown > > [0]PETSC ERROR: ./ex3 on a arch-linux2-c-opt Tue Feb 14 04:23:56 2017 > > [0]PETSC ERROR: Configure options --with-debugging=0 --COPTFLAGS="-O3 -march=native" --CXXOPTFLAGS="-O3 -march=native" --FOPTFLAGS="-O3 -march=native" > > [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 582 in petsc/src/mat/impls/aij/mpi/mpiaij.c > > [0]PETSC ERROR: #2 MatSetValues() line 1190 in petsc/src/mat/interface/matrix.c > > [0]PETSC ERROR: #3 main() line 36 in ex3.c > > [0]PETSC ERROR: No PETSc Option Table entries > > [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- > > > > Here's a working test code: > > > > #include > > #include > > int main(int argc, char** argv) > > { > > PetscErrorCode err; > > err = PetscInitialize(&argc, &argv, NULL, "help"); > > CHKERRQ(err); > > // create a sparse AIJ matrix distributed across MPI > > PetscInt global_width = 6; > > PetscInt global_height = 3; > > Mat A; > > err = MatCreate(MPI_COMM_WORLD, &A); > > CHKERRQ(err); > > err = MatSetType(A, MATMPIAIJ); > > CHKERRQ(err); > > // setup pre-allocation for matrix space > > { > > err = > > MatSetSizes(A, global_height, PETSC_DECIDE, global_height, global_width); > > CHKERRQ(err); > > PetscInt d_nnz[] = {0, 0, 0}; > > PetscInt o_nnz[] = {3, 3, 3}; > > err = MatMPIAIJSetPreallocation(A, 0, d_nnz, 0, o_nnz); > > CHKERRQ(err); > > } > > err = MatSetUp(A); > > CHKERRQ(err); > > // set values inside the matrix > > for (PetscInt row = 0; row < global_height; ++row) > > { > > for (PetscInt col = global_height; col < global_width; ++col) > > { > > err = MatSetValue(A, row, col, 1, INSERT_VALUES); > > CHKERRQ(err); > > } > > } > > err = MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); > > CHKERRQ(err); > > err = MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); > > CHKERRQ(err); > > err = MatView(A, PETSC_VIEWER_STDOUT_WORLD); > > CHKERRQ(err); > > // free memory > > err = MatDestroy(&A); > > CHKERRQ(err); > > // cleanup any internal PETSc data at end of program > > err = PetscFinalize(); > > CHKERRQ(err); > > } > > > > Am I mis-understanding what the d_nnz and o_nnz parameter are supposed to mean? > > -- > > Andrew Ho > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > > > > > > -- > > Andrew Ho > > > > > > -- > Andrew Ho > From jychang48 at gmail.com Wed Feb 15 13:03:49 2017 From: jychang48 at gmail.com (Justin Chang) Date: Wed, 15 Feb 2017 13:03:49 -0600 Subject: [petsc-users] FLOPS vs FLOPS/sec from PETSc -log_summary In-Reply-To: References: Message-ID: "Flops" here simply stands for floating point operations, not floating point operations per second. On Wed, Feb 15, 2017 at 12:58 PM, Ajit Desai wrote: > Hello everyone, > > I am little unclear between Flops and Flops/sec in the performance profile > obtained for my solver using *-log_summary*. > Usually, FLOPS stands for (Floating Point Operations Per Second) then what > is Flops/sec produced by PETSc log_summary and how it is different from > Flops? > > I have included the output for the reference below. > > Max Max/Min > Avg Total > Time (sec): 6.782e+02 1.00003 6.782e+02 > Objects: 7.400e+01 1.02778 7.206e+01 > Flops: 5.451e+11 1.99585 4.518e+11 > 3.181e+14 > Flops/sec: 8.038e+08 1.99585 6.662e+08 > 4.690e+11 > MPI Messages: 0.000e+00 0.00000 0.000e+00 0.000e+00 > MPI Message Lengths: 0.000e+00 0.00000 0.000e+00 0.000e+00 > MPI Reductions: 1.000e+00 1.00000 > > Thanks in advance. > *Ajit Desai* > PhD Scholar, Carleton University, Canada > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Feb 15 13:06:48 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 15 Feb 2017 13:06:48 -0600 Subject: [petsc-users] FLOPS vs FLOPS/sec from PETSc -log_summary In-Reply-To: References: Message-ID: In PETSc FLOPS refers to (F)loating (P)oint (OP)eration(s) :-) Hence the confusing notation in our output. Barry Yes. this is confusing but it ended up this way because no one says, "this calculation took 500 FLOP", that just sounds weird. Even Total FLOP sounds weird. Any suggestions for how to phrase it without the confusion? > On Feb 15, 2017, at 12:58 PM, Ajit Desai wrote: > > Hello everyone, > > I am little unclear between Flops and Flops/sec in the performance profile obtained for my solver using -log_summary. > Usually, FLOPS stands for (Floating Point Operations Per Second) then what is Flops/sec produced by PETSc log_summary and how it is different from Flops? > > I have included the output for the reference below. > > Max Max/Min Avg Total > Time (sec): 6.782e+02 1.00003 6.782e+02 > Objects: 7.400e+01 1.02778 7.206e+01 > Flops: 5.451e+11 1.99585 4.518e+11 3.181e+14 > Flops/sec: 8.038e+08 1.99585 6.662e+08 4.690e+11 > MPI Messages: 0.000e+00 0.00000 0.000e+00 0.000e+00 > MPI Message Lengths: 0.000e+00 0.00000 0.000e+00 0.000e+00 > MPI Reductions: 1.000e+00 1.00000 > > Thanks in advance. > Ajit Desai > PhD Scholar, Carleton University, Canada From bsmith at mcs.anl.gov Wed Feb 15 13:13:29 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 15 Feb 2017 13:13:29 -0600 Subject: [petsc-users] multigrid questions In-Reply-To: References: Message-ID: > On Feb 15, 2017, at 12:19 PM, Matthew Knepley wrote: > > On Wed, Feb 15, 2017 at 10:42 AM, Matt Landreman wrote: > Hi, Petsc developers, > > A few basic questions about geometric and algebraic multigrid in petsc: > > Based on the output of -ksp_view, I see that for both -pc_type mg and -pc_type gamg, the smoothing on each multigrid level is implemented using a ksp with default type chebyshev, associated with a pc of default type sor. I also see in section 4.4.5 of the petsc manual that switching to -mg_levels_ksp_type richardson is sometimes recommended, and I see in the ksp examples makefile that ksp ex28.c uses -mg_levels_ksp_type gmres. > > 1. Does ?textbook? Gauss-Seidel smoothing correspond in petsc to -mg_levels_ksp_type richardson? > > Yes. > > 2. It says in http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPCHEBYSHEV.html that PETSc?s chebyshev iteration only works for symmetric matrices. So to confirm, for any problem with non-symmetric matrices such as convection-diffusion, setting -mg_levels_ksp_type to some non-default value like richardson or gmres is mandatory then? (If so, it would be helpful if this were stated in the manual.) > > Yes, this is true. > > 3. For smoothing with geometric multigrid, is there a way in petsc to do red-black Gauss Seidel? Line relaxation? > > I don't know if Barry has something special, but you could easily do it using PCFIELDSPLIT with the default splitting and > Jacobi on each block. Line relaxation is similar, with different blocking and Block-Jacobi. For optimal implementation of line relaxation (assuming each process is only doing sequential lines) you would write a PCSHELL that looped over the "lines" for your geometry and use inside it a PC created with the tridiagonal matrix for a line. Line relaxation in parallel opens all kinds of questions; it is possible but I suspect it will have poor parallel efficiency. > > Matt > > Thanks, > Matt Landreman > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From aero.aju at gmail.com Wed Feb 15 13:24:18 2017 From: aero.aju at gmail.com (Ajit Desai) Date: Wed, 15 Feb 2017 14:24:18 -0500 Subject: [petsc-users] FLOPS vs FLOPS/sec from PETSc -log_summary In-Reply-To: References: Message-ID: Thanks, Justin and Barry, That makes it clear now and I agree, using FLOP is sounds weird. However, making it clear in the profiling section of PETSc manual would really help. If I am not wrong, I couldn't find any such statement in PETSc manuals (for ver-6.3.6). my suggestion : FLOPc : Floating point operaions count. Also it goes well with PETSc. *Ajit Desai* PhD Scholar, Carleton University, Canada On Wed, Feb 15, 2017 at 2:06 PM, Barry Smith wrote: > > In PETSc FLOPS refers to (F)loating (P)oint (OP)eration(s) :-) Hence > the confusing notation in our output. > > Barry > > Yes. this is confusing but it ended up this way because no one says, "this > calculation took 500 FLOP", that just sounds weird. Even Total FLOP sounds > weird. Any suggestions for how to phrase it without the confusion? > > > > > On Feb 15, 2017, at 12:58 PM, Ajit Desai wrote: > > > > Hello everyone, > > > > I am little unclear between Flops and Flops/sec in the performance > profile obtained for my solver using -log_summary. > > Usually, FLOPS stands for (Floating Point Operations Per Second) then > what is Flops/sec produced by PETSc log_summary and how it is different > from Flops? > > > > I have included the output for the reference below. > > > > Max Max/Min > Avg Total > > Time (sec): 6.782e+02 1.00003 6.782e+02 > > Objects: 7.400e+01 1.02778 7.206e+01 > > Flops: 5.451e+11 1.99585 4.518e+11 > 3.181e+14 > > Flops/sec: 8.038e+08 1.99585 6.662e+08 > 4.690e+11 > > MPI Messages: 0.000e+00 0.00000 0.000e+00 0.000e+00 > > MPI Message Lengths: 0.000e+00 0.00000 0.000e+00 0.000e+00 > > MPI Reductions: 1.000e+00 1.00000 > > > > Thanks in advance. > > Ajit Desai > > PhD Scholar, Carleton University, Canada > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas.drinkt.thee at gmail.com Wed Feb 15 13:25:13 2017 From: lukas.drinkt.thee at gmail.com (Lukas van de Wiel) Date: Wed, 15 Feb 2017 20:25:13 +0100 Subject: [petsc-users] FLOPS vs FLOPS/sec from PETSc -log_summary In-Reply-To: References: Message-ID: In that case, do not care whether it becomes a literary 'tour de force', and best name it FLOP and FLOP/s to avoid any redundancy. Cheers Lukas On Wed, Feb 15, 2017 at 8:06 PM, Barry Smith wrote: > > In PETSc FLOPS refers to (F)loating (P)oint (OP)eration(s) :-) Hence > the confusing notation in our output. > > Barry > > Yes. this is confusing but it ended up this way because no one says, "this > calculation took 500 FLOP", that just sounds weird. Even Total FLOP sounds > weird. Any suggestions for how to phrase it without the confusion? > > > > > On Feb 15, 2017, at 12:58 PM, Ajit Desai wrote: > > > > Hello everyone, > > > > I am little unclear between Flops and Flops/sec in the performance > profile obtained for my solver using -log_summary. > > Usually, FLOPS stands for (Floating Point Operations Per Second) then > what is Flops/sec produced by PETSc log_summary and how it is different > from Flops? > > > > I have included the output for the reference below. > > > > Max Max/Min > Avg Total > > Time (sec): 6.782e+02 1.00003 6.782e+02 > > Objects: 7.400e+01 1.02778 7.206e+01 > > Flops: 5.451e+11 1.99585 4.518e+11 > 3.181e+14 > > Flops/sec: 8.038e+08 1.99585 6.662e+08 > 4.690e+11 > > MPI Messages: 0.000e+00 0.00000 0.000e+00 0.000e+00 > > MPI Message Lengths: 0.000e+00 0.00000 0.000e+00 0.000e+00 > > MPI Reductions: 1.000e+00 1.00000 > > > > Thanks in advance. > > Ajit Desai > > PhD Scholar, Carleton University, Canada > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Feb 15 13:26:11 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 15 Feb 2017 13:26:11 -0600 Subject: [petsc-users] Newbie question : iterative solver - algorithm and performance In-Reply-To: <915837361.5274745.1487163839533@mail.yahoo.com> References: <2007645858.4961090.1487140811648.ref@mail.yahoo.com> <2007645858.4961090.1487140811648@mail.yahoo.com> <915837361.5274745.1487163839533@mail.yahoo.com> Message-ID: > On Feb 15, 2017, at 7:03 AM, lixin chu wrote: > > I think the chapter 4 of the user manual more or less answers my first question already ... > > rgds > lixin > > > On Wednesday, 15 February 2017, 14:40, lixin chu wrote: > > > Hello, > New to PETSc, appreciate any help - > > I have done some experiment with MUMPS (the direct solver for sparse matrix), and very interested to try out PETSc now. I have a large sparse symmetric matrix (3 millions x 3 millions, complex data type). > > Some questions I have: > - I am assuming that I should select one algorithm of "Krylov methods", which algorithm is a good option, GMRES, CG, or others ? (I am not a domain expert, but helping developing a program to test the matrix). GMRES is a "safe" choice. You can try -ksp_type cg -ksp_cg_type symmetric (or if the matrix is hermitian; i.e. transpose A = complex conjugate of A then type hermitian instead of symmetric). > - do all the algorithms support distributed architecture (multiple machines, multiple cores) Yes > - are there any performance test data ? (total run time for example) Run your code with -view_summary and it will print information at the end about where it has spent the time doing the computation. > > > thank you very much, > LX > > From lukas.drinkt.thee at gmail.com Wed Feb 15 13:29:02 2017 From: lukas.drinkt.thee at gmail.com (Lukas van de Wiel) Date: Wed, 15 Feb 2017 20:29:02 +0100 Subject: [petsc-users] Newbie question : iterative solver - algorithm and performance In-Reply-To: References: <2007645858.4961090.1487140811648.ref@mail.yahoo.com> <2007645858.4961090.1487140811648@mail.yahoo.com> <915837361.5274745.1487163839533@mail.yahoo.com> Message-ID: The wonderful people of PETSc have made a comprehensible list of all the included algorithms and whether they work in parallel or not. It is here: http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html Have a great day Lukas On Wed, Feb 15, 2017 at 8:26 PM, Barry Smith wrote: > > > On Feb 15, 2017, at 7:03 AM, lixin chu wrote: > > > > I think the chapter 4 of the user manual more or less answers my first > question already ... > > > > rgds > > lixin > > > > > > On Wednesday, 15 February 2017, 14:40, lixin chu > wrote: > > > > > > Hello, > > New to PETSc, appreciate any help - > > > > I have done some experiment with MUMPS (the direct solver for sparse > matrix), and very interested to try out PETSc now. I have a large sparse > symmetric matrix (3 millions x 3 millions, complex data type). > > > > Some questions I have: > > - I am assuming that I should select one algorithm of "Krylov methods", > which algorithm is a good option, GMRES, CG, or others ? (I am not a domain > expert, but helping developing a program to test the matrix). > > GMRES is a "safe" choice. You can try -ksp_type cg -ksp_cg_type > symmetric (or if the matrix is hermitian; i.e. transpose A = complex > conjugate of A then type hermitian instead of symmetric). > > > > - do all the algorithms support distributed architecture (multiple > machines, multiple cores) > > Yes > > > - are there any performance test data ? (total run time for example) > > Run your code with -view_summary and it will print information at the > end about where it has spent the time doing the computation. > > > > > > > > thank you very much, > > LX > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From patrick.sanan at gmail.com Wed Feb 15 13:28:48 2017 From: patrick.sanan at gmail.com (Patrick Sanan) Date: Wed, 15 Feb 2017 20:28:48 +0100 Subject: [petsc-users] Newbie question : iterative solver - algorithm and performance In-Reply-To: References: <2007645858.4961090.1487140811648.ref@mail.yahoo.com> <2007645858.4961090.1487140811648@mail.yahoo.com> <915837361.5274745.1487163839533@mail.yahoo.com> Message-ID: On Wed, Feb 15, 2017 at 8:26 PM, Barry Smith wrote: > >> On Feb 15, 2017, at 7:03 AM, lixin chu wrote: >> >> I think the chapter 4 of the user manual more or less answers my first question already ... >> >> rgds >> lixin >> >> >> On Wednesday, 15 February 2017, 14:40, lixin chu wrote: >> >> >> Hello, >> New to PETSc, appreciate any help - >> >> I have done some experiment with MUMPS (the direct solver for sparse matrix), and very interested to try out PETSc now. I have a large sparse symmetric matrix (3 millions x 3 millions, complex data type). >> >> Some questions I have: >> - I am assuming that I should select one algorithm of "Krylov methods", which algorithm is a good option, GMRES, CG, or others ? (I am not a domain expert, but helping developing a program to test the matrix). > > GMRES is a "safe" choice. You can try -ksp_type cg -ksp_cg_type symmetric (or if the matrix is hermitian; i.e. transpose A = complex conjugate of A then type hermitian instead of symmetric). > > >> - do all the algorithms support distributed architecture (multiple machines, multiple cores) > > Yes > >> - are there any performance test data ? (total run time for example) > > Run your code with -view_summary and it will print information at the end about where it has spent the time doing the computation. Barry probably meant -log_view here. > > >> >> >> thank you very much, >> LX >> >> > From jed at jedbrown.org Wed Feb 15 13:34:54 2017 From: jed at jedbrown.org (Jed Brown) Date: Wed, 15 Feb 2017 12:34:54 -0700 Subject: [petsc-users] multigrid questions In-Reply-To: References: Message-ID: <87bmu31c4h.fsf@jedbrown.org> Barry Smith writes: > For optimal implementation of line relaxation (assuming each > process is only doing sequential lines) you would write a PCSHELL > that looped over the "lines" for your geometry and use inside it a > PC created with the tridiagonal matrix for a line. Or (less code) assemble a Pmat that skips insertions across lines. That will make a bunch of tridiagonal segments on each process and a direct solver will be trivially cheap (ILU(0) is a direct solver). -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From andrewh0 at uw.edu Wed Feb 15 14:34:54 2017 From: andrewh0 at uw.edu (Andrew Ho) Date: Wed, 15 Feb 2017 12:34:54 -0800 Subject: [petsc-users] PETSc set off-diagonal matrix pre-allocation In-Reply-To: <7FCA7E9C-BACA-47FC-801A-F3F6A96044F2@mcs.anl.gov> References: <7FCA7E9C-BACA-47FC-801A-F3F6A96044F2@mcs.anl.gov> Message-ID: I'm guessing it's because I'm using an optimized build why I get less error checking. But regardless, what does the "local" column size mean in terms of which MPI rank owns what portion of the matrix? Is that the size of the local diagonal block when specifying the non-zeros for pre-allocation? The documentation does not make this part clear at all to me. On Wed, Feb 15, 2017 at 11:01 AM, Barry Smith wrote: > > Strange. I run your example and get an error message as I expect. > > > $ petscmpiexec -n 4 ./ex5 > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Nonconforming object sizes > [0]PETSC ERROR: Sum of local lengths 12 does not equal global length 6, my > local length 3 > likely a call to VecSetSizes() or MatSetSizes() is wrong. > See http://www.mcs.anl.gov/petsc/documentation/faq.html#split > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.7.5-3049-ge39721325b > GIT Date: 2017-02-05 20:43:45 -0600 > [0]PETSC ERROR: ./ex5 on a arch-basic named visitor097-125.wl.anl- > external.org by barrysmith Wed Feb 15 12:55:53 2017 > [0]PETSC ERROR: Configure options --download-mpich PETSC_ARCH=arch-basic > --download-hdf5 --download-triangle > [0]PETSC ERROR: #1 PetscSplitOwnership() line 89 in > /Users/barrysmith/Src/petsc/src/sys/utils/psplit.c > [0]PETSC ERROR: #2 PetscLayoutSetUp() line 137 in > /Users/barrysmith/Src/petsc/src/vec/is/utils/pmap.c > [0]PETSC ERROR: #3 MatMPIAIJSetPreallocation_MPIAIJ() line 2640 in > /Users/barrysmith/Src/petsc/src/mat/impls/aij/mpi/mpiaij.c > [0]PETSC ERROR: #4 MatMPIAIJSetPreallocation() line 3368 in > /Users/barrysmith/Src/petsc/src/mat/impls/aij/mpi/mpiaij.c > [0]PETSC ERROR: #5 main() line 34 in /Users/barrysmith/Src/petsc/ > test-directory/ex5.c > [0]PETSC ERROR: PETSc Option Table entries: > [0]PETSC ERROR: -malloc_test > [0]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint at mcs.anl.gov---------- > [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [1]PETSC ERROR: Nonconforming object sizes > [1]PETSC ERROR: Sum of local lengths 12 does not equal global length 6, my > local length 3 > > > Note that the input you provide is not consistent. You claim the matrix > has a total of 6 rows but then you try to assign 3 rows to the first > process, 3 rows to the second, 3 rows to the third, 3 rows to the fourth > for a total of 12 rows. Hence it should error. Similarly the sum of the > "local sizes" for columns has to sum up to exactly the total number of > columns. > > > > > > On Feb 15, 2017, at 9:52 AM, Andrew Ho wrote: > > > > I don't understand how the matrices are partitioned then. The > documentation online all shows each mpi rank owning entire rows, with no > partitioning along columns. > > > > I've attached an example where I try to partition a 6x7 matrix into 4 > ranks: > > > > rank 0 should own a 3x4 section > > rank 1 should own a 3x3 section > > rank 2 should own a 3x4 section > > rank 3 should own a 3x3 section > > > > However, when I run the example and print out the ownership range and > ownership column range, I get: > > > > rank 0 owns rows/cols: [0,3), [0, 4) > > rank 1 owns rows/cols: [3,6), [4, 7) > > rank 2 owns rows/cols: [6,9), [7, 11) > > rank 3 owns rows/cols: [9,12), [11, 14) > > > > Which makes no sense since these ranges extend beyond the actual size of > the matrix. > > > > On Tue, Feb 14, 2017 at 6:05 PM, Barry Smith wrote: > > > > 1) on a single process all the columns are in the diagonal block and > none in the off diagonal block (note that the column partitioning > corresponding to a partitioning of the vector in the product A *x > > > > 2) on two processes you guessed wrong how many columns PETSc would > put on the first process. You guessed it would put the first three on the > first process and the last three on the second process. > > > > a) it cannot do that, every column has to been owned by some process > (in the vector x above) so it cannot be 3 and 3, it has to be 4 and 3 or 3 > and 4. > > > > b) PETSc puts the "extra" columns on the earlier processes not the > later processes. > > > > For rectangular matrices you really cannot get away with using > "PETSC_DECIDE" for local columns when using preallocation > > , since it may decide something different than what you assume. > > > > I've attached the parallel code that behaves as expected. > > > > Barry > > > > > On Feb 14, 2017, at 1:06 PM, Andrew Ho wrote: > > > > > > The problem isn't only with 1 process, but it seems to be with all > non-square matrices? > > > > > > For example, here I have a two process program which tries to set a > 6x7 matrix to (each process owns 3 rows): > > > > > > 0 0 0 1 1 1 1 > > > 0 0 0 1 1 1 1 > > > 0 0 0 1 1 1 1 > > > 0 0 0 1 1 1 1 > > > 0 0 0 1 1 1 1 > > > 0 0 0 1 1 1 1 > > > > > > #include > > > #include > > > > > > int main(int argc, char** argv) > > > { > > > PetscErrorCode err; > > > err = PetscInitialize(&argc, &argv, NULL, "help"); > > > CHKERRQ(err); > > > > > > int rank, size; > > > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > > > MPI_Comm_size(MPI_COMM_WORLD, &size); > > > if(size != 2) > > > { > > > printf("must run with 2 processes"); > > > MPI_Abort(MPI_COMM_WORLD, -1); > > > } > > > > > > // create a sparse AIJ matrix distributed across MPI > > > Mat A; > > > err = MatCreate(MPI_COMM_WORLD, &A); > > > CHKERRQ(err); > > > err = MatSetType(A, MATMPIAIJ); > > > CHKERRQ(err); > > > // setup pre-allocation for matrix space > > > { > > > err = > > > MatSetSizes(A, 3, PETSC_DECIDE, 6, 7); > > > CHKERRQ(err); > > > if(rank == 0) > > > { > > > PetscInt d_nnz[] = {0, 0, 0}; > > > PetscInt o_nnz[] = {4, 4, 4}; > > > err = MatMPIAIJSetPreallocation(A, 0, d_nnz, 0, o_nnz); > > > CHKERRQ(err); > > > } > > > else > > > { > > > PetscInt d_nnz[] = {3, 3, 3}; > > > PetscInt o_nnz[] = {1, 1, 1}; > > > err = MatMPIAIJSetPreallocation(A, 0, d_nnz, 0, o_nnz); > > > CHKERRQ(err); > > > } > > > } > > > err = MatSetUp(A); > > > CHKERRQ(err); > > > > > > // set values inside the matrix > > > for (PetscInt row = 0; row < 3; ++row) > > > { > > > for (PetscInt col = 3; col < 7; ++col) > > > { > > > err = MatSetValue(A, 3 * rank + row, col, 1, INSERT_VALUES); > > > CHKERRQ(err); > > > } > > > } > > > > > > err = MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); > > > CHKERRQ(err); > > > err = MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); > > > CHKERRQ(err); > > > > > > err = MatView(A, PETSC_VIEWER_STDOUT_WORLD); > > > CHKERRQ(err); > > > > > > // free memory > > > err = MatDestroy(&A); > > > CHKERRQ(err); > > > > > > // cleanup any internal PETSc data at end of program > > > err = PetscFinalize(); > > > CHKERRQ(err); > > > } > > > > > > > > > On Tue, Feb 14, 2017 at 5:26 AM, Matthew Knepley > wrote: > > > On Tue, Feb 14, 2017 at 6:41 AM, Andrew Ho wrote: > > > I have a 3x6 matrix, and I want to set it to (just as an example): > > > > > > 0 0 0 1 1 1 > > > 0 0 0 1 1 1 > > > 0 0 0 1 1 1 > > > > > > From what I can tell, according to the documentation the MPIAIJ > sparsity of this matrix is: > > > > > > I believe this is an inconsistency in PETSc for rectangular matrices. > We divide matrices into diagonal and > > > off-diagonal parts mainly to separate communication from computation. > Thus on 1 proc, we never divide > > > them, even if the matrix is rectangular. Therefore we misinterpret > your preallocation. > > > > > > Barry, do you think we want to try and change this? > > > > > > Matt > > > > > > d_nnz = [0, 0, 0] > > > o_nnz = [3, 3, 3] > > > > > > However, when I do this, I get the following error: > > > > > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > > [0]PETSC ERROR: Argument out of range > > > [0]PETSC ERROR: New nonzero at (0,3) caused a malloc > > > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to > turn off this check > > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/ > documentation/faq.html for trouble shooting. > > > [0]PETSC ERROR: Petsc Release Version 3.7.5, unknown > > > [0]PETSC ERROR: ./ex3 on a arch-linux2-c-opt Tue Feb 14 04:23:56 2017 > > > [0]PETSC ERROR: Configure options --with-debugging=0 --COPTFLAGS="-O3 > -march=native" --CXXOPTFLAGS="-O3 -march=native" --FOPTFLAGS="-O3 > -march=native" > > > [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 582 in > petsc/src/mat/impls/aij/mpi/mpiaij.c > > > [0]PETSC ERROR: #2 MatSetValues() line 1190 in petsc/src/mat/interface/ > matrix.c > > > [0]PETSC ERROR: #3 main() line 36 in ex3.c > > > [0]PETSC ERROR: No PETSc Option Table entries > > > [0]PETSC ERROR: ----------------End of Error Message -------send > entire error message to petsc-maint at mcs.anl.gov---------- > > > > > > Here's a working test code: > > > > > > #include > > > #include > > > int main(int argc, char** argv) > > > { > > > PetscErrorCode err; > > > err = PetscInitialize(&argc, &argv, NULL, "help"); > > > CHKERRQ(err); > > > // create a sparse AIJ matrix distributed across MPI > > > PetscInt global_width = 6; > > > PetscInt global_height = 3; > > > Mat A; > > > err = MatCreate(MPI_COMM_WORLD, &A); > > > CHKERRQ(err); > > > err = MatSetType(A, MATMPIAIJ); > > > CHKERRQ(err); > > > // setup pre-allocation for matrix space > > > { > > > err = > > > MatSetSizes(A, global_height, PETSC_DECIDE, global_height, > global_width); > > > CHKERRQ(err); > > > PetscInt d_nnz[] = {0, 0, 0}; > > > PetscInt o_nnz[] = {3, 3, 3}; > > > err = MatMPIAIJSetPreallocation(A, 0, d_nnz, 0, o_nnz); > > > CHKERRQ(err); > > > } > > > err = MatSetUp(A); > > > CHKERRQ(err); > > > // set values inside the matrix > > > for (PetscInt row = 0; row < global_height; ++row) > > > { > > > for (PetscInt col = global_height; col < global_width; ++col) > > > { > > > err = MatSetValue(A, row, col, 1, INSERT_VALUES); > > > CHKERRQ(err); > > > } > > > } > > > err = MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); > > > CHKERRQ(err); > > > err = MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); > > > CHKERRQ(err); > > > err = MatView(A, PETSC_VIEWER_STDOUT_WORLD); > > > CHKERRQ(err); > > > // free memory > > > err = MatDestroy(&A); > > > CHKERRQ(err); > > > // cleanup any internal PETSc data at end of program > > > err = PetscFinalize(); > > > CHKERRQ(err); > > > } > > > > > > Am I mis-understanding what the d_nnz and o_nnz parameter are supposed > to mean? > > > -- > > > Andrew Ho > > > > > > > > > > > > -- > > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > > -- Norbert Wiener > > > > > > > > > > > > -- > > > Andrew Ho > > > > > > > > > > > > -- > > Andrew Ho > > > > -- Andrew Ho -------------- next part -------------- An HTML attachment was scrubbed... URL: From elbueler at alaska.edu Wed Feb 15 14:57:13 2017 From: elbueler at alaska.edu (Ed Bueler) Date: Wed, 15 Feb 2017 11:57:13 -0900 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: <87efyz1esp.fsf@jedbrown.org> References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> <60D7CDCC-E9D5-472A-9175-64C56DF8B34F@mcs.anl.gov> <87vasc4pz5.fsf@jedbrown.org> <6D880F3B-F880-4AFD-975D-5A8A4F4AA8A9@mcs.anl.gov> <87inoc4l17.fsf@jedbrown.org> <87shng2jk6.fsf@jedbrown.org> <87efyz1esp.fsf@jedbrown.org> Message-ID: Jed -- > My point was that including sliding potentially adds another > stiff/algebraic term so whatever interface we choose better be able to > support at least two stiff terms. Yes, totally agreed w.r.t. the interface design. (The DAE that you are solving in that case is stiff.) However, *my* observation about the current interface design is actually that it was never apparently tested with TS+SNESVI and is awkward in that case. IMHO the design for time-stepping on constrained problems should continue to allow use of SNESVI (in implicit and imex cases) if an LHSfunction (or similar) is provided, *and* have a mechanism where explicit time-solvers can use projection onto the bounds---conceivably this could be a trivialized SNESVI instance---if bounds are present. This is because mere evaluation of LHSFunction and RHSFunction will not otherwise generate the "side effect" of enforcing the constraints which were provided by FormBounds() in using the SNESVI. In both the current and proposed designs in this thread, for implicit and imex methods the constraint enforcement is inside SNESVI *but* for explicit time-stepping it would be a user-written projection in TSPostStep(). Furthermore the LHSFunction might have to be rewritten to allow non-feasible inputs just to support explicit multistage time-stepping. Yuk. Note that there are VIs in L^2 and they *are* "projection onto the bounds" in the just-mentioned sense. See section II.3 of Kinderlehrer and Stampacchia. (Elliptic VIs in W^{1,p} are what one usually sees.) The switch from VI in Sobolev space to VI in L^2 is exactly what you see when you semi-discretize these obstacle-like problems in time only and then compare implicit and explicit timestepping, respectively. So ya'll are having a good argument ... but I am worried about what I asked about anyway. >> q_total = - Gamma H^{n+2} |grad s|^{n-1} grad s + V H > Is V really a prescribed function or the solution of an equation that > involves u? I know that the problem you want me to solve is a big DAE including both mass and momentum conservation. (Do you really think that all this time I haven't noticed?) My interest is in good solvers for what you call the "local problem". This local problem is the one that limits PISM hybrid runs *and* I think it is why Stokes is not really functional yet in anyone's time-stepping model. (If they treat the time-stepping explicitly then they have the SIA-type time-step restriction kick in as a solver convergence problem, because the SIA tells the truth in the majority of the area where the bed is sticky.) Thus my test case code does not waste time generating a real V (i.e. solving simultaneously for H and V at each time step, for the DAE you want me to solve ...). > For the SIA thickness equation, I think it would be interesting to > consider the conductivity tensor in the Newton linearization. > d_u [-div (u^k |grad u|^{p-2} grad u)] * w > = -div (k w^{k-1} |grad u|^{p-2} grad u) > -div (u^k [|grad u|^{p-2} I + (p-2) |grad u|^{p-4} grad u \otimes grad u] grad w) > The first part is transport of the perturbation w. The next is > diffusion of w with an anisotropic conductivity tensor. Yes I have considered it. See http://dx.doi.org/10.1017/jog.2015.3, both the discussion of flux-splitting and the appendix on analytical jacobian. What do *you* get by considering this conductivity tensor? >> So now I'll emphasize what I said before: it is *degenerate* p-bratu. > To my knowledge, "p-Bratu" is just something I invented to combine two > nonlinearities. You don't have an e^u nonlinearity so there is no > Bratu-type nonlinearity. There *is* a term I want to treat explicitly, and it is *not* related to the stiffness etc., but to the coupling to the atmosphere. (I said this before.) Namely, in my form u_t - f(t,u)=g(t,u) the RHS g(t,u) is an elevation dependent mass balance term. Actually it can be (and might as well be) bratu i.e. exponentialish in the thickness. Mostly it is out of my control because it is the altitude dependence of precipitation in the bottom few km of the atmosphere arising from coupling to atmosphere models. My test case linearizes it a la a glaciologist, with an ELA and a lapse rate. By analogy, in other words, the time-dependent form of the I-know-Jed-invented-it p-bratu problem is pretty representative of my problem ... at least if you look in p>2 cases, add additional degeneracy to the diffusivity, make the zero-order term signed, use SNESVI (because the source term is signed and so a VI/NCP must be solved), and you add a sliding flux. I was trying to work by analogy ... Ed On Wed, Feb 15, 2017 at 9:37 AM, Jed Brown wrote: > Ed Bueler writes: > > > Jed -- > > > >>> u_t - div (u^k |grad u|^{p-2} grad u) = g(t,u) > >>Are you also interested in the case with sliding? > > > > You must be getting grumpy. There is this 2009 paper about sliding ... > > Yes I am interested in sliding but I was trying to keep things simple for > > you. ;-) > > My point was that including sliding potentially adds another > stiff/algebraic term so whatever interface we choose better be able to > support at least two stiff terms. > > > The code in question is an under-development example in my book that I am > > trying to use to show off SNESVI functionality in an interesting case and > > yet not get too nasty. See the help string at the top of the file for > what > > it actually does: > > > > https://github.com/bueler/p4pdes/blob/master/c/ch11/ice.c > > > > In summary, the actual flux is > > > > q_total = - Gamma H^{n+2} |grad s|^{n-1} grad s + V H > > Is V really a prescribed function or the solution of an equation that > involves u? > > > and it is all handled implicitly by preference, i.e. in the current API I > > have IFunction = u_t - div q_total. > > > > The real interest of the problem is the failure of long-range > communication > > of low-(spatial)-frequency information when you hit the free boundary, > and > > these barriers can emerge anywhere in the domain. (In a mountain range > you > > can have neighboring small glaciers completely out of phase; how should > > multigrid ... or any other preconditioning one would hope for ... work > for > > all the glaciers in a mountain range? > > This already exists for variable coefficient problems -- the electric > potential in a twisted wire pair is independent despite being nearby > points in the solution of a Poisson equation. > > An important difference from ice sheets is that the Green's functions > are highly nonlocal for the twisted wire pair, but very local for ice > sheets with sticky beds. With slippery beds (ice shelves are the > limiting case), the velocity Green's functions become similar to > Poisson. That is the scenario where multigrid is needed. Otherwise the > solutions are quite local. > > For the SIA thickness equation, I think it would be interesting to > consider the conductivity tensor in the Newton linearization. > > d_u [-div (u^k |grad u|^{p-2} grad u)] * w > = -div (k w^{k-1} |grad u|^{p-2} grad u) > -div (u^k [|grad u|^{p-2} I + (p-2) |grad u|^{p-4} grad u \otimes > grad u] grad w) > > The first part is transport of the perturbation w. The next is > diffusion of w with an anisotropic conductivity tensor. > > > So now I'll emphasize what I said before: it is *degenerate* p-bratu. > > To my knowledge, "p-Bratu" is just something I invented to combine two > nonlinearities. You don't have an e^u nonlinearity so there is no > Bratu-type nonlinearity. > > > Doubly-degenerate because p>2, but the "H^{n+2}" aka "u^k" is the bad > > thing that makes the abrupt free boundary.) Sliding, in this regard, > > is the easy case, so I am throwing it in by generating an artificial > > velocity field; yes this is the coupling that should include the SSA > > in the future ... we already have an SSA solver or two that behave > > surprisingly well, you know, as long as the boundary conditions can be > > figured-out. > > I think sliding is the only phenomena that produces a nonlocal response. > Consider an ice shelf. The thickness and slope at the grounding line > instantaneously affect the seaward edge of the shelf. Grounded SIA has > no such dynamics. > > >> I would just have a PetscOptionsEList() for whether to evaluate the > >> diffusive term on the left or right. Shouldn't be more than a few lines > >> of code. > > > > So I have something like?: > > -ice_diffusive_side left,right > > > > O.k. What if user says > > -ts_type rk -ice_diffusive_side left > > Then I get the current brokenness: code runs and produces nonsense > because > > RK does not call SNESVI (constraint never enforced) *and* diffusive stuff > > is never evaluated (because RK does not call IFunction). > > Yes, I think TSRK should error if IFunction has been specified. > > > So maybe then I also go through possible -ts_types and check whether the > > option combination made sense? Or move some terms to the correct side > > based on what that -ts_type should want? Do I include -ts_type sundials > > because somebody's config might include that third-party thing (even > though > > mine doesn't)? > > I am not suggesting switching on TS type. I guess I felt that having > dependent options was acceptable in exchange for it being easy/reliable > to know which terms are being treated explicitly or implicitly. If the > method takes over that choice, it makes the interface "smarter" which is > not necessarily a good thing. It requires copious documentation and > diagnostics so the user can tell what the hell is happening under the > hood. But I see the utility. > > > That's what I mean by fragile code. > > > > Such things build up! Each time you shove cool stuff in my face I have > to > > get used to seeing ugly stuff turn into beautiful stuff as my aesthetic > > sense follows my growing understanding. But it takes time (learning > curve) > > and all the time it is fragile code (i.e. for me ... because I don't > > appreciate all the motivations behind that part of the API ...) > > > > On the other hand ... I agree that if I am convinced this is really a > good > > API for something I want, then indeed I will suck it up and learn it and > > write it. > > > > Recall my original request was for the ability to know if the TS type I > > have chosen will actually use the SNES I have set ... so long ago it now > > seems. > > I think this is just how design conversations work. > -- Ed Bueler Dept of Math and Stat and Geophysical Institute University of Alaska Fairbanks Fairbanks, AK 99775-6660 301C Chapman and 410D Elvey 907 474-7693 and 907 474-7199 (fax 907 474-5394) -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Feb 15 15:03:10 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 15 Feb 2017 15:03:10 -0600 Subject: [petsc-users] PETSc set off-diagonal matrix pre-allocation In-Reply-To: References: <7FCA7E9C-BACA-47FC-801A-F3F6A96044F2@mcs.anl.gov> Message-ID: <31CA7AC8-BBC1-429D-BBF7-DE2952034ABB@mcs.anl.gov> > On Feb 15, 2017, at 2:34 PM, Andrew Ho wrote: > > I'm guessing it's because I'm using an optimized build why I get less error checking. But regardless, what does the "local" column size mean in terms of which MPI rank owns what portion of the matrix? It means nothing about the ownership of the MATRIX entries, it means the ownership of the VECTOR entries when you do a matrix vector produce A * x. It means which vector entries of x are local. > > Is that the size of the local diagonal block when specifying the non-zeros for pre-allocation? It is the local size passed to MatSetSizes() as the n argument. > > The documentation does not make this part clear at all to me. > > On Wed, Feb 15, 2017 at 11:01 AM, Barry Smith wrote: > > Strange. I run your example and get an error message as I expect. > > > $ petscmpiexec -n 4 ./ex5 > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Nonconforming object sizes > [0]PETSC ERROR: Sum of local lengths 12 does not equal global length 6, my local length 3 > likely a call to VecSetSizes() or MatSetSizes() is wrong. > See http://www.mcs.anl.gov/petsc/documentation/faq.html#split > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.7.5-3049-ge39721325b GIT Date: 2017-02-05 20:43:45 -0600 > [0]PETSC ERROR: ./ex5 on a arch-basic named visitor097-125.wl.anl-external.org by barrysmith Wed Feb 15 12:55:53 2017 > [0]PETSC ERROR: Configure options --download-mpich PETSC_ARCH=arch-basic --download-hdf5 --download-triangle > [0]PETSC ERROR: #1 PetscSplitOwnership() line 89 in /Users/barrysmith/Src/petsc/src/sys/utils/psplit.c > [0]PETSC ERROR: #2 PetscLayoutSetUp() line 137 in /Users/barrysmith/Src/petsc/src/vec/is/utils/pmap.c > [0]PETSC ERROR: #3 MatMPIAIJSetPreallocation_MPIAIJ() line 2640 in /Users/barrysmith/Src/petsc/src/mat/impls/aij/mpi/mpiaij.c > [0]PETSC ERROR: #4 MatMPIAIJSetPreallocation() line 3368 in /Users/barrysmith/Src/petsc/src/mat/impls/aij/mpi/mpiaij.c > [0]PETSC ERROR: #5 main() line 34 in /Users/barrysmith/Src/petsc/test-directory/ex5.c > [0]PETSC ERROR: PETSc Option Table entries: > [0]PETSC ERROR: -malloc_test > [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- > [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [1]PETSC ERROR: Nonconforming object sizes > [1]PETSC ERROR: Sum of local lengths 12 does not equal global length 6, my local length 3 > > > Note that the input you provide is not consistent. You claim the matrix has a total of 6 rows but then you try to assign 3 rows to the first process, 3 rows to the second, 3 rows to the third, 3 rows to the fourth for a total of 12 rows. Hence it should error. Similarly the sum of the "local sizes" for columns has to sum up to exactly the total number of columns. > > > > > > On Feb 15, 2017, at 9:52 AM, Andrew Ho wrote: > > > > I don't understand how the matrices are partitioned then. The documentation online all shows each mpi rank owning entire rows, with no partitioning along columns. > > > > I've attached an example where I try to partition a 6x7 matrix into 4 ranks: > > > > rank 0 should own a 3x4 section > > rank 1 should own a 3x3 section > > rank 2 should own a 3x4 section > > rank 3 should own a 3x3 section > > > > However, when I run the example and print out the ownership range and ownership column range, I get: > > > > rank 0 owns rows/cols: [0,3), [0, 4) > > rank 1 owns rows/cols: [3,6), [4, 7) > > rank 2 owns rows/cols: [6,9), [7, 11) > > rank 3 owns rows/cols: [9,12), [11, 14) > > > > Which makes no sense since these ranges extend beyond the actual size of the matrix. > > > > On Tue, Feb 14, 2017 at 6:05 PM, Barry Smith wrote: > > > > 1) on a single process all the columns are in the diagonal block and none in the off diagonal block (note that the column partitioning corresponding to a partitioning of the vector in the product A *x > > > > 2) on two processes you guessed wrong how many columns PETSc would put on the first process. You guessed it would put the first three on the first process and the last three on the second process. > > > > a) it cannot do that, every column has to been owned by some process (in the vector x above) so it cannot be 3 and 3, it has to be 4 and 3 or 3 and 4. > > > > b) PETSc puts the "extra" columns on the earlier processes not the later processes. > > > > For rectangular matrices you really cannot get away with using "PETSC_DECIDE" for local columns when using preallocation > > , since it may decide something different than what you assume. > > > > I've attached the parallel code that behaves as expected. > > > > Barry > > > > > On Feb 14, 2017, at 1:06 PM, Andrew Ho wrote: > > > > > > The problem isn't only with 1 process, but it seems to be with all non-square matrices? > > > > > > For example, here I have a two process program which tries to set a 6x7 matrix to (each process owns 3 rows): > > > > > > 0 0 0 1 1 1 1 > > > 0 0 0 1 1 1 1 > > > 0 0 0 1 1 1 1 > > > 0 0 0 1 1 1 1 > > > 0 0 0 1 1 1 1 > > > 0 0 0 1 1 1 1 > > > > > > #include > > > #include > > > > > > int main(int argc, char** argv) > > > { > > > PetscErrorCode err; > > > err = PetscInitialize(&argc, &argv, NULL, "help"); > > > CHKERRQ(err); > > > > > > int rank, size; > > > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > > > MPI_Comm_size(MPI_COMM_WORLD, &size); > > > if(size != 2) > > > { > > > printf("must run with 2 processes"); > > > MPI_Abort(MPI_COMM_WORLD, -1); > > > } > > > > > > // create a sparse AIJ matrix distributed across MPI > > > Mat A; > > > err = MatCreate(MPI_COMM_WORLD, &A); > > > CHKERRQ(err); > > > err = MatSetType(A, MATMPIAIJ); > > > CHKERRQ(err); > > > // setup pre-allocation for matrix space > > > { > > > err = > > > MatSetSizes(A, 3, PETSC_DECIDE, 6, 7); > > > CHKERRQ(err); > > > if(rank == 0) > > > { > > > PetscInt d_nnz[] = {0, 0, 0}; > > > PetscInt o_nnz[] = {4, 4, 4}; > > > err = MatMPIAIJSetPreallocation(A, 0, d_nnz, 0, o_nnz); > > > CHKERRQ(err); > > > } > > > else > > > { > > > PetscInt d_nnz[] = {3, 3, 3}; > > > PetscInt o_nnz[] = {1, 1, 1}; > > > err = MatMPIAIJSetPreallocation(A, 0, d_nnz, 0, o_nnz); > > > CHKERRQ(err); > > > } > > > } > > > err = MatSetUp(A); > > > CHKERRQ(err); > > > > > > // set values inside the matrix > > > for (PetscInt row = 0; row < 3; ++row) > > > { > > > for (PetscInt col = 3; col < 7; ++col) > > > { > > > err = MatSetValue(A, 3 * rank + row, col, 1, INSERT_VALUES); > > > CHKERRQ(err); > > > } > > > } > > > > > > err = MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); > > > CHKERRQ(err); > > > err = MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); > > > CHKERRQ(err); > > > > > > err = MatView(A, PETSC_VIEWER_STDOUT_WORLD); > > > CHKERRQ(err); > > > > > > // free memory > > > err = MatDestroy(&A); > > > CHKERRQ(err); > > > > > > // cleanup any internal PETSc data at end of program > > > err = PetscFinalize(); > > > CHKERRQ(err); > > > } > > > > > > > > > On Tue, Feb 14, 2017 at 5:26 AM, Matthew Knepley wrote: > > > On Tue, Feb 14, 2017 at 6:41 AM, Andrew Ho wrote: > > > I have a 3x6 matrix, and I want to set it to (just as an example): > > > > > > 0 0 0 1 1 1 > > > 0 0 0 1 1 1 > > > 0 0 0 1 1 1 > > > > > > From what I can tell, according to the documentation the MPIAIJ sparsity of this matrix is: > > > > > > I believe this is an inconsistency in PETSc for rectangular matrices. We divide matrices into diagonal and > > > off-diagonal parts mainly to separate communication from computation. Thus on 1 proc, we never divide > > > them, even if the matrix is rectangular. Therefore we misinterpret your preallocation. > > > > > > Barry, do you think we want to try and change this? > > > > > > Matt > > > > > > d_nnz = [0, 0, 0] > > > o_nnz = [3, 3, 3] > > > > > > However, when I do this, I get the following error: > > > > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > > [0]PETSC ERROR: Argument out of range > > > [0]PETSC ERROR: New nonzero at (0,3) caused a malloc > > > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check > > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > > [0]PETSC ERROR: Petsc Release Version 3.7.5, unknown > > > [0]PETSC ERROR: ./ex3 on a arch-linux2-c-opt Tue Feb 14 04:23:56 2017 > > > [0]PETSC ERROR: Configure options --with-debugging=0 --COPTFLAGS="-O3 -march=native" --CXXOPTFLAGS="-O3 -march=native" --FOPTFLAGS="-O3 -march=native" > > > [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 582 in petsc/src/mat/impls/aij/mpi/mpiaij.c > > > [0]PETSC ERROR: #2 MatSetValues() line 1190 in petsc/src/mat/interface/matrix.c > > > [0]PETSC ERROR: #3 main() line 36 in ex3.c > > > [0]PETSC ERROR: No PETSc Option Table entries > > > [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- > > > > > > Here's a working test code: > > > > > > #include > > > #include > > > int main(int argc, char** argv) > > > { > > > PetscErrorCode err; > > > err = PetscInitialize(&argc, &argv, NULL, "help"); > > > CHKERRQ(err); > > > // create a sparse AIJ matrix distributed across MPI > > > PetscInt global_width = 6; > > > PetscInt global_height = 3; > > > Mat A; > > > err = MatCreate(MPI_COMM_WORLD, &A); > > > CHKERRQ(err); > > > err = MatSetType(A, MATMPIAIJ); > > > CHKERRQ(err); > > > // setup pre-allocation for matrix space > > > { > > > err = > > > MatSetSizes(A, global_height, PETSC_DECIDE, global_height, global_width); > > > CHKERRQ(err); > > > PetscInt d_nnz[] = {0, 0, 0}; > > > PetscInt o_nnz[] = {3, 3, 3}; > > > err = MatMPIAIJSetPreallocation(A, 0, d_nnz, 0, o_nnz); > > > CHKERRQ(err); > > > } > > > err = MatSetUp(A); > > > CHKERRQ(err); > > > // set values inside the matrix > > > for (PetscInt row = 0; row < global_height; ++row) > > > { > > > for (PetscInt col = global_height; col < global_width; ++col) > > > { > > > err = MatSetValue(A, row, col, 1, INSERT_VALUES); > > > CHKERRQ(err); > > > } > > > } > > > err = MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); > > > CHKERRQ(err); > > > err = MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); > > > CHKERRQ(err); > > > err = MatView(A, PETSC_VIEWER_STDOUT_WORLD); > > > CHKERRQ(err); > > > // free memory > > > err = MatDestroy(&A); > > > CHKERRQ(err); > > > // cleanup any internal PETSc data at end of program > > > err = PetscFinalize(); > > > CHKERRQ(err); > > > } > > > > > > Am I mis-understanding what the d_nnz and o_nnz parameter are supposed to mean? > > > -- > > > Andrew Ho > > > > > > > > > > > > -- > > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > > -- Norbert Wiener > > > > > > > > > > > > -- > > > Andrew Ho > > > > > > > > > > > > -- > > Andrew Ho > > > > > > > -- > Andrew Ho From mfadams at lbl.gov Wed Feb 15 15:44:57 2017 From: mfadams at lbl.gov (Mark Adams) Date: Wed, 15 Feb 2017 16:44:57 -0500 Subject: [petsc-users] multigrid questions In-Reply-To: <87fujf1fnr.fsf@jedbrown.org> References: <87fujf1fnr.fsf@jedbrown.org> Message-ID: > > > Chebyshev(1) is actually just Richardson. Damped Richardson, with, for instance, the/an optimal damping for "the model problem". -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Feb 15 16:10:02 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 15 Feb 2017 16:10:02 -0600 Subject: [petsc-users] FLOPS vs FLOPS/sec from PETSc -log_summary In-Reply-To: References: Message-ID: <35867CE0-2026-400B-8135-A9C5B8566134@mcs.anl.gov> Fixed in master > On Feb 15, 2017, at 1:25 PM, Lukas van de Wiel wrote: > > In that case, do not care whether it becomes a literary 'tour de force', and best name it FLOP and FLOP/s to avoid any redundancy. > > Cheers > Lukas > > On Wed, Feb 15, 2017 at 8:06 PM, Barry Smith wrote: > > In PETSc FLOPS refers to (F)loating (P)oint (OP)eration(s) :-) Hence the confusing notation in our output. > > Barry > > Yes. this is confusing but it ended up this way because no one says, "this calculation took 500 FLOP", that just sounds weird. Even Total FLOP sounds weird. Any suggestions for how to phrase it without the confusion? > > > > > On Feb 15, 2017, at 12:58 PM, Ajit Desai wrote: > > > > Hello everyone, > > > > I am little unclear between Flops and Flops/sec in the performance profile obtained for my solver using -log_summary. > > Usually, FLOPS stands for (Floating Point Operations Per Second) then what is Flops/sec produced by PETSc log_summary and how it is different from Flops? > > > > I have included the output for the reference below. > > > > Max Max/Min Avg Total > > Time (sec): 6.782e+02 1.00003 6.782e+02 > > Objects: 7.400e+01 1.02778 7.206e+01 > > Flops: 5.451e+11 1.99585 4.518e+11 3.181e+14 > > Flops/sec: 8.038e+08 1.99585 6.662e+08 4.690e+11 > > MPI Messages: 0.000e+00 0.00000 0.000e+00 0.000e+00 > > MPI Message Lengths: 0.000e+00 0.00000 0.000e+00 0.000e+00 > > MPI Reductions: 1.000e+00 1.00000 > > > > Thanks in advance. > > Ajit Desai > > PhD Scholar, Carleton University, Canada > > From mvalera at mail.sdsu.edu Wed Feb 15 19:16:23 2017 From: mvalera at mail.sdsu.edu (Manuel Valera) Date: Wed, 15 Feb 2017 17:16:23 -0800 Subject: [petsc-users] A way to distribute 3D arrays. Message-ID: Hello, My question this time is just if there is a way to distribute a 3D array whos located at Zero rank over the processors, if possible using the DMDAs, i'm trying not to do a lot of initialization I/O in parallel. Thanks for your time, Manuel -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Feb 15 20:01:21 2017 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 15 Feb 2017 20:01:21 -0600 Subject: [petsc-users] A way to distribute 3D arrays. In-Reply-To: References: Message-ID: On Wed, Feb 15, 2017 at 7:16 PM, Manuel Valera wrote: > Hello, > > My question this time is just if there is a way to distribute a 3D array > whos located at Zero rank over the processors, if possible using the DMDAs, > i'm trying not to do a lot of initialization I/O in parallel. > 1) You could create a VecScatter to do exactly what you want. This is time consuming and error prone. 2) You could a) Read the data into a Vec from a serial DMDA with those dimensions, and write it out in binary as a Petsc Vec. b) Read in the Petsc Vec for a distributed DMDA of the same dimensions. This will happen in parallel. Thanks, Matt > Thanks for your time, > > Manuel > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From lixin_chu at yahoo.com Wed Feb 15 21:15:08 2017 From: lixin_chu at yahoo.com (lixin chu) Date: Thu, 16 Feb 2017 03:15:08 +0000 (UTC) Subject: [petsc-users] Newbie question : iterative solver - algorithm and performance In-Reply-To: References: <2007645858.4961090.1487140811648.ref@mail.yahoo.com> <2007645858.4961090.1487140811648@mail.yahoo.com> <915837361.5274745.1487163839533@mail.yahoo.com> Message-ID: <886300824.136969.1487214908800@mail.yahoo.com> Thank you all for your help !?rgds LX On Thursday, 16 February 2017, 3:29, Patrick Sanan wrote: On Wed, Feb 15, 2017 at 8:26 PM, Barry Smith wrote: > >> On Feb 15, 2017, at 7:03 AM, lixin chu wrote: >> >> I think the chapter 4 of the user manual more or less answers my first question already ... >> >> rgds >> lixin >> >> >> On Wednesday, 15 February 2017, 14:40, lixin chu wrote: >> >> >> Hello, >> New to PETSc, appreciate any help - >> >> I have done some experiment with MUMPS (the direct solver for sparse matrix), and very interested to try out PETSc now. I have a large sparse symmetric matrix (3 millions x 3 millions, complex data type). >> >> Some questions I have: >> - I am assuming that I should select one algorithm of "Krylov methods", which algorithm is a good option, GMRES, CG, or others ? (I am not a domain expert, but helping developing a program to test the matrix). > >? ? GMRES is a "safe" choice. You can try -ksp_type cg? -ksp_cg_type symmetric? (or if the matrix is hermitian; i.e. transpose A = complex conjugate of A? then type hermitian instead of symmetric). > > >> - do all the algorithms support distributed architecture (multiple machines, multiple cores) > >? ? Yes > >> - are there any performance test data ? (total run time for example) > >? ? Run your code with -view_summary and it will print information at the end about where it has spent the time doing the computation. Barry probably meant -log_view here. > > >> >> >> thank you very much, >> LX >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Feb 15 21:31:50 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 15 Feb 2017 21:31:50 -0600 Subject: [petsc-users] A way to distribute 3D arrays. In-Reply-To: References: Message-ID: <2371E2C6-DB67-420C-8020-A53E2550BB35@mcs.anl.gov> You can do this in the style of VecLoad_Binary_DA() first you take your sequential vector and make it parallel in the "natural" ordering for 3d arrays DMDACreateNaturalVector(da,&natural); VecScatterCreateToZero(natural,&scatter,&veczero); /* veczero is of full size on process 0 and has zero entries on all other processes*/ /* fill up veczero */ VecScatterBegin(scatter,veczero,natural,INSERT_VALUES,SCATTER_REVERSE); VecScatterEnd(scatter,veczero,natural,INSERT_VALUES,SCATTER_REVERSE); and then move it into the PETSc DMDA parallel ordering vector with ierr = DMCreateGlobalVector(da,&xin);CHKERRQ(ierr); ierr = DMDANaturalToGlobalBegin(da,natural,INSERT_VALUES,xin);CHKERRQ(ierr); ierr = DMDANaturalToGlobalEnd(da,natural,INSERT_VALUES,xin);CHKERRQ(ierr); > On Feb 15, 2017, at 7:16 PM, Manuel Valera wrote: > > Hello, > > My question this time is just if there is a way to distribute a 3D array whos located at Zero rank over the processors, if possible using the DMDAs, i'm trying not to do a lot of initialization I/O in parallel. > > Thanks for your time, > > Manuel From lixin_chu at yahoo.com Wed Feb 15 21:57:06 2017 From: lixin_chu at yahoo.com (lixin chu) Date: Thu, 16 Feb 2017 03:57:06 +0000 (UTC) Subject: [petsc-users] Newbie question : sequential and parallel storage, and memory estimation References: <145946111.175320.1487217426005.ref@mail.yahoo.com> Message-ID: <145946111.175320.1487217426005@mail.yahoo.com> Hello,Newbie question again - data storage and distribution related this time? I have briefly looked at the user manual, and searched forum, trying to get some basic understanding of the data storage and distribution. Some questions I have: 1. Sequential vs. Parallel? ? I assume that by 'sequential', we mean the matrix/vector is created in one process only ?? ? Then if this correct, I can assume that 'parallel' means each process will create a sub-matrix/vector for itself ? ? ? Will each process eventually have the complete matrix and the RHS vector after MatAssemblyBegin() and MatAssemblyEnd(), before solve starts ? 2. How the matrix/vector is split among the processes ?? ? I think matrix will be split by blocks of rows ? Does PETSc equally divide the matrix rows (with the last taking the remainder ) ? ? ? If I want to create the matrix in a parallel way, do I have to divide the rows equally, or different process can have different number of rows, and some may not have any rows ? ? ? Does RHS vector also need to be split in the exactly same blocks of rows as the matrix A (same start and end row number for the same process) ? 3.?MAT_STRUCTURALLY_SYMMETRIC? ?Do I have to input the upper triangle portion of non zero values, or lower triangle is also fine ? 4. Given the matrix A and RHS b, data type and sparsity, is there a way to estimate the total RAM needed for the solve phase, let's say for GMRES algorithm ? many thanks !?rgds LX -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Feb 15 22:13:09 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 15 Feb 2017 22:13:09 -0600 Subject: [petsc-users] Newbie question : sequential and parallel storage, and memory estimation In-Reply-To: <145946111.175320.1487217426005@mail.yahoo.com> References: <145946111.175320.1487217426005.ref@mail.yahoo.com> <145946111.175320.1487217426005@mail.yahoo.com> Message-ID: <6A37148B-7473-463B-A4F2-3615554C5FFE@mcs.anl.gov> > On Feb 15, 2017, at 9:57 PM, lixin chu wrote: > > Hello, > Newbie question again - data storage and distribution related this time > > I have briefly looked at the user manual, and searched forum, trying to get some basic understanding of the data storage and distribution. Some questions I have: > > 1. Sequential vs. Parallel > I assume that by 'sequential', we mean the matrix/vector is created in one process only ? > Then if this correct, I can assume that 'parallel' means each process will create a sub-matrix/vector for itself ? In PETSc language parallel means that parts of the matrix/vector are stored on each process. > > Will each process eventually have the complete matrix and the RHS vector after MatAssemblyBegin() and MatAssemblyEnd(), before solve starts ? No, each process contains part of the matrix and part of the vector. If you stored the entire matrix on each process this would limit how large a problem you could run based on the memory had on one process. > > 2. How the matrix/vector is split among the processes ? > I think matrix will be split by blocks of rows ? Yes > Does PETSc equally divide the matrix rows (with the last taking the remainder ) ? More or less. You can set how the rows are divided by using MatSetSizes() and you can use MatGetOwnershipRange() to determine how it decided to split the rows. > > If I want to create the matrix in a parallel way, do I have to divide the rows equally, or different process can have different number of rows, and some may not have any rows ? You can using MatSetSizes() do it anyway you want. BUT if you give some processes many rows and others zero rows you will not be using the parallel machine efficiently since some processes will have lots of work (and memory demands) and others will have no work. > > Does RHS vector also need to be split in the exactly same blocks of rows as the matrix A (same start and end row number for the same process) ? Yes > > > 3. MAT_STRUCTURALLY_SYMMETRIC > Do I have to input the upper triangle portion of non zero values, or lower triangle is also fine ? Structurally symmetric only means the nonzero pattern below the diagonal is the same as above, the numerical values below the diagonal may be different than above so you always need to provide all the values. > > > 4. Given the matrix A and RHS b, data type and sparsity, is there a way to estimate the total RAM needed for the solve phase, let's say for GMRES algorithm ? Not really. It is very complicated based on the preconditioned used. Just for the GMRES solver, not counting memory for the preconditioner it is roughly restart + seven vectors worth of memory. Barry > > many thanks ! > > rgds > LX From jed at jedbrown.org Wed Feb 15 22:13:55 2017 From: jed at jedbrown.org (Jed Brown) Date: Wed, 15 Feb 2017 21:13:55 -0700 Subject: [petsc-users] TS question 1: how to stop explicit methods because they do not use SNES(VI)? In-Reply-To: References: <0991D4AC-57B0-4D3B-93E7-8F42E305CF8C@mcs.anl.gov> <6B4461AE-21E7-4878-B674-CD7631A26EA6@anl.gov> <60D7CDCC-E9D5-472A-9175-64C56DF8B34F@mcs.anl.gov> <87vasc4pz5.fsf@jedbrown.org> <6D880F3B-F880-4AFD-975D-5A8A4F4AA8A9@mcs.anl.gov> <87inoc4l17.fsf@jedbrown.org> <87shng2jk6.fsf@jedbrown.org> <87efyz1esp.fsf@jedbrown.org> Message-ID: <871suy22nw.fsf@jedbrown.org> Ed Bueler writes: > Jed -- > >> My point was that including sliding potentially adds another >> stiff/algebraic term so whatever interface we choose better be able to >> support at least two stiff terms. > > Yes, totally agreed w.r.t. the interface design. (The DAE that you are > solving in that case is stiff.) > > However, *my* observation about the current interface design is actually > that it was never apparently tested with TS+SNESVI and is awkward in that > case. Certainly. > IMHO the design for time-stepping on constrained problems should continue > to allow use of SNESVI (in implicit and imex cases) if an LHSfunction (or > similar) is provided, *and* have a mechanism where explicit time-solvers > can use projection onto the bounds---conceivably this could be a > trivialized SNESVI instance---if bounds are present. This is because mere > evaluation of LHSFunction and RHSFunction will not otherwise generate the > "side effect" of enforcing the constraints which were provided by > FormBounds() in using the SNESVI. > > In both the current and proposed designs in this thread, for implicit and > imex methods the constraint enforcement is inside SNESVI *but* for explicit > time-stepping it would be a user-written projection in TSPostStep(). > Furthermore the LHSFunction might have to be rewritten to allow > non-feasible inputs just to support explicit multistage time-stepping. Yuk. > > Note that there are VIs in L^2 and they *are* "projection onto the bounds" > in the just-mentioned sense. See section II.3 of Kinderlehrer and > Stampacchia. (Elliptic VIs in W^{1,p} are what one usually sees.) The > switch from VI in Sobolev space to VI in L^2 is exactly what you see when > you semi-discretize these obstacle-like problems in time only and then > compare implicit and explicit timestepping, respectively. > > So ya'll are having a good argument ... but I am worried about what I asked > about anyway. Should TS get explicit support for time-dependent VIs? (Note that I joined the conversation late.) >>> q_total = - Gamma H^{n+2} |grad s|^{n-1} grad s + V H >> Is V really a prescribed function or the solution of an equation that >> involves u? > > I know that the problem you want me to solve is a big DAE including both > mass and momentum conservation. (Do you really think that all this time I > haven't noticed?) > > My interest is in good solvers for what you call the "local problem". This > local problem is the one that limits PISM hybrid runs *and* I think it is > why Stokes is not really functional yet in anyone's time-stepping model. > (If they treat the time-stepping explicitly then they have the SIA-type > time-step restriction kick in as a solver convergence problem, because the > SIA tells the truth in the majority of the area where the bed is sticky.) > Thus my test case code does not waste time generating a real V (i.e. > solving simultaneously for H and V at each time step, for the DAE you want > me to solve ...). > >> For the SIA thickness equation, I think it would be interesting to >> consider the conductivity tensor in the Newton linearization. >> d_u [-div (u^k |grad u|^{p-2} grad u)] * w >> = -div (k w^{k-1} |grad u|^{p-2} grad u) >> -div (u^k [|grad u|^{p-2} I + (p-2) |grad u|^{p-4} grad u \otimes > grad u] grad w) >> The first part is transport of the perturbation w. The next is >> diffusion of w with an anisotropic conductivity tensor. > > Yes I have considered it. See http://dx.doi.org/10.1017/jog.2015.3, both > the discussion of flux-splitting and the appendix on analytical jacobian. > What do *you* get by considering this conductivity tensor? Is it singular or degenerate apart from u=0 and grad u=0? (I think this can only happen for p=1.) What is the maximum condition number of the tensor? Degeneracy is a common challenge in subsurface flows and AMG tends to do alright, though it can be sensitive to the ability of the strength of connection measure to diagnose anisotropy for the chosen discretization. I guess my point is that at the level of a robust linear solve, I'm not sure the degeneracy that you face is particularly different from that faced in other applications. If you want a robust nonlinear multigrid, it's important to know what technology would be required for a robust linear solve because you probably need at least that. >>> So now I'll emphasize what I said before: it is *degenerate* p-bratu. >> To my knowledge, "p-Bratu" is just something I invented to combine two >> nonlinearities. You don't have an e^u nonlinearity so there is no >> Bratu-type nonlinearity. > > There *is* a term I want to treat explicitly, and it is *not* related to > the stiffness etc., but to the coupling to the atmosphere. (I said this > before.) Namely, in my form u_t - f(t,u)=g(t,u) the RHS g(t,u) is an > elevation dependent mass balance term. Actually it can be (and might as > well be) bratu i.e. exponentialish in the thickness. Mostly it is out of > my control because it is the altitude dependence of precipitation in the > bottom few km of the atmosphere arising from coupling to atmosphere > models. My test case linearizes it a la a glaciologist, with an ELA and a > lapse rate. > > By analogy, in other words, the time-dependent form of the > I-know-Jed-invented-it p-bratu problem is pretty representative of my > problem ... at least if you look in p>2 cases, add additional degeneracy to > the diffusivity, make the zero-order term signed, use SNESVI (because the > source term is signed and so a VI/NCP must be solved), and you add a > sliding flux. I was trying to work by analogy ... Fair enough. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From lixin_chu at yahoo.com Wed Feb 15 22:24:32 2017 From: lixin_chu at yahoo.com (lixin chu) Date: Thu, 16 Feb 2017 04:24:32 +0000 (UTC) Subject: [petsc-users] Newbie question : sequential and parallel storage, and memory estimation In-Reply-To: <6A37148B-7473-463B-A4F2-3615554C5FFE@mcs.anl.gov> References: <145946111.175320.1487217426005.ref@mail.yahoo.com> <145946111.175320.1487217426005@mail.yahoo.com> <6A37148B-7473-463B-A4F2-3615554C5FFE@mcs.anl.gov> Message-ID: <1766527887.207393.1487219072167@mail.yahoo.com> super fast response, really appreciate Barry ! - MAT_STRUCTURALLY_SYMMETRICI quoted the wrong option - I think it is MAT_SYMMETRIC - so for this, it should ok if I input lower triangle or upper triangle data right ? Back to the matrix and RHS distribution question :- will the result X also follow the same row distribution as A and b ? ?rgds lixin On Thursday, 16 February 2017, 12:13, Barry Smith wrote: > On Feb 15, 2017, at 9:57 PM, lixin chu wrote: > > Hello, > Newbie question again - data storage and distribution related this time > > I have briefly looked at the user manual, and searched forum, trying to get some basic understanding of the data storage and distribution. Some questions I have: > > 1. Sequential vs. Parallel >? ? I assume that by 'sequential', we mean the matrix/vector is created in one process only ? >? ? Then if this correct, I can assume that 'parallel' means each process will create a sub-matrix/vector for itself ? ? In PETSc language parallel means that parts of the matrix/vector are stored on each process. > >? ? Will each process eventually have the complete matrix and the RHS vector after MatAssemblyBegin() and MatAssemblyEnd(), before solve starts ? ? No, each process contains part of the matrix and part of the vector. If you stored the entire matrix on each process this would limit how large a problem you could run based on the memory had on one process. > > 2. How the matrix/vector is split among the processes ? >? ? I think matrix will be split by blocks of rows ? ? Yes > Does PETSc equally divide the matrix rows (with the last taking the remainder ) ? More or less. You can set how the rows are divided by using MatSetSizes()? and you can use MatGetOwnershipRange() to determine how it decided to split the rows. > >? ? If I want to create the matrix in a parallel way, do I have to divide the rows equally, or different process can have different number of rows, and some may not have any rows ? ? You can using MatSetSizes() do it anyway you want. BUT if you give some processes many rows and others zero rows you will not be using the parallel machine efficiently since some processes will have lots of work (and memory demands) and others will have no work. > >? ? Does RHS vector also need to be split in the exactly same blocks of rows as the matrix A (same start and end row number for the same process) ? ? Yes > > > 3. MAT_STRUCTURALLY_SYMMETRIC >? ? Do I have to input the upper triangle portion of non zero values, or lower triangle is also fine ? ? Structurally symmetric only means the nonzero pattern below the diagonal is the same as above, the numerical values below the diagonal may be different than above so you always need to provide all the values. > > > 4. Given the matrix A and RHS b, data type and sparsity, is there a way to estimate the total RAM needed for the solve phase, let's say for GMRES algorithm ? ? Not really. It is very complicated based on the preconditioned used. Just for the GMRES solver, not counting memory for the preconditioner it is roughly restart + seven vectors worth of memory. ? Barry > > many thanks ! >? > rgds > LX -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Feb 15 22:40:45 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 15 Feb 2017 22:40:45 -0600 Subject: [petsc-users] Newbie question : sequential and parallel storage, and memory estimation In-Reply-To: <1766527887.207393.1487219072167@mail.yahoo.com> References: <145946111.175320.1487217426005.ref@mail.yahoo.com> <145946111.175320.1487217426005@mail.yahoo.com> <6A37148B-7473-463B-A4F2-3615554C5FFE@mcs.anl.gov> <1766527887.207393.1487219072167@mail.yahoo.com> Message-ID: <51377AF1-85A7-4786-911C-22DE083F13D9@mcs.anl.gov> > On Feb 15, 2017, at 10:24 PM, lixin chu wrote: > > super fast response, really appreciate Barry ! > > - MAT_STRUCTURALLY_SYMMETRIC > I quoted the wrong option - I think it is MAT_SYMMETRIC - so for this, it should ok if I input lower triangle or upper triangle data right ? For AIJ and BAIJ you MUST provide the full matrix. For SBAIJ matrices (that only store half the matrix) yes you only need to provide the lower or upper (plus diagonal) part of the matrix. > > Back to the matrix and RHS distribution question : > - will the result X also follow the same row distribution as A and b ? For square matrices by default the x and b vectors have the same layout. This can be controlled to be different but for square matrices you likely never want to do that. > > > rgds > lixin > > > On Thursday, 16 February 2017, 12:13, Barry Smith wrote: > > > > > On Feb 15, 2017, at 9:57 PM, lixin chu wrote: > > > > Hello, > > Newbie question again - data storage and distribution related this time > > > > I have briefly looked at the user manual, and searched forum, trying to get some basic understanding of the data storage and distribution. Some questions I have: > > > > 1. Sequential vs. Parallel > > I assume that by 'sequential', we mean the matrix/vector is created in one process only ? > > > Then if this correct, I can assume that 'parallel' means each process will create a sub-matrix/vector for itself ? > > In PETSc language parallel means that parts of the matrix/vector are stored on each process. > > > > Will each process eventually have the complete matrix and the RHS vector after MatAssemblyBegin() and MatAssemblyEnd(), before solve starts ? > > No, each process contains part of the matrix and part of the vector. If you stored the entire matrix on each process this would limit how large a problem you could run based on the memory had on one process. > > > > > 2. How the matrix/vector is split among the processes ? > > I think matrix will be split by blocks of rows ? > > Yes > > > Does PETSc equally divide the matrix rows (with the last taking the remainder ) ? > > More or less. You can set how the rows are divided by using MatSetSizes() and you can use MatGetOwnershipRange() to determine how it decided to split the rows. > > > > > If I want to create the matrix in a parallel way, do I have to divide the rows equally, or different process can have different number of rows, and some may not have any rows ? > > You can using MatSetSizes() do it anyway you want. BUT if you give some processes many rows and others zero rows you will not be using the parallel machine efficiently since some processes will have lots of work (and memory demands) and others will have no work. > > > > Does RHS vector also need to be split in the exactly same blocks of rows as the matrix A (same start and end row number for the same process) ? > > Yes > > > > > > > 3. MAT_STRUCTURALLY_SYMMETRIC > > Do I have to input the upper triangle portion of non zero values, or lower triangle is also fine ? > > Structurally symmetric only means the nonzero pattern below the diagonal is the same as above, the numerical values below the diagonal may be different than above so you always need to provide all the values. > > > > > > > 4. Given the matrix A and RHS b, data type and sparsity, is there a way to estimate the total RAM needed for the solve phase, let's say for GMRES algorithm ? > > Not really. It is very complicated based on the preconditioned used. Just for the GMRES solver, not counting memory for the preconditioner it is roughly restart + seven vectors worth of memory. > > Barry > > > > > > many thanks ! > > > > rgds > > LX > > From dnolte at dim.uchile.cl Wed Feb 15 22:47:10 2017 From: dnolte at dim.uchile.cl (David Nolte) Date: Thu, 16 Feb 2017 01:47:10 -0300 Subject: [petsc-users] usefulness of fieldsplit_type schur and fieldsplit_schur_type user Message-ID: <918e7e1f-e5d1-19f5-fe42-502514b4043d@dim.uchile.cl> Dear all, I am trying to use PETSc to solve a steady Stokes problem, discretized with stable FEM (P2P1) in 2D and 3D. I have been playing around with different preconditioners to get the hang of things. For my 2D case, for example, the following setup works well, where fieldsplit_0 is identified with the velocity dofs and fieldsplit_1 corresponds to the pressure dofs, and the user defined Schur approximation is the pressure mass matrix: -ksp_view -ksp_converged_reason -ksp_type fgmres -ksp_rtol 1.0e-8 -ksp_monitor -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_schur_fact_type upper -pc_fieldsplit_schur_precondition user -fieldsplit_0_ksp_type richardson -fieldsplit_0_ksp_max_it 1 -fieldsplit_0_pc_type hypre -fieldsplit_0_pc_hypre_type boomeramg -fieldsplit_1_ksp_type preonly -fieldsplit_1_pc_type jacobi However, now I wonder what is the difference between -pc_fieldsplit_type schur -pc_fieldsplit_schur_fact_type diag -pc_fieldsplit_schur_precondition user and -pc_fieldsplit_type additive where the preconditioner matrix would be P = [A, 0; 0, ?] (? is the same Schur approximation as above)? And likewise, between "schur_fact_type lower" and "fieldsplit_type multiplicative", with P = [A, 0; B, ?]. (Except for the generality/flexibility of the Schur approach.) Is there any advantage of using the Schur approach over the additive oder multiplicative fieldsplits if I specify the approx. Schur complement ?? Any reason to prefer one method over the other? The Schur factorization causes a (very) slightly longer computation time. Thanks, kind regards David From jed at jedbrown.org Wed Feb 15 22:58:00 2017 From: jed at jedbrown.org (Jed Brown) Date: Wed, 15 Feb 2017 21:58:00 -0700 Subject: [petsc-users] usefulness of fieldsplit_type schur and fieldsplit_schur_type user In-Reply-To: <918e7e1f-e5d1-19f5-fe42-502514b4043d@dim.uchile.cl> References: <918e7e1f-e5d1-19f5-fe42-502514b4043d@dim.uchile.cl> Message-ID: <87y3x6zq93.fsf@jedbrown.org> David Nolte writes: > Dear all, > > I am trying to use PETSc to solve a steady Stokes problem, discretized > with stable FEM (P2P1) in 2D and 3D. I have been playing around with > different preconditioners to get the hang of things. For my 2D case, for > example, the following setup works well, where fieldsplit_0 is > identified with the velocity dofs and fieldsplit_1 corresponds to the > pressure dofs, and the user defined Schur approximation is the pressure > mass matrix: > > -ksp_view > -ksp_converged_reason > -ksp_type fgmres > -ksp_rtol 1.0e-8 > -ksp_monitor > > -pc_type fieldsplit > -pc_fieldsplit_type schur > -pc_fieldsplit_schur_fact_type upper > -pc_fieldsplit_schur_precondition user > > -fieldsplit_0_ksp_type richardson > -fieldsplit_0_ksp_max_it 1 > -fieldsplit_0_pc_type hypre > -fieldsplit_0_pc_hypre_type boomeramg > > -fieldsplit_1_ksp_type preonly > -fieldsplit_1_pc_type jacobi > > However, now I wonder what is the difference between > -pc_fieldsplit_type schur > -pc_fieldsplit_schur_fact_type diag > -pc_fieldsplit_schur_precondition user > and > -pc_fieldsplit_type additive > where the preconditioner matrix would be P = [A, 0; 0, ?] (? is the same > Schur approximation as above)? And likewise, between "schur_fact_type > lower" and "fieldsplit_type multiplicative", with P = [A, 0; B, ?]. > (Except for the generality/flexibility of the Schur approach.) > > Is there any advantage of using the Schur approach over the additive > oder multiplicative fieldsplits if I specify the approx. Schur > complement ?? Any reason to prefer one method over the other? The Schur > factorization causes a (very) slightly longer computation time. Depends if you might want to iterate on the Schur complement. With Schur, the operator for the inner solver becomes the Schur complement S = D - C A^{-1} B where A^{-1} is an iterative solve. This is Schur complement reduction. If you use preonly, this representation isn't used. With additive or multiplicative, that operator is just the block of the outer operator (D) which is nothing like the Schur complement. So the schur option is better for debugging convergence or if you want to iterate on the Schur complement. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From cmpierce at WPI.EDU Fri Feb 17 03:25:25 2017 From: cmpierce at WPI.EDU (Christopher Pierce) Date: Fri, 17 Feb 2017 04:25:25 -0500 Subject: [petsc-users] Krylov-Schur Tolerance Message-ID: <4691ae05-be89-5bdb-33ac-2fd5cef4c978@wpi.edu> Hello All, I'm trying to use the SLEPc Krylov-Schur implementation to solve a general eigenvalue problem. I have a monitor on my solver and the solutions appear to converge correctly when using the approximation for the residual norm in the algorithm. However, when the solutions are displayed and I retrieve the actual residual norm it is very large and increases with the size of the matrices I am working with. In some cases it may be 10^17 times as large as the approximate norm. I also don't get the eigenvalues I would expect for the system I am studying. When I turn on the option "true residual" the solver fails to converge. The residual norm shrinks to some limit (~10^-3) and then sits there for the remaining iterations. As a note, I am solving for the eigenvalues with the smallest real part. I have also tried the RQCG solver on the same problems and appear to get the correct results using it, but I'm looking to use the better scaling of the Krylov-Schur solver. Does anyone know what could be causing this behavior? Thanks, Chris Pierce WPI Center for Computational Nanoscience From jroman at dsic.upv.es Fri Feb 17 03:42:43 2017 From: jroman at dsic.upv.es (Jose E. Roman) Date: Fri, 17 Feb 2017 10:42:43 +0100 Subject: [petsc-users] Krylov-Schur Tolerance In-Reply-To: <4691ae05-be89-5bdb-33ac-2fd5cef4c978@wpi.edu> References: <4691ae05-be89-5bdb-33ac-2fd5cef4c978@wpi.edu> Message-ID: <78DC3258-D0BB-49A9-9246-D15F6E82A9D0@dsic.upv.es> For computing eigenvalues with smallest real part of generalized problems Ax=lambda Bx, it may be better to use a target value (instead of -eps_smallest_real). For instance, if you know that all eigenvalues are positive, use -eps_target 0 -eps_target_magnitude What linear solvers are you using? In the default setting, the coefficient matrix for linear solves will be B, but with target=sigma the coefficient matrix will be A-sigma*B; this may make a difference. Also, in any case, if experiencing convergence problems I would suggest using MUMPS (see section 3.4.1 of SLEPc's users manual). Jose > El 17 feb 2017, a las 10:25, Christopher Pierce escribi?: > > Hello All, > > I'm trying to use the SLEPc Krylov-Schur implementation to solve a > general eigenvalue problem. I have a monitor on my solver and the > solutions appear to converge correctly when using the approximation for > the residual norm in the algorithm. However, when the solutions are > displayed and I retrieve the actual residual norm it is very large and > increases with the size of the matrices I am working with. In some > cases it may be 10^17 times as large as the approximate norm. I also > don't get the eigenvalues I would expect for the system I am studying. > > When I turn on the option "true residual" the solver fails to converge. > The residual norm shrinks to some limit (~10^-3) and then sits there for > the remaining iterations. As a note, I am solving for the eigenvalues > with the smallest real part. I have also tried the RQCG solver on the > same problems and appear to get the correct results using it, but I'm > looking to use the better scaling of the Krylov-Schur solver. > > Does anyone know what could be causing this behavior? > > Thanks, > > Chris Pierce > WPI Center for Computational Nanoscience > > From ahasan94 at gmail.com Fri Feb 17 16:29:29 2017 From: ahasan94 at gmail.com (Mahmud Hasan) Date: Fri, 17 Feb 2017 16:29:29 -0600 Subject: [petsc-users] PETSc user Message-ID: Dear Sir/Madam, Good afternoon. I am trying to learn PETSc. Please add me to email list for future information about PETSc. Thank you. Have a nice day. Regards-- Mahmud Hasan, PhD, PE (Texas and Louisiana) Email: ahasan94 at gmail.com Cell: 832-248-3770 -------------- next part -------------- An HTML attachment was scrubbed... URL: From lixin_chu at yahoo.com Sat Feb 18 05:06:41 2017 From: lixin_chu at yahoo.com (lixin chu) Date: Sat, 18 Feb 2017 11:06:41 +0000 (UTC) Subject: [petsc-users] PCSETUP_FAILED due to FACTOR_NUMERIC_ZEROPIVOT References: <270456863.252483.1487416001868.ref@mail.yahoo.com> Message-ID: <270456863.252483.1487416001868@mail.yahoo.com> Hello, trying my first PETSc program, some matrices ok, but this one showing error: Linear solve did not converge due to DIVERGED_PCSETUP_FAILED iterations 0? ? ? ? ? ? ? ?PCSETUP_FAILED due to FACTOR_NUMERIC_ZEROPIVOT Not sure what I can try (I tried PC and GMRES, both no good) ? /*?* Matrix A?* ? 1 ?0 ?0 ?1 ?1 ?1?* ? 0 ?1 ?1 ?1 ?0 ?1?* ? 1 ?0 ?0 ?0 ?1 ?1?* ? 0 ?0 -1 ?0 ?1 ?1?* ? 1 ?1 ?0 ?1 ?1 ?1?* ? 0 ?1 ?1 ?1 ?0 ?0?*?* Expected inverse result?* 0 ?0 ?0 ?-1 ?1 ?-1?* -1 ?0 ?0 ? 0 ?1 ? 0?* 0 ?0 ?1 ? 0 -1 ? 1?* 1 ?0 -1 ? 0 ?0 ? 0?* 0 -1 ?1 ? 1 -1 ? 2?* 0 ?1 ?0 ? 0 ?0 ?-1?*?*/ Here is my code: Vec ? ? ? ? ? ?x,b; ? ? ? ?? Mat ? ? ? ? ? ?A; ? ? ? ? ?? KSP ? ? ? ? ? solver; ? ? ? ? ? PC ? ? ? ? ? ? pc; ? ? ? ? ? ? ? ?? PetscMPIInt ? ?size; int mpi_rank; bool is_mpi_root = FALSE; /* * ?initialize PETSc and MPI */ PetscInitialize ( &argc, &argv, (char*)0, USAGE); int col_index [6][6] = { {0, 1, 2, 3, 4, 5}, {0, 1, 2, 3, 4, 5}, {0, 1, 2, 3, 4, 5}, {0, 1, 2, 3, 4, 5}, {0, 1, 2, 3, 4, 5}, {0, 1, 2, 3, 4, 5} }; double v [6][6] = { {1., 0., ?0., 1., 1., 1.}, {0., 1., ?1., 1., 0., 1.}, {1., 0., ?0., 0., 1., 1.}, {0., 0., -1., 0., 1., 1.}, {1., 1., ?0., 1., 1., 1.}, {0., 1., ?1., 1., 0., 0.} }; int nz_rows [] = {6, 6, 6, 6, 6, 6}; int N = 6; int row, col; double value = 1.; int m, n; int b_start, b_end; // create x VecCreate ( PETSC_COMM_WORLD, &x ); VecSetSizes ( x, PETSC_DECIDE, N ); VecSetType ( x, VECMPI ); // create b VecCreate ( PETSC_COMM_WORLD, &b ); VecSetSizes ( b, PETSC_DECIDE, N ); VecSetType ( b, VECMPI ); // create A MatCreate ( PETSC_COMM_WORLD, &A ); MatSetSizes ( A, PETSC_DECIDE, PETSC_DECIDE, N, N ); MatSetUp ( A ); for ( row = 0; row < N; row++ ) { MatSetValues ( A, 1, &row, nz_rows[row], col_index[row], v[row], INSERT_VALUES ); } MatAssemblyBegin ( A, MAT_FINAL_ASSEMBLY ) ; MatAssemblyEnd ( A, MAT_FINAL_ASSEMBLY ) ; MatView ( A, PETSC_VIEWER_STDOUT_WORLD ); /* * setting up KSP */ KSPCreate ( PETSC_COMM_WORLD, &solver ); KSPSetOperators ( solver, A, A ); // KSPSetType ( solver, KSPCG ); KSPSetType ( solver, KSPGMRES ); for ( col = 0; col < N; col++ ) { // set b in root process VecGetOwnershipRange ( b , &b_start , &b_end ) ; // clear all values VecSet ( b, 0 ); // set 1 in the correct process. // each process owns a block of b : [b_start, b_end) // index starts from 0 if ( col >= b_start && col < b_end ) VecSetValues ( b, 1, &col, &value, INSERT_VALUES ); // distribute b VecAssemblyBegin ( b ) ; VecAssemblyEnd ( b ) ; // must be called from all processes //VecView ( b, PETSC_VIEWER_STDOUT_WORLD ); KSPSolve ( solver, b, x ); VecView ( x, PETSC_VIEWER_STDOUT_WORLD ); } /* * destroy KSP */ VecDestroy ( &b ); VecDestroy ( &x ); MatDestroy ( &A ); KSPDestroy(&solver); /* * finalize PETSc */ PetscFinalize(); return 0; I am running this with one MPI process, and the output seems to indicate that the A matrix is set correctly: Mat Object: 1 MPI processes? type: seqaijrow 0: (0, 1.) ?(1, 0.) ?(2, 0.) ?(3, 1.) ?(4, 1.) ?(5, 1.)?row 1: (0, 0.) ?(1, 1.) ?(2, 1.) ?(3, 1.) ?(4, 0.) ?(5, 1.)?row 2: (0, 1.) ?(1, 0.) ?(2, 0.) ?(3, 0.) ?(4, 1.) ?(5, 1.)?row 3: (0, 0.) ?(1, 0.) ?(2, -1.) ?(3, 0.) ?(4, 1.) ?(5, 1.)?row 4: (0, 1.) ?(1, 1.) ?(2, 0.) ?(3, 1.) ?(4, 1.) ?(5, 1.)?row 5: (0, 0.) ?(1, 1.) ?(2, 1.) ?(3, 1.) ?(4, 0.) ?(5, 0.)? Thanks for any help !?rgds LX -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Feb 18 11:07:53 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 18 Feb 2017 11:07:53 -0600 Subject: [petsc-users] PCSETUP_FAILED due to FACTOR_NUMERIC_ZEROPIVOT In-Reply-To: <270456863.252483.1487416001868@mail.yahoo.com> References: <270456863.252483.1487416001868.ref@mail.yahoo.com> <270456863.252483.1487416001868@mail.yahoo.com> Message-ID: Because of the zeros on the diagonal the sparse (I)LU factorization failed (sparse (I)LU factorizations generally don't use partial pivoting and hence can fail on nonsingular matrices unlike dense factorizations). Did you run with -pc_type lu ? Or the defaults: it is always useful to run with -ksp_view to see what solver options are being used. Where did this matrix come from? Usually PETSc users have a specific class of problems they are solving and so don't hit upon these matrices. SuperLU (not superlu_dist) does do some partial pivoting and so will usually factor such matrices. You can use ./configure --download-superlu and then run the example with -pc_type lu -pc_factor_mat_solver_package superlu But for large matrices Superlu is pretty slow. Barry > On Feb 18, 2017, at 5:06 AM, lixin chu wrote: > > Hello, > > trying my first PETSc program, some matrices ok, but this one showing error: > > Linear solve did not converge due to DIVERGED_PCSETUP_FAILED iterations 0 > PCSETUP_FAILED due to FACTOR_NUMERIC_ZEROPIVOT > > Not sure what I can try (I tried PC and GMRES, both no good) ? > > > /* > * Matrix A > * 1 0 0 1 1 1 > * 0 1 1 1 0 1 > * 1 0 0 0 1 1 > * 0 0 -1 0 1 1 > * 1 1 0 1 1 1 > * 0 1 1 1 0 0 > * > * Expected inverse result > * 0 0 0 -1 1 -1 > * -1 0 0 0 1 0 > * 0 0 1 0 -1 1 > * 1 0 -1 0 0 0 > * 0 -1 1 1 -1 2 > * 0 1 0 0 0 -1 > * > */ > > Here is my code: > > Vec x,b; > Mat A; > KSP solver; > PC pc; > > PetscMPIInt size; > > int mpi_rank; > bool is_mpi_root = FALSE; > > /* > * initialize PETSc and MPI > */ > PetscInitialize ( &argc, &argv, (char*)0, USAGE); > > > int col_index [6][6] = { > {0, 1, 2, 3, 4, 5}, > {0, 1, 2, 3, 4, 5}, > {0, 1, 2, 3, 4, 5}, > {0, 1, 2, 3, 4, 5}, > {0, 1, 2, 3, 4, 5}, > {0, 1, 2, 3, 4, 5} > }; > > double v [6][6] = { > {1., 0., 0., 1., 1., 1.}, > {0., 1., 1., 1., 0., 1.}, > {1., 0., 0., 0., 1., 1.}, > {0., 0., -1., 0., 1., 1.}, > {1., 1., 0., 1., 1., 1.}, > {0., 1., 1., 1., 0., 0.} > }; > > int nz_rows [] = {6, 6, 6, 6, 6, 6}; > > int N = 6; > int row, col; > double value = 1.; > > int m, n; > > int b_start, b_end; > > // create x > VecCreate ( PETSC_COMM_WORLD, &x ); > VecSetSizes ( x, PETSC_DECIDE, N ); > VecSetType ( x, VECMPI ); > > > // create b > VecCreate ( PETSC_COMM_WORLD, &b ); > VecSetSizes ( b, PETSC_DECIDE, N ); > VecSetType ( b, VECMPI ); > > > // create A > MatCreate ( PETSC_COMM_WORLD, &A ); > MatSetSizes ( A, PETSC_DECIDE, PETSC_DECIDE, N, N ); > > MatSetUp ( A ); > > for ( row = 0; row < N; row++ ) { > MatSetValues ( A, 1, &row, nz_rows[row], col_index[row], v[row], INSERT_VALUES ); > } > MatAssemblyBegin ( A, MAT_FINAL_ASSEMBLY ) ; > MatAssemblyEnd ( A, MAT_FINAL_ASSEMBLY ) ; > > MatView ( A, PETSC_VIEWER_STDOUT_WORLD ); > > /* > * setting up KSP > */ > KSPCreate ( PETSC_COMM_WORLD, &solver ); > KSPSetOperators ( solver, A, A ); > // KSPSetType ( solver, KSPCG ); > KSPSetType ( solver, KSPGMRES ); > > for ( col = 0; col < N; col++ ) { > > // set b in root process > VecGetOwnershipRange ( b , &b_start , &b_end ) ; > > // clear all values > VecSet ( b, 0 ); > // set 1 in the correct process. > // each process owns a block of b : [b_start, b_end) > // index starts from 0 > > if ( col >= b_start && col < b_end ) > VecSetValues ( b, 1, &col, &value, INSERT_VALUES ); > > // distribute b > VecAssemblyBegin ( b ) ; > VecAssemblyEnd ( b ) ; > > // must be called from all processes > //VecView ( b, PETSC_VIEWER_STDOUT_WORLD ); > > KSPSolve ( solver, b, x ); > > VecView ( x, PETSC_VIEWER_STDOUT_WORLD ); > } > > /* > * destroy KSP > */ > VecDestroy ( &b ); > VecDestroy ( &x ); > MatDestroy ( &A ); > KSPDestroy(&solver); > > /* > * finalize PETSc > */ > PetscFinalize(); > > return 0; > > I am running this with one MPI process, and the output seems to indicate that the A matrix is set correctly: > > Mat Object: 1 MPI processes > type: seqaij > row 0: (0, 1.) (1, 0.) (2, 0.) (3, 1.) (4, 1.) (5, 1.) > row 1: (0, 0.) (1, 1.) (2, 1.) (3, 1.) (4, 0.) (5, 1.) > row 2: (0, 1.) (1, 0.) (2, 0.) (3, 0.) (4, 1.) (5, 1.) > row 3: (0, 0.) (1, 0.) (2, -1.) (3, 0.) (4, 1.) (5, 1.) > row 4: (0, 1.) (1, 1.) (2, 0.) (3, 1.) (4, 1.) (5, 1.) > row 5: (0, 0.) (1, 1.) (2, 1.) (3, 1.) (4, 0.) (5, 0.) > > > Thanks for any help ! > > rgds > LX From mvalera at mail.sdsu.edu Sat Feb 18 14:44:13 2017 From: mvalera at mail.sdsu.edu (Manuel Valera) Date: Sat, 18 Feb 2017 12:44:13 -0800 Subject: [petsc-users] A way to distribute 3D arrays. In-Reply-To: <2371E2C6-DB67-420C-8020-A53E2550BB35@mcs.anl.gov> References: <2371E2C6-DB67-420C-8020-A53E2550BB35@mcs.anl.gov> Message-ID: thanks guys that helped a lot! I think i got it know, i copy the code i created in case you want to suggest something or maybe use it as example... I have a bonus question: Im going to operate next in several arrays at the same time, i created them using their slightly different layouts, one DA each, a peer told me different DA are not guaranteed to share the same corners, is this correct? if so, is there a way to enforce that? Thanks SUBROUTINE DAs(da,veczero,localv,array) ! use PetscObjects, only :: ierr ! Umbrella program to update and communicate the arrays in a ! distributed fashion using the DMDA objects from PETSc. ! Manuel Valera 1/20/17 ! Arguments: ! da = DMDA array (3D) already created and setup ! veczero = ! globalv = ! localv = local chunk each processor works in. ! array = the array to be petscified. ONLY 3D ARRAYS as now. ! arrayp = the petsc version of the array, may be not needed. ! cta = the scatter context to translate globalv to localv in DA indices Vec,intent(inout) :: veczero,localv Vec :: natural,globalv PetscScalar,dimension(:,:,:) :: array PetscScalar,pointer,dimension(:,:,:) :: arrayp DM,intent(inout) :: da VecScatter :: cta PetscInt ::xind,yind,zind,xwidth,ywidth,zwidth !Debug: !print*,"INFO BEFORE SCATTER:" call DAinf(da,xind,yind,zind,xwidth,ywidth,zwidth) print*, SIZE(array,1),'x',SIZE(array,2),'x',SIZE(array,3) ! call DMCreateLocalVector(da,localv,ierr) !The following if-block is an attempt to scatter the arrays as vectors !loaded from master node. if (sizex/= 1)then !PETSC Devs suggested: !Buffer 3d array natural has diff ordering than DMDAs: call DMDACreateNaturalVector(da,natural,ierr) call DMDAVecGetArrayF90(da,natural,arrayp,ierr) !**fill up veczero***: !TODO if(rank==0)then arrayp = array(xind+1:xwidth,yind+1:ywidth,zind+1:zwidth) endif call DMDAVecRestoreArrayF90(da,natural,arrayp,ierr) !call DMRestoreNaturalVector(da,natural,ierr) !???? not needed? !Distributing array: call VecScatterCreateToZero(natural,cta,veczero,ierr) call VecScatterBegin(cta,veczero,natural,INSERT_VALUES,SCATTER_REVERSE,ierr) call VecScatterEnd(cta,veczero,natural,INSERT_VALUES,SCATTER_REVERSE,ierr) call DMCreateGlobalVector(da,globalv,ierr) call DMDANaturalToGlobalBegin(da,natural,INSERT_VALUES,globalv,ierr) call DMDANaturalToGlobalEnd(da,natural,INSERT_VALUES,globalv,ierr) call DMGlobalToLocalBegin(da,globalv,INSERT_VALUES,localv,ierr) call DMGlobalToLocalEnd(da,globalv,INSERT_VALUES,localv,ierr) else print*, "This should be called in serial only." call DMDAVecGetArrayF90(da,localv,arrayp,ierr) arrayp = array call DMDAVecRestoreArrayF90(da,localv,arrayp,ierr) call DMRestoreLocalVector(da,localv,ierr) endif call VecDestroy(globalv,ierr) call VecScatterDestroy(cta,ierr) call VecDestroy(natural,ierr) END SUBROUTINE DAs On Wed, Feb 15, 2017 at 7:31 PM, Barry Smith wrote: > > You can do this in the style of > > VecLoad_Binary_DA() > > first you take your sequential vector and make it parallel in the > "natural" ordering for 3d arrays > > DMDACreateNaturalVector(da,&natural); > VecScatterCreateToZero(natural,&scatter,&veczero); /* veczero is of > full size on process 0 and has zero entries on all other processes*/ > /* fill up veczero */ > VecScatterBegin(scatter,veczero,natural,INSERT_VALUES,SCATTER_REVERSE); > VecScatterEnd(scatter,veczero,natural,INSERT_VALUES,SCATTER_REVERSE); > > and then move it into the PETSc DMDA parallel ordering vector with > > ierr = DMCreateGlobalVector(da,&xin);CHKERRQ(ierr); > ierr = DMDANaturalToGlobalBegin(da,natural,INSERT_VALUES,xin); > CHKERRQ(ierr); > ierr = DMDANaturalToGlobalEnd(da,natural,INSERT_VALUES,xin); > CHKERRQ(ierr); > > > > On Feb 15, 2017, at 7:16 PM, Manuel Valera > wrote: > > > > Hello, > > > > My question this time is just if there is a way to distribute a 3D array > whos located at Zero rank over the processors, if possible using the DMDAs, > i'm trying not to do a lot of initialization I/O in parallel. > > > > Thanks for your time, > > > > Manuel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Feb 18 14:49:25 2017 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 18 Feb 2017 14:49:25 -0600 Subject: [petsc-users] A way to distribute 3D arrays. In-Reply-To: References: <2371E2C6-DB67-420C-8020-A53E2550BB35@mcs.anl.gov> Message-ID: On Sat, Feb 18, 2017 at 2:44 PM, Manuel Valera wrote: > thanks guys that helped a lot! > > I think i got it know, i copy the code i created in case you want to > suggest something or maybe use it as example... > > I have a bonus question: Im going to operate next in several arrays at the > same time, i created them using their slightly different layouts, one DA > each, a peer told me different DA are not guaranteed to share the same > corners, is this correct? if so, is there a way to enforce that? Thanks > I do not understand. Do these DAs have the same size, meaning number of vertices in X, Y, and Z? If so, why are you using separate arrays. You can just use 'dof' to put all your data there, and strip out subvectors using VecStride functions, such as http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecStrideScatter.html If the DAs are different sizes, how would they ever have the same corners. Thanks, Matt > SUBROUTINE DAs(da,veczero,localv,array) > > ! use PetscObjects, only :: ierr > > ! Umbrella program to update and communicate the arrays in a > ! distributed fashion using the DMDA objects from PETSc. > ! Manuel Valera 1/20/17 > > ! Arguments: > ! da = DMDA array (3D) already created and setup > ! veczero = > ! globalv = > ! localv = local chunk each processor works in. > ! array = the array to be petscified. ONLY 3D ARRAYS as now. > ! arrayp = the petsc version of the array, may be not needed. > ! cta = the scatter context to translate globalv to localv in DA > indices > > Vec,intent(inout) :: veczero,localv > Vec :: natural,globalv > PetscScalar,dimension(:,:,:) :: array > PetscScalar,pointer,dimension(:,:,:) :: arrayp > DM,intent(inout) :: da > VecScatter :: cta > PetscInt ::xind,yind,zind,xwidth, > ywidth,zwidth > > > > !Debug: > !print*,"INFO BEFORE SCATTER:" > call DAinf(da,xind,yind,zind,xwidth,ywidth,zwidth) > print*, SIZE(array,1),'x',SIZE(array,2),'x',SIZE(array,3) > ! > > call DMCreateLocalVector(da,localv,ierr) > > !The following if-block is an attempt to scatter the arrays as > vectors > !loaded from master node. > > if (sizex/= 1)then > !PETSC Devs suggested: > !Buffer 3d array natural has diff ordering than DMDAs: > call DMDACreateNaturalVector(da,natural,ierr) > call DMDAVecGetArrayF90(da,natural,arrayp,ierr) > !**fill up veczero***: > !TODO > if(rank==0)then > arrayp = array(xind+1:xwidth,yind+1:ywidth,zind+1:zwidth) > endif > call DMDAVecRestoreArrayF90(da,natural,arrayp,ierr) > !call DMRestoreNaturalVector(da,natural,ierr) !???? not > needed? > > !Distributing array: > call VecScatterCreateToZero(natural,cta,veczero,ierr) > call VecScatterBegin(cta,veczero,natural,INSERT_VALUES,SCATTER_ > REVERSE,ierr) > call VecScatterEnd(cta,veczero,natural,INSERT_VALUES,SCATTER_ > REVERSE,ierr) > > > call DMCreateGlobalVector(da,globalv,ierr) > call DMDANaturalToGlobalBegin(da,natural,INSERT_VALUES,globalv, > ierr) > call DMDANaturalToGlobalEnd(da,natural,INSERT_VALUES,globalv,ierr) > > call DMGlobalToLocalBegin(da,globalv,INSERT_VALUES,localv,ierr) > call DMGlobalToLocalEnd(da,globalv,INSERT_VALUES,localv,ierr) > > else > > print*, "This should be called in serial only." > > call DMDAVecGetArrayF90(da,localv,arrayp,ierr) > > arrayp = array > > call DMDAVecRestoreArrayF90(da,localv,arrayp,ierr) > call DMRestoreLocalVector(da,localv,ierr) > endif > > call VecDestroy(globalv,ierr) > call VecScatterDestroy(cta,ierr) > call VecDestroy(natural,ierr) > > > > END SUBROUTINE DAs > > > > On Wed, Feb 15, 2017 at 7:31 PM, Barry Smith wrote: > >> >> You can do this in the style of >> >> VecLoad_Binary_DA() >> >> first you take your sequential vector and make it parallel in the >> "natural" ordering for 3d arrays >> >> DMDACreateNaturalVector(da,&natural); >> VecScatterCreateToZero(natural,&scatter,&veczero); /* veczero is of >> full size on process 0 and has zero entries on all other processes*/ >> /* fill up veczero */ >> VecScatterBegin(scatter,veczero,natural,INSERT_VALUES,SCATT >> ER_REVERSE); >> VecScatterEnd(scatter,veczero,natural,INSERT_VALUES,SCATTER_REVERSE); >> >> and then move it into the PETSc DMDA parallel ordering vector with >> >> ierr = DMCreateGlobalVector(da,&xin);CHKERRQ(ierr); >> ierr = DMDANaturalToGlobalBegin(da,natural,INSERT_VALUES,xin);CHKER >> RQ(ierr); >> ierr = DMDANaturalToGlobalEnd(da,natural,INSERT_VALUES,xin);CHKERRQ >> (ierr); >> >> >> > On Feb 15, 2017, at 7:16 PM, Manuel Valera >> wrote: >> > >> > Hello, >> > >> > My question this time is just if there is a way to distribute a 3D >> array whos located at Zero rank over the processors, if possible using the >> DMDAs, i'm trying not to do a lot of initialization I/O in parallel. >> > >> > Thanks for your time, >> > >> > Manuel >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mvalera at mail.sdsu.edu Sat Feb 18 14:54:57 2017 From: mvalera at mail.sdsu.edu (Manuel Valera) Date: Sat, 18 Feb 2017 12:54:57 -0800 Subject: [petsc-users] A way to distribute 3D arrays. In-Reply-To: References: <2371E2C6-DB67-420C-8020-A53E2550BB35@mcs.anl.gov> Message-ID: Is very easy Matthew, im porting a serial code into a parallel petsc based one and i want to do it in the most natural straightforward way, also im learning petsc at the same time. So the original code uses staggered grid and every variable is stored in a separate array, so i created a DA for each common array of them but they differ in just +/-1 to +/-3 in each dimension. So i guess i should allocate all of them in the same DA and just ignore the extra non used points for some arrays, right? Regards, Manuel On Sat, Feb 18, 2017 at 12:49 PM, Matthew Knepley wrote: > On Sat, Feb 18, 2017 at 2:44 PM, Manuel Valera > wrote: > >> thanks guys that helped a lot! >> >> I think i got it know, i copy the code i created in case you want to >> suggest something or maybe use it as example... >> >> I have a bonus question: Im going to operate next in several arrays at >> the same time, i created them using their slightly different layouts, one >> DA each, a peer told me different DA are not guaranteed to share the same >> corners, is this correct? if so, is there a way to enforce that? Thanks >> > > I do not understand. Do these DAs have the same size, meaning number of > vertices in X, Y, and Z? If so, why are you > using separate arrays. You can just use 'dof' to put all your data there, > and strip out subvectors using VecStride > functions, such as > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/ > VecStrideScatter.html > > If the DAs are different sizes, how would they ever have the same corners. > > Thanks, > > Matt > > >> SUBROUTINE DAs(da,veczero,localv,array) >> >> ! use PetscObjects, only :: ierr >> >> ! Umbrella program to update and communicate the arrays in a >> ! distributed fashion using the DMDA objects from PETSc. >> ! Manuel Valera 1/20/17 >> >> ! Arguments: >> ! da = DMDA array (3D) already created and setup >> ! veczero = >> ! globalv = >> ! localv = local chunk each processor works in. >> ! array = the array to be petscified. ONLY 3D ARRAYS as now. >> ! arrayp = the petsc version of the array, may be not needed. >> ! cta = the scatter context to translate globalv to localv in DA >> indices >> >> Vec,intent(inout) :: veczero,localv >> Vec :: natural,globalv >> PetscScalar,dimension(:,:,:) :: array >> PetscScalar,pointer,dimension(:,:,:) :: arrayp >> DM,intent(inout) :: da >> VecScatter :: cta >> PetscInt ::xind,yind,zind,xwidth,ywidt >> h,zwidth >> >> >> >> !Debug: >> !print*,"INFO BEFORE SCATTER:" >> call DAinf(da,xind,yind,zind,xwidth,ywidth,zwidth) >> print*, SIZE(array,1),'x',SIZE(array,2),'x',SIZE(array,3) >> ! >> >> call DMCreateLocalVector(da,localv,ierr) >> >> !The following if-block is an attempt to scatter the arrays as >> vectors >> !loaded from master node. >> >> if (sizex/= 1)then >> !PETSC Devs suggested: >> !Buffer 3d array natural has diff ordering than DMDAs: >> call DMDACreateNaturalVector(da,natural,ierr) >> call DMDAVecGetArrayF90(da,natural,arrayp,ierr) >> !**fill up veczero***: >> !TODO >> if(rank==0)then >> arrayp = array(xind+1:xwidth,yind+1:ywidth,zind+1:zwidth) >> endif >> call DMDAVecRestoreArrayF90(da,natural,arrayp,ierr) >> !call DMRestoreNaturalVector(da,natural,ierr) !???? not >> needed? >> >> !Distributing array: >> call VecScatterCreateToZero(natural,cta,veczero,ierr) >> call VecScatterBegin(cta,veczero,natural,INSERT_VALUES,SCATTER_RE >> VERSE,ierr) >> call VecScatterEnd(cta,veczero,natural,INSERT_VALUES,SCATTER_REVE >> RSE,ierr) >> >> >> call DMCreateGlobalVector(da,globalv,ierr) >> call DMDANaturalToGlobalBegin(da,natural,INSERT_VALUES,globalv,ie >> rr) >> call DMDANaturalToGlobalEnd(da,natural,INSERT_VALUES,globalv,ierr >> ) >> >> call DMGlobalToLocalBegin(da,globalv,INSERT_VALUES,localv,ierr) >> call DMGlobalToLocalEnd(da,globalv,INSERT_VALUES,localv,ierr) >> >> else >> >> print*, "This should be called in serial only." >> >> call DMDAVecGetArrayF90(da,localv,arrayp,ierr) >> >> arrayp = array >> >> call DMDAVecRestoreArrayF90(da,localv,arrayp,ierr) >> call DMRestoreLocalVector(da,localv,ierr) >> endif >> >> call VecDestroy(globalv,ierr) >> call VecScatterDestroy(cta,ierr) >> call VecDestroy(natural,ierr) >> >> >> >> END SUBROUTINE DAs >> >> >> >> On Wed, Feb 15, 2017 at 7:31 PM, Barry Smith wrote: >> >>> >>> You can do this in the style of >>> >>> VecLoad_Binary_DA() >>> >>> first you take your sequential vector and make it parallel in the >>> "natural" ordering for 3d arrays >>> >>> DMDACreateNaturalVector(da,&natural); >>> VecScatterCreateToZero(natural,&scatter,&veczero); /* veczero is of >>> full size on process 0 and has zero entries on all other processes*/ >>> /* fill up veczero */ >>> VecScatterBegin(scatter,veczero,natural,INSERT_VALUES,SCATT >>> ER_REVERSE); >>> VecScatterEnd(scatter,veczero,natural,INSERT_VALUES,SCATTER_REVERSE); >>> >>> and then move it into the PETSc DMDA parallel ordering vector with >>> >>> ierr = DMCreateGlobalVector(da,&xin);CHKERRQ(ierr); >>> ierr = DMDANaturalToGlobalBegin(da,natural,INSERT_VALUES,xin);CHKER >>> RQ(ierr); >>> ierr = DMDANaturalToGlobalEnd(da,natural,INSERT_VALUES,xin);CHKERRQ >>> (ierr); >>> >>> >>> > On Feb 15, 2017, at 7:16 PM, Manuel Valera >>> wrote: >>> > >>> > Hello, >>> > >>> > My question this time is just if there is a way to distribute a 3D >>> array whos located at Zero rank over the processors, if possible using the >>> DMDAs, i'm trying not to do a lot of initialization I/O in parallel. >>> > >>> > Thanks for your time, >>> > >>> > Manuel >>> >>> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Feb 18 15:03:04 2017 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 18 Feb 2017 15:03:04 -0600 Subject: [petsc-users] A way to distribute 3D arrays. In-Reply-To: References: <2371E2C6-DB67-420C-8020-A53E2550BB35@mcs.anl.gov> Message-ID: On Sat, Feb 18, 2017 at 2:54 PM, Manuel Valera wrote: > Is very easy Matthew, im porting a serial code into a parallel petsc based > one and i want to do it in the most natural straightforward way, also im > learning petsc at the same time. > > So the original code uses staggered grid and every variable is stored in a > separate array, so i created a DA for each common array of them but they > differ in just +/-1 to +/-3 in each dimension. > > So i guess i should allocate all of them in the same DA and just ignore > the extra non used points for some arrays, right? > I have seen many staggered grid implementation in PETSc, but the most successful have been those that used a single DA for everything, and just allowed some points at the boundary to unused. I would advocate this, but before you do it I would make a detailed drawing of the 3 layers around a corner, showing all unknowns and marking them with their indices. This is invaluable when writing the equations. Thanks, Matt > Regards, > > Manuel > > On Sat, Feb 18, 2017 at 12:49 PM, Matthew Knepley > wrote: > >> On Sat, Feb 18, 2017 at 2:44 PM, Manuel Valera >> wrote: >> >>> thanks guys that helped a lot! >>> >>> I think i got it know, i copy the code i created in case you want to >>> suggest something or maybe use it as example... >>> >>> I have a bonus question: Im going to operate next in several arrays at >>> the same time, i created them using their slightly different layouts, one >>> DA each, a peer told me different DA are not guaranteed to share the same >>> corners, is this correct? if so, is there a way to enforce that? Thanks >>> >> >> I do not understand. Do these DAs have the same size, meaning number of >> vertices in X, Y, and Z? If so, why are you >> using separate arrays. You can just use 'dof' to put all your data there, >> and strip out subvectors using VecStride >> functions, such as >> >> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpage >> s/Vec/VecStrideScatter.html >> >> If the DAs are different sizes, how would they ever have the same corners. >> >> Thanks, >> >> Matt >> >> >>> SUBROUTINE DAs(da,veczero,localv,array) >>> >>> ! use PetscObjects, only :: ierr >>> >>> ! Umbrella program to update and communicate the arrays in a >>> ! distributed fashion using the DMDA objects from PETSc. >>> ! Manuel Valera 1/20/17 >>> >>> ! Arguments: >>> ! da = DMDA array (3D) already created and setup >>> ! veczero = >>> ! globalv = >>> ! localv = local chunk each processor works in. >>> ! array = the array to be petscified. ONLY 3D ARRAYS as now. >>> ! arrayp = the petsc version of the array, may be not needed. >>> ! cta = the scatter context to translate globalv to localv in DA >>> indices >>> >>> Vec,intent(inout) :: veczero,localv >>> Vec :: natural,globalv >>> PetscScalar,dimension(:,:,:) :: array >>> PetscScalar,pointer,dimension(:,:,:) :: arrayp >>> DM,intent(inout) :: da >>> VecScatter :: cta >>> PetscInt ::xind,yind,zind,xwidth,ywidt >>> h,zwidth >>> >>> >>> >>> !Debug: >>> !print*,"INFO BEFORE SCATTER:" >>> call DAinf(da,xind,yind,zind,xwidth,ywidth,zwidth) >>> print*, SIZE(array,1),'x',SIZE(array,2),'x',SIZE(array,3) >>> ! >>> >>> call DMCreateLocalVector(da,localv,ierr) >>> >>> !The following if-block is an attempt to scatter the arrays as >>> vectors >>> !loaded from master node. >>> >>> if (sizex/= 1)then >>> !PETSC Devs suggested: >>> !Buffer 3d array natural has diff ordering than DMDAs: >>> call DMDACreateNaturalVector(da,natural,ierr) >>> call DMDAVecGetArrayF90(da,natural,arrayp,ierr) >>> !**fill up veczero***: >>> !TODO >>> if(rank==0)then >>> arrayp = array(xind+1:xwidth,yind+1:ywidth,zind+1:zwidth) >>> endif >>> call DMDAVecRestoreArrayF90(da,natural,arrayp,ierr) >>> !call DMRestoreNaturalVector(da,natural,ierr) !???? not >>> needed? >>> >>> !Distributing array: >>> call VecScatterCreateToZero(natural,cta,veczero,ierr) >>> call VecScatterBegin(cta,veczero,na >>> tural,INSERT_VALUES,SCATTER_REVERSE,ierr) >>> call VecScatterEnd(cta,veczero,natu >>> ral,INSERT_VALUES,SCATTER_REVERSE,ierr) >>> >>> >>> call DMCreateGlobalVector(da,globalv,ierr) >>> call DMDANaturalToGlobalBegin(da,na >>> tural,INSERT_VALUES,globalv,ierr) >>> call DMDANaturalToGlobalEnd(da,natu >>> ral,INSERT_VALUES,globalv,ierr) >>> >>> call DMGlobalToLocalBegin(da,globalv,INSERT_VALUES,localv,ierr) >>> call DMGlobalToLocalEnd(da,globalv,INSERT_VALUES,localv,ierr) >>> >>> else >>> >>> print*, "This should be called in serial only." >>> >>> call DMDAVecGetArrayF90(da,localv,arrayp,ierr) >>> >>> arrayp = array >>> >>> call DMDAVecRestoreArrayF90(da,localv,arrayp,ierr) >>> call DMRestoreLocalVector(da,localv,ierr) >>> endif >>> >>> call VecDestroy(globalv,ierr) >>> call VecScatterDestroy(cta,ierr) >>> call VecDestroy(natural,ierr) >>> >>> >>> >>> END SUBROUTINE DAs >>> >>> >>> >>> On Wed, Feb 15, 2017 at 7:31 PM, Barry Smith wrote: >>> >>>> >>>> You can do this in the style of >>>> >>>> VecLoad_Binary_DA() >>>> >>>> first you take your sequential vector and make it parallel in the >>>> "natural" ordering for 3d arrays >>>> >>>> DMDACreateNaturalVector(da,&natural); >>>> VecScatterCreateToZero(natural,&scatter,&veczero); /* veczero is of >>>> full size on process 0 and has zero entries on all other processes*/ >>>> /* fill up veczero */ >>>> VecScatterBegin(scatter,veczero,natural,INSERT_VALUES,SCATT >>>> ER_REVERSE); >>>> VecScatterEnd(scatter,veczero,natural,INSERT_VALUES,SCATTER_REVERSE); >>>> >>>> and then move it into the PETSc DMDA parallel ordering vector with >>>> >>>> ierr = DMCreateGlobalVector(da,&xin);CHKERRQ(ierr); >>>> ierr = DMDANaturalToGlobalBegin(da,natural,INSERT_VALUES,xin);CHKER >>>> RQ(ierr); >>>> ierr = DMDANaturalToGlobalEnd(da,natural,INSERT_VALUES,xin);CHKERRQ >>>> (ierr); >>>> >>>> >>>> > On Feb 15, 2017, at 7:16 PM, Manuel Valera >>>> wrote: >>>> > >>>> > Hello, >>>> > >>>> > My question this time is just if there is a way to distribute a 3D >>>> array whos located at Zero rank over the processors, if possible using the >>>> DMDAs, i'm trying not to do a lot of initialization I/O in parallel. >>>> > >>>> > Thanks for your time, >>>> > >>>> > Manuel >>>> >>>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From lixin_chu at yahoo.com Sat Feb 18 15:58:32 2017 From: lixin_chu at yahoo.com (lixin chu) Date: Sat, 18 Feb 2017 21:58:32 +0000 (UTC) Subject: [petsc-users] PCSETUP_FAILED due to FACTOR_NUMERIC_ZEROPIVOT In-Reply-To: References: <270456863.252483.1487416001868.ref@mail.yahoo.com> <270456863.252483.1487416001868@mail.yahoo.com> Message-ID: <956969461.383999.1487455112895@mail.yahoo.com> Thank you very much Barry ! I hand crafted this for leaning PETSc APIs. Thought I can get the solution since the solution exists 'mathematically'.? Output of -kep_view shows that it uses "ILU: out-of-place factorization" I think: KSP Object: 1 MPI processes? type: gmres? ? GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement? ? GMRES: happy breakdown tolerance 1e-30? maximum iterations=10000, initial guess is zero? tolerances: ?relative=1e-05, absolute=1e-50, divergence=10000.? left preconditioning? using PRECONDITIONED norm type for convergence testPC Object: 1 MPI processes? type: ilu? ? ILU: out-of-place factorization? ? 0 levels of fill? ? tolerance for zero pivot 2.22045e-14? ? matrix ordering: natural? ? factor fill ratio given 1., needed 1. Will test with real matrices when I learn enough PETSc.?Thanks ! lx On Sunday, 19 February 2017, 1:07, Barry Smith wrote: ? Because of the zeros on the diagonal the sparse (I)LU factorization failed (sparse (I)LU factorizations generally don't use partial pivoting and hence can fail on nonsingular matrices unlike dense factorizations).? Did you run with -pc_type lu ? Or the defaults: it is always useful to run with -ksp_view to see what solver options are being used. ? Where did this matrix come from? Usually PETSc users have a specific class of problems they are solving and so don't hit upon these matrices. ? SuperLU (not superlu_dist) does do some partial pivoting and so will usually factor such matrices. You can use ./configure --download-superlu? ? and then run the example with -pc_type lu -pc_factor_mat_solver_package superlu But for large matrices Superlu is pretty slow. ? Barry > On Feb 18, 2017, at 5:06 AM, lixin chu wrote: > > Hello, > > trying my first PETSc program, some matrices ok, but this one showing error: > > Linear solve did not converge due to DIVERGED_PCSETUP_FAILED iterations 0 >? ? ? ? ? ? ? ? PCSETUP_FAILED due to FACTOR_NUMERIC_ZEROPIVOT > > Not sure what I can try (I tried PC and GMRES, both no good) ? > > > /* >? * Matrix A >? *? 1? 0? 0? 1? 1? 1 >? *? 0? 1? 1? 1? 0? 1 >? *? 1? 0? 0? 0? 1? 1 >? *? 0? 0 -1? 0? 1? 1 >? *? 1? 1? 0? 1? 1? 1 >? *? 0? 1? 1? 1? 0? 0 >? * >? * Expected inverse result >? *??? 0? 0? 0? -1? 1? -1 >? *??? -1? 0? 0? 0? 1? 0 >? *??? 0? 0? 1? 0 -1? 1 >? *??? 1? 0 -1? 0? 0? 0 >? *??? 0 -1? 1? 1 -1? 2 >? *??? 0? 1? 0? 0? 0? -1 >? * >? */ > > Here is my code: > > ??? Vec? ? ? ? ? ? x,b;??? ? ? ? ? > ??? Mat? ? ? ? ? ? A;? ??? ? ? ? ? > ??? KSP? ? ? ? ? solver;? ? ? ? ? > ??? PC? ? ? ? ? ? pc;? ? ? ? ? ? ? ? > > ??? PetscMPIInt? ? size; > > ??? int mpi_rank; > ??? bool is_mpi_root = FALSE; > > ??? /* > ??? *? initialize PETSc and MPI > ??? */ > ??? PetscInitialize ( &argc, &argv, (char*)0, USAGE); > > > ??? int col_index [6][6] = { > ??? ??? ??? {0, 1, 2, 3, 4, 5}, > ??? ??? ??? {0, 1, 2, 3, 4, 5}, > ??? ??? ??? {0, 1, 2, 3, 4, 5}, > ??? ??? ??? {0, 1, 2, 3, 4, 5}, > ??? ??? ??? {0, 1, 2, 3, 4, 5}, > ??? ??? ??? {0, 1, 2, 3, 4, 5} > ??? }; > > ??? double v [6][6] = { > ??? ??? ??? {1., 0.,? 0., 1., 1., 1.}, > ??? ??? ??? {0., 1.,? 1., 1., 0., 1.}, > ??? ??? ??? {1., 0.,? 0., 0., 1., 1.}, > ??? ??? ??? {0., 0., -1., 0., 1., 1.}, > ??? ??? ??? {1., 1.,? 0., 1., 1., 1.}, > ??? ??? ??? {0., 1.,? 1., 1., 0., 0.} > ??? }; > > ??? int nz_rows [] = {6, 6, 6, 6, 6, 6}; > > ??? int N = 6; > ??? int row, col; > ??? double value = 1.; > > ??? int m, n; > > ??? int b_start, b_end; > > ??? // create x > ??? VecCreate ( PETSC_COMM_WORLD, &x ); > ??? VecSetSizes ( x, PETSC_DECIDE, N ); > ??? VecSetType ( x, VECMPI ); > > > ??? // create b > ??? VecCreate ( PETSC_COMM_WORLD, &b ); > ??? VecSetSizes ( b, PETSC_DECIDE, N ); > ??? VecSetType ( b, VECMPI ); > > > ??? // create A > ??? MatCreate ( PETSC_COMM_WORLD, &A ); > ??? MatSetSizes ( A, PETSC_DECIDE, PETSC_DECIDE, N, N ); > > ??? MatSetUp ( A ); > > ??? for ( row = 0; row < N; row++ ) { > ??? ??? MatSetValues ( A, 1, &row, nz_rows[row], col_index[row], v[row], INSERT_VALUES ); > ??? } > ??? MatAssemblyBegin ( A, MAT_FINAL_ASSEMBLY ) ; > ??? MatAssemblyEnd ( A, MAT_FINAL_ASSEMBLY ) ; > > ??? MatView ( A, PETSC_VIEWER_STDOUT_WORLD ); > > ??? /* > ??? * setting up KSP > ??? */ > ??? KSPCreate ( PETSC_COMM_WORLD, &solver ); > ??? KSPSetOperators ( solver, A, A ); > ??? // KSPSetType ( solver, KSPCG ); > ??? KSPSetType ( solver, KSPGMRES ); > > ??? for ( col = 0; col < N; col++ ) { > > ??? ??? // set b in root process > ??? ??? VecGetOwnershipRange ( b , &b_start , &b_end ) ; > > ??? ??? // clear all values > ??? ??? VecSet ( b, 0 ); > ??? ??? // set 1 in the correct process. > ??? ??? // each process owns a block of b : [b_start, b_end) > ??? ??? // index starts from 0 > > ??? ??? if ( col >= b_start && col < b_end ) > ??? ??? ??? VecSetValues ( b, 1, &col, &value, INSERT_VALUES ); > > ??? ??? // distribute b > ??? ??? VecAssemblyBegin ( b ) ; > ??? ??? VecAssemblyEnd ( b ) ; > > ??? ??? // must be called from all processes > ??? ??? //VecView ( b, PETSC_VIEWER_STDOUT_WORLD ); > > ??? ??? KSPSolve ( solver, b, x ); > > ??? ??? VecView ( x, PETSC_VIEWER_STDOUT_WORLD ); > ??? } > > ??? /* > ??? * destroy KSP > ??? */ > ??? VecDestroy ( &b ); > ??? VecDestroy ( &x ); > ??? MatDestroy ( &A ); > ??? KSPDestroy(&solver); > > ??? /* > ??? * finalize PETSc > ??? */ > ??? PetscFinalize(); > > ??? return 0; > > I am running this with one MPI process, and the output seems to indicate that the A matrix is set correctly: > > Mat Object: 1 MPI processes >? type: seqaij > row 0: (0, 1.)? (1, 0.)? (2, 0.)? (3, 1.)? (4, 1.)? (5, 1.) > row 1: (0, 0.)? (1, 1.)? (2, 1.)? (3, 1.)? (4, 0.)? (5, 1.) > row 2: (0, 1.)? (1, 0.)? (2, 0.)? (3, 0.)? (4, 1.)? (5, 1.) > row 3: (0, 0.)? (1, 0.)? (2, -1.)? (3, 0.)? (4, 1.)? (5, 1.) > row 4: (0, 1.)? (1, 1.)? (2, 0.)? (3, 1.)? (4, 1.)? (5, 1.) > row 5: (0, 0.)? (1, 1.)? (2, 1.)? (3, 1.)? (4, 0.)? (5, 0.) > > > Thanks for any help ! >? > rgds > LX -------------- next part -------------- An HTML attachment was scrubbed... URL: From lixin_chu at yahoo.com Sat Feb 18 16:58:20 2017 From: lixin_chu at yahoo.com (lixin chu) Date: Sat, 18 Feb 2017 22:58:20 +0000 (UTC) Subject: [petsc-users] Newbie question : sequential vs. parallel and matrix creation References: <1784911422.373395.1487458700722.ref@mail.yahoo.com> Message-ID: <1784911422.373395.1487458700722@mail.yahoo.com> Hello, Some newbie questions I have wrt matrix creation, thank you for any help: 1. Is it correct to say that a sequential matrix is created in one process (for example, the root process), and then distributed to all processes with MatAssemblyBegin and MatAssemblyEnd ? Or sequential matrix only works with running with 1 MPI process only? 2. For a parallel matrix creation, each process will set its values, so I need to provide the data for each process on all the machine ? 3.?MatSetValues(Mat mat,PetscInt m,const PetscInt idxm[],PetscInt n,const PetscInt idxn[],const PetscScalar v[],InsertMode addv) According to the manual page #59 : This routine inserts or adds a logically dense subblock of dimension m*n into the matrix ... ? ? I am not sure if extracting the non zero elements and forming a 'dense block' of data from a large sparse matrix is efficient. My original matrix data is column major. I am thinking of creating and loading the matrix in a column by column way, with n = 1 and using MAT_ROW_ORIENTED = FALSE. Is it efficient ?? ? I think I need to pre-allocate memory, but the API for parallel matrix?MatMPIAIJSetPreallocation () requires to have the none zero info for DIAGONAL portion and NON-DIAGONAL portion separately.? This seems to add more work when converting my sparse matrix to PETSc format...?4. Ideally, I would like to load the matrix data in the main process only, then distribute to all other processes. What is the best way to do this ?? ?? 5. MatView and MatLoad? ? MatView seems to create one file with data for all processes (I only tested with one machine) :? ? ????Vec Object: 4 MPI processes? ? ? ? type: mpi????????Process [0]????????0.????????Process [1]????????1.????????Process [2]????????2.????????Process [3]????????3. ? ? So do I have to manually distribute this file to all machines ? Many thanks again ! ? ?? ??rgds lixin -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Feb 18 17:46:49 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 18 Feb 2017 17:46:49 -0600 Subject: [petsc-users] Newbie question : sequential vs. parallel and matrix creation In-Reply-To: <1784911422.373395.1487458700722@mail.yahoo.com> References: <1784911422.373395.1487458700722.ref@mail.yahoo.com> <1784911422.373395.1487458700722@mail.yahoo.com> Message-ID: > On Feb 18, 2017, at 4:58 PM, lixin chu wrote: > > Hello, > > Some newbie questions I have wrt matrix creation, thank you for any help: > > 1. Is it correct to say that a sequential matrix is created in one process (for example, the root process), and then distributed to all processes with MatAssemblyBegin and MatAssemblyEnd ? Or sequential matrix only works with running with 1 MPI process only? No, sequential matrix lives on any single process. Process 0 can have a sequential matrix, Process 1 can have a different sequential matrix etc. > > 2. For a parallel matrix creation, each process will set its values, so I need to provide the data for each process on all the machine ? For a parallel matrix any process in that matrix's communication can set any values into the matrix. For good performance each process should set mostly the values owned by that process. > > 3. MatSetValues(Mat mat,PetscInt m,const PetscInt idxm[],PetscInt n,const PetscInt idxn[],const PetscScalar v[],InsertMode addv) > According to the manual page #59 : This routine inserts or adds a logically dense subblock of dimension m*n into the matrix ... > > I am not sure if extracting the non zero elements and forming a 'dense block' of data from a large sparse matrix is efficient. My original matrix data is column major. I am thinking of creating and loading the matrix in a column by column way, with n = 1 and using MAT_ROW_ORIENTED = FALSE. Is it efficient ? I am guessing you are currently creating a sparse matrix, using some sparse matrix data format (compressed sparse column) on one process and you are wanting to know how to reuse this sparse matrix data format within PETSc? The answer is you don't want to do that? You want to throw away your old sparse matrix data format and just keep the code that generates your sparse matrix entries and call MatSetValues() directly to put these entries into a PETSc Mat. What type of discretization are you using? Finite element, finite difference? Something else? Many PETSc examples show you how to use MatSetValues(). You don't call MatSetValues() once with the entire matrix, you call it with small numbers of matrix entries that are natural for your discretization; for example for finite elements you call it one element at a time, for finite difference you usually call it for one one row of the matrix at a time. > I think I need to pre-allocate memory, but the API for parallel matrix MatMPIAIJSetPreallocation () requires to have the none zero info for DIAGONAL portion and NON-DIAGONAL portion separately. This seems to add more work when converting my sparse matrix to PETSc format... Yes there is some work in setting preallocation. > > 4. Ideally, I would like to load the matrix data in the main process only, then distribute to all other processes. What is the best way to do this ? MatLoad() is exactly > > > 5. MatView and MatLoad > MatView seems to create one file with data for all processes (I only tested with one machine) : > Vec Object: 4 MPI processes > type: mpi > Process [0] > 0. > Process [1] > 1. > Process [2] > 2. > Process [3] > 3. > > So do I have to manually distribute this file to all machines ? With MatLoad() only the zeroth process in the MPI_Comm needs access to the file. > > > Many thanks again ! > > > rgds > lixin From lixin_chu at yahoo.com Sat Feb 18 18:55:26 2017 From: lixin_chu at yahoo.com (lixin chu) Date: Sun, 19 Feb 2017 00:55:26 +0000 (UTC) Subject: [petsc-users] Newbie question : sequential vs. parallel and matrix creation References: <1036061598.384658.1487465727006.ref@mail.yahoo.com> Message-ID: <1036061598.384658.1487465727006@mail.yahoo.com> Thank you again for your super responsive reply, Barry ! The matrix data I have is a file from NASTRAN, with DMAP command, column major, one single file. So I think MatView/MatLoad will be the best for my case. I need to develop a program for this conversion. The program will create a parallel matrix. But since the data is only available on root process, it might take a while to populate all the values in the root process and distribute to all brfore MatView can create the file.? I donot think I can create a sequential matrix (MatView will hence create a file for one process ?) and MatLoad into a parallel matrix ? Really appreciate your helpLX Sent from Yahoo Mail on Android On Sun, 19 Feb 2017 at 7:46, Barry Smith wrote: > On Feb 18, 2017, at 4:58 PM, lixin chu wrote: > > Hello, > > Some newbie questions I have wrt matrix creation, thank you for any help: > > 1. Is it correct to say that a sequential matrix is created in one process (for example, the root process), and then distributed to all processes with MatAssemblyBegin and MatAssemblyEnd ? Or sequential matrix only works with running with 1 MPI process only? ? No, sequential matrix lives on any single process. Process 0 can have a sequential matrix, Process 1 can have a different sequential matrix etc. > > 2. For a parallel matrix creation, each process will set its values, so I need to provide the data for each process on all the machine ? ? For a parallel matrix any process in that matrix's communication can set any values into the matrix. For good performance each process should set mostly the values owned by that process. > > 3. MatSetValues(Mat mat,PetscInt m,const PetscInt idxm[],PetscInt n,const PetscInt idxn[],const PetscScalar v[],InsertMode addv) >? According to the manual page #59 : This routine inserts or adds a logically dense subblock of dimension m*n into the matrix ... > >? ? I am not sure if extracting the non zero elements and forming a 'dense block' of data from a large sparse matrix is efficient. My original matrix data is column major. I am thinking of creating and loading the matrix in a column by column way, with n = 1 and using MAT_ROW_ORIENTED = FALSE. Is it efficient ? ? I am guessing you are currently creating a sparse matrix, using some sparse matrix data format (compressed sparse column) on one process and you are wanting to know how to reuse this sparse matrix data format within PETSc? ? ? The answer is you don't want to do that? You want to throw away your old sparse matrix data format and just keep the code that generates your sparse matrix entries and call MatSetValues() directly to put these entries into a PETSc Mat. What type of discretization are you using? Finite element, finite difference? Something else? Many PETSc examples show you how to use MatSetValues(). You don't call MatSetValues() once with the entire matrix, you call it with small numbers of matrix entries that are natural for your discretization; for example for finite elements you call it one element at a time, for finite difference you usually call it for one one row of the matrix at a time. >? ? I think I need to pre-allocate memory, but the API for parallel matrix MatMPIAIJSetPreallocation () requires to have the none zero info for DIAGONAL portion and NON-DIAGONAL portion separately.? This seems to add more work when converting my sparse matrix to PETSc format... ? Yes there is some work in setting preallocation. >? > 4. Ideally, I would like to load the matrix data in the main process only, then distribute to all other processes. What is the best way to do this ? ? MatLoad() is exactly >? ? > > 5. MatView and MatLoad >? ? MatView seems to create one file with data for all processes (I only tested with one machine) : >? ? ? ? Vec Object: 4 MPI processes >? ? ? ? type: mpi >? ? ? ? Process [0] >? ? ? ? 0. >? ? ? ? Process [1] >? ? ? ? 1. >? ? ? ? Process [2] >? ? ? ? 2. >? ? ? ? Process [3] >? ? ? ? 3. > >? ? So do I have to manually distribute this file to all machines ? With MatLoad() only the zeroth process in the MPI_Comm needs access to the file. > > > Many thanks again !? ? >? ? >? > rgds > lixin -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Feb 18 19:04:34 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 18 Feb 2017 19:04:34 -0600 Subject: [petsc-users] Newbie question : sequential vs. parallel and matrix creation In-Reply-To: <1036061598.384658.1487465727006@mail.yahoo.com> References: <1036061598.384658.1487465727006.ref@mail.yahoo.com> <1036061598.384658.1487465727006@mail.yahoo.com> Message-ID: > On Feb 18, 2017, at 6:55 PM, lixin chu wrote: > > Thank you again for your super responsive reply, Barry ! > > The matrix data I have is a file from NASTRAN, with DMAP command, column major, one single file. Write a __sequential__ program that reads in the file and then calls MatView() to a binary file and then in the "real" parallel program (a completely different code then the one used to convert from NASTRAN format to PETSc binary format) use MatLoad() to load the binary file in parallel. Never, never, never write a parallel program that tries to read in a NASTRAN data file! See src/mat/examples/tests/ex32.c or ex78.c for examples of programs that read in a sequential matrix and store it with MatView. Yes you can MatView() a sequential matrix and then read it in in parallel with MatLoad(). Barry > > So I think MatView/MatLoad will be the best for my case. I need to develop a program for this conversion. The program will create a parallel matrix. But since the data is only available on root process, it might take a while to populate all the values in the root process and distribute to all brfore MatView can create the file. > > I donot think I can create a sequential matrix (MatView will hence create a file for one process ?) and MatLoad into a parallel matrix ? > > Really appreciate your help > LX > > Sent from Yahoo Mail on Android > > On Sun, 19 Feb 2017 at 7:46, Barry Smith > wrote: > > > On Feb 18, 2017, at 4:58 PM, lixin chu wrote: > > > > Hello, > > > > Some newbie questions I have wrt matrix creation, thank you for any help: > > > > 1. Is it correct to say that a sequential matrix is created in one process (for example, the root process), and then distributed to all processes with MatAssemblyBegin and MatAssemblyEnd ? Or sequential matrix only works with running with 1 MPI process only? > > No, sequential matrix lives on any single process. Process 0 can have a sequential matrix, Process 1 can have a different sequential matrix etc. > > > > > 2. For a parallel matrix creation, each process will set its values, so I need to provide the data for each process on all the machine ? > > For a parallel matrix any process in that matrix's communication can set any values into the matrix. For good performance each process should set mostly the values owned by that process. > > > > > 3. MatSetValues(Mat mat,PetscInt m,const PetscInt idxm[],PetscInt n,const PetscInt idxn[],const PetscScalar v[],InsertMode addv) > > According to the manual page #59 : This routine inserts or adds a logically dense subblock of dimension m*n into the matrix ... > > > > I am not sure if extracting the non zero elements and forming a 'dense block' of data from a large sparse matrix is efficient. My original matrix data is column major. I am thinking of creating and loading the matrix in a column by column way, with n = 1 and using MAT_ROW_ORIENTED = FALSE. Is it efficient ? > > I am guessing you are currently creating a sparse matrix, using some sparse matrix data format (compressed sparse column) on one process and you are wanting to know how to reuse this sparse matrix data format within PETSc? > > The answer is you don't want to do that? You want to throw away your old sparse matrix data format and just keep the code that generates your sparse matrix entries and call MatSetValues() directly to put these entries into a PETSc Mat. What type of discretization are you using? Finite element, finite difference? Something else? Many PETSc examples show you how to use MatSetValues(). You don't call MatSetValues() once with the entire matrix, you call it with small numbers of matrix entries that are natural for your discretization; for example for finite elements you call it one element at a time, for finite difference you usually call it for one one row of the matrix at a time. > > > I think I need to pre-allocate memory, but the API for parallel matrix MatMPIAIJSetPreallocation () requires to have the none zero info for DIAGONAL portion and NON-DIAGONAL portion separately. This seems to add more work when converting my sparse matrix to PETSc format... > > Yes there is some work in setting preallocation. > > > > 4. Ideally, I would like to load the matrix data in the main process only, then distribute to all other processes. What is the best way to do this ? > > MatLoad() is exactly > > > > > > 5. MatView and MatLoad > > MatView seems to create one file with data for all processes (I only tested with one machine) : > > Vec Object: 4 MPI processes > > type: mpi > > Process [0] > > 0. > > Process [1] > > 1. > > Process [2] > > 2. > > Process [3] > > 3. > > > > So do I have to manually distribute this file to all machines ? > > With MatLoad() only the zeroth process in the MPI_Comm needs access to the file. > > > > > > > > Many thanks again ! > > > > > > rgds > > lixin From lixin_chu at yahoo.com Sat Feb 18 19:06:40 2017 From: lixin_chu at yahoo.com (lixin chu) Date: Sun, 19 Feb 2017 01:06:40 +0000 (UTC) Subject: [petsc-users] Newbie question : sequential vs. parallel and matrix creation In-Reply-To: References: <1036061598.384658.1487465727006.ref@mail.yahoo.com> <1036061598.384658.1487465727006@mail.yahoo.com> Message-ID: <2054752955.362194.1487466400841@mail.yahoo.com> Super, Barry ! Sent from Yahoo Mail on Android On Sun, 19 Feb 2017 at 9:04, Barry Smith wrote: > On Feb 18, 2017, at 6:55 PM, lixin chu wrote: > > Thank you again for your super responsive reply, Barry ! > > The matrix data I have is a file from NASTRAN, with DMAP command, column major, one single file. ? Write a __sequential__ program that reads in the file and then calls MatView() to a binary file and then in the "real" parallel program (a completely different code then the one used to convert from NASTRAN format to PETSc binary format) use MatLoad() to load the binary file in parallel.? Never, never, never write a parallel program that tries to read in a NASTRAN data file! ? See src/mat/examples/tests/ex32.c or ex78.c for examples of programs that read in a sequential matrix and store it with MatView. ? Yes you can MatView() a sequential matrix and then read it in in parallel with MatLoad(). ? Barry > > So I think MatView/MatLoad will be the best for my case. I need to develop a program for this conversion. The program will create a parallel matrix. But since the data is only available on root process, it might take a while to populate all the values in the root process and distribute to all brfore MatView can create the file. > > I donot think I can create a sequential matrix (MatView will hence create a file for one process ?) and MatLoad into a parallel matrix ? > > Really appreciate your help > LX > > Sent from Yahoo Mail on Android > > On Sun, 19 Feb 2017 at 7:46, Barry Smith > wrote: > > > On Feb 18, 2017, at 4:58 PM, lixin chu wrote: > > > > Hello, > > > > Some newbie questions I have wrt matrix creation, thank you for any help: > > > > 1. Is it correct to say that a sequential matrix is created in one process (for example, the root process), and then distributed to all processes with MatAssemblyBegin and MatAssemblyEnd ? Or sequential matrix only works with running with 1 MPI process only? > >? No, sequential matrix lives on any single process. Process 0 can have a sequential matrix, Process 1 can have a different sequential matrix etc. > > > > > 2. For a parallel matrix creation, each process will set its values, so I need to provide the data for each process on all the machine ? > >? For a parallel matrix any process in that matrix's communication can set any values into the matrix. For good performance each process should set mostly the values owned by that process. > > > > > 3. MatSetValues(Mat mat,PetscInt m,const PetscInt idxm[],PetscInt n,const PetscInt idxn[],const PetscScalar v[],InsertMode addv) > >? According to the manual page #59 : This routine inserts or adds a logically dense subblock of dimension m*n into the matrix ... > > > >? ? I am not sure if extracting the non zero elements and forming a 'dense block' of data from a large sparse matrix is efficient. My original matrix data is column major. I am thinking of creating and loading the matrix in a column by column way, with n = 1 and using MAT_ROW_ORIENTED = FALSE. Is it efficient ? > >? I am guessing you are currently creating a sparse matrix, using some sparse matrix data format (compressed sparse column) on one process and you are wanting to know how to reuse this sparse matrix data format within PETSc? > >? ? The answer is you don't want to do that? You want to throw away your old sparse matrix data format and just keep the code that generates your sparse matrix entries and call MatSetValues() directly to put these entries into a PETSc Mat. What type of discretization are you using? Finite element, finite difference? Something else? Many PETSc examples show you how to use MatSetValues(). You don't call MatSetValues() once with the entire matrix, you call it with small numbers of matrix entries that are natural for your discretization; for example for finite elements you call it one element at a time, for finite difference you usually call it for one one row of the matrix at a time. > > >? ? I think I need to pre-allocate memory, but the API for parallel matrix MatMPIAIJSetPreallocation () requires to have the none zero info for DIAGONAL portion and NON-DIAGONAL portion separately.? This seems to add more work when converting my sparse matrix to PETSc format... > >? Yes there is some work in setting preallocation. > >? > > 4. Ideally, I would like to load the matrix data in the main process only, then distribute to all other processes. What is the best way to do this ? > >? MatLoad() is exactly > >? ? > > > > 5. MatView and MatLoad > >? ? MatView seems to create one file with data for all processes (I only tested with one machine) : > >? ? ? ? Vec Object: 4 MPI processes > >? ? ? ? type: mpi > >? ? ? ? Process [0] > >? ? ? ? 0. > >? ? ? ? Process [1] > >? ? ? ? 1. > >? ? ? ? Process [2] > >? ? ? ? 2. > >? ? ? ? Process [3] > >? ? ? ? 3. > > > >? ? So do I have to manually distribute this file to all machines ? > > With MatLoad() only the zeroth process in the MPI_Comm needs access to the file. > > > > > > > > Many thanks again !? ? > >? ? > >? > > rgds > > lixin -------------- next part -------------- An HTML attachment was scrubbed... URL: From jychang48 at gmail.com Sun Feb 19 03:05:33 2017 From: jychang48 at gmail.com (Justin Chang) Date: Sun, 19 Feb 2017 03:05:33 -0600 Subject: [petsc-users] GAMG huge hash being requested Message-ID: Hi all, So I am attempting to employ the DG1 finite element method on the poisson equation using GAMG. When I attempt to solve a problem with roughly 4 million DOFs across 20 cores, i get this error: Traceback (most recent call last): File "pFiredrake.py", line 109, in solve(a==L,solution,options_prefix='fe_',solver_parameters=solver_params) File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 122, in solve _solve_varproblem(*args, **kwargs) File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 152, in _solve_varproblem solver.solve() File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", line 220, in solve self.snes.solve(None, v) File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:172359) petsc4py.PETSc.Error: error code 63 [ 6] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes/interface/snes.c [ 6] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes/impls/ksponly/ksponly.c [ 6] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [ 6] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [ 6] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/pc/interface/precon.c [ 6] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/gamg.c [ 6] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/agg.c [ 6] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/interface/matrix.c [ 6] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [ 6] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [ 6] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [ 6] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [ 6] Argument out of range [ 6] A really huge hash is being requested.. cannot process: 4096000 -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 6 in communicator MPI_COMM_WORLD with errorcode 1. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- Traceback (most recent call last): File "pFiredrake.py", line 109, in Traceback (most recent call last): Traceback (most recent call last): File "pFiredrake.py", line 109, in File "pFiredrake.py", line 109, in solve(a==L,solution,options_prefix='fe_',solver_parameters=solver_params) File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 122, in solve solve(a==L,solution,options_prefix='fe_',solver_parameters=solver_params) Traceback (most recent call last): File "pFiredrake.py", line 109, in Traceback (most recent call last): File "pFiredrake.py", line 109, in _solve_varproblem(*args, **kwargs) solve(a==L,solution,options_prefix='fe_',solver_parameters=solver_params) File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 152, in _solve_varproblem Traceback (most recent call last): File "pFiredrake.py", line 109, in File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 122, in solve Traceback (most recent call last): File "pFiredrake.py", line 109, in Traceback (most recent call last): File "pFiredrake.py", line 109, in solve(a==L,solution,options_prefix='fe_',solver_parameters=solver_params) File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 122, in solve solve(a==L,solution,options_prefix='fe_',solver_parameters=solver_params) File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 122, in solve File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 122, in solve solve(a==L,solution,options_prefix='fe_',solver_parameters=solver_params) File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 122, in solve _solve_varproblem(*args, **kwargs) File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 152, in _solve_varproblem _solve_varproblem(*args, **kwargs) File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 152, in _solve_varproblem _solve_varproblem(*args, **kwargs) File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 152, in _solve_varproblem _solve_varproblem(*args, **kwargs) File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 152, in _solve_varproblem solver.solve() File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", line 220, in solve _solve_varproblem(*args, **kwargs) File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 152, in _solve_varproblem solver.solve() File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", line 220, in solve solver.solve() File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", line 220, in solve solver.solve() File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", line 220, in solve solver.solve() solver.solve() File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", line 220, in solve self.snes.solve(None, v) File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:172359) self.snes.solve(None, v) File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:172359) self.snes.solve(None, v) File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:172359) self.snes.solve(None, v) File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:172359) self.snes.solve(None, v) Traceback (most recent call last): File "pFiredrake.py", line 109, in File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:172359) petsc4py.PETSc.Error solve(a==L,solution,options_prefix='fe_',solver_parameters=solver_params) petsc4py.PETSc.Errorpetsc4py.PETSc.Errorpetsc4py.PETSc.Error: solve(a==L,solution,options_prefix='fe_',solver_parameters=solver_params) : error code 63 [ 3] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes/interface/snes.c [ 3] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes/impls/ksponly/ksponly.c [ 3] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [ 3] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [ 3] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/pc/interface/precon.c [ 3] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/gamg.c [ 3] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/agg.c [ 3] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/interface/matrix.c [ 3] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [ 3] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [ 3] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [ 3] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [ 3] Argument out of range [ 3] A really huge hash is being requested.. cannot process: 4096000 File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", line 220, in solve : error code 63 [ 1] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes/interface/snes.c [ 1] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes/impls/ksponly/ksponly.c [ 1] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [ 1] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [ 1] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/pc/interface/precon.c [ 1] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/gamg.c [ 1] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/agg.c [ 1] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/interface/matrix.c [ 1] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [ 1] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [ 1] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [ 1] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [ 1] Argument out of range [ 1] A really huge hash is being requested.. cannot process: 4096000 : error code 63 [17] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes/interface/snes.c [17] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes/impls/ksponly/ksponly.c [17] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [17] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [17] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/pc/interface/precon.c [17] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/gamg.c [17] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/agg.c [17] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/interface/matrix.c [17] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [17] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [17] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [17] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [17] Argument out of range [17] A really huge hash is being requested.. cannot process: 4096000 solve(a==L,solution,options_prefix='fe_',solver_parameters=solver_params) error code 63 [ 5] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes/interface/snes.c [ 5] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes/impls/ksponly/ksponly.c [ 5] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [ 5] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [ 5] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/pc/interface/precon.c [ 5] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/gamg.c [ 5] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/agg.c [ 5] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/interface/matrix.c [ 5] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [ 5] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [ 5] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [ 5] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [ 5] Argument out of range [ 5] A really huge hash is being requested.. cannot process: 4096000 Traceback (most recent call last): File "pFiredrake.py", line 109, in File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 122, in solve File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 122, in solve _solve_varproblem(*args, **kwargs) Traceback (most recent call last): File "pFiredrake.py", line 109, in File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 122, in solve petsc4py.PETSc.Error: error code 63 [ 9] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes/interface/snes.c [ 9] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes/impls/ksponly/ksponly.c [ 9] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [ 9] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [ 9] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/pc/interface/precon.c [ 9] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/gamg.c [ 9] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/agg.c [ 9] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/interface/matrix.c [ 9] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [ 9] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [ 9] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [ 9] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [ 9] Argument out of range [ 9] A really huge hash is being requested.. cannot process: 4096000 self.snes.solve(None, v) Traceback (most recent call last): File "pFiredrake.py", line 109, in Traceback (most recent call last): File "pFiredrake.py", line 109, in solve(a==L,solution,options_prefix='fe_',solver_parameters=solver_params) _solve_varproblem(*args, **kwargs) Traceback (most recent call last): File "pFiredrake.py", line 109, in solve(a==L,solution,options_prefix='fe_',solver_parameters=solver_params) File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:172359) petsc4py.PETSc.Error: error code 63 [ 4] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes/interface/snes.c [ 4] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes/impls/ksponly/ksponly.c [ 4] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [ 4] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [ 4] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/pc/interface/precon.c [ 4] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/gamg.c [ 4] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/agg.c [ 4] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/interface/matrix.c [ 4] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [ 4] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [ 4] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [ 4] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [ 4] Argument out of range [ 4] A really huge hash is being requested.. cannot process: 4096000 File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 152, in _solve_varproblem File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 122, in solve Traceback (most recent call last): File "pFiredrake.py", line 109, in solve(a==L,solution,options_prefix='fe_',solver_parameters=solver_params) File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 122, in solve File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 152, in _solve_varproblem solve(a==L,solution,options_prefix='fe_',solver_parameters=solver_params) _solve_varproblem(*args, **kwargs) File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 122, in solve solver.solve() _solve_varproblem(*args, **kwargs) solve(a==L,solution,options_prefix='fe_',solver_parameters=solver_params) File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 122, in solve solve(a==L,solution,options_prefix='fe_',solver_parameters=solver_params) File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 122, in solve File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 152, in _solve_varproblem Traceback (most recent call last): File "pFiredrake.py", line 109, in _solve_varproblem(*args, **kwargs) solver.solve() File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", line 220, in solve File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", line 220, in solve Traceback (most recent call last): File "pFiredrake.py", line 109, in File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 152, in _solve_varproblem _solve_varproblem(*args, **kwargs) File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 152, in _solve_varproblem _solve_varproblem(*args, **kwargs) File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 122, in solve _solve_varproblem(*args, **kwargs) File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 152, in _solve_varproblem solver.solve() solve(a==L,solution,options_prefix='fe_',solver_parameters=solver_params) Traceback (most recent call last): File "pFiredrake.py", line 109, in File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 152, in _solve_varproblem self.snes.solve(None, v) File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:172359) self.snes.solve(None, v) solve(a==L,solution,options_prefix='fe_',solver_parameters=solver_params) petsc4py.PETSc.Error: error code 63 [13] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes/interface/snes.c [13] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes/impls/ksponly/ksponly.c [13] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [13] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [13] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/pc/interface/precon.c [13] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/gamg.c [13] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/agg.c [13] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/interface/matrix.c [13] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [13] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [13] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [13] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [13] Argument out of range [13] A really huge hash is being requested.. cannot process: 4096000 solver.solve() Traceback (most recent call last): File "pFiredrake.py", line 109, in File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 152, in _solve_varproblem File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", line 220, in solve _solve_varproblem(*args, **kwargs) File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 152, in _solve_varproblem solver.solve() File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 122, in solve solve(a==L,solution,options_prefix='fe_',solver_parameters=solver_params) solver.solve() File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:172359) petsc4py.PETSc.Error File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 122, in solve : error code 63 [11] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes/interface/snes.c [11] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes/impls/ksponly/ksponly.c [11] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [11] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [11] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/pc/interface/precon.c [11] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/gamg.c [11] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/agg.c [11] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/interface/matrix.c [11] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [11] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [11] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [11] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [11] Argument out of range [11] A really huge hash is being requested.. cannot process: 4096000 File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", line 220, in solve solve(a==L,solution,options_prefix='fe_',solver_parameters=solver_params) solver.solve() File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", line 220, in solve solver.solve() File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", line 220, in solve solver.solve() self.snes.solve(None, v) File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", line 220, in solve _solve_varproblem(*args, **kwargs) File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 122, in solve File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", line 220, in solve _solve_varproblem(*args, **kwargs) self.snes.solve(None, v) File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 122, in solve self.snes.solve(None, v) File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:172359) self.snes.solve(None, v) File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", line 220, in solve File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:172359) petsc4py.PETSc.Error: error code 63 [14] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes/interface/snes.c [14] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes/impls/ksponly/ksponly.c [14] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [14] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [14] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/pc/interface/precon.c [14] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/gamg.c [14] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/agg.c [14] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/interface/matrix.c [14] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [14] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [14] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [14] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [14] Argument out of range [14] A really huge hash is being requested.. cannot process: 4096000 File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 152, in _solve_varproblem self.snes.solve(None, v) petsc4py.PETSc.Error: error code 63 [19] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes/interface/snes.c [19] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes/impls/ksponly/ksponly.c [19] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [19] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [19] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/pc/interface/precon.c [19] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/gamg.c [19] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/agg.c [19] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/interface/matrix.c [19] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [19] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [19] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [19] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [19] Argument out of range [19] A really huge hash is being requested.. cannot process: 4096000 _solve_varproblem(*args, **kwargs) self.snes.solve(None, v) File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 152, in _solve_varproblem File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:172359) _solve_varproblem(*args, **kwargs) File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:172359) petsc4py.PETSc.Error: error code 63 [15] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes/interface/snes.c [15] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes/impls/ksponly/ksponly.c [15] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [15] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [15] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/pc/interface/precon.c [15] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/gamg.c [15] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/agg.c [15] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/interface/matrix.c [15] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [15] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [15] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [15] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [15] Argument out of range [15] A really huge hash is being requested.. cannot process: 4096000 self.snes.solve(None, v) petsc4py.PETSc.Error: error code 63 [ 7] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes/interface/snes.c [ 7] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes/impls/ksponly/ksponly.c [ 7] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [ 7] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [ 7] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/pc/interface/precon.c [ 7] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/gamg.c [ 7] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/agg.c [ 7] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/interface/matrix.c [ 7] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [ 7] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [ 7] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [ 7] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [ 7] Argument out of range [ 7] A really huge hash is being requested.. cannot process: 4096000 solver.solve() File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:172359) petsc4py.PETSc.Error File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 152, in _solve_varproblem : error code 63 [12] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes/interface/snes.c [12] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes/impls/ksponly/ksponly.c [12] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [12] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [12] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/pc/interface/precon.c [12] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/gamg.c [12] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/agg.c [12] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/interface/matrix.c [12] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [12] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [12] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [12] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [12] Argument out of range [12] A really huge hash is being requested.. cannot process: 4096000 File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:172359) solver.solve() File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", line 152, in _solve_varproblem petsc4py.PETSc.Error: File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:172359) error code 63 [ 2] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes/interface/snes.c [ 2] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes/impls/ksponly/ksponly.c [ 2] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [ 2] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [ 2] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/pc/interface/precon.c [ 2] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/gamg.c [ 2] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/agg.c [ 2] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/interface/matrix.c [ 2] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [ 2] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [ 2] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [ 2] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [ 2] Argument out of range [ 2] A really huge hash is being requested.. cannot process: 4096000 petsc4py.PETSc.Error File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", line 220, in solve : error code 63 [ 8] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes/interface/snes.c [ 8] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes/impls/ksponly/ksponly.c [ 8] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [ 8] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [ 8] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/pc/interface/precon.c [ 8] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/gamg.c [ 8] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/agg.c [ 8] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/interface/matrix.c [ 8] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [ 8] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [ 8] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [ 8] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [ 8] Argument out of range [ 8] A really huge hash is being requested.. cannot process: 4096000 File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", line 220, in solve solver.solve() solver.solve() self.snes.solve(None, v) File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", line 220, in solve File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", line 220, in solve self.snes.solve(None, v) File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:172359) File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:172359) petsc4py.PETSc.Error self.snes.solve(None, v) File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:172359) : error code 63 [18] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes/interface/snes.c [18] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes/impls/ksponly/ksponly.c [18] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [18] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [18] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/pc/interface/precon.c [18] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/gamg.c [18] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/agg.c [18] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/interface/matrix.c [18] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [18] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [18] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [18] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [18] Argument out of range [18] A really huge hash is being requested.. cannot process: 4096000 self.snes.solve(None, v) File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:172359) petsc4py.PETSc.Errorpetsc4py.PETSc.Error: error code 63 [ 0] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes/interface/snes.c [ 0] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes/impls/ksponly/ksponly.c [ 0] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [ 0] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [ 0] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/pc/interface/precon.c [ 0] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/gamg.c [ 0] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/agg.c [ 0] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/interface/matrix.c [ 0] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [ 0] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [ 0] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [ 0] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [ 0] Argument out of range [ 0] A really huge hash is being requested.. cannot process: 4096000 : error code 63 [16] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes/interface/snes.c [16] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes/impls/ksponly/ksponly.c [16] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [16] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [16] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/pc/interface/precon.c [16] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/gamg.c [16] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/agg.c [16] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/interface/matrix.c [16] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [16] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [16] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [16] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [16] Argument out of range [16] A really huge hash is being requested.. cannot process: 4096000 petsc4py.PETSc.Error: error code 63 [10] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes/interface/snes.c [10] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes/impls/ksponly/ksponly.c [10] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [10] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ksp/interface/itfunc.c [10] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/pc/interface/precon.c [10] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/gamg.c [10] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/pc/impls/gamg/agg.c [10] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/interface/matrix.c [10] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [10] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c [10] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [10] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/utils/ctable.c [10] Argument out of range [10] A really huge hash is being requested.. cannot process: 4096000 [compute-0-0.local:31182] 19 more processes have sent help message help-mpi-api.txt / mpi-abort [compute-0-0.local:31182] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages However, if I use either ML or HYPRE i do not. When I downscale the problem to roughly 1.7 million dofs, I get no such error with GAMG, but both ML and HYPRE are still faster. Something tells me I need to tune the GAMG parameters. Here's some -GAMG related info that may help: [0] PCSetUp_GAMG(): level 0) N=4096000, n data rows=1, n data cols=1, nnz/row (ave)=55, np=20 [0] PCGAMGFilterGraph(): 78.5842% nnz after filtering, with threshold 0., 55.8 nnz ave. (N=4096000) [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square And if I run with -pc_gamg_threshold 0.08: [0] PCSetUp_GAMG(): level 0) N=4096000, n data rows=1, n data cols=1, nnz/row (ave)=55, np=20 [0] PCGAMGFilterGraph(): 28.5842% nnz after filtering, with threshold 0.08, 55.8 nnz ave. (N=4096000) [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square Any idea what to do here? Thanks, Justin -------------- next part -------------- An HTML attachment was scrubbed... URL: From cmpierce at WPI.EDU Sun Feb 19 04:00:41 2017 From: cmpierce at WPI.EDU (Christopher Pierce) Date: Sun, 19 Feb 2017 05:00:41 -0500 Subject: [petsc-users] Krylov-Schur Tolerance In-Reply-To: <78DC3258-D0BB-49A9-9246-D15F6E82A9D0@dsic.upv.es> References: <4691ae05-be89-5bdb-33ac-2fd5cef4c978@wpi.edu> <78DC3258-D0BB-49A9-9246-D15F6E82A9D0@dsic.upv.es> Message-ID: Thanks, Those changes did improve the tolerances of the solutions. However, I still have the same problem. For certain matrices the error is up to 10^4 times as large as the requested tolerances and when using true residual the solver gets stuck on a certain residual norm the solutions and does not converge. I dumped the settings that I used which I'm attaching here. Chris On 02/17/17 04:42, Jose E. Roman wrote: > For computing eigenvalues with smallest real part of generalized problems Ax=lambda Bx, it may be better to use a target value (instead of -eps_smallest_real). For instance, if you know that all eigenvalues are positive, use -eps_target 0 -eps_target_magnitude > > What linear solvers are you using? In the default setting, the coefficient matrix for linear solves will be B, but with target=sigma the coefficient matrix will be A-sigma*B; this may make a difference. Also, in any case, if experiencing convergence problems I would suggest using MUMPS (see section 3.4.1 of SLEPc's users manual). > > Jose > > > >> El 17 feb 2017, a las 10:25, Christopher Pierce escribi?: >> >> Hello All, >> >> I'm trying to use the SLEPc Krylov-Schur implementation to solve a >> general eigenvalue problem. I have a monitor on my solver and the >> solutions appear to converge correctly when using the approximation for >> the residual norm in the algorithm. However, when the solutions are >> displayed and I retrieve the actual residual norm it is very large and >> increases with the size of the matrices I am working with. In some >> cases it may be 10^17 times as large as the approximate norm. I also >> don't get the eigenvalues I would expect for the system I am studying. >> >> When I turn on the option "true residual" the solver fails to converge. >> The residual norm shrinks to some limit (~10^-3) and then sits there for >> the remaining iterations. As a note, I am solving for the eigenvalues >> with the smallest real part. I have also tried the RQCG solver on the >> same problems and appear to get the correct results using it, but I'm >> looking to use the better scaling of the Krylov-Schur solver. >> >> Does anyone know what could be causing this behavior? >> >> Thanks, >> >> Chris Pierce >> WPI Center for Computational Nanoscience >> >> -------------- next part -------------- EPS Object: 4 MPI processes type: krylovschur Krylov-Schur: 50% of basis vectors kept after restart Krylov-Schur: using the locking variant problem type: generalized symmetric eigenvalue problem selected portion of the spectrum: closest to target: 0. (in magnitude) postprocessing eigenvectors with purification number of eigenvalues (nev): 10 number of column vectors (ncv): 25 maximum dimension of projected problem (mpd): 25 maximum number of iterations: 1000 tolerance: 1e-10 convergence test: relative to the eigenvalue BV Object: 4 MPI processes type: svec 26 columns of global length 12513 vector orthogonalization method: classical Gram-Schmidt orthogonalization refinement: if needed (eta: 0.7071) block orthogonalization method: Gram-Schmidt non-standard inner product Mat Object: 4 MPI processes type: mpiaij rows=12513, cols=12513 total: nonzeros=177931, allocated nonzeros=177931 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines doing matmult as a single matrix-matrix product DS Object: 4 MPI processes type: hep solving the problem with: Implicit QR method (_steqr) ST Object: 4 MPI processes type: sinvert shift: 0. number of matrices: 2 all matrices have different nonzero pattern KSP Object: (st_) 4 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-08, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (st_) 4 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: natural factor fill ratio given 0., needed 0. Factored matrix follows: Mat Object: 4 MPI processes type: mpiaij rows=12513, cols=12513 package used to perform factorization: mumps total: nonzeros=4234311, allocated nonzeros=4234311 total number of mallocs used during MatSetValues calls =0 MUMPS run parameters: SYM (matrix type): 0 PAR (host participation): 1 ICNTL(1) (output for error): 6 ICNTL(2) (output of diagnostic msg): 0 ICNTL(3) (output for global info): 0 ICNTL(4) (level of printing): 0 ICNTL(5) (input mat struct): 0 ICNTL(6) (matrix prescaling): 7 ICNTL(7) (sequentia matrix ordering):7 ICNTL(8) (scalling strategy): 77 ICNTL(10) (max num of refinements): 0 ICNTL(11) (error analysis): 0 ICNTL(12) (efficiency control): 1 ICNTL(13) (efficiency control): 0 ICNTL(14) (percentage of estimated workspace increase): 20 ICNTL(18) (input mat struct): 3 ICNTL(19) (Shur complement info): 0 ICNTL(20) (rhs sparse pattern): 0 ICNTL(21) (solution struct): 1 ICNTL(22) (in-core/out-of-core facility): 0 ICNTL(23) (max size of memory can be allocated locally):0 ICNTL(24) (detection of null pivot rows): 0 ICNTL(25) (computation of a null space basis): 0 ICNTL(26) (Schur options for rhs or solution): 0 ICNTL(27) (experimental parameter): -24 ICNTL(28) (use parallel or sequential ordering): 1 ICNTL(29) (parallel ordering): 0 ICNTL(30) (user-specified set of entries in inv(A)): 0 ICNTL(31) (factors is discarded in the solve phase): 0 ICNTL(33) (compute determinant): 0 CNTL(1) (relative pivoting threshold): 0.01 CNTL(2) (stopping criterion of refinement): 1.49012e-08 CNTL(3) (absolute pivoting threshold): 0. CNTL(4) (value of static pivoting): -1. CNTL(5) (fixation for null pivots): 0. RINFO(1) (local estimated flops for the elimination after analysis): [0] 3.42689e+08 [1] 5.94214e+08 [2] 3.8211e+08 [3] 3.4841e+08 RINFO(2) (local estimated flops for the assembly after factorization): [0] 1.5205e+06 [1] 1.4933e+06 [2] 1.4988e+06 [3] 1.56079e+06 RINFO(3) (local estimated flops for the elimination after factorization): [0] 3.42689e+08 [1] 5.94214e+08 [2] 3.8211e+08 [3] 3.4841e+08 INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): [0] 37 [1] 44 [2] 40 [3] 38 INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): [0] 37 [1] 44 [2] 40 [3] 38 INFO(23) (num of pivots eliminated on this processor after factorization): [0] 3977 [1] 2646 [2] 2409 [3] 3481 RINFOG(1) (global estimated flops for the elimination after analysis): 1.66742e+09 RINFOG(2) (global estimated flops for the assembly after factorization): 6.0734e+06 RINFOG(3) (global estimated flops for the elimination after factorization): 1.66742e+09 (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0.,0.)*(2^0) INFOG(3) (estimated real workspace for factors on all processors after analysis): 4234311 INFOG(4) (estimated integer workspace for factors on all processors after analysis): 169823 INFOG(5) (estimated maximum front size in the complete tree): 925 INFOG(6) (number of nodes in the complete tree): 2357 INFOG(7) (ordering option effectively use after analysis): 4 INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 100 INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 4234311 INFOG(10) (total integer space store the matrix factors after factorization): 169823 INFOG(11) (order of largest frontal matrix after factorization): 925 INFOG(12) (number of off-diagonal pivots): 0 INFOG(13) (number of delayed pivots after factorization): 0 INFOG(14) (number of memory compress after factorization): 0 INFOG(15) (number of steps of iterative refinement after solution): 0 INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 44 INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 159 INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 44 INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 159 INFOG(20) (estimated number of entries in the factors): 4234311 INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 38 INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 142 INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 INFOG(28) (after factorization: number of null pivots encountered): 0 INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 4234311 INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 22, 68 INFOG(32) (after analysis: type of analysis done): 1 INFOG(33) (value used for ICNTL(8)): 7 INFOG(34) (exponent of the determinant if determinant is requested): 0 linear system matrix = precond matrix: Mat Object: 4 MPI processes type: mpiaij rows=12513, cols=12513 total: nonzeros=177931, allocated nonzeros=177931 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines From knepley at gmail.com Sun Feb 19 08:56:13 2017 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 19 Feb 2017 08:56:13 -0600 Subject: [petsc-users] GAMG huge hash being requested In-Reply-To: References: Message-ID: Satish fixed this error. I believe the fix is now in master. Thanks, Matt On Sun, Feb 19, 2017 at 3:05 AM, Justin Chang wrote: > Hi all, > > So I am attempting to employ the DG1 finite element method on the poisson > equation using GAMG. When I attempt to solve a problem with roughly 4 > million DOFs across 20 cores, i get this error: > > Traceback (most recent call last): > File "pFiredrake.py", line 109, in > solve(a==L,solution,options_prefix='fe_',solver_ > parameters=solver_params) > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 122, in solve > _solve_varproblem(*args, **kwargs) > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 152, in _solve_varproblem > solver.solve() > File "/home/jchang23/Software/firedrake/src/firedrake/ > firedrake/variational_solver.py", line 220, in solve > self.snes.solve(None, v) > File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve > (src/petsc4py.PETSc.c:172359) > petsc4py.PETSc.Error: error code 63 > [ 6] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/ > snes/interface/snes.c > [ 6] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/ > snes/impls/ksponly/ksponly.c > [ 6] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [ 6] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [ 6] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ > pc/interface/precon.c > [ 6] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/gamg.c > [ 6] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/agg.c > [ 6] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ > interface/matrix.c > [ 6] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [ 6] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [ 6] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [ 6] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [ 6] Argument out of range > [ 6] A really huge hash is being requested.. cannot process: 4096000 > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 6 in communicator MPI_COMM_WORLD > with errorcode 1. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > Traceback (most recent call last): > File "pFiredrake.py", line 109, in > Traceback (most recent call last): > Traceback (most recent call last): > File "pFiredrake.py", line 109, in > File "pFiredrake.py", line 109, in > solve(a==L,solution,options_prefix='fe_',solver_ > parameters=solver_params) > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 122, in solve > solve(a==L,solution,options_prefix='fe_',solver_ > parameters=solver_params) > Traceback (most recent call last): > File "pFiredrake.py", line 109, in > Traceback (most recent call last): > File "pFiredrake.py", line 109, in > _solve_varproblem(*args, **kwargs) > solve(a==L,solution,options_prefix='fe_',solver_ > parameters=solver_params) > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 152, in _solve_varproblem > Traceback (most recent call last): > File "pFiredrake.py", line 109, in > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 122, in solve > Traceback (most recent call last): > File "pFiredrake.py", line 109, in > Traceback (most recent call last): > File "pFiredrake.py", line 109, in > solve(a==L,solution,options_prefix='fe_',solver_ > parameters=solver_params) > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 122, in solve > solve(a==L,solution,options_prefix='fe_',solver_ > parameters=solver_params) > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 122, in solve > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 122, in solve > solve(a==L,solution,options_prefix='fe_',solver_ > parameters=solver_params) > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 122, in solve > _solve_varproblem(*args, **kwargs) > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 152, in _solve_varproblem > _solve_varproblem(*args, **kwargs) > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 152, in _solve_varproblem > _solve_varproblem(*args, **kwargs) > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 152, in _solve_varproblem > _solve_varproblem(*args, **kwargs) > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 152, in _solve_varproblem > solver.solve() > File "/home/jchang23/Software/firedrake/src/firedrake/ > firedrake/variational_solver.py", line 220, in solve > _solve_varproblem(*args, **kwargs) > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 152, in _solve_varproblem > solver.solve() > File "/home/jchang23/Software/firedrake/src/firedrake/ > firedrake/variational_solver.py", line 220, in solve > solver.solve() > File "/home/jchang23/Software/firedrake/src/firedrake/ > firedrake/variational_solver.py", line 220, in solve > solver.solve() > File "/home/jchang23/Software/firedrake/src/firedrake/ > firedrake/variational_solver.py", line 220, in solve > solver.solve() > solver.solve() > File "/home/jchang23/Software/firedrake/src/firedrake/ > firedrake/variational_solver.py", line 220, in solve > self.snes.solve(None, v) > File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve > (src/petsc4py.PETSc.c:172359) > self.snes.solve(None, v) > File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve > (src/petsc4py.PETSc.c:172359) > self.snes.solve(None, v) > File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve > (src/petsc4py.PETSc.c:172359) > self.snes.solve(None, v) > File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve > (src/petsc4py.PETSc.c:172359) > self.snes.solve(None, v) > Traceback (most recent call last): > File "pFiredrake.py", line 109, in > File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve > (src/petsc4py.PETSc.c:172359) > petsc4py.PETSc.Error solve(a==L,solution,options_prefix='fe_',solver_ > parameters=solver_params) > petsc4py.PETSc.Errorpetsc4py.PETSc.Errorpetsc4py.PETSc.Error: > solve(a==L,solution,options_prefix='fe_',solver_parameters=solver_params) > : error code 63 > [ 3] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/ > snes/interface/snes.c > [ 3] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/ > snes/impls/ksponly/ksponly.c > [ 3] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [ 3] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [ 3] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ > pc/interface/precon.c > [ 3] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/gamg.c > [ 3] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/agg.c > [ 3] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ > interface/matrix.c > [ 3] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [ 3] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [ 3] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [ 3] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [ 3] Argument out of range > [ 3] A really huge hash is being requested.. cannot process: 4096000 > File "/home/jchang23/Software/firedrake/src/firedrake/ > firedrake/variational_solver.py", line 220, in solve > : error code 63 > [ 1] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/ > snes/interface/snes.c > [ 1] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/ > snes/impls/ksponly/ksponly.c > [ 1] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [ 1] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [ 1] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ > pc/interface/precon.c > [ 1] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/gamg.c > [ 1] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/agg.c > [ 1] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ > interface/matrix.c > [ 1] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [ 1] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [ 1] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [ 1] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [ 1] Argument out of range > [ 1] A really huge hash is being requested.. cannot process: 4096000 > : error code 63 > [17] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/ > snes/interface/snes.c > [17] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/ > snes/impls/ksponly/ksponly.c > [17] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [17] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [17] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ > pc/interface/precon.c > [17] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/gamg.c > [17] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/agg.c > [17] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ > interface/matrix.c > [17] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [17] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [17] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [17] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [17] Argument out of range > [17] A really huge hash is being requested.. cannot process: 4096000 > solve(a==L,solution,options_prefix='fe_',solver_ > parameters=solver_params) > error code 63 > [ 5] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/ > snes/interface/snes.c > [ 5] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/ > snes/impls/ksponly/ksponly.c > [ 5] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [ 5] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [ 5] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ > pc/interface/precon.c > [ 5] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/gamg.c > [ 5] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/agg.c > [ 5] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ > interface/matrix.c > [ 5] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [ 5] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [ 5] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [ 5] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [ 5] Argument out of range > [ 5] A really huge hash is being requested.. cannot process: 4096000 > Traceback (most recent call last): > File "pFiredrake.py", line 109, in > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 122, in solve > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 122, in solve > _solve_varproblem(*args, **kwargs) > Traceback (most recent call last): > File "pFiredrake.py", line 109, in > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 122, in solve > petsc4py.PETSc.Error: error code 63 > [ 9] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/ > snes/interface/snes.c > [ 9] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/ > snes/impls/ksponly/ksponly.c > [ 9] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [ 9] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [ 9] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ > pc/interface/precon.c > [ 9] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/gamg.c > [ 9] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/agg.c > [ 9] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ > interface/matrix.c > [ 9] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [ 9] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [ 9] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [ 9] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [ 9] Argument out of range > [ 9] A really huge hash is being requested.. cannot process: 4096000 > self.snes.solve(None, v) > Traceback (most recent call last): > File "pFiredrake.py", line 109, in > Traceback (most recent call last): > File "pFiredrake.py", line 109, in > solve(a==L,solution,options_prefix='fe_',solver_ > parameters=solver_params) > _solve_varproblem(*args, **kwargs) > Traceback (most recent call last): > File "pFiredrake.py", line 109, in > solve(a==L,solution,options_prefix='fe_',solver_ > parameters=solver_params) > File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve > (src/petsc4py.PETSc.c:172359) > petsc4py.PETSc.Error: error code 63 > [ 4] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/ > snes/interface/snes.c > [ 4] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/ > snes/impls/ksponly/ksponly.c > [ 4] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [ 4] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [ 4] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ > pc/interface/precon.c > [ 4] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/gamg.c > [ 4] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/agg.c > [ 4] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ > interface/matrix.c > [ 4] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [ 4] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [ 4] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [ 4] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [ 4] Argument out of range > [ 4] A really huge hash is being requested.. cannot process: 4096000 > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 152, in _solve_varproblem > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 122, in solve > Traceback (most recent call last): > File "pFiredrake.py", line 109, in > solve(a==L,solution,options_prefix='fe_',solver_ > parameters=solver_params) > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 122, in solve > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 152, in _solve_varproblem > solve(a==L,solution,options_prefix='fe_',solver_ > parameters=solver_params) > _solve_varproblem(*args, **kwargs) > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 122, in solve > solver.solve() > _solve_varproblem(*args, **kwargs) > solve(a==L,solution,options_prefix='fe_',solver_ > parameters=solver_params) > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 122, in solve > solve(a==L,solution,options_prefix='fe_',solver_ > parameters=solver_params) > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 122, in solve > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 152, in _solve_varproblem > Traceback (most recent call last): > File "pFiredrake.py", line 109, in > _solve_varproblem(*args, **kwargs) > solver.solve() > File "/home/jchang23/Software/firedrake/src/firedrake/ > firedrake/variational_solver.py", line 220, in solve > File "/home/jchang23/Software/firedrake/src/firedrake/ > firedrake/variational_solver.py", line 220, in solve > Traceback (most recent call last): > File "pFiredrake.py", line 109, in > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 152, in _solve_varproblem > _solve_varproblem(*args, **kwargs) > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 152, in _solve_varproblem > _solve_varproblem(*args, **kwargs) > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 122, in solve > _solve_varproblem(*args, **kwargs) > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 152, in _solve_varproblem > solver.solve() > solve(a==L,solution,options_prefix='fe_',solver_ > parameters=solver_params) > Traceback (most recent call last): > File "pFiredrake.py", line 109, in > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 152, in _solve_varproblem > self.snes.solve(None, v) > File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve > (src/petsc4py.PETSc.c:172359) > self.snes.solve(None, v) > solve(a==L,solution,options_prefix='fe_',solver_ > parameters=solver_params) > petsc4py.PETSc.Error: error code 63 > [13] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/ > snes/interface/snes.c > [13] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/ > snes/impls/ksponly/ksponly.c > [13] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [13] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [13] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ > pc/interface/precon.c > [13] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/gamg.c > [13] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/agg.c > [13] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ > interface/matrix.c > [13] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [13] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [13] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [13] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [13] Argument out of range > [13] A really huge hash is being requested.. cannot process: 4096000 > solver.solve() > Traceback (most recent call last): > File "pFiredrake.py", line 109, in > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 152, in _solve_varproblem > File "/home/jchang23/Software/firedrake/src/firedrake/ > firedrake/variational_solver.py", line 220, in solve > _solve_varproblem(*args, **kwargs) > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 152, in _solve_varproblem > solver.solve() > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 122, in solve > solve(a==L,solution,options_prefix='fe_',solver_ > parameters=solver_params) > solver.solve() > File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve > (src/petsc4py.PETSc.c:172359) > petsc4py.PETSc.Error File "/home/jchang23/Software/ > firedrake/src/firedrake/firedrake/solving.py", line 122, in solve > : error code 63 > [11] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/ > snes/interface/snes.c > [11] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/ > snes/impls/ksponly/ksponly.c > [11] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [11] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [11] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ > pc/interface/precon.c > [11] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/gamg.c > [11] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/agg.c > [11] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ > interface/matrix.c > [11] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [11] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [11] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [11] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [11] Argument out of range > [11] A really huge hash is being requested.. cannot process: 4096000 > File "/home/jchang23/Software/firedrake/src/firedrake/ > firedrake/variational_solver.py", line 220, in solve > solve(a==L,solution,options_prefix='fe_',solver_ > parameters=solver_params) > solver.solve() > File "/home/jchang23/Software/firedrake/src/firedrake/ > firedrake/variational_solver.py", line 220, in solve > solver.solve() > File "/home/jchang23/Software/firedrake/src/firedrake/ > firedrake/variational_solver.py", line 220, in solve > solver.solve() > self.snes.solve(None, v) > File "/home/jchang23/Software/firedrake/src/firedrake/ > firedrake/variational_solver.py", line 220, in solve > _solve_varproblem(*args, **kwargs) > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 122, in solve > File "/home/jchang23/Software/firedrake/src/firedrake/ > firedrake/variational_solver.py", line 220, in solve > _solve_varproblem(*args, **kwargs) > self.snes.solve(None, v) > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 122, in solve > self.snes.solve(None, v) > File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve > (src/petsc4py.PETSc.c:172359) > self.snes.solve(None, v) > File "/home/jchang23/Software/firedrake/src/firedrake/ > firedrake/variational_solver.py", line 220, in solve > File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve > (src/petsc4py.PETSc.c:172359) > petsc4py.PETSc.Error: error code 63 > [14] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/ > snes/interface/snes.c > [14] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/ > snes/impls/ksponly/ksponly.c > [14] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [14] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [14] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ > pc/interface/precon.c > [14] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/gamg.c > [14] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/agg.c > [14] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ > interface/matrix.c > [14] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [14] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [14] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [14] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [14] Argument out of range > [14] A really huge hash is being requested.. cannot process: 4096000 > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 152, in _solve_varproblem > self.snes.solve(None, v) > petsc4py.PETSc.Error: error code 63 > [19] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/ > snes/interface/snes.c > [19] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/ > snes/impls/ksponly/ksponly.c > [19] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [19] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [19] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ > pc/interface/precon.c > [19] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/gamg.c > [19] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/agg.c > [19] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ > interface/matrix.c > [19] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [19] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [19] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [19] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [19] Argument out of range > [19] A really huge hash is being requested.. cannot process: 4096000 > _solve_varproblem(*args, **kwargs) > self.snes.solve(None, v) > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 152, in _solve_varproblem > File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve > (src/petsc4py.PETSc.c:172359) > _solve_varproblem(*args, **kwargs) > File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve > (src/petsc4py.PETSc.c:172359) > petsc4py.PETSc.Error: error code 63 > [15] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/ > snes/interface/snes.c > [15] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/ > snes/impls/ksponly/ksponly.c > [15] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [15] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [15] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ > pc/interface/precon.c > [15] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/gamg.c > [15] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/agg.c > [15] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ > interface/matrix.c > [15] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [15] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [15] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [15] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [15] Argument out of range > [15] A really huge hash is being requested.. cannot process: 4096000 > self.snes.solve(None, v) > petsc4py.PETSc.Error: error code 63 > [ 7] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/ > snes/interface/snes.c > [ 7] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/ > snes/impls/ksponly/ksponly.c > [ 7] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [ 7] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [ 7] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ > pc/interface/precon.c > [ 7] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/gamg.c > [ 7] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/agg.c > [ 7] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ > interface/matrix.c > [ 7] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [ 7] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [ 7] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [ 7] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [ 7] Argument out of range > [ 7] A really huge hash is being requested.. cannot process: 4096000 > solver.solve() > File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve > (src/petsc4py.PETSc.c:172359) > petsc4py.PETSc.Error File "/home/jchang23/Software/ > firedrake/src/firedrake/firedrake/solving.py", line 152, in > _solve_varproblem > : error code 63 > [12] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/ > snes/interface/snes.c > [12] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/ > snes/impls/ksponly/ksponly.c > [12] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [12] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [12] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ > pc/interface/precon.c > [12] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/gamg.c > [12] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/agg.c > [12] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ > interface/matrix.c > [12] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [12] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [12] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [12] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [12] Argument out of range > [12] A really huge hash is being requested.. cannot process: 4096000 > File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve > (src/petsc4py.PETSc.c:172359) > solver.solve() > File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", > line 152, in _solve_varproblem > petsc4py.PETSc.Error: File "PETSc/SNES.pyx", line 537, in > petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:172359) > error code 63 > [ 2] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/ > snes/interface/snes.c > [ 2] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/ > snes/impls/ksponly/ksponly.c > [ 2] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [ 2] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [ 2] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ > pc/interface/precon.c > [ 2] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/gamg.c > [ 2] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/agg.c > [ 2] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ > interface/matrix.c > [ 2] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [ 2] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [ 2] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [ 2] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [ 2] Argument out of range > [ 2] A really huge hash is being requested.. cannot process: 4096000 > petsc4py.PETSc.Error File "/home/jchang23/Software/ > firedrake/src/firedrake/firedrake/variational_solver.py", line 220, in > solve > : error code 63 > [ 8] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/ > snes/interface/snes.c > [ 8] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/ > snes/impls/ksponly/ksponly.c > [ 8] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [ 8] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [ 8] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ > pc/interface/precon.c > [ 8] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/gamg.c > [ 8] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/agg.c > [ 8] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ > interface/matrix.c > [ 8] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [ 8] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [ 8] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [ 8] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [ 8] Argument out of range > [ 8] A really huge hash is being requested.. cannot process: 4096000 > File "/home/jchang23/Software/firedrake/src/firedrake/ > firedrake/variational_solver.py", line 220, in solve > solver.solve() > solver.solve() > self.snes.solve(None, v) > File "/home/jchang23/Software/firedrake/src/firedrake/ > firedrake/variational_solver.py", line 220, in solve > File "/home/jchang23/Software/firedrake/src/firedrake/ > firedrake/variational_solver.py", line 220, in solve > self.snes.solve(None, v) > File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve > (src/petsc4py.PETSc.c:172359) > File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve > (src/petsc4py.PETSc.c:172359) > petsc4py.PETSc.Error self.snes.solve(None, v) > File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve > (src/petsc4py.PETSc.c:172359) > : error code 63 > [18] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/ > snes/interface/snes.c > [18] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/ > snes/impls/ksponly/ksponly.c > [18] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [18] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [18] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ > pc/interface/precon.c > [18] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/gamg.c > [18] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/agg.c > [18] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ > interface/matrix.c > [18] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [18] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [18] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [18] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [18] Argument out of range > [18] A really huge hash is being requested.. cannot process: 4096000 > self.snes.solve(None, v) > File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve > (src/petsc4py.PETSc.c:172359) > petsc4py.PETSc.Errorpetsc4py.PETSc.Error: error code 63 > [ 0] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/ > snes/interface/snes.c > [ 0] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/ > snes/impls/ksponly/ksponly.c > [ 0] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [ 0] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [ 0] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ > pc/interface/precon.c > [ 0] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/gamg.c > [ 0] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/agg.c > [ 0] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ > interface/matrix.c > [ 0] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [ 0] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [ 0] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [ 0] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [ 0] Argument out of range > [ 0] A really huge hash is being requested.. cannot process: 4096000 > : error code 63 > [16] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/ > snes/interface/snes.c > [16] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/ > snes/impls/ksponly/ksponly.c > [16] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [16] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [16] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ > pc/interface/precon.c > [16] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/gamg.c > [16] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/agg.c > [16] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ > interface/matrix.c > [16] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [16] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [16] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [16] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [16] Argument out of range > [16] A really huge hash is being requested.. cannot process: 4096000 > petsc4py.PETSc.Error: error code 63 > [10] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/ > snes/interface/snes.c > [10] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/ > snes/impls/ksponly/ksponly.c > [10] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [10] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ > ksp/interface/itfunc.c > [10] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ > pc/interface/precon.c > [10] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/gamg.c > [10] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ > pc/impls/gamg/agg.c > [10] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ > interface/matrix.c > [10] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [10] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in > /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c > [10] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [10] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ > utils/ctable.c > [10] Argument out of range > [10] A really huge hash is being requested.. cannot process: 4096000 > [compute-0-0.local:31182] 19 more processes have sent help message > help-mpi-api.txt / mpi-abort > [compute-0-0.local:31182] Set MCA parameter "orte_base_help_aggregate" to > 0 to see all help / error messages > > However, if I use either ML or HYPRE i do not. When I downscale the > problem to roughly 1.7 million dofs, I get no such error with GAMG, but > both ML and HYPRE are still faster. > > Something tells me I need to tune the GAMG parameters. Here's some -GAMG > related info that may help: > > [0] PCSetUp_GAMG(): level 0) N=4096000, n data rows=1, n data cols=1, > nnz/row (ave)=55, np=20 > [0] PCGAMGFilterGraph(): 78.5842% nnz after filtering, with threshold > 0., 55.8 nnz ave. (N=4096000) > [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square > > And if I run with -pc_gamg_threshold 0.08: > > [0] PCSetUp_GAMG(): level 0) N=4096000, n data rows=1, n data cols=1, > nnz/row (ave)=55, np=20 > [0] PCGAMGFilterGraph(): 28.5842% nnz after filtering, with threshold > 0.08, 55.8 nnz ave. (N=4096000) > [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square > > Any idea what to do here? > > Thanks, > Justin > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Sun Feb 19 11:24:16 2017 From: jroman at dsic.upv.es (Jose E. Roman) Date: Sun, 19 Feb 2017 18:24:16 +0100 Subject: [petsc-users] Krylov-Schur Tolerance In-Reply-To: References: <4691ae05-be89-5bdb-33ac-2fd5cef4c978@wpi.edu> <78DC3258-D0BB-49A9-9246-D15F6E82A9D0@dsic.upv.es> Message-ID: <18B35373-BFB8-4307-B7E4-3D2C49B699DE@dsic.upv.es> > El 19 feb 2017, a las 11:00, Christopher Pierce escribi?: > > Thanks, > > Those changes did improve the tolerances of the solutions. However, I > still have the same problem. For certain matrices the error is up to > 10^4 times as large as the requested tolerances and when using true > residual the solver gets stuck on a certain residual norm the solutions > and does not converge. I dumped the settings that I used which I'm > attaching here. > > Chris The settings seem correct. I would try to solve the problem as a non-symmetric problem: -eps_gen_non_hermitian Does it give small residuals? If so, the problem may be that your B-matrix is quite bad (high norm or ill-conditioned). Jose From jychang48 at gmail.com Sun Feb 19 12:55:50 2017 From: jychang48 at gmail.com (Justin Chang) Date: Sun, 19 Feb 2017 12:55:50 -0600 Subject: [petsc-users] GAMG huge hash being requested In-Reply-To: References: Message-ID: Okay, it doesn't seem like the Firedrake fork (which is what I am using) has this latest fix. Lawrence, when do you think it's possible you folks can incorporate these fixes? On Sun, Feb 19, 2017 at 8:56 AM, Matthew Knepley wrote: > Satish fixed this error. I believe the fix is now in master. > > Thanks, > > Matt > > On Sun, Feb 19, 2017 at 3:05 AM, Justin Chang wrote: > >> Hi all, >> >> So I am attempting to employ the DG1 finite element method on the poisson >> equation using GAMG. When I attempt to solve a problem with roughly 4 >> million DOFs across 20 cores, i get this error: >> >> Traceback (most recent call last): >> File "pFiredrake.py", line 109, in >> solve(a==L,solution,options_prefix='fe_',solver_parameters= >> solver_params) >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 122, in solve >> _solve_varproblem(*args, **kwargs) >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 152, in _solve_varproblem >> solver.solve() >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", >> line 220, in solve >> self.snes.solve(None, v) >> File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve >> (src/petsc4py.PETSc.c:172359) >> petsc4py.PETSc.Error: error code 63 >> [ 6] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes >> /interface/snes.c >> [ 6] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes >> /impls/ksponly/ksponly.c >> [ 6] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [ 6] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [ 6] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/interface/precon.c >> [ 6] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/gamg.c >> [ 6] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/agg.c >> [ 6] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ >> interface/matrix.c >> [ 6] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [ 6] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [ 6] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [ 6] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [ 6] Argument out of range >> [ 6] A really huge hash is being requested.. cannot process: 4096000 >> ------------------------------------------------------------ >> -------------- >> MPI_ABORT was invoked on rank 6 in communicator MPI_COMM_WORLD >> with errorcode 1. >> >> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >> You may or may not see output from other processes, depending on >> exactly when Open MPI kills them. >> ------------------------------------------------------------ >> -------------- >> Traceback (most recent call last): >> File "pFiredrake.py", line 109, in >> Traceback (most recent call last): >> Traceback (most recent call last): >> File "pFiredrake.py", line 109, in >> File "pFiredrake.py", line 109, in >> solve(a==L,solution,options_prefix='fe_',solver_parameters= >> solver_params) >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 122, in solve >> solve(a==L,solution,options_prefix='fe_',solver_parameters= >> solver_params) >> Traceback (most recent call last): >> File "pFiredrake.py", line 109, in >> Traceback (most recent call last): >> File "pFiredrake.py", line 109, in >> _solve_varproblem(*args, **kwargs) >> solve(a==L,solution,options_prefix='fe_',solver_parameters= >> solver_params) >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 152, in _solve_varproblem >> Traceback (most recent call last): >> File "pFiredrake.py", line 109, in >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 122, in solve >> Traceback (most recent call last): >> File "pFiredrake.py", line 109, in >> Traceback (most recent call last): >> File "pFiredrake.py", line 109, in >> solve(a==L,solution,options_prefix='fe_',solver_parameters= >> solver_params) >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 122, in solve >> solve(a==L,solution,options_prefix='fe_',solver_parameters= >> solver_params) >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 122, in solve >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 122, in solve >> solve(a==L,solution,options_prefix='fe_',solver_parameters= >> solver_params) >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 122, in solve >> _solve_varproblem(*args, **kwargs) >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 152, in _solve_varproblem >> _solve_varproblem(*args, **kwargs) >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 152, in _solve_varproblem >> _solve_varproblem(*args, **kwargs) >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 152, in _solve_varproblem >> _solve_varproblem(*args, **kwargs) >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 152, in _solve_varproblem >> solver.solve() >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", >> line 220, in solve >> _solve_varproblem(*args, **kwargs) >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 152, in _solve_varproblem >> solver.solve() >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", >> line 220, in solve >> solver.solve() >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", >> line 220, in solve >> solver.solve() >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", >> line 220, in solve >> solver.solve() >> solver.solve() >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", >> line 220, in solve >> self.snes.solve(None, v) >> File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve >> (src/petsc4py.PETSc.c:172359) >> self.snes.solve(None, v) >> File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve >> (src/petsc4py.PETSc.c:172359) >> self.snes.solve(None, v) >> File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve >> (src/petsc4py.PETSc.c:172359) >> self.snes.solve(None, v) >> File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve >> (src/petsc4py.PETSc.c:172359) >> self.snes.solve(None, v) >> Traceback (most recent call last): >> File "pFiredrake.py", line 109, in >> File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve >> (src/petsc4py.PETSc.c:172359) >> petsc4py.PETSc.Error solve(a==L,solution,options_pr >> efix='fe_',solver_parameters=solver_params) >> petsc4py.PETSc.Errorpetsc4py.PETSc.Errorpetsc4py.PETSc.Error: >> solve(a==L,solution,options_prefix='fe_',solver_parameters=solver_params) >> : error code 63 >> [ 3] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes >> /interface/snes.c >> [ 3] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes >> /impls/ksponly/ksponly.c >> [ 3] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [ 3] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [ 3] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/interface/precon.c >> [ 3] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/gamg.c >> [ 3] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/agg.c >> [ 3] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ >> interface/matrix.c >> [ 3] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [ 3] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [ 3] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [ 3] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [ 3] Argument out of range >> [ 3] A really huge hash is being requested.. cannot process: 4096000 >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", >> line 220, in solve >> : error code 63 >> [ 1] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes >> /interface/snes.c >> [ 1] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes >> /impls/ksponly/ksponly.c >> [ 1] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [ 1] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [ 1] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/interface/precon.c >> [ 1] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/gamg.c >> [ 1] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/agg.c >> [ 1] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ >> interface/matrix.c >> [ 1] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [ 1] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [ 1] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [ 1] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [ 1] Argument out of range >> [ 1] A really huge hash is being requested.. cannot process: 4096000 >> : error code 63 >> [17] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes >> /interface/snes.c >> [17] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes >> /impls/ksponly/ksponly.c >> [17] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [17] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [17] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/interface/precon.c >> [17] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/gamg.c >> [17] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/agg.c >> [17] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ >> interface/matrix.c >> [17] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [17] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [17] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [17] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [17] Argument out of range >> [17] A really huge hash is being requested.. cannot process: 4096000 >> solve(a==L,solution,options_prefix='fe_',solver_parameters= >> solver_params) >> error code 63 >> [ 5] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes >> /interface/snes.c >> [ 5] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes >> /impls/ksponly/ksponly.c >> [ 5] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [ 5] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [ 5] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/interface/precon.c >> [ 5] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/gamg.c >> [ 5] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/agg.c >> [ 5] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ >> interface/matrix.c >> [ 5] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [ 5] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [ 5] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [ 5] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [ 5] Argument out of range >> [ 5] A really huge hash is being requested.. cannot process: 4096000 >> Traceback (most recent call last): >> File "pFiredrake.py", line 109, in >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 122, in solve >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 122, in solve >> _solve_varproblem(*args, **kwargs) >> Traceback (most recent call last): >> File "pFiredrake.py", line 109, in >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 122, in solve >> petsc4py.PETSc.Error: error code 63 >> [ 9] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes >> /interface/snes.c >> [ 9] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes >> /impls/ksponly/ksponly.c >> [ 9] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [ 9] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [ 9] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/interface/precon.c >> [ 9] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/gamg.c >> [ 9] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/agg.c >> [ 9] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ >> interface/matrix.c >> [ 9] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [ 9] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [ 9] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [ 9] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [ 9] Argument out of range >> [ 9] A really huge hash is being requested.. cannot process: 4096000 >> self.snes.solve(None, v) >> Traceback (most recent call last): >> File "pFiredrake.py", line 109, in >> Traceback (most recent call last): >> File "pFiredrake.py", line 109, in >> solve(a==L,solution,options_prefix='fe_',solver_parameters= >> solver_params) >> _solve_varproblem(*args, **kwargs) >> Traceback (most recent call last): >> File "pFiredrake.py", line 109, in >> solve(a==L,solution,options_prefix='fe_',solver_parameters= >> solver_params) >> File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve >> (src/petsc4py.PETSc.c:172359) >> petsc4py.PETSc.Error: error code 63 >> [ 4] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes >> /interface/snes.c >> [ 4] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes >> /impls/ksponly/ksponly.c >> [ 4] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [ 4] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [ 4] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/interface/precon.c >> [ 4] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/gamg.c >> [ 4] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/agg.c >> [ 4] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ >> interface/matrix.c >> [ 4] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [ 4] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [ 4] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [ 4] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [ 4] Argument out of range >> [ 4] A really huge hash is being requested.. cannot process: 4096000 >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 152, in _solve_varproblem >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 122, in solve >> Traceback (most recent call last): >> File "pFiredrake.py", line 109, in >> solve(a==L,solution,options_prefix='fe_',solver_parameters= >> solver_params) >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 122, in solve >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 152, in _solve_varproblem >> solve(a==L,solution,options_prefix='fe_',solver_parameters= >> solver_params) >> _solve_varproblem(*args, **kwargs) >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 122, in solve >> solver.solve() >> _solve_varproblem(*args, **kwargs) >> solve(a==L,solution,options_prefix='fe_',solver_parameters= >> solver_params) >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 122, in solve >> solve(a==L,solution,options_prefix='fe_',solver_parameters= >> solver_params) >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 122, in solve >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 152, in _solve_varproblem >> Traceback (most recent call last): >> File "pFiredrake.py", line 109, in >> _solve_varproblem(*args, **kwargs) >> solver.solve() >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", >> line 220, in solve >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", >> line 220, in solve >> Traceback (most recent call last): >> File "pFiredrake.py", line 109, in >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 152, in _solve_varproblem >> _solve_varproblem(*args, **kwargs) >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 152, in _solve_varproblem >> _solve_varproblem(*args, **kwargs) >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 122, in solve >> _solve_varproblem(*args, **kwargs) >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 152, in _solve_varproblem >> solver.solve() >> solve(a==L,solution,options_prefix='fe_',solver_parameters= >> solver_params) >> Traceback (most recent call last): >> File "pFiredrake.py", line 109, in >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 152, in _solve_varproblem >> self.snes.solve(None, v) >> File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve >> (src/petsc4py.PETSc.c:172359) >> self.snes.solve(None, v) >> solve(a==L,solution,options_prefix='fe_',solver_parameters= >> solver_params) >> petsc4py.PETSc.Error: error code 63 >> [13] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes >> /interface/snes.c >> [13] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes >> /impls/ksponly/ksponly.c >> [13] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [13] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [13] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/interface/precon.c >> [13] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/gamg.c >> [13] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/agg.c >> [13] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ >> interface/matrix.c >> [13] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [13] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [13] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [13] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [13] Argument out of range >> [13] A really huge hash is being requested.. cannot process: 4096000 >> solver.solve() >> Traceback (most recent call last): >> File "pFiredrake.py", line 109, in >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 152, in _solve_varproblem >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", >> line 220, in solve >> _solve_varproblem(*args, **kwargs) >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 152, in _solve_varproblem >> solver.solve() >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 122, in solve >> solve(a==L,solution,options_prefix='fe_',solver_parameters= >> solver_params) >> solver.solve() >> File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve >> (src/petsc4py.PETSc.c:172359) >> petsc4py.PETSc.Error File "/home/jchang23/Software/fired >> rake/src/firedrake/firedrake/solving.py", line 122, in solve >> : error code 63 >> [11] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes >> /interface/snes.c >> [11] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes >> /impls/ksponly/ksponly.c >> [11] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [11] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [11] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/interface/precon.c >> [11] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/gamg.c >> [11] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/agg.c >> [11] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ >> interface/matrix.c >> [11] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [11] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [11] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [11] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [11] Argument out of range >> [11] A really huge hash is being requested.. cannot process: 4096000 >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", >> line 220, in solve >> solve(a==L,solution,options_prefix='fe_',solver_parameters= >> solver_params) >> solver.solve() >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", >> line 220, in solve >> solver.solve() >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", >> line 220, in solve >> solver.solve() >> self.snes.solve(None, v) >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", >> line 220, in solve >> _solve_varproblem(*args, **kwargs) >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 122, in solve >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", >> line 220, in solve >> _solve_varproblem(*args, **kwargs) >> self.snes.solve(None, v) >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 122, in solve >> self.snes.solve(None, v) >> File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve >> (src/petsc4py.PETSc.c:172359) >> self.snes.solve(None, v) >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", >> line 220, in solve >> File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve >> (src/petsc4py.PETSc.c:172359) >> petsc4py.PETSc.Error: error code 63 >> [14] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes >> /interface/snes.c >> [14] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes >> /impls/ksponly/ksponly.c >> [14] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [14] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [14] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/interface/precon.c >> [14] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/gamg.c >> [14] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/agg.c >> [14] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ >> interface/matrix.c >> [14] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [14] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [14] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [14] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [14] Argument out of range >> [14] A really huge hash is being requested.. cannot process: 4096000 >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 152, in _solve_varproblem >> self.snes.solve(None, v) >> petsc4py.PETSc.Error: error code 63 >> [19] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes >> /interface/snes.c >> [19] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes >> /impls/ksponly/ksponly.c >> [19] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [19] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [19] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/interface/precon.c >> [19] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/gamg.c >> [19] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/agg.c >> [19] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ >> interface/matrix.c >> [19] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [19] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [19] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [19] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [19] Argument out of range >> [19] A really huge hash is being requested.. cannot process: 4096000 >> _solve_varproblem(*args, **kwargs) >> self.snes.solve(None, v) >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 152, in _solve_varproblem >> File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve >> (src/petsc4py.PETSc.c:172359) >> _solve_varproblem(*args, **kwargs) >> File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve >> (src/petsc4py.PETSc.c:172359) >> petsc4py.PETSc.Error: error code 63 >> [15] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes >> /interface/snes.c >> [15] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes >> /impls/ksponly/ksponly.c >> [15] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [15] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [15] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/interface/precon.c >> [15] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/gamg.c >> [15] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/agg.c >> [15] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ >> interface/matrix.c >> [15] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [15] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [15] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [15] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [15] Argument out of range >> [15] A really huge hash is being requested.. cannot process: 4096000 >> self.snes.solve(None, v) >> petsc4py.PETSc.Error: error code 63 >> [ 7] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes >> /interface/snes.c >> [ 7] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes >> /impls/ksponly/ksponly.c >> [ 7] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [ 7] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [ 7] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/interface/precon.c >> [ 7] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/gamg.c >> [ 7] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/agg.c >> [ 7] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ >> interface/matrix.c >> [ 7] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [ 7] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [ 7] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [ 7] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [ 7] Argument out of range >> [ 7] A really huge hash is being requested.. cannot process: 4096000 >> solver.solve() >> File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve >> (src/petsc4py.PETSc.c:172359) >> petsc4py.PETSc.Error File "/home/jchang23/Software/fired >> rake/src/firedrake/firedrake/solving.py", line 152, in _solve_varproblem >> : error code 63 >> [12] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes >> /interface/snes.c >> [12] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes >> /impls/ksponly/ksponly.c >> [12] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [12] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [12] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/interface/precon.c >> [12] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/gamg.c >> [12] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/agg.c >> [12] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ >> interface/matrix.c >> [12] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [12] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [12] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [12] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [12] Argument out of range >> [12] A really huge hash is being requested.. cannot process: 4096000 >> File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve >> (src/petsc4py.PETSc.c:172359) >> solver.solve() >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/solving.py", >> line 152, in _solve_varproblem >> petsc4py.PETSc.Error: File "PETSc/SNES.pyx", line 537, in >> petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:172359) >> error code 63 >> [ 2] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes >> /interface/snes.c >> [ 2] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes >> /impls/ksponly/ksponly.c >> [ 2] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [ 2] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [ 2] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/interface/precon.c >> [ 2] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/gamg.c >> [ 2] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/agg.c >> [ 2] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ >> interface/matrix.c >> [ 2] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [ 2] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [ 2] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [ 2] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [ 2] Argument out of range >> [ 2] A really huge hash is being requested.. cannot process: 4096000 >> petsc4py.PETSc.Error File "/home/jchang23/Software/fired >> rake/src/firedrake/firedrake/variational_solver.py", line 220, in solve >> : error code 63 >> [ 8] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes >> /interface/snes.c >> [ 8] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes >> /impls/ksponly/ksponly.c >> [ 8] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [ 8] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [ 8] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/interface/precon.c >> [ 8] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/gamg.c >> [ 8] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/agg.c >> [ 8] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ >> interface/matrix.c >> [ 8] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [ 8] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [ 8] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [ 8] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [ 8] Argument out of range >> [ 8] A really huge hash is being requested.. cannot process: 4096000 >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", >> line 220, in solve >> solver.solve() >> solver.solve() >> self.snes.solve(None, v) >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", >> line 220, in solve >> File "/home/jchang23/Software/firedrake/src/firedrake/firedrake/variational_solver.py", >> line 220, in solve >> self.snes.solve(None, v) >> File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve >> (src/petsc4py.PETSc.c:172359) >> File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve >> (src/petsc4py.PETSc.c:172359) >> petsc4py.PETSc.Error self.snes.solve(None, v) >> File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve >> (src/petsc4py.PETSc.c:172359) >> : error code 63 >> [18] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes >> /interface/snes.c >> [18] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes >> /impls/ksponly/ksponly.c >> [18] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [18] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [18] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/interface/precon.c >> [18] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/gamg.c >> [18] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/agg.c >> [18] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ >> interface/matrix.c >> [18] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [18] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [18] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [18] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [18] Argument out of range >> [18] A really huge hash is being requested.. cannot process: 4096000 >> self.snes.solve(None, v) >> File "PETSc/SNES.pyx", line 537, in petsc4py.PETSc.SNES.solve >> (src/petsc4py.PETSc.c:172359) >> petsc4py.PETSc.Errorpetsc4py.PETSc.Error: error code 63 >> [ 0] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes >> /interface/snes.c >> [ 0] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes >> /impls/ksponly/ksponly.c >> [ 0] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [ 0] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [ 0] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/interface/precon.c >> [ 0] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/gamg.c >> [ 0] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/agg.c >> [ 0] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ >> interface/matrix.c >> [ 0] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [ 0] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [ 0] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [ 0] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [ 0] Argument out of range >> [ 0] A really huge hash is being requested.. cannot process: 4096000 >> : error code 63 >> [16] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes >> /interface/snes.c >> [16] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes >> /impls/ksponly/ksponly.c >> [16] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [16] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [16] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/interface/precon.c >> [16] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/gamg.c >> [16] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/agg.c >> [16] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ >> interface/matrix.c >> [16] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [16] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [16] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [16] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [16] Argument out of range >> [16] A really huge hash is being requested.. cannot process: 4096000 >> petsc4py.PETSc.Error: error code 63 >> [10] SNESSolve() line 4128 in /tmp/pip-FNpsya-build/src/snes >> /interface/snes.c >> [10] SNESSolve_KSPONLY() line 40 in /tmp/pip-FNpsya-build/src/snes >> /impls/ksponly/ksponly.c >> [10] KSPSolve() line 620 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [10] KSPSetUp() line 393 in /tmp/pip-FNpsya-build/src/ksp/ >> ksp/interface/itfunc.c >> [10] PCSetUp() line 968 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/interface/precon.c >> [10] PCSetUp_GAMG() line 524 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/gamg.c >> [10] PCGAMGCoarsen_AGG() line 955 in /tmp/pip-FNpsya-build/src/ksp/ >> pc/impls/gamg/agg.c >> [10] MatTransposeMatMult() line 9962 in /tmp/pip-FNpsya-build/src/mat/ >> interface/matrix.c >> [10] MatTransposeMatMult_MPIAIJ_MPIAIJ() line 902 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [10] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1676 in >> /tmp/pip-FNpsya-build/src/mat/impls/aij/mpi/mpimatmatmult.c >> [10] PetscTableCreate() line 52 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [10] PetscTableCreateHashSize() line 28 in /tmp/pip-FNpsya-build/src/sys/ >> utils/ctable.c >> [10] Argument out of range >> [10] A really huge hash is being requested.. cannot process: 4096000 >> [compute-0-0.local:31182] 19 more processes have sent help message >> help-mpi-api.txt / mpi-abort >> [compute-0-0.local:31182] Set MCA parameter "orte_base_help_aggregate" to >> 0 to see all help / error messages >> >> However, if I use either ML or HYPRE i do not. When I downscale the >> problem to roughly 1.7 million dofs, I get no such error with GAMG, but >> both ML and HYPRE are still faster. >> >> Something tells me I need to tune the GAMG parameters. Here's some -GAMG >> related info that may help: >> >> [0] PCSetUp_GAMG(): level 0) N=4096000, n data rows=1, n data cols=1, >> nnz/row (ave)=55, np=20 >> [0] PCGAMGFilterGraph(): 78.5842% nnz after filtering, with >> threshold 0., 55.8 nnz ave. (N=4096000) >> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square >> >> And if I run with -pc_gamg_threshold 0.08: >> >> [0] PCSetUp_GAMG(): level 0) N=4096000, n data rows=1, n data cols=1, >> nnz/row (ave)=55, np=20 >> [0] PCGAMGFilterGraph(): 28.5842% nnz after filtering, with >> threshold 0.08, 55.8 nnz ave. (N=4096000) >> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square >> >> Any idea what to do here? >> >> Thanks, >> Justin >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lawrence.mitchell at imperial.ac.uk Sun Feb 19 14:32:48 2017 From: lawrence.mitchell at imperial.ac.uk (Lawrence Mitchell) Date: Sun, 19 Feb 2017 20:32:48 +0000 Subject: [petsc-users] GAMG huge hash being requested In-Reply-To: References: Message-ID: > On 19 Feb 2017, at 18:55, Justin Chang wrote: > > Okay, it doesn't seem like the Firedrake fork (which is what I am using) has this latest fix. Lawrence, when do you think it's possible you folks can incorporate these fixes I'll fast forward our branch pointer on Monday. Lawrence From cmpierce at WPI.EDU Mon Feb 20 04:01:47 2017 From: cmpierce at WPI.EDU (Christopher Pierce) Date: Mon, 20 Feb 2017 05:01:47 -0500 Subject: [petsc-users] Krylov-Schur Tolerance In-Reply-To: <18B35373-BFB8-4307-B7E4-3D2C49B699DE@dsic.upv.es> References: <4691ae05-be89-5bdb-33ac-2fd5cef4c978@wpi.edu> <78DC3258-D0BB-49A9-9246-D15F6E82A9D0@dsic.upv.es> <18B35373-BFB8-4307-B7E4-3D2C49B699DE@dsic.upv.es> Message-ID: <7ee69de1-511c-6dba-58b1-4a330e1beb96@wpi.edu> It seems to give the same results. I exported the matrices to Matlab and checked the estimated condition number of the B matrix which came to ~15 and the 2-norm of the B matrix which was ~10^6. I'm guessing that the large matrix norm is the problem. I glanced over the source for the RQCG solver and it doesn't seem to use a linear solver which is likely why it showed better performance. Do you have any suggestions for dealing with problems like this? Chris On 02/19/17 12:24, Jose E. Roman wrote: >> El 19 feb 2017, a las 11:00, Christopher Pierce escribi?: >> >> Thanks, >> >> Those changes did improve the tolerances of the solutions. However, I >> still have the same problem. For certain matrices the error is up to >> 10^4 times as large as the requested tolerances and when using true >> residual the solver gets stuck on a certain residual norm the solutions >> and does not converge. I dumped the settings that I used which I'm >> attaching here. >> >> Chris > The settings seem correct. > I would try to solve the problem as a non-symmetric problem: -eps_gen_non_hermitian > Does it give small residuals? If so, the problem may be that your B-matrix is quite bad (high norm or ill-conditioned). > > Jose > > From jroman at dsic.upv.es Mon Feb 20 04:33:40 2017 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 20 Feb 2017 11:33:40 +0100 Subject: [petsc-users] Krylov-Schur Tolerance In-Reply-To: <7ee69de1-511c-6dba-58b1-4a330e1beb96@wpi.edu> References: <4691ae05-be89-5bdb-33ac-2fd5cef4c978@wpi.edu> <78DC3258-D0BB-49A9-9246-D15F6E82A9D0@dsic.upv.es> <18B35373-BFB8-4307-B7E4-3D2C49B699DE@dsic.upv.es> <7ee69de1-511c-6dba-58b1-4a330e1beb96@wpi.edu> Message-ID: > El 20 feb 2017, a las 11:01, Christopher Pierce escribi?: > > It seems to give the same results. I exported the matrices to Matlab > and checked the estimated condition number of the B matrix which came to > ~15 and the 2-norm of the B matrix which was ~10^6. I'm guessing that > the large matrix norm is the problem. I glanced over the source for the > RQCG solver and it doesn't seem to use a linear solver which is likely > why it showed better performance. Do you have any suggestions for > dealing with problems like this? > > Chris Send the data files to my personal email and I will make some tests. Jose From lixin_chu at yahoo.com Mon Feb 20 06:27:24 2017 From: lixin_chu at yahoo.com (lixin chu) Date: Mon, 20 Feb 2017 12:27:24 +0000 (UTC) Subject: [petsc-users] Newbie question : complex number with MatView ? References: <1384528687.1324844.1487593644019.ref@mail.yahoo.com> Message-ID: <1384528687.1324844.1487593644019@mail.yahoo.com> Hello,Some questions wrt complex number - I have compiled PETSc with complex data type, but having problems of setting/getting the values: - MatView does not seem to print complex number ?? - Does MATSEQAIJ support row major SetValues for complex number ? I have tested real number, it is ok. I can not get the right values with MatGetRow. Still troubleshooting if it is my program problem... This is how I set the matrix: ? PetscComplex *buffer; // this is the column data buffer, malloc'ed ? // setting the values? buffer [i] = double1 + double2 * PETSC_i; Then call MatSetValues, passing buffer.???The program is ok for real numbers, btw. I use a 5x5 matrix, when setting buffer for column 2, I can see these are the data passed to MatSetValues (row, col):(0, 2) ?1 + 0(1, 2) ?1 + 0(3, 2) ?-1 + 0 But MatGetRow return:(0, 2) ?1. + 1. i (1, 2) ?0. + 1. i(3, 2) ?1. + 0. i (I have used MatRestoreRow after MatGetRow). for ( i = 0; i < rows; i++ ) { MatGetRow ( Z, i, ¤t_doubles, &col_index, &complex_buffer ); if ( current_doubles > 0 ) { for ( j = 0; j < current_doubles; j++ ) { PetscPrintf ( PETSC_COMM_WORLD, "(%d, %d) ?%g + %g i\n", i, col_index[j], PetscRealPart(complex_buffer[j]), PetscImaginaryPart (complex_buffer[j]) ); } } MatRestoreRow( Z, i, ¤t_doubles, &col_index, &complex_buffer );} ?thanks ! LX -------------- next part -------------- An HTML attachment was scrubbed... URL: From bikash at umich.edu Mon Feb 20 07:01:53 2017 From: bikash at umich.edu (Bikash Kanungo) Date: Mon, 20 Feb 2017 08:01:53 -0500 Subject: [petsc-users] Non-hermitian KSP Message-ID: Hi, I'm trying to solve a KSP using GMRES in which the matrix (say, A) is non-hermitian. I'm using a shell matrix for A. In this context I have the following few questions: 1. I guess one needs to provide appropriate user-defined function override for MatMultHermitianTranspose instead of MatMultTranspose (i.e., MATOP_MULT_TRANSPOSE). However, I am unable to find any MATOP_MULT_HERMITIAN_TRANSPOSE flag. So if GMRES expects MatlMultHermitianTranspose then should I override the MatMultTranspose in a way to actually perform MatMultHermitianTranspose? Or am I mistaken in my assumption that GMRES requires MatMultHermitianTranspose instead of MatMultTranspose? 2. For the same problem, I want to provide my own preconditioner matrix (say, P) as a shell a matrix. What operations (e.g., MATOP_MULT, MATOP_MULT_TRANSPOSE etc.) should I override through user-defined functions? Thanks, Bikash -- Bikash S. Kanungo PhD Student Computational Materials Physics Group Mechanical Engineering University of Michigan -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Feb 20 07:10:33 2017 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 20 Feb 2017 07:10:33 -0600 Subject: [petsc-users] Non-hermitian KSP In-Reply-To: References: Message-ID: On Mon, Feb 20, 2017 at 7:01 AM, Bikash Kanungo wrote: > Hi, > > > I'm trying to solve a KSP using GMRES in which the matrix (say, A) is > non-hermitian. I'm using a shell matrix for A. > > In this context I have the following few questions: > > > 1. I guess one needs to provide appropriate user-defined function > override for MatMultHermitianTranspose instead of MatMultTranspose (i.e., > MATOP_MULT_TRANSPOSE). However, I am unable to find any > MATOP_MULT_HERMITIAN_TRANSPOSE flag. So if GMRES expects > MatlMultHermitianTranspose then should I override the MatMultTranspose in a > way to actually perform MatMultHermitianTranspose? Or am I mistaken in my > assumption that GMRES requires MatMultHermitianTranspose instead of > MatMultTranspose? > > GMRES does not use the transpose at all. > > 1. For the same problem, I want to provide my own preconditioner > matrix (say, P) as a shell a matrix. What operations (e.g., MATOP_MULT, > MATOP_MULT_TRANSPOSE etc.) should I override through user-defined > functions? > > A user -defined preconditioner should be implemented using PCSHELL, which is much more straightforward than MATSHELL. Thanks, Matt > Thanks, > Bikash > > -- > Bikash S. Kanungo > PhD Student > Computational Materials Physics Group > Mechanical Engineering > University of Michigan > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Feb 20 10:02:18 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 20 Feb 2017 10:02:18 -0600 Subject: [petsc-users] Newbie question : complex number with MatView ? In-Reply-To: <1384528687.1324844.1487593644019@mail.yahoo.com> References: <1384528687.1324844.1487593644019.ref@mail.yahoo.com> <1384528687.1324844.1487593644019@mail.yahoo.com> Message-ID: PETSc must be configured with --with-scalar-type=complex to use complex numbers. Then all PETSc vectors/matrices contain complex numbers. You cannot mix PETSc objects with real numbers and PETSc objects with complex numbers in the same program. Barry > On Feb 20, 2017, at 6:27 AM, lixin chu wrote: > > Hello, > Some questions wrt complex number - I have compiled PETSc with complex data type, but having problems of setting/getting the values: > > - MatView does not seem to print complex number ? > > - Does MATSEQAIJ support row major SetValues for complex number ? I have tested real number, it is ok. I can not get the right values with MatGetRow. Still troubleshooting if it is my program problem... > > This is how I set the matrix: > > PetscComplex *buffer; // this is the column data buffer, malloc'ed > > // setting the values > buffer [i] = double1 + double2 * PETSC_i; > > Then call MatSetValues, passing buffer. > > The program is ok for real numbers, btw. > > > I use a 5x5 matrix, when setting buffer for column 2, I can see these are the data passed to MatSetValues (row, col): > (0, 2) 1 + 0 > (1, 2) 1 + 0 > (3, 2) -1 + 0 > > > But MatGetRow return: > (0, 2) 1. + 1. i > (1, 2) 0. + 1. i > (3, 2) 1. + 0. i > > (I have used MatRestoreRow after MatGetRow). > > for ( i = 0; i < rows; i++ ) { > MatGetRow ( Z, i, ¤t_doubles, &col_index, &complex_buffer ); > if ( current_doubles > 0 ) { > for ( j = 0; j < current_doubles; j++ ) { > PetscPrintf ( PETSC_COMM_WORLD, "(%d, %d) %g + %g i\n", i, col_index[j], PetscRealPart(complex_buffer[j]), PetscImaginaryPart (complex_buffer[j]) ); > } > } > MatRestoreRow( Z, i, ¤t_doubles, &col_index, &complex_buffer ); > } > > > thanks ! > LX From lixin_chu at yahoo.com Mon Feb 20 18:20:07 2017 From: lixin_chu at yahoo.com (lixin chu) Date: Tue, 21 Feb 2017 00:20:07 +0000 (UTC) Subject: [petsc-users] Newbie question : complex number with MatView ? In-Reply-To: References: <1384528687.1324844.1487593644019.ref@mail.yahoo.com> <1384528687.1324844.1487593644019@mail.yahoo.com> Message-ID: <1585101353.1497118.1487636407463@mail.yahoo.com> Yes, I compiled with complex data type.? Is setting imaginary part 0 ok ? Like this: 1.0 ? 0.0 * PETSC_i? Sent from Yahoo Mail on Android On Tue, 21 Feb 2017 at 0:02, Barry Smith wrote: ? PETSc must be configured with --with-scalar-type=complex to use complex numbers. Then all PETSc vectors/matrices contain complex numbers. You cannot mix PETSc objects with real numbers and PETSc objects with complex numbers in the same program. ? Barry > On Feb 20, 2017, at 6:27 AM, lixin chu wrote: > > Hello, > Some questions wrt complex number - I have compiled PETSc with complex data type, but having problems of setting/getting the values: > > - MatView does not seem to print complex number ? > > - Does MATSEQAIJ support row major SetValues for complex number ? I have tested real number, it is ok. I can not get the right values with MatGetRow. Still troubleshooting if it is my program problem... > > This is how I set the matrix: > >? PetscComplex *buffer; // this is the column data buffer, malloc'ed > >? // setting the values >? buffer [i] = double1 + double2 * PETSC_i; > > Then call MatSetValues, passing buffer. >? > The program is ok for real numbers, btw. > > > I use a 5x5 matrix, when setting buffer for column 2, I can see these are the data passed to MatSetValues (row, col): > (0, 2)? 1 + 0 > (1, 2)? 1 + 0 > (3, 2)? -1 + 0 > > > But MatGetRow return: > (0, 2)? 1. + 1. i > (1, 2)? 0. + 1. i > (3, 2)? 1. + 0. i > > (I have used MatRestoreRow after MatGetRow). > > for ( i = 0; i < rows; i++ ) { > ??? MatGetRow ( Z, i, ?t_doubles, &col_index, &complex_buffer ); > ??? if ( current_doubles > 0 ) { > ??? ??? for ( j = 0; j < current_doubles; j++ ) { > ??? ??? ??? PetscPrintf ( PETSC_COMM_WORLD, "(%d, %d)? %g + %g i\n", i, col_index[j], PetscRealPart(complex_buffer[j]), PetscImaginaryPart (complex_buffer[j]) ); > ??? ??? } > ??? } > ??? MatRestoreRow( Z, i, ?t_doubles, &col_index, &complex_buffer ); > } > >? > thanks ! > LX -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Feb 20 19:09:36 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 20 Feb 2017 19:09:36 -0600 Subject: [petsc-users] Newbie question : complex number with MatView ? In-Reply-To: <1585101353.1497118.1487636407463@mail.yahoo.com> References: <1384528687.1324844.1487593644019.ref@mail.yahoo.com> <1384528687.1324844.1487593644019@mail.yahoo.com> <1585101353.1497118.1487636407463@mail.yahoo.com> Message-ID: <8B0A1678-A008-48F6-9A89-6171AB3EE63C@mcs.anl.gov> Yes, but you do not need to. You can say PetscComplex x = 1.0; > On Feb 20, 2017, at 6:20 PM, lixin chu wrote: > > Yes, I compiled with complex data type. > > Is setting imaginary part 0 ok ? Like this: > > 1.0 ? 0.0 * PETSC_i > > > Sent from Yahoo Mail on Android > > On Tue, 21 Feb 2017 at 0:02, Barry Smith > wrote: > > PETSc must be configured with --with-scalar-type=complex to use complex numbers. Then all PETSc vectors/matrices contain complex numbers. You cannot mix PETSc objects with real numbers and PETSc objects with complex numbers in the same program. > > > Barry > > > On Feb 20, 2017, at 6:27 AM, lixin chu wrote: > > > > Hello, > > Some questions wrt complex number - I have compiled PETSc with complex data type, but having problems of setting/getting the values: > > > > - MatView does not seem to print complex number ? > > > > - Does MATSEQAIJ support row major SetValues for complex number ? I have tested real number, it is ok. I can not get the right values with MatGetRow. Still troubleshooting if it is my program problem... > > > > This is how I set the matrix: > > > > PetscComplex *buffer; // this is the column data buffer, malloc'ed > > > > // setting the values > > buffer [i] = double1 + double2 * PETSC_i; > > > > Then call MatSetValues, passing buffer. > > > > The program is ok for real numbers, btw. > > > > > > I use a 5x5 matrix, when setting buffer for column 2, I can see these are the data passed to MatSetValues (row, col): > > (0, 2) 1 + 0 > > (1, 2) 1 + 0 > > (3, 2) -1 + 0 > > > > > > But MatGetRow return: > > (0, 2) 1. + 1. i > > (1, 2) 0. + 1. i > > (3, 2) 1. + 0. i > > > > (I have used MatRestoreRow after MatGetRow). > > > > for ( i = 0; i < rows; i++ ) { > > MatGetRow ( Z, i, ?t_doubles, &col_index, &complex_buffer ); > > if ( current_doubles > 0 ) { > > for ( j = 0; j < current_doubles; j++ ) { > > PetscPrintf ( PETSC_COMM_WORLD, "(%d, %d) %g + %g i\n", i, col_index[j], PetscRealPart(complex_buffer[j]), PetscImaginaryPart (complex_buffer[j]) ); > > } > > } > > MatRestoreRow( Z, i, ?t_doubles, &col_index, &complex_buffer ); > > } > > > > > > thanks ! > > LX From bikash at umich.edu Mon Feb 20 19:35:48 2017 From: bikash at umich.edu (Bikash Kanungo) Date: Mon, 20 Feb 2017 20:35:48 -0500 Subject: [petsc-users] Non-hermitian KSP In-Reply-To: References: Message-ID: Thank you so much, Matthew. On Feb 20, 2017 8:10 AM, "Matthew Knepley" wrote: > On Mon, Feb 20, 2017 at 7:01 AM, Bikash Kanungo wrote: > >> Hi, >> >> >> I'm trying to solve a KSP using GMRES in which the matrix (say, A) is >> non-hermitian. I'm using a shell matrix for A. >> >> In this context I have the following few questions: >> >> >> 1. I guess one needs to provide appropriate user-defined function >> override for MatMultHermitianTranspose instead of MatMultTranspose (i.e., >> MATOP_MULT_TRANSPOSE). However, I am unable to find any >> MATOP_MULT_HERMITIAN_TRANSPOSE flag. So if GMRES expects >> MatlMultHermitianTranspose then should I override the MatMultTranspose in a >> way to actually perform MatMultHermitianTranspose? Or am I mistaken in my >> assumption that GMRES requires MatMultHermitianTranspose instead of >> MatMultTranspose? >> >> GMRES does not use the transpose at all. > >> >> 1. For the same problem, I want to provide my own preconditioner >> matrix (say, P) as a shell a matrix. What operations (e.g., MATOP_MULT, >> MATOP_MULT_TRANSPOSE etc.) should I override through user-defined >> functions? >> >> A user -defined preconditioner should be implemented using PCSHELL, which > is much more straightforward than MATSHELL. > > Thanks, > > Matt > > >> Thanks, >> Bikash >> >> -- >> Bikash S. Kanungo >> PhD Student >> Computational Materials Physics Group >> Mechanical Engineering >> University of Michigan >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jychang48 at gmail.com Tue Feb 21 00:01:36 2017 From: jychang48 at gmail.com (Justin Chang) Date: Tue, 21 Feb 2017 00:01:36 -0600 Subject: [petsc-users] GAMG huge hash being requested In-Reply-To: References: Message-ID: Okay thanks On Sun, Feb 19, 2017 at 2:32 PM, Lawrence Mitchell < lawrence.mitchell at imperial.ac.uk> wrote: > > > > On 19 Feb 2017, at 18:55, Justin Chang wrote: > > > > Okay, it doesn't seem like the Firedrake fork (which is what I am using) > has this latest fix. Lawrence, when do you think it's possible you folks > can incorporate these fixes > > I'll fast forward our branch pointer on Monday. > > Lawrence > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lawrence.mitchell at imperial.ac.uk Tue Feb 21 07:04:42 2017 From: lawrence.mitchell at imperial.ac.uk (Lawrence Mitchell) Date: Tue, 21 Feb 2017 13:04:42 +0000 Subject: [petsc-users] GAMG huge hash being requested In-Reply-To: References: Message-ID: <709aa52f-3149-f22f-0eed-4ab02e3577e7@imperial.ac.uk> Hi Justin, On 21/02/17 06:01, Justin Chang wrote: > Okay thanks Now done. Cheers, Lawrence -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 473 bytes Desc: OpenPGP digital signature URL: From dnolte at dim.uchile.cl Tue Feb 21 10:07:42 2017 From: dnolte at dim.uchile.cl (David Nolte) Date: Tue, 21 Feb 2017 13:07:42 -0300 Subject: [petsc-users] understanding the LSC preconditioner Message-ID: <74e410b0-c64d-dccf-a653-c0505fb16dd6@dim.uchile.cl> Dear all, new to PETSc, I am trying to use the LSC preconditioner for a Stokes problem (discretized by means of stable FEM). I use the python backend petsc4py. The "automatic" version of the LSC seems to work with the following setup (I think adapted from Matt's tutorial slides): -ksp_view -ksp_converged_reason -ksp_monitor_true_residual -ksp_type fgmres -ksp_rtol 1.0e-8 -pc_type fieldsplit -pc_fieldsplit_detect_saddle_point -pc_fieldsplit_type schur -pc_fieldsplit_schur_fact_type upper -pc_fieldsplit_schur_precondition self -fieldsplit_0_ksp_type preonly -fieldsplit_0_pc_type ml -fieldsplit_1_ksp_type preonly -fieldsplit_1_pc_type lsc -fieldsplit_1_lsc_pc_type ml -fieldsplit_1_lsc_ksp_type preonly In a 3D setup with 250k dofs this converges within 78 iterations. (For reference, upper Schur factorization with ML for the uu-block and Sp = diag(Q), the diagonal of the pressure mass matrix, takes 41 iterations and half of the computation time.) Now I just wanted to check if I can get the same result by building the L-matrix manually with the following piece of python code, where is0, is1 are the index sets corresponding to the velocity and pressure dofs, and A is full the system matrix. Sp = Sp.getSubMatrix(is1, is1) pc.setFieldSplitSchurPreType(PETSc.PC.SchurPreType.USER, Sp) # Sp.setType(PETSc.Mat.Type.SCHURCOMPLEMENT) # necessary? # extract A10 block Bdiv = A.getSubMatrix(is1, is0) # extract A01 block Bgrad = A.getSubMatrix(is0, is1) L = Bdiv L.matMult(Bgrad) Sp.compose('LSC_L', L) Sp.compose('LSC_Lp', L) To my understanding, this should behave similarly to what the LSC preconditioner does when LSC_L is not given. However, I get a segmentation fault during the first iteration: 0 KSP unpreconditioned resid norm 2.963704216563e+01 true resid norm 2.963704216563e+01 ||r(i)||/||b|| 1.000000000000e+00 [1] 2311 segmentation fault (core dumped) python StokesPC/stokespc/stokes_bench.py What am I doing wrong? I appreciate any hints, thanks a lot in advance! Regards, David PS: The log trace is: 0 KSP unpreconditioned resid norm 2.963704216563e+01 true resid norm 2.963704216563e+01 ||r(i)||/||b|| 1.000000000000e+00 [0] 10.0543 Event begin: VecScale [0] 10.0545 Event end: VecScale [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] 10.0545 Event begin: PCApply [0] 10.0545 Event begin: VecScatterBegin [0] 10.0546 Event end: VecScatterBegin [0] 10.0546 Event begin: KSPSolve_FS_Schu [0] 10.0546 Event begin: KSPSetUp [0] 10.0546 Event end: KSPSetUp [0] PCSetUp(): Setting up PC for first time [0] 10.0546 Event begin: PCSetUp [0] 10.0547 Event begin: VecSet [0] 10.055 Event end: VecSet [0] 10.055 Event begin: VecSet [0] 10.0553 Event end: VecSet [0] 10.0553 Event begin: VecSet [0] 10.0553 Event end: VecSet [0] 10.0554 Event end: PCSetUp [0] 10.0554 Event begin: VecSet [0] 10.0554 Event end: VecSet [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] 10.0554 Event begin: KSPSetUp [0] 10.0554 Event end: KSPSetUp [0] PCSetUp(): Setting up PC for first time [0] 10.0554 Event begin: PCSetUp [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783 [0] 10.0555 Event begin: VecSet [0] 10.0557 Event end: VecSet [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783 [0] 10.0557 Event begin: VecSet [0] 10.0558 Event end: VecSet [0] 10.082 Event begin: MatMult [0] 10.1273 Event end: MatMult [0] 10.1277 Event begin: MatMult [0] 10.1739 Event end: MatMult [0] 10.1742 Event begin: MatMult [0] 10.2195 Event end: MatMult [0] 10.2199 Event begin: MatMult [0] 10.2653 Event end: MatMult [0] 10.2657 Event begin: MatMult [0] 10.3113 Event end: MatMult [0] 10.3116 Event begin: MatMult [0] 10.3571 Event end: MatMult [0] 10.3575 Event begin: MatMult [0] 10.403 Event end: MatMult [0] 10.4033 Event begin: MatMult [0] 10.4487 Event end: MatMult [0] 10.4491 Event begin: MatMult [0] 10.4947 Event end: MatMult [0] 10.495 Event begin: MatMult [0] 10.5406 Event end: MatMult From bsmith at mcs.anl.gov Tue Feb 21 11:20:54 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 21 Feb 2017 11:20:54 -0600 Subject: [petsc-users] understanding the LSC preconditioner In-Reply-To: <74e410b0-c64d-dccf-a653-c0505fb16dd6@dim.uchile.cl> References: <74e410b0-c64d-dccf-a653-c0505fb16dd6@dim.uchile.cl> Message-ID: You'll have to figure out what is triggering the segmentation violation. If it is the python that is crashing then likely you can run the entire python program in the debugger and then when it crashes you should be able to see where. Barry > On Feb 21, 2017, at 10:07 AM, David Nolte wrote: > > Dear all, > > new to PETSc, I am trying to use the LSC preconditioner for a Stokes > problem (discretized by means of stable FEM). I use the python backend > petsc4py. > The "automatic" version of the LSC seems to work with the following > setup (I think adapted from Matt's tutorial slides): > > -ksp_view > -ksp_converged_reason > -ksp_monitor_true_residual > -ksp_type fgmres > -ksp_rtol 1.0e-8 > > -pc_type fieldsplit > -pc_fieldsplit_detect_saddle_point > -pc_fieldsplit_type schur > -pc_fieldsplit_schur_fact_type upper > -pc_fieldsplit_schur_precondition self > > -fieldsplit_0_ksp_type preonly > -fieldsplit_0_pc_type ml > > -fieldsplit_1_ksp_type preonly > -fieldsplit_1_pc_type lsc > -fieldsplit_1_lsc_pc_type ml > -fieldsplit_1_lsc_ksp_type preonly > > In a 3D setup with 250k dofs this converges within 78 iterations. (For > reference, upper Schur factorization with ML for the uu-block and Sp = > diag(Q), the diagonal of the pressure mass matrix, takes 41 iterations > and half of the computation time.) > > Now I just wanted to check if I can get the same result by building the > L-matrix manually with the following piece of python code, where is0, > is1 are the index sets corresponding to the velocity and pressure dofs, > and A is full the system matrix. > > Sp = Sp.getSubMatrix(is1, is1) > pc.setFieldSplitSchurPreType(PETSc.PC.SchurPreType.USER, Sp) > # Sp.setType(PETSc.Mat.Type.SCHURCOMPLEMENT) # necessary? > # extract A10 block > Bdiv = A.getSubMatrix(is1, is0) > # extract A01 block > Bgrad = A.getSubMatrix(is0, is1) > L = Bdiv > L.matMult(Bgrad) > Sp.compose('LSC_L', L) > Sp.compose('LSC_Lp', L) > > To my understanding, this should behave similarly to what the LSC > preconditioner does when LSC_L is not given. However, I get a > segmentation fault during the first iteration: > > 0 KSP unpreconditioned resid norm 2.963704216563e+01 true resid norm > 2.963704216563e+01 ||r(i)||/||b|| 1.000000000000e+00 > [1] 2311 segmentation fault (core dumped) python > StokesPC/stokespc/stokes_bench.py > > What am I doing wrong? I appreciate any hints, thanks a lot in advance! > > Regards, > David > > > PS: The log trace is: > 0 KSP unpreconditioned resid norm 2.963704216563e+01 true resid norm > 2.963704216563e+01 ||r(i)||/||b|| 1.000000000000e+00 > [0] 10.0543 Event begin: VecScale > [0] 10.0545 Event end: VecScale > [0] PCSetUp(): Leaving PC with identical preconditioner since operator > is unchanged > [0] 10.0545 Event begin: PCApply > [0] 10.0545 Event begin: VecScatterBegin > [0] 10.0546 Event end: VecScatterBegin > [0] 10.0546 Event begin: KSPSolve_FS_Schu > [0] 10.0546 Event begin: KSPSetUp > [0] 10.0546 Event end: KSPSetUp > [0] PCSetUp(): Setting up PC for first time > [0] 10.0546 Event begin: PCSetUp > [0] 10.0547 Event begin: VecSet > [0] 10.055 Event end: VecSet > [0] 10.055 Event begin: VecSet > [0] 10.0553 Event end: VecSet > [0] 10.0553 Event begin: VecSet > [0] 10.0553 Event end: VecSet > [0] 10.0554 Event end: PCSetUp > [0] 10.0554 Event begin: VecSet > [0] 10.0554 Event end: VecSet > [0] PCSetUp(): Leaving PC with identical preconditioner since operator > is unchanged > [0] 10.0554 Event begin: KSPSetUp > [0] 10.0554 Event end: KSPSetUp > [0] PCSetUp(): Setting up PC for first time > [0] 10.0554 Event begin: PCSetUp > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 > -2080374783 > [0] 10.0555 Event begin: VecSet > [0] 10.0557 Event end: VecSet > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 > -2080374783 > [0] 10.0557 Event begin: VecSet > [0] 10.0558 Event end: VecSet > [0] 10.082 Event begin: MatMult > [0] 10.1273 Event end: MatMult > [0] 10.1277 Event begin: MatMult > [0] 10.1739 Event end: MatMult > [0] 10.1742 Event begin: MatMult > [0] 10.2195 Event end: MatMult > [0] 10.2199 Event begin: MatMult > [0] 10.2653 Event end: MatMult > [0] 10.2657 Event begin: MatMult > [0] 10.3113 Event end: MatMult > [0] 10.3116 Event begin: MatMult > [0] 10.3571 Event end: MatMult > [0] 10.3575 Event begin: MatMult > [0] 10.403 Event end: MatMult > [0] 10.4033 Event begin: MatMult > [0] 10.4487 Event end: MatMult > [0] 10.4491 Event begin: MatMult > [0] 10.4947 Event end: MatMult > [0] 10.495 Event begin: MatMult > [0] 10.5406 Event end: MatMult > > > From dnolte at dim.uchile.cl Tue Feb 21 13:02:01 2017 From: dnolte at dim.uchile.cl (David Nolte) Date: Tue, 21 Feb 2017 16:02:01 -0300 Subject: [petsc-users] understanding the LSC preconditioner In-Reply-To: References: <74e410b0-c64d-dccf-a653-c0505fb16dd6@dim.uchile.cl> Message-ID: <123a2c95-0c1b-31fb-858b-2c4ddc3e2284@dim.uchile.cl> It crashes in ML_matmat_mult(): Program received signal SIGSEGV, Segmentation fault. 0x00007ffff5cdaeb0 in ML_matmat_mult () from /usr/local/petsc-32/lib/libpetsc.so.3.7 (gdb) bt #0 0x00007ffff5cdaeb0 in ML_matmat_mult () from /usr/local/petsc-32/lib/libpetsc.so.3.7 #1 0x00007ffff5cdbf76 in ML_2matmult () from /usr/local/petsc-32/lib/libpetsc.so.3.7 #2 0x00007ffff5ca197a in ML_AGG_Gen_Prolongator () from /usr/local/petsc-32/lib/libpetsc.so.3.7 #3 0x00007ffff5c9fa71 in ML_Gen_MultiLevelHierarchy () from /usr/local/petsc-32/lib/libpetsc.so.3.7 #4 0x00007ffff5ca0484 in ML_Gen_MultiLevelHierarchy_UsingAggregation () from /usr/local/petsc-32/lib/libpetsc.so.3.7 #5 0x00007ffff5780aff in PCSetUp_ML () from /usr/local/petsc-32/lib/libpetsc.so.3.7 #6 0x00007ffff5667fce in PCSetUp () from /usr/local/petsc-32/lib/libpetsc.so.3.7 #7 0x00007ffff578d9f8 in KSPSetUp () from /usr/local/petsc-32/lib/libpetsc.so.3.7 #8 0x00007ffff578e7d8 in KSPSolve () from /usr/local/petsc-32/lib/libpetsc.so.3.7 #9 0x00007ffff578606b in PCApply_LSC () from /usr/local/petsc-32/lib/libpetsc.so.3.7 #10 0x00007ffff5668640 in PCApply () from /usr/local/petsc-32/lib/libpetsc.so.3.7 #11 0x00007ffff57f05a5 in KSPSolve_PREONLY () from /usr/local/petsc-32/lib/libpetsc.so.3.7 #12 0x00007ffff578ea63 in KSPSolve () from /usr/local/petsc-32/lib/libpetsc.so.3.7 #13 0x00007ffff5746cf4 in PCApply_FieldSplit_Schur () from /usr/local/petsc-32/lib/libpetsc.so.3.7 #14 0x00007ffff5668640 in PCApply () from /usr/local/petsc-32/lib/libpetsc.so.3.7 #15 0x00007ffff580ab2d in KSPFGMRESCycle () from /usr/local/petsc-32/lib/libpetsc.so.3.7 #16 0x00007ffff580b900 in KSPSolve_FGMRES () from /usr/local/petsc-32/lib/libpetsc.so.3.7 #17 0x00007ffff578ea63 in KSPSolve () from /usr/local/petsc-32/lib/libpetsc.so.3.7 #18 0x00007ffff6597fbf in __pyx_pf_8petsc4py_5PETSc_3KSP_98solve (__pyx_v_self=0x7fffe4f59830, __pyx_v_b=, __pyx_v_x=) at src/petsc4py.PETSc.c:153555 [...] Is this where it tries to perform the matrix multiplication `Bdiv.matMult(Bgrad)`? When instead of ML I use LU in the PC, -fieldsplit_1_lsc_ksp_type preonly -fieldsplit_1_lsc_pc_type lu I get a "wrong argument" error: File "StokesPC/stokespc/stokes_bench.py", line 242, in solve_petsc ksp.solve(b.vec(), x) File "PETSc/KSP.pyx", line 384, in petsc4py.PETSc.KSP.solve (src/petsc4py.PETSc.c:153555) petsc4py.PETSc.Error: error code 62 Regards, David On 02/21/2017 02:20 PM, Barry Smith wrote: > You'll have to figure out what is triggering the segmentation violation. If it is the python that is crashing then likely you can run the entire python program in the debugger and then when it crashes you should be able to see where. > > Barry > > >> On Feb 21, 2017, at 10:07 AM, David Nolte wrote: >> >> Dear all, >> >> new to PETSc, I am trying to use the LSC preconditioner for a Stokes >> problem (discretized by means of stable FEM). I use the python backend >> petsc4py. >> The "automatic" version of the LSC seems to work with the following >> setup (I think adapted from Matt's tutorial slides): >> >> -ksp_view >> -ksp_converged_reason >> -ksp_monitor_true_residual >> -ksp_type fgmres >> -ksp_rtol 1.0e-8 >> >> -pc_type fieldsplit >> -pc_fieldsplit_detect_saddle_point >> -pc_fieldsplit_type schur >> -pc_fieldsplit_schur_fact_type upper >> -pc_fieldsplit_schur_precondition self >> >> -fieldsplit_0_ksp_type preonly >> -fieldsplit_0_pc_type ml >> >> -fieldsplit_1_ksp_type preonly >> -fieldsplit_1_pc_type lsc >> -fieldsplit_1_lsc_pc_type ml >> -fieldsplit_1_lsc_ksp_type preonly >> >> In a 3D setup with 250k dofs this converges within 78 iterations. (For >> reference, upper Schur factorization with ML for the uu-block and Sp = >> diag(Q), the diagonal of the pressure mass matrix, takes 41 iterations >> and half of the computation time.) >> >> Now I just wanted to check if I can get the same result by building the >> L-matrix manually with the following piece of python code, where is0, >> is1 are the index sets corresponding to the velocity and pressure dofs, >> and A is full the system matrix. >> >> Sp = Sp.getSubMatrix(is1, is1) >> pc.setFieldSplitSchurPreType(PETSc.PC.SchurPreType.USER, Sp) >> # Sp.setType(PETSc.Mat.Type.SCHURCOMPLEMENT) # necessary? >> # extract A10 block >> Bdiv = A.getSubMatrix(is1, is0) >> # extract A01 block >> Bgrad = A.getSubMatrix(is0, is1) >> L = Bdiv >> L.matMult(Bgrad) >> Sp.compose('LSC_L', L) >> Sp.compose('LSC_Lp', L) >> >> To my understanding, this should behave similarly to what the LSC >> preconditioner does when LSC_L is not given. However, I get a >> segmentation fault during the first iteration: >> >> 0 KSP unpreconditioned resid norm 2.963704216563e+01 true resid norm >> 2.963704216563e+01 ||r(i)||/||b|| 1.000000000000e+00 >> [1] 2311 segmentation fault (core dumped) python >> StokesPC/stokespc/stokes_bench.py >> >> What am I doing wrong? I appreciate any hints, thanks a lot in advance! >> >> Regards, >> David >> >> >> PS: The log trace is: >> 0 KSP unpreconditioned resid norm 2.963704216563e+01 true resid norm >> 2.963704216563e+01 ||r(i)||/||b|| 1.000000000000e+00 >> [0] 10.0543 Event begin: VecScale >> [0] 10.0545 Event end: VecScale >> [0] PCSetUp(): Leaving PC with identical preconditioner since operator >> is unchanged >> [0] 10.0545 Event begin: PCApply >> [0] 10.0545 Event begin: VecScatterBegin >> [0] 10.0546 Event end: VecScatterBegin >> [0] 10.0546 Event begin: KSPSolve_FS_Schu >> [0] 10.0546 Event begin: KSPSetUp >> [0] 10.0546 Event end: KSPSetUp >> [0] PCSetUp(): Setting up PC for first time >> [0] 10.0546 Event begin: PCSetUp >> [0] 10.0547 Event begin: VecSet >> [0] 10.055 Event end: VecSet >> [0] 10.055 Event begin: VecSet >> [0] 10.0553 Event end: VecSet >> [0] 10.0553 Event begin: VecSet >> [0] 10.0553 Event end: VecSet >> [0] 10.0554 Event end: PCSetUp >> [0] 10.0554 Event begin: VecSet >> [0] 10.0554 Event end: VecSet >> [0] PCSetUp(): Leaving PC with identical preconditioner since operator >> is unchanged >> [0] 10.0554 Event begin: KSPSetUp >> [0] 10.0554 Event end: KSPSetUp >> [0] PCSetUp(): Setting up PC for first time >> [0] 10.0554 Event begin: PCSetUp >> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 >> -2080374783 >> [0] 10.0555 Event begin: VecSet >> [0] 10.0557 Event end: VecSet >> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 >> -2080374783 >> [0] 10.0557 Event begin: VecSet >> [0] 10.0558 Event end: VecSet >> [0] 10.082 Event begin: MatMult >> [0] 10.1273 Event end: MatMult >> [0] 10.1277 Event begin: MatMult >> [0] 10.1739 Event end: MatMult >> [0] 10.1742 Event begin: MatMult >> [0] 10.2195 Event end: MatMult >> [0] 10.2199 Event begin: MatMult >> [0] 10.2653 Event end: MatMult >> [0] 10.2657 Event begin: MatMult >> [0] 10.3113 Event end: MatMult >> [0] 10.3116 Event begin: MatMult >> [0] 10.3571 Event end: MatMult >> [0] 10.3575 Event begin: MatMult >> [0] 10.403 Event end: MatMult >> [0] 10.4033 Event begin: MatMult >> [0] 10.4487 Event end: MatMult >> [0] 10.4491 Event begin: MatMult >> [0] 10.4947 Event end: MatMult >> [0] 10.495 Event begin: MatMult >> [0] 10.5406 Event end: MatMult >> >> >> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: From bsmith at mcs.anl.gov Tue Feb 21 13:09:31 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 21 Feb 2017 13:09:31 -0600 Subject: [petsc-users] understanding the LSC preconditioner In-Reply-To: <123a2c95-0c1b-31fb-858b-2c4ddc3e2284@dim.uchile.cl> References: <74e410b0-c64d-dccf-a653-c0505fb16dd6@dim.uchile.cl> <123a2c95-0c1b-31fb-858b-2c4ddc3e2284@dim.uchile.cl> Message-ID: Hmm, it is crashing inside trying to build the ML preconditioner. This is certainly not expected, could be some data got corrupted earlier. I'm not sure how to track it down, maybe run the python under valgrind? Barry > On Feb 21, 2017, at 1:02 PM, David Nolte wrote: > > It crashes in ML_matmat_mult(): > > Program received signal SIGSEGV, Segmentation > fault. > > 0x00007ffff5cdaeb0 in ML_matmat_mult () from > /usr/local/petsc-32/lib/libpetsc.so.3.7 > > (gdb) bt > #0 0x00007ffff5cdaeb0 in ML_matmat_mult () from > /usr/local/petsc-32/lib/libpetsc.so.3.7 > > #1 0x00007ffff5cdbf76 in ML_2matmult () from > /usr/local/petsc-32/lib/libpetsc.so.3.7 > > #2 0x00007ffff5ca197a in ML_AGG_Gen_Prolongator () from > /usr/local/petsc-32/lib/libpetsc.so.3.7 > > #3 0x00007ffff5c9fa71 in ML_Gen_MultiLevelHierarchy () from > /usr/local/petsc-32/lib/libpetsc.so.3.7 > > #4 0x00007ffff5ca0484 in ML_Gen_MultiLevelHierarchy_UsingAggregation () > from > /usr/local/petsc-32/lib/libpetsc.so.3.7 > > #5 0x00007ffff5780aff in PCSetUp_ML () from > /usr/local/petsc-32/lib/libpetsc.so.3.7 > > #6 0x00007ffff5667fce in PCSetUp () from > /usr/local/petsc-32/lib/libpetsc.so.3.7 > > #7 0x00007ffff578d9f8 in KSPSetUp () from > /usr/local/petsc-32/lib/libpetsc.so.3.7 > > #8 0x00007ffff578e7d8 in KSPSolve () from > /usr/local/petsc-32/lib/libpetsc.so.3.7 > > #9 0x00007ffff578606b in PCApply_LSC () from > /usr/local/petsc-32/lib/libpetsc.so.3.7 > > #10 0x00007ffff5668640 in PCApply () from > /usr/local/petsc-32/lib/libpetsc.so.3.7 > > #11 0x00007ffff57f05a5 in KSPSolve_PREONLY () from > /usr/local/petsc-32/lib/libpetsc.so.3.7 > #12 0x00007ffff578ea63 in KSPSolve () from > /usr/local/petsc-32/lib/libpetsc.so.3.7 > #13 0x00007ffff5746cf4 in PCApply_FieldSplit_Schur () from > /usr/local/petsc-32/lib/libpetsc.so.3.7 > #14 0x00007ffff5668640 in PCApply () from > /usr/local/petsc-32/lib/libpetsc.so.3.7 > #15 0x00007ffff580ab2d in KSPFGMRESCycle () from > /usr/local/petsc-32/lib/libpetsc.so.3.7 > #16 0x00007ffff580b900 in KSPSolve_FGMRES () from > /usr/local/petsc-32/lib/libpetsc.so.3.7 > #17 0x00007ffff578ea63 in KSPSolve () from > /usr/local/petsc-32/lib/libpetsc.so.3.7 > #18 0x00007ffff6597fbf in __pyx_pf_8petsc4py_5PETSc_3KSP_98solve > (__pyx_v_self=0x7fffe4f59830, __pyx_v_b=, > __pyx_v_x=) > at src/petsc4py.PETSc.c:153555 > [...] > > Is this where it tries to perform the matrix multiplication > `Bdiv.matMult(Bgrad)`? > > When instead of ML I use LU in the PC, > > -fieldsplit_1_lsc_ksp_type preonly > -fieldsplit_1_lsc_pc_type lu > > I get a "wrong argument" error: > > File "StokesPC/stokespc/stokes_bench.py", line 242, in > solve_petsc > ksp.solve(b.vec(), x) > File "PETSc/KSP.pyx", line 384, in petsc4py.PETSc.KSP.solve > (src/petsc4py.PETSc.c:153555) > petsc4py.PETSc.Error: error code 62 > > > Regards, > David > > > On 02/21/2017 02:20 PM, Barry Smith wrote: >> You'll have to figure out what is triggering the segmentation violation. If it is the python that is crashing then likely you can run the entire python program in the debugger and then when it crashes you should be able to see where. >> >> Barry >> >> >>> On Feb 21, 2017, at 10:07 AM, David Nolte wrote: >>> >>> Dear all, >>> >>> new to PETSc, I am trying to use the LSC preconditioner for a Stokes >>> problem (discretized by means of stable FEM). I use the python backend >>> petsc4py. >>> The "automatic" version of the LSC seems to work with the following >>> setup (I think adapted from Matt's tutorial slides): >>> >>> -ksp_view >>> -ksp_converged_reason >>> -ksp_monitor_true_residual >>> -ksp_type fgmres >>> -ksp_rtol 1.0e-8 >>> >>> -pc_type fieldsplit >>> -pc_fieldsplit_detect_saddle_point >>> -pc_fieldsplit_type schur >>> -pc_fieldsplit_schur_fact_type upper >>> -pc_fieldsplit_schur_precondition self >>> >>> -fieldsplit_0_ksp_type preonly >>> -fieldsplit_0_pc_type ml >>> >>> -fieldsplit_1_ksp_type preonly >>> -fieldsplit_1_pc_type lsc >>> -fieldsplit_1_lsc_pc_type ml >>> -fieldsplit_1_lsc_ksp_type preonly >>> >>> In a 3D setup with 250k dofs this converges within 78 iterations. (For >>> reference, upper Schur factorization with ML for the uu-block and Sp = >>> diag(Q), the diagonal of the pressure mass matrix, takes 41 iterations >>> and half of the computation time.) >>> >>> Now I just wanted to check if I can get the same result by building the >>> L-matrix manually with the following piece of python code, where is0, >>> is1 are the index sets corresponding to the velocity and pressure dofs, >>> and A is full the system matrix. >>> >>> Sp = Sp.getSubMatrix(is1, is1) >>> pc.setFieldSplitSchurPreType(PETSc.PC.SchurPreType.USER, Sp) >>> # Sp.setType(PETSc.Mat.Type.SCHURCOMPLEMENT) # necessary? >>> # extract A10 block >>> Bdiv = A.getSubMatrix(is1, is0) >>> # extract A01 block >>> Bgrad = A.getSubMatrix(is0, is1) >>> L = Bdiv >>> L.matMult(Bgrad) >>> Sp.compose('LSC_L', L) >>> Sp.compose('LSC_Lp', L) >>> >>> To my understanding, this should behave similarly to what the LSC >>> preconditioner does when LSC_L is not given. However, I get a >>> segmentation fault during the first iteration: >>> >>> 0 KSP unpreconditioned resid norm 2.963704216563e+01 true resid norm >>> 2.963704216563e+01 ||r(i)||/||b|| 1.000000000000e+00 >>> [1] 2311 segmentation fault (core dumped) python >>> StokesPC/stokespc/stokes_bench.py >>> >>> What am I doing wrong? I appreciate any hints, thanks a lot in advance! >>> >>> Regards, >>> David >>> >>> >>> PS: The log trace is: >>> 0 KSP unpreconditioned resid norm 2.963704216563e+01 true resid norm >>> 2.963704216563e+01 ||r(i)||/||b|| 1.000000000000e+00 >>> [0] 10.0543 Event begin: VecScale >>> [0] 10.0545 Event end: VecScale >>> [0] PCSetUp(): Leaving PC with identical preconditioner since operator >>> is unchanged >>> [0] 10.0545 Event begin: PCApply >>> [0] 10.0545 Event begin: VecScatterBegin >>> [0] 10.0546 Event end: VecScatterBegin >>> [0] 10.0546 Event begin: KSPSolve_FS_Schu >>> [0] 10.0546 Event begin: KSPSetUp >>> [0] 10.0546 Event end: KSPSetUp >>> [0] PCSetUp(): Setting up PC for first time >>> [0] 10.0546 Event begin: PCSetUp >>> [0] 10.0547 Event begin: VecSet >>> [0] 10.055 Event end: VecSet >>> [0] 10.055 Event begin: VecSet >>> [0] 10.0553 Event end: VecSet >>> [0] 10.0553 Event begin: VecSet >>> [0] 10.0553 Event end: VecSet >>> [0] 10.0554 Event end: PCSetUp >>> [0] 10.0554 Event begin: VecSet >>> [0] 10.0554 Event end: VecSet >>> [0] PCSetUp(): Leaving PC with identical preconditioner since operator >>> is unchanged >>> [0] 10.0554 Event begin: KSPSetUp >>> [0] 10.0554 Event end: KSPSetUp >>> [0] PCSetUp(): Setting up PC for first time >>> [0] 10.0554 Event begin: PCSetUp >>> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 >>> -2080374783 >>> [0] 10.0555 Event begin: VecSet >>> [0] 10.0557 Event end: VecSet >>> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 >>> -2080374783 >>> [0] 10.0557 Event begin: VecSet >>> [0] 10.0558 Event end: VecSet >>> [0] 10.082 Event begin: MatMult >>> [0] 10.1273 Event end: MatMult >>> [0] 10.1277 Event begin: MatMult >>> [0] 10.1739 Event end: MatMult >>> [0] 10.1742 Event begin: MatMult >>> [0] 10.2195 Event end: MatMult >>> [0] 10.2199 Event begin: MatMult >>> [0] 10.2653 Event end: MatMult >>> [0] 10.2657 Event begin: MatMult >>> [0] 10.3113 Event end: MatMult >>> [0] 10.3116 Event begin: MatMult >>> [0] 10.3571 Event end: MatMult >>> [0] 10.3575 Event begin: MatMult >>> [0] 10.403 Event end: MatMult >>> [0] 10.4033 Event begin: MatMult >>> [0] 10.4487 Event end: MatMult >>> [0] 10.4491 Event begin: MatMult >>> [0] 10.4947 Event end: MatMult >>> [0] 10.495 Event begin: MatMult >>> [0] 10.5406 Event end: MatMult >>> >>> >>> > > From matt.landreman at gmail.com Wed Feb 22 04:53:31 2017 From: matt.landreman at gmail.com (Matt Landreman) Date: Wed, 22 Feb 2017 05:53:31 -0500 Subject: [petsc-users] Multigrid with defect correction Message-ID: Hi, PETSc developers, 1. Consider solving a linear problem with -pc_type mg, assembling the matrix using KSPSetComputeOperators(ksp,ComputeMatrix,&user) and MatSetValuesStencil as in ksp ex25.c. I?d like Pmat to be different than Amat (a stable low-order discretization and a high-order discretization, respectively.) However the two Mat arguments that PETSc passes to ComputeMatrix appear to be the same object, so Pmat and Amat are forced to be the same. (For example in ex25, J==jac). How do I allow Amat and Pmat to differ? 2. I suppose if the above issue were resolved, the resulting gmres/mg iteration would amount to doing defect correction on only the finest grid. Another defect correction method is to use a different Mat for smoothing and residual computation at every level. (Stable low-order for smoothing, high-order for residual.) This latter approach is recommended on page 103 of Brandt?s multigrid guide ( http://www.wisdom.weizmann.ac.il/~achi/classics.pdf). What would be the best way to do this second kind of defect correction in PETSc mg? Perhaps loop over every multigrid level and call PCMGSetResidual(pc,l,PCMGResidualDefault,Amat) on each level, after calling ComputeMatrix to build the appropriately-sized Amat? 3. Is it at all sensible to do this second kind of defect correction with _algebraic_ multigrid? Perhaps Amat for each level could be formed from the high-order matrix at the fine level by the Galerkin operator R A P, after getting all the restriction matrices created by gamg for Pmat? Thanks, Matt Landreman -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Wed Feb 22 05:51:41 2017 From: jroman at dsic.upv.es (Jose E. Roman) Date: Wed, 22 Feb 2017 12:51:41 +0100 Subject: [petsc-users] Krylov-Schur Tolerance In-Reply-To: References: <4691ae05-be89-5bdb-33ac-2fd5cef4c978@wpi.edu> <78DC3258-D0BB-49A9-9246-D15F6E82A9D0@dsic.upv.es> <18B35373-BFB8-4307-B7E4-3D2C49B699DE@dsic.upv.es> <7ee69de1-511c-6dba-58b1-4a330e1beb96@wpi.edu> Message-ID: > El 20 feb 2017, a las 11:33, Jose E. Roman escribi?: > > >> El 20 feb 2017, a las 11:01, Christopher Pierce escribi?: >> >> It seems to give the same results. I exported the matrices to Matlab >> and checked the estimated condition number of the B matrix which came to >> ~15 and the 2-norm of the B matrix which was ~10^6. I'm guessing that >> the large matrix norm is the problem. I glanced over the source for the >> RQCG solver and it doesn't seem to use a linear solver which is likely >> why it showed better performance. Do you have any suggestions for >> dealing with problems like this? >> >> Chris > > Send the data files to my personal email and I will make some tests. > Jose > The large errors that you report can be explained by the fact that both A and B have large norm, in particular ~10^11 and ~10^5, respectively. Note that the norm of the residual ||A*x-lambda*B*X|| can be made arbitrarily large by increasing the matrix norms. So if you scale the matrices so that e.g. norm(B)=1, you will see a reduction of the reported error in several orders of magnitude. On the other hand, note that in case of solving the problem as a GHEP the returned eigenvector is not normalized to have ||x||_2=1, but to have ||x||_B=1, so when showing the relative residual error, SLEPc should divide by ||x||_2, which is quite small in your case. This can be considered a bug in SLEPc that we will fix for the next release. Doing this in your case would imply increasing a bit the reported errors (much less than the gain from the first comment). In conclusion, you can rely on the computed solution even though the reported residual norms are large in this case. Jose From dnolte at dim.uchile.cl Wed Feb 22 08:03:30 2017 From: dnolte at dim.uchile.cl (David Nolte) Date: Wed, 22 Feb 2017 11:03:30 -0300 Subject: [petsc-users] understanding the LSC preconditioner In-Reply-To: References: <74e410b0-c64d-dccf-a653-c0505fb16dd6@dim.uchile.cl> <123a2c95-0c1b-31fb-858b-2c4ddc3e2284@dim.uchile.cl> Message-ID: <5e68f816-3331-c7ba-e1fc-6f79ffa3c7e8@dim.uchile.cl> Hm, I thought it could be related to my construction of the L matrix, since for example LU -fieldsplit_1_lsc_ksp_type preonly -fieldsplit_1_lsc_pc_type lu doesn't work either: 0 KSP unpreconditioned resid norm 2.963704216563e+01 true resid norm 2.963704216563e+01 ||r(i)||/||b|| 1.000000000000e+00 [0] 14.2691 Event begin: VecScale [0] 14.2693 Event end: VecScale [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] 14.2693 Event begin: PCApply [0] 14.2693 Event begin: VecScatterBegin [0] 14.2693 Event end: VecScatterBegin [0] 14.2693 Event begin: KSPSolve_FS_Schu [0] 14.2693 Event begin: KSPSetUp [0] 14.2694 Event end: KSPSetUp [0] PCSetUp(): Setting up PC for first time [0] 14.2694 Event begin: PCSetUp [0] 14.2694 Event begin: VecSet [0] 14.2696 Event end: VecSet [0] 14.2696 Event begin: VecSet [0] 14.2698 Event end: VecSet [0] 14.2699 Event begin: VecSet [0] 14.2699 Event end: VecSet [0] 14.2699 Event end: PCSetUp [0] 14.2699 Event begin: VecSet [0] 14.27 Event end: VecSet [0] PCSetUp(): Leaving PC with identical preconditioner since operator is unchanged [0] 14.27 Event begin: KSPSetUp [0] 14.27 Event end: KSPSetUp [0] PCSetUp(): Setting up PC for first time [0] 14.27 Event begin: PCSetUp Traceback (most recent call last): File "StokesPC/stokespc/stokes_bench.py", line 331, in plot_residuals=plot_residuals) File "StokesPC/stokespc/stokes_bench.py", line 291, in run_bench compare_LU=compare_LU, results_dir=results_dir) File "StokesPC/stokespc/stokes_bench.py", line 136, in stokes_bench inpfile, P, Sp, W) File "StokesPC/stokespc/stokes_bench.py", line 244, in solve_petsc ksp.solve(b, x) File "PETSc/KSP.pyx", line 384, in petsc4py.PETSc.KSP.solve (src/petsc4py.PETSc.c:153555) petsc4py.PETSc.Error: error code 62 Aborted (core dumped) David On 02/21/2017 04:09 PM, Barry Smith wrote: > Hmm, it is crashing inside trying to build the ML preconditioner. This is certainly not expected, could be some data got corrupted earlier. I'm not sure how to track it down, maybe run the python under valgrind? > > Barry > > >> On Feb 21, 2017, at 1:02 PM, David Nolte wrote: >> >> It crashes in ML_matmat_mult(): >> >> Program received signal SIGSEGV, Segmentation >> fault. >> >> 0x00007ffff5cdaeb0 in ML_matmat_mult () from >> /usr/local/petsc-32/lib/libpetsc.so.3.7 >> >> (gdb) bt >> #0 0x00007ffff5cdaeb0 in ML_matmat_mult () from >> /usr/local/petsc-32/lib/libpetsc.so.3.7 >> >> #1 0x00007ffff5cdbf76 in ML_2matmult () from >> /usr/local/petsc-32/lib/libpetsc.so.3.7 >> >> #2 0x00007ffff5ca197a in ML_AGG_Gen_Prolongator () from >> /usr/local/petsc-32/lib/libpetsc.so.3.7 >> >> #3 0x00007ffff5c9fa71 in ML_Gen_MultiLevelHierarchy () from >> /usr/local/petsc-32/lib/libpetsc.so.3.7 >> >> #4 0x00007ffff5ca0484 in ML_Gen_MultiLevelHierarchy_UsingAggregation () >> from >> /usr/local/petsc-32/lib/libpetsc.so.3.7 >> >> #5 0x00007ffff5780aff in PCSetUp_ML () from >> /usr/local/petsc-32/lib/libpetsc.so.3.7 >> >> #6 0x00007ffff5667fce in PCSetUp () from >> /usr/local/petsc-32/lib/libpetsc.so.3.7 >> >> #7 0x00007ffff578d9f8 in KSPSetUp () from >> /usr/local/petsc-32/lib/libpetsc.so.3.7 >> >> #8 0x00007ffff578e7d8 in KSPSolve () from >> /usr/local/petsc-32/lib/libpetsc.so.3.7 >> >> #9 0x00007ffff578606b in PCApply_LSC () from >> /usr/local/petsc-32/lib/libpetsc.so.3.7 >> >> #10 0x00007ffff5668640 in PCApply () from >> /usr/local/petsc-32/lib/libpetsc.so.3.7 >> >> #11 0x00007ffff57f05a5 in KSPSolve_PREONLY () from >> /usr/local/petsc-32/lib/libpetsc.so.3.7 >> #12 0x00007ffff578ea63 in KSPSolve () from >> /usr/local/petsc-32/lib/libpetsc.so.3.7 >> #13 0x00007ffff5746cf4 in PCApply_FieldSplit_Schur () from >> /usr/local/petsc-32/lib/libpetsc.so.3.7 >> #14 0x00007ffff5668640 in PCApply () from >> /usr/local/petsc-32/lib/libpetsc.so.3.7 >> #15 0x00007ffff580ab2d in KSPFGMRESCycle () from >> /usr/local/petsc-32/lib/libpetsc.so.3.7 >> #16 0x00007ffff580b900 in KSPSolve_FGMRES () from >> /usr/local/petsc-32/lib/libpetsc.so.3.7 >> #17 0x00007ffff578ea63 in KSPSolve () from >> /usr/local/petsc-32/lib/libpetsc.so.3.7 >> #18 0x00007ffff6597fbf in __pyx_pf_8petsc4py_5PETSc_3KSP_98solve >> (__pyx_v_self=0x7fffe4f59830, __pyx_v_b=, >> __pyx_v_x=) >> at src/petsc4py.PETSc.c:153555 >> [...] >> >> Is this where it tries to perform the matrix multiplication >> `Bdiv.matMult(Bgrad)`? >> >> When instead of ML I use LU in the PC, >> >> -fieldsplit_1_lsc_ksp_type preonly >> -fieldsplit_1_lsc_pc_type lu >> >> I get a "wrong argument" error: >> >> File "StokesPC/stokespc/stokes_bench.py", line 242, in >> solve_petsc >> ksp.solve(b.vec(), x) >> File "PETSc/KSP.pyx", line 384, in petsc4py.PETSc.KSP.solve >> (src/petsc4py.PETSc.c:153555) >> petsc4py.PETSc.Error: error code 62 >> >> >> Regards, >> David >> >> >> On 02/21/2017 02:20 PM, Barry Smith wrote: >>> You'll have to figure out what is triggering the segmentation violation. If it is the python that is crashing then likely you can run the entire python program in the debugger and then when it crashes you should be able to see where. >>> >>> Barry >>> >>> >>>> On Feb 21, 2017, at 10:07 AM, David Nolte wrote: >>>> >>>> Dear all, >>>> >>>> new to PETSc, I am trying to use the LSC preconditioner for a Stokes >>>> problem (discretized by means of stable FEM). I use the python backend >>>> petsc4py. >>>> The "automatic" version of the LSC seems to work with the following >>>> setup (I think adapted from Matt's tutorial slides): >>>> >>>> -ksp_view >>>> -ksp_converged_reason >>>> -ksp_monitor_true_residual >>>> -ksp_type fgmres >>>> -ksp_rtol 1.0e-8 >>>> >>>> -pc_type fieldsplit >>>> -pc_fieldsplit_detect_saddle_point >>>> -pc_fieldsplit_type schur >>>> -pc_fieldsplit_schur_fact_type upper >>>> -pc_fieldsplit_schur_precondition self >>>> >>>> -fieldsplit_0_ksp_type preonly >>>> -fieldsplit_0_pc_type ml >>>> >>>> -fieldsplit_1_ksp_type preonly >>>> -fieldsplit_1_pc_type lsc >>>> -fieldsplit_1_lsc_pc_type ml >>>> -fieldsplit_1_lsc_ksp_type preonly >>>> >>>> In a 3D setup with 250k dofs this converges within 78 iterations. (For >>>> reference, upper Schur factorization with ML for the uu-block and Sp = >>>> diag(Q), the diagonal of the pressure mass matrix, takes 41 iterations >>>> and half of the computation time.) >>>> >>>> Now I just wanted to check if I can get the same result by building the >>>> L-matrix manually with the following piece of python code, where is0, >>>> is1 are the index sets corresponding to the velocity and pressure dofs, >>>> and A is full the system matrix. >>>> >>>> Sp = Sp.getSubMatrix(is1, is1) >>>> pc.setFieldSplitSchurPreType(PETSc.PC.SchurPreType.USER, Sp) >>>> # Sp.setType(PETSc.Mat.Type.SCHURCOMPLEMENT) # necessary? >>>> # extract A10 block >>>> Bdiv = A.getSubMatrix(is1, is0) >>>> # extract A01 block >>>> Bgrad = A.getSubMatrix(is0, is1) >>>> L = Bdiv >>>> L.matMult(Bgrad) >>>> Sp.compose('LSC_L', L) >>>> Sp.compose('LSC_Lp', L) >>>> >>>> To my understanding, this should behave similarly to what the LSC >>>> preconditioner does when LSC_L is not given. However, I get a >>>> segmentation fault during the first iteration: >>>> >>>> 0 KSP unpreconditioned resid norm 2.963704216563e+01 true resid norm >>>> 2.963704216563e+01 ||r(i)||/||b|| 1.000000000000e+00 >>>> [1] 2311 segmentation fault (core dumped) python >>>> StokesPC/stokespc/stokes_bench.py >>>> >>>> What am I doing wrong? I appreciate any hints, thanks a lot in advance! >>>> >>>> Regards, >>>> David >>>> >>>> >>>> PS: The log trace is: >>>> 0 KSP unpreconditioned resid norm 2.963704216563e+01 true resid norm >>>> 2.963704216563e+01 ||r(i)||/||b|| 1.000000000000e+00 >>>> [0] 10.0543 Event begin: VecScale >>>> [0] 10.0545 Event end: VecScale >>>> [0] PCSetUp(): Leaving PC with identical preconditioner since operator >>>> is unchanged >>>> [0] 10.0545 Event begin: PCApply >>>> [0] 10.0545 Event begin: VecScatterBegin >>>> [0] 10.0546 Event end: VecScatterBegin >>>> [0] 10.0546 Event begin: KSPSolve_FS_Schu >>>> [0] 10.0546 Event begin: KSPSetUp >>>> [0] 10.0546 Event end: KSPSetUp >>>> [0] PCSetUp(): Setting up PC for first time >>>> [0] 10.0546 Event begin: PCSetUp >>>> [0] 10.0547 Event begin: VecSet >>>> [0] 10.055 Event end: VecSet >>>> [0] 10.055 Event begin: VecSet >>>> [0] 10.0553 Event end: VecSet >>>> [0] 10.0553 Event begin: VecSet >>>> [0] 10.0553 Event end: VecSet >>>> [0] 10.0554 Event end: PCSetUp >>>> [0] 10.0554 Event begin: VecSet >>>> [0] 10.0554 Event end: VecSet >>>> [0] PCSetUp(): Leaving PC with identical preconditioner since operator >>>> is unchanged >>>> [0] 10.0554 Event begin: KSPSetUp >>>> [0] 10.0554 Event end: KSPSetUp >>>> [0] PCSetUp(): Setting up PC for first time >>>> [0] 10.0554 Event begin: PCSetUp >>>> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 >>>> -2080374783 >>>> [0] 10.0555 Event begin: VecSet >>>> [0] 10.0557 Event end: VecSet >>>> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 >>>> -2080374783 >>>> [0] 10.0557 Event begin: VecSet >>>> [0] 10.0558 Event end: VecSet >>>> [0] 10.082 Event begin: MatMult >>>> [0] 10.1273 Event end: MatMult >>>> [0] 10.1277 Event begin: MatMult >>>> [0] 10.1739 Event end: MatMult >>>> [0] 10.1742 Event begin: MatMult >>>> [0] 10.2195 Event end: MatMult >>>> [0] 10.2199 Event begin: MatMult >>>> [0] 10.2653 Event end: MatMult >>>> [0] 10.2657 Event begin: MatMult >>>> [0] 10.3113 Event end: MatMult >>>> [0] 10.3116 Event begin: MatMult >>>> [0] 10.3571 Event end: MatMult >>>> [0] 10.3575 Event begin: MatMult >>>> [0] 10.403 Event end: MatMult >>>> [0] 10.4033 Event begin: MatMult >>>> [0] 10.4487 Event end: MatMult >>>> [0] 10.4491 Event begin: MatMult >>>> [0] 10.4947 Event end: MatMult >>>> [0] 10.495 Event begin: MatMult >>>> [0] 10.5406 Event end: MatMult >>>> >>>> >>>> >> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: From bsmith at mcs.anl.gov Wed Feb 22 13:30:19 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 22 Feb 2017 13:30:19 -0600 Subject: [petsc-users] understanding the LSC preconditioner In-Reply-To: <5e68f816-3331-c7ba-e1fc-6f79ffa3c7e8@dim.uchile.cl> References: <74e410b0-c64d-dccf-a653-c0505fb16dd6@dim.uchile.cl> <123a2c95-0c1b-31fb-858b-2c4ddc3e2284@dim.uchile.cl> <5e68f816-3331-c7ba-e1fc-6f79ffa3c7e8@dim.uchile.cl> Message-ID: > On Feb 22, 2017, at 8:03 AM, David Nolte wrote: > > Hm, I thought it could be related to my construction of the L matrix, > since for example LU > -fieldsplit_1_lsc_ksp_type preonly > -fieldsplit_1_lsc_pc_type lu > > doesn't work either: Yes, it could be. But try with valgrind > > 0 KSP unpreconditioned resid norm 2.963704216563e+01 true resid norm > 2.963704216563e+01 ||r(i)||/||b|| 1.000000000000e+00 > [0] 14.2691 Event begin: VecScale > [0] 14.2693 Event end: VecScale > [0] PCSetUp(): Leaving PC with identical preconditioner since operator > is unchanged > [0] 14.2693 Event begin: PCApply > [0] 14.2693 Event begin: VecScatterBegin > [0] 14.2693 Event end: VecScatterBegin > [0] 14.2693 Event begin: KSPSolve_FS_Schu > [0] 14.2693 Event begin: KSPSetUp > [0] 14.2694 Event end: KSPSetUp > [0] PCSetUp(): Setting up PC for first time > [0] 14.2694 Event begin: PCSetUp > [0] 14.2694 Event begin: VecSet > [0] 14.2696 Event end: VecSet > [0] 14.2696 Event begin: VecSet > [0] 14.2698 Event end: VecSet > [0] 14.2699 Event begin: VecSet > [0] 14.2699 Event end: VecSet > [0] 14.2699 Event end: PCSetUp > [0] 14.2699 Event begin: VecSet > [0] 14.27 Event end: VecSet > [0] PCSetUp(): Leaving PC with identical preconditioner since operator > is unchanged > [0] 14.27 Event begin: KSPSetUp > [0] 14.27 Event end: KSPSetUp > [0] PCSetUp(): Setting up PC for first time > [0] 14.27 Event begin: PCSetUp > Traceback (most recent call last): > File "StokesPC/stokespc/stokes_bench.py", line 331, in > plot_residuals=plot_residuals) > File "StokesPC/stokespc/stokes_bench.py", line 291, in run_bench > compare_LU=compare_LU, results_dir=results_dir) > File "StokesPC/stokespc/stokes_bench.py", line 136, in stokes_bench > inpfile, P, Sp, W) > File "StokesPC/stokespc/stokes_bench.py", line 244, in solve_petsc > ksp.solve(b, x) > File "PETSc/KSP.pyx", line 384, in petsc4py.PETSc.KSP.solve > (src/petsc4py.PETSc.c:153555) > petsc4py.PETSc.Error: error code 62 > Aborted (core dumped) > > > David > > > On 02/21/2017 04:09 PM, Barry Smith wrote: >> Hmm, it is crashing inside trying to build the ML preconditioner. This is certainly not expected, could be some data got corrupted earlier. I'm not sure how to track it down, maybe run the python under valgrind? >> >> Barry >> >> >>> On Feb 21, 2017, at 1:02 PM, David Nolte wrote: >>> >>> It crashes in ML_matmat_mult(): >>> >>> Program received signal SIGSEGV, Segmentation >>> fault. >>> >>> 0x00007ffff5cdaeb0 in ML_matmat_mult () from >>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>> >>> (gdb) bt >>> #0 0x00007ffff5cdaeb0 in ML_matmat_mult () from >>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>> >>> #1 0x00007ffff5cdbf76 in ML_2matmult () from >>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>> >>> #2 0x00007ffff5ca197a in ML_AGG_Gen_Prolongator () from >>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>> >>> #3 0x00007ffff5c9fa71 in ML_Gen_MultiLevelHierarchy () from >>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>> >>> #4 0x00007ffff5ca0484 in ML_Gen_MultiLevelHierarchy_UsingAggregation () >>> from >>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>> >>> #5 0x00007ffff5780aff in PCSetUp_ML () from >>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>> >>> #6 0x00007ffff5667fce in PCSetUp () from >>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>> >>> #7 0x00007ffff578d9f8 in KSPSetUp () from >>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>> >>> #8 0x00007ffff578e7d8 in KSPSolve () from >>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>> >>> #9 0x00007ffff578606b in PCApply_LSC () from >>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>> >>> #10 0x00007ffff5668640 in PCApply () from >>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>> >>> #11 0x00007ffff57f05a5 in KSPSolve_PREONLY () from >>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>> #12 0x00007ffff578ea63 in KSPSolve () from >>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>> #13 0x00007ffff5746cf4 in PCApply_FieldSplit_Schur () from >>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>> #14 0x00007ffff5668640 in PCApply () from >>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>> #15 0x00007ffff580ab2d in KSPFGMRESCycle () from >>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>> #16 0x00007ffff580b900 in KSPSolve_FGMRES () from >>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>> #17 0x00007ffff578ea63 in KSPSolve () from >>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>> #18 0x00007ffff6597fbf in __pyx_pf_8petsc4py_5PETSc_3KSP_98solve >>> (__pyx_v_self=0x7fffe4f59830, __pyx_v_b=, >>> __pyx_v_x=) >>> at src/petsc4py.PETSc.c:153555 >>> [...] >>> >>> Is this where it tries to perform the matrix multiplication >>> `Bdiv.matMult(Bgrad)`? >>> >>> When instead of ML I use LU in the PC, >>> >>> -fieldsplit_1_lsc_ksp_type preonly >>> -fieldsplit_1_lsc_pc_type lu >>> >>> I get a "wrong argument" error: >>> >>> File "StokesPC/stokespc/stokes_bench.py", line 242, in >>> solve_petsc >>> ksp.solve(b.vec(), x) >>> File "PETSc/KSP.pyx", line 384, in petsc4py.PETSc.KSP.solve >>> (src/petsc4py.PETSc.c:153555) >>> petsc4py.PETSc.Error: error code 62 >>> >>> >>> Regards, >>> David >>> >>> >>> On 02/21/2017 02:20 PM, Barry Smith wrote: >>>> You'll have to figure out what is triggering the segmentation violation. If it is the python that is crashing then likely you can run the entire python program in the debugger and then when it crashes you should be able to see where. >>>> >>>> Barry >>>> >>>> >>>>> On Feb 21, 2017, at 10:07 AM, David Nolte wrote: >>>>> >>>>> Dear all, >>>>> >>>>> new to PETSc, I am trying to use the LSC preconditioner for a Stokes >>>>> problem (discretized by means of stable FEM). I use the python backend >>>>> petsc4py. >>>>> The "automatic" version of the LSC seems to work with the following >>>>> setup (I think adapted from Matt's tutorial slides): >>>>> >>>>> -ksp_view >>>>> -ksp_converged_reason >>>>> -ksp_monitor_true_residual >>>>> -ksp_type fgmres >>>>> -ksp_rtol 1.0e-8 >>>>> >>>>> -pc_type fieldsplit >>>>> -pc_fieldsplit_detect_saddle_point >>>>> -pc_fieldsplit_type schur >>>>> -pc_fieldsplit_schur_fact_type upper >>>>> -pc_fieldsplit_schur_precondition self >>>>> >>>>> -fieldsplit_0_ksp_type preonly >>>>> -fieldsplit_0_pc_type ml >>>>> >>>>> -fieldsplit_1_ksp_type preonly >>>>> -fieldsplit_1_pc_type lsc >>>>> -fieldsplit_1_lsc_pc_type ml >>>>> -fieldsplit_1_lsc_ksp_type preonly >>>>> >>>>> In a 3D setup with 250k dofs this converges within 78 iterations. (For >>>>> reference, upper Schur factorization with ML for the uu-block and Sp = >>>>> diag(Q), the diagonal of the pressure mass matrix, takes 41 iterations >>>>> and half of the computation time.) >>>>> >>>>> Now I just wanted to check if I can get the same result by building the >>>>> L-matrix manually with the following piece of python code, where is0, >>>>> is1 are the index sets corresponding to the velocity and pressure dofs, >>>>> and A is full the system matrix. >>>>> >>>>> Sp = Sp.getSubMatrix(is1, is1) >>>>> pc.setFieldSplitSchurPreType(PETSc.PC.SchurPreType.USER, Sp) >>>>> # Sp.setType(PETSc.Mat.Type.SCHURCOMPLEMENT) # necessary? >>>>> # extract A10 block >>>>> Bdiv = A.getSubMatrix(is1, is0) >>>>> # extract A01 block >>>>> Bgrad = A.getSubMatrix(is0, is1) >>>>> L = Bdiv >>>>> L.matMult(Bgrad) >>>>> Sp.compose('LSC_L', L) >>>>> Sp.compose('LSC_Lp', L) >>>>> >>>>> To my understanding, this should behave similarly to what the LSC >>>>> preconditioner does when LSC_L is not given. However, I get a >>>>> segmentation fault during the first iteration: >>>>> >>>>> 0 KSP unpreconditioned resid norm 2.963704216563e+01 true resid norm >>>>> 2.963704216563e+01 ||r(i)||/||b|| 1.000000000000e+00 >>>>> [1] 2311 segmentation fault (core dumped) python >>>>> StokesPC/stokespc/stokes_bench.py >>>>> >>>>> What am I doing wrong? I appreciate any hints, thanks a lot in advance! >>>>> >>>>> Regards, >>>>> David >>>>> >>>>> >>>>> PS: The log trace is: >>>>> 0 KSP unpreconditioned resid norm 2.963704216563e+01 true resid norm >>>>> 2.963704216563e+01 ||r(i)||/||b|| 1.000000000000e+00 >>>>> [0] 10.0543 Event begin: VecScale >>>>> [0] 10.0545 Event end: VecScale >>>>> [0] PCSetUp(): Leaving PC with identical preconditioner since operator >>>>> is unchanged >>>>> [0] 10.0545 Event begin: PCApply >>>>> [0] 10.0545 Event begin: VecScatterBegin >>>>> [0] 10.0546 Event end: VecScatterBegin >>>>> [0] 10.0546 Event begin: KSPSolve_FS_Schu >>>>> [0] 10.0546 Event begin: KSPSetUp >>>>> [0] 10.0546 Event end: KSPSetUp >>>>> [0] PCSetUp(): Setting up PC for first time >>>>> [0] 10.0546 Event begin: PCSetUp >>>>> [0] 10.0547 Event begin: VecSet >>>>> [0] 10.055 Event end: VecSet >>>>> [0] 10.055 Event begin: VecSet >>>>> [0] 10.0553 Event end: VecSet >>>>> [0] 10.0553 Event begin: VecSet >>>>> [0] 10.0553 Event end: VecSet >>>>> [0] 10.0554 Event end: PCSetUp >>>>> [0] 10.0554 Event begin: VecSet >>>>> [0] 10.0554 Event end: VecSet >>>>> [0] PCSetUp(): Leaving PC with identical preconditioner since operator >>>>> is unchanged >>>>> [0] 10.0554 Event begin: KSPSetUp >>>>> [0] 10.0554 Event end: KSPSetUp >>>>> [0] PCSetUp(): Setting up PC for first time >>>>> [0] 10.0554 Event begin: PCSetUp >>>>> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 >>>>> -2080374783 >>>>> [0] 10.0555 Event begin: VecSet >>>>> [0] 10.0557 Event end: VecSet >>>>> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 >>>>> -2080374783 >>>>> [0] 10.0557 Event begin: VecSet >>>>> [0] 10.0558 Event end: VecSet >>>>> [0] 10.082 Event begin: MatMult >>>>> [0] 10.1273 Event end: MatMult >>>>> [0] 10.1277 Event begin: MatMult >>>>> [0] 10.1739 Event end: MatMult >>>>> [0] 10.1742 Event begin: MatMult >>>>> [0] 10.2195 Event end: MatMult >>>>> [0] 10.2199 Event begin: MatMult >>>>> [0] 10.2653 Event end: MatMult >>>>> [0] 10.2657 Event begin: MatMult >>>>> [0] 10.3113 Event end: MatMult >>>>> [0] 10.3116 Event begin: MatMult >>>>> [0] 10.3571 Event end: MatMult >>>>> [0] 10.3575 Event begin: MatMult >>>>> [0] 10.403 Event end: MatMult >>>>> [0] 10.4033 Event begin: MatMult >>>>> [0] 10.4487 Event end: MatMult >>>>> [0] 10.4491 Event begin: MatMult >>>>> [0] 10.4947 Event end: MatMult >>>>> [0] 10.495 Event begin: MatMult >>>>> [0] 10.5406 Event end: MatMult >>>>> >>>>> >>>>> >>> > > From jswenson at mail.smu.edu Wed Feb 22 16:10:50 2017 From: jswenson at mail.smu.edu (Swenson, Jennifer) Date: Wed, 22 Feb 2017 22:10:50 +0000 Subject: [petsc-users] implementing non-local scalar field Message-ID: Dear PETSc Users, As a part of my problem, I need to be able to look up the value of a scalar field (let's call it xi) that is defined via a somewhat non-trivial integral over a second scalar field (let's call it rho). Mathematically we have [cid:07284a17-07f8-426b-85da-461ebc1fe4b8] What we're modeling here is the average energy loss of an energetic ion that is passing through a solid material. Ions can arrive with a specified direction, so to calculate xi at each point, we have to perform line integrals back toward the ion source, and keep track of how much mass the ion has passed through. In a serial algorithm, one can in principle generate xi by working from the top down, subtracting energy from each row of grid points given information on the density field. However, it is not clear how one could develop a robust parallel algorithm, much less one that works within the PETSc framework. I would very grateful for any help you can offer. Sincerely, Jennifer Swenson -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: equation.png Type: image/png Size: 6066 bytes Desc: equation.png URL: From gideon.simpson at gmail.com Wed Feb 22 20:38:18 2017 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Wed, 22 Feb 2017 21:38:18 -0500 Subject: [petsc-users] make file issues Message-ID: <6B63CEAF-E752-46D3-93D5-26B4CA5E15E2@gmail.com> I?ve been trying to use some code that was originally developed under petsc 3.6.x with my new 3.7.5 installation. I?m having an issue in that the way the code is written, it?s spread across several .c files. Here are the essential features of the makefile: all: myprog1 OBJS = functions.o \ utils.o include ${PETSC_DIR}/lib/petsc/conf/variables include ${PETSC_DIR}/lib/petsc/conf/rules myprog1: prog1.o ${OBJS} ${CLINKER} -o myprog1 myprog1.o ${OBJS} ${PETSC_LIB} make all yields: make: *** No rule to make target `myprog1.o', needed by `myprog1'. Stop -gideon From jed at jedbrown.org Wed Feb 22 21:27:32 2017 From: jed at jedbrown.org (Jed Brown) Date: Wed, 22 Feb 2017 20:27:32 -0700 Subject: [petsc-users] make file issues In-Reply-To: <6B63CEAF-E752-46D3-93D5-26B4CA5E15E2@gmail.com> References: <6B63CEAF-E752-46D3-93D5-26B4CA5E15E2@gmail.com> Message-ID: <87poi9bn8b.fsf@jedbrown.org> Gideon Simpson writes: > I?ve been trying to use some code that was originally developed under petsc 3.6.x with my new 3.7.5 installation. I?m having an issue in that the way the code is written, it?s spread across several .c files. Here are the essential features of the makefile: > > all: myprog1 > > OBJS = functions.o \ > utils.o > > include ${PETSC_DIR}/lib/petsc/conf/variables > include ${PETSC_DIR}/lib/petsc/conf/rules > > myprog1: prog1.o ${OBJS} prog1.o or myprog1.o? You may want to use $< or $^ to avoid duplication. > ${CLINKER} -o myprog1 myprog1.o ${OBJS} ${PETSC_LIB} > > > make all yields: > > make: *** No rule to make target `myprog1.o', needed by `myprog1'. Stop > > -gideon -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From balay at mcs.anl.gov Wed Feb 22 21:28:17 2017 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 22 Feb 2017 21:28:17 -0600 Subject: [petsc-users] make file issues In-Reply-To: <6B63CEAF-E752-46D3-93D5-26B4CA5E15E2@gmail.com> References: <6B63CEAF-E752-46D3-93D5-26B4CA5E15E2@gmail.com> Message-ID: On Wed, 22 Feb 2017, Gideon Simpson wrote: > I?ve been trying to use some code that was originally developed under petsc 3.6.x with my new 3.7.5 installation. I?m having an issue in that the way the code is written, it?s spread across several .c files. Here are the essential features of the makefile: > > all: myprog1 > > OBJS = functions.o \ > utils.o > > include ${PETSC_DIR}/lib/petsc/conf/variables > include ${PETSC_DIR}/lib/petsc/conf/rules > > myprog1: prog1.o ${OBJS} > ${CLINKER} -o myprog1 myprog1.o ${OBJS} ${PETSC_LIB} Hm - you have prog1.o listed first - and myprog1.o listed later. What is the name of this sourcefile? Satish > > > make all yields: > > make: *** No rule to make target `myprog1.o', needed by `myprog1'. Stop > > -gideon > > From cmpierce at WPI.EDU Wed Feb 22 21:50:09 2017 From: cmpierce at WPI.EDU (Christopher Pierce) Date: Wed, 22 Feb 2017 22:50:09 -0500 Subject: [petsc-users] Krylov-Schur Tolerance In-Reply-To: References: <4691ae05-be89-5bdb-33ac-2fd5cef4c978@wpi.edu> <78DC3258-D0BB-49A9-9246-D15F6E82A9D0@dsic.upv.es> <18B35373-BFB8-4307-B7E4-3D2C49B699DE@dsic.upv.es> <7ee69de1-511c-6dba-58b1-4a330e1beb96@wpi.edu> Message-ID: <29542086-6ac7-005e-1aed-31c0a5fa44bd@wpi.edu> Thanks, That makes a lot of sense. Is there any simple way to query the true residual norm of the approximate eigenvector in the equivalent single matrix problem? IE ||B^{-1}AX-lambda*X||. I think that would help determine the reliability of the computed solution. Chris On 02/22/17 06:51, Jose E. Roman wrote: >> El 20 feb 2017, a las 11:33, Jose E. Roman escribi?: >> >> >>> El 20 feb 2017, a las 11:01, Christopher Pierce escribi?: >>> >>> It seems to give the same results. I exported the matrices to Matlab >>> and checked the estimated condition number of the B matrix which came to >>> ~15 and the 2-norm of the B matrix which was ~10^6. I'm guessing that >>> the large matrix norm is the problem. I glanced over the source for the >>> RQCG solver and it doesn't seem to use a linear solver which is likely >>> why it showed better performance. Do you have any suggestions for >>> dealing with problems like this? >>> >>> Chris >> Send the data files to my personal email and I will make some tests. >> Jose >> > The large errors that you report can be explained by the fact that both A and B have large norm, in particular ~10^11 and ~10^5, respectively. Note that the norm of the residual ||A*x-lambda*B*X|| can be made arbitrarily large by increasing the matrix norms. So if you scale the matrices so that e.g. norm(B)=1, you will see a reduction of the reported error in several orders of magnitude. > > On the other hand, note that in case of solving the problem as a GHEP the returned eigenvector is not normalized to have ||x||_2=1, but to have ||x||_B=1, so when showing the relative residual error, SLEPc should divide by ||x||_2, which is quite small in your case. This can be considered a bug in SLEPc that we will fix for the next release. Doing this in your case would imply increasing a bit the reported errors (much less than the gain from the first comment). > > In conclusion, you can rely on the computed solution even though the reported residual norms are large in this case. > > Jose > From jswenson at smu.edu Wed Feb 22 21:53:47 2017 From: jswenson at smu.edu (Jennifer Swenson) Date: Wed, 22 Feb 2017 21:53:47 -0600 Subject: [petsc-users] implementing non-local scalar field -- updated Message-ID: <02d801d28d88$719032f0$54b098d0$@smu.edu> Note: This has been updated from the original thread to contain the equation. I apologize for the previous error. Dear PETSc Users, As a part of my problem, I need to be able to look up the value of a scalar field (let's call it xi) that is defined via a somewhat non-trivial integral over a second scalar field (let's call it rho). Mathematically we have xi(x,t) = max( 0.1 - 1/L int{ rho_1[r(s),t] + rho_2[r(s),t] } ds ) What we're modeling here is the average energy loss of an energetic ion that is passing through a solid material. Ions can arrive with a specified direction, so to calculate xi at each point, we have to perform line integrals back toward the ion source, and keep track of how much mass the ion has passed through. In a serial algorithm, one can in principle generate xi by working from the top down, subtracting energy from each row of grid points given information on the density field. However, it is not clear how one could develop a robust parallel algorithm, much less one that works within the PETSc framework. I would very grateful for any help you can offer. Sincerely, Jennifer Swenson -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Wed Feb 22 22:07:44 2017 From: mfadams at lbl.gov (Mark Adams) Date: Wed, 22 Feb 2017 23:07:44 -0500 Subject: [petsc-users] Multigrid with defect correction In-Reply-To: References: Message-ID: On Wed, Feb 22, 2017 at 5:53 AM, Matt Landreman wrote: > Hi, PETSc developers, > > 1. Consider solving a linear problem with -pc_type mg, assembling the > matrix using KSPSetComputeOperators(ksp,ComputeMatrix,&user) and > MatSetValuesStencil as in ksp ex25.c. I?d like Pmat to be different than > Amat (a stable low-order discretization and a high-order discretization, > respectively.) However the two Mat arguments that PETSc passes to > ComputeMatrix appear to be the same object, so Pmat and Amat are forced to > be the same. (For example in ex25, J==jac). How do I allow Amat and Pmat > to differ? > > 2. I suppose if the above issue were resolved, the resulting gmres/mg > iteration would amount to doing defect correction on only the finest grid. > Another defect correction method is to use a different Mat for smoothing > and residual computation at every level. (Stable low-order for smoothing, > high-order for residual.) This latter approach is recommended on page 103 > of Brandt?s multigrid guide (http://www.wisdom.weizmann.ac > .il/~achi/classics.pdf). What would be the best way to do this second > kind of defect correction in PETSc mg? Perhaps loop over every multigrid > level and call PCMGSetResidual(pc,l,PCMGResidualDefault,Amat) on each > level, after calling ComputeMatrix to build the appropriately-sized Amat? > > I am not familiar with this KSPSetComputeOperators interface, but yes you can loop over the levels and change operators. If you want to modify the default algorithm then you can dive in and modify it. > 3. Is it at all sensible to do this second kind of defect correction with > _algebraic_ multigrid? > Not likely, > Perhaps Amat for each level could be formed from the high-order matrix at > the fine level by the Galerkin operator R A P, after getting all the > restriction matrices created by gamg for Pmat? > RAP will kill the high order unless you come up with very clever AMG coarse grid spaces. You probably want to keep all explicit matrices stable/low-order and do high order matrix free. > > Thanks, > Matt Landreman > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gideon.simpson at gmail.com Wed Feb 22 22:24:16 2017 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Wed, 22 Feb 2017 23:24:16 -0500 Subject: [petsc-users] make file issues In-Reply-To: References: <6B63CEAF-E752-46D3-93D5-26B4CA5E15E2@gmail.com> Message-ID: Ok, it should be: myprog1: prog1.o ${OBJS} ${CLINKER} -o myprog1 prog1.o ${OBJS} ${PETSC_LIB} where the application should be called mprog1, and there?s this collection of objects, prog1.o (from prog1.c), along with other object files. -gideon > On Feb 22, 2017, at 10:28 PM, Satish Balay wrote: > > On Wed, 22 Feb 2017, Gideon Simpson wrote: > >> I?ve been trying to use some code that was originally developed under petsc 3.6.x with my new 3.7.5 installation. I?m having an issue in that the way the code is written, it?s spread across several .c files. Here are the essential features of the makefile: >> >> all: myprog1 >> >> OBJS = functions.o \ >> utils.o >> >> include ${PETSC_DIR}/lib/petsc/conf/variables >> include ${PETSC_DIR}/lib/petsc/conf/rules >> >> myprog1: prog1.o ${OBJS} >> ${CLINKER} -o myprog1 myprog1.o ${OBJS} ${PETSC_LIB} > > Hm - you have prog1.o listed first - and myprog1.o listed later. What is the name of this sourcefile? > > Satish >> >> >> make all yields: >> >> make: *** No rule to make target `myprog1.o', needed by `myprog1'. Stop >> >> -gideon -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Wed Feb 22 23:35:32 2017 From: jed at jedbrown.org (Jed Brown) Date: Wed, 22 Feb 2017 22:35:32 -0700 Subject: [petsc-users] make file issues In-Reply-To: References: <6B63CEAF-E752-46D3-93D5-26B4CA5E15E2@gmail.com> Message-ID: <87efypbhaz.fsf@jedbrown.org> Gideon Simpson writes: > Ok, it should be: > > myprog1: prog1.o ${OBJS} > ${CLINKER} -o myprog1 prog1.o ${OBJS} ${PETSC_LIB} ${CLINKER} -o $@ $^ ${PETSC_LIB} ^^ I would spell it this way -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From jroman at dsic.upv.es Thu Feb 23 04:02:13 2017 From: jroman at dsic.upv.es (Jose E. Roman) Date: Thu, 23 Feb 2017 11:02:13 +0100 Subject: [petsc-users] Krylov-Schur Tolerance In-Reply-To: <29542086-6ac7-005e-1aed-31c0a5fa44bd@wpi.edu> References: <4691ae05-be89-5bdb-33ac-2fd5cef4c978@wpi.edu> <78DC3258-D0BB-49A9-9246-D15F6E82A9D0@dsic.upv.es> <18B35373-BFB8-4307-B7E4-3D2C49B699DE@dsic.upv.es> <7ee69de1-511c-6dba-58b1-4a330e1beb96@wpi.edu> <29542086-6ac7-005e-1aed-31c0a5fa44bd@wpi.edu> Message-ID: No, there is no simple way. You can use EPSGetST() followed by STApply() to compute the residual you mention (it will actually be ||A^{-1}B*x-1/lambda*x|| for STSINVERT with target=0). Also, an estimate of this residual norm is what the monitors will show, e.g. -eps_monitor_all. These values can be retrieved with EPSGetErrorEstimates(), for details see section 2.5.3 of the users manual. Jose > El 23 feb 2017, a las 4:50, Christopher Pierce escribi?: > > Thanks, > > That makes a lot of sense. Is there any simple way to query the true > residual norm of the approximate eigenvector in the equivalent single > matrix problem? IE ||B^{-1}AX-lambda*X||. I think that would help > determine the reliability of the computed solution. > > Chris > > > > > > On 02/22/17 06:51, Jose E. Roman wrote: >>> El 20 feb 2017, a las 11:33, Jose E. Roman escribi?: >>> >>> >>>> El 20 feb 2017, a las 11:01, Christopher Pierce escribi?: >>>> >>>> It seems to give the same results. I exported the matrices to Matlab >>>> and checked the estimated condition number of the B matrix which came to >>>> ~15 and the 2-norm of the B matrix which was ~10^6. I'm guessing that >>>> the large matrix norm is the problem. I glanced over the source for the >>>> RQCG solver and it doesn't seem to use a linear solver which is likely >>>> why it showed better performance. Do you have any suggestions for >>>> dealing with problems like this? >>>> >>>> Chris >>> Send the data files to my personal email and I will make some tests. >>> Jose >>> >> The large errors that you report can be explained by the fact that both A and B have large norm, in particular ~10^11 and ~10^5, respectively. Note that the norm of the residual ||A*x-lambda*B*X|| can be made arbitrarily large by increasing the matrix norms. So if you scale the matrices so that e.g. norm(B)=1, you will see a reduction of the reported error in several orders of magnitude. >> >> On the other hand, note that in case of solving the problem as a GHEP the returned eigenvector is not normalized to have ||x||_2=1, but to have ||x||_B=1, so when showing the relative residual error, SLEPc should divide by ||x||_2, which is quite small in your case. This can be considered a bug in SLEPc that we will fix for the next release. Doing this in your case would imply increasing a bit the reported errors (much less than the gain from the first comment). >> >> In conclusion, you can rely on the computed solution even though the reported residual norms are large in this case. >> >> Jose >> > From knepley at gmail.com Thu Feb 23 09:41:04 2017 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 23 Feb 2017 09:41:04 -0600 Subject: [petsc-users] Multigrid with defect correction In-Reply-To: References: Message-ID: On Wed, Feb 22, 2017 at 4:53 AM, Matt Landreman wrote: > Hi, PETSc developers, > > 1. Consider solving a linear problem with -pc_type mg, assembling the > matrix using KSPSetComputeOperators(ksp,ComputeMatrix,&user) and > MatSetValuesStencil as in ksp ex25.c. I?d like Pmat to be different than > Amat (a stable low-order discretization and a high-order discretization, > respectively.) However the two Mat arguments that PETSc passes to > ComputeMatrix appear to be the same object, so Pmat and Amat are forced to > be the same. (For example in ex25, J==jac). How do I allow Amat and Pmat > to differ? > You are correct that this is a shortcoming of the interface. We really should rework it so that (possibly) two operators are constructed. I think the right way to do this would end up having most of the DM infrastructure duplicated for the second matrix, since the DM is holding the construction routine. You can see my example of this idea in PetscDS, which holds two sets of creation routines, one for the Jacobian and one for its preconditioning matrix. However this is not yet universally agreed in PETSc. For the time being, I believe switching to the nonlinear interface in SNES will give you what you want. > 2. I suppose if the above issue were resolved, the resulting gmres/mg > iteration would amount to doing defect correction on only the finest grid. > Another defect correction method is to use a different Mat for smoothing > and residual computation at every level. (Stable low-order for smoothing, > high-order for residual.) This latter approach is recommended on page 103 > of Brandt?s multigrid guide (http://www.wisdom.weizmann. > ac.il/~achi/classics.pdf). What would be the best way to do this second > kind of defect correction in PETSc mg? Perhaps loop over every multigrid > level and call PCMGSetResidual(pc,l,PCMGResidualDefault,Amat) on each > level, after calling ComputeMatrix to build the appropriately-sized Amat? > I do not understand the point here. If you could indeed construct a P and A on each level, then you can do exactly what you want. In fact, this is just Newton with KSPPREONLY, or preconditioned Richardson (I think). It should be easy to build whatever you want since each smoother can be independently configured with options. Can you write exactly what you want in algebraic terms? > 3. Is it at all sensible to do this second kind of defect correction with > _algebraic_ multigrid? Perhaps Amat for each level could be formed from the > high-order matrix at the fine level by the Galerkin operator R A P, after > getting all the restriction matrices created by gamg for Pmat? > Mark responded. Thanks, Matt > Thanks, > Matt Landreman > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Feb 23 09:57:40 2017 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 23 Feb 2017 09:57:40 -0600 Subject: [petsc-users] implementing non-local scalar field -- updated In-Reply-To: <02d801d28d88$719032f0$54b098d0$@smu.edu> References: <02d801d28d88$719032f0$54b098d0$@smu.edu> Message-ID: On Wed, Feb 22, 2017 at 9:53 PM, Jennifer Swenson wrote: > Note: This has been updated from the original thread to contain the > equation. I apologize for the previous error. > > > > Dear PETSc Users, > > > > As a part of my problem, I need to be able to look up the value of a > scalar field (let's call it xi) that is defined via a somewhat non-trivial > integral over a second scalar field (let's call it rho). Mathematically we > have > > > > xi(x,t) = max( 0.1 ? 1/L int{ rho_1[r(s),t] + rho_2[r(s),t] } ds ) > This sounds like the Radon Transform to me: https://en.wikipedia.org/wiki/Radon_transform > What we're modeling here is the average energy loss of an energetic ion > that is passing through a solid material. Ions can arrive with a specified > direction, so to calculate xi at each point, we have to perform line > integrals back toward the ion source, and keep track of how much mass the > ion has passed through. > > > > In a serial algorithm, one can in principle generate xi by working from > the top down, subtracting energy from each row of grid points given > information on the density field. However, it is not clear how one could > develop a robust parallel algorithm, much less one that works within the > PETSc framework. > I cannot understand everything from the above, but how about a simplified model. Suppose you have a single ion source somewhere that radiates isotropically. Now suppose you have a division of space into rectangles, such as would be produced by DMDA. For each rectangle on a given process, I can independently calculate all the paths through it from the known source point, which would provide increments to the line integral. Now it gets complicated, or at least I do not have a simple way to do this. You have to scan the processes in the order in which they are illuminated by the source.This can be calculated for rectangular (probably for convex) regions in a straightforward way. Then you communicate along the chain, updating the increments. I think you could renumber the processes to match some total order from the partial order of illumination and just use MPI_Scan(), but I can not sure. However, this is an old problem, and my first thought is that someone would have already solved it. To the library! Thanks, Matt I would very grateful for any help you can offer. > > > > Sincerely, > > Jennifer Swenson > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Feb 23 10:44:49 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 23 Feb 2017 10:44:49 -0600 Subject: [petsc-users] Multigrid with defect correction In-Reply-To: References: Message-ID: <92430F37-E142-4AD2-B2BF-B35DE2D24719@mcs.anl.gov> > On Feb 22, 2017, at 4:53 AM, Matt Landreman wrote: > > Hi, PETSc developers, > > 1. Consider solving a linear problem with -pc_type mg, assembling the matrix using KSPSetComputeOperators(ksp,ComputeMatrix,&user) and MatSetValuesStencil as in ksp ex25.c. I?d like Pmat to be different than Amat (a stable low-order discretization and a high-order discretization, respectively. Presumably the high-order discretization will be "matrix-free" and you'd like to use a MATSHELL for it? > ) However the two Mat arguments that PETSc passes to ComputeMatrix appear to be the same object, so Pmat and Amat are forced to be the same. (For example in ex25, J==jac). How do I allow Amat and Pmat to differ? It looks like you are using a DMDA and having the DMDA create the matrices for all levels instead of setting them yourself. The problem comes about because of this code: PetscErrorCode KSPSetUp(KSP ksp) { PetscErrorCode ierr; ..... ierr = DMCreateMatrix(ksp->dm,&A);CHKERRQ(ierr); ierr = KSPSetOperators(ksp,A,A);CHKERRQ(ierr); ierr = PetscObjectDereference((PetscObject)A);CHKERRQ(ierr); The easiest way for you to work around this is when you enter you ComputeMatrix() if the two matrices are the same then create your shell matrix amat and call KSPSetOperators(ksp,amat,pmat thus "replacing the first use of the pmat with you new shell matrix. Then just "fill up" the pmat as usual and do what you need for you shell matrix amat. The next time ComputeMatrix() is called for this level the two matrices will already be different so you just set the entries in the pmat and do what you need for the amat shell matrix. This approach is "tacky" but allows you do exactly what you want. Barry > > 2. I suppose if the above issue were resolved, the resulting gmres/mg iteration would amount to doing defect correction on only the finest grid. No, I think > Another defect correction method is to use a different Mat for smoothing and residual computation at every level. (Stable low-order for smoothing, high-order for residual.) This latter approach is recommended on page 103 of Brandt?s multigrid guide (http://www.wisdom.weizmann.ac.il/~achi/classics.pdf). What would be the best way to do this second kind of defect correction in PETSc mg? Perhaps loop over every multigrid level and call PCMGSetResidual(pc,l,PCMGResidualDefault,Amat) on each level, after calling ComputeMatrix to build the appropriately-sized Amat? You could use PCMGSetResidual but I don't think it is necessary. > > > 3. Is it at all sensible to do this second kind of defect correction with _algebraic_ multigrid? Perhaps Amat for each level could be formed from the high-order matrix at the fine level by the Galerkin operator R A P, after getting all the restriction matrices created by gamg for Pmat? > > Thanks, > Matt Landreman From fangbowa at buffalo.edu Thu Feb 23 12:09:55 2017 From: fangbowa at buffalo.edu (Fangbo Wang) Date: Thu, 23 Feb 2017 13:09:55 -0500 Subject: [petsc-users] can I still use collectives like MPI_Bcast, MPI_Scatter on Petsc vectors? Message-ID: thank you very much! -- Fangbo Wang, PhD student Stochastic Geomechanics Research Group Department of Civil, Structural and Environmental Engineering University at Buffalo Email: *fangbowa at buffalo.edu * -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Feb 23 12:23:13 2017 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 23 Feb 2017 12:23:13 -0600 Subject: [petsc-users] can I still use collectives like MPI_Bcast, MPI_Scatter on Petsc vectors? In-Reply-To: References: Message-ID: On Thu, Feb 23, 2017 at 12:09 PM, Fangbo Wang wrote: > thank you very much! > I do not know what you mean about "on PETSc Vectors", but yes you can always call MPI functions whenever you want. Matt > -- > Fangbo Wang, PhD student > Stochastic Geomechanics Research Group > Department of Civil, Structural and Environmental Engineering > University at Buffalo > Email: *fangbowa at buffalo.edu * > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From fangbowa at buffalo.edu Thu Feb 23 12:44:15 2017 From: fangbowa at buffalo.edu (Fangbo Wang) Date: Thu, 23 Feb 2017 13:44:15 -0500 Subject: [petsc-users] can I still use collectives like MPI_Bcast, MPI_Scatter on Petsc vectors? In-Reply-To: References: Message-ID: Assume I need to use Petsc function Matmult, I need to define a Petsc matrix "A" and a Petsc vector "b". Mat A; Vec b; can I do MPI_bcast on this vector b? Thank you very much! On Thu, Feb 23, 2017 at 1:23 PM, Matthew Knepley wrote: > On Thu, Feb 23, 2017 at 12:09 PM, Fangbo Wang > wrote: > >> thank you very much! >> > > I do not know what you mean about "on PETSc Vectors", but yes you can > always call MPI functions whenever you want. > > Matt > > >> -- >> Fangbo Wang, PhD student >> Stochastic Geomechanics Research Group >> Department of Civil, Structural and Environmental Engineering >> University at Buffalo >> Email: *fangbowa at buffalo.edu * >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Fangbo Wang, PhD student Stochastic Geomechanics Research Group Department of Civil, Structural and Environmental Engineering University at Buffalo Email: *fangbowa at buffalo.edu * -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Feb 23 12:48:44 2017 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 23 Feb 2017 12:48:44 -0600 Subject: [petsc-users] can I still use collectives like MPI_Bcast, MPI_Scatter on Petsc vectors? In-Reply-To: References: Message-ID: On Thu, Feb 23, 2017 at 12:44 PM, Fangbo Wang wrote: > Assume I need to use Petsc function Matmult, I need to define a Petsc > matrix "A" and a Petsc vector "b". > > Mat A; > Vec b; > > can I do MPI_bcast on this vector b? > 1) I assume the Bcast has nothing to do with the MatMult. 2) MPI_Bcast applies only to MPI datatypes. Vec is a pointer to an object structure. 3) If you want to Bcast the values of a serial vector to all procs, do this: http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecScatterCreateToAll.html#VecScatterCreateToAll Thanks, Matt > Thank you very much! > > > On Thu, Feb 23, 2017 at 1:23 PM, Matthew Knepley > wrote: > >> On Thu, Feb 23, 2017 at 12:09 PM, Fangbo Wang >> wrote: >> >>> thank you very much! >>> >> >> I do not know what you mean about "on PETSc Vectors", but yes you can >> always call MPI functions whenever you want. >> >> Matt >> >> >>> -- >>> Fangbo Wang, PhD student >>> Stochastic Geomechanics Research Group >>> Department of Civil, Structural and Environmental Engineering >>> University at Buffalo >>> Email: *fangbowa at buffalo.edu * >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > Fangbo Wang, PhD student > Stochastic Geomechanics Research Group > Department of Civil, Structural and Environmental Engineering > University at Buffalo > Email: *fangbowa at buffalo.edu * > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Thu Feb 23 16:23:02 2017 From: jed at jedbrown.org (Jed Brown) Date: Thu, 23 Feb 2017 15:23:02 -0700 Subject: [petsc-users] Multigrid with defect correction In-Reply-To: References: Message-ID: <8760k07dix.fsf@jedbrown.org> Matt Landreman writes: > 3. Is it at all sensible to do this second kind of defect correction with > _algebraic_ multigrid? Perhaps Amat for each level could be formed from the > high-order matrix at the fine level by the Galerkin operator R A P, after > getting all the restriction matrices created by gamg for Pmat? Note that defect correction is most commonly used for transport-dominated processes for which the high order discretization is not h-elliptic. AMG heuristics are typically really bad for such problems so stabilizing a smoother isn't really a relevant issue. Also, it is usually used for strongly nonlinear problems where AMG's setup costs are likely overkill. This is not to say that defect correction AMG won't work, but there is reason to believe that it doesn't matter for many important problems. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From jed at jedbrown.org Thu Feb 23 16:35:58 2017 From: jed at jedbrown.org (Jed Brown) Date: Thu, 23 Feb 2017 15:35:58 -0700 Subject: [petsc-users] implementing non-local scalar field -- updated In-Reply-To: <02d801d28d88$719032f0$54b098d0$@smu.edu> References: <02d801d28d88$719032f0$54b098d0$@smu.edu> Message-ID: <87zihc5ycx.fsf@jedbrown.org> Jennifer Swenson writes: > As a part of my problem, I need to be able to look up the value of a scalar > field (let's call it xi) that is defined via a somewhat non-trivial integral > over a second scalar field (let's call it rho). Mathematically we have > > > > xi(x,t) = max( 0.1 - 1/L int{ rho_1[r(s),t] + rho_2[r(s),t] } ds ) Is the RHS of this expression intended to depend on x? > What we're modeling here is the average energy loss of an energetic ion that > is passing through a solid material. Ions can arrive with a specified > direction, so to calculate xi at each point, we have to perform line > integrals back toward the ion source, and keep track of how much mass the > ion has passed through. Are the ions tracked or do they just appear? > In a serial algorithm, one can in principle generate xi by working from the > top down, subtracting energy from each row of grid points given information > on the density field. However, it is not clear how one could develop a > robust parallel algorithm, much less one that works within the PETSc > framework. It would help to know more about the integral that needs to be done and the relative data sizes. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From gideon.simpson at gmail.com Thu Feb 23 17:20:53 2017 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Thu, 23 Feb 2017 18:20:53 -0500 Subject: [petsc-users] make file issues In-Reply-To: <87efypbhaz.fsf@jedbrown.org> References: <6B63CEAF-E752-46D3-93D5-26B4CA5E15E2@gmail.com> <87efypbhaz.fsf@jedbrown.org> Message-ID: still getting a no rule to make target? -gideon > On Feb 23, 2017, at 12:35 AM, Jed Brown wrote: > > Gideon Simpson writes: > >> Ok, it should be: >> >> myprog1: prog1.o ${OBJS} >> ${CLINKER} -o myprog1 prog1.o ${OBJS} ${PETSC_LIB} > > ${CLINKER} -o $@ $^ ${PETSC_LIB} > > ^^ I would spell it this way -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Thu Feb 23 17:24:56 2017 From: jed at jedbrown.org (Jed Brown) Date: Thu, 23 Feb 2017 16:24:56 -0700 Subject: [petsc-users] make file issues In-Reply-To: References: <6B63CEAF-E752-46D3-93D5-26B4CA5E15E2@gmail.com> <87efypbhaz.fsf@jedbrown.org> Message-ID: <87o9xs5w3b.fsf@jedbrown.org> Gideon Simpson writes: > still getting a no rule to make target? What exactly is the error message? And what is the output of "ls -l" in that directory? My syntax suggestion is just avoiding error-prone duplication. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From gideon.simpson at gmail.com Thu Feb 23 17:25:16 2017 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Thu, 23 Feb 2017 18:25:16 -0500 Subject: [petsc-users] make file issues In-Reply-To: <87o9xs5w3b.fsf@jedbrown.org> References: <6B63CEAF-E752-46D3-93D5-26B4CA5E15E2@gmail.com> <87efypbhaz.fsf@jedbrown.org> <87o9xs5w3b.fsf@jedbrown.org> Message-ID: ugh, never mind?. -gideon > On Feb 23, 2017, at 6:24 PM, Jed Brown wrote: > > Gideon Simpson writes: > >> still getting a no rule to make target? > > What exactly is the error message? And what is the output of "ls -l" in > that directory? > > My syntax suggestion is just avoiding error-prone duplication. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gideon.simpson at gmail.com Thu Feb 23 19:10:19 2017 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Thu, 23 Feb 2017 20:10:19 -0500 Subject: [petsc-users] python3 patch Message-ID: <4C6E2899-5F6B-4822-9C7D-76F15E9C66BD@gmail.com> Does anyone have a patch for the petsc python files (PetscBinaryIO.py in particular), that will allow it to work with python 3? -gideon -------------- next part -------------- An HTML attachment was scrubbed... URL: From dnolte at dim.uchile.cl Fri Feb 24 08:13:19 2017 From: dnolte at dim.uchile.cl (David Nolte) Date: Fri, 24 Feb 2017 11:13:19 -0300 Subject: [petsc-users] understanding the LSC preconditioner In-Reply-To: References: <74e410b0-c64d-dccf-a653-c0505fb16dd6@dim.uchile.cl> <123a2c95-0c1b-31fb-858b-2c4ddc3e2284@dim.uchile.cl> <5e68f816-3331-c7ba-e1fc-6f79ffa3c7e8@dim.uchile.cl> Message-ID: <2a1e3886-ea26-d413-5455-a881af96024c@dim.uchile.cl> At the moment it's not possible for me to rebuild python with the proper configuration for valgrind (--without-pymalloc --with-valgrind). I'm not sure how useful the output is now. I can't say I get it... you can find it in the attachment, if you want to take a look. Valgrind crashes with the following error: $ valgrind -q --log-file=valgrind.log --tool=memcheck --suppressions=valgrind-python.supp python StokesPC/stokespc/stokes_bench.py cr_libinit.c:189 cri_init: sigaction() failed: Invalid argument [1] 80 abort (core dumped) valgrind -q --log-file=valgrind.log --tool=memcheck python On 02/22/2017 04:30 PM, Barry Smith wrote: >> On Feb 22, 2017, at 8:03 AM, David Nolte wrote: >> >> Hm, I thought it could be related to my construction of the L matrix, >> since for example LU >> -fieldsplit_1_lsc_ksp_type preonly >> -fieldsplit_1_lsc_pc_type lu >> >> doesn't work either: > Yes, it could be. But try with valgrind > >> 0 KSP unpreconditioned resid norm 2.963704216563e+01 true resid norm >> 2.963704216563e+01 ||r(i)||/||b|| 1.000000000000e+00 >> [0] 14.2691 Event begin: VecScale >> [0] 14.2693 Event end: VecScale >> [0] PCSetUp(): Leaving PC with identical preconditioner since operator >> is unchanged >> [0] 14.2693 Event begin: PCApply >> [0] 14.2693 Event begin: VecScatterBegin >> [0] 14.2693 Event end: VecScatterBegin >> [0] 14.2693 Event begin: KSPSolve_FS_Schu >> [0] 14.2693 Event begin: KSPSetUp >> [0] 14.2694 Event end: KSPSetUp >> [0] PCSetUp(): Setting up PC for first time >> [0] 14.2694 Event begin: PCSetUp >> [0] 14.2694 Event begin: VecSet >> [0] 14.2696 Event end: VecSet >> [0] 14.2696 Event begin: VecSet >> [0] 14.2698 Event end: VecSet >> [0] 14.2699 Event begin: VecSet >> [0] 14.2699 Event end: VecSet >> [0] 14.2699 Event end: PCSetUp >> [0] 14.2699 Event begin: VecSet >> [0] 14.27 Event end: VecSet >> [0] PCSetUp(): Leaving PC with identical preconditioner since operator >> is unchanged >> [0] 14.27 Event begin: KSPSetUp >> [0] 14.27 Event end: KSPSetUp >> [0] PCSetUp(): Setting up PC for first time >> [0] 14.27 Event begin: PCSetUp >> Traceback (most recent call last): >> File "StokesPC/stokespc/stokes_bench.py", line 331, in >> plot_residuals=plot_residuals) >> File "StokesPC/stokespc/stokes_bench.py", line 291, in run_bench >> compare_LU=compare_LU, results_dir=results_dir) >> File "StokesPC/stokespc/stokes_bench.py", line 136, in stokes_bench >> inpfile, P, Sp, W) >> File "StokesPC/stokespc/stokes_bench.py", line 244, in solve_petsc >> ksp.solve(b, x) >> File "PETSc/KSP.pyx", line 384, in petsc4py.PETSc.KSP.solve >> (src/petsc4py.PETSc.c:153555) >> petsc4py.PETSc.Error: error code 62 >> Aborted (core dumped) >> >> >> David >> >> >> On 02/21/2017 04:09 PM, Barry Smith wrote: >>> Hmm, it is crashing inside trying to build the ML preconditioner. This is certainly not expected, could be some data got corrupted earlier. I'm not sure how to track it down, maybe run the python under valgrind? >>> >>> Barry >>> >>> >>>> On Feb 21, 2017, at 1:02 PM, David Nolte wrote: >>>> >>>> It crashes in ML_matmat_mult(): >>>> >>>> Program received signal SIGSEGV, Segmentation >>>> fault. >>>> >>>> 0x00007ffff5cdaeb0 in ML_matmat_mult () from >>>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>>> >>>> (gdb) bt >>>> #0 0x00007ffff5cdaeb0 in ML_matmat_mult () from >>>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>>> >>>> #1 0x00007ffff5cdbf76 in ML_2matmult () from >>>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>>> >>>> #2 0x00007ffff5ca197a in ML_AGG_Gen_Prolongator () from >>>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>>> >>>> #3 0x00007ffff5c9fa71 in ML_Gen_MultiLevelHierarchy () from >>>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>>> >>>> #4 0x00007ffff5ca0484 in ML_Gen_MultiLevelHierarchy_UsingAggregation () >>>> from >>>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>>> >>>> #5 0x00007ffff5780aff in PCSetUp_ML () from >>>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>>> >>>> #6 0x00007ffff5667fce in PCSetUp () from >>>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>>> >>>> #7 0x00007ffff578d9f8 in KSPSetUp () from >>>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>>> >>>> #8 0x00007ffff578e7d8 in KSPSolve () from >>>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>>> >>>> #9 0x00007ffff578606b in PCApply_LSC () from >>>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>>> >>>> #10 0x00007ffff5668640 in PCApply () from >>>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>>> >>>> #11 0x00007ffff57f05a5 in KSPSolve_PREONLY () from >>>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>>> #12 0x00007ffff578ea63 in KSPSolve () from >>>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>>> #13 0x00007ffff5746cf4 in PCApply_FieldSplit_Schur () from >>>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>>> #14 0x00007ffff5668640 in PCApply () from >>>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>>> #15 0x00007ffff580ab2d in KSPFGMRESCycle () from >>>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>>> #16 0x00007ffff580b900 in KSPSolve_FGMRES () from >>>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>>> #17 0x00007ffff578ea63 in KSPSolve () from >>>> /usr/local/petsc-32/lib/libpetsc.so.3.7 >>>> #18 0x00007ffff6597fbf in __pyx_pf_8petsc4py_5PETSc_3KSP_98solve >>>> (__pyx_v_self=0x7fffe4f59830, __pyx_v_b=, >>>> __pyx_v_x=) >>>> at src/petsc4py.PETSc.c:153555 >>>> [...] >>>> >>>> Is this where it tries to perform the matrix multiplication >>>> `Bdiv.matMult(Bgrad)`? >>>> >>>> When instead of ML I use LU in the PC, >>>> >>>> -fieldsplit_1_lsc_ksp_type preonly >>>> -fieldsplit_1_lsc_pc_type lu >>>> >>>> I get a "wrong argument" error: >>>> >>>> File "StokesPC/stokespc/stokes_bench.py", line 242, in >>>> solve_petsc >>>> ksp.solve(b.vec(), x) >>>> File "PETSc/KSP.pyx", line 384, in petsc4py.PETSc.KSP.solve >>>> (src/petsc4py.PETSc.c:153555) >>>> petsc4py.PETSc.Error: error code 62 >>>> >>>> >>>> Regards, >>>> David >>>> >>>> >>>> On 02/21/2017 02:20 PM, Barry Smith wrote: >>>>> You'll have to figure out what is triggering the segmentation violation. If it is the python that is crashing then likely you can run the entire python program in the debugger and then when it crashes you should be able to see where. >>>>> >>>>> Barry >>>>> >>>>> >>>>>> On Feb 21, 2017, at 10:07 AM, David Nolte wrote: >>>>>> >>>>>> Dear all, >>>>>> >>>>>> new to PETSc, I am trying to use the LSC preconditioner for a Stokes >>>>>> problem (discretized by means of stable FEM). I use the python backend >>>>>> petsc4py. >>>>>> The "automatic" version of the LSC seems to work with the following >>>>>> setup (I think adapted from Matt's tutorial slides): >>>>>> >>>>>> -ksp_view >>>>>> -ksp_converged_reason >>>>>> -ksp_monitor_true_residual >>>>>> -ksp_type fgmres >>>>>> -ksp_rtol 1.0e-8 >>>>>> >>>>>> -pc_type fieldsplit >>>>>> -pc_fieldsplit_detect_saddle_point >>>>>> -pc_fieldsplit_type schur >>>>>> -pc_fieldsplit_schur_fact_type upper >>>>>> -pc_fieldsplit_schur_precondition self >>>>>> >>>>>> -fieldsplit_0_ksp_type preonly >>>>>> -fieldsplit_0_pc_type ml >>>>>> >>>>>> -fieldsplit_1_ksp_type preonly >>>>>> -fieldsplit_1_pc_type lsc >>>>>> -fieldsplit_1_lsc_pc_type ml >>>>>> -fieldsplit_1_lsc_ksp_type preonly >>>>>> >>>>>> In a 3D setup with 250k dofs this converges within 78 iterations. (For >>>>>> reference, upper Schur factorization with ML for the uu-block and Sp = >>>>>> diag(Q), the diagonal of the pressure mass matrix, takes 41 iterations >>>>>> and half of the computation time.) >>>>>> >>>>>> Now I just wanted to check if I can get the same result by building the >>>>>> L-matrix manually with the following piece of python code, where is0, >>>>>> is1 are the index sets corresponding to the velocity and pressure dofs, >>>>>> and A is full the system matrix. >>>>>> >>>>>> Sp = Sp.getSubMatrix(is1, is1) >>>>>> pc.setFieldSplitSchurPreType(PETSc.PC.SchurPreType.USER, Sp) >>>>>> # Sp.setType(PETSc.Mat.Type.SCHURCOMPLEMENT) # necessary? >>>>>> # extract A10 block >>>>>> Bdiv = A.getSubMatrix(is1, is0) >>>>>> # extract A01 block >>>>>> Bgrad = A.getSubMatrix(is0, is1) >>>>>> L = Bdiv >>>>>> L.matMult(Bgrad) >>>>>> Sp.compose('LSC_L', L) >>>>>> Sp.compose('LSC_Lp', L) >>>>>> >>>>>> To my understanding, this should behave similarly to what the LSC >>>>>> preconditioner does when LSC_L is not given. However, I get a >>>>>> segmentation fault during the first iteration: >>>>>> >>>>>> 0 KSP unpreconditioned resid norm 2.963704216563e+01 true resid norm >>>>>> 2.963704216563e+01 ||r(i)||/||b|| 1.000000000000e+00 >>>>>> [1] 2311 segmentation fault (core dumped) python >>>>>> StokesPC/stokespc/stokes_bench.py >>>>>> >>>>>> What am I doing wrong? I appreciate any hints, thanks a lot in advance! >>>>>> >>>>>> Regards, >>>>>> David >>>>>> >>>>>> >>>>>> PS: The log trace is: >>>>>> 0 KSP unpreconditioned resid norm 2.963704216563e+01 true resid norm >>>>>> 2.963704216563e+01 ||r(i)||/||b|| 1.000000000000e+00 >>>>>> [0] 10.0543 Event begin: VecScale >>>>>> [0] 10.0545 Event end: VecScale >>>>>> [0] PCSetUp(): Leaving PC with identical preconditioner since operator >>>>>> is unchanged >>>>>> [0] 10.0545 Event begin: PCApply >>>>>> [0] 10.0545 Event begin: VecScatterBegin >>>>>> [0] 10.0546 Event end: VecScatterBegin >>>>>> [0] 10.0546 Event begin: KSPSolve_FS_Schu >>>>>> [0] 10.0546 Event begin: KSPSetUp >>>>>> [0] 10.0546 Event end: KSPSetUp >>>>>> [0] PCSetUp(): Setting up PC for first time >>>>>> [0] 10.0546 Event begin: PCSetUp >>>>>> [0] 10.0547 Event begin: VecSet >>>>>> [0] 10.055 Event end: VecSet >>>>>> [0] 10.055 Event begin: VecSet >>>>>> [0] 10.0553 Event end: VecSet >>>>>> [0] 10.0553 Event begin: VecSet >>>>>> [0] 10.0553 Event end: VecSet >>>>>> [0] 10.0554 Event end: PCSetUp >>>>>> [0] 10.0554 Event begin: VecSet >>>>>> [0] 10.0554 Event end: VecSet >>>>>> [0] PCSetUp(): Leaving PC with identical preconditioner since operator >>>>>> is unchanged >>>>>> [0] 10.0554 Event begin: KSPSetUp >>>>>> [0] 10.0554 Event end: KSPSetUp >>>>>> [0] PCSetUp(): Setting up PC for first time >>>>>> [0] 10.0554 Event begin: PCSetUp >>>>>> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 >>>>>> -2080374783 >>>>>> [0] 10.0555 Event begin: VecSet >>>>>> [0] 10.0557 Event end: VecSet >>>>>> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 >>>>>> -2080374783 >>>>>> [0] 10.0557 Event begin: VecSet >>>>>> [0] 10.0558 Event end: VecSet >>>>>> [0] 10.082 Event begin: MatMult >>>>>> [0] 10.1273 Event end: MatMult >>>>>> [0] 10.1277 Event begin: MatMult >>>>>> [0] 10.1739 Event end: MatMult >>>>>> [0] 10.1742 Event begin: MatMult >>>>>> [0] 10.2195 Event end: MatMult >>>>>> [0] 10.2199 Event begin: MatMult >>>>>> [0] 10.2653 Event end: MatMult >>>>>> [0] 10.2657 Event begin: MatMult >>>>>> [0] 10.3113 Event end: MatMult >>>>>> [0] 10.3116 Event begin: MatMult >>>>>> [0] 10.3571 Event end: MatMult >>>>>> [0] 10.3575 Event begin: MatMult >>>>>> [0] 10.403 Event end: MatMult >>>>>> [0] 10.4033 Event begin: MatMult >>>>>> [0] 10.4487 Event end: MatMult >>>>>> [0] 10.4491 Event begin: MatMult >>>>>> [0] 10.4947 Event end: MatMult >>>>>> [0] 10.495 Event begin: MatMult >>>>>> [0] 10.5406 Event end: MatMult >>>>>> >>>>>> >>>>>> >> -------------- next part -------------- A non-text attachment was scrubbed... Name: valgrind.log Type: text/x-log Size: 30463 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: From jed at jedbrown.org Fri Feb 24 08:22:53 2017 From: jed at jedbrown.org (Jed Brown) Date: Fri, 24 Feb 2017 07:22:53 -0700 Subject: [petsc-users] understanding the LSC preconditioner In-Reply-To: <2a1e3886-ea26-d413-5455-a881af96024c@dim.uchile.cl> References: <74e410b0-c64d-dccf-a653-c0505fb16dd6@dim.uchile.cl> <123a2c95-0c1b-31fb-858b-2c4ddc3e2284@dim.uchile.cl> <5e68f816-3331-c7ba-e1fc-6f79ffa3c7e8@dim.uchile.cl> <2a1e3886-ea26-d413-5455-a881af96024c@dim.uchile.cl> Message-ID: <87wpcf3bya.fsf@jedbrown.org> David Nolte writes: > At the moment it's not possible for me to rebuild python with the proper > configuration for valgrind (--without-pymalloc --with-valgrind). I'm not > sure how useful the output is now. I can't say I get it... you can find > it in the attachment, if you want to take a look. Oh, the joys of mixed-language memory debugging... You can build PETSc and your code with -fstack-protector -fsanitize=address to hopefully find the memory corruption. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From gideon.simpson at gmail.com Fri Feb 24 11:46:30 2017 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Fri, 24 Feb 2017 12:46:30 -0500 Subject: [petsc-users] python3 update Message-ID: <4BBA33BC-1B01-427F-B709-C5EBF630D4C3@gmail.com> It appears the following modification to petsc_conf.py is sufficient to get PetscBinaryIO.py working: change: if os.environ.has_key('PETSC_DIR?): to if 'PETSC_DIR' in os.environ: and if os.environ.has_key('PETSC_ARCH?): to if 'PETSC_ARCH' in os.environ: -gideon From bsmith at mcs.anl.gov Fri Feb 24 12:26:36 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 24 Feb 2017 12:26:36 -0600 Subject: [petsc-users] python3 update In-Reply-To: <4BBA33BC-1B01-427F-B709-C5EBF630D4C3@gmail.com> References: <4BBA33BC-1B01-427F-B709-C5EBF630D4C3@gmail.com> Message-ID: Thanks, we'll get this into maint and master ASAP. > On Feb 24, 2017, at 11:46 AM, Gideon Simpson wrote: > > It appears the following modification to petsc_conf.py is sufficient to get PetscBinaryIO.py working: > > change: > if os.environ.has_key('PETSC_DIR?): > to > if 'PETSC_DIR' in os.environ: > and > if os.environ.has_key('PETSC_ARCH?): > to > if 'PETSC_ARCH' in os.environ: > > -gideon > From gideon.simpson at gmail.com Fri Feb 24 12:50:14 2017 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Fri, 24 Feb 2017 13:50:14 -0500 Subject: [petsc-users] python3 update In-Reply-To: References: <4BBA33BC-1B01-427F-B709-C5EBF630D4C3@gmail.com> Message-ID: <7D94348E-4077-43F6-92BE-7A87DE5E48F6@gmail.com> yes, though I don?t know if this breaks backwards compatibility with python2. -gideon > On Feb 24, 2017, at 1:26 PM, Barry Smith wrote: > > > Thanks, we'll get this into maint and master ASAP. > > >> On Feb 24, 2017, at 11:46 AM, Gideon Simpson wrote: >> >> It appears the following modification to petsc_conf.py is sufficient to get PetscBinaryIO.py working: >> >> change: >> if os.environ.has_key('PETSC_DIR?): >> to >> if 'PETSC_DIR' in os.environ: >> and >> if os.environ.has_key('PETSC_ARCH?): >> to >> if 'PETSC_ARCH' in os.environ: >> >> -gideon >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Feb 24 14:25:32 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 24 Feb 2017 14:25:32 -0600 Subject: [petsc-users] python3 update In-Reply-To: <7D94348E-4077-43F6-92BE-7A87DE5E48F6@gmail.com> References: <4BBA33BC-1B01-427F-B709-C5EBF630D4C3@gmail.com> <7D94348E-4077-43F6-92BE-7A87DE5E48F6@gmail.com> Message-ID: <1D38DFEF-0A8F-4A36-9AF5-2034F425B13F@mcs.anl.gov> > On Feb 24, 2017, at 12:50 PM, Gideon Simpson wrote: > > yes, though I don?t know if this breaks backwards compatibility with python2. Nope. it worked for me with python2 > > -gideon > >> On Feb 24, 2017, at 1:26 PM, Barry Smith wrote: >> >> >> Thanks, we'll get this into maint and master ASAP. >> >> >>> On Feb 24, 2017, at 11:46 AM, Gideon Simpson wrote: >>> >>> It appears the following modification to petsc_conf.py is sufficient to get PetscBinaryIO.py working: >>> >>> change: >>> if os.environ.has_key('PETSC_DIR?): >>> to >>> if 'PETSC_DIR' in os.environ: >>> and >>> if os.environ.has_key('PETSC_ARCH?): >>> to >>> if 'PETSC_ARCH' in os.environ: >>> >>> -gideon >>> >> > From fande.kong at inl.gov Fri Feb 24 14:40:23 2017 From: fande.kong at inl.gov (Kong, Fande) Date: Fri, 24 Feb 2017 13:40:23 -0700 Subject: [petsc-users] consider an error message as a petscinfo?? Message-ID: Hi All, In MatSolve(), there is a piece of code: * if (mat->errortype) { ierr = PetscInfo1(mat,"MatFactorError %D\n",mat->errortype);CHKERRQ(ierr); ierr = VecSetInf(x);CHKERRQ(ierr); } else { ierr = (*mat->ops->solve)(mat,b,x);CHKERRQ(ierr); }* If a direct solver such as LU or superlu fails to create a factorization, why we do not stop here and throw out an error message here or earlier? Now we just let solver keep doing garbage computation, and finally we have a solution like this: *Vec Object: 1 MPI processes type: mpiProcess [0]inf.inf.inf.inf.inf.inf.inf.inf.inf.inf.inf.inf.inf.inf.inf.inf.* Any particular reason to handle the thing in this way? Fande, -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Feb 24 14:56:13 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 24 Feb 2017 14:56:13 -0600 Subject: [petsc-users] consider an error message as a petscinfo?? In-Reply-To: References: Message-ID: <2F3D5EED-E357-48AF-AC6D-C51A75BFEB34@mcs.anl.gov> Fande, Yes. Say one is doing a timestepping and using a direct solver. With time stepping we do not want to necessarily stop (in fact we almost never) on a failed solve (due to for example a failed factorization). We want the code to continue up the stack until it gets to the time stepper with the information that the linear solver failed; then the timestepper can decide what to do, decrease the time step and try again (common) or switch to another method etc. The same could be true for linear solvers called from within optimization routines. The reason we do the funny business with putting infinity into the vector is so that errors that occur on any process get propagated to all processes during the next inner product or norm computation, otherwise we would need to have some "side channel" communication which MPI doesn't really provide to propagate errors to all processes, this would be a lot of performance crippling communication that would have to take place during the entire solution process. Plus our approach since it doesn't change the flow of control cannot result in lost memory etc. Note that C (and MPI) don't have any exception mechanism that could be used to handle these "exceptional cases" directly. Note that you can use options like -snes_error_if_not_converged or -ksp_error_if_not_converged to force the error to be set as soon as any problem with convergence or zero factorization is found (good for debugging, but not usually for production unless production is solving a linear system only). Barry > On Feb 24, 2017, at 2:40 PM, Kong, Fande wrote: > > Hi All, > > In MatSolve(), there is a piece of code: > > if (mat->errortype) { > ierr = PetscInfo1(mat,"MatFactorError %D\n",mat->errortype);CHKERRQ(ierr); > ierr = VecSetInf(x);CHKERRQ(ierr); > } else { > ierr = (*mat->ops->solve)(mat,b,x);CHKERRQ(ierr); > } > > If a direct solver such as LU or superlu fails to create a factorization, why we do not stop here and throw out an error message here or earlier? Now we just let solver keep doing garbage computation, and finally we have a solution like this: > > Vec Object: 1 MPI processes > type: mpi > Process [0] > inf. > inf. > inf. > inf. > inf. > inf. > inf. > inf. > inf. > inf. > inf. > inf. > inf. > inf. > inf. > inf. > > Any particular reason to handle the thing in this way? > > Fande, > > From fande.kong at inl.gov Fri Feb 24 15:08:53 2017 From: fande.kong at inl.gov (Kong, Fande) Date: Fri, 24 Feb 2017 14:08:53 -0700 Subject: [petsc-users] consider an error message as a petscinfo?? In-Reply-To: <2F3D5EED-E357-48AF-AC6D-C51A75BFEB34@mcs.anl.gov> References: <2F3D5EED-E357-48AF-AC6D-C51A75BFEB34@mcs.anl.gov> Message-ID: Thanks a lot for your explanation, Barry, This makes sense! Fande, On Fri, Feb 24, 2017 at 1:56 PM, Barry Smith wrote: > > Fande, > > Yes. Say one is doing a timestepping and using a direct solver. With > time stepping we do not want to necessarily stop (in fact we almost never) > on a failed solve (due to for example a failed factorization). We want the > code to continue up the stack until it gets to the time stepper with the > information that the linear solver failed; then the timestepper can decide > what to do, decrease the time step and try again (common) or switch to > another method etc. The same could be true for linear solvers called from > within optimization routines. > > The reason we do the funny business with putting infinity into the > vector is so that errors that occur on any process get propagated to all > processes during the next inner product or norm computation, otherwise we > would need to have some "side channel" communication which MPI doesn't > really provide to propagate errors to all processes, this would be a lot of > performance crippling communication that would have to take place during > the entire solution process. Plus our approach since it doesn't change the > flow of control cannot result in lost memory etc. Note that C (and MPI) > don't have any exception mechanism that could be used to handle these > "exceptional cases" directly. > > Note that you can use options like -snes_error_if_not_converged or > -ksp_error_if_not_converged to force the error to be set as soon as any > problem with convergence or zero factorization is found (good for > debugging, but not usually for production unless production is solving a > linear system only). > > Barry > > > On Feb 24, 2017, at 2:40 PM, Kong, Fande wrote: > > > > Hi All, > > > > In MatSolve(), there is a piece of code: > > > > if (mat->errortype) { > > ierr = PetscInfo1(mat,"MatFactorError %D\n",mat->errortype);CHKERRQ( > ierr); > > ierr = VecSetInf(x);CHKERRQ(ierr); > > } else { > > ierr = (*mat->ops->solve)(mat,b,x);CHKERRQ(ierr); > > } > > > > If a direct solver such as LU or superlu fails to create a > factorization, why we do not stop here and throw out an error message here > or earlier? Now we just let solver keep doing garbage computation, and > finally we have a solution like this: > > > > Vec Object: 1 MPI processes > > type: mpi > > Process [0] > > inf. > > inf. > > inf. > > inf. > > inf. > > inf. > > inf. > > inf. > > inf. > > inf. > > inf. > > inf. > > inf. > > inf. > > inf. > > inf. > > > > Any particular reason to handle the thing in this way? > > > > Fande, > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gideon.simpson at gmail.com Fri Feb 24 21:12:08 2017 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Fri, 24 Feb 2017 22:12:08 -0500 Subject: [petsc-users] ghost points, DMs, and vector operations Message-ID: I?ve got a simple problem where I use a DM to handle a representation of a vector complex numbers, storing the real and imaginary components at each lattice point. I also have ghost points at either end, i.e.: DMDACreate1d (PETSC_COMM_WORLD, DM_BOUNDARY_GHOSTED, N, 2 , 1 , NULL, &dm ) ; I have a few related questions: 1. Is there a quick way to zero out the ghost points, other than to do DMGetLocalVector(dm,&local); DMGlobalToLocalBegin(dm,global,INSERT_VALUES,local); DMGlobalToLocalEnd(dm,global,INSERT_VALUES,local); DMDAVecGetArray(dm, local, &array); DMDAGetLocalInfo(dm, &info); if(info.xs == 0){ array[-1].u =0.0; array[-1].v =0.0; } if(info.xs + info.xm == info.mx){ array[info.mx].u =0.0; array[info.mx].v =0.0; } /* call restores after */ My point is, I don?t really need to interfere with any of the entries, just the values in those two ghost points. 2. If I take a dot product of two such vectors (associated with the dm, as arrays of complex numbers in terms of real and imaginary parts), does the dot product operate on the ghost points too, or just the ?real? values? 3. If I apply a VecCopy, VecAXPY, or any similar operation to such vectors, do they also operate on the ghost points? -gideon From jed at jedbrown.org Fri Feb 24 23:30:01 2017 From: jed at jedbrown.org (Jed Brown) Date: Fri, 24 Feb 2017 22:30:01 -0700 Subject: [petsc-users] ghost points, DMs, and vector operations In-Reply-To: References: Message-ID: <871sumzvl2.fsf@jedbrown.org> Gideon Simpson writes: > I?ve got a simple problem where I use a DM to handle a representation of a vector complex numbers, storing the real and imaginary components at each lattice point. I also have ghost points at either end, i.e.: > > DMDACreate1d (PETSC_COMM_WORLD, DM_BOUNDARY_GHOSTED, N, 2 , 1 , NULL, &dm ) ; > > I have a few related questions: > > 1. Is there a quick way to zero out the ghost points, other than to do > > DMGetLocalVector(dm,&local); > DMGlobalToLocalBegin(dm,global,INSERT_VALUES,local); > DMGlobalToLocalEnd(dm,global,INSERT_VALUES,local); > DMDAVecGetArray(dm, local, &array); > DMDAGetLocalInfo(dm, &info); > if(info.xs == 0){ > array[-1].u =0.0; > array[-1].v =0.0; > } > if(info.xs + info.xm == info.mx){ > array[info.mx].u =0.0; > array[info.mx].v =0.0; > } > /* call restores after */ You can do it this way to make the loop independent of the number of ghost points, though you likely have to do something different if you have a different number of ghosts. I.e., the simplification only works for toy problems; for anything real you need to write the loops. for (i=info.gxs; i<0; i++) array[i] = 0; for (i=info.mx; i My point is, I don?t really need to interfere with any of the entries, just the values in those two ghost points. > > 2. If I take a dot product of two such vectors (associated with the dm, as arrays of complex numbers in terms of real and imaginary parts), does the dot product operate on the ghost points too, or just the ?real? values? Local vectors are strictly local, and inner products of local vectors rarely makes sense. You probably want a global vector (DMCreateGlobalVector), in which case the dot product is ... global. > 3. If I apply a VecCopy, VecAXPY, or any similar operation to such vectors, do they also operate on the ghost points? "ghost points" aren't special in any way. Local vectors are serial vectors that contain ghost points. Global vectors are parallel vectors that don't have ghost points. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From gideon.simpson at gmail.com Sat Feb 25 00:02:37 2017 From: gideon.simpson at gmail.com (gideon.simpson at gmail.com) Date: Sat, 25 Feb 2017 01:02:37 -0500 Subject: [petsc-users] ghost points, DMs, and vector operations In-Reply-To: <871sumzvl2.fsf@jedbrown.org> References: <871sumzvl2.fsf@jedbrown.org> Message-ID: <266845AE-691B-4A66-8B84-9E80A283335B@gmail.com> Yes, when I was talking about vector operations, i.e., VecAXPY, I was doing them on the global vectors. So what I'm understanding from you is that the ghost points only appear after I go to the local data structure, is that correct? -gideon > On Feb 25, 2017, at 12:30 AM, Jed Brown wrote: > > Gideon Simpson writes: > >> I?ve got a simple problem where I use a DM to handle a representation of a vector complex numbers, storing the real and imaginary components at each lattice point. I also have ghost points at either end, i.e.: >> >> DMDACreate1d (PETSC_COMM_WORLD, DM_BOUNDARY_GHOSTED, N, 2 , 1 , NULL, &dm ) ; >> >> I have a few related questions: >> >> 1. Is there a quick way to zero out the ghost points, other than to do >> >> DMGetLocalVector(dm,&local); >> DMGlobalToLocalBegin(dm,global,INSERT_VALUES,local); >> DMGlobalToLocalEnd(dm,global,INSERT_VALUES,local); >> DMDAVecGetArray(dm, local, &array); >> DMDAGetLocalInfo(dm, &info); >> if(info.xs == 0){ >> array[-1].u =0.0; >> array[-1].v =0.0; >> } >> if(info.xs + info.xm == info.mx){ >> array[info.mx].u =0.0; >> array[info.mx].v =0.0; >> } >> /* call restores after */ > > You can do it this way to make the loop independent of the number of > ghost points, though you likely have to do something different if you > have a different number of ghosts. I.e., the simplification only works > for toy problems; for anything real you need to write the loops. > > for (i=info.gxs; i<0; i++) array[i] = 0; > for (i=info.mx; i > You can also just VecZeroEntries(local) before the global-to-local. > >> My point is, I don?t really need to interfere with any of the entries, just the values in those two ghost points. >> >> 2. If I take a dot product of two such vectors (associated with the dm, as arrays of complex numbers in terms of real and imaginary parts), does the dot product operate on the ghost points too, or just the ?real? values? > > Local vectors are strictly local, and inner products of local vectors > rarely makes sense. You probably want a global vector > (DMCreateGlobalVector), in which case the dot product is ... global. > >> 3. If I apply a VecCopy, VecAXPY, or any similar operation to such vectors, do they also operate on the ghost points? > > "ghost points" aren't special in any way. Local vectors are serial > vectors that contain ghost points. Global vectors are parallel vectors > that don't have ghost points. From jed at jedbrown.org Sat Feb 25 00:05:27 2017 From: jed at jedbrown.org (Jed Brown) Date: Fri, 24 Feb 2017 23:05:27 -0700 Subject: [petsc-users] ghost points, DMs, and vector operations In-Reply-To: <266845AE-691B-4A66-8B84-9E80A283335B@gmail.com> References: <871sumzvl2.fsf@jedbrown.org> <266845AE-691B-4A66-8B84-9E80A283335B@gmail.com> Message-ID: <87r32myfdk.fsf@jedbrown.org> gideon.simpson at gmail.com writes: > Yes, when I was talking about vector operations, i.e., VecAXPY, I was doing them on the global vectors. So what I'm understanding from you is that the ghost points only appear after I go to the local data structure, is that correct? The concept literally does not exist for global vectors. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From gideon.simpson at gmail.com Sat Feb 25 09:32:25 2017 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Sat, 25 Feb 2017 10:32:25 -0500 Subject: [petsc-users] low dimensional sub-problem Message-ID: <40E35B00-FFD1-4B32-9FB7-FAFD23B9927D@gmail.com> I?ve been continuing working on implementing a projection method problem, which, loosely, looks like the following. Vec y contains the state vector for my system, y? = f(y) which is solved with a TS, using, for now, rk4. I have added to this a TSPostStep which does root finding on the nonlinear problem f(lambda) = f(lambda; y, w) = g(y + lambda * w) - g0 y and w are vectors that are the size of the system (numbers of mesh points), and lambda is a scalar perturbation. Right now, the snes function for solving this amounts to: PetscErrorCode projector(SNES snes, Vec lam, Vec f, void *ctx){ AppCtx * user = (AppCtx *) ctx; const PetscScalar *lam_value; PetscScalar *f_value; VecGetArrayRead(lam, &lam_value); VecGetArray(f, &f_value); VecCopy(user->y, user->w); VecAXPY(user->w, lam_value[0], user->gradMy); f_value[0] = g(user->w) -user->g0; VecRestoreArrayRead(lam, &lam_value); VecRestoreArray(f, &f_value); return 0; } To get this all to work, I constructed the SNES and Vec lam with PETSC_COMM_SELF, effectively putting a copy of the nonlinear problem on each process. Basically, the nonlinear problem is low dimensional, but it parametrically depends on the high dimensional, distributed, vectors y and w. The one thing that bothers me about this is: 1. It would seem that each process is is doing all of these vector operations, which is entirely redundant, as the only thing that really needs to be shared amongst the processes is the value of lambda. Is that correct? 2. Is there an obvious way around this redundancy? -gideon From jed at jedbrown.org Sat Feb 25 09:44:40 2017 From: jed at jedbrown.org (Jed Brown) Date: Sat, 25 Feb 2017 08:44:40 -0700 Subject: [petsc-users] low dimensional sub-problem In-Reply-To: <40E35B00-FFD1-4B32-9FB7-FAFD23B9927D@gmail.com> References: <40E35B00-FFD1-4B32-9FB7-FAFD23B9927D@gmail.com> Message-ID: <87o9xqxok7.fsf@jedbrown.org> Gideon Simpson writes: > I?ve been continuing working on implementing a projection method problem, which, loosely, looks like the following. Vec y contains the state vector for my system, y? = f(y) which is solved with a TS, using, for now, rk4. > > I have added to this a TSPostStep which does root finding on the nonlinear problem > > f(lambda) = f(lambda; y, w) = g(y + lambda * w) - g0 You might want to treat this as a DAE. > y and w are vectors that are the size of the system (numbers of mesh points), and lambda is a scalar perturbation. > > Right now, the snes function for solving this amounts to: > > PetscErrorCode projector(SNES snes, Vec lam, Vec f, void *ctx){ > > AppCtx * user = (AppCtx *) ctx; > const PetscScalar *lam_value; > PetscScalar *f_value; > > VecGetArrayRead(lam, &lam_value); > VecGetArray(f, &f_value); > VecCopy(user->y, user->w); > VecAXPY(user->w, lam_value[0], user->gradMy); > > f_value[0] = g(user->w) -user->g0; > VecRestoreArrayRead(lam, &lam_value); > VecRestoreArray(f, &f_value); > > return 0; > } > > To get this all to work, I constructed the SNES and Vec lam with PETSC_COMM_SELF, effectively putting a copy of the nonlinear problem on each process. Basically, the nonlinear problem is low dimensional, but it parametrically depends on the high dimensional, distributed, vectors y and w. > > The one thing that bothers me about this is: > > 1. It would seem that each process is is doing all of these vector operations, which is entirely redundant, as the only thing that really needs to be shared amongst the processes is the value of lambda. Is that correct? Only the SNES part is redundant, but it's super cheap because it's a scalar problem. The Vec operations are using global (parallel, distributed) vectors, so they are not redundant. Of course it is critical for the SNES on every process to be configured identically and deterministic so that the processes don't diverge. And g(user->w) must return the same value on each process (it probably finishes with an MPI_Allreduce or similar). > 2. Is there an obvious way around this redundancy? > > -gideon -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From gideon.simpson at gmail.com Sat Feb 25 10:12:07 2017 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Sat, 25 Feb 2017 11:12:07 -0500 Subject: [petsc-users] low dimensional sub-problem In-Reply-To: <87o9xqxok7.fsf@jedbrown.org> References: <40E35B00-FFD1-4B32-9FB7-FAFD23B9927D@gmail.com> <87o9xqxok7.fsf@jedbrown.org> Message-ID: <67D6B7CF-70B2-416C-A682-AF9DCBFE91A1@gmail.com> -gideon > On Feb 25, 2017, at 10:44 AM, Jed Brown wrote: > > Gideon Simpson writes: > >> I?ve been continuing working on implementing a projection method problem, which, loosely, looks like the following. Vec y contains the state vector for my system, y? = f(y) which is solved with a TS, using, for now, rk4. >> >> I have added to this a TSPostStep which does root finding on the nonlinear problem >> >> f(lambda) = f(lambda; y, w) = g(y + lambda * w) - g0 > > You might want to treat this as a DAE. Yea, we discussed DAE previously. For the sake of a reviewer, I?m trying to implement a classical method (explicit time step + projection), rather than DAE. > >> y and w are vectors that are the size of the system (numbers of mesh points), and lambda is a scalar perturbation. >> >> Right now, the snes function for solving this amounts to: >> >> PetscErrorCode projector(SNES snes, Vec lam, Vec f, void *ctx){ >> >> AppCtx * user = (AppCtx *) ctx; >> const PetscScalar *lam_value; >> PetscScalar *f_value; >> >> VecGetArrayRead(lam, &lam_value); >> VecGetArray(f, &f_value); >> VecCopy(user->y, user->w); >> VecAXPY(user->w, lam_value[0], user->gradMy); >> >> f_value[0] = g(user->w) -user->g0; >> VecRestoreArrayRead(lam, &lam_value); >> VecRestoreArray(f, &f_value); >> >> return 0; >> } >> >> To get this all to work, I constructed the SNES and Vec lam with PETSC_COMM_SELF, effectively putting a copy of the nonlinear problem on each process. Basically, the nonlinear problem is low dimensional, but it parametrically depends on the high dimensional, distributed, vectors y and w. >> >> The one thing that bothers me about this is: >> >> 1. It would seem that each process is is doing all of these vector operations, which is entirely redundant, as the only thing that really needs to be shared amongst the processes is the value of lambda. Is that correct? > > Only the SNES part is redundant, but it's super cheap because it's a > scalar problem. The Vec operations are using global (parallel, > distributed) vectors, so they are not redundant. Of course it is > critical for the SNES on every process to be configured identically and > deterministic so that the processes don't diverge. And g(user->w) must > return the same value on each process (it probably finishes with an > MPI_Allreduce or similar). So within the root finding problem, when it calls, for instance, VecCopy (which is acting on distributed vectors), it?s not doing that once on process 0, once on process 1, once on process 2, etc. Or it is, but you?re saying it?s too cheap to matter? Two things I?m mildly concerned about is that, since user->w changes at each step of the solver, by virtue of the lambda scalar changing, if the SNES?s are running asynchronously, by virtue of them being PETSC_COMM_SELF constructs, couldn?t there be a collision? The operation g(user->w) does indeed conclude with an MPI_Allreduce, followed by DMDAVecRestoreArray and DMRestoreLocalVector. Lastly, in the TSPostStep, it runs as: SNESSolve(user->snes, NULL,user->lam); VecGetArray(user->lam, &lam_value); VecAXPY(y, lam_value[0], user->gradMy); VecRestoreArray(user->lam, &lam_value); Is there any communication issue with ensuring that all the SNES?s have finished on each process, before proceeding to do the vector operations? I keep wondering if I should just write this as a global, with the data structures stored on process 0, just to avoid this kind of headache. > >> 2. Is there an obvious way around this redundancy? >> >> -gideon -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Feb 25 11:00:42 2017 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 25 Feb 2017 11:00:42 -0600 Subject: [petsc-users] low dimensional sub-problem In-Reply-To: <67D6B7CF-70B2-416C-A682-AF9DCBFE91A1@gmail.com> References: <40E35B00-FFD1-4B32-9FB7-FAFD23B9927D@gmail.com> <87o9xqxok7.fsf@jedbrown.org> <67D6B7CF-70B2-416C-A682-AF9DCBFE91A1@gmail.com> Message-ID: On Sat, Feb 25, 2017 at 10:12 AM, Gideon Simpson wrote: > > -gideon > > On Feb 25, 2017, at 10:44 AM, Jed Brown wrote: > > Gideon Simpson writes: > > I?ve been continuing working on implementing a projection method problem, > which, loosely, looks like the following. Vec y contains the state vector > for my system, y? = f(y) which is solved with a TS, using, for now, rk4. > > I have added to this a TSPostStep which does root finding on the nonlinear > problem > > f(lambda) = f(lambda; y, w) = g(y + lambda * w) - g0 > > > You might want to treat this as a DAE. > > > Yea, we discussed DAE previously. For the sake of a reviewer, I?m trying > to implement a classical method (explicit time step + projection), rather > than DAE. > > > y and w are vectors that are the size of the system (numbers of mesh > points), and lambda is a scalar perturbation. > > Right now, the snes function for solving this amounts to: > > PetscErrorCode projector(SNES snes, Vec lam, Vec f, void *ctx){ > > AppCtx * user = (AppCtx *) ctx; > const PetscScalar *lam_value; > PetscScalar *f_value; > > VecGetArrayRead(lam, &lam_value); > VecGetArray(f, &f_value); > VecCopy(user->y, user->w); > VecAXPY(user->w, lam_value[0], user->gradMy); > > f_value[0] = g(user->w) -user->g0; > VecRestoreArrayRead(lam, &lam_value); > VecRestoreArray(f, &f_value); > > return 0; > } > > To get this all to work, I constructed the SNES and Vec lam with > PETSC_COMM_SELF, effectively putting a copy of the nonlinear problem on > each process. Basically, the nonlinear problem is low dimensional, but it > parametrically depends on the high dimensional, distributed, vectors y and > w. > > The one thing that bothers me about this is: > > 1. It would seem that each process is is doing all of these vector > operations, which is entirely redundant, as the only thing that really > needs to be shared amongst the processes is the value of lambda. Is that > correct? > > > Only the SNES part is redundant, but it's super cheap because it's a > scalar problem. The Vec operations are using global (parallel, > distributed) vectors, so they are not redundant. Of course it is > critical for the SNES on every process to be configured identically and > deterministic so that the processes don't diverge. And g(user->w) must > return the same value on each process (it probably finishes with an > MPI_Allreduce or similar). > > > So within the root finding problem, when it calls, for instance, VecCopy > (which is acting on distributed vectors), it?s not doing that once on > process 0, once on process 1, once on process 2, etc. Or it is, but you?re > saying it?s too cheap to matter? > No they are collective. That is why Jed said its crucial that the same branches are taken on all processes. > Two things I?m mildly concerned about is that, since user->w changes at > each step of the solver, by virtue of the lambda scalar changing, if the > SNES?s are running asynchronously, by virtue of them being PETSC_COMM_SELF > constructs, couldn?t there be a collision? > Yes, as I say above. I think its much safer to run a COMM_WORLD SNES that just has nothing on most procs than to do the former since you are already synchronizing everywhere. The operation g(user->w) does indeed conclude with an MPI_Allreduce, > followed by DMDAVecRestoreArray and DMRestoreLocalVector. > > Lastly, in the TSPostStep, it runs as: > > SNESSolve(user->snes, NULL,user->lam); > VecGetArray(user->lam, &lam_value); > VecAXPY(y, lam_value[0], user->gradMy); > VecRestoreArray(user->lam, &lam_value); > > Is there any communication issue with ensuring that all the SNES?s have > finished on each process, before proceeding to do the vector operations? > No. > I keep wondering if I should just write this as a global, with the data > structures stored on process 0, just to avoid this kind of headache. > Yes. Matt > > 2. Is there an obvious way around this redundancy? > > -gideon > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From matt.landreman at gmail.com Sat Feb 25 14:21:46 2017 From: matt.landreman at gmail.com (Matt Landreman) Date: Sat, 25 Feb 2017 15:21:46 -0500 Subject: [petsc-users] Multigrid with defect correction In-Reply-To: <8760k07dix.fsf@jedbrown.org> References: <8760k07dix.fsf@jedbrown.org> Message-ID: Thanks everyone for the help. On item 1 (using Amat different from Pmat with geometric multigrid), I tried Barry's suggestion but it did not seem to resolve the issue. For example, in ksp ex25.c, I tried adding the following lines after line 112: if (J == jac) { ierr = PetscPrintf(PETSC_COMM_WORLD,"Creating a new Amat\n"); ierr = DMCreateMatrix(da,&J); ierr = KSPSetOperators(ksp,J,jac); } ierr = MatShift(J,1.0); This change should set Amat (J) to be different from Pmat (jac), (specifically Amat=identity matrix), so the solution from KSP should be completely different from the original ex25.c. But viewing the solution vector, the solution is unchanged. It seems PETSc is ignoring the Amat created in this approach. Matt K's suggestion of switching from KSP to SNES does work, allowing Amat to differ from Pmat on the finest multigrid level. (On coarser levels, it seems PETSc still forces Amat=Pmat on entry to computeMatrix). On Jed's comment, the application I have in mind is indeed a convection-dominated equation (a steady linear 3D convection-diffusion equation with smoothly varying anisotropic coefficients and recirculating convection). Gamg and hypre-boomerAMG have been working on it ok if I discretize with low-order upwind differences in Pmat and use Amat=Pmat, but I'd like higher order accuracy. Using gmres with a higher-order discretization in Amat and low-order Pmat works ok, but the number of KSP iterations required gets large as the diffusion gets small compared to convection, even with -pc_type lu. So I'm working to see if geometric mg with defect correction at each level can do better. Thanks, Matt Landreman On Thu, Feb 23, 2017 at 5:23 PM, Jed Brown wrote: > Matt Landreman writes: > > 3. Is it at all sensible to do this second kind of defect correction with > > _algebraic_ multigrid? Perhaps Amat for each level could be formed from > the > > high-order matrix at the fine level by the Galerkin operator R A P, after > > getting all the restriction matrices created by gamg for Pmat? > > Note that defect correction is most commonly used for > transport-dominated processes for which the high order discretization is > not h-elliptic. AMG heuristics are typically really bad for such > problems so stabilizing a smoother isn't really a relevant issue. Also, > it is usually used for strongly nonlinear problems where AMG's setup > costs are likely overkill. This is not to say that defect correction > AMG won't work, but there is reason to believe that it doesn't matter > for many important problems. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Feb 25 14:45:52 2017 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 25 Feb 2017 14:45:52 -0600 Subject: [petsc-users] Multigrid with defect correction In-Reply-To: References: <8760k07dix.fsf@jedbrown.org> Message-ID: On Sat, Feb 25, 2017 at 2:21 PM, Matt Landreman wrote: > Thanks everyone for the help. > > On item 1 (using Amat different from Pmat with geometric multigrid), I > tried Barry's suggestion but it did not seem to resolve the issue. For > example, in ksp ex25.c, I tried adding the following lines after line 112: > if (J == jac) { > > ierr = PetscPrintf(PETSC_COMM_WORLD,"Creating a new Amat\n"); > > ierr = DMCreateMatrix(da,&J); > > ierr = KSPSetOperators(ksp,J,jac); > > } > ierr = MatShift(J,1.0); > > This change should set Amat (J) to be different from Pmat (jac), > (specifically Amat=identity matrix), so the solution from KSP should be > completely different from the original ex25.c. But viewing the solution > vector, the solution is unchanged. It seems PETSc is ignoring the Amat > created in this approach. > > Matt K's suggestion of switching from KSP to SNES does work, allowing Amat > to differ from Pmat on the finest multigrid level. (On coarser levels, it > seems PETSc still forces Amat=Pmat on entry to computeMatrix). > You should not be using computeMatrix() anymore with SNES, which is why it will work. You use the DM callback. See for DA SNES ex5, or for Plex see SNES ex12. Thanks Matt > On Jed's comment, the application I have in mind is indeed a > convection-dominated equation (a steady linear 3D convection-diffusion > equation with smoothly varying anisotropic coefficients and recirculating > convection). Gamg and hypre-boomerAMG have been working on it ok if I > discretize with low-order upwind differences in Pmat and use Amat=Pmat, but > I'd like higher order accuracy. Using gmres with a higher-order > discretization in Amat and low-order Pmat works ok, but the number of KSP > iterations required gets large as the diffusion gets small compared to > convection, even with -pc_type lu. So I'm working to see if geometric mg > with defect correction at each level can do better. > > Thanks, > Matt Landreman > > > > > On Thu, Feb 23, 2017 at 5:23 PM, Jed Brown wrote: > >> Matt Landreman writes: >> > 3. Is it at all sensible to do this second kind of defect correction >> with >> > _algebraic_ multigrid? Perhaps Amat for each level could be formed from >> the >> > high-order matrix at the fine level by the Galerkin operator R A P, >> after >> > getting all the restriction matrices created by gamg for Pmat? >> >> Note that defect correction is most commonly used for >> transport-dominated processes for which the high order discretization is >> not h-elliptic. AMG heuristics are typically really bad for such >> problems so stabilizing a smoother isn't really a relevant issue. Also, >> it is usually used for strongly nonlinear problems where AMG's setup >> costs are likely overkill. This is not to say that defect correction >> AMG won't work, but there is reason to believe that it doesn't matter >> for many important problems. >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Feb 25 16:57:51 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 25 Feb 2017 16:57:51 -0600 Subject: [petsc-users] low dimensional sub-problem In-Reply-To: References: <40E35B00-FFD1-4B32-9FB7-FAFD23B9927D@gmail.com> <87o9xqxok7.fsf@jedbrown.org> <67D6B7CF-70B2-416C-A682-AF9DCBFE91A1@gmail.com> Message-ID: <2A87DDD4-653C-41CE-9762-B66828918B43@mcs.anl.gov> > On Feb 25, 2017, at 11:00 AM, Matthew Knepley wrote: > > On Sat, Feb 25, 2017 at 10:12 AM, Gideon Simpson wrote: > > -gideon > >> On Feb 25, 2017, at 10:44 AM, Jed Brown wrote: >> >> Gideon Simpson writes: >> >>> I?ve been continuing working on implementing a projection method problem, which, loosely, looks like the following. Vec y contains the state vector for my system, y? = f(y) which is solved with a TS, using, for now, rk4. >>> >>> I have added to this a TSPostStep which does root finding on the nonlinear problem >>> >>> f(lambda) = f(lambda; y, w) = g(y + lambda * w) - g0 >> >> You might want to treat this as a DAE. > > Yea, we discussed DAE previously. For the sake of a reviewer, I?m trying to implement a classical method (explicit time step + projection), rather than DAE. > >> >>> y and w are vectors that are the size of the system (numbers of mesh points), and lambda is a scalar perturbation. >>> >>> Right now, the snes function for solving this amounts to: >>> >>> PetscErrorCode projector(SNES snes, Vec lam, Vec f, void *ctx){ >>> >>> AppCtx * user = (AppCtx *) ctx; >>> const PetscScalar *lam_value; >>> PetscScalar *f_value; >>> >>> VecGetArrayRead(lam, &lam_value); >>> VecGetArray(f, &f_value); >>> VecCopy(user->y, user->w); If you do make SNES run on PETSC_COMM_WORLD where lam is now a parallel vector with 1 entry on process 0 and zero entries on all the others (and similar for f) You will need to broadcast the lam_value[0] to all processes before calling the VecAXPY() >>> VecAXPY(user->w, lam_value[0], user->gradMy); >>> >>> f_value[0] = g(user->w) -user->g0; >>> VecRestoreArrayRead(lam, &lam_value); >>> VecRestoreArray(f, &f_value); >>> >>> return 0; >>> } >>> >>> To get this all to work, I constructed the SNES and Vec lam with PETSC_COMM_SELF, effectively putting a copy of the nonlinear problem on each process. Basically, the nonlinear problem is low dimensional, but it parametrically depends on the high dimensional, distributed, vectors y and w. >>> >>> The one thing that bothers me about this is: >>> >>> 1. It would seem that each process is is doing all of these vector operations, which is entirely redundant, as the only thing that really needs to be shared amongst the processes is the value of lambda. Is that correct? >> >> Only the SNES part is redundant, but it's super cheap because it's a >> scalar problem. The Vec operations are using global (parallel, >> distributed) vectors, so they are not redundant. Of course it is >> critical for the SNES on every process to be configured identically and >> deterministic so that the processes don't diverge. And g(user->w) must >> return the same value on each process (it probably finishes with an >> MPI_Allreduce or similar). > > So within the root finding problem, when it calls, for instance, VecCopy (which is acting on distributed vectors), it?s not doing that once on process 0, once on process 1, once on process 2, etc. Or it is, but you?re saying it?s too cheap to matter? Each process is copying ITs part of the vector; which is exactly what you need. > > No they are collective. That is why Jed said its crucial that the same branches are taken on all processes. > > Two things I?m mildly concerned about is that, since user->w changes at each step of the solver, by virtue of the lambda scalar changing, if the SNES?s are running asynchronously, by virtue of them being PETSC_COMM_SELF constructs, couldn?t there be a collision? > > Yes, as I say above. I think its much safer to run a COMM_WORLD SNES that just has nothing on most procs than to do the former > since you are already synchronizing everywhere. > > The operation g(user->w) does indeed conclude with an MPI_Allreduce, followed by DMDAVecRestoreArray and DMRestoreLocalVector. > > Lastly, in the TSPostStep, it runs as: > > SNESSolve(user->snes, NULL,user->lam); > VecGetArray(user->lam, &lam_value); Again here you would need to broadcast this user->lam value to all processes before calling the VecAXPY() since it is only stored on process zero. > VecAXPY(y, lam_value[0], user->gradMy); > VecRestoreArray(user->lam, &lam_value); > > Is there any communication issue with ensuring that all the SNES?s have finished on each process, before proceeding to do the vector operations? > > No. > > I keep wondering if I should just write this as a global, with the data structures stored on process 0, just to avoid this kind of headache. > > Yes. > > Matt > >> >>> 2. Is there an obvious way around this redundancy? >>> >>> -gideon > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From jed at jedbrown.org Sat Feb 25 18:13:33 2017 From: jed at jedbrown.org (Jed Brown) Date: Sat, 25 Feb 2017 17:13:33 -0700 Subject: [petsc-users] Multigrid with defect correction In-Reply-To: References: <8760k07dix.fsf@jedbrown.org> Message-ID: <87innxyfki.fsf@jedbrown.org> Matt Landreman writes: > On Jed's comment, the application I have in mind is indeed a > convection-dominated equation (a steady linear 3D convection-diffusion > equation with smoothly varying anisotropic coefficients and recirculating > convection). Gamg and hypre-boomerAMG have been working on it ok if I > discretize with low-order upwind differences in Pmat and use Amat=Pmat, but > I'd like higher order accuracy. Using gmres with a higher-order > discretization in Amat and low-order Pmat works ok, but the number of KSP > iterations required gets large as the diffusion gets small compared to > convection, even with -pc_type lu. So I'm working to see if geometric mg > with defect correction at each level can do better. You asked about algebraic multigrid. A first order discretization has a ton of numerical diffusion so no matter how hard you try, you can't actually produce a large cell Peclet number. I.e., AMG working on the low-order operator has nothing to do with high Peclet number. With geometric multigrid, you typically rediscretize the coarse operators and defect correction makes sense. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From gideon.simpson at gmail.com Sat Feb 25 18:24:48 2017 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Sat, 25 Feb 2017 19:24:48 -0500 Subject: [petsc-users] low dimensional sub-problem In-Reply-To: <2A87DDD4-653C-41CE-9762-B66828918B43@mcs.anl.gov> References: <40E35B00-FFD1-4B32-9FB7-FAFD23B9927D@gmail.com> <87o9xqxok7.fsf@jedbrown.org> <67D6B7CF-70B2-416C-A682-AF9DCBFE91A1@gmail.com> <2A87DDD4-653C-41CE-9762-B66828918B43@mcs.anl.gov> Message-ID: <0B5B91AC-AB18-4A47-81A1-07F4D78123E9@gmail.com> Yea, I implemented it as suggested by Matt and Barry with the BCast of the element that sits only on process 0, and that seems to work fine. -gideon > On Feb 25, 2017, at 5:57 PM, Barry Smith wrote: > > >> On Feb 25, 2017, at 11:00 AM, Matthew Knepley wrote: >> >> On Sat, Feb 25, 2017 at 10:12 AM, Gideon Simpson wrote: >> >> -gideon >> >>> On Feb 25, 2017, at 10:44 AM, Jed Brown wrote: >>> >>> Gideon Simpson writes: >>> >>>> I?ve been continuing working on implementing a projection method problem, which, loosely, looks like the following. Vec y contains the state vector for my system, y? = f(y) which is solved with a TS, using, for now, rk4. >>>> >>>> I have added to this a TSPostStep which does root finding on the nonlinear problem >>>> >>>> f(lambda) = f(lambda; y, w) = g(y + lambda * w) - g0 >>> >>> You might want to treat this as a DAE. >> >> Yea, we discussed DAE previously. For the sake of a reviewer, I?m trying to implement a classical method (explicit time step + projection), rather than DAE. >> >>> >>>> y and w are vectors that are the size of the system (numbers of mesh points), and lambda is a scalar perturbation. >>>> >>>> Right now, the snes function for solving this amounts to: >>>> >>>> PetscErrorCode projector(SNES snes, Vec lam, Vec f, void *ctx){ >>>> >>>> AppCtx * user = (AppCtx *) ctx; >>>> const PetscScalar *lam_value; >>>> PetscScalar *f_value; >>>> >>>> VecGetArrayRead(lam, &lam_value); >>>> VecGetArray(f, &f_value); >>>> VecCopy(user->y, user->w); > > If you do make SNES run on PETSC_COMM_WORLD where lam is now a parallel vector with 1 entry on process 0 and zero entries on all the others (and similar for f) > You will need to broadcast the lam_value[0] to all processes before calling the VecAXPY() > >>>> VecAXPY(user->w, lam_value[0], user->gradMy); >>>> >>>> f_value[0] = g(user->w) -user->g0; >>>> VecRestoreArrayRead(lam, &lam_value); >>>> VecRestoreArray(f, &f_value); >>>> >>>> return 0; >>>> } >>>> >>>> To get this all to work, I constructed the SNES and Vec lam with PETSC_COMM_SELF, effectively putting a copy of the nonlinear problem on each process. Basically, the nonlinear problem is low dimensional, but it parametrically depends on the high dimensional, distributed, vectors y and w. >>>> >>>> The one thing that bothers me about this is: >>>> >>>> 1. It would seem that each process is is doing all of these vector operations, which is entirely redundant, as the only thing that really needs to be shared amongst the processes is the value of lambda. Is that correct? >>> >>> Only the SNES part is redundant, but it's super cheap because it's a >>> scalar problem. The Vec operations are using global (parallel, >>> distributed) vectors, so they are not redundant. Of course it is >>> critical for the SNES on every process to be configured identically and >>> deterministic so that the processes don't diverge. And g(user->w) must >>> return the same value on each process (it probably finishes with an >>> MPI_Allreduce or similar). >> >> So within the root finding problem, when it calls, for instance, VecCopy (which is acting on distributed vectors), it?s not doing that once on process 0, once on process 1, once on process 2, etc. Or it is, but you?re saying it?s too cheap to matter? > > Each process is copying ITs part of the vector; which is exactly what you need. >> >> No they are collective. That is why Jed said its crucial that the same branches are taken on all processes. >> >> Two things I?m mildly concerned about is that, since user->w changes at each step of the solver, by virtue of the lambda scalar changing, if the SNES?s are running asynchronously, by virtue of them being PETSC_COMM_SELF constructs, couldn?t there be a collision? >> >> Yes, as I say above. I think its much safer to run a COMM_WORLD SNES that just has nothing on most procs than to do the former >> since you are already synchronizing everywhere. >> >> The operation g(user->w) does indeed conclude with an MPI_Allreduce, followed by DMDAVecRestoreArray and DMRestoreLocalVector. >> >> Lastly, in the TSPostStep, it runs as: >> >> SNESSolve(user->snes, NULL,user->lam); >> VecGetArray(user->lam, &lam_value); > > Again here you would need to broadcast this user->lam value to all processes before calling the VecAXPY() since it is only stored on process zero. > >> VecAXPY(y, lam_value[0], user->gradMy); >> VecRestoreArray(user->lam, &lam_value); >> >> Is there any communication issue with ensuring that all the SNES?s have finished on each process, before proceeding to do the vector operations? >> >> No. >> >> I keep wondering if I should just write this as a global, with the data structures stored on process 0, just to avoid this kind of headache. >> >> Yes. >> >> Matt >> >>> >>>> 2. Is there an obvious way around this redundancy? >>>> >>>> -gideon >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener > From bsmith at mcs.anl.gov Sat Feb 25 18:26:34 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 25 Feb 2017 18:26:34 -0600 Subject: [petsc-users] Multigrid with defect correction In-Reply-To: References: <8760k07dix.fsf@jedbrown.org> Message-ID: <01114AA8-3460-4F8A-A579-0A0CB3AA33C3@mcs.anl.gov> > On Feb 25, 2017, at 2:21 PM, Matt Landreman wrote: > > Thanks everyone for the help. > > On item 1 (using Amat different from Pmat with geometric multigrid), I tried Barry's suggestion but it did not seem to resolve the issue. For example, in ksp ex25.c, I tried adding the following lines after line 112: > if (J == jac) { > ierr = PetscPrintf(PETSC_COMM_WORLD,"Creating a new Amat\n"); > ierr = DMCreateMatrix(da,&J); > ierr = KSPSetOperators(ksp,J,jac); > } > ierr = MatShift(J,1.0); > > This change should set Amat (J) to be different from Pmat (jac), (specifically Amat=identity matrix), so the solution from KSP should be completely different from the original ex25.c. But viewing the solution vector, the solution is unchanged. It seems PETSc is ignoring the Amat created in this approach. Ahh, yes a multiple number of errors on my part. 1) there was old code that immediately overwrote the newly set KSPSetOperators() after the call to ComputeMatrix(), hence simply removing the new user provided matrix, fixed in commit d73a0ed5842d2f684a9729e0b9c14ba0e3d554c3 2) the default behavior of PCMG was to use the pmat on each level to compute the residual, not the mat, fixed in commit 13842ffb25245dcab28ec15282db994aa7b56c77 so even after 1) was fixed it would still be using "the wrong" matrix to compute the residuals. 3) Even after 2) was fixed my advice still would not do what I wanted because the default setting of the matrix to compute the residual on each level (the call the PCMGSetResidual() in mg.c) occurs BEFORE any calls to ComputeMatrix() so it would still use the original matrix provided by the DM and not the new one provided by the user. The best solution I see moving forward is to actually properly support providing two DMs to KSP, one responsible for the "true" operator and one responsible for the "approximate" operator used in constructing the preconditioner. Conceptually looking back this makes perfect sense and should have been done the very first day that I introduced the concept of providing a DM to the KSP. Unfortunately this requires a good amount of replumbing; wherever we do refinement/coarsening of DM we need to manage it for both, the horrible DMKSP/SNES/TS construct that Jed needed to introduce to get user callbacks working needs to handle the two DMs in some way, plus TS and SNES need to handle the two DMs. There are additional complications in managing the user callbacks for function evaluations, when they should utilize the first DM and when they should utilize the second; for Newton based TS and SNES solves presumably they should almost always be just using the higher order function, but for nonlinear multigrid and other non-Newton based nonlinear solvers there are presumably places where the lower order function should be called. The end result of the refactorization is very good however since it will then be straightforward to support many cases of the "high order discretization for operators, but low order discretization for preconditioners" paradigms, not just for DMDA but for other DMS without the usual PETSc model and code. Matt, So yes we definitely want to support what you want to do and have always intended to support it eventually, now is the time we start to add the support. I'll start a git branch shortly that begins to make it simple for your case of KSP; extending it everywhere will take more time. Barry > > Matt K's suggestion of switching from KSP to SNES does work, allowing Amat to differ from Pmat on the finest multigrid level. (On coarser levels, it seems PETSc still forces Amat=Pmat on entry to computeMatrix). > > On Jed's comment, the application I have in mind is indeed a convection-dominated equation (a steady linear 3D convection-diffusion equation with smoothly varying anisotropic coefficients and recirculating convection). Gamg and hypre-boomerAMG have been working on it ok if I discretize with low-order upwind differences in Pmat and use Amat=Pmat, but I'd like higher order accuracy. Using gmres with a higher-order discretization in Amat and low-order Pmat works ok, but the number of KSP iterations required gets large as the diffusion gets small compared to convection, even with -pc_type lu. So I'm working to see if geometric mg with defect correction at each level can do better. > > Thanks, > Matt Landreman > > > > > On Thu, Feb 23, 2017 at 5:23 PM, Jed Brown wrote: > Matt Landreman writes: > > 3. Is it at all sensible to do this second kind of defect correction with > > _algebraic_ multigrid? Perhaps Amat for each level could be formed from the > > high-order matrix at the fine level by the Galerkin operator R A P, after > > getting all the restriction matrices created by gamg for Pmat? > > Note that defect correction is most commonly used for > transport-dominated processes for which the high order discretization is > not h-elliptic. AMG heuristics are typically really bad for such > problems so stabilizing a smoother isn't really a relevant issue. Also, > it is usually used for strongly nonlinear problems where AMG's setup > costs are likely overkill. This is not to say that defect correction > AMG won't work, but there is reason to believe that it doesn't matter > for many important problems. > From bsmith at mcs.anl.gov Sat Feb 25 18:27:20 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 25 Feb 2017 18:27:20 -0600 Subject: [petsc-users] low dimensional sub-problem In-Reply-To: <0B5B91AC-AB18-4A47-81A1-07F4D78123E9@gmail.com> References: <40E35B00-FFD1-4B32-9FB7-FAFD23B9927D@gmail.com> <87o9xqxok7.fsf@jedbrown.org> <67D6B7CF-70B2-416C-A682-AF9DCBFE91A1@gmail.com> <2A87DDD4-653C-41CE-9762-B66828918B43@mcs.anl.gov> <0B5B91AC-AB18-4A47-81A1-07F4D78123E9@gmail.com> Message-ID: <8AD874E6-62EE-467A-8943-3FC65C319D10@mcs.anl.gov> Great, thanks for letting us know. Barry > On Feb 25, 2017, at 6:24 PM, Gideon Simpson wrote: > > Yea, I implemented it as suggested by Matt and Barry with the BCast of the element that sits only on process 0, and that seems to work fine. > > -gideon > >> On Feb 25, 2017, at 5:57 PM, Barry Smith wrote: >> >> >>> On Feb 25, 2017, at 11:00 AM, Matthew Knepley wrote: >>> >>> On Sat, Feb 25, 2017 at 10:12 AM, Gideon Simpson wrote: >>> >>> -gideon >>> >>>> On Feb 25, 2017, at 10:44 AM, Jed Brown wrote: >>>> >>>> Gideon Simpson writes: >>>> >>>>> I?ve been continuing working on implementing a projection method problem, which, loosely, looks like the following. Vec y contains the state vector for my system, y? = f(y) which is solved with a TS, using, for now, rk4. >>>>> >>>>> I have added to this a TSPostStep which does root finding on the nonlinear problem >>>>> >>>>> f(lambda) = f(lambda; y, w) = g(y + lambda * w) - g0 >>>> >>>> You might want to treat this as a DAE. >>> >>> Yea, we discussed DAE previously. For the sake of a reviewer, I?m trying to implement a classical method (explicit time step + projection), rather than DAE. >>> >>>> >>>>> y and w are vectors that are the size of the system (numbers of mesh points), and lambda is a scalar perturbation. >>>>> >>>>> Right now, the snes function for solving this amounts to: >>>>> >>>>> PetscErrorCode projector(SNES snes, Vec lam, Vec f, void *ctx){ >>>>> >>>>> AppCtx * user = (AppCtx *) ctx; >>>>> const PetscScalar *lam_value; >>>>> PetscScalar *f_value; >>>>> >>>>> VecGetArrayRead(lam, &lam_value); >>>>> VecGetArray(f, &f_value); >>>>> VecCopy(user->y, user->w); >> >> If you do make SNES run on PETSC_COMM_WORLD where lam is now a parallel vector with 1 entry on process 0 and zero entries on all the others (and similar for f) >> You will need to broadcast the lam_value[0] to all processes before calling the VecAXPY() >> >>>>> VecAXPY(user->w, lam_value[0], user->gradMy); >>>>> >>>>> f_value[0] = g(user->w) -user->g0; >>>>> VecRestoreArrayRead(lam, &lam_value); >>>>> VecRestoreArray(f, &f_value); >>>>> >>>>> return 0; >>>>> } >>>>> >>>>> To get this all to work, I constructed the SNES and Vec lam with PETSC_COMM_SELF, effectively putting a copy of the nonlinear problem on each process. Basically, the nonlinear problem is low dimensional, but it parametrically depends on the high dimensional, distributed, vectors y and w. >>>>> >>>>> The one thing that bothers me about this is: >>>>> >>>>> 1. It would seem that each process is is doing all of these vector operations, which is entirely redundant, as the only thing that really needs to be shared amongst the processes is the value of lambda. Is that correct? >>>> >>>> Only the SNES part is redundant, but it's super cheap because it's a >>>> scalar problem. The Vec operations are using global (parallel, >>>> distributed) vectors, so they are not redundant. Of course it is >>>> critical for the SNES on every process to be configured identically and >>>> deterministic so that the processes don't diverge. And g(user->w) must >>>> return the same value on each process (it probably finishes with an >>>> MPI_Allreduce or similar). >>> >>> So within the root finding problem, when it calls, for instance, VecCopy (which is acting on distributed vectors), it?s not doing that once on process 0, once on process 1, once on process 2, etc. Or it is, but you?re saying it?s too cheap to matter? >> >> Each process is copying ITs part of the vector; which is exactly what you need. >>> >>> No they are collective. That is why Jed said its crucial that the same branches are taken on all processes. >>> >>> Two things I?m mildly concerned about is that, since user->w changes at each step of the solver, by virtue of the lambda scalar changing, if the SNES?s are running asynchronously, by virtue of them being PETSC_COMM_SELF constructs, couldn?t there be a collision? >>> >>> Yes, as I say above. I think its much safer to run a COMM_WORLD SNES that just has nothing on most procs than to do the former >>> since you are already synchronizing everywhere. >>> >>> The operation g(user->w) does indeed conclude with an MPI_Allreduce, followed by DMDAVecRestoreArray and DMRestoreLocalVector. >>> >>> Lastly, in the TSPostStep, it runs as: >>> >>> SNESSolve(user->snes, NULL,user->lam); >>> VecGetArray(user->lam, &lam_value); >> >> Again here you would need to broadcast this user->lam value to all processes before calling the VecAXPY() since it is only stored on process zero. >> >>> VecAXPY(y, lam_value[0], user->gradMy); >>> VecRestoreArray(user->lam, &lam_value); >>> >>> Is there any communication issue with ensuring that all the SNES?s have finished on each process, before proceeding to do the vector operations? >>> >>> No. >>> >>> I keep wondering if I should just write this as a global, with the data structures stored on process 0, just to avoid this kind of headache. >>> >>> Yes. >>> >>> Matt >>> >>>> >>>>> 2. Is there an obvious way around this redundancy? >>>>> >>>>> -gideon >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >> From fsantost at student.ethz.ch Sun Feb 26 10:51:00 2017 From: fsantost at student.ethz.ch (Santos Teixeira Frederico) Date: Sun, 26 Feb 2017 16:51:00 +0000 Subject: [petsc-users] Multi-domain meshes with DMPLEX Message-ID: <682CC3CD7A208742B8C2D116C67199013F3C3A65@MBX23.d.ethz.ch> Hi folks, Attached you find small.msh, a mesh with two prisms (physical domains 43 and 45) and the interface between them (physical domain 40). test_dmplex.cpp loads the mesh and tries to filter each sub-mesh and the interface. I use PETSc master. When I load small.msh, all physical domains are correctly assigned to "Cell Sets" or "Face Sets". 1) Both sub-meshes (prisms with domains 43 and 45) are filtered with DMPlexFilter and saved correctly, but the interface isn't. What am I missing? 2) I can retrieve any specific strata of "Cell Sets" with DMPlexFilter, but this operation does not include any related strata of "Face Sets", as pointed by DMView. How can I do it? 3) Physical domains 43 and 45 could be interpreted as fluid and solid domains in a F.S.I. simulation, for example, and I would like to define fluid and solid sub-problems only on the respective sub-mesh. Do I really need to have fully operational sub-DMPLEX's in order to achieve it? Is there an easier way? My apologies if the explanations are confusing! Best regards, Frederico. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: config_small.txt URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: small.msh Type: application/octet-stream Size: 1762 bytes Desc: small.msh URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test_dmplex.cpp Type: application/octet-stream Size: 4216 bytes Desc: test_dmplex.cpp URL: From knepley at gmail.com Sun Feb 26 20:26:21 2017 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 26 Feb 2017 20:26:21 -0600 Subject: [petsc-users] Multi-domain meshes with DMPLEX In-Reply-To: <682CC3CD7A208742B8C2D116C67199013F3C3A65@MBX23.d.ethz.ch> References: <682CC3CD7A208742B8C2D116C67199013F3C3A65@MBX23.d.ethz.ch> Message-ID: On Sun, Feb 26, 2017 at 10:51 AM, Santos Teixeira Frederico < fsantost at student.ethz.ch> wrote: > Hi folks, > > Attached you find small.msh, a mesh with two prisms (physical domains 43 > and 45) and the interface between them (physical domain 40). > test_dmplex.cpp loads the mesh and tries to filter each sub-mesh and the > interface. I use PETSc master. > > When I load small.msh, all physical domains are correctly assigned to "Cell > Sets" or "Face Sets". > > 1) Both sub-meshes (prisms with domains 43 and 45) are filtered with > DMPlexFilter and saved correctly, but the interface isn't. What am I > missing? > What exactly does "the interface isn't" mean? Does it mean you are not getting what you expect from the submesh? One way to more easily communicate about this is to replace your blocks of Viewer commands with one line DMViewFromOptions() calls. That way you can show me just -dm_view or -dm_view ::ascii_info_detail, or output VTK with -dm_view vtk:mesh.vtu, or HDF5 -dm_view hdf5:mesh.h5, etc. We could start off with sending me just the plain ASCII output for the mesh and submeshes. I do recognize you sent the code, but I have a conference deadline for Monday (I will be there until Friday), so I am unlikely to build it before them. > 2) I can retrieve any specific strata of "Cell Sets" with DMPlexFilter, > but this operation does not include any related strata of "Face Sets", as > pointed by DMView. How can I do it? > Do DMFilter is not propagating labels? Okay, that can go on the bug list. Is this what you mean? > 3) Physical domains 43 and 45 could be interpreted as fluid and solid > domains in a F.S.I. simulation, for example, and I would like to define > fluid and solid sub-problems only on the respective sub-mesh. Do I really > need to have fully operational sub-DMPLEX's in order to achieve it? Is > there an easier way? > I am not sure I understand what is going wrong yet. Thanks, Matt > My apologies if the explanations are confusing! > > Best regards, > Frederico. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From fangbowa at buffalo.edu Sun Feb 26 21:30:44 2017 From: fangbowa at buffalo.edu (Fangbo Wang) Date: Sun, 26 Feb 2017 22:30:44 -0500 Subject: [petsc-users] Why my petsc program get stuck and hang there when assembling the matrix? Message-ID: Hi, I construct a big matrix which is 1.3million by 1.3million which is using approximatly 100GB memory. I have a computer with 500GB memory. I run the Petsc program and it get stuck when finally assembling the matrix. The program is using around 200GB memory only. However, the program just get stuck there. Here is the output message when it gets stuck. . . *previous outputs not shown here* . [12] MatStashScatterBegin_Ref(): No of messages: 15 [12] MatStashScatterBegin_Ref(): Mesg_to: 0: size: 271636416 bytes [12] MatStashScatterBegin_Ref(): Mesg_to: 1: size: 328581552 bytes [12] MatStashScatterBegin_Ref(): Mesg_to: 2: size: 163649328 bytes [12] MatStashScatterBegin_Ref(): Mesg_to: 3: size: 95512224 bytes [12] MatStashScatterBegin_Ref(): Mesg_to: 4: size: 317711616 bytes [12] MatStashScatterBegin_Ref(): Mesg_to: 5: size: 170971776 bytes [12] MatStashScatterBegin_Ref(): Mesg_to: 6: size: 254000064 bytes [12] MatStashScatterBegin_Ref(): Mesg_to: 7: size: 163146720 bytes [12] MatStashScatterBegin_Ref(): Mesg_to: 8: size: 345150048 bytes [12] MatStashScatterBegin_Ref(): Mesg_to: 9: size: 163411584 bytes [12] MatStashScatterBegin_Ref(): Mesg_to: 10: size: 428874816 bytes [12] MatStashScatterBegin_Ref(): Mesg_to: 11: size: 739711296 bytes [12] MatStashScatterBegin_Ref(): Mesg_to: 13: size: 435247344 bytes [12] MatStashScatterBegin_Ref(): Mesg_to: 14: size: 435136752 bytes [12] MatStashScatterBegin_Ref(): Mesg_to: 15: size: 346167552 bytes [14] MatAssemblyBegin_MPIAIJ(): Stash has 263158893 entries, uses 14 mallocs. [8] MatAssemblyBegin_MPIAIJ(): Stash has 286768572 entries, uses 14 mallocs. [12] MatAssemblyBegin_MPIAIJ(): Stash has 291181818 entries, uses 14 mallocs. [13] MatStashScatterBegin_Ref(): No of messages: 15 [13] MatStashScatterBegin_Ref(): Mesg_to: 0: size: 271636416 bytes [13] MatStashScatterBegin_Ref(): Mesg_to: 1: size: 271636416 bytes [13] MatStashScatterBegin_Ref(): Mesg_to: 2: size: 220594464 bytes [13] MatStashScatterBegin_Ref(): Mesg_to: 3: size: 51041952 bytes [13] MatStashScatterBegin_Ref(): Mesg_to: 4: size: 276201408 bytes [13] MatStashScatterBegin_Ref(): Mesg_to: 5: size: 256952256 bytes [13] MatStashScatterBegin_Ref(): Mesg_to: 6: size: 198489024 bytes [13] MatStashScatterBegin_Ref(): Mesg_to: 7: size: 218657760 bytes [13] MatStashScatterBegin_Ref(): Mesg_to: 8: size: 219686880 bytes [13] MatStashScatterBegin_Ref(): Mesg_to: 9: size: 288874752 bytes [13] MatStashScatterBegin_Ref(): Mesg_to: 10: size: 428874816 bytes [13] MatStashScatterBegin_Ref(): Mesg_to: 11: size: 172579968 bytes [13] MatStashScatterBegin_Ref(): Mesg_to: 12: size: 639835680 bytes [13] MatStashScatterBegin_Ref(): Mesg_to: 14: size: 270060144 bytes [13] MatStashScatterBegin_Ref(): Mesg_to: 15: size: 511244160 bytes [13] MatAssemblyBegin_MPIAIJ(): Stash has 268522881 entries, uses 14 mallocs. [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 96812 X 96812; storage space: 89786788 unneeded,7025212 used [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 81 [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 96812) < 0.6. Do not use CompressedRow routines. [5] MatSeqAIJCheckInode(): Found 32271 nodes of 96812. Limit used: 5. Using Inode routines [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 96812 X 96812; storage space: 89841924 unneeded,6970076 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 81 [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 96812) < 0.6. Do not use CompressedRow routines. [4] MatSeqAIJCheckInode(): Found 32272 nodes of 96812. Limit used: 5. Using Inode routines *stuck here!!!!* Any one have ideas on this? Thank you very much! Fangbo Wang -- Fangbo Wang, PhD student Stochastic Geomechanics Research Group Department of Civil, Structural and Environmental Engineering University at Buffalo Email: *fangbowa at buffalo.edu * -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Feb 26 21:42:02 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 26 Feb 2017 21:42:02 -0600 Subject: [petsc-users] Why my petsc program get stuck and hang there when assembling the matrix? In-Reply-To: References: Message-ID: <258CF5B2-CCFD-4AB9-9535-CAE1661018D0@mcs.anl.gov> How are you generating the matrix entries in parallel? In general you can generate any matrix entries on any MPI process and they will be automatically transferred to the MPI process that owns the entries automatically. BUT if a huge number of matrix entries are computed on one process and need to be communicated to another process this may cause gridlock with MPI. Based on the huge size of messages from process 12 it looks like this is what is happening in your code. Ideally most matrix entries are generated on the process they are stored and hence this gridlock does not happen. What type of discretization are you using? Finite differences, finite element, finite volume, spectral, something else? How are you deciding which MPI process should compute which matrix entries? Once we understand this we may be able to suggest a better way to compute the entries. Barry Under normally circumstances 1.3 million unknowns is not a large parallel matrix, there may be special features of your matrix that is making this difficult. > On Feb 26, 2017, at 9:30 PM, Fangbo Wang wrote: > > Hi, > > I construct a big matrix which is 1.3million by 1.3million which is using approximatly 100GB memory. I have a computer with 500GB memory. > > I run the Petsc program and it get stuck when finally assembling the matrix. The program is using around 200GB memory only. However, the program just get stuck there. Here is the output message when it gets stuck. > . > . > previous outputs not shown here > . > [12] MatStashScatterBegin_Ref(): No of messages: 15 > [12] MatStashScatterBegin_Ref(): Mesg_to: 0: size: 271636416 bytes > [12] MatStashScatterBegin_Ref(): Mesg_to: 1: size: 328581552 bytes > [12] MatStashScatterBegin_Ref(): Mesg_to: 2: size: 163649328 bytes > [12] MatStashScatterBegin_Ref(): Mesg_to: 3: size: 95512224 bytes > [12] MatStashScatterBegin_Ref(): Mesg_to: 4: size: 317711616 bytes > [12] MatStashScatterBegin_Ref(): Mesg_to: 5: size: 170971776 bytes > [12] MatStashScatterBegin_Ref(): Mesg_to: 6: size: 254000064 bytes > [12] MatStashScatterBegin_Ref(): Mesg_to: 7: size: 163146720 bytes > [12] MatStashScatterBegin_Ref(): Mesg_to: 8: size: 345150048 bytes > [12] MatStashScatterBegin_Ref(): Mesg_to: 9: size: 163411584 bytes > [12] MatStashScatterBegin_Ref(): Mesg_to: 10: size: 428874816 bytes > [12] MatStashScatterBegin_Ref(): Mesg_to: 11: size: 739711296 bytes > [12] MatStashScatterBegin_Ref(): Mesg_to: 13: size: 435247344 bytes > [12] MatStashScatterBegin_Ref(): Mesg_to: 14: size: 435136752 bytes > [12] MatStashScatterBegin_Ref(): Mesg_to: 15: size: 346167552 bytes > [14] MatAssemblyBegin_MPIAIJ(): Stash has 263158893 entries, uses 14 mallocs. > [8] MatAssemblyBegin_MPIAIJ(): Stash has 286768572 entries, uses 14 mallocs. > [12] MatAssemblyBegin_MPIAIJ(): Stash has 291181818 entries, uses 14 mallocs. > [13] MatStashScatterBegin_Ref(): No of messages: 15 > [13] MatStashScatterBegin_Ref(): Mesg_to: 0: size: 271636416 bytes > [13] MatStashScatterBegin_Ref(): Mesg_to: 1: size: 271636416 bytes > [13] MatStashScatterBegin_Ref(): Mesg_to: 2: size: 220594464 bytes > [13] MatStashScatterBegin_Ref(): Mesg_to: 3: size: 51041952 bytes > [13] MatStashScatterBegin_Ref(): Mesg_to: 4: size: 276201408 bytes > [13] MatStashScatterBegin_Ref(): Mesg_to: 5: size: 256952256 bytes > [13] MatStashScatterBegin_Ref(): Mesg_to: 6: size: 198489024 bytes > [13] MatStashScatterBegin_Ref(): Mesg_to: 7: size: 218657760 bytes > [13] MatStashScatterBegin_Ref(): Mesg_to: 8: size: 219686880 bytes > [13] MatStashScatterBegin_Ref(): Mesg_to: 9: size: 288874752 bytes > [13] MatStashScatterBegin_Ref(): Mesg_to: 10: size: 428874816 bytes > [13] MatStashScatterBegin_Ref(): Mesg_to: 11: size: 172579968 bytes > [13] MatStashScatterBegin_Ref(): Mesg_to: 12: size: 639835680 bytes > [13] MatStashScatterBegin_Ref(): Mesg_to: 14: size: 270060144 bytes > [13] MatStashScatterBegin_Ref(): Mesg_to: 15: size: 511244160 bytes > [13] MatAssemblyBegin_MPIAIJ(): Stash has 268522881 entries, uses 14 mallocs. > [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 96812 X 96812; storage space: 89786788 unneeded,7025212 used > [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 81 > [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 96812) < 0.6. Do not use CompressedRow routines. > [5] MatSeqAIJCheckInode(): Found 32271 nodes of 96812. Limit used: 5. Using Inode routines > [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 96812 X 96812; storage space: 89841924 unneeded,6970076 used > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 81 > [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 96812) < 0.6. Do not use CompressedRow routines. > [4] MatSeqAIJCheckInode(): Found 32272 nodes of 96812. Limit used: 5. Using Inode routines > > stuck here!!!! > > > Any one have ideas on this? Thank you very much! > > > > Fangbo Wang > > > > -- > Fangbo Wang, PhD student > Stochastic Geomechanics Research Group > Department of Civil, Structural and Environmental Engineering > University at Buffalo > Email: fangbowa at buffalo.edu From fangbowa at buffalo.edu Sun Feb 26 22:04:46 2017 From: fangbowa at buffalo.edu (Fangbo Wang) Date: Sun, 26 Feb 2017 23:04:46 -0500 Subject: [petsc-users] Why my petsc program get stuck and hang there when assembling the matrix? In-Reply-To: <258CF5B2-CCFD-4AB9-9535-CAE1661018D0@mcs.anl.gov> References: <258CF5B2-CCFD-4AB9-9535-CAE1661018D0@mcs.anl.gov> Message-ID: My problem is a solid mechanics problem using finite element method to discretize the model ( a 30mX30mX30m soil domain with a building structure on top). I am not manually deciding which MPI process compute which matrix enties. Because I know Petsc can automaticaly communicate between these processors. I am just asking each MPI process generate certain number of matrix entries regardless of which process will finally store them. Actually, I constructed another matrix with same size but generating much less entries, and the code worked. However, it gets stuck when I generate more matrix entries. thank you very much! Any suggestion is highly appreciated. BTW, what is the meaning of "[4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 96812) < 0.6. Do not use CompressedRow routines."? I know compressed row format is commonly used for sparse matrix, why don't use compressed row routines here? Thanks, Fangbo Wang On Sun, Feb 26, 2017 at 10:42 PM, Barry Smith wrote: > > How are you generating the matrix entries in parallel? In general you > can generate any matrix entries on any MPI process and they will be > automatically transferred to the MPI process that owns the entries > automatically. BUT if a huge number of matrix entries are computed on one > process and need to be communicated to another process this may cause > gridlock with MPI. Based on the huge size of messages from process 12 it > looks like this is what is happening in your code. > > Ideally most matrix entries are generated on the process they are stored > and hence this gridlock does not happen. > > What type of discretization are you using? Finite differences, finite > element, finite volume, spectral, something else? How are you deciding > which MPI process should compute which matrix entries? Once we understand > this we may be able to suggest a better way to compute the entries. > > Barry > > Under normally circumstances 1.3 million unknowns is not a large parallel > matrix, there may be special features of your matrix that is making this > difficult. > > > > > On Feb 26, 2017, at 9:30 PM, Fangbo Wang wrote: > > > > Hi, > > > > I construct a big matrix which is 1.3million by 1.3million which is > using approximatly 100GB memory. I have a computer with 500GB memory. > > > > I run the Petsc program and it get stuck when finally assembling the > matrix. The program is using around 200GB memory only. However, the program > just get stuck there. Here is the output message when it gets stuck. > > . > > . > > previous outputs not shown here > > . > > [12] MatStashScatterBegin_Ref(): No of messages: 15 > > [12] MatStashScatterBegin_Ref(): Mesg_to: 0: size: 271636416 bytes > > [12] MatStashScatterBegin_Ref(): Mesg_to: 1: size: 328581552 bytes > > [12] MatStashScatterBegin_Ref(): Mesg_to: 2: size: 163649328 bytes > > [12] MatStashScatterBegin_Ref(): Mesg_to: 3: size: 95512224 bytes > > [12] MatStashScatterBegin_Ref(): Mesg_to: 4: size: 317711616 bytes > > [12] MatStashScatterBegin_Ref(): Mesg_to: 5: size: 170971776 bytes > > [12] MatStashScatterBegin_Ref(): Mesg_to: 6: size: 254000064 bytes > > [12] MatStashScatterBegin_Ref(): Mesg_to: 7: size: 163146720 bytes > > [12] MatStashScatterBegin_Ref(): Mesg_to: 8: size: 345150048 bytes > > [12] MatStashScatterBegin_Ref(): Mesg_to: 9: size: 163411584 bytes > > [12] MatStashScatterBegin_Ref(): Mesg_to: 10: size: 428874816 bytes > > [12] MatStashScatterBegin_Ref(): Mesg_to: 11: size: 739711296 bytes > > [12] MatStashScatterBegin_Ref(): Mesg_to: 13: size: 435247344 bytes > > [12] MatStashScatterBegin_Ref(): Mesg_to: 14: size: 435136752 bytes > > [12] MatStashScatterBegin_Ref(): Mesg_to: 15: size: 346167552 bytes > > [14] MatAssemblyBegin_MPIAIJ(): Stash has 263158893 entries, uses 14 > mallocs. > > [8] MatAssemblyBegin_MPIAIJ(): Stash has 286768572 entries, uses 14 > mallocs. > > [12] MatAssemblyBegin_MPIAIJ(): Stash has 291181818 entries, uses 14 > mallocs. > > [13] MatStashScatterBegin_Ref(): No of messages: 15 > > [13] MatStashScatterBegin_Ref(): Mesg_to: 0: size: 271636416 bytes > > [13] MatStashScatterBegin_Ref(): Mesg_to: 1: size: 271636416 bytes > > [13] MatStashScatterBegin_Ref(): Mesg_to: 2: size: 220594464 bytes > > [13] MatStashScatterBegin_Ref(): Mesg_to: 3: size: 51041952 bytes > > [13] MatStashScatterBegin_Ref(): Mesg_to: 4: size: 276201408 bytes > > [13] MatStashScatterBegin_Ref(): Mesg_to: 5: size: 256952256 bytes > > [13] MatStashScatterBegin_Ref(): Mesg_to: 6: size: 198489024 bytes > > [13] MatStashScatterBegin_Ref(): Mesg_to: 7: size: 218657760 bytes > > [13] MatStashScatterBegin_Ref(): Mesg_to: 8: size: 219686880 bytes > > [13] MatStashScatterBegin_Ref(): Mesg_to: 9: size: 288874752 bytes > > [13] MatStashScatterBegin_Ref(): Mesg_to: 10: size: 428874816 bytes > > [13] MatStashScatterBegin_Ref(): Mesg_to: 11: size: 172579968 bytes > > [13] MatStashScatterBegin_Ref(): Mesg_to: 12: size: 639835680 bytes > > [13] MatStashScatterBegin_Ref(): Mesg_to: 14: size: 270060144 bytes > > [13] MatStashScatterBegin_Ref(): Mesg_to: 15: size: 511244160 bytes > > [13] MatAssemblyBegin_MPIAIJ(): Stash has 268522881 entries, uses 14 > mallocs. > > [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 96812 X 96812; storage space: > 89786788 unneeded,7025212 used > > [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > > [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 81 > > [5] MatCheckCompressedRow(): Found the ratio (num_zerorows > 0)/(num_localrows 96812) < 0.6. Do not use CompressedRow routines. > > [5] MatSeqAIJCheckInode(): Found 32271 nodes of 96812. Limit used: 5. > Using Inode routines > > [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 96812 X 96812; storage space: > 89841924 unneeded,6970076 used > > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 81 > > [4] MatCheckCompressedRow(): Found the ratio (num_zerorows > 0)/(num_localrows 96812) < 0.6. Do not use CompressedRow routines. > > [4] MatSeqAIJCheckInode(): Found 32272 nodes of 96812. Limit used: 5. > Using Inode routines > > > > stuck here!!!! > > > > > > Any one have ideas on this? Thank you very much! > > > > > > > > Fangbo Wang > > > > > > > > -- > > Fangbo Wang, PhD student > > Stochastic Geomechanics Research Group > > Department of Civil, Structural and Environmental Engineering > > University at Buffalo > > Email: fangbowa at buffalo.edu > > -- Fangbo Wang, PhD student Stochastic Geomechanics Research Group Department of Civil, Structural and Environmental Engineering University at Buffalo Email: *fangbowa at buffalo.edu * -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Feb 26 22:15:58 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 26 Feb 2017 22:15:58 -0600 Subject: [petsc-users] Why my petsc program get stuck and hang there when assembling the matrix? In-Reply-To: References: <258CF5B2-CCFD-4AB9-9535-CAE1661018D0@mcs.anl.gov> Message-ID: > On Feb 26, 2017, at 10:04 PM, Fangbo Wang wrote: > > My problem is a solid mechanics problem using finite element method to discretize the model ( a 30mX30mX30m soil domain with a building structure on top). > > I am not manually deciding which MPI process compute which matrix enties. Because I know Petsc can automaticaly communicate between these processors. > I am just asking each MPI process generate certain number of matrix entries regardless of which process will finally store them. The standard way to handle this for finite elements is to partition the elements among the processes and then partition the nodes (rows of the degrees of freedom) subservient to the partitioning of the elements. Otherwise most of the matrix (or vector) entries must be communicated and this is not scalable. So how are you partitioning the elements (for matrix stiffness computations) and the nodes between processes? > > Actually, I constructed another matrix with same size but generating much less entries, and the code worked. However, it gets stuck when I generate more matrix entries. > > thank you very much! Any suggestion is highly appreciated. > > BTW, what is the meaning of "[4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 96812) < 0.6. Do not use CompressedRow routines."? I know compressed row format is commonly used for sparse matrix, why don't use compressed row routines here? This is not important. > > > Thanks, > > > Fangbo Wang > > > > On Sun, Feb 26, 2017 at 10:42 PM, Barry Smith wrote: > > How are you generating the matrix entries in parallel? In general you can generate any matrix entries on any MPI process and they will be automatically transferred to the MPI process that owns the entries automatically. BUT if a huge number of matrix entries are computed on one process and need to be communicated to another process this may cause gridlock with MPI. Based on the huge size of messages from process 12 it looks like this is what is happening in your code. > > Ideally most matrix entries are generated on the process they are stored and hence this gridlock does not happen. > > What type of discretization are you using? Finite differences, finite element, finite volume, spectral, something else? How are you deciding which MPI process should compute which matrix entries? Once we understand this we may be able to suggest a better way to compute the entries. > > Barry > > Under normally circumstances 1.3 million unknowns is not a large parallel matrix, there may be special features of your matrix that is making this difficult. > > > > > On Feb 26, 2017, at 9:30 PM, Fangbo Wang wrote: > > > > Hi, > > > > I construct a big matrix which is 1.3million by 1.3million which is using approximatly 100GB memory. I have a computer with 500GB memory. > > > > I run the Petsc program and it get stuck when finally assembling the matrix. The program is using around 200GB memory only. However, the program just get stuck there. Here is the output message when it gets stuck. > > . > > . > > previous outputs not shown here > > . > > [12] MatStashScatterBegin_Ref(): No of messages: 15 > > [12] MatStashScatterBegin_Ref(): Mesg_to: 0: size: 271636416 bytes > > [12] MatStashScatterBegin_Ref(): Mesg_to: 1: size: 328581552 bytes > > [12] MatStashScatterBegin_Ref(): Mesg_to: 2: size: 163649328 bytes > > [12] MatStashScatterBegin_Ref(): Mesg_to: 3: size: 95512224 bytes > > [12] MatStashScatterBegin_Ref(): Mesg_to: 4: size: 317711616 bytes > > [12] MatStashScatterBegin_Ref(): Mesg_to: 5: size: 170971776 bytes > > [12] MatStashScatterBegin_Ref(): Mesg_to: 6: size: 254000064 bytes > > [12] MatStashScatterBegin_Ref(): Mesg_to: 7: size: 163146720 bytes > > [12] MatStashScatterBegin_Ref(): Mesg_to: 8: size: 345150048 bytes > > [12] MatStashScatterBegin_Ref(): Mesg_to: 9: size: 163411584 bytes > > [12] MatStashScatterBegin_Ref(): Mesg_to: 10: size: 428874816 bytes > > [12] MatStashScatterBegin_Ref(): Mesg_to: 11: size: 739711296 bytes > > [12] MatStashScatterBegin_Ref(): Mesg_to: 13: size: 435247344 bytes > > [12] MatStashScatterBegin_Ref(): Mesg_to: 14: size: 435136752 bytes > > [12] MatStashScatterBegin_Ref(): Mesg_to: 15: size: 346167552 bytes > > [14] MatAssemblyBegin_MPIAIJ(): Stash has 263158893 entries, uses 14 mallocs. > > [8] MatAssemblyBegin_MPIAIJ(): Stash has 286768572 entries, uses 14 mallocs. > > [12] MatAssemblyBegin_MPIAIJ(): Stash has 291181818 entries, uses 14 mallocs. > > [13] MatStashScatterBegin_Ref(): No of messages: 15 > > [13] MatStashScatterBegin_Ref(): Mesg_to: 0: size: 271636416 bytes > > [13] MatStashScatterBegin_Ref(): Mesg_to: 1: size: 271636416 bytes > > [13] MatStashScatterBegin_Ref(): Mesg_to: 2: size: 220594464 bytes > > [13] MatStashScatterBegin_Ref(): Mesg_to: 3: size: 51041952 bytes > > [13] MatStashScatterBegin_Ref(): Mesg_to: 4: size: 276201408 bytes > > [13] MatStashScatterBegin_Ref(): Mesg_to: 5: size: 256952256 bytes > > [13] MatStashScatterBegin_Ref(): Mesg_to: 6: size: 198489024 bytes > > [13] MatStashScatterBegin_Ref(): Mesg_to: 7: size: 218657760 bytes > > [13] MatStashScatterBegin_Ref(): Mesg_to: 8: size: 219686880 bytes > > [13] MatStashScatterBegin_Ref(): Mesg_to: 9: size: 288874752 bytes > > [13] MatStashScatterBegin_Ref(): Mesg_to: 10: size: 428874816 bytes > > [13] MatStashScatterBegin_Ref(): Mesg_to: 11: size: 172579968 bytes > > [13] MatStashScatterBegin_Ref(): Mesg_to: 12: size: 639835680 bytes > > [13] MatStashScatterBegin_Ref(): Mesg_to: 14: size: 270060144 bytes > > [13] MatStashScatterBegin_Ref(): Mesg_to: 15: size: 511244160 bytes > > [13] MatAssemblyBegin_MPIAIJ(): Stash has 268522881 entries, uses 14 mallocs. > > [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 96812 X 96812; storage space: 89786788 unneeded,7025212 used > > [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > > [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 81 > > [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 96812) < 0.6. Do not use CompressedRow routines. > > [5] MatSeqAIJCheckInode(): Found 32271 nodes of 96812. Limit used: 5. Using Inode routines > > [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 96812 X 96812; storage space: 89841924 unneeded,6970076 used > > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 81 > > [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 96812) < 0.6. Do not use CompressedRow routines. > > [4] MatSeqAIJCheckInode(): Found 32272 nodes of 96812. Limit used: 5. Using Inode routines > > > > stuck here!!!! > > > > > > Any one have ideas on this? Thank you very much! > > > > > > > > Fangbo Wang > > > > > > > > -- > > Fangbo Wang, PhD student > > Stochastic Geomechanics Research Group > > Department of Civil, Structural and Environmental Engineering > > University at Buffalo > > Email: fangbowa at buffalo.edu > > > > > -- > Fangbo Wang, PhD student > Stochastic Geomechanics Research Group > Department of Civil, Structural and Environmental Engineering > University at Buffalo > Email: fangbowa at buffalo.edu From s_g at berkeley.edu Sun Feb 26 22:23:52 2017 From: s_g at berkeley.edu (Sanjay Govindjee) Date: Sun, 26 Feb 2017 20:23:52 -0800 Subject: [petsc-users] Why my petsc program get stuck and hang there when assembling the matrix? In-Reply-To: References: <258CF5B2-CCFD-4AB9-9535-CAE1661018D0@mcs.anl.gov> Message-ID: <225f14d0-6688-8688-fe22-982091496e9d@berkeley.edu> I'm not sure it is best to say that "the standard way to handle this" is to partition the elements. Minimization of communication calls for partitioning the nodes (at the expense of performing extra element computations). On 2/26/17 8:15 PM, Barry Smith wrote: > The standard way to handle this for finite elements is to partition the elements among the processes and then partition the nodes (rows of the degrees of freedom) subservient to the partitioning of the elements. Otherwise most of the matrix (or vector) entries must be communicated and this is not scalable. > > So how are you partitioning the elements (for matrix stiffness computations) and the nodes between processes? From jed at jedbrown.org Sun Feb 26 22:30:32 2017 From: jed at jedbrown.org (Jed Brown) Date: Sun, 26 Feb 2017 21:30:32 -0700 Subject: [petsc-users] Why my petsc program get stuck and hang there when assembling the matrix? In-Reply-To: <225f14d0-6688-8688-fe22-982091496e9d@berkeley.edu> References: <258CF5B2-CCFD-4AB9-9535-CAE1661018D0@mcs.anl.gov> <225f14d0-6688-8688-fe22-982091496e9d@berkeley.edu> Message-ID: <877f4cuufr.fsf@jedbrown.org> Sanjay Govindjee writes: > I'm not sure it is best to say that "the standard way to handle this" is > to partition the elements. Minimization of communication calls for > partitioning the nodes (at the expense of performing extra element > computations). For high order elements, that entails far more communication (volume). I think it's fair to say that element partition is by far the most common way to implement, though there is certainly a case for working only with a nodal partition (depending on the intended solver) when using relatively low order elements. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From s_g at berkeley.edu Sun Feb 26 22:37:31 2017 From: s_g at berkeley.edu (Sanjay Govindjee) Date: Sun, 26 Feb 2017 20:37:31 -0800 Subject: [petsc-users] Why my petsc program get stuck and hang there when assembling the matrix? In-Reply-To: <877f4cuufr.fsf@jedbrown.org> References: <258CF5B2-CCFD-4AB9-9535-CAE1661018D0@mcs.anl.gov> <225f14d0-6688-8688-fe22-982091496e9d@berkeley.edu> <877f4cuufr.fsf@jedbrown.org> Message-ID: <06030d52-c541-adb9-77fd-4f7edf9e1978@berkeley.edu> Fair enough. I have only done the estimates/testing for linear and quadratic elements, the work horses of large-scale industrial solid-mechanics computations. On 2/26/17 8:30 PM, Jed Brown wrote: > Sanjay Govindjee writes: > >> I'm not sure it is best to say that "the standard way to handle this" is >> to partition the elements. Minimization of communication calls for >> partitioning the nodes (at the expense of performing extra element >> computations). > For high order elements, that entails far more communication (volume). > > I think it's fair to say that element partition is by far the most > common way to implement, though there is certainly a case for working > only with a nodal partition (depending on the intended solver) when > using relatively low order elements. -- ------------------------------------------------------------------- Sanjay Govindjee, PhD, PE Horace, Dorothy, and Katherine Johnson Professor in Engineering 779 Davis Hall University of California Berkeley, CA 94720-1710 Voice: +1 510 642 6060 FAX: +1 510 643 5264 s_g at berkeley.edu http://www.ce.berkeley.edu/~sanjay ------------------------------------------------------------------- Books: Engineering Mechanics of Deformable Solids: A Presentation with Exercises http://www.oup.com/us/catalog/general/subject/Physics/MaterialsScience/?view=usa&ci=9780199651641 http://ukcatalogue.oup.com/product/9780199651641.do http://amzn.com/0199651647 Engineering Mechanics 3 (Dynamics) 2nd Edition http://www.springer.com/978-3-642-53711-0 http://amzn.com/3642537111 Engineering Mechanics 3, Supplementary Problems: Dynamics http://www.amzn.com/B00SOXN8JU ----------------------------------------------- From fangbowa at buffalo.edu Sun Feb 26 22:37:48 2017 From: fangbowa at buffalo.edu (Fangbo Wang) Date: Sun, 26 Feb 2017 23:37:48 -0500 Subject: [petsc-users] Why my petsc program get stuck and hang there when assembling the matrix? In-Reply-To: References: <258CF5B2-CCFD-4AB9-9535-CAE1661018D0@mcs.anl.gov> Message-ID: I got my finite element mesh from a commercial finite element software ABAQUS. I simply draw the geometry of the model in the graphical interface and assign element types and material properties to different parts of the model, ABAQUS will automatically output the element and node information of the model. Suppose I have 1000 elements in my model and 10 MPI processes, #1 to #100 local element matrices will be computed in MPI process 0; #101 to #200 local element matrices will be computed in MPI process 1; #201 to #300 local element matrices will be computed in MPI process 2; .......... #901 to #1000 local element matrices will be computed in MPI process 9; However, I might get a lot of global matrix indices which I need to send to other processors due to the degree of freedom ordering in the finite element model. This is what I did according to my understanding of finite element and what I have seen. Do you have some nice libraries or packages that can be easily used in scientific computing environment? Thank you very much! Fangbo Wang On Sun, Feb 26, 2017 at 11:15 PM, Barry Smith wrote: > > > On Feb 26, 2017, at 10:04 PM, Fangbo Wang wrote: > > > > My problem is a solid mechanics problem using finite element method to > discretize the model ( a 30mX30mX30m soil domain with a building structure > on top). > > > > I am not manually deciding which MPI process compute which matrix > enties. Because I know Petsc can automaticaly communicate between these > processors. > > I am just asking each MPI process generate certain number of matrix > entries regardless of which process will finally store them. > > The standard way to handle this for finite elements is to partition the > elements among the processes and then partition the nodes (rows of the > degrees of freedom) subservient to the partitioning of the elements. > Otherwise most of the matrix (or vector) entries must be communicated and > this is not scalable. > > So how are you partitioning the elements (for matrix stiffness > computations) and the nodes between processes? > > > > Actually, I constructed another matrix with same size but generating > much less entries, and the code worked. However, it gets stuck when I > generate more matrix entries. > > > > thank you very much! Any suggestion is highly appreciated. > > > > BTW, what is the meaning of "[4] MatCheckCompressedRow(): Found the > ratio (num_zerorows 0)/(num_localrows 96812) < 0.6. Do not use > CompressedRow routines."? I know compressed row format is commonly used for > sparse matrix, why don't use compressed row routines here? > > This is not important. > > > > > > > Thanks, > > > > > > Fangbo Wang > > > > > > > > On Sun, Feb 26, 2017 at 10:42 PM, Barry Smith > wrote: > > > > How are you generating the matrix entries in parallel? In general you > can generate any matrix entries on any MPI process and they will be > automatically transferred to the MPI process that owns the entries > automatically. BUT if a huge number of matrix entries are computed on one > process and need to be communicated to another process this may cause > gridlock with MPI. Based on the huge size of messages from process 12 it > looks like this is what is happening in your code. > > > > Ideally most matrix entries are generated on the process they are > stored and hence this gridlock does not happen. > > > > What type of discretization are you using? Finite differences, finite > element, finite volume, spectral, something else? How are you deciding > which MPI process should compute which matrix entries? Once we understand > this we may be able to suggest a better way to compute the entries. > > > > Barry > > > > Under normally circumstances 1.3 million unknowns is not a large > parallel matrix, there may be special features of your matrix that is > making this difficult. > > > > > > > > > On Feb 26, 2017, at 9:30 PM, Fangbo Wang wrote: > > > > > > Hi, > > > > > > I construct a big matrix which is 1.3million by 1.3million which is > using approximatly 100GB memory. I have a computer with 500GB memory. > > > > > > I run the Petsc program and it get stuck when finally assembling the > matrix. The program is using around 200GB memory only. However, the program > just get stuck there. Here is the output message when it gets stuck. > > > . > > > . > > > previous outputs not shown here > > > . > > > [12] MatStashScatterBegin_Ref(): No of messages: 15 > > > [12] MatStashScatterBegin_Ref(): Mesg_to: 0: size: 271636416 bytes > > > [12] MatStashScatterBegin_Ref(): Mesg_to: 1: size: 328581552 bytes > > > [12] MatStashScatterBegin_Ref(): Mesg_to: 2: size: 163649328 bytes > > > [12] MatStashScatterBegin_Ref(): Mesg_to: 3: size: 95512224 bytes > > > [12] MatStashScatterBegin_Ref(): Mesg_to: 4: size: 317711616 bytes > > > [12] MatStashScatterBegin_Ref(): Mesg_to: 5: size: 170971776 bytes > > > [12] MatStashScatterBegin_Ref(): Mesg_to: 6: size: 254000064 bytes > > > [12] MatStashScatterBegin_Ref(): Mesg_to: 7: size: 163146720 bytes > > > [12] MatStashScatterBegin_Ref(): Mesg_to: 8: size: 345150048 bytes > > > [12] MatStashScatterBegin_Ref(): Mesg_to: 9: size: 163411584 bytes > > > [12] MatStashScatterBegin_Ref(): Mesg_to: 10: size: 428874816 bytes > > > [12] MatStashScatterBegin_Ref(): Mesg_to: 11: size: 739711296 bytes > > > [12] MatStashScatterBegin_Ref(): Mesg_to: 13: size: 435247344 bytes > > > [12] MatStashScatterBegin_Ref(): Mesg_to: 14: size: 435136752 bytes > > > [12] MatStashScatterBegin_Ref(): Mesg_to: 15: size: 346167552 bytes > > > [14] MatAssemblyBegin_MPIAIJ(): Stash has 263158893 entries, uses 14 > mallocs. > > > [8] MatAssemblyBegin_MPIAIJ(): Stash has 286768572 entries, uses 14 > mallocs. > > > [12] MatAssemblyBegin_MPIAIJ(): Stash has 291181818 entries, uses 14 > mallocs. > > > [13] MatStashScatterBegin_Ref(): No of messages: 15 > > > [13] MatStashScatterBegin_Ref(): Mesg_to: 0: size: 271636416 bytes > > > [13] MatStashScatterBegin_Ref(): Mesg_to: 1: size: 271636416 bytes > > > [13] MatStashScatterBegin_Ref(): Mesg_to: 2: size: 220594464 bytes > > > [13] MatStashScatterBegin_Ref(): Mesg_to: 3: size: 51041952 bytes > > > [13] MatStashScatterBegin_Ref(): Mesg_to: 4: size: 276201408 bytes > > > [13] MatStashScatterBegin_Ref(): Mesg_to: 5: size: 256952256 bytes > > > [13] MatStashScatterBegin_Ref(): Mesg_to: 6: size: 198489024 bytes > > > [13] MatStashScatterBegin_Ref(): Mesg_to: 7: size: 218657760 bytes > > > [13] MatStashScatterBegin_Ref(): Mesg_to: 8: size: 219686880 bytes > > > [13] MatStashScatterBegin_Ref(): Mesg_to: 9: size: 288874752 bytes > > > [13] MatStashScatterBegin_Ref(): Mesg_to: 10: size: 428874816 bytes > > > [13] MatStashScatterBegin_Ref(): Mesg_to: 11: size: 172579968 bytes > > > [13] MatStashScatterBegin_Ref(): Mesg_to: 12: size: 639835680 bytes > > > [13] MatStashScatterBegin_Ref(): Mesg_to: 14: size: 270060144 bytes > > > [13] MatStashScatterBegin_Ref(): Mesg_to: 15: size: 511244160 bytes > > > [13] MatAssemblyBegin_MPIAIJ(): Stash has 268522881 entries, uses 14 > mallocs. > > > [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 96812 X 96812; storage > space: 89786788 unneeded,7025212 used > > > [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() > is 0 > > > [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 81 > > > [5] MatCheckCompressedRow(): Found the ratio (num_zerorows > 0)/(num_localrows 96812) < 0.6. Do not use CompressedRow routines. > > > [5] MatSeqAIJCheckInode(): Found 32271 nodes of 96812. Limit used: 5. > Using Inode routines > > > [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 96812 X 96812; storage > space: 89841924 unneeded,6970076 used > > > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() > is 0 > > > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 81 > > > [4] MatCheckCompressedRow(): Found the ratio (num_zerorows > 0)/(num_localrows 96812) < 0.6. Do not use CompressedRow routines. > > > [4] MatSeqAIJCheckInode(): Found 32272 nodes of 96812. Limit used: 5. > Using Inode routines > > > > > > stuck here!!!! > > > > > > > > > Any one have ideas on this? Thank you very much! > > > > > > > > > > > > Fangbo Wang > > > > > > > > > > > > -- > > > Fangbo Wang, PhD student > > > Stochastic Geomechanics Research Group > > > Department of Civil, Structural and Environmental Engineering > > > University at Buffalo > > > Email: fangbowa at buffalo.edu > > > > > > > > > > -- > > Fangbo Wang, PhD student > > Stochastic Geomechanics Research Group > > Department of Civil, Structural and Environmental Engineering > > University at Buffalo > > Email: fangbowa at buffalo.edu > > -- Fangbo Wang, PhD student Stochastic Geomechanics Research Group Department of Civil, Structural and Environmental Engineering University at Buffalo Email: *fangbowa at buffalo.edu * -------------- next part -------------- An HTML attachment was scrubbed... URL: From fsantost at student.ethz.ch Mon Feb 27 00:29:43 2017 From: fsantost at student.ethz.ch (Santos Teixeira Frederico) Date: Mon, 27 Feb 2017 06:29:43 +0000 Subject: [petsc-users] Multi-domain meshes with DMPLEX In-Reply-To: References: <682CC3CD7A208742B8C2D116C67199013F3C3A65@MBX23.d.ethz.ch>, Message-ID: <682CC3CD7A208742B8C2D116C67199013F3C3C00@MBX23.d.ethz.ch> Dear Matt, Thanks for your reply and help. I replaced the code with DMViewFromOptions() as you suggested. Attached you find the code for your reference, and the output of -dm_view ::ascii_info_detail. a) -dm_view of the original mesh DM Object: 1 MPI processes type: plex DM_0x84000000_0 in 3 dimensions: 0-cells: 16 1-cells: 61 2-cells: 80 3-cells: 34 Labels: Cell Sets: 2 strata of sizes (17, 17) Face Sets: 3 strata of sizes (12, 8, 10) depth: 4 strata of sizes (16, 61, 80, 34) b) -dm_view of the sub-mesh 45 obtained by DMPlexFilter sub-mesh 45 DM Object: 1 MPI processes type: plex DM_0x84000000_1 in 3 dimensions: 0-cells: 12 1-cells: 38 2-cells: 44 3-cells: 17 Labels: depth: 4 strata of sizes (12, 38, 44, 17) c) -dm_view of the sub-mesh 43 obtained by DMPlexFilter: sub-mesh 43 DM Object: 1 MPI processes type: plex DM_0x84000000_2 in 3 dimensions: 0-cells: 12 1-cells: 38 2-cells: 44 3-cells: 17 Labels: depth: 4 strata of sizes (12, 38, 44, 17) d) -dm_view of the "sub-mesh 40" (interface between sub-meshes 43 and 45) obtained by DMPlexCreateSubMesh: interface 40 DM Object: 1 MPI processes type: plex DM_0x84000000_3 in 2 dimensions: 0-cells: 0 1-cells: 0 2-cells: 0 (-83895904) Labels: depth: 0 strata of sizes () Questions: 1) the output of (d) is empty. What am I missing? 2) I used the "Cell Sets" DMLabel along with value=45 or 43 to yield the output of (b) and (c), respectively, and they do not contain "Face Sets". Is it expected? I will address the other question in a separate e-mail, because it is not related to the issues I described above. My apologies the confusion. Regards, Frederico. ________________________________ From: Matthew Knepley [knepley at gmail.com] Sent: 26 February 2017 23:26 To: Santos Teixeira Frederico Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Multi-domain meshes with DMPLEX On Sun, Feb 26, 2017 at 10:51 AM, Santos Teixeira Frederico > wrote: Hi folks, Attached you find small.msh, a mesh with two prisms (physical domains 43 and 45) and the interface between them (physical domain 40). test_dmplex.cpp loads the mesh and tries to filter each sub-mesh and the interface. I use PETSc master. When I load small.msh, all physical domains are correctly assigned to "Cell Sets" or "Face Sets". 1) Both sub-meshes (prisms with domains 43 and 45) are filtered with DMPlexFilter and saved correctly, but the interface isn't. What am I missing? What exactly does "the interface isn't" mean? Does it mean you are not getting what you expect from the submesh? One way to more easily communicate about this is to replace your blocks of Viewer commands with one line DMViewFromOptions() calls. That way you can show me just -dm_view or -dm_view ::ascii_info_detail, or output VTK with -dm_view vtk:mesh.vtu, or HDF5 -dm_view hdf5:mesh.h5, etc. We could start off with sending me just the plain ASCII output for the mesh and submeshes. I do recognize you sent the code, but I have a conference deadline for Monday (I will be there until Friday), so I am unlikely to build it before them. 2) I can retrieve any specific strata of "Cell Sets" with DMPlexFilter, but this operation does not include any related strata of "Face Sets", as pointed by DMView. How can I do it? Do DMFilter is not propagating labels? Okay, that can go on the bug list. Is this what you mean? 3) Physical domains 43 and 45 could be interpreted as fluid and solid domains in a F.S.I. simulation, for example, and I would like to define fluid and solid sub-problems only on the respective sub-mesh. Do I really need to have fully operational sub-DMPLEX's in order to achieve it? Is there an easier way? I am not sure I understand what is going wrong yet. Thanks, Matt My apologies if the explanations are confusing! Best regards, Frederico. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test_dmplex.cpp Type: application/octet-stream Size: 2868 bytes Desc: test_dmplex.cpp URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: dm_view_info_detail.txt URL: From mfadams at lbl.gov Mon Feb 27 08:37:50 2017 From: mfadams at lbl.gov (Mark Adams) Date: Mon, 27 Feb 2017 09:37:50 -0500 Subject: [petsc-users] Why my petsc program get stuck and hang there when assembling the matrix? In-Reply-To: References: <258CF5B2-CCFD-4AB9-9535-CAE1661018D0@mcs.anl.gov> Message-ID: Another approach that might be simple, if you have the metadata for the entire mesh locally, is set up a list of elements that your local matrix block-rows/vertices touch but going over all the elements and test if any of its vertices i are: if (i >= start && i < end) list.append(i). Just compute and assemble those elements and tell PETSc to ignore-off-processor-entries. No communication, redundant local work, some setup code and cost. On Sun, Feb 26, 2017 at 11:37 PM, Fangbo Wang wrote: > I got my finite element mesh from a commercial finite element software > ABAQUS. I simply draw the geometry of the model in the graphical interface > and assign element types and material properties to different parts of the > model, ABAQUS will automatically output the element and node information of > the model. > > Suppose I have 1000 elements in my model and 10 MPI processes, > #1 to #100 local element matrices will be computed in MPI process 0; > #101 to #200 local element matrices will be computed in MPI process 1; > #201 to #300 local element matrices will be computed in MPI process 2; > .......... > #901 to #1000 local element matrices will be computed in MPI process 9; > > > However, I might get a lot of global matrix indices which I need to send > to other processors due to the degree of freedom ordering in the finite > element model. > > This is what I did according to my understanding of finite element and > what I have seen. > Do you have some nice libraries or packages that can be easily used in > scientific computing environment? > > Thank you very much! > > > > Fangbo Wang > > > > > On Sun, Feb 26, 2017 at 11:15 PM, Barry Smith wrote: > >> >> > On Feb 26, 2017, at 10:04 PM, Fangbo Wang wrote: >> > >> > My problem is a solid mechanics problem using finite element method to >> discretize the model ( a 30mX30mX30m soil domain with a building structure >> on top). >> > >> > I am not manually deciding which MPI process compute which matrix >> enties. Because I know Petsc can automaticaly communicate between these >> processors. >> > I am just asking each MPI process generate certain number of matrix >> entries regardless of which process will finally store them. >> >> The standard way to handle this for finite elements is to partition the >> elements among the processes and then partition the nodes (rows of the >> degrees of freedom) subservient to the partitioning of the elements. >> Otherwise most of the matrix (or vector) entries must be communicated and >> this is not scalable. >> >> So how are you partitioning the elements (for matrix stiffness >> computations) and the nodes between processes? >> > >> > Actually, I constructed another matrix with same size but generating >> much less entries, and the code worked. However, it gets stuck when I >> generate more matrix entries. >> > >> > thank you very much! Any suggestion is highly appreciated. >> > >> > BTW, what is the meaning of "[4] MatCheckCompressedRow(): Found the >> ratio (num_zerorows 0)/(num_localrows 96812) < 0.6. Do not use >> CompressedRow routines."? I know compressed row format is commonly used for >> sparse matrix, why don't use compressed row routines here? >> >> This is not important. >> >> > >> > >> > Thanks, >> > >> > >> > Fangbo Wang >> > >> > >> > >> > On Sun, Feb 26, 2017 at 10:42 PM, Barry Smith >> wrote: >> > >> > How are you generating the matrix entries in parallel? In general you >> can generate any matrix entries on any MPI process and they will be >> automatically transferred to the MPI process that owns the entries >> automatically. BUT if a huge number of matrix entries are computed on one >> process and need to be communicated to another process this may cause >> gridlock with MPI. Based on the huge size of messages from process 12 it >> looks like this is what is happening in your code. >> > >> > Ideally most matrix entries are generated on the process they are >> stored and hence this gridlock does not happen. >> > >> > What type of discretization are you using? Finite differences, finite >> element, finite volume, spectral, something else? How are you deciding >> which MPI process should compute which matrix entries? Once we understand >> this we may be able to suggest a better way to compute the entries. >> > >> > Barry >> > >> > Under normally circumstances 1.3 million unknowns is not a large >> parallel matrix, there may be special features of your matrix that is >> making this difficult. >> > >> > >> > >> > > On Feb 26, 2017, at 9:30 PM, Fangbo Wang >> wrote: >> > > >> > > Hi, >> > > >> > > I construct a big matrix which is 1.3million by 1.3million which is >> using approximatly 100GB memory. I have a computer with 500GB memory. >> > > >> > > I run the Petsc program and it get stuck when finally assembling the >> matrix. The program is using around 200GB memory only. However, the program >> just get stuck there. Here is the output message when it gets stuck. >> > > . >> > > . >> > > previous outputs not shown here >> > > . >> > > [12] MatStashScatterBegin_Ref(): No of messages: 15 >> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 0: size: 271636416 bytes >> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 1: size: 328581552 bytes >> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 2: size: 163649328 bytes >> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 3: size: 95512224 bytes >> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 4: size: 317711616 bytes >> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 5: size: 170971776 bytes >> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 6: size: 254000064 bytes >> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 7: size: 163146720 bytes >> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 8: size: 345150048 bytes >> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 9: size: 163411584 bytes >> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 10: size: 428874816 bytes >> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 11: size: 739711296 bytes >> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 13: size: 435247344 bytes >> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 14: size: 435136752 bytes >> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 15: size: 346167552 bytes >> > > [14] MatAssemblyBegin_MPIAIJ(): Stash has 263158893 entries, uses 14 >> mallocs. >> > > [8] MatAssemblyBegin_MPIAIJ(): Stash has 286768572 entries, uses 14 >> mallocs. >> > > [12] MatAssemblyBegin_MPIAIJ(): Stash has 291181818 entries, uses 14 >> mallocs. >> > > [13] MatStashScatterBegin_Ref(): No of messages: 15 >> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 0: size: 271636416 bytes >> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 1: size: 271636416 bytes >> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 2: size: 220594464 bytes >> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 3: size: 51041952 bytes >> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 4: size: 276201408 bytes >> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 5: size: 256952256 bytes >> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 6: size: 198489024 bytes >> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 7: size: 218657760 bytes >> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 8: size: 219686880 bytes >> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 9: size: 288874752 bytes >> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 10: size: 428874816 bytes >> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 11: size: 172579968 bytes >> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 12: size: 639835680 bytes >> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 14: size: 270060144 bytes >> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 15: size: 511244160 bytes >> > > [13] MatAssemblyBegin_MPIAIJ(): Stash has 268522881 entries, uses 14 >> mallocs. >> > > [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 96812 X 96812; storage >> space: 89786788 unneeded,7025212 used >> > > [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >> is 0 >> > > [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 81 >> > > [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >> 0)/(num_localrows 96812) < 0.6. Do not use CompressedRow routines. >> > > [5] MatSeqAIJCheckInode(): Found 32271 nodes of 96812. Limit used: 5. >> Using Inode routines >> > > [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 96812 X 96812; storage >> space: 89841924 unneeded,6970076 used >> > > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >> is 0 >> > > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 81 >> > > [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >> 0)/(num_localrows 96812) < 0.6. Do not use CompressedRow routines. >> > > [4] MatSeqAIJCheckInode(): Found 32272 nodes of 96812. Limit used: 5. >> Using Inode routines >> > > >> > > stuck here!!!! >> > > >> > > >> > > Any one have ideas on this? Thank you very much! >> > > >> > > >> > > >> > > Fangbo Wang >> > > >> > > >> > > >> > > -- >> > > Fangbo Wang, PhD student >> > > Stochastic Geomechanics Research Group >> > > Department of Civil, Structural and Environmental Engineering >> > > University at Buffalo >> > > Email: fangbowa at buffalo.edu >> > >> > >> > >> > >> > -- >> > Fangbo Wang, PhD student >> > Stochastic Geomechanics Research Group >> > Department of Civil, Structural and Environmental Engineering >> > University at Buffalo >> > Email: fangbowa at buffalo.edu >> >> > > > -- > Fangbo Wang, PhD student > Stochastic Geomechanics Research Group > Department of Civil, Structural and Environmental Engineering > University at Buffalo > Email: *fangbowa at buffalo.edu * > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas.drinkt.thee at gmail.com Mon Feb 27 09:06:09 2017 From: lukas.drinkt.thee at gmail.com (Lukas van de Wiel) Date: Mon, 27 Feb 2017 16:06:09 +0100 Subject: [petsc-users] Why my petsc program get stuck and hang there when assembling the matrix? In-Reply-To: References: <258CF5B2-CCFD-4AB9-9535-CAE1661018D0@mcs.anl.gov> Message-ID: Spreading the elements over the processors by sheer number is not automatically a safe method, depending on the mesh. Especially with irregular meshes, such as created by Triangle of Gmsh, such a distribution will not reduce the amount of communication, maybe even increase it. There are mature and well-tested partitioning tools available that can divide your mesh into regional partitions. We use Metis/ParMetis. I believe PETSc uses PTScotch. This is an extra step, but it will reduce the communication volume considerably. Cheers Lukas On 2/27/17, Mark Adams wrote: > Another approach that might be simple, if you have the metadata for the > entire mesh locally, is set up a list of elements that your local matrix > block-rows/vertices touch but going over all the elements and test if any > of its vertices i are: if (i >= start && i < end) list.append(i). Just > compute and assemble those elements and tell PETSc to > ignore-off-processor-entries. No communication, redundant local work, some > setup code and cost. > > On Sun, Feb 26, 2017 at 11:37 PM, Fangbo Wang wrote: > >> I got my finite element mesh from a commercial finite element software >> ABAQUS. I simply draw the geometry of the model in the graphical >> interface >> and assign element types and material properties to different parts of >> the >> model, ABAQUS will automatically output the element and node information >> of >> the model. >> >> Suppose I have 1000 elements in my model and 10 MPI processes, >> #1 to #100 local element matrices will be computed in MPI process 0; >> #101 to #200 local element matrices will be computed in MPI process 1; >> #201 to #300 local element matrices will be computed in MPI process 2; >> .......... >> #901 to #1000 local element matrices will be computed in MPI process 9; >> >> >> However, I might get a lot of global matrix indices which I need to send >> to other processors due to the degree of freedom ordering in the finite >> element model. >> >> This is what I did according to my understanding of finite element and >> what I have seen. >> Do you have some nice libraries or packages that can be easily used in >> scientific computing environment? >> >> Thank you very much! >> >> >> >> Fangbo Wang >> >> >> >> >> On Sun, Feb 26, 2017 at 11:15 PM, Barry Smith wrote: >> >>> >>> > On Feb 26, 2017, at 10:04 PM, Fangbo Wang >>> > wrote: >>> > >>> > My problem is a solid mechanics problem using finite element method to >>> discretize the model ( a 30mX30mX30m soil domain with a building >>> structure >>> on top). >>> > >>> > I am not manually deciding which MPI process compute which matrix >>> enties. Because I know Petsc can automaticaly communicate between these >>> processors. >>> > I am just asking each MPI process generate certain number of matrix >>> entries regardless of which process will finally store them. >>> >>> The standard way to handle this for finite elements is to partition >>> the >>> elements among the processes and then partition the nodes (rows of the >>> degrees of freedom) subservient to the partitioning of the elements. >>> Otherwise most of the matrix (or vector) entries must be communicated >>> and >>> this is not scalable. >>> >>> So how are you partitioning the elements (for matrix stiffness >>> computations) and the nodes between processes? >>> > >>> > Actually, I constructed another matrix with same size but generating >>> much less entries, and the code worked. However, it gets stuck when I >>> generate more matrix entries. >>> > >>> > thank you very much! Any suggestion is highly appreciated. >>> > >>> > BTW, what is the meaning of "[4] MatCheckCompressedRow(): Found the >>> ratio (num_zerorows 0)/(num_localrows 96812) < 0.6. Do not use >>> CompressedRow routines."? I know compressed row format is commonly used >>> for >>> sparse matrix, why don't use compressed row routines here? >>> >>> This is not important. >>> >>> > >>> > >>> > Thanks, >>> > >>> > >>> > Fangbo Wang >>> > >>> > >>> > >>> > On Sun, Feb 26, 2017 at 10:42 PM, Barry Smith >>> wrote: >>> > >>> > How are you generating the matrix entries in parallel? In general >>> > you >>> can generate any matrix entries on any MPI process and they will be >>> automatically transferred to the MPI process that owns the entries >>> automatically. BUT if a huge number of matrix entries are computed on >>> one >>> process and need to be communicated to another process this may cause >>> gridlock with MPI. Based on the huge size of messages from process 12 it >>> looks like this is what is happening in your code. >>> > >>> > Ideally most matrix entries are generated on the process they are >>> stored and hence this gridlock does not happen. >>> > >>> > What type of discretization are you using? Finite differences, finite >>> element, finite volume, spectral, something else? How are you deciding >>> which MPI process should compute which matrix entries? Once we >>> understand >>> this we may be able to suggest a better way to compute the entries. >>> > >>> > Barry >>> > >>> > Under normally circumstances 1.3 million unknowns is not a large >>> parallel matrix, there may be special features of your matrix that is >>> making this difficult. >>> > >>> > >>> > >>> > > On Feb 26, 2017, at 9:30 PM, Fangbo Wang >>> wrote: >>> > > >>> > > Hi, >>> > > >>> > > I construct a big matrix which is 1.3million by 1.3million which is >>> using approximatly 100GB memory. I have a computer with 500GB memory. >>> > > >>> > > I run the Petsc program and it get stuck when finally assembling the >>> matrix. The program is using around 200GB memory only. However, the >>> program >>> just get stuck there. Here is the output message when it gets stuck. >>> > > . >>> > > . >>> > > previous outputs not shown here >>> > > . >>> > > [12] MatStashScatterBegin_Ref(): No of messages: 15 >>> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 0: size: 271636416 bytes >>> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 1: size: 328581552 bytes >>> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 2: size: 163649328 bytes >>> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 3: size: 95512224 bytes >>> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 4: size: 317711616 bytes >>> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 5: size: 170971776 bytes >>> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 6: size: 254000064 bytes >>> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 7: size: 163146720 bytes >>> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 8: size: 345150048 bytes >>> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 9: size: 163411584 bytes >>> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 10: size: 428874816 bytes >>> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 11: size: 739711296 bytes >>> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 13: size: 435247344 bytes >>> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 14: size: 435136752 bytes >>> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 15: size: 346167552 bytes >>> > > [14] MatAssemblyBegin_MPIAIJ(): Stash has 263158893 entries, uses 14 >>> mallocs. >>> > > [8] MatAssemblyBegin_MPIAIJ(): Stash has 286768572 entries, uses 14 >>> mallocs. >>> > > [12] MatAssemblyBegin_MPIAIJ(): Stash has 291181818 entries, uses 14 >>> mallocs. >>> > > [13] MatStashScatterBegin_Ref(): No of messages: 15 >>> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 0: size: 271636416 bytes >>> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 1: size: 271636416 bytes >>> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 2: size: 220594464 bytes >>> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 3: size: 51041952 bytes >>> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 4: size: 276201408 bytes >>> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 5: size: 256952256 bytes >>> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 6: size: 198489024 bytes >>> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 7: size: 218657760 bytes >>> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 8: size: 219686880 bytes >>> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 9: size: 288874752 bytes >>> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 10: size: 428874816 bytes >>> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 11: size: 172579968 bytes >>> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 12: size: 639835680 bytes >>> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 14: size: 270060144 bytes >>> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 15: size: 511244160 bytes >>> > > [13] MatAssemblyBegin_MPIAIJ(): Stash has 268522881 entries, uses 14 >>> mallocs. >>> > > [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 96812 X 96812; storage >>> space: 89786788 unneeded,7025212 used >>> > > [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>> is 0 >>> > > [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 81 >>> > > [5] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 0)/(num_localrows 96812) < 0.6. Do not use CompressedRow routines. >>> > > [5] MatSeqAIJCheckInode(): Found 32271 nodes of 96812. Limit used: >>> > > 5. >>> Using Inode routines >>> > > [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 96812 X 96812; storage >>> space: 89841924 unneeded,6970076 used >>> > > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>> is 0 >>> > > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 81 >>> > > [4] MatCheckCompressedRow(): Found the ratio (num_zerorows >>> 0)/(num_localrows 96812) < 0.6. Do not use CompressedRow routines. >>> > > [4] MatSeqAIJCheckInode(): Found 32272 nodes of 96812. Limit used: >>> > > 5. >>> Using Inode routines >>> > > >>> > > stuck here!!!! >>> > > >>> > > >>> > > Any one have ideas on this? Thank you very much! >>> > > >>> > > >>> > > >>> > > Fangbo Wang >>> > > >>> > > >>> > > >>> > > -- >>> > > Fangbo Wang, PhD student >>> > > Stochastic Geomechanics Research Group >>> > > Department of Civil, Structural and Environmental Engineering >>> > > University at Buffalo >>> > > Email: fangbowa at buffalo.edu >>> > >>> > >>> > >>> > >>> > -- >>> > Fangbo Wang, PhD student >>> > Stochastic Geomechanics Research Group >>> > Department of Civil, Structural and Environmental Engineering >>> > University at Buffalo >>> > Email: fangbowa at buffalo.edu >>> >>> >> >> >> -- >> Fangbo Wang, PhD student >> Stochastic Geomechanics Research Group >> Department of Civil, Structural and Environmental Engineering >> University at Buffalo >> Email: *fangbowa at buffalo.edu * >> > From knepley at gmail.com Mon Feb 27 09:10:40 2017 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 27 Feb 2017 09:10:40 -0600 Subject: [petsc-users] Why my petsc program get stuck and hang there when assembling the matrix? In-Reply-To: References: <258CF5B2-CCFD-4AB9-9535-CAE1661018D0@mcs.anl.gov> Message-ID: On Mon, Feb 27, 2017 at 9:06 AM, Lukas van de Wiel < lukas.drinkt.thee at gmail.com> wrote: > Spreading the elements over the processors by sheer number is not > automatically a safe method, depending on the mesh. Especially with > irregular meshes, such as created by Triangle of Gmsh, such a > distribution will not reduce the amount of communication, maybe even > increase it. > > There are mature and well-tested partitioning tools available that can > divide your mesh into regional partitions. We use Metis/ParMetis. I > believe PETSc uses PTScotch. We have interfaces to Chaco, Metis, ParMetis, Party, and PTScotch Matt > This is an extra step, but it will reduce > the communication volume considerably. > > Cheers > Lukas > > On 2/27/17, Mark Adams wrote: > > Another approach that might be simple, if you have the metadata for the > > entire mesh locally, is set up a list of elements that your local matrix > > block-rows/vertices touch but going over all the elements and test if any > > of its vertices i are: if (i >= start && i < end) list.append(i). Just > > compute and assemble those elements and tell PETSc to > > ignore-off-processor-entries. No communication, redundant local work, > some > > setup code and cost. > > > > On Sun, Feb 26, 2017 at 11:37 PM, Fangbo Wang > wrote: > > > >> I got my finite element mesh from a commercial finite element software > >> ABAQUS. I simply draw the geometry of the model in the graphical > >> interface > >> and assign element types and material properties to different parts of > >> the > >> model, ABAQUS will automatically output the element and node information > >> of > >> the model. > >> > >> Suppose I have 1000 elements in my model and 10 MPI processes, > >> #1 to #100 local element matrices will be computed in MPI process 0; > >> #101 to #200 local element matrices will be computed in MPI process 1; > >> #201 to #300 local element matrices will be computed in MPI process 2; > >> .......... > >> #901 to #1000 local element matrices will be computed in MPI process 9; > >> > >> > >> However, I might get a lot of global matrix indices which I need to send > >> to other processors due to the degree of freedom ordering in the finite > >> element model. > >> > >> This is what I did according to my understanding of finite element and > >> what I have seen. > >> Do you have some nice libraries or packages that can be easily used in > >> scientific computing environment? > >> > >> Thank you very much! > >> > >> > >> > >> Fangbo Wang > >> > >> > >> > >> > >> On Sun, Feb 26, 2017 at 11:15 PM, Barry Smith > wrote: > >> > >>> > >>> > On Feb 26, 2017, at 10:04 PM, Fangbo Wang > >>> > wrote: > >>> > > >>> > My problem is a solid mechanics problem using finite element method > to > >>> discretize the model ( a 30mX30mX30m soil domain with a building > >>> structure > >>> on top). > >>> > > >>> > I am not manually deciding which MPI process compute which matrix > >>> enties. Because I know Petsc can automaticaly communicate between these > >>> processors. > >>> > I am just asking each MPI process generate certain number of matrix > >>> entries regardless of which process will finally store them. > >>> > >>> The standard way to handle this for finite elements is to partition > >>> the > >>> elements among the processes and then partition the nodes (rows of the > >>> degrees of freedom) subservient to the partitioning of the elements. > >>> Otherwise most of the matrix (or vector) entries must be communicated > >>> and > >>> this is not scalable. > >>> > >>> So how are you partitioning the elements (for matrix stiffness > >>> computations) and the nodes between processes? > >>> > > >>> > Actually, I constructed another matrix with same size but generating > >>> much less entries, and the code worked. However, it gets stuck when I > >>> generate more matrix entries. > >>> > > >>> > thank you very much! Any suggestion is highly appreciated. > >>> > > >>> > BTW, what is the meaning of "[4] MatCheckCompressedRow(): Found the > >>> ratio (num_zerorows 0)/(num_localrows 96812) < 0.6. Do not use > >>> CompressedRow routines."? I know compressed row format is commonly used > >>> for > >>> sparse matrix, why don't use compressed row routines here? > >>> > >>> This is not important. > >>> > >>> > > >>> > > >>> > Thanks, > >>> > > >>> > > >>> > Fangbo Wang > >>> > > >>> > > >>> > > >>> > On Sun, Feb 26, 2017 at 10:42 PM, Barry Smith > >>> wrote: > >>> > > >>> > How are you generating the matrix entries in parallel? In general > >>> > you > >>> can generate any matrix entries on any MPI process and they will be > >>> automatically transferred to the MPI process that owns the entries > >>> automatically. BUT if a huge number of matrix entries are computed on > >>> one > >>> process and need to be communicated to another process this may cause > >>> gridlock with MPI. Based on the huge size of messages from process 12 > it > >>> looks like this is what is happening in your code. > >>> > > >>> > Ideally most matrix entries are generated on the process they are > >>> stored and hence this gridlock does not happen. > >>> > > >>> > What type of discretization are you using? Finite differences, finite > >>> element, finite volume, spectral, something else? How are you deciding > >>> which MPI process should compute which matrix entries? Once we > >>> understand > >>> this we may be able to suggest a better way to compute the entries. > >>> > > >>> > Barry > >>> > > >>> > Under normally circumstances 1.3 million unknowns is not a large > >>> parallel matrix, there may be special features of your matrix that is > >>> making this difficult. > >>> > > >>> > > >>> > > >>> > > On Feb 26, 2017, at 9:30 PM, Fangbo Wang > >>> wrote: > >>> > > > >>> > > Hi, > >>> > > > >>> > > I construct a big matrix which is 1.3million by 1.3million which is > >>> using approximatly 100GB memory. I have a computer with 500GB memory. > >>> > > > >>> > > I run the Petsc program and it get stuck when finally assembling > the > >>> matrix. The program is using around 200GB memory only. However, the > >>> program > >>> just get stuck there. Here is the output message when it gets stuck. > >>> > > . > >>> > > . > >>> > > previous outputs not shown here > >>> > > . > >>> > > [12] MatStashScatterBegin_Ref(): No of messages: 15 > >>> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 0: size: 271636416 bytes > >>> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 1: size: 328581552 bytes > >>> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 2: size: 163649328 bytes > >>> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 3: size: 95512224 bytes > >>> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 4: size: 317711616 bytes > >>> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 5: size: 170971776 bytes > >>> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 6: size: 254000064 bytes > >>> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 7: size: 163146720 bytes > >>> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 8: size: 345150048 bytes > >>> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 9: size: 163411584 bytes > >>> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 10: size: 428874816 bytes > >>> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 11: size: 739711296 bytes > >>> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 13: size: 435247344 bytes > >>> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 14: size: 435136752 bytes > >>> > > [12] MatStashScatterBegin_Ref(): Mesg_to: 15: size: 346167552 bytes > >>> > > [14] MatAssemblyBegin_MPIAIJ(): Stash has 263158893 entries, uses > 14 > >>> mallocs. > >>> > > [8] MatAssemblyBegin_MPIAIJ(): Stash has 286768572 entries, uses 14 > >>> mallocs. > >>> > > [12] MatAssemblyBegin_MPIAIJ(): Stash has 291181818 entries, uses > 14 > >>> mallocs. > >>> > > [13] MatStashScatterBegin_Ref(): No of messages: 15 > >>> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 0: size: 271636416 bytes > >>> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 1: size: 271636416 bytes > >>> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 2: size: 220594464 bytes > >>> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 3: size: 51041952 bytes > >>> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 4: size: 276201408 bytes > >>> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 5: size: 256952256 bytes > >>> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 6: size: 198489024 bytes > >>> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 7: size: 218657760 bytes > >>> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 8: size: 219686880 bytes > >>> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 9: size: 288874752 bytes > >>> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 10: size: 428874816 bytes > >>> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 11: size: 172579968 bytes > >>> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 12: size: 639835680 bytes > >>> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 14: size: 270060144 bytes > >>> > > [13] MatStashScatterBegin_Ref(): Mesg_to: 15: size: 511244160 bytes > >>> > > [13] MatAssemblyBegin_MPIAIJ(): Stash has 268522881 entries, uses > 14 > >>> mallocs. > >>> > > [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 96812 X 96812; storage > >>> space: 89786788 unneeded,7025212 used > >>> > > [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during > MatSetValues() > >>> is 0 > >>> > > [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 81 > >>> > > [5] MatCheckCompressedRow(): Found the ratio (num_zerorows > >>> 0)/(num_localrows 96812) < 0.6. Do not use CompressedRow routines. > >>> > > [5] MatSeqAIJCheckInode(): Found 32271 nodes of 96812. Limit used: > >>> > > 5. > >>> Using Inode routines > >>> > > [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 96812 X 96812; storage > >>> space: 89841924 unneeded,6970076 used > >>> > > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during > MatSetValues() > >>> is 0 > >>> > > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 81 > >>> > > [4] MatCheckCompressedRow(): Found the ratio (num_zerorows > >>> 0)/(num_localrows 96812) < 0.6. Do not use CompressedRow routines. > >>> > > [4] MatSeqAIJCheckInode(): Found 32272 nodes of 96812. Limit used: > >>> > > 5. > >>> Using Inode routines > >>> > > > >>> > > stuck here!!!! > >>> > > > >>> > > > >>> > > Any one have ideas on this? Thank you very much! > >>> > > > >>> > > > >>> > > > >>> > > Fangbo Wang > >>> > > > >>> > > > >>> > > > >>> > > -- > >>> > > Fangbo Wang, PhD student > >>> > > Stochastic Geomechanics Research Group > >>> > > Department of Civil, Structural and Environmental Engineering > >>> > > University at Buffalo > >>> > > Email: fangbowa at buffalo.edu > >>> > > >>> > > >>> > > >>> > > >>> > -- > >>> > Fangbo Wang, PhD student > >>> > Stochastic Geomechanics Research Group > >>> > Department of Civil, Structural and Environmental Engineering > >>> > University at Buffalo > >>> > Email: fangbowa at buffalo.edu > >>> > >>> > >> > >> > >> -- > >> Fangbo Wang, PhD student > >> Stochastic Geomechanics Research Group > >> Department of Civil, Structural and Environmental Engineering > >> University at Buffalo > >> Email: *fangbowa at buffalo.edu * > >> > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From francesco.casalegno at epfl.ch Mon Feb 27 10:29:24 2017 From: francesco.casalegno at epfl.ch (Casalegno Francesco) Date: Mon, 27 Feb 2017 16:29:24 +0000 Subject: [petsc-users] Number of unknowns/proc for scaling tests Message-ID: <1488212964902.78356@epfl.ch> Hello, I am pretty new to PETSc and I would like to get some advice on choosing the right ratio of unknowns/proc. I have implemented a finite element solver for 3D problems with potentially up to 5B vertices, using CG as iterative solver. To run weak scaling tests I have to keep constant the ratio unknowns/proc, but if that if this ratio is too small I will not see any scaling as the communication overhead would dominate. Similarly, for strong scaling tests I expect to observe nearly-linear scaling only up to a certain number of procs, after which communications will dominate. My question is: what is a reasonable minimum number of unknowns per proc to get good scaling efficiency? I know that for <50k unknowns a direct solver would be preferable, but I do not know if this number is also a reasonable guess for the ratio unknown/proc. Thanks in advance for your help! Francesco -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas.drinkt.thee at gmail.com Mon Feb 27 10:37:19 2017 From: lukas.drinkt.thee at gmail.com (Lukas van de Wiel) Date: Mon, 27 Feb 2017 17:37:19 +0100 Subject: [petsc-users] Number of unknowns/proc for scaling tests In-Reply-To: <1488212964902.78356@epfl.ch> References: <1488212964902.78356@epfl.ch> Message-ID: Hi Casalegno, The answer to your question is in the PETSc FAQ: http://www.mcs.anl.gov/petsc/documentation/faq.html Have a great day Lukas On Mon, Feb 27, 2017 at 5:29 PM, Casalegno Francesco < francesco.casalegno at epfl.ch> wrote: > Hello, > > > I am pretty new to PETSc and I would like to get some advice on choosing > the right ratio of unknowns/proc. > > > I have implemented a finite element solver for 3D problems with > potentially up to 5B vertices, using CG as iterative solver. To run weak > scaling tests I have to keep constant the ratio unknowns/proc, but if that > if this ratio is too small I will not see any scaling as the communication > overhead would dominate. Similarly, for strong scaling tests I expect to > observe nearly-linear scaling only up to a certain number of procs, after > which communications will dominate. > > > My question is: what is a reasonable minimum number of unknowns per proc > to get good scaling efficiency? I know that for <50k unknowns a direct > solver would be preferable, but I do not know if this number is also a > reasonable guess for the ratio unknown/proc. > > > Thanks in advance for your help! > > > Francesco > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Feb 27 11:08:56 2017 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 27 Feb 2017 11:08:56 -0600 Subject: [petsc-users] Number of unknowns/proc for scaling tests In-Reply-To: <1488212964902.78356@epfl.ch> References: <1488212964902.78356@epfl.ch> Message-ID: On Mon, Feb 27, 2017 at 10:29 AM, Casalegno Francesco < francesco.casalegno at epfl.ch> wrote: > Hello, > > I am pretty new to PETSc and I would like to get some advice on choosing > the right ratio of unknowns/proc. > > I have implemented a finite element solver for 3D problems with > potentially up to 5B vertices, using CG as iterative solver. To run weak > scaling tests I have to keep constant the ratio unknowns/proc, but if that > if this ratio is too small I will not see any scaling as the communication > overhead would dominate. Similarly, for strong scaling tests I expect to > observe nearly-linear scaling only up to a certain number of procs, after > which communications will dominate. > > My question is: what is a reasonable minimum number of unknowns per proc > to get good scaling efficiency? I know that for <50k unknowns a direct > solver would be preferable, but I do not know if this number is also a > reasonable guess for the ratio unknown/proc. > We have normally recommended > 10K, but of course this depends on the speed of the processor/memory vs. the network, and the arithmetic intensity of your algorithm. Consider using the "performance spectrum" plot. Here is an example: https://hpgmg.org/2016/06/21/isc16-list/ You run a variety of problem sizes, and you can see where strong scaling starts to tail off on the left edge. Thanks, Matt > Thanks in advance for your help! > > > Francesco > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From gideon.simpson at gmail.com Mon Feb 27 21:18:17 2017 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Mon, 27 Feb 2017 22:18:17 -0500 Subject: [petsc-users] issue with ghost points Message-ID: Not sure if this is a bug or not, but, for the following code: #include #include #include typedef struct{ PetscScalar u; PetscScalar v; }b_node; int main(int argc, char **args) { char dataname[PETSC_MAX_PATH_LEN] = "testvec.bin"; PetscInt N = 8; DM dm; Vec b; PetscScalar mass, eng; PetscViewer viewer; DMDALocalInfo info; b_node * b_array; Vec b_local; PetscInt local_index, local_width, i; PetscScalar u, v; PetscInitialize(&argc,&args,NULL,NULL); PetscOptionsGetInt(NULL,NULL,"-N",&N,NULL); DMDACreate1d(PETSC_COMM_WORLD, DM_BOUNDARY_GHOSTED, N, 2 , 1 , NULL, &dm); DMCreateGlobalVector(dm ,&b); /* Load vector from disk */ PetscViewerBinaryOpen(PETSC_COMM_WORLD,dataname, FILE_MODE_READ, &viewer); VecLoad(b, viewer); PetscViewerDestroy(&viewer); /* Inspect with viewer */ VecView(b, PETSC_VIEWER_STDOUT_WORLD); DMGetLocalVector(dm,&b_local); DMGlobalToLocalBegin(dm,b,INSERT_VALUES,b_local); DMGlobalToLocalEnd(dm,b,INSERT_VALUES,b_local); DMDAVecGetArray(dm, b_local, &b_array); DMDAGetCorners (dm, &local_index, NULL, NULL,&local_width, NULL, NULL); /* Zero out the ghost points */ if(info.xs == 0){ b_array[-1].u =0.0; b_array[-1].v =0.0; } if(info.xs + info.xm == info.mx){ b_array[info.mx].u =0.0; b_array[info.mx].v =0.0; } /* Manually inspect */ for(i=local_index;i From bsmith at mcs.anl.gov Mon Feb 27 21:28:25 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 27 Feb 2017 21:28:25 -0600 Subject: [petsc-users] issue with ghost points In-Reply-To: References: Message-ID: <4F015B0D-2BF6-4C57-A119-EDB64489A945@mcs.anl.gov> You are using the info struct without ever having set values into it You are, at a minimum, missing a call to DMDAGetLocalInfo(). Valgrind would immediately have found the use of uninitialized values. Note that DMDAGetCorners() provides some of the information that DMDAGetLocalInfo() provides but in a slightly different format. Barry > On Feb 27, 2017, at 9:18 PM, Gideon Simpson wrote: > > Not sure if this is a bug or not, but, for the following code: > > #include > #include > #include > > typedef struct{ > PetscScalar u; > PetscScalar v; > }b_node; > > > int main(int argc, char **args) > { > char dataname[PETSC_MAX_PATH_LEN] = "testvec.bin"; > PetscInt N = 8; > DM dm; > Vec b; > PetscScalar mass, eng; > PetscViewer viewer; > DMDALocalInfo info; > b_node * b_array; > Vec b_local; > PetscInt local_index, local_width, i; > PetscScalar u, v; > > > PetscInitialize(&argc,&args,NULL,NULL); > > PetscOptionsGetInt(NULL,NULL,"-N",&N,NULL); > > DMDACreate1d(PETSC_COMM_WORLD, DM_BOUNDARY_GHOSTED, N, 2 , 1 , NULL, &dm); > DMCreateGlobalVector(dm ,&b); > > /* Load vector from disk */ > PetscViewerBinaryOpen(PETSC_COMM_WORLD,dataname, FILE_MODE_READ, &viewer); > VecLoad(b, viewer); > PetscViewerDestroy(&viewer); > > /* Inspect with viewer */ > VecView(b, PETSC_VIEWER_STDOUT_WORLD); > > DMGetLocalVector(dm,&b_local); > DMGlobalToLocalBegin(dm,b,INSERT_VALUES,b_local); > DMGlobalToLocalEnd(dm,b,INSERT_VALUES,b_local); > > DMDAVecGetArray(dm, b_local, &b_array); > DMDAGetCorners (dm, &local_index, NULL, NULL,&local_width, NULL, NULL); > > /* Zero out the ghost points */ > if(info.xs == 0){ > b_array[-1].u =0.0; > b_array[-1].v =0.0; > } > if(info.xs + info.xm == info.mx){ > b_array[info.mx].u =0.0; > b_array[info.mx].v =0.0; > } > > /* Manually inspect */ > for(i=local_index;i u = b_array[i].u; > v = b_array[i].v; > PetscPrintf(PETSC_COMM_WORLD," %i: u = %g, v = %g\n", i, u, v); > } > > DMDAVecRestoreArray (dm,b_local,&b_array); > DMRestoreLocalVector(dm,&b_local); > > VecDestroy(&b); > DMDestroy(&dm) ; > > PetscFinalize(); > return 0; > > } > > I am getting the output > > Vec Object: 1 MPI processes > type: seq > Vec Object:Vec_0x84000000_0 1 MPI processes > type: mpi > Process [0] > 1. > 0. > 0.707107 > 0.707107 > 6.12323e-17 > 1. > -0.707107 > 0.707107 > -1. > 1.22465e-16 > -0.707107 > -0.707107 > -1.83697e-16 > -1. > 0.707107 > -0.707107 > 0: u = 0., v = 0. > 1: u = 0.707107, v = 0.707107 > 2: u = 6.12323e-17, v = 1. > 3: u = -0.707107, v = 0.707107 > 4: u = -1., v = 1.22465e-16 > 5: u = -0.707107, v = -0.707107 > 6: u = -1.83697e-16, v = -1. > 7: u = 0.707107, v = -0.707107 > > The input vector is an alternating sequence of cos( pi / 4 * j), sin(pi/4 * j), for j = 0,1,2?7. What I want to point out is that the first element of this vector, as it is read in, is 1. But when we do the manually looping, it spits out u = 0 at index i = 0. If i turn off the piece of code that zeros out the ghost points, the problem disappears. The reason I am concerned is that this kind of looping through the local vector with ghost points shows up in a piece of my code, and I am now concerned about the output. Do I have a bug here? > > > > -gideon > From gideon.simpson at gmail.com Mon Feb 27 21:34:13 2017 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Mon, 27 Feb 2017 22:34:13 -0500 Subject: [petsc-users] issue with ghost points In-Reply-To: <4F015B0D-2BF6-4C57-A119-EDB64489A945@mcs.anl.gov> References: <4F015B0D-2BF6-4C57-A119-EDB64489A945@mcs.anl.gov> Message-ID: <417438EA-2B62-4866-9BCB-D4853C337B3B@gmail.com> Thanks, as always. Interesting that -Wall (with clang) didn?t catch that. -gideon > On Feb 27, 2017, at 10:28 PM, Barry Smith wrote: > > > You are using the info struct without ever having set values into it You are, at a minimum, missing a call to DMDAGetLocalInfo(). Valgrind would immediately have found the use of uninitialized values. > > Note that DMDAGetCorners() provides some of the information that DMDAGetLocalInfo() provides but in a slightly different format. > > > Barry > > > >> On Feb 27, 2017, at 9:18 PM, Gideon Simpson wrote: >> >> Not sure if this is a bug or not, but, for the following code: >> >> #include >> #include >> #include >> >> typedef struct{ >> PetscScalar u; >> PetscScalar v; >> }b_node; >> >> >> int main(int argc, char **args) >> { >> char dataname[PETSC_MAX_PATH_LEN] = "testvec.bin"; >> PetscInt N = 8; >> DM dm; >> Vec b; >> PetscScalar mass, eng; >> PetscViewer viewer; >> DMDALocalInfo info; >> b_node * b_array; >> Vec b_local; >> PetscInt local_index, local_width, i; >> PetscScalar u, v; >> >> >> PetscInitialize(&argc,&args,NULL,NULL); >> >> PetscOptionsGetInt(NULL,NULL,"-N",&N,NULL); >> >> DMDACreate1d(PETSC_COMM_WORLD, DM_BOUNDARY_GHOSTED, N, 2 , 1 , NULL, &dm); >> DMCreateGlobalVector(dm ,&b); >> >> /* Load vector from disk */ >> PetscViewerBinaryOpen(PETSC_COMM_WORLD,dataname, FILE_MODE_READ, &viewer); >> VecLoad(b, viewer); >> PetscViewerDestroy(&viewer); >> >> /* Inspect with viewer */ >> VecView(b, PETSC_VIEWER_STDOUT_WORLD); >> >> DMGetLocalVector(dm,&b_local); >> DMGlobalToLocalBegin(dm,b,INSERT_VALUES,b_local); >> DMGlobalToLocalEnd(dm,b,INSERT_VALUES,b_local); >> >> DMDAVecGetArray(dm, b_local, &b_array); >> DMDAGetCorners (dm, &local_index, NULL, NULL,&local_width, NULL, NULL); >> >> /* Zero out the ghost points */ >> if(info.xs == 0){ >> b_array[-1].u =0.0; >> b_array[-1].v =0.0; >> } >> if(info.xs + info.xm == info.mx){ >> b_array[info.mx].u =0.0; >> b_array[info.mx].v =0.0; >> } >> >> /* Manually inspect */ >> for(i=local_index;i> u = b_array[i].u; >> v = b_array[i].v; >> PetscPrintf(PETSC_COMM_WORLD," %i: u = %g, v = %g\n", i, u, v); >> } >> >> DMDAVecRestoreArray (dm,b_local,&b_array); >> DMRestoreLocalVector(dm,&b_local); >> >> VecDestroy(&b); >> DMDestroy(&dm) ; >> >> PetscFinalize(); >> return 0; >> >> } >> >> I am getting the output >> >> Vec Object: 1 MPI processes >> type: seq >> Vec Object:Vec_0x84000000_0 1 MPI processes >> type: mpi >> Process [0] >> 1. >> 0. >> 0.707107 >> 0.707107 >> 6.12323e-17 >> 1. >> -0.707107 >> 0.707107 >> -1. >> 1.22465e-16 >> -0.707107 >> -0.707107 >> -1.83697e-16 >> -1. >> 0.707107 >> -0.707107 >> 0: u = 0., v = 0. >> 1: u = 0.707107, v = 0.707107 >> 2: u = 6.12323e-17, v = 1. >> 3: u = -0.707107, v = 0.707107 >> 4: u = -1., v = 1.22465e-16 >> 5: u = -0.707107, v = -0.707107 >> 6: u = -1.83697e-16, v = -1. >> 7: u = 0.707107, v = -0.707107 >> >> The input vector is an alternating sequence of cos( pi / 4 * j), sin(pi/4 * j), for j = 0,1,2?7. What I want to point out is that the first element of this vector, as it is read in, is 1. But when we do the manually looping, it spits out u = 0 at index i = 0. If i turn off the piece of code that zeros out the ghost points, the problem disappears. The reason I am concerned is that this kind of looping through the local vector with ghost points shows up in a piece of my code, and I am now concerned about the output. Do I have a bug here? >> >> >> >> -gideon >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Mon Feb 27 21:57:44 2017 From: jed at jedbrown.org (Jed Brown) Date: Mon, 27 Feb 2017 20:57:44 -0700 Subject: [petsc-users] issue with ghost points In-Reply-To: <417438EA-2B62-4866-9BCB-D4853C337B3B@gmail.com> References: <4F015B0D-2BF6-4C57-A119-EDB64489A945@mcs.anl.gov> <417438EA-2B62-4866-9BCB-D4853C337B3B@gmail.com> Message-ID: <87poi3ge6f.fsf@jedbrown.org> Gideon Simpson writes: > Thanks, as always. > > Interesting that -Wall (with clang) didn?t catch that. gcc warns on this if you turn on optimization. I don't know why clang doesn't. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From gideon.simpson at gmail.com Tue Feb 28 09:57:42 2017 From: gideon.simpson at gmail.com (Gideon Simpson) Date: Tue, 28 Feb 2017 10:57:42 -0500 Subject: [petsc-users] default TS RK Message-ID: <5C8966E9-001A-49B2-B829-37C596410BAD@gmail.com> I just wanted to check, has the default RK solver (when called with -ts_type rk) always been 3rd order (RK3)? -gideon From emconsta at mcs.anl.gov Tue Feb 28 10:08:37 2017 From: emconsta at mcs.anl.gov (Emil Constantinescu) Date: Tue, 28 Feb 2017 10:08:37 -0600 Subject: [petsc-users] default TS RK In-Reply-To: <5C8966E9-001A-49B2-B829-37C596410BAD@gmail.com> References: <5C8966E9-001A-49B2-B829-37C596410BAD@gmail.com> Message-ID: We changed it a couple of years ago. Emil On 2/28/17 9:57 AM, Gideon Simpson wrote: > I just wanted to check, has the default RK solver (when called with -ts_type rk) always been 3rd order (RK3)? > > > -gideon > From jed at jedbrown.org Tue Feb 28 10:10:40 2017 From: jed at jedbrown.org (Jed Brown) Date: Tue, 28 Feb 2017 09:10:40 -0700 Subject: [petsc-users] default TS RK In-Reply-To: <5C8966E9-001A-49B2-B829-37C596410BAD@gmail.com> References: <5C8966E9-001A-49B2-B829-37C596410BAD@gmail.com> Message-ID: <874lzegutb.fsf@jedbrown.org> Gideon Simpson writes: > I just wanted to check, has the default RK solver (when called with -ts_type rk) always been 3rd order (RK3)? Yes, since it was rewritten to be adaptive and extensible in 2013. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From Rodrigo.Felicio at iongeo.com Tue Feb 28 18:17:57 2017 From: Rodrigo.Felicio at iongeo.com (Rodrigo Felicio) Date: Wed, 1 Mar 2017 00:17:57 +0000 Subject: [petsc-users] petsc4py and MPI.COMM_SELF.Spawn Message-ID: <350529B93F4E2F4497FD8DE4E86E84AA16F1DC6F@AUS1EXMBX04.ioinc.ioroot.tld> Dear All, I am new to both pestc (petsc4py) and MPI. I am prototyping an application that acts on data belonging to thousands of different spatial locations, so I decided to use mpi4py to parallelize my computations over said set of points. At each point, however, I have to solve linear systems of the type A x = b, using least-squares. As matrices A are sparse and large, I would like use petsc LSQR solver to obtain x. Trouble is that I cannot get mpi4py to work with petsc4py when using MPI.Spawn(), so that I could solve those linear systems using threads spawned from my main algorithm. I am not sure if this is a problem with my petsc/mpi installation, or if there is some inherent incompatibility between mpi4py and petsc4py on spawned threads. Anyway, I would be really thankful to anyone who could shed some light on this issue. For illustration, the child and parent codes found on the pestc4py spawning demo folder fail for me if I just include the petsc4py initialization in either of them. In fact, just the introduction of the line " from petsc4py import PETSc" is enough to make the code to hang and issue, upon keyboard termination, error msgs of the type "spawned process group was unable to connect back to parent port" . Kind regards Rodrigo ------------------------ CHILD code: ------------------------- # inclusion of petsc4py lines causes code to hang import petsc4py petsc4py.init(sys.argv) from petsc4py import PETSc from mpi4py import MPI from array import array master = MPI.Comm.Get_parent() nprocs = master.Get_size() myrank = master.Get_rank() n = array('i', [0]) master.Bcast([n, MPI.INT], root=0) n = n[0] h = 1.0 / n s = 0.0 for i in range(myrank+1, n+1, nprocs): x = h * (i - 0.5) s += 4.0 / (1.0 + x**2) pi = s * h pi = array('d', [pi]) master.Reduce(sendbuf=[pi, MPI.DOUBLE], recvbuf=None, op=MPI.SUM, root=0) master.Disconnect() --------------------- MASTER CODE: ---------------------- from mpi4py import MPI from array import array from math import pi as PI from sys import argv cmd = 'cpi-worker-py.exe' if len(argv) > 1: cmd = argv[1] print("%s -> %s" % (argv[0], cmd)) worker = MPI.COMM_SELF.Spawn(cmd, None, 5) n = array('i', [100]) worker.Bcast([n,MPI.INT], root=MPI.ROOT) pi = array('d', [0.0]) worker.Reduce(sendbuf=None, recvbuf=[pi, MPI.DOUBLE], op=MPI.SUM, root=MPI.ROOT) pi = pi[0] worker.Disconnect() print('pi: %.16f, error: %.16f' % (pi, abs(PI-pi))) ________________________________ This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender and delete the original. From bsmith at mcs.anl.gov Tue Feb 28 19:45:18 2017 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 28 Feb 2017 19:45:18 -0600 Subject: [petsc-users] petsc4py and MPI.COMM_SELF.Spawn In-Reply-To: <350529B93F4E2F4497FD8DE4E86E84AA16F1DC6F@AUS1EXMBX04.ioinc.ioroot.tld> References: <350529B93F4E2F4497FD8DE4E86E84AA16F1DC6F@AUS1EXMBX04.ioinc.ioroot.tld> Message-ID: <7D62F674-BEE6-43C8-A631-C5207935ED1F@mcs.anl.gov> Always cut and paste the full error message: Based on the fact that import petsc4py doesn't work this means that either petsc4py did not build successfully or the environment is not correct; for example is PYTHONPATH set properly? > On Feb 28, 2017, at 6:17 PM, Rodrigo Felicio wrote: > > Dear All, > > I am new to both pestc (petsc4py) and MPI. I am prototyping an application that acts on data belonging to thousands of different spatial locations, so I decided to use mpi4py to parallelize my computations over said set of points. At each point, however, I have to solve linear systems of the type A x = b, using least-squares. As matrices A are sparse and large, I would like use petsc LSQR solver to obtain x. Trouble is that I cannot get mpi4py to work with petsc4py when using MPI.Spawn(), so that I could solve those linear systems using threads spawned from my main algorithm. I am not sure if this is a problem with my petsc/mpi installation, or if there is some inherent incompatibility between mpi4py and petsc4py on spawned threads. Anyway, I would be really thankful to anyone who could shed some light on this issue. For illustration, the child and parent codes found on the pestc4py spawning demo folder fail for me if I just include the petsc4py initialization in either of them. In fact, just the introduction of the line " from petsc4py import PETSc" is enough to make the code to hang and issue, upon keyboard termination, error msgs of the type "spawned process group was unable to connect back to parent port" . > > Kind regards > Rodrigo > > ------------------------ > CHILD code: > ------------------------- > > # inclusion of petsc4py lines causes code to hang > import petsc4py > petsc4py.init(sys.argv) > from petsc4py import PETSc > from mpi4py import MPI > from array import array > > master = MPI.Comm.Get_parent() > nprocs = master.Get_size() > myrank = master.Get_rank() > > n = array('i', [0]) > master.Bcast([n, MPI.INT], root=0) > n = n[0] > > h = 1.0 / n > s = 0.0 > for i in range(myrank+1, n+1, nprocs): > x = h * (i - 0.5) > s += 4.0 / (1.0 + x**2) > pi = s * h > > pi = array('d', [pi]) > master.Reduce(sendbuf=[pi, MPI.DOUBLE], > recvbuf=None, > op=MPI.SUM, root=0) > > master.Disconnect() > > > --------------------- > MASTER CODE: > ---------------------- > from mpi4py import MPI > from array import array > from math import pi as PI > from sys import argv > > cmd = 'cpi-worker-py.exe' > if len(argv) > 1: cmd = argv[1] > print("%s -> %s" % (argv[0], cmd)) > > worker = MPI.COMM_SELF.Spawn(cmd, None, 5) > > n = array('i', [100]) > worker.Bcast([n,MPI.INT], root=MPI.ROOT) > > pi = array('d', [0.0]) > worker.Reduce(sendbuf=None, > recvbuf=[pi, MPI.DOUBLE], > op=MPI.SUM, root=MPI.ROOT) > pi = pi[0] > > worker.Disconnect() > > print('pi: %.16f, error: %.16f' % (pi, abs(PI-pi))) > > > > > > ________________________________ > > > This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender and delete the original. >