From jzk164 at psu.edu Mon Nov 1 12:35:12 2010 From: jzk164 at psu.edu (jibum kim) Date: Mon, 1 Nov 2010 13:35:12 -0400 Subject: [petsc-users] Boomeramg as a solver Message-ID: Hello, I've used Boomeramg as a preconditioner. Now, I am wondering whether I can use Boomeramg as a solver ? and is it possible that I use Boomeramg as both a preconditioner and a solver? I saw one slide saying that I can use KSPSetType(ksp, KSPRICHARDSON); But when I try it and use -ksp_view, it shows that my solver is Richardson not Boomeramg. Could u clarify this issue ? Thank you. - Jibum -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Nov 1 13:18:40 2010 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 1 Nov 2010 13:18:40 -0500 Subject: [petsc-users] Boomeramg as a solver In-Reply-To: References: Message-ID: 2010/11/1 jibum kim > Hello, > > I've used Boomeramg as a preconditioner. > > Now, I am wondering whether I can use Boomeramg as a solver ? and is it > possible that I use Boomeramg as both a preconditioner and a solver? > > I saw one slide saying that I can use KSPSetType(ksp, KSPRICHARDSON); > > But when I try it and use -ksp_view, it shows that my solver is Richardson > not Boomeramg. > > Could u clarify this issue ? > Use -ksp_type richardson -pc_type hypre -pc_hypre_type boomeramg Matt > > Thank you. > > - Jibum > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Nov 1 13:25:48 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 1 Nov 2010 13:25:48 -0500 Subject: [petsc-users] Boomeramg as a solver In-Reply-To: References: Message-ID: <6D113290-3D2D-4B9B-8C5B-6B321CD79C4A@mcs.anl.gov> On Nov 1, 2010, at 12:35 PM, jibum kim wrote: > Hello, > > I've used Boomeramg as a preconditioner. > > Now, I am wondering whether I can use Boomeramg as a solver ? and is it possible that I use Boomeramg as both a preconditioner and a solver? > > I saw one slide saying that I can use KSPSetType(ksp, KSPRICHARDSON); > > But when I try it and use -ksp_view, it shows that my solver is Richardson not Boomeramg. > > Could u clarify this issue ? Richardson is our name for the iterative process u^{n+1} = u^n + B(b - A u^n) for n=1,...max_it where B is the application of the preconditioner (in other words running a linear iterative method without any Krylov acceleration). In BoomerAMG's case B is one application of the V or W cycle (depending on your choice of BoomerAMG options). So even though it says the "solver" is Richardson it is using BoomerAMG for the entire solution process, that is algebraic multigrid with multiple V (or W cycles). Barry > > Thank you. > > - Jibum From jzk164 at psu.edu Mon Nov 1 14:46:28 2010 From: jzk164 at psu.edu (jibum kim) Date: Mon, 1 Nov 2010 15:46:28 -0400 Subject: [petsc-users] Boomeramg as a solver References: <6D113290-3D2D-4B9B-8C5B-6B321CD79C4A@mcs.anl.gov> Message-ID: <08DC9265EEED476786510A3D5B4D9BD8@jibumpsu> Are you saying that boomeramg with richardson iterations is same as the multigrid without a preconditioner? and if it the case, how can I set an extra preconditioner like ILU ? Thanks, - Jibum ----- Original Message ----- From: "Barry Smith" To: "PETSc users list" Sent: Monday, November 01, 2010 2:25 PM Subject: Re: [petsc-users] Boomeramg as a solver On Nov 1, 2010, at 12:35 PM, jibum kim wrote: > Hello, > > I've used Boomeramg as a preconditioner. > > Now, I am wondering whether I can use Boomeramg as a solver ? and is it > possible that I use Boomeramg as both a preconditioner and a solver? > > I saw one slide saying that I can use KSPSetType(ksp, KSPRICHARDSON); > > But when I try it and use -ksp_view, it shows that my solver is Richardson > not Boomeramg. > > Could u clarify this issue ? Richardson is our name for the iterative process u^{n+1} = u^n + B(b - A u^n) for n=1,...max_it where B is the application of the preconditioner (in other words running a linear iterative method without any Krylov acceleration). In BoomerAMG's case B is one application of the V or W cycle (depending on your choice of BoomerAMG options). So even though it says the "solver" is Richardson it is using BoomerAMG for the entire solution process, that is algebraic multigrid with multiple V (or W cycles). Barry > > Thank you. > > - Jibum From bsmith at mcs.anl.gov Mon Nov 1 14:54:21 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 1 Nov 2010 14:54:21 -0500 Subject: [petsc-users] Boomeramg as a solver In-Reply-To: <08DC9265EEED476786510A3D5B4D9BD8@jibumpsu> References: <6D113290-3D2D-4B9B-8C5B-6B321CD79C4A@mcs.anl.gov> <08DC9265EEED476786510A3D5B4D9BD8@jibumpsu> Message-ID: On Nov 1, 2010, at 2:46 PM, jibum kim wrote: > Are you saying that boomeramg with richardson iterations is same as the multigrid without a preconditioner? I do not know what you mean by multigrid with or without a preconditioner? Multigrid doesn't "take a preconditioner", it is an iterative solver that can be used as a preconditioner but it cannot use other preconditioners. > and if it the case, how can I set an extra preconditioner like ILU ? What do you mean by an "extra preconditioner"? I assume you mean use ILU as a smoother instead of SOR or Jacobi. BoomerAMG has no mechanism to use ILU as a smoother so you cannot do this. Because ML is more flexible it is possible to use PETSc's ILU as a smoother for it. Barry > > Thanks, > > - Jibum > > ----- Original Message ----- From: "Barry Smith" > To: "PETSc users list" > Sent: Monday, November 01, 2010 2:25 PM > Subject: Re: [petsc-users] Boomeramg as a solver > > > > On Nov 1, 2010, at 12:35 PM, jibum kim wrote: > >> Hello, >> >> I've used Boomeramg as a preconditioner. >> >> Now, I am wondering whether I can use Boomeramg as a solver ? and is it possible that I use Boomeramg as both a preconditioner and a solver? >> >> I saw one slide saying that I can use KSPSetType(ksp, KSPRICHARDSON); >> >> But when I try it and use -ksp_view, it shows that my solver is Richardson not Boomeramg. >> >> Could u clarify this issue ? > > Richardson is our name for the iterative process u^{n+1} = u^n + B(b - A u^n) for n=1,...max_it where B is the application of the preconditioner (in other words running a linear iterative method without any Krylov acceleration). In BoomerAMG's case B is one application of the V or W cycle (depending on your choice of BoomerAMG options). So even though it says the "solver" is Richardson it is using BoomerAMG for the entire solution process, that is algebraic multigrid with multiple V (or W cycles). > > Barry > > > > >> >> Thank you. >> >> - Jibum > > From jzk164 at psu.edu Mon Nov 1 15:02:43 2010 From: jzk164 at psu.edu (jibum kim) Date: Mon, 1 Nov 2010 16:02:43 -0400 Subject: [petsc-users] Boomeramg as a solver Message-ID: <2F3A3E510DD842ADB104E00C03D04E07@jibumpsu> Now it is clear. Thank you for your help. - Jibum ----- Original Message ----- From: "jibum kim" To: "PETSc users list" Sent: Monday, November 01, 2010 3:46 PM Subject: Re: [petsc-users] Boomeramg as a solver > Are you saying that boomeramg with richardson iterations is same as the > multigrid without a preconditioner? and if it the case, how can I set an > extra preconditioner like ILU ? > > Thanks, > > - Jibum > > ----- Original Message ----- > From: "Barry Smith" > To: "PETSc users list" > Sent: Monday, November 01, 2010 2:25 PM > Subject: Re: [petsc-users] Boomeramg as a solver > > > > On Nov 1, 2010, at 12:35 PM, jibum kim wrote: > >> Hello, >> >> I've used Boomeramg as a preconditioner. >> >> Now, I am wondering whether I can use Boomeramg as a solver ? and is it >> possible that I use Boomeramg as both a preconditioner and a solver? >> >> I saw one slide saying that I can use KSPSetType(ksp, KSPRICHARDSON); >> >> But when I try it and use -ksp_view, it shows that my solver is >> Richardson not Boomeramg. >> >> Could u clarify this issue ? > > Richardson is our name for the iterative process u^{n+1} = u^n + B(b - > A u^n) for n=1,...max_it where B is the application of the preconditioner > (in other words running a linear iterative method without any Krylov > acceleration). In BoomerAMG's case B is one application of the V or W > cycle (depending on your choice of BoomerAMG options). So even though it > says the "solver" is Richardson it is using BoomerAMG for the entire > solution process, that is algebraic multigrid with multiple V (or W > cycles). > > Barry > > > > >> >> Thank you. >> >> - Jibum > > From andreas.hauffe at tu-dresden.de Wed Nov 3 02:39:13 2010 From: andreas.hauffe at tu-dresden.de (Andreas Hauffe) Date: Wed, 3 Nov 2010 08:39:13 +0100 Subject: [petsc-users] Compiler Warning In-Reply-To: <2F3A3E510DD842ADB104E00C03D04E07@jibumpsu> References: <2F3A3E510DD842ADB104E00C03D04E07@jibumpsu> Message-ID: <201011030839.13586.andreas.hauffe@tu-dresden.de> Hi, during the compilation process I'm getting the following warning. Is this a problem? Warning libfast in: /scratch/ahauffe/petsc-3.1-p5/src/vec/vec/utils vscat.c(322): warning #592: variable "sto" is used before its value is set ierr = PetscMalloc2(1,VecScatter_MPI_ToAll,&sto,size,PetscMPIInt,&sto- >count);CHKERRQ(ierr); Best regards, -- Andreas Hauffe ---------------------------------------------------------------------------------------------------- Technische Universit?t Dresden Institut f?r Luft- und Raumfahrttechnik / Institute of Aerospace Engineering Lehrstuhl f?r Luftfahrzeugtechnik / Chair of Aircraft Engineering D-01062 Dresden Germany phone : (++49)351 463 38496 fax : (++49)351 463 37263 mail : andreas.hauffe at tu-dresden.de Website : http://tu-dresden.de/mw/ilr/lft ---------------------------------------------------------------------------------------------------- From jed at 59A2.org Wed Nov 3 07:33:35 2010 From: jed at 59A2.org (Jed Brown) Date: Wed, 3 Nov 2010 07:33:35 -0500 Subject: [petsc-users] Compiler Warning In-Reply-To: <201011030839.13586.andreas.hauffe@tu-dresden.de> References: <2F3A3E510DD842ADB104E00C03D04E07@jibumpsu> <201011030839.13586.andreas.hauffe@tu-dresden.de> Message-ID: On Wed, Nov 3, 2010 at 02:39, Andreas Hauffe wrote: > Hi, > > during the compilation process I'm getting the following warning. Is this a > problem? > > Warning libfast in: /scratch/ahauffe/petsc-3.1-p5/src/vec/vec/utils > vscat.c(322): warning #592: variable "sto" is used before its value is set > ierr = > PetscMalloc2(1,VecScatter_MPI_ToAll,&sto,size,PetscMPIInt,&sto- > >count);CHKERRQ(ierr); Thanks for pointing this out, I didn't notice when it was introduced (to fix a different warning). http://petsc.cs.iit.edu/petsc/petsc-dev/rev/4ff7b4bc0bea I'm surprised other compilers haven't warned about this, and that we haven't seen any bad behavior because of it. We either need to find a way to tweak the macro to clean up after itself correctly in the case of malloc failure, without dereferencing something that may not exist yet (conflicting goals), or fix any calls that try to build a structure like this with a single malloc. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Nov 3 08:09:35 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 3 Nov 2010 08:09:35 -0500 Subject: [petsc-users] Compiler Warning In-Reply-To: References: <2F3A3E510DD842ADB104E00C03D04E07@jibumpsu> <201011030839.13586.andreas.hauffe@tu-dresden.de> Message-ID: <004A607D-0B1E-48A4-AB3D-DB8A55388CA6@mcs.anl.gov> On Nov 3, 2010, at 7:33 AM, Jed Brown wrote: > On Wed, Nov 3, 2010 at 02:39, Andreas Hauffe wrote: > Hi, > > during the compilation process I'm getting the following warning. Is this a > problem? > > Warning libfast in: /scratch/ahauffe/petsc-3.1-p5/src/vec/vec/utils > vscat.c(322): warning #592: variable "sto" is used before its value is set > ierr = > PetscMalloc2(1,VecScatter_MPI_ToAll,&sto,size,PetscMPIInt,&sto- > >count);CHKERRQ(ierr); > > Thanks for pointing this out, I didn't notice when it was introduced (to fix a different warning). > > http://petsc.cs.iit.edu/petsc/petsc-dev/rev/4ff7b4bc0bea > > I'm surprised other compilers haven't warned about this, and that we haven't seen any bad behavior because of it. > > We either need to find a way to tweak the macro to clean up after itself correctly in the case of malloc failure, without dereferencing something that may not exist yet (conflicting goals), or fix any calls that try to build a structure like this with a single malloc. Are you sure this is not a compiler bug. It seems to me the code is ok, what am I missing? Barry > > Jed From jed at 59A2.org Wed Nov 3 08:57:33 2010 From: jed at 59A2.org (Jed Brown) Date: Wed, 3 Nov 2010 08:57:33 -0500 Subject: [petsc-users] Compiler Warning In-Reply-To: <004A607D-0B1E-48A4-AB3D-DB8A55388CA6@mcs.anl.gov> References: <2F3A3E510DD842ADB104E00C03D04E07@jibumpsu> <201011030839.13586.andreas.hauffe@tu-dresden.de> <004A607D-0B1E-48A4-AB3D-DB8A55388CA6@mcs.anl.gov> Message-ID: On Wed, Nov 3, 2010 at 08:09, Barry Smith wrote: > Are you sure this is not a compiler bug. It seems to me the code is ok, what am I missing? The optimized piece starts with *(r2)=0, but this is sto->count before sto has been initialized. Jed From bsmith at mcs.anl.gov Wed Nov 3 10:54:22 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 3 Nov 2010 10:54:22 -0500 Subject: [petsc-users] Compiler Warning In-Reply-To: References: <2F3A3E510DD842ADB104E00C03D04E07@jibumpsu> <201011030839.13586.andreas.hauffe@tu-dresden.de> <004A607D-0B1E-48A4-AB3D-DB8A55388CA6@mcs.anl.gov> Message-ID: On Nov 3, 2010, at 8:57 AM, Jed Brown wrote: > On Wed, Nov 3, 2010 at 08:09, Barry Smith wrote: >> Are you sure this is not a compiler bug. It seems to me the code is ok, what am I missing? > > The optimized piece starts with *(r2)=0, but this is sto->count before > sto has been initialized. > > Jed Right, this is very bad. We set the r2 and r3 up front so the compiler won't complain about possible not set values? I guess we need to not use PetscMallocN() for structs and values in the structs. Barry From jed at 59A2.org Wed Nov 3 11:39:09 2010 From: jed at 59A2.org (Jed Brown) Date: Wed, 3 Nov 2010 11:39:09 -0500 Subject: [petsc-users] Compiler Warning In-Reply-To: References: <2F3A3E510DD842ADB104E00C03D04E07@jibumpsu> <201011030839.13586.andreas.hauffe@tu-dresden.de> <004A607D-0B1E-48A4-AB3D-DB8A55388CA6@mcs.anl.gov> Message-ID: [moving to petsc-dev] On Wed, Nov 3, 2010 at 10:54, Barry Smith wrote: > Right, this is very bad. We set the r2 and r3 up front so the compiler > won't complain about possible not set values? > I don't know, you wrote that code. I stripped it out of a local build and I don't get any warnings in optimized mode (with gcc or clang). I think those warnings were a compiler bug, unless you only saw them with-errorchecking=0. I guess we need to not use PetscMallocN() for structs and values in the > structs. > Maybe, but only if we really need to set r2 up front. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Nov 3 12:50:46 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 3 Nov 2010 12:50:46 -0500 Subject: [petsc-users] [petsc-dev] Compiler Warning In-Reply-To: References: <2F3A3E510DD842ADB104E00C03D04E07@jibumpsu> <201011030839.13586.andreas.hauffe@tu-dresden.de> <004A607D-0B1E-48A4-AB3D-DB8A55388CA6@mcs.anl.gov> Message-ID: On Nov 3, 2010, at 11:39 AM, Jed Brown wrote: > [moving to petsc-dev] > > On Wed, Nov 3, 2010 at 10:54, Barry Smith wrote: > Right, this is very bad. We set the r2 and r3 up front so the compiler won't complain about possible not set values? > > I don't know, you wrote that code. I stripped it out of a local build and I don't get any warnings in optimized mode (with gcc or clang). I think those warnings were a compiler bug, unless you only saw them with-errorchecking=0. Because of the || *r2 may never get set, but it gets used later in the routine. Maybe compilers shouldn't give the warning but some did. Barry > > I guess we need to not use PetscMallocN() for structs and values in the structs. > > Maybe, but only if we really need to set r2 up front. > > Jed From B.Sanderse at cwi.nl Thu Nov 4 09:43:11 2010 From: B.Sanderse at cwi.nl (Benjamin Sanderse) Date: Thu, 4 Nov 2010 15:43:11 +0100 (CET) Subject: [petsc-users] kronecker products In-Reply-To: Message-ID: <868548161.144466.1288881791177.JavaMail.root@zembox02.zaas.igi.nl> Hi all, I am working on a CFD code which calculates differences, averages, interpolations, etc. by computing matrix-vector and matrix-matrix products. The basis here is formed by (very) sparse matrices and extension to more dimensions is done with kronecker products. This works fine one a single processor; however, before implementing things in parallel I have two questions: - is there a way to compute kronecker products efficiently in parallel with Petsc? - if problems get big, let's say order of 10-100 million unknowns, is it still efficient to store the entire (sparse) matrices? is this commonly done, or is it better to shift to matrix-free methods? btw, the current system I can run on has several hundreds of processors and 24Gb memory per node (8 proc. per node). Thanks, Ben From jed at 59A2.org Thu Nov 4 09:50:43 2010 From: jed at 59A2.org (Jed Brown) Date: Thu, 4 Nov 2010 09:50:43 -0500 Subject: [petsc-users] kronecker products In-Reply-To: <868548161.144466.1288881791177.JavaMail.root@zembox02.zaas.igi.nl> References: <868548161.144466.1288881791177.JavaMail.root@zembox02.zaas.igi.nl> Message-ID: On Thu, Nov 4, 2010 at 09:43, Benjamin Sanderse wrote: > I am working on a CFD code which calculates differences, averages, > interpolations, etc. by computing matrix-vector and matrix-matrix products. > The basis here is formed by (very) sparse matrices and extension to more > dimensions is done with kronecker products. > This works fine one a single processor; however, before implementing things > in parallel I have two questions: > > - is there a way to compute kronecker products efficiently in parallel with > Petsc? > You could store the constitutive pieces in a MatShell. What do you have to do with the result* *(multiply with, solve with, compute singular values of, etc)? What is the relative size and sparsity of each piece? Would you be willing to reorder unknowns in the vector (perhaps with a scatter) for a more efficient implementation? Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From B.Sanderse at cwi.nl Thu Nov 4 09:58:31 2010 From: B.Sanderse at cwi.nl (Benjamin Sanderse) Date: Thu, 4 Nov 2010 15:58:31 +0100 (CET) Subject: [petsc-users] kronecker products In-Reply-To: Message-ID: <1209312240.144889.1288882711931.JavaMail.root@zembox02.zaas.igi.nl> Some matrices are just used to compute matrix-vector products, and some have to be solved for. That's basically it. The matrices are really very sparse; on the order of 10-20 diagonals (typically independent of problem size). As far as I know, reordering is hardly necessary, because I am using a structured grid. Ben ----- Original Message ----- From: "Jed Brown" To: "PETSc users list" Sent: Thursday, November 4, 2010 3:50:43 PM Subject: Re: [petsc-users] kronecker products On Thu, Nov 4, 2010 at 09:43, Benjamin Sanderse wrote: > I am working on a CFD code which calculates differences, averages, > interpolations, etc. by computing matrix-vector and matrix-matrix products. > The basis here is formed by (very) sparse matrices and extension to more > dimensions is done with kronecker products. > This works fine one a single processor; however, before implementing things > in parallel I have two questions: > > - is there a way to compute kronecker products efficiently in parallel with > Petsc? > You could store the constitutive pieces in a MatShell. What do you have to do with the result* *(multiply with, solve with, compute singular values of, etc)? What is the relative size and sparsity of each piece? Would you be willing to reorder unknowns in the vector (perhaps with a scatter) for a more efficient implementation? Jed From aron.ahmadia at kaust.edu.sa Thu Nov 4 10:01:13 2010 From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia) Date: Thu, 4 Nov 2010 18:01:13 +0300 Subject: [petsc-users] kronecker products In-Reply-To: <1209312240.144889.1288882711931.JavaMail.root@zembox02.zaas.igi.nl> References: <1209312240.144889.1288882711931.JavaMail.root@zembox02.zaas.igi.nl> Message-ID: It sounds like you will want to shift to matrix-free methods, but it is hard to make that assessment without seeing your problem formulation. I can tell you that I have seen matrix-free methods implemented more efficiently than if you had kept the entire sparse system, but I cannot tell you if it will be more efficient and worth the effort for your problem. Aron On Thu, Nov 4, 2010 at 5:58 PM, Benjamin Sanderse wrote: > Some matrices are just used to compute matrix-vector products, and some have to be solved for. That's basically it. > The matrices are really very sparse; on the order of 10-20 diagonals (typically independent of problem size). > As far as I know, reordering is hardly necessary, because I am using a structured grid. > > Ben > > ----- Original Message ----- > From: "Jed Brown" > To: "PETSc users list" > Sent: Thursday, November 4, 2010 3:50:43 PM > Subject: Re: [petsc-users] kronecker products > > On Thu, Nov 4, 2010 at 09:43, Benjamin Sanderse wrote: > >> I am working on a CFD code which calculates differences, averages, >> interpolations, etc. by computing matrix-vector and matrix-matrix products. >> The basis here is formed by (very) sparse matrices and extension to more >> dimensions is done with kronecker products. >> This works fine one a single processor; however, before implementing things >> in parallel I have two questions: >> >> - is there a way to compute kronecker products efficiently in parallel with >> Petsc? >> > > You could store the constitutive pieces in a MatShell. ?What do you have to > do with the result* *(multiply with, solve with, compute singular values of, > etc)? ?What is the relative size and sparsity of each piece? ?Would you be > willing to reorder unknowns in the vector (perhaps with a scatter) for a > more efficient implementation? > > Jed > From jed at 59A2.org Thu Nov 4 10:03:15 2010 From: jed at 59A2.org (Jed Brown) Date: Thu, 4 Nov 2010 10:03:15 -0500 Subject: [petsc-users] kronecker products In-Reply-To: <1209312240.144889.1288882711931.JavaMail.root@zembox02.zaas.igi.nl> References: <1209312240.144889.1288882711931.JavaMail.root@zembox02.zaas.igi.nl> Message-ID: On Thu, Nov 4, 2010 at 09:58, Benjamin Sanderse wrote: > Some matrices are just used to compute matrix-vector products, and some > have to be solved for. That's basically it. > The matrices are really very sparse; on the order of 10-20 diagonals > (typically independent of problem size). > As far as I know, reordering is hardly necessary, because I am using a > structured grid. > What are the relative sizes of each piece? How are you producing these matrices? How do you want the vector distributed? Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From B.Sanderse at cwi.nl Thu Nov 4 10:18:54 2010 From: B.Sanderse at cwi.nl (Benjamin Sanderse) Date: Thu, 4 Nov 2010 16:18:54 +0100 (CET) Subject: [petsc-users] kronecker products In-Reply-To: Message-ID: <306819932.145491.1288883934055.JavaMail.root@zembox02.zaas.igi.nl> I am not sure if I understand you correctly. During initialization, several matrices are formed by kronecker products, and the resulting matrices have the size of the number of unknowns; i.e. if n is the number of unknowns then the matrices are n x n, although they typically have only a*n nonzero elements, with a around 10-20. After initialization a time loop starts in which these matrices are used for matrix-vector products (e.g. to calculate a first or second derivative) and things like CG or Multigrid are used to solve a number of matrix-vector equations (e.g. a Poisson equation for the pressure). This is repeated each time step. ----- Original Message ----- From: "Jed Brown" To: "PETSc users list" Sent: Thursday, November 4, 2010 4:03:15 PM Subject: Re: [petsc-users] kronecker products On Thu, Nov 4, 2010 at 09:58, Benjamin Sanderse wrote: > Some matrices are just used to compute matrix-vector products, and some > have to be solved for. That's basically it. > The matrices are really very sparse; on the order of 10-20 diagonals > (typically independent of problem size). > As far as I know, reordering is hardly necessary, because I am using a > structured grid. > What are the relative sizes of each piece? How are you producing these matrices? How do you want the vector distributed? Jed From jed at 59A2.org Thu Nov 4 10:25:58 2010 From: jed at 59A2.org (Jed Brown) Date: Thu, 4 Nov 2010 10:25:58 -0500 Subject: [petsc-users] kronecker products In-Reply-To: <306819932.145491.1288883934055.JavaMail.root@zembox02.zaas.igi.nl> References: <306819932.145491.1288883934055.JavaMail.root@zembox02.zaas.igi.nl> Message-ID: On Thu, Nov 4, 2010 at 10:18, Benjamin Sanderse wrote: > I am not sure if I understand you correctly. During initialization, several > matrices are formed by kronecker products, and the resulting matrices have > the size of the number of unknowns; i.e. if n is the number of unknowns then > the matrices are n x n, although they typically have only a*n nonzero > elements, with a around 10-20. > I'm curious about the structure of these kronecker products. For example, you may have K = A \otimes B Is either one of A or B dense? Are they both large and distributed, or is one much smaller than the other? If they are both distributed, what distribution do you want the vector to have? A different question is, what problem are you solving? E.g. if this is a Galerkin method for stochastic PDE, it gives us some idea about the structure, and the natural follow-up question is what is the size of the (reduced how?) stochastic space, and do the stochastic basis functions have local or global support. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From B.Sanderse at cwi.nl Thu Nov 4 10:39:00 2010 From: B.Sanderse at cwi.nl (Benjamin Sanderse) Date: Thu, 4 Nov 2010 16:39:00 +0100 (CET) Subject: [petsc-users] kronecker products In-Reply-To: Message-ID: <220090183.145998.1288885140540.JavaMail.root@zembox02.zaas.igi.nl> I am using a finite volume method on structured grids and things are really simple. To give you an idea, here is the matrix that calculates du/dx, given a vector with u. For simplicity I use the Matlab way to construct such a thing: diagonal = (1/deltax)*ones(Nx,1); Dux_1D = spdiags([-diagonal diagonal],[0 1],Nx-1,Nx); % differencing matrix in 1D Dux_2D = kron(speye(Ny),Dux_1D)); Dux_3D = kron(speye(Nz),Dux_2D)); Here Nx is the number of points in x-direction, Ny in y-direction, and so on. For simplicity I took a uniform Cartesian grid and did not take boundary conditions into account. Other operators, e.g. involving cross derivatives, are more complicated, but the basic idea should be clear. Notice that here the extension to multi-D is simply with an identity matrix, so both matrices in the kron command are really sparse. Ben ----- Original Message ----- From: "Jed Brown" To: "PETSc users list" Sent: Thursday, November 4, 2010 4:25:58 PM Subject: Re: [petsc-users] kronecker products On Thu, Nov 4, 2010 at 10:18, Benjamin Sanderse wrote: > I am not sure if I understand you correctly. During initialization, several > matrices are formed by kronecker products, and the resulting matrices have > the size of the number of unknowns; i.e. if n is the number of unknowns then > the matrices are n x n, although they typically have only a*n nonzero > elements, with a around 10-20. > I'm curious about the structure of these kronecker products. For example, you may have K = A \otimes B Is either one of A or B dense? Are they both large and distributed, or is one much smaller than the other? If they are both distributed, what distribution do you want the vector to have? A different question is, what problem are you solving? E.g. if this is a Galerkin method for stochastic PDE, it gives us some idea about the structure, and the natural follow-up question is what is the size of the (reduced how?) stochastic space, and do the stochastic basis functions have local or global support. Jed From jed at 59A2.org Thu Nov 4 17:23:11 2010 From: jed at 59A2.org (Jed Brown) Date: Thu, 4 Nov 2010 17:23:11 -0500 Subject: [petsc-users] kronecker products In-Reply-To: <220090183.145998.1288885140540.JavaMail.root@zembox02.zaas.igi.nl> References: <220090183.145998.1288885140540.JavaMail.root@zembox02.zaas.igi.nl> Message-ID: On Thu, Nov 4, 2010 at 10:39, Benjamin Sanderse wrote: > I am using a finite volume method on structured grids and things are really > simple. > To give you an idea, here is the matrix that calculates du/dx, given a > vector with u. For simplicity I use the Matlab way to construct such a > thing: > > diagonal = (1/deltax)*ones(Nx,1); > Dux_1D = spdiags([-diagonal diagonal],[0 1],Nx-1,Nx); % > differencing matrix in 1D > Dux_2D = kron(speye(Ny),Dux_1D)); > Dux_3D = kron(speye(Nz),Dux_2D)); > Just form the matrix, this isn't a good way to build it in parallel. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From sharatnreddy at gmail.com Thu Nov 4 23:22:18 2010 From: sharatnreddy at gmail.com (Sharat N) Date: Thu, 4 Nov 2010 23:22:18 -0500 Subject: [petsc-users] Petsc with Intel Fortran Message-ID: Hi, I configured the petsc system with intel fortran compiler. This time when I did 'make' on any fortran example. It compiled but gave me 1 warning and 1 error. 1) *warning: feupdateenv is not implemented and will always fail* I found out that This warning is due to a mismatch of library types and can be ignored. The warning will not appear if -shared-intel is used. 2) *symbol lookup error: /opt/intel/Compiler/11.1/072/lib/intel64/libifcore.so.5: undefined symbol: __intel_sse2_strlen* * * *This error is not executing my program does anyone know what is the reasons for this error and how can we solve it?* -- Sharat -------------- next part -------------- An HTML attachment was scrubbed... URL: From wujinshan at yahoo.com Fri Nov 5 00:41:22 2010 From: wujinshan at yahoo.com (jinshan wu) Date: Thu, 4 Nov 2010 22:41:22 -0700 (PDT) Subject: [petsc-users] Initial guess for the GMRES solver in PETSC Message-ID: <953125.67135.qm@web112619.mail.gq1.yahoo.com> Hi all, I am using the GMRES linear solver in PETSC. I have a very good initial guess of the true solution. Can I somehow supply it to the solver? It should help the solver speed up the convergence if I can do that. Thanks. Jinshan From aron.ahmadia at kaust.edu.sa Fri Nov 5 01:11:25 2010 From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia) Date: Fri, 5 Nov 2010 09:11:25 +0300 Subject: [petsc-users] Initial guess for the GMRES solver in PETSC In-Reply-To: <953125.67135.qm@web112619.mail.gq1.yahoo.com> References: <953125.67135.qm@web112619.mail.gq1.yahoo.com> Message-ID: Hi Jinshan, You can pass your initial guess in to the KSP object as X, then at the command line set the flag: -ksp_initial_guess_nonzero Happy Computing, Aron On Fri, Nov 5, 2010 at 8:41 AM, jinshan wu wrote: > Hi all, > > I am using the GMRES linear solver in PETSC. I have a very good initial guess of > the true solution. Can I somehow supply it to the solver? It should help the > solver speed up the convergence if I can do that. Thanks. > > Jinshan > > > > From aron.ahmadia at kaust.edu.sa Fri Nov 5 01:18:17 2010 From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia) Date: Fri, 5 Nov 2010 09:18:17 +0300 Subject: [petsc-users] Petsc with Intel Fortran In-Reply-To: References: Message-ID: This is quite odd. The symbol __intel_sse2_strlen should be provided by libirc.a (which should be brought in by icc or icpc, the Intel C/C++ compilers). I am surprised that configure passed with this error. What C compiler are you supplying to PETSc configure? I suggest you send the complete configure.log and the results of the make to petsc-maint at mcs.anl.gov A On Fri, Nov 5, 2010 at 7:22 AM, Sharat N wrote: > Hi, > I configured the petsc system with intel fortran compiler. This time when I > did 'make' on any fortran example. It compiled but gave me 1 warning and 1 > error. > 1)??warning: feupdateenv is not implemented and will always fail > I?found out that?This warning is due to a mismatch of library types and can > be ignored. The warning will not appear if?-shared-intel?is used. > > 2)?symbol lookup error: > /opt/intel/Compiler/11.1/072/lib/intel64/libifcore.so.5: undefined symbol: > __intel_sse2_strlen > This error is not executing my program does anyone know what is the reasons > for this error and how can we solve it? > -- > Sharat From dave.mayhem23 at gmail.com Fri Nov 5 02:54:15 2010 From: dave.mayhem23 at gmail.com (Dave May) Date: Fri, 5 Nov 2010 08:54:15 +0100 Subject: [petsc-users] Initial guess for the GMRES solver in PETSC In-Reply-To: References: <953125.67135.qm@web112619.mail.gq1.yahoo.com> Message-ID: Or you can specify this directly in your code. See http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/KSP/KSPSetInitialGuessNonzero.html Cheers, Dave On 5 November 2010 07:11, Aron Ahmadia wrote: > Hi Jinshan, > > You can pass your initial guess in to the KSP object as X, then at the > command line set the flag: > > -ksp_initial_guess_nonzero > > Happy Computing, > Aron > > On Fri, Nov 5, 2010 at 8:41 AM, jinshan wu wrote: >> Hi all, >> >> I am using the GMRES linear solver in PETSC. I have a very good initial guess of >> the true solution. Can I somehow supply it to the solver? It should help the >> solver speed up the convergence if I can do that. Thanks. >> >> Jinshan >> >> >> >> > From wujinshan at yahoo.com Fri Nov 5 14:30:20 2010 From: wujinshan at yahoo.com (jinshan wu) Date: Fri, 5 Nov 2010 12:30:20 -0700 (PDT) Subject: [petsc-users] Initial guess for the GMRES solver in PETSC In-Reply-To: References: Message-ID: <11727.40309.qm@web112617.mail.gq1.yahoo.com> Aron and Dave, thanks both of you a lot. Jinshan Message: 6 Date: Fri, 5 Nov 2010 08:54:15 +0100 From: Dave May Subject: Re: [petsc-users] Initial guess for the GMRES solver in PETSC To: PETSc users list Message-ID: Content-Type: text/plain; charset=ISO-8859-1 Or you can specify this directly in your code. See http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/KSP/KSPSetInitialGuessNonzero.html Cheers, Dave On 5 November 2010 07:11, Aron Ahmadia wrote: > Hi Jinshan, > > You can pass your initial guess in to the KSP object as X, then at the > command line set the flag: > > -ksp_initial_guess_nonzero > > Happy Computing, > Aron > > On Fri, Nov 5, 2010 at 8:41 AM, jinshan wu wrote: >> Hi all, >> >> I am using the GMRES linear solver in PETSC. I have a very good initial guess >>of >> the true solution. Can I somehow supply it to the solver? It should help the >> solver speed up the convergence if I can do that. Thanks. >> >> Jinshan >> >> >> >> > ------------------------------ _______________________________________________ petsc-users mailing list petsc-users at mcs.anl.gov https://lists.mcs.anl.gov/mailman/listinfo/petsc-users End of petsc-users Digest, Vol 23, Issue 4 ****************************************** From lizs at mail.uc.edu Fri Nov 5 16:22:28 2010 From: lizs at mail.uc.edu (Li, Zhisong (lizs)) Date: Fri, 5 Nov 2010 21:22:28 +0000 Subject: [petsc-users] No warning or error message of segmentation violation Message-ID: <88D7E3BB7E1960428303E760100374511AA9F6FE@BL2PRD0103MB052.prod.exchangelabs.com> Hi, Petsc Team, I use DA object to create structured data array based on global vectors. Recently, I checked my code and found an array with index out of range. But the code can run without any warning or segmentation violation message. At first I thought this might be a cache problem. But later I ran a simpler test code on another machine and it gave no error either. I did not use any periodic boundary setting. This never took place in the past from my experience, as Petsc could always detect any segmentation violation. So what's wrong with it this time? The following is the main part of my sequential test code: DACreate3d(PETSC_COMM_WORLD,DA_NONPERIODIC,DA_STENCIL_BOX,m,n,q,PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,PETSC_NULL,PETSC_NULL,PETSC_NULL,&da); DACreateGlobalVector(da, &elev); VecSet(elev, 3.14159265); DAVecGetArray(da, elev, &ele); PetscPrintf(PETSC_COMM_WORLD, "ele[3][3][%D] is %4.3e\n", m+120, ele[3][3][m+120]); DAVecRestoreArray(da, elev, &ele); VecDestroy(elev); DADestroy(daw); Thank you very much. Zhisong Li -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Nov 5 16:28:06 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 5 Nov 2010 16:28:06 -0500 Subject: [petsc-users] No warning or error message of segmentation violation In-Reply-To: <88D7E3BB7E1960428303E760100374511AA9F6FE@BL2PRD0103MB052.prod.exchangelabs.com> References: <88D7E3BB7E1960428303E760100374511AA9F6FE@BL2PRD0103MB052.prod.exchangelabs.com> Message-ID: <924F2407-9327-44EC-BC0B-0275FFFA0959@mcs.anl.gov> C has no way of detecting out of array access. If you are lucky the code crashes but if you are unlucky it is just accessing some other memory location (and thus doing garbage computation). Valgrind is very good at detecting bad memory access and should be used often even if the code "seems" to run correctly. We test all our codes with valgrind regularly. http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind Barry On Nov 5, 2010, at 4:22 PM, Li, Zhisong (lizs) wrote: > Hi, Petsc Team, > > I use DA object to create structured data array based on global vectors. Recently, I checked my code and found an array with index out of range. But the code can run without any warning or segmentation violation message. At first I thought this might be a cache problem. But later I ran a simpler test code on another machine and it gave no error either. I did not use any periodic boundary setting. > > This never took place in the past from my experience, as Petsc could always detect any segmentation violation. So what's wrong with it this time? The following is the main part of my sequential test code: > > DACreate3d(PETSC_COMM_WORLD,DA_NONPERIODIC,DA_STENCIL_BOX,m,n,q,PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,PETSC_NULL,PETSC_NULL,PETSC_NULL,&da); > DACreateGlobalVector(da, &elev); > VecSet(elev, 3.14159265); > DAVecGetArray(da, elev, &ele); > PetscPrintf(PETSC_COMM_WORLD, "ele[3][3][%D] is %4.3e\n", m+120, ele[3][3][m+120]); > DAVecRestoreArray(da, elev, &ele); > VecDestroy(elev); > DADestroy(daw); > > > Thank you very much. > > > Zhisong Li From knepley at gmail.com Fri Nov 5 16:28:26 2010 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 5 Nov 2010 16:28:26 -0500 Subject: [petsc-users] No warning or error message of segmentation violation In-Reply-To: <88D7E3BB7E1960428303E760100374511AA9F6FE@BL2PRD0103MB052.prod.exchangelabs.com> References: <88D7E3BB7E1960428303E760100374511AA9F6FE@BL2PRD0103MB052.prod.exchangelabs.com> Message-ID: On Fri, Nov 5, 2010 at 4:22 PM, Li, Zhisong (lizs) wrote: > Hi, Petsc Team, > > I use DA object to create structured data array based on global vectors. > Recently, I checked my code and found an array with index out of range. But > the code can run without any warning or segmentation violation message. At > first I thought this might be a cache problem. But later I ran a simpler > test code on another machine and it gave no error either. I did not use any > periodic boundary setting. > > This never took place in the past from my experience, as Petsc could always > detect any segmentation violation. So what's wrong with it this time? The > following is the main part of my sequential test code: > > > DACreate3d(PETSC_COMM_WORLD,DA_NONPERIODIC,DA_STENCIL_BOX,m,n,q,PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,PETSC_NULL,PETSC_NULL,PETSC_NULL,&da); > > DACreateGlobalVector(da, &elev); > VecSet(elev, 3.14159265); > DAVecGetArray(da, elev, &ele); > PetscPrintf(PETSC_COMM_WORLD, "ele[3][3][%D] is %4.3e\n", m+120, > ele[3][3][m+120]); > DAVecRestoreArray(da, elev, &ele); > VecDestroy(elev); > DADestroy(daw); > PETSc detects write errors that take place directly after an array by putting a sentinel value at the end. If it is written over, CHKMEMQ will detect that. We have no way of checking for overwrites of random memory locations. If you try to write in some other users space, the OS will use a SEGV signal, but you can freely overwrite your own space. Matt > Thank you very much. > > > Zhisong Li > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.langr at gmail.com Mon Nov 8 02:05:04 2010 From: daniel.langr at gmail.com (Daniel Langr) Date: Mon, 08 Nov 2010 09:05:04 +0100 Subject: [petsc-users] Mixed 32-bit and 64-bit indices Message-ID: <4CD7AF30.7040604@gmail.com> Hi all, is there any way to use 32-bit integers for indexing rows/columns and 64-bit integers for indexing nonzeros at the same time in PETSc? I have a matrix with 4,559,168,066 nonzeros but only 448,797 rows/columns. For a general CSR scheme, the memory savings are almost 25% when using 32-bit integers compared to 64-bit for indexing columns, which is really a lot of memory. Thanks, Daniel From daniel.langr at gmail.com Mon Nov 8 02:32:25 2010 From: daniel.langr at gmail.com (Daniel Langr) Date: Mon, 08 Nov 2010 09:32:25 +0100 Subject: [petsc-users] MatCreateMPIAIJWithSplitArrays for symmetric matrix? Message-ID: <4CD7B599.1080201@gmail.com> Hi all, Is there an equivalent of MatCreateMPIAIJWithSplitArrays function for symmetric matrices? Or does this function work with symmetric matrices? I work with symmetric matrices that are extremely expensive to compute and I don't know their structure in advance. So to do preallocation, it is not practical to compute a matrix twice - firstly only to get d_nnz and o_nnz arrays and secondly to set values. Now, I solve this problem by storing a matrix to a file system, which is faster but still far from ideal. The best solution I see is to compute a matrix only once, store it in memory as split CSR arrays and then just call something like MatCreateMPISBAIJWithSplitArrays. Thanks, Daniel From jed at 59A2.org Mon Nov 8 04:19:31 2010 From: jed at 59A2.org (Jed Brown) Date: Mon, 8 Nov 2010 11:19:31 +0100 Subject: [petsc-users] MatCreateMPIAIJWithSplitArrays for symmetric matrix? In-Reply-To: <4CD7B599.1080201@gmail.com> References: <4CD7B599.1080201@gmail.com> Message-ID: On Mon, Nov 8, 2010 at 09:32, Daniel Langr wrote: > Is there an equivalent of MatCreateMPIAIJWithSplitArrays function for > symmetric matrices? Or does this function work with symmetric matrices? > There is MatCreateMPISBAIJWithArrays() in petsc-dev. > I work with symmetric matrices that are extremely expensive to compute and > I don't know their structure in advance. So to do preallocation, it is not > practical to compute a matrix twice - firstly only to get d_nnz and o_nnz > arrays and secondly to set values. Now, I solve this problem by storing a > matrix to a file system, which is faster but still far from ideal. The best > solution I see is to compute a matrix only once, store it in memory as split > CSR arrays and then just call something like > MatCreateMPISBAIJWithSplitArrays. > Where do the matrices come from? How do you know how to allocate to create the arrays? Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Mon Nov 8 04:22:25 2010 From: jed at 59A2.org (Jed Brown) Date: Mon, 8 Nov 2010 11:22:25 +0100 Subject: [petsc-users] Mixed 32-bit and 64-bit indices In-Reply-To: <4CD7AF30.7040604@gmail.com> References: <4CD7AF30.7040604@gmail.com> Message-ID: On Mon, Nov 8, 2010 at 09:05, Daniel Langr wrote: > is there any way to use 32-bit integers for indexing rows/columns and > 64-bit integers for indexing nonzeros at the same time in PETSc? I have a > matrix with 4,559,168,066 nonzeros but only 448,797 rows/columns. For a > general CSR scheme, the memory savings are almost 25% when using 32-bit > integers compared to 64-bit for indexing columns, which is really a lot of > memory. Are you running in serial? As long as the local part of any matrix has less than 2B entries (which would cost more than 24 GiB to store) and the global problem has dimension less than 2B, you should be fine using 32-bit indices. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.langr at gmail.com Mon Nov 8 05:14:36 2010 From: daniel.langr at gmail.com (Daniel Langr) Date: Mon, 08 Nov 2010 12:14:36 +0100 Subject: [petsc-users] MatCreateMPIAIJWithSplitArrays for symmetric matrix? In-Reply-To: References: <4CD7B599.1080201@gmail.com> Message-ID: <4CD7DB9C.6070602@gmail.com> Jed, MatCreateMPISBAIJWithArrays() copy the arrays, so it won't help me. I need all available memory for a matrix, so I cannot have it in memory twice. Matrices emerge in new methods in nuclear structure computations (symmetry-adapted no-core shell models). I don't know how to allocate arrays exactly. But, I can generate a matrix into CSR format and then probably reorganize it (split into diagonal and off-diagonal entries). Daniel Dne 8.11.2010 11:19, Jed Brown napsal(a): > On Mon, Nov 8, 2010 at 09:32, Daniel Langr > wrote: > > Is there an equivalent of MatCreateMPIAIJWithSplitArrays function > for symmetric matrices? Or does this function work with symmetric > matrices? > > > There is MatCreateMPISBAIJWithArrays() in petsc-dev. > > I work with symmetric matrices that are extremely expensive to > compute and I don't know their structure in advance. So to do > preallocation, it is not practical to compute a matrix twice - > firstly only to get d_nnz and o_nnz arrays and secondly to set > values. Now, I solve this problem by storing a matrix to a file > system, which is faster but still far from ideal. The best solution > I see is to compute a matrix only once, store it in memory as split > CSR arrays and then just call something like > MatCreateMPISBAIJWithSplitArrays. > > > Where do the matrices come from? How do you know how to allocate to > create the arrays? > > Jed From daniel.langr at gmail.com Mon Nov 8 05:17:47 2010 From: daniel.langr at gmail.com (Daniel Langr) Date: Mon, 08 Nov 2010 12:17:47 +0100 Subject: [petsc-users] Mixed 32-bit and 64-bit indices In-Reply-To: References: <4CD7AF30.7040604@gmail.com> Message-ID: <4CD7DC5B.1070703@gmail.com> In parallel (12 processes minimum), so it should work :) Thanks a lot, Daniel Dne 8.11.2010 11:22, Jed Brown napsal(a): > On Mon, Nov 8, 2010 at 09:05, Daniel Langr > wrote: > > is there any way to use 32-bit integers for indexing rows/columns > and 64-bit integers for indexing nonzeros at the same time in PETSc? > I have a matrix with 4,559,168,066 nonzeros but only 448,797 > rows/columns. For a general CSR scheme, the memory savings are > almost 25% when using 32-bit integers compared to 64-bit for > indexing columns, which is really a lot of memory. > > > Are you running in serial? As long as the local part of any matrix has > less than 2B entries (which would cost more than 24 GiB to store) and > the global problem has dimension less than 2B, you should be fine using > 32-bit indices. > > Jed From jed at 59A2.org Mon Nov 8 05:24:02 2010 From: jed at 59A2.org (Jed Brown) Date: Mon, 8 Nov 2010 12:24:02 +0100 Subject: [petsc-users] MatCreateMPIAIJWithSplitArrays for symmetric matrix? In-Reply-To: <4CD7DB9C.6070602@gmail.com> References: <4CD7B599.1080201@gmail.com> <4CD7DB9C.6070602@gmail.com> Message-ID: On Mon, Nov 8, 2010 at 12:14, Daniel Langr wrote: > MatCreateMPISBAIJWithArrays() copy the arrays, so it won't help me. I need > all available memory for a matrix, so I cannot have it in memory twice. > What preconditioners do you use? Block Jacobi with incomplete factorization, for example, uses the same amount of memory as the matrix, presumably you could free those arrays before factoring. This doesn't help if you use SOR. Matrices emerge in new methods in nuclear structure computations > (symmetry-adapted no-core shell models). I don't know how to allocate arrays > exactly. > Where does the sparsity of the matrix come from? > But, I can generate a matrix into CSR format and then probably reorganize > it (split into diagonal and off-diagonal entries). > It's not clear how you could generate it in CSR format to begin with, considering that if you underallocate, you will be forced to reallocate and copy. It would be possible to have a MatCreateMPISBAIJWithSplitArrays, but it would be very peculiar to be able to create the arrays without being able to preallocate correctly. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.langr at gmail.com Mon Nov 8 05:36:39 2010 From: daniel.langr at gmail.com (Daniel Langr) Date: Mon, 08 Nov 2010 12:36:39 +0100 Subject: [petsc-users] MatCreateMPIAIJWithSplitArrays for symmetric matrix? In-Reply-To: References: <4CD7B599.1080201@gmail.com> <4CD7DB9C.6070602@gmail.com> Message-ID: <4CD7E0C7.7020006@gmail.com> Dne 8.11.2010 12:24, Jed Brown napsal(a): > On Mon, Nov 8, 2010 at 12:14, Daniel Langr > wrote: > > MatCreateMPISBAIJWithArrays() copy the arrays, so it won't help me. > I need all available memory for a matrix, so I cannot have it in > memory twice. > > > What preconditioners do you use? Block Jacobi with incomplete > factorization, for example, uses the same amount of memory as the > matrix, presumably you could free those arrays before factoring. This > doesn't help if you use SOR. I am looking for several eigenvalues and eigenvectors with SLEPc. > Matrices emerge in new methods in nuclear structure computations > (symmetry-adapted no-core shell models). I don't know how to > allocate arrays exactly. > > > Where does the sparsity of the matrix come from? Lie algebra, Slater determinants, SU(3) groups, new bases... But I don't understand all these things much, I just solve their (physicist's) eigenproblems. > But, I can generate a matrix into CSR format and then probably > reorganize it (split into diagonal and off-diagonal entries). > > > It's not clear how you could generate it in CSR format to begin with, > considering that if you underallocate, you will be forced to reallocate > and copy. It would be possible to have a > MatCreateMPISBAIJWithSplitArrays, but it would be very peculiar to be > able to create the arrays without being able to preallocate correctly. I don't know the exact matrix structure, but I need some estimate about the number of nonzeros, since the matrix needs to fit into a memory. And I can simply use 85-90% of available memory for arrays, and just left some space for vectors (but vectors are very small - matrices are not much sparse). If I underestimate arrays' size, I will abort a computation, cause I don't have any more memory. Daniel > Jed From jed at 59A2.org Mon Nov 8 05:45:50 2010 From: jed at 59A2.org (Jed Brown) Date: Mon, 8 Nov 2010 12:45:50 +0100 Subject: [petsc-users] MatCreateMPIAIJWithSplitArrays for symmetric matrix? In-Reply-To: <4CD7E0C7.7020006@gmail.com> References: <4CD7B599.1080201@gmail.com> <4CD7DB9C.6070602@gmail.com> <4CD7E0C7.7020006@gmail.com> Message-ID: On Mon, Nov 8, 2010 at 12:36, Daniel Langr wrote: > I am looking for several eigenvalues and eigenvectors with SLEPc. > Presumably using methods that don't need a solve? Where does the sparsity of the matrix come from? >> > > Lie algebra, Slater determinants, SU(3) groups, new bases... But I don't > understand all these things much, I just solve their (physicist's) > eigenproblems. > This more likely explains what the entries are, but not what the sparsity is. Sparse problems usually have some spatial locality (and/or locality in frequency or ensemble space). It is usually much easier to find a reasonable upper bound for the sparsity than to figure out exactly what the entries are. But I understand that this may be hard, such as if the support of the basis functions grew without bound due to some nonlinearity in the system. > I don't know the exact matrix structure, but I need some estimate about the > number of nonzeros, since the matrix needs to fit into a memory. And I can > simply use 85-90% of available memory for arrays, and just left some space > for vectors (but vectors are very small - matrices are not much sparse). If > I underestimate arrays' size, I will abort a computation, cause I don't have > any more memory. > Why not use MatMPISBAIJSetPreallocation and MatSetOption MAT_NEW_NONZERO_ALLOCATION_ERR to abort if you have underestimated? How would you do better than this yourself? Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.langr at gmail.com Mon Nov 8 06:07:50 2010 From: daniel.langr at gmail.com (Daniel Langr) Date: Mon, 08 Nov 2010 13:07:50 +0100 Subject: [petsc-users] MatCreateMPIAIJWithSplitArrays for symmetric matrix? In-Reply-To: References: <4CD7B599.1080201@gmail.com> <4CD7DB9C.6070602@gmail.com> <4CD7E0C7.7020006@gmail.com> Message-ID: <4CD7E816.10108@gmail.com> Dne 8.11.2010 12:45, Jed Brown napsal(a): > On Mon, Nov 8, 2010 at 12:36, Daniel Langr > wrote: > > I am looking for several eigenvalues and eigenvectors with SLEPc. > > > Presumably using methods that don't need a solve? I don't use a linear solver, I use Lanczos or other iterative algorithm based on sparse matrix-vector multiplication. > Where does the sparsity of the matrix come from? > > > Lie algebra, Slater determinants, SU(3) groups, new bases... But I > don't understand all these things much, I just solve their > (physicist's) eigenproblems. > > > This more likely explains what the entries are, but not what the > sparsity is. Sparse problems usually have some spatial locality (and/or > locality in frequency or ensemble space). It is usually much easier to > find a reasonable upper bound for the sparsity than to figure out > exactly what the entries are. But I understand that this may be hard, > such as if the support of the basis functions grew without bound due to > some nonlinearity in the system. > > I don't know the exact matrix structure, but I need some estimate > about the number of nonzeros, since the matrix needs to fit into a > memory. And I can simply use 85-90% of available memory for arrays, > and just left some space for vectors (but vectors are very small - > matrices are not much sparse). If I underestimate arrays' size, I > will abort a computation, cause I don't have any more memory. > > > Why not use MatMPISBAIJSetPreallocation and MatSetOption > MAT_NEW_NONZERO_ALLOCATION_ERR to abort if you have underestimated? How > would you do better than this yourself? Ok, that's a good idea and I will try it. I can estimate total number of nonzeros (limited by the amount of memory), but I can hardly estimate number of nonzeros per particular rows (d_nnz, o_nnz). So I need to use d_nz and o_nz, right? And for symmetric matrix, what do d_nz and o_nz mean? Is it number of nonzeros for whole matrix row, or just upper triangular part? (I don't see this in manual page.) (That was my point, through MatMPISBAIJSetPreallocation I need to estimate number of nonzeros per row. Not just number of nonzeros of whole local part. If I construct CSR arrays directly, I don't need this row-estimates.) Daniel > Jed From jed at 59A2.org Mon Nov 8 06:31:58 2010 From: jed at 59A2.org (Jed Brown) Date: Mon, 8 Nov 2010 13:31:58 +0100 Subject: [petsc-users] MatCreateMPIAIJWithSplitArrays for symmetric matrix? In-Reply-To: <4CD7E816.10108@gmail.com> References: <4CD7B599.1080201@gmail.com> <4CD7DB9C.6070602@gmail.com> <4CD7E0C7.7020006@gmail.com> <4CD7E816.10108@gmail.com> Message-ID: On Mon, Nov 8, 2010 at 13:07, Daniel Langr wrote: > Ok, that's a good idea and I will try it. I can estimate total number of > nonzeros (limited by the amount of memory), but I can hardly estimate number > of nonzeros per particular rows (d_nnz, o_nnz). So I need to use d_nz and > o_nz, right? And for symmetric matrix, what do d_nz and o_nz mean? Is it > number of nonzeros for whole matrix row, or just upper triangular part? (I > don't see this in manual page.) > It is only for the upper triangular part. If you have no way to guess the number per row, then just set {d,o}_nz, but note that this is pessimistic because the last rows of the matrix necessarily have fewer nonzeros in the upper triangular part. > (That was my point, through MatMPISBAIJSetPreallocation I need to estimate > number of nonzeros per row. Not just number of nonzeros of whole local part. > If I construct CSR arrays directly, I don't need this row-estimates.) > Here's a little secret: if you assemble the matrix row-by-row, in-order, then you need only estimate the average number of nonzeros per row. The implementation is not stupid, so early rows can blow over their initial estimates as long as later rows make up for it by being under the estimate. For example, with a 5-point Laplacian, you can MatSeqSBAIJSetPreallocation(A,1,3,PETSC_NULL) and you will see something like Matrix Object: type: seqsbaij rows=400, cols=400 total: nonzeros=1160, allocated nonzeros=1200 total number of mallocs used during MatSetValues calls =0 block size is 1 So no mallocs even though the early rows have more than 3 nonzeros in the upper triangular part. If you assemble out-of-order, or in a scattered fashion (such as for a finite element method) then this will be costly. Does this help? Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.langr at gmail.com Mon Nov 8 06:47:55 2010 From: daniel.langr at gmail.com (Daniel Langr) Date: Mon, 08 Nov 2010 13:47:55 +0100 Subject: [petsc-users] MatCreateMPIAIJWithSplitArrays for symmetric matrix? In-Reply-To: References: <4CD7B599.1080201@gmail.com> <4CD7DB9C.6070602@gmail.com> <4CD7E0C7.7020006@gmail.com> <4CD7E816.10108@gmail.com> Message-ID: <4CD7F17B.60002@gmail.com> Dne 8.11.2010 13:31, Jed Brown napsal(a): > On Mon, Nov 8, 2010 at 13:07, Daniel Langr > wrote: > > Ok, that's a good idea and I will try it. I can estimate total > number of nonzeros (limited by the amount of memory), but I can > hardly estimate number of nonzeros per particular rows (d_nnz, > o_nnz). So I need to use d_nz and o_nz, right? And for symmetric > matrix, what do d_nz and o_nz mean? Is it number of nonzeros for > whole matrix row, or just upper triangular part? (I don't see this > in manual page.) > > > It is only for the upper triangular part. If you have no way to guess > the number per row, then just set {d,o}_nz, but note that this is > pessimistic because the last rows of the matrix necessarily have fewer > nonzeros in the upper triangular part. > > (That was my point, through MatMPISBAIJSetPreallocation I need to > estimate number of nonzeros per row. Not just number of nonzeros of > whole local part. If I construct CSR arrays directly, I don't need > this row-estimates.) > > > Here's a little secret: if you assemble the matrix row-by-row, in-order, > then you need only estimate the average number of nonzeros per row. The > implementation is not stupid, so early rows can blow over their initial > estimates as long as later rows make up for it by being under the > estimate. For example, with a 5-point Laplacian, you can > MatSeqSBAIJSetPreallocation(A,1,3,PETSC_NULL) and you will see something > like > > Matrix Object: > type: seqsbaij > rows=400, cols=400 > total: nonzeros=1160, allocated nonzeros=1200 > total number of mallocs used during MatSetValues calls =0 > block size is 1 > > So no mallocs even though the early rows have more than 3 nonzeros in > the upper triangular part. If you assemble out-of-order, or in a > scattered fashion (such as for a finite element method) then this will > be costly. Does this help? Great, helps much, that is something I was hoping for :). As for now, we can generate matrix in-order. In the future it seems we will need to generate matrix in "transposed" order, e.g. row-by-row order for the lower triangular part. But then I suppose we will need to use MatMultTranspose() or MatCreateTranspose() if the performance won't suffer much (I have not tried it yet, so I have no idea about it for now.) Thanks a lot Daniel > Jed From jed at 59A2.org Mon Nov 8 07:01:47 2010 From: jed at 59A2.org (Jed Brown) Date: Mon, 8 Nov 2010 14:01:47 +0100 Subject: [petsc-users] MatCreateMPIAIJWithSplitArrays for symmetric matrix? In-Reply-To: <4CD7F17B.60002@gmail.com> References: <4CD7B599.1080201@gmail.com> <4CD7DB9C.6070602@gmail.com> <4CD7E0C7.7020006@gmail.com> <4CD7E816.10108@gmail.com> <4CD7F17B.60002@gmail.com> Message-ID: On Mon, Nov 8, 2010 at 13:47, Daniel Langr wrote: > Great, helps much, that is something I was hoping for :). As for now, we > can generate matrix in-order. In the future it seems we will need to > generate matrix in "transposed" order, e.g. row-by-row order for the lower > triangular part. But then I suppose we will need to use MatMultTranspose() > or MatCreateTranspose() if the performance won't suffer much (I have not > tried it yet, so I have no idea about it for now.) Well, the distinguishing property of symmetric matrices is that they are equal to their transpose. :-) For non-symmetric matrices, MatMultTranspose (perhaps accessed as MatMult via MatCreateTranspose) is about the same speed as MatMult. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.langr at gmail.com Mon Nov 8 07:16:15 2010 From: daniel.langr at gmail.com (Daniel Langr) Date: Mon, 08 Nov 2010 14:16:15 +0100 Subject: [petsc-users] MatCreateMPIAIJWithSplitArrays for symmetric matrix? In-Reply-To: References: <4CD7B599.1080201@gmail.com> <4CD7DB9C.6070602@gmail.com> <4CD7E0C7.7020006@gmail.com> <4CD7E816.10108@gmail.com> <4CD7F17B.60002@gmail.com> Message-ID: <4CD7F81F.6010109@gmail.com> Dne 8.11.2010 14:01, Jed Brown napsal(a): > On Mon, Nov 8, 2010 at 13:47, Daniel Langr > wrote: > > Great, helps much, that is something I was hoping for :). As for > now, we can generate matrix in-order. In the future it seems we will > need to generate matrix in "transposed" order, e.g. row-by-row order > for the lower triangular part. But then I suppose we will need to > use MatMultTranspose() or MatCreateTranspose() if the performance > won't suffer much (I have not tried it yet, so I have no idea about > it for now.) > > > Well, the distinguishing property of symmetric matrices is that they are > equal to their transpose. :-) > > For non-symmetric matrices, MatMultTranspose (perhaps accessed as > MatMult via MatCreateTranspose) is about the same speed as MatMult. > > Jed I meant something else but wrote it wrong, shame on me :). I thought that when I use MatCreateTranspose() for a symmetric matrix, I will be able to preallocate and set values for lower part (instead of upper part) in row-by-row order, as I generate them. But I am not sure about it. Daniel From rongliang.chan at gmail.com Mon Nov 8 19:49:22 2010 From: rongliang.chan at gmail.com (Rongliang Chen) Date: Mon, 8 Nov 2010 18:49:22 -0700 Subject: [petsc-users] Can I use MAT_FINAL_ASSEMBLY instead of MAT_FLUSH_ASSEMBLY? Message-ID: Hi everyone, Can I use MAT_FINAL_ASSEMBLY instead of MAT_FLUSH_ASSEMBLY in the function MatAssemblyBegin/End when switching between ADD_VALUES and INSERT_VALUES in MatSetValues()? Thank you! Regards, Rongliang -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Nov 8 19:57:16 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 8 Nov 2010 19:57:16 -0600 Subject: [petsc-users] Can I use MAT_FINAL_ASSEMBLY instead of MAT_FLUSH_ASSEMBLY? In-Reply-To: References: Message-ID: <52AFD8FB-17C1-4B51-97E9-C77112B69F3C@mcs.anl.gov> On Nov 8, 2010, at 7:49 PM, Rongliang Chen wrote: > Hi everyone, > > Can I use MAT_FINAL_ASSEMBLY instead of MAT_FLUSH_ASSEMBLY in the function MatAssemblyBegin/End when switching between ADD_VALUES and INSERT_VALUES in MatSetValues()? Thank you! > > Regards, > > Rongliang Yes, but generally there is no reason to. It is best to use the flush until your final assembly because then it doesn't do any work that needs to be redone or prematurely free memory that may be needed later. Why do you want to do that? Barry From tim.kroeger at cevis.uni-bremen.de Tue Nov 9 05:11:30 2010 From: tim.kroeger at cevis.uni-bremen.de (Tim Kroeger) Date: Tue, 9 Nov 2010 12:11:30 +0100 (CET) Subject: [petsc-users] Index set with zero size Message-ID: Dear all, Is it allowed to work with an index set (IS) with size 0? In particular, using such an index set in VecScatter and MatGetSubMatrix, I am creating the following objects: v : a vector of size n>0, w : a vector of size 0, A : a matrix of size n\times0. Then I do MatMultAdd (A, w, v, v); This actually seems to change the value of v, although I supposed it not to do so. Are such index sets allowed or not? If not, what about index sets having local size 0 on some processors? Best Regards, Tim -- Dr. Tim Kroeger CeVis -- Center of Complex Systems and Visualization University of Bremen tim.kroeger at cevis.uni-bremen.de Universitaetsallee 29 tim.kroeger at mevis.fraunhofer.de D-28359 Bremen Phone +49-421-218-7710 Germany Fax +49-421-218-4236 From jed at 59A2.org Tue Nov 9 05:28:09 2010 From: jed at 59A2.org (Jed Brown) Date: Tue, 9 Nov 2010 12:28:09 +0100 Subject: [petsc-users] Index set with zero size In-Reply-To: References: Message-ID: On Tue, Nov 9, 2010 at 12:11, Tim Kroeger wrote: > Is it allowed to work with an index set (IS) with size 0? > Yes. In particular, using such an index set in VecScatter and MatGetSubMatrix, I > am creating the following objects: > > v : a vector of size n>0, > w : a vector of size 0, > A : a matrix of size n\times0. > > Then I do > > MatMultAdd (A, w, v, v); > > This actually seems to change the value of v, although I supposed it not to > do so. > It sounds like you have a simple test case for this, could you send it? Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.kroeger at cevis.uni-bremen.de Tue Nov 9 05:49:12 2010 From: tim.kroeger at cevis.uni-bremen.de (Tim Kroeger) Date: Tue, 9 Nov 2010 12:49:12 +0100 (CET) Subject: [petsc-users] Index set with zero size In-Reply-To: References: Message-ID: On Tue, 9 Nov 2010, Jed Brown wrote: > On Tue, Nov 9, 2010 at 12:11, Tim Kroeger wrote: > Is it allowed to work with an index set (IS) with size 0? > > > Yes. Good to hear, thank you. > In particular, using such an index set in VecScatter and MatGetSubMatrix, I am creating the following > objects: > > v : a vector of size n>0, > w : a vector of size 0, > A : a matrix of size n\times0. > > Then I do > > MatMultAdd (A, w, v, v); > > This actually seems to change the value of v, although I supposed it not to do so. > > > It sounds like you have a simple test case for this, could you send it? Well, as you know, I'm using PETSc only via libMesh. Creating a standalone PETSc test case is non-trivial for me. But I'll keep to this and will either find out that I did something wrong or produce a test case for you. It might be imporant to know that the original matrix is a ShellMatrix. Best Regards, Tim -- Dr. Tim Kroeger CeVis -- Center of Complex Systems and Visualization University of Bremen tim.kroeger at cevis.uni-bremen.de Universitaetsallee 29 tim.kroeger at mevis.fraunhofer.de D-28359 Bremen Phone +49-421-218-7710 Germany Fax +49-421-218-4236 From jed at 59A2.org Tue Nov 9 05:53:17 2010 From: jed at 59A2.org (Jed Brown) Date: Tue, 9 Nov 2010 12:53:17 +0100 Subject: [petsc-users] Index set with zero size In-Reply-To: References: Message-ID: On Tue, Nov 9, 2010 at 12:49, Tim Kroeger wrote: > Well, as you know, I'm using PETSc only via libMesh. Creating a standalone > PETSc test case is non-trivial for me. But I'll keep to this and will > either find out that I did something wrong or produce a test case for you. > > It might be imporant to know that the original matrix is a ShellMatrix. > Yes, this is important. Check the implementation of MatMultAdd for the Shell. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.kroeger at cevis.uni-bremen.de Tue Nov 9 05:56:26 2010 From: tim.kroeger at cevis.uni-bremen.de (Tim Kroeger) Date: Tue, 9 Nov 2010 12:56:26 +0100 (CET) Subject: [petsc-users] Index set with zero size In-Reply-To: References: Message-ID: On Tue, 9 Nov 2010, Jed Brown wrote: > On Tue, Nov 9, 2010 at 12:49, Tim Kroeger wrote: > > It might be imporant to know that the original matrix is a ShellMatrix. > > > Yes, this is important. ?Check the implementation of MatMultAdd for the Shell. Thank you, I found it now: The shell matrix does not implement MatMultAdd, only MatMult. Of course, it can't work then. Thank you very much! Best Regards, Tim -- Dr. Tim Kroeger CeVis -- Center of Complex Systems and Visualization University of Bremen tim.kroeger at cevis.uni-bremen.de Universitaetsallee 29 tim.kroeger at mevis.fraunhofer.de D-28359 Bremen Phone +49-421-218-7710 Germany Fax +49-421-218-4236 From jed at 59A2.org Tue Nov 9 06:01:21 2010 From: jed at 59A2.org (Jed Brown) Date: Tue, 9 Nov 2010 13:01:21 +0100 Subject: [petsc-users] Index set with zero size In-Reply-To: References: Message-ID: On Tue, Nov 9, 2010 at 12:56, Tim Kroeger wrote: > Thank you, I found it now: The shell matrix does not implement MatMultAdd, > only MatMult. Of course, it can't work then. Thank you very much! Okay, but then it will error instead of silently modifying the vector. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.kroeger at cevis.uni-bremen.de Tue Nov 9 07:41:41 2010 From: tim.kroeger at cevis.uni-bremen.de (Tim Kroeger) Date: Tue, 9 Nov 2010 14:41:41 +0100 (CET) Subject: [petsc-users] Index set with zero size In-Reply-To: References: Message-ID: On Tue, 9 Nov 2010, Jed Brown wrote: > On Tue, Nov 9, 2010 at 12:56, Tim Kroeger wrote: > Thank you, I found it now: The shell matrix does not implement MatMultAdd, only MatMult. ?Of course, it > can't work then. ?Thank you very much! > > > Okay, but then it will error instead of silently modifying the vector. It didn't. And I did not forget CHKERRABORT() after the call to MatMultAdd(). Also, implementing MatMultAdd() for the shell did not resolve the problem. I had a look into src/mat/impls/submat/submat.c (of petsc-3.1-p4) and found that MatMultAdd_SubMatrix() internally calls MatMult() (line 155), not MatMultAdd(). Later in that method, some VecAYPX() is called (line 159), which might do the required addition. Can you please confirm whether (a) this is correct but explains why leaving MatMultAdd() of the shell undefined does not error; or (b) everything remains as you said before; or (c) this is a bug in that PETSc version? Best Regards, Tim -- Dr. Tim Kroeger CeVis -- Center of Complex Systems and Visualization University of Bremen tim.kroeger at cevis.uni-bremen.de Universitaetsallee 29 tim.kroeger at mevis.fraunhofer.de D-28359 Bremen Phone +49-421-218-7710 Germany Fax +49-421-218-4236 From jed at 59A2.org Tue Nov 9 07:50:45 2010 From: jed at 59A2.org (Jed Brown) Date: Tue, 9 Nov 2010 14:50:45 +0100 Subject: [petsc-users] Index set with zero size In-Reply-To: References: Message-ID: On Tue, Nov 9, 2010 at 14:41, Tim Kroeger wrote: > I had a look into src/mat/impls/submat/submat.c (of petsc-3.1-p4) and found > that MatMultAdd_SubMatrix() internally calls MatMult() (line 155), not > MatMultAdd(). Later in that method, some VecAYPX() is called (line 159), > which might do the required addition. Can you please confirm whether Ah, I missed that you called MatGetSubMatrix on a MatShell in which you hadn't implemented MatGetSubMatrix, therefore you had a MatSubMatrix. Looking at MatMultAdd_SubMatrix, I see that the code is wrong if v2==v3. We don't have "weighted scatters", so I think it needs to get another work vector for a correct answer with "left scaling" and v2==v3. I'll fix this in 3.1 and -dev, but I might not get to it today. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.kroeger at cevis.uni-bremen.de Tue Nov 9 08:43:57 2010 From: tim.kroeger at cevis.uni-bremen.de (Tim Kroeger) Date: Tue, 9 Nov 2010 15:43:57 +0100 (CET) Subject: [petsc-users] Index set with zero size In-Reply-To: References: Message-ID: On Tue, 9 Nov 2010, Jed Brown wrote: > On Tue, Nov 9, 2010 at 14:41, Tim Kroeger wrote: > I had a look into src/mat/impls/submat/submat.c (of petsc-3.1-p4) and found that MatMultAdd_SubMatrix() > internally calls MatMult() (line 155), not MatMultAdd(). ?Later in that method, some VecAYPX() is called > (line 159), which might do the required addition. ?Can you please confirm whether > > > Ah, I missed that you called MatGetSubMatrix on a MatShell in which you hadn't implemented MatGetSubMatrix, > therefore you had a MatSubMatrix. Sorry for not pointing this our more clearly. > Looking at MatMultAdd_SubMatrix, I see that the code is wrong if v2==v3. ?We don't have "weighted scatters", so I > think it needs to get another work vector for a correct answer with "left scaling" and v2==v3. ?I'll fix this in 3.1 > and -dev, but I might not get to it today. Okay, I implemented a workaround on user side (using a temporary vector and MatMult()). This seems to resolve the problem now. I'll leave it like this in the libMesh part, so that libMesh remains compatible with PETSc-3.1-p4. Nevertheless, if you fix this within a few days and launch PETSc-3.1-p5 with this fix (or just send me a patch), let me know, I can then test it. Best Regards, Tim -- Dr. Tim Kroeger CeVis -- Center of Complex Systems and Visualization University of Bremen tim.kroeger at cevis.uni-bremen.de Universitaetsallee 29 tim.kroeger at mevis.fraunhofer.de D-28359 Bremen Phone +49-421-218-7710 Germany Fax +49-421-218-4236 From rongliang.chan at gmail.com Tue Nov 9 11:27:32 2010 From: rongliang.chan at gmail.com (Rongliang Chen) Date: Tue, 9 Nov 2010 10:27:32 -0700 Subject: [petsc-users] Can I use MAT_FINAL_ASSEMBLY instead of MAT_FLUSH_ASSEMBLY? Message-ID: Message: 2 > Date: Mon, 8 Nov 2010 19:57:16 -0600 > From: Barry Smith > Subject: Re: [petsc-users] Can I use MAT_FINAL_ASSEMBLY instead of > MAT_FLUSH_ASSEMBLY? > To: PETSc users list > Message-ID: <52AFD8FB-17C1-4B51-97E9-C77112B69F3C at mcs.anl.gov> > Content-Type: text/plain; charset=us-ascii > > > On Nov 8, 2010, at 7:49 PM, Rongliang Chen wrote: > > > Hi everyone, > > > > Can I use MAT_FINAL_ASSEMBLY instead of MAT_FLUSH_ASSEMBLY in the > function MatAssemblyBegin/End when switching between ADD_VALUES and > INSERT_VALUES in MatSetValues()? Thank you! > > > > Regards, > > > > Rongliang > > Yes, but generally there is no reason to. It is best to use the flush until > your final assembly because then it doesn't do any work that needs to be > redone or prematurely free memory that may be needed later. > > Why do you want to do that? > > Barry > > > Hi Barry, Thank you for your reply. I find that when I use MAT_FINAL_ASSEMBLY instead of MAT_FLUSH_ASSEMBLY in the function MatAssemblyBegin/End, it will cause problem. I do not know why. The reason that I want to use it like this is that I first set the matrix with ADD_VALUES and then I need to zero some rows of the matrix using MatZeroRowsLocalIS(), at last I still need to set some elements of the matrix with INSERT_VALUES. Between MatSetValues() and MatZeroRowsLocalIS() I need to use MAT_FINAL_ASSEMBLY. The order is: ADD_VALUES--->MatZeroRowsLocalIS()--->INSERT_VALUES. Thanks! Regards, Rongliang -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Nov 9 11:31:55 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 9 Nov 2010 11:31:55 -0600 Subject: [petsc-users] Can I use MAT_FINAL_ASSEMBLY instead of MAT_FLUSH_ASSEMBLY? In-Reply-To: References: Message-ID: On Nov 9, 2010, at 11:27 AM, Rongliang Chen wrote: > Message: 2 > Date: Mon, 8 Nov 2010 19:57:16 -0600 > From: Barry Smith > Subject: Re: [petsc-users] Can I use MAT_FINAL_ASSEMBLY instead of > MAT_FLUSH_ASSEMBLY? > To: PETSc users list > Message-ID: <52AFD8FB-17C1-4B51-97E9-C77112B69F3C at mcs.anl.gov> > Content-Type: text/plain; charset=us-ascii > > > On Nov 8, 2010, at 7:49 PM, Rongliang Chen wrote: > > > Hi everyone, > > > > Can I use MAT_FINAL_ASSEMBLY instead of MAT_FLUSH_ASSEMBLY in the function MatAssemblyBegin/End when switching between ADD_VALUES and INSERT_VALUES in MatSetValues()? Thank you! > > > > Regards, > > > > Rongliang > > Yes, but generally there is no reason to. It is best to use the flush until your final assembly because then it doesn't do any work that needs to be redone or prematurely free memory that may be needed later. > > Why do you want to do that? > > Barry > > > > Hi Barry, > > Thank you for your reply. I find that when I use MAT_FINAL_ASSEMBLY instead of MAT_FLUSH_ASSEMBLY in the function MatAssemblyBegin/End, it will > cause problem. I do not know why. "cause problem" is not very helpful. What does that mean? Send all output to petsc-maint at mcs.anl.gov if it doesn't work. > > The reason that I want to use it like this is that I first set the matrix with ADD_VALUES and then I need to zero some rows of the matrix using MatZeroRowsLocalIS(), at last I still need to set some elements of the matrix with INSERT_VALUES. Between MatSetValues() and MatZeroRowsLocalIS() I need to use MAT_FINAL_ASSEMBLY. > The order is: > ADD_VALUES--->MatZeroRowsLocalIS()--->INSERT_VALUES. You need to use FINAL_ASSEMLY before the zero rows calls. Flush is not enough in that case. Barry > > Thanks! > > Regards, > > Rongliang From rongliang.chan at gmail.com Tue Nov 9 11:56:43 2010 From: rongliang.chan at gmail.com (Rongliang Chen) Date: Tue, 9 Nov 2010 10:56:43 -0700 Subject: [petsc-users] Can I use MAT_FINAL_ASSEMBLY instead of MAT_FLUSH_ASSEMBLY? In-Reply-To: References: Message-ID: Yes, I know. So this is why I need to use MAT_FINAL_ASSEMBLY instead of MAT_FLUSH_ASSEMBLY in the function MatAssemblyBegin/End when switching between ADD_VALUES and INSERT_VALUES in MatSetValues(). But when I use the finally assembled matrix to multiply a vector using MatMult. I find that the output of MatMult is wrong if I use multiprocessors. So I doubt that the problem is come from "MAT_FINAL_ASSEMBLY instead of MAT_FLUSH_ASSEMBLY", but I am not sure. Thanks. Regards, Rongliang On Tue, Nov 9, 2010 at 10:31 AM, Barry Smith wrote: > > On Nov 9, 2010, at 11:27 AM, Rongliang Chen wrote: > > > Message: 2 > > Date: Mon, 8 Nov 2010 19:57:16 -0600 > > From: Barry Smith > > Subject: Re: [petsc-users] Can I use MAT_FINAL_ASSEMBLY instead of > > MAT_FLUSH_ASSEMBLY? > > To: PETSc users list > > Message-ID: <52AFD8FB-17C1-4B51-97E9-C77112B69F3C at mcs.anl.gov> > > Content-Type: text/plain; charset=us-ascii > > > > > > On Nov 8, 2010, at 7:49 PM, Rongliang Chen wrote: > > > > > Hi everyone, > > > > > > Can I use MAT_FINAL_ASSEMBLY instead of MAT_FLUSH_ASSEMBLY in the > function MatAssemblyBegin/End when switching between ADD_VALUES and > INSERT_VALUES in MatSetValues()? Thank you! > > > > > > Regards, > > > > > > Rongliang > > > > Yes, but generally there is no reason to. It is best to use the flush > until your final assembly because then it doesn't do any work that needs to > be redone or prematurely free memory that may be needed later. > > > > Why do you want to do that? > > > > Barry > > > > > > > > Hi Barry, > > > > Thank you for your reply. I find that when I use MAT_FINAL_ASSEMBLY > instead of MAT_FLUSH_ASSEMBLY in the function MatAssemblyBegin/End, it will > > cause problem. I do not know why. > > "cause problem" is not very helpful. What does that mean? Send all > output to petsc-maint at mcs.anl.gov if it doesn't work. > > > > The reason that I want to use it like this is that I first set the matrix > with ADD_VALUES and then I need to zero some rows of the matrix using > MatZeroRowsLocalIS(), at last I still need to set some elements of the > matrix with INSERT_VALUES. Between MatSetValues() and MatZeroRowsLocalIS() I > need to use MAT_FINAL_ASSEMBLY. > > The order is: > > ADD_VALUES--->MatZeroRowsLocalIS()--->INSERT_VALUES. > > You need to use FINAL_ASSEMLY before the zero rows calls. Flush is not > enough in that case. > > Barry > > > > > Thanks! > > > > Regards, > > > > Rongliang > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Nov 9 12:33:16 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 9 Nov 2010 12:33:16 -0600 Subject: [petsc-users] Can I use MAT_FINAL_ASSEMBLY instead of MAT_FLUSH_ASSEMBLY? In-Reply-To: References: Message-ID: On Nov 9, 2010, at 11:56 AM, Rongliang Chen wrote: > Yes, I know. So this is why I need to use MAT_FINAL_ASSEMBLY instead of MAT_FLUSH_ASSEMBLY in the function MatAssemblyBegin/End when switching between ADD_VALUES and INSERT_VALUES in MatSetValues(). > But when I use the finally assembled matrix to multiply a vector using MatMult. I find that the output of MatMult is wrong if I use multiprocessors. So I doubt that the problem is come from "MAT_FINAL_ASSEMBLY instead of MAT_FLUSH_ASSEMBLY", but I am not sure. Thanks. Well the problem is either in your code or in PETSc. Send to petsc-maint at mcs.anl.gov a code that demonstrates the problem and we'll look into the problem. Barry > > Regards, > > Rongliang > > On Tue, Nov 9, 2010 at 10:31 AM, Barry Smith wrote: > > On Nov 9, 2010, at 11:27 AM, Rongliang Chen wrote: > > > Message: 2 > > Date: Mon, 8 Nov 2010 19:57:16 -0600 > > From: Barry Smith > > Subject: Re: [petsc-users] Can I use MAT_FINAL_ASSEMBLY instead of > > MAT_FLUSH_ASSEMBLY? > > To: PETSc users list > > Message-ID: <52AFD8FB-17C1-4B51-97E9-C77112B69F3C at mcs.anl.gov> > > Content-Type: text/plain; charset=us-ascii > > > > > > On Nov 8, 2010, at 7:49 PM, Rongliang Chen wrote: > > > > > Hi everyone, > > > > > > Can I use MAT_FINAL_ASSEMBLY instead of MAT_FLUSH_ASSEMBLY in the function MatAssemblyBegin/End when switching between ADD_VALUES and INSERT_VALUES in MatSetValues()? Thank you! > > > > > > Regards, > > > > > > Rongliang > > > > Yes, but generally there is no reason to. It is best to use the flush until your final assembly because then it doesn't do any work that needs to be redone or prematurely free memory that may be needed later. > > > > Why do you want to do that? > > > > Barry > > > > > > > > Hi Barry, > > > > Thank you for your reply. I find that when I use MAT_FINAL_ASSEMBLY instead of MAT_FLUSH_ASSEMBLY in the function MatAssemblyBegin/End, it will > > cause problem. I do not know why. > > "cause problem" is not very helpful. What does that mean? Send all output to petsc-maint at mcs.anl.gov if it doesn't work. > > > > The reason that I want to use it like this is that I first set the matrix with ADD_VALUES and then I need to zero some rows of the matrix using MatZeroRowsLocalIS(), at last I still need to set some elements of the matrix with INSERT_VALUES. Between MatSetValues() and MatZeroRowsLocalIS() I need to use MAT_FINAL_ASSEMBLY. > > The order is: > > ADD_VALUES--->MatZeroRowsLocalIS()--->INSERT_VALUES. > > You need to use FINAL_ASSEMLY before the zero rows calls. Flush is not enough in that case. > > Barry > > > > > Thanks! > > > > Regards, > > > > Rongliang > > From jed at 59A2.org Tue Nov 9 21:18:33 2010 From: jed at 59A2.org (Jed Brown) Date: Wed, 10 Nov 2010 04:18:33 +0100 Subject: [petsc-users] Index set with zero size In-Reply-To: References: Message-ID: On Tue, Nov 9, 2010 at 15:43, Tim Kroeger wrote: > Okay, I implemented a workaround on user side (using a temporary vector and > MatMult()). This seems to resolve the problem now. I'll leave it like this > in the libMesh part, so that libMesh remains compatible with PETSc-3.1-p4. > Nevertheless, if you fix this within a few days and launch PETSc-3.1-p5 > with this fix (or just send me a patch), let me know, I can then test it. I fixed this, and also a bug with nontrivial MatDiagonalScale after MatGetSubMatrix. You can get it from hg, or in the next patch level (I don't know when that will be). Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.langr at gmail.com Wed Nov 10 11:45:30 2010 From: daniel.langr at gmail.com (Daniel Langr) Date: Wed, 10 Nov 2010 18:45:30 +0100 Subject: [petsc-users] 64-bit integer options Message-ID: <4CDADA3A.6020103@gmail.com> Hi all, I have a problem passing an integer option value greater than 2^32 to my program. I compiled PETSc with --with-64-bit-indices=1, but it seems PetscOptionsGetInt() uses aoti() internally; I would expect atol() or another function named PetscOptionsGetLong(). Thanks, Daniel From bsmith at mcs.anl.gov Wed Nov 10 11:51:29 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 10 Nov 2010 11:51:29 -0600 Subject: [petsc-users] 64-bit integer options In-Reply-To: <4CDADA3A.6020103@gmail.com> References: <4CDADA3A.6020103@gmail.com> Message-ID: <1C86A2FA-B6AB-4105-99D5-C28C6B1AC16C@mcs.anl.gov> On Nov 10, 2010, at 11:45 AM, Daniel Langr wrote: > Hi all, > > I have a problem passing an integer option value greater than 2^32 to my program. I compiled PETSc with --with-64-bit-indices=1, but it seems PetscOptionsGetInt() uses aoti() internally; I would expect atol() or another function named PetscOptionsGetLong(). You are correct, this is our error. We will fix it in petsc 3.1 and petsc-dev and get back with you. Barry > > Thanks, > Daniel From daniel.langr at gmail.com Wed Nov 10 11:52:25 2010 From: daniel.langr at gmail.com (Daniel Langr) Date: Wed, 10 Nov 2010 18:52:25 +0100 Subject: [petsc-users] 64-bit integer options In-Reply-To: <1C86A2FA-B6AB-4105-99D5-C28C6B1AC16C@mcs.anl.gov> References: <4CDADA3A.6020103@gmail.com> <1C86A2FA-B6AB-4105-99D5-C28C6B1AC16C@mcs.anl.gov> Message-ID: <4CDADBD9.2030204@gmail.com> Thanks a lot, Daniel Dne 10.11.2010 18:51, Barry Smith napsal(a): > > On Nov 10, 2010, at 11:45 AM, Daniel Langr wrote: > >> Hi all, >> >> I have a problem passing an integer option value greater than 2^32 to my program. I compiled PETSc with --with-64-bit-indices=1, but it seems PetscOptionsGetInt() uses aoti() internally; I would expect atol() or another function named PetscOptionsGetLong(). > > You are correct, this is our error. We will fix it in petsc 3.1 and petsc-dev and get back with you. > > Barry > >> >> Thanks, >> Daniel From jed at 59A2.org Wed Nov 10 11:53:51 2010 From: jed at 59A2.org (Jed Brown) Date: Wed, 10 Nov 2010 18:53:51 +0100 Subject: [petsc-users] 64-bit integer options In-Reply-To: <1C86A2FA-B6AB-4105-99D5-C28C6B1AC16C@mcs.anl.gov> References: <4CDADA3A.6020103@gmail.com> <1C86A2FA-B6AB-4105-99D5-C28C6B1AC16C@mcs.anl.gov> Message-ID: On Wed, Nov 10, 2010 at 18:51, Barry Smith wrote: > You are correct, this is our error. We will fix it in petsc 3.1 and > petsc-dev and get back with you. I have a patch, will push in a moment. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chun.SUN at 3ds.com Wed Nov 10 13:55:39 2010 From: Chun.SUN at 3ds.com (SUN Chun) Date: Wed, 10 Nov 2010 19:55:39 +0000 Subject: [petsc-users] our of core capability? In-Reply-To: <4CDADA3A.6020103@gmail.com> References: <4CDADA3A.6020103@gmail.com> Message-ID: <3AB51330CF1300428AD80035DAF20EE30161F6@AG-DCC-MBX03.dsone.3ds.com> Hi Petsc Developers, Sorry this might doesn't make any sense but I have to ask ... Does PETSc have the capability to use out-of-core disk space if I set a cap for memory usage? I looked around in the src but I could be missing something. What I can think of is to use MatShell if that's the only option... Thanks a lot, Chun This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged. If you are not one of the named recipients or have received this email in error, (i) you should not read, disclose, or copy it, (ii) please notify sender of your receipt by reply email and delete this email and all attachments, (iii) Dassault Systemes does not accept or assume any liability or responsibility for any use of or reliance on this email.For other languages, go to http://www.3ds.com/terms/email-disclaimer. From jed at 59A2.org Wed Nov 10 14:07:47 2010 From: jed at 59A2.org (Jed Brown) Date: Wed, 10 Nov 2010 21:07:47 +0100 Subject: [petsc-users] our of core capability? In-Reply-To: <3AB51330CF1300428AD80035DAF20EE30161F6@AG-DCC-MBX03.dsone.3ds.com> References: <4CDADA3A.6020103@gmail.com> <3AB51330CF1300428AD80035DAF20EE30161F6@AG-DCC-MBX03.dsone.3ds.com> Message-ID: On Wed, Nov 10, 2010 at 20:55, SUN Chun wrote: > Sorry this might doesn't make any sense but I have to ask ... Does PETSc > have the capability to use out-of-core disk space if I set a cap for memory > usage? I looked around in the src but I could be missing something. > There is no explicit out-of-core capability, be warned that it will unavoidably make things much slower (even if written specially for out-of-core) but you could mmap a big file and have the matrices use that, or you can just create a big swap file and let the OS handle it. Note that kernels can actually do a pretty good job of handling paging, so it's not a clear win for an application to manage it themselves. See e.g. http://queue.acm.org/detail.cfm?id=1814327 > What I can think of is to use MatShell if that's the only option... > You can certainly do this, but if it makes sense to store your matrix in an assembled format, you should probably just assemble it and let the OS handle paging. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Wed Nov 10 14:19:57 2010 From: jed at 59A2.org (Jed Brown) Date: Wed, 10 Nov 2010 21:19:57 +0100 Subject: [petsc-users] our of core capability? In-Reply-To: References: <4CDADA3A.6020103@gmail.com> <3AB51330CF1300428AD80035DAF20EE30161F6@AG-DCC-MBX03.dsone.3ds.com> Message-ID: On Wed, Nov 10, 2010 at 21:07, Jed Brown wrote: > See e.g. http://queue.acm.org/detail.cfm?id=1814327 > I probably should have just cited the varnish project. Note that many of the optimizations (mostly data structure) that are good for out-of-core (like in this article) are already good for cache performance. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.kroeger at cevis.uni-bremen.de Thu Nov 11 02:28:27 2010 From: tim.kroeger at cevis.uni-bremen.de (Tim Kroeger) Date: Thu, 11 Nov 2010 09:28:27 +0100 (CET) Subject: [petsc-users] Index set with zero size In-Reply-To: References: Message-ID: On Wed, 10 Nov 2010, Jed Brown wrote: > On Tue, Nov 9, 2010 at 15:43, Tim Kroeger wrote: > Okay, I implemented a workaround on user side (using a temporary vector and MatMult()). ?This seems to > resolve the problem now. ?I'll leave it like this in the libMesh part, so that libMesh remains > compatible with PETSc-3.1-p4. ?Nevertheless, if you fix this within a few days and launch PETSc-3.1-p5 > with this fix (or just send me a patch), let me know, I can then test it. > > > I fixed this, and also a bug with nontrivial MatDiagonalScale after MatGetSubMatrix. ?You can get it from hg, or in > the next patch level (I don't know when that will be). Yes, this fixes it. Thank you very much again. Best Regards, Tim -- Dr. Tim Kroeger CeVis -- Center of Complex Systems and Visualization University of Bremen tim.kroeger at cevis.uni-bremen.de Universitaetsallee 29 tim.kroeger at mevis.fraunhofer.de D-28359 Bremen Phone +49-421-218-7710 Germany Fax +49-421-218-4236 From tim.kroeger at cevis.uni-bremen.de Thu Nov 11 10:03:36 2010 From: tim.kroeger at cevis.uni-bremen.de (Tim Kroeger) Date: Thu, 11 Nov 2010 17:03:36 +0100 (CET) Subject: [petsc-users] Index set with zero size In-Reply-To: References: Message-ID: On Thu, 11 Nov 2010, Tim Kroeger wrote: > On Wed, 10 Nov 2010, Jed Brown wrote: > >> On Tue, Nov 9, 2010 at 15:43, Tim Kroeger >> wrote: >> Okay, I implemented a workaround on user side (using a temporary >> vector and MatMult()). ?This seems to >> resolve the problem now. ?I'll leave it like this in the libMesh >> part, so that libMesh remains >> compatible with PETSc-3.1-p4. ?Nevertheless, if you fix this within a >> few days and launch PETSc-3.1-p5 >> with this fix (or just send me a patch), let me know, I can then test >> it. >> >> >> I fixed this, and also a bug with nontrivial MatDiagonalScale after >> MatGetSubMatrix. ?You can get it from hg, or in >> the next patch level (I don't know when that will be). > > Yes, this fixes it. Thank you very much again. One more question: Does this bux and fix also apply to a submatrix of an MPIAIJ matrix? That is, can I safely use MatMultAdd() having v2==v3 for such a matrix in 3.1.0-p4 or not? (I'm having trouble with such a matrix now, but the error might be on my side.) Best Regards, Tim -- Dr. Tim Kroeger CeVis -- Center of Complex Systems and Visualization University of Bremen tim.kroeger at cevis.uni-bremen.de Universitaetsallee 29 tim.kroeger at mevis.fraunhofer.de D-28359 Bremen Phone +49-421-218-7710 Germany Fax +49-421-218-4236 From jed at 59A2.org Thu Nov 11 10:08:08 2010 From: jed at 59A2.org (Jed Brown) Date: Thu, 11 Nov 2010 17:08:08 +0100 Subject: [petsc-users] Index set with zero size In-Reply-To: References: Message-ID: On Thu, Nov 11, 2010 at 17:03, Tim Kroeger wrote: > One more question: Does this bux and fix also apply to a submatrix of an > MPIAIJ matrix? That is, can I safely use MatMultAdd() having v2==v3 for such > a matrix in 3.1.0-p4 or not? (I'm having trouble with such a matrix now, > but the error might be on my side.) MPIAIJ implements MatGetSubMatrix, so it returns a new, efficient MatMPIAIJ matrix. This patch only affects matrix types that do not implement MatGetSubMatrix, and thus fall back on creating a MatSubMatrix (which just holds a reference to the original matrix and implements multiplication through scatters). So calling MatMultAdd using a matrix of any type other than MatSubMatrix is unrelated to my patch. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From filippo.spiga at disco.unimib.it Thu Nov 11 13:49:07 2010 From: filippo.spiga at disco.unimib.it (Filippo Spiga) Date: Thu, 11 Nov 2010 14:49:07 -0500 Subject: [petsc-users] Logging the same pieces of code several times using PetscLogEventBegin/PetscLogEventEnd Message-ID: <4CDC48B3.5060807@disco.unimib.it> Dear all, I would like to timing the same pieces of code several times using PetscLogEventBegin/PetscLogEventEnd. My code does, more or less, something like that PetscLogEvent logA, logB; PetscLogEventRegister( "Log A", 0, &logA); PetscLogEventRegister( "Log B", 0, &logB); for (i=0...10) { PetscLogEventBegin(logA, 0, 0, 0, 0); ... PetscLogEventEnd(logA, 0, 0, 0, 0); PetscLogEventBegin(logB 0, 0, 0, 0); .. PetscLogEventEnd(logB, 0, 0, 0, 0); } At the end, when I print the summary of all the evens using -log_summary, "Log A" and "Log B" appear many times... as many times as I performed the loop. How can change this behaviour? Thank you very much in advance! Cheers -- Filippo SPIGA, MSc Computer Science From balay at mcs.anl.gov Thu Nov 11 13:55:21 2010 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 11 Nov 2010 13:55:21 -0600 (CST) Subject: [petsc-users] Logging the same pieces of code several times using PetscLogEventBegin/PetscLogEventEnd In-Reply-To: <4CDC48B3.5060807@disco.unimib.it> References: <4CDC48B3.5060807@disco.unimib.it> Message-ID: I suspect you have PetscLogEventRegister() in a loop aswell.. Satish On Thu, 11 Nov 2010, Filippo Spiga wrote: > Dear all, > I would like to timing the same pieces of code several times using > PetscLogEventBegin/PetscLogEventEnd. My code does, more or less, something > like that > > PetscLogEvent logA, logB; > > PetscLogEventRegister( "Log A", 0, &logA); > PetscLogEventRegister( "Log B", 0, &logB); > > for (i=0...10) { > > PetscLogEventBegin(logA, 0, 0, 0, 0); > ... > PetscLogEventEnd(logA, 0, 0, 0, 0); > > PetscLogEventBegin(logB 0, 0, 0, 0); > .. > PetscLogEventEnd(logB, 0, 0, 0, 0); > > } > > At the end, when I print the summary of all the evens using -log_summary, "Log > A" and "Log B" appear many times... as many times as I performed the loop. How > can change this behaviour? > > Thank you very much in advance! > > Cheers > > From chris.bording at gmail.com Thu Nov 11 14:01:33 2010 From: chris.bording at gmail.com (Chris Bording) Date: Thu, 11 Nov 2010 12:01:33 -0800 Subject: [petsc-users] MATAIJ solvers Message-ID: <06A0D8A9-BFA9-4276-AA6F-E21031653138@gmail.com> Hi I am new to petsc and am attempting to parallelize ex13.c (Poisson example) in the ksp tutorials. I have been able to write some simple test codes and get them to work correctly and learn some of the mechanics of what petsc is doing, with respect to creating mpi vectors and matrices and in using the DA functions. I attempting to test the different solvers I am having some trouble... When I try to run with the same solver it fails with a message about Matrix format for mpiaij does not have a built-in PETSc direct solver. Which solvers are available for mpiaij formats? Also I created the Matrix and Vectors using the DACreate2d(dA,PETSC_COMM_WORLD,..... DAGetMatrix(dA,MATMPIAIJ,&A) DACreateGlobalVector(dA,&x) DACreateGlobalVector(dA,&b) .... is this dictating the types of solvers available? Is there another example of the variable Poisson in Parallel? or creating in a variable RHS and the rho variable in ex13 that maps from local to global indices? Thanks Chris Bording From bsmith at mcs.anl.gov Thu Nov 11 14:09:46 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 11 Nov 2010 14:09:46 -0600 Subject: [petsc-users] MATAIJ solvers In-Reply-To: <06A0D8A9-BFA9-4276-AA6F-E21031653138@gmail.com> References: <06A0D8A9-BFA9-4276-AA6F-E21031653138@gmail.com> Message-ID: <69434E52-9D0D-4387-ABB7-7E906930DA49@mcs.anl.gov> On Nov 11, 2010, at 2:01 PM, Chris Bording wrote: > Hi I am new to petsc and am attempting to parallelize ex13.c (Poisson example) in the ksp tutorials. I have been able to write some simple test codes and get them to work correctly and learn some of the mechanics of what petsc is doing, with respect to creating mpi vectors and matrices and in using the DA functions. > I attempting to test the different solvers I am having some trouble... > > When I try to run with the same solver it fails with a message about > Matrix format for mpiaij does not have a built-in PETSc direct solver. > Which solvers are available for mpiaij formats? Running with -help (and having called KSPSetFromOptions() in your code) will print all solvers possible. Generally you don't want to use a direct solver for variable Poisson but if you like you can ./configure PETSc with --download-superlu_dist or --download-mumps --download-blacs --download-scalapack and then run the code with -pc_type lu -pc_factor_mat_solver_package superlu_dist or mumps A good solver for this problem is hypre's BoomerAMG which you can get with ./configure --download-hypre and then run with -pc_type hyre Running with -ksp_view is a good way of seeing exactly what solver it is using. > > Also I created the Matrix and Vectors using the > DACreate2d(dA,PETSC_COMM_WORLD,..... > > DAGetMatrix(dA,MATMPIAIJ,&A) > > DACreateGlobalVector(dA,&x) > DACreateGlobalVector(dA,&b) > > .... > > is this dictating the types of solvers available? No For structured grids like with DA you can also use geometric multigrid, see for example ex22.c and some others in that directory. Barry > > > Is there another example of the variable Poisson in Parallel? > or creating in a variable RHS and the rho variable in ex13 that maps from local to global indices? > > Thanks > Chris Bording From filippo.spiga at disco.unimib.it Thu Nov 11 14:13:47 2010 From: filippo.spiga at disco.unimib.it (Filippo Spiga) Date: Thu, 11 Nov 2010 15:13:47 -0500 Subject: [petsc-users] Logging the same pieces of code several times using PetscLogEventBegin/PetscLogEventEnd In-Reply-To: References: <4CDC48B3.5060807@disco.unimib.it> Message-ID: <4CDC4E7B.60201@disco.unimib.it> Dear Satish, I checked and you are right! There are also other events logged (not only "Log A" and "Log B") inside some functions called many times during the loop. Inside these functions I've found PetscLogEventRegister. My fault. Cheers! Satish Balay wrote: > I suspect you have PetscLogEventRegister() in a loop aswell.. > > Satish -- Filippo SPIGA, MSc Computer Science From m.skates82 at gmail.com Fri Nov 12 09:43:46 2010 From: m.skates82 at gmail.com (Nunion) Date: Fri, 12 Nov 2010 09:43:46 -0600 Subject: [petsc-users] Compile Errors in Make files Message-ID: Hello, I recently started receiving the following error when trying to compile anything... makefile:16: /conf/variables: No such file or directory makefile:17: /conf/rules: No such file or directory makefile:858: /conf/test: No such file or directory make: *** No rule to make target `/conf/test'. Stop. The errors seem to be related to one another. I tried to hardcode this path, but the makefile is not picking up the path. I can't recall changing anything major Thanks! Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Nov 12 10:07:36 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 12 Nov 2010 10:07:36 -0600 Subject: [petsc-users] Compile Errors in Make files In-Reply-To: References: Message-ID: You have not set your PETSC_ARCH and PETSC_DIR On Nov 12, 2010, at 9:43 AM, Nunion wrote: > Hello, > > I recently started receiving the following error when trying to compile anything... > > makefile:16: /conf/variables: No such file or directory > makefile:17: /conf/rules: No such file or directory > makefile:858: /conf/test: No such file or directory > make: *** No rule to make target `/conf/test'. Stop. > > The errors seem to be related to one another. I tried to hardcode this path, but the makefile is not picking up the path. I can't recall changing anything major > > > Thanks! > > Mark From m.skates82 at gmail.com Fri Nov 12 10:26:02 2010 From: m.skates82 at gmail.com (Nunion) Date: Fri, 12 Nov 2010 10:26:02 -0600 Subject: [petsc-users] Compile Errors in Make files In-Reply-To: References: Message-ID: Initially, I thought that was the case as well. However, I saved some output from installation; =============================================================================== Configuring PETSc to compile on your system =============================================================================== TESTING: alternateConfigureLibrary from PETSc.packages.petsc4py(config/PETSc/packages/petsc4py.py:64) Compilers: C Compiler: mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -g3 Fortran Compiler: mpif90 -g Linkers: Static linker: /usr/bin/ar cr Dynamic linker: mpicc -shared -Wall -Wwrite-strings -Wno-strict-aliasing -g3 MPI: Includes: -I/home/walker/MPI/mpich2-1.0.8-install/include X11: Includes: Library: -lX11 BLAS/LAPACK: -llapack -lblas PETSc: PETSC_ARCH: linux-gnu-c-debug PETSC_DIR: /home/walker/Desktop/PETSc/LITE/petsc-3.1-p4 Clanguage: C shared libraries: disabled dynamic libraries: disabled Scalar type: real Precision: double Memory alignment: 16 xxx=========================================================================xxx Configure stage complete. Now build PETSc libraries with: make PETSC_DIR=/home/walker/Desktop/PETSc/LITE/petsc-3.1-p4 PETSC_ARCH=linux-gnu-c-debug all xxx=========================================================================xxx Is this the same information that would be contained in the log file for the configuration? On Fri, Nov 12, 2010 at 10:07 AM, Barry Smith wrote: > > You have not set your PETSC_ARCH and PETSC_DIR > > > On Nov 12, 2010, at 9:43 AM, Nunion wrote: > > > Hello, > > > > I recently started receiving the following error when trying to compile > anything... > > > > makefile:16: /conf/variables: No such file or directory > > makefile:17: /conf/rules: No such file or directory > > makefile:858: /conf/test: No such file or directory > > make: *** No rule to make target `/conf/test'. Stop. > > > > The errors seem to be related to one another. I tried to hardcode this > path, but the makefile is not picking up the path. I can't recall changing > anything major > > > > > > Thanks! > > > > Mark > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Nov 12 10:39:11 2010 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 12 Nov 2010 10:39:11 -0600 (CST) Subject: [petsc-users] Compile Errors in Make files In-Reply-To: References: Message-ID: On Fri, 12 Nov 2010, Nunion wrote: > Initially, I thought that was the case as well. However, I saved some > output from installation; ????? Why don't you try the provided suggestion - and say if it worked or - not? and if it didn't - furnish further relavent info - like the command used, errors obtained, makefile used? And how and where exactly did you set PETSC_ARCH and PETSC_DIR? BTW: The log you qouted mentiones one way of using PETSC_ARCH and PETSC_DIR with 'make' > make PETSC_DIR=/home/walker/Desktop/PETSc/LITE/petsc-3.1-p4 PETSC_ARCH=linux-gnu-c-debug all Satish > On Fri, Nov 12, 2010 at 10:07 AM, Barry Smith wrote: > > > > > You have not set your PETSC_ARCH and PETSC_DIR > > > > > > On Nov 12, 2010, at 9:43 AM, Nunion wrote: > > > > > Hello, > > > > > > I recently started receiving the following error when trying to compile > > anything... > > > > > > makefile:16: /conf/variables: No such file or directory > > > makefile:17: /conf/rules: No such file or directory > > > makefile:858: /conf/test: No such file or directory > > > make: *** No rule to make target `/conf/test'. Stop. > > > > > > The errors seem to be related to one another. I tried to hardcode this > > path, but the makefile is not picking up the path. I can't recall changing > > anything major > > > > > > > > > Thanks! > > > > > > Mark > > > > > From amari at cpht.polytechnique.fr Fri Nov 12 11:13:44 2010 From: amari at cpht.polytechnique.fr (Tahar Amari) Date: Fri, 12 Nov 2010 18:13:44 +0100 Subject: [petsc-users] Compile Errors with ifort 11.1 In-Reply-To: References: Message-ID: <021A6584-4DD7-44D0-BC9C-977E1DA737F9@cpht.polytechnique.fr> Hello, While compiling one of my code with ifort , I have the following error pointing towards petscvec.h . Any idea about this posssible kind of error please ? Any option of ifort to use ? Many thanks Tahar ifort -assume byterecl -I/usr/local/petsc/ -I/usr/local/petsc//include -I/usr/local/petsc//include/finclude -I/usr/local/petsc//macx/include -I/usr/X11R6/include/X11 -DPETSC_HAVE_PARMETIS -DPETSC_USE_DEBUG -DPETSC_USE_LOG -DPETSC_USE_BOPT_g -DPETSC_USE_STACK -DPETSC_AVOID_MPIF_H -c petsc.fpp /usr/local/petsc//include/finclude/petscvec.h(10): error #6418: This name has already been assigned a data type. [NORM_1] integer(kind=selected_int_kind(5)) NORM_1 -------------------------------------------- T. Amari Centre de Physique Theorique Ecole Polytechnique 91128 Palaiseau Cedex France tel : 33 1 69 33 42 52 fax: 33 1 69 33 49 49 email: URL : http://www.cpht.polytechnique.fr/cpht/amari -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Nov 12 11:17:38 2010 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 12 Nov 2010 11:17:38 -0600 (CST) Subject: [petsc-users] Compile Errors with ifort 11.1 In-Reply-To: <021A6584-4DD7-44D0-BC9C-977E1DA737F9@cpht.polytechnique.fr> References: <021A6584-4DD7-44D0-BC9C-977E1DA737F9@cpht.polytechnique.fr> Message-ID: Looks like this is related to the following: http://www.mcs.anl.gov/petsc/petsc-as/documentation/changes/31.html # petsc.h and petscdef.h are replaced with petscsys.h and petscsysdef.h; while petsc.h now includes ALL PETSc include files. Try removing all includes other than 'petsc.h' from your code. Satish On Fri, 12 Nov 2010, Tahar Amari wrote: > Hello, > > While compiling one of my code with ifort , I have the following error pointing > towards petscvec.h . > > Any idea about this posssible kind of error please ? Any option of ifort to use ? > > Many thanks > > Tahar > > > ifort -assume byterecl -I/usr/local/petsc/ -I/usr/local/petsc//include -I/usr/local/petsc//include/finclude -I/usr/local/petsc//macx/include -I/usr/X11R6/include/X11 -DPETSC_HAVE_PARMETIS -DPETSC_USE_DEBUG -DPETSC_USE_LOG -DPETSC_USE_BOPT_g -DPETSC_USE_STACK -DPETSC_AVOID_MPIF_H -c petsc.fpp > /usr/local/petsc//include/finclude/petscvec.h(10): error #6418: This name has already been assigned a data type. [NORM_1] > integer(kind=selected_int_kind(5)) NORM_1 > > > > -------------------------------------------- > T. Amari > Centre de Physique Theorique > Ecole Polytechnique > 91128 Palaiseau Cedex France > tel : 33 1 69 33 42 52 > fax: 33 1 69 33 49 49 > email: > URL : http://www.cpht.polytechnique.fr/cpht/amari > > > > > From amari at cpht.polytechnique.fr Fri Nov 12 11:23:23 2010 From: amari at cpht.polytechnique.fr (Tahar Amari) Date: Fri, 12 Nov 2010 18:23:23 +0100 Subject: [petsc-users] Compile Errors with ifort 11.1 In-Reply-To: References: <021A6584-4DD7-44D0-BC9C-977E1DA737F9@cpht.polytechnique.fr> Message-ID: <6501936B-4961-4C2E-A491-62F0A3655416@cpht.polytechnique.fr> Many thanks. This is solved. Now at link I have the library not found. Any idea ? -L/usr/local/petsc/macx/lib -lpetscsnes -lpetscvec -lpetscmat -lpetsccontrib -lpetscts -lpetscdm -lpetscksp -lpetsc -lmpich -lpmpich -lfmpich -lmpichf90 -lparmetis -lmetis -lfblas -lflapack ld: library not found for -lpetscsnes Tahar Le 12 nov. 2010 ? 18:17, Satish Balay a ?crit : > Looks like this is related to the following: > http://www.mcs.anl.gov/petsc/petsc-as/documentation/changes/31.html > # petsc.h and petscdef.h are replaced with petscsys.h and petscsysdef.h; while petsc.h now includes ALL PETSc include files. > > Try removing all includes other than 'petsc.h' from your code. > > Satish > > On Fri, 12 Nov 2010, Tahar Amari wrote: > >> Hello, >> >> While compiling one of my code with ifort , I have the following error pointing >> towards petscvec.h . >> >> Any idea about this posssible kind of error please ? Any option of ifort to use ? >> >> Many thanks >> >> Tahar >> >> >> ifort -assume byterecl -I/usr/local/petsc/ -I/usr/local/petsc//include -I/usr/local/petsc//include/finclude -I/usr/local/petsc//macx/include -I/usr/X11R6/include/X11 -DPETSC_HAVE_PARMETIS -DPETSC_USE_DEBUG -DPETSC_USE_LOG -DPETSC_USE_BOPT_g -DPETSC_USE_STACK -DPETSC_AVOID_MPIF_H -c petsc.fpp >> /usr/local/petsc//include/finclude/petscvec.h(10): error #6418: This name has already been assigned a data type. [NORM_1] >> integer(kind=selected_int_kind(5)) NORM_1 >> >> >> >> -------------------------------------------- >> T. Amari >> Centre de Physique Theorique >> Ecole Polytechnique >> 91128 Palaiseau Cedex France >> tel : 33 1 69 33 42 52 >> fax: 33 1 69 33 49 49 >> email: >> URL : http://www.cpht.polytechnique.fr/cpht/amari >> >> >> >> >> > -------------------------------------------- T. Amari Centre de Physique Theorique Ecole Polytechnique 91128 Palaiseau Cedex France tel : 33 1 69 33 42 52 fax: 33 1 69 33 49 49 email: URL : http://www.cpht.polytechnique.fr/cpht/amari -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Nov 12 11:26:30 2010 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 12 Nov 2010 11:26:30 -0600 (CST) Subject: [petsc-users] Compile Errors with ifort 11.1 In-Reply-To: <6501936B-4961-4C2E-A491-62F0A3655416@cpht.polytechnique.fr> References: <021A6584-4DD7-44D0-BC9C-977E1DA737F9@cpht.polytechnique.fr> <6501936B-4961-4C2E-A491-62F0A3655416@cpht.polytechnique.fr> Message-ID: Looks like you are not using PETSc makefiles - and upgrading from older release to petsc-3.1 Currently there is only one library for petsc [-lpetsc] Check: cd $PETSC_DIR make getincludedirs getlinklibs Satish On Fri, 12 Nov 2010, Tahar Amari wrote: > Many thanks. This is solved. > > Now at link I have the library not found. Any idea ? > > > -L/usr/local/petsc/macx/lib -lpetscsnes -lpetscvec -lpetscmat -lpetsccontrib -lpetscts -lpetscdm -lpetscksp -lpetsc -lmpich -lpmpich -lfmpich -lmpichf90 -lparmetis -lmetis -lfblas -lflapack > ld: library not found for -lpetscsnes > > > Tahar > > > Le 12 nov. 2010 ? 18:17, Satish Balay a ?crit : > > > Looks like this is related to the following: > > http://www.mcs.anl.gov/petsc/petsc-as/documentation/changes/31.html > > # petsc.h and petscdef.h are replaced with petscsys.h and petscsysdef.h; while petsc.h now includes ALL PETSc include files. > > > > Try removing all includes other than 'petsc.h' from your code. > > > > Satish > > > > On Fri, 12 Nov 2010, Tahar Amari wrote: > > > >> Hello, > >> > >> While compiling one of my code with ifort , I have the following error pointing > >> towards petscvec.h . > >> > >> Any idea about this posssible kind of error please ? Any option of ifort to use ? > >> > >> Many thanks > >> > >> Tahar > >> > >> > >> ifort -assume byterecl -I/usr/local/petsc/ -I/usr/local/petsc//include -I/usr/local/petsc//include/finclude -I/usr/local/petsc//macx/include -I/usr/X11R6/include/X11 -DPETSC_HAVE_PARMETIS -DPETSC_USE_DEBUG -DPETSC_USE_LOG -DPETSC_USE_BOPT_g -DPETSC_USE_STACK -DPETSC_AVOID_MPIF_H -c petsc.fpp > >> /usr/local/petsc//include/finclude/petscvec.h(10): error #6418: This name has already been assigned a data type. [NORM_1] > >> integer(kind=selected_int_kind(5)) NORM_1 > >> > >> > >> > >> -------------------------------------------- > >> T. Amari > >> Centre de Physique Theorique > >> Ecole Polytechnique > >> 91128 Palaiseau Cedex France > >> tel : 33 1 69 33 42 52 > >> fax: 33 1 69 33 49 49 > >> email: > >> URL : http://www.cpht.polytechnique.fr/cpht/amari > >> > >> > >> > >> > >> > > > > -------------------------------------------- > T. Amari > Centre de Physique Theorique > Ecole Polytechnique > 91128 Palaiseau Cedex France > tel : 33 1 69 33 42 52 > fax: 33 1 69 33 49 49 > email: > URL : http://www.cpht.polytechnique.fr/cpht/amari > > > > > From amari at cpht.polytechnique.fr Fri Nov 12 11:32:33 2010 From: amari at cpht.polytechnique.fr (Tahar Amari) Date: Fri, 12 Nov 2010 18:32:33 +0100 Subject: [petsc-users] Compile Errors with ifort 11.1 In-Reply-To: References: <021A6584-4DD7-44D0-BC9C-977E1DA737F9@cpht.polytechnique.fr> <6501936B-4961-4C2E-A491-62F0A3655416@cpht.polytechnique.fr> Message-ID: <2E4B4874-EC33-4DAC-91E1-0E188FBFD2ED@cpht.polytechnique.fr> You are right. Here is what I got timac> make getincludedirs getlinklibs -I/Data/Poub1/petsc-3.1-p5/macx/include -I/Data/Poub1/petsc-3.1-p5/include -I/Data/Poub1/petsc-3.1-p5/macx/include -L/Data/Poub1/petsc-3.1-p5/macx/lib -L/Data/Poub1/petsc-3.1-p5/macx/lib -L/Data/Poub1/petsc-3.1-p5/macx/lib -lpetsc -L/usr/X11R6/lib -lX11 -L/Data/Poub1/petsc-3.1-p5/macx/lib -lparmetis -lmetis -lflapack -lfblas -L/Data/Poub1/petsc-3.1-p5/macx/lib -L/usr/lib/gcc/i686-apple-darwin10/4.2.1/x86_64 -L/usr/lib/i686-apple-darwin10/4.2.1 -L/opt/intel/Compiler/11.1/088/lib -L/opt/intel/Compiler/11.1/088/Frameworks/mkl/lib/em64t -L/usr/lib/gcc/i686-apple-darwin10/4.2.1 -ldl -lpmpich -lmpich -lSystem -lmpichf90 -lifport -lifcore -limf -lsvml -lipgo -lgcc_s.10.5 -lSystemStubs -lmx -lirc -lpthread -ldl -lpmpich -lmpich -lSystem -ldl What does it mean please for me to change in my makefiles ? Do you mean that I should remove any link other than -lpetsc ? Great thanks Tahar Le 12 nov. 2010 ? 18:26, Satish Balay a ?crit : > Looks like you are not using PETSc makefiles - and upgrading from > older release to petsc-3.1 > > Currently there is only one library for petsc [-lpetsc] > > Check: > cd $PETSC_DIR > make getincludedirs getlinklibs > > Satish > > On Fri, 12 Nov 2010, Tahar Amari wrote: > >> Many thanks. This is solved. >> >> Now at link I have the library not found. Any idea ? >> >> >> -L/usr/local/petsc/macx/lib -lpetscsnes -lpetscvec -lpetscmat -lpetsccontrib -lpetscts -lpetscdm -lpetscksp -lpetsc -lmpich -lpmpich -lfmpich -lmpichf90 -lparmetis -lmetis -lfblas -lflapack >> ld: library not found for -lpetscsnes >> >> >> Tahar >> >> >> Le 12 nov. 2010 ? 18:17, Satish Balay a ?crit : >> >>> Looks like this is related to the following: >>> http://www.mcs.anl.gov/petsc/petsc-as/documentation/changes/31.html >>> # petsc.h and petscdef.h are replaced with petscsys.h and petscsysdef.h; while petsc.h now includes ALL PETSc include files. >>> >>> Try removing all includes other than 'petsc.h' from your code. >>> >>> Satish >>> >>> On Fri, 12 Nov 2010, Tahar Amari wrote: >>> >>>> Hello, >>>> >>>> While compiling one of my code with ifort , I have the following error pointing >>>> towards petscvec.h . >>>> >>>> Any idea about this posssible kind of error please ? Any option of ifort to use ? >>>> >>>> Many thanks >>>> >>>> Tahar >>>> >>>> >>>> ifort -assume byterecl -I/usr/local/petsc/ -I/usr/local/petsc//include -I/usr/local/petsc//include/finclude -I/usr/local/petsc//macx/include -I/usr/X11R6/include/X11 -DPETSC_HAVE_PARMETIS -DPETSC_USE_DEBUG -DPETSC_USE_LOG -DPETSC_USE_BOPT_g -DPETSC_USE_STACK -DPETSC_AVOID_MPIF_H -c petsc.fpp >>>> /usr/local/petsc//include/finclude/petscvec.h(10): error #6418: This name has already been assigned a data type. [NORM_1] >>>> integer(kind=selected_int_kind(5)) NORM_1 >>>> >>>> >>>> >>>> -------------------------------------------- >>>> T. Amari >>>> Centre de Physique Theorique >>>> Ecole Polytechnique >>>> 91128 Palaiseau Cedex France >>>> tel : 33 1 69 33 42 52 >>>> fax: 33 1 69 33 49 49 >>>> email: >>>> URL : http://www.cpht.polytechnique.fr/cpht/amari >>>> >>>> >>>> >>>> >>>> >>> >> >> -------------------------------------------- >> T. Amari >> Centre de Physique Theorique >> Ecole Polytechnique >> 91128 Palaiseau Cedex France >> tel : 33 1 69 33 42 52 >> fax: 33 1 69 33 49 49 >> email: >> URL : http://www.cpht.polytechnique.fr/cpht/amari >> >> >> >> >> -------------------------------------------- T. Amari Centre de Physique Theorique Ecole Polytechnique 91128 Palaiseau Cedex France tel : 33 1 69 33 42 52 fax: 33 1 69 33 49 49 email: URL : http://www.cpht.polytechnique.fr/cpht/amari -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Nov 12 11:34:24 2010 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 12 Nov 2010 11:34:24 -0600 (CST) Subject: [petsc-users] Compile Errors with ifort 11.1 In-Reply-To: <2E4B4874-EC33-4DAC-91E1-0E188FBFD2ED@cpht.polytechnique.fr> References: <021A6584-4DD7-44D0-BC9C-977E1DA737F9@cpht.polytechnique.fr> <6501936B-4961-4C2E-A491-62F0A3655416@cpht.polytechnique.fr> <2E4B4874-EC33-4DAC-91E1-0E188FBFD2ED@cpht.polytechnique.fr> Message-ID: Make sure your compiles have all the flags listed here.. Its best to use petsc makefile formats so that this info is automatically picked up from the petsc config files. [and not tweak user makefiles for every install of petsc..] Satish On Fri, 12 Nov 2010, Tahar Amari wrote: > You are right. > > Here is what I got > > > timac> make getincludedirs getlinklibs > -I/Data/Poub1/petsc-3.1-p5/macx/include -I/Data/Poub1/petsc-3.1-p5/include -I/Data/Poub1/petsc-3.1-p5/macx/include > -L/Data/Poub1/petsc-3.1-p5/macx/lib -L/Data/Poub1/petsc-3.1-p5/macx/lib -L/Data/Poub1/petsc-3.1-p5/macx/lib -lpetsc -L/usr/X11R6/lib -lX11 -L/Data/Poub1/petsc-3.1-p5/macx/lib -lparmetis -lmetis -lflapack -lfblas -L/Data/Poub1/petsc-3.1-p5/macx/lib -L/usr/lib/gcc/i686-apple-darwin10/4.2.1/x86_64 -L/usr/lib/i686-apple-darwin10/4.2.1 -L/opt/intel/Compiler/11.1/088/lib -L/opt/intel/Compiler/11.1/088/Frameworks/mkl/lib/em64t -L/usr/lib/gcc/i686-apple-darwin10/4.2.1 -ldl -lpmpich -lmpich -lSystem -lmpichf90 -lifport -lifcore -limf -lsvml -lipgo -lgcc_s.10.5 -lSystemStubs -lmx -lirc -lpthread -ldl -lpmpich -lmpich -lSystem -ldl > > > What does it mean please for me to change in my makefiles ? > > Do you mean that I should remove any link other than -lpetsc ? > > Great thanks > > Tahar > > > > > > > > > Le 12 nov. 2010 ? 18:26, Satish Balay a ?crit : > > > Looks like you are not using PETSc makefiles - and upgrading from > > older release to petsc-3.1 > > > > Currently there is only one library for petsc [-lpetsc] > > > > Check: > > cd $PETSC_DIR > > make getincludedirs getlinklibs > > > > Satish > > > > On Fri, 12 Nov 2010, Tahar Amari wrote: > > > >> Many thanks. This is solved. > >> > >> Now at link I have the library not found. Any idea ? > >> > >> > >> -L/usr/local/petsc/macx/lib -lpetscsnes -lpetscvec -lpetscmat -lpetsccontrib -lpetscts -lpetscdm -lpetscksp -lpetsc -lmpich -lpmpich -lfmpich -lmpichf90 -lparmetis -lmetis -lfblas -lflapack > >> ld: library not found for -lpetscsnes > >> > >> > >> Tahar > >> > >> > >> Le 12 nov. 2010 ? 18:17, Satish Balay a ?crit : > >> > >>> Looks like this is related to the following: > >>> http://www.mcs.anl.gov/petsc/petsc-as/documentation/changes/31.html > >>> # petsc.h and petscdef.h are replaced with petscsys.h and petscsysdef.h; while petsc.h now includes ALL PETSc include files. > >>> > >>> Try removing all includes other than 'petsc.h' from your code. > >>> > >>> Satish > >>> > >>> On Fri, 12 Nov 2010, Tahar Amari wrote: > >>> > >>>> Hello, > >>>> > >>>> While compiling one of my code with ifort , I have the following error pointing > >>>> towards petscvec.h . > >>>> > >>>> Any idea about this posssible kind of error please ? Any option of ifort to use ? > >>>> > >>>> Many thanks > >>>> > >>>> Tahar > >>>> > >>>> > >>>> ifort -assume byterecl -I/usr/local/petsc/ -I/usr/local/petsc//include -I/usr/local/petsc//include/finclude -I/usr/local/petsc//macx/include -I/usr/X11R6/include/X11 -DPETSC_HAVE_PARMETIS -DPETSC_USE_DEBUG -DPETSC_USE_LOG -DPETSC_USE_BOPT_g -DPETSC_USE_STACK -DPETSC_AVOID_MPIF_H -c petsc.fpp > >>>> /usr/local/petsc//include/finclude/petscvec.h(10): error #6418: This name has already been assigned a data type. [NORM_1] > >>>> integer(kind=selected_int_kind(5)) NORM_1 > >>>> > >>>> > >>>> > >>>> -------------------------------------------- > >>>> T. Amari > >>>> Centre de Physique Theorique > >>>> Ecole Polytechnique > >>>> 91128 Palaiseau Cedex France > >>>> tel : 33 1 69 33 42 52 > >>>> fax: 33 1 69 33 49 49 > >>>> email: > >>>> URL : http://www.cpht.polytechnique.fr/cpht/amari > >>>> > >>>> > >>>> > >>>> > >>>> > >>> > >> > >> -------------------------------------------- > >> T. Amari > >> Centre de Physique Theorique > >> Ecole Polytechnique > >> 91128 Palaiseau Cedex France > >> tel : 33 1 69 33 42 52 > >> fax: 33 1 69 33 49 49 > >> email: > >> URL : http://www.cpht.polytechnique.fr/cpht/amari > >> > >> > >> > >> > >> > > -------------------------------------------- > T. Amari > Centre de Physique Theorique > Ecole Polytechnique > 91128 Palaiseau Cedex France > tel : 33 1 69 33 42 52 > fax: 33 1 69 33 49 49 > email: > URL : http://www.cpht.polytechnique.fr/cpht/amari > > > > > From amari at cpht.polytechnique.fr Fri Nov 12 11:40:55 2010 From: amari at cpht.polytechnique.fr (Tahar Amari) Date: Fri, 12 Nov 2010 18:40:55 +0100 Subject: [petsc-users] Compile Errors with ifort 11.1 In-Reply-To: References: <021A6584-4DD7-44D0-BC9C-977E1DA737F9@cpht.polytechnique.fr> <6501936B-4961-4C2E-A491-62F0A3655416@cpht.polytechnique.fr> <2E4B4874-EC33-4DAC-91E1-0E188FBFD2ED@cpht.polytechnique.fr> Message-ID: <0E36C9ED-9082-4302-91B6-023DE9A7367A@cpht.polytechnique.fr> Great thanks, it seems that -lpetsc alone works . I will follow this advice regarding the makefile, for the reason you gave. Where should I get a typical makefile to modify please ? Tahar Le 12 nov. 2010 ? 18:34, Satish Balay a ?crit : > Make sure your compiles have all the flags listed here.. > > Its best to use petsc makefile formats so that this info is > automatically picked up from the petsc config files. [and not tweak > user makefiles for every install of petsc..] > > Satish > > On Fri, 12 Nov 2010, Tahar Amari wrote: > >> You are right. >> >> Here is what I got >> >> >> timac> make getincludedirs getlinklibs >> -I/Data/Poub1/petsc-3.1-p5/macx/include -I/Data/Poub1/petsc-3.1-p5/include -I/Data/Poub1/petsc-3.1-p5/macx/include >> -L/Data/Poub1/petsc-3.1-p5/macx/lib -L/Data/Poub1/petsc-3.1-p5/macx/lib -L/Data/Poub1/petsc-3.1-p5/macx/lib -lpetsc -L/usr/X11R6/lib -lX11 -L/Data/Poub1/petsc-3.1-p5/macx/lib -lparmetis -lmetis -lflapack -lfblas -L/Data/Poub1/petsc-3.1-p5/macx/lib -L/usr/lib/gcc/i686-apple-darwin10/4.2.1/x86_64 -L/usr/lib/i686-apple-darwin10/4.2.1 -L/opt/intel/Compiler/11.1/088/lib -L/opt/intel/Compiler/11.1/088/Frameworks/mkl/lib/em64t -L/usr/lib/gcc/i686-apple-darwin10/4.2.1 -ldl -lpmpich -lmpich -lSystem -lmpichf90 -lifport -lifcore -limf -lsvml -lipgo -lgcc_s.10.5 -lSystemStubs -lmx -lirc -lpthread -ldl -lpmpich -lmpich -lSystem -ldl >> >> >> What does it mean please for me to change in my makefiles ? >> >> Do you mean that I should remove any link other than -lpetsc ? >> >> Great thanks >> >> Tahar >> >> >> >> >> >> >> >> >> Le 12 nov. 2010 ? 18:26, Satish Balay a ?crit : >> >>> Looks like you are not using PETSc makefiles - and upgrading from >>> older release to petsc-3.1 >>> >>> Currently there is only one library for petsc [-lpetsc] >>> >>> Check: >>> cd $PETSC_DIR >>> make getincludedirs getlinklibs >>> >>> Satish >>> >>> On Fri, 12 Nov 2010, Tahar Amari wrote: >>> >>>> Many thanks. This is solved. >>>> >>>> Now at link I have the library not found. Any idea ? >>>> >>>> >>>> -L/usr/local/petsc/macx/lib -lpetscsnes -lpetscvec -lpetscmat -lpetsccontrib -lpetscts -lpetscdm -lpetscksp -lpetsc -lmpich -lpmpich -lfmpich -lmpichf90 -lparmetis -lmetis -lfblas -lflapack >>>> ld: library not found for -lpetscsnes >>>> >>>> >>>> Tahar >>>> >>>> >>>> Le 12 nov. 2010 ? 18:17, Satish Balay a ?crit : >>>> >>>>> Looks like this is related to the following: >>>>> http://www.mcs.anl.gov/petsc/petsc-as/documentation/changes/31.html >>>>> # petsc.h and petscdef.h are replaced with petscsys.h and petscsysdef.h; while petsc.h now includes ALL PETSc include files. >>>>> >>>>> Try removing all includes other than 'petsc.h' from your code. >>>>> >>>>> Satish >>>>> >>>>> On Fri, 12 Nov 2010, Tahar Amari wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> While compiling one of my code with ifort , I have the following error pointing >>>>>> towards petscvec.h . >>>>>> >>>>>> Any idea about this posssible kind of error please ? Any option of ifort to use ? >>>>>> >>>>>> Many thanks >>>>>> >>>>>> Tahar >>>>>> >>>>>> >>>>>> ifort -assume byterecl -I/usr/local/petsc/ -I/usr/local/petsc//include -I/usr/local/petsc//include/finclude -I/usr/local/petsc//macx/include -I/usr/X11R6/include/X11 -DPETSC_HAVE_PARMETIS -DPETSC_USE_DEBUG -DPETSC_USE_LOG -DPETSC_USE_BOPT_g -DPETSC_USE_STACK -DPETSC_AVOID_MPIF_H -c petsc.fpp >>>>>> /usr/local/petsc//include/finclude/petscvec.h(10): error #6418: This name has already been assigned a data type. [NORM_1] >>>>>> integer(kind=selected_int_kind(5)) NORM_1 >>>>>> >>>>>> >>>>>> >>>>>> -------------------------------------------- >>>>>> T. Amari >>>>>> Centre de Physique Theorique >>>>>> Ecole Polytechnique >>>>>> 91128 Palaiseau Cedex France >>>>>> tel : 33 1 69 33 42 52 >>>>>> fax: 33 1 69 33 49 49 >>>>>> email: >>>>>> URL : http://www.cpht.polytechnique.fr/cpht/amari >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>> >>>> -------------------------------------------- >>>> T. Amari >>>> Centre de Physique Theorique >>>> Ecole Polytechnique >>>> 91128 Palaiseau Cedex France >>>> tel : 33 1 69 33 42 52 >>>> fax: 33 1 69 33 49 49 >>>> email: >>>> URL : http://www.cpht.polytechnique.fr/cpht/amari >>>> >>>> >>>> >>>> >>>> >> >> -------------------------------------------- >> T. Amari >> Centre de Physique Theorique >> Ecole Polytechnique >> 91128 Palaiseau Cedex France >> tel : 33 1 69 33 42 52 >> fax: 33 1 69 33 49 49 >> email: >> URL : http://www.cpht.polytechnique.fr/cpht/amari >> >> >> >> >> -------------------------------------------- T. Amari Centre de Physique Theorique Ecole Polytechnique 91128 Palaiseau Cedex France tel : 33 1 69 33 42 52 fax: 33 1 69 33 49 49 email: URL : http://www.cpht.polytechnique.fr/cpht/amari -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Nov 12 11:44:49 2010 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 12 Nov 2010 11:44:49 -0600 (CST) Subject: [petsc-users] Compile Errors with ifort 11.1 In-Reply-To: <0E36C9ED-9082-4302-91B6-023DE9A7367A@cpht.polytechnique.fr> References: <021A6584-4DD7-44D0-BC9C-977E1DA737F9@cpht.polytechnique.fr> <6501936B-4961-4C2E-A491-62F0A3655416@cpht.polytechnique.fr> <2E4B4874-EC33-4DAC-91E1-0E188FBFD2ED@cpht.polytechnique.fr> <0E36C9ED-9082-4302-91B6-023DE9A7367A@cpht.polytechnique.fr> Message-ID: Attaching a minimal sample makefile. The additional info it would need is PETSC_DIR, PETSC_ARCH values for your install of PETSc. All example/tests,tutorial dirs have makefiles in petsc format.. Satish On Fri, 12 Nov 2010, Tahar Amari wrote: > Great thanks, it seems that -lpetsc alone works . > > I will follow this advice regarding the makefile, for the reason you gave. > Where should I get a typical makefile to modify please ? > > Tahar > > > Le 12 nov. 2010 ? 18:34, Satish Balay a ?crit : > > > Make sure your compiles have all the flags listed here.. > > > > Its best to use petsc makefile formats so that this info is > > automatically picked up from the petsc config files. [and not tweak > > user makefiles for every install of petsc..] > > > > Satish > > > > On Fri, 12 Nov 2010, Tahar Amari wrote: > > > >> You are right. > >> > >> Here is what I got > >> > >> > >> timac> make getincludedirs getlinklibs > >> -I/Data/Poub1/petsc-3.1-p5/macx/include -I/Data/Poub1/petsc-3.1-p5/include -I/Data/Poub1/petsc-3.1-p5/macx/include > >> -L/Data/Poub1/petsc-3.1-p5/macx/lib -L/Data/Poub1/petsc-3.1-p5/macx/lib -L/Data/Poub1/petsc-3.1-p5/macx/lib -lpetsc -L/usr/X11R6/lib -lX11 -L/Data/Poub1/petsc-3.1-p5/macx/lib -lparmetis -lmetis -lflapack -lfblas -L/Data/Poub1/petsc-3.1-p5/macx/lib -L/usr/lib/gcc/i686-apple-darwin10/4.2.1/x86_64 -L/usr/lib/i686-apple-darwin10/4.2.1 -L/opt/intel/Compiler/11.1/088/lib -L/opt/intel/Compiler/11.1/088/Frameworks/mkl/lib/em64t -L/usr/lib/gcc/i686-apple-darwin10/4.2.1 -ldl -lpmpich -lmpich -lSystem -lmpichf90 -lifport -lifcore -limf -lsvml -lipgo -lgcc_s.10.5 -lSystemStubs -lmx -lirc -lpthread -ldl -lpmpich -lmpich -lSystem -ldl > >> > >> > >> What does it mean please for me to change in my makefiles ? > >> > >> Do you mean that I should remove any link other than -lpetsc ? > >> > >> Great thanks > >> > >> Tahar > >> > >> > >> > >> > >> > >> > >> > >> > >> Le 12 nov. 2010 ? 18:26, Satish Balay a ?crit : > >> > >>> Looks like you are not using PETSc makefiles - and upgrading from > >>> older release to petsc-3.1 > >>> > >>> Currently there is only one library for petsc [-lpetsc] > >>> > >>> Check: > >>> cd $PETSC_DIR > >>> make getincludedirs getlinklibs > >>> > >>> Satish > >>> > >>> On Fri, 12 Nov 2010, Tahar Amari wrote: > >>> > >>>> Many thanks. This is solved. > >>>> > >>>> Now at link I have the library not found. Any idea ? > >>>> > >>>> > >>>> -L/usr/local/petsc/macx/lib -lpetscsnes -lpetscvec -lpetscmat -lpetsccontrib -lpetscts -lpetscdm -lpetscksp -lpetsc -lmpich -lpmpich -lfmpich -lmpichf90 -lparmetis -lmetis -lfblas -lflapack > >>>> ld: library not found for -lpetscsnes > >>>> > >>>> > >>>> Tahar > >>>> > >>>> > >>>> Le 12 nov. 2010 ? 18:17, Satish Balay a ?crit : > >>>> > >>>>> Looks like this is related to the following: > >>>>> http://www.mcs.anl.gov/petsc/petsc-as/documentation/changes/31.html > >>>>> # petsc.h and petscdef.h are replaced with petscsys.h and petscsysdef.h; while petsc.h now includes ALL PETSc include files. > >>>>> > >>>>> Try removing all includes other than 'petsc.h' from your code. > >>>>> > >>>>> Satish > >>>>> > >>>>> On Fri, 12 Nov 2010, Tahar Amari wrote: > >>>>> > >>>>>> Hello, > >>>>>> > >>>>>> While compiling one of my code with ifort , I have the following error pointing > >>>>>> towards petscvec.h . > >>>>>> > >>>>>> Any idea about this posssible kind of error please ? Any option of ifort to use ? > >>>>>> > >>>>>> Many thanks > >>>>>> > >>>>>> Tahar > >>>>>> > >>>>>> > >>>>>> ifort -assume byterecl -I/usr/local/petsc/ -I/usr/local/petsc//include -I/usr/local/petsc//include/finclude -I/usr/local/petsc//macx/include -I/usr/X11R6/include/X11 -DPETSC_HAVE_PARMETIS -DPETSC_USE_DEBUG -DPETSC_USE_LOG -DPETSC_USE_BOPT_g -DPETSC_USE_STACK -DPETSC_AVOID_MPIF_H -c petsc.fpp > >>>>>> /usr/local/petsc//include/finclude/petscvec.h(10): error #6418: This name has already been assigned a data type. [NORM_1] > >>>>>> integer(kind=selected_int_kind(5)) NORM_1 > >>>>>> > >>>>>> > >>>>>> > >>>>>> -------------------------------------------- > >>>>>> T. Amari > >>>>>> Centre de Physique Theorique > >>>>>> Ecole Polytechnique > >>>>>> 91128 Palaiseau Cedex France > >>>>>> tel : 33 1 69 33 42 52 > >>>>>> fax: 33 1 69 33 49 49 > >>>>>> email: > >>>>>> URL : http://www.cpht.polytechnique.fr/cpht/amari > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>> > >>>> > >>>> -------------------------------------------- > >>>> T. Amari > >>>> Centre de Physique Theorique > >>>> Ecole Polytechnique > >>>> 91128 Palaiseau Cedex France > >>>> tel : 33 1 69 33 42 52 > >>>> fax: 33 1 69 33 49 49 > >>>> email: > >>>> URL : http://www.cpht.polytechnique.fr/cpht/amari > >>>> > >>>> > >>>> > >>>> > >>>> > >> > >> -------------------------------------------- > >> T. Amari > >> Centre de Physique Theorique > >> Ecole Polytechnique > >> 91128 Palaiseau Cedex France > >> tel : 33 1 69 33 42 52 > >> fax: 33 1 69 33 49 49 > >> email: > >> URL : http://www.cpht.polytechnique.fr/cpht/amari > >> > >> > >> > >> > >> > > -------------------------------------------- > T. Amari > Centre de Physique Theorique > Ecole Polytechnique > 91128 Palaiseau Cedex France > tel : 33 1 69 33 42 52 > fax: 33 1 69 33 49 49 > email: > URL : http://www.cpht.polytechnique.fr/cpht/amari > > > > > -------------- next part -------------- CFLAGS = FFLAGS = CPPFLAGS = FPPFLAGS = CLEANFILES = include ${PETSC_DIR}/conf/variables include ${PETSC_DIR}/conf/rules OBJS = ex1.o ex1: ${OBJS} chkopts -${CLINKER} -o ex1 ${OBJS} ${PETSC_LIB} From amari at cpht.polytechnique.fr Fri Nov 12 11:51:09 2010 From: amari at cpht.polytechnique.fr (Tahar Amari) Date: Fri, 12 Nov 2010 18:51:09 +0100 Subject: [petsc-users] Compile Errors with ifort 11.1 In-Reply-To: References: <021A6584-4DD7-44D0-BC9C-977E1DA737F9@cpht.polytechnique.fr> <6501936B-4961-4C2E-A491-62F0A3655416@cpht.polytechnique.fr> <2E4B4874-EC33-4DAC-91E1-0E188FBFD2ED@cpht.polytechnique.fr> <0E36C9ED-9082-4302-91B6-023DE9A7367A@cpht.polytechnique.fr> Message-ID: Thank you very much for your efficient help. Tahar Le 12 nov. 2010 ? 18:44, Satish Balay a ?crit : > Attaching a minimal sample makefile. The additional info it would > need is PETSC_DIR, PETSC_ARCH values for your install of PETSc. > > All example/tests,tutorial dirs have makefiles in petsc format.. > > Satish > > On Fri, 12 Nov 2010, Tahar Amari wrote: > >> Great thanks, it seems that -lpetsc alone works . >> >> I will follow this advice regarding the makefile, for the reason you gave. >> Where should I get a typical makefile to modify please ? >> >> Tahar >> >> >> Le 12 nov. 2010 ? 18:34, Satish Balay a ?crit : >> >>> Make sure your compiles have all the flags listed here.. >>> >>> Its best to use petsc makefile formats so that this info is >>> automatically picked up from the petsc config files. [and not tweak >>> user makefiles for every install of petsc..] >>> >>> Satish >>> >>> On Fri, 12 Nov 2010, Tahar Amari wrote: >>> >>>> You are right. >>>> >>>> Here is what I got >>>> >>>> >>>> timac> make getincludedirs getlinklibs >>>> -I/Data/Poub1/petsc-3.1-p5/macx/include -I/Data/Poub1/petsc-3.1-p5/include -I/Data/Poub1/petsc-3.1-p5/macx/include >>>> -L/Data/Poub1/petsc-3.1-p5/macx/lib -L/Data/Poub1/petsc-3.1-p5/macx/lib -L/Data/Poub1/petsc-3.1-p5/macx/lib -lpetsc -L/usr/X11R6/lib -lX11 -L/Data/Poub1/petsc-3.1-p5/macx/lib -lparmetis -lmetis -lflapack -lfblas -L/Data/Poub1/petsc-3.1-p5/macx/lib -L/usr/lib/gcc/i686-apple-darwin10/4.2.1/x86_64 -L/usr/lib/i686-apple-darwin10/4.2.1 -L/opt/intel/Compiler/11.1/088/lib -L/opt/intel/Compiler/11.1/088/Frameworks/mkl/lib/em64t -L/usr/lib/gcc/i686-apple-darwin10/4.2.1 -ldl -lpmpich -lmpich -lSystem -lmpichf90 -lifport -lifcore -limf -lsvml -lipgo -lgcc_s.10.5 -lSystemStubs -lmx -lirc -lpthread -ldl -lpmpich -lmpich -lSystem -ldl >>>> >>>> >>>> What does it mean please for me to change in my makefiles ? >>>> >>>> Do you mean that I should remove any link other than -lpetsc ? >>>> >>>> Great thanks >>>> >>>> Tahar >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> Le 12 nov. 2010 ? 18:26, Satish Balay a ?crit : >>>> >>>>> Looks like you are not using PETSc makefiles - and upgrading from >>>>> older release to petsc-3.1 >>>>> >>>>> Currently there is only one library for petsc [-lpetsc] >>>>> >>>>> Check: >>>>> cd $PETSC_DIR >>>>> make getincludedirs getlinklibs >>>>> >>>>> Satish >>>>> >>>>> On Fri, 12 Nov 2010, Tahar Amari wrote: >>>>> >>>>>> Many thanks. This is solved. >>>>>> >>>>>> Now at link I have the library not found. Any idea ? >>>>>> >>>>>> >>>>>> -L/usr/local/petsc/macx/lib -lpetscsnes -lpetscvec -lpetscmat -lpetsccontrib -lpetscts -lpetscdm -lpetscksp -lpetsc -lmpich -lpmpich -lfmpich -lmpichf90 -lparmetis -lmetis -lfblas -lflapack >>>>>> ld: library not found for -lpetscsnes >>>>>> >>>>>> >>>>>> Tahar >>>>>> >>>>>> >>>>>> Le 12 nov. 2010 ? 18:17, Satish Balay a ?crit : >>>>>> >>>>>>> Looks like this is related to the following: >>>>>>> http://www.mcs.anl.gov/petsc/petsc-as/documentation/changes/31.html >>>>>>> # petsc.h and petscdef.h are replaced with petscsys.h and petscsysdef.h; while petsc.h now includes ALL PETSc include files. >>>>>>> >>>>>>> Try removing all includes other than 'petsc.h' from your code. >>>>>>> >>>>>>> Satish >>>>>>> >>>>>>> On Fri, 12 Nov 2010, Tahar Amari wrote: >>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> While compiling one of my code with ifort , I have the following error pointing >>>>>>>> towards petscvec.h . >>>>>>>> >>>>>>>> Any idea about this posssible kind of error please ? Any option of ifort to use ? >>>>>>>> >>>>>>>> Many thanks >>>>>>>> >>>>>>>> Tahar >>>>>>>> >>>>>>>> >>>>>>>> ifort -assume byterecl -I/usr/local/petsc/ -I/usr/local/petsc//include -I/usr/local/petsc//include/finclude -I/usr/local/petsc//macx/include -I/usr/X11R6/include/X11 -DPETSC_HAVE_PARMETIS -DPETSC_USE_DEBUG -DPETSC_USE_LOG -DPETSC_USE_BOPT_g -DPETSC_USE_STACK -DPETSC_AVOID_MPIF_H -c petsc.fpp >>>>>>>> /usr/local/petsc//include/finclude/petscvec.h(10): error #6418: This name has already been assigned a data type. [NORM_1] >>>>>>>> integer(kind=selected_int_kind(5)) NORM_1 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -------------------------------------------- >>>>>>>> T. Amari >>>>>>>> Centre de Physique Theorique >>>>>>>> Ecole Polytechnique >>>>>>>> 91128 Palaiseau Cedex France >>>>>>>> tel : 33 1 69 33 42 52 >>>>>>>> fax: 33 1 69 33 49 49 >>>>>>>> email: >>>>>>>> URL : http://www.cpht.polytechnique.fr/cpht/amari >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> -------------------------------------------- >>>>>> T. Amari >>>>>> Centre de Physique Theorique >>>>>> Ecole Polytechnique >>>>>> 91128 Palaiseau Cedex France >>>>>> tel : 33 1 69 33 42 52 >>>>>> fax: 33 1 69 33 49 49 >>>>>> email: >>>>>> URL : http://www.cpht.polytechnique.fr/cpht/amari >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>> >>>> -------------------------------------------- >>>> T. Amari >>>> Centre de Physique Theorique >>>> Ecole Polytechnique >>>> 91128 Palaiseau Cedex France >>>> tel : 33 1 69 33 42 52 >>>> fax: 33 1 69 33 49 49 >>>> email: >>>> URL : http://www.cpht.polytechnique.fr/cpht/amari >>>> >>>> >>>> >>>> >>>> >> >> -------------------------------------------- >> T. Amari >> Centre de Physique Theorique >> Ecole Polytechnique >> 91128 Palaiseau Cedex France >> tel : 33 1 69 33 42 52 >> fax: 33 1 69 33 49 49 >> email: >> URL : http://www.cpht.polytechnique.fr/cpht/amari >> >> >> >> >> > -------------------------------------------- T. Amari Centre de Physique Theorique Ecole Polytechnique 91128 Palaiseau Cedex France tel : 33 1 69 33 42 52 fax: 33 1 69 33 49 49 email: URL : http://www.cpht.polytechnique.fr/cpht/amari -------------- next part -------------- An HTML attachment was scrubbed... URL: From amari at cpht.polytechnique.fr Fri Nov 12 15:31:25 2010 From: amari at cpht.polytechnique.fr (Tahar Amari) Date: Fri, 12 Nov 2010 22:31:25 +0100 Subject: [petsc-users] unstructured mesh In-Reply-To: References: <021A6584-4DD7-44D0-BC9C-977E1DA737F9@cpht.polytechnique.fr> <6501936B-4961-4C2E-A491-62F0A3655416@cpht.polytechnique.fr> <2E4B4874-EC33-4DAC-91E1-0E188FBFD2ED@cpht.polytechnique.fr> <0E36C9ED-9082-4302-91B6-023DE9A7367A@cpht.polytechnique.fr> Message-ID: <63C50A85-FA14-400B-B544-1EAF7C7C3D49@cpht.polytechnique.fr> Hello, I have written a C/C++ code on unstructured mesh (tetrahedra) , using pthreads but not MPI. 1 Is it a good idea to use petsc as a framework as an easier high level library for interface to MPI in order to parallelize the code 2. If yes. I have fields on various location on my mesh ; some variables are defined at cell center, otheres on edges, and others on faces. Does petsc allow to avoid to deal with low level MPI routines to get and set the ghost values between partitions to evaluate some functions such as fluxes at cell faces which needs different variables which may lie on different partitions ? (I have seen on the talk http://www.mcs.anl.gov/petsc/petsc-as/documentation/tutorials/ARSC2010.pdf page 81 some code lines which suggest that it is simple /? Code to communicate nonlocal ghost point data ?/ VecGetArray(F, &f ); /? Code to compute local function components ?/ VecRestoreArray(F, &f ); return 0; Would it be possible to get some advice please ? Thanks Tahar -------------------------------------------- T. Amari Centre de Physique Theorique Ecole Polytechnique 91128 Palaiseau Cedex France tel : 33 1 69 33 42 52 fax: 33 1 69 33 49 49 email: URL : http://www.cpht.polytechnique.fr/cpht/amari -------------- next part -------------- An HTML attachment was scrubbed... URL: From amari at cpht.polytechnique.fr Fri Nov 12 17:49:23 2010 From: amari at cpht.polytechnique.fr (Tahar Amari) Date: Sat, 13 Nov 2010 00:49:23 +0100 Subject: [petsc-users] unstructured mesh followed In-Reply-To: References: <021A6584-4DD7-44D0-BC9C-977E1DA737F9@cpht.polytechnique.fr> <6501936B-4961-4C2E-A491-62F0A3655416@cpht.polytechnique.fr> <2E4B4874-EC33-4DAC-91E1-0E188FBFD2ED@cpht.polytechnique.fr> <0E36C9ED-9082-4302-91B6-023DE9A7367A@cpht.polytechnique.fr> Message-ID: Hello, Reading the doc files, I saw that the DA frameworks contains very interesting functions which allow to specify various stencils for ghost cells and fields. I read that those cannot be used for unstructured meshes. HOwever is there some equivalent for unstructured mesh. This would be very useful. Tahar -------------------------------------------- T. Amari Centre de Physique Theorique Ecole Polytechnique 91128 Palaiseau Cedex France tel : 33 1 69 33 42 52 fax: 33 1 69 33 49 49 email: URL : http://www.cpht.polytechnique.fr/cpht/amari -------------- next part -------------- An HTML attachment was scrubbed... URL: From u.tabak at tudelft.nl Sun Nov 14 10:40:45 2010 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sun, 14 Nov 2010 17:40:45 +0100 Subject: [petsc-users] minres and cholesky or icc preconditioner Message-ID: <4CE0110D.40802@tudelft.nl> Dear all, I am trying to solve some linear systems with minres where the operator matrices can become indefinite depending on \lambda and moreover both the operator matrices and the right hand side is changing in a loop. A = K - \lambda M; I used the basic example, ex1.c, and tried supply some command line arguments to see what is going on for the linear solves with minres. Use of -pc_type cholesky or icc does not return an error however I could not understand this because the operator matrix is not PD(symmetric however) so I did not get how cholesky or icc might be used as a preconditioner in this case. However, the results are fine, but the iteration numbers are a bit high maybe I should play with the tolerance, because I am not looking for the very close approximate solution, a coarse representation of the solution might also help me. This is because I would like to use these solution vectors in some other context mixed with some other vectors. And the other question is that backslash of MATLAB solves these systems faster than umfpack or mumps interface of PETSc, is this due to the fact that my PETSc configuration is the default debug one(I read somewhere in the documentation some time ago, mentioning that there can be important performance differences, but could not locate that right now.)? I would appreciate clarification on these points. Best, Umut -- - Hope is a good thing, maybe the best of things and no good thing ever dies... The Shawshank Redemption, replique of Tim Robbins From bsmith at mcs.anl.gov Sun Nov 14 11:26:01 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 14 Nov 2010 11:26:01 -0600 Subject: [petsc-users] minres and cholesky or icc preconditioner In-Reply-To: <4CE0110D.40802@tudelft.nl> References: <4CE0110D.40802@tudelft.nl> Message-ID: On Nov 14, 2010, at 10:40 AM, Umut Tabak wrote: > Dear all, > > I am trying to solve some linear systems with minres where the operator matrices can become indefinite depending on \lambda and moreover both the operator matrices and the right hand side is changing in a loop. > > A = K - \lambda M; > > I used the basic example, ex1.c, and tried supply some command line arguments to see what is going on for the linear solves with minres. > > Use of -pc_type cholesky or icc does not return an error however I could not understand this because the operator matrix is not PD(symmetric however) so I did not get how cholesky or icc might be used as a preconditioner in this case. By default our ICC automatically shifts the diagonal as needed to force the preconditioner to be positive definite; if you run with -info it will print information about that. For Cholesky so long as non of the pivots are zero it will complete the factorization and produce a good solver. > However, the results are fine, but the iteration numbers are a bit high maybe I should play with the tolerance, because I am not looking for the very close approximate solution, a coarse representation of the solution might also help me. This is because I would like to use these solution vectors in some other context mixed with some other vectors. > > And the other question is that backslash of MATLAB solves these systems faster than umfpack or mumps interface of PETSc, is this due to the fact that my PETSc configuration is the default debug one(I read somewhere in the documentation some time ago, mentioning that there can be important performance differences, but could not locate that right now.)? It could be that. You can do ANOTHER ./configure --with-debugging=0 using a different PETSC_ARCH to build another set of libraries to compare with. It could also be that the time you are counting with PETSc includes the time to generate the matrix or compute other things as well. Barry > > I would appreciate clarification on these points. > > Best, > Umut > > -- > - Hope is a good thing, maybe the best of things > and no good thing ever dies... > The Shawshank Redemption, replique of Tim Robbins > From u.tabak at tudelft.nl Sun Nov 14 11:29:16 2010 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sun, 14 Nov 2010 18:29:16 +0100 Subject: [petsc-users] minres and cholesky or icc preconditioner In-Reply-To: References: <4CE0110D.40802@tudelft.nl> Message-ID: <4CE01C6C.4060304@tudelft.nl> On 11/14/2010 06:26 PM, Barry Smith wrote: > On Nov 14, 2010, at 10:40 AM, Umut Tabak wrote: > > It could be that. You can do ANOTHER ./configure --with-debugging=0 using a different PETSC_ARCH to build another set of libraries to compare with. It could also be that the time you are counting with PETSc includes the time to generate the matrix or compute other things as well. > > Barry > > I read the matrices in binary format, I guess this is a pretty small cost. I will try to use the --with-debugging=0 option to build a faster version. Thx for the cholesky overview. Umut From elbueler at alaska.edu Sun Nov 14 13:46:04 2010 From: elbueler at alaska.edu (Edward Bueler) Date: Sun, 14 Nov 2010 10:46:04 -0900 Subject: [petsc-users] dof>1 vector tools Message-ID: Dear PETSc-- This is an actual user request. I am working on a typical PETSc task involving two nonlinear PDEs coupled together. I notice that there are several tools for Vecs which don't help much when I have a dof>1 Vec from my DA, for which the components have different scaling. Thus VecNorm and VecScale are not really what I want. Is there a plan for component-wise versions of those? For example, is there a better/quicker way to scale one component of a dof>1 DA-derived Vec than writing my own, something like this: typedef struct { PetscReal H, u; } Node; ierr = DACreate1d(PETSC_COMM_WORLD,DA_NONPERIODIC,-M,2,1,PETSC_NULL,&da);CHKERRQ(ierr); ierr = DACreateGlobalVector(da,&v);CHKERRQ(ierr); ... Node *n; ierr = DAVecGetArray(da,v,&n);CHKERRQ(ierr); for (i = xs; i < xs + xm; i++) { n[i].u *= alpha; } ierr = DAVecRestoreArray(da,v,&n);CHKERRQ(ierr); I guess I would want the loop to be replaced by "DAVecScaleDOF(da,v,k,alpha)" for k=0,...,dof-1. Same question for VecNorm: norm just an identified component? Ed -- ********************************** Ed Bueler 474-7693 (fax 474-5394) Dept of Mathematics and Statistics 301C Chapman Hall, Univ of Alaska Fairbanks, AK 99775-6660 www.dms.uaf.edu/~bueler ********************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Sun Nov 14 14:15:41 2010 From: jed at 59A2.org (Jed Brown) Date: Sun, 14 Nov 2010 21:15:41 +0100 Subject: [petsc-users] dof>1 vector tools In-Reply-To: References: Message-ID: On Sun, Nov 14, 2010 at 20:46, Edward Bueler wrote: > Dear PETSc-- > > This is an actual user request. I am working on a typical PETSc task > involving two nonlinear PDEs coupled together. I notice that there are > several tools for Vecs which don't help much when I have a dof>1 Vec from my > DA, for which the components have different scaling. Thus VecNorm and > VecScale are not really what I want. Is there a plan for component-wise > versions of those? > > For example, is there a better/quicker way to scale one component of a > dof>1 DA-derived Vec than writing my own, something like this: > > typedef struct { > PetscReal H, u; > } Node; > ierr = > DACreate1d(PETSC_COMM_WORLD,DA_NONPERIODIC,-M,2,1,PETSC_NULL,&da);CHKERRQ(ierr); > ierr = DACreateGlobalVector(da,&v);CHKERRQ(ierr); > ... > Node *n; > ierr = DAVecGetArray(da,v,&n);CHKERRQ(ierr); > for (i = xs; i < xs + xm; i++) { > n[i].u *= alpha; > } > ierr = DAVecRestoreArray(da,v,&n);CHKERRQ(ierr); > > I guess I would want the loop to be replaced by > "DAVecScaleDOF(da,v,k,alpha)" for k=0,...,dof-1. > It's spelled VecStrideScale(). Same question for VecNorm: norm just an identified component? > You guessed it, VecStrideNorm() But, I actually recommend nondimensionalizing (in the sense of ex48, not in terms on nondimensional numbers which tend to have imprecise definitions for complex geometry or multi-physics) so that these component norms are not needed by the solvers, as the nondimensional system will generally produce a better conditioned algebraic system. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Mon Nov 15 03:54:50 2010 From: zonexo at gmail.com (TAY wee-beng) Date: Mon, 15 Nov 2010 10:54:50 +0100 Subject: [petsc-users] Help with "make: m2c: Command not found" In-Reply-To: References: <021A6584-4DD7-44D0-BC9C-977E1DA737F9@cpht.polytechnique.fr> <6501936B-4961-4C2E-A491-62F0A3655416@cpht.polytechnique.fr> <2E4B4874-EC33-4DAC-91E1-0E188FBFD2ED@cpht.polytechnique.fr> <0E36C9ED-9082-4302-91B6-023DE9A7367A@cpht.polytechnique.fr> Message-ID: <4CE1036A.4080802@gmail.com> Hi, I have a modified makefile originally given to me by the PETSc team. It worked well but there's a small problem. Sometimes, when I have edited a fortran file and tries to rebuild, it gives this error: /opt/openmpi-1.4.1/bin/mpif90 -r8 -w95 -c -O3 -save airfoil.f90 /opt/openmpi-1.4.1/bin/mpif90 -w95 -c -O3 -save grid.f90 m2c -o cell_data.o cell_data.mod make: m2c: Command not found make: *** [cell_data.o] Error 127 However, if I delete all the *.o and *.mod files and build clean, there's no such problems. Is there anyway to solve this problem? I have attached the makefile Thanks! Yours sincerely, TAY wee-beng >> >> -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: makefile URL: From s.kramer at imperial.ac.uk Mon Nov 15 05:41:42 2010 From: s.kramer at imperial.ac.uk (Stephan Kramer) Date: Mon, 15 Nov 2010 11:41:42 +0000 Subject: [petsc-users] Help with "make: m2c: Command not found" In-Reply-To: <4CE1036A.4080802@gmail.com> References: <021A6584-4DD7-44D0-BC9C-977E1DA737F9@cpht.polytechnique.fr> <6501936B-4961-4C2E-A491-62F0A3655416@cpht.polytechnique.fr> <2E4B4874-EC33-4DAC-91E1-0E188FBFD2ED@cpht.polytechnique.fr> <0E36C9ED-9082-4302-91B6-023DE9A7367A@cpht.polytechnique.fr> <4CE1036A.4080802@gmail.com> Message-ID: <4CE11C76.2090306@imperial.ac.uk> On 15/11/10 09:54, TAY wee-beng wrote: > Hi, > > I have a modified makefile originally given to me by the PETSc team. It > worked well but there's a small problem. Sometimes, when I have edited a > fortran file and tries to rebuild, it gives this error: > > /opt/openmpi-1.4.1/bin/mpif90 -r8 -w95 -c -O3 -save airfoil.f90 > /opt/openmpi-1.4.1/bin/mpif90 -w95 -c -O3 -save grid.f90 > m2c -o cell_data.o cell_data.mod > make: m2c: Command not found > make: *** [cell_data.o] Error 127 > > However, if I delete all the *.o and *.mod files and build clean, > there's no such problems. Is there anyway to solve this problem? > > I have attached the makefile > > Thanks! > > Yours sincerely, > > TAY wee-beng > > >>> >>> This is caused by a built-in implicit make rule that tells make it can make object files from modula-2 .mod files by running m2c - obviously not what you want. You can override it by adding the following line to your makefile: %.o: %.mod Make sure the next line doesn't start with a tab. Cheers Stephan From chenleping at yahoo.cn Mon Nov 15 05:52:34 2010 From: chenleping at yahoo.cn (=?gb2312?B?s8LA1sa9?=) Date: Mon, 15 Nov 2010 19:52:34 +0800 (CST) Subject: [petsc-users] I want to quit PETSc users list Message-ID: <327956.14882.qm@web92407.mail.cnh.yahoo.com> I want to quit PETSc users list. Thank you! ________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Nov 15 07:34:11 2010 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 15 Nov 2010 07:34:11 -0600 (CST) Subject: [petsc-users] I want to quit PETSc users list In-Reply-To: <327956.14882.qm@web92407.mail.cnh.yahoo.com> References: <327956.14882.qm@web92407.mail.cnh.yahoo.com> Message-ID: Done unsubscribing.. Just note: All list users can manage their subscription info directly. The relavent info is in the header of each list e-mail. List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Satish On Mon, 15 Nov 2010, ??? wrote: > I want to quit PETSc users list. > > Thank you! > > > > ________________________________ > > > From balay at mcs.anl.gov Mon Nov 15 13:28:19 2010 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 15 Nov 2010 13:28:19 -0600 (CST) Subject: [petsc-users] sending configure.log to mailing lists In-Reply-To: References: Message-ID: On Mon, 15 Nov 2010, Srinath Vadlamani wrote: > Thanks for checking. The team does want configure.log with all help > requests, correct? yeah - all issues requiring configure.log should be sent to petsc-maint. There is some confusion regarding public mailing lists [petsc-dev, petsc-users] and private list [petsc-maint]. Here is some clarification: 1. To reach petsc-developers - any of the above e-mail ids are equivalent, although petsc-dev is supporsed to be for discussions on development version of petsc. 2. We do not want to flood inboxes of all mailing list subscribers with multi-megabyte configure.log or similar attachments. For this reason - any issues that involve configure.log [esp installation issues] are best sent to petsc-maint. 3. Mailing lists [petsc-dev,petsc-users] need subscription - but no such thing for petsc-maint]. Also mailing list emails reach all folks - while petsc-maint reaches only developers. And mailing lists conversation is archived and accessiable by all for future searches. 3. We do want to keep both the public and private conversation channels open - hence the existance of mailing lists as well as petsc-maint. Satish From elbueler at alaska.edu Mon Nov 15 17:38:37 2010 From: elbueler at alaska.edu (Edward Bueler) Date: Mon, 15 Nov 2010 14:38:37 -0900 Subject: [petsc-users] dof>1 vector tools Message-ID: Jed-- > ... > > I guess I would want the loop to be replaced by > > "DAVecScaleDOF(da,v,k,alpha)" for k=0,...,dof-1. > > It's spelled VecStrideScale(). > > > Same question for VecNorm: norm just an identified component? > > You guessed it, VecStrideNorm() RIght! Sorry to miss that. Didn't search on "Stride". > But, I actually recommend nondimensionalizing ... so that these component > norms are not needed by the solvers, as the nondimensional system will > generally produce a better conditioned algebraic system. Did not need the norm for the solvers, actually. I wrote a scaling/descaling "shell" around the residual evaluation routine, so the latter could be dimensional. At this early, prototype stage, where I am at on my code, I don't want the "clutter" of nondimensionalizing PDEs if I am familiar with them in dimensional units. > ... nondimensionalizing (in the sense of ex48, not in terms on > nondimensional numbers ...) ... ex48.c is heavy-going if what one wants is an example showing how PETSc can help with scaling the residual and Jacobian for better conditioning. Related question: Is it true that an instance of DALocalFunction1 will cause trouble if it modifies the solution array? (The values supplied in the "Field *u" argument?) It seems that this is so, which I did not expect for this call-back. Related comment: There is no man page on DALocalFunction1, I think. Thanks! Ed -- ********************************** Ed Bueler 474-7693? ?? (fax 474-5394) Dept of Mathematics and Statistics 301C Chapman Hall, Univ of Alaska Fairbanks, AK 99775-6660 www.dms.uaf.edu/~bueler ********************************** From jed at 59A2.org Tue Nov 16 06:28:37 2010 From: jed at 59A2.org (Jed Brown) Date: Tue, 16 Nov 2010 13:28:37 +0100 Subject: [petsc-users] dof>1 vector tools In-Reply-To: References: Message-ID: On Tue, Nov 16, 2010 at 00:38, Edward Bueler wrote: > Did not need the norm for the solvers, actually. > > I wrote a scaling/descaling "shell" around the residual evaluation > routine, so the latter could be dimensional. At this early, prototype > stage, where I am at on my code, I don't want the "clutter" of > nondimensionalizing PDEs if I am familiar with them in dimensional > units. > Understood, I just type printf("velocity %g m/s\n",velocity * units->meter/units->second); which I find fairly intuitive. > > ... nondimensionalizing (in the sense of ex48, not in terms on > > nondimensional numbers ...) ... > > ex48.c is heavy-going if what one wants is an example showing how > PETSc can help with scaling the residual and Jacobian for better > conditioning. > Understood, a slightly shorter code, albeit with more complicated physics, is in src/ts/examples/tutorials/ex10.c (only petsc-dev). I'll keep your comment in mind and make a simple example handling dimensional issues at some point. > Related question: Is it true that an instance of DALocalFunction1 > will cause trouble if it modifies the solution array? (The values > supplied in the "Field *u" argument?) It seems that this is so, which > I did not expect for this call-back. > In general, this is not right/doesn't make sense. It certainly causes problems with finite differencing. Are you asking about enforcing special constraints (like positivity)? The right thing is probably to use the variational inequality stuff in petsc-dev, but I think it would be useful for PETSc to make a clear statement about if and when the "simple hack" of projecting the trial state into the feasible set is acceptable. It may need to be implemented as a separate callback to work correctly with TS. > Related comment: There is no man page on DALocalFunction1, I think. > Thanks for pointing this out, DASetLocalFunction does not even document the calling convention (though it is in the user's manual, mildly obfuscated). DALocalFunction1 is a funny beast that does not distinguish between the LocalFunction and LocalJacobian calling conventions (both just void*). The use of void* is because the user typically has a multi-component problem in which case arrays of PetscScalar are clunky since they want arrays of some struct, thus always requiring an explicit cast. But passing a Mat in through the same void* is just disgusting. As a petsc-dev issue, now that SNES holds a reference to the DM, perhaps SNES should hold the "local" function pointer. I think it's not so weird to want to solve different systems with the same DM. Having access to subdomain residuals and Jacobians would be useful for less synchronous nonlinear solvers. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Wed Nov 17 07:45:51 2010 From: zonexo at gmail.com (TAY wee-beng) Date: Wed, 17 Nov 2010 14:45:51 +0100 Subject: [petsc-users] Help with "make: m2c: Command not found" In-Reply-To: <4CE11C76.2090306@imperial.ac.uk> References: <021A6584-4DD7-44D0-BC9C-977E1DA737F9@cpht.polytechnique.fr> <6501936B-4961-4C2E-A491-62F0A3655416@cpht.polytechnique.fr> <2E4B4874-EC33-4DAC-91E1-0E188FBFD2ED@cpht.polytechnique.fr> <0E36C9ED-9082-4302-91B6-023DE9A7367A@cpht.polytechnique.fr> <4CE1036A.4080802@gmail.com> <4CE11C76.2090306@imperial.ac.uk> Message-ID: <4CE3DC8F.4030100@gmail.com> Hi Stephan, Thanks, it worked! Yours sincerely, TAY wee-beng On 15-11-2010 12:41, Stephan Kramer wrote: > On 15/11/10 09:54, TAY wee-beng wrote: >> Hi, >> >> I have a modified makefile originally given to me by the PETSc team. It >> worked well but there's a small problem. Sometimes, when I have edited a >> fortran file and tries to rebuild, it gives this error: >> >> /opt/openmpi-1.4.1/bin/mpif90 -r8 -w95 -c -O3 -save airfoil.f90 >> /opt/openmpi-1.4.1/bin/mpif90 -w95 -c -O3 -save grid.f90 >> m2c -o cell_data.o cell_data.mod >> make: m2c: Command not found >> make: *** [cell_data.o] Error 127 >> >> However, if I delete all the *.o and *.mod files and build clean, >> there's no such problems. Is there anyway to solve this problem? >> >> I have attached the makefile >> >> Thanks! >> >> Yours sincerely, >> >> TAY wee-beng >> >> >>>> >>>> > > This is caused by a built-in implicit make rule that tells make it can > make object files > from modula-2 .mod files by running m2c - obviously not what you want. > You can override it by adding the following > line to your makefile: > > %.o: %.mod > > Make sure the next line doesn't start with a tab. > > Cheers > Stephan From zonexo at gmail.com Wed Nov 17 08:04:53 2010 From: zonexo at gmail.com (TAY wee-beng) Date: Wed, 17 Nov 2010 15:04:53 +0100 Subject: [petsc-users] Adding adaptive mesh refinement to current fortran code Message-ID: <4CE3E105.2090509@gmail.com> Hi, I am thinking of adding adaptive mesh refinement to my current fortran code. I am using an immersed boundary code with staggered cartesian grid. I am using PETSc to solve my momentum equations and PETSc/hypre to solve my poisson equation. The code now runs in parallel. Since I'm programming in fortran, some other c++ packages may not be feasible. I am considering packages like paramesh and AGRIF. I wonder if it is possible to integrate them together with PETSc. Is there anything which I need to consider too? Thanks! -- Yours sincerely, TAY wee-beng From Chun.SUN at 3ds.com Wed Nov 17 09:48:38 2010 From: Chun.SUN at 3ds.com (SUN Chun) Date: Wed, 17 Nov 2010 15:48:38 +0000 Subject: [petsc-users] GPU questions In-Reply-To: <3AB51330CF1300428AD80035DAF20EE30161F6@AG-DCC-MBX03.dsone.3ds.com> References: <4CDADA3A.6020103@gmail.com> <3AB51330CF1300428AD80035DAF20EE30161F6@AG-DCC-MBX03.dsone.3ds.com> Message-ID: <3AB51330CF1300428AD80035DAF20EE3016A39@AG-DCC-MBX03.dsone.3ds.com> Hi PETSc developers, I have some questions regarding GPGPU support in PETSc. Sorry if these questions are redundant, I didn't browse the dev code too carefully... 1. The only example I can find is tutorial/ex47. Is there a plan to provide more examples involving KSP with, say, simple Jacobi PC? I thought KSP is supported but PC is not. 2. Would you please comment on the difficulty in supporting PCs? Like ILU, SSOR.... Would you please also comment on the difficulty in supporting external libraries such as ML? 3. I noticed (might be wrong), MatMult in CUDA is implemented in such a way that we copy the lhs and rhs to GPU each time before we do MatVec. I understand that you may have to do this to ensure MatMult being robust, but I'm worried about performance. Is it possible, say, like in KSP, we keep the lhs and intermediate results on the GPU side? 4. Any performance data on Tesla/Fermi card? I saw from the webpage you only have Tesla card? 5. Is there a roadmap, a plan, a timeline..., regarding PETSc and nVidia's collaboration for a final fully GPGPU compatible PETSc? Thanks a lot for your time! Chun This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged. If you are not one of the named recipients or have received this email in error, (i) you should not read, disclose, or copy it, (ii) please notify sender of your receipt by reply email and delete this email and all attachments, (iii) Dassault Systemes does not accept or assume any liability or responsibility for any use of or reliance on this email.For other languages, go to http://www.3ds.com/terms/email-disclaimer. From bsmith at mcs.anl.gov Wed Nov 17 11:28:27 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 17 Nov 2010 11:28:27 -0600 Subject: [petsc-users] Adding adaptive mesh refinement to current fortran code In-Reply-To: <4CE3E105.2090509@gmail.com> References: <4CE3E105.2090509@gmail.com> Message-ID: libmesh has adaptive refinement and all kinds of cool stuff and lives on top of PETSc. Barry But it is C++. On Nov 17, 2010, at 8:04 AM, TAY wee-beng wrote: > Hi, > > I am thinking of adding adaptive mesh refinement to my current fortran code. I am using an immersed boundary code with staggered cartesian grid. I am using PETSc to solve my momentum equations and PETSc/hypre to solve my poisson equation. The code now runs in parallel. > > Since I'm programming in fortran, some other c++ packages may not be feasible. I am considering packages like paramesh and AGRIF. I wonder if it is possible to integrate them together with PETSc. Is there anything which I need to consider too? > > Thanks! > > -- > Yours sincerely, > > TAY wee-beng > From bsmith at mcs.anl.gov Wed Nov 17 11:41:11 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 17 Nov 2010 11:41:11 -0600 Subject: [petsc-users] GPU questions In-Reply-To: <3AB51330CF1300428AD80035DAF20EE3016A39@AG-DCC-MBX03.dsone.3ds.com> References: <4CDADA3A.6020103@gmail.com> <3AB51330CF1300428AD80035DAF20EE30161F6@AG-DCC-MBX03.dsone.3ds.com> <3AB51330CF1300428AD80035DAF20EE3016A39@AG-DCC-MBX03.dsone.3ds.com> Message-ID: On Nov 17, 2010, at 9:48 AM, SUN Chun wrote: > Hi PETSc developers, > > I have some questions regarding GPGPU support in PETSc. Sorry if these questions are redundant, I didn't browse the dev code too carefully... > > 1. The only example I can find is tutorial/ex47. Is there a plan to provide more examples involving KSP with, say, simple Jacobi PC? I thought KSP is supported but PC is not. Any example that uses DAGetMatrix(), DAGetGlobalVector() or DMMG can be run with -da_vec_type cuda -da_mat_type aijcuda to use the GPUS The whole idea behind PETSc is to have the SAME code for different solvers and different systems so we will not have a bunch of examples just for GPUs. > > 2. Would you please comment on the difficulty in supporting PCs? Like ILU, SSOR.... Would you please also comment on the difficulty in supporting external libraries such as ML? We don't have code for triangular solves on the GPU, without those ILU and SSOR cannot run on GPUs. Once someone provides triangular solves for GPUs we can add there use and put ILU and SSOR onto the GPUs with PETSc. Regarding ML that is totally up their developers. Note that Nvidia has a smooth agglomaration algorithm for symmetric problems in CUSP that you can access via PETSc as the PCSACUDA PC (not yet stable, so possibly bugs). > > 3. I noticed (might be wrong), MatMult in CUDA is implemented in such a way that we copy the lhs and rhs to GPU each time before we do MatVec. I understand that you may have to do this to ensure MatMult being robust, but I'm worried about performance. Is it possible, say, like in KSP, we keep the lhs and intermediate results on the GPU side? Look at the code more closes. VecCUDACopyToGPU() (also the MatCUDACopy...) ONLY copy down if the values are NOT already on the GPU. This means once the vectors are on the GPU they remain there and are NOT copied back and forth for each multiply. ierr = MatCUDACopyToGPU(A);CHKERRQ(ierr); ierr = VecCUDACopyToGPU(xx);CHKERRQ(ierr); ierr = VecCUDAAllocateCheck(yy);CHKERRQ(ierr); if (usecprow){ /* use compressed row format */ try { cusp::multiply(*cudastruct->mat,*((Vec_CUDA *)xx->spptr)->GPUarray,*cudastruct->tempvec); ierr = VecSet_SeqCUDA(yy,0.0);CHKERRQ(ierr); thrust::copy(cudastruct->tempvec->begin(),cudastruct->tempvec->end(),thrust::make_permutation_iterator(((Vec_CUDA *)yy->spptr)->GPUarray->begin(),cudastruct->indices->begin())); } catch (char* ex) { SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"CUDA error: %s", ex); } } else { /* do not use compressed row format */ try { cusp::multiply(*cudastruct->mat,*((Vec_CUDA *)xx->spptr)->GPUarray,*((Vec_CUDA *)yy->spptr)->GPUarray); } catch(char* ex) { SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"CUDA error: %s", ex); } } yy->valid_GPU_array = PETSC_CUDA_GPU; ierr = WaitForGPU();CHKERRCUDA(ierr); > > 4. Any performance data on Tesla/Fermi card? I saw from the webpage you only have Tesla card? We don't have good hardware for running benchmarks. The performance is better than just running on the CPU, that is about all I can say. > > 5. Is there a roadmap, a plan, a timeline..., regarding PETSc and nVidia's collaboration for a final fully GPGPU compatible PETSc? What do you mean by final fully GPGPU compatible PETSc? Now some things (vector operations, matrix vector products, Krylov solvers) are fully done on the GPUs. Others are automatically done on the CPU. I imagine it will always be this way, I double ever that EVERYTHING will be done on the GPU but that is ok so long as most things are done on the GPU. If you run with -log_summary it will tell how how many copy to GPUs and copy from GPUs are done in the run and how much time they take. Obviously one wants that number as low as possible. Barry > > > Thanks a lot for your time! > Chun > > > This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged. If you are not one of the named recipients or have received this email in error, (i) you should not read, disclose, or copy it, (ii) please notify sender of your receipt by reply email and delete this email and all attachments, (iii) Dassault Systemes does not accept or assume any liability or responsibility for any use of or reliance on this email.For other languages, go to http://www.3ds.com/terms/email-disclaimer. From Chun.SUN at 3ds.com Wed Nov 17 12:23:41 2010 From: Chun.SUN at 3ds.com (SUN Chun) Date: Wed, 17 Nov 2010 18:23:41 +0000 Subject: [petsc-users] GPU questions In-Reply-To: References: <4CDADA3A.6020103@gmail.com><3AB51330CF1300428AD80035DAF20EE30161F6@AG-DCC-MBX03.dsone.3ds.com><3AB51330CF1300428AD80035DAF20EE3016A39@AG-DCC-MBX03.dsone.3ds.com> Message-ID: <3AB51330CF1300428AD80035DAF20EE3016AB9@AG-DCC-MBX03.dsone.3ds.com> Hi Barry, Thanks for your complete reply. Everything is clear w.r.t. my previous questions. My understanding is the following: As long as I create matrix with type MPIAIJCUDA in the same way as I do with MPIAIJ, it should be performing Mat, Vec, and Ksp on the GPU side, while everything else on the CPU side? (So my code doesn't change except for the MatCreateMPIAIJ -> MatCreateMPIAIJCUDA) Thanks, Chun -----Original Message----- From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Barry Smith Sent: Wednesday, November 17, 2010 12:41 PM To: PETSc users list Subject: Re: [petsc-users] GPU questions On Nov 17, 2010, at 9:48 AM, SUN Chun wrote: > Hi PETSc developers, > > I have some questions regarding GPGPU support in PETSc. Sorry if these questions are redundant, I didn't browse the dev code too carefully... > > 1. The only example I can find is tutorial/ex47. Is there a plan to provide more examples involving KSP with, say, simple Jacobi PC? I thought KSP is supported but PC is not. Any example that uses DAGetMatrix(), DAGetGlobalVector() or DMMG can be run with -da_vec_type cuda -da_mat_type aijcuda to use the GPUS The whole idea behind PETSc is to have the SAME code for different solvers and different systems so we will not have a bunch of examples just for GPUs. > > 2. Would you please comment on the difficulty in supporting PCs? Like ILU, SSOR.... Would you please also comment on the difficulty in supporting external libraries such as ML? We don't have code for triangular solves on the GPU, without those ILU and SSOR cannot run on GPUs. Once someone provides triangular solves for GPUs we can add there use and put ILU and SSOR onto the GPUs with PETSc. Regarding ML that is totally up their developers. Note that Nvidia has a smooth agglomaration algorithm for symmetric problems in CUSP that you can access via PETSc as the PCSACUDA PC (not yet stable, so possibly bugs). > > 3. I noticed (might be wrong), MatMult in CUDA is implemented in such a way that we copy the lhs and rhs to GPU each time before we do MatVec. I understand that you may have to do this to ensure MatMult being robust, but I'm worried about performance. Is it possible, say, like in KSP, we keep the lhs and intermediate results on the GPU side? Look at the code more closes. VecCUDACopyToGPU() (also the MatCUDACopy...) ONLY copy down if the values are NOT already on the GPU. This means once the vectors are on the GPU they remain there and are NOT copied back and forth for each multiply. ierr = MatCUDACopyToGPU(A);CHKERRQ(ierr); ierr = VecCUDACopyToGPU(xx);CHKERRQ(ierr); ierr = VecCUDAAllocateCheck(yy);CHKERRQ(ierr); if (usecprow){ /* use compressed row format */ try { cusp::multiply(*cudastruct->mat,*((Vec_CUDA *)xx->spptr)->GPUarray,*cudastruct->tempvec); ierr = VecSet_SeqCUDA(yy,0.0);CHKERRQ(ierr); thrust::copy(cudastruct->tempvec->begin(),cudastruct->tempvec->end(),thrust::make_permutation_iterator(((Vec_CUDA *)yy->spptr)->GPUarray->begin(),cudastruct->indices->begin())); } catch (char* ex) { SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"CUDA error: %s", ex); } } else { /* do not use compressed row format */ try { cusp::multiply(*cudastruct->mat,*((Vec_CUDA *)xx->spptr)->GPUarray,*((Vec_CUDA *)yy->spptr)->GPUarray); } catch(char* ex) { SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"CUDA error: %s", ex); } } yy->valid_GPU_array = PETSC_CUDA_GPU; ierr = WaitForGPU();CHKERRCUDA(ierr); > > 4. Any performance data on Tesla/Fermi card? I saw from the webpage you only have Tesla card? We don't have good hardware for running benchmarks. The performance is better than just running on the CPU, that is about all I can say. > > 5. Is there a roadmap, a plan, a timeline..., regarding PETSc and nVidia's collaboration for a final fully GPGPU compatible PETSc? What do you mean by final fully GPGPU compatible PETSc? Now some things (vector operations, matrix vector products, Krylov solvers) are fully done on the GPUs. Others are automatically done on the CPU. I imagine it will always be this way, I double ever that EVERYTHING will be done on the GPU but that is ok so long as most things are done on the GPU. If you run with -log_summary it will tell how how many copy to GPUs and copy from GPUs are done in the run and how much time they take. Obviously one wants that number as low as possible. Barry > > > Thanks a lot for your time! > Chun > > > This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged. If you are not one of the named recipients or have received this email in error, (i) you should not read, disclose, or copy it, (ii) please notify sender of your receipt by reply email and delete this email and all attachments, (iii) Dassault Systemes does not accept or assume any liability or responsibility for any use of or reliance on this email.For other languages, go to http://www.3ds.com/terms/email-disclaimer. This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged. If you are not one of the named recipients or have received this email in error, (i) you should not read, disclose, or copy it, (ii) please notify sender of your receipt by reply email and delete this email and all attachments, (iii) Dassault Systemes does not accept or assume any liability or responsibility for any use of or reliance on this email.For other languages, go to http://www.3ds.com/terms/email-disclaimer. From bsmith at mcs.anl.gov Wed Nov 17 12:45:55 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 17 Nov 2010 12:45:55 -0600 Subject: [petsc-users] GPU questions In-Reply-To: <3AB51330CF1300428AD80035DAF20EE3016AB9@AG-DCC-MBX03.dsone.3ds.com> References: <4CDADA3A.6020103@gmail.com><3AB51330CF1300428AD80035DAF20EE30161F6@AG-DCC-MBX03.dsone.3ds.com><3AB51330CF1300428AD80035DAF20EE3016A39@AG-DCC-MBX03.dsone.3ds.com> <3AB51330CF1300428AD80035DAF20EE3016AB9@AG-DCC-MBX03.dsone.3ds.com> Message-ID: <5E1C5225-5F41-4021-B6CA-1EB91FA03033@mcs.anl.gov> On Nov 17, 2010, at 12:23 PM, SUN Chun wrote: > Hi Barry, > > Thanks for your complete reply. Everything is clear w.r.t. my previous questions. > > My understanding is the following: As long as I create matrix with type MPIAIJCUDA in the same way as I do with MPIAIJ, it should be performing Mat, Vec, and Ksp on the GPU side, while everything else on the CPU side? (So my code doesn't change except for the MatCreateMPIAIJ -> MatCreateMPIAIJCUDA) Yes, but currently only the MatMult() takes place on the GPU, all other matrix operations happen on the CPU. Barry > > Thanks, > Chun > > -----Original Message----- > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Barry Smith > Sent: Wednesday, November 17, 2010 12:41 PM > To: PETSc users list > Subject: Re: [petsc-users] GPU questions > > > On Nov 17, 2010, at 9:48 AM, SUN Chun wrote: > >> Hi PETSc developers, >> >> I have some questions regarding GPGPU support in PETSc. Sorry if these questions are redundant, I didn't browse the dev code too carefully... >> >> 1. The only example I can find is tutorial/ex47. Is there a plan to provide more examples involving KSP with, say, simple Jacobi PC? I thought KSP is supported but PC is not. > > Any example that uses DAGetMatrix(), DAGetGlobalVector() or DMMG can be run with -da_vec_type cuda -da_mat_type aijcuda to use the GPUS > > The whole idea behind PETSc is to have the SAME code for different solvers and different systems so we will not have a bunch of examples just for GPUs. > >> >> 2. Would you please comment on the difficulty in supporting PCs? Like ILU, SSOR.... Would you please also comment on the difficulty in supporting external libraries such as ML? > > We don't have code for triangular solves on the GPU, without those ILU and SSOR cannot run on GPUs. Once someone provides triangular solves for GPUs we can add there use and put ILU and SSOR onto the GPUs with PETSc. Regarding ML that is totally up their developers. Note that Nvidia has a smooth agglomaration algorithm for symmetric problems in CUSP that you can access via PETSc as the PCSACUDA PC (not yet stable, so possibly bugs). > >> >> 3. I noticed (might be wrong), MatMult in CUDA is implemented in such a way that we copy the lhs and rhs to GPU each time before we do MatVec. I understand that you may have to do this to ensure MatMult being robust, but I'm worried about performance. Is it possible, say, like in KSP, we keep the lhs and intermediate results on the GPU side? > > Look at the code more closes. VecCUDACopyToGPU() (also the MatCUDACopy...) ONLY copy down if the values are NOT already on the GPU. This means once the vectors are on the GPU they remain there and are NOT copied back and forth for each multiply. > > ierr = MatCUDACopyToGPU(A);CHKERRQ(ierr); > ierr = VecCUDACopyToGPU(xx);CHKERRQ(ierr); > ierr = VecCUDAAllocateCheck(yy);CHKERRQ(ierr); > if (usecprow){ /* use compressed row format */ > try { > cusp::multiply(*cudastruct->mat,*((Vec_CUDA *)xx->spptr)->GPUarray,*cudastruct->tempvec); > ierr = VecSet_SeqCUDA(yy,0.0);CHKERRQ(ierr); > thrust::copy(cudastruct->tempvec->begin(),cudastruct->tempvec->end(),thrust::make_permutation_iterator(((Vec_CUDA *)yy->spptr)->GPUarray->begin(),cudastruct->indices->begin())); > } catch (char* ex) { > SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"CUDA error: %s", ex); > } > } else { /* do not use compressed row format */ > try { > cusp::multiply(*cudastruct->mat,*((Vec_CUDA *)xx->spptr)->GPUarray,*((Vec_CUDA *)yy->spptr)->GPUarray); > } catch(char* ex) { > SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"CUDA error: %s", ex); > } > } > yy->valid_GPU_array = PETSC_CUDA_GPU; > ierr = WaitForGPU();CHKERRCUDA(ierr); > >> >> 4. Any performance data on Tesla/Fermi card? I saw from the webpage you only have Tesla card? > > We don't have good hardware for running benchmarks. The performance is better than just running on the CPU, that is about all I can say. > >> >> 5. Is there a roadmap, a plan, a timeline..., regarding PETSc and nVidia's collaboration for a final fully GPGPU compatible PETSc? > > What do you mean by final fully GPGPU compatible PETSc? Now some things (vector operations, matrix vector products, Krylov solvers) are fully done on the GPUs. Others are automatically done on the CPU. I imagine it will always be this way, I double ever that EVERYTHING will be done on the GPU but that is ok so long as most things are done on the GPU. If you run with -log_summary it will tell how how many copy to GPUs and copy from GPUs are done in the run and how much time they take. Obviously one wants that number as low as possible. > > Barry > >> >> >> Thanks a lot for your time! >> Chun >> >> >> This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged. If you are not one of the named recipients or have received this email in error, (i) you should not read, disclose, or copy it, (ii) please notify sender of your receipt by reply email and delete this email and all attachments, (iii) Dassault Systemes does not accept or assume any liability or responsibility for any use of or reliance on this email.For other languages, go to http://www.3ds.com/terms/email-disclaimer. > > > > This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged. If you are not one of the named recipients or have received this email in error, (i) you should not read, disclose, or copy it, (ii) please notify sender of your receipt by reply email and delete this email and all attachments, (iii) Dassault Systemes does not accept or assume any liability or responsibility for any use of or reliance on this email.For other languages, go to http://www.3ds.com/terms/email-disclaimer. From zonexo at gmail.com Thu Nov 18 04:04:19 2010 From: zonexo at gmail.com (TAY wee-beng) Date: Thu, 18 Nov 2010 11:04:19 +0100 Subject: [petsc-users] Adding adaptive mesh refinement to current fortran code In-Reply-To: References: <4CE3E105.2090509@gmail.com> Message-ID: <4CE4FA23.2070907@gmail.com> Hi Barry, In that case, will it be very difficult to use paramesh together with PETSc? I may thinking that as long as I can create a matrix from paramesh and store it as a PETSc format matrix, everything should be fine when I try to solve Ax=b. Is it the same for the parallel case too? Or is it better if I direct the questions to the paramesh developers? Thanks! Yours sincerely, TAY wee-beng On 17-11-2010 18:28, Barry Smith wrote: > libmesh has adaptive refinement and all kinds of cool stuff and lives on top of PETSc. > > Barry > > But it is C++. > > > On Nov 17, 2010, at 8:04 AM, TAY wee-beng wrote: > >> Hi, >> >> I am thinking of adding adaptive mesh refinement to my current fortran code. I am using an immersed boundary code with staggered cartesian grid. I am using PETSc to solve my momentum equations and PETSc/hypre to solve my poisson equation. The code now runs in parallel. >> >> Since I'm programming in fortran, some other c++ packages may not be feasible. I am considering packages like paramesh and AGRIF. I wonder if it is possible to integrate them together with PETSc. Is there anything which I need to consider too? >> >> Thanks! >> >> -- >> Yours sincerely, >> >> TAY wee-beng >> From bsmith at mcs.anl.gov Thu Nov 18 07:23:53 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 18 Nov 2010 07:23:53 -0600 Subject: [petsc-users] Adding adaptive mesh refinement to current fortran code In-Reply-To: <4CE4FA23.2070907@gmail.com> References: <4CE3E105.2090509@gmail.com> <4CE4FA23.2070907@gmail.com> Message-ID: <0FD52CDE-5412-4DC1-949A-3C2F06CCEC2D@mcs.anl.gov> I don't know. Never used paramesh. Barry On Nov 18, 2010, at 4:04 AM, TAY wee-beng wrote: > Hi Barry, > > In that case, will it be very difficult to use paramesh together with PETSc? > > I may thinking that as long as I can create a matrix from paramesh and store it as a PETSc format matrix, everything should be fine when I try to solve Ax=b. Is it the same for the parallel case too? > > Or is it better if I direct the questions to the paramesh developers? > > Thanks! > > Yours sincerely, > > TAY wee-beng > > > On 17-11-2010 18:28, Barry Smith wrote: >> libmesh has adaptive refinement and all kinds of cool stuff and lives on top of PETSc. >> >> Barry >> >> But it is C++. >> >> >> On Nov 17, 2010, at 8:04 AM, TAY wee-beng wrote: >> >>> Hi, >>> >>> I am thinking of adding adaptive mesh refinement to my current fortran code. I am using an immersed boundary code with staggered cartesian grid. I am using PETSc to solve my momentum equations and PETSc/hypre to solve my poisson equation. The code now runs in parallel. >>> >>> Since I'm programming in fortran, some other c++ packages may not be feasible. I am considering packages like paramesh and AGRIF. I wonder if it is possible to integrate them together with PETSc. Is there anything which I need to consider too? >>> >>> Thanks! >>> >>> -- >>> Yours sincerely, >>> >>> TAY wee-beng >>> From chetan.jhurani at gmail.com Thu Nov 18 16:14:03 2010 From: chetan.jhurani at gmail.com (Chetan Jhurani) Date: Thu, 18 Nov 2010 15:14:03 -0700 Subject: [petsc-users] TS with inhomogeneous rhs Message-ID: <4ce5a52d.0f8de50a.74b5.2cd6@mx.google.com> Hello, Question in brief: what's a good way, using TS, to solve a system of ODEs which looks like this? M U_t = -K U + f(t) In addition, how can the code be kept nearly identical to allow different TS types to be used (EULER, BEULER, TSRUNGE_KUTTA, TSCRANK_NICHOLSON, etc)? I'd also like to take possible advantages of M and K being constants. Background and what I've tried: If f(t) were not present TSSetMatrices seems like the convenient way ahead. In fact, I can still use the same function in presence of f(t) for an explicit time-stepping method like RK or Euler if I use a MatShell for Arhs, and sneak in f(t) there. But I cannot use an implicit method like Backward Euler with a MatShell for Arhs easily. It needs MatFactor to be implemented for the shell matrix (at least). Is there something I'm missing? This is a common system of equation. Most TS examples solve u_t = u_xx, convenient for TSSetMatrices. ts/examples/tests/ex3.c looks like it should work for a system with f(t). But it needs the Jacobian matrix of RHS and it evaluates it using finite differences. In addition, for each time-step it computes an ILU factorization. Thanks, Chetan From bsmith at mcs.anl.gov Thu Nov 18 16:22:39 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 18 Nov 2010 16:22:39 -0600 Subject: [petsc-users] TS with inhomogeneous rhs In-Reply-To: <4ce5a52d.0f8de50a.74b5.2cd6@mx.google.com> References: <4ce5a52d.0f8de50a.74b5.2cd6@mx.google.com> Message-ID: <77DECC8D-FAFE-4708-8007-2C44E79FBD9D@mcs.anl.gov> For this we recommend using petsc-dev http://www.mcs.anl.gov/petsc/petsc-as/developers/index.html and using TSSetIFunction() and TSSetIJacobian(). We are transistioning to use this new interface for everything because it handles more general situations than the old interfaces. Barry On Nov 18, 2010, at 4:14 PM, Chetan Jhurani wrote: > Hello, > > Question in brief: what's a good way, using TS, to solve a system of > ODEs which looks like this? > > M U_t = -K U + f(t) > > In addition, how can the code be kept nearly identical to allow different > TS types to be used (EULER, BEULER, TSRUNGE_KUTTA, TSCRANK_NICHOLSON, > etc)? I'd also like to take possible advantages of M and K being > constants. > > Background and what I've tried: > > If f(t) were not present TSSetMatrices seems like the convenient > way ahead. In fact, I can still use the same function in presence of > f(t) for an explicit time-stepping method like RK or Euler if I use > a MatShell for Arhs, and sneak in f(t) there. > > But I cannot use an implicit method like Backward Euler with a > MatShell for Arhs easily. It needs MatFactor to be implemented > for the shell matrix (at least). > > Is there something I'm missing? This is a common system of equation. > > Most TS examples solve u_t = u_xx, convenient for TSSetMatrices. > > ts/examples/tests/ex3.c looks like it should work for a system > with f(t). But it needs the Jacobian matrix of RHS and it evaluates > it using finite differences. In addition, for each time-step it > computes an ILU factorization. > > Thanks, > > Chetan > > > From jed at 59A2.org Thu Nov 18 16:48:58 2010 From: jed at 59A2.org (Jed Brown) Date: Thu, 18 Nov 2010 23:48:58 +0100 Subject: [petsc-users] TS with inhomogeneous rhs In-Reply-To: <4ce5a52d.0f8de50a.74b5.2cd6@mx.google.com> References: <4ce5a52d.0f8de50a.74b5.2cd6@mx.google.com> Message-ID: On Thu, Nov 18, 2010 at 23:14, Chetan Jhurani wrote: > M U_t = -K U + f(t) > > In addition, how can the code be kept nearly identical to allow different > TS types to be used (EULER, BEULER, TSRUNGE_KUTTA, TSCRANK_NICHOLSON, > etc)? I'd also like to take possible advantages of M and K being > constants. > As Barry says, we are transitioning to a better API for this. You can use TSTHETA (theta=0.5 is Crank-Nicolson, theta=1 is Backward Euler) with TSSetIFunction and TSSetIJacobian (see the man pages or read section 6.1.2 of the users manual). Note that an "explicit" method for your problem above needs to solve with the mass matrix at every time step. Unless it is very cheap (e.g. block diagonal), then it may not be worth it. PETSc's explicit methods do not currently support this, but support for that will be added in the next few months. At present, you would need to do the solve in your RHSFunction. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From lizs at mail.uc.edu Thu Nov 18 18:14:46 2010 From: lizs at mail.uc.edu (Li, Zhisong (lizs)) Date: Fri, 19 Nov 2010 00:14:46 +0000 Subject: [petsc-users] Lengthy time to assemble the matrix Message-ID: <88D7E3BB7E1960428303E760100374511AA9F986@BL2PRD0103MB052.prod.exchangelabs.com> Hi, Petsc Team, I noticed that the Petsc program often cost a lot of time to assemble a matrix, particularly when DOF is large. For example, I write a 3D structured CFD code with a linear system of 40 x 22 x 20 in dimension and DOF = 6. In a sequential computation, it takes more than 20 minutes to finish assembling the matrix before ksp operates. This same problem should take only a few seconds for a commercial CFD software to initialize. The processor is 2.67G Hz in speed with sufficient RAM (6GB). The largest matrix stencil is 19 points. I typically use the function: MatSetValuesStencil(A, 1, &abc, 19, def, val, INSERT_VALUES). I wonder the assembling can be faster if abc is an array of multiple matrix row coordinates. But in the case of DOF >1, this seems difficult to implement. Is this normal for Petsc or do we have any means to improve this? Thank you. Zhisong Li -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Nov 18 18:19:40 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 18 Nov 2010 18:19:40 -0600 Subject: [petsc-users] Lengthy time to assemble the matrix In-Reply-To: <88D7E3BB7E1960428303E760100374511AA9F986@BL2PRD0103MB052.prod.exchangelabs.com> References: <88D7E3BB7E1960428303E760100374511AA9F986@BL2PRD0103MB052.prod.exchangelabs.com> Message-ID: <23D22A1A-4E37-42BA-B850-BB53B8FFF2C7@mcs.anl.gov> This is due to not setting the proper preallocation. If (1) you are using DAGetMatrix() to get your matrix then you have not provided the DA the correct stencil information that corresponds to matrix you are setting. For example you used a stencil width of 1 in the DA but are setting values two widths array into the matrix (2) you are calling MatCreateMPIAIJ() or MatCreateMPIBAIJ() or using MatCreate() and then setting the type. See the manual pages for MatCreateMPIAIJ(), MatMPIAIJSetPreallocation() etc and look in the users manual for the chapter on optimization. With proper allocation the time will come down dramatically. Barry On Nov 18, 2010, at 6:14 PM, Li, Zhisong (lizs) wrote: > Hi, Petsc Team, > > I noticed that the Petsc program often cost a lot of time to assemble a matrix, particularly when DOF is large. For example, I write a 3D structured CFD code with a linear system of 40 x 22 x 20 in dimension and DOF = 6. In a sequential computation, it takes more than 20 minutes to finish assembling the matrix before ksp operates. This same problem should take only a few seconds for a commercial CFD software to initialize. > > The processor is 2.67G Hz in speed with sufficient RAM (6GB). The largest matrix stencil is 19 points. I typically use the function: MatSetValuesStencil(A, 1, &abc, 19, def, val, INSERT_VALUES). I wonder the assembling can be faster if abc is an array of multiple matrix row coordinates. But in the case of DOF >1, this seems difficult to implement. > > Is this normal for Petsc or do we have any means to improve this? > > > Thank you. > > > Zhisong Li From dave.mayhem23 at gmail.com Thu Nov 18 18:25:06 2010 From: dave.mayhem23 at gmail.com (Dave May) Date: Fri, 19 Nov 2010 01:25:06 +0100 Subject: [petsc-users] Lengthy time to assemble the matrix In-Reply-To: <23D22A1A-4E37-42BA-B850-BB53B8FFF2C7@mcs.anl.gov> References: <88D7E3BB7E1960428303E760100374511AA9F986@BL2PRD0103MB052.prod.exchangelabs.com> <23D22A1A-4E37-42BA-B850-BB53B8FFF2C7@mcs.anl.gov> Message-ID: You can easily check whether your matrix preallocation is correct by running your application with the following additional command line args -info | grep malloc If the preallocation is correct you'll see something like this. [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 If the malloc count is not zero, your preallocation algorithm is the problem. Dave On 19 November 2010 01:19, Barry Smith wrote: > > ?This is due to not setting the proper preallocation. ?If > > (1) you are using DAGetMatrix() to get your matrix then you have not provided the DA the correct stencil information that corresponds to matrix you are setting. For example you used a stencil width of 1 in the DA but are setting values two widths array into the matrix > > (2) you are calling MatCreateMPIAIJ() or MatCreateMPIBAIJ() or using MatCreate() and then setting the type. See the manual pages for MatCreateMPIAIJ(), MatMPIAIJSetPreallocation() etc and look in the users manual for the chapter on optimization. > > ?With proper allocation the time will come down dramatically. > > ? Barry > > On Nov 18, 2010, at 6:14 PM, Li, Zhisong (lizs) wrote: > >> ?Hi, Petsc Team, >> >> I noticed that the Petsc program often cost a lot of time to assemble a matrix, particularly when DOF is large. For example, I write a 3D structured CFD code with a linear system of 40 x 22 x 20 in dimension and DOF = 6. In a sequential computation, it takes more than 20 minutes to finish assembling the matrix before ksp operates. This same problem should take only a few seconds for a commercial CFD software to initialize. >> >> The processor is 2.67G Hz in speed with sufficient RAM (6GB). The largest matrix stencil is 19 points. I typically use the function: MatSetValuesStencil(A, 1, &abc, 19, def, val, INSERT_VALUES). I wonder the assembling can be faster if abc is an array of multiple matrix row coordinates. But in the case of DOF >1, this seems difficult to implement. >> >> Is this normal for Petsc or do we have any means to improve this? >> >> >> Thank you. >> >> >> Zhisong Li > > From chetan.jhurani at gmail.com Thu Nov 18 19:07:28 2010 From: chetan.jhurani at gmail.com (Chetan Jhurani) Date: Thu, 18 Nov 2010 18:07:28 -0700 Subject: [petsc-users] TS with inhomogeneous rhs In-Reply-To: References: <4ce5a52d.0f8de50a.74b5.2cd6@mx.google.com> Message-ID: <4ce5cdd2.8e4ee50a.36b9.368b@mx.google.com> Thanks Barry and Jed. TSSetIFunction and TSSetIJacobian from petsc-dev are working perfectly. Chetan From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jed Brown Sent: Thursday, November 18, 2010 3:49 PM To: PETSc users list Subject: Re: [petsc-users] TS with inhomogeneous rhs On Thu, Nov 18, 2010 at 23:14, Chetan Jhurani wrote: M U_t = -K U + f(t) In addition, how can the code be kept nearly identical to allow different TS types to be used (EULER, BEULER, TSRUNGE_KUTTA, TSCRANK_NICHOLSON, etc)? I'd also like to take possible advantages of M and K being constants. As Barry says, we are transitioning to a better API for this. You can use TSTHETA (theta=0.5 is Crank-Nicolson, theta=1 is Backward Euler) with TSSetIFunction and TSSetIJacobian (see the man pages or read section 6.1.2 of the users manual). Note that an "explicit" method for your problem above needs to solve with the mass matrix at every time step. Unless it is very cheap (e.g. block diagonal), then it may not be worth it. PETSc's explicit methods do not currently support this, but support for that will be added in the next few months. At present, you would need to do the solve in your RHSFunction. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From m.skates82 at gmail.com Thu Nov 18 19:54:43 2010 From: m.skates82 at gmail.com (Nunion) Date: Thu, 18 Nov 2010 19:54:43 -0600 Subject: [petsc-users] Trilinos-PETSc Interface Message-ID: I am in the middle of configuring the PETSc/Trilinos interface. However, when I run 'make', the following error is being generated. I have testing and examples enabled. From my knowledge I have disabled all MPI features, i.e. "-D TPL_ENABLE_MPI:BOOL=OFF", however this error is happening while in the section, "Scanning dependencies of target epetraext". Any assistance is greatly appreciated. Thanks! Mark [ 57%] Building CXX object packages/epetraext/src/CMakeFiles/epetraext.dir/block/EpetraExt_MultiPointModelEvaluator.cpp.o [ 57%] Building CXX object packages/epetraext/src/CMakeFiles/epetraext.dir/block/EpetraExt_MultiSerialComm.cpp.o [ 57%] Building CXX object packages/epetraext/src/CMakeFiles/epetraext.dir/block/EpetraExt_BlockDiagMatrix.cpp.o [ 57%] Building CXX object packages/epetraext/src/CMakeFiles/epetraext.dir/block/EpetraExt_PointToBlockDiagPermute.cpp.o [ 57%] Building CXX object packages/epetraext/src/CMakeFiles/epetraext.dir/restrict/EpetraExt_RestrictedMultiVectorWrapper.cpp.o [ 57%] Building CXX object packages/epetraext/src/CMakeFiles/epetraext.dir/restrict/EpetraExt_RestrictedCrsMatrixWrapper.cpp.o [ 57%] Building CXX object packages/epetraext/src/CMakeFiles/epetraext.dir/petsc/EpetraExt_PETScAIJMatrix.cpp.o /home/walker/Desktop/PETSc/_NEW/petsc-3.1-p5/include/private/vecimpl.h:255: error: ?MPI_Win? does not name a type /home/walker/Desktop/trilinos/trilinos-10.4.1-Source/packages/epetraext/src/petsc/EpetraExt_PETScAIJMatrix.h: In member function ?virtual int Epetra_PETScAIJMatrix::NumGlobalNonzeros() const?: /home/walker/Desktop/trilinos/trilinos-10.4.1-Source/packages/epetraext/src/petsc/EpetraExt_PETScAIJMatrix.h:232: warning: converting to ?int? from ?const double? /home/walker/Desktop/trilinos/trilinos-10.4.1-Source/packages/epetraext/src/petsc/EpetraExt_PETScAIJMatrix.cpp: In member function ?virtual int Epetra_PETScAIJMatrix::ExtractDiagonalCopy(Epetra_Vector&) const?: /home/walker/Desktop/trilinos/trilinos-10.4.1-Source/packages/epetraext/src/petsc/EpetraExt_PETScAIJMatrix.cpp:243: error: ?class Epetra_SerialComm? has no member named ?Comm? /home/walker/Desktop/trilinos/trilinos-10.4.1-Source/packages/epetraext/src/petsc/EpetraExt_PETScAIJMatrix.cpp: In member function ?virtual int Epetra_PETScAIJMatrix::Multiply(bool, const Epetra_MultiVector&, Epetra_MultiVector&) const?: /home/walker/Desktop/trilinos/trilinos-10.4.1-Source/packages/epetraext/src/petsc/EpetraExt_PETScAIJMatrix.cpp:292: error: ?class Epetra_SerialComm? has no member named ?Comm? /home/walker/Desktop/trilinos/trilinos-10.4.1-Source/packages/epetraext/src/petsc/EpetraExt_PETScAIJMatrix.cpp:293: error: ?class Epetra_SerialComm? has no member named ?Comm? /home/walker/Desktop/trilinos/trilinos-10.4.1-Source/packages/epetraext/src/petsc/EpetraExt_PETScAIJMatrix.cpp: In member function ?virtual int Epetra_PETScAIJMatrix::LeftScale(const Epetra_Vector&)?: /home/walker/Desktop/trilinos/trilinos-10.4.1-Source/packages/epetraext/src/petsc/EpetraExt_PETScAIJMatrix.cpp:445: error: ?class Epetra_SerialComm? has no member named ?Comm? /home/walker/Desktop/trilinos/trilinos-10.4.1-Source/packages/epetraext/src/petsc/EpetraExt_PETScAIJMatrix.cpp: In member function ?virtual int Epetra_PETScAIJMatrix::RightScale(const Epetra_Vector&)?: /home/walker/Desktop/trilinos/trilinos-10.4.1-Source/packages/epetraext/src/petsc/EpetraExt_PETScAIJMatrix.cpp:465: error: ?class Epetra_SerialComm? has no member named ?Comm? make[2]: *** [packages/epetraext/src/CMakeFiles/epetraext.dir/petsc/EpetraExt_PETScAIJMatrix.cpp.o] Error 1 make[1]: *** [packages/epetraext/src/CMakeFiles/epetraext.dir/all] Error 2 make: *** [all] Error 2 [walker at sys TB2]$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Fri Nov 19 08:15:03 2010 From: zonexo at gmail.com (TAY wee-beng) Date: Fri, 19 Nov 2010 15:15:03 +0100 Subject: [petsc-users] Changing the options in the makefile In-Reply-To: <4CE11C76.2090306@imperial.ac.uk> References: <021A6584-4DD7-44D0-BC9C-977E1DA737F9@cpht.polytechnique.fr> <6501936B-4961-4C2E-A491-62F0A3655416@cpht.polytechnique.fr> <2E4B4874-EC33-4DAC-91E1-0E188FBFD2ED@cpht.polytechnique.fr> <0E36C9ED-9082-4302-91B6-023DE9A7367A@cpht.polytechnique.fr> <4CE1036A.4080802@gmail.com> <4CE11C76.2090306@imperial.ac.uk> Message-ID: <4CE68667.9070107@gmail.com> Hi, I have a modified makefile originally given to me by the PETSc team. It worked well but now I want to change some of the options of the compiler, like using -O3 or -ipo etc instead of -O. I tried to add -O3 in and the compiling command becomes: /opt/openmpi-1.4.1/bin/mpif90 -O3 -c -g -r8 -save -w90 -w -w95 -O -I/home/wtay/Lib/petsc-3.1-p5/include .... So now there's -O3 and -O. How can I change it to only -O3 or -ipo? I have attached the makefile too. Thank you very much Yours sincerely, TAY wee-beng -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: makefile URL: From bsmith at mcs.anl.gov Fri Nov 19 11:37:02 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 19 Nov 2010 11:37:02 -0600 Subject: [petsc-users] Changing the options in the makefile In-Reply-To: <4CE68667.9070107@gmail.com> References: <021A6584-4DD7-44D0-BC9C-977E1DA737F9@cpht.polytechnique.fr> <6501936B-4961-4C2E-A491-62F0A3655416@cpht.polytechnique.fr> <2E4B4874-EC33-4DAC-91E1-0E188FBFD2ED@cpht.polytechnique.fr> <0E36C9ED-9082-4302-91B6-023DE9A7367A@cpht.polytechnique.fr> <4CE1036A.4080802@gmail.com> <4CE11C76.2090306@imperial.ac.uk> <4CE68667.9070107@gmail.com> Message-ID: <2AC3578D-8DF7-4B9B-80C6-0FAEC6BE5CBD@mcs.anl.gov> Edit $PETSC_ARCH/conf/petscvariables and remove the options you don't want from everywhere in that file. Barry On Nov 19, 2010, at 8:15 AM, TAY wee-beng wrote: > Hi, > > I have a modified makefile originally given to me by the PETSc team. It worked well but now I want to change some of the options of the compiler, like using -O3 or -ipo etc instead of -O. > > I tried to add -O3 in and the compiling command becomes: > > /opt/openmpi-1.4.1/bin/mpif90 -O3 -c -g -r8 -save -w90 -w -w95 -O -I/home/wtay/Lib/petsc-3.1-p5/include .... > > So now there's -O3 and -O. How can I change it to only -O3 or -ipo? > > I have attached the makefile too. > > Thank you very much > > Yours sincerely, > > TAY wee-beng > > > From zonexo at gmail.com Fri Nov 19 14:34:56 2010 From: zonexo at gmail.com (Wee-Beng Tay) Date: Fri, 19 Nov 2010 21:34:56 +0100 Subject: [petsc-users] Changing the options in the makefile In-Reply-To: <2AC3578D-8DF7-4B9B-80C6-0FAEC6BE5CBD@mcs.anl.gov> References: <021A6584-4DD7-44D0-BC9C-977E1DA737F9@cpht.polytechnique.fr> <6501936B-4961-4C2E-A491-62F0A3655416@cpht.polytechnique.fr> <2E4B4874-EC33-4DAC-91E1-0E188FBFD2ED@cpht.polytechnique.fr> <0E36C9ED-9082-4302-91B6-023DE9A7367A@cpht.polytechnique.fr> <4CE1036A.4080802@gmail.com> <4CE11C76.2090306@imperial.ac.uk> <4CE68667.9070107@gmail.com> <2AC3578D-8DF7-4B9B-80C6-0FAEC6BE5CBD@mcs.anl.gov> Message-ID: <4CE6DF70.9030100@gmail.com> Hi Barry, Thanks! Yours sincerely, Wee-Beng Tay On 19/11/2010 6:37 PM, Barry Smith wrote: > Edit $PETSC_ARCH/conf/petscvariables and remove the options you don't want from everywhere in that file. > > Barry > > > On Nov 19, 2010, at 8:15 AM, TAY wee-beng wrote: > >> Hi, >> >> I have a modified makefile originally given to me by the PETSc team. It worked well but now I want to change some of the options of the compiler, like using -O3 or -ipo etc instead of -O. >> >> I tried to add -O3 in and the compiling command becomes: >> >> /opt/openmpi-1.4.1/bin/mpif90 -O3 -c -g -r8 -save -w90 -w -w95 -O -I/home/wtay/Lib/petsc-3.1-p5/include .... >> >> So now there's -O3 and -O. How can I change it to only -O3 or -ipo? >> >> I have attached the makefile too. >> >> Thank you very much >> >> Yours sincerely, >> >> TAY wee-beng >> >> >> From chris.bording at gmail.com Mon Nov 22 15:23:58 2010 From: chris.bording at gmail.com (Chris Bording) Date: Mon, 22 Nov 2010 13:23:58 -0800 Subject: [petsc-users] Passing DMMG struct. Message-ID: Hi I am trying to pass the DMMG struct to a function similar to what is in the examples. extern PetscErrorCode Myfunc(DMMG,Vec); int main(int argc,char **argv) { DMMG *dmmg; ...Vec b; Myfunc(dmmg,b) } PetscErrorCode Myfunc(DMMG dmmg, Vec b) { DA da = (DA)dmmg->dm; ..... } error: cannot convert ?_n_DMMG**? to ?_n_DMMG*? for argument ?1? to ?PetscErrorCode Myfunc(_n_DMMG*, _p_Vec*) So how do I pass a DMMG defined structure into my function? Thanks Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Nov 22 15:34:06 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 22 Nov 2010 15:34:06 -0600 Subject: [petsc-users] Passing DMMG struct. In-Reply-To: References: Message-ID: <7CC9EABF-A8CE-4E6D-8DD6-175DA92B40F1@mcs.anl.gov> On Nov 22, 2010, at 3:23 PM, Chris Bording wrote: > Hi I am trying to pass the DMMG struct to a function similar to what is in the examples. > > extern PetscErrorCode Myfunc(DMMG,Vec); > > > int main(int argc,char **argv) > { > DMMG *dmmg; ^^^^^^ Note this > ...Vec b; > > Myfunc(dmmg,b) > } > > PetscErrorCode Myfunc(DMMG *dmmg, Vec b) ^^^ add this > { > DA da = (DA)dmmg->dm; > ..... > } > > error: cannot convert ?_n_DMMG**? to ?_n_DMMG*? for argument ?1? to ?PetscErrorCode Myfunc(_n_DMMG*, _p_Vec*) > > So how do I pass a DMMG defined structure into my function? > > > > Thanks > Chris From chris.bording at gmail.com Mon Nov 22 17:02:18 2010 From: chris.bording at gmail.com (Chris Bording) Date: Mon, 22 Nov 2010 15:02:18 -0800 Subject: [petsc-users] Passing DMMG struct. In-Reply-To: <7CC9EABF-A8CE-4E6D-8DD6-175DA92B40F1@mcs.anl.gov> References: <7CC9EABF-A8CE-4E6D-8DD6-175DA92B40F1@mcs.anl.gov> Message-ID: I made the change in the function so I now get this error PetscErrorCode ComputeRho(DMMG *dmmg, Vec rho) { DA distA = (DA)dmmg->dm; In function ?PetscErrorCode Myfunc(_n_DMMG**, _p_Vec*)?: error: request for member ?dm? in ?* dmmg?, which is of non-class type ?_n_DMMG*? It looks like I have a null pointer to _n_DMMG** instead of _n_DMMG* Chris On Mon, Nov 22, 2010 at 1:34 PM, Barry Smith wrote: > > On Nov 22, 2010, at 3:23 PM, Chris Bording wrote: > > > Hi I am trying to pass the DMMG struct to a function similar to what is > in the examples. > > > > extern PetscErrorCode Myfunc(DMMG,Vec); > > > > > > int main(int argc,char **argv) > > { > > DMMG *dmmg; > ^^^^^^ Note this > > ...Vec b; > > > > Myfunc(dmmg,b) > > } > > > > PetscErrorCode Myfunc(DMMG *dmmg, Vec b) > ^^^ add this > > > { > > DA da = (DA)dmmg->dm; > > ..... > > } > > > > error: cannot convert ?_n_DMMG**? to ?_n_DMMG*? for argument ?1? to > ?PetscErrorCode Myfunc(_n_DMMG*, _p_Vec*) > > > > So how do I pass a DMMG defined structure into my function? > > > > > > > > Thanks > > Chris > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean at mcs.anl.gov Mon Nov 22 17:27:34 2010 From: sean at mcs.anl.gov (Sean Farley) Date: Mon, 22 Nov 2010 17:27:34 -0600 Subject: [petsc-users] Passing DMMG struct. In-Reply-To: References: <7CC9EABF-A8CE-4E6D-8DD6-175DA92B40F1@mcs.anl.gov> Message-ID: > > I made the change in the function so I now get this error > PetscErrorCode ComputeRho(DMMG *dmmg, Vec rho) > { > DA distA = (DA)dmmg->dm; > > In function ?PetscErrorCode Myfunc(_n_DMMG**, _p_Vec*)?: > error: request for member ?dm? in ?* dmmg?, which is of non-class type > ?_n_DMMG*? > > It looks like I have a null pointer to _n_DMMG** instead of _n_DMMG* It seems like your code expects you to pass a DMMG object not a DMMG* object. There are two ways to fix this: 1) In your function 'ComputeRho' change all references of dmmg to *dmmg, i.e.: DA distA = (DA)(*dmmg)->dm; 2) Change your function declaration back to what you had: PetscErrorCode ComputeRho(DMMG dmmg, Vec rho) {...} and then pass the object to your function as so: ComputeRho(*dmmg, b); Do you need your dmmg to be a pointer in your 'main' function? It looks like your declaration could just be: extern PetscErrorCode Myfunc(DMMG,Vec); >>>>>>>>>>>>>>>> int main(int argc,char **argv) { DMMG dmmg; ...Vec b; Myfunc(dmmg,b) } PetscErrorCode Myfunc(DMMG dmmg, Vec b) { DA da = (DA)dmmg->dm; ..... } <<<<<<<<<<<<<<<<< and that would fix everything. Sean -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Mon Nov 22 18:02:03 2010 From: jed at 59A2.org (Jed Brown) Date: Tue, 23 Nov 2010 01:02:03 +0100 Subject: [petsc-users] Passing DMMG struct. In-Reply-To: References: <7CC9EABF-A8CE-4E6D-8DD6-175DA92B40F1@mcs.anl.gov> Message-ID: On Tue, Nov 23, 2010 at 00:27, Sean Farley wrote: > 2) Change your function declaration back to what you had: > > PetscErrorCode ComputeRho(DMMG dmmg, Vec rho) > {...} > > and then pass the object to your function as so: > > ComputeRho(*dmmg, b); > > Do you need your dmmg to be a pointer in your 'main' function? It looks > like your declaration could just be: > DMMG is simultaneously an array representing a full hierarchy and the individual levels in the hierarchy. If you have DMMG *dmmg; then dmmg[0] (same as *dmmg) represents the coarsest level. If your function only works on one level, then you should just pass in a DMMG (not a DMMG*). If your function only needs the dmmg->dm field, then just make your function use a DA, then make that the argument of your function and call it as DMMG *dmmg; // dmmg is initialized somehow ... MyFunc(DMMGGetDA(dmmg),bvec); This passes in the DA on the finest level. DMMG is a weird beast. Once its functionality has been migrated to SNES and KSP (underway in petsc-dev), it will be removed. Jed Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From clemens.domanig at uibk.ac.at Tue Nov 23 03:32:41 2010 From: clemens.domanig at uibk.ac.at (Clemens Domanig) Date: Tue, 23 Nov 2010 10:32:41 +0100 Subject: [petsc-users] Parallel assembly of matrix using OpenMP In-Reply-To: References: Message-ID: <4CEB8A39.8020705@uibk.ac.at> Hi everyone! I'm writing a FEM-program and at the moment I try to parallize the assembly of the stiffness-matrix. The idea would be if I for example use 4 threads I create 4 matrices and 4 rhs-vectors. Each thread fills its matrix and rhs-vector. At the end I add all matrices and vectors. But the Petsc-manual says that Petsc is not thread-safe. Is that the reason why it doesn't work? Usually I get those error-code (after assembling 4-100 elements - everytime different): I.) [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range or II.) *** glibc detected *** ../icona: corrupted double-linked list: 0x00000000166ff730 *** ======= Backtrace: ========= /lib64/libc.so.6[0x3a9fc72405] /lib64/libc.so.6(cfree+0x4b)[0x3a9fc7276b] ../icona[0x426cae] ../icona[0x42df99] ../icona[0x41b829] ../icona[0x416203] ../icona[0x417341] ../icona[0x408954] /lib64/libc.so.6(__libc_start_main+0xf4)[0x3a9fc1d994] ../icona(__gxx_personality_v0+0x3e9)[0x4078f9] ======= Memory map: ======== 00400000-00fe1000 r-xp 00000000 00:1b 143920732 /home/c7031006/icona2.zip_FILES/icona2/icona 011e1000-011e8000 rw-p 00be1000 00:1b 143920732 /home/c7031006/icona2.zip_FILES/icona2/icona 011e8000-0131b000 rw-p 011e8000 00:00 0 16397000-16724000 rw-p 16397000 00:00 0 [heap] If someone could explain - Thank you. Yours sincerely, Clemens Domanig From jed at 59A2.org Tue Nov 23 05:15:02 2010 From: jed at 59A2.org (Jed Brown) Date: Tue, 23 Nov 2010 12:15:02 +0100 Subject: [petsc-users] Parallel assembly of matrix using OpenMP In-Reply-To: <4CEB8A39.8020705@uibk.ac.at> References: <4CEB8A39.8020705@uibk.ac.at> Message-ID: On Tue, Nov 23, 2010 at 10:32, Clemens Domanig wrote: > I'm writing a FEM-program and at the moment I try to parallize the assembly > of the stiffness-matrix. The idea would be if I for example use 4 threads I > create 4 matrices and 4 rhs-vectors. Each thread fills its matrix and > rhs-vector. At the end I add all matrices and vectors. > Adding sparse matrices together is a relatively expensive operation, usually more expensive than assembling the matrix you wanted in the first place. > But the Petsc-manual says that Petsc is not thread-safe. > There is a minor issue of some logging functions that are not thread safe. Wrapping locks around some of those operations would not be a huge deal, but hasn't been done. > Is that the reason why it doesn't work? > The matrix data structures cannot cheaply be mutated in a thread-safe way. We would either need fine-grained locks which have more overhead, and are very tricky when the user does not preallocate correctly (requiring "rollback" logic), or a coarse-grained lock for the whole MatSetValues. If you want to do multi-threaded assembly, you should just put your own lock around your call to MatSetValues. As long as your physics does some work (e.g. is a bit more than a linear constant-coefficient problem on affine elements), this can work fine for a few threads, usually 8-16 or so, before serialization of MatSetValues becomes a bottleneck. Note that you can also use multiple MPI processes to keep your cores busy, there have been a few discussions on this list, and petsc-dev, about the relative merits of threads versus processes. If you tell us a bit about your problem, we may be able to predict how each will perform. This helps to assess the importance of making PETSc thread-safe and making certain kernels perform well with threads, relative to other features. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmgottfried at web.de Wed Nov 24 09:26:36 2010 From: jmgottfried at web.de (Jens-Malte Gottfried) Date: Wed, 24 Nov 2010 16:26:36 +0100 Subject: [petsc-users] petsc rpath Message-ID: <4CED2EAC.2070104@web.de> Hi! I am trying to write a gentoo ebuild for petsc (my current work is already in the science overlay). Is there any possibility to disable writing the RPATH of the to-be-generated libpetsc.so? Currently, the library is built fine but RPATH is set to /usr/lib64:/usr/lib/gcc/x86_64-pc-linux-gnu/4.4.4:/usr/x86_64-pc-linux-gnu/lib which is not neccessary. Before trying to patch the build system, perhaps there is a simple configure option which I did not find. Thanks Jens-Malte Gottfried -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 900 bytes Desc: OpenPGP digital signature URL: From jed at 59A2.org Wed Nov 24 09:47:14 2010 From: jed at 59A2.org (Jed Brown) Date: Wed, 24 Nov 2010 16:47:14 +0100 Subject: [petsc-users] petsc rpath In-Reply-To: <4CED2EAC.2070104@web.de> References: <4CED2EAC.2070104@web.de> Message-ID: On Wed, Nov 24, 2010 at 16:26, Jens-Malte Gottfried wrote: > I am trying to write a gentoo ebuild for petsc (my current work is > already in the science overlay). > Is there any possibility to disable writing the RPATH of the > to-be-generated libpetsc.so? > Currently, the library is built fine but RPATH is set to > > > /usr/lib64:/usr/lib/gcc/x86_64-pc-linux-gnu/4.4.4:/usr/x86_64-pc-linux-gnu/lib > which is not neccessary. Before trying to patch the build system, > perhaps there is a simple configure option which I did not find. > The standard build system does not provide an easy way to do this. If you are using petsc-dev, you use CMake (configure as usual, but then run make from $PETSC_DIR/$PETSC_ARCH instead of from $PETSC_DIR). The CMake build files default to setting RPATH, but it is easy to set the cache variables to disable RPATH. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Nov 24 09:58:35 2010 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 24 Nov 2010 09:58:35 -0600 (CST) Subject: [petsc-users] petsc rpath In-Reply-To: References: <4CED2EAC.2070104@web.de> Message-ID: B1;2601;0cOn Wed, 24 Nov 2010, Jed Brown wrote: > On Wed, Nov 24, 2010 at 16:26, Jens-Malte Gottfried wrote: > > > I am trying to write a gentoo ebuild for petsc (my current work is > > already in the science overlay). > > Is there any possibility to disable writing the RPATH of the > > to-be-generated libpetsc.so? > > Currently, the library is built fine but RPATH is set to > > > > > > /usr/lib64:/usr/lib/gcc/x86_64-pc-linux-gnu/4.4.4:/usr/x86_64-pc-linux-gnu/lib > > which is not neccessary. Before trying to patch the build system, > > perhaps there is a simple configure option which I did not find. > > > > The standard build system does not provide an easy way to do this. If you > are using petsc-dev, you use CMake (configure as usual, but then run make > from $PETSC_DIR/$PETSC_ARCH instead of from $PETSC_DIR). The CMake build > files default to setting RPATH, but it is easy to set the cache variables to > disable RPATH. Some of the above comes from 'gfortran -v' - which checkFortranLibraries() etc do.. For such known systems with known compatibility libraries [ and each language compiler can find each others libs] you can use: [for c++ build] -with-clib-autodetect=0 -with-fortranlib-autodetect=0 LIBS=-lgfortran [for c++ build] -with-clib-autodetect=0 -with-cxxlib-autodetect=0 -with-fortranlib-autodetect=0 LIBS='-lstdc++ -lgfortran' Wrt -rpath for -lpetsc, perhaps commenting out the following line in config/BuildSystem/config/setCompilers.py is sufficient: self.executeTest(self.checkSharedLinkerPaths) Satish From balay at mcs.anl.gov Wed Nov 24 10:19:16 2010 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 24 Nov 2010 10:19:16 -0600 (CST) Subject: [petsc-users] petsc rpath In-Reply-To: References: <4CED2EAC.2070104@web.de> Message-ID: On Wed, 24 Nov 2010, Satish Balay wrote: > B1;2601;0cOn Wed, 24 Nov 2010, Jed Brown wrote: > > > On Wed, Nov 24, 2010 at 16:26, Jens-Malte Gottfried wrote: > > > > > I am trying to write a gentoo ebuild for petsc (my current work is > > > already in the science overlay). > > > Is there any possibility to disable writing the RPATH of the > > > to-be-generated libpetsc.so? > > > Currently, the library is built fine but RPATH is set to > > > > > > > > > /usr/lib64:/usr/lib/gcc/x86_64-pc-linux-gnu/4.4.4:/usr/x86_64-pc-linux-gnu/lib > > > which is not neccessary. Before trying to patch the build system, > > > perhaps there is a simple configure option which I did not find. > > > > > > > The standard build system does not provide an easy way to do this. If you > > are using petsc-dev, you use CMake (configure as usual, but then run make > > from $PETSC_DIR/$PETSC_ARCH instead of from $PETSC_DIR). The CMake build > > files default to setting RPATH, but it is easy to set the cache variables to > > disable RPATH. > > Some of the above comes from 'gfortran -v' - which checkFortranLibraries() etc do.. > > For such known systems with known compatibility libraries [ and each > language compiler can find each others libs] you can use: > > [for c++ build] I meant c build.. > -with-clib-autodetect=0 -with-fortranlib-autodetect=0 LIBS=-lgfortran > > [for c++ build] > -with-clib-autodetect=0 -with-cxxlib-autodetect=0 -with-fortranlib-autodetect=0 LIBS='-lstdc++ -lgfortran' > > Wrt -rpath for -lpetsc, perhaps commenting out the following line in > config/BuildSystem/config/setCompilers.py is sufficient: > > self.executeTest(self.checkSharedLinkerPaths) Actually you need '-L' there - so the following change will remove -rpath asterix:/home/balay/tmp/petsc-dist-test/config/BuildSystem>hg diff diff -r 5fac76781491 config/setCompilers.py --- a/config/setCompilers.py Mon Sep 20 16:15:14 2010 -0500 +++ b/config/setCompilers.py Wed Nov 24 10:12:46 2010 -0600 @@ -1188,7 +1188,7 @@ self.pushLanguage(language) # test '-R' before '-rpath' as sun compilers [c,fortran] don't give proper errors with wrong options. if not Configure.isDarwin(): - testFlags = ['-Wl,-rpath,', '-R','-rpath ' , '-Wl,-R,'] + testFlags = [] else: testFlags = [] # test '-R' before '-Wl,-rpath' for SUN compilers [as cc on linux accepts -Wl,-rpath, but f90 & CC do not. asterix:/home/balay/tmp/petsc-dist-test/config/BuildSystem> Satish From jmgottfried at web.de Wed Nov 24 12:28:19 2010 From: jmgottfried at web.de (Jens-Malte Gottfried) Date: Wed, 24 Nov 2010 19:28:19 +0100 Subject: [petsc-users] petsc rpath In-Reply-To: References: <4CED2EAC.2070104@web.de> Message-ID: <4CED5943.1010304@web.de> Thanks a lot Satish, your patch did the job :-) Am 24.11.2010 17:19, schrieb Satish Balay: > On Wed, 24 Nov 2010, Satish Balay wrote: > >> B1;2601;0cOn Wed, 24 Nov 2010, Jed Brown wrote: >> >>> On Wed, Nov 24, 2010 at 16:26, Jens-Malte Gottfried wrote: >>> >>>> I am trying to write a gentoo ebuild for petsc (my current work is >>>> already in the science overlay). >>>> Is there any possibility to disable writing the RPATH of the >>>> to-be-generated libpetsc.so? >>>> Currently, the library is built fine but RPATH is set to >>>> >>>> >>>> /usr/lib64:/usr/lib/gcc/x86_64-pc-linux-gnu/4.4.4:/usr/x86_64-pc-linux-gnu/lib >>>> which is not neccessary. Before trying to patch the build system, >>>> perhaps there is a simple configure option which I did not find. >>>> >>> The standard build system does not provide an easy way to do this. If you >>> are using petsc-dev, you use CMake (configure as usual, but then run make >>> from $PETSC_DIR/$PETSC_ARCH instead of from $PETSC_DIR). The CMake build >>> files default to setting RPATH, but it is easy to set the cache variables to >>> disable RPATH. >> Some of the above comes from 'gfortran -v' - which checkFortranLibraries() etc do.. >> >> For such known systems with known compatibility libraries [ and each >> language compiler can find each others libs] you can use: >> >> [for c++ build] > I meant c build.. > >> -with-clib-autodetect=0 -with-fortranlib-autodetect=0 LIBS=-lgfortran >> >> [for c++ build] >> -with-clib-autodetect=0 -with-cxxlib-autodetect=0 -with-fortranlib-autodetect=0 LIBS='-lstdc++ -lgfortran' >> >> Wrt -rpath for -lpetsc, perhaps commenting out the following line in >> config/BuildSystem/config/setCompilers.py is sufficient: >> >> self.executeTest(self.checkSharedLinkerPaths) > Actually you need '-L' there - so the following change will remove -rpath > > asterix:/home/balay/tmp/petsc-dist-test/config/BuildSystem>hg diff > diff -r 5fac76781491 config/setCompilers.py > --- a/config/setCompilers.py Mon Sep 20 16:15:14 2010 -0500 > +++ b/config/setCompilers.py Wed Nov 24 10:12:46 2010 -0600 > @@ -1188,7 +1188,7 @@ > self.pushLanguage(language) > # test '-R' before '-rpath' as sun compilers [c,fortran] don't give proper errors with wrong options. > if not Configure.isDarwin(): > - testFlags = ['-Wl,-rpath,', '-R','-rpath ' , '-Wl,-R,'] > + testFlags = [] > else: > testFlags = [] > # test '-R' before '-Wl,-rpath' for SUN compilers [as cc on linux accepts -Wl,-rpath, but f90 & CC do not. > asterix:/home/balay/tmp/petsc-dist-test/config/BuildSystem> > > > > Satish -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 900 bytes Desc: OpenPGP digital signature URL: From dalcinl at gmail.com Wed Nov 24 13:40:19 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 24 Nov 2010 16:40:19 -0300 Subject: [petsc-users] petsc rpath In-Reply-To: <4CED5943.1010304@web.de> References: <4CED2EAC.2070104@web.de> <4CED5943.1010304@web.de> Message-ID: Perhaps you should add a --with-rpath option to ./configure ? For reference, MPICH2 did that (>=1.3), I believe the idea was to make the life of distro packagers easier. On 24 November 2010 15:28, Jens-Malte Gottfried wrote: > Thanks a lot Satish, your patch did the job :-) > > Am 24.11.2010 17:19, schrieb Satish Balay: >> On Wed, 24 Nov 2010, Satish Balay wrote: >> >>> B1;2601;0cOn Wed, 24 Nov 2010, Jed Brown wrote: >>> >>>> On Wed, Nov 24, 2010 at 16:26, Jens-Malte Gottfried wrote: >>>> >>>>> I am trying to write a gentoo ebuild for petsc (my current work is >>>>> already in the science overlay). >>>>> Is there any possibility to disable writing the RPATH of the >>>>> to-be-generated libpetsc.so? >>>>> Currently, the library is built fine but RPATH is set to >>>>> >>>>> >>>>> /usr/lib64:/usr/lib/gcc/x86_64-pc-linux-gnu/4.4.4:/usr/x86_64-pc-linux-gnu/lib >>>>> which is not neccessary. Before trying to patch the build system, >>>>> perhaps there is a simple configure option which I did not find. >>>>> >>>> The standard build system does not provide an easy way to do this. ?If you >>>> are using petsc-dev, you use CMake (configure as usual, but then run make >>>> from $PETSC_DIR/$PETSC_ARCH instead of from $PETSC_DIR). ?The CMake build >>>> files default to setting RPATH, but it is easy to set the cache variables to >>>> disable RPATH. >>> Some of the above comes from 'gfortran -v' - which checkFortranLibraries() etc do.. >>> >>> For such known systems with known compatibility libraries [ and each >>> language compiler can find each others libs] you can use: >>> >>> [for c++ build] >> I meant c build.. >> >>> -with-clib-autodetect=0 -with-fortranlib-autodetect=0 LIBS=-lgfortran >>> >>> [for c++ build] >>> -with-clib-autodetect=0 -with-cxxlib-autodetect=0 -with-fortranlib-autodetect=0 LIBS='-lstdc++ -lgfortran' >>> >>> Wrt -rpath for -lpetsc, perhaps commenting out the following line in >>> config/BuildSystem/config/setCompilers.py is sufficient: >>> >>> ? ? self.executeTest(self.checkSharedLinkerPaths) >> Actually you need '-L' there - so the following change will remove -rpath >> >> asterix:/home/balay/tmp/petsc-dist-test/config/BuildSystem>hg diff >> diff -r 5fac76781491 config/setCompilers.py >> --- a/config/setCompilers.py ? ?Mon Sep 20 16:15:14 2010 -0500 >> +++ b/config/setCompilers.py ? ?Wed Nov 24 10:12:46 2010 -0600 >> @@ -1188,7 +1188,7 @@ >> ? ? ? ?self.pushLanguage(language) >> ? ? ? ?# test '-R' before '-rpath' as sun compilers [c,fortran] don't give proper errors with wrong options. >> ? ? ? ?if not Configure.isDarwin(): >> - ? ? ? ?testFlags = ['-Wl,-rpath,', '-R','-rpath ' , '-Wl,-R,'] >> + ? ? ? ?testFlags = [] >> ? ? ? ?else: >> ? ? ? ? ?testFlags = [] >> ? ? ? ?# test '-R' before '-Wl,-rpath' for SUN compilers [as cc on linux accepts -Wl,-rpath, but ?f90 & CC do not. >> asterix:/home/balay/tmp/petsc-dist-test/config/BuildSystem> >> >> >> >> Satish > > -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From khalid_eee at yahoo.com Thu Nov 25 02:11:06 2010 From: khalid_eee at yahoo.com (khalid ashraf) Date: Thu, 25 Nov 2010 00:11:06 -0800 (PST) Subject: [petsc-users] DAGetCorners; xs,ys,zs return 0 for all the processors Message-ID: <842748.55775.qm@web112615.mail.gq1.yahoo.com> Hi, I am having trouble with writing to global vector indexes using DA . I am using the following function to get the starting point of a particular processor. DAGetCorners(da,&xs,&ys,&zs,&xm,&ym,&zm); And then use it to iterate through the global indices. for (k=zs; k From jed at 59A2.org Thu Nov 25 02:37:21 2010 From: jed at 59A2.org (Jed Brown) Date: Thu, 25 Nov 2010 09:37:21 +0100 Subject: [petsc-users] DAGetCorners; xs,ys,zs return 0 for all the processors In-Reply-To: <842748.55775.qm@web112615.mail.gq1.yahoo.com> References: <842748.55775.qm@web112615.mail.gq1.yahoo.com> Message-ID: What communicator did you create your DA on? Can you show us what that call looks like? Jed On Nov 25, 2010 9:19 AM, "khalid ashraf" wrote: Hi, I am having trouble with writing to global vector indexes using DA . I am using the following function to get the starting point of a particular processor. DAGetCorners(da,&xs,&ys,&zs,&xm,&ym,&zm); And then use it to iterate through the global indices. for (k=zs; k From khalid_eee at yahoo.com Thu Nov 25 03:51:05 2010 From: khalid_eee at yahoo.com (khalid ashraf) Date: Thu, 25 Nov 2010 01:51:05 -0800 (PST) Subject: [petsc-users] Global DA index using DAGetCorners Message-ID: <427842.22585.qm@web112604.mail.gq1.yahoo.com> Sorry for the spam. Just to illustrate the problem. The following code produces the following output when run on 4 processors. Code: ierr=DAGetCorners(appctx->da,&xs,&ys,&zs,&xm,&ym,&zm);CHKERRQ(ierr); for (k=zs; k From jed at 59A2.org Thu Nov 25 04:16:04 2010 From: jed at 59A2.org (Jed Brown) Date: Thu, 25 Nov 2010 11:16:04 +0100 Subject: [petsc-users] Global DA index using DAGetCorners In-Reply-To: <427842.22585.qm@web112604.mail.gq1.yahoo.com> References: <427842.22585.qm@web112604.mail.gq1.yahoo.com> Message-ID: On Thu, Nov 25, 2010 at 10:51, khalid ashraf wrote: > Sorry for the spam. Just to illustrate the problem. The following code > produces the > following output when run on 4 processors. > > Code: > ierr=DAGetCorners(appctx->da,&xs,&ys,&zs,&xm,&ym,&zm);CHKERRQ(ierr); > for (k=zs; k for (j=ys; j for (i=xs; i { > Vec_A[k][j][i] = (PetscReal)(k*my*mx+j*mx+i); > }}} > > Output: > Vec_A (mx=16 X my=16 X mz=4): > You have not shown how you produce this output. > 0 1 2 3 4 5 6 7 > 16 17 18 19 20 21 22 23 > 32 33 34 35 36 37 38 39 > 48 49 and so on > > I don't understand why does the array value jump to 16 after 7. Meaning > why is the 8th element of the vector is 16. Thanks. > You are seeing the "PETSc ordering" instead of the "Natural ordering", see Figure 9 of the Users Manual http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manual.pdf#page=50 DAVecGetArray sets up the indexing so you can write your code in terms of the natural ordering even though the actual data are not stored that way. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From khalid_eee at yahoo.com Thu Nov 25 04:24:52 2010 From: khalid_eee at yahoo.com (khalid ashraf) Date: Thu, 25 Nov 2010 02:24:52 -0800 (PST) Subject: [petsc-users] DAGetCorners; xs,ys,zs return 0 for all the processors Message-ID: <588844.78157.qm@web112619.mail.gq1.yahoo.com> Here is the call to DA creation: ierr = DACreate3d(PETSC_COMM_WORLD,DA_NONPERIODIC,DA_STENCIL_BOX,appctx.l,appctx.m,appctx.n, PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,PETSC_NULL,PETSC_NULL,PETSC_NULL, &appctx.da);CHKERRQ(ierr); Khalid -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Thu Nov 25 04:35:43 2010 From: jed at 59A2.org (Jed Brown) Date: Thu, 25 Nov 2010 11:35:43 +0100 Subject: [petsc-users] DAGetCorners; xs,ys,zs return 0 for all the processors In-Reply-To: <588844.78157.qm@web112619.mail.gq1.yahoo.com> References: <588844.78157.qm@web112619.mail.gq1.yahoo.com> Message-ID: On Thu, Nov 25, 2010 at 11:24, khalid ashraf wrote: > Here is the call to DA creation: > > ierr = > DACreate3d(PETSC_COMM_WORLD,DA_NONPERIODIC,DA_STENCIL_BOX,appctx.l,appctx.m,appctx.n, > > PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,PETSC_NULL,PETSC_NULL,PETSC_NULL, > &appctx.da);CHKERRQ(ierr); > If you created the DA as above and then call DAGetCorners(appctx.da,&xs,&ys,&zs,&xm,&ym,&zm); then only on rank=0 will xs,ys,zs all be equal to zero. Your other message about PETSc versus natural ordering indicates that this is working correctly. Is it working now? If not, please send more context, including whatever code you are using to print output. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From khalid_eee at yahoo.com Thu Nov 25 19:50:21 2010 From: khalid_eee at yahoo.com (khalid ashraf) Date: Thu, 25 Nov 2010 17:50:21 -0800 (PST) Subject: [petsc-users] Global DA index using DAGetCorners Message-ID: <113982.45603.qm@web112617.mail.gq1.yahoo.com> It doesn't seem like the problem is with the natural ordering. Since in Fig.9 of the manual, the highest number of the grid should be the same(30 in fig. 9) for natural and petsc ordering. But I get a lower number when I print it. I also tried the petsc to natural conversion using the following code, { ia[0]=(PetscInt)((k)*(my)*(mx)+(j)*(mx)+(i)); AOPetscToApplication(ao,1,ia); w_localptr[k][j][i] = ia[0]; PetscPrintf(PETSC_COMM_WORLD,"%d\n",ia[0]); } But this gives even lower number for the highest index. Anyway my goal is to be able to determine the global natural index of the grid within the for loops in the following way: for (k=zs; k half of the grid along x {Do something} if natural_global_coord[k'][j'][i'] > half of the grid along y {Do something} etc.. }}} Could you please suggest what would be the simplest way to achieve this. Thanks in advance. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Nov 25 20:31:17 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 25 Nov 2010 20:31:17 -0600 Subject: [petsc-users] Global DA index using DAGetCorners In-Reply-To: <113982.45603.qm@web112617.mail.gq1.yahoo.com> References: <113982.45603.qm@web112617.mail.gq1.yahoo.com> Message-ID: On Nov 25, 2010, at 7:50 PM, khalid ashraf wrote: > It doesn't seem like the problem is with the natural ordering. Since in Fig.9 of the manual, > the highest number of the grid should be the same(30 in fig. 9) for natural and petsc ordering. > But I get a lower number when I print it. > > I also tried the petsc to natural conversion using the following code, > { > ia[0]=(PetscInt)((k)*(my)*(mx)+(j)*(mx)+(i)); This give the natural number of the location in ia[0] > AOPetscToApplication(ao,1,ia); So it makes no sense to call AOPetscToApplication() to map it to something else. It is already in the natural (i.e. application ordering) > w_localptr[k][j][i] = ia[0]; > PetscPrintf(PETSC_COMM_WORLD,"%d\n",ia[0]); > } > But this gives even lower number for the highest index. > > Anyway my goal is to be able to determine the global natural index of the > grid within the for loops in the following way: > for (k=zs; k for (j=ys; j for (i=xs; i natural_global_coordinate(k',j',i')=f(k,j,i); > if natural_global_coord[k'][j'][i'] > half of the grid along x {Do something} > if natural_global_coord[k'][j'][i'] > half of the grid along y {Do something} i > m/2 means means it is in the right half of the grid along x (where m is the number of grid points in the x direction). To determine these things you don't need to do anything about mapping between orderings. You are chasing yourself in circles to do something that is essentially trivial. With the loop structure you have above, the i,j,k ARE exactly the logical locations in the global X, Y, Z grid. It couldn't be simplier. Barry > etc.. > }}} > Could you please suggest what would be the simplest way to achieve this. > Thanks in advance. > > > > > From khalid_eee at yahoo.com Thu Nov 25 23:36:08 2010 From: khalid_eee at yahoo.com (khalid ashraf) Date: Thu, 25 Nov 2010 21:36:08 -0800 (PST) Subject: [petsc-users] Printing parallel Vector in VTK format In-Reply-To: References: Message-ID: <804366.55079.qm@web112616.mail.gq1.yahoo.com> Thanks Barry. I get the correct output when I use VecView(w,PETSC_VIEWER_STDOUT_WORLD) But if I use the VecView_VTK function from the file ksp/ksp/examples/tutorials/ex29.c then I get different results. For the code: if (i Subject: [petsc-users] DAGetCorners; xs,ys,zs return 0 for all the processors To: petsc-users at mcs.anl.gov Message-ID: <588844.78157.qm at web112619.mail.gq1.yahoo.com> Content-Type: text/plain; charset="us-ascii" Here is the call to DA creation: ierr = DACreate3d(PETSC_COMM_WORLD,DA_NONPERIODIC,DA_STENCIL_BOX,appctx.l,appctx.m,appctx.n, PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,PETSC_NULL,PETSC_NULL,PETSC_NULL, &appctx.da);CHKERRQ(ierr); Khalid -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 2 Date: Thu, 25 Nov 2010 11:35:43 +0100 From: Jed Brown Subject: Re: [petsc-users] DAGetCorners; xs,ys,zs return 0 for all the processors To: PETSc users list Message-ID: Content-Type: text/plain; charset="utf-8" On Thu, Nov 25, 2010 at 11:24, khalid ashraf wrote: > Here is the call to DA creation: > > ierr = >DACreate3d(PETSC_COMM_WORLD,DA_NONPERIODIC,DA_STENCIL_BOX,appctx.l,appctx.m,appctx.n, >, > > PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,PETSC_NULL,PETSC_NULL,PETSC_NULL, > &appctx.da);CHKERRQ(ierr); > If you created the DA as above and then call DAGetCorners(appctx.da,&xs,&ys,&zs,&xm,&ym,&zm); then only on rank=0 will xs,ys,zs all be equal to zero. Your other message about PETSc versus natural ordering indicates that this is working correctly. Is it working now? If not, please send more context, including whatever code you are using to print output. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ _______________________________________________ petsc-users mailing list petsc-users at mcs.anl.gov https://lists.mcs.anl.gov/mailman/listinfo/petsc-users End of petsc-users Digest, Vol 23, Issue 28 ******************************************* -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Fri Nov 26 03:13:31 2010 From: jed at 59A2.org (Jed Brown) Date: Fri, 26 Nov 2010 10:13:31 +0100 Subject: [petsc-users] Printing parallel Vector in VTK format In-Reply-To: <804366.55079.qm@web112616.mail.gq1.yahoo.com> References: <804366.55079.qm@web112616.mail.gq1.yahoo.com> Message-ID: On Fri, Nov 26, 2010 at 06:36, khalid ashraf wrote: > Thanks Barry. I get the correct output when I use > VecView(w,PETSC_VIEWER_STDOUT_WORLD) > > But if I use the VecView_VTK function from the file > ksp/ksp/examples/tutorials/ex29.c > > then I get different results. > This is a crucial piece of information, if you stated this in the first email, you would have gotten a good answer to your first email. The problem is that this function and the copy in ex50.c, never worked in parallel. Clearly the person who wrote it misunderstood the VTK legacy format because nothing similar can work in parallel. See src/snes/examples/tutorials/ex5.c for the correct way to do it: PetscViewer viewer; ierr = PetscViewerCreate(PETSC_COMM_WORLD, &viewer);CHKERRQ(ierr); ierr = PetscViewerSetType(viewer, PETSCVIEWERASCII);CHKERRQ(ierr); ierr = PetscViewerFileSetName(viewer, "ex5_sol.vtk");CHKERRQ(ierr); ierr = PetscViewerSetFormat(viewer, PETSC_VIEWER_ASCII_VTK);CHKERRQ(ierr); ierr = DAView(user.da, viewer);CHKERRQ(ierr); ierr = VecView(x, viewer);CHKERRQ(ierr); ierr = PetscViewerDestroy(viewer);CHKERRQ(ierr); Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Nov 26 10:42:04 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 26 Nov 2010 10:42:04 -0600 Subject: [petsc-users] Printing parallel Vector in VTK format In-Reply-To: References: <804366.55079.qm@web112616.mail.gq1.yahoo.com> Message-ID: <8A1D2053-2CC5-44E6-AC92-AAE60D7CF966@mcs.anl.gov> Jed, Can you please remove that bad code from those two examples if you haven't yet. Thanks Barry On Nov 26, 2010, at 3:13 AM, Jed Brown wrote: > On Fri, Nov 26, 2010 at 06:36, khalid ashraf wrote: > Thanks Barry. I get the correct output when I use > VecView(w,PETSC_VIEWER_STDOUT_WORLD) > > But if I use the VecView_VTK function from the file > ksp/ksp/examples/tutorials/ex29.c > > then I get different results. > > This is a crucial piece of information, if you stated this in the first email, you would have gotten a good answer to your first email. > > The problem is that this function and the copy in ex50.c, never worked in parallel. Clearly the person who wrote it misunderstood the VTK legacy format because nothing similar can work in parallel. See src/snes/examples/tutorials/ex5.c for the correct way to do it: > > PetscViewer viewer; > ierr = PetscViewerCreate(PETSC_COMM_WORLD, &viewer);CHKERRQ(ierr); > ierr = PetscViewerSetType(viewer, PETSCVIEWERASCII);CHKERRQ(ierr); > ierr = PetscViewerFileSetName(viewer, "ex5_sol.vtk");CHKERRQ(ierr); > ierr = PetscViewerSetFormat(viewer, PETSC_VIEWER_ASCII_VTK);CHKERRQ(ierr); > ierr = DAView(user.da, viewer);CHKERRQ(ierr); > ierr = VecView(x, viewer);CHKERRQ(ierr); > ierr = PetscViewerDestroy(viewer);CHKERRQ(ierr); > > Jed From khalid_eee at yahoo.com Fri Nov 26 17:02:14 2010 From: khalid_eee at yahoo.com (khalid ashraf) Date: Fri, 26 Nov 2010 15:02:14 -0800 (PST) Subject: [petsc-users] Printing parallel Vector in VTK format In-Reply-To: References: Message-ID: <613573.73681.qm@web112607.mail.gq1.yahoo.com> Thanks Jed and Barry. It works fine now. Khalid ________________________________ From: "petsc-users-request at mcs.anl.gov" To: petsc-users at mcs.anl.gov Sent: Fri, November 26, 2010 10:00:19 AM Subject: petsc-users Digest, Vol 23, Issue 29 Send petsc-users mailing list submissions to petsc-users at mcs.anl.gov To subscribe or unsubscribe via the World Wide Web, visit https://lists.mcs.anl.gov/mailman/listinfo/petsc-users or, via email, send a message with subject or body 'help' to petsc-users-request at mcs.anl.gov You can reach the person managing the list at petsc-users-owner at mcs.anl.gov When replying, please edit your Subject line so it is more specific than "Re: Contents of petsc-users digest..." Today's Topics: 1. Global DA index using DAGetCorners (khalid ashraf) 2. Re: Global DA index using DAGetCorners (Barry Smith) 3. Re: Printing parallel Vector in VTK format (khalid ashraf) 4. Re: Printing parallel Vector in VTK format (Jed Brown) 5. Re: Printing parallel Vector in VTK format (Barry Smith) ---------------------------------------------------------------------- Message: 1 Date: Thu, 25 Nov 2010 17:50:21 -0800 (PST) From: khalid ashraf Subject: [petsc-users] Global DA index using DAGetCorners To: petsc-users at mcs.anl.gov Message-ID: <113982.45603.qm at web112617.mail.gq1.yahoo.com> Content-Type: text/plain; charset="us-ascii" It doesn't seem like the problem is with the natural ordering. Since in Fig.9 of the manual, the highest number of the grid should be the same(30 in fig. 9) for natural and petsc ordering. But I get a lower number when I print it. I also tried the petsc to natural conversion using the following code, { ia[0]=(PetscInt)((k)*(my)*(mx)+(j)*(mx)+(i)); AOPetscToApplication(ao,1,ia); w_localptr[k][j][i] = ia[0]; PetscPrintf(PETSC_COMM_WORLD,"%d\n",ia[0]); } But this gives even lower number for the highest index. Anyway my goal is to be able to determine the global natural index of the grid within the for loops in the following way: for (k=zs; k half of the grid along x {Do something} if natural_global_coord[k'][j'][i'] > half of the grid along y {Do something} etc.. }}} Could you please suggest what would be the simplest way to achieve this. Thanks in advance. -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 2 Date: Thu, 25 Nov 2010 20:31:17 -0600 From: Barry Smith Subject: Re: [petsc-users] Global DA index using DAGetCorners To: PETSc users list Message-ID: Content-Type: text/plain; charset=us-ascii On Nov 25, 2010, at 7:50 PM, khalid ashraf wrote: > It doesn't seem like the problem is with the natural ordering. Since in Fig.9 >of the manual, > the highest number of the grid should be the same(30 in fig. 9) for natural and >petsc ordering. > But I get a lower number when I print it. > > I also tried the petsc to natural conversion using the following code, > { > ia[0]=(PetscInt)((k)*(my)*(mx)+(j)*(mx)+(i)); This give the natural number of the location in ia[0] > AOPetscToApplication(ao,1,ia); So it makes no sense to call AOPetscToApplication() to map it to something else. It is already in the natural (i.e. application ordering) > w_localptr[k][j][i] = ia[0]; > PetscPrintf(PETSC_COMM_WORLD,"%d\n",ia[0]); > } > But this gives even lower number for the highest index. > > Anyway my goal is to be able to determine the global natural index of the > grid within the for loops in the following way: > for (k=zs; k for (j=ys; j for (i=xs; i natural_global_coordinate(k',j',i')=f(k,j,i); > if natural_global_coord[k'][j'][i'] > half of the grid along x {Do something} > if natural_global_coord[k'][j'][i'] > half of the grid along y {Do something} i > m/2 means means it is in the right half of the grid along x (where m is the number of grid points in the x direction). To determine these things you don't need to do anything about mapping between orderings. You are chasing yourself in circles to do something that is essentially trivial. With the loop structure you have above, the i,j,k ARE exactly the logical locations in the global X, Y, Z grid. It couldn't be simplier. Barry > etc.. > }}} > Could you please suggest what would be the simplest way to achieve this. > Thanks in advance. > > > > > ------------------------------ Message: 3 Date: Thu, 25 Nov 2010 21:36:08 -0800 (PST) From: khalid ashraf Subject: Re: [petsc-users] Printing parallel Vector in VTK format To: petsc-users at mcs.anl.gov Message-ID: <804366.55079.qm at web112616.mail.gq1.yahoo.com> Content-Type: text/plain; charset="us-ascii" Thanks Barry. I get the correct output when I use VecView(w,PETSC_VIEWER_STDOUT_WORLD) But if I use the VecView_VTK function from the file ksp/ksp/examples/tutorials/ex29.c then I get different results. For the code: if (i Subject: [petsc-users] DAGetCorners; xs,ys,zs return 0 for all the processors To: petsc-users at mcs.anl.gov Message-ID: <588844.78157.qm at web112619.mail.gq1.yahoo.com> Content-Type: text/plain; charset="us-ascii" Here is the call to DA creation: ierr = DACreate3d(PETSC_COMM_WORLD,DA_NONPERIODIC,DA_STENCIL_BOX,appctx.l,appctx.m,appctx.n, PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,PETSC_NULL,PETSC_NULL,PETSC_NULL, &appctx.da);CHKERRQ(ierr); Khalid -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 2 Date: Thu, 25 Nov 2010 11:35:43 +0100 From: Jed Brown Subject: Re: [petsc-users] DAGetCorners; xs,ys,zs return 0 for all the processors To: PETSc users list Message-ID: Content-Type: text/plain; charset="utf-8" On Thu, Nov 25, 2010 at 11:24, khalid ashraf wrote: > Here is the call to DA creation: > > ierr = >DACreate3d(PETSC_COMM_WORLD,DA_NONPERIODIC,DA_STENCIL_BOX,appctx.l,appctx.m,appctx.n, > >, > > PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,PETSC_NULL,PETSC_NULL,PETSC_NULL, > &appctx.da);CHKERRQ(ierr); > If you created the DA as above and then call DAGetCorners(appctx.da,&xs,&ys,&zs,&xm,&ym,&zm); then only on rank=0 will xs,ys,zs all be equal to zero. Your other message about PETSc versus natural ordering indicates that this is working correctly. Is it working now? If not, please send more context, including whatever code you are using to print output. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ _______________________________________________ petsc-users mailing list petsc-users at mcs.anl.gov https://lists.mcs.anl.gov/mailman/listinfo/petsc-users End of petsc-users Digest, Vol 23, Issue 28 ******************************************* -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 4 Date: Fri, 26 Nov 2010 10:13:31 +0100 From: Jed Brown Subject: Re: [petsc-users] Printing parallel Vector in VTK format To: PETSc users list Message-ID: Content-Type: text/plain; charset="utf-8" On Fri, Nov 26, 2010 at 06:36, khalid ashraf wrote: > Thanks Barry. I get the correct output when I use > VecView(w,PETSC_VIEWER_STDOUT_WORLD) > > But if I use the VecView_VTK function from the file > ksp/ksp/examples/tutorials/ex29.c > > then I get different results. > This is a crucial piece of information, if you stated this in the first email, you would have gotten a good answer to your first email. The problem is that this function and the copy in ex50.c, never worked in parallel. Clearly the person who wrote it misunderstood the VTK legacy format because nothing similar can work in parallel. See src/snes/examples/tutorials/ex5.c for the correct way to do it: PetscViewer viewer; ierr = PetscViewerCreate(PETSC_COMM_WORLD, &viewer);CHKERRQ(ierr); ierr = PetscViewerSetType(viewer, PETSCVIEWERASCII);CHKERRQ(ierr); ierr = PetscViewerFileSetName(viewer, "ex5_sol.vtk");CHKERRQ(ierr); ierr = PetscViewerSetFormat(viewer, PETSC_VIEWER_ASCII_VTK);CHKERRQ(ierr); ierr = DAView(user.da, viewer);CHKERRQ(ierr); ierr = VecView(x, viewer);CHKERRQ(ierr); ierr = PetscViewerDestroy(viewer);CHKERRQ(ierr); Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 5 Date: Fri, 26 Nov 2010 10:42:04 -0600 From: Barry Smith Subject: Re: [petsc-users] Printing parallel Vector in VTK format To: PETSc users list Message-ID: <8A1D2053-2CC5-44E6-AC92-AAE60D7CF966 at mcs.anl.gov> Content-Type: text/plain; charset=us-ascii Jed, Can you please remove that bad code from those two examples if you haven't yet. Thanks Barry On Nov 26, 2010, at 3:13 AM, Jed Brown wrote: > On Fri, Nov 26, 2010 at 06:36, khalid ashraf wrote: > Thanks Barry. I get the correct output when I use > VecView(w,PETSC_VIEWER_STDOUT_WORLD) > > But if I use the VecView_VTK function from the file > ksp/ksp/examples/tutorials/ex29.c > > then I get different results. > > This is a crucial piece of information, if you stated this in the first email, >you would have gotten a good answer to your first email. > > The problem is that this function and the copy in ex50.c, never worked in >parallel. Clearly the person who wrote it misunderstood the VTK legacy format >because nothing similar can work in parallel. See >src/snes/examples/tutorials/ex5.c for the correct way to do it: > > PetscViewer viewer; > ierr = PetscViewerCreate(PETSC_COMM_WORLD, &viewer);CHKERRQ(ierr); > ierr = PetscViewerSetType(viewer, PETSCVIEWERASCII);CHKERRQ(ierr); > ierr = PetscViewerFileSetName(viewer, "ex5_sol.vtk");CHKERRQ(ierr); > ierr = PetscViewerSetFormat(viewer, PETSC_VIEWER_ASCII_VTK);CHKERRQ(ierr); > ierr = DAView(user.da, viewer);CHKERRQ(ierr); > ierr = VecView(x, viewer);CHKERRQ(ierr); > ierr = PetscViewerDestroy(viewer);CHKERRQ(ierr); > > Jed ------------------------------ _______________________________________________ petsc-users mailing list petsc-users at mcs.anl.gov https://lists.mcs.anl.gov/mailman/listinfo/petsc-users End of petsc-users Digest, Vol 23, Issue 29 ******************************************* -------------- next part -------------- An HTML attachment was scrubbed... URL: From lizs at mail.uc.edu Sat Nov 27 19:09:28 2010 From: lizs at mail.uc.edu (Li, Zhisong (lizs)) Date: Sun, 28 Nov 2010 01:09:28 +0000 Subject: [petsc-users] matrix computation costs large memory Message-ID: <88D7E3BB7E1960428303E760100374511AA9FC21@BL2PRD0103MB052.prod.exchangelabs.com> Hi, Petsc Team, I have a Petsc code calculating simple structured grid. The dimension is 96 x 25 x 22 with DOF = 6. When running in parallel or sequential processing on a shared memory machine, it costs about 2.5GB RAM when assembling the matrix and more than 5GB when KSP is solving with GMRES. In the code I use the basic DAGetmatrix(da, MATAIJ, &M) to define the matrix, and it is supposed to be a sparse matrix. Is this normal for Petsc application? If not, how to improve this? Thank you. Zhisong Li -------------- next part -------------- An HTML attachment was scrubbed... URL: From khalid_eee at yahoo.com Sat Nov 27 21:33:51 2010 From: khalid_eee at yahoo.com (khalid ashraf) Date: Sat, 27 Nov 2010 19:33:51 -0800 (PST) Subject: [petsc-users] Supplying the RHS Vector to the DMMG solver from outside Message-ID: <24688.1756.qm@web112613.mail.gq1.yahoo.com> Hi, I am trying to use the DMMG solver in /ksp/ksp/examples/tutorials/ex22.c where I have to use the PETSC defined function ComputeRHS. Here instead of setting the constant value h, I want to copy the values from another Vector (associated with another DA, but same size). Hence I added the line opt 2) as shown below. I can see that the value has been copied into b, but the DMMG solve produces 0. Whereas opt 1) works fine. It seems like the problem could be due to copying from a vector associated with another DA. Could you please suggest what can be done to make it work. ComputeRHS(DMMG,Vec) { PetscFunctionBegin; h = 1.0/((mx-1)*(my-1)*(mz-1)); opt 1) PETSC_Provided: ierr = VecSet(b,h);CHKERRQ(ierr); opt 2) Value from my Vector: ierr= VecCopy(globalVec,b); VecView(b); //I can see that the value has been copied to b PetscFunctionReturn(0); } I have also tried copying element by element using row=0; for (k=zs; k From bsmith at mcs.anl.gov Sat Nov 27 21:39:50 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 27 Nov 2010 21:39:50 -0600 Subject: [petsc-users] Supplying the RHS Vector to the DMMG solver from outside In-Reply-To: <24688.1756.qm@web112613.mail.gq1.yahoo.com> References: <24688.1756.qm@web112613.mail.gq1.yahoo.com> Message-ID: <6BDF0ACC-5048-41E0-B433-8FD6B78D16A5@mcs.anl.gov> On Nov 27, 2010, at 9:33 PM, khalid ashraf wrote: > Hi, I am trying to use the DMMG solver in /ksp/ksp/examples/tutorials/ex22.c > where I have to use the PETSC defined function ComputeRHS. > Here instead of setting the constant value h, I want to copy the values from another Vector (associated with another DA, but same size). Hence I added the line opt 2) as shown below. This should work fine. What are the results when you run with -ksp_monitor_true_residual -ksp_converged_reason No error messages are generated when you run it? Barry > I can see that the value has been copied into b, but the DMMG solve produces 0. Whereas opt 1) works fine. It seems like the problem could be due to copying from a vector associated with another DA. Could you please suggest what can be done to make it work. > > ComputeRHS(DMMG,Vec) > { > PetscFunctionBegin; > h = 1.0/((mx-1)*(my-1)*(mz-1)); > > opt 1) PETSC_Provided: ierr = VecSet(b,h);CHKERRQ(ierr); > opt 2) Value from my Vector: ierr= VecCopy(globalVec,b); > > VecView(b); //I can see that the value has been copied to b > > PetscFunctionReturn(0); > } > > I have also tried copying element by element using > > row=0; > for (k=zs; k for (j=ys; j for (i=xs; i temp=vec_globalptr[k][j][i]; > VecSetValue(b,row,temp,INSERT_VALUES); > row=row+1; > } > ierr = VecAssemblyBegin(b);CHKERRQ(ierr); > ierr = VecAssemblyEnd(b);CHKERRQ(ierr); > > In this case as well I can see that the values have been added into vector b, but the DMMG solve produces all zeros for DMMGGetx(); Any suggestions ? > > Thanks in advance. > > Khalid > > > From bsmith at mcs.anl.gov Sat Nov 27 22:02:49 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 27 Nov 2010 22:02:49 -0600 Subject: [petsc-users] matrix computation costs large memory In-Reply-To: <88D7E3BB7E1960428303E760100374511AA9FC21@BL2PRD0103MB052.prod.exchangelabs.com> References: <88D7E3BB7E1960428303E760100374511AA9FC21@BL2PRD0103MB052.prod.exchangelabs.com> Message-ID: If you are using the default DACreate3d() with a stencil width of one and a star stencil then each row of the matrix has 7*6*6 nonzeros because it has to assume each each of the six dof is coupled with any of the six dof in any of the six directions. The number of nonzeros in the matrix is then roughly 96 x 25 x 22 * 6 * 7*6*6, with the AIJ format it takes essentially 12 bytes per nonzero so you end up with 958,003,200 = 1 gigabyte. If you used the DA_STENCIL_BOX then the 7 is replaced with 27 so it takes roughly 3.7 gigabytes for a matrix. If you used --with-64-bit-indices=1 when configuring PETSc then the 12 is replaced with 16 and the matrix would take 1.3 gigabytes. If you used the default PC ILU(0) then the preconditioner takes basically the same amount of space as the matrix which explains why the memory usage would double during the solve. Unless my computations above are wrong I cannot explain why it is taking 2.5 g just for the original matrix. Did you ever put into the matrix a row using more then the stencil width that you provided? For example if you set a stencil width of 1 but on some boundaries you actually put entries into the matrix of stencil width 2 this would blow up the memory usage a lot. Or are you using a stencil width of 2? How do you know it is using 2.5 gig originally? Regardless: if the little 6 by 6 blocks are actually dense then you should switch to the BAIJ format, since the memory usage drops from 12 bytes per nonzero to essentially 8 bytes per nonzero. If the 6 by 6 blocks are sparse which is very likely then you should use the routine DASetBlockFills() to indicate the sparsity of these little blocks. This can save you an enormous amount of memory. Barry On Nov 27, 2010, at 7:09 PM, Li, Zhisong (lizs) wrote: > Hi, Petsc Team, > > I have a Petsc code calculating simple structured grid. The dimension is 96 x 25 x 22 with DOF = 6. When running in parallel or sequential processing on a shared memory machine, it costs about 2.5GB RAM when assembling the matrix and more than 5GB when KSP is solving with GMRES. In the code I use the basic DAGetmatrix(da, MATAIJ, &M) to define the matrix, and it is supposed to be a sparse matrix. Is this normal for Petsc application? If not, how to improve this? > > > Thank you. > > > Zhisong Li From khalid_eee at yahoo.com Sat Nov 27 22:02:56 2010 From: khalid_eee at yahoo.com (khalid ashraf) Date: Sat, 27 Nov 2010 20:02:56 -0800 (PST) Subject: [petsc-users] Supplying the RHS Vector to the DMMG solver from outside Message-ID: <517451.40673.qm@web112618.mail.gq1.yahoo.com> Here is the output with opt 1) VecSet(b,h): 0 KSP preconditioned resid norm 5.656854249492e+00 true resid norm 5.656854249492e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.256073966947e-15 true resid norm 8.881784197001e-16 ||Ae||/||Ax|| 1.570092458684e-16 Linear solve converged due to CONVERGED_RTOL iterations Process [0] 0.666667 0.666667 Process [1] 0.666667 0.666667 Process [2] 0.666667 0.666667 Process [3] 0.666667 0.666667 0 KSP preconditioned resid norm 5.656854249492e+00 true resid norm 5.656854249492e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.256073966947e-15 true resid norm 8.881784197001e-16 ||Ae||/||Ax|| 1.570092458684e-16 Linear solve converged due to CONVERGED_RTOL iterations 1 0 KSP preconditioned resid norm 5.656854249492e+00 true resid norm 5.656854249492e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.256073966947e-15 true resid norm 8.881784197001e-16 ||Ae||/||Ax|| 1.570092458684e-16 Linear solve converged due to CONVERGED_RTOL iterations 1 Here is the output with opt 2) Copied from another function: Linear solve converged due to CONVERGED_ATOL iterations 0 Process [0] 0 0 Process [1] 0 0 Process [2] 0 0 Process [3] 0 0 0 KSP preconditioned resid norm 7.515606252360e+01 true resid norm 7.515606252360e+01 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.014114995652e+01 true resid norm 2.014114995652e+01 ||Ae||/||Ax|| 2.679910213523e-01 2 KSP preconditioned resid norm 6.102231836870e+00 true resid norm 6.102231836870e+00 ||Ae||/||Ax|| 8.119413966044e-02 3 KSP preconditioned resid norm 3.582243925876e-01 true resid norm 3.582243925876e-01 ||Ae||/||Ax|| 4.766407134156e-03 4 KSP preconditioned resid norm 9.891345584245e-02 true resid norm 9.891345584245e-02 ||Ae||/||Ax|| 1.316107477176e-03 5 KSP preconditioned resid norm 3.103414625576e-14 true resid norm 2.787743265007e-14 ||Ae||/||Ax|| 3.709272640689e-16 Linear solve converged due to CONVERGED_RTOL iterations 5 0 KSP preconditioned resid norm 7.567218624719e+01 true resid norm 7.567218624719e+01 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.027946648035e+01 true resid norm 2.027946648035e+01 ||Ae||/||Ax|| 2.679910213523e-01 2 KSP preconditioned resid norm 6.144138058565e+00 true resid norm 6.144138058565e+00 ||Ae||/||Ax|| 8.119413966044e-02 3 KSP preconditioned resid norm 3.606844483858e-01 true resid norm 3.606844483858e-01 ||Ae||/||Ax|| 4.766407134156e-03 4 KSP preconditioned resid norm 9.959273013421e-02 true resid norm 9.959273013421e-02 ||Ae||/||Ax|| 1.316107477176e-03 5 KSP preconditioned resid norm 2.724152110884e-14 true resid norm 3.587687035382e-14 ||Ae||/||Ax|| 4.741090767039e-16 Linear solve converged due to CONVERGED_RTOL iterations 5 Here the printed zeros are the print of DMMGGetx() Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Nov 28 11:09:00 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 28 Nov 2010 11:09:00 -0600 Subject: [petsc-users] Supplying the RHS Vector to the DMMG solver from outside In-Reply-To: <517451.40673.qm@web112618.mail.gq1.yahoo.com> References: <517451.40673.qm@web112618.mail.gq1.yahoo.com> Message-ID: <1D0309EA-1A68-4D84-86A9-1BD31D3D507A@mcs.anl.gov> You are not providing full information to determine the problem. Why are there three linear solves? You never said anything about three linear solves. Why are you printing the solution after the first linear solve? Likely that "first" linear solve where you got CONVERGED_ATOL the right hand side is never set or it is set to zero. Barry On Nov 27, 2010, at 10:02 PM, khalid ashraf wrote: > Here is the output with opt 1) VecSet(b,h): > > 0 KSP preconditioned resid norm 5.656854249492e+00 true resid norm 5.656854249492e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.256073966947e-15 true resid norm 8.881784197001e-16 ||Ae||/||Ax|| 1.570092458684e-16 > Linear solve converged due to CONVERGED_RTOL iterations > Process [0] > 0.666667 > 0.666667 > Process [1] > 0.666667 > 0.666667 > Process [2] > 0.666667 > 0.666667 > Process [3] > 0.666667 > 0.666667 > 0 KSP preconditioned resid norm 5.656854249492e+00 true resid norm 5.656854249492e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.256073966947e-15 true resid norm 8.881784197001e-16 ||Ae||/||Ax|| 1.570092458684e-16 > Linear solve converged due to CONVERGED_RTOL iterations 1 > 0 KSP preconditioned resid norm 5.656854249492e+00 true resid norm 5.656854249492e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.256073966947e-15 true resid norm 8.881784197001e-16 ||Ae||/||Ax|| 1.570092458684e-16 > Linear solve converged due to CONVERGED_RTOL iterations 1 > > > Here is the output with opt 2) Copied from another function: > > > Linear solve converged due to CONVERGED_ATOL iterations 0 > Process [0] > 0 > 0 > Process [1] > 0 > 0 > Process [2] > 0 > 0 > Process [3] > 0 > 0 > 0 KSP preconditioned resid norm 7.515606252360e+01 true resid norm 7.515606252360e+01 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.014114995652e+01 true resid norm 2.014114995652e+01 ||Ae||/||Ax|| 2.679910213523e-01 > 2 KSP preconditioned resid norm 6.102231836870e+00 true resid norm 6.102231836870e+00 ||Ae||/||Ax|| 8.119413966044e-02 > 3 KSP preconditioned resid norm 3.582243925876e-01 true resid norm 3.582243925876e-01 ||Ae||/||Ax|| 4.766407134156e-03 > 4 KSP preconditioned resid norm 9.891345584245e-02 true resid norm 9.891345584245e-02 ||Ae||/||Ax|| 1.316107477176e-03 > 5 KSP preconditioned resid norm 3.103414625576e-14 true resid norm 2.787743265007e-14 ||Ae||/||Ax|| 3.709272640689e-16 > Linear solve converged due to CONVERGED_RTOL iterations 5 > 0 KSP preconditioned resid norm 7.567218624719e+01 true resid norm 7.567218624719e+01 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.027946648035e+01 true resid norm 2.027946648035e+01 ||Ae||/||Ax|| 2.679910213523e-01 > 2 KSP preconditioned resid norm 6.144138058565e+00 true resid norm 6.144138058565e+00 ||Ae||/||Ax|| 8.119413966044e-02 > 3 KSP preconditioned resid norm 3.606844483858e-01 true resid norm 3.606844483858e-01 ||Ae||/||Ax|| 4.766407134156e-03 > 4 KSP preconditioned resid norm 9.959273013421e-02 true resid norm 9.959273013421e-02 ||Ae||/||Ax|| 1.316107477176e-03 > 5 KSP preconditioned resid norm 2.724152110884e-14 true resid norm 3.587687035382e-14 ||Ae||/||Ax|| 4.741090767039e-16 > Linear solve converged due to CONVERGED_RTOL iterations 5 > > > Here the printed zeros are the print of DMMGGetx() > > Thanks. > > From rongliang.chan at gmail.com Sun Nov 28 14:11:50 2010 From: rongliang.chan at gmail.com (Rongliang Chen) Date: Sun, 28 Nov 2010 13:11:50 -0700 Subject: [petsc-users] Why MatMult(A, x, Ax) != A*x? Message-ID: Hi, I am having trouble with using the function MatMult in PETSc. The result of MatMult(A, x, Ax) (*using multiprocessors*) is not equal to the result of A*x computed in Matlab. The format of the matrix A is AIJ and is obtained by function SNESDefaultComputeJacobian. I compute A*x in Matlab with the following A and x and compare it with Ax. ....................... ierr = MatMult(*A, x, Ax);CHKERRQ(ierr); sprintf(filename,"x.m"); ierr = PetscViewerASCIIOpen(PETSC_COMM_WORLD,filename,&view->viewer);CHKERRQ(ierr); ierr = PetscViewerSetFormat(view->viewer, PETSC_VIEWER_ASCII_MATLAB);CHKERRQ(ierr); ierr = VecView(x, view->viewer);CHKERRQ(ierr); sprintf(filename,"Ax.m"); ierr = PetscViewerASCIIOpen(PETSC_COMM_WORLD,filename,&view->viewer);CHKERRQ(ierr); ierr = PetscViewerSetFormat(view->viewer, PETSC_VIEWER_ASCII_MATLAB);CHKERRQ(ierr); ierr = VecView(Ax, view->viewer);CHKERRQ(ierr); ..................... What kind of bug can cause this problem? Thanks. Best, Rongliang -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Sun Nov 28 14:18:59 2010 From: jed at 59A2.org (Jed Brown) Date: Sun, 28 Nov 2010 21:18:59 +0100 Subject: [petsc-users] Why MatMult(A, x, Ax) != A*x? In-Reply-To: References: Message-ID: On Sun, Nov 28, 2010 at 21:11, Rongliang Chen wrote: > The format of the matrix A is AIJ and is obtained by function > SNESDefaultComputeJacobian. > I compute A*x in Matlab with the following A and x and compare it with Ax. > > ....................... > ierr = MatMult(*A, x, Ax);CHKERRQ(ierr); > > sprintf(filename,"x.m"); > ierr = > PetscViewerASCIIOpen(PETSC_COMM_WORLD,filename,&view->viewer);CHKERRQ(ierr); > ierr = PetscViewerSetFormat(view->viewer, > PETSC_VIEWER_ASCII_MATLAB);CHKERRQ(ierr); > ierr = VecView(x, view->viewer);CHKERRQ(ierr); > > sprintf(filename,"Ax.m"); > ierr = > PetscViewerASCIIOpen(PETSC_COMM_WORLD,filename,&view->viewer);CHKERRQ(ierr); > ierr = PetscViewerSetFormat(view->viewer, > PETSC_VIEWER_ASCII_MATLAB);CHKERRQ(ierr); > ierr = VecView(Ax, view->viewer);CHKERRQ(ierr); > Are the vectors and matrices obtained from a DA (DACreateGlobalVector, DAGetMatrix)? How are you comparing the matrix that PETSc sees with the matrix that Matlab sees? I suggest writing the matrix and both vectors to a PETSc binary file PetscViewer viewer = PETSC_VIEWER_BINARY(PETSC_COMM_WORLD); MatView(A,viewer); VecView(x,viewer); MatMult(A,x,y); VecView(y,viewer); and read them into Matlab with [A,x,y] = PetscBinaryRead('binaryoutput') norm(A*x - y) % This should be small Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Sun Nov 28 14:20:41 2010 From: jed at 59A2.org (Jed Brown) Date: Sun, 28 Nov 2010 21:20:41 +0100 Subject: [petsc-users] Why MatMult(A, x, Ax) != A*x? In-Reply-To: References: Message-ID: On Sun, Nov 28, 2010 at 21:18, Jed Brown wrote: > PetscViewer viewer = PETSC_VIEWER_BINARY(PETSC_COMM_WORLD); > This line should read: PetscViewer viewer = PETSC_VIEWER_BINARY_(PETSC_COMM_WORLD); You can use PetscViewerBinaryOpen for more control, and to specify the filename. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From pengxwang at hotmail.com Mon Nov 29 11:46:57 2010 From: pengxwang at hotmail.com (Peter Wang) Date: Mon, 29 Nov 2010 11:46:57 -0600 Subject: [petsc-users] Should the vactor be partitioned? Message-ID: I am trying to solver a sparse linear system (Ax=b) by PETSc. I noticed the parallel partitioning of the matrix is determined by PETSc at runtime by calling MatGetOwnerShipRange(). I am wondering if the right hand side vector (b) should be partitioned by Calling VecGetOwnerShipRange(), Since I didn't see it is called in example code of ex2f.F on webpage of : http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-2.3.3/src/ksp/ksp/examples/tutorials/ex2f.F.html . Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Nov 29 11:50:59 2010 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 29 Nov 2010 11:50:59 -0600 (CST) Subject: [petsc-users] Should the vactor be partitioned? In-Reply-To: References: Message-ID: If you are using the default partitioning for both Mat and Vec - then they will match. [provided the sizes match] Satish On Mon, 29 Nov 2010, Peter Wang wrote: > > I am trying to solver a sparse linear system (Ax=b) by PETSc. > > I noticed the parallel partitioning of the matrix is determined by PETSc at runtime by calling MatGetOwnerShipRange(). > > I am wondering if the right hand side vector (b) should be partitioned by Calling VecGetOwnerShipRange(), Since I didn't see it is called in example code of ex2f.F on webpage of : http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-2.3.3/src/ksp/ksp/examples/tutorials/ex2f.F.html . > > Thanks. > From pengxwang at hotmail.com Mon Nov 29 12:05:14 2010 From: pengxwang at hotmail.com (Peter Wang) Date: Mon, 29 Nov 2010 12:05:14 -0600 Subject: [petsc-users] Should the vactor be partitioned? In-Reply-To: References: Message-ID: Thanks. One more question. Should the vactor (b) be assembled after calling VecSetValues() ? From: pengxwang at hotmail.com To: petsc-users at mcs.anl.gov Date: Mon, 29 Nov 2010 11:46:57 -0600 Subject: [petsc-users] Should the vactor be partitioned? I am trying to solver a sparse linear system (Ax=b) by PETSc. I noticed the parallel partitioning of the matrix is determined by PETSc at runtime by calling MatGetOwnerShipRange(). I am wondering if the right hand side vector (b) should be partitioned by Calling VecGetOwnerShipRange(), Since I didn't see it is called in example code of ex2f.F on webpage of : http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-2.3.3/src/ksp/ksp/examples/tutorials/ex2f.F.html . Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Mon Nov 29 12:06:17 2010 From: jed at 59A2.org (Jed Brown) Date: Mon, 29 Nov 2010 19:06:17 +0100 Subject: [petsc-users] Should the vactor be partitioned? In-Reply-To: References: Message-ID: On Mon, Nov 29, 2010 at 19:05, Peter Wang wrote: > Should the vactor (b) be assembled after calling VecSetValues() ? Yes. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenway at utias.utoronto.ca Mon Nov 29 13:01:17 2010 From: kenway at utias.utoronto.ca (Gaetan Kenway) Date: Mon, 29 Nov 2010 14:01:17 -0500 Subject: [petsc-users] ASM Preconditioner with SNES Message-ID: <4CF3F87D.4090105@utias.utoronto.ca> Hello I am wondering how one sets options for the sub-ksp contexts when using the ASM preconditoner for the krylov solve in a SNES. After I create the snes context, set the FormFunction routine, set the snesSetJacobian routine, I run something like this: call SNESGetKSP(snes,ksp,ierr) call KSPSetType(ksp,KSPGMRES,ierr) call KSPGMRESSetRestart(ksp, 80, ierr) call KSPGetPC(ksp,pc,ierr) call PCSetType( pc, PCASM, ierr) call PCASMSetOverlap(pc,3,ierr) All good up until here. Elsewhere where I use just a Krylov solve (for the adjoint of the non-linear system), I would normally use call PCASMGetSubKSP( pc, nlocal, first, subksp, ierr) call KSPGetPC( subksp, subpc, ierr) call PCSetType( subpc, PCILU, ierr) call PCFactorSetLevels( subpc, fillLevel , ierr) call KSPSetType(subksp, KSPPREONLY, ierr) However, calling the PCASMGetSubKSP gives an error, specifically Error Code 58: "Operation done in wrong order". Any thoughts on how to set these options for the subksp in the ASM pre-conditioner? Thank you, Gaetan From jed at 59A2.org Mon Nov 29 13:08:33 2010 From: jed at 59A2.org (Jed Brown) Date: Mon, 29 Nov 2010 20:08:33 +0100 Subject: [petsc-users] ASM Preconditioner with SNES In-Reply-To: <4CF3F87D.4090105@utias.utoronto.ca> References: <4CF3F87D.4090105@utias.utoronto.ca> Message-ID: On Mon, Nov 29, 2010 at 20:01, Gaetan Kenway wrote: > I am wondering how one sets options for the sub-ksp contexts when using the > ASM preconditoner for the krylov solve in a SNES. After I create the snes > context, set the FormFunction routine, set the snesSetJacobian routine, I > run something like this: > > call SNESGetKSP(snes,ksp,ierr) > call KSPSetType(ksp,KSPGMRES,ierr) > call KSPGMRESSetRestart(ksp, 80, ierr) > call KSPGetPC(ksp,pc,ierr) > call PCSetType( pc, PCASM, ierr) > call PCASMSetOverlap(pc,3,ierr) > > All good up until here. Elsewhere where I use just a Krylov solve (for the > adjoint of the non-linear system), I would normally use > > call PCASMGetSubKSP( pc, nlocal, first, subksp, ierr) > call KSPGetPC( subksp, subpc, ierr) > call PCSetType( subpc, PCILU, ierr) > call PCFactorSetLevels( subpc, fillLevel , ierr) > call KSPSetType(subksp, KSPPREONLY, ierr) > Note that you are basically just setting defaults, but maybe this is a model for where your application takes active control of the algorithm? However, calling the PCASMGetSubKSP gives an error, specifically Error Code > 58: "Operation done in wrong order". > The these cannot be created (or they wouldn't be useful yet) because we don't have a matrix yet, so we can't know how big the overlap 3 system will be. You can call these functions after PCSetUp, but you need to set a matrix (with correct nonzero, not necessarily correct values) to do PCSetUp. So you should either find a way to provide that information before SNESSolve() and call PCSetUp before PCASMGetSubKSP, or use the options database to customize the inner solvers. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From kjf198 at cse.psu.edu Mon Nov 29 13:15:42 2010 From: kjf198 at cse.psu.edu (Kelly Fermoyle) Date: Mon, 29 Nov 2010 14:15:42 -0500 (EST) Subject: [petsc-users] MatPermuteSparsify hanging Message-ID: <58888.130.203.38.76.1291058142.squirrel@mail.cse.psu.edu> Hi, I'm having a problem with MatPermuteSparsify. The function is hanging indefinitely, but does not give an error of any kind. Below is my relevant code. It definitely gets through MatGetOrdering fine, but never prints the line following MatPermuteSparsify. I tried it in the debugger, and it said that aprun was calling poll(), I don't know what that means. Thanks! Kelly PetscInitialize(&argc, &argv, (char*)0, "Loads a linear system and solves it with given solver options.\n"); PetscViewer mat_view; PetscViewerBinaryOpen(PETSC_COMM_WORLD, mat_file, FILE_MODE_READ, &mat_view); MatLoad(mat_view, MATSEQAIJ, &A); PetscViewerDestroy(mat_view); PetscViewer vec_view; PetscViewerBinaryOpen(PETSC_COMM_WORLD, vec_file, FILE_MODE_READ, &vec_view); VecLoad(vec_view, VECSEQ, &b); PetscViewerDestroy(vec_view); ierr = VecDuplicate(b, &x); CHKERRQ(ierr); if(strcmp(reordering, "1wd") == 0){ type = MATORDERING_1WD; } else if(strcmp(reordering, "natural") == 0) { type = MATORDERING_NATURAL; } else if(strcmp(reordering, "nd") == 0) { type = MATORDERING_ND; } else if(strcmp(reordering, "qmd") == 0) { type = MATORDERING_QMD; } else if(strcmp(reordering, "rcm") == 0) { type = MATORDERING_RCM; } else { printf("%s is not a valid reordering!\n", reordering); exit(-1); } MatGetOrdering(A, type, &rowp, &colp); MatPermuteSparsify(A, -1, 0, threshold, rowp, colp, &P); printf("Got sparse version.\n"); From bsmith at mcs.anl.gov Mon Nov 29 13:21:03 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 29 Nov 2010 13:21:03 -0600 Subject: [petsc-users] MatPermuteSparsify hanging In-Reply-To: <58888.130.203.38.76.1291058142.squirrel@mail.cse.psu.edu> References: <58888.130.203.38.76.1291058142.squirrel@mail.cse.psu.edu> Message-ID: Use ./programname -start_in_debugger noxterm then type cont in the debugger, after it seems to be "hanging" then use control c and type where, this should show where the program is "hanging". Likely it is not hanging, it is just taking forever. Barry On Nov 29, 2010, at 1:15 PM, Kelly Fermoyle wrote: > Hi, > > I'm having a problem with MatPermuteSparsify. The function is hanging > indefinitely, but does not give an error of any kind. Below is my relevant > code. It definitely gets through MatGetOrdering fine, but never prints the > line following MatPermuteSparsify. I tried it in the debugger, and it said > that aprun was calling poll(), I don't know what that means. > > Thanks! > > Kelly > > > > PetscInitialize(&argc, &argv, (char*)0, > "Loads a linear system and solves it with given solver options.\n"); > > PetscViewer mat_view; > PetscViewerBinaryOpen(PETSC_COMM_WORLD, mat_file, FILE_MODE_READ, > &mat_view); > MatLoad(mat_view, MATSEQAIJ, &A); > PetscViewerDestroy(mat_view); > > PetscViewer vec_view; > PetscViewerBinaryOpen(PETSC_COMM_WORLD, vec_file, FILE_MODE_READ, > &vec_view); > VecLoad(vec_view, VECSEQ, &b); > PetscViewerDestroy(vec_view); > ierr = VecDuplicate(b, &x); CHKERRQ(ierr); > > if(strcmp(reordering, "1wd") == 0){ > type = MATORDERING_1WD; > } else if(strcmp(reordering, "natural") == 0) { > type = MATORDERING_NATURAL; > } else if(strcmp(reordering, "nd") == 0) { > type = MATORDERING_ND; > } else if(strcmp(reordering, "qmd") == 0) { > type = MATORDERING_QMD; > } else if(strcmp(reordering, "rcm") == 0) { > type = MATORDERING_RCM; > } else { > printf("%s is not a valid reordering!\n", reordering); > exit(-1); > } > MatGetOrdering(A, type, &rowp, &colp); > MatPermuteSparsify(A, -1, 0, threshold, rowp, colp, &P); > printf("Got sparse version.\n"); > From jed at 59A2.org Mon Nov 29 13:22:01 2010 From: jed at 59A2.org (Jed Brown) Date: Mon, 29 Nov 2010 20:22:01 +0100 Subject: [petsc-users] MatPermuteSparsify hanging In-Reply-To: <58888.130.203.38.76.1291058142.squirrel@mail.cse.psu.edu> References: <58888.130.203.38.76.1291058142.squirrel@mail.cse.psu.edu> Message-ID: On Mon, Nov 29, 2010 at 20:15, Kelly Fermoyle wrote: > I tried it in the debugger, and it said > that aprun was calling poll(), I don't know what that means. > Type "backtrace full" and send the results. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From kjf198 at cse.psu.edu Mon Nov 29 13:28:55 2010 From: kjf198 at cse.psu.edu (Kelly Fermoyle) Date: Mon, 29 Nov 2010 14:28:55 -0500 (EST) Subject: [petsc-users] MatPermuteSparsify hanging In-Reply-To: References: <58888.130.203.38.76.1291058142.squirrel@mail.cse.psu.edu> Message-ID: <58964.130.203.38.76.1291058935.squirrel@mail.cse.psu.edu> Barry, I am using a Cray machine and it requires that I call as aprun -n 1 ./prog_name, which does not seem to work with -start_in_debugger. Jed, Below is the output of the full backtrace: -Kelly [Thread debugging using libthread_db enabled] [New Thread 47303838703712 (LWP 30863)] Program received signal SIGINT, Interrupt. [Switching to Thread 47303838703712 (LWP 30863)] 0x00002b05c868197f in poll () from /lib64/libc.so.6 (gdb) backtrace full #0 0x00002b05c868197f in poll () from /lib64/libc.so.6 No symbol table info available. #1 0x0000000000405ec6 in control_loop (argc=20, argv=0x7fffe2814538) at aprun.c:1137 l = (List) 0x58b6d0 i = 4 pollRet = 0 ret = 0 numClients = 0 currNumPoll = 4 connAttempt = 0 controlConnClosed = 0 readFail = 0 apsysIndex = 0 pipeInt = 0 nRead = 0 dropConnCheck = 0 xferCnt = 0 chkpntBlocked = 1 apsys_socket_temp = 0 buf = '\0' , "?gW\000\000\000\000\000`hW\000\000\000\000\000\220?X\000\000\000\000\000 `W\000\000\000\000\000 `W\000\000\000\000\000\000?W\000\000\000\000\000/usr/bin/gzip --fast -c -- ./SolveTest.exe", '\0' emsg = 0x0 tcpInfoMsg = 0x0 tcpInfo = {tcpi_state = 0 '\0', tcpi_ca_state = 0 '\0', tcpi_retransmits = 0 '\0', tcpi_probes = 0 '\0', tcpi_backoff = 0 '\0', tcpi_options = 0 '\0', tcpi_snd_wscale = 0 '\0', tcpi_rcv_wscale = 0 '\0', tcpi_rto = 0, tcpi_ato = 0, tcpi_snd_mss = 0, tcpi_rcv_mss = 0, tcpi_unacked = 0, tcpi_sacked = 0, tcpi_lost = 0, tcpi_retrans = 0, tcpi_fackets = 0, tcpi_last_data_sent = 0, tcpi_last_ack_sent = 0, tcpi_last_data_recv = 0, tcpi_last_ack_recv = 5731168, tcpi_pmtu = 0, tcpi_rcv_ssthresh = 1, tcpi_rtt = 1, tcpi_rttvar = 0, tcpi_snd_ssthresh = 0, tcpi_snd_cwnd = 0, tcpi_advmss = 0, tcpi_reordering = 0} timeNow = 0 lastConnLog = 0 restartLaunch = (XMLRPC_REQUEST) 0x0 #2 0x0000000000403be7 in main (argc=20, argv=0x7fffe2814538) at aprun.c:225 maxNumPoll = 1024 rlim = {rlim_cur = 1024, rlim_max = 1024} launch = (XMLRPC_REQUEST) 0x576980 emsg = 0x0 (gdb) > On Mon, Nov 29, 2010 at 20:15, Kelly Fermoyle wrote: > >> I tried it in the debugger, and it said >> that aprun was calling poll(), I don't know what that means. >> > > Type "backtrace full" and send the results. > > Jed > From jed at 59A2.org Mon Nov 29 13:32:11 2010 From: jed at 59A2.org (Jed Brown) Date: Mon, 29 Nov 2010 20:32:11 +0100 Subject: [petsc-users] MatPermuteSparsify hanging In-Reply-To: <58964.130.203.38.76.1291058935.squirrel@mail.cse.psu.edu> References: <58888.130.203.38.76.1291058142.squirrel@mail.cse.psu.edu> <58964.130.203.38.76.1291058935.squirrel@mail.cse.psu.edu> Message-ID: On Mon, Nov 29, 2010 at 20:28, Kelly Fermoyle wrote: > I am using a Cray machine and it requires that I call as aprun -n 1 > ./prog_name, which does not seem to work with -start_in_debugger. > Can you reproduce with the same code on a local machine (e.g. desktop/laptop) where you can get a proper debugging session? > Below is the output of the full backtrace: > This trace is not even in your code > #2 0x0000000000403be7 in main (argc=20, argv=0x7fffe2814538) at aprun.c:225 Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Nov 29 13:33:01 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 29 Nov 2010 13:33:01 -0600 Subject: [petsc-users] MatPermuteSparsify hanging In-Reply-To: <58964.130.203.38.76.1291058935.squirrel@mail.cse.psu.edu> References: <58888.130.203.38.76.1291058142.squirrel@mail.cse.psu.edu> <58964.130.203.38.76.1291058935.squirrel@mail.cse.psu.edu> Message-ID: Since it is a Cray it should have TotalView debugger installed. You can run in that and then see where it is when it "hangs". Barry On Nov 29, 2010, at 1:28 PM, Kelly Fermoyle wrote: > Barry, > > I am using a Cray machine and it requires that I call as aprun -n 1 > ./prog_name, which does not seem to work with -start_in_debugger. > > Jed, > > Below is the output of the full backtrace: > > -Kelly > > > [Thread debugging using libthread_db enabled] > [New Thread 47303838703712 (LWP 30863)] > > Program received signal SIGINT, Interrupt. > [Switching to Thread 47303838703712 (LWP 30863)] > 0x00002b05c868197f in poll () from /lib64/libc.so.6 > (gdb) backtrace full > #0 0x00002b05c868197f in poll () from /lib64/libc.so.6 > No symbol table info available. > #1 0x0000000000405ec6 in control_loop (argc=20, argv=0x7fffe2814538) at > aprun.c:1137 > l = (List) 0x58b6d0 > i = 4 > pollRet = 0 > ret = 0 > numClients = 0 > currNumPoll = 4 > connAttempt = 0 > controlConnClosed = 0 > readFail = 0 > apsysIndex = 0 > pipeInt = 0 > nRead = 0 > dropConnCheck = 0 > xferCnt = 0 > chkpntBlocked = 1 > apsys_socket_temp = 0 > buf = '\0' , > "?gW\000\000\000\000\000`hW\000\000\000\000\000\220?X\000\000\000\000\000 > `W\000\000\000\000\000 > `W\000\000\000\000\000\000?W\000\000\000\000\000/usr/bin/gzip --fast -c > -- ./SolveTest.exe", '\0' > emsg = 0x0 > tcpInfoMsg = 0x0 > tcpInfo = {tcpi_state = 0 '\0', tcpi_ca_state = 0 '\0', tcpi_retransmits > = 0 '\0', tcpi_probes = 0 '\0', > tcpi_backoff = 0 '\0', tcpi_options = 0 '\0', tcpi_snd_wscale = 0 '\0', > tcpi_rcv_wscale = 0 '\0', tcpi_rto = 0, tcpi_ato = 0, > tcpi_snd_mss = 0, tcpi_rcv_mss = 0, tcpi_unacked = 0, tcpi_sacked = 0, > tcpi_lost = 0, tcpi_retrans = 0, tcpi_fackets = 0, > tcpi_last_data_sent = 0, tcpi_last_ack_sent = 0, tcpi_last_data_recv = > 0, tcpi_last_ack_recv = 5731168, tcpi_pmtu = 0, > tcpi_rcv_ssthresh = 1, tcpi_rtt = 1, tcpi_rttvar = 0, tcpi_snd_ssthresh > = 0, tcpi_snd_cwnd = 0, tcpi_advmss = 0, tcpi_reordering = 0} > timeNow = 0 > lastConnLog = 0 > restartLaunch = (XMLRPC_REQUEST) 0x0 > #2 0x0000000000403be7 in main (argc=20, argv=0x7fffe2814538) at aprun.c:225 > maxNumPoll = 1024 > rlim = {rlim_cur = 1024, rlim_max = 1024} > launch = (XMLRPC_REQUEST) 0x576980 > emsg = 0x0 > (gdb) > > > > > > > >> On Mon, Nov 29, 2010 at 20:15, Kelly Fermoyle wrote: >> >>> I tried it in the debugger, and it said >>> that aprun was calling poll(), I don't know what that means. >>> >> >> Type "backtrace full" and send the results. >> >> Jed >> > From kjf198 at cse.psu.edu Mon Nov 29 16:23:01 2010 From: kjf198 at cse.psu.edu (Kelly Fermoyle) Date: Mon, 29 Nov 2010 17:23:01 -0500 (EST) Subject: [petsc-users] MatPermuteSparsify hanging Message-ID: <59491.130.203.38.76.1291069381.squirrel@mail.cse.psu.edu> This is what I got on a desktop. It looks a lot more reasonable. I stopped it after a while and it appears to be in MatSetValues. [Thread 0x40200940 (LWP 26776) exited] Program received signal SIGINT, Interrupt. 0x0000000000accb16 in PetscMemcpy (a=0x2aaab7b499b0, b=0x2aaab6de4938, n=1796808) at memc.c:104 104 memcpy((char*)(a),(char*)(b),n); (gdb) backtrace full #0 0x0000000000accb16 in PetscMemcpy (a=0x2aaab7b499b0, b=0x2aaab6de4938, n=1796808) at memc.c:104 al = 46912714873264 bl = 46912700827960 nl = 1796808 #1 0x000000000069b8f9 in MatSetValues_SeqAIJ (A=0x1430490, m=1, im=0x7fffffffd270, n=21, in=0x2aaab2c76200, v=0x2aaab2d9a200, is=INSERT_VALUES) at aij.c:225 len = 224601 new_j = 0x2aaab7d01200 Ain = 0x1431620 new_nz = 1755215 new_i = 0x2aaab83b4200 new_a = 0x2aaab6f9c200 a = 0x1431620 rp = 0x2aaab89c5d48 k = 0 low = 18 high = 21 t = 18 ii = 59410 row = 52376 nrow = 21 i = 21 col = 52922 l = 0 rmax = 21 N = 37 imax = 0x2aaab2e0f200 ai = 0x2aaab8aa2200 ailen = 0x2aaab2e4a200 ierr = 0 aj = 0x2aaab83ef200 nonew = 0 lastcol = 52922 ap = 0x2aaab6de4890 value = 25472.555765646161 aa = 0x2aaab6237200 ignorezeroentries = PETSC_FALSE roworiented = PETSC_TRUE #2 0x000000000061c324 in MatSetValues (mat=0x1430490, m=1, idxm=0x7fffffffd270, n=21, idxn=0x2aaab2c76200, v=0x2aaab2d9a200, addv=INSERT_VALUES) at matrix.c:921 ierr = 0 #3 0x00000000006426d7 in MatPermuteSparsify (A=0x141a1a0, band=-1, frac=0, tol=0.001, rowp=0x13fde70, colp=0x142e630, B=0x7fffffffd678) at matrix.c:4157 irowp = 0x142f050 icolp = 0x142fa70 rows = 0x2aaab2cb1200 cols = 0x2aaab2cec200 M = 59409 N = 59409 locRowStart = 0 locRowEnd = 59409 nz = 21 newNz = 21 cwork = 0x2aaab5271a24 cnew = 0x2aaab2c76200 vwork = 0x2aaab4bdd248 vnew = 0x2aaab2d9a200 bw = 2970 issize = 59409 row = 27126 locRow = 27126 newRow = 52376 col = 21 newCol = 52382 ierr = 0 #4 0x000000000043822f in main (argc=17, argv=0x7fffffffd7a8) at SolveTest.cpp:54 ksp = 0x7fff00000001 colp = 0x142e630 ---Type to continue, or q to quit--- vec_file = "/gpfs/home/kjf198/group/kjf198/SYMMETRIC_MATRICES/20-0.vec\000\000\000\000\000\000\200\325\377\377\377\177\000\000i4\271\255\252*\000\000\250\065\062\001\000\000\000\000\023\000\000\000\252*\000\000\300\065\062\001\000\000\000\000\001\000\000\000\000\000\000\000\250\065\062\001\000\000\000\000\221 \226\256\252*\000\000\303\312\363\000\000\000\000\000\340,\226\256\252*\000\000\250\335\037\001\000\000\000\000\303\312\363\000\000\000\000\000\060\326\377\377\377\177\000\000\000\260\036\001\000\000\000\000\070\330\377\377\377\177\000\000b.\226\256\252*\000\000\000\330\377\377\377\177\000\000\000\260\036\001\000\000\000\000@\326\377\377\377\177\000\000\201\065\343M4\000\000\000\000\260\036\001\000\000\000\000@\326\377\377\377\177\000\000\000\260\036\001\000\000\000\000\t\204C\000\000\000\000" mat_file = "/gpfs/home/kjf198/group/kjf198/SYMMETRIC_MATRICES/20-0.mat\000\377\377\177\000\000\000\020\214\256\252*\000\000>\340\215\256\252*\000\000`\023\214\256\252*\000\000\260\325\377\377\377\177\000\000\000\020\214\256\000\000\000\000\300?\257\252*\000\000`\263\252\252\252*\000\000\320\325\377\377\377\177\000\000\000\260\252\252\252*\000\000\023\246A\000\000\000\000\000`\263\252\252\252*\000\000\360\325\377\377\377\177\000\000\020\224\214\256\252*\000\000\000\020\214\256\252*\000\000\360\233\340M4\000\000\000\340\024\214\256\252*\000\000`\020\273\256\000\000\000\000y:<\a\000\000\000\000\000\260\036\001\000\000\000\000H[/\000\000\000\000\000\b(2\001\000\000\000\000\021\000\000\000\000\000\000\000\250\327\377\377\377\177\000\000\070\330\377\377\377\177\000" reordering = "rcm", '\000' b = 0x13f6e30 rowp = 0x13fde70 ierr = 0 timestep = 20 x = 0x13fb6c0 start = 1.1098831058709664e-312 type = 0xecbb5a "rcm" iters = 15513927 A = 0x141a1a0 P = 0x1430490 residual = 2.1861090613857222e-317 threshold = 0.001 mat_view = 0x13fb6c0 vec_view = 0x13fb6c0 iteration = 0 end = 7.6648711891784312e-317 > On Mon, Nov 29, 2010 at 20:28, Kelly Fermoyle wrote: > >> I am using a Cray machine and it requires that I call as aprun -n 1 >> ./prog_name, which does not seem to work with -start_in_debugger. >> > > Can you reproduce with the same code on a local machine (e.g. > desktop/laptop) where you can get a proper debugging session? > > >> Below is the output of the full backtrace: >> > > This trace is not even in your code > >> #2 0x0000000000403be7 in main (argc=20, argv=0x7fffe2814538) at > aprun.c:225 > > Jed > From jed at 59A2.org Mon Nov 29 16:28:57 2010 From: jed at 59A2.org (Jed Brown) Date: Mon, 29 Nov 2010 23:28:57 +0100 Subject: [petsc-users] MatPermuteSparsify hanging In-Reply-To: <59491.130.203.38.76.1291069381.squirrel@mail.cse.psu.edu> References: <59491.130.203.38.76.1291069381.squirrel@mail.cse.psu.edu> Message-ID: On Mon, Nov 29, 2010 at 23:23, Kelly Fermoyle wrote: > This is what I got on a desktop. It looks a lot more reasonable. I stopped > it after a while and it appears to be in MatSetValues. > Looks like the code for this function does not use the new permutation to preallocate. That would need to be fixed to make this function fast enough for general use. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Nov 29 16:30:12 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 29 Nov 2010 16:30:12 -0600 Subject: [petsc-users] MatPermuteSparsify hanging In-Reply-To: <59491.130.203.38.76.1291069381.squirrel@mail.cse.psu.edu> References: <59491.130.203.38.76.1291069381.squirrel@mail.cse.psu.edu> Message-ID: <52F96BF4-61D9-4084-BBAF-C4CF6C393070@mcs.anl.gov> Ok, Then the problem is not a fundamental bug but a performance bug (that makes the routine unusable). It is not properly preallocating the new compressed matrix so setting values into it is taking FOREVER (hence the routine is unusable) so who wrote this routine and who is going to fix it. Kelly, What are you using this routine for? It is a pretty obscure that is hardly ever used. Do you need it? Barry On Nov 29, 2010, at 4:23 PM, Kelly Fermoyle wrote: > This is what I got on a desktop. It looks a lot more reasonable. I stopped > it after a while and it appears to be in MatSetValues. > > > > [Thread 0x40200940 (LWP 26776) exited] > > Program received signal SIGINT, Interrupt. > 0x0000000000accb16 in PetscMemcpy (a=0x2aaab7b499b0, b=0x2aaab6de4938, > n=1796808) at memc.c:104 > 104 memcpy((char*)(a),(char*)(b),n); > (gdb) backtrace full > #0 0x0000000000accb16 in PetscMemcpy (a=0x2aaab7b499b0, b=0x2aaab6de4938, > n=1796808) at memc.c:104 > al = 46912714873264 > bl = 46912700827960 > nl = 1796808 > #1 0x000000000069b8f9 in MatSetValues_SeqAIJ (A=0x1430490, m=1, > im=0x7fffffffd270, n=21, in=0x2aaab2c76200, v=0x2aaab2d9a200, > is=INSERT_VALUES) at aij.c:225 > len = 224601 > new_j = 0x2aaab7d01200 > Ain = 0x1431620 > new_nz = 1755215 > new_i = 0x2aaab83b4200 > new_a = 0x2aaab6f9c200 > a = 0x1431620 > rp = 0x2aaab89c5d48 > k = 0 > low = 18 > high = 21 > t = 18 > ii = 59410 > row = 52376 > nrow = 21 > i = 21 > col = 52922 > l = 0 > rmax = 21 > N = 37 > imax = 0x2aaab2e0f200 > ai = 0x2aaab8aa2200 > ailen = 0x2aaab2e4a200 > ierr = 0 > aj = 0x2aaab83ef200 > nonew = 0 > lastcol = 52922 > ap = 0x2aaab6de4890 > value = 25472.555765646161 > aa = 0x2aaab6237200 > ignorezeroentries = PETSC_FALSE > roworiented = PETSC_TRUE > #2 0x000000000061c324 in MatSetValues (mat=0x1430490, m=1, > idxm=0x7fffffffd270, n=21, idxn=0x2aaab2c76200, v=0x2aaab2d9a200, > addv=INSERT_VALUES) at matrix.c:921 > ierr = 0 > #3 0x00000000006426d7 in MatPermuteSparsify (A=0x141a1a0, band=-1, > frac=0, tol=0.001, rowp=0x13fde70, colp=0x142e630, > B=0x7fffffffd678) at matrix.c:4157 > irowp = 0x142f050 > icolp = 0x142fa70 > rows = 0x2aaab2cb1200 > cols = 0x2aaab2cec200 > M = 59409 > N = 59409 > locRowStart = 0 > locRowEnd = 59409 > nz = 21 > newNz = 21 > cwork = 0x2aaab5271a24 > cnew = 0x2aaab2c76200 > vwork = 0x2aaab4bdd248 > vnew = 0x2aaab2d9a200 > bw = 2970 > issize = 59409 > row = 27126 > locRow = 27126 > newRow = 52376 > col = 21 > newCol = 52382 > ierr = 0 > #4 0x000000000043822f in main (argc=17, argv=0x7fffffffd7a8) at > SolveTest.cpp:54 > ksp = 0x7fff00000001 > colp = 0x142e630 > ---Type to continue, or q to quit--- > vec_file = > "/gpfs/home/kjf198/group/kjf198/SYMMETRIC_MATRICES/20-0.vec\000\000\000\000\000\000\200\325\377\377\377\177\000\000i4\271\255\252*\000\000\250\065\062\001\000\000\000\000\023\000\000\000\252*\000\000\300\065\062\001\000\000\000\000\001\000\000\000\000\000\000\000\250\065\062\001\000\000\000\000\221 > \226\256\252*\000\000\303\312\363\000\000\000\000\000\340,\226\256\252*\000\000\250\335\037\001\000\000\000\000\303\312\363\000\000\000\000\000\060\326\377\377\377\177\000\000\000\260\036\001\000\000\000\000\070\330\377\377\377\177\000\000b.\226\256\252*\000\000\000\330\377\377\377\177\000\000\000\260\036\001\000\000\000\000@\326\377\377\377\177\000\000\201\065\343M4\000\000\000\000\260\036\001\000\000\000\000@\326\377\377\377\177\000\000\000\260\036\001\000\000\000\000\t\204C\000\000\000\000" > mat_file = > "/gpfs/home/kjf198/group/kjf198/SYMMETRIC_MATRICES/20-0.mat\000\377\377\177\000\000\000\020\214\256\252*\000\000>\340\215\256\252*\000\000`\023\214\256\252*\000\000\260\325\377\377\377\177\000\000\000\020\214\256\000\000\000\000\300?\257\252*\000\000`\263\252\252\252*\000\000\320\325\377\377\377\177\000\000\000\260\252\252\252*\000\000\023\246A\000\000\000\000\000`\263\252\252\252*\000\000\360\325\377\377\377\177\000\000\020\224\214\256\252*\000\000\000\020\214\256\252*\000\000\360\233\340M4\000\000\000\340\024\214\256\252*\000\000`\020\273\256\000\000\000\000y:<\a\000\000\000\000\000\260\036\001\000\000\000\000H[/\000\000\000\000\000\b(2\001\000\000\000\000\021\000\000\000\000\000\000\000\250\327\377\377\377\177\000\000\070\330\377\377\377\177\000" > reordering = "rcm", '\000' > b = 0x13f6e30 > rowp = 0x13fde70 > ierr = 0 > timestep = 20 > x = 0x13fb6c0 > start = 1.1098831058709664e-312 > type = 0xecbb5a "rcm" > iters = 15513927 > A = 0x141a1a0 > P = 0x1430490 > residual = 2.1861090613857222e-317 > threshold = 0.001 > mat_view = 0x13fb6c0 > vec_view = 0x13fb6c0 > iteration = 0 > end = 7.6648711891784312e-317 > > > >> On Mon, Nov 29, 2010 at 20:28, Kelly Fermoyle wrote: >> >>> I am using a Cray machine and it requires that I call as aprun -n 1 >>> ./prog_name, which does not seem to work with -start_in_debugger. >>> >> >> Can you reproduce with the same code on a local machine (e.g. >> desktop/laptop) where you can get a proper debugging session? >> >> >>> Below is the output of the full backtrace: >>> >> >> This trace is not even in your code >> >>> #2 0x0000000000403be7 in main (argc=20, argv=0x7fffe2814538) at >> aprun.c:225 >> >> Jed >> > From jed at 59A2.org Mon Nov 29 16:32:13 2010 From: jed at 59A2.org (Jed Brown) Date: Mon, 29 Nov 2010 23:32:13 +0100 Subject: [petsc-users] MatPermuteSparsify hanging In-Reply-To: <52F96BF4-61D9-4084-BBAF-C4CF6C393070@mcs.anl.gov> References: <59491.130.203.38.76.1291069381.squirrel@mail.cse.psu.edu> <52F96BF4-61D9-4084-BBAF-C4CF6C393070@mcs.anl.gov> Message-ID: On Mon, Nov 29, 2010 at 23:30, Barry Smith wrote: > who wrote this routine changeset: 19:eb3da2ea4a05 parent: 16:4babff491f33 user: knepley at spinetta.mcs.anl.gov date: Thu Oct 11 06:00:09 2001 -0600 summary: bk-changeset-1.21.1.1 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Mon Nov 29 16:34:15 2010 From: jed at 59A2.org (Jed Brown) Date: Mon, 29 Nov 2010 23:34:15 +0100 Subject: [petsc-users] MatPermuteSparsify hanging In-Reply-To: References: <59491.130.203.38.76.1291069381.squirrel@mail.cse.psu.edu> <52F96BF4-61D9-4084-BBAF-C4CF6C393070@mcs.anl.gov> Message-ID: On Mon, Nov 29, 2010 at 23:32, Jed Brown wrote: > On Mon, Nov 29, 2010 at 23:30, Barry Smith wrote: > >> who wrote this routine > > > changeset: 19:eb3da2ea4a05 > parent: 16:4babff491f33 > user: knepley at spinetta.mcs.anl.gov > date: Thu Oct 11 06:00:09 2001 -0600 > summary: bk-changeset-1.21.1.1 > MatMultConstrained also still exists from this changeset, is marked as "beginner", but has zero implementations in PETSc. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Nov 29 16:58:59 2010 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 29 Nov 2010 16:58:59 -0600 Subject: [petsc-users] MatPermuteSparsify hanging In-Reply-To: References: <59491.130.203.38.76.1291069381.squirrel@mail.cse.psu.edu> <52F96BF4-61D9-4084-BBAF-C4CF6C393070@mcs.anl.gov> Message-ID: I wrote that to test out someone's idea for a preconditioner. It was not great. I left it in for other people to do testing. It is my current opinion that fixing it is more work than it is worth. Matt On Mon, Nov 29, 2010 at 4:32 PM, Jed Brown wrote: > On Mon, Nov 29, 2010 at 23:30, Barry Smith wrote: > >> who wrote this routine > > > changeset: 19:eb3da2ea4a05 > parent: 16:4babff491f33 > user: knepley at spinetta.mcs.anl.gov > date: Thu Oct 11 06:00:09 2001 -0600 > summary: bk-changeset-1.21.1.1 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Nov 29 16:59:54 2010 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 29 Nov 2010 16:59:54 -0600 Subject: [petsc-users] MatPermuteSparsify hanging In-Reply-To: References: <59491.130.203.38.76.1291069381.squirrel@mail.cse.psu.edu> <52F96BF4-61D9-4084-BBAF-C4CF6C393070@mcs.anl.gov> Message-ID: On Mon, Nov 29, 2010 at 4:34 PM, Jed Brown wrote: > On Mon, Nov 29, 2010 at 23:32, Jed Brown wrote: > >> On Mon, Nov 29, 2010 at 23:30, Barry Smith wrote: >> >>> who wrote this routine >> >> >> changeset: 19:eb3da2ea4a05 >> parent: 16:4babff491f33 >> user: knepley at spinetta.mcs.anl.gov >> date: Thu Oct 11 06:00:09 2001 -0600 >> summary: bk-changeset-1.21.1.1 >> > > MatMultConstrained also still exists from this changeset, is marked as > "beginner", but has zero implementations in PETSc. > I thought Barry ripped that out long ago. Matt > Jed > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From vkuhlem at emory.edu Mon Nov 29 17:04:23 2010 From: vkuhlem at emory.edu (Verena Kuhlemann) Date: Mon, 29 Nov 2010 18:04:23 -0500 Subject: [petsc-users] gmres restart Message-ID: Hello, I have a (hopefully) quick question about setting the restart for gmres. I set the restart to be 50, but when I use KSPView it looks like GMRES still uses a restart of 30. It is possible that I overwrite my settings somehow? Here is the relevant part of my code: KSPCreate(PETSC_COMM_WORLD,&ksp); KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN); KSPGMRESSetRestart(ksp,50); KSPSetInitialGuessNonzero(ksp,PETSC_TRUE); KSPSetTolerances(ksp,1.e-12,1.e-12,PETSC_DEFAULT,100000); KSPSetUp(ksp); KSPGetPC(ksp,&pc); PCSetType(pc,PCNONE); KSPView(ksp,PETSC_VIEWER_STDOUT_WORLD); KSPSolve(ksp,b,x); Thanks, Verena -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Nov 29 17:08:07 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 29 Nov 2010 17:08:07 -0600 Subject: [petsc-users] gmres restart In-Reply-To: References: Message-ID: <1924121E-17B9-4E7A-B1A6-F974644D580B@mcs.anl.gov> On Nov 29, 2010, at 5:04 PM, Verena Kuhlemann wrote: > Hello, > > I have a (hopefully) quick question about setting the restart for gmres. > > I set the restart to be 50, but when I use KSPView it looks like GMRES > still uses a restart of 30. It is possible that I overwrite my settings > somehow? Here is the relevant part of my code: > > KSPCreate(PETSC_COMM_WORLD,&ksp); > KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN); KSPSetType(ksp,KSPGMRES); > KSPGMRESSetRestart(ksp,50); You have not set the type to GMRES by this point, therefor the setting of restart is ignored. Must set the type before this. BTW: we highly recommend just calling KSPSetFromOptions() and setting parameters from the options database, much nicer IOHO then hardwiring in the source code. Barry > KSPSetInitialGuessNonzero(ksp,PETSC_TRUE); > KSPSetTolerances(ksp,1.e-12,1.e-12,PETSC_DEFAULT,100000); > KSPSetUp(ksp); > KSPGetPC(ksp,&pc); > PCSetType(pc,PCNONE); > KSPView(ksp,PETSC_VIEWER_STDOUT_WORLD); > KSPSolve(ksp,b,x); > > > Thanks, > Verena From knepley at gmail.com Mon Nov 29 17:08:50 2010 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 29 Nov 2010 17:08:50 -0600 Subject: [petsc-users] gmres restart In-Reply-To: References: Message-ID: On Mon, Nov 29, 2010 at 5:04 PM, Verena Kuhlemann wrote: > Hello, > > I have a (hopefully) quick question about setting the restart for gmres. > > I set the restart to be 50, but when I use KSPView it looks like GMRES > still uses a restart of 30. It is possible that I overwrite my settings > somehow? Here is the relevant part of my code: > > KSPCreate(PETSC_COMM_WORLD,&ksp); > KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN); > KSPGMRESSetRestart(ksp,50); > This is ignored because you have not yet set the KSP type. > KSPSetInitialGuessNonzero(ksp,PETSC_TRUE); > KSPSetTolerances(ksp,1.e-12,1.e-12,PETSC_DEFAULT,100000); > KSPSetUp(ksp); > If you move this before the restart call, the type wil be set and it will work. Matt > KSPGetPC(ksp,&pc); > PCSetType(pc,PCNONE); > KSPView(ksp,PETSC_VIEWER_STDOUT_WORLD); > KSPSolve(ksp,b,x); > > > Thanks, > Verena > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From vkuhlem at emory.edu Mon Nov 29 18:01:50 2010 From: vkuhlem at emory.edu (Verena Kuhlemann) Date: Mon, 29 Nov 2010 19:01:50 -0500 Subject: [petsc-users] gmres restart In-Reply-To: <1924121E-17B9-4E7A-B1A6-F974644D580B@mcs.anl.gov> References: <1924121E-17B9-4E7A-B1A6-F974644D580B@mcs.anl.gov> Message-ID: On Mon, Nov 29, 2010 at 6:08 PM, Barry Smith wrote: > > On Nov 29, 2010, at 5:04 PM, Verena Kuhlemann wrote: > > > Hello, > > > > I have a (hopefully) quick question about setting the restart for gmres. > > > > I set the restart to be 50, but when I use KSPView it looks like GMRES > > still uses a restart of 30. It is possible that I overwrite my settings > > somehow? Here is the relevant part of my code: > > > > KSPCreate(PETSC_COMM_WORLD,&ksp); > > KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN); > > KSPSetType(ksp,KSPGMRES); > > > KSPGMRESSetRestart(ksp,50); > > You have not set the type to GMRES by this point, therefor the setting of > restart is ignored. Must set the type before this. > Ah, I thought I don't have to set it, since GMRES is the default. Now it works, thanks. > > BTW: we highly recommend just calling KSPSetFromOptions() and setting > parameters from the options database, much nicer IOHO then hardwiring in the > source code. > > Barry > > > > KSPSetInitialGuessNonzero(ksp,PETSC_TRUE); > > KSPSetTolerances(ksp,1.e-12,1.e-12,PETSC_DEFAULT,100000); > > KSPSetUp(ksp); > > KSPGetPC(ksp,&pc); > > PCSetType(pc,PCNONE); > > KSPView(ksp,PETSC_VIEWER_STDOUT_WORLD); > > KSPSolve(ksp,b,x); > > > > > > Thanks, > > Verena > > -- Verena Kuhlemann Ph.D. Candidate Mathematics and Computer Science Department Emory University -------------- next part -------------- An HTML attachment was scrubbed... URL: From rongliang.chan at gmail.com Mon Nov 29 20:26:45 2010 From: rongliang.chan at gmail.com (Rongliang Chen) Date: Mon, 29 Nov 2010 19:26:45 -0700 Subject: [petsc-users] petsc-users Digest, Vol 23, Issue 32 In-Reply-To: References: Message-ID: Hi Jed, Thank you for your reply. The Matrix A and Vector x are created by MatCreateMPIAIJ and VecCreateMPI and matrix A is obtained by SNESDefaultComputeJacobian. I compared the results using the following routine: .......................................................... sprintf(filename,"A.m"); ierr = PetscViewerASCIIOpen(PETSC_COMM_WORLD,filename,&view->viewer);CHKERRQ(ierr); ierr = PetscViewerSetFormat(view->viewer, PETSC_VIEWER_ASCII_MATLAB);CHKERRQ(ierr); ierr = MatView(A, view->viewer);CHKERRQ(ierr); Vec y; ierr = VecDuplicate(x, &y);CHKERRQ(ierr); ierr = MatMult(A, x, y);CHKERRQ(ierr); sprintf(filename,"x.m"); ierr = PetscViewerASCIIOpen(PETSC_COMM_WORLD,filename,&view->viewer);CHKERRQ(ierr); ierr = PetscViewerSetFormat(view->viewer, PETSC_VIEWER_ASCII_MATLAB);CHKERRQ(ierr); ierr = VecView(x, view->viewer);CHKERRQ(ierr); sprintf(filename,"y.m"); ierr = PetscViewerASCIIOpen(PETSC_COMM_WORLD,filename,&view->viewer);CHKERRQ(ierr); ierr = PetscViewerSetFormat(view->viewer, PETSC_VIEWER_ASCII_MATLAB);CHKERRQ(ierr); ierr = VecView(y, view->viewer);CHKERRQ(ierr); ........................................................... In the Matlab, I first load the files A.m, x.m and y.m into A, x and y, and then do "error=norm(A*x - y)". The 'error' is very big(about 200). Following your suggestion, I can write the matrix and vectors into binary files, but I do not how to load these binary files into matlab matrix and vector. I do not understand the command: [A,x,y] = PetscBinaryRead('binaryoutput'), can this command load the binary files into matlab matrix and vector? Thanks. Best, Rongliang > Message: 2 > Date: Sun, 28 Nov 2010 21:18:59 +0100 > From: Jed Brown > Subject: Re: [petsc-users] Why MatMult(A, x, Ax) != A*x? > To: PETSc users list > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > On Sun, Nov 28, 2010 at 21:11, Rongliang Chen >wrote: > > > The format of the matrix A is AIJ and is obtained by function > > SNESDefaultComputeJacobian. > > I compute A*x in Matlab with the following A and x and compare it with > Ax. > > > > ....................... > > ierr = MatMult(*A, x, Ax);CHKERRQ(ierr); > > > > sprintf(filename,"x.m"); > > ierr = > > > PetscViewerASCIIOpen(PETSC_COMM_WORLD,filename,&view->viewer);CHKERRQ(ierr); > > ierr = PetscViewerSetFormat(view->viewer, > > PETSC_VIEWER_ASCII_MATLAB);CHKERRQ(ierr); > > ierr = VecView(x, view->viewer);CHKERRQ(ierr); > > > > sprintf(filename,"Ax.m"); > > ierr = > > > PetscViewerASCIIOpen(PETSC_COMM_WORLD,filename,&view->viewer);CHKERRQ(ierr); > > ierr = PetscViewerSetFormat(view->viewer, > > PETSC_VIEWER_ASCII_MATLAB);CHKERRQ(ierr); > > ierr = VecView(Ax, view->viewer);CHKERRQ(ierr); > > > > Are the vectors and matrices obtained from a DA (DACreateGlobalVector, > DAGetMatrix)? How are you comparing the matrix that PETSc sees with the > matrix that Matlab sees? I suggest writing the matrix and both vectors to > a > PETSc binary file > > PetscViewer viewer = PETSC_VIEWER_BINARY(PETSC_COMM_WORLD); > MatView(A,viewer); > VecView(x,viewer); > MatMult(A,x,y); > VecView(y,viewer); > > and read them into Matlab with > > [A,x,y] = PetscBinaryRead('binaryoutput') > norm(A*x - y) % This should be small > > Jed > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20101128/d97eefaa/attachment-0001.htm > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Nov 29 20:57:04 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 29 Nov 2010 20:57:04 -0600 Subject: [petsc-users] gmres restart In-Reply-To: References: <1924121E-17B9-4E7A-B1A6-F974644D580B@mcs.anl.gov> Message-ID: On Nov 29, 2010, at 6:01 PM, Verena Kuhlemann wrote: > > > On Mon, Nov 29, 2010 at 6:08 PM, Barry Smith wrote: > > On Nov 29, 2010, at 5:04 PM, Verena Kuhlemann wrote: > > > Hello, > > > > I have a (hopefully) quick question about setting the restart for gmres. > > > > I set the restart to be 50, but when I use KSPView it looks like GMRES > > still uses a restart of 30. It is possible that I overwrite my settings > > somehow? Here is the relevant part of my code: > > > > KSPCreate(PETSC_COMM_WORLD,&ksp); > > KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN); > > KSPSetType(ksp,KSPGMRES); > > > KSPGMRESSetRestart(ksp,50); > > You have not set the type to GMRES by this point, therefor the setting of restart is ignored. Must set the type before this. > > Ah, I thought I don't have to set it, since GMRES is the default. Now it works, thanks. The thing is it delays picking the default as long as possible, thus it was not set at that point. But you are right, it would be reasonable to be known the whole time. Barry > > > > > > > > > > > BTW: we highly recommend just calling KSPSetFromOptions() and setting parameters from the options database, much nicer IOHO then hardwiring in the source code. > > Barry > > > > KSPSetInitialGuessNonzero(ksp,PETSC_TRUE); > > KSPSetTolerances(ksp,1.e-12,1.e-12,PETSC_DEFAULT,100000); > > KSPSetUp(ksp); > > KSPGetPC(ksp,&pc); > > PCSetType(pc,PCNONE); > > KSPView(ksp,PETSC_VIEWER_STDOUT_WORLD); > > KSPSolve(ksp,b,x); > > > > > > Thanks, > > Verena > > > > > -- > Verena Kuhlemann > Ph.D. Candidate > Mathematics and Computer Science Department > Emory University > From bsmith at mcs.anl.gov Mon Nov 29 21:01:15 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 29 Nov 2010 21:01:15 -0600 Subject: [petsc-users] petsc-users Digest, Vol 23, Issue 32 In-Reply-To: References: Message-ID: On Nov 29, 2010, at 8:26 PM, Rongliang Chen wrote: > > Following your suggestion, I can write the matrix and vectors into binary files, but I do not how to load > these binary files into matlab matrix and vector. > I do not understand the command: [A,x,y] = PetscBinaryRead('binaryoutput'), can this command load > the binary files into matlab matrix and vector? Thanks. If you first view into the binary file the A, then the x vector than the y vector then yes > [A,x,y] = PetscBinaryRead('binaryoutput') reads in the matrix and the two vectors. Barry > > Best, > > Rongliang > > > Message: 2 > Date: Sun, 28 Nov 2010 21:18:59 +0100 > From: Jed Brown > Subject: Re: [petsc-users] Why MatMult(A, x, Ax) != A*x? > To: PETSc users list > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > On Sun, Nov 28, 2010 at 21:11, Rongliang Chen wrote: > > > The format of the matrix A is AIJ and is obtained by function > > SNESDefaultComputeJacobian. > > I compute A*x in Matlab with the following A and x and compare it with Ax. > > > > ....................... > > ierr = MatMult(*A, x, Ax);CHKERRQ(ierr); > > > > sprintf(filename,"x.m"); > > ierr = > > PetscViewerASCIIOpen(PETSC_COMM_WORLD,filename,&view->viewer);CHKERRQ(ierr); > > ierr = PetscViewerSetFormat(view->viewer, > > PETSC_VIEWER_ASCII_MATLAB);CHKERRQ(ierr); > > ierr = VecView(x, view->viewer);CHKERRQ(ierr); > > > > sprintf(filename,"Ax.m"); > > ierr = > > PetscViewerASCIIOpen(PETSC_COMM_WORLD,filename,&view->viewer);CHKERRQ(ierr); > > ierr = PetscViewerSetFormat(view->viewer, > > PETSC_VIEWER_ASCII_MATLAB);CHKERRQ(ierr); > > ierr = VecView(Ax, view->viewer);CHKERRQ(ierr); > > > > Are the vectors and matrices obtained from a DA (DACreateGlobalVector, > DAGetMatrix)? How are you comparing the matrix that PETSc sees with the > matrix that Matlab sees? I suggest writing the matrix and both vectors to a > PETSc binary file > > PetscViewer viewer = PETSC_VIEWER_BINARY(PETSC_COMM_WORLD); > MatView(A,viewer); > VecView(x,viewer); > MatMult(A,x,y); > VecView(y,viewer); > > and read them into Matlab with > > [A,x,y] = PetscBinaryRead('binaryoutput') > norm(A*x - y) % This should be small > > Jed > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > From jed at 59A2.org Tue Nov 30 02:29:17 2010 From: jed at 59A2.org (Jed Brown) Date: Tue, 30 Nov 2010 09:29:17 +0100 Subject: [petsc-users] petsc-users Digest, Vol 23, Issue 32 In-Reply-To: References: Message-ID: On Tue, Nov 30, 2010 at 03:26, Rongliang Chen wrote: > [A,x,y] = PetscBinaryRead('binaryoutput'), can this command load > the binary files into matlab matrix and vector? > Yes, this reads the binary files. You will need to add $PETSC_DIR/bin/matlab to your path (see "help addpath"). You should also call PetscViewerDestroy() when you are done with each viewer. In the source you quoted, this is never called. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From m.skates82 at gmail.com Tue Nov 30 03:46:35 2010 From: m.skates82 at gmail.com (Nunion) Date: Tue, 30 Nov 2010 03:46:35 -0600 Subject: [petsc-users] Installing Hypre with PETSc [Complex] Message-ID: Hello all, when attempting to install Hypre with PETSc (for complex arithmetic), I got a message saying that configuration could not proceed forward because Hypre is not configured for complex arithmetic. Does anyone know of a fix around this issue as I would like to use BoomerAMG interfaced with PETSc. Thanks, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Tue Nov 30 07:44:28 2010 From: jed at 59A2.org (Jed Brown) Date: Tue, 30 Nov 2010 14:44:28 +0100 Subject: [petsc-users] Installing Hypre with PETSc [Complex] In-Reply-To: References: Message-ID: On Tue, Nov 30, 2010 at 10:46, Nunion wrote: > when attempting to install Hypre with PETSc (for complex arithmetic), I got > a message saying that configuration could not proceed forward because Hypre > is not configured for complex arithmetic. Does anyone know of a fix around > this issue as I would like to use BoomerAMG interfaced with PETSc. >From the Hypre Users' Manual: "The hypre library currently does not support complex-valued systems." Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From kjf198 at cse.psu.edu Tue Nov 30 09:49:58 2010 From: kjf198 at cse.psu.edu (Kelly Fermoyle) Date: Tue, 30 Nov 2010 10:49:58 -0500 (EST) Subject: [petsc-users] MatPermuteSparsify hanging In-Reply-To: References: <59491.130.203.38.76.1291069381.squirrel@mail.cse.psu.edu> <52F96BF4-61D9-4084-BBAF-C4CF6C393070@mcs.anl.gov> Message-ID: <60467.130.203.38.76.1291132198.squirrel@mail.cse.psu.edu> Okay, I don't actually need it. I ran it until it finally completed (45 minutes for 1 million nonzeros) and it didn't seem to help my preconditioner. Thanks for looking into it. Seems like you could leave it in as a Developer function and note that it is not efficient. > On Mon, Nov 29, 2010 at 4:34 PM, Jed Brown wrote: > >> On Mon, Nov 29, 2010 at 23:32, Jed Brown wrote: >> >>> On Mon, Nov 29, 2010 at 23:30, Barry Smith wrote: >>> >>>> who wrote this routine >>> >>> >>> changeset: 19:eb3da2ea4a05 >>> parent: 16:4babff491f33 >>> user: knepley at spinetta.mcs.anl.gov >>> date: Thu Oct 11 06:00:09 2001 -0600 >>> summary: bk-changeset-1.21.1.1 >>> >> >> MatMultConstrained also still exists from this changeset, is marked as >> "beginner", but has zero implementations in PETSc. >> > > I thought Barry ripped that out long ago. > > Matt > > >> Jed >> > > > > -- > What most experimenters take for granted before they begin their > experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener > From rongliang.chan at gmail.com Tue Nov 30 14:42:33 2010 From: rongliang.chan at gmail.com (Rongliang Chen) Date: Tue, 30 Nov 2010 13:42:33 -0700 Subject: [petsc-users] Why MatMult(A, x, Ax) != A*x? In-Reply-To: References: Message-ID: Hi, I know that I should call PetscViewerDestroy() when I have done with each viewer. But if I called this I just can view the last Matrix and vector. What I want to do is that I let it do three steps of SNES, view the jacobian of each step. I found that A*x is correct at the first step and it's wrong for the second and third step. I also tried using the binary output and input, and it has the same results. But If I destroy the matrix A after the first SNES, recreate it and then do the second and third SNES, it will be alright. Thanks. Best, Rongliang > ------------------------------ > > Message: 3 > Date: Tue, 30 Nov 2010 09:29:17 +0100 > From: Jed Brown > Subject: Re: [petsc-users] petsc-users Digest, Vol 23, Issue 32 > To: PETSc users list > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > On Tue, Nov 30, 2010 at 03:26, Rongliang Chen >wrote: > > > [A,x,y] = PetscBinaryRead('binaryoutput'), can this command load > > the binary files into matlab matrix and vector? > > > > Yes, this reads the binary files. You will need to add > $PETSC_DIR/bin/matlab to your path (see "help addpath"). > > You should also call PetscViewerDestroy() when you are done with each > viewer. In the source you quoted, this is never called. > > Jed > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20101130/c1eb3531/attachment-0001.htm > > > > ------------------------------ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Tue Nov 30 14:52:45 2010 From: jed at 59A2.org (Jed Brown) Date: Tue, 30 Nov 2010 21:52:45 +0100 Subject: [petsc-users] Why MatMult(A, x, Ax) != A*x? In-Reply-To: References: Message-ID: On Tue, Nov 30, 2010 at 21:42, Rongliang Chen wrote: > I know that I should call PetscViewerDestroy() when I have done with each > viewer. But if I called this I just can view the last Matrix and vector. > What I want > to do is that I let it do three steps of SNES, view the jacobian of each > step. > Maybe view it to different files. The code you posted for viewing is certainly not correct, especially if called in a loop. > I found that A*x is correct at the first step and it's wrong for the second > and > third step. > I also tried using the binary output and input, and it has the same > results. > The problem is most likely that you are picking these pieces out of the files differently than they are put in. MatMult being incorrect is by far the least likely scenario. I don't know why you want to do this, but you need to find a way to manage the files so that you can be certain that you are reading (from Matlab) exactly what you think you are reading. One way to do this is to use a binary file that gets all three items together. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From khalid_eee at yahoo.com Tue Nov 30 14:58:50 2010 From: khalid_eee at yahoo.com (khalid ashraf) Date: Tue, 30 Nov 2010 12:58:50 -0800 (PST) Subject: [petsc-users] Accessing array indices in 3D Message-ID: <908472.67694.qm@web112602.mail.gq1.yahoo.com> Hi, I am writing the following loop for accessing the vectors in a 3D grid. for (k=zs; k From bsmith at mcs.anl.gov Tue Nov 30 15:02:55 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 30 Nov 2010 15:02:55 -0600 Subject: [petsc-users] Accessing array indices in 3D In-Reply-To: <908472.67694.qm@web112602.mail.gq1.yahoo.com> References: <908472.67694.qm@web112602.mail.gq1.yahoo.com> Message-ID: <011574D7-E49D-46B9-A2BD-7D0507F563CA@mcs.anl.gov> Where are you getting these u_localptr arrays? What are you passing in as the Vec into DAVecGetArray()? If you pass in global vectors you CANNOT access ghost points in the arrays (they don't exist) you need to pass in local vectors obtained with DAGetLocalVector() or DACreateLocalVector() to access ghost points. Barry On Nov 30, 2010, at 2:58 PM, khalid ashraf wrote: > Hi, I am writing the following loop for accessing the vectors in a 3D grid. > > for (k=zs; k for (j=ys; j for (i=xs; i if (i<(mx-1)) > { > Cx=u_localptr[k][j][i+1]-u_localptr[k][j][i]; > } > else > { > Cx=u_localptr[k][j][i]-u_localptr[k][j][i-1]; > } > if (j<(my-1)) > { > // Cy=v_localptr[k][j+1][i]-v_localptr[k][j][i]; > Cy=v_localptr[k][j][i]-v_localptr[k][j][i]; > } > else > { > Cy=v_localptr[k][j][i]-v_localptr[k][j-1][i]; > } > if (k<(mz-1)) > { > // Cz=w_localptr[k+1][j][i]-w_localptr[k][j][i]; > Cz=w_localptr[k][j][i]-w_localptr[k][j][i]; > } > else > { > Cz=w_localptr[k][j][i]-w_localptr[k-1][j][i]; > } > In this code if I uncomment the two commented lines, then the program gives memory access out of range error when run on multiple processors. Could you please tell me what is the problem with the two commented out lines. > > Thanks. > > Khalid > > From khalid_eee at yahoo.com Tue Nov 30 16:32:54 2010 From: khalid_eee at yahoo.com (khalid ashraf) Date: Tue, 30 Nov 2010 14:32:54 -0800 (PST) Subject: [petsc-users] Accessing array indices in 3D Message-ID: <308887.88166.qm@web112605.mail.gq1.yahoo.com> Thanks Barry. It works fine. Another question, I have a Periodic Boundary condition along x and y. So I want to use the two commented out line below. for (j=ys; j From knepley at gmail.com Tue Nov 30 16:36:10 2010 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 30 Nov 2010 16:36:10 -0600 Subject: [petsc-users] Accessing array indices in 3D In-Reply-To: <308887.88166.qm@web112605.mail.gq1.yahoo.com> References: <308887.88166.qm@web112605.mail.gq1.yahoo.com> Message-ID: You need to indicate that the DA is periodic when you create it. Matt On Tue, Nov 30, 2010 at 4:32 PM, khalid ashraf wrote: > *Thanks Barry. It works fine. Another question, I have a Periodic Boundary* > > *condition along x and y. So I want to use the two commented out line below.* > > * for (j=ys; j > *** // Cy=v_localptr[k][j][i]-v_localptr[k][0][i];** }** > * > > *Is it possible to include these non-local points [k][j][0] or [k][0][i]. Or I have to * > > *include the boundary points in the grid ? The same question when forming the A matrix with * > > *PBC. How to include the non-neighbour elements within the if statement. * > > * > * > > *Thanks.* > > * > * > > *Khalid* > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Nov 30 16:38:14 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 30 Nov 2010 16:38:14 -0600 Subject: [petsc-users] Accessing array indices in 3D In-Reply-To: <308887.88166.qm@web112605.mail.gq1.yahoo.com> References: <308887.88166.qm@web112605.mail.gq1.yahoo.com> Message-ID: <02236E32-00B0-428C-A09A-A74F3C6B0723@mcs.anl.gov> On Nov 30, 2010, at 4:32 PM, khalid ashraf wrote: > Thanks Barry. It works fine. Another question, I have a Periodic Boundary > condition along x and y. So I want to use the two commented out line below. > for (j=ys; j > for (i=xs; i > if (i<(mx-1)) > > { > > Cx=u_localptr[k][j][i+1]-u_localptr[k][j][i]; > > } > > else > > { > > // Cx=u_localptr[k][j][i]-u_localptr[k][j][0]; > > } > > if (j<(my-1)) > > { > > Cy=v_localptr[k][j+1][i]-v_localptr[k][j][i]; > > } > > else > > { > // Cy=v_localptr[k][j][i]-v_localptr[k][0][i]; > > } > > > Is it possible to include these non-local points [k][j][0] or [k][0][i]. Why do you call those points nonlocal? The are local on the appropriate process. 0 is just the first entry. [k][j][-1] would be a ghost point and just like all other ghost points would be available in the ghosted (local) vectors. Also the ghost points on the other side are at locations [mx] or [my] or [mz] > Or I have to > include the boundary points in the grid ? The same question when forming the A matrix with > PBC. How to include the non-neighbour elements within the if statement. You can use MatSetValuesStencil() for setting values in the matrix and include the ghost points into the stencil, you do nothing special. Things just work with periodic boundaries, don't overthink things and get yourself in a tangle. Barry > > Thanks. > > Khalid > > From rongliang.chan at gmail.com Tue Nov 30 19:13:35 2010 From: rongliang.chan at gmail.com (Rongliang Chen) Date: Tue, 30 Nov 2010 18:13:35 -0700 Subject: [petsc-users] Why MatMult(A, x, Ax) != A*x? Message-ID: Hi, I have tried the following method: Vec Ax; ierr = VecDuplicate(x, &Ax);CHKERRQ(ierr); ierr = MatMult(*A, x, Ax);CHKERRQ(ierr); PetscViewer viewer = PETSC_VIEWER_BINARY_(PETSC_COMM_WORLD); ierr = MatView(*A,viewer);CHKERRQ(ierr); ierr = VecView(x,viewer);CHKERRQ(ierr); ierr = VecView(Ax,viewer);CHKERRQ(ierr); // ierr = PetscViewerDestroy(viewer);CHKERRQ(ierr); (If I use this line, I just can obtain A,x,y and I can not get the A1, x1, y1 (see below.)). After do two SNES, I use the following commands in matlab: >> [A,x,y,A1,x1,y1] = PetscBinaryRead('binaryoutput'); >> norm(A*x-y) ans = 1.2000e-13 >> norm(A1*x1-y1) ans = 2.6612e+03 The reason that I want to do this is that I doubt that there is a bug in my code and I want to following these information to find out the bug. There is a very strange thing in my code:(I use GMRES with ASM preconditioner to solve the linear system.) Solving Shape Optimization problem 0 SNES norm 1.1359606379e+02, 0 KSP its last norm 0.0000000000e+00. 1 SNES norm 4.1335342946e+01, 1 KSP its last norm 2.3855348597e-03. 2 SNES norm *4.1085343652e+01, 13 KSP* its last norm 3.4528582271e-02. *SNES diverged DIVERGED_LS_FAILURE.* I think there is some thing wrong with the second SNES, because its KSP should be 1 or 2. If I destroy the Jacobian matrix here and recreate it, then continue the SNES, it will be alright: 0 SNES norm *4.1085343652e+01,* 0 KSP its last norm 0.0000000000e+00. 1 SNES norm 6.9950501590e+00, 1 KSP its last norm 2.8278185399e-03. 2 SNES norm 1.3445253365e-01, 1 KSP its last norm 2.3243944231e-04. 3 SNES norm 5.0914645029e-04, 1 KSP its last norm 5.0954271956e-06. ................. So I output the matrix and vectors into matlab and do the matrix and vector multiplication and I found that the result is different from the MatMult in Petsc. Best, Rongliang ------------------------------ > > Message: 2 > Date: Tue, 30 Nov 2010 21:52:45 +0100 > From: Jed Brown > Subject: Re: [petsc-users] Why MatMult(A, x, Ax) != A*x? > To: PETSc users list > Message-ID: > > > > Content-Type: text/plain; charset="utf-8" > > On Tue, Nov 30, 2010 at 21:42, Rongliang Chen >wrote: > > > I know that I should call PetscViewerDestroy() when I have done with each > > viewer. But if I called this I just can view the last Matrix and vector. > > What I want > > to do is that I let it do three steps of SNES, view the jacobian of each > > step. > > > > Maybe view it to different files. The code you posted for viewing is > certainly not correct, especially if called in a loop. > > > > I found that A*x is correct at the first step and it's wrong for the > second > > and > > third step. > > I also tried using the binary output and input, and it has the same > > results. > > > > The problem is most likely that you are picking these pieces out of the > files differently than they are put in. MatMult being incorrect is by far > the least likely scenario. I don't know why you want to do this, but you > need to find a way to manage the files so that you can be certain that you > are reading (from Matlab) exactly what you think you are reading. One way > to do this is to use a binary file that gets all three items together. > > Jed > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Nov 30 19:37:46 2010 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 30 Nov 2010 19:37:46 -0600 Subject: [petsc-users] Why MatMult(A, x, Ax) != A*x? In-Reply-To: References: Message-ID: On Tue, Nov 30, 2010 at 7:13 PM, Rongliang Chen wrote: > Hi, > > I have tried the following method: > > Vec Ax; > ierr = VecDuplicate(x, &Ax);CHKERRQ(ierr); > ierr = MatMult(*A, x, Ax);CHKERRQ(ierr); > PetscViewer viewer = PETSC_VIEWER_BINARY_(PETSC_COMM_WORLD); > ierr = MatView(*A,viewer);CHKERRQ(ierr); > ierr = VecView(x,viewer);CHKERRQ(ierr); > ierr = VecView(Ax,viewer);CHKERRQ(ierr); > // ierr = PetscViewerDestroy(viewer);CHKERRQ(ierr); (If I use this > line, I just can obtain A,x,y and I can not get the A1, x1, y1 (see > below.)). > > After do two SNES, I use the following commands in matlab: > > >> [A,x,y,A1,x1,y1] = PetscBinaryRead('binaryoutput'); > >> norm(A*x-y) > > ans = > > 1.2000e-13 > > >> norm(A1*x1-y1) > > ans = > > 2.6612e+03 > > The reason that I want to do this is that I doubt that there is a bug in my > code > and I want to following these information to find out the bug. > When you create the Jacobian, do you remember to zero out the matrix? This is a common bug when creating the matrix after the initial iterate. Matt > There is a very strange thing in my code:(I use GMRES with ASM > preconditioner to solve the linear system.) > > Solving Shape Optimization problem > 0 SNES norm 1.1359606379e+02, 0 KSP its last norm 0.0000000000e+00. > 1 SNES norm 4.1335342946e+01, 1 KSP its last norm 2.3855348597e-03. > 2 SNES norm *4.1085343652e+01, 13 KSP* its last norm 3.4528582271e-02. > *SNES diverged DIVERGED_LS_FAILURE.* > > I think there is some thing wrong with the second SNES, because its KSP > should > be 1 or 2. If I destroy the Jacobian matrix here and recreate it, then > continue the > SNES, it will be alright: > > 0 SNES norm *4.1085343652e+01,* 0 KSP its last norm 0.0000000000e+00. > 1 SNES norm 6.9950501590e+00, 1 KSP its last norm 2.8278185399e-03. > 2 SNES norm 1.3445253365e-01, 1 KSP its last norm 2.3243944231e-04. > 3 SNES norm 5.0914645029e-04, 1 KSP its last norm 5.0954271956e-06. > ................. > So I output the matrix and vectors into matlab and do the matrix and vector > > multiplication and I found that the result is different from the MatMult in > Petsc. > > Best, > > Rongliang > > ------------------------------ >> >> Message: 2 >> Date: Tue, 30 Nov 2010 21:52:45 +0100 >> From: Jed Brown >> Subject: Re: [petsc-users] Why MatMult(A, x, Ax) != A*x? >> To: PETSc users list >> Message-ID: >> >> > >> Content-Type: text/plain; charset="utf-8" >> >> On Tue, Nov 30, 2010 at 21:42, Rongliang Chen > >wrote: >> >> > I know that I should call PetscViewerDestroy() when I have done with >> each >> > viewer. But if I called this I just can view the last Matrix and vector. >> > What I want >> > to do is that I let it do three steps of SNES, view the jacobian of each >> > step. >> > >> >> Maybe view it to different files. The code you posted for viewing is >> certainly not correct, especially if called in a loop. >> >> >> > I found that A*x is correct at the first step and it's wrong for the >> second >> > and >> > third step. >> > I also tried using the binary output and input, and it has the same >> > results. >> > >> >> The problem is most likely that you are picking these pieces out of the >> files differently than they are put in. MatMult being incorrect is by far >> the least likely scenario. I don't know why you want to do this, but you >> need to find a way to manage the files so that you can be certain that you >> are reading (from Matlab) exactly what you think you are reading. One way >> to do this is to use a binary file that gets all three items together. >> >> Jed >> >> >> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Nov 30 21:31:22 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 30 Nov 2010 21:31:22 -0600 Subject: [petsc-users] Why MatMult(A, x, Ax) != A*x? In-Reply-To: References: Message-ID: On Nov 30, 2010, at 7:13 PM, Rongliang Chen wrote: > Hi, > > I have tried the following method: > > Vec Ax; > ierr = VecDuplicate(x, &Ax);CHKERRQ(ierr); > ierr = MatMult(*A, x, Ax);CHKERRQ(ierr); > PetscViewer viewer = PETSC_VIEWER_BINARY_(PETSC_COMM_WORLD); > ierr = MatView(*A,viewer);CHKERRQ(ierr); > ierr = VecView(x,viewer);CHKERRQ(ierr); > ierr = VecView(Ax,viewer);CHKERRQ(ierr); > // ierr = PetscViewerDestroy(viewer);CHKERRQ(ierr); (If I use this line, I just can obtain A,x,y and I can not get the A1, x1, y1 (see below.)). > > After do two SNES, I use the following commands in matlab: > > >> [A,x,y,A1,x1,y1] = PetscBinaryRead('binaryoutput'); > >> norm(A*x-y) > > ans = > > 1.2000e-13 > > >> norm(A1*x1-y1) > > ans = > > 2.6612e+03 > > The reason that I want to do this is that I doubt that there is a bug in my code > and I want to following these information to find out the bug. > There is a very strange thing in my code:(I use GMRES with ASM preconditioner to solve the linear system.) > > Solving Shape Optimization problem > 0 SNES norm 1.1359606379e+02, 0 KSP its last norm 0.0000000000e+00. > 1 SNES norm 4.1335342946e+01, 1 KSP its last norm 2.3855348597e-03. > 2 SNES norm 4.1085343652e+01, 13 KSP its last norm 3.4528582271e-02. > SNES diverged DIVERGED_LS_FAILURE. You are going about this all wrong. Forget about checking that A*x = A*x. That is not the bug and not the way to find the bug. Run with -snes_monitor -ksp_monitor_true_residual -snes_converged_reason -ksp_converged_reason -snes_ls_monitor likely either the Jacobian is wrong or the linear solver is not working. Look at the output and if confused send the output to petsc-maint at mcs.anl.gov Barry > > I think there is some thing wrong with the second SNES, because its KSP should > be 1 or 2. If I destroy the Jacobian matrix here and recreate it, then continue the > SNES, it will be alright: > > 0 SNES norm 4.1085343652e+01, 0 KSP its last norm 0.0000000000e+00. > 1 SNES norm 6.9950501590e+00, 1 KSP its last norm 2.8278185399e-03. > 2 SNES norm 1.3445253365e-01, 1 KSP its last norm 2.3243944231e-04. > 3 SNES norm 5.0914645029e-04, 1 KSP its last norm 5.0954271956e-06. > ................. > So I output the matrix and vectors into matlab and do the matrix and vector > multiplication and I found that the result is different from the MatMult in Petsc. > > Best, > > Rongliang > > ------------------------------ > > Message: 2 > Date: Tue, 30 Nov 2010 21:52:45 +0100 > From: Jed Brown > Subject: Re: [petsc-users] Why MatMult(A, x, Ax) != A*x? > To: PETSc users list > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > On Tue, Nov 30, 2010 at 21:42, Rongliang Chen wrote: > > > I know that I should call PetscViewerDestroy() when I have done with each > > viewer. But if I called this I just can view the last Matrix and vector. > > What I want > > to do is that I let it do three steps of SNES, view the jacobian of each > > step. > > > > Maybe view it to different files. The code you posted for viewing is > certainly not correct, especially if called in a loop. > > > > I found that A*x is correct at the first step and it's wrong for the second > > and > > third step. > > I also tried using the binary output and input, and it has the same > > results. > > > > The problem is most likely that you are picking these pieces out of the > files differently than they are put in. MatMult being incorrect is by far > the least likely scenario. I don't know why you want to do this, but you > need to find a way to manage the files so that you can be certain that you > are reading (from Matlab) exactly what you think you are reading. One way > to do this is to use a binary file that gets all three items together. > > Jed > > From pengxwang at hotmail.com Tue Nov 30 22:33:30 2010 From: pengxwang at hotmail.com (Peter Wang) Date: Tue, 30 Nov 2010 22:33:30 -0600 Subject: [petsc-users] column index in MatSetValues() Message-ID: I am trying to create a matrix and insert values to it. The martix is supposed to be as following: 1 0 0 0 0 2 0 0 0 0 3 0 0 0 0 4 array coef[] is the diagonal value of the matrix, snr[] is the index of the row, rnr[] is the index of column. However, I always get the wrong results. It shows the Column too large: col 4607182418800017408 max 3! I cheked the value of rnr[]. The output snr and rnr is correct: snr= 0 1 2 3 rnr= 0 1 2 3 It seems there is something wrong when MatSetValues() is called. Following is a part of the error information. The information is shown at each loop of do II=Istart,Iend-1 The output (if any) follows: snr= 0 1 2 3 rnr= 0 1 2 3 8.....Check after MatGetOwnershipRange() Istart= 0 Iend= 4 II= 0 1 0 0 [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: Column too large: col 4607182418800017408 max 3! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 5, Mon Sep 27 11:51:54 CDT 2010 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Debug_PETSc_MatCreate_20101130 on a linux-gnu named compute-1-35.hpc.local.uwm by pwang_a Tue Nov 30 22:27:03 2010 [0]PETSC ERROR: Libraries linked from /sharedapps/uwm/ceas/gcc-4.4.3/petsc/3.1-p5-v1/lib [0]PETSC ERROR: Configure run at Fri Oct 8 12:59:16 2010 [0]PETSC ERROR: Configure options --prefix=/sharedapps/uwm/ceas/gcc-4.4.3/petsc/3.1-p5-v1 --with-mpi-dir=/sharedapps/uwm/common/gcc-4.4.3/openmpi/1.3.2-v1 --with-blas-lapack-dir=/sharedapps/uwm/ceas/gcc-4.4.3/lapack/3.2.2-v1/lib --with-64-bit-indices=1 --with-64-bit-pointers=1 --with-large-file-io=1 --with-x=0 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatSetValues_SeqAIJ() line 193 in src/mat/impls/aij/seq/aij.c [0]PETSC ERROR: MatSetValues() line 992 in src/mat/interface/matrix.c !The code is as following: !============================= program Debug_PETSc_MatCreate_20101130 implicit none ! #include "finclude/petscsys.h" #include "finclude/petscvec.h" #include "finclude/petscmat.h" #include "finclude/petscpc.h" #include "finclude/petscksp.h" ! Variables !- - - - - - - - - - - - - - - - - - - - - - - - - - - - - ! PETSc Variables !- - - - - - - - - - - - - - - - - - - - - - - - - - - - - real*8 norm PetscInt i,j,II,JJ,its !,m,n PetscInt Istart,Iend,ione PetscErrorCode ierr PetscMPIInt myid,numprocs PetscTruth flg PetscScalar v,one,neg_one Vec x,b,u Mat A_petsc KSP ksp PetscInt,parameter:: n_nz=4 !- - - - - - - - - - - - - - - - - - - - - - - - - - - - - ! Other Variables !- - - - - - - - - - - - - - - - - - - - - - - - - - - - - !parameter::n_nz=4 Real*8::Coef(n_nz) PetscInt::snr(n_nz),rnr(n_nz) data Coef /1., 2., 3. , 4./ data snr /0, 1, 2, 3/ data rnr /0, 1 , 2, 3/ ! Body of Debug_PETSc_MatCreate_20101130 ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ! Beginning of program ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - call PetscInitialize(PETSC_NULL_CHARACTER,ierr) call MPI_Comm_rank(PETSC_COMM_WORLD,myid,ierr) call MPI_Comm_size(PETSC_COMM_WORLD,numprocs,ierr) write(*,"('snr=',4i4)")snr write(*,"('rnr=',4i4)")rnr call MatCreate(PETSC_COMM_WORLD,A_Petsc,ierr) call MatSetSizes(A_Petsc,PETSC_DECIDE,PETSC_DECIDE,n_nz,n_nz,ierr) !n_nz-1??? call MatSetFromOptions(A_Petsc,ierr) ! write(*,*)A_petsc call MatGetOwnershipRange(A_Petsc,Istart,Iend,ierr) write(*,'(1a,1i7,1a,1i7)') & '8.....Check after MatGetOwnershipRange() Istart=',Istart,' Iend=',Iend do II=Istart,Iend-1 ione=II+1 !(Coef,snr,rnr are 1-based row and column numbers, shifting them to 0-based) write(*,'(1a,4i7)')'II=',II,ione,snr(ione),rnr(ione) !output snr and rnr for error check call MatSetValues(A_Petsc,1,snr(Ione),1,rnr(Ione),Coef(Ione),INSERT_VALUES,ierr) enddo write(*,'(1a)')'9.....Check after MatSetValues()' call MatAssemblyBegin(A_petsc,MAT_FINAL_ASSEMBLY,ierr) call MatAssemblyEnd(A_Petsc,MAT_FINAL_ASSEMBLY,ierr) write(*,'(1a)')'10.....Check after MatCreate()' call MatView(A_Petsc,PETSC_VIEWER_STDOUT_WORLD,ierr) ! call KSPDestroy(ksp,ierr) ! call VecDestroy(u,ierr) ! call VecDestroy(x,ierr) ! call VecDestroy(b,ierr) call MatDestroy(A_petsc,ierr) call PetscFinalize(ierr) end program Debug_PETSc_MatCreate_20101130 !===================================== -------------- next part -------------- An HTML attachment was scrubbed... URL: