From bernardosk at gmail.com Sat May 1 08:34:50 2010 From: bernardosk at gmail.com (Bernardo Rocha) Date: Sat, 1 May 2010 10:34:50 -0300 Subject: [petsc-users] Sparse matrix vector multiplication performance Message-ID: Hi everyone, I?m wondering if it is possible to measure the CPU time of the sparse matrix vector multiplications within a iterative solver such as the CG in PETSc? I?ve been reading the manual, but it seems that the options -log_summary and similar ones only track the convergence behaviour. So, how one could check the performance of matrix vector multiplications? Thanks in advance, Bernardo M. R. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Sat May 1 08:59:43 2010 From: jed at 59A2.org (Jed Brown) Date: Sat, 01 May 2010 15:59:43 +0200 Subject: [petsc-users] Sparse matrix vector multiplication performance In-Reply-To: References: Message-ID: <87zl0jyb74.fsf@59A2.org> On Sat, 1 May 2010 10:34:50 -0300, Bernardo Rocha wrote: > Hi everyone, > > I?m wondering if it is possible to measure the CPU time of the sparse matrix > vector multiplications within a iterative solver such as the CG in PETSc? > > I?ve been reading the manual, but it seems that the options -log_summary and > similar ones only track the convergence behaviour. Actually, -log_summary is exactly what you are looking for, look for the MatMult event. You should see something like the following: ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ [...] MatMult 1268 1.0 3.5579e+01 1.1 5.18e+09 1.1 5.1e+04 8.9e+03 0.0e+00 12 32 13 18 0 12 32 13 18 0 1124 Jed From jed at 59A2.org Sat May 1 09:47:43 2010 From: jed at 59A2.org (Jed Brown) Date: Sat, 01 May 2010 16:47:43 +0200 Subject: [petsc-users] about snes and finite difference jacobian approximations In-Reply-To: <201004302217070315525@yahoo.cn> References: <201004301512433337340@yahoo.cn> <201004302217070315525@yahoo.cn> Message-ID: <87wrvny8z4.fsf@59A2.org> On Fri, 30 Apr 2010 22:17:09 +0800, "=?gb2312?B?s8LA1sa9o6hMZXBpbmcgQ2hlbqOp?=" wrote: > Dear petsc teams, > When I use snes, I find the Jacobian matrix computation is an imposible misson, > however,I don't konw how to use "Finite Difference Jacobian Approximations", for example, > I modified the example "ex1f.F" and the function FormJacobian() remain unchanged,as follows, > call FormJacobian(snes,x,J,J,flag,dummy,ierr) > call MatGetColoring(J,MATCOLORING_SL,iscoloring,ierr) > call MatFDColoringCreate(J,iscoloring,fdcoloring,ierr) > call ISColoringDestroy(iscoloring,ierr) > call MatFDColoringSetFromOptions(fdcoloring,ierr) > call SNESSetJacobian(snes,J,J,SNESDefaultComputeJacobianColor, fdcoloring,ierr) > but, > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Object is in wrong state! > [0]PETSC ERROR: Must call MatFDColoringSetFunction()! You have not given the MatFDColoring any information about your problem so it has nothing to finite difference. When modifying ex1f.F, you dropped the mandatory call to MatFDColoringSetFunction(). > and I don't understand > /* > This initializes the nonzero structure of the Jacobian. This is artificial > because clearly if we had a routine to compute the Jacobian we wouldn?t > need to use finite differences. > */ > so I don't how to define the function FormJacobian(). Usually the nonzero structure is available based on the mesh and discretization that you are using, and does not depend on the physics of your problem. If you are using a DA, DAGetMatrix() will initialize the matrix so you can replace FormJacobian() with DAGetMatrix(). A more realistic example is in snes/examples/tutorials/ex5.c. Jed PS: Either your message took a day to deliver or it's "Date" header is incorrect. From u.tabak at tudelft.nl Sat May 1 15:27:15 2010 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sat, 01 May 2010 22:27:15 +0200 Subject: [petsc-users] Yet another inversion question Message-ID: <4BDC8EA3.7030300@tudelft.nl> Dear all, I need some terms in my algorithm such as A^{-1} B so and inverted matrix multiplied by another one(A and B pretty sparse, however inversion makes all the sparsity lost). Explicit inverse is not the way to go, however I thought of solving for the columns of B thorugh KSP and put them into another matrix as columns(which might be expensive as well I am not sure)and eventually I will get the term I would like. I know these kinds of terms should be avoided if possible but for the moment I can not avoid them so what would be the best way to compute this dense symmetric resulting matrix. I did a first try to solve for the columns of B, with the umfpack interface and direct solve, which seemed to be fastest on a matrix of size 5544( to solve for all the columns of B). About 1m11sec. to do the above mentioned solves. I was wondering if there some more efficient ways to make this operation faster than what I could accomplish now. Otherwise I should leave this way of thinking. Sth like, for(PetscInt indx=0; indx References: <4BDC8EA3.7030300@tudelft.nl> Message-ID: <8E0C47E0-4FA9-4548-B89B-25E55980FFA6@mcs.anl.gov> Umut, You do not want to use MatGetColumnVector() on a sparse matrix it is very slow. Follow the directions at http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#invertmatrix in your case don't use the identity for B but use your B as a dense matrix. For applying many rows, a direct solver will always win over using an iterative solver for each column. Barry On May 1, 2010, at 3:27 PM, Umut Tabak wrote: > Dear all, > > I need some terms in my algorithm such as > > A^{-1} B > > so and inverted matrix multiplied by another one(A and B pretty > sparse, however inversion makes all the sparsity lost). Explicit > inverse is not the way to go, however I thought of solving for the > columns of B thorugh KSP and put them into another matrix as > columns(which might be expensive as well I am not sure)and > eventually I will get the term I would like. I know these kinds of > terms should be avoided if possible but for the moment I can not > avoid them so what would be the best way to compute this dense > symmetric resulting matrix. > > I did a first try to solve for the columns of B, with the umfpack > interface and direct solve, which seemed to be fastest on a matrix > of size 5544( to solve for all the columns of B). About 1m11sec. to > do the above mentioned solves. I was wondering if there some more > efficient ways to make this operation faster than what I could > accomplish now. Otherwise I should leave this way of thinking. > Sth like, > > for(PetscInt indx=0; indx { > //cout << "Column : " << indx << endl; > // retrieve column > MatGetColumnVector(Mss, u, indx); > /* Solve linear system > */ > ierr = KSPSolve(ksp,u,x);CHKERRQ(ierr); > } > > Best regards, > Umut > > > > > From jed at 59A2.org Sat May 1 15:46:26 2010 From: jed at 59A2.org (Jed Brown) Date: Sat, 01 May 2010 22:46:26 +0200 Subject: [petsc-users] Yet another inversion question In-Reply-To: <4BDC8EA3.7030300@tudelft.nl> References: <4BDC8EA3.7030300@tudelft.nl> Message-ID: <87pr1fxsd9.fsf@59A2.org> On Sat, 01 May 2010 22:27:15 +0200, Umut Tabak wrote: > Dear all, > > I need some terms in my algorithm such as > > A^{-1} B You say that both A and B are sparse, what are you going to do with the product C = A^{-1} B? The most common thing is to apply it to some vectors or perhaps compute a few eigen/singular values/vectors. You rarely need an explicit representation of the product to do such things. > so what would be the best way to compute this dense symmetric > resulting matrix. It is extremely unlikely that C is symmetric. > I did a first try to solve for the columns of B, with the umfpack > interface and direct solve This will work, you could also use the lower level interface to factor A, MatConvert() B to MATDENSE, then MatMatSolve() the result. Internally this will do something very similar to what you have already done, it's questionable whether it would actually be faster (it will certainly use more memory). Jed From u.tabak at tudelft.nl Sat May 1 15:59:22 2010 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sat, 01 May 2010 22:59:22 +0200 Subject: [petsc-users] Yet another inversion question In-Reply-To: <87pr1fxsd9.fsf@59A2.org> References: <4BDC8EA3.7030300@tudelft.nl> <87pr1fxsd9.fsf@59A2.org> Message-ID: <4BDC962A.2000203@tudelft.nl> Jed Brown wrote: > You say that both A and B are sparse, what are you going to do with the > product C = A^{-1} B? The most common thing is to apply it to some > vectors or perhaps compute a few eigen/singular values/vectors. You > rarely need an explicit representation of the product to do such things. > > Dear Barry and Jed, Thanks for the swift replies. Barry, that goes really slower than what I tried. That is not the way to go I suppose. Jed, I have some kind of a symmetrization(or call transformation) matrix for symmetrizing an unsymmetric problem. I need these terms in this explicit symmetrization matrix. I guess this is also a rare case. I could not device an analytical expression to get rid of these kinds of terms yet. That is why I try different approaches, not with high expectations though. > It is extremely unlikely that C is symmetric. > > Yep, you are right ;) > This will work, you could also use the lower level interface to factor > A, MatConvert() B to MATDENSE, then MatMatSolve() the result. > Internally this will do something very similar to what you have already > done, it's questionable whether it would actually be faster (it will > certainly use more memory). > I will see what this does. Thanks and best, Umut From bsmith at mcs.anl.gov Sat May 1 17:00:22 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 1 May 2010 17:00:22 -0500 Subject: [petsc-users] Yet another inversion question In-Reply-To: <4BDC962A.2000203@tudelft.nl> References: <4BDC8EA3.7030300@tudelft.nl> <87pr1fxsd9.fsf@59A2.org> <4BDC962A.2000203@tudelft.nl> Message-ID: <410542CE-8A53-41C9-B390-7C1055CE396C@mcs.anl.gov> On May 1, 2010, at 3:59 PM, Umut Tabak wrote: > Jed Brown wrote: >> You say that both A and B are sparse, what are you going to do with >> the >> product C = A^{-1} B? The most common thing is to apply it to some >> vectors or perhaps compute a few eigen/singular values/vectors. You >> rarely need an explicit representation of the product to do such >> things. >> >> > Dear Barry and Jed, > > Thanks for the swift replies. > > Barry, > > that goes really slower than what I tried. That is not the way to go > I suppose. This cannot be! Perhaps you used the PETSc default solver for my suggestion but UMFPACK for your version? With a direct solve "all" the work will be in the factorization and triangular solves so if you use UMFPACK for both they both do the same amount of work. Barry > > Jed, > > I have some kind of a symmetrization(or call transformation) matrix > for symmetrizing an unsymmetric problem. I need these terms in this > explicit symmetrization matrix. I guess this is also a rare case. I > could not device an analytical expression to get rid of these kinds > of terms yet. That is why I try different approaches, not with high > expectations though. >> It is extremely unlikely that C is symmetric. >> >> > Yep, you are right ;) >> This will work, you could also use the lower level interface to >> factor >> A, MatConvert() B to MATDENSE, then MatMatSolve() the result. >> Internally this will do something very similar to what you have >> already >> done, it's questionable whether it would actually be faster (it will >> certainly use more memory). >> > I will see what this does. > > Thanks and best, > Umut From jeffrey.scott.brown at gmail.com Sat May 1 18:25:27 2010 From: jeffrey.scott.brown at gmail.com (Jeff Brown) Date: Sat, 1 May 2010 17:25:27 -0600 Subject: [petsc-users] distributed arrays Message-ID: I'm new to PetSc, but have been following your mailing list for several months. I'm using fortran95 if it changes the answer, but from reading the manual it looks like only syntax changes. I'm interested in using distributed arrays in my work, but I need to be able to specify 2D grids that are not simply NX by NY. I'm using a finite difference formulation and I would like to be able to use a completely arbitrary number of processors, like NP = 17. For NP=17 on a square underlying finite difference grid (for the following illustration, the side of a sqaure=1), the optimum configuration (minimizing boundary surface to interior volume) is the following: [ 3 x 3 grid, each individual square is 1/3 x 3/17 on a side], followed by a [2x4 grid where each individual square is 1/2 x 2/17 on a side] I've illustrated this below, with A and B being the 1/3 x 3/17 grids and C and D being the 1/2 by 2/17 cells AAA BBB AAA CC DD CC DD AAA BBB AAA CC DD CC DD BBB AAA BBB CC DD CC DD BBB AAA BBB DD CC DD CC AAA BBB AAA DD CC DD CC AAA BBB AAA DD CC DD CC Is there a way to specify this distribution of grid cells using distributed arrays? ------------------------- At different stages in computing with the distributed arrays, I would like to be able to group sets of ghost values together for communication. These will be different groups at different stages of the computation. Let me illustrate with an example. Step 1: Calculate A[east], B[east], C[east], D[east] on the ghost interface, each of A, B, C, D is a distributed array of doubles [east] means calculating the ghost interface to the east of each cell Step 2: Transmit (A,B,C,D)[east] together. Let's pick the stuff on processor 1 for reference. If I were implementing this in raw MPI, I would create a combined buffer = {A1east, B1east, C1east, D1east} and then send the total buffer Step 3: Overlapping calculations of A[west], B[west], C[west], D[west] Step 4: Transmit (A,B,C,D)[west] together Step 5: Overlapping calculations of A[north], B[north], C[north], D[north] Step 6: Transmit (A,B,C,D)[north] Step 7: Overlapping calculations of A[south], B[south], C[south], D[south] Step 8: Transmit (A,B,C,D)[south] Step 9: Overlapping calcluations of A[center], B[center], C[center], D[center] Step 10: Wait on east, west, north, south At different stages, A, B, C, D might be A, B, C, D, E, F (where ABCD are the same as above). Is there a good way to do this? Suggestions? --Jeff Brown -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat May 1 18:50:32 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 1 May 2010 18:50:32 -0500 Subject: [petsc-users] distributed arrays In-Reply-To: References: Message-ID: <7E4E2853-DCDD-4333-ACCA-EA5D166A9D89@mcs.anl.gov> On May 1, 2010, at 6:25 PM, Jeff Brown wrote: > I'm new to PetSc, but have been following your mailing list for > several months. > I'm using fortran95 if it changes the answer, but from reading the > manual it looks like only syntax changes. > > > I'm interested in using distributed arrays in my work, but I need to > be able to specify 2D grids that are not simply NX by NY. > I'm using a finite difference formulation and I would like to be > able to use a completely arbitrary number of processors, > like NP = 17. > > For NP=17 on a square underlying finite difference grid (for the > following illustration, the side of a sqaure=1), > the optimum configuration (minimizing boundary surface to interior > volume) is the following: > > [ 3 x 3 grid, each individual square is 1/3 x 3/17 on a side], > followed by a > [2x4 grid where each individual square is 1/2 x 2/17 on a side] > > I've illustrated this below, with A and B being the 1/3 x 3/17 grids > and C and D being the 1/2 by 2/17 cells > AAA BBB AAA CC DD CC DD > AAA BBB AAA CC DD CC DD > BBB AAA BBB CC DD CC DD > BBB AAA BBB DD CC DD CC > AAA BBB AAA DD CC DD CC > AAA BBB AAA DD CC DD CC > > Is there a way to specify this distribution of grid cells using > distributed arrays? > > No, the PETSc DA can only handle tensor product grids. > ------------------------- > > At different stages in computing with the distributed arrays, I > would like to be able to group sets of ghost values together for > communication. > These will be different groups at different stages of the computation. > > Let me illustrate with an example. > Step 1: Calculate A[east], B[east], C[east], D[east] on the ghost > interface, > each of A, B, C, D is a distributed array of doubles > [east] means calculating the ghost interface to the east of > each cell > Step 2: Transmit (A,B,C,D)[east] together. > Let's pick the stuff on processor 1 for reference. > If I were implementing this in raw MPI, I would create a > combined buffer = {A1east, B1east, C1east, D1east} and then send the > total buffer > Step 3: Overlapping calculations of A[west], B[west], C[west], D[west] > Step 4: Transmit (A,B,C,D)[west] together > Step 5: Overlapping calculations of A[north], B[north], C[north], > D[north] > Step 6: Transmit (A,B,C,D)[north] > Step 7: Overlapping calculations of A[south], B[south], C[south], > D[south] > Step 8: Transmit (A,B,C,D)[south] > Step 9: Overlapping calcluations of A[center], B[center], C[center], > D[center] > Step 10: Wait on east, west, north, south > > At different stages, A, B, C, D might be A, B, C, D, E, F (where > ABCD are the same as above). > > Is there a good way to do this? The PETSc DA provides a simple interface for ghost points for tensor product grids. It doesn't have this level of control of when communication takes place. Of course, you can write code to manage the decomposition and the ghost point updates yourself directly using MPI (or some package that does this) and still "wrap" the arrays as PETSc Vecs and so still use the various PETSc solvers (but not using the DAs). Barry > Suggestions? > > --Jeff Brown > > > > From chenleping at yahoo.cn Sat May 1 02:35:25 2010 From: chenleping at yahoo.cn (=?gb2312?B?s8LA1sa9o6hMZXBpbmcgQ2hlbqOp?=) Date: Sat, 1 May 2010 15:35:25 +0800 Subject: [petsc-users] about finite difference jacobian approximations Message-ID: <201005011535200026274@yahoo.cn> hello, petsc teams, When I use finite difference jacobian approximations, I only find the computation methods and results of jacobian matrix cannot influence iterative process of snes. Is it right? How should I compute the jacobian matrix? thanks, leping 2010-05-01 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Sun May 2 07:10:25 2010 From: jed at 59A2.org (Jed Brown) Date: Sun, 02 May 2010 14:10:25 +0200 Subject: [petsc-users] about finite difference jacobian approximations In-Reply-To: <201005011535200026274@yahoo.cn> References: <201005011535200026274@yahoo.cn> Message-ID: <87ljc2y05q.fsf@59A2.org> On Sat, 1 May 2010 15:35:25 +0800, "=?gb2312?B?s8LA1sa9o6hMZXBpbmcgQ2hlbqOp?=" wrote: > hello, petsc teams, > > When I use finite difference jacobian approximations, I only find the > computation methods and results of jacobian matrix cannot influence > iterative process of snes. I don't know what you're asking. The accuracy of the Jacobian certainly can affect the convergence rate; for simple or well-conditioned problems, finite difference Jacobians are usually reliable. Analytic Jacobians, or Jacobians from automatic differentiation (AD) are usually better when available and not too expensive to compute. Another common thing is to assemble a "cheap" preconditioning matrix (dropping terms, or with a lower order discretization) for use with a matrix-free (usually finite difference or AD) operator. Jed From jed at 59A2.org Sun May 2 07:13:21 2010 From: jed at 59A2.org (Jed Brown) Date: Sun, 02 May 2010 14:13:21 +0200 Subject: [petsc-users] Yet another inversion question In-Reply-To: <410542CE-8A53-41C9-B390-7C1055CE396C@mcs.anl.gov> References: <4BDC8EA3.7030300@tudelft.nl> <87pr1fxsd9.fsf@59A2.org> <4BDC962A.2000203@tudelft.nl> <410542CE-8A53-41C9-B390-7C1055CE396C@mcs.anl.gov> Message-ID: <87iq76y00u.fsf@59A2.org> On Sat, 1 May 2010 17:00:22 -0500, Barry Smith wrote: > This cannot be! Perhaps you used the PETSc default solver for my > suggestion but UMFPACK for your version? With a direct solve "all" the > work will be in the factorization and triangular solves so if you use > UMFPACK for both they both do the same amount of work. For what it's worth, extracting a column should cost a fair amount less than doing a solve since the factored matrix will usually have a lot more entries. Jed From chenleping at yahoo.cn Sat May 1 07:31:11 2010 From: chenleping at yahoo.cn (=?gb2312?B?s8LA1sa9o6hMZXBpbmcgQ2hlbqOp?=) Date: Sat, 1 May 2010 20:31:11 +0800 Subject: [petsc-users] about finite difference jacobian approximations References: <201005011535200026274@yahoo.cn> Message-ID: <201005012031087256640@yahoo.cn> hello, petsc teams, If I want to use finite difference jacobian approximations,how should I define the function FormJacobian()? I don't get analytic jacobians. thanks, leping 2010-05-02 ???? Jed Brown ????? 2010-05-02 20:10:20 ???? chenleping; PETSc users list ??? ??? Re: [petsc-users] about finite difference jacobian approximations On Sat, 1 May 2010 15:35:25 +0800, "=?gb2312?B?s8LA1sa9o6hMZXBpbmcgQ2hlbqOp?=" wrote: > hello, petsc teams, > > When I use finite difference jacobian approximations, I only find the > computation methods and results of jacobian matrix cannot influence > iterative process of snes. I don't know what you're asking. The accuracy of the Jacobian certainly can affect the convergence rate; for simple or well-conditioned problems, finite difference Jacobians are usually reliable. Analytic Jacobians, or Jacobians from automatic differentiation (AD) are usually better when available and not too expensive to compute. Another common thing is to assemble a "cheap" preconditioning matrix (dropping terms, or with a lower order discretization) for use with a matrix-free (usually finite difference or AD) operator. Jed -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Sun May 2 07:41:25 2010 From: jed at 59A2.org (Jed Brown) Date: Sun, 02 May 2010 14:41:25 +0200 Subject: [petsc-users] about finite difference jacobian approximations In-Reply-To: <201005012031087256640@yahoo.cn> References: <201005011535200026274@yahoo.cn> <201005012031087256640@yahoo.cn> Message-ID: <87bpcyxyq2.fsf@59A2.org> On Sat, 1 May 2010 20:31:11 +0800, "=?gb2312?B?s8LA1sa9o6hMZXBpbmcgQ2hlbqOp?=" wrote: > hello, petsc teams, > > If I want to use finite difference jacobian approximations,how should I define > the function FormJacobian()? I don't get analytic jacobians. You don't write FormJacobian(), you just define the nonzero structure of the matrix you want. As I said before, look at http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/src/snes/examples/tutorials/ex5.c.html#line150 Jed PS: You should set your computer's clock forward one day (it thinks today is May 1 so your message will show up in a very strange order with many email clients). Jed From rxk at cfdrc.com Sun May 2 13:26:21 2010 From: rxk at cfdrc.com (Ravi Kannan) Date: Sun, 2 May 2010 13:26:21 -0500 Subject: [petsc-users] updating the host value using the ghost values Message-ID: <000601caea24$f6f62850$e4e278f0$@com> Hi all, Does anyone have a small function (petsc function), which takes in a vector (having host + ghost cell values of a scalar) and then updates the host values using the ghost values. The converse is well known : update the ghost using the host. But how about updating the host from the ghost values? Thanks in advance Ravi Ravi Kannan, PhD Research Engineer CFD Research Corporation 215 Wynn Drive, Suite 501 Huntsville, AL 35805 (256)726-4851 rxk at cfdrc.com ____________________________ Confidentiality Notice The information contained in this communication and its attachments is intended only for the use of the individual to whom it is addressed and may contain information that is legally privileged, confidential, or exempt from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, copying, or reliance on this communication is strictly prohibited. If you have received this communication in error, please notify confidentiality at cfdrc.com (256-726-4800) and delete the communication without retaining any copies. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Sun May 2 13:35:53 2010 From: jed at 59A2.org (Jed Brown) Date: Sun, 02 May 2010 20:35:53 +0200 Subject: [petsc-users] updating the host value using the ghost values In-Reply-To: <000601caea24$f6f62850$e4e278f0$@com> References: <000601caea24$f6f62850$e4e278f0$@com> Message-ID: <874oiqxiba.fsf@59A2.org> On Sun, 2 May 2010 13:26:21 -0500, "Ravi Kannan" wrote: > Does anyone have a small function (petsc function), which takes in a vector > (having host + ghost cell values of a scalar) and then updates the host > values using the ghost values. The converse is well known : update the > ghost using the host. But how about updating the host from the ghost values? To the sum the values on the owning process, you can use DALocalToGlobalBegin(da,local,global) or VecScatterBegin(scatter,local,global,ADD_VALUES,SCATTER_REVERSE). INSERT_VALUES doesn't make sense in this context because shared values are often ghosted multiple times. Jed From bsmith at mcs.anl.gov Sun May 2 14:10:35 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 2 May 2010 14:10:35 -0500 Subject: [petsc-users] Yet another inversion question In-Reply-To: <87iq76y00u.fsf@59A2.org> References: <4BDC8EA3.7030300@tudelft.nl> <87pr1fxsd9.fsf@59A2.org> <4BDC962A.2000203@tudelft.nl> <410542CE-8A53-41C9-B390-7C1055CE396C@mcs.anl.gov> <87iq76y00u.fsf@59A2.org> Message-ID: On May 2, 2010, at 7:13 AM, Jed Brown wrote: > On Sat, 1 May 2010 17:00:22 -0500, Barry Smith > wrote: >> This cannot be! Perhaps you used the PETSc default solver for my >> suggestion but UMFPACK for your version? With a direct solve "all" >> the >> work will be in the factorization and triangular solves so if you use >> UMFPACK for both they both do the same amount of work. > > For what it's worth, extracting a column should cost a fair amount > less > than doing a solve since the factored matrix will usually have a lot > more entries. Perhaps. But the current get column code is terribly inefficient. So is the issue that MatMatSolve(A,B,X) requires TWO dense matrices B and X, instead of just one and that is too much memory? Perhaps we could modify MatMatSolve(A,B,X) to allow an in-place version (i.e. B == X) that uses only 1 or several work vectors then you no longer have this memory issue? Should I add this? Barry > > Jed From jed at 59A2.org Sun May 2 14:36:40 2010 From: jed at 59A2.org (Jed Brown) Date: Sun, 02 May 2010 21:36:40 +0200 Subject: [petsc-users] Yet another inversion question In-Reply-To: References: <4BDC8EA3.7030300@tudelft.nl> <87pr1fxsd9.fsf@59A2.org> <4BDC962A.2000203@tudelft.nl> <410542CE-8A53-41C9-B390-7C1055CE396C@mcs.anl.gov> <87iq76y00u.fsf@59A2.org> Message-ID: <871vduxfhz.fsf@59A2.org> On Sun, 2 May 2010 14:10:35 -0500, Barry Smith wrote: > Perhaps. But the current get column code is terribly inefficient. > > So is the issue that MatMatSolve(A,B,X) requires TWO dense > matrices B and X, instead of just one and that is too much memory? > > Perhaps we could modify MatMatSolve(A,B,X) to allow an in-place > version (i.e. B == X) that uses only 1 or several work vectors then > you no longer have this memory issue? Should I add this? It would be easy to make MatGetColumnVector faster (use better than linear search, maybe specialize for AIJ). If it's really important to make this fast then MatMatSolve for sparse B might make sense, but I think it's well worth benchmarking first because I'm not convinced that MatGetColumnVector will take a significant amount of the time (at least after easy speedups) compared to the solves. SuperLU_Dist (and maybe other direct solvers?) have a MatMatSolve API, if this indicates that it actually solves multiple systems at once, then the performance difference over solving lots of single-vector systems might make specializing worthwhile. But the purist in me still thinks the solution is to provide support for doing whatever users want to do with these resulting matrices (or just document as the use cases come up). Jed From sander.land at comlab.ox.ac.uk Mon May 3 08:15:39 2010 From: sander.land at comlab.ox.ac.uk (Sander Land) Date: Mon, 3 May 2010 14:15:39 +0100 Subject: [petsc-users] free unused allocated nonzeros after overestimating preallocation Message-ID: Hi, I am preallocating a matrix using MatMPIAIJSetPreallocation(HalfM,27,PETSC_NULL,26,PETSC_NULL); Since I have no easy way to know whether the entries will end up in petsc's 'diagonal' or 'nondiagonal'. After assembling the matrix this results in: Matrix Object: type=mpiaij, rows=9801, cols=9801 total: nonzeros=231601, allocated nonzeros=519453 [0] Local rows 9801 nz 231601 nz alloced 519453 mem 6551364, not using I-node routines [0] on-diagonal part: nz 231601 [0] off-diagonal part: nz 0 On 1 processor, and some numbers split between on/off on more, as expected. So far no problem since I can temporarily spare the memory for 1 such matrix. Next, I would like to compress the storage somehow, to free the memory not used, since I know this nonzero pattern will not change, but I can't find a way to do this in petsc. Worse, when I use MatConvert/MatDuplicate to initialize my other matrices, they all seem to inherit this property of using about twice as much memory as needed. This is a problem when taking the number of rows to 1 million +. What is the easiest way to free the unused memory? Thanks, Sander Land From knepley at gmail.com Mon May 3 08:20:44 2010 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 3 May 2010 08:20:44 -0500 Subject: [petsc-users] free unused allocated nonzeros after overestimating preallocation In-Reply-To: References: Message-ID: On Mon, May 3, 2010 at 8:15 AM, Sander Land wrote: > Hi, > > I am preallocating a matrix using > MatMPIAIJSetPreallocation(HalfM,27,PETSC_NULL,26,PETSC_NULL); > Since I have no easy way to know whether the entries will end up in > petsc's 'diagonal' or 'nondiagonal'. After assembling the matrix this > results in: > > Matrix Object: > type=mpiaij, rows=9801, cols=9801 > total: nonzeros=231601, allocated nonzeros=519453 > [0] Local rows 9801 nz 231601 nz alloced 519453 mem 6551364, not > using I-node routines > [0] on-diagonal part: nz 231601 > [0] off-diagonal part: nz 0 > > On 1 processor, and some numbers split between on/off on more, as > expected. So far no problem since I can temporarily spare the memory > for 1 such matrix. > Next, I would like to compress the storage somehow, to free the memory > not used, since I know this nonzero pattern will not change, but I > can't find a way to do this in petsc. > Worse, when I use MatConvert/MatDuplicate to initialize my other > matrices, they all seem to inherit this property of using about twice > as much memory as needed. This is a problem when taking the number of > rows to 1 million +. > > What is the easiest way to free the unused memory? > MatAssemblyBegin/End() frees the unused memory, so you must be calculating the amount of memory needed incorrectly. Remember that memory is necessary not only for the values, but the indexing structures. Matt > Thanks, > Sander Land > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From neckel at in.tum.de Mon May 3 08:21:34 2010 From: neckel at in.tum.de (Tobias Neckel) Date: Mon, 03 May 2010 15:21:34 +0200 Subject: [petsc-users] free unused allocated nonzeros after overestimating preallocation In-Reply-To: References: Message-ID: <4BDECDDE.8090302@in.tum.de> Hi, > I am preallocating a matrix using > MatMPIAIJSetPreallocation(HalfM,27,PETSC_NULL,26,PETSC_NULL); > Since I have no easy way to know whether the entries will end up in > petsc's 'diagonal' or 'nondiagonal'. After assembling the matrix this > results in: > > Matrix Object: > type=mpiaij, rows=9801, cols=9801 > total: nonzeros=231601, allocated nonzeros=519453 > [0] Local rows 9801 nz 231601 nz alloced 519453 mem 6551364, not > using I-node routines > [0] on-diagonal part: nz 231601 > [0] off-diagonal part: nz 0 > > On 1 processor, and some numbers split between on/off on more, as > expected. So far no problem since I can temporarily spare the memory > for 1 such matrix. > Next, I would like to compress the storage somehow, to free the memory > not used, since I know this nonzero pattern will not change, but I > can't find a way to do this in petsc. > Worse, when I use MatConvert/MatDuplicate to initialize my other > matrices, they all seem to inherit this property of using about twice > as much memory as needed. This is a problem when taking the number of > rows to 1 million +. > > What is the easiest way to free the unused memory? Why not iterating once to count how many entries you really need (and storing the counts per row in a vector), creating the matrix with the exact number of necessary entries, and then iterating a second time? Seems easier than squeezing out sth. Best regards Tobias From jed at 59A2.org Mon May 3 08:40:41 2010 From: jed at 59A2.org (Jed Brown) Date: Mon, 03 May 2010 15:40:41 +0200 Subject: [petsc-users] free unused allocated nonzeros after overestimating preallocation In-Reply-To: References: Message-ID: <87ljc1w1ba.fsf@59A2.org> On Mon, 3 May 2010 08:20:44 -0500, Matthew Knepley wrote: > MatAssemblyBegin/End() frees the unused memory, so you must be > calculating the amount of memory needed incorrectly. It compacts the values but does not free the extra memory. It would be easy to add a matrix option to allocate new space for the entries and column indices, then copy the entries over, but requires too much memory to do it by default. MatDuplicate on the other hand only allocates for the used values, but copies the "maxnz" field so it will incorrectly report more allocation than there really is. Jed From knepley at gmail.com Mon May 3 08:45:59 2010 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 3 May 2010 08:45:59 -0500 Subject: [petsc-users] free unused allocated nonzeros after overestimating preallocation In-Reply-To: <87ljc1w1ba.fsf@59A2.org> References: <87ljc1w1ba.fsf@59A2.org> Message-ID: On Mon, May 3, 2010 at 8:40 AM, Jed Brown wrote: > On Mon, 3 May 2010 08:20:44 -0500, Matthew Knepley > wrote: > > MatAssemblyBegin/End() frees the unused memory, so you must be > > calculating the amount of memory needed incorrectly. > > It compacts the values but does not free the extra memory. It would be > easy to add a matrix option to allocate new space for the entries and > column indices, then copy the entries over, but requires too much memory > to do it by default. > Yep, does not free by default, which would mean making a copy. > MatDuplicate on the other hand only allocates for the used values, but > copies the "maxnz" field so it will incorrectly report more allocation > than there really is. I can fix this. Matt > > Jed > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From sander.land at comlab.ox.ac.uk Mon May 3 09:10:43 2010 From: sander.land at comlab.ox.ac.uk (Sander Land) Date: Mon, 3 May 2010 15:10:43 +0100 Subject: [petsc-users] free unused allocated nonzeros after overestimating preallocation In-Reply-To: <87ljc1w1ba.fsf@59A2.org> References: <87ljc1w1ba.fsf@59A2.org> Message-ID: Thanks for the quick answers everyone. I was using 8 matrices of 1.03 million rows each, and my memory usage went off to 3+ GB, whereas I was expecting around twice (8 ? 1030301?27?sizeof(double) < 1 GB) Looking at the actual mem usage of the duplicated matrices, it seems that this is not inherited, but just more than I was expecting in the first place, and the wrong "allocated nonzeros" entry threw me off. Matrix Object: type=mpiaij, rows=1030301, cols=1030301 total: nonzeros=27270901, allocated nonzeros=54605953 [0] Local rows 1030301 nz 27270901 nz alloced 54605953 mem 688245364, not using I-node routines MatDuplicate'd : Matrix Object: type=mpiaij, rows=1030301, cols=1030301 total: nonzeros=27270901, allocated nonzeros=54605953 [0] Local rows 1030301 nz 27270901 nz alloced 54605953 mem 360224748, not using I-node routines I guess MAT_SHARE_NONZERO_PATTERN could help reduce this, but my current version (ubuntu package) doesn't have this, so I'm going to recompile from sources. On Mon, May 3, 2010 at 2:40 PM, Jed Brown wrote: > On Mon, 3 May 2010 08:20:44 -0500, Matthew Knepley wrote: >> MatAssemblyBegin/End() frees the unused memory, so you must be >> calculating the amount of memory needed incorrectly. > > It compacts the values but does not free the extra memory. ?It would be > easy to add a matrix option to allocate new space for the entries and > column indices, then copy the entries over, but requires too much memory > to do it by default. > > MatDuplicate on the other hand only allocates for the used values, but > copies the "maxnz" field so it will incorrectly report more allocation > than there really is. > > Jed > From sander.land at comlab.ox.ac.uk Mon May 3 11:08:01 2010 From: sander.land at comlab.ox.ac.uk (Sander Land) Date: Mon, 3 May 2010 17:08:01 +0100 Subject: [petsc-users] free unused allocated nonzeros after overestimating preallocation In-Reply-To: References: <87ljc1w1ba.fsf@59A2.org> Message-ID: Quick follow-up question: is MAT_SHARE_NONZERO_PATTERN fully implemented? Using a quick 'grep', it seems only baij/seq and sbaij/seq even check for the flag. From knepley at gmail.com Mon May 3 10:49:38 2010 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 3 May 2010 10:49:38 -0500 Subject: [petsc-users] DAVecGetArray, ghost access In-Reply-To: References: Message-ID: 2010/4/29 ????? ??????? > Dear PETSc team, all! > > I'm involved in developing PETSc-assisted code and recently came across the > following problem. > > As described in user manual (rev.3.1, p.48), elements of vector that is > managed by DA object can be directly accessed by DAVecGetArray(...) and > DAVecRestoreArray(...) where the vector is either local or global. Besides, > there's written that GHOSTED values can be accessed in this way. It seems to > be not applicable for global vectors, but for local ones only. > > As far as I understand, it's proposed to get ghosts by creating a new local > vector, that contains full local part of global vector plus ghosts. > Therefore the memory is just twice we really need, not to mention excessive > data coping. > > So the question arises. In what way can user get access to ghosted values > without creating additional DA local vector? > That does not really make sense. Ghosted values exist in the memory of other processes. So either 1) You think copying takes to much time Measure the time. It is insignificant. 2) You want to use less memory You can build a VecScatter to get only the values you want. Since we solve mostly implicit problems, the memory has not been an issue. Matt RSVP > Best Wishes, Pavel V. Chouvakhov. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon May 3 11:25:04 2010 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 3 May 2010 11:25:04 -0500 Subject: [petsc-users] free unused allocated nonzeros after overestimating preallocation In-Reply-To: References: <87ljc1w1ba.fsf@59A2.org> Message-ID: No, its not done for AIJ. Matt On Mon, May 3, 2010 at 11:08 AM, Sander Land wrote: > Quick follow-up question: is MAT_SHARE_NONZERO_PATTERN fully implemented? > Using a quick 'grep', it seems only baij/seq and sbaij/seq even check > for the flag. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From lizs at mail.uc.edu Tue May 4 16:44:23 2010 From: lizs at mail.uc.edu (Li, Zhisong (lizs)) Date: Tue, 4 May 2010 21:44:23 +0000 Subject: [petsc-users] How to save a vector value of multiple DOF to a file? Message-ID: <88D7E3BB7E1960428303E76010037451019C69@BL2PRD0103MB055.prod.exchangelabs.com> Hi, I want to setup a function in the main function to save the data to a file. For example suppose the vector x in ts/tutorials/ex7 consists of 3 DOF as typedef struct { PetscScalar u, v, p; } Field; I first followed the advised method to collect all values from a parallel vector into a vector on the 0th processor: VecScatterCreateToZero(x, &scatter, &y); VecScatterBegin(scatter,x,y,INSERT_VALUES,SCATTER_FORWARD); VecScatterEnd(scatter,x,y,INSERT_VALUES,SCATTER_FORWARD); VecScatterDestroy(scatter); and then use Field **z; DAVecGetArray(da, y, &z); and PetscViewerASCIIOpen(comm,"./result.txt",&viewer); PetscViewerASCIIPrintf(viewer, "%G ", &z[j][i].u ); to finish. But I always either meet error for the line "DAVecGetArray(da, y, &z);" or save a file of all extremely small values(e.g. 8.652e-317), which is obviously wrong. So what is the correct way to do this? Where can I find a similar example to follow? I also wonder if "DAGetGlobalVector( )" will do an equal job to collect parallel values? Thank you. Best Regards, Zhisong Li -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue May 4 17:17:21 2010 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 4 May 2010 17:17:21 -0500 Subject: [petsc-users] How to save a vector value of multiple DOF to a file? In-Reply-To: <88D7E3BB7E1960428303E76010037451019C69@BL2PRD0103MB055.prod.exchangelabs.com> References: <88D7E3BB7E1960428303E76010037451019C69@BL2PRD0103MB055.prod.exchangelabs.com> Message-ID: On Tue, May 4, 2010 at 4:44 PM, Li, Zhisong (lizs) wrote: > > Hi, > > I want to setup a function in the main function to save the data to a file. > For example suppose the vector x in ts/tutorials/ex7 consists of 3 DOF as > > typedef struct { > PetscScalar u, v, p; > > } Field; > > I first followed the advised method to collect all values from a parallel > vector into a vector on the 0th processor: > > VecScatterCreateToZero(x, &scatter, &y); > VecScatterBegin(scatter,x,y,INSERT_VALUES,SCATTER_FORWARD); > VecScatterEnd(scatter,x,y,INSERT_VALUES,SCATTER_FORWARD); > VecScatterDestroy(scatter); > > and then use > Field **z; > DAVecGetArray(da, y, &z); > > and > PetscViewerASCIIOpen(comm,"./result.txt",&viewer); > PetscViewerASCIIPrintf(viewer, "%G ", &z[j][i].u ); > > to finish. But I always either meet error for the line "DAVecGetArray(da, > y, &z);" or save a file of all extremely small values(e.g. 8.652e-317), > which is obviously wrong. So what is the correct way to do this? Where can I > find a similar example to follow? > What error? This is correct way to do it, if you do not just use VecView(). Matt > I also wonder if "DAGetGlobalVector( )" will do an equal job to collect > parallel values? > > > Thank you. > > > Best Regards, > > > Zhisong Li > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From chenleping at yahoo.cn Wed May 5 10:02:52 2010 From: chenleping at yahoo.cn (=?gb2312?B?s8LA1sa9o6hMZXBpbmcgQ2hlbqOp?=) Date: Wed, 5 May 2010 23:02:52 +0800 Subject: [petsc-users] about snes Message-ID: <201005052302363263062@yahoo.cn> petsc teams, When I use SNES, I find my program iterated many times but x vector and residual vector remained unchanged, that is to say,no convergence.I used finite difference jacobian approximations, but I cannot confirm the jacobian matrix structure is right.Whether the matrix structure will influence the convergence of SNES or not? Why does my x vector and residual vector remained unchanged? call FormJacobian(snes,x,J,J,flag,dummy,ierr) call MatGetColoring(J,MATCOLORING_SL,iscoloring,ierr) call MatFDColoringCreate(J,iscoloring,fdcoloring,ierr) call ISColoringDestroy(iscoloring,ierr) call MatFDColoringSetFromOptions(fdcoloring,ierr) call MatFDColoringSetFunction(fdcoloring,FormFunction, & PETSC_NULL_OBJECT,ierr) call SNESSetJacobian(snes,J,J,SNESDefaultComputeJacobianColor, & fdcoloring,ierr) call SNESGetKSP(snes,ksp,ierr) call KSPGetPC(ksp,pc,ierr) call SNESSolve(snes,PETSC_NULL_OBJECT,petscu,ierr) Thanks, leping -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed May 5 10:08:23 2010 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 5 May 2010 10:08:23 -0500 Subject: [petsc-users] about snes In-Reply-To: <201005052302363263062@yahoo.cn> References: <201005052302363263062@yahoo.cn> Message-ID: Try using -snes_mf which will approximate the Jacobian automatically. Matt 2010/5/5 ????Leping Chen? > petsc teams, > > When I use SNES, I find my program iterated many times but x vector and > residual vector > > remained unchanged, that is to say,no convergence.I used > finite difference jacobian approximations, > > but I cannot confirm the jacobian matrix structure is right.Whether the > matrix structure will influence > > the convergence of SNES or not? Why does my x vector and residual vector remained > unchanged? > > call FormJacobian(snes,x,J,J,flag,dummy,ierr) > call MatGetColoring(J,MATCOLORING_SL,iscoloring,ierr) > call MatFDColoringCreate(J,iscoloring,fdcoloring,ierr) > call ISColoringDestroy(iscoloring,ierr) > call MatFDColoringSetFromOptions(fdcoloring,ierr) > call MatFDColoringSetFunction(fdcoloring,FormFunction, > & PETSC_NULL_OBJECT,ierr) > call SNESSetJacobian(snes,J,J,SNESDefaultComputeJacobianColor, > & fdcoloring,ierr) > call SNESGetKSP(snes,ksp,ierr) > call KSPGetPC(ksp,pc,ierr) > call SNESSolve(snes,PETSC_NULL_OBJECT,petscu,ierr) > Thanks, > leping > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From lizs at mail.uc.edu Wed May 5 13:17:52 2010 From: lizs at mail.uc.edu (Li, Zhisong (lizs)) Date: Wed, 5 May 2010 18:17:52 +0000 Subject: [petsc-users] save a vector value of multiple DOF to a file Message-ID: <88D7E3BB7E1960428303E76010037451019C9D@BL2PRD0103MB055.prod.exchangelabs.com> Hi, I temporarily remove all the file opening/writing operations and found that it's probably "DAVecGetArray()" that's causing the trouble when run on 2 processors. It looks okay when in serial processing. I don't use VecView() at all, as X-display doesn't function well for parallel processing. I also tried to add these lines to first map the global vector xx to a newly created global vector x , before I scatter x to y: DACreateNaturalVector(da, &x); DAGlobalToNaturalBegin(da, xx,INSERT_VALUES, x); DAGlobalToNaturalEnd(da, xx,INSERT_VALUES, x); But this doesn't help. The error message: [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Arguments are incompatible! [1]PETSC ERROR: Vector local size 0 is not compatible with DA local sizes 4284 4437 ! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 1, Thu Apr 8 14:16:50 CDT 2010 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: ./ex7 on a linux-gnu named custard by verhoff Wed May 5 14:13:17 2010 [1]PETSC ERROR: Libraries linked from /home/verhoff/petsc/linux-gnu-c-debug/lib [1]PETSC ERROR: Configure run at Tue Apr 27 11:40:02 2010 [1]PETSC ERROR: Configure options --download-c-blas-lapack=1 --with-cc=gcc --download-mpich=1 [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: DAVecGetArray() line 53 in src/dm/da/src/dagetarray.c [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Arguments are incompatible! [1]PETSC ERROR: Vector local size 0 is not compatible with DA local sizes 4284 4437 ! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 1, Thu Apr 8 14:16:50 CDT 2010 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: ./ex7 on a linux-gnu named custard by verhoff Wed May 5 14:13:17 2010 [1]PETSC ERROR: Libraries linked from /home/verhoff/petsc/linux-gnu-c-debug/lib [1]PETSC ERROR: Configure run at Tue Apr 27 11:40:02 2010 [1]PETSC ERROR: Configure options --download-c-blas-lapack=1 --with-cc=gcc --download-mpich=1 [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: DAVecRestoreArray() line 108 in src/dm/da/src/dagetarray.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Arguments are incompatible! [0]PETSC ERROR: Vector local size 8721 is not compatible with DA local sizes 4437 4590 ! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 1, Thu Apr 8 14:16:50 CDT 2010 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./ex7 on a linux-gnu named custard by verhoff Wed May 5 14:13:17 2010 [0]PETSC ERROR: Libraries linked from /home/verhoff/petsc/linux-gnu-c-debug/lib [0]PETSC ERROR: Configure run at Tue Apr 27 11:40:02 2010 [0]PETSC ERROR: Configure options --download-c-blas-lapack=1 --with-cc=gcc --download-mpich=1 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: DAVecGetArray() line 53 in src/dm/da/src/dagetarray.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Arguments are incompatible! [0]PETSC ERROR: Vector local size 8721 is not compatible with DA local sizes 4437 4590 ! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 1, Thu Apr 8 14:16:50 CDT 2010 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./ex7 on a linux-gnu named custard by verhoff Wed May 5 14:13:17 2010 [0]PETSC ERROR: Libraries linked from /home/verhoff/petsc/linux-gnu-c-debug/lib [0]PETSC ERROR: Configure run at Tue Apr 27 11:40:02 2010 [0]PETSC ERROR: Configure options --download-c-blas-lapack=1 --with-cc=gcc --download-mpich=1 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: DAVecRestoreArray() line 108 in src/dm/da/src/dagetarray.c Thanks, Regards, Zhisong Li > Hi, > > I want to setup a function in the main function to save the data to a file. > For example suppose the vector x in ts/tutorials/ex7 consists of 3 DOF as > > typedef struct { > PetscScalar u, v, p; > } Field; > > I first followed the advised method to collect all values from a parallel > vector into a vector on the 0th processor: > > VecScatterCreateToZero(x, &scatter, &y); > VecScatterBegin(scatter,x,y,INSERT_VALUES,SCATTER_FORWARD); > VecScatterEnd(scatter,x,y,INSERT_VALUES,SCATTER_FORWARD); > VecScatterDestroy(scatter); > > and then use > Field **z; > DAVecGetArray(da, y, &z); > > and > PetscViewerASCIIOpen(comm,"./result.txt",&viewer); > PetscViewerASCIIPrintf(viewer, "%G ", &z[j][i].u ); > > to finish. But I always either meet error for the line "DAVecGetArray(da, > y, &z);" or save a file of all extremely small values(e.g. 8.652e-317), > which is obviously wrong. So what is the correct way to do this? Where can I > find a similar example to follow? > What error? This is correct way to do it, if you do not just use VecView(). Matt > I also wonder if "DAGetGlobalVector( )" will do an equal job to collect > parallel values? > > > Thank you. > > > Best Regards, > > > Zhisong Li From knepley at gmail.com Wed May 5 13:21:43 2010 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 5 May 2010 13:21:43 -0500 Subject: [petsc-users] save a vector value of multiple DOF to a file In-Reply-To: <88D7E3BB7E1960428303E76010037451019C9D@BL2PRD0103MB055.prod.exchangelabs.com> References: <88D7E3BB7E1960428303E76010037451019C9D@BL2PRD0103MB055.prod.exchangelabs.com> Message-ID: On Wed, May 5, 2010 at 1:17 PM, Li, Zhisong (lizs) wrote: > Hi, > > I temporarily remove all the file opening/writing operations and found that > it's probably "DAVecGetArray()" that's causing the trouble when run on 2 > processors. It looks okay when in serial processing. > > I don't use VecView() at all, as X-display doesn't function well for > parallel processing. > VecView() has nothing to do with X. It is the generic output function for Vecs. Please consult the manual. > I also tried to add these lines to first map the global vector xx to a > newly created global vector x , before I scatter x to y: > > DACreateNaturalVector(da, &x); > DAGlobalToNaturalBegin(da, xx,INSERT_VALUES, x); > DAGlobalToNaturalEnd(da, xx,INSERT_VALUES, x); > > But this doesn't help. > > The error message: > > [1]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [1]PETSC ERROR: Arguments are incompatible! > [1]PETSC ERROR: Vector local size 0 is not compatible with DA local sizes > 4284 4437 > ! > This is your problem. The Vec you pass into DAVecGetArray() does not actually come from the DA at all, and in fact has size 0. You can get properly sized Vecs using DAGetGlobalVector(). What Vec are you trying to access (since it obviously cannot have valid values in it)? Matt > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 1, Thu Apr 8 14:16:50 > CDT 2010 > [1]PETSC ERROR: See docs/changes/index.html for recent updates. > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [1]PETSC ERROR: See docs/index.html for manual pages. > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: ./ex7 on a linux-gnu named custard by verhoff Wed May 5 > 14:13:17 2010 > [1]PETSC ERROR: Libraries linked from > /home/verhoff/petsc/linux-gnu-c-debug/lib > [1]PETSC ERROR: Configure run at Tue Apr 27 11:40:02 2010 > [1]PETSC ERROR: Configure options --download-c-blas-lapack=1 --with-cc=gcc > --download-mpich=1 > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: DAVecGetArray() line 53 in src/dm/da/src/dagetarray.c > [1]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [1]PETSC ERROR: Arguments are incompatible! > [1]PETSC ERROR: Vector local size 0 is not compatible with DA local sizes > 4284 4437 > ! > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 1, Thu Apr 8 14:16:50 > CDT 2010 > [1]PETSC ERROR: See docs/changes/index.html for recent updates. > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [1]PETSC ERROR: See docs/index.html for manual pages. > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: ./ex7 on a linux-gnu named custard by verhoff Wed May 5 > 14:13:17 2010 > [1]PETSC ERROR: Libraries linked from > /home/verhoff/petsc/linux-gnu-c-debug/lib > [1]PETSC ERROR: Configure run at Tue Apr 27 11:40:02 2010 > [1]PETSC ERROR: Configure options --download-c-blas-lapack=1 --with-cc=gcc > --download-mpich=1 > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: DAVecRestoreArray() line 108 in src/dm/da/src/dagetarray.c > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Arguments are incompatible! > [0]PETSC ERROR: Vector local size 8721 is not compatible with DA local > sizes 4437 4590 > ! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 1, Thu Apr 8 14:16:50 > CDT 2010 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./ex7 on a linux-gnu named custard by verhoff Wed May 5 > 14:13:17 2010 > [0]PETSC ERROR: Libraries linked from > /home/verhoff/petsc/linux-gnu-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Apr 27 11:40:02 2010 > [0]PETSC ERROR: Configure options --download-c-blas-lapack=1 --with-cc=gcc > --download-mpich=1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: DAVecGetArray() line 53 in src/dm/da/src/dagetarray.c > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Arguments are incompatible! > [0]PETSC ERROR: Vector local size 8721 is not compatible with DA local > sizes 4437 4590 > ! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 1, Thu Apr 8 14:16:50 > CDT 2010 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./ex7 on a linux-gnu named custard by verhoff Wed May 5 > 14:13:17 2010 > [0]PETSC ERROR: Libraries linked from > /home/verhoff/petsc/linux-gnu-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Apr 27 11:40:02 2010 > [0]PETSC ERROR: Configure options --download-c-blas-lapack=1 --with-cc=gcc > --download-mpich=1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: DAVecRestoreArray() line 108 in src/dm/da/src/dagetarray.c > > > Thanks, > > Regards, > > Zhisong Li > > > > > > > > > > Hi, > > > > I want to setup a function in the main function to save the data to a > file. > > For example suppose the vector x in ts/tutorials/ex7 consists of 3 DOF as > > > > typedef struct { > > PetscScalar u, v, p; > > } Field; > > > > I first followed the advised method to collect all values from a parallel > > vector into a vector on the 0th processor: > > > > VecScatterCreateToZero(x, &scatter, &y); > > VecScatterBegin(scatter,x,y,INSERT_VALUES,SCATTER_FORWARD); > > VecScatterEnd(scatter,x,y,INSERT_VALUES,SCATTER_FORWARD); > > VecScatterDestroy(scatter); > > > > and then use > > Field **z; > > DAVecGetArray(da, y, &z); > > > > and > > PetscViewerASCIIOpen(comm,"./result.txt",&viewer); > > PetscViewerASCIIPrintf(viewer, "%G ", &z[j][i].u ); > > > > to finish. But I always either meet error for the line "DAVecGetArray(da, > > y, &z);" or save a file of all extremely small values(e.g. 8.652e-317), > > which is obviously wrong. So what is the correct way to do this? Where > can I > > find a similar example to follow? > > > > What error? This is correct way to do it, if you do not just use VecView(). > > Matt > > > > I also wonder if "DAGetGlobalVector( )" will do an equal job to collect > > parallel values? > > > > > > Thank you. > > > > > > Best Regards, > > > > > > Zhisong Li > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed May 5 13:40:24 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 5 May 2010 13:40:24 -0500 Subject: [petsc-users] about snes In-Reply-To: References: <201005052302363263062@yahoo.cn> Message-ID: <6F86103E-9659-4B92-82C1-39C46B2885F4@mcs.anl.gov> On May 5, 2010, at 10:08 AM, Matthew Knepley wrote: > Try using -snes_mf which will approximate the Jacobian automatically. You can also use -snes_type test -snes_test_display (for a small problem). Also try runs with -snes_monitor -ksp_monitor -ksp_converged_reason -snes_converged_reason and try also -info for lots of information about the line search. You can also try running with -pc_type lu to eliminate any chance that failure of the linear solver is the problem. Barry > > Matt > > 2010/5/5 ????Leping Chen? > petsc teams, > > When I use SNES, I find my program iterated many times but x vector and residual vector > > remained unchanged, that is to say,no convergence.I used finite difference jacobian approximations, > > but I cannot confirm the jacobian matrix structure is right.Whether the matrix structure will influence > > the convergence of SNES or not? Why does my x vector and residual vector remained unchanged? > > call FormJacobian(snes,x,J,J,flag,dummy,ierr) > call MatGetColoring(J,MATCOLORING_SL,iscoloring,ierr) > call MatFDColoringCreate(J,iscoloring,fdcoloring,ierr) > call ISColoringDestroy(iscoloring,ierr) > call MatFDColoringSetFromOptions(fdcoloring,ierr) > call MatFDColoringSetFunction(fdcoloring,FormFunction, > & PETSC_NULL_OBJECT,ierr) > call SNESSetJacobian(snes,J,J,SNESDefaultComputeJacobianColor, > & fdcoloring,ierr) > call SNESGetKSP(snes,ksp,ierr) > call KSPGetPC(ksp,pc,ierr) > call SNESSolve(snes,PETSC_NULL_OBJECT,petscu,ierr) > Thanks, > leping > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From lizs at mail.uc.edu Wed May 5 20:46:47 2010 From: lizs at mail.uc.edu (Li, Zhisong (lizs)) Date: Thu, 6 May 2010 01:46:47 +0000 Subject: [petsc-users] DAE user-defined Jacobian based on discretized variables Message-ID: <88D7E3BB7E1960428303E76010037451019D52@BL2PRD0103MB055.prod.exchangelabs.com> Hello, PETSc Team, >From previous discussion with you on DAE solver, for some 2D/3D PDE equations (such as N-S equation), we can first discretized them before computing the Jacobian, if my understanding is correct. Suppose we can derive the 3x3 Jacobian matrix elements: ddF1/ddU, ddF2/ddV, ...... as J[0][0] = 0.5*(x[j][i+1].u-x[j][i-1].u)+...;... J[0][1] = 0.5*(x[j+1][i].v-x[j-1][i].v)+..., J[0][2]=..., ..., J[2][2] = ..... The resulting Jacobian matrix A depends on the discretized variables x[][].u, x[][].v, ...or say, the location index [j][i]. That's A=A[j][i]. In the function of TSSetIJacobian(), the FormJacobian routine can only return one matrix, rather than an array of matrices (one for each grid point) as we need. My question is that how to construct the correct Jacobian in this case? Thank you very much. Regards, Zhisong Li -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed May 5 21:16:12 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 5 May 2010 21:16:12 -0500 Subject: [petsc-users] DAE user-defined Jacobian based on discretized variables In-Reply-To: <88D7E3BB7E1960428303E76010037451019D52@BL2PRD0103MB055.prod.exchangelabs.com> References: <88D7E3BB7E1960428303E76010037451019D52@BL2PRD0103MB055.prod.exchangelabs.com> Message-ID: On May 5, 2010, at 8:46 PM, Li, Zhisong (lizs) wrote: > Hello, PETSc Team, > > >From previous discussion with you on DAE solver, for some 2D/3D PDE equations (such as N-S equation), we can first discretized them before computing the Jacobian, if my understanding is correct. > > Suppose we can derive the 3x3 Jacobian matrix elements: ddF1/ddU, ddF2/ddV, ...... as > > J[0][0] = 0.5*(x[j][i+1].u-x[j][i-1].u)+...;... J[0][1] = 0.5*(x[j+1][i].v-x[j-1][i].v)+..., J[0][2]=..., ..., J[2][2] = ..... > > The resulting Jacobian matrix A depends on the discretized variables x[][].u, x[][].v, ...or say, the location index [j][i]. That's A=A[j][i]. In the function of TSSetIJacobian(), the FormJacobian routine can only return one matrix, rather than an array of matrices (one for each grid point) as we need. My question is that how to construct the correct Jacobian in this case? You need to assemble the Jacobian in the usual finite element way. That assembled Jacobian is what is passed to the set Jacobian routines. (see any finite element book on assembling the global stiffness matrix from element stiffness matrices). Barry > > Thank you very much. > > > Regards, > > > Zhisong Li > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lizs at mail.uc.edu Thu May 6 11:47:07 2010 From: lizs at mail.uc.edu (Li, Zhisong (lizs)) Date: Thu, 6 May 2010 16:47:07 +0000 Subject: [petsc-users] save a vector value of multiple DOF to a file Message-ID: <88D7E3BB7E1960428303E76010037451019D7E@BL2PRD0103MB055.prod.exchangelabs.com> Hi, Matt, First let me state the problem again: I use the following code to write a global solution vector x to a file, which is in 2D domain with multiple DOF (u, v, p) as in a typical CFD problem (like snes/ex19.c): VecScatterCreateToZero(x, &scatter, &y); VecScatterBegin(scatter, x, y, INSERT_VALUES,SCATTER_FORWARD); VecScatterEnd(scatter, x, y, INSERT_VALUES,SCATTER_FORWARD); VecScatterDestroy(scatter); DAVecGetArray(da,y,&ww); PetscViewerASCIIOpen(PETSC_COMM_WORLD,"./result.txt",&viewer); PetscPrintf(PETSC_COMM_WORLD," %G\n ", ww[15][17].p); This works correctly for sequential processing. After scattering, I can find both x and y size as 8721 using VecGetSize(). But for parallel processing, DAVecGetArray() produces errors. If I don't use DAVecGetArray(), how can I access the data and write it to a file? I also tried the following code as you suggested: DAGetGlobalVector(da, &x); DAVecGetArray(da, x, &ww); It gives no error for sequential or parallel running, but the value are all 0 (wrong). I think writing data to a file should be a common utility for PETSc, can you just find me a sample code that I can follow? Thank you. Regards, Zhisong Li > >The error message: >> > >[1]PETSC ERROR: --------------------- Error Message > >------------------------------------ > >[1]PETSC ERROR: Arguments are incompatible! > >[1]PETSC ERROR: Vector local size 0 is not compatible with DA local sizes > >4284 4437 > >! >This is your problem. The Vec you pass into DAVecGetArray() does not >actually come from >the DA at all, and in fact has size 0. You can get properly sized Vecs using >DAGetGlobalVector(). >What Vec are you trying to access (since it obviously cannot have valid >values in it)? > Matt From jed at 59A2.org Thu May 6 12:14:27 2010 From: jed at 59A2.org (Jed Brown) Date: Thu, 06 May 2010 19:14:27 +0200 Subject: [petsc-users] save a vector value of multiple DOF to a file In-Reply-To: <88D7E3BB7E1960428303E76010037451019D7E@BL2PRD0103MB055.prod.exchangelabs.com> References: <88D7E3BB7E1960428303E76010037451019D7E@BL2PRD0103MB055.prod.exchangelabs.com> Message-ID: <874oilq7f0.fsf@59A2.org> On Thu, 6 May 2010 16:47:07 +0000, "Li, Zhisong (lizs)" wrote: > > Hi, Matt, > > First let me state the problem again: > > I use the following code to write a global solution vector x to a file, which is in 2D domain with multiple DOF (u, v, p) as in a typical CFD problem (like snes/ex19.c): > > VecScatterCreateToZero(x, &scatter, &y); > VecScatterBegin(scatter, x, y, INSERT_VALUES,SCATTER_FORWARD); > VecScatterEnd(scatter, x, y, INSERT_VALUES,SCATTER_FORWARD); > VecScatterDestroy(scatter); vout - output SEQVEC that is large enough to scatter into on processor 0 and of length zero on all other processors http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Vec/VecScatterCreateToZero.html > DAVecGetArray(da,y,&ww); The DA is still global, the vector is serial and only on one process, it is not compatible with the DA. It is also in the "PETSc ordering", see the user's manual. > I also tried the following code as you suggested: > > DAGetGlobalVector(da, &x); > DAVecGetArray(da, x, &ww); > > It gives no error for sequential or parallel running, but the value are all 0 (wrong). The values are correct, you never put anything in this vector x. > I think writing data to a file should be a common utility for PETSc, can you just find me a sample code that I can follow? See the manual page: http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Vec/VecView.html PetscViewerASCIIOpen(PETSC_COMM_WORLD,"./result.txt",&viewer); VecView(x,viewer); or similar with a binary viewer Jed From rlmackie862 at gmail.com Thu May 6 12:34:02 2010 From: rlmackie862 at gmail.com (Randall Mackie) Date: Thu, 6 May 2010 10:34:02 -0700 Subject: [petsc-users] Bug in DAGetGlobalIndicesF90 ? Message-ID: I just upgraded to version 3.1-p2, and have discovered that a call to DAGetGlobalIndicesF90 that use to work fine is now bombing out with the following message: [7]PETSC ERROR: ------------------------------------------------------------------------ [7]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[7]PETSCERROR: or try http://valgrind.org on GNU/ linux and Apple Mac OS X to find memory corruption errors [7]PETSC ERROR: likely location of problem given in stack below [7]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [7]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [7]PETSC ERROR: INSTEAD the line number of the start of the function [7]PETSC ERROR: is given. [7]PETSC ERROR: [7] F90Array1dCreate line 52 src/sys/f90-src/f90_cwrap.c [7]PETSC ERROR: --------------------- Error Message ------------------------------------ [7]PETSC ERROR: Signal received! Calls to DAGetGlobalIndices (the old non-F90 version) seem to work fine. The call to DAGetGlobalIndicesF90 worked fine in the previous version I was using (3.0.0-p7). Can you take a look and see if you see some problem? Thanks, Randy Mackie -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu May 6 12:48:13 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 6 May 2010 12:48:13 -0500 Subject: [petsc-users] Bug in DAGetGlobalIndicesF90 ? In-Reply-To: References: Message-ID: Switching this message to petsc-maint at mcs.anl.gov Does VecGetArrayF90() work correctly? Please send configure.log to petsc-maint at mcs.anl.gov Barry On May 6, 2010, at 12:34 PM, Randall Mackie wrote: > I just upgraded to version 3.1-p2, and have discovered that a call to DAGetGlobalIndicesF90 that use to work > fine is now bombing out with the following message: > > [7]PETSC ERROR: ------------------------------------------------------------------------ > [7]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[7]PETSC ERROR: or try http://valgrind.org on GNU/ > linux and Apple Mac OS X to find memory corruption errors > [7]PETSC ERROR: likely location of problem given in stack below > [7]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [7]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [7]PETSC ERROR: INSTEAD the line number of the start of the function > [7]PETSC ERROR: is given. > [7]PETSC ERROR: [7] F90Array1dCreate line 52 src/sys/f90-src/f90_cwrap.c > [7]PETSC ERROR: --------------------- Error Message ------------------------------------ > [7]PETSC ERROR: Signal received! > > > Calls to DAGetGlobalIndices (the old non-F90 version) seem to work fine. The call to DAGetGlobalIndicesF90 > worked fine in the previous version I was using (3.0.0-p7). > > > Can you take a look and see if you see some problem? > > Thanks, Randy Mackie -------------- next part -------------- An HTML attachment was scrubbed... URL: From rlmackie862 at gmail.com Thu May 6 13:00:24 2010 From: rlmackie862 at gmail.com (Randall Mackie) Date: Thu, 6 May 2010 11:00:24 -0700 Subject: [petsc-users] Bug in DAGetGlobalIndicesF90 ? In-Reply-To: References: Message-ID: Barry, Calls to VecGetArrayF90 also produce errors: [7]PETSC ERROR: ------------------------------------------------------------------------ [7]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[7]PETSCERROR: or try http://valgrind.org on GNU/ linux and Apple Mac OS X to find memory corruption errors [7]PETSC ERROR: likely location of problem given in stack below [7]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [55]PETSC ERROR: ------------------------------------------------------------------------ [55]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [55]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [55]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[55]PETSCERROR: or try http://valgrind.org on GN U/linux and Apple Mac OS X to find memory corruption errors [55]PETSC ERROR: likely location of problem given in stack below The configure log was sent in a separate email. Randy Mackie On Thu, May 6, 2010 at 10:48 AM, Barry Smith wrote: > > Switching this message to petsc-maint at mcs.anl.gov > > Does VecGetArrayF90() work correctly? Please send configure.log to > petsc-maint at mcs.anl.gov > > Barry > > On May 6, 2010, at 12:34 PM, Randall Mackie wrote: > > I just upgraded to version 3.1-p2, and have discovered that a call to > DAGetGlobalIndicesF90 that use to work > fine is now bombing out with the following message: > > [7]PETSC ERROR: > ------------------------------------------------------------------------ > [7]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [7]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[7]PETSCERROR: or try > http://valgrind.org on GNU/ > linux and Apple Mac OS X to find memory corruption errors > [7]PETSC ERROR: likely location of problem given in stack below > [7]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [7]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [7]PETSC ERROR: INSTEAD the line number of the start of the function > [7]PETSC ERROR: is given. > [7]PETSC ERROR: [7] F90Array1dCreate line 52 src/sys/f90-src/f90_cwrap.c > [7]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [7]PETSC ERROR: Signal received! > > > Calls to DAGetGlobalIndices (the old non-F90 version) seem to work fine. > The call to DAGetGlobalIndicesF90 > worked fine in the previous version I was using (3.0.0-p7). > > > Can you take a look and see if you see some problem? > > Thanks, Randy Mackie > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rlmackie862 at gmail.com Thu May 6 13:19:49 2010 From: rlmackie862 at gmail.com (Randall Mackie) Date: Thu, 06 May 2010 11:19:49 -0700 Subject: [petsc-users] Question about Fortran Kernels In-Reply-To: References: Message-ID: <4BE30845.20200@gmail.com> I have another question about the Fortran Kernels: If you do ./configure --help, it says: --with-fortran-kernels= And so I've been using generic for some time. After upgrading to PETSc version 3.1-p2 (or trying to upgrade), I saw a message at the end of a test run that said for complex numbers, I should use --with-fortran-kernels=1 for better performance. But setting that to 1 was not mentioned with the --help flag, and it doesn't explain what the bgl refers to either. Can someone please clarify the options with fortran kernels? Thanks, Randy From bsmith at mcs.anl.gov Thu May 6 13:25:43 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 6 May 2010 13:25:43 -0500 Subject: [petsc-users] Question about Fortran Kernels In-Reply-To: <4BE30845.20200@gmail.com> References: <4BE30845.20200@gmail.com> Message-ID: On May 6, 2010, at 1:19 PM, Randall Mackie wrote: > I have another question about the Fortran Kernels: > > If you do ./configure --help, it says: > > --with-fortran-kernels= Hmm, are you sure this is with 3.1-p2 ? Now it should print -with-fortran-kernels= meaning 0 for off and 1 for on. Maybe the next patch has the correct message. > > And so I've been using generic for some time. > > > After upgrading to PETSc version 3.1-p2 (or trying to upgrade), I saw > a message at the end of a test run that said for complex numbers, > I should use --with-fortran-kernels=1 for better performance. > > But setting that to 1 was not mentioned with the --help flag, and it > doesn't explain what the bgl refers to either. That is outdated and shouldn't be mentioned. You can use either 0 or 1. Barry > > Can someone please clarify the options with fortran kernels? > > Thanks, Randy > From rlmackie862 at gmail.com Thu May 6 13:29:32 2010 From: rlmackie862 at gmail.com (Randall Mackie) Date: Thu, 06 May 2010 11:29:32 -0700 Subject: [petsc-users] Question about Fortran Kernels In-Reply-To: References: <4BE30845.20200@gmail.com> Message-ID: <4BE30A8C.1030802@gmail.com> You are correct, the current version (3.1-p2) returns --with-fortran-kernels= Didn't realize this had changed between the older and newer versions. Randy On 5/6/2010 11:25 AM, Barry Smith wrote: > > On May 6, 2010, at 1:19 PM, Randall Mackie wrote: > >> I have another question about the Fortran Kernels: >> >> If you do ./configure --help, it says: >> >> --with-fortran-kernels= > > Hmm, are you sure this is with 3.1-p2 ? Now it should print -with-fortran-kernels= meaning 0 for off and 1 for on. Maybe the next patch has the correct message. > >> >> And so I've been using generic for some time. >> >> >> After upgrading to PETSc version 3.1-p2 (or trying to upgrade), I saw >> a message at the end of a test run that said for complex numbers, >> I should use --with-fortran-kernels=1 for better performance. >> >> But setting that to 1 was not mentioned with the --help flag, and it >> doesn't explain what the bgl refers to either. > > That is outdated and shouldn't be mentioned. You can use either 0 or 1. > > Barry > >> >> Can someone please clarify the options with fortran kernels? >> >> Thanks, Randy >> > From lizs at mail.uc.edu Thu May 6 13:33:01 2010 From: lizs at mail.uc.edu (Li, Zhisong (lizs)) Date: Thu, 6 May 2010 18:33:01 +0000 Subject: [petsc-users] DAE user-defined Jacobian based on discretized variables Message-ID: <88D7E3BB7E1960428303E76010037451019D97@BL2PRD0103MB055.prod.exchangelabs.com> Hello, Barry, Thank you for your reply. It's very helpful. I took some FEM courses before. For actual implementation, I think we still need to know the solution variable order corresponding to the global stiffness matrix. Here is a two-dimensional 3 DOF problem. We specify 3 equations for each grid point: for (j = yints; j < yinte; j++) { for (i = xints; i < xinte; i++) { f[j][i].u =.......; f[j][i].v =......; f[j][i].p =....; } } f is actually a 3-dimensional array. How can we know its order? Whether it's [f[0][0].u, f[0][1].u, ... f[9][9].u; f[0][0].v, f[0][1].v,...f[9][9].v; f[0][0].p, f[0][1].p,...f[9][9].p] or [f[0][0].u, f[0][0].v, f[0][0].p; f[0][1].u, f[0][1].v, f[0][1].p; ..... ] or any other order? What's the default ordering in PETSc for this case? Based on the i, j loop priority? We need to know these info in order to assemble the stiffness matrix. Thank you. Zhisong Li On May 5, 2010, at 8:46 PM, Li, Zhisong (lizs) wrote: You need to assemble the Jacobian in the usual finite element way. That assembled Jacobian is what is passed to the set Jacobian routines. (see any finite element book on assembling the global stiffness matrix from element stiffness matrices). Barry From bsmith at mcs.anl.gov Thu May 6 13:37:16 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 6 May 2010 13:37:16 -0500 Subject: [petsc-users] DAE user-defined Jacobian based on discretized variables In-Reply-To: <88D7E3BB7E1960428303E76010037451019D97@BL2PRD0103MB055.prod.exchangelabs.com> References: <88D7E3BB7E1960428303E76010037451019D97@BL2PRD0103MB055.prod.exchangelabs.com> Message-ID: On May 6, 2010, at 1:33 PM, Li, Zhisong (lizs) wrote: > Hello, Barry, > > Thank you for your reply. It's very helpful. > > I took some FEM courses before. For actual implementation, I think we still need to know the solution variable order corresponding to the global stiffness matrix. Here is a two-dimensional 3 DOF problem. We specify 3 equations for each grid point: > > for (j = yints; j < yinte; j++) { > for (i = xints; i < xinte; i++) { > f[j][i].u =.......; > f[j][i].v =......; > f[j][i].p =....; } } > > f is actually a 3-dimensional array. How can we know its order? Whether it's > > [f[0][0].u, f[0][1].u, ... f[9][9].u; f[0][0].v, f[0][1].v,...f[9][9].v; f[0][0].p, f[0][1].p,...f[9][9].p] > > or > > [f[0][0].u, f[0][0].v, f[0][0].p; f[0][1].u, f[0][1].v, f[0][1].p; ..... ] > > or any other order? > > What's the default ordering in PETSc for this case? Based on the i, j loop priority? We need to know these info in order to assemble the stiffness matrix. That is totally up to you. So long as you use the same ordering everywhere you can use any that you want. We highly recommend the second ordering where the u,v,p are stored interlaced. Barry > > > Thank you. > > > Zhisong Li > > > > > > On May 5, 2010, at 8:46 PM, Li, Zhisong (lizs) wrote: > > > > You need to assemble the Jacobian in the usual finite element way. That assembled Jacobian is what is passed to the set Jacobian routines. (see any finite element book on assembling the global stiffness matrix from element stiffness matrices). > > > Barry > From mishrabinit at gmail.com Fri May 7 04:33:39 2010 From: mishrabinit at gmail.com (binitranjan mishra) Date: Fri, 7 May 2010 15:03:39 +0530 Subject: [petsc-users] Parallel SPMV code Message-ID: Hi all I have to actually time the performance of petsc's SPMV kernel, that is the MatMult() function. This is the basic code I have edited for the same. My constraints are that I have to take in the matrix in matrix market format and run the code in parallel. Hence the code. However, I am facing a few problems : 1. The executable is taking a very very long time to run. Is this okay? 2. The timings for different processors is very unbalanced. I am not sure if noting down the longest time in this case would be fair. 3. As of now, I have entered a fixed integer = 10 for all vector values. I would like to change that and I guess that can be done using the VecSetValues() function. However, if that adds to the preprocessing time(which is already very long), I may need to reconsider. Looking forward to a quick reply. Thank you very much Regards Binit http://sites.google.com/site/mishrabinit "I am merely a figment of my imagination" main(int in,char *inn[] ) {((in=='+'-'+'+'/'/'/')&& (inn['+'-'+']=")(Cjoju\ !S/!Njtisb!\niuuq;00tjuft/hpphmf/dpn0tjuf0njtisbcjoju\n#Ju!jt!tjnqmf!cvu!opu!boz!tjnqmfs#!!);" )&&main('/'/'/'+'/'/'/', inn))||(!!in&&(inn[ '-'-'-'][in]==0x29)&& !!'(' &&putchar(012))|| ('/'/'/'&& putchar(inn['-'-'-' ][ in]-'/'/'/'+'+'-'+')&&main(in+'/'/'/'+'+'-'+',inn));} -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex72.c Type: text/x-c Size: 3388 bytes Desc: not available URL: From bsmith at mcs.anl.gov Fri May 7 07:37:21 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 7 May 2010 07:37:21 -0500 Subject: [petsc-users] Parallel SPMV code In-Reply-To: References: Message-ID: <793987ED-4C87-4518-84B6-BA429882254C@mcs.anl.gov> On May 7, 2010, at 4:33 AM, binitranjan mishra wrote: > Hi all > > I have to actually time the performance of petsc's SPMV kernel, that is the MatMult() function. This is the basic code I have edited for the same. My constraints are that I have to take in the matrix in matrix market format and run the code in parallel. Hence the code. If you remain in those constraints it will take forever to load. You should run the program on one process that reads in the ASCII file and then save it with the binary viewer. Then your real code will run in parallel and use MatLoad() to quickly load in the matrix in parallel. > > However, I am facing a few problems : > 1. The executable is taking a very very long time to run. Is this okay? > 2. The timings for different processors is very unbalanced. I am not sure if noting down the longest time in this case would be fair. Put a MPI_Barrier() right before you start the timer. > 3. As of now, I have entered a fixed integer = 10 for all vector values It doesn't matter this won't change the speed. > . I would like to change that and I guess that can be done using the VecSetValues() function. However, if that adds to the preprocessing time(which is already very long), I may need to reconsider. > > Looking forward to a quick reply. Thank you very much > > Regards > Binit > http://sites.google.com/site/mishrabinit > "I am merely a figment of my imagination" > > main(int in,char *inn[] ) {((in=='+'-'+'+'/'/'/')&& (inn['+'-'+']=")(Cjoju\ > !S/!Njtisb!\niuuq;00tjuft/hpphmf/dpn0tjuf0njtisbcjoju\n#Ju!jt!tjnqmf!cvu!opu!boz!tjnqmfs#!!);" )&&main('/'/'/'+'/'/'/', > inn))||(!!in&&(inn[ '-'-'-'][in]==0x29)&& !!'(' &&putchar(012))|| ('/'/'/'&& > putchar(inn['-'-'-' ][ in]-'/'/'/'+'+'-'+')&&main(in+'/'/'/'+'+'-'+',inn));} > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Z.Sheng at tudelft.nl Fri May 7 10:16:41 2010 From: Z.Sheng at tudelft.nl (Zhifeng Sheng - EWI) Date: Fri, 7 May 2010 17:16:41 +0200 Subject: [petsc-users] Compute economic singular value decomposition for sparse matrix? References: <88D7E3BB7E1960428303E76010037451019D97@BL2PRD0103MB055.prod.exchangelabs.com> Message-ID: <947C04CD618D16429440FED56EAE47BA06B39C@SRV564.tudelft.net> Dear all I need to approximate a matrix with its economic singular value decomposition. Is there any function in Petsc which does this? If not, does anyone have a suggestion on which package I should look into? Thanks and best regards Zhifeng -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Fri May 7 10:20:22 2010 From: jed at 59A2.org (Jed Brown) Date: Fri, 07 May 2010 17:20:22 +0200 Subject: [petsc-users] Compute economic singular value decomposition for sparse matrix? In-Reply-To: <947C04CD618D16429440FED56EAE47BA06B39C@SRV564.tudelft.net> References: <88D7E3BB7E1960428303E76010037451019D97@BL2PRD0103MB055.prod.exchangelabs.com> <947C04CD618D16429440FED56EAE47BA06B39C@SRV564.tudelft.net> Message-ID: <877hnfpwll.fsf@59A2.org> On Fri, 7 May 2010 17:16:41 +0200, "Zhifeng Sheng - EWI" wrote: > Dear all > > I need to approximate a matrix with its economic singular value decomposition. Is there any function in Petsc which does this? > > If not, does anyone have a suggestion on which package I should look into? http://www.grycap.upv.es/slepc/ From jed at 59A2.org Fri May 7 11:36:53 2010 From: jed at 59A2.org (Jed Brown) Date: Fri, 07 May 2010 18:36:53 +0200 Subject: [petsc-users] Compute economic singular value decomposition for sparse matrix? In-Reply-To: <947C04CD618D16429440FED56EAE47BA06B39D@SRV564.tudelft.net> References: <88D7E3BB7E1960428303E76010037451019D97@BL2PRD0103MB055.prod.exchangelabs.com> <947C04CD618D16429440FED56EAE47BA06B39C@SRV564.tudelft.net> <877hnfpwll.fsf@59A2.org> <947C04CD618D16429440FED56EAE47BA06B39D@SRV564.tudelft.net> Message-ID: <871vdnpt22.fsf@59A2.org> On Fri, 7 May 2010 18:30:23 +0200, "Zhifeng Sheng - EWI" wrote: > > > Hi, > > This library does not work with visual studio, does it? I tried to install it but not yet successful. I don't use Windows so I can't (personally) confirm whether it does or does not work. It uses the same configure and build system as PETSc so it should work anywhere that PETSc does. Jed From torres.pedrozpk at gmail.com Fri May 7 13:06:14 2010 From: torres.pedrozpk at gmail.com (Pedro Torres) Date: Fri, 7 May 2010 15:06:14 -0300 Subject: [petsc-users] speedup on KSPSetUp() using qmd reordering Message-ID: Hello, I have a sparse linear system, block dense, one block for each process, and solve with cg and block jacobi (ICC(0)) as preconditioners, with different reordering. During my tests I found quasi linear speed on kspsetup() and KSPSetUpOnBlocks() using 1wd,nd,rcm reorderings, but using qmd I get superlinear speedup. My CPU is a Xeon 5410. Is that possible or something is going wrong?, and if I want to explain that, what functions on this stage should I monitor??. I really appreciate any advice. Thanks in advance. -- Pedro Torres GESAR/UERJ Rua Fonseca Teles 121, S?o Crist?v?o Rio de Janeiro - Brasil -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri May 7 13:11:33 2010 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 7 May 2010 14:11:33 -0400 Subject: [petsc-users] speedup on KSPSetUp() using qmd reordering In-Reply-To: References: Message-ID: On Fri, May 7, 2010 at 2:06 PM, Pedro Torres wrote: > Hello, > > I have a sparse linear system, block dense, one block for each process, and > solve with cg and block jacobi (ICC(0)) as preconditioners, with different > reordering. During my tests I found quasi linear speed on kspsetup() > and KSPSetUpOnBlocks() using 1wd,nd,rcm reorderings, but using qmd I get > superlinear speedup. My CPU is a Xeon 5410. > > Is that possible or something is going wrong?, and if I want to explain > that, what functions on this stage should I monitor??. I really > appreciate any advice. > It is possible (but unlikely) that the ordering on a larger number of processes creates a better preconditioner, particularly because ICC(0) is so unpredictable. You can try and separate arithmetic efficiency from algorithmic efficiency by looking at the number of iterates between these runs. Matt > Thanks in advance. > > -- > Pedro Torres > GESAR/UERJ > Rua Fonseca Teles 121, S?o Crist?v?o > Rio de Janeiro - Brasil > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From torres.pedrozpk at gmail.com Fri May 7 14:32:22 2010 From: torres.pedrozpk at gmail.com (Pedro Torres) Date: Fri, 7 May 2010 16:32:22 -0300 Subject: [petsc-users] speedup on KSPSetUp() using qmd reordering In-Reply-To: References: Message-ID: 2010/5/7 Matthew Knepley > On Fri, May 7, 2010 at 2:06 PM, Pedro Torres wrote: > >> Hello, >> >> I have a sparse linear system, block dense, one block for each process, >> and solve with cg and block jacobi (ICC(0)) as preconditioners, with >> different reordering. During my tests I found quasi linear speed on >> kspsetup() and KSPSetUpOnBlocks() using 1wd,nd,rcm reorderings, but using >> qmd I get superlinear speedup. My CPU is a Xeon 5410. >> >> Is that possible or something is going wrong?, and if I want to explain >> that, what functions on this stage should I monitor??. I really >> appreciate any advice. >> > > It is possible (but unlikely) that the ordering on a larger number of > processes creates a better > preconditioner, particularly because ICC(0) is so unpredictable. You can > try and separate > arithmetic efficiency from algorithmic efficiency by looking at the number > of iterates between these > runs. > Thanks for a swift response. I found that given an ordering, the number of iterations is almost uniform,with just a little increase, but 1wd have the minimun numbers of iterations. May be I should look in the ratio of cache miss of MatCholeskyFactorNumeric to explain this behavior, rigth?. Thanks a lot! > Matt > > >> Thanks in advance. >> >> -- >> Pedro Torres >> GESAR/UERJ >> Rua Fonseca Teles 121, S?o Crist?v?o >> Rio de Janeiro - Brasil >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri May 7 14:37:00 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 7 May 2010 14:37:00 -0500 Subject: [petsc-users] speedup on KSPSetUp() using qmd reordering In-Reply-To: References: Message-ID: <7CFC1CAC-C315-4E51-9C21-93A88436ED47@mcs.anl.gov> Send the output from -log_summary to petsc-maint at mcs.anl.gov on the various approaches and we'll comment on it. Barry On May 7, 2010, at 2:32 PM, Pedro Torres wrote: > > > 2010/5/7 Matthew Knepley > On Fri, May 7, 2010 at 2:06 PM, Pedro Torres wrote: > Hello, > > I have a sparse linear system, block dense, one block for each process, and solve with cg and block jacobi (ICC(0)) as preconditioners, with different reordering. During my tests I found quasi linear speed on kspsetup() and KSPSetUpOnBlocks() using 1wd,nd,rcm reorderings, but using qmd I get superlinear speedup. My CPU is a Xeon 5410. > > Is that possible or something is going wrong?, and if I want to explain that, what functions on this stage should I monitor??. I really > appreciate any advice. > > It is possible (but unlikely) that the ordering on a larger number of processes creates a better > preconditioner, particularly because ICC(0) is so unpredictable. You can try and separate > arithmetic efficiency from algorithmic efficiency by looking at the number of iterates between these > runs. > Thanks for a swift response. I found that given an ordering, the number of iterations is almost uniform,with just a little increase, but 1wd have the minimun numbers of iterations. May be I should look > in the ratio of cache miss of MatCholeskyFactorNumeric to explain this behavior, rigth?. > Thanks a lot! > Matt > > Thanks in advance. > > -- > Pedro Torres > GESAR/UERJ > Rua Fonseca Teles 121, S?o Crist?v?o > Rio de Janeiro - Brasil > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From irfan.khan at gatech.edu Fri May 7 14:57:03 2010 From: irfan.khan at gatech.edu (irfan.khan at gatech.edu) Date: Fri, 7 May 2010 15:57:03 -0400 (EDT) Subject: [petsc-users] Adding MatAij rows In-Reply-To: <538256097.687721273262108501.JavaMail.root@mail8.gatech.edu> Message-ID: <29690493.688181273262223164.JavaMail.root@mail8.gatech.edu> Petsc Team, Is there a function that can be used to add two rows and put the results into a third row of the same parallel matrix (MPIAIJ)? i.e A[k][l] = A[i][l] + A[j][l] Thank you Irfan -- PhD Candidate G.W. Woodruff School of Mechanical Engineering Georgia Institute of Technology Atlanta, GA (30332) -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri May 7 15:03:42 2010 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 7 May 2010 16:03:42 -0400 Subject: [petsc-users] Adding MatAij rows In-Reply-To: <29690493.688181273262223164.JavaMail.root@mail8.gatech.edu> References: <538256097.687721273262108501.JavaMail.root@mail8.gatech.edu> <29690493.688181273262223164.JavaMail.root@mail8.gatech.edu> Message-ID: On Fri, May 7, 2010 at 3:57 PM, wrote: > Petsc Team, > > Is there a function that can be used to add two rows and put the results > into a third row of the same parallel matrix (MPIAIJ)? i.e > > A[k][l] = A[i][l] + A[j][l] > No, since it would entail messing with the nonzero structure. You could easily use a second matrix to get this effect from subsequent MatMult() calls. Matt > Thank you > Irfan > > -- > PhD Candidate > G.W. Woodruff School of Mechanical Engineering > Georgia Institute of Technology > Atlanta, GA (30332) > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmitrey15 at ukr.net Sun May 9 09:42:39 2010 From: dmitrey15 at ukr.net (Dmitrey) Date: Sun, 09 May 2010 17:42:39 +0300 Subject: [petsc-users] any ways to run PETSc4py on several CPU? Message-ID: <4BE6C9DF.3080807@ukr.net> hi all, are there any ways to run PETSc4py on several CPU, i.e. something like mpirun -np 4? Currently I have >>> print PETSc.COMM_WORLD.Get_size() 1 Thank you in advance, D. From knepley at gmail.com Mon May 10 07:06:46 2010 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 10 May 2010 08:06:46 -0400 Subject: [petsc-users] any ways to run PETSc4py on several CPU? In-Reply-To: <4BE6C9DF.3080807@ukr.net> References: <4BE6C9DF.3080807@ukr.net> Message-ID: You use mpirun. Matt On Sun, May 9, 2010 at 10:42 AM, Dmitrey wrote: > hi all, > are there any ways to run PETSc4py on several CPU, i.e. something like > mpirun -np 4? > > Currently I have > > >>> print PETSc.COMM_WORLD.Get_size() > 1 > > Thank you in advance, D. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon May 10 08:32:19 2010 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 10 May 2010 08:32:19 -0500 (CDT) Subject: [petsc-users] any ways to run PETSc4py on several CPU? In-Reply-To: References: <4BE6C9DF.3080807@ukr.net> Message-ID: Note: this might not work with some older MPI impls like mpich1. However newer ones like mpich2 should work. mpiexec -n 4 python foo.py Satish On Mon, 10 May 2010, Matthew Knepley wrote: > You use mpirun. > > Matt > > On Sun, May 9, 2010 at 10:42 AM, Dmitrey wrote: > > > hi all, > > are there any ways to run PETSc4py on several CPU, i.e. something like > > mpirun -np 4? > > > > Currently I have > > > > >>> print PETSc.COMM_WORLD.Get_size() > > 1 > > > > Thank you in advance, D. > > > > > > > From irfan.khan at gatech.edu Mon May 10 11:07:06 2010 From: irfan.khan at gatech.edu (irfan.khan at gatech.edu) Date: Mon, 10 May 2010 12:07:06 -0400 (EDT) Subject: [petsc-users] MAT_REUSE_MATRIX In-Reply-To: <1900423573.1029011273507437069.JavaMail.root@mail8.gatech.edu> Message-ID: <652484597.1030151273507626129.JavaMail.root@mail8.gatech.edu> Hi Petsc Team, When using the subroutine MatMatMult(Mat A,Mat B,PetscReal fill,Mat *C) with option MAT_REUSE_MATRIX, Can the C matrix be any one of A or B? Thank you Irfan ----- Original Message ----- From: "Matthew Knepley" To: "PETSc users list" Sent: Friday, May 7, 2010 4:03:42 PM GMT -05:00 US/Canada Eastern Subject: Re: [petsc-users] Adding MatAij rows On Fri, May 7, 2010 at 3:57 PM, < irfan.khan at gatech.edu > wrote: Petsc Team, Is there a function that can be used to add two rows and put the results into a third row of the same parallel matrix (MPIAIJ)? i.e A[k][l] = A[i][l] + A[j][l] No, since it would entail messing with the nonzero structure. You could easily use a second matrix to get this effect from subsequent MatMult() calls. Matt Thank you Irfan -- PhD Candidate G.W. Woodruff School of Mechanical Engineering Georgia Institute of Technology Atlanta, GA (30332) -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- PhD Candidate G.W. Woodruff School of Mechanical Engineering Georgia Institute of Technology Atlanta, GA (30332) From knepley at gmail.com Mon May 10 11:26:14 2010 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 10 May 2010 12:26:14 -0400 Subject: [petsc-users] MAT_REUSE_MATRIX In-Reply-To: <652484597.1030151273507626129.JavaMail.root@mail8.gatech.edu> References: <1900423573.1029011273507437069.JavaMail.root@mail8.gatech.edu> <652484597.1030151273507626129.JavaMail.root@mail8.gatech.edu> Message-ID: On Mon, May 10, 2010 at 12:07 PM, wrote: > Hi Petsc Team, > > When using the subroutine MatMatMult(Mat A,Mat B,PetscReal fill,Mat *C) > with option MAT_REUSE_MATRIX, Can the C matrix be any one of A or B? > I do not think that is safe. I believe C needs to be another matrix, but it can be reused for successive multiplies. Matt > Thank you > Irfan > > ----- Original Message ----- > From: "Matthew Knepley" > To: "PETSc users list" > Sent: Friday, May 7, 2010 4:03:42 PM GMT -05:00 US/Canada Eastern > Subject: Re: [petsc-users] Adding MatAij rows > > On Fri, May 7, 2010 at 3:57 PM, < irfan.khan at gatech.edu > wrote: > > > > > > > Petsc Team, > > Is there a function that can be used to add two rows and put the results > into a third row of the same parallel matrix (MPIAIJ)? i.e > > A[k][l] = A[i][l] + A[j][l] > > > > No, since it would entail messing with the nonzero structure. You could > easily > use a second matrix to get this effect from subsequent MatMult() calls. > > > Matt > > > > > > Thank you > Irfan > > -- > PhD Candidate > G.W. Woodruff School of Mechanical Engineering > Georgia Institute of Technology > Atlanta, GA (30332) > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- > PhD Candidate > G.W. Woodruff School of Mechanical Engineering > Georgia Institute of Technology > Atlanta, GA (30332) > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From irfan.khan at gatech.edu Mon May 10 12:20:33 2010 From: irfan.khan at gatech.edu (irfan.khan at gatech.edu) Date: Mon, 10 May 2010 13:20:33 -0400 (EDT) Subject: [petsc-users] MAT_REUSE_MATRIX In-Reply-To: <213642763.1050791273511963207.JavaMail.root@mail8.gatech.edu> Message-ID: <1590693802.1051181273512032966.JavaMail.root@mail8.gatech.edu> I get the following error at the call for MatMatMult(Mat A,Mat B,MatReuse scall, PetscReal fill,Mat *C). Error: Container does not exit! Is the probable cause of this error an invalid C matrix?. Can I use MAT_REUSE_MATRIX in MatMatMult() and use an existing C matrix. The call works fine when I use MAT_INITIAL_MATRIX and use a new matrix. Here are the sequence of steps. /* Generating A & B matrices by copying the global stiffness matrix */ ierr = MatConvert(global->Amat, MATSAME, MAT_INITIAL_MATRIX, &A);CHKERRQ(ierr); ierr = MatConvert(global->Amat, MATSAME, MAT_INITIAL_MATRIX, &B);CHKERRQ(ierr); /* Zero all the entries of the A matrix */ ierr = MatZeroEntries(A);CHKERRQ(ierr); /* Fill in the diagonal entry and one off-diagonal entry of A matrix */ for(i = 0; i < bc.numPeriodic; i++){ if(bc.periodNode[i] > -1){ for(j = 0; j < dofn; j++){ row = local2Petsc[bc.periodNode[i]*dofn+j]; col = bc.periodPair[i]*dofn+j; /* The diagonal Entry */ ierr =MatSetValue(A,row,row,1.0,ADD_VALUES);CHKERRQ(ierr); /* The entry for periodic node */ ierr =MatSetValue(A,row,col,1.0,ADD_VALUES);CHKERRQ(ierr); } } } /* Assemble the new matrices */ ierr = MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); ierr = MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); ierr = MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); ierr = MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); /* Perform matrix multiplication of A and B(Amat) to add rows of periodic * nodes. Save the new matrix in Amat */ ierr = MatMatMult(A,B,MAT_REUSE_MATRIX,PETSC_DEFAULT,&global->Amat);CHKERRQ(ierr); ierr = MatDestroy(A);CHKERRQ(ierr); ierr = MatDestroy(B);CHKERRQ(ierr); Thank you Irfan ----- Original Message ----- From: "Matthew Knepley" To: "PETSc users list" Sent: Monday, May 10, 2010 12:26:14 PM GMT -05:00 US/Canada Eastern Subject: Re: [petsc-users] MAT_REUSE_MATRIX On Mon, May 10, 2010 at 12:07 PM, < irfan.khan at gatech.edu > wrote: Hi Petsc Team, When using the subroutine MatMatMult(Mat A,Mat B,PetscReal fill,Mat *C) with option MAT_REUSE_MATRIX, Can the C matrix be any one of A or B? I do not think that is safe. I believe C needs to be another matrix, but it can be reused for successive multiplies. Matt Thank you Irfan ----- Original Message ----- From: "Matthew Knepley" < knepley at gmail.com > To: "PETSc users list" < petsc-users at mcs.anl.gov > Sent: Friday, May 7, 2010 4:03:42 PM GMT -05:00 US/Canada Eastern Subject: Re: [petsc-users] Adding MatAij rows On Fri, May 7, 2010 at 3:57 PM, < irfan.khan at gatech.edu > wrote: Petsc Team, Is there a function that can be used to add two rows and put the results into a third row of the same parallel matrix (MPIAIJ)? i.e A[k][l] = A[i][l] + A[j][l] No, since it would entail messing with the nonzero structure. You could easily use a second matrix to get this effect from subsequent MatMult() calls. Matt Thank you Irfan -- PhD Candidate G.W. Woodruff School of Mechanical Engineering Georgia Institute of Technology Atlanta, GA (30332) -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- PhD Candidate G.W. Woodruff School of Mechanical Engineering Georgia Institute of Technology Atlanta, GA (30332) -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- PhD Candidate G.W. Woodruff School of Mechanical Engineering Georgia Institute of Technology Atlanta, GA (30332) From knepley at gmail.com Mon May 10 13:12:01 2010 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 10 May 2010 14:12:01 -0400 Subject: [petsc-users] MAT_REUSE_MATRIX In-Reply-To: <1590693802.1051181273512032966.JavaMail.root@mail8.gatech.edu> References: <213642763.1050791273511963207.JavaMail.root@mail8.gatech.edu> <1590693802.1051181273512032966.JavaMail.root@mail8.gatech.edu> Message-ID: On Mon, May 10, 2010 at 1:20 PM, wrote: > I get the following error at the call for MatMatMult(Mat A,Mat B,MatReuse > scall, PetscReal fill,Mat *C). > > Error: Container does not exit! > The idea here is that you use MAT_INITIAL_MATRIX the first time, so C will be created, and then the same C and MAT_REUSE_MATRIX the next time. Matt > Is the probable cause of this error an invalid C matrix?. Can I use > MAT_REUSE_MATRIX in MatMatMult() and use an existing C matrix. The call > works fine when I use MAT_INITIAL_MATRIX and use a new matrix. > > Here are the sequence of steps. > > /* Generating A & B matrices by copying the global stiffness matrix > */ > ierr = MatConvert(global->Amat, MATSAME, MAT_INITIAL_MATRIX, > &A);CHKERRQ(ierr); > ierr = MatConvert(global->Amat, MATSAME, MAT_INITIAL_MATRIX, > &B);CHKERRQ(ierr); > > /* Zero all the entries of the A matrix */ > ierr = MatZeroEntries(A);CHKERRQ(ierr); > > /* Fill in the diagonal entry and one off-diagonal entry of A matrix > */ > for(i = 0; i < bc.numPeriodic; i++){ > if(bc.periodNode[i] > -1){ > for(j = 0; j < dofn; j++){ > row = local2Petsc[bc.periodNode[i]*dofn+j]; > col = bc.periodPair[i]*dofn+j; > /* The diagonal Entry */ > ierr > =MatSetValue(A,row,row,1.0,ADD_VALUES);CHKERRQ(ierr); > /* The entry for periodic node */ > ierr > =MatSetValue(A,row,col,1.0,ADD_VALUES);CHKERRQ(ierr); > } > } > } > > /* Assemble the new matrices */ > ierr = MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > > /* Perform matrix multiplication of A and B(Amat) to add rows of > periodic > * nodes. Save the new matrix in Amat */ > ierr = > MatMatMult(A,B,MAT_REUSE_MATRIX,PETSC_DEFAULT,&global->Amat);CHKERRQ(ierr); > > ierr = MatDestroy(A);CHKERRQ(ierr); > ierr = MatDestroy(B);CHKERRQ(ierr); > > > Thank you > Irfan > > ----- Original Message ----- > From: "Matthew Knepley" > To: "PETSc users list" > Sent: Monday, May 10, 2010 12:26:14 PM GMT -05:00 US/Canada Eastern > Subject: Re: [petsc-users] MAT_REUSE_MATRIX > > On Mon, May 10, 2010 at 12:07 PM, < irfan.khan at gatech.edu > wrote: > > > > Hi Petsc Team, > > When using the subroutine MatMatMult(Mat A,Mat B,PetscReal fill,Mat *C) > with option MAT_REUSE_MATRIX, Can the C matrix be any one of A or B? > > > > I do not think that is safe. I believe C needs to be another matrix, but it > can be > reused for successive multiplies. > > > Matt > > > Thank you > Irfan > > ----- Original Message ----- > From: "Matthew Knepley" < knepley at gmail.com > > To: "PETSc users list" < petsc-users at mcs.anl.gov > > Sent: Friday, May 7, 2010 4:03:42 PM GMT -05:00 US/Canada Eastern > Subject: Re: [petsc-users] Adding MatAij rows > > On Fri, May 7, 2010 at 3:57 PM, < irfan.khan at gatech.edu > wrote: > > > > > > > Petsc Team, > > Is there a function that can be used to add two rows and put the results > into a third row of the same parallel matrix (MPIAIJ)? i.e > > A[k][l] = A[i][l] + A[j][l] > > > > No, since it would entail messing with the nonzero structure. You could > easily > use a second matrix to get this effect from subsequent MatMult() calls. > > > Matt > > > > > > Thank you > Irfan > > -- > PhD Candidate > G.W. Woodruff School of Mechanical Engineering > Georgia Institute of Technology > Atlanta, GA (30332) > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- > PhD Candidate > G.W. Woodruff School of Mechanical Engineering > Georgia Institute of Technology > Atlanta, GA (30332) > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- > PhD Candidate > G.W. Woodruff School of Mechanical Engineering > Georgia Institute of Technology > Atlanta, GA (30332) > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From recrusader at gmail.com Mon May 10 16:08:12 2010 From: recrusader at gmail.com (Yujie) Date: Mon, 10 May 2010 16:08:12 -0500 Subject: [petsc-users] Whether can one set the row and col sizes of a Mat to zero when creating it? Message-ID: Dear PETSc developers, Whether can one set the row and col sizes of a Mat to zero when creating it? How about a Vec? If we can, how about the object of Mat and Vec? thanks a lot. Regards, Yujie From bsmith at mcs.anl.gov Mon May 10 16:16:21 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 10 May 2010 16:16:21 -0500 Subject: [petsc-users] Whether can one set the row and col sizes of a Mat to zero when creating it? In-Reply-To: References: Message-ID: <546C917A-98C6-47E7-B529-97F44BE83079@mcs.anl.gov> Yes, Mats and Vecs can have zero sizes. They are just like other Mats and Vecs but with no space allocated for data. Since these are not commonly used they may have glitches, let us know about them and we will fix them. Barry On May 10, 2010, at 4:08 PM, Yujie wrote: > Dear PETSc developers, > > Whether can one set the row and col sizes of a Mat to zero when creating it? > How about a Vec? > > If we can, how about the object of Mat and Vec? thanks a lot. > > Regards, > Yujie From recrusader at gmail.com Mon May 10 16:41:42 2010 From: recrusader at gmail.com (Yujie) Date: Mon, 10 May 2010 16:41:42 -0500 Subject: [petsc-users] Whether can one set the row and col sizes of a Mat to zero when creating it? In-Reply-To: <546C917A-98C6-47E7-B529-97F44BE83079@mcs.anl.gov> References: <546C917A-98C6-47E7-B529-97F44BE83079@mcs.anl.gov> Message-ID: Dear Barry, Thank you for your reply. One can create an array with zero sizes. When he wants to set the values of the array, he can use " for(i=0; i wrote: > > ?Yes, Mats and Vecs can have zero sizes. They are just like other Mats and Vecs but with no space allocated for data. > > ?Since these are not commonly used they may have glitches, let us know about them and we will fix them. > > ? Barry > > On May 10, 2010, at 4:08 PM, Yujie wrote: > >> Dear PETSc developers, >> >> Whether can one set the row and col sizes of a Mat to zero when creating it? >> How about a Vec? >> >> If we can, how about the object of Mat and Vec? thanks a lot. >> >> Regards, >> Yujie > > From bsmith at mcs.anl.gov Mon May 10 16:55:53 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 10 May 2010 16:55:53 -0500 Subject: [petsc-users] Whether can one set the row and col sizes of a Mat to zero when creating it? In-Reply-To: References: <546C917A-98C6-47E7-B529-97F44BE83079@mcs.anl.gov> Message-ID: You cannot call MatVecSetValues() on a zero size array in the same way that you cannot do v[0] = 2 on an array with size 0. PETSc only allows you to call VecMatSetValues() on indices that the Vec Mat has, since an empty Vec Mat has no indices you cannot call SetValues Barry On May 10, 2010, at 4:41 PM, Yujie wrote: > Dear Barry, > > Thank you for your reply. > > One can create an array with zero sizes. When he wants to set the > values of the array, he can use > " > for(i=0; i " > he doesn't need to consider whether the size of the array is zero > since "loop for" can handle it even if the size of the array is zero. > > Whether is the operation done for Mat and Vec? I mean > one can use > " > Mat(Vec)SetValues() > " > regardless of the size of Mat and Vec. > > Thanks again. > > Regards, > Yujie > > On Mon, May 10, 2010 at 4:16 PM, Barry Smith wrote: >> >> Yes, Mats and Vecs can have zero sizes. They are just like other Mats and Vecs but with no space allocated for data. >> >> Since these are not commonly used they may have glitches, let us know about them and we will fix them. >> >> Barry >> >> On May 10, 2010, at 4:08 PM, Yujie wrote: >> >>> Dear PETSc developers, >>> >>> Whether can one set the row and col sizes of a Mat to zero when creating it? >>> How about a Vec? >>> >>> If we can, how about the object of Mat and Vec? thanks a lot. >>> >>> Regards, >>> Yujie >> >> From recrusader at gmail.com Mon May 10 17:00:09 2010 From: recrusader at gmail.com (Yujie) Date: Mon, 10 May 2010 17:00:09 -0500 Subject: [petsc-users] Whether can one set the row and col sizes of a Mat to zero when creating it? In-Reply-To: References: <546C917A-98C6-47E7-B529-97F44BE83079@mcs.anl.gov> Message-ID: I got it. Thank you very much, Barry. Regards, Yujie On Mon, May 10, 2010 at 4:55 PM, Barry Smith wrote: > > ?You cannot call MatVecSetValues() on a zero size array in the same way that you cannot do v[0] = 2 on an array with size 0. > > ?PETSc only allows you to call VecMatSetValues() on indices that the Vec Mat has, since an empty Vec Mat has no indices you cannot call SetValues > > ? Barry > > On May 10, 2010, at 4:41 PM, Yujie wrote: > >> Dear Barry, >> >> Thank you for your reply. >> >> One can create an array with zero sizes. When he wants to set the >> values of the array, he can use >> " >> for(i=0; i> " >> he doesn't need to consider whether the size of the array is zero >> since "loop for" can handle it even if the size of the array is zero. >> >> Whether is the operation done for Mat and Vec? I mean >> one can use >> " >> Mat(Vec)SetValues() >> " >> regardless of the size of Mat and Vec. >> >> Thanks again. >> >> Regards, >> Yujie >> >> On Mon, May 10, 2010 at 4:16 PM, Barry Smith wrote: >>> >>> ?Yes, Mats and Vecs can have zero sizes. They are just like other Mats and Vecs but with no space allocated for data. >>> >>> ?Since these are not commonly used they may have glitches, let us know about them and we will fix them. >>> >>> ? Barry >>> >>> On May 10, 2010, at 4:08 PM, Yujie wrote: >>> >>>> Dear PETSc developers, >>>> >>>> Whether can one set the row and col sizes of a Mat to zero when creating it? >>>> How about a Vec? >>>> >>>> If we can, how about the object of Mat and Vec? thanks a lot. >>>> >>>> Regards, >>>> Yujie >>> >>> > > From recrusader at gmail.com Mon May 10 20:31:25 2010 From: recrusader at gmail.com (Yujie) Date: Mon, 10 May 2010 20:31:25 -0500 Subject: [petsc-users] logical operation for Vec Message-ID: Dear PETSc Developers, I am wondering whether there is a function for Vec. Assuming Vec v; double d_test; This function can compare each values (v(i)) of v with d_test and return the logical values contained in Vec v_logi (such as if(v(i)>d_test) v_logi(i)=0; else v_logi(i)=1; ) Thanks a lot. Regards, Yujie From bsmith at mcs.anl.gov Mon May 10 20:36:11 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 10 May 2010 20:36:11 -0500 Subject: [petsc-users] logical operation for Vec In-Reply-To: References: Message-ID: No, nothing like that. But this is easily done by call VecGetArray() looping over the array and applying the check to each entry. barry On May 10, 2010, at 8:31 PM, Yujie wrote: > Dear PETSc Developers, > > I am wondering whether there is a function for Vec. > > Assuming > > Vec v; > > double d_test; > > This function can compare each values (v(i)) of v with d_test and > return the logical values contained in Vec v_logi (such as > if(v(i)>d_test) > v_logi(i)=0; > else > v_logi(i)=1; ) > > Thanks a lot. > > Regards, > Yujie From recrusader at gmail.com Mon May 10 20:36:57 2010 From: recrusader at gmail.com (Yujie) Date: Mon, 10 May 2010 20:36:57 -0500 Subject: [petsc-users] logical operation for Vec In-Reply-To: References: Message-ID: Thanks, Barry. I got it. Yujie On Mon, May 10, 2010 at 8:36 PM, Barry Smith wrote: > > ?No, nothing like that. But this is easily done by call VecGetArray() looping over the array and applying the check to each entry. > > ?barry > > On May 10, 2010, at 8:31 PM, Yujie wrote: > >> Dear PETSc Developers, >> >> I am wondering whether there is a function for Vec. >> >> Assuming >> >> Vec v; >> >> double d_test; >> >> This function can compare each values (v(i)) of v with d_test and >> return the logical values contained in Vec v_logi (such as >> if(v(i)>d_test) >> ?v_logi(i)=0; >> else >> ?v_logi(i)=1; ?) >> >> Thanks a lot. >> >> Regards, >> Yujie > > From zonexo at gmail.com Tue May 11 20:09:17 2010 From: zonexo at gmail.com (Wee-Beng TAY) Date: Wed, 12 May 2010 09:09:17 +0800 Subject: [petsc-users] Error with file linking Message-ID: <4BE9FFBD.7080705@gmail.com> Hi, my code works fine with petsc-3.0. However I have problem linking when using petsc3.1. I used the same linking command as that given by make ex1f, which is : /app1/mvapich/current/bin/mpif90 -r8 -w95 -fPIC -g -O3 -o a.out global.o grid.o flux_area.o bc.o bc_impl.o bc_semi.o set_matrix.o inter_step.o mom_disz.o poisson.o petsc_sub.o cell_data.o fractional.o ns2d_c.o -Wl,-rpath,/home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib -L/home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib -lpetsc -Wl,-rpath,/home/svu/g0306332/lib/hypre-2.6.0b_atlas5/lib -L/home/svu/g0306332/lib/hypre-2.6.0b_atlas5/lib -lHYPRE -lmpichcxx -lstdc++ -Wl,-rpath,/app1/intel/mkl/10.0.5.025/lib/em64t -L/app1/intel/mkl/10.0.5.025/lib/em64t -lmkl_lapack -lmkl -lguide -lpthread -L/app1/mvapich2/1.4/lib -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -ldl -lmpich -lpthread -lrdmacm -libverbs -libumad -lrt -lgcc_s -lmpichf90 -L/app1/intel/Compiler/11.0/074/lib/intel64 -lifport -lifcore -limf -lsvml -lm -lipgo -lirc -lirc_s -lm -lmpichcxx -lstdc++ -lmpichcxx -lstdc++ -ldl -lmpich -lpthread -lrdmacm -libverbs -libumad -lrt -lgcc_s -ldl The compiling command is: /app1/mvapich/current/bin/mpif90 -r8 -132 -fPIC -g -w90 -w -w95 -c -O3 -I/home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/include -I/home/svu/g0306332/codes/petsc-3.1-p0/include -I/home/svu/g0306332/lib/hypre-2.6.0b_atlas5/include -I/app1/mvapich2/1.4/include -I/app1/mvapich2/current/include -I/home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/include -I/home/svu/g0306332/lib/hypre-2.6.0b_atlas5/include -I/app1/mvapich2/1.4/include -I/app1/mvapich2/current/include The error msgs are: /home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib/libpetsc.a(mpinit.o): In function `PetscOpenMPSpawn': /home/svu/g0306332/codes/petsc-3.1-p0/src/sys/objects/mpinit.c:72: undefined reference to `MPI_Comm_get_parent' /home/svu/g0306332/codes/petsc-3.1-p0/src/sys/objects/mpinit.c:79: undefined reference to `MPI_Comm_spawn' /home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib/libpetsc.a(gr2.o): In function `DAArrayMPIIO': /home/svu/g0306332/codes/petsc-3.1-p0/src/dm/da/src/gr2.c:482: undefined reference to `MPI_Type_get_extent' /home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib/libpetsc.a(binv.o): In function `PetscViewerBinaryMPIIO': /home/svu/g0306332/codes/petsc-3.1-p0/src/sys/viewer/impls/binary/binv.c:563: undefined reference to `MPI_Type_get_extent' /home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib/libpetsc.a(vpscat.o): In function `VecScatterDestroy_PtoP': /home/svu/g0306332/codes/petsc-3.1-p0/src/vec/vec/utils/vpscat.c:169: undefined reference to `MPI_Win_free' /home/svu/g0306332/codes/petsc-3.1-p0/src/vec/vec/utils/vpscat.c:170: undefined reference to `MPI_Win_free' /home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib/libpetsc.a(vpscat.o): In function `VecScatterBegin_1': /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:64: undefined reference to `MPI_Win_fence' /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:67: undefined reference to `MPI_Put' /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:69: undefined reference to `MPI_Win_fence' /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:52: undefined reference to `MPI_Alltoallw' /home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib/libpetsc.a(vpscat.o): In function `VecScatterCreateCommon_PtoS': /home/svu/g0306332/codes/petsc-3.1-p0/src/vec/vec/utils/vpscat.c:1896: undefined reference to `MPI_Win_create' /home/svu/g0306332/codes/petsc-3.1-p0/src/vec/vec/utils/vpscat.c:1909: undefined reference to `MPI_Win_create' /home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib/libpetsc.a(vpscat.o): In function `VecScatterBegin_12': /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:64: undefined reference to `MPI_Win_fence' /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:67: undefined reference to `MPI_Put' /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:69: undefined reference to `MPI_Win_fence' /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:52: undefined reference to `MPI_Alltoallw' /home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib/libpetsc.a(vpscat.o): In function `VecScatterBegin_8': /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:64: undefined reference to `MPI_Win_fence' /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:67: undefined reference to `MPI_Put' /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:69: undefined reference to `MPI_Win_fence' /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:52: undefined reference to `MPI_Alltoallw' /home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib/libpetsc.a(vpscat.o): In function `VecScatterBegin_7': /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:64: undefined reference to `MPI_Win_fence' /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:67: undefined reference to `MPI_Put' /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:69: undefined reference to `MPI_Win_fence' /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:52: undefined reference to `MPI_Alltoallw' /home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib/libpetsc.a(vpscat.o): In function `VecScatterBegin_6': /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:64: undefined reference to `MPI_Win_fence' /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:67: undefined reference to `MPI_Put' /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:69: undefined reference to `MPI_Win_fence' /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:52: undefined reference to `MPI_Alltoallw' /home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib/libpetsc.a(vpscat.o): In function `VecScatterBegin_5': /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:64: undefined reference to `MPI_Win_fence' /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:67: undefined reference to `MPI_Put' /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:69: undefined reference to `MPI_Win_fence' /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:52: undefined reference to `MPI_Alltoallw' /home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib/libpetsc.a(vpscat.o): In function `VecScatterBegin_4': /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:64: undefined reference to `MPI_Win_fence' /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:67: undefined reference to `MPI_Put' /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:69: undefined reference to `MPI_Win_fence' /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:52: undefined reference to `MPI_Alltoallw' /home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib/libpetsc.a(vpscat.o): In function `VecScatterBegin_3': /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:64: undefined reference to `MPI_Win_fence' /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:67: undefined reference to `MPI_Put' /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:69: undefined reference to `MPI_Win_fence' /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:52: undefined reference to `MPI_Alltoallw' /home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib/libpetsc.a(vpscat.o): In function `VecScatterBegin_2': /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:64: undefined reference to `MPI_Win_fence' /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:67: undefined reference to `MPI_Put' /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:69: undefined reference to `MPI_Win_fence' /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:52: undefined reference to `MPI_Alltoallw' /home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib/libpetsc.a(vecio.o): In function `VecLoad_Binary': /home/svu/g0306332/codes/petsc-3.1-p0/src/vec/vec/utils/vecio.c:276: undefined reference to `MPI_Type_get_extent' May I know what's wrong? -- Thank you very much and have a nice day! Yours sincerely, Wee-Beng Tay From balay at mcs.anl.gov Tue May 11 20:42:01 2010 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 11 May 2010 20:42:01 -0500 (CDT) Subject: [petsc-users] Error with file linking In-Reply-To: <4BE9FFBD.7080705@gmail.com> References: <4BE9FFBD.7080705@gmail.com> Message-ID: This is too confusing.. On Wed, 12 May 2010, Wee-Beng TAY wrote: > Hi, > > my code works fine with petsc-3.0. However I have problem linking when using > petsc3.1. I used the same linking command as that given by make ex1f, which is > : This is not the compile for ex1f - I see ' a.out global.o ...' > > /app1/mvapich/current/bin/mpif90 -r8 -w95 -fPIC -g -O3 -o a.out global.o > grid.o flux_area.o bc.o bc_impl.o bc_semi.o set_matrix.o inter_step.o > mom_disz.o poisson.o petsc_sub.o cell_data.o fractional.o ns2d_c.o > -Wl,-rpath,/home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib > -L/home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib -lpetsc > -Wl,-rpath,/home/svu/g0306332/lib/hypre-2.6.0b_atlas5/lib > -L/home/svu/g0306332/lib/hypre-2.6.0b_atlas5/lib -lHYPRE -lmpichcxx -lstdc++ > -Wl,-rpath,/app1/intel/mkl/10.0.5.025/lib/em64t > -L/app1/intel/mkl/10.0.5.025/lib/em64t -lmkl_lapack -lmkl -lguide -lpthread > -L/app1/mvapich2/1.4/lib -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -ldl -lmpich > -lpthread -lrdmacm -libverbs -libumad -lrt -lgcc_s -lmpichf90 > -L/app1/intel/Compiler/11.0/074/lib/intel64 -lifport -lifcore -limf -lsvml -lm > -lipgo -lirc -lirc_s -lm -lmpichcxx -lstdc++ -lmpichcxx -lstdc++ -ldl -lmpich > -lpthread -lrdmacm -libverbs -libumad -lrt -lgcc_s -ldl > > The compiling command is: > > /app1/mvapich/current/bin/mpif90 -r8 -132 -fPIC -g -w90 -w -w95 -c -O3 > -I/home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/include > -I/home/svu/g0306332/codes/petsc-3.1-p0/include > -I/home/svu/g0306332/lib/hypre-2.6.0b_atlas5/include > -I/app1/mvapich2/1.4/include -I/app1/mvapich2/current/include > -I/home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/include > -I/home/svu/g0306332/lib/hypre-2.6.0b_atlas5/include > -I/app1/mvapich2/1.4/include -I/app1/mvapich2/current/include The above compile command is incomplete. Why truncate it? What about the link command? > > The error msgs are: > > /home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib/libpetsc.a(mpinit.o): > In function `PetscOpenMPSpawn': > /home/svu/g0306332/codes/petsc-3.1-p0/src/sys/objects/mpinit.c:72: undefined > reference to `MPI_Comm_get_parent' Clearly you are not linking with the library that provides MPI_Comm_get_parent. Why not use petsc makefiles - and avoid these hasseles? Satish > /home/svu/g0306332/codes/petsc-3.1-p0/src/sys/objects/mpinit.c:79: undefined > reference to `MPI_Comm_spawn' > /home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib/libpetsc.a(gr2.o): > In function `DAArrayMPIIO': > /home/svu/g0306332/codes/petsc-3.1-p0/src/dm/da/src/gr2.c:482: undefined > reference to `MPI_Type_get_extent' > /home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib/libpetsc.a(binv.o): > In function `PetscViewerBinaryMPIIO': > /home/svu/g0306332/codes/petsc-3.1-p0/src/sys/viewer/impls/binary/binv.c:563: > undefined reference to `MPI_Type_get_extent' > /home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib/libpetsc.a(vpscat.o): > In function `VecScatterDestroy_PtoP': > /home/svu/g0306332/codes/petsc-3.1-p0/src/vec/vec/utils/vpscat.c:169: > undefined reference to `MPI_Win_free' > /home/svu/g0306332/codes/petsc-3.1-p0/src/vec/vec/utils/vpscat.c:170: > undefined reference to `MPI_Win_free' > /home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib/libpetsc.a(vpscat.o): > In function `VecScatterBegin_1': > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:64: > undefined reference to `MPI_Win_fence' > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:67: > undefined reference to `MPI_Put' > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:69: > undefined reference to `MPI_Win_fence' > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:52: > undefined reference to `MPI_Alltoallw' > /home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib/libpetsc.a(vpscat.o): > In function `VecScatterCreateCommon_PtoS': > /home/svu/g0306332/codes/petsc-3.1-p0/src/vec/vec/utils/vpscat.c:1896: > undefined reference to `MPI_Win_create' > /home/svu/g0306332/codes/petsc-3.1-p0/src/vec/vec/utils/vpscat.c:1909: > undefined reference to `MPI_Win_create' > /home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib/libpetsc.a(vpscat.o): > In function `VecScatterBegin_12': > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:64: > undefined reference to `MPI_Win_fence' > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:67: > undefined reference to `MPI_Put' > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:69: > undefined reference to `MPI_Win_fence' > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:52: > undefined reference to `MPI_Alltoallw' > /home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib/libpetsc.a(vpscat.o): > In function `VecScatterBegin_8': > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:64: > undefined reference to `MPI_Win_fence' > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:67: > undefined reference to `MPI_Put' > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:69: > undefined reference to `MPI_Win_fence' > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:52: > undefined reference to `MPI_Alltoallw' > /home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib/libpetsc.a(vpscat.o): > In function `VecScatterBegin_7': > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:64: > undefined reference to `MPI_Win_fence' > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:67: > undefined reference to `MPI_Put' > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:69: > undefined reference to `MPI_Win_fence' > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:52: > undefined reference to `MPI_Alltoallw' > /home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib/libpetsc.a(vpscat.o): > In function `VecScatterBegin_6': > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:64: > undefined reference to `MPI_Win_fence' > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:67: > undefined reference to `MPI_Put' > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:69: > undefined reference to `MPI_Win_fence' > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:52: > undefined reference to `MPI_Alltoallw' > /home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib/libpetsc.a(vpscat.o): > In function `VecScatterBegin_5': > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:64: > undefined reference to `MPI_Win_fence' > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:67: > undefined reference to `MPI_Put' > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:69: > undefined reference to `MPI_Win_fence' > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:52: > undefined reference to `MPI_Alltoallw' > /home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib/libpetsc.a(vpscat.o): > In function `VecScatterBegin_4': > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:64: > undefined reference to `MPI_Win_fence' > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:67: > undefined reference to `MPI_Put' > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:69: > undefined reference to `MPI_Win_fence' > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:52: > undefined reference to `MPI_Alltoallw' > /home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib/libpetsc.a(vpscat.o): > In function `VecScatterBegin_3': > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:64: > undefined reference to `MPI_Win_fence' > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:67: > undefined reference to `MPI_Put' > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:69: > undefined reference to `MPI_Win_fence' > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:52: > undefined reference to `MPI_Alltoallw' > /home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib/libpetsc.a(vpscat.o): > In function `VecScatterBegin_2': > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:64: > undefined reference to `MPI_Win_fence' > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:67: > undefined reference to `MPI_Put' > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:69: > undefined reference to `MPI_Win_fence' > /home/svu/g0306332/codes/petsc-3.1-p0/include/../src/vec/vec/utils/vpscat.h:52: > undefined reference to `MPI_Alltoallw' > /home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib/libpetsc.a(vecio.o): > In function `VecLoad_Binary': > /home/svu/g0306332/codes/petsc-3.1-p0/src/vec/vec/utils/vecio.c:276: undefined > reference to `MPI_Type_get_extent' > > May I know what's wrong? > > From balay at mcs.anl.gov Tue May 11 20:55:32 2010 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 11 May 2010 20:55:32 -0500 (CDT) Subject: [petsc-users] Error with file linking In-Reply-To: References: <4BE9FFBD.7080705@gmail.com> Message-ID: Lets step back and deal with this primary issue. Have you attempted to use makefiles in petsc format? What problems have you encountered? Perhaps they are fixable. The reason to use this format is to make your makefile as portable as possible [and avoid such issues]. Satish On Tue, 11 May 2010, Satish Balay wrote: > > Why not use petsc makefiles - and avoid these hasseles? > From zonexo at gmail.com Wed May 12 11:41:32 2010 From: zonexo at gmail.com (Wee-Beng TAY) Date: Thu, 13 May 2010 00:41:32 +0800 Subject: [petsc-users] Error with file linking In-Reply-To: References: <4BE9FFBD.7080705@gmail.com> Message-ID: <4BEADA3C.8000205@gmail.com> Hi Satish, Sorry for the confusion. I have attached my makefile. This was used to compile and link my code before I used PETSc 3.1. I have several fortran files. Some contain PETSc commands and need to use PETSc include files. Other do not. In the end, they all have to be linked to form my final code. I used to copy part of the compiling and linking command from "make ex1f" to compile my code: /app1/mvapich2/current/bin/mpif90 -c -O3 -I/home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/include -I/home/svu/g0306332/codes/petsc-3.1-p0/include -I/home/svu/g0306332/lib/hypre-2.6.0b_atlas5/include -I/app1/mvapich2/1.4/include -I/app1/mvapich2/current/include -I/home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/include -I/home/svu/g0306332/lib/hypre-2.6.0b_atlas5/include -I/app1/mvapich2/1.4/include -I/app1/mvapich2/current/include -o ex1f.o ex1f.F /app1/mvapich2/current/bin/mpif90 -O3 -o ex1f ex1f.o -Wl,-rpath,/home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib -L/home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib -lpetsc -Wl,-rpath,/home/svu/g0306332/lib/hypre-2.6.0b_atlas5/lib -L/home/svu/g0306332/lib/hypre-2.6.0b_atlas5/lib -lHYPRE -lmpichcxx -lstdc++ -Wl,-rpath,/app1/intel/mkl/10.0.5.025/lib/em64t -L/app1/intel/mkl/10.0.5.025/lib/em64t -lmkl_lapack -lmkl -lguide -lpthread -L/app1/mvapich2/1.4/lib -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -ldl -lmpich -lpthread -lrdmacm -libverbs -libumad -lrt -lgcc_s -lmpichf90 -L/app1/intel/Compiler/11.0/074/lib/intel64 -lifport -lifcore -limf -lsvml -lm -lipgo -lirc -lirc_s -lm -lmpichcxx -lstdc++ -lmpichcxx -lstdc++ -ldl -lmpich -lpthread -lrdmacm -libverbs -libumad -lrt -lgcc_s -ldl However, it doesn't seem to work now. I tried to look at chapter 14 but I don't really understand since I know very little about makefiles. Can you give a rough guide for my case? I will try to read up on the manual to get more info. Moreover, I am confused by the use of: #include "finclude/petsc.h" etc. In my fortran files, do I simply use petsc.h or I have to use specific *.h depending on what petsc commands are used? Thank you very much and have a nice day! Yours sincerely, Wee-Beng Tay On 12/5/2010 9:55 AM, Satish Balay wrote: > Lets step back and deal with this primary issue. > > Have you attempted to use makefiles in petsc format? What problems > have you encountered? Perhaps they are fixable. > > The reason to use this format is to make your makefile as portable as > possible [and avoid such issues]. > > Satish > > On Tue, 11 May 2010, Satish Balay wrote: > > >> Why not use petsc makefiles - and avoid these hasseles? >> >> > -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: makefile_atlas5 URL: From balay at mcs.anl.gov Wed May 12 12:27:39 2010 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 12 May 2010 12:27:39 -0500 (CDT) Subject: [petsc-users] Error with file linking In-Reply-To: <4BEADA3C.8000205@gmail.com> References: <4BE9FFBD.7080705@gmail.com> <4BEADA3C.8000205@gmail.com> Message-ID: Attaching the modified makefile. It might need a couple of iterations of fixing. Hopefully its clear why it would work. Note - if not for different flags required for different sources (opt/opt2/opt3) - you can build all petsc/non-petsc sources with petsc makefile targets. > In my fortran files, do I simply use petsc.h or I have to use specific *.h > depending on what petsc commands are used? With petsc-3.1 you just have to use petsc.h for all F77 interface stuff, and petsc.h90 for all f90 interface stuff [like VecGetArrayF90()] Satish On Thu, 13 May 2010, Wee-Beng TAY wrote: > Hi Satish, > > Sorry for the confusion. > > I have attached my makefile. This was used to compile and link my code before > I used PETSc 3.1. > > I have several fortran files. Some contain PETSc commands and need to use > PETSc include files. Other do not. In the end, they all have to be linked to > form my final code. I used to copy part of the compiling and linking command > from "make ex1f" to compile my code: > > /app1/mvapich2/current/bin/mpif90 -c -O3 > -I/home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/include > -I/home/svu/g0306332/codes/petsc-3.1-p0/include > -I/home/svu/g0306332/lib/hypre-2.6.0b_atlas5/include > -I/app1/mvapich2/1.4/include -I/app1/mvapich2/current/include > -I/home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/include > -I/home/svu/g0306332/lib/hypre-2.6.0b_atlas5/include > -I/app1/mvapich2/1.4/include -I/app1/mvapich2/current/include -o ex1f.o > ex1f.F > > /app1/mvapich2/current/bin/mpif90 -O3 -o ex1f ex1f.o > -Wl,-rpath,/home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib > -L/home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib -lpetsc > -Wl,-rpath,/home/svu/g0306332/lib/hypre-2.6.0b_atlas5/lib > -L/home/svu/g0306332/lib/hypre-2.6.0b_atlas5/lib -lHYPRE -lmpichcxx -lstdc++ > -Wl,-rpath,/app1/intel/mkl/10.0.5.025/lib/em64t > -L/app1/intel/mkl/10.0.5.025/lib/em64t -lmkl_lapack -lmkl -lguide -lpthread > -L/app1/mvapich2/1.4/lib -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -ldl -lmpich > -lpthread -lrdmacm -libverbs -libumad -lrt -lgcc_s -lmpichf90 > -L/app1/intel/Compiler/11.0/074/lib/intel64 -lifport -lifcore -limf -lsvml -lm > -lipgo -lirc -lirc_s -lm -lmpichcxx -lstdc++ -lmpichcxx -lstdc++ -ldl -lmpich > -lpthread -lrdmacm -libverbs -libumad -lrt -lgcc_s -ldl > > However, it doesn't seem to work now. I tried to look at chapter 14 but I > don't really understand since I know very little about makefiles. > > Can you give a rough guide for my case? I will try to read up on the manual to > get more info. > > Moreover, I am confused by the use of: > > #include "finclude/petsc.h" etc. > > In my fortran files, do I simply use petsc.h or I have to use specific *.h > depending on what petsc commands are used? > > Thank you very much and have a nice day! > > Yours sincerely, > > Wee-Beng Tay > > > On 12/5/2010 9:55 AM, Satish Balay wrote: > > Lets step back and deal with this primary issue. > > > > Have you attempted to use makefiles in petsc format? What problems > > have you encountered? Perhaps they are fixable. > > > > The reason to use this format is to make your makefile as portable as > > possible [and avoid such issues]. > > > > Satish > > > > On Tue, 11 May 2010, Satish Balay wrote: > > > > > > > Why not use petsc makefiles - and avoid these hasseles? > > > > > > > > -------------- next part -------------- # PETSC_DIR and PETSC_ARCH should be set either in env or in this makeifle CFLAGS = FFLAGS = -r8 -132 -fPIC -g -w90 -w -w95 -c -O3 # instead of opt2 add these flags to default petsc compile target for .F/.f CPPFLAGS = FPPFLAGS = CLEANFILES = a.out include ${PETSC_DIR}/conf/variables include ${PETSC_DIR}/conf/rules opt = -r8 -w95 -c -O3 opt3 = -w95 -c -O3 libs =/home/svu/g0306332/lib/tecio64.a /home/svu/g0306332/lib/linux64.a OBJS = global.o grid.o flux_area.o bc.o bc_impl.o bc_semi.o set_matrix.o inter_step.o mom_disz.o poisson.o petsc_sub.o cell_data.o fractional.o ns2d_c.o # perhaps -r8 -w95 -fPIC -g -O3 flags not needed here? a.out : $(OBJS) $(FLINKER) -r8 -w95 -fPIC -g -O3 -o a.out $(OBJS) $(libs) $(PETSC_LIB) #compiled these sources with default petsc targets global.o : global.F petsc_sub.o : petsc_sub.F poisson.o ns2d_c.o : ns2d_c.f90 cell_data.o fractional.o # sources compiled separately - as they need different opt/opt3 compiler flags. grid.o : grid.f90 global.o $(FC) $(opt3) grid.f90 flux_area.o : flux_area.f90 global.o $(FC) $(opt) flux_area.f90 bc.o : bc.f90 flux_area.o $(FC) $(opt) bc.f90 bc_impl.o : bc_impl.f90 bc.o $(FC) $(opt) bc_impl.f90 bc_semi.o : bc_semi.f90 bc.o $(FC) $(opt) bc_semi.f90 set_matrix.o : set_matrix.f90 bc_semi.o bc_impl.o $(FC) $(opt) set_matrix.f90 inter_step.o : inter_step.f90 bc_impl.o $(FC) $(opt) inter_step.f90 mom_disz.o : mom_disz.f90 bc_impl.o bc_semi.o set_matrix.o $(FC) $(opt) mom_disz.f90 poisson.o : poisson.f90 set_matrix.o $(FC) $(opt) poisson.f90 cell_data.o : cell_data.f90 grid.o set_matrix.o $(FC) $(opt) cell_data.f90 fractional.o : fractional.f90 mom_disz.o petsc_sub.o inter_step.o $(FC) $(opt) fractional.f90 From balay at mcs.anl.gov Wed May 12 12:34:07 2010 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 12 May 2010 12:34:07 -0500 (CDT) Subject: [petsc-users] Error with file linking In-Reply-To: References: <4BE9FFBD.7080705@gmail.com> <4BEADA3C.8000205@gmail.com> Message-ID: On Wed, 12 May 2010, Satish Balay wrote: > Attaching the modified makefile. It might need a couple of iterations > of fixing. Also should remove '-c' from FFLAGS Satish From mader at utias.utoronto.ca Wed May 12 12:39:22 2010 From: mader at utias.utoronto.ca (C.A.(Sandy) Mader) Date: Wed, 12 May 2010 13:39:22 -0400 Subject: [petsc-users] Issues with subsequent GMRES Solves Message-ID: <1273685962.20896.40.camel@tuff> I am using PETSc to solve an adjoint system for a CFD code. The issue I am having is that if I try to run successive cfd/adjoint solutions, the adjoint fails to converge after the first cfd/adjoint solution cycle. i.e. Solve CFD (ok) Solve adjoint system with multiple RHS's (ok) Solve CFD at new point (ok) Solve adjoint system at new point with multiple RHS's (GMRES iterations stall) PETSc is not used to solve the CFD system, only the adjoint system. I am currently using the following solver options: 'Adjoint solver type': 'GMRES',\ 'adjoint relative tolerance':1e-6,\ 'adjoint absolute tolerance':1e-16,\ 'adjoint max iterations': 1500,\ 'adjoint restart iteration' : 80,\ 'adjoint monitor step': 10,\ 'Preconditioner Side': 'right',\ 'Matrix Ordering': 'NestedDissection',\ 'Global Preconditioner Type': 'Additive Schwartz',\ 'Local Preconditioner Type' : 'ILU',\ 'ILU Fill Levels': 3,\ 'ASM Overlap' : 5,\ Also, I am using the fortran interface to PETSc. As a sample, I have included two convergence histories. Both are at the same converged CFD point(this case is ~1.2million cell so the PETSc system is approx 6million degrees of freedom). The first is an example where the adjoint system is solved the first time through the cycle. The second is an example of where the adjoint is solved the second time through the cycle: 1) as first point in cycle # ... KSP properties: # type : gmres # tolerances : rel = 1.0E-10 # abs = 1.0E-16 # div = 1.0E+05 # max.iter. : 1500 # precond.type: asm Solving ADjoint Transpose with PETSc... 0 KSP Residual norm 0.1392E+00 10 KSP Residual norm 0.6394E-01 20 KSP Residual norm 0.6106E-01 30 KSP Residual norm 0.6019E-01 40 KSP Residual norm 0.5941E-01 50 KSP Residual norm 0.5876E-01 60 KSP Residual norm 0.5602E-01 70 KSP Residual norm 0.4915E-01 80 KSP Residual norm 0.3994E-01 90 KSP Residual norm 0.3892E-01 100 KSP Residual norm 0.3854E-01 110 KSP Residual norm 0.3794E-01 120 KSP Residual norm 0.3717E-01 130 KSP Residual norm 0.3630E-01 140 KSP Residual norm 0.3415E-01 ... 900 KSP Residual norm 0.2437E-09 910 KSP Residual norm 0.1452E-09 920 KSP Residual norm 0.1025E-09 930 KSP Residual norm 0.6875E-10 940 KSP Residual norm 0.4141E-10 950 KSP Residual norm 0.2317E-10 960 KSP Residual norm 0.1559E-10 Solving ADjoint Transpose with PETSc time (s) = 391.43 Norm of error = 0.1366E-10 Iterations = 965 ------------------------------------------------ PETSc solver converged after 965 iterations. ------------------------------------------------ 2) As second point in cycle # ... KSP properties: # type : gmres # tolerances : rel = 1.0E-10 # abs = 1.0E-16 # div = 1.0E+05 # max.iter. : 1500 # precond.type: asm Solving ADjoint Transpose with PETSc... 0 KSP Residual norm 0.1392E+00 10 KSP Residual norm 0.6400E-01 20 KSP Residual norm 0.6140E-01 30 KSP Residual norm 0.6060E-01 40 KSP Residual norm 0.5995E-01 50 KSP Residual norm 0.5974E-01 60 KSP Residual norm 0.5971E-01 70 KSP Residual norm 0.5957E-01 80 KSP Residual norm 0.5906E-01 90 KSP Residual norm 0.5906E-01 100 KSP Residual norm 0.5906E-01 110 KSP Residual norm 0.5903E-01 120 KSP Residual norm 0.5901E-01 130 KSP Residual norm 0.5901E-01 140 KSP Residual norm 0.5901E-01 ... 1400 KSP Residual norm 0.5895E-01 1410 KSP Residual norm 0.5895E-01 1420 KSP Residual norm 0.5895E-01 1430 KSP Residual norm 0.5895E-01 1440 KSP Residual norm 0.5895E-01 1450 KSP Residual norm 0.5895E-01 1460 KSP Residual norm 0.5895E-01 1470 KSP Residual norm 0.5895E-01 1480 KSP Residual norm 0.5895E-01 1490 KSP Residual norm 0.5895E-01 1500 KSP Residual norm 0.5895E-01 Solving ADjoint Transpose with PETSc time (s) = 516.59 Norm of error = 0.5895E-01 Iterations = 1500 ------------------------------------------------ PETSc solver converged after 1500 iterations. ------------------------------------------------ I have tried both resetting the operators between cycles and completely destroying the KSP object between cycles. Both give the same result. Is there a step I am missing to properly reset the system for the subsequent solves? Thanks in advance for your input... Sandy -- C.A.(Sandy) Mader Ph.D Candidate Multidisciplinary Design Optimization Laboratory University of Toronto Institute for Aerospace Studies mader at utias.utoronto.ca From bsmith at mcs.anl.gov Wed May 12 12:48:47 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 12 May 2010 12:48:47 -0500 Subject: [petsc-users] Issues with subsequent GMRES Solves In-Reply-To: <1273685962.20896.40.camel@tuff> References: <1273685962.20896.40.camel@tuff> Message-ID: <669D1DEB-346A-48E6-9CE8-A288162B69C4@mcs.anl.gov> Is it the same matrix or a different matrix that is solved with the "second adjoint system"? Or is it just a different right hand side? If it is a different system how do you know it will even converge for the new matrix? Maybe the new matrix is much more difficult? Is it possible the "second adjoint system" is singular? Given that this is a very strong preconditioner the convergence for the first adjoint system is pretty poor. Does this happen also for much smaller problems? Recommend running much smaller problem to see what happens convergence wise, then run that much smaller problem with -pc_type lu to see if the direct solver handles it happily. On May 12, 2010, at 12:39 PM, C.A.(Sandy) Mader wrote: > I am using PETSc to solve an adjoint system for a CFD code. > The issue I am having is that if I try to run successive cfd/adjoint > solutions, the adjoint fails to converge after the first cfd/adjoint > solution cycle. i.e. > Solve CFD (ok) > Solve adjoint system with multiple RHS's (ok) > Solve CFD at new point (ok) > Solve adjoint system at new point with multiple RHS's (GMRES iterations > stall) > > PETSc is not used to solve the CFD system, only the adjoint system. > > I am currently using the following solver options: > 'Adjoint solver type': 'GMRES',\ > 'adjoint relative tolerance':1e-6,\ > 'adjoint absolute tolerance':1e-16,\ > 'adjoint max iterations': 1500,\ > 'adjoint restart iteration' : 80,\ > 'adjoint monitor step': 10,\ > 'Preconditioner Side': 'right',\ > 'Matrix Ordering': 'NestedDissection',\ > 'Global Preconditioner Type': 'Additive Schwartz',\ > 'Local Preconditioner Type' : 'ILU',\ > 'ILU Fill Levels': 3,\ > 'ASM Overlap' : 5,\ > > Also, I am using the fortran interface to PETSc. > > As a sample, I have included two convergence histories. Both are at the > same converged CFD point(this case is ~1.2million cell so the PETSc > system is approx 6million degrees of freedom). The first is an example > where the adjoint system is solved the first time through the cycle. The > second is an example of where the adjoint is solved the second time > through the cycle: > 1) as first point in cycle > # ... KSP properties: > # type : gmres > # tolerances : rel = 1.0E-10 > # abs = 1.0E-16 > # div = 1.0E+05 > # max.iter. : 1500 > # precond.type: asm > Solving ADjoint Transpose with PETSc... > 0 KSP Residual norm 0.1392E+00 > 10 KSP Residual norm 0.6394E-01 > 20 KSP Residual norm 0.6106E-01 > 30 KSP Residual norm 0.6019E-01 > 40 KSP Residual norm 0.5941E-01 > 50 KSP Residual norm 0.5876E-01 > 60 KSP Residual norm 0.5602E-01 > 70 KSP Residual norm 0.4915E-01 > 80 KSP Residual norm 0.3994E-01 > 90 KSP Residual norm 0.3892E-01 > 100 KSP Residual norm 0.3854E-01 > 110 KSP Residual norm 0.3794E-01 > 120 KSP Residual norm 0.3717E-01 > 130 KSP Residual norm 0.3630E-01 > 140 KSP Residual norm 0.3415E-01 > ... > 900 KSP Residual norm 0.2437E-09 > 910 KSP Residual norm 0.1452E-09 > 920 KSP Residual norm 0.1025E-09 > 930 KSP Residual norm 0.6875E-10 > 940 KSP Residual norm 0.4141E-10 > 950 KSP Residual norm 0.2317E-10 > 960 KSP Residual norm 0.1559E-10 > Solving ADjoint Transpose with PETSc time (s) = 391.43 > Norm of error = 0.1366E-10 Iterations = 965 > ------------------------------------------------ > PETSc solver converged after 965 iterations. > ------------------------------------------------ > > 2) As second point in cycle > # ... KSP properties: > # type : gmres > # tolerances : rel = 1.0E-10 > # abs = 1.0E-16 > # div = 1.0E+05 > # max.iter. : 1500 > # precond.type: asm > Solving ADjoint Transpose with PETSc... > 0 KSP Residual norm 0.1392E+00 > 10 KSP Residual norm 0.6400E-01 > 20 KSP Residual norm 0.6140E-01 > 30 KSP Residual norm 0.6060E-01 > 40 KSP Residual norm 0.5995E-01 > 50 KSP Residual norm 0.5974E-01 > 60 KSP Residual norm 0.5971E-01 > 70 KSP Residual norm 0.5957E-01 > 80 KSP Residual norm 0.5906E-01 > 90 KSP Residual norm 0.5906E-01 > 100 KSP Residual norm 0.5906E-01 > 110 KSP Residual norm 0.5903E-01 > 120 KSP Residual norm 0.5901E-01 > 130 KSP Residual norm 0.5901E-01 > 140 KSP Residual norm 0.5901E-01 > ... > 1400 KSP Residual norm 0.5895E-01 > 1410 KSP Residual norm 0.5895E-01 > 1420 KSP Residual norm 0.5895E-01 > 1430 KSP Residual norm 0.5895E-01 > 1440 KSP Residual norm 0.5895E-01 > 1450 KSP Residual norm 0.5895E-01 > 1460 KSP Residual norm 0.5895E-01 > 1470 KSP Residual norm 0.5895E-01 > 1480 KSP Residual norm 0.5895E-01 > 1490 KSP Residual norm 0.5895E-01 > 1500 KSP Residual norm 0.5895E-01 > Solving ADjoint Transpose with PETSc time (s) = 516.59 > Norm of error = 0.5895E-01 Iterations = 1500 > ------------------------------------------------ > PETSc solver converged after 1500 iterations. > ------------------------------------------------ > I have tried both resetting the operators between cycles and completely > destroying the KSP object between cycles. Both give the same result. Is > there a step I am missing to properly reset the system for the > subsequent solves? > > Thanks in advance for your input... > > Sandy > -- > C.A.(Sandy) Mader > Ph.D Candidate > Multidisciplinary Design Optimization Laboratory > University of Toronto Institute for Aerospace Studies > mader at utias.utoronto.ca > From zonexo at gmail.com Wed May 12 12:51:42 2010 From: zonexo at gmail.com (Wee-Beng TAY) Date: Thu, 13 May 2010 01:51:42 +0800 Subject: [petsc-users] Error with file linking In-Reply-To: References: <4BE9FFBD.7080705@gmail.com> <4BEADA3C.8000205@gmail.com> Message-ID: <4BEAEAAE.9050103@gmail.com> Hi Satish, I tried to run it, with the -c present and absent. However, nothing happened. I run using "make". I also specified "export PETSC_DIR=/home/svu/g0306332/codes/petsc-3.1-p0/" and "export PETSC_ARCH=atlas5-mpi-nodebug" before that. Did I miss out something? Also supposed that I have the same flag for all sources. In that case, how can I build them? Thank you very much and have a nice day! Yours sincerely, Wee-Beng Tay On 13/5/2010 1:27 AM, Satish Balay wrote: > Attaching the modified makefile. It might need a couple of iterations > of fixing. > > Hopefully its clear why it would work. Note - if not for different > flags required for different sources (opt/opt2/opt3) - you can build > all petsc/non-petsc sources with petsc makefile targets. > > >> In my fortran files, do I simply use petsc.h or I have to use specific *.h >> depending on what petsc commands are used? >> > With petsc-3.1 you just have to use petsc.h for all F77 interface stuff, and > petsc.h90 for all f90 interface stuff [like VecGetArrayF90()] > > Satish > > On Thu, 13 May 2010, Wee-Beng TAY wrote: > > >> Hi Satish, >> >> Sorry for the confusion. >> >> I have attached my makefile. This was used to compile and link my code before >> I used PETSc 3.1. >> >> I have several fortran files. Some contain PETSc commands and need to use >> PETSc include files. Other do not. In the end, they all have to be linked to >> form my final code. I used to copy part of the compiling and linking command >> from "make ex1f" to compile my code: >> >> /app1/mvapich2/current/bin/mpif90 -c -O3 >> -I/home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/include >> -I/home/svu/g0306332/codes/petsc-3.1-p0/include >> -I/home/svu/g0306332/lib/hypre-2.6.0b_atlas5/include >> -I/app1/mvapich2/1.4/include -I/app1/mvapich2/current/include >> -I/home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/include >> -I/home/svu/g0306332/lib/hypre-2.6.0b_atlas5/include >> -I/app1/mvapich2/1.4/include -I/app1/mvapich2/current/include -o ex1f.o >> ex1f.F >> >> /app1/mvapich2/current/bin/mpif90 -O3 -o ex1f ex1f.o >> -Wl,-rpath,/home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib >> -L/home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib -lpetsc >> -Wl,-rpath,/home/svu/g0306332/lib/hypre-2.6.0b_atlas5/lib >> -L/home/svu/g0306332/lib/hypre-2.6.0b_atlas5/lib -lHYPRE -lmpichcxx -lstdc++ >> -Wl,-rpath,/app1/intel/mkl/10.0.5.025/lib/em64t >> -L/app1/intel/mkl/10.0.5.025/lib/em64t -lmkl_lapack -lmkl -lguide -lpthread >> -L/app1/mvapich2/1.4/lib -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -ldl -lmpich >> -lpthread -lrdmacm -libverbs -libumad -lrt -lgcc_s -lmpichf90 >> -L/app1/intel/Compiler/11.0/074/lib/intel64 -lifport -lifcore -limf -lsvml -lm >> -lipgo -lirc -lirc_s -lm -lmpichcxx -lstdc++ -lmpichcxx -lstdc++ -ldl -lmpich >> -lpthread -lrdmacm -libverbs -libumad -lrt -lgcc_s -ldl >> >> However, it doesn't seem to work now. I tried to look at chapter 14 but I >> don't really understand since I know very little about makefiles. >> >> Can you give a rough guide for my case? I will try to read up on the manual to >> get more info. >> >> Moreover, I am confused by the use of: >> >> #include "finclude/petsc.h" etc. >> >> In my fortran files, do I simply use petsc.h or I have to use specific *.h >> depending on what petsc commands are used? >> >> Thank you very much and have a nice day! >> >> Yours sincerely, >> >> Wee-Beng Tay >> >> >> On 12/5/2010 9:55 AM, Satish Balay wrote: >> >>> Lets step back and deal with this primary issue. >>> >>> Have you attempted to use makefiles in petsc format? What problems >>> have you encountered? Perhaps they are fixable. >>> >>> The reason to use this format is to make your makefile as portable as >>> possible [and avoid such issues]. >>> >>> Satish >>> >>> On Tue, 11 May 2010, Satish Balay wrote: >>> >>> >>> >>>> Why not use petsc makefiles - and avoid these hasseles? >>>> >>>> >>>> >>> >>> From balay at mcs.anl.gov Wed May 12 12:56:59 2010 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 12 May 2010 12:56:59 -0500 (CDT) Subject: [petsc-users] Error with file linking In-Reply-To: <4BEAEAAE.9050103@gmail.com> References: <4BE9FFBD.7080705@gmail.com> <4BEADA3C.8000205@gmail.com> <4BEAEAAE.9050103@gmail.com> Message-ID: try the attached makefile. If you have problems copy/paste the *complete* make session from your terminal. Satish On Thu, 13 May 2010, Wee-Beng TAY wrote: > Hi Satish, > > I tried to run it, with the -c present and absent. However, nothing happened. > I run using "make". I also specified "export > PETSC_DIR=/home/svu/g0306332/codes/petsc-3.1-p0/" and "export > PETSC_ARCH=atlas5-mpi-nodebug" before that. > > Did I miss out something? > > Also supposed that I have the same flag for all sources. In that case, how can > I build them? > > Thank you very much and have a nice day! > > Yours sincerely, > > Wee-Beng Tay > > > On 13/5/2010 1:27 AM, Satish Balay wrote: > > Attaching the modified makefile. It might need a couple of iterations > > of fixing. > > > > Hopefully its clear why it would work. Note - if not for different > > flags required for different sources (opt/opt2/opt3) - you can build > > all petsc/non-petsc sources with petsc makefile targets. > > > > > > > In my fortran files, do I simply use petsc.h or I have to use specific *.h > > > depending on what petsc commands are used? > > > > > With petsc-3.1 you just have to use petsc.h for all F77 interface stuff, and > > petsc.h90 for all f90 interface stuff [like VecGetArrayF90()] > > > > Satish > > > > On Thu, 13 May 2010, Wee-Beng TAY wrote: > > > > > > > Hi Satish, > > > > > > Sorry for the confusion. > > > > > > I have attached my makefile. This was used to compile and link my code > > > before > > > I used PETSc 3.1. > > > > > > I have several fortran files. Some contain PETSc commands and need to use > > > PETSc include files. Other do not. In the end, they all have to be linked > > > to > > > form my final code. I used to copy part of the compiling and linking > > > command > > > from "make ex1f" to compile my code: > > > > > > /app1/mvapich2/current/bin/mpif90 -c -O3 > > > -I/home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/include > > > -I/home/svu/g0306332/codes/petsc-3.1-p0/include > > > -I/home/svu/g0306332/lib/hypre-2.6.0b_atlas5/include > > > -I/app1/mvapich2/1.4/include -I/app1/mvapich2/current/include > > > -I/home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/include > > > -I/home/svu/g0306332/lib/hypre-2.6.0b_atlas5/include > > > -I/app1/mvapich2/1.4/include -I/app1/mvapich2/current/include -o ex1f.o > > > ex1f.F > > > > > > /app1/mvapich2/current/bin/mpif90 -O3 -o ex1f ex1f.o > > > -Wl,-rpath,/home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib > > > -L/home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib -lpetsc > > > -Wl,-rpath,/home/svu/g0306332/lib/hypre-2.6.0b_atlas5/lib > > > -L/home/svu/g0306332/lib/hypre-2.6.0b_atlas5/lib -lHYPRE -lmpichcxx > > > -lstdc++ > > > -Wl,-rpath,/app1/intel/mkl/10.0.5.025/lib/em64t > > > -L/app1/intel/mkl/10.0.5.025/lib/em64t -lmkl_lapack -lmkl -lguide > > > -lpthread > > > -L/app1/mvapich2/1.4/lib -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -ldl > > > -lmpich > > > -lpthread -lrdmacm -libverbs -libumad -lrt -lgcc_s -lmpichf90 > > > -L/app1/intel/Compiler/11.0/074/lib/intel64 -lifport -lifcore -limf -lsvml > > > -lm > > > -lipgo -lirc -lirc_s -lm -lmpichcxx -lstdc++ -lmpichcxx -lstdc++ -ldl > > > -lmpich > > > -lpthread -lrdmacm -libverbs -libumad -lrt -lgcc_s -ldl > > > > > > However, it doesn't seem to work now. I tried to look at chapter 14 but I > > > don't really understand since I know very little about makefiles. > > > > > > Can you give a rough guide for my case? I will try to read up on the > > > manual to > > > get more info. > > > > > > Moreover, I am confused by the use of: > > > > > > #include "finclude/petsc.h" etc. > > > > > > In my fortran files, do I simply use petsc.h or I have to use specific *.h > > > depending on what petsc commands are used? > > > > > > Thank you very much and have a nice day! > > > > > > Yours sincerely, > > > > > > Wee-Beng Tay > > > > > > > > > On 12/5/2010 9:55 AM, Satish Balay wrote: > > > > > > > Lets step back and deal with this primary issue. > > > > > > > > Have you attempted to use makefiles in petsc format? What problems > > > > have you encountered? Perhaps they are fixable. > > > > > > > > The reason to use this format is to make your makefile as portable as > > > > possible [and avoid such issues]. > > > > > > > > Satish > > > > > > > > On Tue, 11 May 2010, Satish Balay wrote: > > > > > > > > > > > > > > > > > Why not use petsc makefiles - and avoid these hasseles? > > > > > > > > > > > > > > > > > > > > > > > -------------- next part -------------- ALL: a.out # PETSC_DIR and PETSC_ARCH should be set either in env or in this makeifle PETSC_DIR=/home/svu/g0306332/codes/petsc-3.1-p0 PETSC_ARCH=atlas5-mpi-nodebug CFLAGS = FFLAGS = -r8 -132 -fPIC -g -w90 -w -w95 -O3 # instead of opt2 add these flags to default petsc compile target for .F/.f CPPFLAGS = FPPFLAGS = CLEANFILES = a.out include ${PETSC_DIR}/conf/variables include ${PETSC_DIR}/conf/rules opt = -r8 -w95 -c -O3 opt3 = -w95 -c -O3 libs =/home/svu/g0306332/lib/tecio64.a /home/svu/g0306332/lib/linux64.a OBJS = global.o grid.o flux_area.o bc.o bc_impl.o bc_semi.o set_matrix.o inter_step.o mom_disz.o poisson.o petsc_sub.o cell_data.o fractional.o ns2d_c.o a.out : $(OBJS) $(FLINKER) -o a.out $(OBJS) $(libs) $(PETSC_LIB) #compiled these sources with default petsc targets global.o : global.F petsc_sub.o : petsc_sub.F poisson.o ns2d_c.o : ns2d_c.f90 cell_data.o fractional.o # sources compiled separately - as they need different opt/opt3 compiler flags. grid.o : grid.f90 global.o $(FC) $(opt3) grid.f90 flux_area.o : flux_area.f90 global.o $(FC) $(opt) flux_area.f90 bc.o : bc.f90 flux_area.o $(FC) $(opt) bc.f90 bc_impl.o : bc_impl.f90 bc.o $(FC) $(opt) bc_impl.f90 bc_semi.o : bc_semi.f90 bc.o $(FC) $(opt) bc_semi.f90 set_matrix.o : set_matrix.f90 bc_semi.o bc_impl.o $(FC) $(opt) set_matrix.f90 inter_step.o : inter_step.f90 bc_impl.o $(FC) $(opt) inter_step.f90 mom_disz.o : mom_disz.f90 bc_impl.o bc_semi.o set_matrix.o $(FC) $(opt) mom_disz.f90 poisson.o : poisson.f90 set_matrix.o $(FC) $(opt) poisson.f90 cell_data.o : cell_data.f90 grid.o set_matrix.o $(FC) $(opt) cell_data.f90 fractional.o : fractional.f90 mom_disz.o petsc_sub.o inter_step.o $(FC) $(opt) fractional.f90 From mader at utias.utoronto.ca Wed May 12 13:00:17 2010 From: mader at utias.utoronto.ca (C.A.(Sandy) Mader) Date: Wed, 12 May 2010 14:00:17 -0400 Subject: [petsc-users] Issues with subsequent GMRES Solves In-Reply-To: <669D1DEB-346A-48E6-9CE8-A288162B69C4@mcs.anl.gov> References: <1273685962.20896.40.camel@tuff> <669D1DEB-346A-48E6-9CE8-A288162B69C4@mcs.anl.gov> Message-ID: <1273687217.20896.52.camel@tuff> See responses in text below: On Wed, 2010-05-12 at 12:48 -0500, Barry Smith wrote: > Is it the same matrix or a different matrix that is solved with the "second adjoint system"? Or is it just a different right hand side? It is a different matrix. In this case I solve the CFD at a given flow condition, then solve the adjoint twice (two different RHS) both converge. Then I solve the CFD case for a second flow condition, followed by two more adjoint solutions. Neither of which solve. However, if I solver directly for the second flow condition, the adojint system will solve... > > If it is a different system how do you know it will even converge for the new matrix? Maybe the new matrix is much more difficult? see above answer, if the same system will solve under one set of conditions but not under another... > > Is it possible the "second adjoint system" is singular? The second system will solve if I solve it on its own. > Given that this is a very strong preconditioner the convergence for the first adjoint system is pretty poor. We use an approximate preconditioner to reduce the memory requirements of the solver. I do not have this problem if I use the exact matrix as the preconditioner but it limits my problem size.... > > Does this happen also for much smaller problems? Recommend running much smaller problem to see what happens convergence wise, then run that much smaller problem with -pc_type lu to see if the direct solver handles it happily. I have tried it on a smaller case and this problem does not occur... > > > > On May 12, 2010, at 12:39 PM, C.A.(Sandy) Mader wrote: > > > I am using PETSc to solve an adjoint system for a CFD code. > > The issue I am having is that if I try to run successive cfd/adjoint > > solutions, the adjoint fails to converge after the first cfd/adjoint > > solution cycle. i.e. > > Solve CFD (ok) > > Solve adjoint system with multiple RHS's (ok) > > Solve CFD at new point (ok) > > Solve adjoint system at new point with multiple RHS's (GMRES iterations > > stall) > > > > PETSc is not used to solve the CFD system, only the adjoint system. > > > > I am currently using the following solver options: > > 'Adjoint solver type': 'GMRES',\ > > 'adjoint relative tolerance':1e-6,\ > > 'adjoint absolute tolerance':1e-16,\ > > 'adjoint max iterations': 1500,\ > > 'adjoint restart iteration' : 80,\ > > 'adjoint monitor step': 10,\ > > 'Preconditioner Side': 'right',\ > > 'Matrix Ordering': 'NestedDissection',\ > > 'Global Preconditioner Type': 'Additive Schwartz',\ > > 'Local Preconditioner Type' : 'ILU',\ > > 'ILU Fill Levels': 3,\ > > 'ASM Overlap' : 5,\ > > > > Also, I am using the fortran interface to PETSc. > > > > As a sample, I have included two convergence histories. Both are at the > > same converged CFD point(this case is ~1.2million cell so the PETSc > > system is approx 6million degrees of freedom). The first is an example > > where the adjoint system is solved the first time through the cycle. The > > second is an example of where the adjoint is solved the second time > > through the cycle: > > 1) as first point in cycle > > # ... KSP properties: > > # type : gmres > > # tolerances : rel = 1.0E-10 > > # abs = 1.0E-16 > > # div = 1.0E+05 > > # max.iter. : 1500 > > # precond.type: asm > > Solving ADjoint Transpose with PETSc... > > 0 KSP Residual norm 0.1392E+00 > > 10 KSP Residual norm 0.6394E-01 > > 20 KSP Residual norm 0.6106E-01 > > 30 KSP Residual norm 0.6019E-01 > > 40 KSP Residual norm 0.5941E-01 > > 50 KSP Residual norm 0.5876E-01 > > 60 KSP Residual norm 0.5602E-01 > > 70 KSP Residual norm 0.4915E-01 > > 80 KSP Residual norm 0.3994E-01 > > 90 KSP Residual norm 0.3892E-01 > > 100 KSP Residual norm 0.3854E-01 > > 110 KSP Residual norm 0.3794E-01 > > 120 KSP Residual norm 0.3717E-01 > > 130 KSP Residual norm 0.3630E-01 > > 140 KSP Residual norm 0.3415E-01 > > ... > > 900 KSP Residual norm 0.2437E-09 > > 910 KSP Residual norm 0.1452E-09 > > 920 KSP Residual norm 0.1025E-09 > > 930 KSP Residual norm 0.6875E-10 > > 940 KSP Residual norm 0.4141E-10 > > 950 KSP Residual norm 0.2317E-10 > > 960 KSP Residual norm 0.1559E-10 > > Solving ADjoint Transpose with PETSc time (s) = 391.43 > > Norm of error = 0.1366E-10 Iterations = 965 > > ------------------------------------------------ > > PETSc solver converged after 965 iterations. > > ------------------------------------------------ > > > > 2) As second point in cycle > > # ... KSP properties: > > # type : gmres > > # tolerances : rel = 1.0E-10 > > # abs = 1.0E-16 > > # div = 1.0E+05 > > # max.iter. : 1500 > > # precond.type: asm > > Solving ADjoint Transpose with PETSc... > > 0 KSP Residual norm 0.1392E+00 > > 10 KSP Residual norm 0.6400E-01 > > 20 KSP Residual norm 0.6140E-01 > > 30 KSP Residual norm 0.6060E-01 > > 40 KSP Residual norm 0.5995E-01 > > 50 KSP Residual norm 0.5974E-01 > > 60 KSP Residual norm 0.5971E-01 > > 70 KSP Residual norm 0.5957E-01 > > 80 KSP Residual norm 0.5906E-01 > > 90 KSP Residual norm 0.5906E-01 > > 100 KSP Residual norm 0.5906E-01 > > 110 KSP Residual norm 0.5903E-01 > > 120 KSP Residual norm 0.5901E-01 > > 130 KSP Residual norm 0.5901E-01 > > 140 KSP Residual norm 0.5901E-01 > > ... > > 1400 KSP Residual norm 0.5895E-01 > > 1410 KSP Residual norm 0.5895E-01 > > 1420 KSP Residual norm 0.5895E-01 > > 1430 KSP Residual norm 0.5895E-01 > > 1440 KSP Residual norm 0.5895E-01 > > 1450 KSP Residual norm 0.5895E-01 > > 1460 KSP Residual norm 0.5895E-01 > > 1470 KSP Residual norm 0.5895E-01 > > 1480 KSP Residual norm 0.5895E-01 > > 1490 KSP Residual norm 0.5895E-01 > > 1500 KSP Residual norm 0.5895E-01 > > Solving ADjoint Transpose with PETSc time (s) = 516.59 > > Norm of error = 0.5895E-01 Iterations = 1500 > > ------------------------------------------------ > > PETSc solver converged after 1500 iterations. > > ------------------------------------------------ > > I have tried both resetting the operators between cycles and completely > > destroying the KSP object between cycles. Both give the same result. Is > > there a step I am missing to properly reset the system for the > > subsequent solves? > > > > Thanks in advance for your input... > > > > Sandy > > -- > > C.A.(Sandy) Mader > > Ph.D Candidate > > Multidisciplinary Design Optimization Laboratory > > University of Toronto Institute for Aerospace Studies > > mader at utias.utoronto.ca > > > > From bsmith at mcs.anl.gov Wed May 12 13:05:45 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 12 May 2010 13:05:45 -0500 Subject: [petsc-users] Issues with subsequent GMRES Solves In-Reply-To: <1273687217.20896.52.camel@tuff> References: <1273685962.20896.40.camel@tuff> <669D1DEB-346A-48E6-9CE8-A288162B69C4@mcs.anl.gov> <1273687217.20896.52.camel@tuff> Message-ID: Are you calling KSPSetOperators(ksp,A,B,DIFFERENT_NONZERO_PATTERN); before the second adjoint solve? Or are you trying to reuse a preconditioner from the first set of solves? When the "second adjoint" converges and when it doesn't converge are you using the exact same matrix A, matrix B and right hand side? Barry On May 12, 2010, at 1:00 PM, C.A.(Sandy) Mader wrote: > See responses in text below: > On Wed, 2010-05-12 at 12:48 -0500, Barry Smith wrote: >> Is it the same matrix or a different matrix that is solved with the "second adjoint system"? Or is it just a different right hand side? > It is a different matrix. In this case I solve the CFD at a given flow > condition, then solve the adjoint twice (two different RHS) both > converge. Then I solve the CFD case for a second flow condition, > followed by two more adjoint solutions. Neither of which solve. However, > if I solver directly for the second flow condition, the adojint system > will solve... >> >> If it is a different system how do you know it will even converge for the new matrix? Maybe the new matrix is much more difficult? > see above answer, if the same system will solve under one set of > conditions but not under another... >> >> Is it possible the "second adjoint system" is singular? > The second system will solve if I solve it on its own. > >> Given that this is a very strong preconditioner the convergence for the first adjoint system is pretty poor. > We use an approximate preconditioner to reduce the memory requirements > of the solver. I do not have this problem if I use the exact matrix as > the preconditioner but it limits my problem size.... >> >> Does this happen also for much smaller problems? Recommend running much smaller problem to see what happens convergence wise, then run that much smaller problem with -pc_type lu to see if the direct solver handles it happily. > I have tried it on a smaller case and this problem does not occur... >> >> >> >> On May 12, 2010, at 12:39 PM, C.A.(Sandy) Mader wrote: >> >>> I am using PETSc to solve an adjoint system for a CFD code. >>> The issue I am having is that if I try to run successive cfd/adjoint >>> solutions, the adjoint fails to converge after the first cfd/adjoint >>> solution cycle. i.e. >>> Solve CFD (ok) >>> Solve adjoint system with multiple RHS's (ok) >>> Solve CFD at new point (ok) >>> Solve adjoint system at new point with multiple RHS's (GMRES iterations >>> stall) >>> >>> PETSc is not used to solve the CFD system, only the adjoint system. >>> >>> I am currently using the following solver options: >>> 'Adjoint solver type': 'GMRES',\ >>> 'adjoint relative tolerance':1e-6,\ >>> 'adjoint absolute tolerance':1e-16,\ >>> 'adjoint max iterations': 1500,\ >>> 'adjoint restart iteration' : 80,\ >>> 'adjoint monitor step': 10,\ >>> 'Preconditioner Side': 'right',\ >>> 'Matrix Ordering': 'NestedDissection',\ >>> 'Global Preconditioner Type': 'Additive Schwartz',\ >>> 'Local Preconditioner Type' : 'ILU',\ >>> 'ILU Fill Levels': 3,\ >>> 'ASM Overlap' : 5,\ >>> >>> Also, I am using the fortran interface to PETSc. >>> >>> As a sample, I have included two convergence histories. Both are at the >>> same converged CFD point(this case is ~1.2million cell so the PETSc >>> system is approx 6million degrees of freedom). The first is an example >>> where the adjoint system is solved the first time through the cycle. The >>> second is an example of where the adjoint is solved the second time >>> through the cycle: >>> 1) as first point in cycle >>> # ... KSP properties: >>> # type : gmres >>> # tolerances : rel = 1.0E-10 >>> # abs = 1.0E-16 >>> # div = 1.0E+05 >>> # max.iter. : 1500 >>> # precond.type: asm >>> Solving ADjoint Transpose with PETSc... >>> 0 KSP Residual norm 0.1392E+00 >>> 10 KSP Residual norm 0.6394E-01 >>> 20 KSP Residual norm 0.6106E-01 >>> 30 KSP Residual norm 0.6019E-01 >>> 40 KSP Residual norm 0.5941E-01 >>> 50 KSP Residual norm 0.5876E-01 >>> 60 KSP Residual norm 0.5602E-01 >>> 70 KSP Residual norm 0.4915E-01 >>> 80 KSP Residual norm 0.3994E-01 >>> 90 KSP Residual norm 0.3892E-01 >>> 100 KSP Residual norm 0.3854E-01 >>> 110 KSP Residual norm 0.3794E-01 >>> 120 KSP Residual norm 0.3717E-01 >>> 130 KSP Residual norm 0.3630E-01 >>> 140 KSP Residual norm 0.3415E-01 >>> ... >>> 900 KSP Residual norm 0.2437E-09 >>> 910 KSP Residual norm 0.1452E-09 >>> 920 KSP Residual norm 0.1025E-09 >>> 930 KSP Residual norm 0.6875E-10 >>> 940 KSP Residual norm 0.4141E-10 >>> 950 KSP Residual norm 0.2317E-10 >>> 960 KSP Residual norm 0.1559E-10 >>> Solving ADjoint Transpose with PETSc time (s) = 391.43 >>> Norm of error = 0.1366E-10 Iterations = 965 >>> ------------------------------------------------ >>> PETSc solver converged after 965 iterations. >>> ------------------------------------------------ >>> >>> 2) As second point in cycle >>> # ... KSP properties: >>> # type : gmres >>> # tolerances : rel = 1.0E-10 >>> # abs = 1.0E-16 >>> # div = 1.0E+05 >>> # max.iter. : 1500 >>> # precond.type: asm >>> Solving ADjoint Transpose with PETSc... >>> 0 KSP Residual norm 0.1392E+00 >>> 10 KSP Residual norm 0.6400E-01 >>> 20 KSP Residual norm 0.6140E-01 >>> 30 KSP Residual norm 0.6060E-01 >>> 40 KSP Residual norm 0.5995E-01 >>> 50 KSP Residual norm 0.5974E-01 >>> 60 KSP Residual norm 0.5971E-01 >>> 70 KSP Residual norm 0.5957E-01 >>> 80 KSP Residual norm 0.5906E-01 >>> 90 KSP Residual norm 0.5906E-01 >>> 100 KSP Residual norm 0.5906E-01 >>> 110 KSP Residual norm 0.5903E-01 >>> 120 KSP Residual norm 0.5901E-01 >>> 130 KSP Residual norm 0.5901E-01 >>> 140 KSP Residual norm 0.5901E-01 >>> ... >>> 1400 KSP Residual norm 0.5895E-01 >>> 1410 KSP Residual norm 0.5895E-01 >>> 1420 KSP Residual norm 0.5895E-01 >>> 1430 KSP Residual norm 0.5895E-01 >>> 1440 KSP Residual norm 0.5895E-01 >>> 1450 KSP Residual norm 0.5895E-01 >>> 1460 KSP Residual norm 0.5895E-01 >>> 1470 KSP Residual norm 0.5895E-01 >>> 1480 KSP Residual norm 0.5895E-01 >>> 1490 KSP Residual norm 0.5895E-01 >>> 1500 KSP Residual norm 0.5895E-01 >>> Solving ADjoint Transpose with PETSc time (s) = 516.59 >>> Norm of error = 0.5895E-01 Iterations = 1500 >>> ------------------------------------------------ >>> PETSc solver converged after 1500 iterations. >>> ------------------------------------------------ >>> I have tried both resetting the operators between cycles and completely >>> destroying the KSP object between cycles. Both give the same result. Is >>> there a step I am missing to properly reset the system for the >>> subsequent solves? >>> >>> Thanks in advance for your input... >>> >>> Sandy >>> -- >>> C.A.(Sandy) Mader >>> Ph.D Candidate >>> Multidisciplinary Design Optimization Laboratory >>> University of Toronto Institute for Aerospace Studies >>> mader at utias.utoronto.ca >>> >> >> > From hxie at umn.edu Wed May 12 13:19:51 2010 From: hxie at umn.edu (hxie at umn.edu) Date: 12 May 2010 13:19:51 -0500 Subject: [petsc-users] direct solver for preconditioner In-Reply-To: References: Message-ID: Hi, Is the mumps the best option as a direct solver? Should the size of the linear system be less than 1d5 for mumps? Should I run the following to set mumps as the solver for the preconditioner? mpirun -np 6 ~/bin/cats3d.1 -pc_type asm -ksp_type gmres -sub_ksp_type preonly -sub_pc_type lu -sub_pc_factor_mat_solver_package mumps Bests, Hui From bsmith at mcs.anl.gov Wed May 12 13:24:37 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 12 May 2010 13:24:37 -0500 Subject: [petsc-users] direct solver for preconditioner In-Reply-To: References: Message-ID: <647031DD-31A7-4CF9-A9E6-F56AE20B9A05@mcs.anl.gov> On May 12, 2010, at 1:19 PM, hxie at umn.edu wrote: > Hi, > > Is the mumps the best option as a direct solver? There is no "best option" for any solver; it always depends on the problem you are solving and there is no "recipe" to select the best. We have found that MUMPS generally works better than others. > Should the size of the linear system be less than 1d5 for mumps? There is no hard limit on the size. It again depends on the problem. But, of course, the larger the problem, the more fill and hence longer time it will take or it may run out of memory. > Should I run the following to set mumps as the solver for the preconditioner? > > mpirun -np 6 ~/bin/cats3d.1 -pc_type asm -ksp_type gmres -sub_ksp_type preonly -sub_pc_type lu -sub_pc_factor_mat_solver_package mumps You can certainly use these options. Often, when using overlapping Schwarz we have found the using a direct solver on the subproblems is OVER KILL, that is it takes a lot longer time and doesn't improve the convergence rate of the linear solver over using ILU for each subproblem. This is, of course, problem specific and there may be cases where using a direct solver is best. Barry > > Bests, > Hui From mader at utias.utoronto.ca Wed May 12 13:55:36 2010 From: mader at utias.utoronto.ca (C.A.(Sandy) Mader) Date: Wed, 12 May 2010 14:55:36 -0400 Subject: [petsc-users] Issues with subsequent GMRES Solves In-Reply-To: References: <1273685962.20896.40.camel@tuff> <669D1DEB-346A-48E6-9CE8-A288162B69C4@mcs.anl.gov> <1273687217.20896.52.camel@tuff> Message-ID: <1273690536.20896.57.camel@tuff> See responses in text below: On Wed, 2010-05-12 at 13:05 -0500, Barry Smith wrote: > Are you calling KSPSetOperators(ksp,A,B,DIFFERENT_NONZERO_PATTERN); before the second adjoint solve? Or are you trying to reuse a preconditioner from the first set of solves? I am calling: call KSPSetOperators(ksp,dRdWT,dRdWPreT, & DIFFERENT_NONZERO_PATTERN,PETScIerr) > > When the "second adjoint" converges and when it doesn't converge are you using the exact same matrix A, matrix B and right hand side? As far as I know yes. For the small case (the one that works), I printed them out and compared them and they were identical. I haven't been able to run the same comparison for the larger case... > > Barry > > > > On May 12, 2010, at 1:00 PM, C.A.(Sandy) Mader wrote: > > > See responses in text below: > > On Wed, 2010-05-12 at 12:48 -0500, Barry Smith wrote: > >> Is it the same matrix or a different matrix that is solved with the "second adjoint system"? Or is it just a different right hand side? > > It is a different matrix. In this case I solve the CFD at a given flow > > condition, then solve the adjoint twice (two different RHS) both > > converge. Then I solve the CFD case for a second flow condition, > > followed by two more adjoint solutions. Neither of which solve. However, > > if I solver directly for the second flow condition, the adojint system > > will solve... > >> > >> If it is a different system how do you know it will even converge for the new matrix? Maybe the new matrix is much more difficult? > > see above answer, if the same system will solve under one set of > > conditions but not under another... > >> > >> Is it possible the "second adjoint system" is singular? > > The second system will solve if I solve it on its own. > > > >> Given that this is a very strong preconditioner the convergence for the first adjoint system is pretty poor. > > We use an approximate preconditioner to reduce the memory requirements > > of the solver. I do not have this problem if I use the exact matrix as > > the preconditioner but it limits my problem size.... > >> > >> Does this happen also for much smaller problems? Recommend running much smaller problem to see what happens convergence wise, then run that much smaller problem with -pc_type lu to see if the direct solver handles it happily. > > I have tried it on a smaller case and this problem does not occur... > >> > >> > >> > >> On May 12, 2010, at 12:39 PM, C.A.(Sandy) Mader wrote: > >> > >>> I am using PETSc to solve an adjoint system for a CFD code. > >>> The issue I am having is that if I try to run successive cfd/adjoint > >>> solutions, the adjoint fails to converge after the first cfd/adjoint > >>> solution cycle. i.e. > >>> Solve CFD (ok) > >>> Solve adjoint system with multiple RHS's (ok) > >>> Solve CFD at new point (ok) > >>> Solve adjoint system at new point with multiple RHS's (GMRES iterations > >>> stall) > >>> > >>> PETSc is not used to solve the CFD system, only the adjoint system. > >>> > >>> I am currently using the following solver options: > >>> 'Adjoint solver type': 'GMRES',\ > >>> 'adjoint relative tolerance':1e-6,\ > >>> 'adjoint absolute tolerance':1e-16,\ > >>> 'adjoint max iterations': 1500,\ > >>> 'adjoint restart iteration' : 80,\ > >>> 'adjoint monitor step': 10,\ > >>> 'Preconditioner Side': 'right',\ > >>> 'Matrix Ordering': 'NestedDissection',\ > >>> 'Global Preconditioner Type': 'Additive Schwartz',\ > >>> 'Local Preconditioner Type' : 'ILU',\ > >>> 'ILU Fill Levels': 3,\ > >>> 'ASM Overlap' : 5,\ > >>> > >>> Also, I am using the fortran interface to PETSc. > >>> > >>> As a sample, I have included two convergence histories. Both are at the > >>> same converged CFD point(this case is ~1.2million cell so the PETSc > >>> system is approx 6million degrees of freedom). The first is an example > >>> where the adjoint system is solved the first time through the cycle. The > >>> second is an example of where the adjoint is solved the second time > >>> through the cycle: > >>> 1) as first point in cycle > >>> # ... KSP properties: > >>> # type : gmres > >>> # tolerances : rel = 1.0E-10 > >>> # abs = 1.0E-16 > >>> # div = 1.0E+05 > >>> # max.iter. : 1500 > >>> # precond.type: asm > >>> Solving ADjoint Transpose with PETSc... > >>> 0 KSP Residual norm 0.1392E+00 > >>> 10 KSP Residual norm 0.6394E-01 > >>> 20 KSP Residual norm 0.6106E-01 > >>> 30 KSP Residual norm 0.6019E-01 > >>> 40 KSP Residual norm 0.5941E-01 > >>> 50 KSP Residual norm 0.5876E-01 > >>> 60 KSP Residual norm 0.5602E-01 > >>> 70 KSP Residual norm 0.4915E-01 > >>> 80 KSP Residual norm 0.3994E-01 > >>> 90 KSP Residual norm 0.3892E-01 > >>> 100 KSP Residual norm 0.3854E-01 > >>> 110 KSP Residual norm 0.3794E-01 > >>> 120 KSP Residual norm 0.3717E-01 > >>> 130 KSP Residual norm 0.3630E-01 > >>> 140 KSP Residual norm 0.3415E-01 > >>> ... > >>> 900 KSP Residual norm 0.2437E-09 > >>> 910 KSP Residual norm 0.1452E-09 > >>> 920 KSP Residual norm 0.1025E-09 > >>> 930 KSP Residual norm 0.6875E-10 > >>> 940 KSP Residual norm 0.4141E-10 > >>> 950 KSP Residual norm 0.2317E-10 > >>> 960 KSP Residual norm 0.1559E-10 > >>> Solving ADjoint Transpose with PETSc time (s) = 391.43 > >>> Norm of error = 0.1366E-10 Iterations = 965 > >>> ------------------------------------------------ > >>> PETSc solver converged after 965 iterations. > >>> ------------------------------------------------ > >>> > >>> 2) As second point in cycle > >>> # ... KSP properties: > >>> # type : gmres > >>> # tolerances : rel = 1.0E-10 > >>> # abs = 1.0E-16 > >>> # div = 1.0E+05 > >>> # max.iter. : 1500 > >>> # precond.type: asm > >>> Solving ADjoint Transpose with PETSc... > >>> 0 KSP Residual norm 0.1392E+00 > >>> 10 KSP Residual norm 0.6400E-01 > >>> 20 KSP Residual norm 0.6140E-01 > >>> 30 KSP Residual norm 0.6060E-01 > >>> 40 KSP Residual norm 0.5995E-01 > >>> 50 KSP Residual norm 0.5974E-01 > >>> 60 KSP Residual norm 0.5971E-01 > >>> 70 KSP Residual norm 0.5957E-01 > >>> 80 KSP Residual norm 0.5906E-01 > >>> 90 KSP Residual norm 0.5906E-01 > >>> 100 KSP Residual norm 0.5906E-01 > >>> 110 KSP Residual norm 0.5903E-01 > >>> 120 KSP Residual norm 0.5901E-01 > >>> 130 KSP Residual norm 0.5901E-01 > >>> 140 KSP Residual norm 0.5901E-01 > >>> ... > >>> 1400 KSP Residual norm 0.5895E-01 > >>> 1410 KSP Residual norm 0.5895E-01 > >>> 1420 KSP Residual norm 0.5895E-01 > >>> 1430 KSP Residual norm 0.5895E-01 > >>> 1440 KSP Residual norm 0.5895E-01 > >>> 1450 KSP Residual norm 0.5895E-01 > >>> 1460 KSP Residual norm 0.5895E-01 > >>> 1470 KSP Residual norm 0.5895E-01 > >>> 1480 KSP Residual norm 0.5895E-01 > >>> 1490 KSP Residual norm 0.5895E-01 > >>> 1500 KSP Residual norm 0.5895E-01 > >>> Solving ADjoint Transpose with PETSc time (s) = 516.59 > >>> Norm of error = 0.5895E-01 Iterations = 1500 > >>> ------------------------------------------------ > >>> PETSc solver converged after 1500 iterations. > >>> ------------------------------------------------ > >>> I have tried both resetting the operators between cycles and completely > >>> destroying the KSP object between cycles. Both give the same result. Is > >>> there a step I am missing to properly reset the system for the > >>> subsequent solves? > >>> > >>> Thanks in advance for your input... > >>> > >>> Sandy > >>> -- > >>> C.A.(Sandy) Mader > >>> Ph.D Candidate > >>> Multidisciplinary Design Optimization Laboratory > >>> University of Toronto Institute for Aerospace Studies > >>> mader at utias.utoronto.ca > >>> > >> > >> > > > > From bsmith at mcs.anl.gov Wed May 12 14:02:05 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 12 May 2010 14:02:05 -0500 Subject: [petsc-users] Issues with subsequent GMRES Solves In-Reply-To: <1273690536.20896.57.camel@tuff> References: <1273685962.20896.40.camel@tuff> <669D1DEB-346A-48E6-9CE8-A288162B69C4@mcs.anl.gov> <1273687217.20896.52.camel@tuff> <1273690536.20896.57.camel@tuff> Message-ID: <86EBC94F-5E20-484C-8366-90EF8F1A0622@mcs.anl.gov> On May 12, 2010, at 1:55 PM, C.A.(Sandy) Mader wrote: > See responses in text below: > On Wed, 2010-05-12 at 13:05 -0500, Barry Smith wrote: >> Are you calling KSPSetOperators(ksp,A,B,DIFFERENT_NONZERO_PATTERN); before the second adjoint solve? Or are you trying to reuse a preconditioner from the first set of solves? > I am calling: > call KSPSetOperators(ksp,dRdWT,dRdWPreT, & > DIFFERENT_NONZERO_PATTERN,PETScIerr) When you use this flag it totally rebuilds the preconditioner, there is nothing else you need to do, so there is no "reason" that it shouldn't behave the same whether it is the first or second time you do a solve. Note: you can always destroy the KSP and make a new one to see if that makes a difference (it should not). Have you run the code with valgrind http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind to make sure there is no subtle memory corruption causing problems. Is the first residual norm of the "good" and bad second adjoint iteration the same? What about the second? You can use MatView() with a binary viewer to save all the matrices for the "good" and bad second adjoint, load them into Matlab and see if they are the same. Same with the right hand side. Barry >> >> When the "second adjoint" converges and when it doesn't converge are you using the exact same matrix A, matrix B and right hand side? > As far as I know yes. For the small case (the one that works), I printed > them out and compared them and they were identical. I haven't been able > to run the same comparison for the larger case... >> >> Barry >> >> >> >> On May 12, 2010, at 1:00 PM, C.A.(Sandy) Mader wrote: >> >>> See responses in text below: >>> On Wed, 2010-05-12 at 12:48 -0500, Barry Smith wrote: >>>> Is it the same matrix or a different matrix that is solved with the "second adjoint system"? Or is it just a different right hand side? >>> It is a different matrix. In this case I solve the CFD at a given flow >>> condition, then solve the adjoint twice (two different RHS) both >>> converge. Then I solve the CFD case for a second flow condition, >>> followed by two more adjoint solutions. Neither of which solve. However, >>> if I solver directly for the second flow condition, the adojint system >>> will solve... >>>> >>>> If it is a different system how do you know it will even converge for the new matrix? Maybe the new matrix is much more difficult? >>> see above answer, if the same system will solve under one set of >>> conditions but not under another... >>>> >>>> Is it possible the "second adjoint system" is singular? >>> The second system will solve if I solve it on its own. >>> >>>> Given that this is a very strong preconditioner the convergence for the first adjoint system is pretty poor. >>> We use an approximate preconditioner to reduce the memory requirements >>> of the solver. I do not have this problem if I use the exact matrix as >>> the preconditioner but it limits my problem size.... >>>> >>>> Does this happen also for much smaller problems? Recommend running much smaller problem to see what happens convergence wise, then run that much smaller problem with -pc_type lu to see if the direct solver handles it happily. >>> I have tried it on a smaller case and this problem does not occur... >>>> >>>> >>>> >>>> On May 12, 2010, at 12:39 PM, C.A.(Sandy) Mader wrote: >>>> >>>>> I am using PETSc to solve an adjoint system for a CFD code. >>>>> The issue I am having is that if I try to run successive cfd/adjoint >>>>> solutions, the adjoint fails to converge after the first cfd/adjoint >>>>> solution cycle. i.e. >>>>> Solve CFD (ok) >>>>> Solve adjoint system with multiple RHS's (ok) >>>>> Solve CFD at new point (ok) >>>>> Solve adjoint system at new point with multiple RHS's (GMRES iterations >>>>> stall) >>>>> >>>>> PETSc is not used to solve the CFD system, only the adjoint system. >>>>> >>>>> I am currently using the following solver options: >>>>> 'Adjoint solver type': 'GMRES',\ >>>>> 'adjoint relative tolerance':1e-6,\ >>>>> 'adjoint absolute tolerance':1e-16,\ >>>>> 'adjoint max iterations': 1500,\ >>>>> 'adjoint restart iteration' : 80,\ >>>>> 'adjoint monitor step': 10,\ >>>>> 'Preconditioner Side': 'right',\ >>>>> 'Matrix Ordering': 'NestedDissection',\ >>>>> 'Global Preconditioner Type': 'Additive Schwartz',\ >>>>> 'Local Preconditioner Type' : 'ILU',\ >>>>> 'ILU Fill Levels': 3,\ >>>>> 'ASM Overlap' : 5,\ >>>>> >>>>> Also, I am using the fortran interface to PETSc. >>>>> >>>>> As a sample, I have included two convergence histories. Both are at the >>>>> same converged CFD point(this case is ~1.2million cell so the PETSc >>>>> system is approx 6million degrees of freedom). The first is an example >>>>> where the adjoint system is solved the first time through the cycle. The >>>>> second is an example of where the adjoint is solved the second time >>>>> through the cycle: >>>>> 1) as first point in cycle >>>>> # ... KSP properties: >>>>> # type : gmres >>>>> # tolerances : rel = 1.0E-10 >>>>> # abs = 1.0E-16 >>>>> # div = 1.0E+05 >>>>> # max.iter. : 1500 >>>>> # precond.type: asm >>>>> Solving ADjoint Transpose with PETSc... >>>>> 0 KSP Residual norm 0.1392E+00 >>>>> 10 KSP Residual norm 0.6394E-01 >>>>> 20 KSP Residual norm 0.6106E-01 >>>>> 30 KSP Residual norm 0.6019E-01 >>>>> 40 KSP Residual norm 0.5941E-01 >>>>> 50 KSP Residual norm 0.5876E-01 >>>>> 60 KSP Residual norm 0.5602E-01 >>>>> 70 KSP Residual norm 0.4915E-01 >>>>> 80 KSP Residual norm 0.3994E-01 >>>>> 90 KSP Residual norm 0.3892E-01 >>>>> 100 KSP Residual norm 0.3854E-01 >>>>> 110 KSP Residual norm 0.3794E-01 >>>>> 120 KSP Residual norm 0.3717E-01 >>>>> 130 KSP Residual norm 0.3630E-01 >>>>> 140 KSP Residual norm 0.3415E-01 >>>>> ... >>>>> 900 KSP Residual norm 0.2437E-09 >>>>> 910 KSP Residual norm 0.1452E-09 >>>>> 920 KSP Residual norm 0.1025E-09 >>>>> 930 KSP Residual norm 0.6875E-10 >>>>> 940 KSP Residual norm 0.4141E-10 >>>>> 950 KSP Residual norm 0.2317E-10 >>>>> 960 KSP Residual norm 0.1559E-10 >>>>> Solving ADjoint Transpose with PETSc time (s) = 391.43 >>>>> Norm of error = 0.1366E-10 Iterations = 965 >>>>> ------------------------------------------------ >>>>> PETSc solver converged after 965 iterations. >>>>> ------------------------------------------------ >>>>> >>>>> 2) As second point in cycle >>>>> # ... KSP properties: >>>>> # type : gmres >>>>> # tolerances : rel = 1.0E-10 >>>>> # abs = 1.0E-16 >>>>> # div = 1.0E+05 >>>>> # max.iter. : 1500 >>>>> # precond.type: asm >>>>> Solving ADjoint Transpose with PETSc... >>>>> 0 KSP Residual norm 0.1392E+00 >>>>> 10 KSP Residual norm 0.6400E-01 >>>>> 20 KSP Residual norm 0.6140E-01 >>>>> 30 KSP Residual norm 0.6060E-01 >>>>> 40 KSP Residual norm 0.5995E-01 >>>>> 50 KSP Residual norm 0.5974E-01 >>>>> 60 KSP Residual norm 0.5971E-01 >>>>> 70 KSP Residual norm 0.5957E-01 >>>>> 80 KSP Residual norm 0.5906E-01 >>>>> 90 KSP Residual norm 0.5906E-01 >>>>> 100 KSP Residual norm 0.5906E-01 >>>>> 110 KSP Residual norm 0.5903E-01 >>>>> 120 KSP Residual norm 0.5901E-01 >>>>> 130 KSP Residual norm 0.5901E-01 >>>>> 140 KSP Residual norm 0.5901E-01 >>>>> ... >>>>> 1400 KSP Residual norm 0.5895E-01 >>>>> 1410 KSP Residual norm 0.5895E-01 >>>>> 1420 KSP Residual norm 0.5895E-01 >>>>> 1430 KSP Residual norm 0.5895E-01 >>>>> 1440 KSP Residual norm 0.5895E-01 >>>>> 1450 KSP Residual norm 0.5895E-01 >>>>> 1460 KSP Residual norm 0.5895E-01 >>>>> 1470 KSP Residual norm 0.5895E-01 >>>>> 1480 KSP Residual norm 0.5895E-01 >>>>> 1490 KSP Residual norm 0.5895E-01 >>>>> 1500 KSP Residual norm 0.5895E-01 >>>>> Solving ADjoint Transpose with PETSc time (s) = 516.59 >>>>> Norm of error = 0.5895E-01 Iterations = 1500 >>>>> ------------------------------------------------ >>>>> PETSc solver converged after 1500 iterations. >>>>> ------------------------------------------------ >>>>> I have tried both resetting the operators between cycles and completely >>>>> destroying the KSP object between cycles. Both give the same result. Is >>>>> there a step I am missing to properly reset the system for the >>>>> subsequent solves? >>>>> >>>>> Thanks in advance for your input... >>>>> >>>>> Sandy >>>>> -- >>>>> C.A.(Sandy) Mader >>>>> Ph.D Candidate >>>>> Multidisciplinary Design Optimization Laboratory >>>>> University of Toronto Institute for Aerospace Studies >>>>> mader at utias.utoronto.ca >>>>> >>>> >>>> >>> >> >> > From fischej at umich.edu Wed May 12 14:54:42 2010 From: fischej at umich.edu (John-Michael Fischer) Date: Wed, 12 May 2010 15:54:42 -0400 Subject: [petsc-users] using petsc functions in structs? Message-ID: Hi, I'm fighting this issue. In the follow code which is macro for a struct divide I get three errors about casting PetscErrorCode types into the name of the struct. This is the code struct IMPACT_Moment Divide(struct IMPACT_Moment * M1, struct IMPACT_Moment * M3) { // does M1./M3 and returns M2 PetscErrorCode ierr; struct IMPACT_Moment M2; M2.Nx = M1->Nx; M2.Ny = M1->Ny; M2.coords[0] = M1->coords[0]; M2.coords[1] = M1->coords[1]; 78: ierr = VecCopy(M1->xaxis, M2.xaxis);CHKERRQ(ierr); 79: ierr = VecCopy(M1->yaxis, M2.yaxis);CHKERRQ(ierr); 81: ierr = MatCopy(M1->values, M2.values, DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr); ...... goes on and these are the only three error messages I get IMPACTA_Moments.h: In function ?IMPACT_Moment Divide(IMPACT_Moment*, IMPACT_Moment*)?: IMPACTA_Moments.h:78: error: conversion from ?PetscErrorCode? to non-scalar type ?IMPACT_Moment? requested IMPACTA_Moments.h:79: error: conversion from ?PetscErrorCode? to non-scalar type ?IMPACT_Moment? requested IMPACTA_Moments.h:81: error: conversion from ?PetscErrorCode? to non-scalar type ?IMPACT_Moment? requested Any thoughts? JM -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Wed May 12 15:00:37 2010 From: jed at 59A2.org (Jed Brown) Date: Wed, 12 May 2010 22:00:37 +0200 Subject: [petsc-users] using petsc functions in structs? In-Reply-To: References: Message-ID: <874oicriu2.fsf@59A2.org> On Wed, 12 May 2010 15:54:42 -0400, John-Michael Fischer wrote: > Hi, I'm fighting this issue. > > In the follow code which is macro for a struct divide I get three errors about casting PetscErrorCode types into the name of the struct. > > This is the code > struct IMPACT_Moment Divide(struct IMPACT_Moment * M1, struct IMPACT_Moment * M3) > { // does M1./M3 and returns M2 > PetscErrorCode ierr; > struct IMPACT_Moment M2; > M2.Nx = M1->Nx; > M2.Ny = M1->Ny; > M2.coords[0] = M1->coords[0]; > M2.coords[1] = M1->coords[1]; > > 78: ierr = VecCopy(M1->xaxis, M2.xaxis);CHKERRQ(ierr); > 79: ierr = VecCopy(M1->yaxis, M2.yaxis);CHKERRQ(ierr); > > 81: ierr = MatCopy(M1->values, M2.values, DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr); > ...... goes on > > > and these are the only three error messages I get > > IMPACTA_Moments.h: In function ?IMPACT_Moment Divide(IMPACT_Moment*, IMPACT_Moment*)?: > IMPACTA_Moments.h:78: error: conversion from ?PetscErrorCode? to non-scalar type ?IMPACT_Moment? requested > IMPACTA_Moments.h:79: error: conversion from ?PetscErrorCode? to non-scalar type ?IMPACT_Moment? requested > IMPACTA_Moments.h:81: error: conversion from ?PetscErrorCode? to non-scalar type ?IMPACT_Moment? requested The CHKERRQ macro expects that the enclosing function returns PetscErrorCode. Because your code has to handle errors somehow, your functions should either return PetscErrorCode (int) or you should use CHKERRXX and C++ exceptions (if you are writing C++ and prefer to use exceptions). Jed From fischej at umich.edu Wed May 12 15:01:34 2010 From: fischej at umich.edu (John-Michael Fischer) Date: Wed, 12 May 2010 16:01:34 -0400 Subject: [petsc-users] using petsc functions in structs? In-Reply-To: <874oicriu2.fsf@59A2.org> References: <874oicriu2.fsf@59A2.org> Message-ID: <7897F15A-0CD7-4E9D-A853-8F0749CD0341@umich.edu> you just saved me a lot of time -- thanks. On May 12, 2010, at 4:00 PM, Jed Brown wrote: > On Wed, 12 May 2010 15:54:42 -0400, John-Michael Fischer wrote: >> Hi, I'm fighting this issue. >> >> In the follow code which is macro for a struct divide I get three errors about casting PetscErrorCode types into the name of the struct. >> >> This is the code >> struct IMPACT_Moment Divide(struct IMPACT_Moment * M1, struct IMPACT_Moment * M3) >> { // does M1./M3 and returns M2 >> PetscErrorCode ierr; >> struct IMPACT_Moment M2; >> M2.Nx = M1->Nx; >> M2.Ny = M1->Ny; >> M2.coords[0] = M1->coords[0]; >> M2.coords[1] = M1->coords[1]; >> >> 78: ierr = VecCopy(M1->xaxis, M2.xaxis);CHKERRQ(ierr); >> 79: ierr = VecCopy(M1->yaxis, M2.yaxis);CHKERRQ(ierr); >> >> 81: ierr = MatCopy(M1->values, M2.values, DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr); >> ...... goes on >> >> >> and these are the only three error messages I get >> >> IMPACTA_Moments.h: In function ?IMPACT_Moment Divide(IMPACT_Moment*, IMPACT_Moment*)?: >> IMPACTA_Moments.h:78: error: conversion from ?PetscErrorCode? to non-scalar type ?IMPACT_Moment? requested >> IMPACTA_Moments.h:79: error: conversion from ?PetscErrorCode? to non-scalar type ?IMPACT_Moment? requested >> IMPACTA_Moments.h:81: error: conversion from ?PetscErrorCode? to non-scalar type ?IMPACT_Moment? requested > > The CHKERRQ macro expects that the enclosing function returns > PetscErrorCode. Because your code has to handle errors somehow, your > functions should either return PetscErrorCode (int) or you should use > CHKERRXX and C++ exceptions (if you are writing C++ and prefer to use > exceptions). > > Jed > > From hxie at umn.edu Wed May 12 17:18:29 2010 From: hxie at umn.edu (hxie at umn.edu) Date: 12 May 2010 17:18:29 -0500 Subject: [petsc-users] Change entries after MatCopy In-Reply-To: References: Message-ID: Hi, If I use MatCopy to get a new matrix, can i use MatSetValues to change some entries of that matrix and do the assemble after that? Or if I just need to change some diagonal entries, can I use MatDiagonalSet to change the entries without assembling? Thanks. Bests, Hui From dalcinl at gmail.com Wed May 12 17:24:42 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 12 May 2010 19:24:42 -0300 Subject: [petsc-users] Change entries after MatCopy In-Reply-To: References: Message-ID: On 12 May 2010 19:18, wrote: > Hi, > > If I use MatCopy to get a new matrix, can i use MatSetValues to change some > entries of that matrix and do the assemble after that? Or if I just need to > change some diagonal entries, can I use MatDiagonalSet to change the entries > without assembling? Thanks. > Yes and Yes. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From mader at utias.utoronto.ca Thu May 13 09:20:49 2010 From: mader at utias.utoronto.ca (C.A.(Sandy) Mader) Date: Thu, 13 May 2010 10:20:49 -0400 Subject: [petsc-users] Issues with subsequent GMRES Solves In-Reply-To: <86EBC94F-5E20-484C-8366-90EF8F1A0622@mcs.anl.gov> References: <1273685962.20896.40.camel@tuff> <669D1DEB-346A-48E6-9CE8-A288162B69C4@mcs.anl.gov> <1273687217.20896.52.camel@tuff> <1273690536.20896.57.camel@tuff> <86EBC94F-5E20-484C-8366-90EF8F1A0622@mcs.anl.gov> Message-ID: <1273760449.26903.4.camel@tuff> I was able to get a version of the code working last night by destroying all of the PETSc objects between subsequent adjoint setup calls. Based on some of my previous results I suspect that I am having an issue with resetting the matrices used in the linear solve. I am running some more tests now to confirm this... Sandy On Wed, 2010-05-12 at 14:02 -0500, Barry Smith wrote: > On May 12, 2010, at 1:55 PM, C.A.(Sandy) Mader wrote: > > > See responses in text below: > > On Wed, 2010-05-12 at 13:05 -0500, Barry Smith wrote: > >> Are you calling KSPSetOperators(ksp,A,B,DIFFERENT_NONZERO_PATTERN); before the second adjoint solve? Or are you trying to reuse a preconditioner from the first set of solves? > > I am calling: > > call KSPSetOperators(ksp,dRdWT,dRdWPreT, & > > DIFFERENT_NONZERO_PATTERN,PETScIerr) > > When you use this flag it totally rebuilds the preconditioner, there is nothing else you need to do, so there is no "reason" that it shouldn't behave the same whether it is the first or second time you do a solve. Note: you can always destroy the KSP and make a new one to see if that makes a difference (it should not). > > Have you run the code with valgrind http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind to make sure there is no subtle memory corruption causing problems. > > Is the first residual norm of the "good" and bad second adjoint iteration the same? What about the second? > > You can use MatView() with a binary viewer to save all the matrices for the "good" and bad second adjoint, load them into Matlab and see if they are the same. Same with the right hand side. > > > Barry > > >> > >> When the "second adjoint" converges and when it doesn't converge are you using the exact same matrix A, matrix B and right hand side? > > As far as I know yes. For the small case (the one that works), I printed > > them out and compared them and they were identical. I haven't been able > > to run the same comparison for the larger case... > >> > >> Barry > >> > >> > >> > >> On May 12, 2010, at 1:00 PM, C.A.(Sandy) Mader wrote: > >> > >>> See responses in text below: > >>> On Wed, 2010-05-12 at 12:48 -0500, Barry Smith wrote: > >>>> Is it the same matrix or a different matrix that is solved with the "second adjoint system"? Or is it just a different right hand side? > >>> It is a different matrix. In this case I solve the CFD at a given flow > >>> condition, then solve the adjoint twice (two different RHS) both > >>> converge. Then I solve the CFD case for a second flow condition, > >>> followed by two more adjoint solutions. Neither of which solve. However, > >>> if I solver directly for the second flow condition, the adojint system > >>> will solve... > >>>> > >>>> If it is a different system how do you know it will even converge for the new matrix? Maybe the new matrix is much more difficult? > >>> see above answer, if the same system will solve under one set of > >>> conditions but not under another... > >>>> > >>>> Is it possible the "second adjoint system" is singular? > >>> The second system will solve if I solve it on its own. > >>> > >>>> Given that this is a very strong preconditioner the convergence for the first adjoint system is pretty poor. > >>> We use an approximate preconditioner to reduce the memory requirements > >>> of the solver. I do not have this problem if I use the exact matrix as > >>> the preconditioner but it limits my problem size.... > >>>> > >>>> Does this happen also for much smaller problems? Recommend running much smaller problem to see what happens convergence wise, then run that much smaller problem with -pc_type lu to see if the direct solver handles it happily. > >>> I have tried it on a smaller case and this problem does not occur... > >>>> > >>>> > >>>> > >>>> On May 12, 2010, at 12:39 PM, C.A.(Sandy) Mader wrote: > >>>> > >>>>> I am using PETSc to solve an adjoint system for a CFD code. > >>>>> The issue I am having is that if I try to run successive cfd/adjoint > >>>>> solutions, the adjoint fails to converge after the first cfd/adjoint > >>>>> solution cycle. i.e. > >>>>> Solve CFD (ok) > >>>>> Solve adjoint system with multiple RHS's (ok) > >>>>> Solve CFD at new point (ok) > >>>>> Solve adjoint system at new point with multiple RHS's (GMRES iterations > >>>>> stall) > >>>>> > >>>>> PETSc is not used to solve the CFD system, only the adjoint system. > >>>>> > >>>>> I am currently using the following solver options: > >>>>> 'Adjoint solver type': 'GMRES',\ > >>>>> 'adjoint relative tolerance':1e-6,\ > >>>>> 'adjoint absolute tolerance':1e-16,\ > >>>>> 'adjoint max iterations': 1500,\ > >>>>> 'adjoint restart iteration' : 80,\ > >>>>> 'adjoint monitor step': 10,\ > >>>>> 'Preconditioner Side': 'right',\ > >>>>> 'Matrix Ordering': 'NestedDissection',\ > >>>>> 'Global Preconditioner Type': 'Additive Schwartz',\ > >>>>> 'Local Preconditioner Type' : 'ILU',\ > >>>>> 'ILU Fill Levels': 3,\ > >>>>> 'ASM Overlap' : 5,\ > >>>>> > >>>>> Also, I am using the fortran interface to PETSc. > >>>>> > >>>>> As a sample, I have included two convergence histories. Both are at the > >>>>> same converged CFD point(this case is ~1.2million cell so the PETSc > >>>>> system is approx 6million degrees of freedom). The first is an example > >>>>> where the adjoint system is solved the first time through the cycle. The > >>>>> second is an example of where the adjoint is solved the second time > >>>>> through the cycle: > >>>>> 1) as first point in cycle > >>>>> # ... KSP properties: > >>>>> # type : gmres > >>>>> # tolerances : rel = 1.0E-10 > >>>>> # abs = 1.0E-16 > >>>>> # div = 1.0E+05 > >>>>> # max.iter. : 1500 > >>>>> # precond.type: asm > >>>>> Solving ADjoint Transpose with PETSc... > >>>>> 0 KSP Residual norm 0.1392E+00 > >>>>> 10 KSP Residual norm 0.6394E-01 > >>>>> 20 KSP Residual norm 0.6106E-01 > >>>>> 30 KSP Residual norm 0.6019E-01 > >>>>> 40 KSP Residual norm 0.5941E-01 > >>>>> 50 KSP Residual norm 0.5876E-01 > >>>>> 60 KSP Residual norm 0.5602E-01 > >>>>> 70 KSP Residual norm 0.4915E-01 > >>>>> 80 KSP Residual norm 0.3994E-01 > >>>>> 90 KSP Residual norm 0.3892E-01 > >>>>> 100 KSP Residual norm 0.3854E-01 > >>>>> 110 KSP Residual norm 0.3794E-01 > >>>>> 120 KSP Residual norm 0.3717E-01 > >>>>> 130 KSP Residual norm 0.3630E-01 > >>>>> 140 KSP Residual norm 0.3415E-01 > >>>>> ... > >>>>> 900 KSP Residual norm 0.2437E-09 > >>>>> 910 KSP Residual norm 0.1452E-09 > >>>>> 920 KSP Residual norm 0.1025E-09 > >>>>> 930 KSP Residual norm 0.6875E-10 > >>>>> 940 KSP Residual norm 0.4141E-10 > >>>>> 950 KSP Residual norm 0.2317E-10 > >>>>> 960 KSP Residual norm 0.1559E-10 > >>>>> Solving ADjoint Transpose with PETSc time (s) = 391.43 > >>>>> Norm of error = 0.1366E-10 Iterations = 965 > >>>>> ------------------------------------------------ > >>>>> PETSc solver converged after 965 iterations. > >>>>> ------------------------------------------------ > >>>>> > >>>>> 2) As second point in cycle > >>>>> # ... KSP properties: > >>>>> # type : gmres > >>>>> # tolerances : rel = 1.0E-10 > >>>>> # abs = 1.0E-16 > >>>>> # div = 1.0E+05 > >>>>> # max.iter. : 1500 > >>>>> # precond.type: asm > >>>>> Solving ADjoint Transpose with PETSc... > >>>>> 0 KSP Residual norm 0.1392E+00 > >>>>> 10 KSP Residual norm 0.6400E-01 > >>>>> 20 KSP Residual norm 0.6140E-01 > >>>>> 30 KSP Residual norm 0.6060E-01 > >>>>> 40 KSP Residual norm 0.5995E-01 > >>>>> 50 KSP Residual norm 0.5974E-01 > >>>>> 60 KSP Residual norm 0.5971E-01 > >>>>> 70 KSP Residual norm 0.5957E-01 > >>>>> 80 KSP Residual norm 0.5906E-01 > >>>>> 90 KSP Residual norm 0.5906E-01 > >>>>> 100 KSP Residual norm 0.5906E-01 > >>>>> 110 KSP Residual norm 0.5903E-01 > >>>>> 120 KSP Residual norm 0.5901E-01 > >>>>> 130 KSP Residual norm 0.5901E-01 > >>>>> 140 KSP Residual norm 0.5901E-01 > >>>>> ... > >>>>> 1400 KSP Residual norm 0.5895E-01 > >>>>> 1410 KSP Residual norm 0.5895E-01 > >>>>> 1420 KSP Residual norm 0.5895E-01 > >>>>> 1430 KSP Residual norm 0.5895E-01 > >>>>> 1440 KSP Residual norm 0.5895E-01 > >>>>> 1450 KSP Residual norm 0.5895E-01 > >>>>> 1460 KSP Residual norm 0.5895E-01 > >>>>> 1470 KSP Residual norm 0.5895E-01 > >>>>> 1480 KSP Residual norm 0.5895E-01 > >>>>> 1490 KSP Residual norm 0.5895E-01 > >>>>> 1500 KSP Residual norm 0.5895E-01 > >>>>> Solving ADjoint Transpose with PETSc time (s) = 516.59 > >>>>> Norm of error = 0.5895E-01 Iterations = 1500 > >>>>> ------------------------------------------------ > >>>>> PETSc solver converged after 1500 iterations. > >>>>> ------------------------------------------------ > >>>>> I have tried both resetting the operators between cycles and completely > >>>>> destroying the KSP object between cycles. Both give the same result. Is > >>>>> there a step I am missing to properly reset the system for the > >>>>> subsequent solves? > >>>>> > >>>>> Thanks in advance for your input... > >>>>> > >>>>> Sandy > >>>>> -- > >>>>> C.A.(Sandy) Mader > >>>>> Ph.D Candidate > >>>>> Multidisciplinary Design Optimization Laboratory > >>>>> University of Toronto Institute for Aerospace Studies > >>>>> mader at utias.utoronto.ca > >>>>> > >>>> > >>>> > >>> > >> > >> > > > > From jed at 59A2.org Thu May 13 09:26:25 2010 From: jed at 59A2.org (Jed Brown) Date: Thu, 13 May 2010 16:26:25 +0200 Subject: [petsc-users] Issues with subsequent GMRES Solves In-Reply-To: <1273760449.26903.4.camel@tuff> References: <1273685962.20896.40.camel@tuff> <669D1DEB-346A-48E6-9CE8-A288162B69C4@mcs.anl.gov> <1273687217.20896.52.camel@tuff> <1273690536.20896.57.camel@tuff> <86EBC94F-5E20-484C-8366-90EF8F1A0622@mcs.anl.gov> <1273760449.26903.4.camel@tuff> Message-ID: <87632rri7i.fsf@59A2.org> On Thu, 13 May 2010 10:20:49 -0400, "C.A.(Sandy) Mader" wrote: > I was able to get a version of the code working last night by destroying > all of the PETSc objects between subsequent adjoint setup calls. Based > on some of my previous results I suspect that I am having an issue with > resetting the matrices used in the linear solve. I am running some more > tests now to confirm this... Try reusing all the data structures, but place a MatZeroEntries() before your reassembly. Perhaps the reassembly isn't setting all the same entries, or is using ADD_VALUES? Jed From elroal at upvnet.upv.es Thu May 13 05:10:10 2010 From: elroal at upvnet.upv.es (Eloy Romero) Date: Thu, 13 May 2010 12:10:10 +0200 Subject: [petsc-users] Error using the same ksp with different operator sizes Message-ID: <4BEBD002.4060904@upvnet.upv.es> Dear all, I have a code that using the same ksp, solves some matrices with different size in that way: ierr = KSPCreate(PETSC_COMM_WORLD, &ksp); ... ierr = KSPSetOperators(ksp, A, A, DIFFERENT_NONZERO_PATTERN); CHKERRQ(ierr); ierr = KSPSolve(ksp, v0, v1); CHKERRQ(ierr); ... ierr = KSPSetOperators(ksp, B, B, DIFFERENT_NONZERO_PATTERN); CHKERRQ(ierr); ierr = KSPSolve(ksp, w0, w1); CHKERRQ(ierr); ... The KSPSolve is called with vectors of appropriated length. However I obtain the following error in the second call to KSPSolve: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Nonconforming object sizes! [0]PETSC ERROR: Mat mat,Vec x: global dim 1083 66! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 0, unknown [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./veccomp_ex2 on a linux-gnu named yah by eloy Thu May 13 11:27:31 2010 [0]PETSC ERROR: Libraries linked from /usr/src/petsc-release-3.1/linux-gnu-c-debug/lib [0]PETSC ERROR: Configure run at Wed Mar 31 13:34:33 2010 [0]PETSC ERROR: Configure options --with-blas-lapack-dir="[/usr/lib]" --with-clanguage=C --with-shared=0 --with-mpi=1 --with-cc=mpicc --with-cxx=mpicxx --download-hypre=ifneeded --with-hypre=1 --with-fc=mpif90 --with-mumps --download-mumps=ifneeded --download-blacs=ifneeded --download-scalapack=ifneeded --download-parmetis=ifneeded --download-superlu_dist=ifneeded --with-superlu_dist --with-ml --download-ml=ifneeded --with-mpi-dir=/usr [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatMult() line 1889 in src/mat/interface/matrix.c [0]PETSC ERROR: PCApplyBAorAB() line 585 in src/ksp/pc/interface/precon.c [0]PETSC ERROR: GMREScycle() line 161 in src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: KSPSolve_GMRES() line 241 in src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: main() line 59 in veccomp_ex2.c I don't find in the reference that one can or cannot use the ksp with matrices of different size, so I don't know if it is really a bug. In my application, a ksp configured by the user is given, and I would like to use that configuration for solving all the linear systems. Creating new ksps and configuring them with KSPSetFromOptions is not a complete solution because the user could modify the ksp by code. From knepley at gmail.com Thu May 13 09:41:03 2010 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 13 May 2010 10:41:03 -0400 Subject: [petsc-users] Error using the same ksp with different operator sizes In-Reply-To: <4BEBD002.4060904@upvnet.upv.es> References: <4BEBD002.4060904@upvnet.upv.es> Message-ID: On Thu, May 13, 2010 at 6:10 AM, Eloy Romero wrote: > Dear all, > > I have a code that using the same ksp, solves some matrices with different > size in that way: > > ierr = KSPCreate(PETSC_COMM_WORLD, &ksp); > ... > ierr = KSPSetOperators(ksp, A, A, DIFFERENT_NONZERO_PATTERN); > CHKERRQ(ierr); > ierr = KSPSolve(ksp, v0, v1); CHKERRQ(ierr); > ... > ierr = KSPSetOperators(ksp, B, B, DIFFERENT_NONZERO_PATTERN); > CHKERRQ(ierr); > ierr = KSPSolve(ksp, w0, w1); CHKERRQ(ierr); > ... > > The KSPSolve is called with vectors of appropriated length. However I > obtain the following error in the second call to KSPSolve: > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Nonconforming object sizes! > [0]PETSC ERROR: Mat mat,Vec x: global dim 1083 66! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 0, unknown > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./veccomp_ex2 on a linux-gnu named yah by eloy Thu May 13 > 11:27:31 2010 > [0]PETSC ERROR: Libraries linked from > /usr/src/petsc-release-3.1/linux-gnu-c-debug/lib > [0]PETSC ERROR: Configure run at Wed Mar 31 13:34:33 2010 > [0]PETSC ERROR: Configure options --with-blas-lapack-dir="[/usr/lib]" > --with-clanguage=C --with-shared=0 --with-mpi=1 --with-cc=mpicc > --with-cxx=mpicxx --download-hypre=ifneeded --with-hypre=1 --with-fc=mpif90 > --with-mumps --download-mumps=ifneeded --download-blacs=ifneeded > --download-scalapack=ifneeded --download-parmetis=ifneeded > --download-superlu_dist=ifneeded --with-superlu_dist --with-ml > --download-ml=ifneeded --with-mpi-dir=/usr > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: MatMult() line 1889 in src/mat/interface/matrix.c > [0]PETSC ERROR: PCApplyBAorAB() line 585 in src/ksp/pc/interface/precon.c > [0]PETSC ERROR: GMREScycle() line 161 in src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: KSPSolve_GMRES() line 241 in > src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: main() line 59 in veccomp_ex2.c > > I don't find in the reference that one can or cannot use the ksp with > matrices of different size, so I don't know if it is really a bug. > You can only use a KSP to solve one system size. > In my application, a ksp configured by the user is given, and I would like > to use that configuration for solving all the linear systems. Creating new > ksps and configuring them with KSPSetFromOptions is not a complete solution > because the user could modify the ksp by code. > You can replicate all the KSP settings in the new KSP. They are all available by the API. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From elroal at upvnet.upv.es Thu May 13 10:07:13 2010 From: elroal at upvnet.upv.es (Eloy Romero) Date: Thu, 13 May 2010 17:07:13 +0200 Subject: [petsc-users] Error using the same ksp with different operator sizes In-Reply-To: References: <4BEBD002.4060904@upvnet.upv.es> Message-ID: <4BEC15A1.7070309@upvnet.upv.es> On 05/13/2010 04:41 PM, Matthew Knepley wrote: > > In my application, a ksp configured by the user is given, and I > would like to use that configuration for solving all the linear > systems. Creating new ksps and configuring them with > KSPSetFromOptions is not a complete solution because the user > could modify the ksp by code. > > > You can replicate all the KSP settings in the new KSP. They are all > available by the API. > > Matt > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener Are you suggesting that I have to call _all_ KSPGet* routines and then the corresponding KSPSet* routines in order to copy a ksp object? Thanks for your time, Eloy -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu May 13 10:23:46 2010 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 13 May 2010 11:23:46 -0400 Subject: [petsc-users] Error using the same ksp with different operator sizes In-Reply-To: <4BEC15A1.7070309@upvnet.upv.es> References: <4BEBD002.4060904@upvnet.upv.es> <4BEC15A1.7070309@upvnet.upv.es> Message-ID: On Thu, May 13, 2010 at 11:07 AM, Eloy Romero wrote: > On 05/13/2010 04:41 PM, Matthew Knepley wrote: > > > >> In my application, a ksp configured by the user is given, and I would like >> to use that configuration for solving all the linear systems. Creating new >> ksps and configuring them with KSPSetFromOptions is not a complete solution >> because the user could modify the ksp by code. >> > > You can replicate all the KSP settings in the new KSP. They are all > available by the API. > > Matt > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > Are you suggesting that I have to call _all_ KSPGet* routines and then the > corresponding KSPSet* routines in order to copy a ksp object? > For any setting which you wish to replicate, yes. Matt > Thanks for your time, > > Eloy > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Thu May 13 11:31:54 2010 From: zonexo at gmail.com (Wee-Beng TAY) Date: Fri, 14 May 2010 00:31:54 +0800 Subject: [petsc-users] Error with file linking In-Reply-To: References: <4BE9FFBD.7080705@gmail.com> <4BEADA3C.8000205@gmail.com> <4BEAEAAE.9050103@gmail.com> Message-ID: <4BEC297A.8050302@gmail.com> Hi Satish, It finally worked! Thank you very much and have a nice day! Yours sincerely, Wee-Beng Tay On 13/5/2010 1:56 AM, Satish Balay wrote: > try the attached makefile. If you have problems copy/paste the > *complete* make session from your terminal. > > Satish > > On Thu, 13 May 2010, Wee-Beng TAY wrote: > > >> Hi Satish, >> >> I tried to run it, with the -c present and absent. However, nothing happened. >> I run using "make". I also specified "export >> PETSC_DIR=/home/svu/g0306332/codes/petsc-3.1-p0/" and "export >> PETSC_ARCH=atlas5-mpi-nodebug" before that. >> >> Did I miss out something? >> >> Also supposed that I have the same flag for all sources. In that case, how can >> I build them? >> >> Thank you very much and have a nice day! >> >> Yours sincerely, >> >> Wee-Beng Tay >> >> >> On 13/5/2010 1:27 AM, Satish Balay wrote: >> >>> Attaching the modified makefile. It might need a couple of iterations >>> of fixing. >>> >>> Hopefully its clear why it would work. Note - if not for different >>> flags required for different sources (opt/opt2/opt3) - you can build >>> all petsc/non-petsc sources with petsc makefile targets. >>> >>> >>> >>>> In my fortran files, do I simply use petsc.h or I have to use specific *.h >>>> depending on what petsc commands are used? >>>> >>>> >>> With petsc-3.1 you just have to use petsc.h for all F77 interface stuff, and >>> petsc.h90 for all f90 interface stuff [like VecGetArrayF90()] >>> >>> Satish >>> >>> On Thu, 13 May 2010, Wee-Beng TAY wrote: >>> >>> >>> >>>> Hi Satish, >>>> >>>> Sorry for the confusion. >>>> >>>> I have attached my makefile. This was used to compile and link my code >>>> before >>>> I used PETSc 3.1. >>>> >>>> I have several fortran files. Some contain PETSc commands and need to use >>>> PETSc include files. Other do not. In the end, they all have to be linked >>>> to >>>> form my final code. I used to copy part of the compiling and linking >>>> command >>>> from "make ex1f" to compile my code: >>>> >>>> /app1/mvapich2/current/bin/mpif90 -c -O3 >>>> -I/home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/include >>>> -I/home/svu/g0306332/codes/petsc-3.1-p0/include >>>> -I/home/svu/g0306332/lib/hypre-2.6.0b_atlas5/include >>>> -I/app1/mvapich2/1.4/include -I/app1/mvapich2/current/include >>>> -I/home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/include >>>> -I/home/svu/g0306332/lib/hypre-2.6.0b_atlas5/include >>>> -I/app1/mvapich2/1.4/include -I/app1/mvapich2/current/include -o ex1f.o >>>> ex1f.F >>>> >>>> /app1/mvapich2/current/bin/mpif90 -O3 -o ex1f ex1f.o >>>> -Wl,-rpath,/home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib >>>> -L/home/svu/g0306332/codes/petsc-3.1-p0/atlas5-mpi-nodebug/lib -lpetsc >>>> -Wl,-rpath,/home/svu/g0306332/lib/hypre-2.6.0b_atlas5/lib >>>> -L/home/svu/g0306332/lib/hypre-2.6.0b_atlas5/lib -lHYPRE -lmpichcxx >>>> -lstdc++ >>>> -Wl,-rpath,/app1/intel/mkl/10.0.5.025/lib/em64t >>>> -L/app1/intel/mkl/10.0.5.025/lib/em64t -lmkl_lapack -lmkl -lguide >>>> -lpthread >>>> -L/app1/mvapich2/1.4/lib -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -ldl >>>> -lmpich >>>> -lpthread -lrdmacm -libverbs -libumad -lrt -lgcc_s -lmpichf90 >>>> -L/app1/intel/Compiler/11.0/074/lib/intel64 -lifport -lifcore -limf -lsvml >>>> -lm >>>> -lipgo -lirc -lirc_s -lm -lmpichcxx -lstdc++ -lmpichcxx -lstdc++ -ldl >>>> -lmpich >>>> -lpthread -lrdmacm -libverbs -libumad -lrt -lgcc_s -ldl >>>> >>>> However, it doesn't seem to work now. I tried to look at chapter 14 but I >>>> don't really understand since I know very little about makefiles. >>>> >>>> Can you give a rough guide for my case? I will try to read up on the >>>> manual to >>>> get more info. >>>> >>>> Moreover, I am confused by the use of: >>>> >>>> #include "finclude/petsc.h" etc. >>>> >>>> In my fortran files, do I simply use petsc.h or I have to use specific *.h >>>> depending on what petsc commands are used? >>>> >>>> Thank you very much and have a nice day! >>>> >>>> Yours sincerely, >>>> >>>> Wee-Beng Tay >>>> >>>> >>>> On 12/5/2010 9:55 AM, Satish Balay wrote: >>>> >>>> >>>>> Lets step back and deal with this primary issue. >>>>> >>>>> Have you attempted to use makefiles in petsc format? What problems >>>>> have you encountered? Perhaps they are fixable. >>>>> >>>>> The reason to use this format is to make your makefile as portable as >>>>> possible [and avoid such issues]. >>>>> >>>>> Satish >>>>> >>>>> On Tue, 11 May 2010, Satish Balay wrote: >>>>> >>>>> >>>>> >>>>> >>>>>> Why not use petsc makefiles - and avoid these hasseles? >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> From hxie at umn.edu Thu May 13 15:43:17 2010 From: hxie at umn.edu (hxie at umn.edu) Date: 13 May 2010 15:43:17 -0500 Subject: [petsc-users] Change entries after MatCopy In-Reply-To: References: Message-ID: > >On 12 May 2010 19:18, wrote: >> Hi, >> >> If I use MatCopy to get a new matrix, can i use MatSetValues to change >> some >> entries of that matrix and do the assemble after that? Or if I just >> need to >> change some diagonal entries, can I use MatDiagonalSet to change the >> entries >> without assembling? Thanks. >> > >Yes and Yes. The first way does no work. I will try the second way. > > > From hxie at umn.edu Thu May 13 15:50:40 2010 From: hxie at umn.edu (hxie at umn.edu) Date: 13 May 2010 15:50:40 -0500 Subject: [petsc-users] Change entries after MatCopy In-Reply-To: References: Message-ID: On May 13 2010, hxie at umn.edu wrote: > >> >>On 12 May 2010 19:18, wrote: >>> Hi, >>> >>> If I use MatCopy to get a new matrix, can i use MatSetValues to change >>> some >>> entries of that matrix and do the assemble after that? Or if I just >>> need to >>> change some diagonal entries, can I use MatDiagonalSet to change the >>> entries >>> without assembling? Thanks. >>> >> >>Yes and Yes. > >The first way does no work. I will try the second way. Sorry. I made a mistake in the variable declaration. The first way works. Thanks. > >> >> >> > From dalcinl at gmail.com Fri May 14 14:37:26 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 14 May 2010 16:37:26 -0300 Subject: [petsc-users] Change entries after MatCopy In-Reply-To: References: Message-ID: On 13 May 2010 17:50, wrote: > On May 13 2010, hxie at umn.edu wrote: > >> >>> >>> On 12 May 2010 19:18, ? wrote: >>>> >>>> Hi, >>>> >>>> If I use MatCopy to get a new matrix, can i use MatSetValues to change >>>> some >>>> entries of that matrix and do the assemble after that? Or if I just need >>>> to >>>> change some diagonal entries, can I use MatDiagonalSet to change the >>>> entries >>>> without assembling? Thanks. >>>> >>> >>> Yes and Yes. >> >> The first way does no work. I will try the second way. > > Sorry. I made a mistake in the variable declaration. The first way works. > Thanks. > Next time you ask something, please make it explicit what "does not work" actually means. Such vague statements are not helpful for any of us in order to help people. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From dalcinl at gmail.com Fri May 14 14:58:49 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 14 May 2010 16:58:49 -0300 Subject: [petsc-users] [petsc4py] going to stop support for petsc-2.3.2 and petsc-2.3.3 Message-ID: Maintaining backward compatibility for petsc4py against petsc-2.3.2 and petsc-2.3.3 is making my head explode. Every feature from newer PETSc releases I want to support forces me to emulate the call using the older API's (and in some cases private data structures), or write some extra code in order to generate a runtime error about the unsupported feature/call/whatever. Additionally, Barry seems to have lost control of his fingers, and make changes to petsc-dev that affect petsc4py at its heart (like the COOKIE->CLASSID rename and the SETERRQ macro redefine). If any of you out there is still interested in running 2.3.2/2.3.3 from Python, speak now and offer help for maintaining this. If you one shows up, I'll COMPLETELY REMOVE all traces of 2.3.2/2.3.3 backward compatibility support from petsc4py repositories (at petsc.cs.iit.edu and Google Code). -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From u.tabak at tudelft.nl Sat May 15 08:00:07 2010 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sat, 15 May 2010 15:00:07 +0200 Subject: [petsc-users] addition of two matrices Message-ID: <4BEE9AD7.3030805@tudelft.nl> Dear all, How can I add two matrices? More openly, I have an expression A^T * M* A + B^T *K*B I find M*A and K*B with MatMatMult then form the 1st and 2nd expression with MatMatMultTranspose and I would like to add the two eventually. I could not find a function in the documentation, maybe missed. Any help is appreciated. Umut From jed at 59A2.org Sat May 15 08:22:07 2010 From: jed at 59A2.org (Jed Brown) Date: Sat, 15 May 2010 15:22:07 +0200 Subject: [petsc-users] addition of two matrices In-Reply-To: <4BEE9AD7.3030805@tudelft.nl> References: <4BEE9AD7.3030805@tudelft.nl> Message-ID: <871vddpaf4.fsf@59A2.org> On Sat, 15 May 2010 15:00:07 +0200, Umut Tabak wrote: > Dear all, > > How can I add two matrices? More openly, I have an expression > > A^T * M* A + B^T *K*B > > I find M*A and K*B with MatMatMult then form the 1st and 2nd expression > with MatMatMultTranspose and I would like to add the two eventually. I > could not find a function in the documentation, maybe missed. Look at the docs for MatPtAP() and MatAXPY(). Jed From xy2102 at columbia.edu Sun May 16 08:53:11 2010 From: xy2102 at columbia.edu ((Rebecca) Xuefei YUAN) Date: Sun, 16 May 2010 09:53:11 -0400 Subject: [petsc-users] bluegene make error Message-ID: <20100516095311.cvcqbcmxeokg8ckk@cubmail.cc.columbia.edu> Dear all, I got a bluegene make error on DAGetArray() and DARestoreArray() as rebeccaxyf at fen1:~/scratch/linux/May16/twmgoreggt$ make twmgoreggt mpixlcxx_r -o twmgoreggt.o -c -qmaxmem=-1 -g -g -+ -I/home/rebeccaxyf/scratch/petsc-dev/arch-bgp-ibm-dbg/include -I/home/rebeccaxyf/scratch/petsc-dev/include -D__INSDIR__= twmgoreggt.c "twmgoreggt.c", line 2996.48: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". "twmgoreggt.c", line 2996.48: 1540-1205 (I) The error occurred while converting to parameter 2 of "DAGetArray(DA, PetscTruth, void **)". "twmgoreggt.c", line 2997.47: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". "twmgoreggt.c", line 2997.47: 1540-1205 (I) The error occurred while converting to parameter 2 of "DAGetArray(DA, PetscTruth, void **)". "twmgoreggt.c", line 3042.41: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". "twmgoreggt.c", line 3042.41: 1540-1205 (I) The error occurred while converting to parameter 2 of "DAGetArray(DA, PetscTruth, void **)". "twmgoreggt.c", line 3048.52: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". "twmgoreggt.c", line 3048.52: 1540-1205 (I) The error occurred while converting to parameter 2 of "DARestoreArray(DA, PetscTruth, void **)". "twmgoreggt.c", line 3049.51: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". "twmgoreggt.c", line 3049.51: 1540-1205 (I) The error occurred while converting to parameter 2 of "DARestoreArray(DA, PetscTruth, void **)". "twmgoreggt.c", line 3058.39: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". "twmgoreggt.c", line 3058.39: 1540-1205 (I) The error occurred while converting to parameter 2 of "DAGetArray(DA, PetscTruth, void **)". "twmgoreggt.c", line 3065.57: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". "twmgoreggt.c", line 3065.57: 1540-1205 (I) The error occurred while converting to parameter 2 of "DAGetArray(DA, PetscTruth, void **)". "twmgoreggt.c", line 3066.58: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". "twmgoreggt.c", line 3066.58: 1540-1205 (I) The error occurred while converting to parameter 2 of "DAGetArray(DA, PetscTruth, void **)". "twmgoreggt.c", line 3067.60: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". "twmgoreggt.c", line 3067.60: 1540-1205 (I) The error occurred while converting to parameter 2 of "DAGetArray(DA, PetscTruth, void **)". "twmgoreggt.c", line 3221.61: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". "twmgoreggt.c", line 3221.61: 1540-1205 (I) The error occurred while converting to parameter 2 of "DARestoreArray(DA, PetscTruth, void **)". "twmgoreggt.c", line 3222.62: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". "twmgoreggt.c", line 3222.62: 1540-1205 (I) The error occurred while converting to parameter 2 of "DARestoreArray(DA, PetscTruth, void **)". "twmgoreggt.c", line 3223.64: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". "twmgoreggt.c", line 3223.64: 1540-1205 (I) The error occurred while converting to parameter 2 of "DARestoreArray(DA, PetscTruth, void **)". "twmgoreggt.c", line 3226.39: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". "twmgoreggt.c", line 3226.39: 1540-1205 (I) The error occurred while converting to parameter 2 of "DAGetArray(DA, PetscTruth, void **)". "twmgoreggt.c", line 3387.45: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". "twmgoreggt.c", line 3387.45: 1540-1205 (I) The error occurred while converting to parameter 2 of "DARestoreArray(DA, PetscTruth, void **)". "twmgoreggt.c", line 3388.43: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". "twmgoreggt.c", line 3388.43: 1540-1205 (I) The error occurred while converting to parameter 2 of "DARestoreArray(DA, PetscTruth, void **)". "twmgoreggt.c", line 3389.43: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". "twmgoreggt.c", line 3389.43: 1540-1205 (I) The error occurred while converting to parameter 2 of "DARestoreArray(DA, PetscTruth, void **)". make: [twmgoreggt.o] Error 1 (ignored) mpixlcxx_r -qmaxmem=-1 -g -g -+ -o twmgoreggt.exe twmgoreggt.o -Wl,-rpath,/home/rebeccaxyf/scratch/petsc-dev/arch-bgp-ibm-dbg/lib -L/home/rebeccaxyf/scratch/petsc-dev/arch-bgp-ibm-dbg/lib -lpetsc -L/opt/share/math_libraries/lapack/ppc64/IBM/ -llapack -lblas -lnsl -lrt -L/bgsys/drivers/V1R4M0_320_2009-090815P/ppc/comm/default/lib -L/bgsys/drivers/V1R4M0_320_2009-090815P/ppc/comm/sys/lib -L/bgsys/drivers/V1R4M0_320_2009-090815P/ppc/runtime/SPI -L/opt/ibmcmp/xlsmp/bg/1.7/bglib -L/opt/ibmcmp/xlmass/bg/4.4/bglib -L/opt/ibmcmp/vac/bg/9.0/bglib -L/opt/ibmcmp/vacpp/bg/9.0/bglib -L/bgsys/drivers/V1R4M0_320_2009-090815P/ppc/gnu-linux/lib/gcc/powerpc-bgp-linux/4.1.2 -L/bgsys/drivers/V1R4M0_320_2009-090815P/ppc/gnu-linux/powerpc-bgp-linux/lib -L/opt/ibmcmp/xlf/bg/11.1/bglib -Wl,-rpath,/opt/ibmcmp/lib/bg/bglib -ldl -Wl,-rpath,/bgsys/drivers/V1R4M0_320_2009-090815P/ppc/comm/default/lib -lmpich.cnk -lopa -Wl,-rpath,/bgsys/drivers/V1R4M0_320_2009-090815P/ppc/comm/sys/lib -ldcmf.cnk -ldcmfcoll.cnk -lpthread -Wl,-rpath,/bgsys/drivers/V1R4M0_320_2009-090815P/ppc/runtime/SPI -lSPI.cna -lrt -lxlopt -lxl -lgcc_eh -lxlf90_r -lxlomp_ser -lxlfmath -lm -lcxxmpich.cnk -libmc++ -lstdc++ -lcxxmpich.cnk -libmc++ -lstdc++ -ldl -lmpich.cnk -lopa -ldcmf.cnk -ldcmfcoll.cnk -lpthread -lSPI.cna -lrt -lxlopt -lxl -lgcc_eh -ldl /opt/ibmcmp/vacpp/bg/9.0/bin/bgxlC_r: 1501-228 (W) input file twmgoreggt.o not found make: [twmgoreggt] Error 252 (ignored) /bin/rm -f twmgoreggt.o How could I solve this problem? Thanks a lot! Rebecca -- (Rebecca) Xuefei YUAN Department of Applied Physics and Applied Mathematics Columbia University Tel:917-399-8032 www.columbia.edu/~xy2102 From bsmith at mcs.anl.gov Sun May 16 11:50:08 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 16 May 2010 11:50:08 -0500 Subject: [petsc-users] bluegene make error In-Reply-To: <20100516095311.cvcqbcmxeokg8ckk@cubmail.cc.columbia.edu> References: <20100516095311.cvcqbcmxeokg8ckk@cubmail.cc.columbia.edu> Message-ID: Your calling sequence to DAGetArray() isn't correct. Likely you mean to call DAVecGetArray() not DAGetArray(). Barry On May 16, 2010, at 8:53 AM, (Rebecca) Xuefei YUAN wrote: > Dear all, > > I got a bluegene make error on DAGetArray() and DARestoreArray() as > > rebeccaxyf at fen1:~/scratch/linux/May16/twmgoreggt$ make twmgoreggt > mpixlcxx_r -o twmgoreggt.o -c -qmaxmem=-1 -g -g -+ -I/home/rebeccaxyf/scratch/petsc-dev/arch-bgp-ibm-dbg/include -I/home/rebeccaxyf/scratch/petsc-dev/include -D__INSDIR__= twmgoreggt.c > "twmgoreggt.c", line 2996.48: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". > "twmgoreggt.c", line 2996.48: 1540-1205 (I) The error occurred while converting to parameter 2 of "DAGetArray(DA, PetscTruth, void **)". > "twmgoreggt.c", line 2997.47: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". > "twmgoreggt.c", line 2997.47: 1540-1205 (I) The error occurred while converting to parameter 2 of "DAGetArray(DA, PetscTruth, void **)". > "twmgoreggt.c", line 3042.41: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". > "twmgoreggt.c", line 3042.41: 1540-1205 (I) The error occurred while converting to parameter 2 of "DAGetArray(DA, PetscTruth, void **)". > "twmgoreggt.c", line 3048.52: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". > "twmgoreggt.c", line 3048.52: 1540-1205 (I) The error occurred while converting to parameter 2 of "DARestoreArray(DA, PetscTruth, void **)". > "twmgoreggt.c", line 3049.51: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". > "twmgoreggt.c", line 3049.51: 1540-1205 (I) The error occurred while converting to parameter 2 of "DARestoreArray(DA, PetscTruth, void **)". > "twmgoreggt.c", line 3058.39: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". > "twmgoreggt.c", line 3058.39: 1540-1205 (I) The error occurred while converting to parameter 2 of "DAGetArray(DA, PetscTruth, void **)". > "twmgoreggt.c", line 3065.57: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". > "twmgoreggt.c", line 3065.57: 1540-1205 (I) The error occurred while converting to parameter 2 of "DAGetArray(DA, PetscTruth, void **)". > "twmgoreggt.c", line 3066.58: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". > "twmgoreggt.c", line 3066.58: 1540-1205 (I) The error occurred while converting to parameter 2 of "DAGetArray(DA, PetscTruth, void **)". > "twmgoreggt.c", line 3067.60: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". > "twmgoreggt.c", line 3067.60: 1540-1205 (I) The error occurred while converting to parameter 2 of "DAGetArray(DA, PetscTruth, void **)". > "twmgoreggt.c", line 3221.61: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". > "twmgoreggt.c", line 3221.61: 1540-1205 (I) The error occurred while converting to parameter 2 of "DARestoreArray(DA, PetscTruth, void **)". > "twmgoreggt.c", line 3222.62: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". > "twmgoreggt.c", line 3222.62: 1540-1205 (I) The error occurred while converting to parameter 2 of "DARestoreArray(DA, PetscTruth, void **)". > "twmgoreggt.c", line 3223.64: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". > "twmgoreggt.c", line 3223.64: 1540-1205 (I) The error occurred while converting to parameter 2 of "DARestoreArray(DA, PetscTruth, void **)". > "twmgoreggt.c", line 3226.39: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". > "twmgoreggt.c", line 3226.39: 1540-1205 (I) The error occurred while converting to parameter 2 of "DAGetArray(DA, PetscTruth, void **)". > "twmgoreggt.c", line 3387.45: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". > "twmgoreggt.c", line 3387.45: 1540-1205 (I) The error occurred while converting to parameter 2 of "DARestoreArray(DA, PetscTruth, void **)". > "twmgoreggt.c", line 3388.43: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". > "twmgoreggt.c", line 3388.43: 1540-1205 (I) The error occurred while converting to parameter 2 of "DARestoreArray(DA, PetscTruth, void **)". > "twmgoreggt.c", line 3389.43: 1540-0256 (S) A parameter of type "PetscTruth" cannot be initialized with an expression of type "PetscDataType". > "twmgoreggt.c", line 3389.43: 1540-1205 (I) The error occurred while converting to parameter 2 of "DARestoreArray(DA, PetscTruth, void **)". > make: [twmgoreggt.o] Error 1 (ignored) > mpixlcxx_r -qmaxmem=-1 -g -g -+ -o twmgoreggt.exe twmgoreggt.o -Wl,-rpath,/home/rebeccaxyf/scratch/petsc-dev/arch-bgp-ibm-dbg/lib -L/home/rebeccaxyf/scratch/petsc-dev/arch-bgp-ibm-dbg/lib -lpetsc -L/opt/share/math_libraries/lapack/ppc64/IBM/ -llapack -lblas -lnsl -lrt -L/bgsys/drivers/V1R4M0_320_2009-090815P/ppc/comm/default/lib -L/bgsys/drivers/V1R4M0_320_2009-090815P/ppc/comm/sys/lib -L/bgsys/drivers/V1R4M0_320_2009-090815P/ppc/runtime/SPI -L/opt/ibmcmp/xlsmp/bg/1.7/bglib -L/opt/ibmcmp/xlmass/bg/4.4/bglib -L/opt/ibmcmp/vac/bg/9.0/bglib -L/opt/ibmcmp/vacpp/bg/9.0/bglib -L/bgsys/drivers/V1R4M0_320_2009-090815P/ppc/gnu-linux/lib/gcc/powerpc-bgp-linux/4.1.2 -L/bgsys/drivers/V1R4M0_320_2009-090815P/ppc/gnu-linux/powerpc-bgp-linux/lib -L/opt/ibmcmp/xlf/bg/11.1/bglib -Wl,-rpath,/opt/ibmcmp/lib/bg/bglib -ldl -Wl,-rpath,/bgsys/drivers/V1R4M0_320_2009-090815P/ppc/comm/default/lib -lmpich.cnk -lopa -Wl,-rpath,/bgsys/drivers/V1R4M0_320_2009-090815P/ppc/comm/sys/lib -ldcmf.cnk -ldcmfcoll.cnk -lpthread -Wl,-rpath,/bgsys/drivers/V1R4M0_320_2009-090815P/ppc/runtime/SPI -lSPI.cna -lrt -lxlopt -lxl -lgcc_eh -lxlf90_r -lxlomp_ser -lxlfmath -lm -lcxxmpich.cnk -libmc++ -lstdc++ -lcxxmpich.cnk -libmc++ -lstdc++ -ldl -lmpich.cnk -lopa -ldcmf.cnk -ldcmfcoll.cnk -lpthread -lSPI.cna -lrt -lxlopt -lxl -lgcc_eh -ldl > /opt/ibmcmp/vacpp/bg/9.0/bin/bgxlC_r: 1501-228 (W) input file twmgoreggt.o not found > make: [twmgoreggt] Error 252 (ignored) > /bin/rm -f twmgoreggt.o > > How could I solve this problem? > > Thanks a lot! > > Rebecca > > > > -- > (Rebecca) Xuefei YUAN > Department of Applied Physics and Applied Mathematics > Columbia University > Tel:917-399-8032 > www.columbia.edu/~xy2102 > From kenway at utias.utoronto.ca Mon May 17 18:34:02 2010 From: kenway at utias.utoronto.ca (Gaetan Kenway) Date: Mon, 17 May 2010 19:34:02 -0400 Subject: [petsc-users] PetscInitialize with MPI Groups Message-ID: <4BF1D26A.3000104@utias.utoronto.ca> Hello I use PETSc in both fortran and in Python using the petsc4py bindings. I currently have an issue with initializing PETSc when using MPI groups. I am using a code with two parts: an aero part and an structural part. I wish to only use PETSc on one of the processor groups, say the aero side. I've attached a simple python script that replicates the behavior I see. Basically, when you initialize PETSc on only a subset of MPI_COMM_WORLD, the program hangs. However, if the processors that are NOT being initialized with petsc are at a MPI_BARRIER, it appears to work. Note: any combination of nProc_aero and nProc_struct that add up to 4, ( (1,3), (2,2) or (3,1) ) give the same behavior. The test.py script as supplied should hang with run with mpirun -np 4 python test.py However, if line 37 is uncommented, it will work. This is very similar to my actual problem. After I take the communicator, comm, corresponding to the aero processors, I pass it to fortran (using mpi4py) and then use: PETSC_COMM_WORLD = comm call PetscInitialize(PETSC_NULL_CHARACTER,ierr) However, again, only if the struct processors have a MPI_BARRIER call which corresponds to the PetscInitialize call will the process carry on as expected. If the other process exits before an MPI_BARRIER is called, the program simply hangs indefinitely. Currently, the workaround is to call an MPI_BARRIER on the other processors while the init is being called. However, I don think this is correct. Any guidance would be greatly appreciated. Gaetan Kenway Ph.D Candidate University of Toronto Institute for Aerospace Studies -------------- next part -------------- A non-text attachment was scrubbed... Name: test.py Type: text/x-python Size: 1113 bytes Desc: not available URL: From dalcinl at gmail.com Mon May 17 19:57:20 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 17 May 2010 21:57:20 -0300 Subject: [petsc-users] PetscInitialize with MPI Groups In-Reply-To: <4BF1D26A.3000104@utias.utoronto.ca> References: <4BF1D26A.3000104@utias.utoronto.ca> Message-ID: On 17 May 2010 20:34, Gaetan Kenway wrote: > Hello > > I use PETSc in both fortran and in Python using the petsc4py bindings. ?I > currently have an issue with initializing PETSc when using MPI groups. I am > using a code with two parts: an aero part and an structural part. I wish to > only use PETSc on one of the processor groups, ?say the aero side. I've > attached a simple python script that replicates the behavior I see. > ?Basically, when you initialize PETSc on only a subset of MPI_COMM_WORLD, > the program hangs. However, if the processors that are NOT being initialized > with petsc are at a MPI_BARRIER, it appears to work. ?Note: any combination > of nProc_aero and nProc_struct that add up to 4, ( (1,3), (2,2) or (3,1) ) > give the same behavior. > > The test.py script as supplied should hang with run with > > mpirun -np 4 python test.py > > However, if line 37 is uncommented, it will work. > > > This is very similar to my actual problem. After I take the communicator, > comm, corresponding to the aero processors, I pass it to fortran (using > mpi4py) and then use: > > PETSC_COMM_WORLD = comm > call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > > However, again, only if the struct processors have a MPI_BARRIER call which > corresponds to the PetscInitialize call will the process carry on as > expected. ?If the other process exits before an MPI_BARRIER is called, the > program simply hangs indefinitely. > > Currently, the workaround is to call an MPI_BARRIER on the other processors > while the init is being called. However, I don think this is correct. > > Any guidance would be greatly appreciated. > > Gaetan Kenway > Ph.D Candidate > University of Toronto Institute for Aerospace Studies > Basically, the comment line near the end: if is_aero == True: from petsc4py import PETSc # This will call PETSc initialize on JUST the aero processors is wrong. When you "from petsc4py import PETSc", PETSc is actually initialized in MPI_COMM_WORLD, just because you do not have (currently) any way to set PETSC_COMM_WORLD = comm from Python side before the call to PetscInitalize(). Then it likely hangs at a collective MPI_Barrier(PETSC_COMM_WORLD/*==MPI_COMM_WORLD*/) Adding some support for this would be more or less trivial, for example something like: if is_aero: import petsc4py; petsc4py.init(comm=subcomm) from petsc4py import PETSc else: pass Would this work for you? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From u.tabak at tudelft.nl Tue May 18 02:52:33 2010 From: u.tabak at tudelft.nl (Umut Tabak) Date: Tue, 18 May 2010 09:52:33 +0200 Subject: [petsc-users] Advice on how to tackle with in place operation for a Matrix Message-ID: <4BF24741.1050801@tudelft.nl> Dear all, I would like to do a Gram-Schmidt orthogonalization of a matrix with say 10 columns, I wrote a function for this, However, I do not do this in place for the moment. I create a new matrix for the orthogonal(or M-orthogonal columns). And set the columns of that new matrix, I could not see how I could do this in place, where I read the original matrix in binary format from a file. Then its assembly is finalized I suppose, is there still a way to make changes on that matrix in place? The code is below. A simple translation of a matlab script(more efficient implementations are possible I am sure.) int mgrasch_Morth(Mat &inMat, Mat &outMat, Mat M ) { // read size of the matrix PetscErrorCode ierr; PetscInt r,c; ierr = MatGetSize(inMat, &r, &c); CHKERRQ(ierr); ierr = MatDuplicate(inMat, MAT_COPY_VALUES, &outMat); CHKERRQ(ierr); // set up the necessary workspace Vec Q,R,dummy1,dummy2,dummy3,orthoVec; PetscScalar val1, val2; // ierr = VecCreate(MPI_COMM_SELF, &Q); CHKERRQ(ierr); ierr = VecCreate(MPI_COMM_SELF, &R); CHKERRQ(ierr); // dummy vectors for matrix multiplication ierr = VecCreate(MPI_COMM_SELF, &dummy1); CHKERRQ(ierr); ierr = VecCreate(MPI_COMM_SELF, &dummy2); CHKERRQ(ierr); ierr = VecCreate(MPI_COMM_SELF, &dummy2); CHKERRQ(ierr); ierr = VecCreate(MPI_COMM_SELF, &dummy3); CHKERRQ(ierr); // ierr = VecCreate(MPI_COMM_SELF, &orthoVec); CHKERRQ(ierr); // ierr = VecSetSizes(Q, r, PETSC_DECIDE); CHKERRQ(ierr); ierr = VecSetSizes(R, r, PETSC_DECIDE); CHKERRQ(ierr); ierr = VecSetSizes(dummy1, r, PETSC_DECIDE); CHKERRQ(ierr); ierr = VecSetSizes(dummy2, r, PETSC_DECIDE); CHKERRQ(ierr); ierr = VecSetSizes(dummy3, r, PETSC_DECIDE); CHKERRQ(ierr); ierr = VecSetSizes(orthoVec, r, PETSC_DECIDE); CHKERRQ(ierr); // ierr = VecSetFromOptions(R); CHKERRQ(ierr); ierr = VecSetFromOptions(Q); CHKERRQ(ierr); ierr = VecSetFromOptions(dummy1); CHKERRQ(ierr); // for M * A(:,m) ierr = VecSetFromOptions(dummy2); CHKERRQ(ierr); // for M * Q ierr = VecSetFromOptions(dummy3); CHKERRQ(ierr); // for A(:,m) above ierr = VecSetFromOptions(orthoVec); CHKERRQ(ierr); // orthogonalized result vec // create the arrays for MatSetValues // row PetscInt rowArr[r]; for(int k=0;k!=r;k++) rowArr[k]=k; // to set one column at a time PetscInt C = 1; PetscInt colArr[1]; PetscScalar valsOrthoVec[r]; // for(int i=0;i!=c;i++){ MatGetColumnVector(inMat, Q, i); for(int j=i+1;j!=c;j++){ ierr = MatGetColumnVector(inMat, dummy3, j); CHKERRQ(ierr); ierr = MatMult(M,dummy3,dummy1); CHKERRQ(ierr); // val1 -> Q'* M * A(:,m) ierr = VecTDot(Q, dummy1,&val1); CHKERRQ(ierr); ierr = MatMult(M,Q,dummy2); CHKERRQ(ierr); // val2 -> Q'* M * Q ierr = VecTDot(Q,dummy2,&val2); CHKERRQ(ierr); // ierr = VecAXPY(dummy3, -val1/val2, Q); CHKERRQ(ierr); ierr = VecCopy(dummy3, orthoVec); CHKERRQ(ierr); // set the orthogonal vector colArr[0] = j; // ierr = VecGetValues(orthoVec, r, rowArr, valsOrthoVec); CHKERRQ(ierr); ierr = MatSetValues(outMat, r, rowArr, C, colArr, valsOrthoVec, INSERT_VALUES); CHKERRQ(ierr); } } ierr = MatAssemblyBegin(outMat, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); ierr = MatAssemblyEnd(outMat, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); return 0; } From knepley at gmail.com Tue May 18 03:30:13 2010 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 18 May 2010 03:30:13 -0500 Subject: [petsc-users] Advice on how to tackle with in place operation for a Matrix In-Reply-To: <4BF24741.1050801@tudelft.nl> References: <4BF24741.1050801@tudelft.nl> Message-ID: 1) This must be a dense matrix, so you could directly copy in the values with a strided loop 2) Is there a reason for the output to be a matrix, rather than a collection of vectors? Matt On Tue, May 18, 2010 at 2:52 AM, Umut Tabak wrote: > Dear all, > > I would like to do a Gram-Schmidt orthogonalization of a matrix with say 10 > columns, I wrote a function for this, However, I do not do this in place for > the moment. I create a new matrix for the orthogonal(or M-orthogonal > columns). And set the columns of that new matrix, I could not see how I > could do this in place, where I read the original matrix in binary format > from a file. Then its assembly is finalized I suppose, is there still a way > to make changes on that matrix in place? > > The code is below. > > A simple translation of a matlab script(more efficient implementations are > possible I am sure.) > > int mgrasch_Morth(Mat &inMat, Mat &outMat, Mat M ) > { > // read size of the matrix > PetscErrorCode ierr; > PetscInt r,c; > ierr = MatGetSize(inMat, &r, &c); > CHKERRQ(ierr); > ierr = MatDuplicate(inMat, MAT_COPY_VALUES, &outMat); CHKERRQ(ierr); > // set up the necessary workspace > Vec Q,R,dummy1,dummy2,dummy3,orthoVec; > PetscScalar val1, val2; > // > ierr = VecCreate(MPI_COMM_SELF, &Q); CHKERRQ(ierr); > ierr = VecCreate(MPI_COMM_SELF, &R); CHKERRQ(ierr); > > // dummy vectors for matrix multiplication > ierr = VecCreate(MPI_COMM_SELF, &dummy1); CHKERRQ(ierr); > ierr = VecCreate(MPI_COMM_SELF, &dummy2); CHKERRQ(ierr); > ierr = VecCreate(MPI_COMM_SELF, &dummy2); CHKERRQ(ierr); > ierr = VecCreate(MPI_COMM_SELF, &dummy3); CHKERRQ(ierr); > // > ierr = VecCreate(MPI_COMM_SELF, &orthoVec); CHKERRQ(ierr); > // > ierr = VecSetSizes(Q, r, PETSC_DECIDE); CHKERRQ(ierr); > ierr = VecSetSizes(R, r, PETSC_DECIDE); CHKERRQ(ierr); > ierr = VecSetSizes(dummy1, r, PETSC_DECIDE); CHKERRQ(ierr); > ierr = VecSetSizes(dummy2, r, PETSC_DECIDE); CHKERRQ(ierr); > ierr = VecSetSizes(dummy3, r, PETSC_DECIDE); CHKERRQ(ierr); > ierr = VecSetSizes(orthoVec, r, PETSC_DECIDE); CHKERRQ(ierr); > // > ierr = VecSetFromOptions(R); CHKERRQ(ierr); > ierr = VecSetFromOptions(Q); CHKERRQ(ierr); > ierr = VecSetFromOptions(dummy1); CHKERRQ(ierr); // for M * A(:,m) > ierr = VecSetFromOptions(dummy2); CHKERRQ(ierr); // for M * Q > ierr = VecSetFromOptions(dummy3); CHKERRQ(ierr); // for A(:,m) above > ierr = VecSetFromOptions(orthoVec); CHKERRQ(ierr); // orthogonalized > result vec > // create the arrays for MatSetValues > // row > PetscInt rowArr[r]; > for(int k=0;k!=r;k++) > rowArr[k]=k; > // to set one column at a time > PetscInt C = 1; > PetscInt colArr[1]; > PetscScalar valsOrthoVec[r]; > // > for(int i=0;i!=c;i++){ > MatGetColumnVector(inMat, Q, i); > for(int j=i+1;j!=c;j++){ > ierr = MatGetColumnVector(inMat, dummy3, j); CHKERRQ(ierr); > ierr = MatMult(M,dummy3,dummy1); CHKERRQ(ierr); > // val1 -> Q'* M * A(:,m) > ierr = VecTDot(Q, dummy1,&val1); CHKERRQ(ierr); > ierr = MatMult(M,Q,dummy2); CHKERRQ(ierr); > // val2 -> Q'* M * Q > ierr = VecTDot(Q,dummy2,&val2); CHKERRQ(ierr); > // > ierr = VecAXPY(dummy3, -val1/val2, Q); CHKERRQ(ierr); > ierr = VecCopy(dummy3, orthoVec); CHKERRQ(ierr); > // set the orthogonal vector > colArr[0] = j; > // > ierr = VecGetValues(orthoVec, r, rowArr, valsOrthoVec); > CHKERRQ(ierr); > ierr = MatSetValues(outMat, r, rowArr, C, colArr, valsOrthoVec, > INSERT_VALUES); CHKERRQ(ierr); > } > } > ierr = MatAssemblyBegin(outMat, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > ierr = MatAssemblyEnd(outMat, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > return 0; > } > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From u.tabak at tudelft.nl Tue May 18 03:58:39 2010 From: u.tabak at tudelft.nl (Umut Tabak) Date: Tue, 18 May 2010 10:58:39 +0200 Subject: [petsc-users] Advice on how to tackle with in place operation for a Matrix In-Reply-To: References: <4BF24741.1050801@tudelft.nl> Message-ID: <4BF256BF.4090403@tudelft.nl> Matthew Knepley wrote: > 1) This must be a dense matrix, so you could directly copy in the > values with a strided loop > Dear Matt, Thanks for the immediate reply, I guess I did not understand this. > 2) Is there a reason for the output to be a matrix, rather than a > collection of vectors? > Collection of vectors? not sure on this point. After the orthogonalization, I extract some parts of the matrix and continue operation with these extracted parts. Are there better ways of conducting operations for these kinds of tasks? From knepley at gmail.com Tue May 18 04:08:30 2010 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 18 May 2010 04:08:30 -0500 Subject: [petsc-users] Advice on how to tackle with in place operation for a Matrix In-Reply-To: <4BF256BF.4090403@tudelft.nl> References: <4BF24741.1050801@tudelft.nl> <4BF256BF.4090403@tudelft.nl> Message-ID: On Tue, May 18, 2010 at 3:58 AM, Umut Tabak wrote: > Matthew Knepley wrote: > >> 1) This must be a dense matrix, so you could directly copy in the values >> with a strided loop >> >> Dear Matt, > Thanks for the immediate reply, I guess I did not understand this. > > 2) Is there a reason for the output to be a matrix, rather than a >> collection of vectors? >> >> Collection of vectors? not sure on this point. After the > orthogonalization, I extract some parts of the matrix and continue operation > with these extracted parts. Are there better ways of conducting operations > for these kinds of tasks? > Well, you generally do not want the parallel organization of a MPIDENSE when you make these sorts of updates. You want to access 1 column at a time, with the layout of a vector. Thus is generally makes more sense to me to use a Vec[] than a MPIDENSE. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue May 18 11:36:20 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 18 May 2010 11:36:20 -0500 Subject: [petsc-users] Advice on how to tackle with in place operation for a Matrix In-Reply-To: References: <4BF24741.1050801@tudelft.nl> <4BF256BF.4090403@tudelft.nl> Message-ID: If you do want to do this with a dense matrix then just use MatGetArray() to directly access the matrix entries. Then you can skip the MatGetColumn() and MatSetValues() stuff. But I agree with Matt that generally if you are doing orthogonalization it is a set of vectors you are doing it to. You then decide if you want to represent that collection of vectors as a dense matrix (which is the Matlab way) or as an array of vectors. Barry On May 18, 2010, at 4:08 AM, Matthew Knepley wrote: > On Tue, May 18, 2010 at 3:58 AM, Umut Tabak wrote: > Matthew Knepley wrote: > 1) This must be a dense matrix, so you could directly copy in the values with a strided loop > > Dear Matt, > Thanks for the immediate reply, I guess I did not understand this. > > 2) Is there a reason for the output to be a matrix, rather than a collection of vectors? > > Collection of vectors? not sure on this point. After the orthogonalization, I extract some parts of the matrix and continue operation with these extracted parts. Are there better ways of conducting operations for these kinds of tasks? > > Well, you generally do not want the parallel organization of a MPIDENSE when you make > these sorts of updates. You want to access 1 column at a time, with the layout of a vector. > Thus is generally makes more sense to me to use a Vec[] than a MPIDENSE. > > Matt > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From u.tabak at tudelft.nl Wed May 19 10:15:23 2010 From: u.tabak at tudelft.nl (Umut Tabak) Date: Wed, 19 May 2010 17:15:23 +0200 Subject: [petsc-users] a question for iterative solver users Message-ID: <4BF4008B.70900@tudelft.nl> Dear all, What could be the benefit of iterative solvers(specifically cg or pcg), as a factor, in comparison to direct solvers, on reasonably large sparse systems if one has a well conditioned operator matrix. This question might be a bit abstract(like the one above) though I will fire anyway ;) : where does cg or iterative methods really start to beat direct solvers, again for a well conditioned operator matrix? I am asking this from matrix size point of view. I could not make, really, use of iterative solvers due to ill-conditioned matrices, however at some point of my algorithm, I have a matrix which seems to be well conditioned but still MATLAB's backslash beats the pcg. This was the reason of the above questions, in a bit awkward order. I would like to hear experiences from the general user community if possible. Best, Umut From bsmith at mcs.anl.gov Wed May 19 11:29:24 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 19 May 2010 11:29:24 -0500 Subject: [petsc-users] a question for iterative solver users In-Reply-To: <4BF4008B.70900@tudelft.nl> References: <4BF4008B.70900@tudelft.nl> Message-ID: <04D1D047-4B83-4E62-B2B8-81E1696609A2@mcs.anl.gov> For sparse problems coming from a N = n by n by n 3D grid discretization of PDEs direct solvers end up requiring n^4 memory and n^6 = N^2 work. The best iterative solvers (for example multigrid) if they work well on the problem require only n^3 = N memory and n^3 = N work. So for largish n you can see that direct solvers are no longer practical. From my experiences under the right conditions iterative solvers can win at even 10,000 unknowns and definitely by 50,000 and up they easily win but it is very problem dependent. This is exactly why PETSc provides the same interface for direct and iterative solvers, because you cannot know in advance which will win. Barry On May 19, 2010, at 10:15 AM, Umut Tabak wrote: > Dear all, > > What could be the benefit of iterative solvers(specifically cg or pcg), as a factor, in comparison to direct solvers, on reasonably large sparse systems if one has a well conditioned operator matrix. > > This question might be a bit abstract(like the one above) though I will fire anyway ;) : where does cg or iterative methods really start to beat direct solvers, again for a well conditioned operator matrix? I am asking this from matrix size point of view. > > I could not make, really, use of iterative solvers due to ill-conditioned matrices, however at some point of my algorithm, I have a matrix which seems to be well conditioned but still MATLAB's backslash beats the pcg. This was the reason of the above questions, in a bit awkward order. > > I would like to hear experiences from the general user community if possible. > > Best, > Umut From renzhengyong at gmail.com Wed May 19 13:50:50 2010 From: renzhengyong at gmail.com (RenZhengYong) Date: Wed, 19 May 2010 20:50:50 +0200 Subject: [petsc-users] complex number issues Message-ID: Hi, PETSc team, I wrote a C++ code dealing with electromagnetic problems, which can smoothly call PETSc complied by using " --with-scalar-type=complex --with-clanguage=cxx" . To improve the performances of the C++ code, I changed the complex number of type from std::complex to double _Complex (C99 standard, roughly 30~50 times faster compared to std::complex). But when I find that the C99 complex number head file will lead to the following kinds of errors" /usr/include/c++/4.3/complex:57: error: expected identifier before ?__complex__? /usr/include/c++/4.3/complex:58: error: expected identifier before ?__complex__? /usr/include/c++/4.3/complex:58: error: expected unqualified-id before ? From balay at mcs.anl.gov Wed May 19 13:56:04 2010 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 19 May 2010 13:56:04 -0500 (CDT) Subject: [petsc-users] complex number issues In-Reply-To: References: Message-ID: I don't understand the code change - but if you want to use c99 complex - you can build petsc with: --with-scalar-type=complex --with-clanguage=c [which is the default for clanguage] And no code-change needed for PETSc. However if your code uses c++ - you might have to address that.. Satish On Wed, 19 May 2010, RenZhengYong wrote: > Hi, PETSc team, > I wrote a C++ code dealing with electromagnetic problems, which can smoothly > call PETSc complied by using " --with-scalar-type=complex > --with-clanguage=cxx" . > To improve the performances of the C++ code, I changed the complex number of > type from std::complex to double _Complex (C99 standard, roughly > 30~50 times faster compared to std::complex). But when I find that > the C99 complex number head file will lead to the following > kinds of errors" > > /usr/include/c++/4.3/complex:57: error: expected identifier before > ?__complex__? > /usr/include/c++/4.3/complex:58: error: expected identifier before > ?__complex__? > /usr/include/c++/4.3/complex:58: error: expected unqualified-id before ? token > /usr/include/c++/4.3/complex:59: error: expected identifier before > ?__complex__? > /usr/include/c++/4.3/complex:59: error: expected unqualified-id before ? token > /usr/include/c++/4.3/complex:60: error: expected identifier before > ?__complex__? > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:157: error: > expected unqualified-id before ?__complex__? > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:306: error: > ?PetscScalar? does not name a type > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:339: error: > ?PetscScalar? does not name a type > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: warning: > ?PetscGlobalSum? initialized and declared ?extern? > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: error: > ?PetscScalar? was not declared in this scope > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: error: > expected primary-expression before ?,? token > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: error: > ?PetscScalar? was not declared in this scope > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: error: > expected primary-expression before ?,? token > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: error: > expected primary-expression before ?)? token > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: error: > initializer expression list treated as compound expression > > > Could you please give me some idea to deal with these problems? > > Kind regards, > Zhengyong Ren > > From renzhengyong at gmail.com Thu May 20 07:24:55 2010 From: renzhengyong at gmail.com (RenZhengYong) Date: Thu, 20 May 2010 14:24:55 +0200 Subject: [petsc-users] complex number issues In-Reply-To: References: Message-ID: On Wed, May 19, 2010 at 8:56 PM, Satish Balay wrote: > I don't understand the code change - but if you want to use c99 > complex - you can build petsc with: > > --with-scalar-type=complex --with-clanguage=c [which is the default for > clanguage] > > And no code-change needed for PETSc. However if your code uses c++ - > you might have to address that.. > Badly, my code uses C++. I compiled PETSc with "--with-scalar-type=complex --with-clanguage=c" and I added include ${PETSC_DIR}/conf/variables include ${PETSC_DIR}/conf/rules in my makefile. So, could it possible to specify g++ for my code and to link c-version petsc? kind regards, Zhengyong > > Satish > > On Wed, 19 May 2010, RenZhengYong wrote: > > > Hi, PETSc team, > > I wrote a C++ code dealing with electromagnetic problems, which can > smoothly > > call PETSc complied by using " --with-scalar-type=complex > > --with-clanguage=cxx" . > > To improve the performances of the C++ code, I changed the complex number > of > > type from std::complex to double _Complex (C99 standard, roughly > > 30~50 times faster compared to std::complex). But when I find > that > > the C99 complex number head file will lead to the following > > kinds of errors" > > > > /usr/include/c++/4.3/complex:57: error: expected identifier before > > ?__complex__? > > /usr/include/c++/4.3/complex:58: error: expected identifier before > > ?__complex__? > > /usr/include/c++/4.3/complex:58: error: expected unqualified-id before > ? > token > > /usr/include/c++/4.3/complex:59: error: expected identifier before > > ?__complex__? > > /usr/include/c++/4.3/complex:59: error: expected unqualified-id before > ? > token > > /usr/include/c++/4.3/complex:60: error: expected identifier before > > ?__complex__? > > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:157: error: > > expected unqualified-id before ?__complex__? > > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:306: error: > > ?PetscScalar? does not name a type > > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:339: error: > > ?PetscScalar? does not name a type > > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: warning: > > ?PetscGlobalSum? initialized and declared ?extern? > > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: error: > > ?PetscScalar? was not declared in this scope > > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: error: > > expected primary-expression before ?,? token > > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: error: > > ?PetscScalar? was not declared in this scope > > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: error: > > expected primary-expression before ?,? token > > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: error: > > expected primary-expression before ?)? token > > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: error: > > initializer expression list treated as compound expression > > > > > > Could you please give me some idea to deal with these problems? > > > > Kind regards, > > Zhengyong Ren > > > > > -- Zhengyong Ren AUG Group, Institute of Geophysics Department of Geosciences, ETH Zurich NO H 47 Sonneggstrasse 5 CH-8092, Z?rich, Switzerland Tel: +41 44 633 37561 e-mail: zhengyong.ren at aug.ig.erdw.ethz.ch Gmail: renzhengyong at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From renzhengyong at gmail.com Thu May 20 07:33:19 2010 From: renzhengyong at gmail.com (RenZhengYong) Date: Thu, 20 May 2010 14:33:19 +0200 Subject: [petsc-users] complex number issues In-Reply-To: References: Message-ID: On Wed, May 19, 2010 at 8:56 PM, Satish Balay wrote: > I don't understand the code change - > I do not understand either. In my codes, there are a lots of mathematically complex number operators such exp, *, +, - etc. I do not understand why c++ std::complex <> is so slow compared to C99 complex number even with "-O3" flag on. but if you want to use c99 > complex - you can build petsc with: > > --with-scalar-type=complex --with-clanguage=c [which is the default for > clanguage] > > And no code-change needed for PETSc. However if your code uses c++ - > you might have to address that.. > > Satish > > On Wed, 19 May 2010, RenZhengYong wrote: > > > Hi, PETSc team, > > I wrote a C++ code dealing with electromagnetic problems, which can > smoothly > > call PETSc complied by using " --with-scalar-type=complex > > --with-clanguage=cxx" . > > To improve the performances of the C++ code, I changed the complex number > of > > type from std::complex to double _Complex (C99 standard, roughly > > 30~50 times faster compared to std::complex). But when I find > that > > the C99 complex number head file will lead to the following > > kinds of errors" > > > > /usr/include/c++/4.3/complex:57: error: expected identifier before > > ?__complex__? > > /usr/include/c++/4.3/complex:58: error: expected identifier before > > ?__complex__? > > /usr/include/c++/4.3/complex:58: error: expected unqualified-id before > ? > token > > /usr/include/c++/4.3/complex:59: error: expected identifier before > > ?__complex__? > > /usr/include/c++/4.3/complex:59: error: expected unqualified-id before > ? > token > > /usr/include/c++/4.3/complex:60: error: expected identifier before > > ?__complex__? > > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:157: error: > > expected unqualified-id before ?__complex__? > > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:306: error: > > ?PetscScalar? does not name a type > > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:339: error: > > ?PetscScalar? does not name a type > > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: warning: > > ?PetscGlobalSum? initialized and declared ?extern? > > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: error: > > ?PetscScalar? was not declared in this scope > > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: error: > > expected primary-expression before ?,? token > > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: error: > > ?PetscScalar? was not declared in this scope > > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: error: > > expected primary-expression before ?,? token > > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: error: > > expected primary-expression before ?)? token > > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: error: > > initializer expression list treated as compound expression > > > > > > Could you please give me some idea to deal with these problems? > > > > Kind regards, > > Zhengyong Ren > > > > > -- Zhengyong Ren AUG Group, Institute of Geophysics Department of Geosciences, ETH Zurich NO H 47 Sonneggstrasse 5 CH-8092, Z?rich, Switzerland Tel: +41 44 633 37561 e-mail: zhengyong.ren at aug.ig.erdw.ethz.ch Gmail: renzhengyong at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu May 20 13:54:45 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 20 May 2010 13:54:45 -0500 Subject: [petsc-users] complex number issues In-Reply-To: References: Message-ID: <4791A9BF-EF34-4F54-B986-4B84D5D47DE5@mcs.anl.gov> On May 20, 2010, at 7:24 AM, RenZhengYong wrote: > > On Wed, May 19, 2010 at 8:56 PM, Satish Balay wrote: > I don't understand the code change - but if you want to use c99 > complex - you can build petsc with: > > --with-scalar-type=complex --with-clanguage=c [which is the default for clanguage] > > And no code-change needed for PETSc. However if your code uses c++ - > you might have to address that.. > Badly, my code uses C++. I compiled PETSc with "--with-scalar-type=complex --with-clanguage=c" and I > added > include ${PETSC_DIR}/conf/variables > include ${PETSC_DIR}/conf/rules > in my makefile. So, could it possible to specify g++ for my code and to link c-version petsc? > So long as you do not use any C++ feature of PETSc you can do this. BUT C's complex numbers are C's and may not work with a C++ compiler. With C++ code you may simply need to use the C++ complex type, this has nothing to do with PETSc. How do you know that C's complex is 50 times faster than C++'s? That is not our experience. Barry > kind regards, > Zhengyong > > Satish > > On Wed, 19 May 2010, RenZhengYong wrote: > > > Hi, PETSc team, > > I wrote a C++ code dealing with electromagnetic problems, which can smoothly > > call PETSc complied by using " --with-scalar-type=complex > > --with-clanguage=cxx" . > > To improve the performances of the C++ code, I changed the complex number of > > type from std::complex to double _Complex (C99 standard, roughly > > 30~50 times faster compared to std::complex). But when I find that > > the C99 complex number head file will lead to the following > > kinds of errors" > > > > /usr/include/c++/4.3/complex:57: error: expected identifier before > > ?__complex__? > > /usr/include/c++/4.3/complex:58: error: expected identifier before > > ?__complex__? > > /usr/include/c++/4.3/complex:58: error: expected unqualified-id before ? > token > > /usr/include/c++/4.3/complex:59: error: expected identifier before > > ?__complex__? > > /usr/include/c++/4.3/complex:59: error: expected unqualified-id before ? > token > > /usr/include/c++/4.3/complex:60: error: expected identifier before > > ?__complex__? > > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:157: error: > > expected unqualified-id before ?__complex__? > > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:306: error: > > ?PetscScalar? does not name a type > > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:339: error: > > ?PetscScalar? does not name a type > > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: warning: > > ?PetscGlobalSum? initialized and declared ?extern? > > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: error: > > ?PetscScalar? was not declared in this scope > > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: error: > > expected primary-expression before ?,? token > > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: error: > > ?PetscScalar? was not declared in this scope > > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: error: > > expected primary-expression before ?,? token > > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: error: > > expected primary-expression before ?)? token > > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: error: > > initializer expression list treated as compound expression > > > > > > Could you please give me some idea to deal with these problems? > > > > Kind regards, > > Zhengyong Ren > > > > > > > > -- > Zhengyong Ren > AUG Group, Institute of Geophysics > Department of Geosciences, ETH Zurich > NO H 47 Sonneggstrasse 5 > CH-8092, Z?rich, Switzerland > Tel: +41 44 633 37561 > e-mail: zhengyong.ren at aug.ig.erdw.ethz.ch > Gmail: renzhengyong at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From renzhengyong at gmail.com Thu May 20 14:17:47 2010 From: renzhengyong at gmail.com (RenZhengYong) Date: Thu, 20 May 2010 21:17:47 +0200 Subject: [petsc-users] complex number issues In-Reply-To: <4791A9BF-EF34-4F54-B986-4B84D5D47DE5@mcs.anl.gov> References: <4791A9BF-EF34-4F54-B986-4B84D5D47DE5@mcs.anl.gov> Message-ID: On Thu, May 20, 2010 at 8:54 PM, Barry Smith wrote: > > On May 20, 2010, at 7:24 AM, RenZhengYong wrote: > > > On Wed, May 19, 2010 at 8:56 PM, Satish Balay wrote: > >> I don't understand the code change - but if you want to use c99 >> complex - you can build petsc with: >> >> --with-scalar-type=complex --with-clanguage=c [which is the default for >> clanguage] >> >> And no code-change needed for PETSc. However if your code uses c++ - >> you might have to address that.. >> > Badly, my code uses C++. I compiled PETSc with > "--with-scalar-type=complex --with-clanguage=c" and I > added > include ${PETSC_DIR}/conf/variables > include ${PETSC_DIR}/conf/rules > in my makefile. So, could it possible to specify g++ for my code and to > link c-version petsc? > > So long as you do not use any C++ feature of PETSc you can do this. BUT > C's complex numbers are C's and may not work with a C++ compiler. With C++ > code you may simply need to use the C++ complex type, this has nothing to do > with PETSc. > > Several minutes ago, I successfully linked the c-version petsc in my c++ code which uses C99 complex number. I did not change anything, except for writing a proper makefile to call petsc as a library, :). Now, it works. How do you know that C's complex is 50 times faster than C++'s? That is > not our experience. > I wrote a small code which only has two loops. In the inner loop, there are several complex-value based vector's operations such as vector's product and cross. with "-O3" flag on, the code using C99 complex number is much faster than C++'s complex class. I do not know the reason either. > Barry > > > kind regards, > Zhengyong > >> >> Satish >> >> On Wed, 19 May 2010, RenZhengYong wrote: >> >> > Hi, PETSc team, >> > I wrote a C++ code dealing with electromagnetic problems, which can >> smoothly >> > call PETSc complied by using " --with-scalar-type=complex >> > --with-clanguage=cxx" . >> > To improve the performances of the C++ code, I changed the complex >> number of >> > type from std::complex to double _Complex (C99 standard, roughly >> > 30~50 times faster compared to std::complex). But when I find >> that >> > the C99 complex number head file will lead to the following >> > kinds of errors" >> > >> > /usr/include/c++/4.3/complex:57: error: expected identifier before >> > ?__complex__? >> > /usr/include/c++/4.3/complex:58: error: expected identifier before >> > ?__complex__? >> > /usr/include/c++/4.3/complex:58: error: expected unqualified-id before >> ?> > token >> > /usr/include/c++/4.3/complex:59: error: expected identifier before >> > ?__complex__? >> > /usr/include/c++/4.3/complex:59: error: expected unqualified-id before >> ?> > token >> > /usr/include/c++/4.3/complex:60: error: expected identifier before >> > ?__complex__? >> > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:157: error: >> > expected unqualified-id before ?__complex__? >> > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:306: error: >> > ?PetscScalar? does not name a type >> > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:339: error: >> > ?PetscScalar? does not name a type >> > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: warning: >> > ?PetscGlobalSum? initialized and declared ?extern? >> > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: error: >> > ?PetscScalar? was not declared in this scope >> > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: error: >> > expected primary-expression before ?,? token >> > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: error: >> > ?PetscScalar? was not declared in this scope >> > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: error: >> > expected primary-expression before ?,? token >> > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: error: >> > expected primary-expression before ?)? token >> > /home/Zhengyong/packages/petsc-3.1-p1/include/petscmath.h:469: error: >> > initializer expression list treated as compound expression >> > >> > >> > Could you please give me some idea to deal with these problems? >> > >> > Kind regards, >> > Zhengyong Ren >> > >> > >> > > > > -- > Zhengyong Ren > AUG Group, Institute of Geophysics > Department of Geosciences, ETH Zurich > NO H 47 Sonneggstrasse 5 > CH-8092, Z?rich, Switzerland > Tel: +41 44 633 37561 > e-mail: zhengyong.ren at aug.ig.erdw.ethz.ch > Gmail: renzhengyong at gmail.com > > > -- Zhengyong Ren AUG Group, Institute of Geophysics Department of Geosciences, ETH Zurich NO H 47 Sonneggstrasse 5 CH-8092, Z?rich, Switzerland Tel: +41 44 633 37561 e-mail: zhengyong.ren at aug.ig.erdw.ethz.ch Gmail: renzhengyong at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From recrusader at gmail.com Thu May 20 15:32:06 2010 From: recrusader at gmail.com (Yujie) Date: Thu, 20 May 2010 15:32:06 -0500 Subject: [petsc-users] several MPIDense functions Message-ID: Dear PETSc developers, Currently, I am using PETSc-3.0.0-p8 version for my application. There are not MatCojugate, MatRealPart, MatImaginaryPart for SEQ and MPIDense. Since some revisions in 3.1 version result in large changes in my codes, I don't want to update PETSc to 3.1 version currently. Thank you very much for your previous help. I have added relevant codes for SEQDense. Now, I use similar method to add corresponding functions for MPIDense. However, when I compiled the codes, I get the following errors: " libfast in: /home/yujie/codes/petsc-3.0.0-p8_complex_dbg/src/mat/impls/dense/seq/ftn-custom libfast in: /home/yujie/codes/petsc-3.0.0-p8_complex_dbg/src/mat/impls/dense/mpi mpidense.c:1580: error: invalid conversion from 'PetscErrorCode (*)(_p_Mat*)' to 'PetscErrorCode (*)(_p_Mat*, _p_Mat*, _p_Mat*)' mpidense.c:1580: error: invalid conversion from 'PetscErrorCode (*)(_p_Mat*)' to 'PetscErrorCode (*)(_p_Mat*, PetscInt, const PetscScalar*)' /usr/bin/ar: mpidense.o: No such file or directory libfast in: /home/yujie/codes/petsc-3.0.0-p8_complex_dbg/src/mat/impls/dense/mpi/ftn-auto " Could you give me some help? Thanks a lot. The codes are as following, for "static struct _MatOps MatOps_Values" I add " /*95*/ 0, 0, 0, 0, 0, /*100*/0, 0, 0, MatConjugate_MPIDense, //by Yujie 0, /*105*/0, MatRealPart_MPIDense, //by Yujie MatImaginaryPart_MPIDense, 0, 0, 0 }; //here is Line 1580!!! " The codes for relevant functions: " //---------------------------------------------------------------------------------------- #undef __FUNCT__ #define __FUNCT__ "MatConjugate_MPIDense" static PetscErrorCode MatConjugate_MPIDense(Mat mat) { Mat_MPIDense *a = (Mat_MPIDense *)mat->data; PetscFunctionBegin; MatConjugate(a->A); PetscFunctionReturn(0); } #undef __FUNCT__ #define __FUNCT__ "MatRealPart_MPIDense" static PetscErrorCode MatRealPart_MPIDense(Mat A) { Mat_MPIDense *a = (Mat_MPIDense*)A->data; PetscFunctionBegin; MatRealPart(a->A); PetscFunctionReturn(0); } #undef __FUNCT__ #define __FUNCT__ "MatImaginaryPart_MPIDense" static PetscErrorCode MatImaginaryPart_MPIDense(Mat A) { Mat_MPIDense *a = (Mat_MPIDense*)A->data; PetscFunctionBegin; MatImaginaryPart(a->A); PetscFunctionReturn(0); } //------------------------------------------------------------------------------------------- " Regards, Yujie From jed at 59A2.org Thu May 20 15:58:57 2010 From: jed at 59A2.org (Jed Brown) Date: Thu, 20 May 2010 22:58:57 +0200 Subject: [petsc-users] several MPIDense functions In-Reply-To: References: Message-ID: <8739xmgui6.fsf@59A2.org> On Thu, 20 May 2010 15:32:06 -0500, Yujie wrote: > Dear PETSc developers, > > Currently, I am using PETSc-3.0.0-p8 version for my application. There > are not MatCojugate, MatRealPart, MatImaginaryPart for SEQ and > MPIDense. > > Since some revisions in 3.1 version result in large changes in my > codes, I don't want to update PETSc to 3.1 version currently. This is almost certainly an indexing error, I suggest looking carefully at the definition in struct _MatOps. Your error message has that line number because that is the end of the initializer and GCC's parser doesn't retain the information required to easily produce a better message. Clang is better in this case, you should see something like: mpidense.c:1580:8: warning: incompatible pointer types initializing 'PetscErrorCode (Mat)', expected 'PetscErrorCode (*)(Mat, Mat, PetscReal, Mat *)' [-pedantic] MatConjugate_MPIDense, ^~~~~~~~~~~~~~~~~~~~~ 1 diagnostic generated. Jed From recrusader at gmail.com Thu May 20 16:17:20 2010 From: recrusader at gmail.com (Yujie) Date: Thu, 20 May 2010 16:17:20 -0500 Subject: [petsc-users] several MPIDense functions In-Reply-To: <8739xmgui6.fsf@59A2.org> References: <8739xmgui6.fsf@59A2.org> Message-ID: Thank you very much for your reply, Jed. Actually, I have checked struct_MatOps in matimpl.h file. I find "PetscErrorCode(*)(Mat, Mat, Mat)" is for (*ptapnumeric); PetscErrorCode(*)(Mat, PetscInt, const PetscScalar) is for (*setvaluesrow)" Therefore, I revise the codes in MPIDense.c from " /*100*/0, 0, 0, MatConjugate_MPIDense, /*105*/0, 0, MatRealPart_MPIDense, MatImaginaryPart_MPIDense, 0, " to " /*100*/0, 0, 0, 0, //here add "0" MatConjugate_MPIDense, 0, //here add "0" /*105*/0, 0, MatRealPart_MPIDense, MatImaginaryPart_MPIDense, 0, " It works! However, when I compare the codes in seqdense.c and mpidense.c. They are different. For successful compilation In seqdense.c " /*100*/0, 0, 0, MatConjugate_SeqDense, MatSetSizes_SeqDense, /*104*/0, MatRealPart_SeqDense, MatImaginaryPart_SeqDense, 0, 0, " In mpidense.c " /*100*/0, 0, 0, 0, MatConjugate_MPIDense, //by Yujie, 05/19/10 UT 0, /*105*/0, 0, MatRealPart_MPIDense, //by Yujie, 05/19/10 UT MatImaginaryPart_MPIDense, 0, " At least, there are "0" and "MatSetSizes" between MatConjugate and MatRealPart in seqdense.c; there are three "0" between MatConjugate and MatRealPart (why not be two "0") in mpidense.c? Thanks a lot. Regards, Yujie On Thu, May 20, 2010 at 3:58 PM, Jed Brown wrote: > On Thu, 20 May 2010 15:32:06 -0500, Yujie wrote: >> Dear PETSc developers, >> >> Currently, I am using PETSc-3.0.0-p8 version for my application. There >> are not MatCojugate, MatRealPart, MatImaginaryPart for SEQ and >> MPIDense. >> >> Since some revisions in 3.1 version result in large changes in my >> codes, I don't want to update PETSc to 3.1 version currently. > > This is almost certainly an indexing error, I suggest looking carefully > at the definition in struct _MatOps. ?Your error message has that line > number because that is the end of the initializer and GCC's parser > doesn't retain the information required to easily produce a better > message. ?Clang is better in this case, you should see something like: > > mpidense.c:1580:8: warning: incompatible pointer types initializing 'PetscErrorCode (Mat)', expected 'PetscErrorCode (*)(Mat, Mat, PetscReal, Mat *)' > ? ? ?[-pedantic] > ? ? ? MatConjugate_MPIDense, > ? ? ? ^~~~~~~~~~~~~~~~~~~~~ > 1 diagnostic generated. > > > Jed > From jed at 59A2.org Thu May 20 16:20:15 2010 From: jed at 59A2.org (Jed Brown) Date: Thu, 20 May 2010 23:20:15 +0200 Subject: [petsc-users] several MPIDense functions In-Reply-To: <8739xmgui6.fsf@59A2.org> References: <8739xmgui6.fsf@59A2.org> Message-ID: <87wruyfey8.fsf@59A2.org> On Thu, 20 May 2010 22:58:57 +0200, Jed Brown wrote: > This is almost certainly an indexing error, I suggest looking carefully > at the definition in struct _MatOps. It's sad that we can't use C99 designated initializers here. Jed From jed at 59a2.org Thu May 20 16:22:00 2010 From: jed at 59a2.org (Jed Brown) Date: Thu, 20 May 2010 23:22:00 +0200 Subject: [petsc-users] several MPIDense functions In-Reply-To: References: <8739xmgui6.fsf@59A2.org> Message-ID: <87tyq2fevb.fsf@59A2.org> On Thu, 20 May 2010 16:17:20 -0500, Yujie wrote: > At least, there are "0" and "MatSetSizes" between MatConjugate and > MatRealPart in seqdense.c; > there are three "0" between MatConjugate and MatRealPart (why not be > two "0") in mpidense.c? The indexing is wrong, run in a debugger and see which function gets called. It won't be the one you expect (but the skew should tell you where you are off). Or just use petsc-3.1 which has this feature. Jed From recrusader at gmail.com Thu May 20 18:23:51 2010 From: recrusader at gmail.com (Yujie) Date: Thu, 20 May 2010 18:23:51 -0500 Subject: [petsc-users] several MPIDense functions In-Reply-To: <87tyq2fevb.fsf@59A2.org> References: <8739xmgui6.fsf@59A2.org> <87tyq2fevb.fsf@59A2.org> Message-ID: Dear Jed, Thank you for your help. I have debugged the codes. However, the errors appear when running Line 135 of the following codes: "126: PetscErrorCode MatRealPart(Mat mat) 127: { 133: if (!mat->assembled) SETERRQ(PETSC_ERR_ARG_WRONGSTATE,"Not for unassembled matrix"); 134: if (mat->factor) SETERRQ(PETSC_ERR_ARG_WRONGSTATE,"Not for factored matrix"); 135: if (!mat->ops->realpart) SETERRQ1(PETSC_ERR_SUP,"Mat type %s",((PetscObject)mat)->type_name); 136: MatPreallocated(mat); 137: (*mat->ops->realpart)(mat); 138: return(0); 139: }" That mean there is not MatRealPart function for MPIDense. How to check which functions I have added some functions for? Actually, I am always curious that I didn't change the functions before MatConjugate() in struct MatOps_Values of MPIDense.c. I just added MatConjugate, MatRealPart, MatImaginaryPart at the end of MatOps_Values. I don't know where is the errors from? Could you give me any comments? thanks again. Regards, Yujie On Thu, May 20, 2010 at 4:22 PM, Jed Brown wrote: > On Thu, 20 May 2010 16:17:20 -0500, Yujie wrote: >> At least, there are "0" and "MatSetSizes" between MatConjugate and >> MatRealPart in seqdense.c; >> there are three "0" between MatConjugate and MatRealPart (why not be >> two "0") in mpidense.c? > > The indexing is wrong, run in a debugger and see which function gets > called. ?It won't be the one you expect (but the skew should tell you > where you are off). ?Or just use petsc-3.1 which has this feature. > > Jed > From jed at 59a2.org Thu May 20 18:31:48 2010 From: jed at 59a2.org (Jed Brown) Date: Fri, 21 May 2010 01:31:48 +0200 Subject: [petsc-users] several MPIDense functions In-Reply-To: References: <8739xmgui6.fsf@59A2.org> <87tyq2fevb.fsf@59A2.org> Message-ID: <87ljbef8uz.fsf@59A2.org> On Thu, 20 May 2010 18:23:51 -0500, Yujie wrote: > Dear Jed, > > Thank you for your help. I have debugged the codes. However, the > errors appear when running Line 135 of the following codes: > > "126: PetscErrorCode MatRealPart(Mat mat) > 127: { > > 133: if (!mat->assembled) SETERRQ(PETSC_ERR_ARG_WRONGSTATE,"Not for > unassembled matrix"); > 134: if (mat->factor) SETERRQ(PETSC_ERR_ARG_WRONGSTATE,"Not for > factored matrix"); > 135: if (!mat->ops->realpart) SETERRQ1(PETSC_ERR_SUP,"Mat type > %s",((PetscObject)mat)->type_name); > 136: MatPreallocated(mat); > 137: (*mat->ops->realpart)(mat); > 138: return(0); > 139: }" > > That mean there is not MatRealPart function for MPIDense. How to check > which functions I have added some functions for? For best readability, do (gdb) set print pretty Then look at the ops table. (gdb) print *mat->ops Locate the slots that you are actually setting, and the ones that you meant to set. Jed From Chun.SUN at 3ds.com Fri May 21 10:37:39 2010 From: Chun.SUN at 3ds.com (SUN Chun) Date: Fri, 21 May 2010 11:37:39 -0400 Subject: [petsc-users] a subtle usage of Vec In-Reply-To: <87ljbef8uz.fsf@59A2.org> References: <8739xmgui6.fsf@59A2.org><87tyq2fevb.fsf@59A2.org> <87ljbef8uz.fsf@59A2.org> Message-ID: <2545DC7A42DF804AAAB2ADA5043D57DA9EC3FD@CORP-CLT-EXB01.ds> Hi PETSc developers, I know this might be not good but I just want to hear confirmation from you guys. I'm creating Vec with arrays (both SEQ and MPI) which I allocate myself. However, I'm keeping the pointer, and later on whenever I want to change that vector's values, instead of calling VecSetValues, I just use my pointer and directly access it. And feeling guilty afterwards, I call VecAssembly even though I know it should do nothing. So far this hasn't produce any problem to me but I'm afraid it will do so. Would you please confirm that this usage is acceptable or not? If not acceptable, what's the potential threat? The reason is the following: In the same code, I have a KSP object which I keep using with multiple solves. At some point, I'm getting a -8, which is fine. What is not fine is, if I delete this KSP object, create a new one, re-setup from options, I got convergence. But setting the same bunch of these options on the old KSP object still gives me -8. I'm suspecting memory corruption at some level, but all I can suspect is the Vec usage. Thanks a lot, Chun This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged. If you are not one of the named recipients or have received this email in error, (i) you should not read, disclose, or copy it, (ii) please notify sender of your receipt by reply email and delete this email and all attachments, (iii) Dassault Systemes does not accept or assume any liability or responsibility for any use of or reliance on this email.For other languages, go to http://www.3ds.com/terms/email-disclaimer. From bsmith at mcs.anl.gov Fri May 21 10:59:26 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 21 May 2010 10:59:26 -0500 Subject: [petsc-users] a subtle usage of Vec In-Reply-To: <2545DC7A42DF804AAAB2ADA5043D57DA9EC3FD@CORP-CLT-EXB01.ds> References: <8739xmgui6.fsf@59A2.org><87tyq2fevb.fsf@59A2.org> <87ljbef8uz.fsf@59A2.org> <2545DC7A42DF804AAAB2ADA5043D57DA9EC3FD@CORP-CLT-EXB01.ds> Message-ID: <45A89EF5-9085-463D-96E9-DA5DDE382D81@mcs.anl.gov> On May 21, 2010, at 10:37 AM, SUN Chun wrote: > Hi PETSc developers, > > > I know this might be not good but I just want to hear confirmation from you guys. I'm creating Vec with arrays (both SEQ and MPI) which I allocate myself. However, I'm keeping the pointer, and later on whenever I want to change that vector's values, instead of calling VecSetValues, I just use my pointer and directly access it. And feeling guilty afterwards, I call VecAssembly even though I know it should do nothing. So far this hasn't produce any problem to me but I'm afraid it will do so. Would you please confirm that this usage is acceptable or not? If not acceptable, what's the potential threat? > You DO NEED to call the VecAssembly. Make sure you call that after every time you change values. Then I think it should be ok. > The reason is the following: In the same code, I have a KSP object which I keep using with multiple solves. At some point, I'm getting a -8, which is fine. What is not fine is, if I delete this KSP object, create a new one, re-setup from options, I got convergence. But setting the same bunch of these options on the old KSP object still gives me -8. I'm suspecting memory corruption at some level, but all I can suspect is the Vec usage. Run with valgrind to check for memory corruption http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind During the KSP solve you do not know which vectors are being used so you shouldn't be changing values directly in your array. (For example in a MatMult() or PCApply() if you are using a matshell or pcshell). Barry > > > Thanks a lot, > Chun > > > This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged. If you are not one of the named recipients or have received this email in error, (i) you should not read, disclose, or copy it, (ii) please notify sender of your receipt by reply email and delete this email and all attachments, (iii) Dassault Systemes does not accept or assume any liability or responsibility for any use of or reliance on this email.For other languages, go to http://www.3ds.com/terms/email-disclaimer. From hxie at umn.edu Sun May 23 13:01:00 2010 From: hxie at umn.edu (hxie at umn.edu) Date: 23 May 2010 13:01:00 -0500 Subject: [petsc-users] left and right preconditioners In-Reply-To: References: Message-ID: Hi, I want to use both left and right preconditioners for a linear system. Both the preconditioners are based on part of the matrix. How can I implement it? I can construct the preconditioner matrices. Thanks. Bests, Hui From bsmith at mcs.anl.gov Sun May 23 13:13:59 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 23 May 2010 13:13:59 -0500 Subject: [petsc-users] left and right preconditioners In-Reply-To: References: Message-ID: The PETSc Krylov methods are not coded to take directly both a left and right preconditioner. Therefor the way to use both left and right preconditioning together is to "fake" it by using a Mat that actually applies right preconditioner followed by the operator followed by the left preconditioner. Take a look at src/ksp/pc/impls/eisen/eisen.c and mimic that code. (basically copy all the subroutines and in that file and modify each one to your specific needs.) Barry On May 23, 2010, at 1:01 PM, hxie at umn.edu wrote: > Hi, > > I want to use both left and right preconditioners for a linear system. Both the preconditioners are based on part of the matrix. How can I implement it? I can construct the preconditioner matrices. Thanks. > > Bests, > Hui From hxie at umn.edu Sun May 23 22:06:40 2010 From: hxie at umn.edu (hxie at umn.edu) Date: 23 May 2010 22:06:40 -0500 Subject: [petsc-users] left and right preconditioners In-Reply-To: References: Message-ID: Hi, Thanks. That is not easy for me. I will move the right preconditioner to the left which will lead to a similar preconditioned system with almost the same eigenvalues. Is there any example to do the multiplicative composite preconditioners on different preconditioner matrices from the original matrix? Bests, Hui ------------------ Barry Smith bsmith at mcs.anl.gov Sun May 23 13:13:59 CDT 2010 * Previous message: [petsc-users] left and right preconditioners * Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] The PETSc Krylov methods are not coded to take directly both a left and right preconditioner. Therefor the way to use both left and right preconditioning together is to "fake" it by using a Mat that actually applies right preconditioner followed by the operator followed by the left preconditioner. Take a look at src/ksp/pc/impls/eisen/eisen.c and mimic that code. (basically copy all the subroutines and in that file and modify each one to your specific needs.) Barry On May 23, 2010, at 1:01 PM, hxie at umn.edu wrote: > Hi, > > I want to use both left and right preconditioners for a linear system. > Both the preconditioners are based on part of the matrix. How can I > implement it? I can construct the preconditioner matrices. Thanks. > > Bests, > Hui From mirzadeh at gmail.com Mon May 24 13:04:06 2010 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Mon, 24 May 2010 11:04:06 -0700 Subject: [petsc-users] Interfacing PETSc Message-ID: Dear all, My name is Mohammad and I am a PhD student at the University of California, Santa Barbara. I am currently working on a code that uses OcTree data structures to generate adaptive Cartesian grids for finite difference approximations to PDEs. I am interested in making my code parallel by using PETSc. However there are some issues I am not sure about. My code is relatively a huge one, above 35-40K of lines, and is written in c++ so that it enables us to use all good features like objects and templates. There are several places I can benefit from PETSc. The most obvious one, and easiest to implement, is just to use PETSc as my linear solver. i.e so everything in serial but only solve the linear system in parallel using PETSc. However, this is not enough and I'd like to do more. The problem, however, is that I am using lots of data structures and objects and my question is the possibility of using PETSc along with them, i.e is it possible to write a kind of interface that translates my objects to PETSc variables? For instance, I have an object called OctCell, which is an octree type of cell which has a lot of information inside (what are the neighbors, what are connected nodes, who is the parent of this cell, who are the childs, etc). My grid, is then generated consisting of such (and other objects like OctNode and so on) objects, i.e my grid is an array of such objects. Right now, the template feature of c++ allows me to do this very naturally. Now I'm wondering if it is possible to use array data structures in PETSc instead of array types and still pass my objects as some sort of template? (I know PETSc is written in c which does not support template -- still I'm hopeful!) This way, my grid generation would also run in parallel which saves me a considerable amount of time. Thanks in advance and sorry for the lengthy email. -M -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon May 24 18:07:28 2010 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 25 May 2010 09:07:28 +1000 Subject: [petsc-users] Interfacing PETSc In-Reply-To: References: Message-ID: On Tue, May 25, 2010 at 4:04 AM, Mohammad Mirzadeh wrote: > Dear all, > > My name is Mohammad and I am a PhD student at the University of California, > Santa Barbara. I am currently working on a code that uses OcTree data > structures to generate adaptive Cartesian grids for finite difference > approximations to PDEs. I am interested in making my code parallel by using > PETSc. However there are some issues I am not sure about. > Parallelizing an octree is not trivial, and several packages already exist: - DealII from Wolfgang Bangreuth, et. al. - p4est from Carsten Burstedde and Lucas Wilcox - Dendro from George Biros, et. al. I would first understand why I was not using these packages. That would help us respond intelligently. DealII uses PETSc for solvers already, and p4est should be able to interface soon. Another example is the PetFMM code which manages a uniform, parallel octree using the Sieve communication structures from PETSc to calculate a Fast Multipole transform. Matt > My code is relatively a huge one, above 35-40K of lines, and is written in > c++ so that it enables us to use all good features like objects and > templates. There are several places I can benefit from PETSc. The most > obvious one, and easiest to implement, is just to use PETSc as my linear > solver. i.e so everything in serial but only solve the linear system in > parallel using PETSc. However, this is not enough and I'd like to do more. > The problem, however, is that I am using lots of data structures and objects > and my question is the possibility of using PETSc along with them, i.e is > it possible to write a kind of interface that translates my objects to PETSc > variables? > > For instance, I have an object called OctCell, which is an octree type of > cell which has a lot of information inside (what are the neighbors, what are > connected nodes, who is the parent of this cell, who are the childs, etc). > My grid, is then generated consisting of such (and other objects like > OctNode and so on) objects, i.e my grid is an array of such objects. Right > now, the template feature of c++ allows me to do this very naturally. Now > I'm wondering if it is possible to use array data structures in PETSc > instead of array types and still pass my objects as some sort of template? > (I know PETSc is written in c which does not support template -- still I'm > hopeful!) This way, my grid generation would also run in parallel which > saves me a considerable amount of time. > > Thanks in advance and sorry for the lengthy email. > > -M > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From recrusader at gmail.com Mon May 24 20:12:38 2010 From: recrusader at gmail.com (Yujie) Date: Mon, 24 May 2010 20:12:38 -0500 Subject: [petsc-users] MATSEQ* vs MATMPI* in MatLoad Message-ID: Dear PETSc Developers, If I output a binary mat file with MATMPIAIJ, whether I can read it using MatLoad with MATSEQAIJ? Thanks a lot. Regards, Yujie From bsmith at mcs.anl.gov Mon May 24 20:20:15 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 24 May 2010 20:20:15 -0500 Subject: [petsc-users] MATSEQ* vs MATMPI* in MatLoad In-Reply-To: References: Message-ID: <1DC9DD6E-ACB6-45E4-9C1A-35ECE22C72B1@mcs.anl.gov> Sometimes the best way to find something out with computer software is "to just try it". It is faster than asking and often one learns things better through experimentation then by simple being told something. The answer is yes. Barry On May 24, 2010, at 8:12 PM, Yujie wrote: > Dear PETSc Developers, > > If I output a binary mat file with MATMPIAIJ, whether I can read it > using MatLoad with MATSEQAIJ? Thanks a lot. > > Regards, > Yujie From recrusader at gmail.com Mon May 24 20:26:55 2010 From: recrusader at gmail.com (Yujie) Date: Mon, 24 May 2010 20:26:55 -0500 Subject: [petsc-users] MATSEQ* vs MATMPI* in MatLoad In-Reply-To: <1DC9DD6E-ACB6-45E4-9C1A-35ECE22C72B1@mcs.anl.gov> References: <1DC9DD6E-ACB6-45E4-9C1A-35ECE22C72B1@mcs.anl.gov> Message-ID: Thanks so much for your help, Barry. I think I should establish an environment for function tests ;). Regards, Yujie On Mon, May 24, 2010 at 8:20 PM, Barry Smith wrote: > > ?Sometimes the best way to find something out with computer software is "to just try it". It is faster than asking and often one learns things better through experimentation then by simple being told something. > > ? The answer is yes. > > ? Barry > > On May 24, 2010, at 8:12 PM, Yujie wrote: > >> Dear PETSc Developers, >> >> If I output a binary mat file with MATMPIAIJ, whether I can read it >> using MatLoad with MATSEQAIJ? Thanks a lot. >> >> Regards, >> Yujie > > From mark.cheeseman at kaust.edu.sa Wed May 26 07:53:36 2010 From: mark.cheeseman at kaust.edu.sa (Mark Cheeseman) Date: Wed, 26 May 2010 15:53:36 +0300 Subject: [petsc-users] 2nd order time-stepping in PETSc Message-ID: Hello, I am a new user of PETSC and I have a hopefully not too embarassing question. I am trying to integrate and timestep the 3D acoustic wave equation: u_tt = ( c2 ) * LAPLACIAN( u ) Does any of the timestepping operators in PETSc work with a second-order derivative with the one above? Normally, I would just expand u_tt as a finite difference approximation using Leapfrog (or CN or RK ...) and explicitly solve for the u field at timestep n+1. However I would like to use the vector-matrix formulation in PETSc -my goal is to be able to solve the acoustic wave equation explicitly or implicitly (the user would decide with a command-line option). Thanks, Mark -- Mark Patrick Cheeseman Research Scientist KSL (KAUST Supercomputing Laboratory) Building 1, Office #126 King Abdullah University of Science & Technology Thuwal 23955-6900 Kingdom of Saudi Arabia EMAIL : mark.cheeseman at kaust.edu.sa PHONE : +966 (2) 808 0221 (office) +966 (54) 470 1082 (mobile) SKYPE : mark.patrick.cheeseman -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Wed May 26 08:33:25 2010 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Wed, 26 May 2010 08:33:25 -0500 Subject: [petsc-users] 2nd order time-stepping in PETSc In-Reply-To: References: Message-ID: Mark: You need convert your equation into 1st order ODEs and use petsc TS method (see examples under ~petsc/src/ts/examples/ ), or use petsc KSP/SNES method with your own time step control. Hong > I am a new user of PETSC and I have a hopefully not too embarassing > question.? I am trying to integrate and timestep the 3D acoustic wave > equation: > > u_tt = ( c2 ) * LAPLACIAN( u ) > > Does any of the timestepping operators in PETSc work with a second-order > derivative with the one above?? Normally, I would just expand u_tt as a > finite difference approximation using Leapfrog (or CN or RK ...) and > explicitly solve for the u field at timestep n+1.? However I would like to > use the vector-matrix formulation in PETSc -my goal is to be able to solve > the acoustic wave equation explicitly or implicitly (the user would decide > with a command-line option). > > Thanks, > Mark > > -- > Mark Patrick Cheeseman > > Research Scientist > KSL (KAUST Supercomputing Laboratory) > Building 1, Office #126 > King Abdullah University of Science & Technology > Thuwal 23955-6900 > Kingdom of Saudi Arabia > > EMAIL ? : mark.cheeseman at kaust.edu.sa > PHONE : +966 ? (2) 808 0221 (office) > ? ? ? ? ? ? ? +966 (54) 470 1082 (mobile) > SKYPE : mark.patrick.cheeseman > From recrusader at gmail.com Wed May 26 09:12:00 2010 From: recrusader at gmail.com (Yujie) Date: Wed, 26 May 2010 09:12:00 -0500 Subject: [petsc-users] about MatLoad Message-ID: Dear PETSc Developers, Now, I have an existing MPIAIJ mat (I can't change its local dimension). I have a data file for it. I need to use MatLoad to read this file. However, MatLoad decides the local dimension of mat it uses. This local dimension is different with that of my existing mat when I read the file. Do you have any method to help me read this file for the existing mat? thanks a lot. Regards, Yujie From bsmith at mcs.anl.gov Wed May 26 13:04:10 2010 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 26 May 2010 13:04:10 -0500 Subject: [petsc-users] about MatLoad In-Reply-To: References: Message-ID: I am sorry we do not currently have this functionality. We hope to have it in the next few weeks. Barry On May 26, 2010, at 9:12 AM, Yujie wrote: > Dear PETSc Developers, > > Now, I have an existing MPIAIJ mat (I can't change its local > dimension). I have a data file for it. I need to use MatLoad to read > this file. However, MatLoad decides the local dimension of mat it > uses. This local dimension is different with that of my existing mat > when I read the file. Do you have any method to help me read this file > for the existing mat? thanks a lot. > > Regards, > Yujie From recrusader at gmail.com Wed May 26 13:05:02 2010 From: recrusader at gmail.com (Yujie) Date: Wed, 26 May 2010 13:05:02 -0500 Subject: [petsc-users] about MatLoad In-Reply-To: References: Message-ID: Thanks, Barry. I have use "MatGetRow" and "MatSetValues" for it. Regards, Yujie On Wed, May 26, 2010 at 1:04 PM, Barry Smith wrote: > > ? I am sorry we do not currently have this functionality. We hope to have it in the next few weeks. > > ? ?Barry > > On May 26, 2010, at 9:12 AM, Yujie wrote: > >> Dear PETSc Developers, >> >> Now, I have an existing MPIAIJ mat (I can't change its local >> dimension). ?I have a data file for it. I need to use MatLoad to read >> this file. However, MatLoad decides the local dimension of mat it >> uses. This local dimension is different with that of my existing mat >> when I read the file. Do you have any method to help me read this file >> for the existing mat? thanks a lot. >> >> Regards, >> Yujie > > From knepley at gmail.com Wed May 26 16:58:57 2010 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 27 May 2010 07:58:57 +1000 Subject: [petsc-users] 2nd order time-stepping in PETSc In-Reply-To: References: Message-ID: On Wed, May 26, 2010 at 11:33 PM, Hong Zhang wrote: > Mark: > > You need convert your equation into 1st order ODEs and use petsc > TS method (see examples under ~petsc/src/ts/examples/ ), > or use petsc KSP/SNES method with your own time step control. > > Hong > > > I am a new user of PETSC and I have a hopefully not too embarassing > > question. I am trying to integrate and timestep the 3D acoustic wave > > equation: > > > > u_tt = ( c2 ) * LAPLACIAN( u ) > > > > Does any of the timestepping operators in PETSc work with a second-order > > derivative with the one above? Normally, I would just expand u_tt as a > > finite difference approximation using Leapfrog (or CN or RK ...) and > > explicitly solve for the u field at timestep n+1. However I would like > to > > use the vector-matrix formulation in PETSc -my goal is to be able to > solve > > the acoustic wave equation explicitly or implicitly (the user would > decide > > with a command-line option). > > It sounds like you could formulate it to solve for timestep n+1, and then the RHS would depend on timestep n (which PETSc would give you), and timestep n-1. You could save t^{n-1} yourself with a custom Monitor and get it in your RHS function through the user context. Matt > > Thanks, > > Mark > > > > -- > > Mark Patrick Cheeseman > > > > Research Scientist > > KSL (KAUST Supercomputing Laboratory) > > Building 1, Office #126 > > King Abdullah University of Science & Technology > > Thuwal 23955-6900 > > Kingdom of Saudi Arabia > > > > EMAIL : mark.cheeseman at kaust.edu.sa > > PHONE : +966 (2) 808 0221 (office) > > +966 (54) 470 1082 (mobile) > > SKYPE : mark.patrick.cheeseman > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalcinl at gmail.com Wed May 26 19:48:56 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 26 May 2010 21:48:56 -0300 Subject: [petsc-users] 2nd order time-stepping in PETSc In-Reply-To: References: Message-ID: On 26 May 2010 18:58, Matthew Knepley wrote: > On Wed, May 26, 2010 at 11:33 PM, Hong Zhang wrote: >> >> Mark: >> >> You need convert your equation into 1st order ODEs and use petsc >> TS method (see examples under ~petsc/src/ts/examples/ ), >> or use petsc KSP/SNES method with your own time step control. >> >> Hong >> >> > I am a new user of PETSC and I have a hopefully not too embarassing >> > question.? I am trying to integrate and timestep the 3D acoustic wave >> > equation: >> > >> > u_tt = ( c2 ) * LAPLACIAN( u ) >> > >> > Does any of the timestepping operators in PETSc work with a second-order >> > derivative with the one above?? Normally, I would just expand u_tt as a >> > finite difference approximation using Leapfrog (or CN or RK ...) and >> > explicitly solve for the u field at timestep n+1.? However I would like >> > to >> > use the vector-matrix formulation in PETSc -my goal is to be able to >> > solve >> > the acoustic wave equation explicitly or implicitly (the user would >> > decide >> > with a command-line option). >> > > It sounds like you could formulate it to solve for timestep n+1, and then > the RHS > would depend on timestep n (which PETSc would give you), and timestep n-1. > You > could save t^{n-1} yourself with a custom Monitor and get it in your RHS > function > through the user context. > ?? Matt > What about using TSSet{Pre|Post}Step() ? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From jed at 59A2.org Thu May 27 06:03:47 2010 From: jed at 59A2.org (Jed Brown) Date: Thu, 27 May 2010 13:03:47 +0200 Subject: [petsc-users] 2nd order time-stepping in PETSc In-Reply-To: References: Message-ID: <87vda9y5bg.fsf@59A2.org> On Wed, 26 May 2010 21:48:56 -0300, Lisandro Dalcin wrote: > On 26 May 2010 18:58, Matthew Knepley wrote: > > It sounds like you could formulate it to solve for timestep n+1, and > > then the RHS would depend on timestep n (which PETSc would give > > you), and timestep n-1. You could save t^{n-1} yourself with a > > custom Monitor and get it in your RHS function through the user > > context. ?? Matt > > > > What about using TSSet{Pre|Post}Step() ? The problem is that this sort of ad-hoc dependence on the last step causes the application to assume things about the integrator in use. You can of course reduce the second-order problem to a first-order system, but at the expense of increasing the length of the Krylov vectors and ruining potential symmetry. This is currently the only general method supported by TS, but I don't like it. An alternative is to have the method perform differencing in auxiliary variables, as I proposed a few weeks ago in a different context: http://lists.mcs.anl.gov/pipermail/petsc-dev/2010-April/002671.html Some integrators, like RADAU, recognize special orderings and a flag for this, but I find this especially awkward in parallel and we don't live in an F77 world, so I would prefer to write this capability into the API. Specifically in this case, we have x_tt + f(x) = 0 which with time step \Delta t ~ 1/a and mass matrix M, yields a Jacobian J(x) = a^2 M + f'(x) (I am ignoring the details of the time discretization which involves precise definition of a and the right hand side.) We can transform this as x_t - y = 0 y_t + f(x) = 0 but we now have a Jacobian J(x,y) = [aM, -M; f'(x), aM] which is somewhat unfortunate. The proposal in the message referenced above is not sufficient to transform this into the system you want, but consider /* Set by the user before TSSolve(), called by the integrator at each * stage (maybe lazily). */ typedef PetscErrorCode (*TSAuxiliaryFunction)(TS,Vec X,Vec X_t,Vec Y,void *ctx); PetscErrorCode TSSetAuxiliaryFunction(TS,TSAuxiliaryFunction,void*); /* Called by the user in TSIFunction(), Y is provided directly be the * user's function, Y_t is determined by the integrator using some * differencing of the Y values at prior steps/stages. (This is a * virtual function to be defined by supported implementations.) */ PetscErrorCode TSGetAuxiliaryVector(TS,Vec *Y,Vec *Y_t); The material energy in the referenced message would by Y = Y(X), for Hamiltonian systems you might have Y = Y(X_t) = X_t. I don't think it is possible to avoid leaving the user responsible for differentiating through their auxiliary function when computing a Jacobian, but this is easy for the easy cases (like the wave equation) and when it is actually difficult, I think (naively) it is true complexity and not API awkwardness. What do you think? Jed From luca.mangani at hslu.ch Thu May 27 02:20:55 2010 From: luca.mangani at hslu.ch (Luca Mangani) Date: Thu, 27 May 2010 09:20:55 +0200 Subject: [petsc-users] Many issues solving matrix derived from a CFD coupled Solver Message-ID: <4BFE1D57.7000409@hslu.ch> Good Morning, After many steps I'm not able to solve my coupled matrix with iterative solver. The matrix is defined as the discretization of the Navier Stokes equation in a pressure based form. Also for a small case of 70K cells the linear solver blows up (GMRES(ILU BJACOBI) , BCGS(ILU BJACOBI)) and HYPRE+BOOMERAMG). For the boomerAMG I use this options: -ksp_monitor_true_residual -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_max_iter 5 -pc_hypre_boomeramg_cycle_type V -pc_hypre_boomeramg_relax_type_all SOR/Jacobi -pc_hypre_boomeramg_grid_sweeps_down 3 -pc_hypre_boomeramg_grid_sweeps_up 0 -pc_hypre_boomeramg_grid_sweeps_coarse 3 -pc_hypre_boomeramg_rtol 0.0001 -pc_hypre_boomeramg_coarsen_type modifiedRuge-Stueben -pc_hypre_boomeramg_relax_type_coarse SOR/Jacobi -pc_hypre_boomeramg_interp_type multipass-wts -ksp_max_it 60 -ksp_type richardson -ksp_view but without any success. That's really frustrating because of direct solvers are able to solve the matrix without any problem. Do you have any suggestion of why can I have all this issues? Your help and support could be really appreciated. Thanks in advance Luca -- Hochschule Luzern Technik& Architektur Technikumstrasse 21, CH-6048 Horw T +41 41 349 33 11, F +41 41 349 39 60 Maschinentechnik CC Fluidmechanik und Hydromaschinen Dr. Luca Mangani - PhD Senior Research Fellow T direkt +41 (0) 413493 248 luca.mangani at hslu.ch www.hslu.ch/fmhm From jed at 59A2.org Thu May 27 16:27:29 2010 From: jed at 59A2.org (Jed Brown) Date: Thu, 27 May 2010 23:27:29 +0200 Subject: [petsc-users] Many issues solving matrix derived from a CFD coupled Solver In-Reply-To: <4BFE1D57.7000409@hslu.ch> References: <4BFE1D57.7000409@hslu.ch> Message-ID: <87k4qpxcfy.fsf@59A2.org> Some aspects of your questions come up frequenty, especially the unrealistic expectations from black-box iterative solvers. Hopefully this somewhat in-depth reply will be useful. On Thu, 27 May 2010 09:20:55 +0200, Luca Mangani wrote: > Good Morning, > > After many steps I'm not able to solve my coupled matrix with > iterative solver. The matrix is defined as the discretization of the > Navier Stokes equation in a pressure based form. This form is indefinite. While we often talk about there being no scalable black-box solvers, the statement is especially true for indefinite systems. As an example of the spectacular failure of black-box methods for elliptic systems when applied to other problems, I have observed BoomerAMG being massively singular when applied to a mixed finite element discretization of incompressible flow. This resulted in preconditioned residuals falling in essentially textbook manner, but the unpreconditioned residuals did not converge at all (after satisfying the boundary conditions). It is very important to check this. For *some* incompressible flow problems, it is possible to order unknowns such that incomplete factorization causes useful fill into the pressure-pressure block. Unfortunately, incomplete factorization has very unpredictable properties and is certainly not a robust solution. When playing with orderings, usually you would like all velocity unknowns in a subdomain to come before the pressure unknowns, a number of automatic orderings are also available with -pc_factor_mat_ordering_type. You may also try changing the shift behavior, see -pc_factor_shift_type, using an ILU with pivoting, or using sparse approximate inverse -pc_hypre_type parasails or -pc_type spai. While some of these black-box approaches will be better (worse) than others, and depending on your needs, perhaps even "good enough", none will scale to large problem size or be robust to parameters. They may all fail completely. There are essentially two scalable approaches: 1. Schur complement methods This class contains classical methods like Uzawa and SIMPLE, as well as newer approaches based on approximate commutators. See Elman's 2008 paper "A taxonomy and comparison of parallel block multi-level preconditioners for the incompressible Navier-Stokes equations" for a current overview. These methods break the problem into definite operators for which conventional preconditioners are suitable. It is relatively easy to implement using PCFieldSplit, but note that the approximation to the Schur complement is a crucial choice and involves problem-specific information. 2. Coupled multilevel methods This class contains multigrid methods based on Vanka smoothers as well as domain decomposition methods. It is crucial to define subdomains, interpolants, and smoothers/subdomain solvers that are compatible with the inf-sup condition. These stability requirements are highly dependent on your spatial discretization, therefore it really cannot be encapsulated in a black-box solver. For the multigrid end of the spectrum, see papers by Brandt in the finite difference context or Rannacher and Turek for certain finite element discretizations. At the DD end, you might like papers by Jing Li for the non-overlapping case or Xiao-Chuan Cai for overlapping cases. Note that 1-level Schwarz methods essentially only require a choice of stable subdomains which you can do with PCASMSetLocalSubdomains(). Many other methods can be posed as a multilevel Schwarz method in which case you can use PCMG and define your own problem-specific interpolants. There are currently no scalable algorithms with convergence rates that are independent of Reynolds number, the best cases with the latter class (coupled multigrid or DD) seem to be somewhat less sensitive than the Schur-complement methods. As usual, there are difficulties in the presence of large variation in coefficients or high aspect ratio, though some methods are indpendent or very weakly dependent on these parameters. The methods in class 1 almost always involve more frequent communication, therefore their scalability is likely to be somewhat worse than class 2, though some people have demonstrated quite good weak scalability using class 1. Benzi, Golub, and Liesen's 2005 Acta Numerica review paper may also be of interest. More things are likely to work with non-mixed discretizations, but those discretizations are generally less robust. A group at Sandia has had some luck applying ML to stabilized Q1-Q1 finite element methods, it is important to keep all fields coupled (PETSc will inform ML correctly if you order unknowns [u0,v0,w0,p0,u1,...] and call MatSetBlockSize or use BAIJ, note that this ordering is better for memory performance and smoother strength anyway). Their experience is that this approach is usually faster than approximate commutators, but sometimes fails to converge at very moderate parameter choices. Note that this Q1-Q1 discretization does not exactly enforce incompressibility and this divergence error can be quite large on under-resolved meshes or in the presence of non-smooth boundary conditions. Galerkin least squares methods offer a way to produce a positive definite system from a velocity-pressure formulation, therefore standard solvers can be applied. But note that it generally commits very large divergence errors (see 99% mass loss a few years ago, the newer methods can keep it to a few percent for simple problems), this can be circumvented by choosing inf-sup stable spaces, but then the solver issues become somewhat more complicated again. Once you have decided on a discretization and algorithm from the literature that you would like to implement (or your own variant), we can give suggestions. Please recognize that scalable solvers for incompressible flow is very much an active research topic, you cannot expect black-box solvers to produce good results. If you do not need very large problem sizes, you might want to stick with a direct solver like MUMPS; if do not need extreme parameter choices, you might get by with naive 1-level domain decomposition or coupled algebraic multigrid (especially with collocated discretization). Jed From recrusader at gmail.com Thu May 27 22:02:02 2010 From: recrusader at gmail.com (Yujie) Date: Thu, 27 May 2010 22:02:02 -0500 Subject: [petsc-users] about VecScatterCreate() Message-ID: Dear PETSc Developers, I got the following information from VecScatterCreate() manpage. " PetscErrorCode VecScatterCreate(Vec xin,IS ix,Vec yin,IS iy,VecScatter *newctx) Collective on Vec Input Parameters xin - a vector that defines the shape (parallel data layout of the vector) of vectors from which we scatter yin - a vector that defines the shape (parallel data layout of the vector) of vectors to which we scatter ix - the indices of xin to scatter (if PETSC_NULL scatters all values) iy - the indices of yin to hold results (if PETSC_NULL fills entire vector yin) " My question is about "ix". If xin and yin are parallel Vec, how about ix? Is "ix" an IS only containing local index set or a parallel IS containing all the index set on all the processors (each local index set on its processor)? Thanks a lot. Regards, Yujie From knepley at gmail.com Thu May 27 22:07:02 2010 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 28 May 2010 13:07:02 +1000 Subject: [petsc-users] about VecScatterCreate() In-Reply-To: References: Message-ID: On Fri, May 28, 2010 at 1:02 PM, Yujie wrote: > Dear PETSc Developers, > > I got the following information from VecScatterCreate() manpage. > " > PetscErrorCode VecScatterCreate(Vec xin,IS ix,Vec yin,IS iy,VecScatter > *newctx) > > Collective on Vec > > Input Parameters > xin - a vector that defines the shape (parallel data layout of > the > vector) of vectors from which we scatter > yin - a vector that defines the shape (parallel data layout of > the > vector) of vectors to which we scatter > ix - the indices of xin to scatter (if PETSC_NULL scatters all > values) > iy - the indices of yin to hold results (if PETSC_NULL fills > entire > vector yin) > " > My question is about "ix". If xin and yin are parallel Vec, how about ix? > > Is "ix" an IS only containing local index set or a parallel IS > containing all the index set on all the processors (each local index > set on its processor)? Thanks a lot. > ix contains some indices for that process. The indices can be different on each process. We take the union. Matt > Regards, > Yujie > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From recrusader at gmail.com Thu May 27 22:15:41 2010 From: recrusader at gmail.com (Yujie) Date: Thu, 27 May 2010 22:15:41 -0500 Subject: [petsc-users] about VecScatterCreate() In-Reply-To: References: Message-ID: Thanks, Matt. I think the manpage should point out this. In MatGetSubmatrix(). "isrow" and "iscol" have good explanations. Regards, Yujie On Thu, May 27, 2010 at 10:07 PM, Matthew Knepley wrote: > On Fri, May 28, 2010 at 1:02 PM, Yujie wrote: >> >> Dear PETSc Developers, >> >> I got the following information from VecScatterCreate() manpage. >> " >> PetscErrorCode ?VecScatterCreate(Vec xin,IS ix,Vec yin,IS iy,VecScatter >> *newctx) >> >> Collective on Vec >> >> Input Parameters >> ? ? ? ?xin ? ? - a vector that defines the shape (parallel data layout of >> the >> vector) of vectors from which we scatter >> ? ? ? ?yin ? ? - a vector that defines the shape (parallel data layout of >> the >> vector) of vectors to which we scatter >> ? ? ? ?ix ? ? ?- the indices of xin to scatter (if PETSC_NULL scatters all >> values) >> ? ? ? ?iy ? ? ?- the indices of yin to hold results (if PETSC_NULL fills >> entire >> vector yin) >> " >> My question is about "ix". If xin and yin are parallel Vec, how about ix? >> >> Is "ix" ?an IS only containing local index set or a parallel IS >> containing all the index set on all the processors (each local index >> set on its processor)? Thanks a lot. > > ix contains some indices for that process. The indices can be different on > each process. > We take the union. > ?? Matt > >> >> Regards, >> Yujie > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener > From jed at 59A2.org Mon May 31 06:18:47 2010 From: jed at 59A2.org (Jed Brown) Date: Mon, 31 May 2010 13:18:47 +0200 Subject: [petsc-users] Many issues solving matrix derived from a CFD coupled Solver In-Reply-To: <4C038F15.5070203@hslu.ch> References: <4BFE1D57.7000409@hslu.ch> <87k4qpxcfy.fsf@59A2.org> <4C038F15.5070203@hslu.ch> Message-ID: <87y6f0wc88.fsf@59A2.org> On Mon, 31 May 2010 12:27:33 +0200, Luca Mangani wrote: > Hi Jed, > > first of all thank you for your reply and your support. > > Also in these days a tried different configurations but without any success. > > > Once you have decided on a discretization and algorithm from the > > literature that you would like to implement (or your own variant), we > > can give suggestions. > My solver is based on finite volume discretization with colocated > arrangement for the calculated variables. (I have a paper as reference > and if you wnat I can send you). I suggest finding a preconditioner in the literature that was demonstrated to work for a similar discretization. > The dicretized equations are based on incompressible NS equations in > pressure based form. The matrix is built as discretization of the > velocity and pressure equation solved in a coupled way. The matrix is a > sparse matrix of a dimension 4*Ncells, and the diagonal is always filled > (no saddle point). Your matrix has (after suitable permutation), the form J = [A B; C D] with A nonsymmetric definite (the symmetric part is positive semidefinite) and D negative semidefinite (and probably symmetric). This is an indefinite system and usually referred to more specifically as a (generalized) saddle point problem; see Benzi, Golub & Liesen 2005 if you don't believe me. > The direct solver works well but there isn't any chance to have > convergence for iterative solvers. Chosing bcgs+blockJacobi as PC I > get solution but only for very small cases ( Ncells=5000). Mainly AMG > solvers should be the best solution looking to the other works, but it > does not work at the moment. AMG will definitely not work if the block structure is not respected. I recommend ordering unknowns so that all 4 dofs at each cell come together [u0,v0,w0,p0,u1,v1,w1,p1,...] and call MatSetBlockSize(A,4) (you could also use BAIJ). Then try something like -pc_type ml -pc_ml_BlockScaling -pc_ml_EnergyMinimization 2 and something similar with BoomerAMG. Also let us know if -pc_type asm -sub_pc_type lu works in parallel. If this doesn't work, you will have to follow up on some of my suggestions. Iterative solvers aren't magic and black-box solvers rarely solve interesting problems. Note: I would always use GMRES with a large subspace at this stage. Jed From jed at 59A2.org Mon May 31 08:41:23 2010 From: jed at 59A2.org (Jed Brown) Date: Mon, 31 May 2010 15:41:23 +0200 Subject: [petsc-users] Many issues solving matrix derived from a CFD coupled Solver In-Reply-To: <4C03A581.4010308@hslu.ch> References: <4BFE1D57.7000409@hslu.ch> <87k4qpxcfy.fsf@59A2.org> <4C038F15.5070203@hslu.ch> <87y6f0wc88.fsf@59A2.org> <4C03A581.4010308@hslu.ch> Message-ID: <87sk58w5mk.fsf@59A2.org> On Mon, 31 May 2010 14:03:13 +0200, Luca Mangani wrote: > Dear Jed, > > > The dicretized equations are based on incompressible NS equations in > > pressure based form. The matrix is built as discretization of the > > velocity and pressure equation solved in a coupled way. The matrix is a > > sparse matrix of a dimension 4*Ncells, and the diagonal is always filled > > (no saddle point). > > I solve for the pressure the pressure-equation not continuity, so should > I have no saddle point problem. What is sitting in the pressure-pressure block (D as described in my last message)? Usually this is a stabilization term and does not make the system positive definite. It is possible to have a generalized saddle point form without a Lagrangian. > > AMG will definitely not work if the block structure is not respected. I > > recommend ordering unknowns so that all 4 dofs at each cell come > > together [u0,v0,w0,p0,u1,v1,w1,p1,...] > The pattern and the matrix is arranged exactly in that way > ([u0,v0,w0,p0,u1,v1,w1,p1,...] ) and I received no benefit again with > the options you suggest (ml multigrid). This is not surprising since this approach is quite finicky and very sensitive to the discretization and stabilization. You can try adding (with ML) options like -pc_ml_maxNlevels 2 -mg_levels_pc_type ilu -mg_levels_ksp_type gmres -ksp_type fgmres but I'm not especially optimistic about this. > For the moment the solver is serial. So I cannot do your try, (sorry for > that). You can run ASM in serial with -pc_type asm -pc_asm_blocks 8 -sub_pc_type lu. You could also save the system with -ksp_view_binary and load it in parallel with src/ksp/ksp/examples/tutorials/ex10. > I've noticed also that solving the same coupled matrix with the same > solver type, (same tolerance and PC (GMRES + ILU)) with MATLAB and > comparing, PETSC is slower Matlab is probably pivoting which helps for robustness, but isn't fast or bandwidth friendly, -pc_type hypre -pc_hypre_type euclid is a different ILU that you could try. If ILU is going to be efficient, the pressure dofs probably need to come last (on each subdomain). > and does not converge also with the optimized compiled version, maybe this should be useful. The convergence behavior is different for a PETSC_ARCH built --with-debugging=0 than --with-debugging=1 (default)? If so, please make sure the matrix is actually the same in both cases, save the system with -ksp_view_binary (presumably this occurs for a modest problem size). Send configure.log for both cases, the options you were using, the symptoms you observed, and a link to your saved system to petsc-maint at mcs.anl.gov. Jed From gdiso at ustc.edu Mon May 31 11:11:41 2010 From: gdiso at ustc.edu (Gong Ding) Date: Tue, 1 Jun 2010 00:11:41 +0800 Subject: [petsc-users] Parallel LU decomposition Message-ID: <99EF2E6FB06042769CF1EACA6D152EF1@cogendaeda> Dear all, I need to solve Poisson's equation many times with same Matrix A and different RHS b. The most efficient way is do pre-factorization I think. For serial problem, the PCLU seems work as I want to. However, how to do for parallel situation? Should I use an external LU solver such as superlu_dist? Thanks Gong Ding From jed at 59A2.org Mon May 31 11:18:44 2010 From: jed at 59A2.org (Jed Brown) Date: Mon, 31 May 2010 18:18:44 +0200 Subject: [petsc-users] Parallel LU decomposition In-Reply-To: <99EF2E6FB06042769CF1EACA6D152EF1@cogendaeda> References: <99EF2E6FB06042769CF1EACA6D152EF1@cogendaeda> Message-ID: <87ocfwvycb.fsf@59A2.org> On Tue, 1 Jun 2010 00:11:41 +0800, "Gong Ding" wrote: > Dear all, > I need to solve Poisson's equation many times with same Matrix A and different RHS b. > The most efficient way is do pre-factorization I think. > > For serial problem, the PCLU seems work as I want to. > However, how to do for parallel situation? > Should I use an external LU solver such as superlu_dist? Sure, or MUMPS or PaStiX. As the problem size gets bigger, eventually a multigrid will beat even the triangular solves. Jed From jed at 59A2.org Mon May 31 12:08:54 2010 From: jed at 59A2.org (Jed Brown) Date: Mon, 31 May 2010 19:08:54 +0200 Subject: [petsc-users] Parallel LU decomposition In-Reply-To: <29E44F18A5F34BFC8475F86780F12A7F@cogendaeda> References: <99EF2E6FB06042769CF1EACA6D152EF1@cogendaeda> <87ocfwvycb.fsf@59A2.org> <29E44F18A5F34BFC8475F86780F12A7F@cogendaeda> Message-ID: <87eigsvw0p.fsf@59A2.org> On Tue, 1 Jun 2010 00:50:35 +0800, "Gong Ding" wrote: > Any detailed example? > I'd like to use PETSC routine to build parallel AIJ matrix A and parallel vector b. > and then how to call MUMPS to do the factorization? Almost all examples assemble a parallel matrix and vector. To solve multiple systems, just call KSPSolve() multiple times, the factorization (or other setup) will only be redone after you call KSPSetOperators(). In this case, the parallel solve is -ksp_type preonly -pc_type lu -pc_factor_mat_solver_package mumps The equivalent code is PC pc; ierr = KSPSetType(ksp,KSPPREONLY);CHKERRQ(ierr); ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); ierr = PCSetType(pc,PCLU);CHKERRQ(ierr); ierr = PCFactorSetMatSolverPackage(pc,MATSOLVERMUMPS);CHKERRQ(ierr); if you are allergic to the options database. A complete example is src/ksp/ksp/examples/tutorials/ex16.c I recommend using this interface rather than the lower level interfaces that deal directly with the factors. Jed