From C.Klaij at marin.nl Tue Mar 1 02:28:12 2011 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Tue, 1 Mar 2011 08:28:12 +0000 Subject: [petsc-users] PCDiagonalScale In-Reply-To: References: , , Message-ID: Barry, I do not want to use it for the reason given in the manual. Instead I'm trying to use SIMPLE as a preconditioner for a colocated, cell-centered FV discretization of the incompressible Navier-Stokes equations. I'm using a matfree approach for the matrix and the (right) preconditioner. I would like to scale the system as D A M D^{-1} z = D b but I get the feeling that PCDiagonalScaleSet with GCR doesn't do anything. How about Jed's comment: "GCR also tolerates a variable preconditioner and it does not have such a check. I don't know if that means it can use diagonal scaling or just that someone forgot to check, but you could try it." Is it just that the check is missing for GCR so that it doesn't apply diagonal scaling and doesn't tell me? Chris Message: 3 Date: Thu, 24 Feb 2011 08:08:42 -0600 From: Barry Smith Subject: Re: [petsc-users] PCDiagonalScale To: PETSc users list Message-ID: <699188CA-750D-446A-AD58-C62CB2A75CE2 at mcs.anl.gov> Content-Type: text/plain; charset=us-ascii On Feb 23, 2011, at 9:45 AM, Klaij, Christiaan wrote: > I'm trying to understand the use of PCDiagonalScale since I want to apply additional diagonal scaling when solving my linear system. Do you want to use it exactly for the reason given in PCSetDiagonalScale - Indicates the left scaling to use to apply an additional left and right scaling as needed by certain time-stepping codes. Logically Collective on PC Input Parameters: + pc - the preconditioner context - s - scaling vector Level: intermediate Notes: The system solved via the Krylov method is $ D M A D^{-1} y = D M b for left preconditioning or $ D A M D^{-1} z = D b for right preconditioning dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:C.Klaij at marin.nl T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl From gianmail at gmail.com Tue Mar 1 07:23:23 2011 From: gianmail at gmail.com (Gianluca Meneghello) Date: Tue, 1 Mar 2011 14:23:23 +0100 Subject: [petsc-users] Tridiagonal and pentadiagonal matrices In-Reply-To: References: Message-ID: Hong, thanks for your answer. Please let me ask for some clarifications: - in order to use LAPACK routines for band and tridiagonal matrices, I cannot provide a Petsc Matrix directly to the routine. I must extract the values from the matrix, store them in an array and then provide the array to the LAPACK routine. To access data faster, I can use MatGetArray, provided I know the matrix structure in advance. Am I wrong? - can I ask you some more details (or where to find an example) on the use of the private data structure for efficient assembly? - is there a place where I can find the description of the various matrix types (like this, but with descriptions http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatType.html)? Thanks again Gianluca On 28 February 2011 18:26, Hong Zhang wrote: > Gianluca : >> >> I was looking for the best way to solve tridiagonal and pentadiagonal >> matrix in Petsc. Is there a specific matrix format/solver for these >> kind of systems I should use? > > No. For sequential matrix, you may use LAPACK routines for band or > tridiagonal matrices. >> >> The tridiagonal/pentadiagonal matrix I have to solve corresponds to >> the main 3/5 diagonals of a bigger matrix (if it can help, I'm trying >> to solve a system using block-line Gauss Seidel). I've seen there is >> an easy way to obtain the main diagonal of the matrix >> (MatGetDiagonal). Is there an equivalent way to extract the other >> data? > > You may use MatGetSubMatrix(). For efficient assemble of your submatrix, > you may look into the private date structure (AIJ?) and obtain your submatrix. > For aij format, check aij.h or mpiaij.h for its datastructure. > > Hong >> >> Thanks >> >> Gianluca >> >> -- >> "[Je pense que] l'homme est un monde qui vaut des fois les mondes et >> que les plus ardentes ambitions sont celles qui ont eu l'orgueil de >> l'Anonymat" -- Non omnibus, sed mihi et tibi >> Amedeo Modigliani >> > -- "[Je pense que] l'homme est un monde qui vaut des fois les mondes et que les plus ardentes ambitions sont celles qui ont eu l'orgueil de l'Anonymat" -- Non omnibus, sed mihi et tibi Amedeo Modigliani From u.tabak at tudelft.nl Tue Mar 1 10:44:39 2011 From: u.tabak at tudelft.nl (Umut Tabak) Date: Tue, 01 Mar 2011 17:44:39 +0100 Subject: [petsc-users] how to persist -ksp_monitor_draw Message-ID: <4D6D2277.4000902@tudelft.nl> Dear list, I use the ksp_monitor_draw option for the residual history of the iterative solvers. However, I could not find how to persist the figure on my screen, it just flashes and disappears. I suppose you use gnuplot to plot the info, there is the 'persist' option however how can I use in PETSc? Best regards, Umut -- If I have a thousand ideas and only one turns out to be good, I am satisfied. Alfred Nobel From jed at 59A2.org Tue Mar 1 10:49:18 2011 From: jed at 59A2.org (Jed Brown) Date: Tue, 1 Mar 2011 08:49:18 -0800 Subject: [petsc-users] how to persist -ksp_monitor_draw In-Reply-To: <4D6D2277.4000902@tudelft.nl> References: <4D6D2277.4000902@tudelft.nl> Message-ID: On Tue, Mar 1, 2011 at 08:44, Umut Tabak wrote: > I use the ksp_monitor_draw option for the residual history of the iterative > solvers. However, I could not find how to persist the figure on my screen, > it just flashes and disappears. -draw_pause SEC will leave it on the screen for SEC seconds. Use negative SEC to leave it on the screen indefinitely. > I suppose you use gnuplot to plot the info, there is the 'persist' option > however how can I use in PETSc? X11, not gnuplot. -------------- next part -------------- An HTML attachment was scrubbed... URL: From u.tabak at tudelft.nl Tue Mar 1 10:59:00 2011 From: u.tabak at tudelft.nl (Umut Tabak) Date: Tue, 01 Mar 2011 17:59:00 +0100 Subject: [petsc-users] how to persist -ksp_monitor_draw In-Reply-To: References: <4D6D2277.4000902@tudelft.nl> Message-ID: <4D6D25D4.1010900@tudelft.nl> On 03/01/2011 05:49 PM, Jed Brown wrote: > On Tue, Mar 1, 2011 at 08:44, Umut Tabak > wrote: > > I use the ksp_monitor_draw option for the residual history of the > iterative solvers. However, I could not find how to persist the > figure on my screen, it just flashes and disappears. > > > -draw_pause SEC will leave it on the screen for SEC seconds. Use > negative SEC to leave it on the screen indefinitely. I guess there is a problem with my X11 settings or I am using that in a wrong way, if I use ./ex1 -ksp_type minres -ksp_monitor_draw -draw_pause -1 I get the empty figure hanging on the screen. After clicking with the mouse on the figure, the history advances at each mouse click, is not this strange? I do not get that displayed after the above command. Best, Umut -- If I have a thousand ideas and only one turns out to be good, I am satisfied. Alfred Nobel -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Mar 1 11:07:14 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 1 Mar 2011 11:07:14 -0600 Subject: [petsc-users] how to persist -ksp_monitor_draw In-Reply-To: <4D6D25D4.1010900@tudelft.nl> References: <4D6D2277.4000902@tudelft.nl> <4D6D25D4.1010900@tudelft.nl> Message-ID: <4B7E8036-BA31-4FC0-A7C5-262EE8CB5A42@mcs.anl.gov> On Mar 1, 2011, at 10:59 AM, Umut Tabak wrote: > On 03/01/2011 05:49 PM, Jed Brown wrote: >> On Tue, Mar 1, 2011 at 08:44, Umut Tabak wrote: >> I use the ksp_monitor_draw option for the residual history of the iterative solvers. However, I could not find how to persist the figure on my screen, it just flashes and disappears. >> >> -draw_pause SEC will leave it on the screen for SEC seconds. Use negative SEC to leave it on the screen indefinitely. >> > I guess there is a problem with my X11 settings or I am using that in a wrong way, if I use > > ./ex1 -ksp_type minres -ksp_monitor_draw -draw_pause -1 > > I get the empty figure hanging on the screen. After clicking with the mouse on the figure, the history advances at each mouse click, is not this strange? It is suppose to advance at each mouse click. But it is odd that before the first click it is "empty". Is it possible that it has just plotted the first iteration and is not truly "empty"? Note that the -ksp_monitor_draw draws the plot at EACH iteration, it does not wait for all the iterations and then draw one plot at the end. Barry > I do not get that displayed after the above command. > > Best, > Umut > > -- > If I have a thousand ideas and only one turns out to be good, > I am satisfied. > Alfred Nobel > From u.tabak at tudelft.nl Tue Mar 1 11:10:01 2011 From: u.tabak at tudelft.nl (Umut Tabak) Date: Tue, 01 Mar 2011 18:10:01 +0100 Subject: [petsc-users] how to persist -ksp_monitor_draw In-Reply-To: <4B7E8036-BA31-4FC0-A7C5-262EE8CB5A42@mcs.anl.gov> References: <4D6D2277.4000902@tudelft.nl> <4D6D25D4.1010900@tudelft.nl> <4B7E8036-BA31-4FC0-A7C5-262EE8CB5A42@mcs.anl.gov> Message-ID: <4D6D2869.2020100@tudelft.nl> On 03/01/2011 06:07 PM, Barry Smith wrote: > > > It is suppose to advance at each mouse click. But it is odd that before the first click it is "empty". Is it possible that it has just plotted the first iteration and is not truly "empty"? > There is a plus like sign somewhere close to the origin. Indeed it is not empty, however it does not advance evenly, say 10 or 20 iterations per click, however I can see what I would like to see. Thanks Umut From bsmith at mcs.anl.gov Tue Mar 1 11:13:55 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 1 Mar 2011 11:13:55 -0600 Subject: [petsc-users] how to persist -ksp_monitor_draw In-Reply-To: <4D6D2869.2020100@tudelft.nl> References: <4D6D2277.4000902@tudelft.nl> <4D6D25D4.1010900@tudelft.nl> <4B7E8036-BA31-4FC0-A7C5-262EE8CB5A42@mcs.anl.gov> <4D6D2869.2020100@tudelft.nl> Message-ID: On Mar 1, 2011, at 11:10 AM, Umut Tabak wrote: > On 03/01/2011 06:07 PM, Barry Smith wrote: >> >> >> It is suppose to advance at each mouse click. But it is odd that before the first click it is "empty". Is it possible that it has just plotted the first iteration and is not truly "empty"? >> > There is a plus like sign somewhere close to the origin. Indeed it is not empty, however it does not advance evenly, say 10 or 20 iterations per click, however I can see what I would like to see. > Thanks > Umut That's right. It starts by showing every iteration, then with more iterations it saves a bunch and does them together. This is so you don't have to do 200 hundred clicks if it takes 200 iterations. Barry From balay at mcs.anl.gov Tue Mar 1 11:17:54 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 1 Mar 2011 11:17:54 -0600 (CST) Subject: [petsc-users] how to persist -ksp_monitor_draw In-Reply-To: References: <4D6D2277.4000902@tudelft.nl> <4D6D25D4.1010900@tudelft.nl> <4B7E8036-BA31-4FC0-A7C5-262EE8CB5A42@mcs.anl.gov> <4D6D2869.2020100@tudelft.nl> Message-ID: On Tue, 1 Mar 2011, Barry Smith wrote: > > On Mar 1, 2011, at 11:10 AM, Umut Tabak wrote: > > > On 03/01/2011 06:07 PM, Barry Smith wrote: > >> > >> > >> It is suppose to advance at each mouse click. But it is odd that before the first click it is "empty". Is it possible that it has just plotted the first iteration and is not truly "empty"? > >> > > There is a plus like sign somewhere close to the origin. Indeed it is not empty, however it does not advance evenly, say 10 or 20 iterations per click, however I can see what I would like to see. > > Thanks > > Umut > > That's right. It starts by showing every iteration, then with more iterations it saves a bunch and does them together. This is so you don't have to do 200 hundred clicks if it takes 200 iterations. In this case '--draw_pause 2' could be usefull [instead of clicking each time..] > > Barry > > From u.tabak at tudelft.nl Tue Mar 1 14:45:50 2011 From: u.tabak at tudelft.nl (Umut Tabak) Date: Tue, 01 Mar 2011 21:45:50 +0100 Subject: [petsc-users] minres and gmres difference Message-ID: <4D6D5AFE.9070702@tudelft.nl> Dear list, While testing an ill-conditioned system solution with minres and gmres(default 30 restarts, and no other options) in PETSc, I came up with the result that the system is solved in 155 iterations by minres to a relative default tolerance but it could not be solved by gmres. The original system is indefinite and had a condition number on the order of 10e+6, with a diagonal conditioning left and right, I could lower the condition number to the order of 10e+4 (however still indefinite with negative eigenvalues) and try to solve this system. On the other hand, I am also trying to understand the details of minres and gmres, so they both minimize the 2 norm of residual by solving a least squares problem where the latter is for unsymmetric systems, and the other is using a 3-term recurrence relation similar to cg (which I am trying to figure out now.). I would expect both to give the same result in PETSc however gmres fails. I wanted to check also gmres because I read in "Iterative Krylov Methods for Large Linear Systems" van der Vorst, Chap.6 Gmres and Minres, page 86 where it is mentioned that gmres could be more robust and less vulnerable to rounding errors than minres. What could be the reason for this? Any comments are appreciated Best, Umut From knepley at gmail.com Tue Mar 1 15:17:35 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 1 Mar 2011 15:17:35 -0600 Subject: [petsc-users] minres and gmres difference In-Reply-To: <4D6D5AFE.9070702@tudelft.nl> References: <4D6D5AFE.9070702@tudelft.nl> Message-ID: On Tue, Mar 1, 2011 at 2:45 PM, Umut Tabak wrote: > Dear list, > > While testing an ill-conditioned system solution with minres and > gmres(default 30 restarts, and no other options) in PETSc, I came up with > the result that the system is solved in 155 iterations by minres to a > relative default tolerance but it could not be solved by gmres. > > The original system is indefinite and had a condition number on the order > of 10e+6, with a diagonal conditioning left and right, I could lower the > condition number to the order of 10e+4 (however still indefinite with > negative eigenvalues) and try to solve this system. On the other hand, I am > also trying to understand the details of minres and gmres, so they both > minimize the 2 norm of residual by solving a least squares problem where the > latter is for unsymmetric systems, and the other is using a 3-term > recurrence relation similar to cg (which I am trying to figure out now.). I > would expect both to give the same result in PETSc however gmres fails. I > wanted to check also gmres because I read in > > "Iterative Krylov Methods for Large Linear Systems" van der Vorst, Chap.6 > Gmres and Minres, page 86 > > where it is mentioned that gmres could be more robust and less vulnerable > to rounding errors than minres. > Henk did not count on different orhogonalization strategies. Try -ksp_gmres_modifiedgramschmidt Matt > What could be the reason for this? Any comments are appreciated > > Best, > Umut > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From juan.gonzalez at accelogic.com Tue Mar 1 15:24:37 2011 From: juan.gonzalez at accelogic.com (Juan Gonzalez, Ph.D.) Date: Tue, 1 Mar 2011 16:24:37 -0500 Subject: [petsc-users] Seeking PETSc and NAS Parallel Benchmark Expert for short (2 months) consulting opportunity Message-ID: <000901cbd857$11871190$349534b0$@gonzalez@accelogic.com> Friends: Accelogic is seeking a specialist in both PETSc and the NAS benchmarks who can provide high-level advice to the company (starting immediately until mid-May). Someone who can help with code development on the implementation of the benchmarks for different accelerated platforms, and who understands performance issues. You would work with system developers on integration issues to implement the benchmarks and produce a final report with the benchmark results. If you want to come to Weston and have two months of fun in sunny Florida, or if you are a graduate student or a consultant and want to take this as a part time opportunity, you are a fit candidate. It is also your choice whether or not to work remotely. This opportunity would require at least an initial visit (for kick-off) to our headquarters in Weston, FL, if you decide to work remotely. The only requirement is to have immediate availability, and to commit to the delivery of the results by the first week of May. The pay will be highly competitive, including a special bonus based on performance and timeliness, and commeasurable with experience and potential impact of your work. There is the possibility of extending your engagement with other projects as a consultant or even as a full-time employee. If you are interested, please send your CV and a quick email to anna.hernandez at accelogic.com explaining your specific experience working with NAS benchmarks. Important note: It is less important that you submit the perfect resume, and more important that you apply as soon as possible. Thanks! Juan Gonzalez, Ph.D. President & Chief Scientist Accelogic LLC 1830 Main St, Suite 204 Weston, FL 33326 Ph: +1.954.888.4711 Fx: +1.954.208.0018 www.accelogic.com accelogic "A pessimist sees the difficulty in every opportunity; an optimist sees the opportunity in every difficulty" - Sir Winston Churchill CONFIDENTIALITY: This email (including any attachments) may contain confidential, proprietary and privileged information, and unauthorized disclosure or use is prohibited. If you received this email in error, please notify the sender and delete this email from your system. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 2416 bytes Desc: not available URL: From bsmith at mcs.anl.gov Tue Mar 1 16:01:45 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 1 Mar 2011 16:01:45 -0600 Subject: [petsc-users] minres and gmres difference In-Reply-To: <4D6D5AFE.9070702@tudelft.nl> References: <4D6D5AFE.9070702@tudelft.nl> Message-ID: <533E048B-0039-4F13-8EA0-65EC995035D9@mcs.anl.gov> Also what happens when you crank the restart of GMRES to above 155? I bet it converges. I am not implying that this would be practical but it would tell you something, there is no theory for GMRES with restart. Barry On Mar 1, 2011, at 2:45 PM, Umut Tabak wrote: > Dear list, > > While testing an ill-conditioned system solution with minres and gmres(default 30 restarts, and no other options) in PETSc, I came up with the result that the system is solved in 155 iterations by minres to a relative default tolerance but it could not be solved by gmres. > > The original system is indefinite and had a condition number on the order of 10e+6, with a diagonal conditioning left and right, I could lower the condition number to the order of 10e+4 (however still indefinite with negative eigenvalues) and try to solve this system. On the other hand, I am also trying to understand the details of minres and gmres, so they both minimize the 2 norm of residual by solving a least squares problem where the latter is for unsymmetric systems, and the other is using a 3-term recurrence relation similar to cg (which I am trying to figure out now.). I would expect both to give the same result in PETSc however gmres fails. I wanted to check also gmres because I read in > > "Iterative Krylov Methods for Large Linear Systems" van der Vorst, Chap.6 Gmres and Minres, page 86 > > where it is mentioned that gmres could be more robust and less vulnerable to rounding errors than minres. > > What could be the reason for this? Any comments are appreciated > > Best, > Umut From u.tabak at tudelft.nl Tue Mar 1 16:12:52 2011 From: u.tabak at tudelft.nl (Umut Tabak) Date: Tue, 01 Mar 2011 23:12:52 +0100 Subject: [petsc-users] minres and gmres difference In-Reply-To: References: <4D6D5AFE.9070702@tudelft.nl> Message-ID: <4D6D6F64.1050607@tudelft.nl> On 03/01/2011 10:17 PM, Matthew Knepley wrote: > On Tue, Mar 1, 2011 at 2:45 PM, Umut Tabak > wrote: > > Henk did not count on different orhogonalization strategies. Try > -ksp_gmres_modifiedgramschmidt > It is better at the end, with 10000 iterations though, which makes it far from useful for this matrix. Umut -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Mar 1 16:08:35 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 1 Mar 2011 16:08:35 -0600 Subject: [petsc-users] minres and gmres difference In-Reply-To: <4D6D6F64.1050607@tudelft.nl> References: <4D6D5AFE.9070702@tudelft.nl> <4D6D6F64.1050607@tudelft.nl> Message-ID: <44124372-92BA-48BD-890C-E5C4B3F027F1@mcs.anl.gov> On Mar 1, 2011, at 4:12 PM, Umut Tabak wrote: > On 03/01/2011 10:17 PM, Matthew Knepley wrote: >> On Tue, Mar 1, 2011 at 2:45 PM, Umut Tabak wrote: >> >> Henk did not count on different orhogonalization strategies. Try -ksp_gmres_modifiedgramschmidt >> > It is better at the end, with 10000 iterations though, which makes it far from useful for this matrix. That 10000 is the default maxium that we allow, so I think your run with the other orgonalization is meaningless. Change the restart like I suggest in my email. Barry > Umut From u.tabak at tudelft.nl Tue Mar 1 16:24:12 2011 From: u.tabak at tudelft.nl (Umut Tabak) Date: Tue, 01 Mar 2011 23:24:12 +0100 Subject: [petsc-users] minres and gmres difference In-Reply-To: <44124372-92BA-48BD-890C-E5C4B3F027F1@mcs.anl.gov> References: <4D6D5AFE.9070702@tudelft.nl> <4D6D6F64.1050607@tudelft.nl> <44124372-92BA-48BD-890C-E5C4B3F027F1@mcs.anl.gov> Message-ID: <4D6D720C.8090202@tudelft.nl> On 03/01/2011 11:08 PM, Barry Smith wrote: > 10000 is the default maxium that we allow, so I think your run with the other orgonalization is meaningless. Change the restart like I suggest in my email. > Yep, then converges, so that you iterate for 146 times(for 155 restart). But it seems to me that with a low dimensional subspace it can not catch the information. And needs a lot of vectors and orthogonalization which makes it costly, are there any further comments? Best, Umut From jed at 59A2.org Tue Mar 1 16:29:47 2011 From: jed at 59A2.org (Jed Brown) Date: Tue, 1 Mar 2011 14:29:47 -0800 Subject: [petsc-users] minres and gmres difference In-Reply-To: <4D6D720C.8090202@tudelft.nl> References: <4D6D5AFE.9070702@tudelft.nl> <4D6D6F64.1050607@tudelft.nl> <44124372-92BA-48BD-890C-E5C4B3F027F1@mcs.anl.gov> <4D6D720C.8090202@tudelft.nl> Message-ID: On Tue, Mar 1, 2011 at 14:24, Umut Tabak wrote: > Yep, then converges, so that you iterate for 146 times(for 155 restart). > But it seems to me that with a low dimensional subspace it can not catch the > information. And needs a lot of vectors and orthogonalization which makes it > costly, are there any further comments? 1. Make a better preconditioner 2. Use a different Krylov method that does not require storing a fully subspace. Such methods cannot have a monotone residual and are less robust to inexact Jacobians, but can be better if you need a large number of Krylov iterations. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Mar 2 12:57:38 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 2 Mar 2011 12:57:38 -0600 Subject: [petsc-users] PCDiagonalScale In-Reply-To: References: , , Message-ID: On Mar 1, 2011, at 2:28 AM, Klaij, Christiaan wrote: > Barry, > > I do not want to use it for the reason given in the manual. > > Instead I'm trying to use SIMPLE as a preconditioner for a colocated, cell-centered FV discretization of the incompressible Navier-Stokes equations. I'm using a matfree approach for the matrix and the (right) preconditioner. I would like to scale the system as D A M D^{-1} z = D b but I get the feeling that PCDiagonalScaleSet with GCR doesn't do anything. > > How about Jed's comment: > > "GCR also tolerates a variable preconditioner and it does not have such a check. I don't know if that means it can use diagonal scaling or just that someone forgot to check, but you could try it." KSPSolve_GCR() is not coded for using that extra diagonal scaling and does not check to see if you have requested it. So, as you saw, if you turn it on it has no effect. I have added an error check in petsc-dev so it will tell you that it does not use it. Thank you for letting us know about this bug. You may be able to add support for it. The code is in src/ksp/ksp/impls/gcr/grc.c Note that the current code has the lines ierr = PCApply( pc, r, s ); CHKERRQ(ierr); /* s = B^{-1} r */ ierr = MatMult( A, s, v ); CHKERRQ(ierr); /* v = A s */ to fix the code to do the diagonal scaling you need to do a few things 1) replace the two lines above with a single call to KSP_PCApplyBAorAB() this will incorporate the scaling in the application of A*B so instead it provides D A B D^{-1} 2) this is more difficult. Understand the code well enough to fix it so that it properly takes into account the diagonal scaling if the initial guess is nonzero and also the fact that you are solving D A B D^{-1} y = D b instead of A B z = b and hence A x = b where x = B z so x = B *D^{-1} y Barry > > Is it just that the check is missing for GCR so that it doesn't apply diagonal scaling and doesn't tell me? > > Chris > > > Message: 3 > Date: Thu, 24 Feb 2011 08:08:42 -0600 > From: Barry Smith > Subject: Re: [petsc-users] PCDiagonalScale > To: PETSc users list > Message-ID: <699188CA-750D-446A-AD58-C62CB2A75CE2 at mcs.anl.gov> > Content-Type: text/plain; charset=us-ascii > > > On Feb 23, 2011, at 9:45 AM, Klaij, Christiaan wrote: > >> I'm trying to understand the use of PCDiagonalScale since I want to apply additional diagonal scaling when solving my linear system. > > Do you want to use it exactly for the reason given in > PCSetDiagonalScale - Indicates the left scaling to use to apply an additional left and right > scaling as needed by certain time-stepping codes. > > Logically Collective on PC > > Input Parameters: > + pc - the preconditioner context > - s - scaling vector > > Level: intermediate > > Notes: The system solved via the Krylov method is > $ D M A D^{-1} y = D M b for left preconditioning or > $ D A M D^{-1} z = D b for right preconditioning > > > dr. ir. Christiaan Klaij > CFD Researcher > Research & Development > E mailto:C.Klaij at marin.nl > T +31 317 49 33 44 > > MARIN > 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands > T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl > From brtnfld at uiuc.edu Wed Mar 2 16:25:08 2011 From: brtnfld at uiuc.edu (M. Scot Breitenfeld) Date: Wed, 02 Mar 2011 16:25:08 -0600 Subject: [petsc-users] AOCreateBasic is not scaling Message-ID: <4D6EC3C4.4080407@uiuc.edu> Hi, First, thanks for the suggestion on using MPISBAIJ for my A matrix, it seems to have cut down on my memory and assembly time. For a 1.5 million dof problem: # procs: 2 4 8 16 ---------------------------------------------------------------- Assembly (sec): 245 124 63 86 Solver (sec): 924 578 326 680 Memory (GB): 2.5 1.4 .877 .565 The problem I have is the amount of time it's taking in AOCreateBasic, it takes longer then assembling, # procs: 2 4 8 16 --------------------------------------------------------------------- AOCreateBasic (sec): .6 347 170 197 Is there something that I can change or something I can look for that might be causing this increase in time as I go from 2 to 4 processors (at least it scales from 4 to 8 processors). I read in the archive that AOCreateBasic is not meant to be scalable so maybe there is nothing I can do. Thanks, Scot From knepley at gmail.com Wed Mar 2 16:29:24 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 2 Mar 2011 16:29:24 -0600 Subject: [petsc-users] AOCreateBasic is not scaling In-Reply-To: <4D6EC3C4.4080407@uiuc.edu> References: <4D6EC3C4.4080407@uiuc.edu> Message-ID: On Wed, Mar 2, 2011 at 4:25 PM, M. Scot Breitenfeld wrote: > Hi, > > First, thanks for the suggestion on using MPISBAIJ for my A matrix, it > seems to have cut down on my memory and assembly time. For a 1.5 million > dof problem: > > # procs: 2 4 8 16 > ---------------------------------------------------------------- > Assembly (sec): 245 124 63 86 > Solver (sec): 924 578 326 680 > > Memory (GB): 2.5 1.4 .877 .565 > > The problem I have is the amount of time it's taking in AOCreateBasic, > it takes longer then assembling, > > # procs: 2 4 8 16 > --------------------------------------------------------------------- > AOCreateBasic (sec): .6 347 170 197 > > Is there something that I can change or something I can look for that > might be causing this increase in time as I go from 2 to 4 processors > (at least it scales from 4 to 8 processors). I read in the archive that > AOCreateBasic is not meant to be scalable so maybe there is nothing I > can do. > Yes, this is non-scalable. What are you using it for? Matt > Thanks, > Scot > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From brtnfld at uiuc.edu Wed Mar 2 16:52:48 2011 From: brtnfld at uiuc.edu (M. Scot Breitenfeld) Date: Wed, 02 Mar 2011 16:52:48 -0600 Subject: [petsc-users] AOCreateBasic is not scaling In-Reply-To: References: <4D6EC3C4.4080407@uiuc.edu> Message-ID: <4D6ECA40.5080409@uiuc.edu> I don't number my global degree's of freedom from low-high continuously per processor as PETSc uses for ordering, but I use the natural ordering of the application, I then use AOcreateBasic to obtain the mapping between the PETSc and my ordering. CALL VecGetOwnershipRange(b, low, high, ierr) icnt = 0 DO mi = 1, mctr ! these are the nodes local to processor mi_global = myglobal(mi) irowx = 3*mi_global-2 irowy = 3*mi_global-1 irowz = 3*mi_global mappings(icnt+1:icnt+3) = (/ & nrow_global(row_from_dof(1,mi))-1, & nrow_global(row_from_dof(2,mi))-1, & nrow_global(row_from_dof(3,mi))-1 & /) petscOrdering(icnt+1:icnt+3) = (/low+icnt,low+icnt+1,low+icnt+2/) icnt = icnt + 3 END DO CALL AOCreateBasic(PETSC_COMM_WORLD, icnt, mappings, petscOrdering, toao, ierr) DO mi = mctr+1, myn ! these are the ghost nodes not on this processor mi_global = myglobal(mi) mappings(icnt+1:icnt+3) = (/ & nrow_global(row_from_dof(1,mi))-1, & nrow_global(row_from_dof(2,mi))-1, & nrow_global(row_from_dof(3,mi))-1 & /) icnt = icnt + 3 ENDDO CALL AOApplicationToPetsc(toao, 3*myn, mappings, ierr) CALL AODestroy(toao, ierr) I then use mapping to input the values into the correct row as wanted by PETSc On 03/02/2011 04:29 PM, Matthew Knepley wrote: > On Wed, Mar 2, 2011 at 4:25 PM, M. Scot Breitenfeld > wrote: > > Hi, > > First, thanks for the suggestion on using MPISBAIJ for my A matrix, it > seems to have cut down on my memory and assembly time. For a 1.5 > million > dof problem: > > # procs: 2 4 8 16 > ---------------------------------------------------------------- > Assembly (sec): 245 124 63 86 > Solver (sec): 924 578 326 680 > > Memory (GB): 2.5 1.4 .877 .565 > > The problem I have is the amount of time it's taking in AOCreateBasic, > it takes longer then assembling, > > # procs: 2 4 8 16 > --------------------------------------------------------------------- > AOCreateBasic (sec): .6 347 170 197 > > Is there something that I can change or something I can look for that > might be causing this increase in time as I go from 2 to 4 processors > (at least it scales from 4 to 8 processors). I read in the archive > that > AOCreateBasic is not meant to be scalable so maybe there is nothing I > can do. > > > Yes, this is non-scalable. What are you using it for? > > Matt > > > Thanks, > Scot > > > > > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener From rxk at cfdrc.com Wed Mar 2 17:27:32 2011 From: rxk at cfdrc.com (Ravi Kannan) Date: Wed, 2 Mar 2011 17:27:32 -0600 Subject: [petsc-users] multi-core machines in market In-Reply-To: <4D6ECA40.5080409@uiuc.edu> References: <4D6EC3C4.4080407@uiuc.edu> <4D6ECA40.5080409@uiuc.edu> Message-ID: <002101cbd931$677aff50$3670fdf0$@com> Dear All, Could anyone be kind enough to give me some statistics/information on the commercially available multi-core platforms, including the nodes with the maximum number of cores (4,8,16???). Thanks Ravi _________________________________________ Ravi Kannan CFD Research Corporation Research Engineer 256.726.4851 rxk at cfdrc.com _________________________________________ From jdbst21 at gmail.com Wed Mar 2 18:12:08 2011 From: jdbst21 at gmail.com (Joshua Booth) Date: Wed, 2 Mar 2011 19:12:08 -0500 Subject: [petsc-users] Factor Matrix Message-ID: In petsc, I am able to get a cholesky factor matrix from MatCholeskyFactor, but I am unable to copy or get values from it after. Why is this? Is there a work around? Joshua Booth -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Mar 2 18:36:35 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 2 Mar 2011 18:36:35 -0600 Subject: [petsc-users] Factor Matrix In-Reply-To: References: Message-ID: On Mar 2, 2011, at 6:12 PM, Joshua Booth wrote: > In petsc, I am able to get a cholesky factor matrix from MatCholeskyFactor, but I am unable to copy Do you mean MatCopy() ? > or get values from it after. What do you want to use the values for? Normally they are only used in triangular solvers with MatSolve(). The data is stored in a matrix specific format. You would have to include the private include files and access the data directly if you need it. src/mat/impls/sbaij/seq/sbaij.h Barry > Why is this? Is there a work around? > > Joshua Booth From jdbst21 at gmail.com Wed Mar 2 18:52:17 2011 From: jdbst21 at gmail.com (Joshua Booth) Date: Wed, 2 Mar 2011 19:52:17 -0500 Subject: [petsc-users] Factor Matrix In-Reply-To: References: Message-ID: In reply, Yes , I meant with MatCopy or MatConvert. I need the values so I can store them in a larger matrix for latter use. It is in process of working on an experimental multilevel method. Josh On Wed, Mar 2, 2011 at 7:36 PM, Barry Smith wrote: > > On Mar 2, 2011, at 6:12 PM, Joshua Booth wrote: > > > In petsc, I am able to get a cholesky factor matrix from > MatCholeskyFactor, but I am unable to copy > > Do you mean MatCopy() ? > > > or get values from it after. > > What do you want to use the values for? Normally they are only used in > triangular solvers with MatSolve(). > > The data is stored in a matrix specific format. You would have to include > the private include files and access the data directly if you need it. > src/mat/impls/sbaij/seq/sbaij.h > > > Barry > > > Why is this? Is there a work around? > > > > Joshua Booth > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Mar 2 21:26:19 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 2 Mar 2011 21:26:19 -0600 Subject: [petsc-users] Factor Matrix In-Reply-To: References: Message-ID: On Mar 2, 2011, at 6:52 PM, Joshua Booth wrote: > In reply, > > Yes , I meant with MatCopy or MatConvert. > > I need the values so I can store them in a larger matrix for latter use. It is in process of working on an experimental multilevel method. Store the LU factors or C factors of a given matrix inside a larger matrix. Strange stuff. This is exactly the case where you should include src/mat/impls/sbaij/seq/sbaij.h and access the data structure directly to do what you want with the values. Barry > > Josh > > > On Wed, Mar 2, 2011 at 7:36 PM, Barry Smith wrote: > > On Mar 2, 2011, at 6:12 PM, Joshua Booth wrote: > > > In petsc, I am able to get a cholesky factor matrix from MatCholeskyFactor, but I am unable to copy > > Do you mean MatCopy() ? > > > or get values from it after. > > What do you want to use the values for? Normally they are only used in triangular solvers with MatSolve(). > > The data is stored in a matrix specific format. You would have to include the private include files and access the data directly if you need it. src/mat/impls/sbaij/seq/sbaij.h > > > Barry > > > Why is this? Is there a work around? > > > > Joshua Booth > > From loic.gouarin at math.u-psud.fr Thu Mar 3 06:52:32 2011 From: loic.gouarin at math.u-psud.fr (gouarin) Date: Thu, 03 Mar 2011 13:52:32 +0100 Subject: [petsc-users] DMComposite and ksp Message-ID: <4D6F8F10.1040104@math.u-psud.fr> Hi, I'm still working on my stokes problem with 2 DM (one for the velocities and an other for the pressure). I do the best preallocation for my matrix. After the construction of my matrix, I do a matrix vector operation and I compare the result for one process and 2 processes. If I pay attention to the numbering of my fields between these 2 problems, I have the same result. But when I do just 2 iterations of my ksp solver, I don't have the same result at the end and of course the residual is not the same. Do you know what I do wrong ? Thanks, Loic -- Loic Gouarin Laboratoire de Math?matiques Universit? Paris-Sud B?timent 425 91405 Orsay Cedex France Tel: (+33) 1 69 15 60 14 Fax: (+33) 1 69 15 67 18 From agrayver at gfz-potsdam.de Thu Mar 3 05:51:32 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Thu, 03 Mar 2011 12:51:32 +0100 Subject: [petsc-users] Calling convention from windows VC++ and IFC Message-ID: <4D6F80C4.4070200@gfz-potsdam.de> Hello, I built PETSc-3.1 using VC++ 2008 and IFC 10.1 with the following configuration line: ./config/configure.py --with-cc='win32fe cl' --with-fc='win32fe ifort' --with-cxx='win32fe cl' --download-f-blas-lapack=1 --with-precision=double --with-scalar-type=complex --with-clanguage=cxx --with-mpi-include="/cygdrive/d/Dev/HPC_2008_SDK/Include" --with-mpi-lib=["/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpi.lib","/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib"] --useThreads=0 --with-shared=0 Note there is C++ as main language. Now, when I'm trying to build examples I always get similar errors: Error 2 error LNK2019: unresolved external symbol ZGESVD referenced in function "int __cdecl KSPComputeExtremeSingularValues_GMRES(struct _p_KSP *,double *,double *)" (?KSPComputeExtremeSingularValues_GMRES@@YAHPEAU_p_KSP@@PEAN1 at Z) libpetsc.lib Error 3 error LNK2019: unresolved external symbol ZGEEV referenced in function "int __cdecl KSPComputeEigenvalues_GMRES(struct _p_KSP *,int,double *,double *,int *)" (?KSPComputeEigenvalues_GMRES@@YAHPEAU_p_KSP@@HPEAN1PEAH at Z) libpetsc.lib Error 4 error LNK2001: unresolved external symbol ZGEEV libpetsc.lib Error 5 error LNK2019: unresolved external symbol DZNRM2 referenced in function "int __cdecl VecNorm_MPI(struct _p_Vec *,enum NormType,double *)" (?VecNorm_MPI@@YAHPEAU_p_Vec@@W4NormType@@PEAN at Z) libpetsc.lib Error 6 error LNK2001: unresolved external symbol DZNRM2 libpetsc.lib Error 7 error LNK2019: unresolved external symbol DZASUM referenced in function "int __cdecl VecNorm_Seq(struct _p_Vec *,enum NormType,double *)" (?VecNorm_Seq@@YAHPEAU_p_Vec@@W4NormType@@PEAN at Z) libpetsc.lib ... It look like calling conventions are in disagreement. Any ideas? May be I should specify conventions myself in configuration line? If so, how to do that? From balay at mcs.anl.gov Thu Mar 3 07:49:01 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 3 Mar 2011 07:49:01 -0600 (CST) Subject: [petsc-users] Calling convention from windows VC++ and IFC In-Reply-To: <4D6F80C4.4070200@gfz-potsdam.de> References: <4D6F80C4.4070200@gfz-potsdam.de> Message-ID: can you send the corresponding configure.log and make.log to petsc-maint at mcs.anl.gov? Satish On Thu, 3 Mar 2011, Alexander Grayver wrote: > Hello, > > I built PETSc-3.1 using VC++ 2008 and IFC 10.1 with the following > configuration line: > > ./config/configure.py --with-cc='win32fe cl' --with-fc='win32fe ifort' > --with-cxx='win32fe cl' --download-f-blas-lapack=1 --with-precision=double > --with-scalar-type=complex --with-clanguage=cxx > --with-mpi-include="/cygdrive/d/Dev/HPC_2008_SDK/Include" > --with-mpi-lib=["/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpi.lib","/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib"] > --useThreads=0 --with-shared=0 > > Note there is C++ as main language. > > Now, when I'm trying to build examples I always get similar errors: > > Error 2 error LNK2019: unresolved external symbol ZGESVD referenced in > function "int __cdecl KSPComputeExtremeSingularValues_GMRES(struct _p_KSP > *,double *,double *)" > (?KSPComputeExtremeSingularValues_GMRES@@YAHPEAU_p_KSP@@PEAN1 at Z) > libpetsc.lib > Error 3 error LNK2019: unresolved external symbol ZGEEV referenced in > function "int __cdecl KSPComputeEigenvalues_GMRES(struct _p_KSP *,int,double > *,double *,int *)" (?KSPComputeEigenvalues_GMRES@@YAHPEAU_p_KSP@@HPEAN1PEAH at Z) > libpetsc.lib > Error 4 error LNK2001: unresolved external symbol ZGEEV libpetsc.lib > Error 5 error LNK2019: unresolved external symbol DZNRM2 referenced in > function "int __cdecl VecNorm_MPI(struct _p_Vec *,enum NormType,double *)" > (?VecNorm_MPI@@YAHPEAU_p_Vec@@W4NormType@@PEAN at Z) libpetsc.lib > Error 6 error LNK2001: unresolved external symbol DZNRM2 > libpetsc.lib > Error 7 error LNK2019: unresolved external symbol DZASUM referenced in > function "int __cdecl VecNorm_Seq(struct _p_Vec *,enum NormType,double *)" > (?VecNorm_Seq@@YAHPEAU_p_Vec@@W4NormType@@PEAN at Z) libpetsc.lib > ... > > It look like calling conventions are in disagreement. Any ideas? May be I > should specify conventions myself in configuration line? If so, how to do > that? > From bsmith at mcs.anl.gov Thu Mar 3 08:19:46 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 3 Mar 2011 08:19:46 -0600 Subject: [petsc-users] DMComposite and ksp In-Reply-To: <4D6F8F10.1040104@math.u-psud.fr> References: <4D6F8F10.1040104@math.u-psud.fr> Message-ID: <020F96D8-7F8C-45C1-9896-6A786D3F7A36@mcs.anl.gov> On Mar 3, 2011, at 6:52 AM, gouarin wrote: > Hi, > > I'm still working on my stokes problem with 2 DM (one for the velocities and an other for the pressure). I do the best preallocation for my matrix. > After the construction of my matrix, I do a matrix vector operation and I compare the result for one process and 2 processes. If I pay attention to the numbering of my fields between these 2 problems, I have the same result. > > But when I do just 2 iterations of my ksp solver, I don't have the same result at the end and of course the residual is not the same. What preconditioner are you using? Unless you use Jacobi you will always get a different set of iterations. (And even with Jacobi due to different order of operations you will eventually see a different convergence route between 1 and 2 processes. You cannot be concerned about this, the things to be concerned about are: 1) Do 1 and 2 processes eventually converge to the same answer (bad news if they do not) 2) Does 2 process converge almost as quickly to the answer as one process (bad news if adding more processes slows the convergence greatly so the solution time ends up being longer). Barry > > Do you know what I do wrong ? > > Thanks, > Loic > > -- > Loic Gouarin > Laboratoire de Math?matiques > Universit? Paris-Sud > B?timent 425 > 91405 Orsay Cedex > France > Tel: (+33) 1 69 15 60 14 > Fax: (+33) 1 69 15 67 18 > From agrayver at gfz-potsdam.de Thu Mar 3 08:46:50 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Thu, 03 Mar 2011 15:46:50 +0100 Subject: [petsc-users] Calling convention from windows VC++ and IFC In-Reply-To: References: <4D6F80C4.4070200@gfz-potsdam.de> Message-ID: <4D6FA9DA.7070403@gfz-potsdam.de> I've sent them, thanks! I also must note that this is a problem for fortran examples only! Because when I've done "make test" after building the situation was: mt at agrayver /cygdrive/d/dev/petsc-3.1-p7 $ make PETSC_DIR=/cygdrive/d/dev/petsc-3.1-p7 PETSC_ARCH=cygwin-cxx-debug test Running test examples to verify correct installation C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 MPI process C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 MPI processes --------------Error detected during compile or link!----------------------- See http://www.mcs.anl.gov/petsc/petsc-2/documentation/troubleshooting.html /cygdrive/d/dev/petsc-3.1-p7/bin/win32fe/win32fe ifort -c -MT -Z7 -fpp -I/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/include -I/cygdrive/d/dev/petsc-3.1-p7/include -I/cygdrive/d/Dev/HPC_2008_SDK/Include -I/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/include -I/cygdrive/d/Dev/HPC_2008_SDK/Include -o ex5f.o ex5f.F /cygdrive/d/dev/petsc-3.1-p7/bin/win32fe/win32fe ifort -MT -Z7 -fpp -o ex5f ex5f.o -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib -lpetsc -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib -lflapack -lfblas /cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpi.lib /cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib Gdi32.lib User32.lib Advapi32.lib Kernel32.lib Ws2_32.lib libpetsc.lib(fuser.o) : error LNK2019: unresolved external symbol __imp_GetUserNameA referenced in function "int __cdecl PetscGetUserName(char * const,unsigned__int64)" (?PetscGetUserName@@YAHQEAD_K at Z) libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol __imp_ReleaseDC referenced in function "int __cdecl PetscDrawSetDoubleBuffer_Win32(struct_p_PetscDraw *)" (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol __imp_BitBlt referenced in function "int __cdecl PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol __imp_ExtFloodFill referenced in function "int __cdecl PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol __imp_SelectObject referenced in function "int __cdecl PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol __imp_CreateCompatibleBitmap referenced in function "int __cdecl PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol __imp_CreateCompatibleDC referenced in function "int __cdecl PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol __imp_GetDC referenced in function "int __cdecl PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol __imp_UpdateWindow referenced in function "int __cdecl PetscDrawRectangle_Win32(struct _p_PetscDraw *,double,double,double,double,int,int,int,int)" (?PetscDrawRectangle_Win32@@YAHPEAU_p_PetscDraw@@NNNNHHHH at Z) On 03.03.2011 14:49, Satish Balay wrote: > can you send the corresponding configure.log and make.log to petsc-maint at mcs.anl.gov? > > Satish > > On Thu, 3 Mar 2011, Alexander Grayver wrote: > >> Hello, >> >> I built PETSc-3.1 using VC++ 2008 and IFC 10.1 with the following >> configuration line: >> >> ./config/configure.py --with-cc='win32fe cl' --with-fc='win32fe ifort' >> --with-cxx='win32fe cl' --download-f-blas-lapack=1 --with-precision=double >> --with-scalar-type=complex --with-clanguage=cxx >> --with-mpi-include="/cygdrive/d/Dev/HPC_2008_SDK/Include" >> --with-mpi-lib=["/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpi.lib","/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib"] >> --useThreads=0 --with-shared=0 >> >> Note there is C++ as main language. >> >> Now, when I'm trying to build examples I always get similar errors: >> >> Error 2 error LNK2019: unresolved external symbol ZGESVD referenced in >> function "int __cdecl KSPComputeExtremeSingularValues_GMRES(struct _p_KSP >> *,double *,double *)" >> (?KSPComputeExtremeSingularValues_GMRES@@YAHPEAU_p_KSP@@PEAN1 at Z) >> libpetsc.lib >> Error 3 error LNK2019: unresolved external symbol ZGEEV referenced in >> function "int __cdecl KSPComputeEigenvalues_GMRES(struct _p_KSP *,int,double >> *,double *,int *)" (?KSPComputeEigenvalues_GMRES@@YAHPEAU_p_KSP@@HPEAN1PEAH at Z) >> libpetsc.lib >> Error 4 error LNK2001: unresolved external symbol ZGEEV libpetsc.lib >> Error 5 error LNK2019: unresolved external symbol DZNRM2 referenced in >> function "int __cdecl VecNorm_MPI(struct _p_Vec *,enum NormType,double *)" >> (?VecNorm_MPI@@YAHPEAU_p_Vec@@W4NormType@@PEAN at Z) libpetsc.lib >> Error 6 error LNK2001: unresolved external symbol DZNRM2 >> libpetsc.lib >> Error 7 error LNK2019: unresolved external symbol DZASUM referenced in >> function "int __cdecl VecNorm_Seq(struct _p_Vec *,enum NormType,double *)" >> (?VecNorm_Seq@@YAHPEAU_p_Vec@@W4NormType@@PEAN at Z) libpetsc.lib >> ... >> >> It look like calling conventions are in disagreement. Any ideas? May be I >> should specify conventions myself in configuration line? If so, how to do >> that? >> From balay at mcs.anl.gov Thu Mar 3 09:26:34 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 3 Mar 2011 09:26:34 -0600 (CST) Subject: [petsc-users] Calling convention from windows VC++ and IFC In-Reply-To: <4D6FA9DA.7070403@gfz-potsdam.de> References: <4D6F80C4.4070200@gfz-potsdam.de> <4D6FA9DA.7070403@gfz-potsdam.de> Message-ID: perhaps you can remove '--CXX_CXXFLAGS=-Gz' and retry. Satish On Thu, 3 Mar 2011, Alexander Grayver wrote: > I've sent them, thanks! > > I also must note that this is a problem for fortran examples only! Because > when I've done "make test" after building the situation was: > > mt at agrayver /cygdrive/d/dev/petsc-3.1-p7 > $ make PETSC_DIR=/cygdrive/d/dev/petsc-3.1-p7 PETSC_ARCH=cygwin-cxx-debug test > Running test examples to verify correct installation > C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 MPI > process > C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 MPI > processes > --------------Error detected during compile or link!----------------------- > See http://www.mcs.anl.gov/petsc/petsc-2/documentation/troubleshooting.html > /cygdrive/d/dev/petsc-3.1-p7/bin/win32fe/win32fe ifort -c -MT -Z7 -fpp > -I/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/include > -I/cygdrive/d/dev/petsc-3.1-p7/include -I/cygdrive/d/Dev/HPC_2008_SDK/Include > -I/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/include > -I/cygdrive/d/Dev/HPC_2008_SDK/Include -o ex5f.o ex5f.F > > /cygdrive/d/dev/petsc-3.1-p7/bin/win32fe/win32fe ifort -MT -Z7 -fpp -o ex5f > ex5f.o -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib > -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib -lpetsc > -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib -lflapack -lfblas > /cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpi.lib > /cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib Gdi32.lib User32.lib > Advapi32.lib Kernel32.lib Ws2_32.lib > > libpetsc.lib(fuser.o) : error LNK2019: unresolved external symbol > __imp_GetUserNameA referenced in function "int __cdecl PetscGetUserName(char * > const,unsigned__int64)" (?PetscGetUserName@@YAHQEAD_K at Z) > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol > __imp_ReleaseDC referenced in function "int __cdecl > PetscDrawSetDoubleBuffer_Win32(struct_p_PetscDraw *)" > (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol > __imp_BitBlt referenced in function "int __cdecl > PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" > (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol > __imp_ExtFloodFill referenced in function "int __cdecl > PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" > (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol > __imp_SelectObject referenced in function "int __cdecl > PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" > (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol > __imp_CreateCompatibleBitmap referenced in function "int __cdecl > PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" > (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol > __imp_CreateCompatibleDC referenced in function "int __cdecl > PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" > (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol > __imp_GetDC referenced in function "int __cdecl > PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" > (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol > __imp_UpdateWindow referenced in function "int __cdecl > PetscDrawRectangle_Win32(struct _p_PetscDraw > *,double,double,double,double,int,int,int,int)" > (?PetscDrawRectangle_Win32@@YAHPEAU_p_PetscDraw@@NNNNHHHH at Z) > > On 03.03.2011 14:49, Satish Balay wrote: > > can you send the corresponding configure.log and make.log to > > petsc-maint at mcs.anl.gov? > > > > Satish > > > > On Thu, 3 Mar 2011, Alexander Grayver wrote: > > > > > Hello, > > > > > > I built PETSc-3.1 using VC++ 2008 and IFC 10.1 with the following > > > configuration line: > > > > > > ./config/configure.py --with-cc='win32fe cl' --with-fc='win32fe ifort' > > > --with-cxx='win32fe cl' --download-f-blas-lapack=1 > > > --with-precision=double > > > --with-scalar-type=complex --with-clanguage=cxx > > > --with-mpi-include="/cygdrive/d/Dev/HPC_2008_SDK/Include" > > > > > > --with-mpi-lib=["/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpi.lib","/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib"] > > > --useThreads=0 --with-shared=0 > > > > > > Note there is C++ as main language. > > > > > > Now, when I'm trying to build examples I always get similar errors: > > > > > > Error 2 error LNK2019: unresolved external symbol ZGESVD referenced > > > in > > > function "int __cdecl KSPComputeExtremeSingularValues_GMRES(struct _p_KSP > > > *,double *,double *)" > > > (?KSPComputeExtremeSingularValues_GMRES@@YAHPEAU_p_KSP@@PEAN1 at Z) > > > libpetsc.lib > > > Error 3 error LNK2019: unresolved external symbol ZGEEV referenced > > > in > > > function "int __cdecl KSPComputeEigenvalues_GMRES(struct _p_KSP > > > *,int,double > > > *,double *,int *)" > > > (?KSPComputeEigenvalues_GMRES@@YAHPEAU_p_KSP@@HPEAN1PEAH at Z) > > > libpetsc.lib > > > Error 4 error LNK2001: unresolved external symbol ZGEEV > > > libpetsc.lib > > > Error 5 error LNK2019: unresolved external symbol DZNRM2 referenced > > > in > > > function "int __cdecl VecNorm_MPI(struct _p_Vec *,enum NormType,double *)" > > > (?VecNorm_MPI@@YAHPEAU_p_Vec@@W4NormType@@PEAN at Z) libpetsc.lib > > > Error 6 error LNK2001: unresolved external symbol DZNRM2 > > > libpetsc.lib > > > Error 7 error LNK2019: unresolved external symbol DZASUM referenced > > > in > > > function "int __cdecl VecNorm_Seq(struct _p_Vec *,enum NormType,double *)" > > > (?VecNorm_Seq@@YAHPEAU_p_Vec@@W4NormType@@PEAN at Z) libpetsc.lib > > > ... > > > > > > It look like calling conventions are in disagreement. Any ideas? May be I > > > should specify conventions myself in configuration line? If so, how to do > > > that? > > > > > From agrayver at gfz-potsdam.de Thu Mar 3 09:33:41 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Thu, 03 Mar 2011 16:33:41 +0100 Subject: [petsc-users] Calling convention from windows VC++ and IFC In-Reply-To: References: <4D6F80C4.4070200@gfz-potsdam.de> <4D6FA9DA.7070403@gfz-potsdam.de> Message-ID: <4D6FB4D5.6040708@gfz-potsdam.de> Actually, I've added this flag after sending first message to mailing list (sorry for inconveniences) because I though it could help, unfortunately it didn't. The problem probably could be reproduced compiling petsc using --with-clanguage=cxx and VC++ and then trying to run FORTRAN examples. On 03.03.2011 16:26, Satish Balay wrote: > perhaps you can remove '--CXX_CXXFLAGS=-Gz' and retry. > > Satish > > On Thu, 3 Mar 2011, Alexander Grayver wrote: > >> I've sent them, thanks! >> >> I also must note that this is a problem for fortran examples only! Because >> when I've done "make test" after building the situation was: >> >> mt at agrayver /cygdrive/d/dev/petsc-3.1-p7 >> $ make PETSC_DIR=/cygdrive/d/dev/petsc-3.1-p7 PETSC_ARCH=cygwin-cxx-debug test >> Running test examples to verify correct installation >> C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 MPI >> process >> C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 MPI >> processes >> --------------Error detected during compile or link!----------------------- >> See http://www.mcs.anl.gov/petsc/petsc-2/documentation/troubleshooting.html >> /cygdrive/d/dev/petsc-3.1-p7/bin/win32fe/win32fe ifort -c -MT -Z7 -fpp >> -I/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/include >> -I/cygdrive/d/dev/petsc-3.1-p7/include -I/cygdrive/d/Dev/HPC_2008_SDK/Include >> -I/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/include >> -I/cygdrive/d/Dev/HPC_2008_SDK/Include -o ex5f.o ex5f.F >> >> /cygdrive/d/dev/petsc-3.1-p7/bin/win32fe/win32fe ifort -MT -Z7 -fpp -o ex5f >> ex5f.o -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib >> -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib -lpetsc >> -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib -lflapack -lfblas >> /cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpi.lib >> /cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib Gdi32.lib User32.lib >> Advapi32.lib Kernel32.lib Ws2_32.lib >> >> libpetsc.lib(fuser.o) : error LNK2019: unresolved external symbol >> __imp_GetUserNameA referenced in function "int __cdecl PetscGetUserName(char * >> const,unsigned__int64)" (?PetscGetUserName@@YAHQEAD_K at Z) >> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol >> __imp_ReleaseDC referenced in function "int __cdecl >> PetscDrawSetDoubleBuffer_Win32(struct_p_PetscDraw *)" >> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol >> __imp_BitBlt referenced in function "int __cdecl >> PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" >> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol >> __imp_ExtFloodFill referenced in function "int __cdecl >> PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" >> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol >> __imp_SelectObject referenced in function "int __cdecl >> PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" >> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol >> __imp_CreateCompatibleBitmap referenced in function "int __cdecl >> PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" >> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol >> __imp_CreateCompatibleDC referenced in function "int __cdecl >> PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" >> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol >> __imp_GetDC referenced in function "int __cdecl >> PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" >> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol >> __imp_UpdateWindow referenced in function "int __cdecl >> PetscDrawRectangle_Win32(struct _p_PetscDraw >> *,double,double,double,double,int,int,int,int)" >> (?PetscDrawRectangle_Win32@@YAHPEAU_p_PetscDraw@@NNNNHHHH at Z) >> >> On 03.03.2011 14:49, Satish Balay wrote: >>> can you send the corresponding configure.log and make.log to >>> petsc-maint at mcs.anl.gov? >>> >>> Satish >>> >>> On Thu, 3 Mar 2011, Alexander Grayver wrote: >>> >>>> Hello, >>>> >>>> I built PETSc-3.1 using VC++ 2008 and IFC 10.1 with the following >>>> configuration line: >>>> >>>> ./config/configure.py --with-cc='win32fe cl' --with-fc='win32fe ifort' >>>> --with-cxx='win32fe cl' --download-f-blas-lapack=1 >>>> --with-precision=double >>>> --with-scalar-type=complex --with-clanguage=cxx >>>> --with-mpi-include="/cygdrive/d/Dev/HPC_2008_SDK/Include" >>>> >>>> --with-mpi-lib=["/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpi.lib","/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib"] >>>> --useThreads=0 --with-shared=0 >>>> >>>> Note there is C++ as main language. >>>> >>>> Now, when I'm trying to build examples I always get similar errors: >>>> >>>> Error 2 error LNK2019: unresolved external symbol ZGESVD referenced >>>> in >>>> function "int __cdecl KSPComputeExtremeSingularValues_GMRES(struct _p_KSP >>>> *,double *,double *)" >>>> (?KSPComputeExtremeSingularValues_GMRES@@YAHPEAU_p_KSP@@PEAN1 at Z) >>>> libpetsc.lib >>>> Error 3 error LNK2019: unresolved external symbol ZGEEV referenced >>>> in >>>> function "int __cdecl KSPComputeEigenvalues_GMRES(struct _p_KSP >>>> *,int,double >>>> *,double *,int *)" >>>> (?KSPComputeEigenvalues_GMRES@@YAHPEAU_p_KSP@@HPEAN1PEAH at Z) >>>> libpetsc.lib >>>> Error 4 error LNK2001: unresolved external symbol ZGEEV >>>> libpetsc.lib >>>> Error 5 error LNK2019: unresolved external symbol DZNRM2 referenced >>>> in >>>> function "int __cdecl VecNorm_MPI(struct _p_Vec *,enum NormType,double *)" >>>> (?VecNorm_MPI@@YAHPEAU_p_Vec@@W4NormType@@PEAN at Z) libpetsc.lib >>>> Error 6 error LNK2001: unresolved external symbol DZNRM2 >>>> libpetsc.lib >>>> Error 7 error LNK2019: unresolved external symbol DZASUM referenced >>>> in >>>> function "int __cdecl VecNorm_Seq(struct _p_Vec *,enum NormType,double *)" >>>> (?VecNorm_Seq@@YAHPEAU_p_Vec@@W4NormType@@PEAN at Z) libpetsc.lib >>>> ... >>>> >>>> It look like calling conventions are in disagreement. Any ideas? May be I >>>> should specify conventions myself in configuration line? If so, how to do >>>> that? >>>> >> From C.Klaij at marin.nl Thu Mar 3 09:36:00 2011 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Thu, 3 Mar 2011 15:36:00 +0000 Subject: [petsc-users] PCDiagonalScale Message-ID: Barry, Thanks for your reply. Good to know it's indeed a missing check. I'm not following your suggestion to modify PETSc; it means I would have to modify it on a number of machines, some of which I can't even access. For the time being, I'm considering scaling the system in my code before passing it to PETSc. Besides, this would allow me to use FGMRES in case GCR gives trouble. Chris Date: Wed, 2 Mar 2011 12:57:38 -0600 From: Barry Smith Subject: Re: [petsc-users] PCDiagonalScale To: PETSc users list Message-ID: Content-Type: text/plain; charset=us-ascii On Mar 1, 2011, at 2:28 AM, Klaij, Christiaan wrote: > Barry, > > I do not want to use it for the reason given in the manual. > > Instead I'm trying to use SIMPLE as a preconditioner for a colocated, cell-centered FV discretization of the incompressible Navier-Stokes equations. I'm using a matfree approach for the matrix and the (right) preconditioner. I would like to scale the system as D A M D^{-1} z = D b but I get the feeling that PCDiagonalScaleSet with GCR doesn't do anything. > > How about Jed's comment: > > "GCR also tolerates a variable preconditioner and it does not have such a check. I don't know if that means it can use diagonal scaling or just that someone forgot to check, but you could try it." KSPSolve_GCR() is not coded for using that extra diagonal scaling and does not check to see if you have requested it. So, as you saw, if you turn it on it has no effect. I have added an error check in petsc-dev so it will tell you that it does not use it. Thank you for letting us know about this bug. You may be able to add support for it. The code is in src/ksp/ksp/impls/gcr/grc.c Note that the current code has the lines ierr = PCApply( pc, r, s ); CHKERRQ(ierr); /* s = B^{-1} r */ ierr = MatMult( A, s, v ); CHKERRQ(ierr); /* v = A s */ to fix the code to do the diagonal scaling you need to do a few things 1) replace the two lines above with a single call to KSP_PCApplyBAorAB() this will incorporate the scaling in the application of A*B so instead it provides D A B D^{-1} 2) this is more difficult. Understand the code well enough to fix it so that it properly takes into account the diagonal scaling if the initial guess is nonzero and also the fact that you are solving D A B D^{-1} y = D b instead of A B z = b and hence A x = b where x = B z so x = B *D^{-1} y Barry dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:C.Klaij at marin.nl T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl From balay at mcs.anl.gov Thu Mar 3 09:43:24 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 3 Mar 2011 09:43:24 -0600 (CST) Subject: [petsc-users] Calling convention from windows VC++ and IFC In-Reply-To: <4D6FB4D5.6040708@gfz-potsdam.de> References: <4D6F80C4.4070200@gfz-potsdam.de> <4D6FA9DA.7070403@gfz-potsdam.de> <4D6FB4D5.6040708@gfz-potsdam.de> Message-ID: Currently I don't have access to ifort to try. For one - its strange that c/c++ examples work - but not fortran - esp because all blas calls are from the same c/c++ PETSc code. One thing you can try: cd src/ksp/ksp/examples/tutorials make ex2 make ex2f [if ex2 works and ex2f fails - edit makefile - and change target ex2f: from using FLINKER to CLINKER and see if it makes a difference] Satish On Thu, 3 Mar 2011, Alexander Grayver wrote: > Actually, I've added this flag after sending first message to mailing list > (sorry for inconveniences) because I though it could help, unfortunately it > didn't. > > The problem probably could be reproduced compiling petsc using > --with-clanguage=cxx and VC++ and then trying to run FORTRAN examples. > > On 03.03.2011 16:26, Satish Balay wrote: > > perhaps you can remove '--CXX_CXXFLAGS=-Gz' and retry. > > > > Satish > > > > On Thu, 3 Mar 2011, Alexander Grayver wrote: > > > > > I've sent them, thanks! > > > > > > I also must note that this is a problem for fortran examples only! Because > > > when I've done "make test" after building the situation was: > > > > > > mt at agrayver /cygdrive/d/dev/petsc-3.1-p7 > > > $ make PETSC_DIR=/cygdrive/d/dev/petsc-3.1-p7 PETSC_ARCH=cygwin-cxx-debug > > > test > > > Running test examples to verify correct installation > > > C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 MPI > > > process > > > C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 MPI > > > processes > > > --------------Error detected during compile or > > > link!----------------------- > > > See > > > http://www.mcs.anl.gov/petsc/petsc-2/documentation/troubleshooting.html > > > /cygdrive/d/dev/petsc-3.1-p7/bin/win32fe/win32fe ifort -c -MT -Z7 -fpp > > > -I/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/include > > > -I/cygdrive/d/dev/petsc-3.1-p7/include > > > -I/cygdrive/d/Dev/HPC_2008_SDK/Include > > > -I/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/include > > > -I/cygdrive/d/Dev/HPC_2008_SDK/Include -o ex5f.o ex5f.F > > > > > > /cygdrive/d/dev/petsc-3.1-p7/bin/win32fe/win32fe ifort -MT -Z7 -fpp -o > > > ex5f > > > ex5f.o -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib > > > -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib -lpetsc > > > -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib -lflapack -lfblas > > > /cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpi.lib > > > /cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib Gdi32.lib User32.lib > > > Advapi32.lib Kernel32.lib Ws2_32.lib > > > > > > libpetsc.lib(fuser.o) : error LNK2019: unresolved external symbol > > > __imp_GetUserNameA referenced in function "int __cdecl > > > PetscGetUserName(char * > > > const,unsigned__int64)" (?PetscGetUserName@@YAHQEAD_K at Z) > > > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol > > > __imp_ReleaseDC referenced in function "int __cdecl > > > PetscDrawSetDoubleBuffer_Win32(struct_p_PetscDraw *)" > > > (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) > > > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol > > > __imp_BitBlt referenced in function "int __cdecl > > > PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" > > > (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) > > > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol > > > __imp_ExtFloodFill referenced in function "int __cdecl > > > PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" > > > (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) > > > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol > > > __imp_SelectObject referenced in function "int __cdecl > > > PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" > > > (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) > > > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol > > > __imp_CreateCompatibleBitmap referenced in function "int __cdecl > > > PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" > > > (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) > > > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol > > > __imp_CreateCompatibleDC referenced in function "int __cdecl > > > PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" > > > (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) > > > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol > > > __imp_GetDC referenced in function "int __cdecl > > > PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" > > > (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) > > > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol > > > __imp_UpdateWindow referenced in function "int __cdecl > > > PetscDrawRectangle_Win32(struct _p_PetscDraw > > > *,double,double,double,double,int,int,int,int)" > > > (?PetscDrawRectangle_Win32@@YAHPEAU_p_PetscDraw@@NNNNHHHH at Z) > > > > > > On 03.03.2011 14:49, Satish Balay wrote: > > > > can you send the corresponding configure.log and make.log to > > > > petsc-maint at mcs.anl.gov? > > > > > > > > Satish > > > > > > > > On Thu, 3 Mar 2011, Alexander Grayver wrote: > > > > > > > > > Hello, > > > > > > > > > > I built PETSc-3.1 using VC++ 2008 and IFC 10.1 with the following > > > > > configuration line: > > > > > > > > > > ./config/configure.py --with-cc='win32fe cl' --with-fc='win32fe ifort' > > > > > --with-cxx='win32fe cl' --download-f-blas-lapack=1 > > > > > --with-precision=double > > > > > --with-scalar-type=complex --with-clanguage=cxx > > > > > --with-mpi-include="/cygdrive/d/Dev/HPC_2008_SDK/Include" > > > > > > > > > > > > > > > --with-mpi-lib=["/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpi.lib","/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib"] > > > > > --useThreads=0 --with-shared=0 > > > > > > > > > > Note there is C++ as main language. > > > > > > > > > > Now, when I'm trying to build examples I always get similar errors: > > > > > > > > > > Error 2 error LNK2019: unresolved external symbol ZGESVD > > > > > referenced > > > > > in > > > > > function "int __cdecl KSPComputeExtremeSingularValues_GMRES(struct > > > > > _p_KSP > > > > > *,double *,double *)" > > > > > (?KSPComputeExtremeSingularValues_GMRES@@YAHPEAU_p_KSP@@PEAN1 at Z) > > > > > libpetsc.lib > > > > > Error 3 error LNK2019: unresolved external symbol ZGEEV > > > > > referenced > > > > > in > > > > > function "int __cdecl KSPComputeEigenvalues_GMRES(struct _p_KSP > > > > > *,int,double > > > > > *,double *,int *)" > > > > > (?KSPComputeEigenvalues_GMRES@@YAHPEAU_p_KSP@@HPEAN1PEAH at Z) > > > > > libpetsc.lib > > > > > Error 4 error LNK2001: unresolved external symbol ZGEEV > > > > > libpetsc.lib > > > > > Error 5 error LNK2019: unresolved external symbol DZNRM2 > > > > > referenced > > > > > in > > > > > function "int __cdecl VecNorm_MPI(struct _p_Vec *,enum NormType,double > > > > > *)" > > > > > (?VecNorm_MPI@@YAHPEAU_p_Vec@@W4NormType@@PEAN at Z) libpetsc.lib > > > > > Error 6 error LNK2001: unresolved external symbol DZNRM2 > > > > > libpetsc.lib > > > > > Error 7 error LNK2019: unresolved external symbol DZASUM > > > > > referenced > > > > > in > > > > > function "int __cdecl VecNorm_Seq(struct _p_Vec *,enum NormType,double > > > > > *)" > > > > > (?VecNorm_Seq@@YAHPEAU_p_Vec@@W4NormType@@PEAN at Z) libpetsc.lib > > > > > ... > > > > > > > > > > It look like calling conventions are in disagreement. Any ideas? May > > > > > be I > > > > > should specify conventions myself in configuration line? If so, how to > > > > > do > > > > > that? > > > > > > > > > > From balay at mcs.anl.gov Thu Mar 3 09:48:28 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 3 Mar 2011 09:48:28 -0600 (CST) Subject: [petsc-users] Calling convention from windows VC++ and IFC In-Reply-To: References: <4D6F80C4.4070200@gfz-potsdam.de> <4D6FA9DA.7070403@gfz-potsdam.de> <4D6FB4D5.6040708@gfz-potsdam.de> Message-ID: Oh - I see that with --CXX_CXXFLAGS=-Gz - you get different errors - not blaslapack errors. > > > > libpetsc.lib(fuser.o) : error LNK2019: unresolved external symbol > > > > __imp_GetUserNameA referenced in function "int __cdecl > > > > PetscGetUserName(char * > > > > const,unsigned__int64)" (?PetscGetUserName@@YAHQEAD_K at Z) This is due to compiling C interface with -Gz - so now fortran interface is unable to link with the c interface. Satish On Thu, 3 Mar 2011, Satish Balay wrote: > Currently I don't have access to ifort to try. > > For one - its strange that c/c++ examples work - but not fortran - esp > because all blas calls are from the same c/c++ PETSc code. > > One thing you can try: > > cd src/ksp/ksp/examples/tutorials > make ex2 > make ex2f > > [if ex2 works and ex2f fails - edit makefile - and change target ex2f: > from using FLINKER to CLINKER and see if it makes a difference] > > Satish > > On Thu, 3 Mar 2011, Alexander Grayver wrote: > > > Actually, I've added this flag after sending first message to mailing list > > (sorry for inconveniences) because I though it could help, unfortunately it > > didn't. > > > > The problem probably could be reproduced compiling petsc using > > --with-clanguage=cxx and VC++ and then trying to run FORTRAN examples. > > > > On 03.03.2011 16:26, Satish Balay wrote: > > > perhaps you can remove '--CXX_CXXFLAGS=-Gz' and retry. > > > > > > Satish > > > > > > On Thu, 3 Mar 2011, Alexander Grayver wrote: > > > > > > > I've sent them, thanks! > > > > > > > > I also must note that this is a problem for fortran examples only! Because > > > > when I've done "make test" after building the situation was: > > > > > > > > mt at agrayver /cygdrive/d/dev/petsc-3.1-p7 > > > > $ make PETSC_DIR=/cygdrive/d/dev/petsc-3.1-p7 PETSC_ARCH=cygwin-cxx-debug > > > > test > > > > Running test examples to verify correct installation > > > > C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 MPI > > > > process > > > > C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 MPI > > > > processes > > > > --------------Error detected during compile or > > > > link!----------------------- > > > > See > > > > http://www.mcs.anl.gov/petsc/petsc-2/documentation/troubleshooting.html > > > > /cygdrive/d/dev/petsc-3.1-p7/bin/win32fe/win32fe ifort -c -MT -Z7 -fpp > > > > -I/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/include > > > > -I/cygdrive/d/dev/petsc-3.1-p7/include > > > > -I/cygdrive/d/Dev/HPC_2008_SDK/Include > > > > -I/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/include > > > > -I/cygdrive/d/Dev/HPC_2008_SDK/Include -o ex5f.o ex5f.F > > > > > > > > /cygdrive/d/dev/petsc-3.1-p7/bin/win32fe/win32fe ifort -MT -Z7 -fpp -o > > > > ex5f > > > > ex5f.o -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib > > > > -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib -lpetsc > > > > -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib -lflapack -lfblas > > > > /cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpi.lib > > > > /cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib Gdi32.lib User32.lib > > > > Advapi32.lib Kernel32.lib Ws2_32.lib > > > > > > > > libpetsc.lib(fuser.o) : error LNK2019: unresolved external symbol > > > > __imp_GetUserNameA referenced in function "int __cdecl > > > > PetscGetUserName(char * > > > > const,unsigned__int64)" (?PetscGetUserName@@YAHQEAD_K at Z) > > > > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol > > > > __imp_ReleaseDC referenced in function "int __cdecl > > > > PetscDrawSetDoubleBuffer_Win32(struct_p_PetscDraw *)" > > > > (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) > > > > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol > > > > __imp_BitBlt referenced in function "int __cdecl > > > > PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" > > > > (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) > > > > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol > > > > __imp_ExtFloodFill referenced in function "int __cdecl > > > > PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" > > > > (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) > > > > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol > > > > __imp_SelectObject referenced in function "int __cdecl > > > > PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" > > > > (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) > > > > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol > > > > __imp_CreateCompatibleBitmap referenced in function "int __cdecl > > > > PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" > > > > (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) > > > > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol > > > > __imp_CreateCompatibleDC referenced in function "int __cdecl > > > > PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" > > > > (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) > > > > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol > > > > __imp_GetDC referenced in function "int __cdecl > > > > PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" > > > > (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) > > > > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol > > > > __imp_UpdateWindow referenced in function "int __cdecl > > > > PetscDrawRectangle_Win32(struct _p_PetscDraw > > > > *,double,double,double,double,int,int,int,int)" > > > > (?PetscDrawRectangle_Win32@@YAHPEAU_p_PetscDraw@@NNNNHHHH at Z) > > > > > > > > On 03.03.2011 14:49, Satish Balay wrote: > > > > > can you send the corresponding configure.log and make.log to > > > > > petsc-maint at mcs.anl.gov? > > > > > > > > > > Satish > > > > > > > > > > On Thu, 3 Mar 2011, Alexander Grayver wrote: > > > > > > > > > > > Hello, > > > > > > > > > > > > I built PETSc-3.1 using VC++ 2008 and IFC 10.1 with the following > > > > > > configuration line: > > > > > > > > > > > > ./config/configure.py --with-cc='win32fe cl' --with-fc='win32fe ifort' > > > > > > --with-cxx='win32fe cl' --download-f-blas-lapack=1 > > > > > > --with-precision=double > > > > > > --with-scalar-type=complex --with-clanguage=cxx > > > > > > --with-mpi-include="/cygdrive/d/Dev/HPC_2008_SDK/Include" > > > > > > > > > > > > > > > > > > --with-mpi-lib=["/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpi.lib","/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib"] > > > > > > --useThreads=0 --with-shared=0 > > > > > > > > > > > > Note there is C++ as main language. > > > > > > > > > > > > Now, when I'm trying to build examples I always get similar errors: > > > > > > > > > > > > Error 2 error LNK2019: unresolved external symbol ZGESVD > > > > > > referenced > > > > > > in > > > > > > function "int __cdecl KSPComputeExtremeSingularValues_GMRES(struct > > > > > > _p_KSP > > > > > > *,double *,double *)" > > > > > > (?KSPComputeExtremeSingularValues_GMRES@@YAHPEAU_p_KSP@@PEAN1 at Z) > > > > > > libpetsc.lib > > > > > > Error 3 error LNK2019: unresolved external symbol ZGEEV > > > > > > referenced > > > > > > in > > > > > > function "int __cdecl KSPComputeEigenvalues_GMRES(struct _p_KSP > > > > > > *,int,double > > > > > > *,double *,int *)" > > > > > > (?KSPComputeEigenvalues_GMRES@@YAHPEAU_p_KSP@@HPEAN1PEAH at Z) > > > > > > libpetsc.lib > > > > > > Error 4 error LNK2001: unresolved external symbol ZGEEV > > > > > > libpetsc.lib > > > > > > Error 5 error LNK2019: unresolved external symbol DZNRM2 > > > > > > referenced > > > > > > in > > > > > > function "int __cdecl VecNorm_MPI(struct _p_Vec *,enum NormType,double > > > > > > *)" > > > > > > (?VecNorm_MPI@@YAHPEAU_p_Vec@@W4NormType@@PEAN at Z) libpetsc.lib > > > > > > Error 6 error LNK2001: unresolved external symbol DZNRM2 > > > > > > libpetsc.lib > > > > > > Error 7 error LNK2019: unresolved external symbol DZASUM > > > > > > referenced > > > > > > in > > > > > > function "int __cdecl VecNorm_Seq(struct _p_Vec *,enum NormType,double > > > > > > *)" > > > > > > (?VecNorm_Seq@@YAHPEAU_p_Vec@@W4NormType@@PEAN at Z) libpetsc.lib > > > > > > ... > > > > > > > > > > > > It look like calling conventions are in disagreement. Any ideas? May > > > > > > be I > > > > > > should specify conventions myself in configuration line? If so, how to > > > > > > do > > > > > > that? > > > > > > > > > > > > > > > > From Chun.SUN at 3ds.com Thu Mar 3 09:48:33 2011 From: Chun.SUN at 3ds.com (SUN Chun) Date: Thu, 3 Mar 2011 15:48:33 +0000 Subject: [petsc-users] Calling convention from windows VC++ and IFC In-Reply-To: <4D6FB4D5.6040708@gfz-potsdam.de> References: <4D6F80C4.4070200@gfz-potsdam.de> <4D6FA9DA.7070403@gfz-potsdam.de> <4D6FB4D5.6040708@gfz-potsdam.de> Message-ID: <3AB51330CF1300428AD80035DAF20EE3065DAD1F@AG-DCC-MBX03.dsone.3ds.com> Maybe, you want to look at petscblaslapack.h and turn flag on/off from the #ifdefs in that file. From the original message it seems to me the convention issue is between calling BLAS routine. -----Original Message----- From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Alexander Grayver Sent: Thursday, March 03, 2011 10:34 AM To: PETSc users list Subject: Re: [petsc-users] Calling convention from windows VC++ and IFC Actually, I've added this flag after sending first message to mailing list (sorry for inconveniences) because I though it could help, unfortunately it didn't. The problem probably could be reproduced compiling petsc using --with-clanguage=cxx and VC++ and then trying to run FORTRAN examples. On 03.03.2011 16:26, Satish Balay wrote: > perhaps you can remove '--CXX_CXXFLAGS=-Gz' and retry. > > Satish > > On Thu, 3 Mar 2011, Alexander Grayver wrote: > >> I've sent them, thanks! >> >> I also must note that this is a problem for fortran examples only! Because >> when I've done "make test" after building the situation was: >> >> mt at agrayver /cygdrive/d/dev/petsc-3.1-p7 >> $ make PETSC_DIR=/cygdrive/d/dev/petsc-3.1-p7 PETSC_ARCH=cygwin-cxx-debug test >> Running test examples to verify correct installation >> C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 MPI >> process >> C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 MPI >> processes >> --------------Error detected during compile or link!----------------------- >> See http://www.mcs.anl.gov/petsc/petsc-2/documentation/troubleshooting.html >> /cygdrive/d/dev/petsc-3.1-p7/bin/win32fe/win32fe ifort -c -MT -Z7 -fpp >> -I/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/include >> -I/cygdrive/d/dev/petsc-3.1-p7/include -I/cygdrive/d/Dev/HPC_2008_SDK/Include >> -I/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/include >> -I/cygdrive/d/Dev/HPC_2008_SDK/Include -o ex5f.o ex5f.F >> >> /cygdrive/d/dev/petsc-3.1-p7/bin/win32fe/win32fe ifort -MT -Z7 -fpp -o ex5f >> ex5f.o -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib >> -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib -lpetsc >> -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib -lflapack -lfblas >> /cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpi.lib >> /cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib Gdi32.lib User32.lib >> Advapi32.lib Kernel32.lib Ws2_32.lib >> >> libpetsc.lib(fuser.o) : error LNK2019: unresolved external symbol >> __imp_GetUserNameA referenced in function "int __cdecl PetscGetUserName(char * >> const,unsigned__int64)" (?PetscGetUserName@@YAHQEAD_K at Z) >> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol >> __imp_ReleaseDC referenced in function "int __cdecl >> PetscDrawSetDoubleBuffer_Win32(struct_p_PetscDraw *)" >> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol >> __imp_BitBlt referenced in function "int __cdecl >> PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" >> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol >> __imp_ExtFloodFill referenced in function "int __cdecl >> PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" >> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol >> __imp_SelectObject referenced in function "int __cdecl >> PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" >> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol >> __imp_CreateCompatibleBitmap referenced in function "int __cdecl >> PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" >> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol >> __imp_CreateCompatibleDC referenced in function "int __cdecl >> PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" >> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol >> __imp_GetDC referenced in function "int __cdecl >> PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" >> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol >> __imp_UpdateWindow referenced in function "int __cdecl >> PetscDrawRectangle_Win32(struct _p_PetscDraw >> *,double,double,double,double,int,int,int,int)" >> (?PetscDrawRectangle_Win32@@YAHPEAU_p_PetscDraw@@NNNNHHHH at Z) >> >> On 03.03.2011 14:49, Satish Balay wrote: >>> can you send the corresponding configure.log and make.log to >>> petsc-maint at mcs.anl.gov? >>> >>> Satish >>> >>> On Thu, 3 Mar 2011, Alexander Grayver wrote: >>> >>>> Hello, >>>> >>>> I built PETSc-3.1 using VC++ 2008 and IFC 10.1 with the following >>>> configuration line: >>>> >>>> ./config/configure.py --with-cc='win32fe cl' --with-fc='win32fe ifort' >>>> --with-cxx='win32fe cl' --download-f-blas-lapack=1 >>>> --with-precision=double >>>> --with-scalar-type=complex --with-clanguage=cxx >>>> --with-mpi-include="/cygdrive/d/Dev/HPC_2008_SDK/Include" >>>> >>>> --with-mpi-lib=["/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpi.lib","/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib"] >>>> --useThreads=0 --with-shared=0 >>>> >>>> Note there is C++ as main language. >>>> >>>> Now, when I'm trying to build examples I always get similar errors: >>>> >>>> Error 2 error LNK2019: unresolved external symbol ZGESVD referenced >>>> in >>>> function "int __cdecl KSPComputeExtremeSingularValues_GMRES(struct _p_KSP >>>> *,double *,double *)" >>>> (?KSPComputeExtremeSingularValues_GMRES@@YAHPEAU_p_KSP@@PEAN1 at Z) >>>> libpetsc.lib >>>> Error 3 error LNK2019: unresolved external symbol ZGEEV referenced >>>> in >>>> function "int __cdecl KSPComputeEigenvalues_GMRES(struct _p_KSP >>>> *,int,double >>>> *,double *,int *)" >>>> (?KSPComputeEigenvalues_GMRES@@YAHPEAU_p_KSP@@HPEAN1PEAH at Z) >>>> libpetsc.lib >>>> Error 4 error LNK2001: unresolved external symbol ZGEEV >>>> libpetsc.lib >>>> Error 5 error LNK2019: unresolved external symbol DZNRM2 referenced >>>> in >>>> function "int __cdecl VecNorm_MPI(struct _p_Vec *,enum NormType,double *)" >>>> (?VecNorm_MPI@@YAHPEAU_p_Vec@@W4NormType@@PEAN at Z) libpetsc.lib >>>> Error 6 error LNK2001: unresolved external symbol DZNRM2 >>>> libpetsc.lib >>>> Error 7 error LNK2019: unresolved external symbol DZASUM referenced >>>> in >>>> function "int __cdecl VecNorm_Seq(struct _p_Vec *,enum NormType,double *)" >>>> (?VecNorm_Seq@@YAHPEAU_p_Vec@@W4NormType@@PEAN at Z) libpetsc.lib >>>> ... >>>> >>>> It look like calling conventions are in disagreement. Any ideas? May be I >>>> should specify conventions myself in configuration line? If so, how to do >>>> that? >>>> >> From agrayver at gfz-potsdam.de Thu Mar 3 09:50:42 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Thu, 03 Mar 2011 16:50:42 +0100 Subject: [petsc-users] Calling convention from windows VC++ and IFC In-Reply-To: References: <4D6F80C4.4070200@gfz-potsdam.de> <4D6FA9DA.7070403@gfz-potsdam.de> <4D6FB4D5.6040708@gfz-potsdam.de> Message-ID: <4D6FB8D2.8020202@gfz-potsdam.de> Really! Thanks Satish. Errors apparently are different when I use stdcall convention for compiling instead of default one. But the question is how to combine these interfaces now? On 03.03.2011 16:48, Satish Balay wrote: > Oh - I see that with --CXX_CXXFLAGS=-Gz - you get different errors - not blaslapack errors. > >>>>> libpetsc.lib(fuser.o) : error LNK2019: unresolved external symbol >>>>> __imp_GetUserNameA referenced in function "int __cdecl >>>>> PetscGetUserName(char * >>>>> const,unsigned__int64)" (?PetscGetUserName@@YAHQEAD_K at Z) > This is due to compiling C interface with -Gz - so now fortran > interface is unable to link with the c interface. > > Satish > > On Thu, 3 Mar 2011, Satish Balay wrote: > >> Currently I don't have access to ifort to try. >> >> For one - its strange that c/c++ examples work - but not fortran - esp >> because all blas calls are from the same c/c++ PETSc code. >> >> One thing you can try: >> >> cd src/ksp/ksp/examples/tutorials >> make ex2 >> make ex2f >> >> [if ex2 works and ex2f fails - edit makefile - and change target ex2f: >> from using FLINKER to CLINKER and see if it makes a difference] >> >> Satish >> >> On Thu, 3 Mar 2011, Alexander Grayver wrote: >> >>> Actually, I've added this flag after sending first message to mailing list >>> (sorry for inconveniences) because I though it could help, unfortunately it >>> didn't. >>> >>> The problem probably could be reproduced compiling petsc using >>> --with-clanguage=cxx and VC++ and then trying to run FORTRAN examples. >>> >>> On 03.03.2011 16:26, Satish Balay wrote: >>>> perhaps you can remove '--CXX_CXXFLAGS=-Gz' and retry. >>>> >>>> Satish >>>> >>>> On Thu, 3 Mar 2011, Alexander Grayver wrote: >>>> >>>>> I've sent them, thanks! >>>>> >>>>> I also must note that this is a problem for fortran examples only! Because >>>>> when I've done "make test" after building the situation was: >>>>> >>>>> mt at agrayver /cygdrive/d/dev/petsc-3.1-p7 >>>>> $ make PETSC_DIR=/cygdrive/d/dev/petsc-3.1-p7 PETSC_ARCH=cygwin-cxx-debug >>>>> test >>>>> Running test examples to verify correct installation >>>>> C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 MPI >>>>> process >>>>> C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 MPI >>>>> processes >>>>> --------------Error detected during compile or >>>>> link!----------------------- >>>>> See >>>>> http://www.mcs.anl.gov/petsc/petsc-2/documentation/troubleshooting.html >>>>> /cygdrive/d/dev/petsc-3.1-p7/bin/win32fe/win32fe ifort -c -MT -Z7 -fpp >>>>> -I/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/include >>>>> -I/cygdrive/d/dev/petsc-3.1-p7/include >>>>> -I/cygdrive/d/Dev/HPC_2008_SDK/Include >>>>> -I/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/include >>>>> -I/cygdrive/d/Dev/HPC_2008_SDK/Include -o ex5f.o ex5f.F >>>>> >>>>> /cygdrive/d/dev/petsc-3.1-p7/bin/win32fe/win32fe ifort -MT -Z7 -fpp -o >>>>> ex5f >>>>> ex5f.o -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib >>>>> -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib -lpetsc >>>>> -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib -lflapack -lfblas >>>>> /cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpi.lib >>>>> /cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib Gdi32.lib User32.lib >>>>> Advapi32.lib Kernel32.lib Ws2_32.lib >>>>> >>>>> libpetsc.lib(fuser.o) : error LNK2019: unresolved external symbol >>>>> __imp_GetUserNameA referenced in function "int __cdecl >>>>> PetscGetUserName(char * >>>>> const,unsigned__int64)" (?PetscGetUserName@@YAHQEAD_K at Z) >>>>> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol >>>>> __imp_ReleaseDC referenced in function "int __cdecl >>>>> PetscDrawSetDoubleBuffer_Win32(struct_p_PetscDraw *)" >>>>> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >>>>> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol >>>>> __imp_BitBlt referenced in function "int __cdecl >>>>> PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" >>>>> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >>>>> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol >>>>> __imp_ExtFloodFill referenced in function "int __cdecl >>>>> PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" >>>>> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >>>>> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol >>>>> __imp_SelectObject referenced in function "int __cdecl >>>>> PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" >>>>> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >>>>> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol >>>>> __imp_CreateCompatibleBitmap referenced in function "int __cdecl >>>>> PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" >>>>> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >>>>> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol >>>>> __imp_CreateCompatibleDC referenced in function "int __cdecl >>>>> PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" >>>>> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >>>>> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol >>>>> __imp_GetDC referenced in function "int __cdecl >>>>> PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" >>>>> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >>>>> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external symbol >>>>> __imp_UpdateWindow referenced in function "int __cdecl >>>>> PetscDrawRectangle_Win32(struct _p_PetscDraw >>>>> *,double,double,double,double,int,int,int,int)" >>>>> (?PetscDrawRectangle_Win32@@YAHPEAU_p_PetscDraw@@NNNNHHHH at Z) >>>>> >>>>> On 03.03.2011 14:49, Satish Balay wrote: >>>>>> can you send the corresponding configure.log and make.log to >>>>>> petsc-maint at mcs.anl.gov? >>>>>> >>>>>> Satish >>>>>> >>>>>> On Thu, 3 Mar 2011, Alexander Grayver wrote: >>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> I built PETSc-3.1 using VC++ 2008 and IFC 10.1 with the following >>>>>>> configuration line: >>>>>>> >>>>>>> ./config/configure.py --with-cc='win32fe cl' --with-fc='win32fe ifort' >>>>>>> --with-cxx='win32fe cl' --download-f-blas-lapack=1 >>>>>>> --with-precision=double >>>>>>> --with-scalar-type=complex --with-clanguage=cxx >>>>>>> --with-mpi-include="/cygdrive/d/Dev/HPC_2008_SDK/Include" >>>>>>> >>>>>>> >>>>>>> --with-mpi-lib=["/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpi.lib","/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib"] >>>>>>> --useThreads=0 --with-shared=0 >>>>>>> >>>>>>> Note there is C++ as main language. >>>>>>> >>>>>>> Now, when I'm trying to build examples I always get similar errors: >>>>>>> >>>>>>> Error 2 error LNK2019: unresolved external symbol ZGESVD >>>>>>> referenced >>>>>>> in >>>>>>> function "int __cdecl KSPComputeExtremeSingularValues_GMRES(struct >>>>>>> _p_KSP >>>>>>> *,double *,double *)" >>>>>>> (?KSPComputeExtremeSingularValues_GMRES@@YAHPEAU_p_KSP@@PEAN1 at Z) >>>>>>> libpetsc.lib >>>>>>> Error 3 error LNK2019: unresolved external symbol ZGEEV >>>>>>> referenced >>>>>>> in >>>>>>> function "int __cdecl KSPComputeEigenvalues_GMRES(struct _p_KSP >>>>>>> *,int,double >>>>>>> *,double *,int *)" >>>>>>> (?KSPComputeEigenvalues_GMRES@@YAHPEAU_p_KSP@@HPEAN1PEAH at Z) >>>>>>> libpetsc.lib >>>>>>> Error 4 error LNK2001: unresolved external symbol ZGEEV >>>>>>> libpetsc.lib >>>>>>> Error 5 error LNK2019: unresolved external symbol DZNRM2 >>>>>>> referenced >>>>>>> in >>>>>>> function "int __cdecl VecNorm_MPI(struct _p_Vec *,enum NormType,double >>>>>>> *)" >>>>>>> (?VecNorm_MPI@@YAHPEAU_p_Vec@@W4NormType@@PEAN at Z) libpetsc.lib >>>>>>> Error 6 error LNK2001: unresolved external symbol DZNRM2 >>>>>>> libpetsc.lib >>>>>>> Error 7 error LNK2019: unresolved external symbol DZASUM >>>>>>> referenced >>>>>>> in >>>>>>> function "int __cdecl VecNorm_Seq(struct _p_Vec *,enum NormType,double >>>>>>> *)" >>>>>>> (?VecNorm_Seq@@YAHPEAU_p_Vec@@W4NormType@@PEAN at Z) libpetsc.lib >>>>>>> ... >>>>>>> >>>>>>> It look like calling conventions are in disagreement. Any ideas? May >>>>>>> be I >>>>>>> should specify conventions myself in configuration line? If so, how to >>>>>>> do >>>>>>> that? >>>>>>> >>> >> From jdbst21 at gmail.com Thu Mar 3 10:56:59 2011 From: jdbst21 at gmail.com (Joshua Booth) Date: Thu, 3 Mar 2011 11:56:59 -0500 Subject: [petsc-users] MatGetSubMatrices error Message-ID: Hello, I am running a job on two processors and a matmpiaij. I want to set a submatrix of this on the one processor (rank==1). So I made Mat *localB; If(rank==1) -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Mar 3 11:01:08 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 3 Mar 2011 11:01:08 -0600 (CST) Subject: [petsc-users] Calling convention from windows VC++ and IFC In-Reply-To: <4D6FB8D2.8020202@gfz-potsdam.de> References: <4D6F80C4.4070200@gfz-potsdam.de> <4D6FA9DA.7070403@gfz-potsdam.de> <4D6FB4D5.6040708@gfz-potsdam.de> <4D6FB8D2.8020202@gfz-potsdam.de> Message-ID: You can try adding the following to /cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/include/petscconf.h and rebuild the libraries [witout rerunning configure] and see if the fortran examples work. #ifndef PETSC_STDCALL #define PETSC_STDCALL __stdcall #endif Ideally configure should set this if -Gz was specified to CC aswell - but changing CFLAGS is tricky on windows, and there is no equivalent of CXX_CXXFLAGS - so the alternative would be COPTFLAGS. However you could try the default build [witout any -Gz] - and send all the logs to petsc-maint - and we can try figuring out the issue. Satish On Thu, 3 Mar 2011, Alexander Grayver wrote: > Really! Thanks Satish. Errors apparently are different when I use stdcall > convention for compiling instead of default one. But the question is how to > combine these interfaces now? > > On 03.03.2011 16:48, Satish Balay wrote: > > Oh - I see that with --CXX_CXXFLAGS=-Gz - you get different errors - not > > blaslapack errors. > > > > > > > > libpetsc.lib(fuser.o) : error LNK2019: unresolved external symbol > > > > > > __imp_GetUserNameA referenced in function "int __cdecl > > > > > > PetscGetUserName(char * > > > > > > const,unsigned__int64)" (?PetscGetUserName@@YAHQEAD_K at Z) > > This is due to compiling C interface with -Gz - so now fortran > > interface is unable to link with the c interface. > > > > Satish > > > > On Thu, 3 Mar 2011, Satish Balay wrote: > > > > > Currently I don't have access to ifort to try. > > > > > > For one - its strange that c/c++ examples work - but not fortran - esp > > > because all blas calls are from the same c/c++ PETSc code. > > > > > > One thing you can try: > > > > > > cd src/ksp/ksp/examples/tutorials > > > make ex2 > > > make ex2f > > > > > > [if ex2 works and ex2f fails - edit makefile - and change target ex2f: > > > from using FLINKER to CLINKER and see if it makes a difference] > > > > > > Satish > > > > > > On Thu, 3 Mar 2011, Alexander Grayver wrote: > > > > > > > Actually, I've added this flag after sending first message to mailing > > > > list > > > > (sorry for inconveniences) because I though it could help, unfortunately > > > > it > > > > didn't. > > > > > > > > The problem probably could be reproduced compiling petsc using > > > > --with-clanguage=cxx and VC++ and then trying to run FORTRAN examples. > > > > > > > > On 03.03.2011 16:26, Satish Balay wrote: > > > > > perhaps you can remove '--CXX_CXXFLAGS=-Gz' and retry. > > > > > > > > > > Satish > > > > > > > > > > On Thu, 3 Mar 2011, Alexander Grayver wrote: > > > > > > > > > > > I've sent them, thanks! > > > > > > > > > > > > I also must note that this is a problem for fortran examples only! > > > > > > Because > > > > > > when I've done "make test" after building the situation was: > > > > > > > > > > > > mt at agrayver /cygdrive/d/dev/petsc-3.1-p7 > > > > > > $ make PETSC_DIR=/cygdrive/d/dev/petsc-3.1-p7 > > > > > > PETSC_ARCH=cygwin-cxx-debug > > > > > > test > > > > > > Running test examples to verify correct installation > > > > > > C/C++ example src/snes/examples/tutorials/ex19 run successfully with > > > > > > 1 MPI > > > > > > process > > > > > > C/C++ example src/snes/examples/tutorials/ex19 run successfully with > > > > > > 2 MPI > > > > > > processes > > > > > > --------------Error detected during compile or > > > > > > link!----------------------- > > > > > > See > > > > > > http://www.mcs.anl.gov/petsc/petsc-2/documentation/troubleshooting.html > > > > > > /cygdrive/d/dev/petsc-3.1-p7/bin/win32fe/win32fe ifort -c -MT -Z7 > > > > > > -fpp > > > > > > -I/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/include > > > > > > -I/cygdrive/d/dev/petsc-3.1-p7/include > > > > > > -I/cygdrive/d/Dev/HPC_2008_SDK/Include > > > > > > -I/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/include > > > > > > -I/cygdrive/d/Dev/HPC_2008_SDK/Include -o ex5f.o ex5f.F > > > > > > > > > > > > /cygdrive/d/dev/petsc-3.1-p7/bin/win32fe/win32fe ifort -MT -Z7 -fpp > > > > > > -o > > > > > > ex5f > > > > > > ex5f.o -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib > > > > > > -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib -lpetsc > > > > > > -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib -lflapack > > > > > > -lfblas > > > > > > /cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpi.lib > > > > > > /cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib Gdi32.lib > > > > > > User32.lib > > > > > > Advapi32.lib Kernel32.lib Ws2_32.lib > > > > > > > > > > > > libpetsc.lib(fuser.o) : error LNK2019: unresolved external symbol > > > > > > __imp_GetUserNameA referenced in function "int __cdecl > > > > > > PetscGetUserName(char * > > > > > > const,unsigned__int64)" (?PetscGetUserName@@YAHQEAD_K at Z) > > > > > > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external > > > > > > symbol > > > > > > __imp_ReleaseDC referenced in function "int __cdecl > > > > > > PetscDrawSetDoubleBuffer_Win32(struct_p_PetscDraw *)" > > > > > > (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) > > > > > > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external > > > > > > symbol > > > > > > __imp_BitBlt referenced in function "int __cdecl > > > > > > PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" > > > > > > (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) > > > > > > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external > > > > > > symbol > > > > > > __imp_ExtFloodFill referenced in function "int __cdecl > > > > > > PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" > > > > > > (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) > > > > > > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external > > > > > > symbol > > > > > > __imp_SelectObject referenced in function "int __cdecl > > > > > > PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" > > > > > > (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) > > > > > > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external > > > > > > symbol > > > > > > __imp_CreateCompatibleBitmap referenced in function "int __cdecl > > > > > > PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" > > > > > > (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) > > > > > > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external > > > > > > symbol > > > > > > __imp_CreateCompatibleDC referenced in function "int __cdecl > > > > > > PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" > > > > > > (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) > > > > > > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external > > > > > > symbol > > > > > > __imp_GetDC referenced in function "int __cdecl > > > > > > PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" > > > > > > (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) > > > > > > libpetsc.lib(win32draw.o) : error LNK2019: unresolved external > > > > > > symbol > > > > > > __imp_UpdateWindow referenced in function "int __cdecl > > > > > > PetscDrawRectangle_Win32(struct _p_PetscDraw > > > > > > *,double,double,double,double,int,int,int,int)" > > > > > > (?PetscDrawRectangle_Win32@@YAHPEAU_p_PetscDraw@@NNNNHHHH at Z) > > > > > > > > > > > > On 03.03.2011 14:49, Satish Balay wrote: > > > > > > > can you send the corresponding configure.log and make.log to > > > > > > > petsc-maint at mcs.anl.gov? > > > > > > > > > > > > > > Satish > > > > > > > > > > > > > > On Thu, 3 Mar 2011, Alexander Grayver wrote: > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > I built PETSc-3.1 using VC++ 2008 and IFC 10.1 with the > > > > > > > > following > > > > > > > > configuration line: > > > > > > > > > > > > > > > > ./config/configure.py --with-cc='win32fe cl' --with-fc='win32fe > > > > > > > > ifort' > > > > > > > > --with-cxx='win32fe cl' --download-f-blas-lapack=1 > > > > > > > > --with-precision=double > > > > > > > > --with-scalar-type=complex --with-clanguage=cxx > > > > > > > > --with-mpi-include="/cygdrive/d/Dev/HPC_2008_SDK/Include" > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --with-mpi-lib=["/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpi.lib","/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib"] > > > > > > > > --useThreads=0 --with-shared=0 > > > > > > > > > > > > > > > > Note there is C++ as main language. > > > > > > > > > > > > > > > > Now, when I'm trying to build examples I always get similar > > > > > > > > errors: > > > > > > > > > > > > > > > > Error 2 error LNK2019: unresolved external symbol ZGESVD > > > > > > > > referenced > > > > > > > > in > > > > > > > > function "int __cdecl > > > > > > > > KSPComputeExtremeSingularValues_GMRES(struct > > > > > > > > _p_KSP > > > > > > > > *,double *,double *)" > > > > > > > > (?KSPComputeExtremeSingularValues_GMRES@@YAHPEAU_p_KSP@@PEAN1 at Z) > > > > > > > > libpetsc.lib > > > > > > > > Error 3 error LNK2019: unresolved external symbol ZGEEV > > > > > > > > referenced > > > > > > > > in > > > > > > > > function "int __cdecl KSPComputeEigenvalues_GMRES(struct _p_KSP > > > > > > > > *,int,double > > > > > > > > *,double *,int *)" > > > > > > > > (?KSPComputeEigenvalues_GMRES@@YAHPEAU_p_KSP@@HPEAN1PEAH at Z) > > > > > > > > libpetsc.lib > > > > > > > > Error 4 error LNK2001: unresolved external symbol ZGEEV > > > > > > > > libpetsc.lib > > > > > > > > Error 5 error LNK2019: unresolved external symbol DZNRM2 > > > > > > > > referenced > > > > > > > > in > > > > > > > > function "int __cdecl VecNorm_MPI(struct _p_Vec *,enum > > > > > > > > NormType,double > > > > > > > > *)" > > > > > > > > (?VecNorm_MPI@@YAHPEAU_p_Vec@@W4NormType@@PEAN at Z) > > > > > > > > libpetsc.lib > > > > > > > > Error 6 error LNK2001: unresolved external symbol DZNRM2 > > > > > > > > libpetsc.lib > > > > > > > > Error 7 error LNK2019: unresolved external symbol DZASUM > > > > > > > > referenced > > > > > > > > in > > > > > > > > function "int __cdecl VecNorm_Seq(struct _p_Vec *,enum > > > > > > > > NormType,double > > > > > > > > *)" > > > > > > > > (?VecNorm_Seq@@YAHPEAU_p_Vec@@W4NormType@@PEAN at Z) > > > > > > > > libpetsc.lib > > > > > > > > ... > > > > > > > > > > > > > > > > It look like calling conventions are in disagreement. Any ideas? > > > > > > > > May > > > > > > > > be I > > > > > > > > should specify conventions myself in configuration line? If so, > > > > > > > > how to > > > > > > > > do > > > > > > > > that? > > > > > > > > > > > > > > > > > From jdbst21 at gmail.com Thu Mar 3 11:01:58 2011 From: jdbst21 at gmail.com (Joshua Booth) Date: Thu, 3 Mar 2011 12:01:58 -0500 Subject: [petsc-users] MatGetSubMatrices error In-Reply-To: References: Message-ID: > > > Hello, I am running a job on two processors and a matmpiaij. I want to set a submatrix of this on the one processor (rank==1). So I made Mat *localB; If(rank==1) { IS irow; IS icol; ISCreateGeneral(PETSC_COMM_SELF, end_pos-start_pos+1, order_array, &irow); ISCreateGeneral(PETSC_COMM_SELF, end_pos-start_pos+1, order_array, &icol); MatGetSubMatrices(A, 1, &irow, &icol, MAT_INITIAL_MATRIX, &localB); } else { IS irow; IS icol; ISCreateStride(PETSC_COMM_SELF, 0, 0, 1, &irow); ISCreateStride(PETSC_COMM_SELF, 0, 0, 1, &icol); MatGetSubMatrices(A,0,&irow,&icol,MAT_INITIAL_MATRIX,&localB); } However, I get a segmentation violation from MatGetSubMatrices_MPIAIJ_Local on rank=0 when I track back with gdb. I have already use MatGetSubMatrices on all cores than destroy. Joshua Booth -------------- next part -------------- An HTML attachment was scrubbed... URL: From loic.gouarin at math.u-psud.fr Thu Mar 3 11:06:14 2011 From: loic.gouarin at math.u-psud.fr (gouarin) Date: Thu, 03 Mar 2011 18:06:14 +0100 Subject: [petsc-users] DMComposite and ksp In-Reply-To: <020F96D8-7F8C-45C1-9896-6A786D3F7A36@mcs.anl.gov> References: <4D6F8F10.1040104@math.u-psud.fr> <020F96D8-7F8C-45C1-9896-6A786D3F7A36@mcs.anl.gov> Message-ID: <4D6FCA86.3040303@math.u-psud.fr> Thanks for the reply. My Dirichlet condition was wrong for the multi domain. I fixed it and now I have the same iteration number for my KSP. I'm very sorry. Loic On 03/03/2011 15:19, Barry Smith wrote: > On Mar 3, 2011, at 6:52 AM, gouarin wrote: > >> Hi, >> >> I'm still working on my stokes problem with 2 DM (one for the velocities and an other for the pressure). I do the best preallocation for my matrix. >> After the construction of my matrix, I do a matrix vector operation and I compare the result for one process and 2 processes. If I pay attention to the numbering of my fields between these 2 problems, I have the same result. >> >> But when I do just 2 iterations of my ksp solver, I don't have the same result at the end and of course the residual is not the same. > What preconditioner are you using? Unless you use Jacobi you will always get a different set of iterations. (And even with Jacobi due to different order of operations you will eventually see a different convergence route between 1 and 2 processes. You cannot be concerned about this, the things to be concerned about are: > > 1) Do 1 and 2 processes eventually converge to the same answer (bad news if they do not) > > 2) Does 2 process converge almost as quickly to the answer as one process (bad news if adding more processes slows the convergence greatly so the solution time ends up being longer). > > Barry > > > >> Do you know what I do wrong ? >> >> Thanks, >> Loic >> >> -- >> Loic Gouarin >> Laboratoire de Math?matiques >> Universit? Paris-Sud >> B?timent 425 >> 91405 Orsay Cedex >> France >> Tel: (+33) 1 69 15 60 14 >> Fax: (+33) 1 69 15 67 18 >> -- Loic Gouarin Laboratoire de Math?matiques Universit? Paris-Sud B?timent 425 91405 Orsay Cedex France Tel: (+33) 1 69 15 60 14 Fax: (+33) 1 69 15 67 18 From knepley at gmail.com Thu Mar 3 12:18:07 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 3 Mar 2011 12:18:07 -0600 Subject: [petsc-users] AOCreateBasic is not scaling In-Reply-To: <4D6ECA40.5080409@uiuc.edu> References: <4D6EC3C4.4080407@uiuc.edu> <4D6ECA40.5080409@uiuc.edu> Message-ID: On Wed, Mar 2, 2011 at 4:52 PM, M. Scot Breitenfeld wrote: > I don't number my global degree's of freedom from low-high continuously > per processor as PETSc uses for ordering, but I use the natural ordering > of the application, I then use AOcreateBasic to obtain the mapping > between the PETSc and my ordering. > I would suggest using the LocalToGlobalMapping functions, which are scalable. AO is designed for complete global permutations. Thanks, Matt > CALL VecGetOwnershipRange(b, low, high, ierr) > > icnt = 0 > > DO mi = 1, mctr ! these are the nodes local to processor > mi_global = myglobal(mi) > > irowx = 3*mi_global-2 > irowy = 3*mi_global-1 > irowz = 3*mi_global > > mappings(icnt+1:icnt+3) = (/ & > nrow_global(row_from_dof(1,mi))-1, & > nrow_global(row_from_dof(2,mi))-1, & > nrow_global(row_from_dof(3,mi))-1 & > /) > > petscOrdering(icnt+1:icnt+3) = (/low+icnt,low+icnt+1,low+icnt+2/) > > icnt = icnt + 3 > END DO > > CALL AOCreateBasic(PETSC_COMM_WORLD, icnt, mappings, petscOrdering, > toao, ierr) > > DO mi = mctr+1, myn ! these are the ghost nodes not on this processor > > mi_global = myglobal(mi) > > mappings(icnt+1:icnt+3) = (/ & > nrow_global(row_from_dof(1,mi))-1, & > nrow_global(row_from_dof(2,mi))-1, & > nrow_global(row_from_dof(3,mi))-1 & > /) > > icnt = icnt + 3 > ENDDO > CALL AOApplicationToPetsc(toao, 3*myn, mappings, ierr) > > CALL AODestroy(toao, ierr) > > I then use mapping to input the values into the correct row as wanted > by PETSc > > > On 03/02/2011 04:29 PM, Matthew Knepley wrote: > > On Wed, Mar 2, 2011 at 4:25 PM, M. Scot Breitenfeld > > wrote: > > > > Hi, > > > > First, thanks for the suggestion on using MPISBAIJ for my A matrix, > it > > seems to have cut down on my memory and assembly time. For a 1.5 > > million > > dof problem: > > > > # procs: 2 4 8 16 > > ---------------------------------------------------------------- > > Assembly (sec): 245 124 63 86 > > Solver (sec): 924 578 326 680 > > > > Memory (GB): 2.5 1.4 .877 .565 > > > > The problem I have is the amount of time it's taking in > AOCreateBasic, > > it takes longer then assembling, > > > > # procs: 2 4 8 16 > > --------------------------------------------------------------------- > > AOCreateBasic (sec): .6 347 170 197 > > > > Is there something that I can change or something I can look for that > > might be causing this increase in time as I go from 2 to 4 processors > > (at least it scales from 4 to 8 processors). I read in the archive > > that > > AOCreateBasic is not meant to be scalable so maybe there is nothing I > > can do. > > > > > > Yes, this is non-scalable. What are you using it for? > > > > Matt > > > > > > Thanks, > > Scot > > > > > > > > > > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which > > their experiments lead. > > -- Norbert Wiener > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Mar 3 12:30:28 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 3 Mar 2011 12:30:28 -0600 Subject: [petsc-users] MatGetSubMatrices error In-Reply-To: References: Message-ID: Mail the stack trace to petsc-maint at mcs.anl.gov. Matt On Thu, Mar 3, 2011 at 11:01 AM, Joshua Booth wrote: > >> > Hello, > > I am running a job on two processors and a matmpiaij. I want to set a > submatrix of this on the one processor (rank==1). > > So I made > > Mat *localB; > If(rank==1) > { > IS irow; > IS icol; > ISCreateGeneral(PETSC_COMM_SELF, end_pos-start_pos+1, order_array, > &irow); > ISCreateGeneral(PETSC_COMM_SELF, end_pos-start_pos+1, order_array, > &icol); > MatGetSubMatrices(A, 1, &irow, &icol, MAT_INITIAL_MATRIX, &localB); > } > else > { > IS irow; > IS icol; > ISCreateStride(PETSC_COMM_SELF, 0, 0, 1, &irow); > ISCreateStride(PETSC_COMM_SELF, 0, 0, 1, &icol); > MatGetSubMatrices(A,0,&irow,&icol,MAT_INITIAL_MATRIX,&localB); > > } > However, I get a segmentation violation from MatGetSubMatrices_MPIAIJ_Local > on rank=0 when I track back with gdb. > I have already use MatGetSubMatrices on all cores than destroy. > > Joshua Booth > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From tomjan at jay.au.poznan.pl Fri Mar 4 01:14:33 2011 From: tomjan at jay.au.poznan.pl (Tomasz Jankowski) Date: Fri, 4 Mar 2011 08:14:33 +0100 (CET) Subject: [petsc-users] where is PCFactorGetMatrix in petsc-3.1-p7 release Message-ID: Hello, im trying tu use PCFactorGetMatrix which is described in petsc-3.1-p7/src/ksp/pc/examples/tutorials/ex1.c but i got "error: PCGetFactoredMatrix was not declared in this scope" quick grep through whole petsc-3.1-p7 shows lack of this function. ??? tom ######################################################## # tomjan at jay.au.poznan.pl # # jay.au.poznan.pl/~tomjan/ # ######################################################## From agrayver at gfz-potsdam.de Fri Mar 4 04:27:38 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Fri, 04 Mar 2011 11:27:38 +0100 Subject: [petsc-users] Calling convention from windows VC++ and IFC In-Reply-To: <4D6FE3AD.5020704@gfz-potsdam.de> References: <4D6F80C4.4070200@gfz-potsdam.de> <4D6FA9DA.7070403@gfz-potsdam.de> <4D6FB4D5.6040708@gfz-potsdam.de> <4D6FB8D2.8020202@gfz-potsdam.de> <4D6FE3AD.5020704@gfz-potsdam.de> Message-ID: <4D70BE9A.4090106@gfz-potsdam.de> I did that. Today I reconfigured everything from scratch with the following line: ./config/configure.py --with-cc='win32fe cl' --with-fc='win32fe ifort' --with-cxx='win32fe cl' --with-windows-graphics=0 --download-f-blas-lapack=1 --with-precision=double --with-scalar-type=complex --with-clanguage=cxx --with-mpi-include="/cygdrive/d/Dev/HPC_2008_SDK/Include" --with-mpi-lib=["/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpi.lib","/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib"] --useThreads=0 Note I canceled out win32draw subsystem. I also added one line to this file /petsc-3.1-p7/src/sys/utils/fuser.c: #undef PETSC_HAVE_GET_USER_NAME After configuration and compilation I tried FORTRAN example ex5f (make test) and my linker crashed! Running test examples to verify correct installation C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 MPI process C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 MPI processes --------------Error detected during compile or link!----------------------- See http://www.mcs.anl.gov/petsc/petsc-2/documentation/troubleshooting.html /cygdrive/d/dev/petsc-3.1-p7/bin/win32fe/win32fe ifort -c -MT -Z7 -fpp -I/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/include -I/cygdrive/d/dev/petsc-3.1-p7/include -I/cygdrive/d/Dev/HPC_2008_SDK/Include -I/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/include -I/cygdrive/d/Dev/HPC_2008_SDK/Include -o ex5f.o ex5f.F /cygdrive/d/dev/petsc-3.1-p7/bin/win32fe/win32fe ifort -MT -Z7 -fpp -o ex5f ex5f.o -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib -lpetsc -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib -lflapack -lfblas /cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpi.lib /cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib Gdi32.lib User32.lib Advapi32.lib Kernel32.lib Ws2_32.lib LINK : fatal error LNK1000: Internal error during IMAGE::BuildImage Then I deleted option "-Z7" from linker and... at last it works! BLAS/LAPACK seem good now (don't know what happened:) ). I would suggest that the problem is that my windows is x64 and PETSc supply 32-bit libraries for win32draw subsystem and GetUserNameA function. Probably this issue should be taken into account by build system. Great thanks for patience and help to all and especially to Sr. Satish Balay. You do great job. On 03.03.2011 19:53, Alexander Grayver wrote: > Thanks Satish, > > I've tried that. Still doesn't work. The same error. Probably changes > were not applied properly. > I built PETSc without any -Gz and sent logs to petsc-maint at mcs.anl.gov > Errors now are the same as in the first message of the thread. > > Alexander > > On 03.03.2011 18:01, Satish Balay wrote: >> You can try adding the following to >> /cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/include/petscconf.h >> and rebuild the libraries [witout rerunning configure] and >> see if the fortran examples work. >> >> #ifndef PETSC_STDCALL >> #define PETSC_STDCALL __stdcall >> #endif >> >> Ideally configure should set this if -Gz was specified to CC aswell - >> but changing CFLAGS is tricky on windows, and there is no equivalent >> of CXX_CXXFLAGS - so the alternative would be COPTFLAGS. >> >> However you could try the default build [witout any -Gz] - and send >> all the logs to petsc-maint - and we can try figuring out the issue. >> >> Satish >> >> On Thu, 3 Mar 2011, Alexander Grayver wrote: >> >>> Really! Thanks Satish. Errors apparently are different when I use >>> stdcall >>> convention for compiling instead of default one. But the question is >>> how to >>> combine these interfaces now? >>> >>> On 03.03.2011 16:48, Satish Balay wrote: >>>> Oh - I see that with --CXX_CXXFLAGS=-Gz - you get different errors >>>> - not >>>> blaslapack errors. >>>> >>>>>>>> libpetsc.lib(fuser.o) : error LNK2019: unresolved external symbol >>>>>>>> __imp_GetUserNameA referenced in function "int __cdecl >>>>>>>> PetscGetUserName(char * >>>>>>>> const,unsigned__int64)" (?PetscGetUserName@@YAHQEAD_K at Z) >>>> This is due to compiling C interface with -Gz - so now fortran >>>> interface is unable to link with the c interface. >>>> >>>> Satish >>>> >>>> On Thu, 3 Mar 2011, Satish Balay wrote: >>>> >>>>> Currently I don't have access to ifort to try. >>>>> >>>>> For one - its strange that c/c++ examples work - but not fortran - >>>>> esp >>>>> because all blas calls are from the same c/c++ PETSc code. >>>>> >>>>> One thing you can try: >>>>> >>>>> cd src/ksp/ksp/examples/tutorials >>>>> make ex2 >>>>> make ex2f >>>>> >>>>> [if ex2 works and ex2f fails - edit makefile - and change target >>>>> ex2f: >>>>> from using FLINKER to CLINKER and see if it makes a difference] >>>>> >>>>> Satish >>>>> >>>>> On Thu, 3 Mar 2011, Alexander Grayver wrote: >>>>> >>>>>> Actually, I've added this flag after sending first message to >>>>>> mailing >>>>>> list >>>>>> (sorry for inconveniences) because I though it could help, >>>>>> unfortunately >>>>>> it >>>>>> didn't. >>>>>> >>>>>> The problem probably could be reproduced compiling petsc using >>>>>> --with-clanguage=cxx and VC++ and then trying to run FORTRAN >>>>>> examples. >>>>>> >>>>>> On 03.03.2011 16:26, Satish Balay wrote: >>>>>>> perhaps you can remove '--CXX_CXXFLAGS=-Gz' and retry. >>>>>>> >>>>>>> Satish >>>>>>> >>>>>>> On Thu, 3 Mar 2011, Alexander Grayver wrote: >>>>>>> >>>>>>>> I've sent them, thanks! >>>>>>>> >>>>>>>> I also must note that this is a problem for fortran examples only! >>>>>>>> Because >>>>>>>> when I've done "make test" after building the situation was: >>>>>>>> >>>>>>>> mt at agrayver /cygdrive/d/dev/petsc-3.1-p7 >>>>>>>> $ make PETSC_DIR=/cygdrive/d/dev/petsc-3.1-p7 >>>>>>>> PETSC_ARCH=cygwin-cxx-debug >>>>>>>> test >>>>>>>> Running test examples to verify correct installation >>>>>>>> C/C++ example src/snes/examples/tutorials/ex19 run successfully >>>>>>>> with >>>>>>>> 1 MPI >>>>>>>> process >>>>>>>> C/C++ example src/snes/examples/tutorials/ex19 run successfully >>>>>>>> with >>>>>>>> 2 MPI >>>>>>>> processes >>>>>>>> --------------Error detected during compile or >>>>>>>> link!----------------------- >>>>>>>> See >>>>>>>> http://www.mcs.anl.gov/petsc/petsc-2/documentation/troubleshooting.html >>>>>>>> >>>>>>>> /cygdrive/d/dev/petsc-3.1-p7/bin/win32fe/win32fe ifort -c -MT -Z7 >>>>>>>> -fpp >>>>>>>> -I/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/include >>>>>>>> -I/cygdrive/d/dev/petsc-3.1-p7/include >>>>>>>> -I/cygdrive/d/Dev/HPC_2008_SDK/Include >>>>>>>> -I/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/include >>>>>>>> -I/cygdrive/d/Dev/HPC_2008_SDK/Include -o ex5f.o ex5f.F >>>>>>>> >>>>>>>> /cygdrive/d/dev/petsc-3.1-p7/bin/win32fe/win32fe ifort -MT -Z7 >>>>>>>> -fpp >>>>>>>> -o >>>>>>>> ex5f >>>>>>>> ex5f.o -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib >>>>>>>> -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib -lpetsc >>>>>>>> -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib -lflapack >>>>>>>> -lfblas >>>>>>>> /cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpi.lib >>>>>>>> /cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib Gdi32.lib >>>>>>>> User32.lib >>>>>>>> Advapi32.lib Kernel32.lib Ws2_32.lib >>>>>>>> >>>>>>>> libpetsc.lib(fuser.o) : error LNK2019: unresolved external symbol >>>>>>>> __imp_GetUserNameA referenced in function "int __cdecl >>>>>>>> PetscGetUserName(char * >>>>>>>> const,unsigned__int64)" (?PetscGetUserName@@YAHQEAD_K at Z) >>>>>>>> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external >>>>>>>> symbol >>>>>>>> __imp_ReleaseDC referenced in function "int __cdecl >>>>>>>> PetscDrawSetDoubleBuffer_Win32(struct_p_PetscDraw *)" >>>>>>>> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >>>>>>>> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external >>>>>>>> symbol >>>>>>>> __imp_BitBlt referenced in function "int __cdecl >>>>>>>> PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" >>>>>>>> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >>>>>>>> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external >>>>>>>> symbol >>>>>>>> __imp_ExtFloodFill referenced in function "int __cdecl >>>>>>>> PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" >>>>>>>> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >>>>>>>> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external >>>>>>>> symbol >>>>>>>> __imp_SelectObject referenced in function "int __cdecl >>>>>>>> PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" >>>>>>>> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >>>>>>>> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external >>>>>>>> symbol >>>>>>>> __imp_CreateCompatibleBitmap referenced in function "int __cdecl >>>>>>>> PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" >>>>>>>> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >>>>>>>> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external >>>>>>>> symbol >>>>>>>> __imp_CreateCompatibleDC referenced in function "int __cdecl >>>>>>>> PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" >>>>>>>> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >>>>>>>> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external >>>>>>>> symbol >>>>>>>> __imp_GetDC referenced in function "int __cdecl >>>>>>>> PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" >>>>>>>> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >>>>>>>> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external >>>>>>>> symbol >>>>>>>> __imp_UpdateWindow referenced in function "int __cdecl >>>>>>>> PetscDrawRectangle_Win32(struct _p_PetscDraw >>>>>>>> *,double,double,double,double,int,int,int,int)" >>>>>>>> (?PetscDrawRectangle_Win32@@YAHPEAU_p_PetscDraw@@NNNNHHHH at Z) >>>>>>>> >>>>>>>> On 03.03.2011 14:49, Satish Balay wrote: >>>>>>>>> can you send the corresponding configure.log and make.log to >>>>>>>>> petsc-maint at mcs.anl.gov? >>>>>>>>> >>>>>>>>> Satish >>>>>>>>> >>>>>>>>> On Thu, 3 Mar 2011, Alexander Grayver wrote: >>>>>>>>> >>>>>>>>>> Hello, >>>>>>>>>> >>>>>>>>>> I built PETSc-3.1 using VC++ 2008 and IFC 10.1 with the >>>>>>>>>> following >>>>>>>>>> configuration line: >>>>>>>>>> >>>>>>>>>> ./config/configure.py --with-cc='win32fe cl' --with-fc='win32fe >>>>>>>>>> ifort' >>>>>>>>>> --with-cxx='win32fe cl' --download-f-blas-lapack=1 >>>>>>>>>> --with-precision=double >>>>>>>>>> --with-scalar-type=complex --with-clanguage=cxx >>>>>>>>>> --with-mpi-include="/cygdrive/d/Dev/HPC_2008_SDK/Include" >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> --with-mpi-lib=["/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpi.lib","/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib"] >>>>>>>>>> >>>>>>>>>> --useThreads=0 --with-shared=0 >>>>>>>>>> >>>>>>>>>> Note there is C++ as main language. >>>>>>>>>> >>>>>>>>>> Now, when I'm trying to build examples I always get similar >>>>>>>>>> errors: >>>>>>>>>> >>>>>>>>>> Error 2 error LNK2019: unresolved external symbol ZGESVD >>>>>>>>>> referenced >>>>>>>>>> in >>>>>>>>>> function "int __cdecl >>>>>>>>>> KSPComputeExtremeSingularValues_GMRES(struct >>>>>>>>>> _p_KSP >>>>>>>>>> *,double *,double *)" >>>>>>>>>> (?KSPComputeExtremeSingularValues_GMRES@@YAHPEAU_p_KSP@@PEAN1 at Z) >>>>>>>>>> libpetsc.lib >>>>>>>>>> Error 3 error LNK2019: unresolved external symbol ZGEEV >>>>>>>>>> referenced >>>>>>>>>> in >>>>>>>>>> function "int __cdecl KSPComputeEigenvalues_GMRES(struct _p_KSP >>>>>>>>>> *,int,double >>>>>>>>>> *,double *,int *)" >>>>>>>>>> (?KSPComputeEigenvalues_GMRES@@YAHPEAU_p_KSP@@HPEAN1PEAH at Z) >>>>>>>>>> libpetsc.lib >>>>>>>>>> Error 4 error LNK2001: unresolved external symbol ZGEEV >>>>>>>>>> libpetsc.lib >>>>>>>>>> Error 5 error LNK2019: unresolved external symbol DZNRM2 >>>>>>>>>> referenced >>>>>>>>>> in >>>>>>>>>> function "int __cdecl VecNorm_MPI(struct _p_Vec *,enum >>>>>>>>>> NormType,double >>>>>>>>>> *)" >>>>>>>>>> (?VecNorm_MPI@@YAHPEAU_p_Vec@@W4NormType@@PEAN at Z) >>>>>>>>>> libpetsc.lib >>>>>>>>>> Error 6 error LNK2001: unresolved external symbol DZNRM2 >>>>>>>>>> libpetsc.lib >>>>>>>>>> Error 7 error LNK2019: unresolved external symbol DZASUM >>>>>>>>>> referenced >>>>>>>>>> in >>>>>>>>>> function "int __cdecl VecNorm_Seq(struct _p_Vec *,enum >>>>>>>>>> NormType,double >>>>>>>>>> *)" >>>>>>>>>> (?VecNorm_Seq@@YAHPEAU_p_Vec@@W4NormType@@PEAN at Z) >>>>>>>>>> libpetsc.lib >>>>>>>>>> ... >>>>>>>>>> >>>>>>>>>> It look like calling conventions are in disagreement. Any ideas? >>>>>>>>>> May >>>>>>>>>> be I >>>>>>>>>> should specify conventions myself in configuration line? If so, >>>>>>>>>> how to >>>>>>>>>> do >>>>>>>>>> that? >>>>>>>>>> >>> > From agrayver at gfz-potsdam.de Thu Mar 3 12:53:33 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Thu, 03 Mar 2011 19:53:33 +0100 Subject: [petsc-users] Calling convention from windows VC++ and IFC In-Reply-To: References: <4D6F80C4.4070200@gfz-potsdam.de> <4D6FA9DA.7070403@gfz-potsdam.de> <4D6FB4D5.6040708@gfz-potsdam.de> <4D6FB8D2.8020202@gfz-potsdam.de> Message-ID: <4D6FE3AD.5020704@gfz-potsdam.de> Thanks Satish, I've tried that. Still doesn't work. The same error. Probably changes were not applied properly. I built PETSc without any -Gz and sent logs to petsc-maint at mcs.anl.gov Errors now are the same as in the first message of the thread. Alexander On 03.03.2011 18:01, Satish Balay wrote: > You can try adding the following to > /cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/include/petscconf.h > and rebuild the libraries [witout rerunning configure] and > see if the fortran examples work. > > #ifndef PETSC_STDCALL > #define PETSC_STDCALL __stdcall > #endif > > Ideally configure should set this if -Gz was specified to CC aswell - > but changing CFLAGS is tricky on windows, and there is no equivalent > of CXX_CXXFLAGS - so the alternative would be COPTFLAGS. > > However you could try the default build [witout any -Gz] - and send > all the logs to petsc-maint - and we can try figuring out the issue. > > Satish > > On Thu, 3 Mar 2011, Alexander Grayver wrote: > >> Really! Thanks Satish. Errors apparently are different when I use stdcall >> convention for compiling instead of default one. But the question is how to >> combine these interfaces now? >> >> On 03.03.2011 16:48, Satish Balay wrote: >>> Oh - I see that with --CXX_CXXFLAGS=-Gz - you get different errors - not >>> blaslapack errors. >>> >>>>>>> libpetsc.lib(fuser.o) : error LNK2019: unresolved external symbol >>>>>>> __imp_GetUserNameA referenced in function "int __cdecl >>>>>>> PetscGetUserName(char * >>>>>>> const,unsigned__int64)" (?PetscGetUserName@@YAHQEAD_K at Z) >>> This is due to compiling C interface with -Gz - so now fortran >>> interface is unable to link with the c interface. >>> >>> Satish >>> >>> On Thu, 3 Mar 2011, Satish Balay wrote: >>> >>>> Currently I don't have access to ifort to try. >>>> >>>> For one - its strange that c/c++ examples work - but not fortran - esp >>>> because all blas calls are from the same c/c++ PETSc code. >>>> >>>> One thing you can try: >>>> >>>> cd src/ksp/ksp/examples/tutorials >>>> make ex2 >>>> make ex2f >>>> >>>> [if ex2 works and ex2f fails - edit makefile - and change target ex2f: >>>> from using FLINKER to CLINKER and see if it makes a difference] >>>> >>>> Satish >>>> >>>> On Thu, 3 Mar 2011, Alexander Grayver wrote: >>>> >>>>> Actually, I've added this flag after sending first message to mailing >>>>> list >>>>> (sorry for inconveniences) because I though it could help, unfortunately >>>>> it >>>>> didn't. >>>>> >>>>> The problem probably could be reproduced compiling petsc using >>>>> --with-clanguage=cxx and VC++ and then trying to run FORTRAN examples. >>>>> >>>>> On 03.03.2011 16:26, Satish Balay wrote: >>>>>> perhaps you can remove '--CXX_CXXFLAGS=-Gz' and retry. >>>>>> >>>>>> Satish >>>>>> >>>>>> On Thu, 3 Mar 2011, Alexander Grayver wrote: >>>>>> >>>>>>> I've sent them, thanks! >>>>>>> >>>>>>> I also must note that this is a problem for fortran examples only! >>>>>>> Because >>>>>>> when I've done "make test" after building the situation was: >>>>>>> >>>>>>> mt at agrayver /cygdrive/d/dev/petsc-3.1-p7 >>>>>>> $ make PETSC_DIR=/cygdrive/d/dev/petsc-3.1-p7 >>>>>>> PETSC_ARCH=cygwin-cxx-debug >>>>>>> test >>>>>>> Running test examples to verify correct installation >>>>>>> C/C++ example src/snes/examples/tutorials/ex19 run successfully with >>>>>>> 1 MPI >>>>>>> process >>>>>>> C/C++ example src/snes/examples/tutorials/ex19 run successfully with >>>>>>> 2 MPI >>>>>>> processes >>>>>>> --------------Error detected during compile or >>>>>>> link!----------------------- >>>>>>> See >>>>>>> http://www.mcs.anl.gov/petsc/petsc-2/documentation/troubleshooting.html >>>>>>> /cygdrive/d/dev/petsc-3.1-p7/bin/win32fe/win32fe ifort -c -MT -Z7 >>>>>>> -fpp >>>>>>> -I/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/include >>>>>>> -I/cygdrive/d/dev/petsc-3.1-p7/include >>>>>>> -I/cygdrive/d/Dev/HPC_2008_SDK/Include >>>>>>> -I/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/include >>>>>>> -I/cygdrive/d/Dev/HPC_2008_SDK/Include -o ex5f.o ex5f.F >>>>>>> >>>>>>> /cygdrive/d/dev/petsc-3.1-p7/bin/win32fe/win32fe ifort -MT -Z7 -fpp >>>>>>> -o >>>>>>> ex5f >>>>>>> ex5f.o -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib >>>>>>> -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib -lpetsc >>>>>>> -L/cygdrive/d/dev/petsc-3.1-p7/cygwin-cxx-debug/lib -lflapack >>>>>>> -lfblas >>>>>>> /cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpi.lib >>>>>>> /cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib Gdi32.lib >>>>>>> User32.lib >>>>>>> Advapi32.lib Kernel32.lib Ws2_32.lib >>>>>>> >>>>>>> libpetsc.lib(fuser.o) : error LNK2019: unresolved external symbol >>>>>>> __imp_GetUserNameA referenced in function "int __cdecl >>>>>>> PetscGetUserName(char * >>>>>>> const,unsigned__int64)" (?PetscGetUserName@@YAHQEAD_K at Z) >>>>>>> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external >>>>>>> symbol >>>>>>> __imp_ReleaseDC referenced in function "int __cdecl >>>>>>> PetscDrawSetDoubleBuffer_Win32(struct_p_PetscDraw *)" >>>>>>> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >>>>>>> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external >>>>>>> symbol >>>>>>> __imp_BitBlt referenced in function "int __cdecl >>>>>>> PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" >>>>>>> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >>>>>>> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external >>>>>>> symbol >>>>>>> __imp_ExtFloodFill referenced in function "int __cdecl >>>>>>> PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" >>>>>>> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >>>>>>> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external >>>>>>> symbol >>>>>>> __imp_SelectObject referenced in function "int __cdecl >>>>>>> PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" >>>>>>> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >>>>>>> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external >>>>>>> symbol >>>>>>> __imp_CreateCompatibleBitmap referenced in function "int __cdecl >>>>>>> PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" >>>>>>> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >>>>>>> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external >>>>>>> symbol >>>>>>> __imp_CreateCompatibleDC referenced in function "int __cdecl >>>>>>> PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" >>>>>>> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >>>>>>> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external >>>>>>> symbol >>>>>>> __imp_GetDC referenced in function "int __cdecl >>>>>>> PetscDrawSetDoubleBuffer_Win32(struct _p_PetscDraw *)" >>>>>>> (?PetscDrawSetDoubleBuffer_Win32@@YAHPEAU_p_PetscDraw@@@Z) >>>>>>> libpetsc.lib(win32draw.o) : error LNK2019: unresolved external >>>>>>> symbol >>>>>>> __imp_UpdateWindow referenced in function "int __cdecl >>>>>>> PetscDrawRectangle_Win32(struct _p_PetscDraw >>>>>>> *,double,double,double,double,int,int,int,int)" >>>>>>> (?PetscDrawRectangle_Win32@@YAHPEAU_p_PetscDraw@@NNNNHHHH at Z) >>>>>>> >>>>>>> On 03.03.2011 14:49, Satish Balay wrote: >>>>>>>> can you send the corresponding configure.log and make.log to >>>>>>>> petsc-maint at mcs.anl.gov? >>>>>>>> >>>>>>>> Satish >>>>>>>> >>>>>>>> On Thu, 3 Mar 2011, Alexander Grayver wrote: >>>>>>>> >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> I built PETSc-3.1 using VC++ 2008 and IFC 10.1 with the >>>>>>>>> following >>>>>>>>> configuration line: >>>>>>>>> >>>>>>>>> ./config/configure.py --with-cc='win32fe cl' --with-fc='win32fe >>>>>>>>> ifort' >>>>>>>>> --with-cxx='win32fe cl' --download-f-blas-lapack=1 >>>>>>>>> --with-precision=double >>>>>>>>> --with-scalar-type=complex --with-clanguage=cxx >>>>>>>>> --with-mpi-include="/cygdrive/d/Dev/HPC_2008_SDK/Include" >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> --with-mpi-lib=["/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpi.lib","/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib"] >>>>>>>>> --useThreads=0 --with-shared=0 >>>>>>>>> >>>>>>>>> Note there is C++ as main language. >>>>>>>>> >>>>>>>>> Now, when I'm trying to build examples I always get similar >>>>>>>>> errors: >>>>>>>>> >>>>>>>>> Error 2 error LNK2019: unresolved external symbol ZGESVD >>>>>>>>> referenced >>>>>>>>> in >>>>>>>>> function "int __cdecl >>>>>>>>> KSPComputeExtremeSingularValues_GMRES(struct >>>>>>>>> _p_KSP >>>>>>>>> *,double *,double *)" >>>>>>>>> (?KSPComputeExtremeSingularValues_GMRES@@YAHPEAU_p_KSP@@PEAN1 at Z) >>>>>>>>> libpetsc.lib >>>>>>>>> Error 3 error LNK2019: unresolved external symbol ZGEEV >>>>>>>>> referenced >>>>>>>>> in >>>>>>>>> function "int __cdecl KSPComputeEigenvalues_GMRES(struct _p_KSP >>>>>>>>> *,int,double >>>>>>>>> *,double *,int *)" >>>>>>>>> (?KSPComputeEigenvalues_GMRES@@YAHPEAU_p_KSP@@HPEAN1PEAH at Z) >>>>>>>>> libpetsc.lib >>>>>>>>> Error 4 error LNK2001: unresolved external symbol ZGEEV >>>>>>>>> libpetsc.lib >>>>>>>>> Error 5 error LNK2019: unresolved external symbol DZNRM2 >>>>>>>>> referenced >>>>>>>>> in >>>>>>>>> function "int __cdecl VecNorm_MPI(struct _p_Vec *,enum >>>>>>>>> NormType,double >>>>>>>>> *)" >>>>>>>>> (?VecNorm_MPI@@YAHPEAU_p_Vec@@W4NormType@@PEAN at Z) >>>>>>>>> libpetsc.lib >>>>>>>>> Error 6 error LNK2001: unresolved external symbol DZNRM2 >>>>>>>>> libpetsc.lib >>>>>>>>> Error 7 error LNK2019: unresolved external symbol DZASUM >>>>>>>>> referenced >>>>>>>>> in >>>>>>>>> function "int __cdecl VecNorm_Seq(struct _p_Vec *,enum >>>>>>>>> NormType,double >>>>>>>>> *)" >>>>>>>>> (?VecNorm_Seq@@YAHPEAU_p_Vec@@W4NormType@@PEAN at Z) >>>>>>>>> libpetsc.lib >>>>>>>>> ... >>>>>>>>> >>>>>>>>> It look like calling conventions are in disagreement. Any ideas? >>>>>>>>> May >>>>>>>>> be I >>>>>>>>> should specify conventions myself in configuration line? If so, >>>>>>>>> how to >>>>>>>>> do >>>>>>>>> that? >>>>>>>>> >> -------------- next part -------------- A non-text attachment was scrubbed... Name: conf.zip Type: application/x-zip-compressed Size: 851440 bytes Desc: not available URL: From Chao.Yang at Colorado.EDU Fri Mar 4 03:42:54 2011 From: Chao.Yang at Colorado.EDU (Chao Yang) Date: Fri, 4 Mar 2011 17:42:54 +0800 Subject: [petsc-users] MatSOR for matrix of seqcusp type Message-ID: <27D403D8-741F-4271-A52F-98C8E4DDAC1E@Colorado.EDU> Dear PETSc team, I am developing a code using PETSc on GPU. In the code an SOR preconditioner is needed. Is there an available version of SOR for matrix of seqcusp type in the current petsc-dev? Thanks! Best regards, Chao Yang From bsmith at mcs.anl.gov Fri Mar 4 07:40:50 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 4 Mar 2011 07:40:50 -0600 Subject: [petsc-users] where is PCFactorGetMatrix in petsc-3.1-p7 release In-Reply-To: References: Message-ID: <6A9D1CA4-ABA9-4C6F-91A9-7D143F6549F8@mcs.anl.gov> Tom, See http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manual.pdf sections 13.8 and 13.9 for easy ways to find functions etc in PETSc. On Mar 4, 2011, at 1:14 AM, Tomasz Jankowski wrote: > Hello, > > im trying tu use PCFactorGetMatrix which is described > in petsc-3.1-p7/src/ksp/pc/examples/tutorials/ex1.c > but i got "error: PCGetFactoredMatrix was not declared in this scope" It is PCFactorGetMatrix() not PCGetFactoredMatrix() Barry > quick grep through whole petsc-3.1-p7 shows lack of this function. > > ??? > > tom > > ######################################################## > # tomjan at jay.au.poznan.pl # > # jay.au.poznan.pl/~tomjan/ # > ######################################################## From bsmith at mcs.anl.gov Fri Mar 4 07:43:36 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 4 Mar 2011 07:43:36 -0600 Subject: [petsc-users] MatSOR for matrix of seqcusp type In-Reply-To: <27D403D8-741F-4271-A52F-98C8E4DDAC1E@Colorado.EDU> References: <27D403D8-741F-4271-A52F-98C8E4DDAC1E@Colorado.EDU> Message-ID: On Mar 4, 2011, at 3:42 AM, Chao Yang wrote: > Dear PETSc team, > > I am developing a code using PETSc on GPU. In the code an SOR preconditioner is needed. > Is there an available version of SOR for matrix of seqcusp type in the current petsc-dev? No. The problem is that SOR is a sequential algorithm, there is not much parallelism in it and GPUs require/use a great deal of parallelism to get good performance. Do you have alternatives? How about diagonal scaled CG or Bi-CG-stab? Barry > Thanks! > > Best regards, > > Chao Yang From brtnfld at uiuc.edu Fri Mar 4 15:14:38 2011 From: brtnfld at uiuc.edu (M. Scot Breitenfeld) Date: Fri, 04 Mar 2011 15:14:38 -0600 Subject: [petsc-users] AOCreateBasic is not scaling In-Reply-To: References: <4D6EC3C4.4080407@uiuc.edu> <4D6ECA40.5080409@uiuc.edu> Message-ID: <4D71563E.3060407@uiuc.edu> On 03/03/2011 12:18 PM, Matthew Knepley wrote: > On Wed, Mar 2, 2011 at 4:52 PM, M. Scot Breitenfeld > wrote: > > I don't number my global degree's of freedom from low-high > continuously > per processor as PETSc uses for ordering, but I use the natural > ordering > of the application, I then use AOcreateBasic to obtain the mapping > between the PETSc and my ordering. > > > I would suggest using the LocalToGlobalMapping functions, which are > scalable. > AO is designed for complete global permutations. I don't understand how I can avoid not using AO if my global dof per processor are not arranged in PETSc global ordering (continuous row ordering, i.e. proc1_ 0-n, proc2_n+1:m, proc3_m+1:p, etc...). In the LocalToGlobalMapping routines, doesn't the "GlobalMapping" part mean PETSc ordering and not my application's ordering. I thought I understood the difference between AO and LocalToGlobalMapping but now I'm confused. I tried to use the LocalToGlobalMapping routines and the solution values are correct but the ordering corresponds the global node ordering, not how I partitioned the mesh. In other words, the values are returned in the same ordering as for a serial run, which makes sense since this is how PETSc orders the rows. If I had used PETSc ordering then this would be fine. Is the moral of the story, if I want to get scalability I need to rearrange my global dof in PETSc ordering so that I can use LocalToGlobalMapping? > > Thanks, > > Matt > > > CALL VecGetOwnershipRange(b, low, high, ierr) > > icnt = 0 > > DO mi = 1, mctr ! these are the nodes local to processor > mi_global = myglobal(mi) > > irowx = 3*mi_global-2 > irowy = 3*mi_global-1 > irowz = 3*mi_global > > mappings(icnt+1:icnt+3) = (/ & > nrow_global(row_from_dof(1,mi))-1, & > nrow_global(row_from_dof(2,mi))-1, & > nrow_global(row_from_dof(3,mi))-1 & > /) > > petscOrdering(icnt+1:icnt+3) = > (/low+icnt,low+icnt+1,low+icnt+2/) > > icnt = icnt + 3 > END DO > > CALL AOCreateBasic(PETSC_COMM_WORLD, icnt, mappings, petscOrdering, > toao, ierr) > > DO mi = mctr+1, myn ! these are the ghost nodes not on this processor > > mi_global = myglobal(mi) > > mappings(icnt+1:icnt+3) = (/ & > nrow_global(row_from_dof(1,mi))-1, & > nrow_global(row_from_dof(2,mi))-1, & > nrow_global(row_from_dof(3,mi))-1 & > /) > > icnt = icnt + 3 > ENDDO > CALL AOApplicationToPetsc(toao, 3*myn, mappings, ierr) > > CALL AODestroy(toao, ierr) > > I then use mapping to input the values into the correct row as wanted > by PETSc > > > On 03/02/2011 04:29 PM, Matthew Knepley wrote: > > On Wed, Mar 2, 2011 at 4:25 PM, M. Scot Breitenfeld > > > >> wrote: > > > > Hi, > > > > First, thanks for the suggestion on using MPISBAIJ for my A > matrix, it > > seems to have cut down on my memory and assembly time. For a 1.5 > > million > > dof problem: > > > > # procs: 2 4 8 16 > > ---------------------------------------------------------------- > > Assembly (sec): 245 124 63 86 > > Solver (sec): 924 578 326 680 > > > > Memory (GB): 2.5 1.4 .877 .565 > > > > The problem I have is the amount of time it's taking in > AOCreateBasic, > > it takes longer then assembling, > > > > # procs: 2 4 8 > 16 > > > --------------------------------------------------------------------- > > AOCreateBasic (sec): .6 347 170 197 > > > > Is there something that I can change or something I can look > for that > > might be causing this increase in time as I go from 2 to 4 > processors > > (at least it scales from 4 to 8 processors). I read in the > archive > > that > > AOCreateBasic is not meant to be scalable so maybe there is > nothing I > > can do. > > > > > > Yes, this is non-scalable. What are you using it for? > > > > Matt > > > > > > Thanks, > > Scot > > > > > > > > > > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which > > their experiments lead. > > -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener From knepley at gmail.com Fri Mar 4 15:20:06 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 4 Mar 2011 15:20:06 -0600 Subject: [petsc-users] AOCreateBasic is not scaling In-Reply-To: <4D71563E.3060407@uiuc.edu> References: <4D6EC3C4.4080407@uiuc.edu> <4D6ECA40.5080409@uiuc.edu> <4D71563E.3060407@uiuc.edu> Message-ID: On Fri, Mar 4, 2011 at 3:14 PM, M. Scot Breitenfeld wrote: > On 03/03/2011 12:18 PM, Matthew Knepley wrote: > > On Wed, Mar 2, 2011 at 4:52 PM, M. Scot Breitenfeld > > wrote: > > > > I don't number my global degree's of freedom from low-high > > continuously > > per processor as PETSc uses for ordering, but I use the natural > > ordering > > of the application, I then use AOcreateBasic to obtain the mapping > > between the PETSc and my ordering. > > > > > > I would suggest using the LocalToGlobalMapping functions, which are > > scalable. > > AO is designed for complete global permutations. > I don't understand how I can avoid not using AO if my global dof per > processor are not arranged in PETSc global ordering (continuous row > ordering, i.e. proc1_ 0-n, proc2_n+1:m, proc3_m+1:p, etc...). In the > LocalToGlobalMapping routines, doesn't the "GlobalMapping" part mean > PETSc ordering and not my application's ordering. > > I thought I understood the difference between AO and > LocalToGlobalMapping but now I'm confused. I tried to use the > LocalToGlobalMapping routines and the solution values are correct but > the ordering corresponds the global node ordering, not how I partitioned > the mesh. In other words, the values are returned in the same ordering > as for a serial run, which makes sense since this is how PETSc orders > the rows. If I had used PETSc ordering then this would be fine. > > Is the moral of the story, if I want to get scalability I need to > rearrange my global dof in PETSc ordering so that I can use > LocalToGlobalMapping? I am having a really hard time understanding what you want? If you want Natural Ordering or any other crazy ordering on input/output go ahead and use AO there because the non-scalability is amortized over the run. The PETSc ordering should be used for all globally assembled structures in the solve because its efficient, and there is no reason for the user to care about these structures. For integration/assembly, use local orderings since that is all you need for a PDE. If you have an exotic equation that really does need global information, I would like to hear about it, but it would most likely be non-scalable on its own. Matt > > > > Thanks, > > > > Matt > > > > > > CALL VecGetOwnershipRange(b, low, high, ierr) > > > > icnt = 0 > > > > DO mi = 1, mctr ! these are the nodes local to processor > > mi_global = myglobal(mi) > > > > irowx = 3*mi_global-2 > > irowy = 3*mi_global-1 > > irowz = 3*mi_global > > > > mappings(icnt+1:icnt+3) = (/ & > > nrow_global(row_from_dof(1,mi))-1, & > > nrow_global(row_from_dof(2,mi))-1, & > > nrow_global(row_from_dof(3,mi))-1 & > > /) > > > > petscOrdering(icnt+1:icnt+3) = > > (/low+icnt,low+icnt+1,low+icnt+2/) > > > > icnt = icnt + 3 > > END DO > > > > CALL AOCreateBasic(PETSC_COMM_WORLD, icnt, mappings, petscOrdering, > > toao, ierr) > > > > DO mi = mctr+1, myn ! these are the ghost nodes not on this > processor > > > > mi_global = myglobal(mi) > > > > mappings(icnt+1:icnt+3) = (/ & > > nrow_global(row_from_dof(1,mi))-1, & > > nrow_global(row_from_dof(2,mi))-1, & > > nrow_global(row_from_dof(3,mi))-1 & > > /) > > > > icnt = icnt + 3 > > ENDDO > > CALL AOApplicationToPetsc(toao, 3*myn, mappings, ierr) > > > > CALL AODestroy(toao, ierr) > > > > I then use mapping to input the values into the correct row as > wanted > > by PETSc > > > > > > On 03/02/2011 04:29 PM, Matthew Knepley wrote: > > > On Wed, Mar 2, 2011 at 4:25 PM, M. Scot Breitenfeld > > > > > >> wrote: > > > > > > Hi, > > > > > > First, thanks for the suggestion on using MPISBAIJ for my A > > matrix, it > > > seems to have cut down on my memory and assembly time. For a > 1.5 > > > million > > > dof problem: > > > > > > # procs: 2 4 8 16 > > > > ---------------------------------------------------------------- > > > Assembly (sec): 245 124 63 86 > > > Solver (sec): 924 578 326 680 > > > > > > Memory (GB): 2.5 1.4 .877 .565 > > > > > > The problem I have is the amount of time it's taking in > > AOCreateBasic, > > > it takes longer then assembling, > > > > > > # procs: 2 4 8 > > 16 > > > > > --------------------------------------------------------------------- > > > AOCreateBasic (sec): .6 347 170 197 > > > > > > Is there something that I can change or something I can look > > for that > > > might be causing this increase in time as I go from 2 to 4 > > processors > > > (at least it scales from 4 to 8 processors). I read in the > > archive > > > that > > > AOCreateBasic is not meant to be scalable so maybe there is > > nothing I > > > can do. > > > > > > > > > Yes, this is non-scalable. What are you using it for? > > > > > > Matt > > > > > > > > > Thanks, > > > Scot > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > What most experimenters take for granted before they begin their > > > experiments is infinitely more interesting than any results to > which > > > their experiments lead. > > > -- Norbert Wiener > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which > > their experiments lead. > > -- Norbert Wiener > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From ning.an at ghiocel-tech.com Fri Mar 4 16:40:16 2011 From: ning.an at ghiocel-tech.com (ning.an at ghiocel-tech.com) Date: 4 Mar 2011 22:40:16 -0000 Subject: [petsc-users] how to write the symmetric matrix in PETSc binary matrix format directly Message-ID: <20110304224016.25954.qmail@ns25.webmasters.com> Hello There, I am new to PETSc and programming with FORTRAN. For the huge sparse matrix (200,000x200,000 symmetric), it is so slow to set the matrix in PETSc by reading them in matrix market format, which is far beyond our patience. Therefore, I guess that it would be much fast to load the huge matrix, if the matrix is in PETSc binary matrix format directly. Please help on how to write the symmetric matrix in PETSc binary matrix format directly? Both the guidance and examples are welcome. Ning -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Mar 4 16:44:43 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 4 Mar 2011 16:44:43 -0600 Subject: [petsc-users] how to write the symmetric matrix in PETSc binary matrix format directly In-Reply-To: <20110304224016.25954.qmail@ns25.webmasters.com> References: <20110304224016.25954.qmail@ns25.webmasters.com> Message-ID: On Fri, Mar 4, 2011 at 4:40 PM, wrote: > Hello There, > > I am new to PETSc and programming with FORTRAN. For the huge sparse > matrix (200,000x200,000 symmetric), it is so slow to set the matrix in > PETSc by reading them in matrix market format, which is far beyond our > patience. Therefore, I guess that it would be much fast to load the huge > matrix, if the matrix is in PETSc binary matrix format > directly. > It is probably slow because you have not preallocated the matrix correctly: http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#efficient-assembly > Please help on how to write the symmetric matrix in PETSc binary matrix > format directly? > You just save it using MatView(). For example: http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/src/ksp/pc/examples/tutorials/ex2.c.html Matt > Both the guidance and examples are welcome. > > Ning -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Mar 4 16:59:05 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 4 Mar 2011 16:59:05 -0600 Subject: [petsc-users] how to write the symmetric matrix in PETSc binary matrix format directly In-Reply-To: <20110304224016.25954.qmail@ns25.webmasters.com> References: <20110304224016.25954.qmail@ns25.webmasters.com> Message-ID: <1FEA952C-0DDD-4967-A61A-44402597A980@mcs.anl.gov> The exact binary file layout for matrices is given in the manual page for MatLoad(). Barry On Mar 4, 2011, at 4:40 PM, ning.an at ghiocel-tech.com wrote: > Hello There, > > I am new to PETSc and programming with FORTRAN. For the huge sparse matrix (200,000x200,000 symmetric), it is so slow to set the matrix in PETSc by reading them in matrix market format, which is far beyond our patience. Therefore, I guess that it would be much fast to load the huge matrix, if the matrix is in PETSc binary matrix format > directly. > > Please help on how to write the symmetric matrix in PETSc binary matrix format directly? > Both the guidance and examples are welcome. > > Ning From ning.an at ghiocel-tech.com Fri Mar 4 17:24:22 2011 From: ning.an at ghiocel-tech.com (ning.an at ghiocel-tech.com) Date: 4 Mar 2011 23:24:22 -0000 Subject: [petsc-users] how to write the symmetric matrix in PETSc binary Message-ID: <20110304232422.29903.qmail@ns25.webmasters.com> Dear Knepley, Thanks for your quick response. Yes, I have not pre-allocated sufficient space for the matrix, since I don't know the max. bandwidth. I got this matrix from other person. My task is to test PETSc solver for huge matrix size. I'm going to ask the person to give me that number. base on your experience, how fast could it be if I use "call MatCreateSeqAIJ(comm,n,n,rowmax,PETSC_NULL,&mat)" ? Since I don't have the PETSc matrix yet, I couldn't use MatView() to write it in the binary format. Do you have another suggestion to generate matrix file in PETSc binary format? I'm not familiar with C. I couldn't dig it out from the source code. It is so appreciated If you get me an example. Have a good weekend. ning ----- Original Message ----- From: knepley at gmail.comTo: petsc-users at mcs.anl.gov Sent: 3/4/11 5:44 PM Subject: Re: [petsc-users] how to write the symmetric matrix in PETSc binary On Fri, Mar 4, 2011 at 4:40 PM, wrote: Hello There, I am new to PETSc and programming with FORTRAN. For the huge sparse matrix (200,000x200,000 symmetric), it is so slow to set the matrix in PETSc by reading them in matrix market format, which is far beyond our patience. Therefore, I guess that it would be much fast to load the huge matrix, if the matrix is in PETSc binary matrix format directly. It is probably slow because you have not preallocated the matrix correctly: http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#efficient-assembly Please help on how to write the symmetric matrix in PETSc binary matrix format directly? You just save it using MatView(). For example: http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/src/ksp/pc/examples/tutorials/ex2.c.html Matt Both the guidance and examples are welcome. Ning-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Mar 4 17:34:20 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 4 Mar 2011 17:34:20 -0600 Subject: [petsc-users] how to write the symmetric matrix in PETSc binary In-Reply-To: <20110304232422.29903.qmail@ns25.webmasters.com> References: <20110304232422.29903.qmail@ns25.webmasters.com> Message-ID: On Fri, Mar 4, 2011 at 5:24 PM, wrote: > Dear Knepley, > > Thanks for your quick response. > > Yes, I have not pre-allocated sufficient space for the matrix, > since I don't know the max. bandwidth. I got this matrix from other person. > My task is to test PETSc solver for huge matrix size. I'm going to ask the > person to give me that number. base on your experience, how fast could it be > if I use "call MatCreateSeqAIJ(comm,n,n,rowmax,PETSC_NULL,&mat)" ? That will be fast as long as no row is greater than rowmax long. However, why not just count first? This is easy and fast. > > Since I don't have the PETSc matrix yet, I couldn't use > MatView() to write it in the binary format. Do you have another suggestion > to generate matrix file in PETSc binary format? I'm not familiar with C. I > couldn't dig it out from the source code. It is so appreciated If you get me > an example. > The C is the only reference. Matt > Have a good weekend. > > ning > > ----- Original Message ----- > *From:* knepley at gmail.com > *To:* petsc-users at mcs.anl.gov > *Sent:* 3/4/11 5:44 PM > *Subject:* Re: [petsc-users] how to write the symmetric matrix in PETSc > binary > > On Fri, Mar 4, 2011 at 4:40 PM, wrote: > >> Hello There, >> >> I am new to PETSc and programming with FORTRAN. For the huge sparse >> matrix (200,000x200,000 symmetric), it is so slow to set the matrix in >> PETSc by reading them in matrix market format, which is far beyond our >> patience. Therefore, I guess that it would be much fast to load the huge >> matrix, if the matrix is in PETSc binary matrix format >> directly. >> > > It is probably slow because you have not preallocated the matrix > correctly: > http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#efficient-assembly > > >> Please help on how to write the symmetric matrix in PETSc binary matrix >> format directly? >> > > You just save it using MatView(). For example: > > > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/src/ksp/pc/examples/tutorials/ex2.c.html > > Matt > > >> Both the guidance and examples are welcome. >> >> Ning > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Mar 4 18:49:37 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 4 Mar 2011 18:49:37 -0600 (CST) Subject: [petsc-users] how to write the symmetric matrix in PETSc binary In-Reply-To: References: <20110304232422.29903.qmail@ns25.webmasters.com> Message-ID: check http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#sparse-matrix-ascii-format satish On Fri, 4 Mar 2011, Matthew Knepley wrote: > On Fri, Mar 4, 2011 at 5:24 PM, wrote: > > > Dear Knepley, > > > > Thanks for your quick response. > > > > Yes, I have not pre-allocated sufficient space for the matrix, > > since I don't know the max. bandwidth. I got this matrix from other person. > > My task is to test PETSc solver for huge matrix size. I'm going to ask the > > person to give me that number. base on your experience, how fast could it be > > if I use "call MatCreateSeqAIJ(comm,n,n,rowmax,PETSC_NULL,&mat)" ? > > > That will be fast as long as no row is greater than rowmax long. However, > why not just count first? This is easy and fast. > > > > > > Since I don't have the PETSc matrix yet, I couldn't use > > MatView() to write it in the binary format. Do you have another suggestion > > to generate matrix file in PETSc binary format? I'm not familiar with C. I > > couldn't dig it out from the source code. It is so appreciated If you get me > > an example. > > > > The C is the only reference. > > Matt > > > > Have a good weekend. > > > > ning > > > > ----- Original Message ----- > > *From:* knepley at gmail.com > > *To:* petsc-users at mcs.anl.gov > > *Sent:* 3/4/11 5:44 PM > > *Subject:* Re: [petsc-users] how to write the symmetric matrix in PETSc > > binary > > > > On Fri, Mar 4, 2011 at 4:40 PM, wrote: > > > >> Hello There, > >> > >> I am new to PETSc and programming with FORTRAN. For the huge sparse > >> matrix (200,000x200,000 symmetric), it is so slow to set the matrix in > >> PETSc by reading them in matrix market format, which is far beyond our > >> patience. Therefore, I guess that it would be much fast to load the huge > >> matrix, if the matrix is in PETSc binary matrix format > >> directly. > >> > > > > It is probably slow because you have not preallocated the matrix > > correctly: > > http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#efficient-assembly > > > > > >> Please help on how to write the symmetric matrix in PETSc binary matrix > >> format directly? > >> > > > > You just save it using MatView(). For example: > > > > > > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/src/ksp/pc/examples/tutorials/ex2.c.html > > > > Matt > > > > > >> Both the guidance and examples are welcome. > >> > >> Ning > > > > -- > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which their > > experiments lead. > > -- Norbert Wiener > > > > > > From bsmith at mcs.anl.gov Fri Mar 4 20:02:42 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 4 Mar 2011 20:02:42 -0600 Subject: [petsc-users] how to write the symmetric matrix in PETSc binary In-Reply-To: <20110304232422.29903.qmail@ns25.webmasters.com> References: <20110304232422.29903.qmail@ns25.webmasters.com> Message-ID: <6FE30426-3119-48D5-AAD3-47347789F699@mcs.anl.gov> Did you not get this mail? Subject: Re: [petsc-users] how to write the symmetric matrix in PETSc binary matrix format directly From: Barry Smith Date: March 4, 2011 4:59:05 PM CST To: PETSc users list The exact binary file layout for matrices is given in the manual page for MatLoad(). Barry Is the petsc mail irretrievably broken and and we should switch to pigeons? On Mar 4, 2011, at 4:40 PM, ning.an at ghiocel-tech.com wrote: > Hello There, > > I am new to PETSc and programming with FORTRAN. For the huge sparse matrix (200,000x200,000 symmetric), it is so slow to set the matrix in PETSc by reading them in matrix market format, which is far beyond our patience. Therefore, I guess that it would be much fast to load the huge matrix, if the matrix is in PETSc binary matrix format > directly. > > Please help on how to write the symmetric matrix in PETSc binary matrix format directly? > Both the guidance and examples are welcome. > > Ning On Mar 4, 2011, at 5:24 PM, ning.an at ghiocel-tech.com wrote: > Dear Knepley, > > Thanks for your quick response. > > Yes, I have not pre-allocated sufficient space for the matrix, since I don't know the max. bandwidth. I got this matrix from other person. My task is to test PETSc solver for huge matrix size. I'm going to ask the person to give me that number. base on your experience, how fast could it be if I use "call MatCreateSeqAIJ(comm,n,n,rowmax,PETSC_NULL,&mat)" ? > > Since I don't have the PETSc matrix yet, I couldn't use MatView() to write it in the binary format. Do you have another suggestion to generate matrix file in PETSc binary format? I'm not familiar with C. I couldn't dig it out from the source code. It is so appreciated If you get me an example. > > Have a good weekend. > > ning > > ----- Original Message ----- > From: knepley at gmail.com > To: petsc-users at mcs.anl.gov > Sent: 3/4/11 5:44 PM > Subject: Re: [petsc-users] how to write the symmetric matrix in PETSc binary > > On Fri, Mar 4, 2011 at 4:40 PM, wrote: > Hello There, > > I am new to PETSc and programming with FORTRAN. For the huge sparse matrix (200,000x200,000 symmetric), it is so slow to set the matrix in PETSc by reading them in matrix market format, which is far beyond our patience. Therefore, I guess that it would be much fast to load the huge matrix, if the matrix is in PETSc binary matrix format > directly. > > It is probably slow because you have not preallocated the matrix correctly: http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#efficient-assembly > > Please help on how to write the symmetric matrix in PETSc binary matrix format directly? > > You just save it using MatView(). For example: > > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/src/ksp/pc/examples/tutorials/ex2.c.html > > Matt > > Both the guidance and examples are welcome. > > Ning > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From Chao.Yang at Colorado.EDU Fri Mar 4 20:56:07 2011 From: Chao.Yang at Colorado.EDU (Chao Yang) Date: Sat, 5 Mar 2011 10:56:07 +0800 Subject: [petsc-users] MatSOR for matrix of seqcusp type In-Reply-To: References: <27D403D8-741F-4271-A52F-98C8E4DDAC1E@Colorado.EDU> Message-ID: Hi, Barry, So it's just like MatSolve, too sequential. Thanks for the hint! Chao On Mar 4, 2011, at 9:43 PM, Barry Smith wrote: > > On Mar 4, 2011, at 3:42 AM, Chao Yang wrote: > >> Dear PETSc team, >> >> I am developing a code using PETSc on GPU. In the code an SOR preconditioner is needed. >> Is there an available version of SOR for matrix of seqcusp type in the current petsc-dev? > > No. The problem is that SOR is a sequential algorithm, there is not much parallelism in it and GPUs require/use a great deal of parallelism to get good performance. > Do you have alternatives? How about diagonal scaled CG or Bi-CG-stab? > > > Barry > > > >> Thanks! >> >> Best regards, >> >> Chao Yang > From bsmith at mcs.anl.gov Fri Mar 4 21:03:02 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 4 Mar 2011 21:03:02 -0600 Subject: [petsc-users] MatSOR for matrix of seqcusp type In-Reply-To: References: <27D403D8-741F-4271-A52F-98C8E4DDAC1E@Colorado.EDU> Message-ID: On Mar 4, 2011, at 8:56 PM, Chao Yang wrote: > Hi, Barry, > > So it's just like MatSolve, too sequential. Yup. BTW: we hope to have a MatSolve() for SeqAIJ fairly soon. It won't be great but hopefully a bit better than using the CPU. Barry > Thanks for the hint! > > Chao > > On Mar 4, 2011, at 9:43 PM, Barry Smith wrote: > >> >> On Mar 4, 2011, at 3:42 AM, Chao Yang wrote: >> >>> Dear PETSc team, >>> >>> I am developing a code using PETSc on GPU. In the code an SOR preconditioner is needed. >>> Is there an available version of SOR for matrix of seqcusp type in the current petsc-dev? >> >> No. The problem is that SOR is a sequential algorithm, there is not much parallelism in it and GPUs require/use a great deal of parallelism to get good performance. >> Do you have alternatives? How about diagonal scaled CG or Bi-CG-stab? >> >> >> Barry >> >> >> >>> Thanks! >>> >>> Best regards, >>> >>> Chao Yang >> > From Chao.Yang at Colorado.EDU Sat Mar 5 01:30:45 2011 From: Chao.Yang at Colorado.EDU (Chao Yang) Date: Sat, 5 Mar 2011 15:30:45 +0800 Subject: [petsc-users] MatSOR for matrix of seqcusp type In-Reply-To: References: <27D403D8-741F-4271-A52F-98C8E4DDAC1E@Colorado.EDU> Message-ID: Good to know. Thanks a lot. Have a good weekend! Chao On Mar 5, 2011, at 11:03 AM, Barry Smith wrote: > > On Mar 4, 2011, at 8:56 PM, Chao Yang wrote: > >> Hi, Barry, >> >> So it's just like MatSolve, too sequential. > > Yup. > > BTW: we hope to have a MatSolve() for SeqAIJ fairly soon. It won't be great but hopefully a bit better than using the CPU. > > Barry > >> Thanks for the hint! >> >> Chao >> >> On Mar 4, 2011, at 9:43 PM, Barry Smith wrote: >> >>> >>> On Mar 4, 2011, at 3:42 AM, Chao Yang wrote: >>> >>>> Dear PETSc team, >>>> >>>> I am developing a code using PETSc on GPU. In the code an SOR preconditioner is needed. >>>> Is there an available version of SOR for matrix of seqcusp type in the current petsc-dev? >>> >>> No. The problem is that SOR is a sequential algorithm, there is not much parallelism in it and GPUs require/use a great deal of parallelism to get good performance. >>> Do you have alternatives? How about diagonal scaled CG or Bi-CG-stab? >>> >>> >>> Barry >>> >>> >>> >>>> Thanks! >>>> >>>> Best regards, >>>> >>>> Chao Yang >>> >> > From agrayver at gfz-potsdam.de Sun Mar 6 09:39:49 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Sun, 06 Mar 2011 16:39:49 +0100 Subject: [petsc-users] loosing preallocation information Message-ID: <4D73AAC5.3010407@gfz-potsdam.de> Hello, I'm newbie in PETSc. I use PETSc from FORTRAN under Windows, my build configuration is: Configure options: --with-cc="win32fe cl" --with-fc="win32fe ifort" --with-cxx="win32fe cl" --with-windows-graphics=0 --download-f-blas-lapack=1 --with-precision=double --with-scalar-type=complex --with-clanguage=cxx --with-mpi-include=/cygdrive/d/Dev/HPC_2008_SDK/Include --with-mpi-lib="[/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpi.lib,/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib]" --with-shared=0 --with-debugging=0 --with-fortran-kernels=1 --useThreads=0 I wrote this code: call MatCreateSeqAIJ(PETSC_COMM_SELF, n, m, 2, PETSC_NULL, Gmat, ierr) CHKERRQ(ierr) call cpu_time(timetest(1)) do irow=1,UBOUND(G, dim=1) call MatSetValue(Gmat, ir(i)-1, ic(i)-1, G(irow), INSERT_VALUES, ierr) enddo call cpu_time(timetest(2)) write(*,*) 'Time of setting matrix G was ', timetest(2) - timetest(1), ' seconds' call MatAssemblyBegin(Gmat,MAT_FINAL_ASSEMBLY,ierr) call MatAssemblyEnd(Gmat,MAT_FINAL_ASSEMBLY,ierr) It gives me that for UBOUND(G, dim=1) = 195551: Time of setting matrix G was 620.525177700000 seconds I read user manual and FAQ and I know that I use inefficient way of setting matrix values, but still it seems very suspicios. I run code with -info options and it gave: [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 98304 X 32768; storage space: 1279009 unneeded,195551 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 98304 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 2 [0] Mat_CheckInode(): Found 98304 nodes out of 98304 rows. Not using Inode routines [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374781 [0] PetscCommDuplicate(): returning tag 2147483646 Why there is so much mallocs? I thought I provided information for preallocation by setting 2 nonzeros for each row in MatCreateSeqAIJ. Isn't that enough? And why there is so much "unneeded" elements? Thanks in advance. Regards, Alexander From jed at 59A2.org Sun Mar 6 09:49:22 2011 From: jed at 59A2.org (Jed Brown) Date: Sun, 6 Mar 2011 07:49:22 -0800 Subject: [petsc-users] loosing preallocation information In-Reply-To: <4D73AAC5.3010407@gfz-potsdam.de> References: <4D73AAC5.3010407@gfz-potsdam.de> Message-ID: On Sun, Mar 6, 2011 at 07:39, Alexander Grayver wrote: > Hello, > > I'm newbie in PETSc. > > I use PETSc from FORTRAN under Windows, my build configuration is: > Configure options: --with-cc="win32fe cl" --with-fc="win32fe ifort" > --with-cxx="win32fe cl" --with-windows-graphics=0 --download-f-blas-lapack=1 > --with-precision=double --with-scalar-type=complex --with-clanguage=cxx > --with-mpi-include=/cygdrive/d/Dev/HPC_2008_SDK/Include > --with-mpi-lib="[/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpi.lib,/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib]" > --with-shared=0 --with-debugging=0 --with-fortran-kernels=1 --useThreads=0 > > I wrote this code: > > call MatCreateSeqAIJ(PETSC_COMM_SELF, n, m, 2, PETSC_NULL, > Gmat, ierr) > CHKERRQ(ierr) > > call cpu_time(timetest(1)) > > do irow=1,UBOUND(G, dim=1) > call MatSetValue(Gmat, ir(i)-1, ic(i)-1, G(irow),e > INSERT_VALUES, ierr) > enddo > > call cpu_time(timetest(2)) > write(*,*) 'Time of setting matrix G was ', timetest(2) - > timetest(1), ' seconds' > > call MatAssemblyBegin(Gmat,MAT_FINAL_ASSEMBLY,ierr) > call MatAssemblyEnd(Gmat,MAT_FINAL_ASSEMBLY,ierr) > Are you sure this is how your code looks, in particular that there is no MatAssembly before setting the values. Could you send the whole file. If you are familiar with debuggers, you could call MatSetOption(Gmat,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE,ierr) and then run with -start_in_debugger to find the first insertion that was not preallocated. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jack.poulson at gmail.com Sun Mar 6 22:36:06 2011 From: jack.poulson at gmail.com (Jack Poulson) Date: Sun, 6 Mar 2011 22:36:06 -0600 Subject: [petsc-users] In-place MUMPS call? Message-ID: The documentation seems to suggest that I can choose a sparse-direct solver and factor a MATMPISBAIJ matrix A out-of-place like this: Mat F; MatGetFactor( A, MAT_SOLVER_MUMPS, MAT_FACTOR_CHOLESKY, &F ); MatFactorInfo cholInfo; cholInfo.fill = 3.0; // arbitrary choice cholInfo.dtcol = 0.5; // arbitrary choice IS perm; ISCreateStride( comm, size, 0, 1, &perm ); MatCholeskyFactorSymbolic( F, A, perm, &cholInfo ); MatCholeskyFactorNumeric( F, A, &cholInfo ); but the in-place alternative, MatCholeskyFactor, does not seem to take in choices for the solver. The documentation for these routines is sparse because of the recommendations to use the KSP interfaces, but I need the direct access for a somewhat complicated preconditioner that I'm working on. Is there a way to tell MatCholeskyFactor which approach to use (i.e., MAT_SOLVER_MUMPS), or, failing that, can I destroy my original matrix 'A' and keep the factor 'F' around for solves? Thanks in advance for any clarification, Jack -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Sun Mar 6 22:50:51 2011 From: jed at 59A2.org (Jed Brown) Date: Sun, 6 Mar 2011 20:50:51 -0800 Subject: [petsc-users] In-place MUMPS call? In-Reply-To: References: Message-ID: On Sun, Mar 6, 2011 at 20:36, Jack Poulson wrote: > The documentation seems to suggest that I can choose a sparse-direct solver > and factor a MATMPISBAIJ matrix A out-of-place like this: > > Mat F; > MatGetFactor( A, MAT_SOLVER_MUMPS, MAT_FACTOR_CHOLESKY, &F ); > MatFactorInfo cholInfo; > cholInfo.fill = 3.0; // arbitrary choice > cholInfo.dtcol = 0.5; // arbitrary choice > IS perm; > ISCreateStride( comm, size, 0, 1, &perm ); > MatCholeskyFactorSymbolic( F, A, perm, &cholInfo ); > MatCholeskyFactorNumeric( F, A, &cholInfo ); > > but the in-place alternative, MatCholeskyFactor > I don't know of any sparse direct solver that actually does in-place factorization. An example of what MatCholeskyFactor does to make itself look in-place: PetscErrorCode MatCholeskyFactor_SeqSBAIJ(Mat A,IS perm,const MatFactorInfo *info) { PetscErrorCode ierr; Mat C; PetscFunctionBegin; ierr = MatGetFactor(A,"petsc",MAT_FACTOR_CHOLESKY,&C);CHKERRQ(ierr); ierr = MatCholeskyFactorSymbolic(C,A,perm,info);CHKERRQ(ierr); ierr = MatCholeskyFactorNumeric(C,A,info);CHKERRQ(ierr); A->ops->solve = C->ops->solve; A->ops->solvetranspose = C->ops->solvetranspose; ierr = MatHeaderMerge(A,C);CHKERRQ(ierr); PetscFunctionReturn(0); } Is there a way to tell MatCholeskyFactor which approach to use (i.e., > MAT_SOLVER_MUMPS), > No, maybe this should be added. > or, failing that, can I destroy my original matrix 'A' and keep the factor > 'F' around for solves? > Do this, there is no performance penalty. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jack.poulson at gmail.com Sun Mar 6 22:54:02 2011 From: jack.poulson at gmail.com (Jack Poulson) Date: Sun, 6 Mar 2011 22:54:02 -0600 Subject: [petsc-users] In-place MUMPS call? In-Reply-To: References: Message-ID: On Sun, Mar 6, 2011 at 10:50 PM, Jed Brown wrote: > On Sun, Mar 6, 2011 at 20:36, Jack Poulson wrote: > >> The documentation seems to suggest that I can choose a sparse-direct >> solver and factor a MATMPISBAIJ matrix A out-of-place like this: >> >> Mat F; >> MatGetFactor( A, MAT_SOLVER_MUMPS, MAT_FACTOR_CHOLESKY, &F ); >> MatFactorInfo cholInfo; >> cholInfo.fill = 3.0; // arbitrary choice >> cholInfo.dtcol = 0.5; // arbitrary choice >> IS perm; >> ISCreateStride( comm, size, 0, 1, &perm ); >> MatCholeskyFactorSymbolic( F, A, perm, &cholInfo ); >> MatCholeskyFactorNumeric( F, A, &cholInfo ); >> >> but the in-place alternative, MatCholeskyFactor >> > > I don't know of any sparse direct solver that actually does in-place > factorization. An example of what MatCholeskyFactor does to make itself look > in-place: > > PetscErrorCode MatCholeskyFactor_SeqSBAIJ(Mat A,IS perm,const MatFactorInfo > *info) > { > PetscErrorCode ierr; > Mat C; > > PetscFunctionBegin; > ierr = MatGetFactor(A,"petsc",MAT_FACTOR_CHOLESKY,&C);CHKERRQ(ierr); > ierr = MatCholeskyFactorSymbolic(C,A,perm,info);CHKERRQ(ierr); > ierr = MatCholeskyFactorNumeric(C,A,info);CHKERRQ(ierr); > A->ops->solve = C->ops->solve; > A->ops->solvetranspose = C->ops->solvetranspose; > ierr = MatHeaderMerge(A,C);CHKERRQ(ierr); > PetscFunctionReturn(0); > } > > Is there a way to tell MatCholeskyFactor which approach to use (i.e., >> MAT_SOLVER_MUMPS), >> > > No, maybe this should be added. > > >> or, failing that, can I destroy my original matrix 'A' and keep the factor >> 'F' around for solves? >> > > Do this, there is no performance penalty. > Thanks Jed. I was aware that the factorization was not actually in-place, but the fact that the symbolic and numeric factorizations required the original matrix to be around made me unsure of when I was allowed to free it. This interface should be perfectly sufficient. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jack.poulson at gmail.com Sun Mar 6 23:03:21 2011 From: jack.poulson at gmail.com (Jack Poulson) Date: Sun, 6 Mar 2011 23:03:21 -0600 Subject: [petsc-users] Input and output distributions of vectors for PCShellSetApply 'apply' routine Message-ID: Sorry to hit the list so quickly again, but the PCShellSetApply documentation doesn't say anything about how the 'xin' and 'xout' vectors will be distributed, or if it's up to the user to determine it. Given that a distributed Mat implies a right and left vector distribution, I would assume that the 'xin' should be distributed like the 'left' vector distribution for a left preconditioner. Is it up to the application routine to determine the 'xout' vector distribution? If so, is there a performance penalty for making it something other than the 'right' vector distribution implied by the matrix registered with the KSP? If not, what distribution does it need to be? Jack -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Sun Mar 6 23:23:36 2011 From: jed at 59A2.org (Jed Brown) Date: Sun, 6 Mar 2011 21:23:36 -0800 Subject: [petsc-users] Input and output distributions of vectors for PCShellSetApply 'apply' routine In-Reply-To: References: Message-ID: On Sun, Mar 6, 2011 at 21:03, Jack Poulson wrote: > Sorry to hit the list so quickly again, but the PCShellSetApply > documentation doesn't say anything about how the 'xin' and 'xout' vectors > will be distributed, or if it's up to the user to determine it. > When you create a parallel matrix, MatSetSizes(A,m,n,M,N) is always called (one way or another, perhaps as part of MatCreateMPIAIJ or similar). The pair (m,M) defines the row distribution (m locally owned out of M global) and (n,N) defines the column distribution. This is also the distribution of the vectors obtained using MatGetVecs(). Most KSPs only work for square systems in which case the left and right Vecs necessarily have the same distribution. LSQR is the only KSP I can think of that works with non-square systems. With LSQR, PCApply is called after MatMultTranspose so its input should be in the column distribution of the operator. -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Mon Mar 7 07:50:36 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Mon, 07 Mar 2011 14:50:36 +0100 Subject: [petsc-users] loosing preallocation information In-Reply-To: References: <4D73AAC5.3010407@gfz-potsdam.de> Message-ID: <4D74E2AC.9000505@gfz-potsdam.de> Hi Jed, It's getting even stranger. When I run this code under linux I've got error: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: nnz cannot be greater than row length: local row 98 value 1455203639 rowlength 343! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 14:26:37 CST 2010 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /home/mt/agrayver/mt-soft/multiem/INV3D/_tiger/em_model on a openmpi-i named glic by agrayver Mon Mar 7 14:49:43 2011 [0]PETSC ERROR: Libraries linked from /panfs/panfs.gfz-hpcc.cluster/home/mt/agrayver/lib/petsc-3.1-p7/openmpi-intel-complex-debug/lib [0]PETSC ERROR: Configure run at Fri Mar 4 12:43:58 2011 [0]PETSC ERROR: Configure options --with-petsc-arch=openmpi-intel-complex-debug --with-mpi-dir=/opt/mpi/intel/openmpi-1.4.2 --with-scalar-type=complex --with-blas-lapack-dir=/opt/intel/Compiler/11.1/072/mkl/lib/em64t --with-precision=double --with-x=0 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatSeqAIJSetPreallocation_SeqAIJ() line 3003 in src/mat/impls/aij/seq/aij.c [0]PETSC ERROR: MatCreateSeqAIJ() line 2906 in src/mat/impls/aij/seq/aij.c Then I traced it under debugger both on Windows and Linux and realized that it doesn't crash under Windows only by luck. The call stack if the folowing: [C] MatSeqAIJSetPreallocation_SeqAIJ, FP=7fffb4513880 [C] MatCreateSeqAIJ, FP=7fffb4513900 [C] matcreateseqaij_, FP=7fffb4513960 [F90] MODELING_MOD`modeling, FP=7fffb4517740 As far as I understand after debugging the problem is that nnz in MatSeqAIJSetPreallocation isn't 0 and I go through this code: if (nnz) { for (i=0; irmap->n; i++) { if (nnz[i] < 0) SETERRQ2(PETSC_ERR_ARG_OUTOFRANGE,"nnz cannot be less than 0: local row %d value %d",i,nnz[i]); if (nnz[i] > B->cmap->n) SETERRQ3(PETSC_ERR_ARG_OUTOFRANGE,"nnz cannot be greater than row length: local row %d value %d rowlength %d",i,nnz[i],B->cmap->n); } } But why is nnz nonzero? I passed PETSC_NULL. It seems like macros CHKFORTRANNULLINTEGER in matcreateseqaij_ should set it to zero, but it doesn't. Thanks. On 06.03.2011 16:49, Jed Brown wrote: > On Sun, Mar 6, 2011 at 07:39, Alexander Grayver > > wrote: > > Hello, > > I'm newbie in PETSc. > > I use PETSc from FORTRAN under Windows, my build configuration is: > Configure options: --with-cc="win32fe cl" --with-fc="win32fe > ifort" --with-cxx="win32fe cl" --with-windows-graphics=0 > --download-f-blas-lapack=1 --with-precision=double > --with-scalar-type=complex --with-clanguage=cxx > --with-mpi-include=/cygdrive/d/Dev/HPC_2008_SDK/Include > --with-mpi-lib="[/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpi.lib,/cygdrive/d/Dev/HPC_2008_SDK/Lib/amd64/msmpifec.lib]" > --with-shared=0 --with-debugging=0 --with-fortran-kernels=1 > --useThreads=0 > > I wrote this code: > > call MatCreateSeqAIJ(PETSC_COMM_SELF, n, m, 2, > PETSC_NULL, Gmat, ierr) > CHKERRQ(ierr) > > call cpu_time(timetest(1)) > > do irow=1,UBOUND(G, dim=1) > call MatSetValue(Gmat, ir(i)-1, ic(i)-1, > G(irow),e INSERT_VALUES, ierr) > enddo > > call cpu_time(timetest(2)) > write(*,*) 'Time of setting matrix G was ', > timetest(2) - timetest(1), ' seconds' > > call MatAssemblyBegin(Gmat,MAT_FINAL_ASSEMBLY,ierr) > call MatAssemblyEnd(Gmat,MAT_FINAL_ASSEMBLY,ierr) > > > Are you sure this is how your code looks, in particular that there is > no MatAssembly before setting the values. Could you send the whole file. > > If you are familiar with debuggers, you could call > > MatSetOption(Gmat,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE,ierr) > > and then run with -start_in_debugger to find the first insertion that > was not preallocated. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Mon Mar 7 07:59:54 2011 From: jed at 59A2.org (Jed Brown) Date: Mon, 7 Mar 2011 05:59:54 -0800 Subject: [petsc-users] loosing preallocation information In-Reply-To: <4D74E2AC.9000505@gfz-potsdam.de> References: <4D73AAC5.3010407@gfz-potsdam.de> <4D74E2AC.9000505@gfz-potsdam.de> Message-ID: Looks like a memory error. Best bet is to run in Valgrind. On Mar 7, 2011 5:52 AM, "Alexander Grayver" wrote: Hi Jed, It's getting even stranger. When I run this code under linux I've got error: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: nnz cannot be greater than row length: local row 98 value 1455203639 rowlength 343! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 14:26:37 CST 2010 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /home/mt/agrayver/mt-soft/multiem/INV3D/_tiger/em_model on a openmpi-i named glic by agrayver Mon Mar 7 14:49:43 2011 [0]PETSC ERROR: Libraries linked from /panfs/panfs.gfz-hpcc.cluster/home/mt/agrayver/lib/petsc-3.1-p7/openmpi-intel-complex-debug/lib [0]PETSC ERROR: Configure run at Fri Mar 4 12:43:58 2011 [0]PETSC ERROR: Configure options --with-petsc-arch=openmpi-intel-complex-debug --with-mpi-dir=/opt/mpi/intel/openmpi-1.4.2 --with-scalar-type=complex --with-blas-lapack-dir=/opt/intel/Compiler/11.1/072/mkl/lib/em64t --with-precision=double --with-x=0 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatSeqAIJSetPreallocation_SeqAIJ() line 3003 in src/mat/impls/aij/seq/aij.c [0]PETSC ERROR: MatCreateSeqAIJ() line 2906 in src/mat/impls/aij/seq/aij.c Then I traced it under debugger both on Windows and Linux and realized that it doesn't crash under Windows only by luck. The call stack if the folowing: [C] MatSeqAIJSetPreallocation_SeqAIJ, FP=7fffb4513880 [C] MatCreateSeqAIJ, FP=7fffb4513900 [C] matcreateseqaij_, FP=7fffb4513960 [F90] MODELING_MOD`modeling, FP=7fffb4517740 As far as I understand after debugging the problem is that nnz in MatSeqAIJSetPreallocation isn't 0 and I go through this code: if (nnz) { for (i=0; irmap->n; i++) { if (nnz[i] < 0) SETERRQ2(PETSC_ERR_ARG_OUTOFRANGE,"nnz cannot be less than 0: local row %d value %d",i,nnz[i]); if (nnz[i] > B->cmap->n) SETERRQ3(PETSC_ERR_ARG_OUTOFRANGE,"nnz cannot be greater than row length: local row %d value %d rowlength %d",i,nnz[i],B->cmap->n); } } But why is nnz nonzero? I passed PETSC_NULL. It seems like macros CHKFORTRANNULLINTEGER in matcreateseqaij_ should set it to zero, but it doesn't. Thanks. On 06.03.2011 16:49, Jed Brown wrote: > > On Sun, Mar 6, 2011 at 07:39, Alexander Grayver wrote: >> >> Hello, >... -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Mon Mar 7 08:04:24 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Mon, 07 Mar 2011 15:04:24 +0100 Subject: [petsc-users] loosing preallocation information In-Reply-To: References: <4D73AAC5.3010407@gfz-potsdam.de> <4D74E2AC.9000505@gfz-potsdam.de> Message-ID: <4D74E5E8.5060801@gfz-potsdam.de> Jed, but why do I get in line 3003 in src/mat/impls/aij/seq/aij.c at all? As far as understand, if I don't pass nnz (NULL in C/C++ or PETSC_NULL in fortran) I have to avoid this loop in aij.c code. I don't have much experience with valgrind. Could you provide me with right parameters to run my code under valgrind? Thanks a lot for your help. On 07.03.2011 14:59, Jed Brown wrote: > > Looks like a memory error. Best bet is to run in Valgrind. > >> On Mar 7, 2011 5:52 AM, "Alexander Grayver" > > wrote: >> >> Hi Jed, >> >> It's getting even stranger. >> >> When I run this code under linux I've got error: >> >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Argument out of range! >> [0]PETSC ERROR: nnz cannot be greater than row length: local row 98 >> value 1455203639 rowlength 343! >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 >> 14:26:37 CST 2010 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: >> /home/mt/agrayver/mt-soft/multiem/INV3D/_tiger/em_model on a >> openmpi-i named glic by agrayver Mon Mar 7 14:49:43 2011 >> [0]PETSC ERROR: Libraries linked from >> /panfs/panfs.gfz-hpcc.cluster/home/mt/agrayver/lib/petsc-3.1-p7/openmpi-intel-complex-debug/lib >> [0]PETSC ERROR: Configure run at Fri Mar 4 12:43:58 2011 >> [0]PETSC ERROR: Configure options >> --with-petsc-arch=openmpi-intel-complex-debug >> --with-mpi-dir=/opt/mpi/intel/openmpi-1.4.2 >> --with-scalar-type=complex >> --with-blas-lapack-dir=/opt/intel/Compiler/11.1/072/mkl/lib/em64t >> --with-precision=double --with-x=0 >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: MatSeqAIJSetPreallocation_SeqAIJ() line 3003 in >> src/mat/impls/aij/seq/aij.c >> [0]PETSC ERROR: MatCreateSeqAIJ() line 2906 in >> src/mat/impls/aij/seq/aij.c >> >> >> Then I traced it under debugger both on Windows and Linux and >> realized that it doesn't crash under Windows only by luck. >> >> The call stack if the folowing: >> [C] MatSeqAIJSetPreallocation_SeqAIJ, FP=7fffb4513880 >> [C] MatCreateSeqAIJ, FP=7fffb4513900 >> [C] matcreateseqaij_, FP=7fffb4513960 >> [F90] MODELING_MOD`modeling, FP=7fffb4517740 >> >> As far as I understand after debugging the problem is that nnz in >> MatSeqAIJSetPreallocation isn't 0 and I go through this code: >> if (nnz) { >> for (i=0; irmap->n; i++) { >> if (nnz[i] < 0) SETERRQ2(PETSC_ERR_ARG_OUTOFRANGE,"nnz cannot >> be less than 0: local row %d value %d",i,nnz[i]); >> if (nnz[i] > B->cmap->n) >> SETERRQ3(PETSC_ERR_ARG_OUTOFRANGE,"nnz cannot be greater than row >> length: local row %d value %d rowlength %d",i,nnz[i],B->cmap->n); >> } >> } >> >> But why is nnz nonzero? I passed PETSC_NULL. It seems like macros >> CHKFORTRANNULLINTEGER in matcreateseqaij_ should set it to zero, but >> it doesn't. >> >> >> Thanks. >> >> On 06.03.2011 16:49, Jed Brown wrote: >> >> >> > >> > On Sun, Mar 6, 2011 at 07:39, Alexander Grayver >> > wrote: >> >> >> >> Hello, >> >... >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From gaurish108 at gmail.com Mon Mar 7 08:20:31 2011 From: gaurish108 at gmail.com (Gaurish Telang) Date: Mon, 7 Mar 2011 09:20:31 -0500 Subject: [petsc-users] Not getting scalability. Message-ID: Hi, I have been testing PETSc's scalability on clusters for matrices of sizes 2000, 10,000, uptill 60,000. All I did was try to solve Ax=b for these matrices. I found that the solution time dips if I use upto 16 or 32 processors. However for a larger number of processors however the solution time seems to go up rather than down. IS there anyway I can make my code strongly scalable ? I am measuring the total time (sec) and KSP_SOLVE time in the -log_summary output. Both times show the same behaviour described above. Gaurish -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen.wornom at inria.fr Mon Mar 7 08:29:21 2011 From: stephen.wornom at inria.fr (Stephen Wornom) Date: Mon, 07 Mar 2011 15:29:21 +0100 Subject: [petsc-users] Not getting scalability. In-Reply-To: References: Message-ID: <4D74EBC1.5050209@inria.fr> Gaurish Telang wrote: > Hi, > > I have been testing PETSc's scalability on clusters for matrices of > sizes 2000, 10,000, uptill 60,000. > > All I did was try to solve Ax=b for these matrices. I found that the > solution time dips if I use upto 16 or 32 processors. However for a > larger number of processors however the solution time seems to go up > rather than down. IS there anyway I can make my code strongly scalable ? As the number of processors increases, the work/processor decreases. Therefore the communication time (mpi) will increase and the total time will increase slightly. Hope this helps, Stephen > > I am measuring the total time (sec) and KSP_SOLVE time in the > -log_summary output. Both times show the same behaviour described > above. > > Gaurish > -- stephen.wornom at inria.fr 2004 route des lucioles - BP93 Sophia Antipolis 06902 CEDEX Tel: 04 92 38 50 54 Fax: 04 97 15 53 51 -------------- next part -------------- A non-text attachment was scrubbed... Name: stephen_wornom.vcf Type: text/x-vcard Size: 160 bytes Desc: not available URL: From knepley at gmail.com Mon Mar 7 08:35:28 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 7 Mar 2011 08:35:28 -0600 Subject: [petsc-users] loosing preallocation information In-Reply-To: <4D74E5E8.5060801@gfz-potsdam.de> References: <4D73AAC5.3010407@gfz-potsdam.de> <4D74E2AC.9000505@gfz-potsdam.de> <4D74E5E8.5060801@gfz-potsdam.de> Message-ID: On Mon, Mar 7, 2011 at 8:04 AM, Alexander Grayver wrote: > Jed, but why do I get in line 3003 in src/mat/impls/aij/seq/aij.c at all? > As far as understand, if I don't pass nnz (NULL in C/C++ or PETSC_NULL in > fortran) I have to avoid this loop in aij.c code. > This is the problem. Fortran does not allow this kind of flexible checking, so there are different kinds of "NULL"s. You need to pass PETSC_NULL_INTEGER for nnz. Matt > I don't have much experience with valgrind. Could you provide me with right > parameters to run my code under valgrind? > Thanks a lot for your help. > > > On 07.03.2011 14:59, Jed Brown wrote: > > Looks like a memory error. Best bet is to run in Valgrind. > > On Mar 7, 2011 5:52 AM, "Alexander Grayver" > wrote: > > Hi Jed, > > It's getting even stranger. > > When I run this code under linux I've got error: > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Argument out of range! > [0]PETSC ERROR: nnz cannot be greater than row length: local row 98 value > 1455203639 rowlength 343! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 14:26:37 > CST 2010 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: /home/mt/agrayver/mt-soft/multiem/INV3D/_tiger/em_model on > a openmpi-i named glic by agrayver Mon Mar 7 14:49:43 2011 > [0]PETSC ERROR: Libraries linked from > /panfs/panfs.gfz-hpcc.cluster/home/mt/agrayver/lib/petsc-3.1-p7/openmpi-intel-complex-debug/lib > [0]PETSC ERROR: Configure run at Fri Mar 4 12:43:58 2011 > [0]PETSC ERROR: Configure options > --with-petsc-arch=openmpi-intel-complex-debug > --with-mpi-dir=/opt/mpi/intel/openmpi-1.4.2 --with-scalar-type=complex > --with-blas-lapack-dir=/opt/intel/Compiler/11.1/072/mkl/lib/em64t > --with-precision=double --with-x=0 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: MatSeqAIJSetPreallocation_SeqAIJ() line 3003 in > src/mat/impls/aij/seq/aij.c > [0]PETSC ERROR: MatCreateSeqAIJ() line 2906 in src/mat/impls/aij/seq/aij.c > > > Then I traced it under debugger both on Windows and Linux and realized that > it doesn't crash under Windows only by luck. > > The call stack if the folowing: > [C] MatSeqAIJSetPreallocation_SeqAIJ, FP=7fffb4513880 > [C] MatCreateSeqAIJ, FP=7fffb4513900 > [C] matcreateseqaij_, FP=7fffb4513960 > [F90] MODELING_MOD`modeling, FP=7fffb4517740 > > As far as I understand after debugging the problem is that nnz in > MatSeqAIJSetPreallocation isn't 0 and I go through this code: > if (nnz) { > for (i=0; irmap->n; i++) { > if (nnz[i] < 0) SETERRQ2(PETSC_ERR_ARG_OUTOFRANGE,"nnz cannot be > less than 0: local row %d value %d",i,nnz[i]); > if (nnz[i] > B->cmap->n) SETERRQ3(PETSC_ERR_ARG_OUTOFRANGE,"nnz > cannot be greater than row length: local row %d value %d rowlength > %d",i,nnz[i],B->cmap->n); > } > } > > But why is nnz nonzero? I passed PETSC_NULL. It seems like macros > CHKFORTRANNULLINTEGER in matcreateseqaij_ should set it to zero, but it > doesn't. > > > Thanks. > > On 06.03.2011 16:49, Jed Brown wrote: > > > > > > On Sun, Mar 6, 2011 at 07:39, Alexander Grayver > wrote: > >> > >> Hello, > >... > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Mar 7 08:38:06 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 7 Mar 2011 08:38:06 -0600 Subject: [petsc-users] Not getting scalability. In-Reply-To: References: Message-ID: On Mon, Mar 7, 2011 at 8:20 AM, Gaurish Telang wrote: > Hi, > > I have been testing PETSc's scalability on clusters for matrices of sizes > 2000, 10,000, uptill 60,000. > 1) These matrices are incredibly small. We usually recommend 10,000 unknowns/process for weak scaling. You might get some benefit from a shared memory implementation on a multicore. > All I did was try to solve Ax=b for these matrices. I found that the > solution time dips if I use upto 16 or 32 processors. However for a larger > number of processors however the solution time seems to go up rather than > down. IS there anyway I can make my code strongly scalable ? > 2) These are small enough that direct factorization should be the fastest alternative. I would try UMFPack, SuperLU, and MUMPS. Matt > I am measuring the total time (sec) and KSP_SOLVE time in the -log_summary > output. Both times show the same behaviour described above. > > Gaurish > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Mon Mar 7 08:39:09 2011 From: dave.mayhem23 at gmail.com (Dave May) Date: Mon, 7 Mar 2011 15:39:09 +0100 Subject: [petsc-users] Not getting scalability. In-Reply-To: <4D74EBC1.5050209@inria.fr> References: <4D74EBC1.5050209@inria.fr> Message-ID: And depending on how you configured your solver, it might make a difference what preconditioner you were using. Were you 1) measuring the time taken to perform a fixed number of Krylov iterations?, or 2) measuring the time taken for the solve to converge? If you did the latter and used the default petsc preconditioner (block jacobi/ilu in parallel) the number of Krylov iterations required to converge will increase when you perform strong scaling, as the blocks size will decrease with increasing number of cpus, thereby making the preconditioner weaker. Cheers, Dave On 7 March 2011 15:29, Stephen Wornom wrote: > Gaurish Telang wrote: >> >> Hi, >> >> I have been testing PETSc's scalability on clusters for matrices of sizes >> 2000, 10,000, uptill 60,000. >> >> All I did was try to solve Ax=b for these matrices. I found that the >> solution time dips if I use upto 16 or 32 processors. However for a larger >> number of processors however the solution time seems to go up rather than >> down. IS there anyway I can make my code strongly scalable ? > > As the number of processors increases, the work/processor decreases. > Therefore the communication time (mpi) will increase and the total time will > increase slightly. > Hope this helps, > Stephen >> >> I am measuring the total time (sec) and KSP_SOLVE time in the -log_summary >> output. Both times show the same behaviour described above. >> Gaurish >> > > > -- > stephen.wornom at inria.fr > 2004 route des lucioles - BP93 > Sophia Antipolis > 06902 CEDEX > > Tel: 04 92 38 50 54 > Fax: 04 97 15 53 51 > > From agrayver at gfz-potsdam.de Mon Mar 7 08:41:16 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Mon, 07 Mar 2011 15:41:16 +0100 Subject: [petsc-users] loosing preallocation information In-Reply-To: References: <4D73AAC5.3010407@gfz-potsdam.de> <4D74E2AC.9000505@gfz-potsdam.de> <4D74E5E8.5060801@gfz-potsdam.de> Message-ID: <4D74EE8C.8040106@gfz-potsdam.de> It works! Great, Matt. I guess there is no other way to know all these small issues but to get all possible errors? :) Jed, Matt, thank you a lot! On 07.03.2011 15:35, Matthew Knepley wrote: > On Mon, Mar 7, 2011 at 8:04 AM, Alexander Grayver > > wrote: > > Jed, but why do I get in line 3003 in src/mat/impls/aij/seq/aij.c > at all? As far as understand, if I don't pass nnz (NULL in C/C++ > or PETSC_NULL in fortran) I have to avoid this loop in aij.c code. > > > This is the problem. Fortran does not allow this kind of flexible > checking, so there are different kinds of "NULL"s. You > need to pass PETSC_NULL_INTEGER for nnz. > > Matt > > I don't have much experience with valgrind. Could you provide me > with right parameters to run my code under valgrind? > Thanks a lot for your help. > > > On 07.03.2011 14:59, Jed Brown wrote: >> >> Looks like a memory error. Best bet is to run in Valgrind. >> >>> On Mar 7, 2011 5:52 AM, "Alexander Grayver" >>> > wrote: >>> >>> Hi Jed, >>> >>> It's getting even stranger. >>> >>> When I run this code under linux I've got error: >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> ------------------------------------ >>> [0]PETSC ERROR: Argument out of range! >>> [0]PETSC ERROR: nnz cannot be greater than row length: local row >>> 98 value 1455203639 rowlength 343! >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 >>> 14:26:37 CST 2010 >>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [0]PETSC ERROR: See docs/index.html for manual pages. >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: >>> /home/mt/agrayver/mt-soft/multiem/INV3D/_tiger/em_model on a >>> openmpi-i named glic by agrayver Mon Mar 7 14:49:43 2011 >>> [0]PETSC ERROR: Libraries linked from >>> /panfs/panfs.gfz-hpcc.cluster/home/mt/agrayver/lib/petsc-3.1-p7/openmpi-intel-complex-debug/lib >>> [0]PETSC ERROR: Configure run at Fri Mar 4 12:43:58 2011 >>> [0]PETSC ERROR: Configure options >>> --with-petsc-arch=openmpi-intel-complex-debug >>> --with-mpi-dir=/opt/mpi/intel/openmpi-1.4.2 >>> --with-scalar-type=complex >>> --with-blas-lapack-dir=/opt/intel/Compiler/11.1/072/mkl/lib/em64t --with-precision=double >>> --with-x=0 >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: MatSeqAIJSetPreallocation_SeqAIJ() line 3003 in >>> src/mat/impls/aij/seq/aij.c >>> [0]PETSC ERROR: MatCreateSeqAIJ() line 2906 in >>> src/mat/impls/aij/seq/aij.c >>> >>> >>> Then I traced it under debugger both on Windows and Linux and >>> realized that it doesn't crash under Windows only by luck. >>> >>> The call stack if the folowing: >>> [C] MatSeqAIJSetPreallocation_SeqAIJ, FP=7fffb4513880 >>> [C] MatCreateSeqAIJ, FP=7fffb4513900 >>> [C] matcreateseqaij_, FP=7fffb4513960 >>> [F90] MODELING_MOD`modeling, FP=7fffb4517740 >>> >>> As far as I understand after debugging the problem is that nnz >>> in MatSeqAIJSetPreallocation isn't 0 and I go through this code: >>> if (nnz) { >>> for (i=0; irmap->n; i++) { >>> if (nnz[i] < 0) SETERRQ2(PETSC_ERR_ARG_OUTOFRANGE,"nnz >>> cannot be less than 0: local row %d value %d",i,nnz[i]); >>> if (nnz[i] > B->cmap->n) >>> SETERRQ3(PETSC_ERR_ARG_OUTOFRANGE,"nnz cannot be greater than >>> row length: local row %d value %d rowlength >>> %d",i,nnz[i],B->cmap->n); >>> } >>> } >>> >>> But why is nnz nonzero? I passed PETSC_NULL. It seems like >>> macros CHKFORTRANNULLINTEGER in matcreateseqaij_ should set it >>> to zero, but it doesn't. >>> >>> >>> Thanks. >>> >>> On 06.03.2011 16:49, Jed Brown wrote: >>> >>> >>> > >>> > On Sun, Mar 6, 2011 at 07:39, Alexander Grayver >>> > wrote: >>> >> >>> >> Hello, >>> >... >>> > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From gaurish108 at gmail.com Mon Mar 7 08:43:44 2011 From: gaurish108 at gmail.com (Gaurish Telang) Date: Mon, 7 Mar 2011 09:43:44 -0500 Subject: [petsc-users] Not getting scalability. In-Reply-To: References: Message-ID: Oh thank you, this was helpful. I am interested in iterative solvers, so what is the minimum matrix size you think that strong scalability will show up for such methods? On Mon, Mar 7, 2011 at 9:38 AM, Matthew Knepley wrote: > On Mon, Mar 7, 2011 at 8:20 AM, Gaurish Telang wrote: > >> Hi, >> >> I have been testing PETSc's scalability on clusters for matrices of sizes >> 2000, 10,000, uptill 60,000. >> > > 1) These matrices are incredibly small. We usually recommend 10,000 > unknowns/process for weak scaling. You > might get some benefit from a shared memory implementation on a > multicore. > > >> All I did was try to solve Ax=b for these matrices. I found that the >> solution time dips if I use upto 16 or 32 processors. However for a larger >> number of processors however the solution time seems to go up rather than >> down. IS there anyway I can make my code strongly scalable ? >> > > 2) These are small enough that direct factorization should be the fastest > alternative. I would try UMFPack, SuperLU, and MUMPS. > > Matt > > >> I am measuring the total time (sec) and KSP_SOLVE time in the -log_summary >> output. Both times show the same behaviour described above. >> >> Gaurish >> > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Mar 7 09:02:12 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 7 Mar 2011 09:02:12 -0600 Subject: [petsc-users] Not getting scalability. In-Reply-To: References: Message-ID: On Mon, Mar 7, 2011 at 8:43 AM, Gaurish Telang wrote: > Oh thank you, this was helpful. I am interested in iterative solvers, so > what is the minimum matrix size you > think that strong scalability will show up for such methods? > Such blanket predictions are not worth much for strong scaling since they depend on the architecture, interconnect, etc. What is most important is to understand the timing output in -log_summary and see what is not scaling correctly. Dave pointed out that linear iterations must also scale correctly. Matt > > On Mon, Mar 7, 2011 at 9:38 AM, Matthew Knepley wrote: > >> On Mon, Mar 7, 2011 at 8:20 AM, Gaurish Telang wrote: >> >>> Hi, >>> >>> I have been testing PETSc's scalability on clusters for matrices of sizes >>> 2000, 10,000, uptill 60,000. >>> >> >> 1) These matrices are incredibly small. We usually recommend 10,000 >> unknowns/process for weak scaling. You >> might get some benefit from a shared memory implementation on a >> multicore. >> >> >>> All I did was try to solve Ax=b for these matrices. I found that the >>> solution time dips if I use upto 16 or 32 processors. However for a larger >>> number of processors however the solution time seems to go up rather than >>> down. IS there anyway I can make my code strongly scalable ? >>> >> >> 2) These are small enough that direct factorization should be the fastest >> alternative. I would try UMFPack, SuperLU, and MUMPS. >> >> Matt >> >> >>> I am measuring the total time (sec) and KSP_SOLVE time in the >>> -log_summary output. Both times show the same behaviour described above. >>> >>> Gaurish >>> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From sylvain.desroziers at gmail.com Mon Mar 7 09:11:49 2011 From: sylvain.desroziers at gmail.com (Sylvain Desroziers) Date: Mon, 7 Mar 2011 16:11:49 +0100 Subject: [petsc-users] KSP and SuperLu, Umfpack Message-ID: Hi, I'm trying to use SuperLu and Umfpack LU solvers with Ksp. Using "-ksp_type preonly -pc_type lu -pc_factor_mat_solver_package superlu [or umpfpack]", there's no problem. But I don't know how to configure ksp without command line options... How to choose and switch external p?ckage ? Thanks, Sylvain -- -- (\__/) (='.'=) (")_(") Ceci est un petit lapin. Copiez/collez-le dans votre signature pour l'aider ? dominer le monde -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Mar 7 09:15:27 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 7 Mar 2011 09:15:27 -0600 Subject: [petsc-users] KSP and SuperLu, Umfpack In-Reply-To: References: Message-ID: On Mon, Mar 7, 2011 at 9:11 AM, Sylvain Desroziers < sylvain.desroziers at gmail.com> wrote: > Hi, > > I'm trying to use SuperLu and Umfpack LU solvers with Ksp. Using "-ksp_type > preonly -pc_type lu -pc_factor_mat_solver_package superlu [or umpfpack]", > there's no problem. > > But I don't know how to configure ksp without command line options... How > to choose and switch external p?ckage ? > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/PC/PCFactorSetMatSolverPackage.html Matt > Thanks, > > Sylvain > > -- > -- > (\__/) > (='.'=) > (")_(") > Ceci est un petit lapin. Copiez/collez-le dans votre signature pour > l'aider ? dominer le monde > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From ecoon at lanl.gov Mon Mar 7 09:48:20 2011 From: ecoon at lanl.gov (Ethan Coon) Date: Mon, 07 Mar 2011 08:48:20 -0700 Subject: [petsc-users] loosing preallocation information In-Reply-To: <4D74EE8C.8040106@gfz-potsdam.de> References: <4D73AAC5.3010407@gfz-potsdam.de> <4D74E2AC.9000505@gfz-potsdam.de> <4D74E5E8.5060801@gfz-potsdam.de> <4D74EE8C.8040106@gfz-potsdam.de> Message-ID: <1299512900.24152.4.camel@echo.lanl.gov> On Mon, 2011-03-07 at 15:41 +0100, Alexander Grayver wrote: > It works! Great, Matt. > > I guess there is no other way to know all these small issues but to > get all possible errors? :) > PETSc in Fortran has a lot of oddities due to the difference between the languages (this "null pointer" thing and different ways of looking at array bounds being the primary two). If you haven't seen it yet, you should really look through Chapter 10 of the manual http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manual.pdf which explains most of them. Trust me when I say it will save you a lot of frustrated time (speaking as someone who learned it the hard way!) Ethan > Jed, Matt, thank you a lot! > > On 07.03.2011 15:35, Matthew Knepley wrote: > > On Mon, Mar 7, 2011 at 8:04 AM, Alexander Grayver > > wrote: > > Jed, but why do I get in line 3003 in > > src/mat/impls/aij/seq/aij.c at all? As far as understand, if > > I don't pass nnz (NULL in C/C++ or PETSC_NULL in fortran) I > > have to avoid this loop in aij.c code. > > > > > > > > This is the problem. Fortran does not allow this kind of flexible > > checking, so there are different kinds of "NULL"s. You > > need to pass PETSC_NULL_INTEGER for nnz. > > > > > > Matt > > > > I don't have much experience with valgrind. Could you > > provide me with right parameters to run my code under > > valgrind? > > Thanks a lot for your help. > > > > > > On 07.03.2011 14:59, Jed Brown wrote: > > > Looks like a memory error. Best bet is to run in Valgrind. > > > > > > > On Mar 7, 2011 5:52 AM, "Alexander Grayver" > > > > wrote: > > > > > > > > Hi Jed, > > > > > > > > It's getting even stranger. > > > > > > > > When I run this code under linux I've got error: > > > > > > > > [0]PETSC ERROR: --------------------- Error Message > > > > ------------------------------------ > > > > [0]PETSC ERROR: Argument out of range! > > > > [0]PETSC ERROR: nnz cannot be greater than row length: > > > > local row 98 value 1455203639 rowlength 343! > > > > [0]PETSC ERROR: > > > > ------------------------------------------------------------------------ > > > > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, > > > > Mon Dec 20 14:26:37 CST 2010 > > > > [0]PETSC ERROR: See docs/changes/index.html for recent > > > > updates. > > > > [0]PETSC ERROR: See docs/faq.html for hints about > > > > trouble shooting. > > > > [0]PETSC ERROR: See docs/index.html for manual pages. > > > > [0]PETSC ERROR: > > > > ------------------------------------------------------------------------ > > > > [0]PETSC > > > > ERROR: /home/mt/agrayver/mt-soft/multiem/INV3D/_tiger/em_model on a openmpi-i named glic by agrayver Mon Mar 7 14:49:43 2011 > > > > [0]PETSC ERROR: Libraries linked > > > > from /panfs/panfs.gfz-hpcc.cluster/home/mt/agrayver/lib/petsc-3.1-p7/openmpi-intel-complex-debug/lib > > > > [0]PETSC ERROR: Configure run at Fri Mar 4 12:43:58 > > > > 2011 > > > > [0]PETSC ERROR: Configure options > > > > --with-petsc-arch=openmpi-intel-complex-debug > > > > --with-mpi-dir=/opt/mpi/intel/openmpi-1.4.2 > > > > --with-scalar-type=complex > > > > --with-blas-lapack-dir=/opt/intel/Compiler/11.1/072/mkl/lib/em64t --with-precision=double --with-x=0 > > > > [0]PETSC ERROR: > > > > ------------------------------------------------------------------------ > > > > [0]PETSC ERROR: MatSeqAIJSetPreallocation_SeqAIJ() line > > > > 3003 in src/mat/impls/aij/seq/aij.c > > > > [0]PETSC ERROR: MatCreateSeqAIJ() line 2906 in > > > > src/mat/impls/aij/seq/aij.c > > > > > > > > > > > > Then I traced it under debugger both on Windows and > > > > Linux and realized that it doesn't crash under Windows > > > > only by luck. > > > > > > > > The call stack if the folowing: > > > > [C] MatSeqAIJSetPreallocation_SeqAIJ, FP=7fffb4513880 > > > > [C] MatCreateSeqAIJ, FP=7fffb4513900 > > > > [C] matcreateseqaij_, FP=7fffb4513960 > > > > [F90] MODELING_MOD`modeling, FP=7fffb4517740 > > > > > > > > As far as I understand after debugging the problem is > > > > that nnz in MatSeqAIJSetPreallocation isn't 0 and I go > > > > through this code: > > > > if (nnz) { > > > > for (i=0; irmap->n; i++) { > > > > if (nnz[i] < 0) > > > > SETERRQ2(PETSC_ERR_ARG_OUTOFRANGE,"nnz cannot be less > > > > than 0: local row %d value %d",i,nnz[i]); > > > > if (nnz[i] > B->cmap->n) > > > > SETERRQ3(PETSC_ERR_ARG_OUTOFRANGE,"nnz cannot be greater > > > > than row length: local row %d value %d rowlength % > > > > d",i,nnz[i],B->cmap->n); > > > > } > > > > } > > > > > > > > But why is nnz nonzero? I passed PETSC_NULL. It seems > > > > like macros CHKFORTRANNULLINTEGER in matcreateseqaij_ > > > > should set it to zero, but it doesn't. > > > > > > > > > > > > > > > > Thanks. > > > > > > > > On 06.03.2011 16:49, Jed Brown wrote: > > > > > > > > > > > > > > > > > > On Sun, Mar 6, 2011 at 07:39, Alexander Grayver > > > > wrote: > > > > >> > > > > >> Hello, > > > > >... > > > > > > > > > > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which > > their experiments lead. > > -- Norbert Wiener > -- ------------------------------------ Ethan Coon Post-Doctoral Researcher Applied Mathematics - T-5 Los Alamos National Laboratory 505-665-8289 http://www.ldeo.columbia.edu/~ecoon/ ------------------------------------ From hzhang at mcs.anl.gov Mon Mar 7 09:48:24 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Mon, 7 Mar 2011 09:48:24 -0600 Subject: [petsc-users] KSP and SuperLu, Umfpack In-Reply-To: References: Message-ID: See petsc-dev/src/ksp/ksp/examples/tutorials/ex52.c (attached). Hong On Mon, Mar 7, 2011 at 9:15 AM, Matthew Knepley wrote: > On Mon, Mar 7, 2011 at 9:11 AM, Sylvain Desroziers > wrote: >> >> Hi, >> >> I'm trying to use SuperLu and Umfpack LU solvers with Ksp. Using >> "-ksp_type preonly -pc_type lu -pc_factor_mat_solver_package superlu [or >> umpfpack]", there's no problem. >> >> But I don't know how to configure ksp without command line options... How >> to choose and switch external p?ckage ? > > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/PC/PCFactorSetMatSolverPackage.html > ?? Matt > >> >> Thanks, >> >> Sylvain >> >> -- >> -- >> (\__/) >> (='.'=) >> (")_(") >> Ceci est un petit lapin.? Copiez/collez-le dans votre signature pour >> l'aider ? dominer le monde > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener > -------------- next part -------------- A non-text attachment was scrubbed... Name: ex52.c Type: text/x-csrc Size: 11226 bytes Desc: not available URL: From bsmith at mcs.anl.gov Mon Mar 7 11:32:40 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 7 Mar 2011 11:32:40 -0600 Subject: [petsc-users] Input and output distributions of vectors for PCShellSetApply 'apply' routine In-Reply-To: References: Message-ID: On Mar 6, 2011, at 11:03 PM, Jack Poulson wrote: > Sorry to hit the list so quickly again, but the PCShellSetApply documentation doesn't say anything about how the 'xin' and 'xout' vectors will be distributed, or if it's up to the user to determine it. > > Given that a distributed Mat implies a right and left vector distribution, I would assume that the 'xin' should be distributed like the 'left' vector distribution for a left preconditioner. > > Is it up to the application routine to determine the 'xout' vector distribution? If so, is there a performance penalty for making it something other than the 'right' vector distribution implied by the matrix registered with the KSP? If not, what distribution does it need to be? The PCShell doesn't determine the vector layouts at all. That is determined by the KSP that is using the PC. So basically your PC should handle any distribution; if your PCShell can only handle certain distributions then you need to make sure that the Vecs and Mats you provide to KSP or SNES (or TS) match what the PCShell needs. Barry > > Jack From brtnfld at uiuc.edu Mon Mar 7 11:39:27 2011 From: brtnfld at uiuc.edu (M. Scot Breitenfeld) Date: Mon, 07 Mar 2011 11:39:27 -0600 Subject: [petsc-users] AOCreateBasic is not scaling In-Reply-To: References: <4D6EC3C4.4080407@uiuc.edu> <4D6ECA40.5080409@uiuc.edu> <4D71563E.3060407@uiuc.edu> Message-ID: <4D75184F.9080100@uiuc.edu> On 03/04/2011 03:20 PM, Matthew Knepley wrote: > On Fri, Mar 4, 2011 at 3:14 PM, M. Scot Breitenfeld > wrote: > > On 03/03/2011 12:18 PM, Matthew Knepley wrote: > > On Wed, Mar 2, 2011 at 4:52 PM, M. Scot Breitenfeld > > > >> wrote: > > > > I don't number my global degree's of freedom from low-high > > continuously > > per processor as PETSc uses for ordering, but I use the natural > > ordering > > of the application, I then use AOcreateBasic to obtain the > mapping > > between the PETSc and my ordering. > > > > > > I would suggest using the LocalToGlobalMapping functions, which are > > scalable. > > AO is designed for complete global permutations. > I don't understand how I can avoid not using AO if my global dof per > processor are not arranged in PETSc global ordering (continuous row > ordering, i.e. proc1_ 0-n, proc2_n+1:m, proc3_m+1:p, etc...). In the > LocalToGlobalMapping routines, doesn't the "GlobalMapping" part mean > PETSc ordering and not my application's ordering. > > I thought I understood the difference between AO and > LocalToGlobalMapping but now I'm confused. I tried to use the > LocalToGlobalMapping routines and the solution values are correct but > the ordering corresponds the global node ordering, not how I > partitioned > the mesh. In other words, the values are returned in the same ordering > as for a serial run, which makes sense since this is how PETSc orders > the rows. If I had used PETSc ordering then this would be fine. > > Is the moral of the story, if I want to get scalability I need to > rearrange my global dof in PETSc ordering so that I can use > LocalToGlobalMapping? > > > I am having a really hard time understanding what you want? If you > want Natural > Ordering or any other crazy ordering on input/output go ahead and use > AO there > because the non-scalability is amortized over the run. The PETSc > ordering should > be used for all globally assembled structures in the solve because its > efficient, and > there is no reason for the user to care about these structures. For > integration/assembly, > use local orderings since that is all you need for a PDE. If you have > an exotic > equation that really does need global information, I would like to > hear about it, but > it would most likely be non-scalable on its own. I don't think I'm doing anything crazy, just probably a misunderstanding on my part, I'll try to explain it again: Take this 1D example (dof matches the node numbering) where 'o' are nodes Global--0********4********3********1********2********5 ______o--------o--------o--------o--------o--------o Local---0********1********2********0********1********2 ______|------PROC0---------|-------PROC1-----------| PROC0: indices = 0,4,3, input = 0, 1, 2 PROC1: indices = 1,2,5, input = 0, 1, 2 CALL ISLocalToGlobalMappingCreate(PETSC_COMM_WORLD, 3, indices, mapping, ierr) CALL ISLocalToGlobalMappingApply(mapping,3,input,output,ierr) PROC0: output = 0,4,3, input = 0,1,2 PROC1: output = 1,2,5, input = 0,1,2 CALL VecCreateMPI(PETSC_COMM_WORLD,3, 6, b,ierr) CALL MatCreateMPISBAIJ(PETSC_COMM_WORLD, 1, & 3, 3, & 6, 6, & 0, d_nnz, 0, o_nnz, A, ierr) CALL MatSetLocalToGlobalMapping(A, mapping, ierr) CALL VecSetLocalToGlobalMapping(b, mapping, ierr) ... use MatSetValuesLocal, VecSetValuesLocal to fill arrays. ... solve and place solution in 'b' vector Now it is my understanding that when I use VecGetValues (and is also what I observe) PROC0: b will have solutions for global nodes 0,1,2 PROC1: b will have solutions for global nodes 3,4,5 But, I would like to have, PROC0: b will have solutions for global nodes 0,4,3 PROC1: b will have solutions for global nodes 1,2,5 It is my understanding that I either need to use AO to get the these values, or I should have renumber the nodes such that, Global--0********1********2********3********4********5 ______o--------o--------o--------o--------o--------o Local---0********1********2********0********1********2 ______|------PROC0---------|-------PROC1-----------| Scot > > Matt > > > > > > Thanks, > > > > Matt > > > > > > CALL VecGetOwnershipRange(b, low, high, ierr) > > > > icnt = 0 > > > > DO mi = 1, mctr ! these are the nodes local to processor > > mi_global = myglobal(mi) > > > > irowx = 3*mi_global-2 > > irowy = 3*mi_global-1 > > irowz = 3*mi_global > > > > mappings(icnt+1:icnt+3) = (/ & > > nrow_global(row_from_dof(1,mi))-1, & > > nrow_global(row_from_dof(2,mi))-1, & > > nrow_global(row_from_dof(3,mi))-1 & > > /) > > > > petscOrdering(icnt+1:icnt+3) = > > (/low+icnt,low+icnt+1,low+icnt+2/) > > > > icnt = icnt + 3 > > END DO > > > > CALL AOCreateBasic(PETSC_COMM_WORLD, icnt, mappings, > petscOrdering, > > toao, ierr) > > > > DO mi = mctr+1, myn ! these are the ghost nodes not on this > processor > > > > mi_global = myglobal(mi) > > > > mappings(icnt+1:icnt+3) = (/ & > > nrow_global(row_from_dof(1,mi))-1, & > > nrow_global(row_from_dof(2,mi))-1, & > > nrow_global(row_from_dof(3,mi))-1 & > > /) > > > > icnt = icnt + 3 > > ENDDO > > CALL AOApplicationToPetsc(toao, 3*myn, mappings, ierr) > > > > CALL AODestroy(toao, ierr) > > > > I then use mapping to input the values into the correct row > as wanted > > by PETSc > > > > > > On 03/02/2011 04:29 PM, Matthew Knepley wrote: > > > On Wed, Mar 2, 2011 at 4:25 PM, M. Scot Breitenfeld > > > > > > > > >>> wrote: > > > > > > Hi, > > > > > > First, thanks for the suggestion on using MPISBAIJ for > my A > > matrix, it > > > seems to have cut down on my memory and assembly time. > For a 1.5 > > > million > > > dof problem: > > > > > > # procs: 2 4 8 > 16 > > > > ---------------------------------------------------------------- > > > Assembly (sec): 245 124 63 86 > > > Solver (sec): 924 578 326 680 > > > > > > Memory (GB): 2.5 1.4 .877 .565 > > > > > > The problem I have is the amount of time it's taking in > > AOCreateBasic, > > > it takes longer then assembling, > > > > > > # procs: 2 4 8 > > 16 > > > > > > --------------------------------------------------------------------- > > > AOCreateBasic (sec): .6 347 170 197 > > > > > > Is there something that I can change or something I > can look > > for that > > > might be causing this increase in time as I go from 2 to 4 > > processors > > > (at least it scales from 4 to 8 processors). I read in the > > archive > > > that > > > AOCreateBasic is not meant to be scalable so maybe > there is > > nothing I > > > can do. > > > > > > > > > Yes, this is non-scalable. What are you using it for? > > > > > > Matt > > > > > > > > > Thanks, > > > Scot > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > What most experimenters take for granted before they begin > their > > > experiments is infinitely more interesting than any > results to which > > > their experiments lead. > > > -- Norbert Wiener > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which > > their experiments lead. > > -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener From jack.poulson at gmail.com Mon Mar 7 11:45:27 2011 From: jack.poulson at gmail.com (Jack Poulson) Date: Mon, 7 Mar 2011 11:45:27 -0600 Subject: [petsc-users] Input and output distributions of vectors for PCShellSetApply 'apply' routine In-Reply-To: References: Message-ID: On Mon, Mar 7, 2011 at 11:32 AM, Barry Smith wrote: > > On Mar 6, 2011, at 11:03 PM, Jack Poulson wrote: > > > Sorry to hit the list so quickly again, but the PCShellSetApply > documentation doesn't say anything about how the 'xin' and 'xout' vectors > will be distributed, or if it's up to the user to determine it. > > > > Given that a distributed Mat implies a right and left vector > distribution, I would assume that the 'xin' should be distributed like the > 'left' vector distribution for a left preconditioner. > > > > Is it up to the application routine to determine the 'xout' vector > distribution? If so, is there a performance penalty for making it something > other than the 'right' vector distribution implied by the matrix registered > with the KSP? If not, what distribution does it need to be? > > The PCShell doesn't determine the vector layouts at all. That is > determined by the KSP that is using the PC. So basically your PC should > handle any distribution; if your PCShell can only handle certain > distributions then you need to make sure that the Vecs and Mats you provide > to KSP or SNES (or TS) match what the PCShell needs. > > Barry > > It turns out that I had simply misunderstood the way PETSc was distributing matrices under the hood. I was assuming that it was best to only distribute the rows of the matrix using something along the lines of MatSetSizes( A, N/nproc, N, N, N ), but Jed informed me that the local size arguments for MatSetSizes only constrain the left and right vector distributions implied by the matrix distribution (through rmap and cmap, respectively). Also, I now know that KSP expects the right and left vector distributions to be the same for square matrices, so I should instead create my matrix along the lines of MatSetSizes( A, N/nproc, N/nproc, N, N ), and then both the in and out vectors for the PCShell application routine should be distributed in the same manner. Thank you both for clearing up my misconceptions. Jack -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Mar 7 11:57:52 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 7 Mar 2011 11:57:52 -0600 Subject: [petsc-users] AOCreateBasic is not scaling In-Reply-To: <4D75184F.9080100@uiuc.edu> References: <4D6EC3C4.4080407@uiuc.edu> <4D6ECA40.5080409@uiuc.edu> <4D71563E.3060407@uiuc.edu> <4D75184F.9080100@uiuc.edu> Message-ID: On Mon, Mar 7, 2011 at 11:39 AM, M. Scot Breitenfeld wrote: > On 03/04/2011 03:20 PM, Matthew Knepley wrote: > > On Fri, Mar 4, 2011 at 3:14 PM, M. Scot Breitenfeld > > wrote: > > > > On 03/03/2011 12:18 PM, Matthew Knepley wrote: > > > On Wed, Mar 2, 2011 at 4:52 PM, M. Scot Breitenfeld > > > > > >> wrote: > > > > > > I don't number my global degree's of freedom from low-high > > > continuously > > > per processor as PETSc uses for ordering, but I use the natural > > > ordering > > > of the application, I then use AOcreateBasic to obtain the > > mapping > > > between the PETSc and my ordering. > > > > > > > > > I would suggest using the LocalToGlobalMapping functions, which are > > > scalable. > > > AO is designed for complete global permutations. > > I don't understand how I can avoid not using AO if my global dof per > > processor are not arranged in PETSc global ordering (continuous row > > ordering, i.e. proc1_ 0-n, proc2_n+1:m, proc3_m+1:p, etc...). In the > > LocalToGlobalMapping routines, doesn't the "GlobalMapping" part mean > > PETSc ordering and not my application's ordering. > > > > I thought I understood the difference between AO and > > LocalToGlobalMapping but now I'm confused. I tried to use the > > LocalToGlobalMapping routines and the solution values are correct but > > the ordering corresponds the global node ordering, not how I > > partitioned > > the mesh. In other words, the values are returned in the same > ordering > > as for a serial run, which makes sense since this is how PETSc orders > > the rows. If I had used PETSc ordering then this would be fine. > > > > Is the moral of the story, if I want to get scalability I need to > > rearrange my global dof in PETSc ordering so that I can use > > LocalToGlobalMapping? > > > > > > I am having a really hard time understanding what you want? If you > > want Natural > > Ordering or any other crazy ordering on input/output go ahead and use > > AO there > > because the non-scalability is amortized over the run. The PETSc > > ordering should > > be used for all globally assembled structures in the solve because its > > efficient, and > > there is no reason for the user to care about these structures. For > > integration/assembly, > > use local orderings since that is all you need for a PDE. If you have > > an exotic > > equation that really does need global information, I would like to > > hear about it, but > > it would most likely be non-scalable on its own. > I don't think I'm doing anything crazy, just probably a misunderstanding > on my part, I'll try to explain it again: > > Take this 1D example (dof matches the node numbering) where 'o' are nodes > > Global--0********4********3********1********2********5 > ______o--------o--------o--------o--------o--------o > Local---0********1********2********0********1********2 > ______|------PROC0---------|-------PROC1-----------| > > PROC0: indices = 0,4,3, input = 0, 1, 2 > PROC1: indices = 1,2,5, input = 0, 1, 2 > > CALL ISLocalToGlobalMappingCreate(PETSC_COMM_WORLD, 3, indices, mapping, > ierr) > CALL ISLocalToGlobalMappingApply(mapping,3,input,output,ierr) > > PROC0: output = 0,4,3, input = 0,1,2 > PROC1: output = 1,2,5, input = 0,1,2 > > CALL VecCreateMPI(PETSC_COMM_WORLD,3, 6, b,ierr) > > CALL MatCreateMPISBAIJ(PETSC_COMM_WORLD, 1, & > 3, 3, & > 6, 6, & > 0, d_nnz, 0, o_nnz, A, ierr) > > CALL MatSetLocalToGlobalMapping(A, mapping, ierr) > CALL VecSetLocalToGlobalMapping(b, mapping, ierr) > > ... use MatSetValuesLocal, VecSetValuesLocal to fill arrays. > > ... solve and place solution in 'b' vector > > Now it is my understanding that when I use VecGetValues (and is also > what I observe) > > PROC0: b will have solutions for global nodes 0,1,2 > PROC1: b will have solutions for global nodes 3,4,5 > > But, I would like to have, > > PROC0: b will have solutions for global nodes 0,4,3 > PROC1: b will have solutions for global nodes 1,2,5 > > It is my understanding that I either need to use AO to get the > these values, or I should have renumber the nodes such that, > Yes, however the global numbering during a solve is really only important to the PETSc internals, and it designed to be efficient. You can use AO for global reordering on input/output. As I said, this is amortized over the entire computation, and thus some non-scalability is usually no problem. If it is, consider renumbering because the entire solve would run slowly with an arbitrary global ordering. Matt > Global--0********1********2********3********4********5 > ______o--------o--------o--------o--------o--------o > Local---0********1********2********0********1********2 > ______|------PROC0---------|-------PROC1-----------| > > Scot > > > > > > > Matt > > > > > > > > > > Thanks, > > > > > > Matt > > > > > > > > > CALL VecGetOwnershipRange(b, low, high, ierr) > > > > > > icnt = 0 > > > > > > DO mi = 1, mctr ! these are the nodes local to processor > > > mi_global = myglobal(mi) > > > > > > irowx = 3*mi_global-2 > > > irowy = 3*mi_global-1 > > > irowz = 3*mi_global > > > > > > mappings(icnt+1:icnt+3) = (/ & > > > nrow_global(row_from_dof(1,mi))-1, & > > > nrow_global(row_from_dof(2,mi))-1, & > > > nrow_global(row_from_dof(3,mi))-1 & > > > /) > > > > > > petscOrdering(icnt+1:icnt+3) = > > > (/low+icnt,low+icnt+1,low+icnt+2/) > > > > > > icnt = icnt + 3 > > > END DO > > > > > > CALL AOCreateBasic(PETSC_COMM_WORLD, icnt, mappings, > > petscOrdering, > > > toao, ierr) > > > > > > DO mi = mctr+1, myn ! these are the ghost nodes not on this > > processor > > > > > > mi_global = myglobal(mi) > > > > > > mappings(icnt+1:icnt+3) = (/ & > > > nrow_global(row_from_dof(1,mi))-1, & > > > nrow_global(row_from_dof(2,mi))-1, & > > > nrow_global(row_from_dof(3,mi))-1 & > > > /) > > > > > > icnt = icnt + 3 > > > ENDDO > > > CALL AOApplicationToPetsc(toao, 3*myn, mappings, ierr) > > > > > > CALL AODestroy(toao, ierr) > > > > > > I then use mapping to input the values into the correct row > > as wanted > > > by PETSc > > > > > > > > > On 03/02/2011 04:29 PM, Matthew Knepley wrote: > > > > On Wed, Mar 2, 2011 at 4:25 PM, M. Scot Breitenfeld > > > > > > > > > > > > >>> wrote: > > > > > > > > Hi, > > > > > > > > First, thanks for the suggestion on using MPISBAIJ for > > my A > > > matrix, it > > > > seems to have cut down on my memory and assembly time. > > For a 1.5 > > > > million > > > > dof problem: > > > > > > > > # procs: 2 4 8 > > 16 > > > > > > ---------------------------------------------------------------- > > > > Assembly (sec): 245 124 63 86 > > > > Solver (sec): 924 578 326 680 > > > > > > > > Memory (GB): 2.5 1.4 .877 .565 > > > > > > > > The problem I have is the amount of time it's taking in > > > AOCreateBasic, > > > > it takes longer then assembling, > > > > > > > > # procs: 2 4 > 8 > > > 16 > > > > > > > > > --------------------------------------------------------------------- > > > > AOCreateBasic (sec): .6 347 170 197 > > > > > > > > Is there something that I can change or something I > > can look > > > for that > > > > might be causing this increase in time as I go from 2 to > 4 > > > processors > > > > (at least it scales from 4 to 8 processors). I read in > the > > > archive > > > > that > > > > AOCreateBasic is not meant to be scalable so maybe > > there is > > > nothing I > > > > can do. > > > > > > > > > > > > Yes, this is non-scalable. What are you using it for? > > > > > > > > Matt > > > > > > > > > > > > Thanks, > > > > Scot > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > What most experimenters take for granted before they begin > > their > > > > experiments is infinitely more interesting than any > > results to which > > > > their experiments lead. > > > > -- Norbert Wiener > > > > > > > > > > > > > > > -- > > > What most experimenters take for granted before they begin their > > > experiments is infinitely more interesting than any results to > which > > > their experiments lead. > > > -- Norbert Wiener > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which > > their experiments lead. > > -- Norbert Wiener > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From sylvain.desroziers at gmail.com Mon Mar 7 14:53:36 2011 From: sylvain.desroziers at gmail.com (Sylvain Desroziers) Date: Mon, 7 Mar 2011 21:53:36 +0100 Subject: [petsc-users] KSP and SuperLu, Umfpack In-Reply-To: References: Message-ID: Thanks!!! :) 2011/3/7 Hong Zhang > See petsc-dev/src/ksp/ksp/examples/tutorials/ex52.c (attached). > > Hong > > On Mon, Mar 7, 2011 at 9:15 AM, Matthew Knepley wrote: > > On Mon, Mar 7, 2011 at 9:11 AM, Sylvain Desroziers > > wrote: > >> > >> Hi, > >> > >> I'm trying to use SuperLu and Umfpack LU solvers with Ksp. Using > >> "-ksp_type preonly -pc_type lu -pc_factor_mat_solver_package superlu [or > >> umpfpack]", there's no problem. > >> > >> But I don't know how to configure ksp without command line options... > How > >> to choose and switch external p?ckage ? > > > > > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/PC/PCFactorSetMatSolverPackage.html > > Matt > > > >> > >> Thanks, > >> > >> Sylvain > >> > >> -- > >> -- > >> (\__/) > >> (='.'=) > >> (")_(") > >> Ceci est un petit lapin. Copiez/collez-le dans votre signature pour > >> l'aider ? dominer le monde > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments > > is infinitely more interesting than any results to which their > experiments > > lead. > > -- Norbert Wiener > > > -- -- (\__/) (='.'=) (")_(") Ceci est un petit lapin. Copiez/collez-le dans votre signature pour l'aider ? dominer le monde -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Mar 7 16:58:20 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 7 Mar 2011 16:58:20 -0600 Subject: [petsc-users] loosing preallocation information In-Reply-To: <1299512900.24152.4.camel@echo.lanl.gov> References: <4D73AAC5.3010407@gfz-potsdam.de> <4D74E2AC.9000505@gfz-potsdam.de> <4D74E5E8.5060801@gfz-potsdam.de> <4D74EE8C.8040106@gfz-potsdam.de> <1299512900.24152.4.camel@echo.lanl.gov> Message-ID: <68DDAB34-3AB8-4321-B6E0-F576E8627A23@mcs.anl.gov> On Mar 7, 2011, at 9:48 AM, Ethan Coon wrote: > On Mon, 2011-03-07 at 15:41 +0100, Alexander Grayver wrote: >> It works! Great, Matt. >> >> I guess there is no other way to know all these small issues but to >> get all possible errors? :) This should not have happened. Here is void PETSC_STDCALL matcreateseqaij_(MPI_Comm *comm,PetscInt *m,PetscInt *n,PetscInt *nz, PetscInt *nnz,Mat *newmat,PetscErrorCode *ierr) { CHKFORTRANNULLINTEGER(nnz); *ierr = MatCreateSeqAIJ(MPI_Comm_f2c(*(MPI_Fint *)&*comm),*m,*n,*nz,nnz,newmat); } and here is #define CHKFORTRANNULLINTEGER(a) \ if (FORTRANNULL(a) || FORTRANNULLDOUBLE(a) || FORTRANNULLSCALAR(a) || FORTRANNULLREAL(a) || FORTRANNULLOBJECT(a) || FORTRANNULLFUNCTION(a)) { \ PetscError(PETSC_COMM_SELF,__LINE__,"fortran_interface_unknown_file",__FILE__,__SDIR__,PETSC_ERR_ARG_WRONG,PETSC_ERROR_INITIAL, \ "Use PETSC_NULL_INTEGER"); *ierr = 1; return; } \ else if (FORTRANNULLINTEGER(a)) { a = PETSC_NULL; } If you passed in PETSC_NULL instead of PETSC_NULL_INTEGER it should have printed a (slightly helpful) error message right at this point, it should not have gotten to that place later in the code where it stopped. Are you sure you passed in PETSC_NULL? And did you include petsc.h or petscsys.h to make sure that PETSC_NULL was defined or did you pass 0 in for nnz? I'd like to know so we can fix any PETSc bug there might be there. Barry >> > > PETSc in Fortran has a lot of oddities due to the difference between the > languages (this "null pointer" thing and different ways of looking at > array bounds being the primary two). If you haven't seen it yet, you > should really look through Chapter 10 of the manual > > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manual.pdf > > which explains most of them. Trust me when I say it will save you a lot > of frustrated time (speaking as someone who learned it the hard way!) > > Ethan > >> Jed, Matt, thank you a lot! >> >> On 07.03.2011 15:35, Matthew Knepley wrote: >>> On Mon, Mar 7, 2011 at 8:04 AM, Alexander Grayver >>> wrote: >>> Jed, but why do I get in line 3003 in >>> src/mat/impls/aij/seq/aij.c at all? As far as understand, if >>> I don't pass nnz (NULL in C/C++ or PETSC_NULL in fortran) I >>> have to avoid this loop in aij.c code. >>> >>> >>> >>> This is the problem. Fortran does not allow this kind of flexible >>> checking, so there are different kinds of "NULL"s. You >>> need to pass PETSC_NULL_INTEGER for nnz. >>> >>> >>> Matt >>> >>> I don't have much experience with valgrind. Could you >>> provide me with right parameters to run my code under >>> valgrind? >>> Thanks a lot for your help. >>> >>> >>> On 07.03.2011 14:59, Jed Brown wrote: >>>> Looks like a memory error. Best bet is to run in Valgrind. >>>> >>>>> On Mar 7, 2011 5:52 AM, "Alexander Grayver" >>>>> wrote: >>>>> >>>>> Hi Jed, >>>>> >>>>> It's getting even stranger. >>>>> >>>>> When I run this code under linux I've got error: >>>>> >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> ------------------------------------ >>>>> [0]PETSC ERROR: Argument out of range! >>>>> [0]PETSC ERROR: nnz cannot be greater than row length: >>>>> local row 98 value 1455203639 rowlength 343! >>>>> [0]PETSC ERROR: >>>>> ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, >>>>> Mon Dec 20 14:26:37 CST 2010 >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent >>>>> updates. >>>>> [0]PETSC ERROR: See docs/faq.html for hints about >>>>> trouble shooting. >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>>> [0]PETSC ERROR: >>>>> ------------------------------------------------------------------------ >>>>> [0]PETSC >>>>> ERROR: /home/mt/agrayver/mt-soft/multiem/INV3D/_tiger/em_model on a openmpi-i named glic by agrayver Mon Mar 7 14:49:43 2011 >>>>> [0]PETSC ERROR: Libraries linked >>>>> from /panfs/panfs.gfz-hpcc.cluster/home/mt/agrayver/lib/petsc-3.1-p7/openmpi-intel-complex-debug/lib >>>>> [0]PETSC ERROR: Configure run at Fri Mar 4 12:43:58 >>>>> 2011 >>>>> [0]PETSC ERROR: Configure options >>>>> --with-petsc-arch=openmpi-intel-complex-debug >>>>> --with-mpi-dir=/opt/mpi/intel/openmpi-1.4.2 >>>>> --with-scalar-type=complex >>>>> --with-blas-lapack-dir=/opt/intel/Compiler/11.1/072/mkl/lib/em64t --with-precision=double --with-x=0 >>>>> [0]PETSC ERROR: >>>>> ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: MatSeqAIJSetPreallocation_SeqAIJ() line >>>>> 3003 in src/mat/impls/aij/seq/aij.c >>>>> [0]PETSC ERROR: MatCreateSeqAIJ() line 2906 in >>>>> src/mat/impls/aij/seq/aij.c >>>>> >>>>> >>>>> Then I traced it under debugger both on Windows and >>>>> Linux and realized that it doesn't crash under Windows >>>>> only by luck. >>>>> >>>>> The call stack if the folowing: >>>>> [C] MatSeqAIJSetPreallocation_SeqAIJ, FP=7fffb4513880 >>>>> [C] MatCreateSeqAIJ, FP=7fffb4513900 >>>>> [C] matcreateseqaij_, FP=7fffb4513960 >>>>> [F90] MODELING_MOD`modeling, FP=7fffb4517740 >>>>> >>>>> As far as I understand after debugging the problem is >>>>> that nnz in MatSeqAIJSetPreallocation isn't 0 and I go >>>>> through this code: >>>>> if (nnz) { >>>>> for (i=0; irmap->n; i++) { >>>>> if (nnz[i] < 0) >>>>> SETERRQ2(PETSC_ERR_ARG_OUTOFRANGE,"nnz cannot be less >>>>> than 0: local row %d value %d",i,nnz[i]); >>>>> if (nnz[i] > B->cmap->n) >>>>> SETERRQ3(PETSC_ERR_ARG_OUTOFRANGE,"nnz cannot be greater >>>>> than row length: local row %d value %d rowlength % >>>>> d",i,nnz[i],B->cmap->n); >>>>> } >>>>> } >>>>> >>>>> But why is nnz nonzero? I passed PETSC_NULL. It seems >>>>> like macros CHKFORTRANNULLINTEGER in matcreateseqaij_ >>>>> should set it to zero, but it doesn't. >>>>> >>>>> >>>>> >>>>> Thanks. >>>>> >>>>> On 06.03.2011 16:49, Jed Brown wrote: >>>>> >>>>> >>>>>> >>>>>> On Sun, Mar 6, 2011 at 07:39, Alexander Grayver >>>>> wrote: >>>>>>> >>>>>>> Hello, >>>>>> ... >>>>> >>>>> >>> >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which >>> their experiments lead. >>> -- Norbert Wiener >> > > -- > ------------------------------------ > Ethan Coon > Post-Doctoral Researcher > Applied Mathematics - T-5 > Los Alamos National Laboratory > 505-665-8289 > > http://www.ldeo.columbia.edu/~ecoon/ > ------------------------------------ > From agrayver at gfz-potsdam.de Tue Mar 8 02:16:52 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Tue, 08 Mar 2011 09:16:52 +0100 Subject: [petsc-users] loosing preallocation information In-Reply-To: <68DDAB34-3AB8-4321-B6E0-F576E8627A23@mcs.anl.gov> References: <4D73AAC5.3010407@gfz-potsdam.de> <4D74E2AC.9000505@gfz-potsdam.de> <4D74E5E8.5060801@gfz-potsdam.de> <4D74EE8C.8040106@gfz-potsdam.de> <1299512900.24152.4.camel@echo.lanl.gov> <68DDAB34-3AB8-4321-B6E0-F576E8627A23@mcs.anl.gov> Message-ID: <4D75E5F4.3050504@gfz-potsdam.de> Hello Barry, I'm absolutely sure that I passed PETSC_NULL and #include "finclude/petscsys.h" is defined in the beginning of my subroutine. This line write(*,*) PETSC_NULL, " ", sizeof(PETSC_NULL) gives me 0 4 I also was surprised this macro didn't work properly, but I thought it was my misunderstanding of PETSc code. I use IFC 11.1 and ICC 11.1 as compilers under 64x linux. My configline is: ./configure --with-petsc-arch=openmpi-intel-complex-debug --with-mpi-dir=/opt/mpi/intel/openmpi-1.4.2 --with-scalar-type=complex --with-blas-lapack-dir=/opt/intel/Compiler/11.1/072/mkl/lib/em64t --with-precision=double --with-x=0 If you need any information let me know. On 07.03.2011 23:58, Barry Smith wrote: > On Mar 7, 2011, at 9:48 AM, Ethan Coon wrote: > >> On Mon, 2011-03-07 at 15:41 +0100, Alexander Grayver wrote: >>> It works! Great, Matt. >>> >>> I guess there is no other way to know all these small issues but to >>> get all possible errors? :) > This should not have happened. Here is > > void PETSC_STDCALL matcreateseqaij_(MPI_Comm *comm,PetscInt *m,PetscInt *n,PetscInt *nz, > PetscInt *nnz,Mat *newmat,PetscErrorCode *ierr) > { > CHKFORTRANNULLINTEGER(nnz); > *ierr = MatCreateSeqAIJ(MPI_Comm_f2c(*(MPI_Fint *)&*comm),*m,*n,*nz,nnz,newmat); > } > > and here is > > #define CHKFORTRANNULLINTEGER(a) \ > if (FORTRANNULL(a) || FORTRANNULLDOUBLE(a) || FORTRANNULLSCALAR(a) || FORTRANNULLREAL(a) || FORTRANNULLOBJECT(a) || FORTRANNULLFUNCTION(a)) { \ > PetscError(PETSC_COMM_SELF,__LINE__,"fortran_interface_unknown_file",__FILE__,__SDIR__,PETSC_ERR_ARG_WRONG,PETSC_ERROR_INITIAL, \ > "Use PETSC_NULL_INTEGER"); *ierr = 1; return; } \ > else if (FORTRANNULLINTEGER(a)) { a = PETSC_NULL; } > > If you passed in PETSC_NULL instead of PETSC_NULL_INTEGER it should have printed a (slightly helpful) error message right at this point, it should not have gotten to that place later in the code where it stopped. > > Are you sure you passed in PETSC_NULL? And did you include petsc.h or petscsys.h to make sure that PETSC_NULL was defined or did you pass 0 in for nnz? > > I'd like to know so we can fix any PETSc bug there might be there. > > > > Barry > > >> PETSc in Fortran has a lot of oddities due to the difference between the >> languages (this "null pointer" thing and different ways of looking at >> array bounds being the primary two). If you haven't seen it yet, you >> should really look through Chapter 10 of the manual >> >> http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manual.pdf >> >> which explains most of them. Trust me when I say it will save you a lot >> of frustrated time (speaking as someone who learned it the hard way!) >> >> Ethan >> >>> Jed, Matt, thank you a lot! >>> >>> On 07.03.2011 15:35, Matthew Knepley wrote: >>>> On Mon, Mar 7, 2011 at 8:04 AM, Alexander Grayver >>>> wrote: >>>> Jed, but why do I get in line 3003 in >>>> src/mat/impls/aij/seq/aij.c at all? As far as understand, if >>>> I don't pass nnz (NULL in C/C++ or PETSC_NULL in fortran) I >>>> have to avoid this loop in aij.c code. >>>> >>>> >>>> >>>> This is the problem. Fortran does not allow this kind of flexible >>>> checking, so there are different kinds of "NULL"s. You >>>> need to pass PETSC_NULL_INTEGER for nnz. >>>> >>>> >>>> Matt >>>> >>>> I don't have much experience with valgrind. Could you >>>> provide me with right parameters to run my code under >>>> valgrind? >>>> Thanks a lot for your help. >>>> >>>> >>>> On 07.03.2011 14:59, Jed Brown wrote: >>>>> Looks like a memory error. Best bet is to run in Valgrind. >>>>> >>>>>> On Mar 7, 2011 5:52 AM, "Alexander Grayver" >>>>>> wrote: >>>>>> >>>>>> Hi Jed, >>>>>> >>>>>> It's getting even stranger. >>>>>> >>>>>> When I run this code under linux I've got error: >>>>>> >>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>> ------------------------------------ >>>>>> [0]PETSC ERROR: Argument out of range! >>>>>> [0]PETSC ERROR: nnz cannot be greater than row length: >>>>>> local row 98 value 1455203639 rowlength 343! >>>>>> [0]PETSC ERROR: >>>>>> ------------------------------------------------------------------------ >>>>>> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, >>>>>> Mon Dec 20 14:26:37 CST 2010 >>>>>> [0]PETSC ERROR: See docs/changes/index.html for recent >>>>>> updates. >>>>>> [0]PETSC ERROR: See docs/faq.html for hints about >>>>>> trouble shooting. >>>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>>>> [0]PETSC ERROR: >>>>>> ------------------------------------------------------------------------ >>>>>> [0]PETSC >>>>>> ERROR: /home/mt/agrayver/mt-soft/multiem/INV3D/_tiger/em_model on a openmpi-i named glic by agrayver Mon Mar 7 14:49:43 2011 >>>>>> [0]PETSC ERROR: Libraries linked >>>>>> from /panfs/panfs.gfz-hpcc.cluster/home/mt/agrayver/lib/petsc-3.1-p7/openmpi-intel-complex-debug/lib >>>>>> [0]PETSC ERROR: Configure run at Fri Mar 4 12:43:58 >>>>>> 2011 >>>>>> [0]PETSC ERROR: Configure options >>>>>> --with-petsc-arch=openmpi-intel-complex-debug >>>>>> --with-mpi-dir=/opt/mpi/intel/openmpi-1.4.2 >>>>>> --with-scalar-type=complex >>>>>> --with-blas-lapack-dir=/opt/intel/Compiler/11.1/072/mkl/lib/em64t --with-precision=double --with-x=0 >>>>>> [0]PETSC ERROR: >>>>>> ------------------------------------------------------------------------ >>>>>> [0]PETSC ERROR: MatSeqAIJSetPreallocation_SeqAIJ() line >>>>>> 3003 in src/mat/impls/aij/seq/aij.c >>>>>> [0]PETSC ERROR: MatCreateSeqAIJ() line 2906 in >>>>>> src/mat/impls/aij/seq/aij.c >>>>>> >>>>>> >>>>>> Then I traced it under debugger both on Windows and >>>>>> Linux and realized that it doesn't crash under Windows >>>>>> only by luck. >>>>>> >>>>>> The call stack if the folowing: >>>>>> [C] MatSeqAIJSetPreallocation_SeqAIJ, FP=7fffb4513880 >>>>>> [C] MatCreateSeqAIJ, FP=7fffb4513900 >>>>>> [C] matcreateseqaij_, FP=7fffb4513960 >>>>>> [F90] MODELING_MOD`modeling, FP=7fffb4517740 >>>>>> >>>>>> As far as I understand after debugging the problem is >>>>>> that nnz in MatSeqAIJSetPreallocation isn't 0 and I go >>>>>> through this code: >>>>>> if (nnz) { >>>>>> for (i=0; irmap->n; i++) { >>>>>> if (nnz[i]< 0) >>>>>> SETERRQ2(PETSC_ERR_ARG_OUTOFRANGE,"nnz cannot be less >>>>>> than 0: local row %d value %d",i,nnz[i]); >>>>>> if (nnz[i]> B->cmap->n) >>>>>> SETERRQ3(PETSC_ERR_ARG_OUTOFRANGE,"nnz cannot be greater >>>>>> than row length: local row %d value %d rowlength % >>>>>> d",i,nnz[i],B->cmap->n); >>>>>> } >>>>>> } >>>>>> >>>>>> But why is nnz nonzero? I passed PETSC_NULL. It seems >>>>>> like macros CHKFORTRANNULLINTEGER in matcreateseqaij_ >>>>>> should set it to zero, but it doesn't. >>>>>> >>>>>> >>>>>> >>>>>> Thanks. >>>>>> >>>>>> On 06.03.2011 16:49, Jed Brown wrote: >>>>>> >>>>>> >>>>>>> On Sun, Mar 6, 2011 at 07:39, Alexander Grayver >>>>>> wrote: >>>>>>>> Hello, >>>>>>> ... >>>>>> >>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which >>>> their experiments lead. >>>> -- Norbert Wiener >> -- >> ------------------------------------ >> Ethan Coon >> Post-Doctoral Researcher >> Applied Mathematics - T-5 >> Los Alamos National Laboratory >> 505-665-8289 >> >> http://www.ldeo.columbia.edu/~ecoon/ >> ------------------------------------ >> From tomjan at jay.au.poznan.pl Tue Mar 8 02:39:53 2011 From: tomjan at jay.au.poznan.pl (Tomasz Jankowski) Date: Tue, 8 Mar 2011 09:39:53 +0100 (CET) Subject: [petsc-users] how to use MatSetValues? Message-ID: hello, Could someone give me a hand with MatSetValues function? suppose such example sparse matrix 0 1.1 0 0 0 2.2 0 0 0 0 0 0 0 0 3.3 0 0 4.4 0 0 0 0 0 5.5 0 after storing it in row, col and val arrays we have row[0] = 0;col[0]=1;val[0]=1.1; row[1] = 1;col[1]=0;val[1]=2.2; row[2] = 2;col[2]=4;val[2]=3.3; row[3] = 3;col[3]=2;val[3]=4.4; row[3] = 4;col[3]=3;val[4]=5.5; question is: how to use MatSetValues with it? I thought that 'MatSetValues(xx,5,row,1,col,val,INSERT_VALUES);' would be good but it doesn't .(I suppose I don't fully anderstand how it works...) thanks, tom ######################################################## # tomjan at jay.au.poznan.pl # # jay.au.poznan.pl/~tomjan/ # ######################################################## From gianmail at gmail.com Tue Mar 8 02:56:03 2011 From: gianmail at gmail.com (Gianluca Meneghello) Date: Tue, 8 Mar 2011 09:56:03 +0100 Subject: [petsc-users] how to use MatSetValues? In-Reply-To: References: Message-ID: I think m and n are the number of rows and columns you are inserting in the matrix. If you insert one value at a time, that would be MatSetValues(xx,1,row,1,col,val,INSERT_VALUES); I hope it helps Gianluca On 8 March 2011 09:39, Tomasz Jankowski wrote: > hello, > > Could someone give me a hand with MatSetValues function? > > suppose such example sparse matrix > > 0 ? 1.1 0 ? 0 ? 0 > 2.2 0 ? 0 ? 0 ? 0 > 0 ? 0 ? 0 ? 0 ? 3.3 > 0 ? 0 ? 4.4 0 ? 0 > 0 ? 0 ? 0 ? 5.5 0 > > after storing it in row, col and val arrays we have > > row[0] = 0;col[0]=1;val[0]=1.1; > row[1] = 1;col[1]=0;val[1]=2.2; > row[2] = 2;col[2]=4;val[2]=3.3; > row[3] = 3;col[3]=2;val[3]=4.4; > row[3] = 4;col[3]=3;val[4]=5.5; > > question is: how to use MatSetValues with it? > > I thought that 'MatSetValues(xx,5,row,1,col,val,INSERT_VALUES);' would be > good but it doesn't .(I suppose I don't fully anderstand how it works...) > > thanks, > > tom > > ######################################################## > # ? ? ? ? ? ? ? tomjan at jay.au.poznan.pl ? ? ? ? ? ? ? ?# > # ? ? ? ? ? ? ?jay.au.poznan.pl/~tomjan/ ? ? ? ? ? ? ? # > ######################################################## > -- "[Je pense que] l'homme est un monde qui vaut des fois les mondes et que les plus ardentes ambitions sont celles qui ont eu l'orgueil de l'Anonymat" -- Non omnibus, sed mihi et tibi Amedeo Modigliani From tomjan at jay.au.poznan.pl Tue Mar 8 03:16:28 2011 From: tomjan at jay.au.poznan.pl (Tomasz Jankowski) Date: Tue, 8 Mar 2011 10:16:28 +0100 (CET) Subject: [petsc-users] how to use MatSetValues? In-Reply-To: References: Message-ID: > I think m and n are the number of rows and columns you are inserting > in the matrix. If you insert one value at a time, that would be > > MatSetValues(xx,1,row,1,col,val,INSERT_VALUES); > > I hope it helps > > Gianluca hello Gianluca, thanks for re. your sugestion doesn't work. my code Mat xx; PetscInitialize(&argc,&argv,(char *)0,help); MatCreateSeqAIJ(PETSC_COMM_SELF,5,5,PETSC_DECIDE,PETSC_NULL,&xx); PetscInt row [] ={0, 1, 2, 3, 4}; PetscInt col [] = {1, 0, 4, 2, 3}; PetscScalar val[] = {1.1,2.2,3.3,4.4,5.5}; MatSetValues(xx,5,row,5,col,val,INSERT_VALUES); MatAssemblyBegin(xx,MAT_FINAL_ASSEMBLY); MatAssemblyEnd(xx,MAT_FINAL_ASSEMBLY); MatView(xx,PETSC_VIEWER_STDOUT_WORLD); PetscFinalize(); and here few examples of results MatSetValues(xx,1,row,1,col,val,INSERT_VALUES); row 0: (1, 1.1) row 1: row 2: row 3: row 4: MatSetValues(xx,5,row,1,col,val,INSERT_VALUES); row 0: (1, 1.1) row 1: (1, 2.2) row 2: (1, 3.3) row 3: (1, 4.4) row 4: (1, 5.5) MatSetValues(xx,5,row,5,col,val,INSERT_VALUES); row 0: (0, 2.2) (1, 1.1) (2, 4.4) (3, 5.5) (4, 3.3) row 1: (0, 4.94066e-324) (1, 2.07411e-317) (2, 1.4822e-323) (3, 2.07356e-317) (4, 4.24399e-314) row 2: (0, 6.36599e-314) (1, 2.122e-314) (2, 2.07402e-317) (3, 1.5957e-316) (4, 1.18832e-312) row 3: (0, 6.95332e-310) (1, 2.07367e-317) (2, 0) (3, 1.20407e-312) (4, 0) row 4: (0, 6.95332e-310) (1, 0) (2, 2.07378e-317) (3, 0) (4, 2.122e-314) tom ######################################################## # tomjan at jay.au.poznan.pl # # jay.au.poznan.pl/~tomjan/ # ######################################################## On Tue, 8 Mar 2011, Gianluca Meneghello wrote: > I think m and n are the number of rows and columns you are inserting > in the matrix. If you insert one value at a time, that would be > > MatSetValues(xx,1,row,1,col,val,INSERT_VALUES); > > I hope it helps > > Gianluca > > On 8 March 2011 09:39, Tomasz Jankowski wrote: >> hello, >> >> Could someone give me a hand with MatSetValues function? >> >> suppose such example sparse matrix >> >> 0 ? 1.1 0 ? 0 ? 0 >> 2.2 0 ? 0 ? 0 ? 0 >> 0 ? 0 ? 0 ? 0 ? 3.3 >> 0 ? 0 ? 4.4 0 ? 0 >> 0 ? 0 ? 0 ? 5.5 0 >> >> after storing it in row, col and val arrays we have >> >> row[0] = 0;col[0]=1;val[0]=1.1; >> row[1] = 1;col[1]=0;val[1]=2.2; >> row[2] = 2;col[2]=4;val[2]=3.3; >> row[3] = 3;col[3]=2;val[3]=4.4; >> row[3] = 4;col[3]=3;val[4]=5.5; >> >> question is: how to use MatSetValues with it? >> >> I thought that 'MatSetValues(xx,5,row,1,col,val,INSERT_VALUES);' would be >> good but it doesn't .(I suppose I don't fully anderstand how it works...) >> >> thanks, >> >> tom >> >> ######################################################## >> # ? ? ? ? ? ? ? tomjan at jay.au.poznan.pl ? ? ? ? ? ? ? ?# >> # ? ? ? ? ? ? ?jay.au.poznan.pl/~tomjan/ ? ? ? ? ? ? ? # >> ######################################################## >> > > > > -- > "[Je pense que] l'homme est un monde qui vaut des fois les mondes et > que les plus ardentes ambitions sont celles qui ont eu l'orgueil de > l'Anonymat" -- Non omnibus, sed mihi et tibi > Amedeo Modigliani > From gianmail at gmail.com Tue Mar 8 03:23:34 2011 From: gianmail at gmail.com (Gianluca Meneghello) Date: Tue, 8 Mar 2011 10:23:34 +0100 Subject: [petsc-users] how to use MatSetValues? In-Reply-To: References: Message-ID: You should do a loop over the values for (int i = 0; i < 5; i++) MatSetValues(xx,1,row[i],1,col[i],val[i],INSERT_VALUES); On 8 March 2011 10:16, Tomasz Jankowski wrote: >> I think m and n are the number of rows and columns you are inserting >> in the matrix. If you insert one value at a time, that would be >> >> MatSetValues(xx,1,row,1,col,val,INSERT_VALUES); >> >> I hope it helps >> >> Gianluca > > hello Gianluca, thanks for re. > > your sugestion doesn't work. > > my code > > ?Mat xx; > ?PetscInitialize(&argc,&argv,(char *)0,help); > > ?MatCreateSeqAIJ(PETSC_COMM_SELF,5,5,PETSC_DECIDE,PETSC_NULL,&xx); > > ?PetscInt row [] ={0, 1, 2, 3, 4}; > ?PetscInt col [] = {1, 0, 4, 2, 3}; > ?PetscScalar val[] = {1.1,2.2,3.3,4.4,5.5}; > > ?MatSetValues(xx,5,row,5,col,val,INSERT_VALUES); > > ?MatAssemblyBegin(xx,MAT_FINAL_ASSEMBLY); > ?MatAssemblyEnd(xx,MAT_FINAL_ASSEMBLY); > > ?MatView(xx,PETSC_VIEWER_STDOUT_WORLD); > ?PetscFinalize(); > > and here few examples of results > > ?MatSetValues(xx,1,row,1,col,val,INSERT_VALUES); > > row 0: (1, 1.1) > row 1: > row 2: > row 3: > row 4: > > > ?MatSetValues(xx,5,row,1,col,val,INSERT_VALUES); > > row 0: (1, 1.1) > row 1: (1, 2.2) > row 2: (1, 3.3) > row 3: (1, 4.4) > row 4: (1, 5.5) > > ?MatSetValues(xx,5,row,5,col,val,INSERT_VALUES); > > row 0: (0, 2.2) ?(1, 1.1) ?(2, 4.4) ?(3, 5.5) ?(4, 3.3) > row 1: (0, 4.94066e-324) ?(1, 2.07411e-317) ?(2, 1.4822e-323) ?(3, > 2.07356e-317) ?(4, 4.24399e-314) > row 2: (0, 6.36599e-314) ?(1, 2.122e-314) ?(2, 2.07402e-317) ?(3, > 1.5957e-316) ?(4, 1.18832e-312) > row 3: (0, 6.95332e-310) ?(1, 2.07367e-317) ?(2, 0) ?(3, 1.20407e-312) ?(4, > 0) > row 4: (0, 6.95332e-310) ?(1, 0) ?(2, 2.07378e-317) ?(3, 0) ?(4, 2.122e-314) > > tom > > ######################################################## > # ? ? ? ? ? ? ? tomjan at jay.au.poznan.pl ? ? ? ? ? ? ? ?# > # ? ? ? ? ? ? ?jay.au.poznan.pl/~tomjan/ ? ? ? ? ? ? ? # > ######################################################## > > On Tue, 8 Mar 2011, Gianluca Meneghello wrote: > >> I think m and n are the number of rows and columns you are inserting >> in the matrix. If you insert one value at a time, that would be >> >> MatSetValues(xx,1,row,1,col,val,INSERT_VALUES); >> >> I hope it helps >> >> Gianluca >> >> On 8 March 2011 09:39, Tomasz Jankowski wrote: >>> >>> hello, >>> >>> Could someone give me a hand with MatSetValues function? >>> >>> suppose such example sparse matrix >>> >>> 0 ? 1.1 0 ? 0 ? 0 >>> 2.2 0 ? 0 ? 0 ? 0 >>> 0 ? 0 ? 0 ? 0 ? 3.3 >>> 0 ? 0 ? 4.4 0 ? 0 >>> 0 ? 0 ? 0 ? 5.5 0 >>> >>> after storing it in row, col and val arrays we have >>> >>> row[0] = 0;col[0]=1;val[0]=1.1; >>> row[1] = 1;col[1]=0;val[1]=2.2; >>> row[2] = 2;col[2]=4;val[2]=3.3; >>> row[3] = 3;col[3]=2;val[3]=4.4; >>> row[3] = 4;col[3]=3;val[4]=5.5; >>> >>> question is: how to use MatSetValues with it? >>> >>> I thought that 'MatSetValues(xx,5,row,1,col,val,INSERT_VALUES);' would be >>> good but it doesn't .(I suppose I don't fully anderstand how it works...) >>> >>> thanks, >>> >>> tom >>> >>> ######################################################## >>> # ? ? ? ? ? ? ? tomjan at jay.au.poznan.pl ? ? ? ? ? ? ? ?# >>> # ? ? ? ? ? ? ?jay.au.poznan.pl/~tomjan/ ? ? ? ? ? ? ? # >>> ######################################################## >>> >> >> >> >> -- >> "[Je pense que] l'homme est un monde qui vaut des fois les mondes et >> que les plus ardentes ambitions sont celles qui ont eu l'orgueil de >> l'Anonymat" -- Non omnibus, sed mihi et tibi >> Amedeo Modigliani > -- "[Je pense que] l'homme est un monde qui vaut des fois les mondes et que les plus ardentes ambitions sont celles qui ont eu l'orgueil de l'Anonymat" -- Non omnibus, sed mihi et tibi Amedeo Modigliani From tomjan at jay.au.poznan.pl Tue Mar 8 03:38:51 2011 From: tomjan at jay.au.poznan.pl (Tomasz Jankowski) Date: Tue, 8 Mar 2011 10:38:51 +0100 (CET) Subject: [petsc-users] how to use MatSetValues? In-Reply-To: References: Message-ID: > You should do a loop over the values > > for (int i = 0; i < 5; i++) > MatSetValues(xx,1,row[i],1,col[i],val[i],INSERT_VALUES); here is the point Gianluca, I don't want to store one by one values because It will take the ages with hundreds of milions of values(even with preallocation). I would like to set size of row,col and val arrays to e.g. one milion than fill them with data and such prepared large chunk of data pass to MatSetValues. What is suggested by petsc's developers. But it doesn't work properly for me - I dont know what i'm doing wrong... tom > > > On 8 March 2011 10:16, Tomasz Jankowski wrote: >>> I think m and n are the number of rows and columns you are inserting >>> in the matrix. If you insert one value at a time, that would be >>> >>> MatSetValues(xx,1,row,1,col,val,INSERT_VALUES); >>> >>> I hope it helps >>> >>> Gianluca >> >> hello Gianluca, thanks for re. >> >> your sugestion doesn't work. >> >> my code >> >> ?Mat xx; >> ?PetscInitialize(&argc,&argv,(char *)0,help); >> >> ?MatCreateSeqAIJ(PETSC_COMM_SELF,5,5,PETSC_DECIDE,PETSC_NULL,&xx); >> >> ?PetscInt row [] ={0, 1, 2, 3, 4}; >> ?PetscInt col [] = {1, 0, 4, 2, 3}; >> ?PetscScalar val[] = {1.1,2.2,3.3,4.4,5.5}; >> >> ?MatSetValues(xx,5,row,5,col,val,INSERT_VALUES); >> >> ?MatAssemblyBegin(xx,MAT_FINAL_ASSEMBLY); >> ?MatAssemblyEnd(xx,MAT_FINAL_ASSEMBLY); >> >> ?MatView(xx,PETSC_VIEWER_STDOUT_WORLD); >> ?PetscFinalize(); >> >> and here few examples of results >> >> ?MatSetValues(xx,1,row,1,col,val,INSERT_VALUES); >> >> row 0: (1, 1.1) >> row 1: >> row 2: >> row 3: >> row 4: >> >> >> ?MatSetValues(xx,5,row,1,col,val,INSERT_VALUES); >> >> row 0: (1, 1.1) >> row 1: (1, 2.2) >> row 2: (1, 3.3) >> row 3: (1, 4.4) >> row 4: (1, 5.5) >> >> ?MatSetValues(xx,5,row,5,col,val,INSERT_VALUES); >> >> row 0: (0, 2.2) ?(1, 1.1) ?(2, 4.4) ?(3, 5.5) ?(4, 3.3) >> row 1: (0, 4.94066e-324) ?(1, 2.07411e-317) ?(2, 1.4822e-323) ?(3, >> 2.07356e-317) ?(4, 4.24399e-314) >> row 2: (0, 6.36599e-314) ?(1, 2.122e-314) ?(2, 2.07402e-317) ?(3, >> 1.5957e-316) ?(4, 1.18832e-312) >> row 3: (0, 6.95332e-310) ?(1, 2.07367e-317) ?(2, 0) ?(3, 1.20407e-312) ?(4, >> 0) >> row 4: (0, 6.95332e-310) ?(1, 0) ?(2, 2.07378e-317) ?(3, 0) ?(4, 2.122e-314) >> >> tom >> >> ######################################################## >> # ? ? ? ? ? ? ? tomjan at jay.au.poznan.pl ? ? ? ? ? ? ? ?# >> # ? ? ? ? ? ? ?jay.au.poznan.pl/~tomjan/ ? ? ? ? ? ? ? # >> ######################################################## >> >> On Tue, 8 Mar 2011, Gianluca Meneghello wrote: >> >>> I think m and n are the number of rows and columns you are inserting >>> in the matrix. If you insert one value at a time, that would be >>> >>> MatSetValues(xx,1,row,1,col,val,INSERT_VALUES); >>> >>> I hope it helps >>> >>> Gianluca >>> >>> On 8 March 2011 09:39, Tomasz Jankowski wrote: >>>> >>>> hello, >>>> >>>> Could someone give me a hand with MatSetValues function? >>>> >>>> suppose such example sparse matrix >>>> >>>> 0 ? 1.1 0 ? 0 ? 0 >>>> 2.2 0 ? 0 ? 0 ? 0 >>>> 0 ? 0 ? 0 ? 0 ? 3.3 >>>> 0 ? 0 ? 4.4 0 ? 0 >>>> 0 ? 0 ? 0 ? 5.5 0 >>>> >>>> after storing it in row, col and val arrays we have >>>> >>>> row[0] = 0;col[0]=1;val[0]=1.1; >>>> row[1] = 1;col[1]=0;val[1]=2.2; >>>> row[2] = 2;col[2]=4;val[2]=3.3; >>>> row[3] = 3;col[3]=2;val[3]=4.4; >>>> row[3] = 4;col[3]=3;val[4]=5.5; >>>> >>>> question is: how to use MatSetValues with it? >>>> >>>> I thought that 'MatSetValues(xx,5,row,1,col,val,INSERT_VALUES);' would be >>>> good but it doesn't .(I suppose I don't fully anderstand how it works...) >>>> >>>> thanks, >>>> >>>> tom >>>> >>>> ######################################################## >>>> # ? ? ? ? ? ? ? tomjan at jay.au.poznan.pl ? ? ? ? ? ? ? ?# >>>> # ? ? ? ? ? ? ?jay.au.poznan.pl/~tomjan/ ? ? ? ? ? ? ? # >>>> ######################################################## >>>> >>> >>> >>> >>> -- >>> "[Je pense que] l'homme est un monde qui vaut des fois les mondes et >>> que les plus ardentes ambitions sont celles qui ont eu l'orgueil de >>> l'Anonymat" -- Non omnibus, sed mihi et tibi >>> Amedeo Modigliani >> > > > > -- > "[Je pense que] l'homme est un monde qui vaut des fois les mondes et > que les plus ardentes ambitions sont celles qui ont eu l'orgueil de > l'Anonymat" -- Non omnibus, sed mihi et tibi > Amedeo Modigliani > From gianmail at gmail.com Tue Mar 8 04:02:43 2011 From: gianmail at gmail.com (Gianluca Meneghello) Date: Tue, 8 Mar 2011 11:02:43 +0100 Subject: [petsc-users] how to use MatSetValues? In-Reply-To: References: Message-ID: I'm not an expert, but I think you have to provide two vector for row and cols and a matrix for values corresponding to all combinations of rows and cols. The entry v is required to be a "logically two dimensional array of values", so this is my understanding. If this is true, in your case it would require to pass the whole full matrix in v, zeros included, given that you have 1 value per row and 1 value per col. That said, I have no idea on whether the zeros are then taken away or not once the matrix is assembled. You may insert "block" of values, but I still think each block is required to be a full rectangular matrix. There can alway be the possibility that I'm completely wrong about what I'm saying... this is how I used it so far. Something I'm doing is to set the values line by line by using MatSetValues(xx,1,row,ncols,col,val,INSERT_VALUES); where row has 1 elements, cols and val ncols elements. Actually, I asked a similar question some time ago, and I was redirected to the possibility of assembling the matrix accessing directly the private structure of the matrix. See ~petsc/src/mat/examples/tutorials/ex12.c for seqaij format. I hope this helps Gianluca On 8 March 2011 10:38, Tomasz Jankowski wrote: >> You should do a loop over the values >> >> for (int i = 0; i < 5; i++) >> MatSetValues(xx,1,row[i],1,col[i],val[i],INSERT_VALUES); > > here is the point Gianluca, > I don't want to store one by one values because It will take the ages with > hundreds of milions of values(even with preallocation). I would like to set > size of row,col and val arrays to e.g. one milion than fill them with > data and such prepared large chunk of data pass to MatSetValues. What is > suggested by petsc's developers. But it doesn't work properly for me - I > dont know what i'm doing wrong... > > tom > >> >> >> On 8 March 2011 10:16, Tomasz Jankowski wrote: >>>> >>>> I think m and n are the number of rows and columns you are inserting >>>> in the matrix. If you insert one value at a time, that would be >>>> >>>> MatSetValues(xx,1,row,1,col,val,INSERT_VALUES); >>>> >>>> I hope it helps >>>> >>>> Gianluca >>> >>> hello Gianluca, thanks for re. >>> >>> your sugestion doesn't work. >>> >>> my code >>> >>> ?Mat xx; >>> ?PetscInitialize(&argc,&argv,(char *)0,help); >>> >>> ?MatCreateSeqAIJ(PETSC_COMM_SELF,5,5,PETSC_DECIDE,PETSC_NULL,&xx); >>> >>> ?PetscInt row [] ={0, 1, 2, 3, 4}; >>> ?PetscInt col [] = {1, 0, 4, 2, 3}; >>> ?PetscScalar val[] = {1.1,2.2,3.3,4.4,5.5}; >>> >>> ?MatSetValues(xx,5,row,5,col,val,INSERT_VALUES); >>> >>> ?MatAssemblyBegin(xx,MAT_FINAL_ASSEMBLY); >>> ?MatAssemblyEnd(xx,MAT_FINAL_ASSEMBLY); >>> >>> ?MatView(xx,PETSC_VIEWER_STDOUT_WORLD); >>> ?PetscFinalize(); >>> >>> and here few examples of results >>> >>> ?MatSetValues(xx,1,row,1,col,val,INSERT_VALUES); >>> >>> row 0: (1, 1.1) >>> row 1: >>> row 2: >>> row 3: >>> row 4: >>> >>> >>> ?MatSetValues(xx,5,row,1,col,val,INSERT_VALUES); >>> >>> row 0: (1, 1.1) >>> row 1: (1, 2.2) >>> row 2: (1, 3.3) >>> row 3: (1, 4.4) >>> row 4: (1, 5.5) >>> >>> ?MatSetValues(xx,5,row,5,col,val,INSERT_VALUES); >>> >>> row 0: (0, 2.2) ?(1, 1.1) ?(2, 4.4) ?(3, 5.5) ?(4, 3.3) >>> row 1: (0, 4.94066e-324) ?(1, 2.07411e-317) ?(2, 1.4822e-323) ?(3, >>> 2.07356e-317) ?(4, 4.24399e-314) >>> row 2: (0, 6.36599e-314) ?(1, 2.122e-314) ?(2, 2.07402e-317) ?(3, >>> 1.5957e-316) ?(4, 1.18832e-312) >>> row 3: (0, 6.95332e-310) ?(1, 2.07367e-317) ?(2, 0) ?(3, 1.20407e-312) >>> ?(4, >>> 0) >>> row 4: (0, 6.95332e-310) ?(1, 0) ?(2, 2.07378e-317) ?(3, 0) ?(4, >>> 2.122e-314) >>> >>> tom >>> >>> ######################################################## >>> # ? ? ? ? ? ? ? tomjan at jay.au.poznan.pl ? ? ? ? ? ? ? ?# >>> # ? ? ? ? ? ? ?jay.au.poznan.pl/~tomjan/ ? ? ? ? ? ? ? # >>> ######################################################## >>> >>> On Tue, 8 Mar 2011, Gianluca Meneghello wrote: >>> >>>> I think m and n are the number of rows and columns you are inserting >>>> in the matrix. If you insert one value at a time, that would be >>>> >>>> MatSetValues(xx,1,row,1,col,val,INSERT_VALUES); >>>> >>>> I hope it helps >>>> >>>> Gianluca >>>> >>>> On 8 March 2011 09:39, Tomasz Jankowski wrote: >>>>> >>>>> hello, >>>>> >>>>> Could someone give me a hand with MatSetValues function? >>>>> >>>>> suppose such example sparse matrix >>>>> >>>>> 0 ? 1.1 0 ? 0 ? 0 >>>>> 2.2 0 ? 0 ? 0 ? 0 >>>>> 0 ? 0 ? 0 ? 0 ? 3.3 >>>>> 0 ? 0 ? 4.4 0 ? 0 >>>>> 0 ? 0 ? 0 ? 5.5 0 >>>>> >>>>> after storing it in row, col and val arrays we have >>>>> >>>>> row[0] = 0;col[0]=1;val[0]=1.1; >>>>> row[1] = 1;col[1]=0;val[1]=2.2; >>>>> row[2] = 2;col[2]=4;val[2]=3.3; >>>>> row[3] = 3;col[3]=2;val[3]=4.4; >>>>> row[3] = 4;col[3]=3;val[4]=5.5; >>>>> >>>>> question is: how to use MatSetValues with it? >>>>> >>>>> I thought that 'MatSetValues(xx,5,row,1,col,val,INSERT_VALUES);' would >>>>> be >>>>> good but it doesn't .(I suppose I don't fully anderstand how it >>>>> works...) >>>>> >>>>> thanks, >>>>> >>>>> tom >>>>> >>>>> ######################################################## >>>>> # ? ? ? ? ? ? ? tomjan at jay.au.poznan.pl ? ? ? ? ? ? ? ?# >>>>> # ? ? ? ? ? ? ?jay.au.poznan.pl/~tomjan/ ? ? ? ? ? ? ? # >>>>> ######################################################## >>>>> >>>> >>>> >>>> >>>> -- >>>> "[Je pense que] l'homme est un monde qui vaut des fois les mondes et >>>> que les plus ardentes ambitions sont celles qui ont eu l'orgueil de >>>> l'Anonymat" -- Non omnibus, sed mihi et tibi >>>> Amedeo Modigliani >>> >> >> >> >> -- >> "[Je pense que] l'homme est un monde qui vaut des fois les mondes et >> que les plus ardentes ambitions sont celles qui ont eu l'orgueil de >> l'Anonymat" -- Non omnibus, sed mihi et tibi >> Amedeo Modigliani > -- "[Je pense que] l'homme est un monde qui vaut des fois les mondes et que les plus ardentes ambitions sont celles qui ont eu l'orgueil de l'Anonymat" -- Non omnibus, sed mihi et tibi Amedeo Modigliani From tim.kroeger at cevis.uni-bremen.de Tue Mar 8 04:09:18 2011 From: tim.kroeger at cevis.uni-bremen.de (Tim Kroeger) Date: Tue, 8 Mar 2011 11:09:18 +0100 Subject: [petsc-users] Condition number of a matrix Message-ID: Dear PETSc team, Is there some possibility in PETSc to compute (an approximation to) the condition number of a matrix? The matrix is a rank-1 modification of a sparse matrix, and I am anyway using SuperLU_Dist to compute a factorization of the matrix. The matrix is about 70000 times 70000. Best Regards, Tim -- Dr. Tim Kroeger CeVis -- Center of Complex Systems and Visualization University of Bremen tim.kroeger at cevis.uni-bremen.de Universitaetsallee 29 tim.kroeger at mevis.fraunhofer.de D-28359 Bremen Phone +49-421-218-59246 Germany Fax +49-421-218-59277 From tomjan at jay.au.poznan.pl Tue Mar 8 04:38:58 2011 From: tomjan at jay.au.poznan.pl (Tomasz Jankowski) Date: Tue, 8 Mar 2011 11:38:58 +0100 (CET) Subject: [petsc-users] Reading matrices into PETSc In-Reply-To: <4D11C5C9.1050805@tudelft.nl> References: <4D11C5C9.1050805@tudelft.nl> Message-ID: > Matlab pseudo code > A = diag([1 1 1]) > > So table is sth like this > i j val > 0 0 1 > 1 1 1 > 2 2 1 > > see this > > Mat A; // of course, A should be created first and assembled after the set > operation > int m =3; > int n = 3; > PetscInt indxm [] ={0, 1, 2}; > PetscInt indxn [] = {0, 1, 2}; > PetscScalar vals[] = {1.,1.,1.} > MatSetValues(A, m, idxm, n, idxn, vals, INSERT_VALUES); you example doen't work the result matrix is: row 0: (0, 1.1) (1, 1.2) (2, 1.3) row 1: (0, 2.07356e-317) (1, 2.122e-314) (2, 9.88131e-324) row 2: (0, 2.122e-314) (1, 9.88131e-324) (2, 1.40122e-316) tom here is the code for this example Mat xx; PetscInitialize(&argc,&argv,(char *)0,help); MatCreateSeqAIJ(PETSC_COMM_SELF,3,3,1,PETSC_NULL,&xx); int m =3; int n = 3; PetscInt indxm [] ={0, 1, 2}; PetscInt indxn [] = {0, 1, 2}; PetscScalar vals[] = {1.1,1.2,1.3}; MatSetValues(xx,m,indxm,n,indxn,vals,INSERT_VALUES); MatAssemblyBegin(xx,MAT_FINAL_ASSEMBLY); MatAssemblyEnd(xx,MAT_FINAL_ASSEMBLY); MatView(xx,PETSC_VIEWER_STDOUT_WORLD); PetscFinalize(); return EXIT_SUCCESS; From tomjan at jay.au.poznan.pl Tue Mar 8 05:13:48 2011 From: tomjan at jay.au.poznan.pl (Tomasz Jankowski) Date: Tue, 8 Mar 2011 12:13:48 +0100 (CET) Subject: [petsc-users] how to use MatSetValues? In-Reply-To: References: Message-ID: > I'm not an expert, but I think you have to provide two vector for row > and cols and a matrix for values corresponding to all combinations of > rows and cols. The entry v is required to be a "logically two > dimensional array of values", so this is my understanding. > > If this is true, in your case it would require to pass the whole full > matrix in v, zeros included, given that you have 1 value per row and 1 > value per col. That said, I have no idea on whether the zeros are then > taken away or not once the matrix is assembled. You may insert "block" > of values, but I still think each block is required to be a full > rectangular matrix. > > There can alway be the possibility that I'm completely wrong about > what I'm saying... this is how I used it so far. It seems that you are right Gianluca, int m =3; int n = 3; PetscInt indxm [] ={0, 1, 2}; PetscInt indxn [] = {0, 1, 2}; PetscScalar vals[][3] = {{1.1,0,0},{0,1.2,0},{0,0,1.3}}; MatSetValues(xx,m,indxm,n,indxn,*vals,INSERT_VALUES); only such code gives expected result row 0: (0, 1.1) (1, 0) (2, 0) row 1: (0, 0) (1, 1.2) (2, 0) row 2: (0, 0) (1, 0) (2, 1.3) > Something I'm doing is to set the values line by line by using > MatSetValues(xx,1,row,ncols,col,val,INSERT_VALUES); > where row has 1 elements, cols and val ncols elements. I'm building 100mln x 100mln matrices and bigger. what means that with linebyline scheme i have to 100mln times allocate array with size equal to 100mln with only few nonzero entries. not efficient too much i think... > Actually, I asked a similar question some time ago, and I was > redirected to the possibility of assembling the matrix accessing > directly the private structure of the matrix. > See ~petsc/src/mat/examples/tutorials/ex12.c for seqaij format. maybe it will be the solution... 'll check it again, thanks Gianluca for posting. tom From gianmail at gmail.com Tue Mar 8 05:21:31 2011 From: gianmail at gmail.com (Gianluca Meneghello) Date: Tue, 8 Mar 2011 12:21:31 +0100 Subject: [petsc-users] how to use MatSetValues? In-Reply-To: References: Message-ID: > I'm building 100mln x 100mln matrices and bigger. what means that with > linebyline scheme i have to 100mln times allocate array with size equal to > 100mln with only few nonzero entries. > not efficient too much i think... You don't have to allocate a full array for each line, just an array with the nonzero elements and a corresponding array with the row indices. That is, to store a line like [ 0 0 0 1 0 0 2 0 0 0 3 0 0] the two array would be icols[] = { 3 , 6, 10 }; ivals[] = { 1 , 2, 3}; and the call for the value would be MatSetValues(xx,1,irow,3,icols,*ivals,INSERT_VALUES); If there are multiple lines with the same nonzero structure they can be stored together. Gianluca On 8 March 2011 12:13, Tomasz Jankowski wrote: >> I'm not an expert, but I think you have to provide two vector for row >> and cols and a matrix for values corresponding to all combinations of >> rows and cols. The entry v is required to be a "logically two >> dimensional array of values", so this is my understanding. >> >> If this is true, in your case it would require to pass the whole full >> matrix in v, zeros included, given that you have 1 value per row and 1 >> value per col. That said, I have no idea on whether the zeros are then >> taken away or not once the matrix is assembled. You may insert "block" >> of values, but I still think each block is required to be a full >> rectangular matrix. >> >> There can alway be the possibility that I'm completely wrong about >> what I'm saying... this is how I used it so far. > > It seems that you are right Gianluca, > ?int m =3; > ?int n = 3; > ?PetscInt indxm [] ={0, 1, 2}; > ?PetscInt indxn [] = {0, 1, 2}; > ?PetscScalar vals[][3] = {{1.1,0,0},{0,1.2,0},{0,0,1.3}}; > ?MatSetValues(xx,m,indxm,n,indxn,*vals,INSERT_VALUES); > > only such code gives expected result > row 0: (0, 1.1) ?(1, 0) ?(2, 0) > row 1: (0, 0) ?(1, 1.2) ?(2, 0) > row 2: (0, 0) ?(1, 0) ?(2, 1.3) > > >> Something I'm doing is to set the values line by line by using >> MatSetValues(xx,1,row,ncols,col,val,INSERT_VALUES); >> where row has 1 elements, cols and val ncols elements. > > I'm building 100mln x 100mln matrices and bigger. what means that with > linebyline scheme i have to 100mln times allocate array with size equal to > 100mln with only few nonzero entries. > not efficient too much i think... > > >> Actually, I asked a similar question some time ago, and I was >> redirected to the possibility of assembling the matrix accessing >> directly the private structure of the matrix. >> See ~petsc/src/mat/examples/tutorials/ex12.c for seqaij format. > > maybe it will be the solution... 'll check it > > again, thanks Gianluca for posting. > > tom > > -- "[Je pense que] l'homme est un monde qui vaut des fois les mondes et que les plus ardentes ambitions sont celles qui ont eu l'orgueil de l'Anonymat" -- Non omnibus, sed mihi et tibi Amedeo Modigliani From jroman at dsic.upv.es Tue Mar 8 05:35:22 2011 From: jroman at dsic.upv.es (Jose E. Roman) Date: Tue, 8 Mar 2011 12:35:22 +0100 Subject: [petsc-users] Condition number of a matrix In-Reply-To: References: Message-ID: <3649F076-CE75-42B0-A069-D4A56CF2197A@dsic.upv.es> On 08/03/2011, Tim Kroeger wrote: > Dear PETSc team, > > Is there some possibility in PETSc to compute (an approximation to) the condition number of a matrix? The matrix is a rank-1 modification of a sparse matrix, and I am anyway using SuperLU_Dist to compute a factorization of the matrix. The matrix is about 70000 times 70000. > > Best Regards, > > Tim > > -- > Dr. Tim Kroeger > CeVis -- Center of Complex Systems and Visualization > University of Bremen tim.kroeger at cevis.uni-bremen.de > Universitaetsallee 29 tim.kroeger at mevis.fraunhofer.de > D-28359 Bremen Phone +49-421-218-59246 > Germany Fax +49-421-218-59277 You can compute the condition number as the ratio of the largest to the smallest singular value (or eigenvalue in case of symmetric matrices). With SLEPc you can compute either eigenvalues or singular values. Here is an example that computes the condition number: http://www.grycap.upv.es/slepc/documentation/current/src/examples/ex8.c.html PETSc itself provides some tools for approximating the extreme singular values of the preconditioned operator, see KSPComputeExtremeSingularValues. Jose From tomjan at jay.au.poznan.pl Tue Mar 8 05:58:39 2011 From: tomjan at jay.au.poznan.pl (Tomasz Jankowski) Date: Tue, 8 Mar 2011 12:58:39 +0100 (CET) Subject: [petsc-users] how to use MatSetValues? In-Reply-To: References: Message-ID: >> I'm building 100mln x 100mln matrices and bigger. what means that with >> linebyline scheme i have to 100mln times allocate array with size equal to >> 100mln with only few nonzero entries. >> not efficient too much i think... > > You don't have to allocate a full array for each line, just an array > with the nonzero elements and a corresponding array with the row > indices. > > That is, to store a line like > [ 0 0 0 1 0 0 2 0 0 0 3 0 0] > > the two array would be > > icols[] = { 3 , 6, 10 }; > ivals[] = { 1 , 2, 3}; > > and the call for the value would be > > MatSetValues(xx,1,irow,3,icols,*ivals,INSERT_VALUES); > > If there are multiple lines with the same nonzero structure they can > be stored together. yes, yes. of course you are absolutely right. I did a mistake here. But I was only my supposedly thinking about 'linebyline' scheme. In fact each entry of my matrix 'appear' (when the data file is processed) in random order. so, it seems that with MatSetValues I could only use 'onebyone' scheme :-( i hope I will find how to access private structure of matrix. tom From hung.thanh.nguyen at petrell.no Tue Mar 8 06:44:33 2011 From: hung.thanh.nguyen at petrell.no (Hung Thanh Nguyen) Date: Tue, 08 Mar 2011 13:44:33 +0100 Subject: [petsc-users] How to install Petsc-3.1-p7 with window-intel-mkl_blas95.lib -mkl_lapack95 and --mkl_blacs_intelmpi.lip or mkl_blacs_mpich2.lib Message-ID: Dear PETSc team I try to install PETSc on window Intel MKL and include mkl_blas95.lib, mkl_lapack95.lib and mkl_blacs_intelmpi.lib ( or mkl_mpich2.lib). Intel MKL compiler 11.6-067. The following config-procedure ..... $ ./config/configure.py / --with-petsc-arch=intel-32-static / --with-f-blas95-lapack-dir=/intel/Compiler/11.1/067/mkl/ia32/lib / --with-mpich2-dir= =/intel/Compiler/11.1/067/mkl/ia32/lib I am also try: $ ./config/configure.py / --with-petsc-arch=intel-32-static / --with-f-blas-lapack-dir=/intel/Compiler/11.1/067/mkl/ia32/lib / --with-mpi-dir= =/intel/Compiler/11.1/067/mkl/ia32/lib I got error message: Could not find a functional BLAS. Run with -with-blas-lib= to indicate the library containing BLAS ... What I am doing wrong? Hung T Nguyen Physis and techonology Inst. University in Troms? -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.kroeger at cevis.uni-bremen.de Tue Mar 8 07:10:50 2011 From: tim.kroeger at cevis.uni-bremen.de (Tim Kroeger) Date: Tue, 8 Mar 2011 14:10:50 +0100 Subject: [petsc-users] Condition number of a matrix In-Reply-To: <3649F076-CE75-42B0-A069-D4A56CF2197A@dsic.upv.es> References: <3649F076-CE75-42B0-A069-D4A56CF2197A@dsic.upv.es> Message-ID: On Tue, 8 Mar 2011, Jose E. Roman wrote: > On 08/03/2011, Tim Kroeger wrote: > >> Is there some possibility in PETSc to compute (an approximation to) >> the condition number of a matrix? The matrix is a rank-1 >> modification of a sparse matrix, and I am anyway using SuperLU_Dist >> to compute a factorization of the matrix. The matrix is about >> 70000 times 70000. > > You can compute the condition number as the ratio of the largest to > the smallest singular value (or eigenvalue in case of symmetric > matrices). With SLEPc you can compute either eigenvalues or singular > values. Here is an example that computes the condition number: > http://www.grycap.upv.es/slepc/documentation/current/src/examples/ex8.c.html > > PETSc itself provides some tools for approximating the extreme > singular values of the preconditioned operator, see > KSPComputeExtremeSingularValues. Thank you for your help. Following the notes in the manual page for KSPComputeExtremeSingularValues(), I tried -ksp_monitor_singular_value as a first approach, but this causes my application to crash. I should say that the matrix is a ShellMat, consisting of a sum of a rank-1 matrix and a sparse matrix, where the sparse matrix has been factored by SuperLU_Dist. (I think this was somehow misleading in my previous email.) I suspect that my ShellMat misses some operation that is required, but I have no idea which that is. Could you please let me know which operations are required for a ShellMat to enable -ksp_monitor_singular_value? Best Regards, Tim -- Dr. Tim Kroeger CeVis -- Center of Complex Systems and Visualization University of Bremen tim.kroeger at cevis.uni-bremen.de Universitaetsallee 29 tim.kroeger at mevis.fraunhofer.de D-28359 Bremen Phone +49-421-218-59246 Germany Fax +49-421-218-59277 From knepley at gmail.com Tue Mar 8 07:14:53 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 8 Mar 2011 07:14:53 -0600 Subject: [petsc-users] Reading matrices into PETSc In-Reply-To: References: <4D11C5C9.1050805@tudelft.nl> Message-ID: On Tue, Mar 8, 2011 at 4:38 AM, Tomasz Jankowski wrote: > Matlab pseudo code >> A = diag([1 1 1]) >> >> So table is sth like this >> i j val >> 0 0 1 >> 1 1 1 >> 2 2 1 >> >> see this >> >> Mat A; // of course, A should be created first and assembled after the set >> operation >> int m =3; >> int n = 3; >> PetscInt indxm [] ={0, 1, 2}; >> PetscInt indxn [] = {0, 1, 2}; >> PetscScalar vals[] = {1.,1.,1.} >> MatSetValues(A, m, idxm, n, idxn, vals, INSERT_VALUES); >> > > > you example doen't work > the result matrix is: > > row 0: (0, 1.1) (1, 1.2) (2, 1.3) > row 1: (0, 2.07356e-317) (1, 2.122e-314) (2, 9.88131e-324) > row 2: (0, 2.122e-314) (1, 9.88131e-324) (2, 1.40122e-316) > > tom > > here is the code for this example > > Mat xx; > PetscInitialize(&argc,&argv,(char *)0,help); > > MatCreateSeqAIJ(PETSC_COMM_SELF,3,3,1,PETSC_NULL,&xx); > > int m =3; > int n = 3; > ^^^^^^^^^^ This is wrong. You are claiming to provide a m x n array of values, but you only give 3, not 9 values. If you want a diagonal matrix, either change to MatSetDiagonal(), or use m = 1; n = 1; MatSetValues(xx,m,indxm,n,indxn,vals,INSERT_VALUES); MatSetValues(xx,m,&indxm[1],n,&indxn[1],&vals[1],INSERT_VALUES); MatSetValues(xx,m,&indxm[2],n,&indxn[2],&vals[2],INSERT_VALUES); You set rows of values, or logically dense matrices of values (like element matrices) at a time. Matt > PetscInt indxm [] ={0, 1, 2}; > PetscInt indxn [] = {0, 1, 2}; > PetscScalar vals[] = {1.1,1.2,1.3}; > > MatSetValues(xx,m,indxm,n,indxn,vals,INSERT_VALUES); > > MatAssemblyBegin(xx,MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(xx,MAT_FINAL_ASSEMBLY); > > MatView(xx,PETSC_VIEWER_STDOUT_WORLD); > PetscFinalize(); > > return EXIT_SUCCESS; > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Mar 8 07:16:23 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 8 Mar 2011 07:16:23 -0600 Subject: [petsc-users] Condition number of a matrix In-Reply-To: References: <3649F076-CE75-42B0-A069-D4A56CF2197A@dsic.upv.es> Message-ID: On Tue, Mar 8, 2011 at 7:10 AM, Tim Kroeger wrote: > On Tue, 8 Mar 2011, Jose E. Roman wrote: > > On 08/03/2011, Tim Kroeger wrote: >> >> Is there some possibility in PETSc to compute (an approximation to) the >>> condition number of a matrix? The matrix is a rank-1 modification of a >>> sparse matrix, and I am anyway using SuperLU_Dist to compute a factorization >>> of the matrix. The matrix is about 70000 times 70000. >>> >> >> You can compute the condition number as the ratio of the largest to the >> smallest singular value (or eigenvalue in case of symmetric matrices). With >> SLEPc you can compute either eigenvalues or singular values. Here is an >> example that computes the condition number: >> http://www.grycap.upv.es/slepc/documentation/current/src/examples/ex8.c.html >> >> PETSc itself provides some tools for approximating the extreme singular >> values of the preconditioned operator, see KSPComputeExtremeSingularValues. >> > > Thank you for your help. Following the notes in the manual page for > KSPComputeExtremeSingularValues(), I tried -ksp_monitor_singular_value as a > first approach, but this causes my application to crash. I should say that > the matrix is a ShellMat, consisting of a sum of a rank-1 matrix and a > sparse matrix, where the sparse matrix has been factored by SuperLU_Dist. > (I think this was somehow misleading in my previous email.) I suspect that > my ShellMat misses some operation that is required, but I have no idea which > that is. Could you please let me know which operations are required for a > ShellMat to enable -ksp_monitor_singular_value? > It does not have to do with a shell matrix. It uses the projection of the operator into the Krylov space in the solver. If there is a problem, mail all the error output (and possibly the stack trace) to petsc-maint. Matt > Best Regards, > > Tim > > -- > Dr. Tim Kroeger > CeVis -- Center of Complex Systems and Visualization > University of Bremen tim.kroeger at cevis.uni-bremen.de > Universitaetsallee 29 tim.kroeger at mevis.fraunhofer.de > D-28359 Bremen Phone +49-421-218-59246 > Germany Fax +49-421-218-59277 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Mar 8 07:25:27 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 8 Mar 2011 07:25:27 -0600 Subject: [petsc-users] How to install Petsc-3.1-p7 with window-intel-mkl_blas95.lib -mkl_lapack95 and --mkl_blacs_intelmpi.lip or mkl_blacs_mpich2.lib In-Reply-To: References: Message-ID: We cannot see what happened without the log. Please send configure.log to petsc-maint at mcs.anl.gov. Matt On Tue, Mar 8, 2011 at 6:44 AM, Hung Thanh Nguyen < hung.thanh.nguyen at petrell.no> wrote: > Dear PETSc team > > I try to install PETSc on window Intel MKL and include mkl_blas95.lib, > mkl_lapack95.lib and mkl_blacs_intelmpi.lib ( or mkl_mpich2.lib). > > Intel MKL compiler 11.6-067. The following config-procedure > > ?.. > > *$ ./config/configure.py / --with-petsc-arch=intel-32-static / > --with-f-blas95-lapack-dir=/intel/Compiler/11.1/067/mkl/ia32/lib / > --with-mpich2-dir=* > > *=/intel/Compiler/11.1/067/mkl/ia32/lib* > > > > I am also try: > > *$ ./config/configure.py / --with-petsc-arch=intel-32-static / > --with-f-blas-lapack-dir=/intel/Compiler/11.1/067/mkl/ia32/lib / > --with-mpi-dir=* > > *=/intel/Compiler/11.1/067/mkl/ia32/lib* > > > > *I got error message:* > > *Could not find a functional BLAS. Run with ?with-blas-lib= to > indicate the library containing BLAS ? * > > * * > > What I am doing wrong? > > > > Hung T Nguyen > > Physis and techonology Inst. University in Troms? > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Mar 8 07:40:52 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 8 Mar 2011 07:40:52 -0600 Subject: [petsc-users] how to use MatSetValues? In-Reply-To: References: Message-ID: <35CFDA08-4EAB-4B5F-AFB8-36AAEC9108C1@mcs.anl.gov> On Mar 8, 2011, at 5:58 AM, Tomasz Jankowski wrote: >>> I'm building 100mln x 100mln matrices and bigger. what means that with >>> linebyline scheme i have to 100mln times allocate array with size equal to >>> 100mln with only few nonzero entries. >>> not efficient too much i think... >> >> You don't have to allocate a full array for each line, just an array >> with the nonzero elements and a corresponding array with the row >> indices. >> >> That is, to store a line like >> [ 0 0 0 1 0 0 2 0 0 0 3 0 0] >> >> the two array would be >> >> icols[] = { 3 , 6, 10 }; >> ivals[] = { 1 , 2, 3}; >> >> and the call for the value would be >> >> MatSetValues(xx,1,irow,3,icols,*ivals,INSERT_VALUES); >> >> If there are multiple lines with the same nonzero structure they can >> be stored together. > > yes, yes. of course you are absolutely right. > I did a mistake here. But I was only my supposedly thinking about 'linebyline' scheme. In fact each entry of my matrix 'appear' (when the data file is processed) in random order. so, it seems that with MatSetValues I could only use 'onebyone' scheme :-( > If you are reading in the data from an ASCII file then (1) you should sweep through the file twice, the first time counting the number of nonzeros per row, then do proper preallocation of the matrix with MatCreateSeqAIJ() then (2) sweep through again calling the MatSetValues(). You should then call MatView() with a binary viewer and save the matrix in binary format. Now your "real" PETSc code would use MatLoad() to quickly load the binary file (either in parallel or sequential) to do whatever calculation you want to do. Barry The "extra" cost of two sweeps through the file and using the binary file will save you HUGE amounts of time, you have to do it this way. This is discussed in numerious previous petsc-users email at the archive. > i hope I will find how to access private structure of matrix. > > tom > From steiger at purdue.edu Wed Mar 9 08:47:57 2011 From: steiger at purdue.edu (Sebastian Steiger) Date: Wed, 09 Mar 2011 09:47:57 -0500 Subject: [petsc-users] Scalability of AO ? Message-ID: <4D77931D.4080706@purdue.edu> Hello PETSc experts I have a parallel application that builds extensively on PETSc functionality and also uses the AO commands AOCreateMapping and AOApplicationToPetsc. We are currently doing some benchmarks on jaguar, the world's second-fastest computer, where we find some interior eigenvalues of a really large matrix (in conjunction with SLEPc). The application runs fine when using 40'000 cores and a matrix size of 400 million. There are 20 million AO-indices. However, when I scale up to 80'000 / 800 million / 40 million, I am running out of memory (jaguar has 1.3GB/core). I am pretty sure that in our own code all vectors have only the size of the local degrees of freedom, which stays constant at around 10'000. I figured out that I am running out of memory when I call AOCreateMapping. When I look inside aomapping.c, I see a comment "get all indices on all processors" near line 330 and some MPI_Allgatherv's. That seems like the routine AOApplicationToPetsc is not scalable. Not thinking for it too long, it seems to me that the task of creating this mapping could be done without having the need of communicating all indices to all processors (I am using PETSC_NULL for the mypetsc argument). Let me know what you think about this. Best Sebastian From bsmith at mcs.anl.gov Wed Mar 9 09:11:38 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 9 Mar 2011 09:11:38 -0600 Subject: [petsc-users] Scalability of AO ? In-Reply-To: <4D77931D.4080706@purdue.edu> References: <4D77931D.4080706@purdue.edu> Message-ID: Sebastian, The AOCreateMapping() is badly named (thanks Matt), it is a "generalization" of AOCreateBasic() which is (still) badly named but not as badly named as AOCreateMapping(). The thing to understand is that the AO operations are handled by function table table dispatch (runtime polymorphism just like the Vec operations, Mat operations etc). So, in theory there can be multiple AO implementations (concrete subclasses). Different implementations can have different performance characteristics; for example those defined by AOCreateBasic() trade memory for speed, by storing the entire mapping on each process they are fast (since AOApplicationToPetsc() and AOPetscToApplication()) for this implementation don't require any parallel communication. The name "Basic" is suppose to convey that it is a simple implementation without bells-and-wistles like scalability to very large problems. To get what you want with memory scalability there needs to be another implementation of AO that does not store all values on all processes. This is possible to write and requires possibly communication during the AOApplicationToPetsc() and AOPetscToApplication() to determine values that are not know locally. It is not terribly difficult to write but is clearly much more complicated than those operations defined with AOCreateBasic(). So why doesn't it already exist? Well most people write their code to avoid the need for AOApplicationToPetsc() and AOPetscToApplication() functionality and those who have use AO don't run problems on such a huge system like you do. So, what should we do? Well the best thing to do would be to implement a memory scalable version of the AO. One way to do this is to divide up the numbers 0 to N across the processes and have process 0 know the mapping for application ordering numbers from 0 to N/p-1, process 1 know for application ordering numbers from N/p to 2N/p-1 etc then when any process needs to know the PETSc number for application number a it determines which process knows the PETSc number of a (by simply determining for which k the k*N/p <= a < (k+1)*N/p-1 satisfies) and then sending a message to process k to get the PETSc number. Similarly to handle the mapping from PETSc to application one would divide up the knowledge of each application number for each PETSc number in the same way. One could make it even more sophisticated and have each process "cache" the information about each partner it has already checked on so that it need not communicate if the user asks again for the same value. Under normal circumstances I don't think this is needed because one shouldn't be normally calling AOApplicationToPetsc() or AOPetscToApplication() many times. So who will write the memory scalable implementation? Maybe it is time to finally get it done. Barry On Mar 9, 2011, at 8:47 AM, Sebastian Steiger wrote: > Hello PETSc experts > > I have a parallel application that builds extensively on PETSc > functionality and also uses the AO commands AOCreateMapping and > AOApplicationToPetsc. We are currently doing some benchmarks on jaguar, > the world's second-fastest computer, where we find some interior > eigenvalues of a really large matrix (in conjunction with SLEPc). > > The application runs fine when using 40'000 cores and a matrix size of > 400 million. There are 20 million AO-indices. However, when I scale up > to 80'000 / 800 million / 40 million, I am running out of memory (jaguar > has 1.3GB/core). I am pretty sure that in our own code all vectors have > only the size of the local degrees of freedom, which stays constant at > around 10'000. > > I figured out that I am running out of memory when I call > AOCreateMapping. When I look inside aomapping.c, I see a comment "get > all indices on all processors" near line 330 and some MPI_Allgatherv's. > That seems like the routine AOApplicationToPetsc is not scalable. > > Not thinking for it too long, it seems to me that the task of creating > this mapping could be done without having the need of communicating all > indices to all processors (I am using PETSC_NULL for the mypetsc > argument). Let me know what you think about this. > > Best > Sebastian From steiger at purdue.edu Wed Mar 9 10:03:58 2011 From: steiger at purdue.edu (Sebastian Steiger) Date: Wed, 09 Mar 2011 11:03:58 -0500 Subject: [petsc-users] Scalability of AO ? In-Reply-To: References: <4D77931D.4080706@purdue.edu> Message-ID: <4D77A4EE.8080905@purdue.edu> Hi Barry Thanks for this useful insight - after 2 years of employing most of PETSc's functionality we have finally found a corner that doesn't work with process counts that about 15 machines in the world can provide :-) I was thinking of something similar to your outline of the memory-scalable implementation. Since our code performs the call to this AO function very few times (1-3x), a little communication overhead is really not too bad for us. Best Sebastian On 03/09/2011 10:11 AM, Barry Smith wrote: > > Sebastian, > > The AOCreateMapping() is badly named (thanks Matt), it is a "generalization" of AOCreateBasic() which is (still) badly named but not as badly named as AOCreateMapping(). > > The thing to understand is that the AO operations are handled by function table table dispatch (runtime polymorphism just like the Vec operations, Mat operations etc). So, in theory there can be multiple AO implementations (concrete subclasses). Different implementations can have different performance characteristics; for example those defined by AOCreateBasic() trade memory for speed, by storing the entire mapping on each process they are fast (since AOApplicationToPetsc() and AOPetscToApplication()) for this implementation don't require any parallel communication. The name "Basic" is suppose to convey that it is a simple implementation without bells-and-wistles like scalability to very large problems. > > To get what you want with memory scalability there needs to be another implementation of AO that does not store all values on all processes. This is possible to write and requires possibly communication during the > AOApplicationToPetsc() and AOPetscToApplication() to determine values that are not know locally. It is not terribly difficult to write but is clearly much more complicated than those operations defined with AOCreateBasic(). > > So why doesn't it already exist? Well most people write their code to avoid the need for AOApplicationToPetsc() and AOPetscToApplication() functionality and those who have use AO don't run problems on such a huge system like you do. > > So, what should we do? Well the best thing to do would be to implement a memory scalable version of the AO. One way to do this is to divide up the numbers 0 to N across the processes and have process 0 know the mapping for application ordering numbers from 0 to N/p-1, process 1 know for application ordering numbers from N/p to 2N/p-1 etc then when any process needs to know the PETSc number for application number a it determines which process knows the PETSc number of a (by simply determining for which k the k*N/p <= a < (k+1)*N/p-1 satisfies) and then sending a message to process k to get the PETSc number. Similarly to handle the mapping from PETSc to application one would divide up the knowledge of each application number for each PETSc number in the same way. One could make it even more sophisticated and have each process "cache" the information about each partner it has already checked on so that it need not communicate if the user asks again for the same value. Un der normal circumstances I don't think this is needed because one shouldn't be normally calling AOApplicationToPetsc() or AOPetscToApplication() many times. > > So who will write the memory scalable implementation? Maybe it is time to finally get it done. > > > Barry > > > > > On Mar 9, 2011, at 8:47 AM, Sebastian Steiger wrote: > >> Hello PETSc experts >> >> I have a parallel application that builds extensively on PETSc >> functionality and also uses the AO commands AOCreateMapping and >> AOApplicationToPetsc. We are currently doing some benchmarks on jaguar, >> the world's second-fastest computer, where we find some interior >> eigenvalues of a really large matrix (in conjunction with SLEPc). >> >> The application runs fine when using 40'000 cores and a matrix size of >> 400 million. There are 20 million AO-indices. However, when I scale up >> to 80'000 / 800 million / 40 million, I am running out of memory (jaguar >> has 1.3GB/core). I am pretty sure that in our own code all vectors have >> only the size of the local degrees of freedom, which stays constant at >> around 10'000. >> >> I figured out that I am running out of memory when I call >> AOCreateMapping. When I look inside aomapping.c, I see a comment "get >> all indices on all processors" near line 330 and some MPI_Allgatherv's. >> That seems like the routine AOApplicationToPetsc is not scalable. >> >> Not thinking for it too long, it seems to me that the task of creating >> this mapping could be done without having the need of communicating all >> indices to all processors (I am using PETSC_NULL for the mypetsc >> argument). Let me know what you think about this. >> >> Best >> Sebastian > From travis.fisher at nasa.gov Wed Mar 9 10:53:58 2011 From: travis.fisher at nasa.gov (Travis C. Fisher) Date: Wed, 9 Mar 2011 11:53:58 -0500 Subject: [petsc-users] KSP Residual for GMRES Message-ID: <4D77B0A6.7070600@nasa.gov> I am running a Navier Stokes Solver that uses implicit Runge Kutta for timestepping. I am using SNES to perform preconditioned JFNK. As a sanity check, I am running in serial and using the FD generated Jacobian at the beginning of a time step as the preconditioner matrix for the linear solves, and superlu_dist as the preconditioner. This means that the linear solve (GMRES) for the first Newton iteration of the first stage should converge almost exactly. I do not observe this for large time steps. While it does converge to the Eisenstat-Walker convergence criterion, the KSP residual norm is not small. It only decreases four orders of magnitude. My question is whether or not this is the preconditioned residual? The solver spits out the same residual whether or not I use left or right preconditioning. I should also mention that in this case I have turned off all viscous cross terms and the problem is essentially 1-D, so I have taken as much care as possible to ensure that the FD jacobian is a proper representation of the system. Additionally, I observe the same problem in the inviscid case with both the FD jacobian and the exact jacobian that I have computed. I appreciate any insight you can give me into this observation. Thanks, Travis Fisher From dave.mayhem23 at gmail.com Wed Mar 9 10:58:19 2011 From: dave.mayhem23 at gmail.com (Dave May) Date: Wed, 9 Mar 2011 17:58:19 +0100 Subject: [petsc-users] KSP Residual for GMRES In-Reply-To: <4D77B0A6.7070600@nasa.gov> References: <4D77B0A6.7070600@nasa.gov> Message-ID: GMRES uses left preconditioning so -ksp_monitor will report the preconditioned residual You can use -ksp_monitor_true_residual if you don't know whether the preconditioner is using left or right preconditioning. This will display both residuals. On 9 March 2011 17:53, Travis C. Fisher wrote: > I am running a Navier Stokes Solver that uses implicit Runge Kutta for > timestepping. I am using SNES to perform preconditioned JFNK. As a sanity > check, I am running in serial and using the FD generated Jacobian at the > beginning of a time step as the preconditioner matrix for the linear solves, > and superlu_dist as the preconditioner. This means that the linear solve > (GMRES) for the first Newton iteration of the first stage should converge > almost exactly. I do not observe this for large time steps. While it does > converge to the Eisenstat-Walker convergence criterion, the KSP residual > norm is not small. It only decreases four orders of magnitude. My question > is whether or not this is the preconditioned residual? The solver spits out > the same residual whether or not I use left or right preconditioning. I > should also mention that in this case I have turned off all viscous cross > terms and the problem is essentially 1-D, so I have taken as much care as > possible to ensure that the FD jacobian is a proper representation of the > system. Additionally, I observe the same problem in the inviscid case with > both the FD jacobian and the exact jacobian that I have computed. I > appreciate any insight you can give me into this observation. > > Thanks, > > Travis Fisher > From Robert.Ellis at geosoft.com Wed Mar 9 12:15:46 2011 From: Robert.Ellis at geosoft.com (Robert Ellis) Date: Wed, 9 Mar 2011 18:15:46 +0000 Subject: [petsc-users] MKL library - how to force static linking Message-ID: <18205E5ECD2A1A4584F2BFC0BCBDE95526D576AC@exchange.geosoft.com> Hi PETSc Experts, Environment: Win, Cygwin, ifort, MKL to produce Win exes. I'm using the MKL blas-lapack with the configure parameter --with-blas-lapack-dir=/cygdrive/c/mkl/lib/intel64. The resultant executable requires MKL dlls to run, specifically fmpich2.dll, libiomp5md.dll, mkl_core.dll, mkl_intel_thread.dll, mpich2mpi.dll. Can I force a static PETSc build using the MKL libraries? (BTW, using --download-f-blas-lapack produces a statically linked exe.) Thanks for any help, Cheers, Rob -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Mar 9 12:31:11 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 9 Mar 2011 12:31:11 -0600 (CST) Subject: [petsc-users] MKL library - how to force static linking In-Reply-To: <18205E5ECD2A1A4584F2BFC0BCBDE95526D576AC@exchange.geosoft.com> References: <18205E5ECD2A1A4584F2BFC0BCBDE95526D576AC@exchange.geosoft.com> Message-ID: If you know the correct MLK library list for static linking- you can try specifying with --with-blas-lapack-lib option Satish On Wed, 9 Mar 2011, Robert Ellis wrote: > Hi PETSc Experts, > > Environment: Win, Cygwin, ifort, MKL to produce Win exes. > > I'm using the MKL blas-lapack with the configure parameter --with-blas-lapack-dir=/cygdrive/c/mkl/lib/intel64. The resultant executable requires MKL dlls to run, specifically fmpich2.dll, libiomp5md.dll, mkl_core.dll, mkl_intel_thread.dll, mpich2mpi.dll. Can I force a static PETSc build using the MKL libraries? (BTW, using --download-f-blas-lapack produces a statically linked exe.) > > Thanks for any help, > Cheers, > Rob > From hzhang at mcs.anl.gov Wed Mar 9 15:47:36 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Wed, 9 Mar 2011 15:47:36 -0600 Subject: [petsc-users] Scalability of AO ? In-Reply-To: <4D77A4EE.8080905@purdue.edu> References: <4D77931D.4080706@purdue.edu> <4D77A4EE.8080905@purdue.edu> Message-ID: Sebastian: > > Thanks for this useful insight - after 2 years of employing most of > PETSc's functionality we have finally found a corner that doesn't work > with process counts that about 15 machines in the world can provide :-) > > I was thinking of something similar to your outline of the > memory-scalable implementation. Since our code performs the call to this > AO function very few times (1-3x), a little communication overhead is > really not too bad for us. Are you planing to do something on your side? Otherwise, I can implement what Barry suggested in petsc library. Hong > > Best > Sebastian > > > > On 03/09/2011 10:11 AM, Barry Smith wrote: >> >> ? Sebastian, >> >> ? ? The AOCreateMapping() is badly named (thanks Matt), it is a "generalization" of AOCreateBasic() which is (still) badly named but not as badly named as AOCreateMapping(). >> >> ? ? ?The thing to understand is that the AO operations are handled by function table table dispatch (runtime polymorphism just like the Vec operations, Mat operations etc). So, in theory there can be multiple AO implementations (concrete subclasses). ?Different implementations can have different performance characteristics; for example those defined by AOCreateBasic() trade memory for speed, by storing the entire mapping on each process they are fast (since AOApplicationToPetsc() and AOPetscToApplication()) for this implementation don't require any parallel communication. The name "Basic" is suppose to convey that it is a simple implementation without bells-and-wistles like scalability to very large problems. >> >> ? ? To get what you want with memory scalability there needs to be another implementation of AO that does not store all values on all processes. This is possible to write and requires possibly communication during the >> AOApplicationToPetsc() and AOPetscToApplication() to determine values that are not know locally. It is not terribly difficult to write but is clearly much more complicated than those operations defined with AOCreateBasic(). >> >> ? ? So why doesn't it already exist? Well most people write their code to avoid the need for AOApplicationToPetsc() and AOPetscToApplication() functionality and those who have use AO don't run problems on such a huge system like you do. >> >> ? ? So, what should we do? Well the best thing to do would be to implement a memory scalable version of the AO. One way to do this is to divide up the numbers 0 to N across the processes and have process 0 know the mapping for application ordering numbers from 0 to N/p-1, process 1 know for application ordering numbers from N/p to 2N/p-1 etc then when any process needs to know the PETSc number for application number a it determines which process knows the PETSc number of a (by simply determining for which k the ?k*N/p <= a < ?(k+1)*N/p-1 satisfies) and then sending a message to process k to get the PETSc number. Similarly to handle the mapping from PETSc to application one would divide up the knowledge of each application number for each PETSc number in the same way. One could make it even more sophisticated and have each process "cache" the information about each partner it has already checked on so that it need not communicate if the user asks again for the same value. Un > der normal circumstances I don't think this is needed because one shouldn't be normally calling AOApplicationToPetsc() or AOPetscToApplication() many times. >> >> ? So who will write the memory scalable implementation? Maybe it is time to finally get it done. >> >> >> ? ? Barry >> >> >> >> >> On Mar 9, 2011, at 8:47 AM, Sebastian Steiger wrote: >> >>> Hello PETSc experts >>> >>> I have a parallel application that builds extensively on PETSc >>> functionality and also uses the AO commands AOCreateMapping and >>> AOApplicationToPetsc. We are currently doing some benchmarks on jaguar, >>> the world's second-fastest computer, where we find some interior >>> eigenvalues of a really large matrix (in conjunction with SLEPc). >>> >>> The application runs fine when using 40'000 cores and a matrix size of >>> 400 million. There are 20 million AO-indices. However, when I scale up >>> to 80'000 / 800 million / 40 million, I am running out of memory (jaguar >>> has 1.3GB/core). I am pretty sure that in our own code all vectors have >>> only the size of the local degrees of freedom, which stays constant at >>> around 10'000. >>> >>> I figured out that I am running out of memory when I call >>> AOCreateMapping. When I look inside aomapping.c, I see a comment "get >>> all indices on all processors" near line 330 and some MPI_Allgatherv's. >>> That seems like the routine AOApplicationToPetsc is not scalable. >>> >>> Not thinking for it too long, it seems to me that the task of creating >>> this mapping could be done without having the need of communicating all >>> indices to all processors (I am using PETSC_NULL for the mypetsc >>> argument). Let me know what you think about this. >>> >>> Best >>> Sebastian >> > > From Robert.Ellis at geosoft.com Wed Mar 9 15:53:51 2011 From: Robert.Ellis at geosoft.com (Robert Ellis) Date: Wed, 9 Mar 2011 21:53:51 +0000 Subject: [petsc-users] MKL library - how to force static linking In-Reply-To: References: <18205E5ECD2A1A4584F2BFC0BCBDE95526D576AC@exchange.geosoft.com> Message-ID: <18205E5ECD2A1A4584F2BFC0BCBDE95526D579F7@exchange.geosoft.com> Hi Satish, I've looked at the Intel docs and find that to have ifort use static MKL libraries I need to add /Qopenmp-link:static to the ifort command. Given this information, is the correct form for the config script (options preceded by - and space separated)? --FOPTFLAGS='-Qopenmp-link:static -O3' and the libraries I need are: mkl_intel_lp64.lib, mkl_intel_thread.lib, mkl_core.lib, libiomp5mt.lib Is the correct form full-pathname-comma-separated? --with-blas-lapack-lib='/cygdrive/ c/mkl/lib/intel64/mkl_intel_lp64.lib,/cygdrive/c/mkl/lib/intel64/mkl_intel_thread.lib,/cygdrive/c/mkl/lib/intel64/mkl_core.lib,/cygdrive/c/Program\ Files\ \(x86\)/Intel/ComposerXE-2011/compiler/lib/intel64/libiomp5mt.lib' I've tried various things but keep getting the error Invalid library: /cygdrive/c/mkl/lib/intel64/mkl_intel_lp64.lib,/cygdrive/c/mkl/lib/intel64/mkl_intel_thread.lib,/cygdrive/c/mkl/lib/intel64/mkl_core.lib,/cygdrive/c/Program\ Files\ \(x86\)/Intel/ComposerXE-2011/compiler/lib/intel64/libiomp5mt.lib for key with-blas-lapack-lib The following link is helpful for those of us trying to link MKL statically http://software.intel.com/en-us/forums/showthread.php?t=64254 Cheers, Rob -----Original Message----- From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Satish Balay Sent: Wednesday, March 09, 2011 10:31 AM To: PETSc users list Subject: Re: [petsc-users] MKL library - how to force static linking If you know the correct MLK library list for static linking- you can try specifying with --with-blas-lapack-lib option Satish On Wed, 9 Mar 2011, Robert Ellis wrote: > Hi PETSc Experts, > > Environment: Win, Cygwin, ifort, MKL to produce Win exes. > > I'm using the MKL blas-lapack with the configure parameter --with-blas-lapack-dir=/cygdrive/c/mkl/lib/intel64. The resultant executable requires MKL dlls to run, specifically fmpich2.dll, libiomp5md.dll, mkl_core.dll, mkl_intel_thread.dll, mpich2mpi.dll. Can I force a static PETSc build using the MKL libraries? (BTW, using --download-f-blas-lapack produces a statically linked exe.) > > Thanks for any help, > Cheers, > Rob > From balay at mcs.anl.gov Wed Mar 9 16:36:34 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 9 Mar 2011 16:36:34 -0600 (CST) Subject: [petsc-users] MKL library - how to force static linking In-Reply-To: <18205E5ECD2A1A4584F2BFC0BCBDE95526D579F7@exchange.geosoft.com> References: <18205E5ECD2A1A4584F2BFC0BCBDE95526D576AC@exchange.geosoft.com> <18205E5ECD2A1A4584F2BFC0BCBDE95526D579F7@exchange.geosoft.com> Message-ID: On Wed, 9 Mar 2011, Robert Ellis wrote: > Hi Satish, > > I've looked at the Intel docs and find that to have ifort use static MKL libraries I need to add /Qopenmp-link:static to the ifort command. Given this information, is the correct form for the config script (options preceded by - and space separated)? > > --FOPTFLAGS='-Qopenmp-link:static -O3' Sure this is fine. You might also want to add in similar COPTFLAGS. > and the libraries I need are: mkl_intel_lp64.lib, mkl_intel_thread.lib, mkl_core.lib, libiomp5mt.lib > > Is the correct form full-pathname-comma-separated? > > --with-blas-lapack-lib='/cygdrive/ > c/mkl/lib/intel64/mkl_intel_lp64.lib,/cygdrive/c/mkl/lib/intel64/mkl_intel_thread.lib,/cygdrive/c/mkl/lib/intel64/mkl_core.lib,/cygdrive/c/Program\ Files\ \(x86\)/Intel/ComposerXE-2011/compiler/lib/intel64/libiomp5mt.lib' > > I've tried various things but keep getting the error > > Invalid library: /cygdrive/c/mkl/lib/intel64/mkl_intel_lp64.lib,/cygdrive/c/mkl/lib/intel64/mkl_intel_thread.lib,/cygdrive/c/mkl/lib/intel64/mkl_core.lib,/cygdrive/c/Program\ Files\ \(x86\)/Intel/ComposerXE-2011/compiler/lib/intel64/libiomp5mt.lib for key with-blas-lapack-lib For one the above should be a python list. And then - configure has trouble with 'spaces' and 'braces' in these paths. Suggest doing: - create a dir for the required libs - say mkdir $(HOME)/MKL copy the required files over here. and now use: --with-blas-lapack-lib=[$(HOME)/MKL/mkl_intel_lp64.lib,$(HOME)/MKL/mkl_intel_thread.lib,...] Also the following might work - but I'm not sure.. --with-blas-lapack-lib="-L$(HOME)/MKL mkl_intel_lp64.lib mkl_intel_thread.lib ..." If you have issues - send configure.log to petsc-maint Satish > > > The following link is helpful for those of us trying to link MKL statically http://software.intel.com/en-us/forums/showthread.php?t=64254 > > Cheers, > Rob > > > > -----Original Message----- > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Satish Balay > Sent: Wednesday, March 09, 2011 10:31 AM > To: PETSc users list > Subject: Re: [petsc-users] MKL library - how to force static linking > > If you know the correct MLK library list for static linking- you can > try specifying with --with-blas-lapack-lib option > > Satish > > > On Wed, 9 Mar 2011, Robert Ellis wrote: > > > Hi PETSc Experts, > > > > Environment: Win, Cygwin, ifort, MKL to produce Win exes. > > > > I'm using the MKL blas-lapack with the configure parameter --with-blas-lapack-dir=/cygdrive/c/mkl/lib/intel64. The resultant executable requires MKL dlls to run, specifically fmpich2.dll, libiomp5md.dll, mkl_core.dll, mkl_intel_thread.dll, mpich2mpi.dll. Can I force a static PETSc build using the MKL libraries? (BTW, using --download-f-blas-lapack produces a statically linked exe.) > > > > Thanks for any help, > > Cheers, > > Rob > > > > From kenway at utias.utoronto.ca Thu Mar 10 14:44:32 2011 From: kenway at utias.utoronto.ca (Gaetan Kenway) Date: Thu, 10 Mar 2011 15:44:32 -0500 Subject: [petsc-users] PCASMSetLocalSubdomains Question Message-ID: Hello I am trying to setup a Additive Schwartz Preconditioner using multiple blocks on each processor. My code is a 3D Multiblock CFD code and what I am trying to do is to put each CFD block into its own subdomain, even if there are more than 1 block on each processor. The PETSc help on this function wasn't helpful nor was the one example. I don't understand the difference between "is" and "is_local". Are you supposed to set 1 or both or neither? As an example what I'm currently doing, for a given processor I want to split into ndom subdomains I don the following PetscFortranAddr,dimension(:),allocatable :: is_local allocate(is_local(ndom)) do nn=1,ndom ! Loop over block on this processor ! create the index set for this domain -> Indicies contains the global indices for the the current subdomain -> Code not shown call ISCreateGeneral(PETSC_COMM_WORLD,nindices,indices,is_local(nn),PETScIerr) asdf call PCASMSetLocalSubdomains(pc,ndom,PETSC_NULL_OBJECT,is_local,PETScIerr) This dies with error code 85: Argument is null that should not be The traceback I get is: [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] PetscObjectReference line 112 src/sys/objects/inherit.c [0]PETSC ERROR: [0] PCASMSetLocalSubdomains_ASM line 525 src/ksp/pc/impls/asm/asm.c [0]PETSC ERROR: [0] PCASMSetLocalSubdomains line 699 src/ksp/pc/impls/asm/asm.c I'm trying to use this with a block AIJ Matrix. Does the setlocalsubdomains work with block AIJ Matrices? Thank you, Gaetan -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Mar 10 15:16:19 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 10 Mar 2011 15:16:19 -0600 Subject: [petsc-users] PCASMSetLocalSubdomains Question In-Reply-To: References: Message-ID: On Thu, Mar 10, 2011 at 2:44 PM, Gaetan Kenway wrote: > Hello > > I am trying to setup a Additive Schwartz Preconditioner using multiple > blocks on each processor. My code is a 3D Multiblock CFD code and what I am > trying to do is to put each CFD block into its own subdomain, even if there > are more than 1 block on each processor. The PETSc help on this function > wasn't helpful nor was the one example. > > I don't understand the difference between "is" and "is_local". Are you > supposed to set 1 or both or neither? > If you want multiple blocks/process AND you want to be able to do RASM, then you need to specify both 1) the overlapping blocks 2) the non-overlapping parts You need 2), because I have no idea how to calculate 2) given only 1). We could have an alternative interface that took 2) and an overlap, and figured it out, however currently that only works for 1 block/process, and specfiying both 1) and 2) is more general (I needed it for a specific PC). > As an example what I'm currently doing, for a given processor I want to > split into ndom subdomains I don the following > > > PetscFortranAddr,dimension(:),allocatable :: is_local > > allocate(is_local(ndom)) > do nn=1,ndom ! Loop over block on this processor > ! create the index set for this domain -> Indicies contains the global > indices for the the current subdomain -> Code not shown > > call > ISCreateGeneral(PETSC_COMM_WORLD,nindices,indices,is_local(nn),PETScIerr) > asdf > > call PCASMSetLocalSubdomains(pc,ndom,PETSC_NULL_OBJECT,is_local,PETScIerr) > > > This dies with error code 85: Argument is null that should not be > This should work. Something is wrong with the Fortran translation. Are you using 3.1 or dev? It looks correct in dev. Matt > The traceback I get is: > > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] PetscObjectReference line 112 src/sys/objects/inherit.c > [0]PETSC ERROR: [0] PCASMSetLocalSubdomains_ASM line 525 > src/ksp/pc/impls/asm/asm.c > [0]PETSC ERROR: [0] PCASMSetLocalSubdomains line 699 > src/ksp/pc/impls/asm/asm.c > > I'm trying to use this with a block AIJ Matrix. Does the setlocalsubdomains > work with block AIJ Matrices? > > Thank you, > > Gaetan > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From rongliang.chan at gmail.com Thu Mar 10 17:06:35 2011 From: rongliang.chan at gmail.com (Rongliang Chen) Date: Thu, 10 Mar 2011 16:06:35 -0700 Subject: [petsc-users] Warning in PCCompositeAddPC Message-ID: Hi PETSc experts, When I used the following functions: ierr = PCCompositeAddPC(finepc,PCSHELL);CHKERRQ(ierr); ierr = PCCompositeAddPC(finepc,PCASM);CHKERRQ(ierr); I got the following warning: joab.c:340: warning: passing argument 2 of ?PCCompositeAddPC? discards qualifiers from pointer target type joab.c:341: warning: passing argument 2 of ?PCCompositeAddPC? discards qualifiers from pointer target type Can you tell me how to fix it? Thank you. Best, Rongliang UCB -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Mar 10 17:11:57 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 10 Mar 2011 17:11:57 -0600 Subject: [petsc-users] Warning in PCCompositeAddPC In-Reply-To: References: Message-ID: On Thu, Mar 10, 2011 at 5:06 PM, Rongliang Chen wrote: > Hi PETSc experts, > > When I used the following functions: > > ierr = PCCompositeAddPC(finepc,PCSHELL);CHKERRQ(ierr); > ierr = PCCompositeAddPC(finepc,PCASM);CHKERRQ(ierr); > You can ignore these warnings about 'const' for char *. I believe this is fixed in petsc-dev. Matt > I got the following warning: > > joab.c:340: warning: passing argument 2 of ?PCCompositeAddPC? discards > qualifiers from pointer target type > joab.c:341: warning: passing argument 2 of ?PCCompositeAddPC? discards > qualifiers from pointer target type > > Can you tell me how to fix it? Thank you. > > Best, > > Rongliang > > UCB > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From steiger at purdue.edu Fri Mar 11 09:52:02 2011 From: steiger at purdue.edu (Sebastian Steiger) Date: Fri, 11 Mar 2011 10:52:02 -0500 Subject: [petsc-users] Which preconditioners are scalable? Message-ID: <4D7A4522.8060605@purdue.edu> Hello PETSc developers I'm doing some scaling benchmarks and I found that the parallel asm preconditioner, my favorite preconditioner, has a limit in the number of cores it can handle. I am doing a numerical experiment where I scale up the size of my matrix by roughly the same factor as the number of CPUs employed. When I look at which function used how much memory using PETSc's routine PetscMallocDumpLog, I see the following: Function name N=300 N=600 increase ====================================================================== MatGetSubMatrices_MPIAIJ_Local 75'912'016 134'516'928 1.77 MatIncreaseOverlap_MPIAIJ_Once 168'288'288 346'870'832 2.06 MatIncreaseOverlap_MPIAIJ_Receive 2'918'960 5'658'160 1.94 The matrix sizes are 6'899'904 and 14'224'896, respectively. Above N~5000 CPUs I am running out of memory. Here's my question now: Is the asm preconditioner limited from the algorithm point of view, or is it the implementation? I thought that 'only' the local matrices, plus some constant overlap with neighbors, are solved, so that memory consumption should stay constant when I scale up with a constant number of rows per process. Best Sebastian From knepley at gmail.com Fri Mar 11 10:56:12 2011 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 11 Mar 2011 10:56:12 -0600 Subject: [petsc-users] Which preconditioners are scalable? In-Reply-To: <4D7A4522.8060605@purdue.edu> References: <4D7A4522.8060605@purdue.edu> Message-ID: On Fri, Mar 11, 2011 at 9:52 AM, Sebastian Steiger wrote: > Hello PETSc developers > > I'm doing some scaling benchmarks and I found that the parallel asm > preconditioner, my favorite preconditioner, has a limit in the number of > cores it can handle. > > I am doing a numerical experiment where I scale up the size of my matrix > by roughly the same factor as the number of CPUs employed. When I look > at which function used how much memory using PETSc's routine > PetscMallocDumpLog, I see the following: > > Function name N=300 N=600 increase > ====================================================================== > MatGetSubMatrices_MPIAIJ_Local 75'912'016 134'516'928 1.77 > MatIncreaseOverlap_MPIAIJ_Once 168'288'288 346'870'832 2.06 > MatIncreaseOverlap_MPIAIJ_Receive 2'918'960 5'658'160 1.94 > > The matrix sizes are 6'899'904 and 14'224'896, respectively. Above > N~5000 CPUs I am running out of memory. > We have run ASM on 224,000 processors of the XT5 at ORNL, so something else is going on here. The best thing to do here is send us -log_summary. For attachments, we usually recommend petsc-maint at mcs.anl.gov. Matt > Here's my question now: Is the asm preconditioner limited from the > algorithm point of view, or is it the implementation? I thought that > 'only' the local matrices, plus some constant overlap with neighbors, > are solved, so that memory consumption should stay constant when I scale > up with a constant number of rows per process. > > Best > Sebastian > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Mar 11 13:00:49 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 11 Mar 2011 13:00:49 -0600 Subject: [petsc-users] Which preconditioners are scalable? In-Reply-To: <4D7A4522.8060605@purdue.edu> References: <4D7A4522.8060605@purdue.edu> Message-ID: <541EC601-1C04-4047-AB26-FA50C02180BA@mcs.anl.gov> Having trouble understanding your email. What is N? Is that the number of processes? What does the notation 5'912'016 mean? Are the numbers in your table from a particular process? Or are they summed over all processes? The intention is that the ASM is memory scalable so that if for example you double the number of processes and double the total number of nonzeros in the matrix (probably by doubling the total number of rows and columns in the matrix) each process should require essentially the same amount of memory. But what happens in practice for a particular problem will, to some degree, depend on the amount of coupling between processes in the matrix (hence how much bigger the local overlapped matrix is then the original matrix on that process) and depend on how the domain is sliced up. But even with a "bad" slicing I would not expect the amount of local memory needed to double. I think you need to determine more completely what all this memory is being used for. If you find a scalability problem we definitely want to fix it because we intend as you say: > I thought that > 'only' the local matrices, plus some constant overlap with neighbors, > are solved, so that memory consumption should stay constant when I scale > up with a constant number of rows per process. Barry On Mar 11, 2011, at 9:52 AM, Sebastian Steiger wrote: > Hello PETSc developers > > I'm doing some scaling benchmarks and I found that the parallel asm > preconditioner, my favorite preconditioner, has a limit in the number of > cores it can handle. > > I am doing a numerical experiment where I scale up the size of my matrix > by roughly the same factor as the number of CPUs employed. When I look > at which function used how much memory using PETSc's routine > PetscMallocDumpLog, I see the following: > > Function name N=300 N=600 increase > ====================================================================== > MatGetSubMatrices_MPIAIJ_Local 75'912'016 134'516'928 1.77 > MatIncreaseOverlap_MPIAIJ_Once 168'288'288 346'870'832 2.06 > MatIncreaseOverlap_MPIAIJ_Receive 2'918'960 5'658'160 1.94 > > The matrix sizes are 6'899'904 and 14'224'896, respectively. Above > N~5000 CPUs I am running out of memory. > > Here's my question now: Is the asm preconditioner limited from the > algorithm point of view, or is it the implementation? I thought that > 'only' the local matrices, plus some constant overlap with neighbors, > are solved, so that memory consumption should stay constant when I scale > up with a constant number of rows per process. > > Best > Sebastian > From steiger at purdue.edu Fri Mar 11 14:35:08 2011 From: steiger at purdue.edu (Sebastian Steiger) Date: Fri, 11 Mar 2011 15:35:08 -0500 Subject: [petsc-users] Which preconditioners are scalable? In-Reply-To: <541EC601-1C04-4047-AB26-FA50C02180BA@mcs.anl.gov> References: <4D7A4522.8060605@purdue.edu> <541EC601-1C04-4047-AB26-FA50C02180BA@mcs.anl.gov> Message-ID: <4D7A877C.70103@purdue.edu> Hi Barry and Matt > What is N? Is that the number of processes? Yes > What does the notation 5'912'016 mean? It means there were 5658160 bytes allocated. I introduced the 's for readability. > Are the numbers in your table from a particular process? Or are they summed over all processes? Only process 0. > The intention is that the ASM is memory scalable so that if for example you double the number >of processes and double the total number of nonzeros in the matrix (probably by doubling the total >number of rows and columns in the matrix) each process should require essentially the same amount >of memory. But what happens in practice for a particular problem will, to some degree, depend on >the amount of coupling between processes in the matrix (hence how much bigger the local overlapped >matrix is then the original matrix on that process) and depend on how the domain is sliced up. >But even with a "bad" slicing I would not expect the amount of local memory needed to double. >I think you need to determine more completely what all this memory is being used for. Doubling the total number of rows and nonzeros is what I think I'm doing. Every row has about 40 nonzeros in this example. The coupling / slicing should be fine since I am using pretty much the same system for another calculation where I compute interior eigenstates in a matrix with the same sparsity. There I do not use ASM and I can scale up to 80000 cores without memory problems anymore (after I have done a workaround for not using AOCreateMapping, see my report earlier this week). Also when I turn off ASM and use no preconditioning at all, or when I use the Jacobi preconditioner, then memory stays constant at about 30MB/core. But then the convergence deteriorates... Matt: > We have run ASM on 224,000 processors of the XT5 at ORNL, so > something else is going on here. The best thing to do here is send us > -log_summary. For attachments, we usually recommend > petsc-maint at mcs.anl.gov. My data also comes from the XT5, but it's important for me to know that there are cases where it scales to 224000 processors. I will post more complete profiling information to petsc-maint at mcs.anl.gov in a couple of minutes. Best Sebastian > > Barry > > > > > On Mar 11, 2011, at 9:52 AM, Sebastian Steiger wrote: > >> Hello PETSc developers >> >> I'm doing some scaling benchmarks and I found that the parallel asm >> preconditioner, my favorite preconditioner, has a limit in the number of >> cores it can handle. >> >> I am doing a numerical experiment where I scale up the size of my matrix >> by roughly the same factor as the number of CPUs employed. When I look >> at which function used how much memory using PETSc's routine >> PetscMallocDumpLog, I see the following: >> >> Function name N=300 N=600 increase >> ====================================================================== >> MatGetSubMatrices_MPIAIJ_Local 75'912'016 134'516'928 1.77 >> MatIncreaseOverlap_MPIAIJ_Once 168'288'288 346'870'832 2.06 >> MatIncreaseOverlap_MPIAIJ_Receive 2'918'960 5'658'160 1.94 >> >> The matrix sizes are 6'899'904 and 14'224'896, respectively. Above >> N~5000 CPUs I am running out of memory. >> >> Here's my question now: Is the asm preconditioner limited from the >> algorithm point of view, or is it the implementation? I thought that >> 'only' the local matrices, plus some constant overlap with neighbors, >> are solved, so that memory consumption should stay constant when I scale >> up with a constant number of rows per process. >> >> Best >> Sebastian >> > From bsmith at mcs.anl.gov Fri Mar 11 14:51:40 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 11 Mar 2011 14:51:40 -0600 Subject: [petsc-users] Which preconditioners are scalable? In-Reply-To: <4D7A877C.70103@purdue.edu> References: <4D7A4522.8060605@purdue.edu> <541EC601-1C04-4047-AB26-FA50C02180BA@mcs.anl.gov> <4D7A877C.70103@purdue.edu> Message-ID: Good. Then either (1) the matrix is strange in that using a level overlap of 1 behaviors differently for larger problems (which seems unlikely) or (2) there is a memory bug in the code that scarves up much more memory then it should. This requires deeper analysis of the implementation. Do we have arrays that grow in size with the number of processes in the code? Are you doing the basic ASM with one block per process? Barry On Mar 11, 2011, at 2:35 PM, Sebastian Steiger wrote: > Hi Barry and Matt > >> What is N? Is that the number of processes? > Yes > >> What does the notation 5'912'016 mean? > It means there were 5658160 bytes allocated. I introduced the 's for > readability. > > >> Are the numbers in your table from a particular process? Or are they summed over all processes? > Only process 0. > > >> The intention is that the ASM is memory scalable so that if for example you double the number >> of processes and double the total number of nonzeros in the matrix > (probably by doubling the total >> number of rows and columns in the matrix) each process should require > essentially the same amount >> of memory. But what happens in practice for a particular problem will, > to some degree, depend on >> the amount of coupling between processes in the matrix (hence how much > bigger the local overlapped >> matrix is then the original matrix on that process) and depend on how > the domain is sliced up. >> But even with a "bad" slicing I would not expect the amount of local > memory needed to double. >> I think you need to determine more completely what all this memory is > being used for. > > Doubling the total number of rows and nonzeros is what I think I'm > doing. Every row has about 40 nonzeros in this example. The coupling / > slicing should be fine since I am using pretty much the same system for > another calculation where I compute interior eigenstates in a matrix > with the same sparsity. There I do not use ASM and I can scale up to > 80000 cores without memory problems anymore (after I have done a > workaround for not using AOCreateMapping, see my report earlier this week). > > Also when I turn off ASM and use no preconditioning at all, or when I > use the Jacobi preconditioner, then memory stays constant at about > 30MB/core. But then the convergence deteriorates... > > > > Matt: > >> We have run ASM on 224,000 processors of the XT5 at ORNL, so >> something else is going on here. The best thing to do here is send us >> -log_summary. For attachments, we usually recommend >> petsc-maint at mcs.anl.gov. > > > My data also comes from the XT5, but it's important for me to know that > there are cases where it scales to 224000 processors. I will post more > complete profiling information to petsc-maint at mcs.anl.gov in a couple of > minutes. > > > Best > Sebastian > > > > > > > > > > > > > > >> >> Barry >> >> >> >> >> On Mar 11, 2011, at 9:52 AM, Sebastian Steiger wrote: >> >>> Hello PETSc developers >>> >>> I'm doing some scaling benchmarks and I found that the parallel asm >>> preconditioner, my favorite preconditioner, has a limit in the number of >>> cores it can handle. >>> >>> I am doing a numerical experiment where I scale up the size of my matrix >>> by roughly the same factor as the number of CPUs employed. When I look >>> at which function used how much memory using PETSc's routine >>> PetscMallocDumpLog, I see the following: >>> >>> Function name N=300 N=600 increase >>> ====================================================================== >>> MatGetSubMatrices_MPIAIJ_Local 75'912'016 134'516'928 1.77 >>> MatIncreaseOverlap_MPIAIJ_Once 168'288'288 346'870'832 2.06 >>> MatIncreaseOverlap_MPIAIJ_Receive 2'918'960 5'658'160 1.94 >>> >>> The matrix sizes are 6'899'904 and 14'224'896, respectively. Above >>> N~5000 CPUs I am running out of memory. >>> >>> Here's my question now: Is the asm preconditioner limited from the >>> algorithm point of view, or is it the implementation? I thought that >>> 'only' the local matrices, plus some constant overlap with neighbors, >>> are solved, so that memory consumption should stay constant when I scale >>> up with a constant number of rows per process. >>> >>> Best >>> Sebastian >>> >> > From steiger at purdue.edu Fri Mar 11 15:00:32 2011 From: steiger at purdue.edu (Sebastian Steiger) Date: Fri, 11 Mar 2011 16:00:32 -0500 Subject: [petsc-users] Which preconditioners are scalable? In-Reply-To: References: <4D7A4522.8060605@purdue.edu> <541EC601-1C04-4047-AB26-FA50C02180BA@mcs.anl.gov> <4D7A877C.70103@purdue.edu> Message-ID: <4D7A8D70.8010007@purdue.edu> > Good. Then either (1) the matrix is strange in that using a level overlap of 1 behaviors differently for larger problems (which seems unlikely) or (2) there is a memory bug in the code that scarves up much more memory then it should. I've just posted detailed data to petsc-maint. > This requires deeper analysis of the implementation. Do we have arrays that grow in size with the number of processes in the code? I'm pretty sure that I do not have such arrays on my side. For simulations employing iterative solutions without ASM memory consumption is small and roughly constant when I scale up. > Are you doing the basic ASM with one block per process? Yes. Best Sebastian > > > Barry > > > On Mar 11, 2011, at 2:35 PM, Sebastian Steiger wrote: > >> Hi Barry and Matt >> >>> What is N? Is that the number of processes? >> Yes >> >>> What does the notation 5'912'016 mean? >> It means there were 5658160 bytes allocated. I introduced the 's for >> readability. >> >> >>> Are the numbers in your table from a particular process? Or are they summed over all processes? >> Only process 0. >> >> >>> The intention is that the ASM is memory scalable so that if for example you double the number >>> of processes and double the total number of nonzeros in the matrix >> (probably by doubling the total >>> number of rows and columns in the matrix) each process should require >> essentially the same amount >>> of memory. But what happens in practice for a particular problem will, >> to some degree, depend on >>> the amount of coupling between processes in the matrix (hence how much >> bigger the local overlapped >>> matrix is then the original matrix on that process) and depend on how >> the domain is sliced up. >>> But even with a "bad" slicing I would not expect the amount of local >> memory needed to double. >>> I think you need to determine more completely what all this memory is >> being used for. >> >> Doubling the total number of rows and nonzeros is what I think I'm >> doing. Every row has about 40 nonzeros in this example. The coupling / >> slicing should be fine since I am using pretty much the same system for >> another calculation where I compute interior eigenstates in a matrix >> with the same sparsity. There I do not use ASM and I can scale up to >> 80000 cores without memory problems anymore (after I have done a >> workaround for not using AOCreateMapping, see my report earlier this week). >> >> Also when I turn off ASM and use no preconditioning at all, or when I >> use the Jacobi preconditioner, then memory stays constant at about >> 30MB/core. But then the convergence deteriorates... >> >> >> >> Matt: >> >>> We have run ASM on 224,000 processors of the XT5 at ORNL, so >>> something else is going on here. The best thing to do here is send us >>> -log_summary. For attachments, we usually recommend >>> petsc-maint at mcs.anl.gov. >> >> >> My data also comes from the XT5, but it's important for me to know that >> there are cases where it scales to 224000 processors. I will post more >> complete profiling information to petsc-maint at mcs.anl.gov in a couple of >> minutes. >> >> >> Best >> Sebastian >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>> >>> Barry >>> >>> >>> >>> >>> On Mar 11, 2011, at 9:52 AM, Sebastian Steiger wrote: >>> >>>> Hello PETSc developers >>>> >>>> I'm doing some scaling benchmarks and I found that the parallel asm >>>> preconditioner, my favorite preconditioner, has a limit in the number of >>>> cores it can handle. >>>> >>>> I am doing a numerical experiment where I scale up the size of my matrix >>>> by roughly the same factor as the number of CPUs employed. When I look >>>> at which function used how much memory using PETSc's routine >>>> PetscMallocDumpLog, I see the following: >>>> >>>> Function name N=300 N=600 increase >>>> ====================================================================== >>>> MatGetSubMatrices_MPIAIJ_Local 75'912'016 134'516'928 1.77 >>>> MatIncreaseOverlap_MPIAIJ_Once 168'288'288 346'870'832 2.06 >>>> MatIncreaseOverlap_MPIAIJ_Receive 2'918'960 5'658'160 1.94 >>>> >>>> The matrix sizes are 6'899'904 and 14'224'896, respectively. Above >>>> N~5000 CPUs I am running out of memory. >>>> >>>> Here's my question now: Is the asm preconditioner limited from the >>>> algorithm point of view, or is it the implementation? I thought that >>>> 'only' the local matrices, plus some constant overlap with neighbors, >>>> are solved, so that memory consumption should stay constant when I scale >>>> up with a constant number of rows per process. >>>> >>>> Best >>>> Sebastian >>>> >>> >> > From agrayver at gfz-potsdam.de Sat Mar 12 08:17:55 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Sat, 12 Mar 2011 15:17:55 +0100 Subject: [petsc-users] MatMatMult takes much time Message-ID: <4D7B8093.1070009@gfz-potsdam.de> Hello, call MatMatMult(S,G,MAT_INITIAL_MATRIX,PETSC_DEFAULT_DOUBLE_PRECISION,J,ierr);CHKERRQ(ierr) I use this operation to compute matrix-matrix product. It's correct, but quite slow. For instance, my matrix S is a dense matrix size of <6x762048 complex double> and matrix G is a sparse matrix with 2 nonzeros per row and size of <762048x254016 complex double> Time spent on S*G was ~22 secs with 32 processes over 4 nodes. There is infiniband between nodes. Is it a real life or I'm doing something wrong? Regards, Alexander From bsmith at mcs.anl.gov Sat Mar 12 08:54:25 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 12 Mar 2011 08:54:25 -0600 Subject: [petsc-users] MatMatMult takes much time In-Reply-To: <4D7B8093.1070009@gfz-potsdam.de> References: <4D7B8093.1070009@gfz-potsdam.de> Message-ID: <8B6848F7-F8BD-4965-B0E9-5BEF07DA089D@mcs.anl.gov> A*B where A is sparse and B dense can be implemented efficiently since it is a bunch of sparse matrix vector products (one for each column of B) S*G where S is dense is unlikley to ever be particularly fast: Here is how it is currently implemented #undef __FUNCT__ #define __FUNCT__ "MatMatMultNumeric_MPIDense_MPIAIJ" /* Computes (B'*A')' since computing B*A directly is untenable n p p ( ) ( ) ( ) m ( A ) * n ( B ) = m ( C ) ( ) ( ) ( ) */ PetscErrorCode MatMatMultNumeric_MPIDense_MPIAIJ(Mat A,Mat B,Mat C) { PetscErrorCode ierr; Mat At,Bt,Ct; PetscFunctionBegin; ierr = MatTranspose(A,MAT_INITIAL_MATRIX,&At);CHKERRQ(ierr); ierr = MatTranspose(B,MAT_INITIAL_MATRIX,&Bt);CHKERRQ(ierr); ierr = MatMatMult(Bt,At,MAT_INITIAL_MATRIX,1.0,&Ct);CHKERRQ(ierr); ierr = MatDestroy(At);CHKERRQ(ierr); ierr = MatDestroy(Bt);CHKERRQ(ierr); ierr = MatTranspose(Ct,MAT_REUSE_MATRIX,&C);CHKERRQ(ierr); ierr = MatDestroy(Ct);CHKERRQ(ierr); PetscFunctionReturn(0); } If you can suggest a way to implement it much more efficiently please let us know. Are you sure you need this product and not A*B? Barry On Mar 12, 2011, at 8:17 AM, Alexander Grayver wrote: > Hello, > > call MatMatMult(S,G,MAT_INITIAL_MATRIX,PETSC_DEFAULT_DOUBLE_PRECISION,J,ierr);CHKERRQ(ierr) > > I use this operation to compute matrix-matrix product. It's correct, but quite slow. > > For instance, my matrix S is a dense matrix size of <6x762048 complex double> and matrix G is a sparse matrix with 2 nonzeros per row and size of <762048x254016 complex double> > Time spent on S*G was ~22 secs with 32 processes over 4 nodes. There is infiniband between nodes. > > Is it a real life or I'm doing something wrong? > > Regards, > Alexander From agrayver at gfz-potsdam.de Sat Mar 12 09:06:49 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Sat, 12 Mar 2011 16:06:49 +0100 Subject: [petsc-users] MatMatMult takes much time In-Reply-To: <8B6848F7-F8BD-4965-B0E9-5BEF07DA089D@mcs.anl.gov> References: <4D7B8093.1070009@gfz-potsdam.de> <8B6848F7-F8BD-4965-B0E9-5BEF07DA089D@mcs.anl.gov> Message-ID: <4D7B8C09.7060705@gfz-potsdam.de> On 12.03.2011 15:54, Barry Smith wrote: > A*B where A is sparse and B dense can be implemented efficiently since it is a bunch of sparse matrix vector products (one for each column of B) > > S*G where S is dense is unlikley to ever be particularly fast: Here is how it is currently implemented > > #undef __FUNCT__ > #define __FUNCT__ "MatMatMultNumeric_MPIDense_MPIAIJ" > /* > Computes (B'*A')' since computing B*A directly is untenable Ok, it explains why the code: call MatTranspose(G,MAT_INITIAL_MATRIX,GT,ierr);CHKERRQ(ierr) call MatTranspose(S,MAT_INITIAL_MATRIX,ST,ierr);CHKERRQ(ierr) call MatMatMult(GT,ST,MAT_INITIAL_MATRIX,PETSC_DEFAULT_DOUBLE_PRECISION,JT,ierr);CHKERRQ(ierr) works faster than just call MatMatMult(S,G,MAT_INITIAL_MATRIX,PETSC_DEFAULT_DOUBLE_PRECISION,J,ierr);CHKERRQ(ierr) > n p p > ( ) ( ) ( ) > m ( A ) * n ( B ) = m ( C ) > ( ) ( ) ( ) > > */ > PetscErrorCode MatMatMultNumeric_MPIDense_MPIAIJ(Mat A,Mat B,Mat C) > { > PetscErrorCode ierr; > Mat At,Bt,Ct; > > PetscFunctionBegin; > ierr = MatTranspose(A,MAT_INITIAL_MATRIX,&At);CHKERRQ(ierr); > ierr = MatTranspose(B,MAT_INITIAL_MATRIX,&Bt);CHKERRQ(ierr); > ierr = MatMatMult(Bt,At,MAT_INITIAL_MATRIX,1.0,&Ct);CHKERRQ(ierr); > ierr = MatDestroy(At);CHKERRQ(ierr); > ierr = MatDestroy(Bt);CHKERRQ(ierr); > ierr = MatTranspose(Ct,MAT_REUSE_MATRIX,&C);CHKERRQ(ierr); > ierr = MatDestroy(Ct);CHKERRQ(ierr); > PetscFunctionReturn(0); > } > > If you can suggest a way to implement it much more efficiently please let us know. > > Are you sure you need this product and not A*B? Unfortunately I can't suggest a better way. Well actually I need A*B, but my initial matrices have to be transposed for that. But probably there is a way to avoid forming dense matrix at all, this way requires more code, that is why I tried another way with full dense matrix first. Thanks Barry. > > Barry > > > On Mar 12, 2011, at 8:17 AM, Alexander Grayver wrote: > >> Hello, >> >> call MatMatMult(S,G,MAT_INITIAL_MATRIX,PETSC_DEFAULT_DOUBLE_PRECISION,J,ierr);CHKERRQ(ierr) >> >> I use this operation to compute matrix-matrix product. It's correct, but quite slow. >> >> For instance, my matrix S is a dense matrix size of<6x762048 complex double> and matrix G is a sparse matrix with 2 nonzeros per row and size of<762048x254016 complex double> >> Time spent on S*G was ~22 secs with 32 processes over 4 nodes. There is infiniband between nodes. >> >> Is it a real life or I'm doing something wrong? >> >> Regards, >> Alexander From gdiso at ustc.edu Mon Mar 14 01:16:18 2011 From: gdiso at ustc.edu (Gong Ding) Date: Mon, 14 Mar 2011 14:16:18 +0800 (CST) Subject: [petsc-users] Howto understand the output of -ksp_monitor_singular_value Message-ID: <17437093.74221300083378914.JavaMail.coremail@mail.ustc.edu> Hi, I use -ksp_monitor_singular_value to view the approximate condition number of my matrix. The output during the GMRES iteration looks like 0 KSP Residual norm 1.072677079362e+02 % max 1 min 1 max/min 1 1 KSP Residual norm 2.558517256216e+01 % max 0.949986 min 0.949986 max/min 1 2 KSP Residual norm 7.952391160495e+00 % max 1.00443 min 0.53867 max/min 1.86465 3 KSP Residual norm 3.147058569003e+00 % max 1.01529 min 0.418892 max/min 2.42375 4 KSP Residual norm 1.213876539809e+00 % max 1.04559 min 0.365424 max/min 2.86131 5 KSP Residual norm 4.398018777420e-01 % max 1.0717 min 0.341344 max/min 3.13966 6 KSP Residual norm 1.681000721101e-01 % max 1.07182 min 0.333065 max/min 3.21806 7 KSP Residual norm 6.008534584667e-02 % max 1.07287 min 0.330693 max/min 3.24432 8 KSP Residual norm 2.317545636598e-02 % max 1.07476 min 0.32966 max/min 3.26019 9 KSP Residual norm 8.732395511044e-03 % max 1.0798 min 0.32928 max/min 3.27929 10 KSP Residual norm 3.543098250574e-03 % max 1.08891 min 0.328903 max/min 3.31073 11 KSP Residual norm 1.322114569421e-03 % max 1.1002 min 0.328779 max/min 3.34632 12 KSP Residual norm 4.278882091191e-04 % max 1.12562 min 0.328666 max/min 3.4248 13 KSP Residual norm 1.503136522978e-04 % max 1.13558 min 0.328572 max/min 3.45611 14 KSP Residual norm 5.410712876997e-05 % max 1.1531 min 0.32832 max/min 3.51211 15 KSP Residual norm 2.200160155820e-05 % max 1.16627 min 0.328089 max/min 3.55474 16 KSP Residual norm 9.689610704862e-06 % max 1.17549 min 0.327714 max/min 3.58694 17 KSP Residual norm 4.077334723619e-06 % max 1.18316 min 0.32763 max/min 3.61126 18 KSP Residual norm 1.522015252918e-06 % max 1.18949 min 0.327542 max/min 3.63158 In the first iteration, the min and max singular value are both 1, and the ratio of max/min increase quickly in the next iterations. >From above values, how to extract the condition number of (preconditioned) matrix? From jed at 59A2.org Mon Mar 14 07:15:58 2011 From: jed at 59A2.org (Jed Brown) Date: Mon, 14 Mar 2011 13:15:58 +0100 Subject: [petsc-users] Howto understand the output of -ksp_monitor_singular_value In-Reply-To: <17437093.74221300083378914.JavaMail.coremail@mail.ustc.edu> References: <17437093.74221300083378914.JavaMail.coremail@mail.ustc.edu> Message-ID: 2011/3/14 Gong Ding > > > >From above values, how to extract the condition number of (preconditioned) > matrix? > > The estimate comes from the Krylov polynomial which increases by one degree on each iteration. The final condition number estimate will almost always be the biggest and most accurate. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Mar 14 08:19:13 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 14 Mar 2011 08:19:13 -0500 Subject: [petsc-users] Howto understand the output of -ksp_monitor_singular_value In-Reply-To: <17437093.74221300083378914.JavaMail.coremail@mail.ustc.edu> References: <17437093.74221300083378914.JavaMail.coremail@mail.ustc.edu> Message-ID: <0DA70944-63F2-408D-87FE-8295E579F035@mcs.anl.gov> Arnoldi is converging very nicely for you so the condition number is likely 3.something You can run with -ksp_rtol 1.e-13 -ksp_gmres_restart 50 to get tighter convergence. Barry On Mar 14, 2011, at 1:16 AM, Gong Ding wrote: > Hi, > I use -ksp_monitor_singular_value to view the approximate condition number of my matrix. > The output during the GMRES iteration looks like > > 0 KSP Residual norm 1.072677079362e+02 % max 1 min 1 max/min 1 > 1 KSP Residual norm 2.558517256216e+01 % max 0.949986 min 0.949986 max/min 1 > 2 KSP Residual norm 7.952391160495e+00 % max 1.00443 min 0.53867 max/min 1.86465 > 3 KSP Residual norm 3.147058569003e+00 % max 1.01529 min 0.418892 max/min 2.42375 > 4 KSP Residual norm 1.213876539809e+00 % max 1.04559 min 0.365424 max/min 2.86131 > 5 KSP Residual norm 4.398018777420e-01 % max 1.0717 min 0.341344 max/min 3.13966 > 6 KSP Residual norm 1.681000721101e-01 % max 1.07182 min 0.333065 max/min 3.21806 > 7 KSP Residual norm 6.008534584667e-02 % max 1.07287 min 0.330693 max/min 3.24432 > 8 KSP Residual norm 2.317545636598e-02 % max 1.07476 min 0.32966 max/min 3.26019 > 9 KSP Residual norm 8.732395511044e-03 % max 1.0798 min 0.32928 max/min 3.27929 > 10 KSP Residual norm 3.543098250574e-03 % max 1.08891 min 0.328903 max/min 3.31073 > 11 KSP Residual norm 1.322114569421e-03 % max 1.1002 min 0.328779 max/min 3.34632 > 12 KSP Residual norm 4.278882091191e-04 % max 1.12562 min 0.328666 max/min 3.4248 > 13 KSP Residual norm 1.503136522978e-04 % max 1.13558 min 0.328572 max/min 3.45611 > 14 KSP Residual norm 5.410712876997e-05 % max 1.1531 min 0.32832 max/min 3.51211 > 15 KSP Residual norm 2.200160155820e-05 % max 1.16627 min 0.328089 max/min 3.55474 > 16 KSP Residual norm 9.689610704862e-06 % max 1.17549 min 0.327714 max/min 3.58694 > 17 KSP Residual norm 4.077334723619e-06 % max 1.18316 min 0.32763 max/min 3.61126 > 18 KSP Residual norm 1.522015252918e-06 % max 1.18949 min 0.327542 max/min 3.63158 > > > In the first iteration, the min and max singular value are both 1, and the ratio of max/min increase quickly > in the next iterations. > >> From above values, how to extract the condition number of (preconditioned) matrix? > From ning.an at ghiocel-tech.com Mon Mar 14 13:56:01 2011 From: ning.an at ghiocel-tech.com (ning.an at ghiocel-tech.com) Date: 14 Mar 2011 18:56:01 -0000 Subject: [petsc-users] Is there any code to demostrate how to write a code to use direct solvers, such as superlu, sptools, mumps... Message-ID: <20110314185601.1784.qmail@ns25.webmasters.com> Dear PETSc Expert, I'm a newbie to use PETSc. I'd like to test few direct complex linear equation via PETSc interface. I don't find any example from the website. Please tell me where I can find one or more example, so I can get a littile bit understanding, and try to write the similar code. Thanks. Ning -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Mar 14 14:06:04 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 14 Mar 2011 14:06:04 -0500 Subject: [petsc-users] Is there any code to demostrate how to write a code to use direct solvers, such as superlu, sptools, mumps... In-Reply-To: <20110314185601.1784.qmail@ns25.webmasters.com> References: <20110314185601.1784.qmail@ns25.webmasters.com> Message-ID: On Mon, Mar 14, 2011 at 1:56 PM, wrote: > Dear PETSc Expert, > > I'm a newbie to use PETSc. I'd like to test few direct complex > linear equation via PETSc interface. I don't find any example from the > website. Please tell me where I can find one or more example, so I can get a > littile bit understanding, and try to write the similar code. > KSP ex10 does exactly what you want. Just load in your matrix using MatLoad(). Matt > Thanks. > > Ning > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From austin at txcorp.com Mon Mar 14 15:17:35 2011 From: austin at txcorp.com (Travis Austin) Date: Mon, 14 Mar 2011 13:17:35 -0700 Subject: [petsc-users] ASM for High-Order FEM In-Reply-To: References: Message-ID: <417292B2-69A4-4247-BDAF-F153020419E2@txcorp.com> Hi, Often for high-order finite elements the subdomains in an additive Schwarz method (using PCASM) are defined according to the elements since they can contain a large number of degrees of freedom. This is an approach used extensively by Paul Fischer of Argonne National Lab in his work. I'm trying to code this up for an example that I have using high-order finite elements but I'm running into a few issues. Below is a trivial partitioning of a 1D problem that illustrates what I am trying to overcome. Processor P0 is going to have 3 ASM subdomains and each subdomain will fully be on the processor. However, processor P1 with 4 ASM subdomains will have one subdomain that is partly on processor P0. Is there a way that I can define the local subdomain in the ASM preconditioning for processor P1 corresponding to the middle most element so that the ASM preconditioning part for this element is performed on processor P1? Can I define an index set for processor P1 that includes the degree of freedom from the P0/P1 interface and expect that PETSc will handle that? P0 / ---------------------------------------\ | | | | | | | | |---x---x---X---x---x---X---x---x---X---x---x---X---x---x---X---x---x---X---x---x---| | | | | | | | | \--------------------------------------------------/ P1 Thanks, Travis Austin ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Travis Austin, Ph.D. Tech-X Corporation 5621 Arapahoe Ave, Suite A Boulder, CO 80303 austin at txcorp.com ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ On Oct 6, 2010, at 11:36 AM, Jed Brown wrote: > http://59A2.org/na/Brown-EfficientNonlinearSolversNodalHighOrder3D-2010.pdf -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Mar 14 15:23:35 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 14 Mar 2011 15:23:35 -0500 Subject: [petsc-users] ASM for High-Order FEM In-Reply-To: <417292B2-69A4-4247-BDAF-F153020419E2@txcorp.com> References: <417292B2-69A4-4247-BDAF-F153020419E2@txcorp.com> Message-ID: <3EEF39F1-EF8E-4D76-B95F-8B4B37203E19@mcs.anl.gov> On Mar 14, 2011, at 3:17 PM, Travis Austin wrote: > Hi, > > Often for high-order finite elements the subdomains in an additive Schwarz method (using PCASM) are defined according to the elements > since they can contain a large number of degrees of freedom. This is an approach used extensively by Paul Fischer of Argonne National > Lab in his work. I'm trying to code this up for an example that I have using high-order finite elements but I'm running into a few issues. > > Below is a trivial partitioning of a 1D problem that illustrates what I am trying to overcome. Processor P0 is going to have 3 ASM subdomains > and each subdomain will fully be on the processor. However, processor P1 with 4 ASM subdomains will have one subdomain that is partly on > processor P0. > > Is there a way that I can define the local subdomain in the ASM preconditioning for processor P1 corresponding to the middle most element so > that the ASM preconditioning part for this element is performed on processor P1? > > Can I define an index set for processor P1 that includes the degree of freedom from the P0/P1 interface and expect that PETSc will handle > that? Yes > > P0 > > / ---------------------------------------\ > > | | | | | | | | > |---x---x---X---x---x---X---x---x---X---x---x---X---x---x---X---x---x---X---x---x---| > | | | | | | | | > > \--------------------------------------------------/ > > > P1 The IS used to defined subdomains is done with global indices so on which process a degree of freedom lives in the matrix doesn't matter in regards to making subdomains for ASM. Barry > > Thanks, > Travis Austin > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > Travis Austin, Ph.D. > Tech-X Corporation > 5621 Arapahoe Ave, Suite A > Boulder, CO 80303 > austin at txcorp.com > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > > On Oct 6, 2010, at 11:36 AM, Jed Brown wrote: > >> http://59A2.org/na/Brown-EfficientNonlinearSolversNodalHighOrder3D-2010.pdf > From jed at 59A2.org Mon Mar 14 15:24:07 2011 From: jed at 59A2.org (Jed Brown) Date: Mon, 14 Mar 2011 21:24:07 +0100 Subject: [petsc-users] ASM for High-Order FEM In-Reply-To: <417292B2-69A4-4247-BDAF-F153020419E2@txcorp.com> References: <417292B2-69A4-4247-BDAF-F153020419E2@txcorp.com> Message-ID: On Mon, Mar 14, 2011 at 21:17, Travis Austin wrote: > Can I define an index set for processor P1 that includes the degree of > freedom from the P0/P1 interface and expect that PETSc will handle > that? > Did you have problems with PCASMSetLocalSubdomains()? It should work fine. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Mar 14 15:25:33 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 14 Mar 2011 15:25:33 -0500 Subject: [petsc-users] ASM for High-Order FEM In-Reply-To: <417292B2-69A4-4247-BDAF-F153020419E2@txcorp.com> References: <417292B2-69A4-4247-BDAF-F153020419E2@txcorp.com> Message-ID: On Mon, Mar 14, 2011 at 3:17 PM, Travis Austin wrote: > Hi, > > Often for high-order finite elements the subdomains in an additive Schwarz > method (using PCASM) are defined according to the elements > since they can contain a large number of degrees of freedom. This is an > approach used extensively by Paul Fischer of Argonne National > Lab in his work. I'm trying to code this up for an example that I have > using high-order finite elements but I'm running into a few issues. > > Below is a trivial partitioning of a 1D problem that illustrates what I am > trying to overcome. Processor P0 is going to have 3 ASM subdomains > and each subdomain will fully be on the processor. However, processor P1 > with 4 ASM subdomains will have one subdomain that is partly on > processor P0. > > Is there a way that I can define the local subdomain in the ASM > preconditioning for processor P1 corresponding to the middle most element > so > that the ASM preconditioning part for this element is performed on > processor P1? > > Can I define an index set for processor P1 that includes the degree of > freedom from the P0/P1 interface and expect that PETSc will handle > that? > You can define the block explicitly for ASM, http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/PC/PCASMSetLocalSubdomains.html If you want a single block per process, you can use PETSC_NULL for islocal and only define the overlapping blocks. Matt > P0 > > / ---------------------------------------\ > > | | | | > | | | | > > |---x---x---X---x---x---X---x---x---X---x---x---X---x---x---X---x---x---X---x---x---| > | | | | > | | | | > > > \--------------------------------------------------/ > > > P1 > > Thanks, > Travis Austin > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > Travis Austin, Ph.D. > Tech-X Corporation > 5621 Arapahoe Ave, Suite A > Boulder, CO 80303 > austin at txcorp.com > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > > On Oct 6, 2010, at 11:36 AM, Jed Brown wrote: > > http://59A2.org/na/Brown-EfficientNonlinearSolversNodalHighOrder3D-2010.pdf > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From ning.an at ghiocel-tech.com Mon Mar 14 15:36:42 2011 From: ning.an at ghiocel-tech.com (ning.an at ghiocel-tech.com) Date: 14 Mar 2011 20:36:42 -0000 Subject: [petsc-users] Is there any code to demostrate how to write a Message-ID: <20110314203642.3426.qmail@ns25.webmasters.com> Hi, Matt, Yes, it is just what I'm looking for. Thanks a lot. :) Ning ----- Original Message ----- From: knepley at gmail.comTo: petsc-users at mcs.anl.gov Sent: 3/14/11 3:06 PM Subject: SPAM -> Re: [petsc-users] Is there any code to demostrate how to write a On Mon, Mar 14, 2011 at 1:56 PM, wrote: Dear PETSc Expert, I'm a newbie to use PETSc. I'd like to test few direct complex linear equation via PETSc interface. I don't find any example from the website. Please tell me where I can find one or more example, so I can get a littile bit understanding, and try to write the similar code. KSP ex10 does exactly what you want. Just load in your matrix using MatLoad(). Matt Thanks. Ning -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From austin at txcorp.com Mon Mar 14 15:41:12 2011 From: austin at txcorp.com (Travis Austin) Date: Mon, 14 Mar 2011 13:41:12 -0700 Subject: [petsc-users] ASM for High-Order FEM Message-ID: Hi, Thanks for your responses. I saw a previous thread on this from Matt. If you want multiple blocks/process AND you want to be able to do RASM, then you need to specify both 1) the overlapping blocks 2) the non-overlapping parts You need 2), because I have no idea how to calculate 2) given only 1). We could have an alternative interface that took 2) and an overlap, and figured it out, however currently that only works for 1 block/process, and specfiying both 1) and 2) is more general (I needed it for a specific PC). The overlapping blocks are global indices and the non-overlapping are local indices? Is that right? Thanks, Travis ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Travis Austin, Ph.D. Tech-X Corporation 5621 Arapahoe Ave, Suite A Boulder, CO 80303 austin at txcorp.com ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Mar 14 15:42:10 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 14 Mar 2011 15:42:10 -0500 Subject: [petsc-users] ASM for High-Order FEM In-Reply-To: References: Message-ID: On Mon, Mar 14, 2011 at 3:41 PM, Travis Austin wrote: > Hi, > > Thanks for your responses. I saw a previous thread on this from Matt. > > I*f you want multiple blocks/process AND you want to be able to do > RASM, then > you need to specify both > > 1) the overlapping blocks > > 2) the non-overlapping parts > > You need 2), because I have no idea how to calculate 2) given only 1). > We > could have an alternative interface that > took 2) and an overlap, and figured it out, however currently that only > works for 1 block/process, and specfiying > both 1) and 2) is more general (I needed it for a specific PC).* > > The overlapping blocks are global indices and the non-overlapping are local > indices? Is that right? > No, both are global. Matt > Thanks, > Travis > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > Travis Austin, Ph.D. > Tech-X Corporation > 5621 Arapahoe Ave, Suite A > Boulder, CO 80303 > austin at txcorp.com > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Mon Mar 14 15:44:40 2011 From: jed at 59A2.org (Jed Brown) Date: Mon, 14 Mar 2011 21:44:40 +0100 Subject: [petsc-users] ASM for High-Order FEM In-Reply-To: References: Message-ID: On Mon, Mar 14, 2011 at 21:41, Travis Austin wrote: > The overlapping blocks are global indices and the non-overlapping are local > indices? Is that right? No, they are both global indices, "is_local" just defines the non-overlapping part of the domain so that RASM knows what to update. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Mar 14 15:47:13 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 14 Mar 2011 15:47:13 -0500 Subject: [petsc-users] ASM for High-Order FEM In-Reply-To: References: Message-ID: <1262E7A3-0FE2-482C-BBFE-D7D0E9823369@mcs.anl.gov> I am adding to the manual page the fact that both is and is_local are in global numbering. barry On Mar 14, 2011, at 3:42 PM, Matthew Knepley wrote: > On Mon, Mar 14, 2011 at 3:41 PM, Travis Austin wrote: > Hi, > > Thanks for your responses. I saw a previous thread on this from Matt. > > If you want multiple blocks/process AND you want to be able to do RASM, then > you need to specify both > > 1) the overlapping blocks > > 2) the non-overlapping parts > > You need 2), because I have no idea how to calculate 2) given only 1). We > could have an alternative interface that > took 2) and an overlap, and figured it out, however currently that only > works for 1 block/process, and specfiying > both 1) and 2) is more general (I needed it for a specific PC). > > The overlapping blocks are global indices and the non-overlapping are local indices? Is that right? > > No, both are global. > > Matt > > Thanks, > Travis > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > Travis Austin, Ph.D. > Tech-X Corporation > 5621 Arapahoe Ave, Suite A > Boulder, CO 80303 > austin at txcorp.com > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From Eric.Preston at itt.com Mon Mar 14 18:25:08 2011 From: Eric.Preston at itt.com (Preston, Eric - IS) Date: Mon, 14 Mar 2011 19:25:08 -0400 Subject: [petsc-users] Building with MKL 10.3 Message-ID: Greetings, I had some difficulty compiling PETSc with MKL 10.3 (on 64-bit linux). The config system definitely wasn't able to deal with this version. For a sequential, 32-bit integer compile the Intel link advisor tells me to use the following: -Wl,--start-group $(MKLROOT)/lib/intel64/libmkl_intel_lp64.a $(MKLROOT)/lib/intel64/libmkl_sequential.a $(MKLROOT)/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread So I added this line to BlasLapack.py (in generateGuesses, around line 210): yield ('User specified MKL10.3 Linux-intel64 installation root', None, [os.path.join(dir,'lib','intel64','libmkl_lapack95_lp64.a'),'mkl_intel_lp64', 'mkl_sequential', 'mkl_core', 'pthread'], 1) #efp Note in this distribution the LAPACK libs are called libmkl_lapack95_ilp64.a and libmkl_lapack95_lp64.a (for 64 and 32-bit integers, respectively). Also note this version uses /lib/intel64 instead of /lib/emt64. It came with Intel ComposerXE 2011 (the icc version is 12.0.2). I was able the build the PETSC libraries in this way, however to use the multi-threaded MKL I had to change 'libmkl_sequential.a' to 'libmkl_intel_thread.a' AND add the compiler option -openmp. To get this to work I had to add CFLAGS=-openmp. Now on to adding mpi... Eric Preston This e-mail and any files transmitted with it may be proprietary and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the sender. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of ITT Corporation. The recipient should check this e-mail and any attachments for the presence of viruses. ITT accepts no liability for any damage caused by any virus transmitted by this e-mail. From bsmith at mcs.anl.gov Mon Mar 14 18:48:21 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 14 Mar 2011 18:48:21 -0500 Subject: [petsc-users] Building with MKL 10.3 In-Reply-To: References: Message-ID: Eric, With the current PETSc design that uses one MPI process per core I am not sure that it makes sense to use multi-threaded MKL since the processes are already using all the cores. I could be wrong. Barry On Mar 14, 2011, at 6:25 PM, Preston, Eric - IS wrote: > > Greetings, > > I had some difficulty compiling PETSc with MKL 10.3 (on 64-bit linux). The config system definitely wasn't able to deal with this version. For a sequential, 32-bit integer compile the Intel link advisor tells me to use the following: > > -Wl,--start-group $(MKLROOT)/lib/intel64/libmkl_intel_lp64.a $(MKLROOT)/lib/intel64/libmkl_sequential.a $(MKLROOT)/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread > > So I added this line to BlasLapack.py (in generateGuesses, around line 210): > > yield ('User specified MKL10.3 Linux-intel64 installation root', None, [os.path.join(dir,'lib','intel64','libmkl_lapack95_lp64.a'),'mkl_intel_lp64', 'mkl_sequential', 'mkl_core', 'pthread'], 1) #efp > > Note in this distribution the LAPACK libs are called libmkl_lapack95_ilp64.a and libmkl_lapack95_lp64.a (for 64 and 32-bit integers, respectively). Also note this version uses /lib/intel64 instead of /lib/emt64. It came with Intel ComposerXE 2011 (the icc version is 12.0.2). > > I was able the build the PETSC libraries in this way, however to use the multi-threaded MKL I had to change 'libmkl_sequential.a' to 'libmkl_intel_thread.a' AND add the compiler option -openmp. To get this to work I had to add CFLAGS=-openmp. > > Now on to adding mpi... > > Eric Preston > > > This e-mail and any files transmitted with it may be proprietary and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the sender. > Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of ITT Corporation. The recipient should check this e-mail and any attachments for the presence of viruses. ITT accepts no liability for any damage caused by any virus transmitted by this e-mail. From dalcinl at gmail.com Mon Mar 14 19:48:48 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 14 Mar 2011 21:48:48 -0300 Subject: [petsc-users] Building with MKL 10.3 In-Reply-To: References: Message-ID: On 14 March 2011 20:48, Barry Smith wrote: > > ?Eric, > > ? With the current PETSc design that uses one MPI process per core I am not sure that it makes sense to use multi-threaded MKL since the processes are already using all the cores. I could be wrong. > > ? Barry > Well, I think that PETSc should support sequential codes using multi-threaded blas-lapack and perhaps some OpenMP. Moreover, perhaps we could get some little speedup in MatMult for these kinds of users/applications. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From knepley at gmail.com Mon Mar 14 19:54:04 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 14 Mar 2011 19:54:04 -0500 Subject: [petsc-users] Building with MKL 10.3 In-Reply-To: References: Message-ID: On Mon, Mar 14, 2011 at 7:48 PM, Lisandro Dalcin wrote: > On 14 March 2011 20:48, Barry Smith wrote: > > > > Eric, > > > > With the current PETSc design that uses one MPI process per core I am > not sure that it makes sense to use multi-threaded MKL since the processes > are already using all the cores. I could be wrong. > > > > Barry > > > > Well, I think that PETSc should support sequential codes using > multi-threaded blas-lapack and perhaps some OpenMP. Moreover, perhaps > we could get some little speedup in MatMult for these kinds of > users/applications. > I have never ever seen convincing evidence of this. First, you would need enough bandwidth to satisfy 2+ cores. This is almost never the case. But suppose you do have this. Then you would need a convincing reason to use threads instead of MPI processes, which would mean data reuse. But there is very little reue here; it is mostly streaming. Matt > -- > Lisandro Dalcin > --------------- > CIMEC (INTEC/CONICET-UNL) > Predio CONICET-Santa Fe > Colectora RN 168 Km 472, Paraje El Pozo > 3000 Santa Fe, Argentina > Tel: +54-342-4511594 (ext 1011) > Tel/Fax: +54-342-4511169 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Mon Mar 14 20:13:17 2011 From: jed at 59A2.org (Jed Brown) Date: Tue, 15 Mar 2011 02:13:17 +0100 Subject: [petsc-users] Building with MKL 10.3 In-Reply-To: References: Message-ID: Matt, it makes sense for small subdomains in 3D with wide stencils. It's not difficult to cut the memory by a third or so with 8 subdomains. Of course you also need a preconditioner that makes sense on multicore and a threading scheme with decent memory guarantees on NUMA. On Mar 15, 2011 1:54 AM, "Matthew Knepley" wrote: On Mon, Mar 14, 2011 at 7:48 PM, Lisandro Dalcin wrote: > > On 14 March 2011 20:48, Barry Smith wrote: > > > > Eric, > > > > With th... I have never ever seen convincing evidence of this. First, you would need enough bandwidth to satisfy 2+ cores. This is almost never the case. But suppose you do have this. Then you would need a convincing reason to use threads instead of MPI processes, which would mean data reuse. But there is very little reue here; it is mostly streaming. Matt > -- > > > > Lisandro Dalcin > > --------------- > > CIMEC (INTEC/CONICET-UNL) > > Predio CONICET-Santa Fe > > Colecto... > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Mar 14 20:23:09 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 14 Mar 2011 20:23:09 -0500 Subject: [petsc-users] Building with MKL 10.3 In-Reply-To: References: Message-ID: <7B5D0AAE-2B6E-4DB5-AA76-21EB0DE18F15@mcs.anl.gov> On Mar 14, 2011, at 7:48 PM, Lisandro Dalcin wrote: > On 14 March 2011 20:48, Barry Smith wrote: >> >> Eric, >> >> With the current PETSc design that uses one MPI process per core I am not sure that it makes sense to use multi-threaded MKL since the processes are already using all the cores. I could be wrong. >> >> Barry >> > > Well, I think that PETSc should support sequential codes using > multi-threaded blas-lapack That is support and someone can do it just by using parallel blas. > and perhaps some OpenMP. Again users can use openmp in their sequential PETSc code, we don't stop them. Barry > Moreover, perhaps > we could get some little speedup in MatMult for these kinds of > users/applications. If you aren't willing to pay the entry fee to parallel computing with MPI then you shouldn't be allowed any benifits of parallel PETSc :-). Actually we will eventually support the "hybrid" model. > > -- > Lisandro Dalcin > --------------- > CIMEC (INTEC/CONICET-UNL) > Predio CONICET-Santa Fe > Colectora RN 168 Km 472, Paraje El Pozo > 3000 Santa Fe, Argentina > Tel: +54-342-4511594 (ext 1011) > Tel/Fax: +54-342-4511169 From hung.thanh.nguyen at petrell.no Tue Mar 15 07:33:16 2011 From: hung.thanh.nguyen at petrell.no (Hung Thanh Nguyen) Date: Tue, 15 Mar 2011 13:33:16 +0100 Subject: [petsc-users] Problem with Petcs link to window Intel MKl mkl_blas95.lib, mkl_lapack95.lib and mkl_blas_intelmpi.lib Message-ID: Dear PETSc use I try to install PETSc on window Intel MKL and include mkl_blas95.lib, mkl_lapack95.lib and mkl_blacs_intelmpi.lib ( or mkl_mpich2.lib). Intel MKL compiler 11.6-067. The following config-procedure ..... $ ./config/configure.py / --with-petsc-arch=intel-32-static / --with-f-blas95-lapack-dir=/intel/Compiler/11.1/067/mkl/ia32/lib / --with-mpich2-dir= =/intel/Compiler/11.1/067/mkl/ia32/lib I am also try: $ ./config/configure.py / --with-petsc-arch=intel-32-static / --with-f-blas-lapack-dir=/intel/Compiler/11.1/067/mkl/ia32/lib / --with-mpi-dir= =/intel/Compiler/11.1/067/mkl/ia32/lib I got error message: Could not locate a functional C compiler What I am doing wrong? Thanks Hung T Nguyen Physis and techonology Inst. University in Troms? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 58421 bytes Desc: configure.log URL: From agrayver at gfz-potsdam.de Tue Mar 15 07:47:26 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Tue, 15 Mar 2011 13:47:26 +0100 Subject: [petsc-users] Problem with Petcs link to window Intel MKl mkl_blas95.lib, mkl_lapack95.lib and mkl_blas_intelmpi.lib In-Reply-To: References: Message-ID: <4D7F5FDE.9090304@gfz-potsdam.de> I would suggest that you have to define the name of the compiler which will be used. In case of Intel compiler I guess it should be: --with-cc='win32fe icc' --with-fc='win32fe ifort' --with-cxx='win32fe icc' On 15.03.2011 13:33, Hung Thanh Nguyen wrote: > > Dear PETSc use > > I try to install PETSc on window Intel MKL and include mkl_blas95.lib, > mkl_lapack95.lib and mkl_blacs_intelmpi.lib ( or mkl_mpich2.lib). > > Intel MKL compiler 11.6-067. The following config-procedure > > ..... > > /$ ./config/configure.py / --with-petsc-arch=intel-32-static / > --with-f-blas95-lapack-dir=/intel/Compiler/11.1/067/mkl/ia32/lib / > --with-mpich2-dir=/ > > /=/intel/Compiler/11.1/067/mkl/ia32/lib/ > > I am also try: > > /$ ./config/configure.py / --with-petsc-arch=intel-32-static / > --with-f-blas-lapack-dir=/intel/Compiler/11.1/067/mkl/ia32/lib / > --with-mpi-dir=/ > > /=/intel/Compiler/11.1/067/mkl/ia32/lib/ > > *I got error message:* > > *Could not locate a functional C compiler * > > ** > > What I am doing wrong? > > Thanks > > Hung T Nguyen > > Physis and techonology Inst. University in Troms? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yifan_zhang at brown.edu Tue Mar 15 08:21:31 2011 From: yifan_zhang at brown.edu (Zhang, Yifan) Date: Tue, 15 Mar 2011 09:21:31 -0400 Subject: [petsc-users] Example of PCShell Message-ID: Dear Petsc users, I am trying to preconditioning spectral method with 2nd order finite difference. So just writing to ask if anyone got a good Petsc examples of using PCShell in fortran other than ex15.F . Thanks very much. Best, -- Yifan Zhang Division of Applied Mathematics Brown University, Providence, RI02912 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Tue Mar 15 08:28:25 2011 From: jed at 59A2.org (Jed Brown) Date: Tue, 15 Mar 2011 14:28:25 +0100 Subject: [petsc-users] Example of PCShell In-Reply-To: References: Message-ID: On Tue, Mar 15, 2011 at 14:21, Zhang, Yifan wrote: > I am trying to preconditioning spectral method with 2nd order finite > difference. > You might be interested in https://github.com/jedbrown/spectral-petsc which does this for Chebyshev spectral methods where differentiation is implemented using FFTW. There are examples for a scalar elliptic problem and 3D Stokes. It may need updates to work with petsc-3.1 or petsc-dev. > So just writing to ask if anyone got a good Petsc examples of > using PCShell in fortran other than ex15.F . > There is src/snes/examples/tutorials/ex33f.F, but that example can't be built because the common block definition is missing. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hung.thanh.nguyen at petrell.no Tue Mar 15 08:25:38 2011 From: hung.thanh.nguyen at petrell.no (Hung Thanh Nguyen) Date: Tue, 15 Mar 2011 14:25:38 +0100 Subject: [petsc-users] Problem with Petcs link to window Intel MKl mkl_blas95.lib, mkl_lapack95.lib and mkl_blas_intelmpi.lib In-Reply-To: <4D7F5FDE.9090304@gfz-potsdam.de> References: <4D7F5FDE.9090304@gfz-potsdam.de> Message-ID: Hi petcs-using Thank for fast responding Grayver. I try: ./config/configure.py --with-cc='win32fe icc' --with-fc='win32fe ifort' --with-cxx='win32fe icc' and I got the following error message: Could not found C-compilers I try : ./config/configure.py --with-cc='win32fe icc' --with-fc='win32fe ifort' --with-cxx='win32fe icc' and got the error message: C compiler you provided with --with-cc='win32fe icc' does not work, try -with-batch I try again and got the new following error message: ./config/configure.py -with-batch --with-cc='win32fe icc' --with-fc='win32fe f95' --with-cxx='win32fe icc' and got the new following error message: UNABLE to execute Binaries for congig/configure.py I Hope that you can help me to solve the problem. Thank a lot. Hung T. Nguyen From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Alexander Grayver Sent: 15. mars 2011 13:47 To: PETSc users list Subject: Re: [petsc-users] Problem with Petcs link to window Intel MKl mkl_blas95.lib, mkl_lapack95.lib and mkl_blas_intelmpi.lib I would suggest that you have to define the name of the compiler which will be used. In case of Intel compiler I guess it should be: --with-cc='win32fe icc' --with-fc='win32fe ifort' --with-cxx='win32fe icc' On 15.03.2011 13:33, Hung Thanh Nguyen wrote: Dear PETSc use I try to install PETSc on window Intel MKL and include mkl_blas95.lib, mkl_lapack95.lib and mkl_blacs_intelmpi.lib ( or mkl_mpich2.lib). Intel MKL compiler 11.6-067. The following config-procedure ..... $ ./config/configure.py / --with-petsc-arch=intel-32-static / --with-f-blas95-lapack-dir=/intel/Compiler/11.1/067/mkl/ia32/lib / --with-mpich2-dir= =/intel/Compiler/11.1/067/mkl/ia32/lib I am also try: $ ./config/configure.py / --with-petsc-arch=intel-32-static / --with-f-blas-lapack-dir=/intel/Compiler/11.1/067/mkl/ia32/lib / --with-mpi-dir= =/intel/Compiler/11.1/067/mkl/ia32/lib I got error message: Could not locate a functional C compiler What I am doing wrong? Thanks Hung T Nguyen Physis and techonology Inst. University in Troms? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 34615 bytes Desc: configure.log URL: From balay at mcs.anl.gov Tue Mar 15 08:51:17 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 15 Mar 2011 08:51:17 -0500 (CDT) Subject: [petsc-users] Problem with Petcs link to window Intel MKl mkl_blas95.lib, mkl_lapack95.lib and mkl_blas_intelmpi.lib In-Reply-To: References: <4D7F5FDE.9090304@gfz-potsdam.de> Message-ID: for one - intel compiler on windows is icl [not icc]. Secondly what compilers do you wish to use with petsc? > --with-f-blas95-lapack-dir=/intel/Compiler/11.1/067/mkl/ia32/lib / --with-mpich2-dir= =/intel/Compiler/11.1/067/mkl/ia32/lib Also there are no --with-f-blas95-lapack-dir --with-mpich2-dir configure options to petsc. Check the installation instructions at: http://www.mcs.anl.gov/petsc/petsc-as/documentation/installation.html Use options: --with-blas-lapack-lib --with-mpi-dir Note: If you don't really need petsc on windows - suggest using linux instead and avoid some pain.. If you still have issues send configure.log to petsc-maint. satish On Tue, 15 Mar 2011, Hung Thanh Nguyen wrote: > Hi petcs-using > > Thank for fast responding Grayver. I try: > ./config/configure.py --with-cc='win32fe icc' --with-fc='win32fe ifort' --with-cxx='win32fe icc' > > and I got the following error message: > Could not found C-compilers > I try : > ./config/configure.py --with-cc='win32fe icc' --with-fc='win32fe ifort' --with-cxx='win32fe icc' > > and got the error message: > C compiler you provided with --with-cc='win32fe icc' does not work, try -with-batch > > I try again and got the new following error message: > ./config/configure.py -with-batch --with-cc='win32fe icc' --with-fc='win32fe f95' --with-cxx='win32fe icc' > and got the new following error message: > UNABLE to execute Binaries for congig/configure.py > > I Hope that you can help me to solve the problem. Thank a lot. > Hung T. Nguyen > > > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Alexander Grayver > Sent: 15. mars 2011 13:47 > To: PETSc users list > Subject: Re: [petsc-users] Problem with Petcs link to window Intel MKl mkl_blas95.lib, mkl_lapack95.lib and mkl_blas_intelmpi.lib > > I would suggest that you have to define the name of the compiler which will be used. In case of Intel compiler I guess it should be: > > --with-cc='win32fe icc' --with-fc='win32fe ifort' --with-cxx='win32fe icc' > > On 15.03.2011 13:33, Hung Thanh Nguyen wrote: > Dear PETSc use > I try to install PETSc on window Intel MKL and include mkl_blas95.lib, mkl_lapack95.lib and mkl_blacs_intelmpi.lib ( or mkl_mpich2.lib). > Intel MKL compiler 11.6-067. The following config-procedure > ..... > $ ./config/configure.py / --with-petsc-arch=intel-32-static / --with-f-blas95-lapack-dir=/intel/Compiler/11.1/067/mkl/ia32/lib / --with-mpich2-dir= > =/intel/Compiler/11.1/067/mkl/ia32/lib > > I am also try: > $ ./config/configure.py / --with-petsc-arch=intel-32-static / --with-f-blas-lapack-dir=/intel/Compiler/11.1/067/mkl/ia32/lib / --with-mpi-dir= > =/intel/Compiler/11.1/067/mkl/ia32/lib > > I got error message: > Could not locate a functional C compiler > > What I am doing wrong? > > Thanks > Hung T Nguyen > Physis and techonology Inst. University in Troms? > > > > From csnataraj at gmail.com Tue Mar 15 10:36:23 2011 From: csnataraj at gmail.com (Natarajan CS) Date: Tue, 15 Mar 2011 10:36:23 -0500 Subject: [petsc-users] Building with MKL 10.3 In-Reply-To: <7B5D0AAE-2B6E-4DB5-AA76-21EB0DE18F15@mcs.anl.gov> References: <7B5D0AAE-2B6E-4DB5-AA76-21EB0DE18F15@mcs.anl.gov> Message-ID: Hello, I am neither a regular Petsc user nor contributor so preemptive apologies if I am completely off the line here. I am not sure if the original poster had hyper-threading in mind when he asked about multi-threading, in case that was the idea, I don't think using petsc with MKL (HT) is going to give any benefit, I don't think MKL is really resource insensitive. Also I wonder what percentage of the code is actually blas/lapack intensive to make any significant dent in wall cock? of course +1 to everything else posed above. Cheers, C.S.N On Mon, Mar 14, 2011 at 8:23 PM, Barry Smith wrote: > > On Mar 14, 2011, at 7:48 PM, Lisandro Dalcin wrote: > > > On 14 March 2011 20:48, Barry Smith wrote: > >> > >> Eric, > >> > >> With the current PETSc design that uses one MPI process per core I am > not sure that it makes sense to use multi-threaded MKL since the processes > are already using all the cores. I could be wrong. > >> > >> Barry > >> > > > > Well, I think that PETSc should support sequential codes using > > multi-threaded blas-lapack > > That is support and someone can do it just by using parallel blas. > > > and perhaps some OpenMP. > > Again users can use openmp in their sequential PETSc code, we don't stop > them. > > > Barry > > > > Moreover, perhaps > > we could get some little speedup in MatMult for these kinds of > > users/applications. > > If you aren't willing to pay the entry fee to parallel computing with > MPI then you shouldn't be allowed any benifits of parallel PETSc :-). > Actually we will eventually support the "hybrid" model. > > > > > -- > > Lisandro Dalcin > > --------------- > > CIMEC (INTEC/CONICET-UNL) > > Predio CONICET-Santa Fe > > Colectora RN 168 Km 472, Paraje El Pozo > > 3000 Santa Fe, Argentina > > Tel: +54-342-4511594 (ext 1011) > > Tel/Fax: +54-342-4511169 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Tue Mar 15 10:48:20 2011 From: jed at 59A2.org (Jed Brown) Date: Tue, 15 Mar 2011 16:48:20 +0100 Subject: [petsc-users] Building with MKL 10.3 In-Reply-To: References: <7B5D0AAE-2B6E-4DB5-AA76-21EB0DE18F15@mcs.anl.gov> Message-ID: On Tue, Mar 15, 2011 at 16:36, Natarajan CS wrote: > Also I wonder what percentage of the code is actually blas/lapack intensive > to make any significant dent in wall cock? Rather little of PETSc is dependent on dense linear algebra. Some third-party direct solvers use it in their numerical factorization routines. Otherwise, it's mostly just vector operations which tend to be bandwidth limited anyway and are not so sensitive to implementation. Also, it is much more common for the majority of run time to be in matrix kernels than pure vector operations. Note that while HT is effective at covering stalls due to irregular memory access, it's not so good for tight kernels or purely bandwidth-limited tasks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From csnataraj at gmail.com Tue Mar 15 11:18:56 2011 From: csnataraj at gmail.com (Natarajan CS) Date: Tue, 15 Mar 2011 11:18:56 -0500 Subject: [petsc-users] Building with MKL 10.3 In-Reply-To: References: <7B5D0AAE-2B6E-4DB5-AA76-21EB0DE18F15@mcs.anl.gov> Message-ID: Thanks for the clarification Jed. I am involved in numerical software development but never have come across code that drastically changes performance based on blas/lapack implementation (for exactly the same reason you mentioned), so I was trying to learn what the issue might be. I certainly agree with your comment on HT, the idea is to maintain computational throughput, I guess if you have a high ILP or full registers it really doesn't help! On Tue, Mar 15, 2011 at 10:48 AM, Jed Brown wrote: > On Tue, Mar 15, 2011 at 16:36, Natarajan CS wrote: > >> Also I wonder what percentage of the code is actually blas/lapack >> intensive to make any significant dent in wall cock? > > > Rather little of PETSc is dependent on dense linear algebra. Some > third-party direct solvers use it in their numerical factorization routines. > Otherwise, it's mostly just vector operations which tend to be bandwidth > limited anyway and are not so sensitive to implementation. Also, it is much > more common for the majority of run time to be in matrix kernels than pure > vector operations. Note that while HT is effective at covering stalls due to > irregular memory access, it's not so good for tight kernels or purely > bandwidth-limited tasks. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Eric.Preston at itt.com Tue Mar 15 15:55:25 2011 From: Eric.Preston at itt.com (Preston, Eric - IS) Date: Tue, 15 Mar 2011 16:55:25 -0400 Subject: [petsc-users] Building with MKL 10.3 In-Reply-To: References: Message-ID: As the original poster, I can say I didn't give much thought to using the threaded vs. non-threaded MKL libraries, I was just sharing a solution to building with a different MKL variant (which should probably be incorporated into the config scripts.) However, the discussion it spawned was informative. The main consideration there was not performance but compatibility with your other libraries. I know on windows, for instance, different run-time libraries are used in each case and all your components must be compiled with the same option. Thankfully, I'm not using windows for this project, so it might not make any difference. Eric Preston > Hello, > I am neither a regular Petsc user nor contributor so preemptive >apologies if I am completely off the line here. > >I am not sure if the original poster had hyper-threading in mind when he >asked about multi-threading, in case that was the idea, I don't think using >petsc with MKL (HT) is going to give any benefit, I don't think MKL is >really resource insensitive. > >Also I wonder what percentage of the code is actually blas/lapack intensive >to make any significant dent in wall cock? > >of course +1 to everything else posed above. > >Cheers, > >C.S.N This e-mail and any files transmitted with it may be proprietary and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the sender. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of ITT Corporation. The recipient should check this e-mail and any attachments for the presence of viruses. ITT accepts no liability for any damage caused by any virus transmitted by this e-mail. From Robert.Ellis at geosoft.com Tue Mar 15 16:30:41 2011 From: Robert.Ellis at geosoft.com (Robert Ellis) Date: Tue, 15 Mar 2011 21:30:41 +0000 Subject: [petsc-users] Building with MKL 10.3 In-Reply-To: References: Message-ID: <18205E5ECD2A1A4584F2BFC0BCBDE95526D59A46@exchange.geosoft.com> All, Coincidentally, I have just spent much of the last two weeks testing the effect of the latest MKL blas-lapack release on a large domain decomposition style PETSc application, MPICH2, Windows, Intel Fortran, VS 2010 C++, all static linked, no hyperthreading, multi-blade, latest dual hex core Xeons, 25GB ram. Each core contains a domain and a shell is used for the matrix operations. Regardless of setting the number of threads for MKL or OMP, the MKL performance was worse than simply using --download-f-blas-lapack=1. My interpretation is that the decomposition of one domain per core effectively saturates the hardware and performance is actually degraded using the more sophisticated MKL. And, yes, I know that Windows is less than ideal for this type of work but there are other constraints... Cheers, Rob -----Original Message----- From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Preston, Eric - IS Sent: Tuesday, March 15, 2011 1:55 PM To: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Building with MKL 10.3 As the original poster, I can say I didn't give much thought to using the threaded vs. non-threaded MKL libraries, I was just sharing a solution to building with a different MKL variant (which should probably be incorporated into the config scripts.) However, the discussion it spawned was informative. The main consideration there was not performance but compatibility with your other libraries. I know on windows, for instance, different run-time libraries are used in each case and all your components must be compiled with the same option. Thankfully, I'm not using windows for this project, so it might not make any difference. Eric Preston > Hello, > I am neither a regular Petsc user nor contributor so preemptive >apologies if I am completely off the line here. > >I am not sure if the original poster had hyper-threading in mind when he >asked about multi-threading, in case that was the idea, I don't think using >petsc with MKL (HT) is going to give any benefit, I don't think MKL is >really resource insensitive. > >Also I wonder what percentage of the code is actually blas/lapack intensive >to make any significant dent in wall cock? > >of course +1 to everything else posed above. > >Cheers, > >C.S.N This e-mail and any files transmitted with it may be proprietary and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the sender. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of ITT Corporation. The recipient should check this e-mail and any attachments for the presence of viruses. ITT accepts no liability for any damage caused by any virus transmitted by this e-mail. From jed at 59A2.org Tue Mar 15 16:35:31 2011 From: jed at 59A2.org (Jed Brown) Date: Tue, 15 Mar 2011 22:35:31 +0100 Subject: [petsc-users] Building with MKL 10.3 In-Reply-To: <18205E5ECD2A1A4584F2BFC0BCBDE95526D59A46@exchange.geosoft.com> References: <18205E5ECD2A1A4584F2BFC0BCBDE95526D59A46@exchange.geosoft.com> Message-ID: On Tue, Mar 15, 2011 at 22:30, Robert Ellis wrote: > Regardless of setting the number of threads for MKL or OMP, the MKL > performance was worse than simply using --download-f-blas-lapack=1. Interesting. Does this statement include using just one thread, perhaps with a non-threaded MKL? Also, when you used threading, were you putting an MPI process on every core or were you making sure that you had enough cores for num_mpi_processes * num_mkl_threads? -------------- next part -------------- An HTML attachment was scrubbed... URL: From csnataraj at gmail.com Tue Mar 15 17:20:25 2011 From: csnataraj at gmail.com (Natarajan CS) Date: Tue, 15 Mar 2011 17:20:25 -0500 Subject: [petsc-users] Building with MKL 10.3 In-Reply-To: References: <18205E5ECD2A1A4584F2BFC0BCBDE95526D59A46@exchange.geosoft.com> Message-ID: Thanks Eric and Rob. Indeed! Was MKL_DYNAMIC set to default (true)? It looks like using 1 thread per core (sequential MKL) is the right thing to do as baseline. I would think that the performance of #cores = num_mpi_processes * num_mkl_threads might be <= #cores = num_mpi_processes case (# cores const) unless some cache effects come into play (Not sure what, I would think the mkl installation should weed these issues out). P.S : Out of curiosity have you also tested your app on Nehalem? Any difference between Nehalem vs Westmere for similar bandwidth? On Tue, Mar 15, 2011 at 4:35 PM, Jed Brown wrote: > On Tue, Mar 15, 2011 at 22:30, Robert Ellis wrote: > >> Regardless of setting the number of threads for MKL or OMP, the MKL >> performance was worse than simply using --download-f-blas-lapack=1. > > > Interesting. Does this statement include using just one thread, perhaps > with a non-threaded MKL? Also, when you used threading, were you putting an > MPI process on every core or were you making sure that you had enough cores > for num_mpi_processes * num_mkl_threads? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.G.Ellis at Shaw.ca Tue Mar 15 17:32:44 2011 From: Robert.G.Ellis at Shaw.ca (Rob Ellis) Date: Tue, 15 Mar 2011 15:32:44 -0700 Subject: [petsc-users] Building with MKL 10.3 In-Reply-To: References: <18205E5ECD2A1A4584F2BFC0BCBDE95526D59A46@exchange.geosoft.com> Message-ID: <006301cbe360$e77b63a0$b6722ae0$@G.Ellis@Shaw.ca> Yes, MKL_DYNAMIC was set to true. No, I haven't tested on Nehalem. I'm currently comparing sequential MKL with --download-f-blas-lapack=1. Rob From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Natarajan CS Sent: Tuesday, March 15, 2011 3:20 PM To: PETSc users list Cc: Robert Ellis Subject: Re: [petsc-users] Building with MKL 10.3 Thanks Eric and Rob. Indeed! Was MKL_DYNAMIC set to default (true)? It looks like using 1 thread per core (sequential MKL) is the right thing to do as baseline. I would think that the performance of #cores = num_mpi_processes * num_mkl_threads might be <= #cores = num_mpi_processes case (# cores const) unless some cache effects come into play (Not sure what, I would think the mkl installation should weed these issues out). P.S : Out of curiosity have you also tested your app on Nehalem? Any difference between Nehalem vs Westmere for similar bandwidth? On Tue, Mar 15, 2011 at 4:35 PM, Jed Brown wrote: On Tue, Mar 15, 2011 at 22:30, Robert Ellis wrote: Regardless of setting the number of threads for MKL or OMP, the MKL performance was worse than simply using --download-f-blas-lapack=1. Interesting. Does this statement include using just one thread, perhaps with a non-threaded MKL? Also, when you used threading, were you putting an MPI process on every core or were you making sure that you had enough cores for num_mpi_processes * num_mkl_threads? -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Ellis at geosoft.com Wed Mar 16 16:22:11 2011 From: Robert.Ellis at geosoft.com (Robert Ellis) Date: Wed, 16 Mar 2011 21:22:11 +0000 Subject: [petsc-users] Building with MKL 10.3 In-Reply-To: <006301cbe360$e77b63a0$b6722ae0$@G.Ellis@Shaw.ca> References: <18205E5ECD2A1A4584F2BFC0BCBDE95526D59A46@exchange.geosoft.com> <006301cbe360$e77b63a0$b6722ae0$@G.Ellis@Shaw.ca> Message-ID: <18205E5ECD2A1A4584F2BFC0BCBDE95526D5B17C@exchange.geosoft.com> Hi All, For those still interested in this thread, timing tests with MKL indicate that sequential MKL performs approximately the same as parallel MKL with NUM_THREADS=1, which isn't too surprising. What is a bit surprising is that MKL always, at least for this application, gives significantly slower performance than direct compilation of the code from --download-f-blas-lapack=1. My conclusion is that if your code is written with explicit parallelization, in this case using PETSc, and fully utilizes your hardware, using sophisticated libraries may actually harm performance. Keep it simple! Now a question: all my tests used MPICH2. Does anyone think using Intel MPI would significantly improve the performance of MKL with PETSc? Cheers, Rob From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Rob Ellis Sent: Tuesday, March 15, 2011 3:33 PM To: 'PETSc users list' Subject: Re: [petsc-users] Building with MKL 10.3 Yes, MKL_DYNAMIC was set to true. No, I haven't tested on Nehalem. I'm currently comparing sequential MKL with --download-f-blas-lapack=1. Rob From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Natarajan CS Sent: Tuesday, March 15, 2011 3:20 PM To: PETSc users list Cc: Robert Ellis Subject: Re: [petsc-users] Building with MKL 10.3 Thanks Eric and Rob. Indeed! Was MKL_DYNAMIC set to default (true)? It looks like using 1 thread per core (sequential MKL) is the right thing to do as baseline. I would think that the performance of #cores = num_mpi_processes * num_mkl_threads might be <= #cores = num_mpi_processes case (# cores const) unless some cache effects come into play (Not sure what, I would think the mkl installation should weed these issues out). P.S : Out of curiosity have you also tested your app on Nehalem? Any difference between Nehalem vs Westmere for similar bandwidth? On Tue, Mar 15, 2011 at 4:35 PM, Jed Brown > wrote: On Tue, Mar 15, 2011 at 22:30, Robert Ellis > wrote: Regardless of setting the number of threads for MKL or OMP, the MKL performance was worse than simply using --download-f-blas-lapack=1. Interesting. Does this statement include using just one thread, perhaps with a non-threaded MKL? Also, when you used threading, were you putting an MPI process on every core or were you making sure that you had enough cores for num_mpi_processes * num_mkl_threads? -------------- next part -------------- An HTML attachment was scrubbed... URL: From csnataraj at gmail.com Wed Mar 16 21:24:19 2011 From: csnataraj at gmail.com (Natarajan CS) Date: Wed, 16 Mar 2011 21:24:19 -0500 Subject: [petsc-users] Building with MKL 10.3 In-Reply-To: <18205E5ECD2A1A4584F2BFC0BCBDE95526D5B17C@exchange.geosoft.com> References: <18205E5ECD2A1A4584F2BFC0BCBDE95526D59A46@exchange.geosoft.com> <18205E5ECD2A1A4584F2BFC0BCBDE95526D5B17C@exchange.geosoft.com> Message-ID: Hello Rob, Thanks for the update, this might be very valuable for other developers out there! I am still a little surprised by the performance of MKL, I use it for a variety of problems and haven't come across a situation where performance has been penalized! Maybe there are more experienced developers out here who have seen otherwise. Have you tried ATLAS by any chance? When the MPI performance is concerned, I wouldn't be very surprised if intelMPI turns out to be faster than MPICH2. All this is of course fabric dependent but has been as I said in my experience for a variety of problems. Also, I think intelMPI is a customized version of MPICH2, so may not be a big surprise there There are very experienced people in this forum who might be able to say otherwise and give more accurate answers Cheers, C.S.N On Wed, Mar 16, 2011 at 4:22 PM, Robert Ellis wrote: > Hi All, > > > > For those still interested in this thread, timing tests with MKL indicate > that sequential MKL performs approximately the same as parallel MKL with > NUM_THREADS=1, which isn't too surprising. What is a bit surprising is that > MKL always, at least for this application, gives significantly slower > performance than direct compilation of the code from > --download-f-blas-lapack=1. My conclusion is that if your code is written > with explicit parallelization, in this case using PETSc, and fully utilizes > your hardware, using sophisticated libraries may actually harm performance. > Keep it simple! > > > > Now a question: all my tests used MPICH2. Does anyone think using Intel MPI > would significantly improve the performance of MKL with PETSc? > > > > Cheers, > > Rob > > > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Rob Ellis > *Sent:* Tuesday, March 15, 2011 3:33 PM > *To:* 'PETSc users list' > > *Subject:* Re: [petsc-users] Building with MKL 10.3 > > > > Yes, MKL_DYNAMIC was set to true. No, I haven't tested on Nehalem. I'm > currently comparing sequential MKL with --download-f-blas-lapack=1. > > Rob > > > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Natarajan CS > > *Sent:* Tuesday, March 15, 2011 3:20 PM > *To:* PETSc users list > *Cc:* Robert Ellis > *Subject:* Re: [petsc-users] Building with MKL 10.3 > > > > Thanks Eric and Rob. > > > Indeed! Was MKL_DYNAMIC set to default (true)? It looks like using 1 thread > per core (sequential MKL) is the right thing to do as baseline. > I would think that the performance of #cores = num_mpi_processes * > num_mkl_threads might be <= #cores = num_mpi_processes case (# cores const) > unless some cache effects come into play (Not sure what, I would think the > mkl installation should weed these issues out). > > > > P.S : > Out of curiosity have you also tested your app on Nehalem? Any difference > between Nehalem vs Westmere for similar bandwidth? > > On Tue, Mar 15, 2011 at 4:35 PM, Jed Brown wrote: > > On Tue, Mar 15, 2011 at 22:30, Robert Ellis > wrote: > > Regardless of setting the number of threads for MKL or OMP, the MKL > performance was worse than simply using --download-f-blas-lapack=1. > > > > Interesting. Does this statement include using just one thread, perhaps > with a non-threaded MKL? Also, when you used threading, were you putting an > MPI process on every core or were you making sure that you had enough cores > for num_mpi_processes * num_mkl_threads? > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gshy2014 at gmail.com Wed Mar 16 23:16:56 2011 From: gshy2014 at gmail.com (Gu Shiyuan) Date: Wed, 16 Mar 2011 23:16:56 -0500 Subject: [petsc-users] how to update the matrix in KSP Message-ID: Hi, I want to solve a linear eq. Kx=b where K is changed at each time step and the preconditioner is a shell function and doesn't change. After I change K, do I need to explicitly call KSP/PCSetOperator again to reset the matrix inside the KSP solver? Does KSP hold copies of the matrices or just pointers to the matrices? i.e., PCSetOperator(pc,K,K,SAME_PRECONDITIONER); ..............///other setup KSPSolve(..); ...........////change K; /////////Is PCSetOperator(......) needed here? And what about if I want to change the preconditioner? do I need to destroy the PC/KSP object and re-create and setup everything? or KSP/PCSetOperator will release the memory space associated with the old matrices(e.g., ILU factors)? Thanks. On Wed, Mar 16, 2011 at 12:00 PM, wrote: > Send petsc-users mailing list submissions to > petsc-users at mcs.anl.gov > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.mcs.anl.gov/mailman/listinfo/petsc-users > or, via email, send a message with subject or body 'help' to > petsc-users-request at mcs.anl.gov > > You can reach the person managing the list at > petsc-users-owner at mcs.anl.gov > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of petsc-users digest..." > > > Today's Topics: > > 1. Re: Building with MKL 10.3 (Preston, Eric - IS) > 2. Re: Building with MKL 10.3 (Robert Ellis) > 3. Re: Building with MKL 10.3 (Jed Brown) > 4. Re: Building with MKL 10.3 (Natarajan CS) > 5. Re: Building with MKL 10.3 (Rob Ellis) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 15 Mar 2011 16:55:25 -0400 > From: "Preston, Eric - IS" > Subject: Re: [petsc-users] Building with MKL 10.3 > To: "petsc-users at mcs.anl.gov" > Message-ID: > > Content-Type: text/plain; charset="us-ascii" > > > As the original poster, I can say I didn't give much thought to using the > threaded vs. non-threaded MKL libraries, I was just sharing a solution to > building with a different MKL variant (which should probably be incorporated > into the config scripts.) However, the discussion it spawned was > informative. The main consideration there was not performance but > compatibility with your other libraries. I know on windows, for instance, > different run-time libraries are used in each case and all your components > must be compiled with the same option. Thankfully, I'm not using windows for > this project, so it might not make any difference. > > Eric Preston > > > > Hello, > > I am neither a regular Petsc user nor contributor so preemptive > >apologies if I am completely off the line here. > > > >I am not sure if the original poster had hyper-threading in mind when he > >asked about multi-threading, in case that was the idea, I don't think > using > >petsc with MKL (HT) is going to give any benefit, I don't think MKL is > >really resource insensitive. > > > >Also I wonder what percentage of the code is actually blas/lapack > intensive > >to make any significant dent in wall cock? > > > >of course +1 to everything else posed above. > > > >Cheers, > > > >C.S.N > > > This e-mail and any files transmitted with it may be proprietary and are > intended solely for the use of the individual or entity to whom they are > addressed. If you have received this e-mail in error please notify the > sender. > Please note that any views or opinions presented in this e-mail are solely > those of the author and do not necessarily represent those of ITT > Corporation. The recipient should check this e-mail and any attachments for > the presence of viruses. ITT accepts no liability for any damage caused by > any virus transmitted by this e-mail. > > > ------------------------------ > > Message: 2 > Date: Tue, 15 Mar 2011 21:30:41 +0000 > From: Robert Ellis > Subject: Re: [petsc-users] Building with MKL 10.3 > To: PETSc users list > Message-ID: > <18205E5ECD2A1A4584F2BFC0BCBDE95526D59A46 at exchange.geosoft.com> > Content-Type: text/plain; charset="us-ascii" > > All, > > Coincidentally, I have just spent much of the last two weeks testing the > effect of the latest MKL blas-lapack release on a large domain decomposition > style PETSc application, MPICH2, Windows, Intel Fortran, VS 2010 C++, all > static linked, no hyperthreading, multi-blade, latest dual hex core Xeons, > 25GB ram. Each core contains a domain and a shell is used for the matrix > operations. Regardless of setting the number of threads for MKL or OMP, the > MKL performance was worse than simply using --download-f-blas-lapack=1. My > interpretation is that the decomposition of one domain per core effectively > saturates the hardware and performance is actually degraded using the more > sophisticated MKL. > > And, yes, I know that Windows is less than ideal for this type of work but > there are other constraints... > > Cheers, > Rob > > > -----Original Message----- > From: petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] On Behalf Of Preston, Eric - IS > Sent: Tuesday, March 15, 2011 1:55 PM > To: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Building with MKL 10.3 > > > As the original poster, I can say I didn't give much thought to using the > threaded vs. non-threaded MKL libraries, I was just sharing a solution to > building with a different MKL variant (which should probably be incorporated > into the config scripts.) However, the discussion it spawned was > informative. The main consideration there was not performance but > compatibility with your other libraries. I know on windows, for instance, > different run-time libraries are used in each case and all your components > must be compiled with the same option. Thankfully, I'm not using windows for > this project, so it might not make any difference. > > Eric Preston > > > > Hello, > > I am neither a regular Petsc user nor contributor so preemptive > >apologies if I am completely off the line here. > > > >I am not sure if the original poster had hyper-threading in mind when he > >asked about multi-threading, in case that was the idea, I don't think > using > >petsc with MKL (HT) is going to give any benefit, I don't think MKL is > >really resource insensitive. > > > >Also I wonder what percentage of the code is actually blas/lapack > intensive > >to make any significant dent in wall cock? > > > >of course +1 to everything else posed above. > > > >Cheers, > > > >C.S.N > > > This e-mail and any files transmitted with it may be proprietary and are > intended solely for the use of the individual or entity to whom they are > addressed. If you have received this e-mail in error please notify the > sender. > Please note that any views or opinions presented in this e-mail are solely > those of the author and do not necessarily represent those of ITT > Corporation. The recipient should check this e-mail and any attachments for > the presence of viruses. ITT accepts no liability for any damage caused by > any virus transmitted by this e-mail. > > > ------------------------------ > > Message: 3 > Date: Tue, 15 Mar 2011 22:35:31 +0100 > From: Jed Brown > Subject: Re: [petsc-users] Building with MKL 10.3 > To: PETSc users list > Cc: Robert Ellis > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > On Tue, Mar 15, 2011 at 22:30, Robert Ellis >wrote: > > > Regardless of setting the number of threads for MKL or OMP, the MKL > > performance was worse than simply using --download-f-blas-lapack=1. > > > Interesting. Does this statement include using just one thread, perhaps > with > a non-threaded MKL? Also, when you used threading, were you putting an MPI > process on every core or were you making sure that you had enough cores for > num_mpi_processes * num_mkl_threads? > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20110315/693c6d6c/attachment-0001.htm > > > > ------------------------------ > > Message: 4 > Date: Tue, 15 Mar 2011 17:20:25 -0500 > From: Natarajan CS > Subject: Re: [petsc-users] Building with MKL 10.3 > To: PETSc users list > Cc: Robert Ellis > Message-ID: > > Content-Type: text/plain; charset="iso-8859-1" > > Thanks Eric and Rob. > > Indeed! Was MKL_DYNAMIC set to default (true)? It looks like using 1 thread > per core (sequential MKL) is the right thing to do as baseline. > I would think that the performance of #cores = num_mpi_processes * > num_mkl_threads might be <= #cores = num_mpi_processes case (# cores const) > unless some cache effects come into play (Not sure what, I would think the > mkl installation should weed these issues out). > > > P.S : > Out of curiosity have you also tested your app on Nehalem? Any difference > between Nehalem vs Westmere for similar bandwidth? > > On Tue, Mar 15, 2011 at 4:35 PM, Jed Brown wrote: > > > On Tue, Mar 15, 2011 at 22:30, Robert Ellis >wrote: > > > >> Regardless of setting the number of threads for MKL or OMP, the MKL > >> performance was worse than simply using --download-f-blas-lapack=1. > > > > > > Interesting. Does this statement include using just one thread, perhaps > > with a non-threaded MKL? Also, when you used threading, were you putting > an > > MPI process on every core or were you making sure that you had enough > cores > > for num_mpi_processes * num_mkl_threads? > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20110315/96b06f5f/attachment-0001.htm > > > > ------------------------------ > > Message: 5 > Date: Tue, 15 Mar 2011 15:32:44 -0700 > From: "Rob Ellis" > Subject: Re: [petsc-users] Building with MKL 10.3 > To: "'PETSc users list'" > Message-ID: <006301cbe360$e77b63a0$b6722ae0$@G.Ellis at Shaw.ca> > Content-Type: text/plain; charset="us-ascii" > > Yes, MKL_DYNAMIC was set to true. No, I haven't tested on Nehalem. I'm > currently comparing sequential MKL with --download-f-blas-lapack=1. > > Rob > > > > From: petsc-users-bounces at mcs.anl.gov > [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Natarajan CS > Sent: Tuesday, March 15, 2011 3:20 PM > To: PETSc users list > Cc: Robert Ellis > Subject: Re: [petsc-users] Building with MKL 10.3 > > > > Thanks Eric and Rob. > > Indeed! Was MKL_DYNAMIC set to default (true)? It looks like using 1 thread > per core (sequential MKL) is the right thing to do as baseline. > I would think that the performance of #cores = num_mpi_processes * > num_mkl_threads might be <= #cores = num_mpi_processes case (# cores const) > unless some cache effects come into play (Not sure what, I would think the > mkl installation should weed these issues out). > > > > P.S : > Out of curiosity have you also tested your app on Nehalem? Any difference > between Nehalem vs Westmere for similar bandwidth? > > On Tue, Mar 15, 2011 at 4:35 PM, Jed Brown wrote: > > On Tue, Mar 15, 2011 at 22:30, Robert Ellis > wrote: > > Regardless of setting the number of threads for MKL or OMP, the MKL > performance was worse than simply using --download-f-blas-lapack=1. > > > > Interesting. Does this statement include using just one thread, perhaps > with > a non-threaded MKL? Also, when you used threading, were you putting an MPI > process on every core or were you making sure that you had enough cores for > num_mpi_processes * num_mkl_threads? > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20110315/d848575a/attachment-0001.htm > > > > ------------------------------ > > _______________________________________________ > petsc-users mailing list > petsc-users at mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/petsc-users > > > End of petsc-users Digest, Vol 27, Issue 33 > ******************************************* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aron.ahmadia at kaust.edu.sa Thu Mar 17 03:24:49 2011 From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia) Date: Thu, 17 Mar 2011 09:24:49 +0100 Subject: [petsc-users] Building with MKL 10.3 In-Reply-To: References: <18205E5ECD2A1A4584F2BFC0BCBDE95526D59A46@exchange.geosoft.com> <18205E5ECD2A1A4584F2BFC0BCBDE95526D5B17C@exchange.geosoft.com> Message-ID: I am very suspicious of any results where the fortran blas is out-performing the MKL. A On Thu, Mar 17, 2011 at 3:24 AM, Natarajan CS wrote: > Hello Rob, > ???? Thanks for the update, this might be very valuable for other developers > out there! > I am still a little surprised by the performance of MKL, I use it for a > variety of problems and haven't come across a situation where performance > has been penalized! Maybe there are more experienced developers out here who > have seen otherwise. Have you tried ATLAS by any chance? > > When the MPI performance is concerned, I wouldn't be very surprised if > intelMPI turns out to be faster than MPICH2. All this is of course fabric > dependent but has been as I said in my experience for a variety of problems. > Also, I think intelMPI is a customized version of MPICH2, so may not be a > big surprise there > > ?There are very experienced people in this forum who might be able to say > otherwise and give more accurate answers > > Cheers, > > C.S.N > > > On Wed, Mar 16, 2011 at 4:22 PM, Robert Ellis > wrote: >> >> Hi All, >> >> >> >> For those still interested in this thread, timing tests with MKL indicate >> that sequential MKL performs approximately the same as parallel MKL with >> NUM_THREADS=1, which isn't too surprising. What is a bit surprising is that >> MKL always, at least for this application, gives significantly slower >> performance than direct compilation of the code from >> ?--download-f-blas-lapack=1. My conclusion is that if your code is written >> with explicit parallelization, in this case using PETSc, and fully utilizes >> your hardware, ?using sophisticated libraries may actually harm performance. >> Keep it simple! >> >> >> >> Now a question: all my tests used MPICH2. Does anyone think using Intel >> MPI would significantly improve the performance of MKL with PETSc? >> >> >> >> Cheers, >> >> Rob >> >> >> >> From: petsc-users-bounces at mcs.anl.gov >> [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Rob Ellis >> Sent: Tuesday, March 15, 2011 3:33 PM >> To: 'PETSc users list' >> >> Subject: Re: [petsc-users] Building with MKL 10.3 >> >> >> >> Yes, MKL_DYNAMIC was set to true. No, I haven't tested on Nehalem. I'm >> currently comparing sequential MKL with --download-f-blas-lapack=1. >> >> Rob >> >> >> >> From: petsc-users-bounces at mcs.anl.gov >> [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Natarajan CS >> >> Sent: Tuesday, March 15, 2011 3:20 PM >> To: PETSc users list >> Cc: Robert Ellis >> Subject: Re: [petsc-users] Building with MKL 10.3 >> >> >> >> Thanks Eric and Rob. >> >> Indeed! Was MKL_DYNAMIC set to default (true)? It looks like using 1 >> thread per core (sequential MKL) is the right thing to do as baseline. >> ?I would think that the performance of #cores =? num_mpi_processes * >> num_mkl_threads might be <= #cores = num_mpi_processes case (# cores const) >> unless some cache effects come into play (Not sure what, I would think the >> mkl installation should weed these issues out). >> >> P.S : >> Out of curiosity have you also tested your app on Nehalem? Any difference >> between Nehalem vs Westmere for similar bandwidth? >> >> On Tue, Mar 15, 2011 at 4:35 PM, Jed Brown wrote: >> >> On Tue, Mar 15, 2011 at 22:30, Robert Ellis >> wrote: >> >> Regardless of setting the number of threads for MKL or OMP, the MKL >> performance was worse than simply using --download-f-blas-lapack=1. >> >> >> >> Interesting. Does this statement include using just one thread, perhaps >> with a non-threaded MKL? Also, when you used threading, were you putting an >> MPI process on every core or were you making sure that you had enough cores >> for num_mpi_processes * num_mkl_threads? >> >> > From jed at 59A2.org Thu Mar 17 06:08:12 2011 From: jed at 59A2.org (Jed Brown) Date: Thu, 17 Mar 2011 12:08:12 +0100 Subject: [petsc-users] how to update the matrix in KSP In-Reply-To: References: Message-ID: On Thu, Mar 17, 2011 at 05:16, Gu Shiyuan wrote: > I want to solve a linear eq. Kx=b where K is changed at each time step and > the preconditioner is a shell function and doesn't change. After I change K, > do I need to explicitly call KSP/PCSetOperator again to reset the matrix > inside the KSP solver? > Does KSP hold copies of the matrices or just pointers to the matrices? > i.e., > > PCSetOperator(pc,K,K,SAME_PRECONDITIONER); > It is "PCSetOperators". SAME_PRECONDITIONER means that the PC will not be updated for the new matrix. > ..............///other setup > KSPSolve(..); > ...........////change K; > /////////Is PCSetOperator(......) needed here? > You should be calling KSPSetOperators() to inform the KSP that the operator has changed. Rebuilding the preconditioner is a separate matter which you control using the MatStructure flag. > And what about if I want to change the preconditioner? do I need to destroy > the PC/KSP object and re-create and setup everything? > No, when you pass in SAME_NONZERO_PATTERN, numeric factorization is performed again using the same data structures. With DIFFERENT_NONZERO_PATTERN, some internal structures are reallocated. For many preconditioners, there is very little performance difference between these. If you want to change the type of the preconditioner, just call PCSetType(). This will free things like ILU factors and then switch to the new PCType. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wlowrie at uw.edu Thu Mar 17 15:10:07 2011 From: wlowrie at uw.edu (Weston Lowrie) Date: Thu, 17 Mar 2011 13:10:07 -0700 Subject: [petsc-users] Breadth-First Search (BFS) Message-ID: Hi Petsc-Users, Does anyone know if there is a implementation of a Breadth-First Search in any of the PETSc libraries? Or any other related libraries? Preferably a parallel algorithm. Thanks, Wes -- Weston Lowrie Ph.D. Candidate / Research Assistant Aeronautics and Astronautics Dept. University of Washington, Box 352250 Seattle, WA 98195-2250 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Mar 17 16:00:14 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 17 Mar 2011 16:00:14 -0500 Subject: [petsc-users] Breadth-First Search (BFS) In-Reply-To: References: Message-ID: On Mar 17, 2011, at 3:10 PM, Weston Lowrie wrote: > Hi Petsc-Users, > Does anyone know if there is a implementation of a Breadth-First Search in any of the PETSc libraries? Not the kind of stuff we do. > Or any other related libraries? Preferably a parallel algorithm. > > Thanks, > Wes > > > -- > Weston Lowrie > Ph.D. Candidate / Research Assistant > Aeronautics and Astronautics Dept. > University of Washington, Box 352250 > Seattle, WA 98195-2250 > From balay at mcs.anl.gov Thu Mar 17 21:06:37 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 17 Mar 2011 21:06:37 -0500 (CDT) Subject: [petsc-users] petsc-3.1-p8.tar.gz now available Message-ID: Dear PETSc users, Patch-8 update to petsc-3.1 is now available for download. http://www.mcs.anl.gov/petsc/petsc-as/download/index.html PS: I'm attempting to announce the availability of patch updates as well - as some folks expreseed interest in getting them. Satish From zhaonanavril at gmail.com Fri Mar 18 00:09:33 2011 From: zhaonanavril at gmail.com (NAN ZHAO) Date: Thu, 17 Mar 2011 23:09:33 -0600 Subject: [petsc-users] example code error! Message-ID: Hi everyone, I tried to learn using MATCreateMPIBAIJ and find a example in /src/mat/tests/ex92. I complied it and run it, but I got the following error, Can anyone give some insight on this kind of error? [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: ! [0]PETSC ERROR: Lower triangular value cannot be set for sbaij format. Ignoring these values, run with -mat_ignore_lower_triangular or call MatSetOption(mat,MAT_IGNORE_LOWER_TRIANGULAR)! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 2.3.3, Patch 15, Tue Sep 23 10:02:49 CDT 2008 HG revision: 31306062cd1a6f6a2496fccb4878f485c9b91760 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./ex92 on a linux-gnu named localhost:localdomain by nan Thu Mar 17 23:05:57 2011 [0]PETSC ERROR: Libraries linked from /home/nan/ufesworkspace/petsc-2.3.3-p15/LINUX_MPI/lib/linux-gnu-cxx-debug [0]PETSC ERROR: Configure run at Mon Jun 14 09:57:43 2010 [0]PETSC ERROR: Configure options --prefix=/home/nan/ufesworkspace/petsc-2.3.3-p15/LINUX_MPI --with-fc=0 --with-clanguage=cxx --with-x11=0 --with-shared=0 --with-debugging=1 COPTFLAGS=-O3 --with-cc=mpicc --with-cxx=mpicxx --with-mpi-dir=/usr/lib/mpich2 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatSetValues_MPISBAIJ_MatScalar() line 265 in src/mat/impls/sbaij/mpi/mpisbaij.c [0]PETSC ERROR: MatSetValues() line 828 in src/mat/interface/matrix.c [0]PETSC ERROR: MatConvert_Basic() line 34 in src/mat/utils/convert.c [0]PETSC ERROR: MatConvert() line 3179 in src/mat/interface/matrix.c [0]PETSC ERROR: main() line 89 in src/mat/examples/tests/ex92.c Thanks, Nan -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Mar 18 04:38:23 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 18 Mar 2011 04:38:23 -0500 Subject: [petsc-users] example code error! In-Reply-To: References: Message-ID: <237B9176-5C85-452E-9365-DBE8AE307B54@mcs.anl.gov> You really should NOT be learning anything from Version 2.3.3, Patch 15, Tue Sep 23 10:02:49 CDT 2008. That was released 2 years 4 months ago. You will be learning the wrong things, you need to install PETSc 3.1 or petsc-dev Barry On Mar 18, 2011, at 12:09 AM, NAN ZHAO wrote: > Hi everyone, > > I tried to learn using MATCreateMPIBAIJ and find a example in /src/mat/tests/ex92. I complied it and run it, but I got the following error, Can anyone give some insight on this kind of error? > > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: ! > [0]PETSC ERROR: Lower triangular value cannot be set for sbaij format. Ignoring these values, run with -mat_ignore_lower_triangular or call MatSetOption(mat,MAT_IGNORE_LOWER_TRIANGULAR)! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 2.3.3, Patch 15, Tue Sep 23 10:02:49 CDT 2008 HG revision: 31306062cd1a6f6a2496fccb4878f485c9b91760 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ./ex92 on a linux-gnu named localhost:localdomain by nan Thu Mar 17 23:05:57 2011 > [0]PETSC ERROR: Libraries linked from /home/nan/ufesworkspace/petsc-2.3.3-p15/LINUX_MPI/lib/linux-gnu-cxx-debug > [0]PETSC ERROR: Configure run at Mon Jun 14 09:57:43 2010 > [0]PETSC ERROR: Configure options --prefix=/home/nan/ufesworkspace/petsc-2.3.3-p15/LINUX_MPI --with-fc=0 --with-clanguage=cxx --with-x11=0 --with-shared=0 --with-debugging=1 COPTFLAGS=-O3 --with-cc=mpicc --with-cxx=mpicxx --with-mpi-dir=/usr/lib/mpich2 > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatSetValues_MPISBAIJ_MatScalar() line 265 in src/mat/impls/sbaij/mpi/mpisbaij.c > [0]PETSC ERROR: MatSetValues() line 828 in src/mat/interface/matrix.c > [0]PETSC ERROR: MatConvert_Basic() line 34 in src/mat/utils/convert.c > [0]PETSC ERROR: MatConvert() line 3179 in src/mat/interface/matrix.c > [0]PETSC ERROR: main() line 89 in src/mat/examples/tests/ex92.c > > Thanks, > Nan From jed at 59A2.org Fri Mar 18 05:18:05 2011 From: jed at 59A2.org (Jed Brown) Date: Fri, 18 Mar 2011 11:18:05 +0100 Subject: [petsc-users] example code error! In-Reply-To: References: Message-ID: On Fri, Mar 18, 2011 at 06:09, NAN ZHAO wrote: > [0]PETSC ERROR: Lower triangular value cannot be set for sbaij format. > Ignoring these values, run with -mat_ignore_lower_triangular or call > MatSetOption(mat,MAT_IGNORE_LOWER_TRIANGULAR)! > The problem is that MatConvert_Basic was not setting MAT_IGNORE_LOWER_TRIANGULAR when converting to SBAIJ, I fixed this in petsc-dev. Since you said you want BAIJ (no "S"), this won't be a problem in your own code. And you should definitely upgrade from 2.3.3. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gianmail at gmail.com Fri Mar 18 06:07:29 2011 From: gianmail at gmail.com (Gianluca Meneghello) Date: Fri, 18 Mar 2011 12:07:29 +0100 Subject: [petsc-users] -snes_type test Message-ID: Hi, trying to check the jacobian of my nonlinear solver with -snes_type test, I came up with an error in SNES solve. The same error does not appear with -snes_type ls. Using the latest dev version (downloaded this morning), I tried a tutorial in order to check whether it was a general problem or a problem related to my own code. The code in src/snes/examples/tutorials/ex1.c reports the same error. In this last case, the output of ./ex1 -snes_type test is Testing hand-coded Jacobian, if the ratio is O(1.e-8), the hand-coded Jacobian is probably correct. Run with -snes_test_display to show difference of hand-coded and finite difference Jacobian. Norm of matrix ratio 1.08967e-07 difference 2.43658e-07 Norm of matrix ratio 7.6194e-08 difference 3.4075e-07 Norm of matrix ratio 7.6211e-08 difference 3.40826e-07 [0]PETSC ERROR: SNESSolve() line 2360 in src/snes/interface/snes.c [0]PETSC ERROR: main() line 144 in src/snes/examples/tutorials/ex1.c application called MPI_Abort(MPI_COMM_WORLD, 73) - process 0 [unset]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 73) - process 0 the same error was given using a much older (last december I think) dev version. Any suggestion? Thanks Gianluca -- "[Je pense que] l'homme est un monde qui vaut des fois les mondes et que les plus ardentes ambitions sont celles qui ont eu l'orgueil de l'Anonymat" -- Non omnibus, sed mihi et tibi Amedeo Modigliani From jed at 59A2.org Fri Mar 18 06:12:01 2011 From: jed at 59A2.org (Jed Brown) Date: Fri, 18 Mar 2011 12:12:01 +0100 Subject: [petsc-users] -snes_type test In-Reply-To: References: Message-ID: On Fri, Mar 18, 2011 at 12:07, Gianluca Meneghello wrote: > esting hand-coded Jacobian, if the ratio is > O(1.e-8), the hand-coded Jacobian is probably correct. > Run with -snes_test_display to show difference > of hand-coded and finite difference Jacobian. > Norm of matrix ratio 1.08967e-07 difference 2.43658e-07 > Norm of matrix ratio 7.6194e-08 difference 3.4075e-07 > Norm of matrix ratio 7.6211e-08 difference 3.40826e-07 > Your Jacobian looks good so you don't need to use TEST. > [0]PETSC ERROR: SNESSolve() line 2360 in src/snes/interface/snes.c > [0]PETSC ERROR: main() line 144 in src/snes/examples/tutorials/ex1.c > application called MPI_Abort(MPI_COMM_WORLD, 73) - process 0 > [unset]: aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 73) - process 0 > This is intentional, the end of SNESSolve_Test() has /* Return error code cause Jacobian not good */ PetscFunctionReturn(PETSC_ERR_ARG_WRONGSTATE); Perhaps it should be done differently, maybe even solve the Newton step and continue, but that's not the current behavior. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gianmail at gmail.com Fri Mar 18 06:41:09 2011 From: gianmail at gmail.com (Gianluca Meneghello) Date: Fri, 18 Mar 2011 12:41:09 +0100 Subject: [petsc-users] -snes_type test In-Reply-To: References: Message-ID: Thanks Jed. One more thing: what does the three lines stand for then? I was thinking it was three Newton steps, but I understand it's not... The output I sent was actually the output of one of your examples. The one from my own code is Testing hand-coded Jacobian, if the ratio is O(1.e-8), the hand-coded Jacobian is probably correct. Run with -snes_test_display to show difference of hand-coded and finite difference Jacobian. Norm of matrix ratio 9.17257e-08 difference 2.58876e-05 Norm of matrix ratio 1.45865e-06 difference 0.000430034 Norm of matrix ratio 1.47148e-06 difference 0.000429436 [0]PETSC ERROR: SNESSolve() line 2361 in src/snes/interface/snes.c [0]PETSC ERROR: main() line 71 in jactest.cpp application called MPI_Abort(MPI_COMM_WORLD, 73) - process 0 [unset]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 73) - process 0 so I guess I have to look for an error somewhere! Thanks again Gianluca On 18 March 2011 12:12, Jed Brown wrote: > On Fri, Mar 18, 2011 at 12:07, Gianluca Meneghello > wrote: >> >> esting hand-coded Jacobian, if the ratio is >> O(1.e-8), the hand-coded Jacobian is probably correct. >> Run with -snes_test_display to show difference >> of hand-coded and finite difference Jacobian. >> Norm of matrix ratio 1.08967e-07 difference 2.43658e-07 >> Norm of matrix ratio 7.6194e-08 difference 3.4075e-07 >> Norm of matrix ratio 7.6211e-08 difference 3.40826e-07 > > Your Jacobian looks good so you don't need to use TEST. > >> >> [0]PETSC ERROR: SNESSolve() line 2360 in src/snes/interface/snes.c >> [0]PETSC ERROR: main() line 144 in src/snes/examples/tutorials/ex1.c >> application called MPI_Abort(MPI_COMM_WORLD, 73) - process 0 >> [unset]: aborting job: >> application called MPI_Abort(MPI_COMM_WORLD, 73) - process 0 > > This is intentional, the end of SNESSolve_Test() has > ? /* > ? ? ? ? ?Return error code cause Jacobian not good > ? */ > ? PetscFunctionReturn(PETSC_ERR_ARG_WRONGSTATE); > Perhaps it should be done differently, maybe even solve the Newton step and > continue, but that's not the current behavior. -- "[Je pense que] l'homme est un monde qui vaut des fois les mondes et que les plus ardentes ambitions sont celles qui ont eu l'orgueil de l'Anonymat" -- Non omnibus, sed mihi et tibi Amedeo Modigliani From jed at 59A2.org Fri Mar 18 06:54:33 2011 From: jed at 59A2.org (Jed Brown) Date: Fri, 18 Mar 2011 12:54:33 +0100 Subject: [petsc-users] -snes_type test In-Reply-To: References: Message-ID: On Fri, Mar 18, 2011 at 12:41, Gianluca Meneghello wrote: > One more thing: what does the three lines stand for then? I was > thinking it was three Newton steps, but I understand it's not... > It is your Jacobian evaluated at user-defined state, constant -1.0, and constant 1.0. I added this to the output for petsc-dev: $ ./ex5 -snes_type test Testing hand-coded Jacobian, if the ratio is O(1.e-8), the hand-coded Jacobian is probably correct. Run with -snes_test_display to show difference of hand-coded and finite difference Jacobian. Norm of matrix ratio 2.77166e-08 difference 2.17871e-07 (user-defined state) Norm of matrix ratio 1.34123e-08 difference 1.23213e-07 (constant state -1.0) Norm of matrix ratio 1.31875e-07 difference 9.05494e-07 (constant state 1.0) [0]PETSC ERROR: SNESSolve() line 2360 in /home/jed/petsc/src/snes/interface/snes.c [0]PETSC ERROR: main() line 220 in src/snes/examples/tutorials/ex5.c > > The output I sent was actually the output of one of your examples. The > one from my own code is > > Testing hand-coded Jacobian, if the ratio is > O(1.e-8), the hand-coded Jacobian is probably correct. > Run with -snes_test_display to show difference > of hand-coded and finite difference Jacobian. > Norm of matrix ratio 9.17257e-08 difference 2.58876e-05 > Is your starting state zero? > Norm of matrix ratio 1.45865e-06 difference 0.000430034 > Norm of matrix ratio 1.47148e-06 difference 0.000429436 > If your problem is ill-conditioned, this might come from differencing error. Try with -mat_fd_type ds to use a slightly more robust scheme. You can also run with -ksp_rtol 1e-12 -pc_type lu -snes_monitor and check if you see quadratic convergence. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gianmail at gmail.com Fri Mar 18 08:03:02 2011 From: gianmail at gmail.com (Gianluca Meneghello) Date: Fri, 18 Mar 2011 14:03:02 +0100 Subject: [petsc-users] -snes_type test In-Reply-To: References: Message-ID: Yes, my starting state is 0, not such a good guess... what do you mean by constant +1 and constant -1? Is it the starting/user-defined state vector +1 and -1? My problem should not be ill posed --- it's the Jacobian of the Navier Stokes operator, with good boundary conditions, at least theoretically --- but still I tried what you suggested. Here is the output using -mat_fd_type_ds -snes_type test: Testing hand-coded Jacobian, if the ratio is O(1.e-8), the hand-coded Jacobian is probably correct. Run with -snes_test_display to show difference of hand-coded and finite difference Jacobian. Norm of matrix ratio 9.17279e-09 difference 2.58883e-06 Norm of matrix ratio 8.79304e-08 difference 2.59233e-05 Norm of matrix ratio 8.87045e-08 difference 2.58875e-05 while using -ksp_rtol 1e-12 -pc_type lu -snes_monitor 0 SNES Function norm 2.981436587230e+00 1 SNES Function norm 1.487397936082e+00 2 SNES Function norm 3.490136703474e-01 3 SNES Function norm 1.303397000344e-01 4 SNES Function norm 3.181190030949e-02 5 SNES Function norm 5.280858553562e-03 6 SNES Function norm 3.168168082141e-04 7 SNES Function norm 1.217775965490e-06 8 SNES Function norm 1.559322624068e-11 do I have to deduce that it's ill posed? thanks gianluca On 18 March 2011 12:54, Jed Brown wrote: > On Fri, Mar 18, 2011 at 12:41, Gianluca Meneghello > wrote: >> >> One more thing: what does the three lines stand for then? I was >> thinking it was three Newton steps, but I understand it's not... > > It is your Jacobian evaluated at user-defined state, constant -1.0, and > constant 1.0. I added this to the output for petsc-dev: > $ ./ex5 -snes_type test > Testing hand-coded Jacobian, if the ratio is > O(1.e-8), the hand-coded Jacobian is probably correct. > Run with -snes_test_display to show difference > of hand-coded and finite difference Jacobian. > Norm of matrix ratio 2.77166e-08 difference 2.17871e-07 (user-defined state) > Norm of matrix ratio 1.34123e-08 difference 1.23213e-07 (constant state > -1.0) > Norm of matrix ratio 1.31875e-07 difference 9.05494e-07 (constant state 1.0) > [0]PETSC ERROR: SNESSolve() line 2360 in > /home/jed/petsc/src/snes/interface/snes.c > [0]PETSC ERROR: main() line 220 in src/snes/examples/tutorials/ex5.c > >> >> The output I sent was actually the output of one of your examples. The >> one from my own code is >> >> Testing hand-coded Jacobian, if the ratio is >> O(1.e-8), the hand-coded Jacobian is probably correct. >> Run with -snes_test_display to show difference >> of hand-coded and finite difference Jacobian. >> Norm of matrix ratio 9.17257e-08 difference 2.58876e-05 > > Is your starting state zero? > >> >> Norm of matrix ratio 1.45865e-06 difference 0.000430034 >> Norm of matrix ratio 1.47148e-06 difference 0.000429436 > > If your problem is ill-conditioned, this might come from differencing error. > Try with -mat_fd_type ds to use a slightly more robust scheme. You can also > run with -ksp_rtol 1e-12 -pc_type lu -snes_monitor and check if you see > quadratic convergence. -- "[Je pense que] l'homme est un monde qui vaut des fois les mondes et que les plus ardentes ambitions sont celles qui ont eu l'orgueil de l'Anonymat" -- Non omnibus, sed mihi et tibi Amedeo Modigliani From gianmail at gmail.com Fri Mar 18 08:16:31 2011 From: gianmail at gmail.com (Gianluca Meneghello) Date: Fri, 18 Mar 2011 14:16:31 +0100 Subject: [petsc-users] -snes_type test In-Reply-To: References: Message-ID: Sorry, one more thing... is there a way to see/save/load into matlab the difference between the computed jacobian and the fd one, apart from output to screen? thanks gianlu On 18 March 2011 14:03, Gianluca Meneghello wrote: > Yes, my starting state is 0, not such a good guess... > > what do you mean by constant +1 and constant -1? Is it the > starting/user-defined state vector +1 and -1? > > My problem should not be ill posed --- it's the Jacobian of the Navier > Stokes operator, with good boundary conditions, at least theoretically > --- but still I tried what you suggested. Here is the output > > using -mat_fd_type_ds -snes_type test: > > ? Testing hand-coded Jacobian, if the ratio is > ? O(1.e-8), the hand-coded Jacobian is probably correct. > ? Run with -snes_test_display to show difference > ? of hand-coded and finite difference Jacobian. > ? Norm of matrix ratio 9.17279e-09 difference 2.58883e-06 > ? Norm of matrix ratio 8.79304e-08 difference 2.59233e-05 > ? Norm of matrix ratio 8.87045e-08 difference 2.58875e-05 > > while using -ksp_rtol 1e-12 -pc_type lu -snes_monitor > > ?0 SNES Function norm 2.981436587230e+00 > ?1 SNES Function norm 1.487397936082e+00 > ?2 SNES Function norm 3.490136703474e-01 > ?3 SNES Function norm 1.303397000344e-01 > ?4 SNES Function norm 3.181190030949e-02 > ?5 SNES Function norm 5.280858553562e-03 > ?6 SNES Function norm 3.168168082141e-04 > ?7 SNES Function norm 1.217775965490e-06 > ?8 SNES Function norm 1.559322624068e-11 > > > do I have to deduce that it's ill posed? > thanks > > gianluca > > On 18 March 2011 12:54, Jed Brown wrote: >> On Fri, Mar 18, 2011 at 12:41, Gianluca Meneghello >> wrote: >>> >>> One more thing: what does the three lines stand for then? I was >>> thinking it was three Newton steps, but I understand it's not... >> >> It is your Jacobian evaluated at user-defined state, constant -1.0, and >> constant 1.0. I added this to the output for petsc-dev: >> $ ./ex5 -snes_type test >> Testing hand-coded Jacobian, if the ratio is >> O(1.e-8), the hand-coded Jacobian is probably correct. >> Run with -snes_test_display to show difference >> of hand-coded and finite difference Jacobian. >> Norm of matrix ratio 2.77166e-08 difference 2.17871e-07 (user-defined state) >> Norm of matrix ratio 1.34123e-08 difference 1.23213e-07 (constant state >> -1.0) >> Norm of matrix ratio 1.31875e-07 difference 9.05494e-07 (constant state 1.0) >> [0]PETSC ERROR: SNESSolve() line 2360 in >> /home/jed/petsc/src/snes/interface/snes.c >> [0]PETSC ERROR: main() line 220 in src/snes/examples/tutorials/ex5.c >> >>> >>> The output I sent was actually the output of one of your examples. The >>> one from my own code is >>> >>> Testing hand-coded Jacobian, if the ratio is >>> O(1.e-8), the hand-coded Jacobian is probably correct. >>> Run with -snes_test_display to show difference >>> of hand-coded and finite difference Jacobian. >>> Norm of matrix ratio 9.17257e-08 difference 2.58876e-05 >> >> Is your starting state zero? >> >>> >>> Norm of matrix ratio 1.45865e-06 difference 0.000430034 >>> Norm of matrix ratio 1.47148e-06 difference 0.000429436 >> >> If your problem is ill-conditioned, this might come from differencing error. >> Try with -mat_fd_type ds to use a slightly more robust scheme. You can also >> run with -ksp_rtol 1e-12 -pc_type lu -snes_monitor and check if you see >> quadratic convergence. > > > > -- > "[Je pense que] l'homme est un monde qui vaut des fois les mondes et > que les plus ardentes ambitions sont celles qui ont eu l'orgueil de > l'Anonymat" -- Non omnibus, sed mihi et tibi > Amedeo Modigliani > -- "[Je pense que] l'homme est un monde qui vaut des fois les mondes et que les plus ardentes ambitions sont celles qui ont eu l'orgueil de l'Anonymat" -- Non omnibus, sed mihi et tibi Amedeo Modigliani From jed at 59A2.org Fri Mar 18 08:23:16 2011 From: jed at 59A2.org (Jed Brown) Date: Fri, 18 Mar 2011 14:23:16 +0100 Subject: [petsc-users] -snes_type test In-Reply-To: References: Message-ID: On Fri, Mar 18, 2011 at 14:03, Gianluca Meneghello wrote: > what do you mean by constant +1 and constant -1? Is it the > starting/user-defined state vector +1 and -1? > > My problem should not be ill posed > Not ill-posed, just ill-conditioned, which can be the result of poor scaling. For example, changing the units of velocity or pressure so that these parts of the Jacobian were "the same size" might improve the conditioning of the system. Depending on the solver, it may or may not make a convergence difference. > > using -mat_fd_type_ds -snes_type test: > > Testing hand-coded Jacobian, if the ratio is > O(1.e-8), the hand-coded Jacobian is probably correct. > Run with -snes_test_display to show difference > of hand-coded and finite difference Jacobian. > Norm of matrix ratio 9.17279e-09 difference 2.58883e-06 > Norm of matrix ratio 8.79304e-08 difference 2.59233e-05 > Norm of matrix ratio 8.87045e-08 difference 2.58875e-05 > This numbers are an order of magnitude smaller than before which indicates that the differencing is sensitive to choice of step. In the best case, cond=1, this differencing can give you sqrt(machine_epsilon) which for double precision means you could get 7 or 8 digits correct. You lose precision when the conditioning of the operator degrades (because of the mesh, physics, or just poor relative scaling of fields) and if the step size is not chosen well. The default step size algorithm "wp" is cheaper to compute in parallel, "ds" is more robust. > while using -ksp_rtol 1e-12 -pc_type lu -snes_monitor > > 0 SNES Function norm 2.981436587230e+00 > 1 SNES Function norm 1.487397936082e+00 > 2 SNES Function norm 3.490136703474e-01 > 3 SNES Function norm 1.303397000344e-01 > 4 SNES Function norm 3.181190030949e-02 > 5 SNES Function norm 5.280858553562e-03 > 6 SNES Function norm 3.168168082141e-04 > 7 SNES Function norm 1.217775965490e-06 > 8 SNES Function norm 1.559322624068e-11 > This looks like quadratic convergence so your Jacobian is probably correct. > is there a way to see/save/load into matlab the difference between the computed jacobian and the fd one, apart from output to screen? Not as currently written, but it would be simple to add. If you are interested, look at src/snes/impls/test/snestest.c. You could add a runtime option -snes_test_display_matlab and then view to a Matlab file inside SNESSolve_Test(). Let us know if you get lost (preferably on petsc-dev). -------------- next part -------------- An HTML attachment was scrubbed... URL: From u.tabak at tudelft.nl Sat Mar 19 11:36:20 2011 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sat, 19 Mar 2011 17:36:20 +0100 Subject: [petsc-users] petsc for MATLAB Message-ID: <4D84DB84.2050600@tudelft.nl> Dear all, There was a posting about PETSc for MATLAB some time ago? I tried to find that message among the user list messages but could not find, how can I use this interface and is all the functionality also present for this MATLAB interface? Greetings, Umut -- If I have a thousand ideas and only one turns out to be good, I am satisfied. Alfred Nobel From dave.mayhem23 at gmail.com Sat Mar 19 12:01:38 2011 From: dave.mayhem23 at gmail.com (Dave May) Date: Sat, 19 Mar 2011 18:01:38 +0100 Subject: [petsc-users] petsc for MATLAB In-Reply-To: <4D84DB84.2050600@tudelft.nl> References: <4D84DB84.2050600@tudelft.nl> Message-ID: Umut, There is quite a lot of functionality which is accessible via the matlab interface (see bin/matlab/classes). If you want to play, take a look though the examples here bin/matlab/classes/examples/tutorials These should give you a good starting point. On 19 March 2011 17:36, Umut Tabak wrote: > Dear all, > > There was a posting about PETSc for MATLAB some time ago? I tried to find > that message among the user list messages but could not find, how can I use > this interface and is all the functionality also present for this MATLAB > interface? > > Greetings, > Umut > > -- > If I have a thousand ideas and only one turns out to be good, > I am satisfied. > Alfred Nobel > > From u.tabak at tudelft.nl Sat Mar 19 12:09:50 2011 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sat, 19 Mar 2011 18:09:50 +0100 Subject: [petsc-users] petsc for MATLAB In-Reply-To: References: <4D84DB84.2050600@tudelft.nl> Message-ID: <4D84E35E.6060500@tudelft.nl> On 03/19/2011 06:01 PM, Dave May wrote: > Umut, > There is quite a lot of functionality which is accessible via the > matlab interface (see bin/matlab/classes). If you want to play, take a > look though the examples here > bin/matlab/classes/examples/tutorials > These should give you a good starting point. > Thanks for the info ;-) Dave. From sean at mcs.anl.gov Sat Mar 19 13:12:44 2011 From: sean at mcs.anl.gov (Sean Farley) Date: Sat, 19 Mar 2011 13:12:44 -0500 Subject: [petsc-users] petsc for MATLAB In-Reply-To: <4D84DB84.2050600@tudelft.nl> References: <4D84DB84.2050600@tudelft.nl> Message-ID: > > There was a posting about PETSc for MATLAB some time ago? I tried to find > that message among the user list messages but could not find, how can I use > this interface and is all the functionality also present for this MATLAB > interface? Barry gave a talk on this about 2 months ago. I made a video of the presentation (a very bad one) and posted it (along with the slides) here: http://vimeo.com/18768859 Hope that's a good enough starting point. Barry would have to chime in if there is any other tutorial he's written. Sean -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdbst21 at gmail.com Sun Mar 20 10:03:34 2011 From: jdbst21 at gmail.com (Joshua Booth) Date: Sun, 20 Mar 2011 11:03:34 -0400 Subject: [petsc-users] Question about MatGetSubMatrices Message-ID: Hi reading about MatGetsubMatrices, that it must be called on all processors the same number of times. Does this mean all processors in PETSC_COMM_WORLD or on all processors that contain the Parent Matrix. For example. If I am using four processors, but only rank=0 has a seq matrix A and it wants a seq subA on rank=0, do the other three processors need to make a call? Thanks Josh -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Mar 20 10:15:25 2011 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 20 Mar 2011 10:15:25 -0500 Subject: [petsc-users] Question about MatGetSubMatrices In-Reply-To: References: Message-ID: On Sun, Mar 20, 2011 at 10:03 AM, Joshua Booth wrote: > Hi reading about MatGetsubMatrices, that it must be called on all > processors the same number of times. > Does this mean all processors in PETSC_COMM_WORLD or on all processors that > contain the Parent Matrix. > All processes in the Mat communicator. > For example. If I am using four processors, but only rank=0 has a seq > matrix A and it wants a seq subA on rank=0, do the other three processors > need to make a call? > No. Matt > Thanks > > Josh > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From u.tabak at tudelft.nl Mon Mar 21 05:15:26 2011 From: u.tabak at tudelft.nl (Umut Tabak) Date: Mon, 21 Mar 2011 11:15:26 +0100 Subject: [petsc-users] -pc_factor_shift_type command line Message-ID: <4D87253E.2010805@tudelft.nl> Dear all, What are the options for -pc_factor_shift_type? From the manual page, I understand I should use the -help option with the binary, but I guess the source code should include this function to get the help right? I am using a different laptop so that is the reason I did not have the chance to check that quickly. Is that the case, to get the help the function should be used in the source file? Or how can learn these options? Best regards, Umut -- If I have a thousand ideas and only one turns out to be good, I am satisfied. Alfred Nobel From knepley at gmail.com Mon Mar 21 05:25:30 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 21 Mar 2011 05:25:30 -0500 Subject: [petsc-users] -pc_factor_shift_type command line In-Reply-To: <4D87253E.2010805@tudelft.nl> References: <4D87253E.2010805@tudelft.nl> Message-ID: On Mon, Mar 21, 2011 at 5:15 AM, Umut Tabak wrote: > Dear all, > > What are the options for -pc_factor_shift_type? > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/PC/PCFactorSetShiftType.html Matt > From the manual page, I understand I should use the -help option with the > binary, but I guess the source code should include this function to get the > help right? I am using a different laptop so that is the reason I did not > have the chance to check that quickly. Is that the case, to get the help the > function should be used in the source file? Or how can learn these options? > > Best regards, > Umut > > -- > If I have a thousand ideas and only one turns out to be good, > I am satisfied. > Alfred Nobel > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Mon Mar 21 06:14:30 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Mon, 21 Mar 2011 12:14:30 +0100 Subject: [petsc-users] MatAYPX for dense and sparse matrices Message-ID: <4D873316.8060209@gfz-potsdam.de> Hello! I have two matrices L and J. L is the sparse with 2 nonzeros per row maximum (quite often there are no values in a row at all) and J is the dense matrix. I need to add matrix L to matrix J so that: J = J + L I try to use: call MatAYPX(J,one,L,SUBSET_NONZERO_PATTERN,ierr) Because it seems most obvious and efficient way. and program crashes with this callstack: unknown :: BLASaxpy_ dense.c :: MatAXPY_SeqDense axpy.c :: MatAXPY axpy.c :: MatAYPX axpyf.c :: MATAYPX MODELING_MOD::MODELING I figured out the only one working combination: call MatAXPY(LT,one,JT,DIFFERENT_NONZERO_PATTERN,ierr) But this is very slow and inefficient. What actually could be done in such a situation? Thanks in advance. Regards, Alexander From aron.ahmadia at kaust.edu.sa Mon Mar 21 06:23:11 2011 From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia) Date: Mon, 21 Mar 2011 14:23:11 +0300 Subject: [petsc-users] MatAYPX for dense and sparse matrices In-Reply-To: <4D873316.8060209@gfz-potsdam.de> References: <4D873316.8060209@gfz-potsdam.de> Message-ID: Alex, If you are adding a sparse matrix to a dense one, you are better off just iterating through the values of your sparse matrix and adding them to your dense matrix. As far as I know, there are no routines in PETSc that will do this automatically for you (and this sort of thing is really not PETSc's strength). A On Mon, Mar 21, 2011 at 2:14 PM, Alexander Grayver wrote: > Hello! > > I have two matrices L and J. L is the sparse with 2 nonzeros per row > maximum (quite often there are no values in a row at all) and J is the dense > matrix. I need to add matrix L to matrix J so that: > > J = J + L > > I try to use: > call MatAYPX(J,one,L,SUBSET_NONZERO_PATTERN,ierr) > > Because it seems most obvious and efficient way. > > and program crashes with this callstack: > unknown :: BLASaxpy_ > dense.c :: MatAXPY_SeqDense > axpy.c :: MatAXPY > axpy.c :: MatAYPX > axpyf.c :: MATAYPX > MODELING_MOD::MODELING > > I figured out the only one working combination: > call MatAXPY(LT,one,JT,DIFFERENT_NONZERO_PATTERN,ierr) > > But this is very slow and inefficient. > > What actually could be done in such a situation? > > Thanks in advance. > > Regards, > Alexander > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Mon Mar 21 06:24:51 2011 From: jed at 59A2.org (Jed Brown) Date: Mon, 21 Mar 2011 12:24:51 +0100 Subject: [petsc-users] MatAYPX for dense and sparse matrices In-Reply-To: <4D873316.8060209@gfz-potsdam.de> References: <4D873316.8060209@gfz-potsdam.de> Message-ID: On Mon, Mar 21, 2011 at 12:14, Alexander Grayver wrote: > I have two matrices L and J. L is the sparse with 2 nonzeros per row > maximum (quite often there are no values in a row at all) and J is the dense > matrix. I need to add matrix L to matrix J so that: > > J = J + L > > I try to use: > call MatAYPX(J,one,L,SUBSET_NONZERO_PATTERN,ierr) > The dispatch for this is not set up for matrices of different type. From C, you can #include /* declaration of MatAXPY_Basic */ MatAXPY_Basic(J,1.0,L,SUBSET_NONZERO_PATTERN); There is not a Fortran binding for this function, but you could write the wrapper (basically just copy from src/mat/utils/ftn-auto/axpyf.c). Alternatively, the code for this is really simple, just iterate through the rows of L and insert them into J. -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Mon Mar 21 06:31:27 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Mon, 21 Mar 2011 12:31:27 +0100 Subject: [petsc-users] MatAYPX for dense and sparse matrices In-Reply-To: References: <4D873316.8060209@gfz-potsdam.de> Message-ID: <4D87370F.4000604@gfz-potsdam.de> On 21.03.2011 12:24, Jed Brown wrote: > On Mon, Mar 21, 2011 at 12:14, Alexander Grayver > > wrote: > > I have two matrices L and J. L is the sparse with 2 nonzeros per > row maximum (quite often there are no values in a row at all) and > J is the dense matrix. I need to add matrix L to matrix J so that: > > J = J + L > > I try to use: > call MatAYPX(J,one,L,SUBSET_NONZERO_PATTERN,ierr) > > > The dispatch for this is not set up for matrices of different type. > From C, you can > > #include /* declaration of MatAXPY_Basic */ > MatAXPY_Basic(J,1.0,L,SUBSET_NONZERO_PATTERN); > > There is not a Fortran binding for this function, but you could write > the wrapper (basically just copy from src/mat/utils/ftn-auto/axpyf.c). Would I have to rebuild PETSc from scratch to make this wrapper working? > Alternatively, the code for this is really simple, just iterate > through the rows of L and insert them into J. Do you mean using MatGetOwnershipRange+MatGetSubMatrices+MatGetRow? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Mon Mar 21 06:37:29 2011 From: jed at 59A2.org (Jed Brown) Date: Mon, 21 Mar 2011 12:37:29 +0100 Subject: [petsc-users] MatAYPX for dense and sparse matrices In-Reply-To: <4D87370F.4000604@gfz-potsdam.de> References: <4D873316.8060209@gfz-potsdam.de> <4D87370F.4000604@gfz-potsdam.de> Message-ID: On Mon, Mar 21, 2011 at 12:31, Alexander Grayver wrote: > Would I have to rebuild PETSc from scratch to make this wrapper working? > You would put it in your project and link it with your Fortran code. > > > Alternatively, the code for this is really simple, just iterate through > the rows of L and insert them into J. > > Do you mean using MatGetOwnershipRange+MatGetSubMatrices+MatGetRow? > Just translate the following (taken from MatAXPY_Basic) to Fortran: ierr = MatGetSize(X,&m,&n);CHKERRQ(ierr); ierr = MatGetOwnershipRange(X,&start,&end);CHKERRQ(ierr); for (i = start; i < end; i++) { ierr = MatGetRow(X,i,&ncols,&row,&vals);CHKERRQ(ierr); ierr = MatSetValues(Y,1,&i,ncols,row,vals,ADD_VALUES);CHKERRQ(ierr); ierr = MatRestoreRow(X,i,&ncols,&row,&vals);CHKERRQ(ierr); } ierr = MatAssemblyBegin(Y,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); ierr = MatAssemblyEnd(Y,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Mon Mar 21 06:40:49 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Mon, 21 Mar 2011 12:40:49 +0100 Subject: [petsc-users] MatAYPX for dense and sparse matrices In-Reply-To: References: <4D873316.8060209@gfz-potsdam.de> <4D87370F.4000604@gfz-potsdam.de> Message-ID: <4D873941.6010701@gfz-potsdam.de> On 21.03.2011 12:37, Jed Brown wrote: > On Mon, Mar 21, 2011 at 12:31, Alexander Grayver > > wrote: > > Would I have to rebuild PETSc from scratch to make this wrapper > working? > > > You would put it in your project and link it with your Fortran code. > > > >> Alternatively, the code for this is really simple, just iterate >> through the rows of L and insert them into J. > Do you mean using MatGetOwnershipRange+MatGetSubMatrices+MatGetRow? > > > Just translate the following (taken from MatAXPY_Basic) to Fortran: > > ierr = MatGetSize(X,&m,&n);CHKERRQ(ierr); > ierr = MatGetOwnershipRange(X,&start,&end);CHKERRQ(ierr); > for (i = start; i < end; i++) { > ierr = MatGetRow(X,i,&ncols,&row,&vals);CHKERRQ(ierr); > ierr = MatSetValues(Y,1,&i,ncols,row,vals,ADD_VALUES);CHKERRQ(ierr); > ierr = MatRestoreRow(X,i,&ncols,&row,&vals);CHKERRQ(ierr); > } > ierr = MatAssemblyBegin(Y,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = MatAssemblyEnd(Y,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > Thanks Jed, it looks simple enough for me (PETSc is great). One more question. I get matrix J out of MatMatMult operation and then I have to add L matrix up, wouldn't there be a conflict using MatAssemblyBegin/MatAssemblyEnd again? Generally, is it allowed to set new values into the matrix after MatAssemblyBegin/MatAssemblyEnd already has been done once? Thanks a lot. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alejandro.aragon at gmail.com Mon Mar 21 06:43:42 2011 From: alejandro.aragon at gmail.com (=?iso-8859-1?Q?Alejandro_Marcos_Arag=F3n?=) Date: Mon, 21 Mar 2011 12:43:42 +0100 Subject: [petsc-users] custom sparse matrix Message-ID: <4018865C-BDA1-49FA-ADDE-69AC51B977F8@gmail.com> Hi everyone, I'm new to the list and to PETSc. I was wondering if this is possible: in my code I have my own sparse matrix data structure, so instead of creating a sparse matrix from PETSc and copying all the elements, I was wondering if there is a way to specify the storage of the elements. My custom data structure is a three-array compressed row format (or compressed column storage). Thank you, Alejandro M. Arag?n, Ph.D. From knepley at gmail.com Mon Mar 21 06:45:25 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 21 Mar 2011 06:45:25 -0500 Subject: [petsc-users] MatAYPX for dense and sparse matrices In-Reply-To: <4D873941.6010701@gfz-potsdam.de> References: <4D873316.8060209@gfz-potsdam.de> <4D87370F.4000604@gfz-potsdam.de> <4D873941.6010701@gfz-potsdam.de> Message-ID: On Mon, Mar 21, 2011 at 6:40 AM, Alexander Grayver wrote: > On 21.03.2011 12:37, Jed Brown wrote: > > On Mon, Mar 21, 2011 at 12:31, Alexander Grayver wrote: > >> Would I have to rebuild PETSc from scratch to make this wrapper working? >> > > You would put it in your project and link it with your Fortran code. > > >> >> >> Alternatively, the code for this is really simple, just iterate through >> the rows of L and insert them into J. >> >> Do you mean using MatGetOwnershipRange+MatGetSubMatrices+MatGetRow? >> > > Just translate the following (taken from MatAXPY_Basic) to Fortran: > > ierr = MatGetSize(X,&m,&n);CHKERRQ(ierr); > ierr = MatGetOwnershipRange(X,&start,&end);CHKERRQ(ierr); > for (i = start; i < end; i++) { > ierr = MatGetRow(X,i,&ncols,&row,&vals);CHKERRQ(ierr); > ierr = MatSetValues(Y,1,&i,ncols,row,vals,ADD_VALUES);CHKERRQ(ierr); > ierr = MatRestoreRow(X,i,&ncols,&row,&vals);CHKERRQ(ierr); > } > ierr = MatAssemblyBegin(Y,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = MatAssemblyEnd(Y,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > > Thanks Jed, it looks simple enough for me (PETSc is great). > > One more question. I get matrix J out of MatMatMult operation and then I > have to add L matrix up, wouldn't there be a conflict using > MatAssemblyBegin/MatAssemblyEnd again? Generally, is it allowed to set new > values into the matrix after MatAssemblyBegin/MatAssemblyEnd already has > been done once? > No, you can use them. However, if you are adding to existing values on your process, nothing happens. Matt > Thanks a lot. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Mon Mar 21 06:45:37 2011 From: jed at 59A2.org (Jed Brown) Date: Mon, 21 Mar 2011 12:45:37 +0100 Subject: [petsc-users] MatAYPX for dense and sparse matrices In-Reply-To: <4D873941.6010701@gfz-potsdam.de> References: <4D873316.8060209@gfz-potsdam.de> <4D87370F.4000604@gfz-potsdam.de> <4D873941.6010701@gfz-potsdam.de> Message-ID: On Mon, Mar 21, 2011 at 12:40, Alexander Grayver wrote: > One more question. I get matrix J out of MatMatMult operation and then I > have to add L matrix up, wouldn't there be a conflict using > MatAssemblyBegin/MatAssemblyEnd again? Generally, is it allowed to set new > values into the matrix after MatAssemblyBegin/MatAssemblyEnd already has > been done once? > Yes, you can assemble again. Just set the values, then call MatAssembly{Begin,End}. It is not okay to set values, skip (re)assembling, and call MatMult() or something. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Mar 21 06:46:06 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 21 Mar 2011 06:46:06 -0500 Subject: [petsc-users] custom sparse matrix In-Reply-To: <4018865C-BDA1-49FA-ADDE-69AC51B977F8@gmail.com> References: <4018865C-BDA1-49FA-ADDE-69AC51B977F8@gmail.com> Message-ID: 2011/3/21 Alejandro Marcos Arag?n > Hi everyone, > > I'm new to the list and to PETSc. I was wondering if this is possible: in > my code I have my own sparse matrix data structure, so instead of creating a > sparse matrix from PETSc and copying all the elements, I was wondering if > there is a way to specify the storage of the elements. My custom data > structure is a three-array compressed row format (or compressed column > storage). > That is the PETSc AIJ format, so it should be easy to substitute. Matt > Thank you, > > Alejandro M. Arag?n, Ph.D. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Mon Mar 21 07:49:55 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Mon, 21 Mar 2011 13:49:55 +0100 Subject: [petsc-users] MatAYPX for dense and sparse matrices In-Reply-To: References: <4D873316.8060209@gfz-potsdam.de> <4D87370F.4000604@gfz-potsdam.de> <4D873941.6010701@gfz-potsdam.de> Message-ID: <4D874973.4080701@gfz-potsdam.de> On 21.03.2011 12:45, Matthew Knepley wrote: > On Mon, Mar 21, 2011 at 6:40 AM, Alexander Grayver > > wrote: > > On 21.03.2011 12 :37, Jed Brown wrote: >> On Mon, Mar 21, 2011 at 12:31, Alexander Grayver >> > wrote: >> >> Would I have to rebuild PETSc from scratch to make this >> wrapper working? >> >> >> You would put it in your project and link it with your Fortran code. >> >> >> >>> Alternatively, the code for this is really simple, just >>> iterate through the rows of L and insert them into J. >> Do you mean using >> MatGetOwnershipRange+MatGetSubMatrices+MatGetRow? >> >> >> Just translate the following (taken from MatAXPY_Basic) to Fortran: >> >> ierr = MatGetSize(X,&m,&n);CHKERRQ(ierr); >> ierr = MatGetOwnershipRange(X,&start,&end);CHKERRQ(ierr); >> for (i = start; i < end; i++) { >> ierr = MatGetRow(X,i,&ncols,&row,&vals);CHKERRQ(ierr); >> ierr = >> MatSetValues(Y,1,&i,ncols,row,vals,ADD_VALUES);CHKERRQ(ierr); >> ierr = MatRestoreRow(X,i,&ncols,&row,&vals);CHKERRQ(ierr); >> } >> ierr = MatAssemblyBegin(Y,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >> ierr = MatAssemblyEnd(Y,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >> > Thanks Jed, it looks simple enough for me (PETSc is great). > > One more question. I get matrix J out of MatMatMult operation and > then I have to add L matrix up, wouldn't there be a conflict using > MatAssemblyBegin/MatAssemblyEnd again? Generally, is it allowed to > set new values into the matrix after > MatAssemblyBegin/MatAssemblyEnd already has been done once? > > > No, you can use them. However, if you are adding to existing values on > your process, nothing happens. > > Matt > > Thanks a lot. > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener Matt Jed, Thanks a lot! -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Mar 21 08:14:03 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 21 Mar 2011 08:14:03 -0500 Subject: [petsc-users] custom sparse matrix In-Reply-To: <4018865C-BDA1-49FA-ADDE-69AC51B977F8@gmail.com> References: <4018865C-BDA1-49FA-ADDE-69AC51B977F8@gmail.com> Message-ID: <0955C1F3-3337-490C-9E8E-21EB90F702F1@mcs.anl.gov> On Mar 21, 2011, at 6:43 AM, Alejandro Marcos Arag?n wrote: > Hi everyone, > > I'm new to the list and to PETSc. I was wondering if this is possible: in my code I have my own sparse matrix data structure, so instead of creating a sparse matrix from PETSc and copying all the elements, I was wondering if there is a way to specify the storage of the elements. My custom data structure is a three-array compressed row format (or compressed column storage). If it truly is CSR you an use MatCreateSeqAIJWithArrays() or MatCreateMPIAIJWithArrays(). Otherwise you can use MatCreateShell() and provide your own function operations for MatMult() and whatever you will need. Generally keeping your own sparse matrix data structure is overrated, yes you may save a little time and a little memory but then you use all the flexibility and power of PETSc sparse matrices; why bother with PETSc in that case. Barry > > Thank you, > > Alejandro M. Arag?n, Ph.D. From alejandro.aragon at gmail.com Mon Mar 21 10:14:02 2011 From: alejandro.aragon at gmail.com (=?iso-8859-1?Q?Alejandro_Marcos_Arag=F3n?=) Date: Mon, 21 Mar 2011 16:14:02 +0100 Subject: [petsc-users] custom sparse matrix In-Reply-To: <0955C1F3-3337-490C-9E8E-21EB90F702F1@mcs.anl.gov> References: <4018865C-BDA1-49FA-ADDE-69AC51B977F8@gmail.com> <0955C1F3-3337-490C-9E8E-21EB90F702F1@mcs.anl.gov> Message-ID: <8602AD17-B5E7-47E7-BA77-27D5E299D095@gmail.com> Hi Barry, Thanks for your answer. I'll look into those functions, but instead of copying, do you know if there is a way to give the address of the arrays? I want to use the PETSc parallel solver capabilities, but I don't want to copy something that I already have. Now, I'm not familiar with the MatCreateShell() function that you mentioned. Could you expand on this? Thanks again Alejandro On Mar 21, 2011, at 2:14 PM, Barry Smith wrote: > > On Mar 21, 2011, at 6:43 AM, Alejandro Marcos Arag?n wrote: > >> Hi everyone, >> >> I'm new to the list and to PETSc. I was wondering if this is possible: in my code I have my own sparse matrix data structure, so instead of creating a sparse matrix from PETSc and copying all the elements, I was wondering if there is a way to specify the storage of the elements. My custom data structure is a three-array compressed row format (or compressed column storage). > > If it truly is CSR you an use MatCreateSeqAIJWithArrays() or MatCreateMPIAIJWithArrays(). Otherwise you can use MatCreateShell() and provide your own function operations for MatMult() and whatever you will need. > > Generally keeping your own sparse matrix data structure is overrated, yes you may save a little time and a little memory but then you use all the flexibility and power of PETSc sparse matrices; why bother with PETSc in that case. > > Barry > >> >> Thank you, >> >> Alejandro M. Arag?n, Ph.D. > From bsmith at mcs.anl.gov Mon Mar 21 10:19:28 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 21 Mar 2011 10:19:28 -0500 Subject: [petsc-users] custom sparse matrix In-Reply-To: <8602AD17-B5E7-47E7-BA77-27D5E299D095@gmail.com> References: <4018865C-BDA1-49FA-ADDE-69AC51B977F8@gmail.com> <0955C1F3-3337-490C-9E8E-21EB90F702F1@mcs.anl.gov> <8602AD17-B5E7-47E7-BA77-27D5E299D095@gmail.com> Message-ID: <4E12F39B-4768-4DD1-B7BC-B5A3182B082B@mcs.anl.gov> On Mar 21, 2011, at 10:14 AM, Alejandro Marcos Arag?n wrote: > Hi Barry, > > Thanks for your answer. I'll look into those functions, but instead of copying, do you know if there is a way to give the address of the arrays? MatCreateSeqAIJWithArrays() will not copy the arrays but just use them. In parallel MatCreateMPIAIJWithSplitArrays() will not copy the arrays, just use them but you need to have the correct format for the arrays (in parallel we use a "non-standard" parallel compressed sparse row format that likely you don't use but check the manual page http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-3.0.0/docs/manualpages/Mat/MatCreateMPIAIJWithSplitArrays.html > I want to use the PETSc parallel solver capabilities, but I don't want to copy something that I already have. Now, I'm not familiar with the MatCreateShell() function that you mentioned. Could you expand on this? http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-3.0.0/docs/manualpages/Mat/MatCreateShell.html > > Thanks again > > Alejandro > > > On Mar 21, 2011, at 2:14 PM, Barry Smith wrote: > >> >> On Mar 21, 2011, at 6:43 AM, Alejandro Marcos Arag?n wrote: >> >>> Hi everyone, >>> >>> I'm new to the list and to PETSc. I was wondering if this is possible: in my code I have my own sparse matrix data structure, so instead of creating a sparse matrix from PETSc and copying all the elements, I was wondering if there is a way to specify the storage of the elements. My custom data structure is a three-array compressed row format (or compressed column storage). >> >> If it truly is CSR you an use MatCreateSeqAIJWithArrays() or MatCreateMPIAIJWithArrays(). Otherwise you can use MatCreateShell() and provide your own function operations for MatMult() and whatever you will need. >> >> Generally keeping your own sparse matrix data structure is overrated, yes you may save a little time and a little memory but then you use all the flexibility and power of PETSc sparse matrices; why bother with PETSc in that case. >> >> Barry >> >>> >>> Thank you, >>> >>> Alejandro M. Arag?n, Ph.D. >> > From hung.thanh.nguyen at petrell.no Mon Mar 21 10:26:23 2011 From: hung.thanh.nguyen at petrell.no (Hung Thanh Nguyen) Date: Mon, 21 Mar 2011 16:26:23 +0100 Subject: [petsc-users] Problem with Petcs link to window Intel MKl mkl_blas95.lib, mkl_lapack95.lib and mkl_blas_intelmpi.lib Message-ID: Dear Petcs user I havea problem to Install PETCs on windows Intel MKl -platform. During last 2-weeks I spent to build PETCs on window Intel mkl - compiler I try different configure -procedure, for example: 1)./config/configure.py --with-cc='win32fe icc ' --with-fc='win32fe ifort' --with-cxx='win32fe icpc' -download-blas-lapack=1 Got the error message Cannot run executables created with C. If this machine uses a batch system to submit jobs you witl need to configure .................with the assitional option -with-batch ...Therefore, I try: 2)./config/configure.py -with-batch --with-cc='win32fe icc ' --with-fc='win32fe ifort' --with-cxx='win32fe icpc' -download-blas-lapack=1 And got the error message Fortran compiler you provide with -with-fc=win32fe ifort does not work I am also try the other way to configure as 3) ./config/configure.py --with-cc='win32fe icc' --with-fc='win32fe ifort' --with-cxx='win32fe icc' and I got the following error message: Could not found C-compilers 4)./config/configure.py -with-batch --with-cc='win32fe icl' --with-fc='win32fe ifort' --with-cxx='win32fe icl' --download-blas-lapack=1 and I got the following error message C compiler you provided with-cc=win32fe icl does not work 5)./config/configure.py --with-cc='win32fe icl' --with-fc='win32fe ifort' --with-cxx='win32fe icl' --download-blas-lapack=1 and I got the following error message C compiler you provided with-cc=win32fe icl does not work 6) ./config/configure.py --with-cc='win32fe cl' --with-fc='win32fe ifort' --with-cxx='win32fe cl' --download-f-blas-lapack=1 and I got the following error message C compiler you provided with-cc=win32fe cl does not work (Please see configure.log file in attenment) I hope someone can give me some advise to solve this problem. Thanks very much Hung T. Nguyen Research in Numerical Analys University in Troms?, Norway -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 27794 bytes Desc: configure.log URL: From jed at 59A2.org Mon Mar 21 10:27:19 2011 From: jed at 59A2.org (Jed Brown) Date: Mon, 21 Mar 2011 16:27:19 +0100 Subject: [petsc-users] custom sparse matrix In-Reply-To: <4E12F39B-4768-4DD1-B7BC-B5A3182B082B@mcs.anl.gov> References: <4018865C-BDA1-49FA-ADDE-69AC51B977F8@gmail.com> <0955C1F3-3337-490C-9E8E-21EB90F702F1@mcs.anl.gov> <8602AD17-B5E7-47E7-BA77-27D5E299D095@gmail.com> <4E12F39B-4768-4DD1-B7BC-B5A3182B082B@mcs.anl.gov> Message-ID: 2011/3/21 Barry Smith > > On Mar 21, 2011, at 10:14 AM, Alejandro Marcos Arag?n wrote: > > > Hi Barry, > > > > Thanks for your answer. I'll look into those functions, but instead of > copying, do you know if there is a way to give the address of the arrays? > > MatCreateSeqAIJWithArrays() will not copy the arrays but just use them. In > parallel MatCreateMPIAIJWithSplitArrays() will not copy the arrays, just use > them but you need to have the correct format for the arrays (in parallel we > use a "non-standard" parallel compressed sparse row format that likely you > don't use but check the manual page > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-3.0.0/docs/manualpages/Mat/MatCreateMPIAIJWithSplitArrays.html There is slightly more info in the more recent man page http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/Mat/MatCreateMPIAIJWithSplitArrays.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Mar 21 11:31:23 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 21 Mar 2011 11:31:23 -0500 Subject: [petsc-users] Problem with Petcs link to window Intel MKl mkl_blas95.lib, mkl_lapack95.lib and mkl_blas_intelmpi.lib In-Reply-To: References: Message-ID: On Mon, Mar 21, 2011 at 10:26 AM, Hung Thanh Nguyen < hung.thanh.nguyen at petrell.no> wrote: > Dear Petcs user > > > > I havea problem to Install PETCs on windows Intel MKl ?platform. During > last 2-weeks I spent to build PETCs on window Intel mkl ? compiler > > I try different configure ?procedure, for example: > > > > *1)./config/configure.py --with-cc=?win32fe icc ? --with-fc=?win32fe > ifort? --with-cxx=?win32fe icpc? -download-blas-lapack=1 * > > * * > > Got the error message > > > > Cannot run executables created with C. If this machine uses a batch system > to submit jobs you witl need to configure ?????..with the assitional option > ?with-batch > > ?Therefore, I try: > Your compilers do not appear to work in this shell. Can you compile a simple program? Thanks, Matt > > > *2)./config/configure.py ?with-batch --with-cc=?win32fe icc ? > --with-fc=?win32fe ifort? --with-cxx=?win32fe icpc? > -download-blas-lapack=1 * > > And got the error message > > > > Fortran compiler you provide with ?with-fc=win32fe ifort does not work > > > > I am also try the other way to configure as > > > > *3) ./config/configure.py --with-cc='win32fe icc' --with-fc='win32fe > ifort' --with-cxx='win32fe icc'* > > > and I got the following error message: > > *Could not found C-compilers* > > * * > > *4)./config/configure.py ?with-batch --with-cc='win32fe icl' > --with-fc='win32fe ifort' --with-cxx='win32fe icl' > --download-blas-lapack=1 * > > and I got the following error message > > C compiler you provided with?cc=win32fe icl does not work > > *5)./config/configure.py --with-cc='win32fe icl' --with-fc='win32fe > ifort' --with-cxx='win32fe icl' --download-blas-lapack=1* > > and I got the following error message > > C compiler you provided with?cc=win32fe icl does not work > > *6) ./config/configure.py --with-cc='win32fe cl' --with-fc='win32fe > ifort' --with-cxx='win32fe cl' --download-f-blas-lapack=1* > > and I got the following error message > > C compiler you provided with?cc=win32fe cl does not work > > (Please see configure.log file in attenment) > > > > I hope someone can give me some advise to solve this problem. Thanks very > much > > > > Hung T. Nguyen > > Research in Numerical Analys > > University in Troms?, Norway > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Mar 21 11:39:53 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 21 Mar 2011 11:39:53 -0500 Subject: [petsc-users] custom sparse matrix In-Reply-To: References: <4018865C-BDA1-49FA-ADDE-69AC51B977F8@gmail.com> <0955C1F3-3337-490C-9E8E-21EB90F702F1@mcs.anl.gov> <8602AD17-B5E7-47E7-BA77-27D5E299D095@gmail.com> <4E12F39B-4768-4DD1-B7BC-B5A3182B082B@mcs.anl.gov> Message-ID: On Mon, Mar 21, 2011 at 10:27 AM, Jed Brown wrote: > 2011/3/21 Barry Smith > >> >> On Mar 21, 2011, at 10:14 AM, Alejandro Marcos Arag?n wrote: >> >> > Hi Barry, >> > >> > Thanks for your answer. I'll look into those functions, but instead of >> copying, do you know if there is a way to give the address of the arrays? >> > I would encourage you to instead use the MatSetValues() interface for building your matrix directly, because: 1) It works for arbitrary parallel layout, and does all communication for you automatically 2) You can switch between formats to optimize for your architecture 3) You can switch to higher performance block matrix classes without changing your code Matt > MatCreateSeqAIJWithArrays() will not copy the arrays but just use them. In >> parallel MatCreateMPIAIJWithSplitArrays() will not copy the arrays, just use >> them but you need to have the correct format for the arrays (in parallel we >> use a "non-standard" parallel compressed sparse row format that likely you >> don't use but check the manual page >> http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-3.0.0/docs/manualpages/Mat/MatCreateMPIAIJWithSplitArrays.html > > > There is slightly more info in the more recent man page > > > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/Mat/MatCreateMPIAIJWithSplitArrays.html > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From ram at ibrae.ac.ru Mon Mar 21 20:11:02 2011 From: ram at ibrae.ac.ru (=?KOI8-R?B?4czFy9PFyiDy0drBzs/X?=) Date: Tue, 22 Mar 2011 04:11:02 +0300 Subject: [petsc-users] VecView doesn't work properly with DA global vectors Message-ID: Hi Is it ok, that VecView doesn't work properly with vectors, obtained with DACreateGlobalVector function, or im doing something wrong? In my program I create 5x5 da and global vector, corresponding to this da. Then each of 4 process writes its rank to its own local elements using DAVecGet(Restore)Array functions. Then i use VecView to visualize my global vector. And i see, that each process owns elements with different values. Where is the mistake? Thank you very much! Alexey Ryazanov. CODE: #include "petscksp.h" #include "petscda.h" static char help[] = "VecVeiw doesn't work properly with DAGlobalVectors"; int main(int argc,char **args) { Vec GlobalVec; DA da; PetscInt rank, dof, stencil_width, M, N, m, n, p, x, y, z, i, j; PetscScalar **array; PetscErrorCode ierr; PetscInitialize(&argc,&args,(char *)0,help); MPI_Comm_rank(PETSC_COMM_WORLD, &rank); M=N=5; dof=1; stencil_width=1; ierr = DACreate2d(PETSC_COMM_WORLD, DA_NONPERIODIC, DA_STENCIL_BOX, M, N, PETSC_DECIDE, PETSC_DECIDE, dof, stencil_width, PETSC_NULL, PETSC_NULL, &da); CHKERRQ(ierr); ierr = DACreateGlobalVector(da,&GlobalVec); CHKERRQ(ierr); ierr = DAGetCorners(da,&x,&y,&z,&m,&n,&p); CHKERRQ(ierr); ierr = DAVecGetArray(da, GlobalVec, &array); CHKERRQ(ierr); for(i=y; i From bsmith at mcs.anl.gov Mon Mar 21 20:28:33 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 21 Mar 2011 20:28:33 -0500 Subject: [petsc-users] VecView doesn't work properly with DA global vectors In-Reply-To: References: Message-ID: <2D6B0F6C-6359-4958-ACD4-E554F416813A@mcs.anl.gov> On Mar 21, 2011, at 8:11 PM, ??????? ??????? wrote: > Hi > > Is it ok, that VecView doesn't work properly with vectors, obtained with DACreateGlobalVector function, or im doing something wrong? > In my program I create 5x5 da and global vector, corresponding to this da. > Then each of 4 process writes its rank to its own local elements using DAVecGet(Restore)Array functions. > Then i use VecView to visualize my global vector. > And i see, that each process owns elements with different values. > Where is the mistake? > VecView() for DM global vectors present the vector entries in the global natural ordering. Since PETSc divides the domain with DA into squareish blocks when you print the values on a process (by calling DAVecGetArray) it is not in the global natural ordering. Hence the two are different. Barry > Thank you very much! > > Alexey Ryazanov. > > > > CODE: > > #include "petscksp.h" > #include "petscda.h" > static char help[] = "VecVeiw doesn't work properly with DAGlobalVectors"; > int main(int argc,char **args) > { > Vec GlobalVec; > DA da; > PetscInt rank, dof, stencil_width, M, N, m, n, p, x, y, z, i, j; > PetscScalar **array; > PetscErrorCode ierr; > > PetscInitialize(&argc,&args,(char *)0,help); > MPI_Comm_rank(PETSC_COMM_WORLD, &rank); > > M=N=5; dof=1; stencil_width=1; > ierr = DACreate2d(PETSC_COMM_WORLD, DA_NONPERIODIC, DA_STENCIL_BOX, M, N, PETSC_DECIDE, PETSC_DECIDE, dof, stencil_width, PETSC_NULL, PETSC_NULL, &da); > CHKERRQ(ierr); > ierr = DACreateGlobalVector(da,&GlobalVec); CHKERRQ(ierr); > > ierr = DAGetCorners(da,&x,&y,&z,&m,&n,&p); CHKERRQ(ierr); > > ierr = DAVecGetArray(da, GlobalVec, &array); CHKERRQ(ierr); > for(i=y; i for(j=x; j array[i][j]=rank; > ierr = DAVecRestoreArray(da, GlobalVec, &array); CHKERRQ(ierr); > > ierr = VecView(GlobalVec, PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr); > > ierr = VecDestroy(GlobalVec); CHKERRQ(ierr); > ierr = DADestroy(da); CHKERRQ(ierr); > ierr = PetscFinalize(); CHKERRQ(ierr); > return 0; > } > > > OUTPUT: > > Process [0] > 0 > 0 > 0 > 1 > 1 > 0 > 0 > 0 > 1 > Process [1] > 1 > 0 > 0 > 0 > 1 > 1 > Process [2] > 2 > 2 > 2 > 3 > 3 > 2 > Process [3] > 2 > 2 > 3 > 3 > From jed at 59A2.org Mon Mar 21 20:39:07 2011 From: jed at 59A2.org (Jed Brown) Date: Tue, 22 Mar 2011 02:39:07 +0100 Subject: [petsc-users] VecView doesn't work properly with DA global vectors In-Reply-To: <2D6B0F6C-6359-4958-ACD4-E554F416813A@mcs.anl.gov> References: <2D6B0F6C-6359-4958-ACD4-E554F416813A@mcs.anl.gov> Message-ID: You can set the "native" format to avoid doing this reordering. http://www.mcs.anl.gov/petsc/petsc-2/snapshots/petsc-dev/docs/manualpages/Viewer/PetscViewerSetFormat.html On Mar 22, 2011 2:28 AM, "Barry Smith" wrote: On Mar 21, 2011, at 8:11 PM, ??????? ??????? wrote: > Hi > > Is it ok, that VecView doesn't work ... VecView() for DM global vectors present the vector entries in the global natural ordering. Since PETSc divides the domain with DA into squareish blocks when you print the values on a process (by calling DAVecGetArray) it is not in the global natural ordering. Hence the two are different. Barry > Thank you very much! > > Alexey Ryazanov. > > > > CODE: > > #include "petscksp.h" > #incl... -------------- next part -------------- An HTML attachment was scrubbed... URL: From ram at ibrae.ac.ru Tue Mar 22 04:41:46 2011 From: ram at ibrae.ac.ru (=?KOI8-R?B?4czFy9PFyiDy0drBzs/X?=) Date: Tue, 22 Mar 2011 12:41:46 +0300 Subject: [petsc-users] VecView doesn't work properly with DA global vectors In-Reply-To: <2D6B0F6C-6359-4958-ACD4-E554F416813A@mcs.anl.gov> References: <2D6B0F6C-6359-4958-ACD4-E554F416813A@mcs.anl.gov> Message-ID: 2011/3/22 Barry Smith > > > VecView() for DM global vectors present the vector entries in the global > natural ordering. Since PETSc divides the domain with DA into squareish > blocks when you print the values on a process (by calling DAVecGetArray) it > is not in the global natural ordering. Hence the two are different. > > Barry > Thank You for you answer! Yes now I can see from the output, that VecView uses natural ordering, but its also very importaint to understand where (on which processor) elements are finaly stored. If I got it in a right way, VecView uses natural ordering and gives the WRONG information about actual distribution of memory among processors. And elemens are really splitted according the DA implementation. Am I right? Alexey -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Tue Mar 22 05:00:01 2011 From: jed at 59A2.org (Jed Brown) Date: Tue, 22 Mar 2011 11:00:01 +0100 Subject: [petsc-users] VecView doesn't work properly with DA global vectors In-Reply-To: References: <2D6B0F6C-6359-4958-ACD4-E554F416813A@mcs.anl.gov> Message-ID: On Tue, Mar 22, 2011 at 10:41, ??????? ??????? wrote: > Thank You for you answer! Yes now I can see from the output, that VecView > uses natural ordering, but its also very importaint to understand where (on > which processor) elements are finaly stored. If I got it in a right way, > VecView uses natural ordering and gives the WRONG information about actual > distribution of memory among processors. The natural ordering is different from the PETSc ordering. If you want to view the Vec in the order it is stored in, use PetscViewerPushFormat(viewer,PETSC_VIEWER_NATIVE); VecView(X,viewer); PetscViewerPopFormat(viewer); -------------- next part -------------- An HTML attachment was scrubbed... URL: From vkuhlem at emory.edu Tue Mar 22 10:58:21 2011 From: vkuhlem at emory.edu (Verena Kuhlemann) Date: Tue, 22 Mar 2011 11:58:21 -0400 Subject: [petsc-users] hypre Message-ID: Hello, I was wondering which version of HYPRE one should use with Petsc. I need to download it before the configuration of Petsc as --download-hypre=1 is not working for me. Thanks, Verena -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Mar 22 11:00:42 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 22 Mar 2011 11:00:42 -0500 Subject: [petsc-users] hypre In-Reply-To: References: Message-ID: On Tue, Mar 22, 2011 at 10:58 AM, Verena Kuhlemann wrote: > Hello, > > I was wondering which version of HYPRE one should use with Petsc. I need to > download it before > the configuration of Petsc as --download-hypre=1 is not working for me. > http://ftp.mcs.anl.gov/pub/petsc/externalpackages/hypre-2.7.0b.tar.gz Matt > Thanks, > Verena > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Mar 22 11:02:57 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 22 Mar 2011 11:02:57 -0500 (CDT) Subject: [petsc-users] hypre In-Reply-To: References: Message-ID: Which version of PETSc are you using? The URL for hypre is listed in config/PETSc/packages/hypre.py - for any given version of petsc. for petsc-3.1 its: http://ftp.mcs.anl.gov/pub/petsc/externalpackages/hypre-2.6.0b.tar.gz Satish On Tue, 22 Mar 2011, Verena Kuhlemann wrote: > Hello, > > I was wondering which version of HYPRE one should use with Petsc. I need to > download it before > the configuration of Petsc as --download-hypre=1 is not working for me. > > Thanks, > Verena > From bsmith at mcs.anl.gov Tue Mar 22 11:09:59 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 22 Mar 2011 11:09:59 -0500 Subject: [petsc-users] hypre In-Reply-To: References: Message-ID: <04440227-7807-4EDA-96D6-F44A337840C0@mcs.anl.gov> We need to make the specific tar file version more transparent: When the tarball is generated should a nice list of the tarballs for each external packages be automatically generated and put somehow in the document? Should we have a tool that extracts that information from the appropriate XXX.py upon request? How to tell people about this tool? If we have to tell them about the tool we might as just well tell them to look in the appropriate config/PETSc/packages/XXXX.py or config/BuildSystem/config/packagesXXX.py directly? Barry On Mar 22, 2011, at 11:02 AM, Satish Balay wrote: > Which version of PETSc are you using? The URL for hypre is listed in > config/PETSc/packages/hypre.py - for any given version of petsc. > > for petsc-3.1 its: http://ftp.mcs.anl.gov/pub/petsc/externalpackages/hypre-2.6.0b.tar.gz > > Satish > > On Tue, 22 Mar 2011, Verena Kuhlemann wrote: > >> Hello, >> >> I was wondering which version of HYPRE one should use with Petsc. I need to >> download it before >> the configuration of Petsc as --download-hypre=1 is not working for me. >> >> Thanks, >> Verena >> > From knepley at gmail.com Tue Mar 22 11:13:33 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 22 Mar 2011 11:13:33 -0500 Subject: [petsc-users] hypre In-Reply-To: <04440227-7807-4EDA-96D6-F44A337840C0@mcs.anl.gov> References: <04440227-7807-4EDA-96D6-F44A337840C0@mcs.anl.gov> Message-ID: On Tue, Mar 22, 2011 at 11:09 AM, Barry Smith wrote: > > We need to make the specific tar file version more transparent: > > When the tarball is generated should a nice list of the tarballs for each > external packages be automatically generated and put somehow in the > document? Should we have a tool that extracts that information from the > appropriate XXX.py upon request? How to tell people about this tool? If we > have to tell them about the tool we might as just well tell them to look in > the appropriate config/PETSc/packages/XXXX.py or > config/BuildSystem/config/packagesXXX.py directly? > How about ./configure --packgeInfo hypre ==> Hypre 2.6.0b http://ftp.mcs.anl.gov/pub/petsc/externalpackages/hypre-2.6.0b.tar.gz Matt > Barry > > > > On Mar 22, 2011, at 11:02 AM, Satish Balay wrote: > > > Which version of PETSc are you using? The URL for hypre is listed in > > config/PETSc/packages/hypre.py - for any given version of petsc. > > > > for petsc-3.1 its: > http://ftp.mcs.anl.gov/pub/petsc/externalpackages/hypre-2.6.0b.tar.gz > > > > Satish > > > > On Tue, 22 Mar 2011, Verena Kuhlemann wrote: > > > >> Hello, > >> > >> I was wondering which version of HYPRE one should use with Petsc. I need > to > >> download it before > >> the configuration of Petsc as --download-hypre=1 is not working for me. > >> > >> Thanks, > >> Verena > >> > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Mar 22 11:22:44 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 22 Mar 2011 11:22:44 -0500 (CDT) Subject: [petsc-users] hypre In-Reply-To: <04440227-7807-4EDA-96D6-F44A337840C0@mcs.anl.gov> References: <04440227-7807-4EDA-96D6-F44A337840C0@mcs.anl.gov> Message-ID: If --download-package doesn't work - it prints the URL of the package. If we need to document this further - 'configure --help' should print the defulat URLs for packages. Satish On Tue, 22 Mar 2011, Barry Smith wrote: > > We need to make the specific tar file version more transparent: > > When the tarball is generated should a nice list of the tarballs for each external packages be automatically generated and put somehow in the document? Should we have a tool that extracts that information from the appropriate XXX.py upon request? How to tell people about this tool? If we have to tell them about the tool we might as just well tell them to look in the appropriate config/PETSc/packages/XXXX.py or config/BuildSystem/config/packagesXXX.py directly? > > Barry > > > > On Mar 22, 2011, at 11:02 AM, Satish Balay wrote: > > > Which version of PETSc are you using? The URL for hypre is listed in > > config/PETSc/packages/hypre.py - for any given version of petsc. > > > > for petsc-3.1 its: http://ftp.mcs.anl.gov/pub/petsc/externalpackages/hypre-2.6.0b.tar.gz > > > > Satish > > > > On Tue, 22 Mar 2011, Verena Kuhlemann wrote: > > > >> Hello, > >> > >> I was wondering which version of HYPRE one should use with Petsc. I need to > >> download it before > >> the configuration of Petsc as --download-hypre=1 is not working for me. > >> > >> Thanks, > >> Verena > >> > > > > From bsmith at mcs.anl.gov Tue Mar 22 11:36:17 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 22 Mar 2011 11:36:17 -0500 Subject: [petsc-users] hypre In-Reply-To: References: <04440227-7807-4EDA-96D6-F44A337840C0@mcs.anl.gov> Message-ID: On Mar 22, 2011, at 11:22 AM, Satish Balay wrote: > If --download-package doesn't work - it prints the URL of the package. > > If we need to document this further - 'configure --help' should print > the defulat URLs for packages. Yes it should print it! The problem is that --help also prints a ton of stuff that gets confusing and in the way of important things. Better way to organize the output? Barry > Satish > > On Tue, 22 Mar 2011, Barry Smith wrote: > >> >> We need to make the specific tar file version more transparent: >> >> When the tarball is generated should a nice list of the tarballs for each external packages be automatically generated and put somehow in the document? Should we have a tool that extracts that information from the appropriate XXX.py upon request? How to tell people about this tool? If we have to tell them about the tool we might as just well tell them to look in the appropriate config/PETSc/packages/XXXX.py or config/BuildSystem/config/packagesXXX.py directly? >> >> Barry >> >> >> >> On Mar 22, 2011, at 11:02 AM, Satish Balay wrote: >> >>> Which version of PETSc are you using? The URL for hypre is listed in >>> config/PETSc/packages/hypre.py - for any given version of petsc. >>> >>> for petsc-3.1 its: http://ftp.mcs.anl.gov/pub/petsc/externalpackages/hypre-2.6.0b.tar.gz >>> >>> Satish >>> >>> On Tue, 22 Mar 2011, Verena Kuhlemann wrote: >>> >>>> Hello, >>>> >>>> I was wondering which version of HYPRE one should use with Petsc. I need to >>>> download it before >>>> the configuration of Petsc as --download-hypre=1 is not working for me. >>>> >>>> Thanks, >>>> Verena >>>> >>> >> >> > From knepley at gmail.com Tue Mar 22 11:37:32 2011 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 22 Mar 2011 11:37:32 -0500 Subject: [petsc-users] hypre In-Reply-To: References: <04440227-7807-4EDA-96D6-F44A337840C0@mcs.anl.gov> Message-ID: On Tue, Mar 22, 2011 at 11:36 AM, Barry Smith wrote: > > On Mar 22, 2011, at 11:22 AM, Satish Balay wrote: > > > If --download-package doesn't work - it prints the URL of the package. > > > > If we need to document this further - 'configure --help' should print > > the defulat URLs for packages. > > Yes it should print it! The problem is that --help also prints a ton of > stuff that gets confusing and in the way of important things. Better way to > organize the output? > Okay, no one replied to my other suggestion. How about ./configure --help Hypre prints only help for the Hypre section Matt > Barry > > > > Satish > > > > On Tue, 22 Mar 2011, Barry Smith wrote: > > > >> > >> We need to make the specific tar file version more transparent: > >> > >> When the tarball is generated should a nice list of the tarballs for > each external packages be automatically generated and put somehow in the > document? Should we have a tool that extracts that information from the > appropriate XXX.py upon request? How to tell people about this tool? If we > have to tell them about the tool we might as just well tell them to look in > the appropriate config/PETSc/packages/XXXX.py or > config/BuildSystem/config/packagesXXX.py directly? > >> > >> Barry > >> > >> > >> > >> On Mar 22, 2011, at 11:02 AM, Satish Balay wrote: > >> > >>> Which version of PETSc are you using? The URL for hypre is listed in > >>> config/PETSc/packages/hypre.py - for any given version of petsc. > >>> > >>> for petsc-3.1 its: > http://ftp.mcs.anl.gov/pub/petsc/externalpackages/hypre-2.6.0b.tar.gz > >>> > >>> Satish > >>> > >>> On Tue, 22 Mar 2011, Verena Kuhlemann wrote: > >>> > >>>> Hello, > >>>> > >>>> I was wondering which version of HYPRE one should use with Petsc. I > need to > >>>> download it before > >>>> the configuration of Petsc as --download-hypre=1 is not working for > me. > >>>> > >>>> Thanks, > >>>> Verena > >>>> > >>> > >> > >> > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Mar 22 11:51:13 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 22 Mar 2011 11:51:13 -0500 (CDT) Subject: [petsc-users] hypre In-Reply-To: References: <04440227-7807-4EDA-96D6-F44A337840C0@mcs.anl.gov> Message-ID: On Tue, 22 Mar 2011, Matthew Knepley wrote: > On Tue, Mar 22, 2011 at 11:36 AM, Barry Smith wrote: > > > > > On Mar 22, 2011, at 11:22 AM, Satish Balay wrote: > > > > > If --download-package doesn't work - it prints the URL of the package. > > > > > > If we need to document this further - 'configure --help' should print > > > the defulat URLs for packages. > > > > Yes it should print it! The problem is that --help also prints a ton of > > stuff that gets confusing and in the way of important things. Better way to > > organize the output? Grep is a good tool to pipe configure output. > > > > Okay, no one replied to my other suggestion. How about > > ./configure --help Hypre > > prints only help for the Hypre section I'm fine with this as long as --help also prints it. One can obtain a better 'section help' with grep [unless you are implementing 'grep i' features with the above] For eg: configure --help |grep -i url [should get all download-pacakge URLs] Satish > > Matt > > > > Barry > > > > > > > Satish > > > > > > On Tue, 22 Mar 2011, Barry Smith wrote: > > > > > >> > > >> We need to make the specific tar file version more transparent: > > >> > > >> When the tarball is generated should a nice list of the tarballs for > > each external packages be automatically generated and put somehow in the > > document? Should we have a tool that extracts that information from the > > appropriate XXX.py upon request? How to tell people about this tool? If we > > have to tell them about the tool we might as just well tell them to look in > > the appropriate config/PETSc/packages/XXXX.py or > > config/BuildSystem/config/packagesXXX.py directly? > > >> > > >> Barry > > >> > > >> > > >> > > >> On Mar 22, 2011, at 11:02 AM, Satish Balay wrote: > > >> > > >>> Which version of PETSc are you using? The URL for hypre is listed in > > >>> config/PETSc/packages/hypre.py - for any given version of petsc. > > >>> > > >>> for petsc-3.1 its: > > http://ftp.mcs.anl.gov/pub/petsc/externalpackages/hypre-2.6.0b.tar.gz > > >>> > > >>> Satish > > >>> > > >>> On Tue, 22 Mar 2011, Verena Kuhlemann wrote: > > >>> > > >>>> Hello, > > >>>> > > >>>> I was wondering which version of HYPRE one should use with Petsc. I > > need to > > >>>> download it before > > >>>> the configuration of Petsc as --download-hypre=1 is not working for > > me. > > >>>> > > >>>> Thanks, > > >>>> Verena > > >>>> > > >>> > > >> > > >> > > > > > > > > > > From bsmith at mcs.anl.gov Tue Mar 22 12:50:15 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 22 Mar 2011 12:50:15 -0500 Subject: [petsc-users] hypre In-Reply-To: References: <04440227-7807-4EDA-96D6-F44A337840C0@mcs.anl.gov> Message-ID: On Mar 22, 2011, at 11:37 AM, Matthew Knepley wrote: > On Tue, Mar 22, 2011 at 11:36 AM, Barry Smith wrote: > > On Mar 22, 2011, at 11:22 AM, Satish Balay wrote: > > > If --download-package doesn't work - it prints the URL of the package. > > > > If we need to document this further - 'configure --help' should print > > the defulat URLs for packages. > > Yes it should print it! The problem is that --help also prints a ton of stuff that gets confusing and in the way of important things. Better way to organize the output? > > Okay, no one replied to my other suggestion. How about > > ./configure --help Hypre > > prints only help for the Hypre section I think this is a good approach. This general mechanism will allow getting help for any particular subpart that people want help in. Most PETSc users don't know grep from a hole in the wall so that is not a good solution. Barry > > Matt > > Barry > > > > Satish > > > > On Tue, 22 Mar 2011, Barry Smith wrote: > > > >> > >> We need to make the specific tar file version more transparent: > >> > >> When the tarball is generated should a nice list of the tarballs for each external packages be automatically generated and put somehow in the document? Should we have a tool that extracts that information from the appropriate XXX.py upon request? How to tell people about this tool? If we have to tell them about the tool we might as just well tell them to look in the appropriate config/PETSc/packages/XXXX.py or config/BuildSystem/config/packagesXXX.py directly? > >> > >> Barry > >> > >> > >> > >> On Mar 22, 2011, at 11:02 AM, Satish Balay wrote: > >> > >>> Which version of PETSc are you using? The URL for hypre is listed in > >>> config/PETSc/packages/hypre.py - for any given version of petsc. > >>> > >>> for petsc-3.1 its: http://ftp.mcs.anl.gov/pub/petsc/externalpackages/hypre-2.6.0b.tar.gz > >>> > >>> Satish > >>> > >>> On Tue, 22 Mar 2011, Verena Kuhlemann wrote: > >>> > >>>> Hello, > >>>> > >>>> I was wondering which version of HYPRE one should use with Petsc. I need to > >>>> download it before > >>>> the configuration of Petsc as --download-hypre=1 is not working for me. > >>>> > >>>> Thanks, > >>>> Verena > >>>> > >>> > >> > >> > > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From balay at mcs.anl.gov Tue Mar 22 17:09:12 2011 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 22 Mar 2011 17:09:12 -0500 (CDT) Subject: [petsc-users] hypre In-Reply-To: References: Message-ID: On Tue, 22 Mar 2011, Verena Kuhlemann wrote: > Thanks for the answer. > > On Mar 22, 2011, at 12:02 PM, Satish Balay wrote: > > > Which version of PETSc are you using? The URL for hypre is listed in > > config/PETSc/packages/hypre.py - for any given version of petsc. > > > > This file does not exist in my version. Do you know what might have gone > wrong? You have corrupted PETSc sources. Suggest starting with a fresh tarball. [you might aswell start off with petsc-3.1-p8] > > > > for petsc-3.1 its: > > http://ftp.mcs.anl.gov/pub/petsc/externalpackages/hypre-2.6.0b.tar.gz > > > > I am using Petsc-3.1, so I used the hypre version you suggested. > When I try to compile one of my programs, here is what I get: > > mpicc -o defaultras2_hypre.o -c -g > -I/home/vkuhlem/research/petsc-3.1-p4/withhypre/include > -I/home/vkuhlem/research/petsc-3.1-p4/include > -I/home/vkuhlem/research/petsc-3.1-p4/withhypre/include > -I/home/vkuhlem/research/ParMetis-3.1.1/include -D__INSDIR__= > defaultras2_hypre.c > mpicc -g -o defaultras2_hypre defaultras2_hypre.o > -Wl,-rpath,/home/vkuhlem/research/petsc-3.1-p4/withhypre/lib > -L/home/vkuhlem/research/petsc-3.1-p4/withhypre/lib -lpetsc -lX11 > -Wl,-rpath,/home/vkuhlem/research/petsc-3.1-p4/withhypre/lib > -L/home/vkuhlem/research/petsc-3.1-p4/withhypre/lib -lHYPRE -lmv -lmpath > -lstdc++ -Wl,-rpath,/home/vkuhlem/research/ParMetis-3.1.1/lib > -L/home/vkuhlem/research/ParMetis-3.1.1/lib -lparmetis -lmetis -lflapack > -lfblas -L/opt/pathscale/lib/3.2 -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -ldl > -lmpich -lpscrt -lgcc_s -lmpichf90 -lmpichabiglue_psc -lpathfstart > -lpathfortran -lmv -lmpath -lm -lm -lm -lm -lmv -lmpath -lstdc++ -lmv -lmpath > -lstdc++ -ldl -lmpich -lpscrt -lgcc_s -ldl > /home/vkuhlem/research/petsc-3.1-p4/withhypre/lib/libpetsc.a(mhyp.o): In > function `MatZeroEntries_HYPREStruct_3d': > /home/vkuhlem/research/petsc-3.1-p4/src/dm/da/utils/mhyp.c:397: undefined > reference to `hypre_StructMatrixClearBoxValues' > collect2: ld returned 1 exit status > make: [defaultras2_hypre] Error 1 (ignored) > /bin/rm -f defaultras2_hypre.o > > > Did I configure Petsc wrong? I attached the configure.log. configure.log is too big to be flooded on the mailing list. For such issues requring configure.log - please use petsc-maint at mcs.anl.gov Satish From ckontzialis at lycos.com Tue Mar 22 21:46:25 2011 From: ckontzialis at lycos.com (Kontsantinos Kontzialis) Date: Wed, 23 Mar 2011 04:46:25 +0200 Subject: [petsc-users] matrix free and preconditioner Message-ID: <4D895F01.5050800@lycos.com> Dear all, I'm using a matrix free formulation of the Newton-Krylov method. My code is running when I use a constant matrix as a preconditioner. However, when I want to use another matrix (problem specific) I read to the manual that I have to use the -snes_mf_operator option, with appropriately setting snessetjacobian function. I do that in my code as follows: ierr = MatCreateSNESMF(sys.snes,&sys.J); CHKERRQ(ierr); ierr = SNESSetJacobian(sys.snes, sys.J, sys.P, jacobian_matrix,&sys); CHKERRQ(ierr); I have preallocated space for matrix sys.P, but when I start running I get the following error: [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Null argument, when expecting valid pointer! [1]PETSC ERROR: Null Object: Parameter # 1! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 - Ignored: 14:26:37 CST 2010 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: ./hoac_forth on a linux-gnu named localhost by kontzialis Wed Mar 23 04:42:52 2011 [1]PETSC ERROR: Libraries linked from /home/kontzialis/PETSC/petsc-3.1-p7/linux-gnu-c-debug/lib [1]PETSC ERROR: Configure run at Mon Feb 21 14:55:24 2011 [1]PETSC ERROR: Configure options --with-debugging=1 --with-cc=mpicc --with-fc=mpif90 --with-shared=1 --with-shared-libraries --wi[0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Null argument, when expecting valid pointer! [0]PETSC ERROR: Null Object: Parameter # 1! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 14:26:37 CST 2010 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./hoac_forth on a linux-gnu named localhost by kontzialis Wed Mar 23 04:42:52 2011 [0]PETSC ERROR: Libraries linked from /home/kontzialis/PETSC/petsc-3.1-p7/linux-gnu-c-debug/lib [0]PETSC ERROR: Configure run at Mon Feb 21 14:55:24 2011 [0]PETSC ERROR: Configure options --with-debugging=1 --with-cc=mpicc --with-fc=mpif90 --with-shared=1 --with-shared-libraries --with-large-file-io=1 --with-precision=double --with-blacs=1 --download-blacs=yes --download-f-blas-lapack=yes --with-plapack=1 --download-plapack=yes --with-scalapack=1 --download-scalapack=yes --with-superlu=1 --download-superlu=yes --with-superlu_dist=1 --download-superlu_dist=yes --with-ml=1 --download-ml=yes --with-umfpack=1 --download-umfpack=yes --with-mpi=1 --download-mpich=1 --with-sundials=1 --download-sundials=1 --with-parmetis=1 --download-parmetis=1 --with-hypre=1 --download-hypre=1 [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: PetscObjectGetComm() line 34 in src/sys/objects/gcomm.c [1]PETSC ERROR: VecNormBegin() line 495 in src/vec/vec/utils/comb.c [1]PETSC ERROR: MatMFFDCompute_WP() line 73 in src/mat/impls/mffd/wp.c [1]PETSC ERROR: MatMult_MFFD() line 315 in src/mat/impls/mffd/mffd.c [1]PETSC ERROR: MatMult() line 1899 in src/mat/interface/matrix.c [1]PETSC ERROR: PCApplyBAorAB() line 585 in src/ksp/pc/interface/precon.c [1]PETSC ERROR: Gth-large-file-io=1 --with-precision=double --with-blacs=1 --download-blacs=yes --download-f-blas-lapack=yes --with-plapack=1 --download-plapack=yes --with-scalapack=1 --download-scalapack=yes --with-superlu=1 --download-superlu=yes --with-superlu_dist=1 --download-superlu_dist=yes --with-ml=1 --download-ml=yes --with-umfpack=1 --download-umfpack=yes --with-mpi=1 --download-mpich=1 --with-sundials=1 --download-sundials=1 --with-parmetis=1 --download-parmetis=1 --with-hypre=1 --download-hypre=1 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: PetscObjectGetComm() line 34 in src/sys/objects/gcomm.c [0]PETSC ERROR: VecNormBegin() line 495 in src/vec/vec/utils/comb.c [0]PETSC ERROR: MatMFFDCompute_WP() line 73 in src/mat/impls/mffd/wp.c [0]PETSC ERROR: MatMult_MFFD() line 315 in src/mat/impls/mffd/mffd.c [0]PETSC ERROR: MatMult() line 1899 in src/mat/interface/matrix.c [0]PETSC ERROR: PCApplyBAorAB() line 585 in src/ksp/pc/interface/precon.c [0]PETSC ERROR: GMREScycle() line 161 in src/ksp/ksp/impls/gmres/gmres.c [1]PETSC ERROR: KSPSolve_GMRES() line 241 in src/ksp/ksp/impls/gmres/gmres.c [1]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c [1]PETSC ERROR: SNES_KSPSolve() line 2944 in src/snes/interface/snes.c [1]PETSC ERROR: SNESSolve_LS() line 191 in src/snes/impls/ls/ls.c [1]PETSC ERROR: SNESSolve() line 2255 in src/snes/interface/snes.c [1]PETSC ERROR: User provided function() line 33 in "unknowndirectory/"../crank_nickolson.c [1]PETSC ERROR: User provided function() line 84 in "unknowndirectory/"../implicit_time.c [1]PETSC ERROR: User provided function() line 1158 in "unknowndirectory/"../hoac.c MREScycle() line 161 in src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: KSPSolve_GMRES() line 241 in src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: SNES_KSPSolve() line 2944 in src/snes/interface/snes.c [0]PETSC ERROR: SNESSolve_LS() line 191 in src/snes/impls/ls/ls.c [0]PETSC ERROR: SNESSolve() line 2255 in src/snes/interface/snes.c [0]PETSC ERROR: User provided function() line 33 in "unknowndirectory/"../crank_nickolson.c [0]PETSC ERROR: User provided function() line 84 in "unknowndirectory/"../implicit_time.c [0]PETSC ERROR: User provided function() line 1158 in "unknowndirectory/"../hoac.c Please help. Costas - Done. matrix free and preconditioner.eml Subject: matrix free and preconditioner From: Kontsantinos Kontzialis Date: Wed, 23 Mar 2011 04:43:43 +0200 To: petsc-users-request at mcs.anl.gov Dear Petsc team, I'm using a matrix free formulation of the Newton-Krylov method. My code is running when I use a constant matrix as a preconditioner. However, when I want to use another matrix (problem specific) I read to the manual that I have to use the -snes_mf_operator option, with appropriately setting snessetjacobian function. I do that in my code as follows: ierr = MatCreateSNESMF(sys.snes, &sys.J); CHKERRQ(ierr); ierr = SNESSetJacobian(sys.snes, sys.J, sys.P, jacobian_matrix, &sys); CHKERRQ(ierr); I have preallocated space for matrix sys.P, but when I start running I get the following error: [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Null argument, when expecting valid pointer! [1]PETSC ERROR: Null Object: Parameter # 1! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 14:26:37 CST 2010 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: ./hoac_forth on a linux-gnu named localhost by kontzialis Wed Mar 23 04:42:52 2011 [1]PETSC ERROR: Libraries linked from /home/kontzialis/PETSC/petsc-3.1-p7/linux-gnu-c-debug/lib [1]PETSC ERROR: Configure run at Mon Feb 21 14:55:24 2011 [1]PETSC ERROR: Configure options --with-debugging=1 --with-cc=mpicc --with-fc=mpif90 --with-shared=1 --with-shared-libraries --wi[0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Null argument, when expecting valid pointer! [0]PETSC ERROR: Null Object: Parameter # 1! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 14:26:37 CST 2010 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./hoac_forth on a linux-gnu named localhost by kontzialis Wed Mar 23 04:42:52 2011 [0]PETSC ERROR: Libraries linked from /home/kontzialis/PETSC/petsc-3.1-p7/linux-gnu-c-debug/lib [0]PETSC ERROR: Configure run at Mon Feb 21 14:55:24 2011 [0]PETSC ERROR: Configure options --with-debugging=1 --with-cc=mpicc --with-fc=mpif90 --with-shared=1 --with-shared-libraries --with-large-file-io=1 --with-precision=double --with-blacs=1 --download-blacs=yes --download-f-blas-lapack=yes --with-plapack=1 --download-plapack=yes --with-scalapack=1 --download-scalapack=yes --with-superlu=1 --download-superlu=yes --with-superlu_dist=1 --download-superlu_dist=yes --with-ml=1 --download-ml=yes --with-umfpack=1 --download-umfpack=yes --with-mpi=1 --download-mpich=1 --with-sundials=1 --download-sundials=1 --with-parmetis=1 --download-parmetis=1 --with-hypre=1 --download-hypre=1 [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: PetscObjectGetComm() line 34 in src/sys/objects/gcomm.c [1]PETSC ERROR: VecNormBegin() line 495 in src/vec/vec/utils/comb.c [1]PETSC ERROR: MatMFFDCompute_WP() line 73 in src/mat/impls/mffd/wp.c [1]PETSC ERROR: MatMult_MFFD() line 315 in src/mat/impls/mffd/mffd.c [1]PETSC ERROR: MatMult() line 1899 in src/mat/interface/matrix.c [1]PETSC ERROR: PCApplyBAorAB() line 585 in src/ksp/pc/interface/precon.c [1]PETSC ERROR: Gth-large-file-io=1 --with-precision=double --with-blacs=1 --download-blacs=yes --download-f-blas-lapack=yes --with-plapack=1 --download-plapack=yes --with-scalapack=1 --download-scalapack=yes --with-superlu=1 --download-superlu=yes --with-superlu_dist=1 --download-superlu_dist=yes --with-ml=1 --download-ml=yes --with-umfpack=1 --download-umfpack=yes --with-mpi=1 --download-mpich=1 --with-sundials=1 --download-sundials=1 --with-parmetis=1 --download-parmetis=1 --with-hypre=1 --download-hypre=1 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: PetscObjectGetComm() line 34 in src/sys/objects/gcomm.c [0]PETSC ERROR: VecNormBegin() line 495 in src/vec/vec/utils/comb.c [0]PETSC ERROR: MatMFFDCompute_WP() line 73 in src/mat/impls/mffd/wp.c [0]PETSC ERROR: MatMult_MFFD() line 315 in src/mat/impls/mffd/mffd.c [0]PETSC ERROR: MatMult() line 1899 in src/mat/interface/matrix.c [0]PETSC ERROR: PCApplyBAorAB() line 585 in src/ksp/pc/interface/precon.c [0]PETSC ERROR: GMREScycle() line 161 in src/ksp/ksp/impls/gmres/gmres.c [1]PETSC ERROR: KSPSolve_GMRES() line 241 in src/ksp/ksp/impls/gmres/gmres.c [1]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c [1]PETSC ERROR: SNES_KSPSolve() line 2944 in src/snes/interface/snes.c [1]PETSC ERROR: SNESSolve_LS() line 191 in src/snes/impls/ls/ls.c [1]PETSC ERROR: SNESSolve() line 2255 in src/snes/interface/snes.c [1]PETSC ERROR: User provided function() line 33 in "unknowndirectory/"../crank_nickolson.c [1]PETSC ERROR: User provided function() line 84 in "unknowndirectory/"../implicit_time.c [1]PETSC ERROR: User provided function() line 1158 in "unknowndirectory/"../hoac.c MREScycle() line 161 in src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: KSPSolve_GMRES() line 241 in src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: SNES_KSPSolve() line 2944 in src/snes/interface/snes.c [0]PETSC ERROR: SNESSolve_LS() line 191 in src/snes/impls/ls/ls.c [0]PETSC ERROR: SNESSolve() line 2255 in src/snes/interface/snes.c [0]PETSC ERROR: User provided function() line 33 in "unknowndirectory/"../crank_nickolson.c [0]PETSC ERROR: User provided function() line 84 in "unknowndirectory/"../implicit_time.c [0]PETSC ERROR: User provided function() line 1158 in "unknowndirectory/"../hoac.c Please help. Costas -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Mar 22 22:29:50 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 22 Mar 2011 22:29:50 -0500 Subject: [petsc-users] matrix free and preconditioner In-Reply-To: <4D895F01.5050800@lycos.com> References: <4D895F01.5050800@lycos.com> Message-ID: Call MatMFFDSetFromOptions(sys.J); after the MatCreateSNESMF(). Your jacobian_matrix() HAS to call MatAssemblyBegin/End() on the first matrix passed into it (this one passed in will be sys.J) as well as whatever it does for the sys.P matrix. If this does not resolve the problem please let us know. Barry On Mar 22, 2011, at 9:46 PM, Kontsantinos Kontzialis wrote: > Dear all, > > I'm using a matrix free formulation of the Newton-Krylov method. > My code is running when I use a constant matrix as a preconditioner. > However, when I want to use another matrix (problem specific) I read to the > manual that I have to use the -snes_mf_operator option, with > appropriately setting > snessetjacobian function. I do that in my code as follows: > ierr = MatCreateSNESMF(sys.snes, &sys.J); > CHKERRQ(ierr); > ierr = SNESSetJacobian(sys.snes, sys.J, sys.P, jacobian_matrix, &sys); > CHKERRQ(ierr); > I have preallocated space for matrix sys.P, but when I start running I > get the following error: > [1]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [1]PETSC ERROR: Null argument, when expecting valid pointer! > [1]PETSC ERROR: Null Object: Parameter # 1! > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 > > - Ignored: > 14:26:37 CST 2010 > [1]PETSC ERROR: See docs/changes/index.html for recent updates. > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [1]PETSC ERROR: See docs/index.html for manual pages. > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: ./hoac_forth on a linux-gnu named localhost by > kontzialis Wed Mar 23 04:42:52 2011 > [1]PETSC ERROR: Libraries linked from > /home/kontzialis/PETSC/petsc-3.1-p7/linux-gnu-c-debug/lib > [1]PETSC ERROR: Configure run at Mon Feb 21 14:55:24 2011 > [1]PETSC ERROR: Configure options --with-debugging=1 --with-cc=mpicc > --with-fc=mpif90 --with-shared=1 --with-shared-libraries --wi[0]PETSC > ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Null argument, when expecting valid pointer! > [0]PETSC ERROR: Null Object: Parameter # 1! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 > 14:26:37 CST 2010 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./hoac_forth on a linux-gnu named localhost by > kontzialis Wed Mar 23 04:42:52 2011 > [0]PETSC ERROR: Libraries linked from > /home/kontzialis/PETSC/petsc-3.1-p7/linux-gnu-c-debug/lib > [0]PETSC ERROR: Configure run at Mon Feb 21 14:55:24 2011 > [0]PETSC ERROR: Configure options --with-debugging=1 --with-cc=mpicc > --with-fc=mpif90 --with-shared=1 --with-shared-libraries > --with-large-file-io=1 --with-precision=double --with-blacs=1 > --download-blacs=yes --download-f-blas-lapack=yes --with-plapack=1 > --download-plapack=yes --with-scalapack=1 --download-scalapack=yes > --with-superlu=1 --download-superlu=yes --with-superlu_dist=1 > --download-superlu_dist=yes --with-ml=1 --download-ml=yes > --with-umfpack=1 --download-umfpack=yes --with-mpi=1 --download-mpich=1 > --with-sundials=1 --download-sundials=1 --with-parmetis=1 > --download-parmetis=1 --with-hypre=1 --download-hypre=1 > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: PetscObjectGetComm() line 34 in src/sys/objects/gcomm.c > [1]PETSC ERROR: VecNormBegin() line 495 in src/vec/vec/utils/comb.c > [1]PETSC ERROR: MatMFFDCompute_WP() line 73 in src/mat/impls/mffd/wp.c > [1]PETSC ERROR: MatMult_MFFD() line 315 in src/mat/impls/mffd/mffd.c > [1]PETSC ERROR: MatMult() line 1899 in src/mat/interface/matrix.c > [1]PETSC ERROR: PCApplyBAorAB() line 585 in src/ksp/pc/interface/precon.c > [1]PETSC ERROR: Gth-large-file-io=1 --with-precision=double > --with-blacs=1 --download-blacs=yes --download-f-blas-lapack=yes > --with-plapack=1 --download-plapack=yes --with-scalapack=1 > --download-scalapack=yes --with-superlu=1 --download-superlu=yes > --with-superlu_dist=1 --download-superlu_dist=yes --with-ml=1 > --download-ml=yes --with-umfpack=1 --download-umfpack=yes --with-mpi=1 > --download-mpich=1 --with-sundials=1 --download-sundials=1 > --with-parmetis=1 --download-parmetis=1 --with-hypre=1 --download-hypre=1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: PetscObjectGetComm() line 34 in src/sys/objects/gcomm.c > [0]PETSC ERROR: VecNormBegin() line 495 in src/vec/vec/utils/comb.c > [0]PETSC ERROR: MatMFFDCompute_WP() line 73 in src/mat/impls/mffd/wp.c > [0]PETSC ERROR: MatMult_MFFD() line 315 in src/mat/impls/mffd/mffd.c > [0]PETSC ERROR: MatMult() line 1899 in src/mat/interface/matrix.c > [0]PETSC ERROR: PCApplyBAorAB() line 585 in src/ksp/pc/interface/precon.c > [0]PETSC ERROR: GMREScycle() line 161 in src/ksp/ksp/impls/gmres/gmres.c > [1]PETSC ERROR: KSPSolve_GMRES() line 241 in src/ksp/ksp/impls/gmres/gmres.c > [1]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c > [1]PETSC ERROR: SNES_KSPSolve() line 2944 in src/snes/interface/snes.c > [1]PETSC ERROR: SNESSolve_LS() line 191 in src/snes/impls/ls/ls.c > [1]PETSC ERROR: SNESSolve() line 2255 in src/snes/interface/snes.c > [1]PETSC ERROR: User provided function() line 33 in > "unknowndirectory/"../crank_nickolson.c > [1]PETSC ERROR: User provided function() line 84 in > "unknowndirectory/"../implicit_time.c > [1]PETSC ERROR: User provided function() line 1158 in > "unknowndirectory/"../hoac.c > MREScycle() line 161 in src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: KSPSolve_GMRES() line 241 in src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: SNES_KSPSolve() line 2944 in src/snes/interface/snes.c > [0]PETSC ERROR: SNESSolve_LS() line 191 in src/snes/impls/ls/ls.c > [0]PETSC ERROR: SNESSolve() line 2255 in src/snes/interface/snes.c > [0]PETSC ERROR: User provided function() line 33 in > "unknowndirectory/"../crank_nickolson.c > [0]PETSC ERROR: User provided function() line 84 in > "unknowndirectory/"../implicit_time.c > [0]PETSC ERROR: User provided function() line 1158 in > "unknowndirectory/"../hoac.c > > Please help. > > Costas > > > - Done. > > > > matrix free and preconditioner.eml > Subject: matrix free and preconditioner > From: Kontsantinos Kontzialis > Date: Wed, 23 Mar 2011 04:43:43 +0200 > To: petsc-users-request at mcs.anl.gov > Dear Petsc team, > > I'm using a matrix free formulation of the Newton-Krylov method. > My code is running when I use a constant matrix as a preconditioner. > However, when I want to use another matrix (problem specific) I read to the > manual that I have to use the -snes_mf_operator option, with appropriately setting > snessetjacobian function. I do that in my code as follows: > > ierr = MatCreateSNESMF(sys.snes, &sys.J); > CHKERRQ(ierr); > > ierr = SNESSetJacobian(sys.snes, sys.J, sys.P, jacobian_matrix, &sys); > CHKERRQ(ierr); > > I have preallocated space for matrix sys.P, but when I start running I get the following error: > > [1]PETSC ERROR: --------------------- Error Message ------------------------------------ > [1]PETSC ERROR: Null argument, when expecting valid pointer! > [1]PETSC ERROR: Null Object: Parameter # 1! > [1]PETSC ERROR: ------------------------------------------------------------------------ > [1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 14:26:37 CST 2010 > [1]PETSC ERROR: See docs/changes/index.html for recent updates. > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [1]PETSC ERROR: See docs/index.html for manual pages. > [1]PETSC ERROR: ------------------------------------------------------------------------ > [1]PETSC ERROR: ./hoac_forth on a linux-gnu named localhost by kontzialis Wed Mar 23 04:42:52 2011 > [1]PETSC ERROR: Libraries linked from /home/kontzialis/PETSC/petsc-3.1-p7/linux-gnu-c-debug/lib > [1]PETSC ERROR: Configure run at Mon Feb 21 14:55:24 2011 > [1]PETSC ERROR: Configure options --with-debugging=1 --with-cc=mpicc --with-fc=mpif90 --with-shared=1 --with-shared-libraries --wi[0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Null argument, when expecting valid pointer! > [0]PETSC ERROR: Null Object: Parameter # 1! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 14:26:37 CST 2010 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ./hoac_forth on a linux-gnu named localhost by kontzialis Wed Mar 23 04:42:52 2011 > [0]PETSC ERROR: Libraries linked from /home/kontzialis/PETSC/petsc-3.1-p7/linux-gnu-c-debug/lib > [0]PETSC ERROR: Configure run at Mon Feb 21 14:55:24 2011 > [0]PETSC ERROR: Configure options --with-debugging=1 --with-cc=mpicc --with-fc=mpif90 --with-shared=1 --with-shared-libraries --with-large-file-io=1 --with-precision=double --with-blacs=1 --download-blacs=yes --download-f-blas-lapack=yes --with-plapack=1 --download-plapack=yes --with-scalapack=1 --download-scalapack=yes --with-superlu=1 --download-superlu=yes --with-superlu_dist=1 --download-superlu_dist=yes --with-ml=1 --download-ml=yes --with-umfpack=1 --download-umfpack=yes --with-mpi=1 --download-mpich=1 --with-sundials=1 --download-sundials=1 --with-parmetis=1 --download-parmetis=1 --with-hypre=1 --download-hypre=1 > [1]PETSC ERROR: ------------------------------------------------------------------------ > [1]PETSC ERROR: PetscObjectGetComm() line 34 in src/sys/objects/gcomm.c > [1]PETSC ERROR: VecNormBegin() line 495 in src/vec/vec/utils/comb.c > [1]PETSC ERROR: MatMFFDCompute_WP() line 73 in src/mat/impls/mffd/wp.c > [1]PETSC ERROR: MatMult_MFFD() line 315 in src/mat/impls/mffd/mffd.c > [1]PETSC ERROR: MatMult() line 1899 in src/mat/interface/matrix.c > [1]PETSC ERROR: PCApplyBAorAB() line 585 in src/ksp/pc/interface/precon.c > [1]PETSC ERROR: Gth-large-file-io=1 --with-precision=double --with-blacs=1 --download-blacs=yes --download-f-blas-lapack=yes --with-plapack=1 --download-plapack=yes --with-scalapack=1 --download-scalapack=yes --with-superlu=1 --download-superlu=yes --with-superlu_dist=1 --download-superlu_dist=yes --with-ml=1 --download-ml=yes --with-umfpack=1 --download-umfpack=yes --with-mpi=1 --download-mpich=1 --with-sundials=1 --download-sundials=1 --with-parmetis=1 --download-parmetis=1 --with-hypre=1 --download-hypre=1 > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: PetscObjectGetComm() line 34 in src/sys/objects/gcomm.c > [0]PETSC ERROR: VecNormBegin() line 495 in src/vec/vec/utils/comb.c > [0]PETSC ERROR: MatMFFDCompute_WP() line 73 in src/mat/impls/mffd/wp.c > [0]PETSC ERROR: MatMult_MFFD() line 315 in src/mat/impls/mffd/mffd.c > [0]PETSC ERROR: MatMult() line 1899 in src/mat/interface/matrix.c > [0]PETSC ERROR: PCApplyBAorAB() line 585 in src/ksp/pc/interface/precon.c > [0]PETSC ERROR: GMREScycle() line 161 in src/ksp/ksp/impls/gmres/gmres.c > [1]PETSC ERROR: KSPSolve_GMRES() line 241 in src/ksp/ksp/impls/gmres/gmres.c > [1]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c > [1]PETSC ERROR: SNES_KSPSolve() line 2944 in src/snes/interface/snes.c > [1]PETSC ERROR: SNESSolve_LS() line 191 in src/snes/impls/ls/ls.c > [1]PETSC ERROR: SNESSolve() line 2255 in src/snes/interface/snes.c > [1]PETSC ERROR: User provided function() line 33 in "unknowndirectory/"../crank_nickolson.c > [1]PETSC ERROR: User provided function() line 84 in "unknowndirectory/"../implicit_time.c > [1]PETSC ERROR: User provided function() line 1158 in "unknowndirectory/"../hoac.c > MREScycle() line 161 in src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: KSPSolve_GMRES() line 241 in src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: SNES_KSPSolve() line 2944 in src/snes/interface/snes.c > [0]PETSC ERROR: SNESSolve_LS() line 191 in src/snes/impls/ls/ls.c > [0]PETSC ERROR: SNESSolve() line 2255 in src/snes/interface/snes.c > [0]PETSC ERROR: User provided function() line 33 in "unknowndirectory/"../crank_nickolson.c > [0]PETSC ERROR: User provided function() line 84 in "unknowndirectory/"../implicit_time.c > [0]PETSC ERROR: User provided function() line 1158 in "unknowndirectory/"../hoac.c > > Please help. > > Costas From alejandro.aragon at gmail.com Wed Mar 23 04:59:11 2011 From: alejandro.aragon at gmail.com (=?iso-8859-1?Q?Alejandro_Marcos_Arag=F3n?=) Date: Wed, 23 Mar 2011 10:59:11 +0100 Subject: [petsc-users] MatCreateMPIAIJWithSplitArrays Message-ID: Hi PETSc users, My code uses a custom sparse matrix format that uses a hashed container for the entries of the matrix. The interface can then create a matrix in compressed row (or compressed column) storage format. That is, it can give me the three arrays needed to represent the sparse matrix in these formats. I would like to use the PETSc parallel solver, so then I thought that it would be good to try the MatCreateMPIAIJWithSplitArrays function so that I don't have to copy the values to the PETSc matrix again. Now my question to you is that I really don't get the point of having a diagonal and off-diagonal blocks of the sparse matrix. In the compressed row storage format, there is no distinction between these two blocks. Besides, I don't think there is a clear way to determine which is the boundary between these two blocks. Can someone point me how I should use this function, or if there is a better function that can take the three arrays that I have at this point? Also, since the sparse matrix in each process is the result of a finite element assembly routine, some rows are overlapped among the processes (there are several finite element nodes shared among the processes). At this point using the MatSetValues with the ADD_VALUES flag works fine, but I want to make sure that if I use the MatCreateMPIAIJWithSplitArrays (where I need to set the number of local rows) I can still get this behavior. In other words, if I sum the number of local rows in each process, I get a total number of rows that is greater than the number of global rows because of superposition. Thank you all, Alejandro M. Arag?n, Ph.D. From jed at 59A2.org Wed Mar 23 06:34:36 2011 From: jed at 59A2.org (Jed Brown) Date: Wed, 23 Mar 2011 12:34:36 +0100 Subject: [petsc-users] MatCreateMPIAIJWithSplitArrays In-Reply-To: References: Message-ID: 2011/3/23 Alejandro Marcos Arag?n > My code uses a custom sparse matrix format that uses a hashed container for > the entries of the matrix. The interface can then create a matrix in > compressed row (or compressed column) storage format. That is, it can give > me the three arrays needed to represent the sparse matrix in these formats. > I would like to use the PETSc parallel solver, so then I thought that it > would be good to try the MatCreateMPIAIJWithSplitArrays function so that I > don't have to copy the values to the PETSc matrix again. > Have your hashed format generate CSR rows and insert those one row at a time using MatSetValues(). It will be very fast and avoid the copy of using MatCreateMPIAIJWithArrays. > > Now my question to you is that I really don't get the point of having a > diagonal and off-diagonal blocks of the sparse matrix. > Storing the diagonal and off-diagonal part separately makes MatMult more tolerant to network latency and permits block Jacobi preconditioning without needing to copy out the local blocks. > In the compressed row storage format, there is no distinction between these > two blocks. Besides, I don't think there is a clear way to determine which > is the boundary between these two blocks. Can someone point me how I should > use this function, or if there is a better function that can take the three > arrays that I have at this point? > > Also, since the sparse matrix in each process is the result of a finite > element assembly routine, some rows are overlapped among the processes > (there are several finite element nodes shared among the processes). At this > point using the MatSetValues with the ADD_VALUES flag works fine, but I want > to make sure that if I use the MatCreateMPIAIJWithSplitArrays (where I need > to set the number of local rows) I can still get this behavior. In other > words, if I sum the number of local rows in each process, I get a total > number of rows that is greater than the number of global rows because of > superposition. > Just use MatSetValues(), it will take care of parallel assembly. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ram at ibrae.ac.ru Wed Mar 23 07:10:18 2011 From: ram at ibrae.ac.ru (=?KOI8-R?B?4czFy9PFyiDy0drBzs/X?=) Date: Wed, 23 Mar 2011 15:10:18 +0300 Subject: [petsc-users] VecView doesn't work properly with DA global vectors In-Reply-To: References: <2D6B0F6C-6359-4958-ACD4-E554F416813A@mcs.anl.gov> Message-ID: Thank You, Jed! I'll follow your advice. Now im having some problems with using nondefault viewers, but i think i can cope with it. Thanks 2011/3/22 Jed Brown > On Tue, Mar 22, 2011 at 10:41, ??????? ??????? wrote: > >> Thank You for you answer! Yes now I can see from the output, that VecView >> uses natural ordering, but its also very importaint to understand where (on >> which processor) elements are finaly stored. If I got it in a right way, >> VecView uses natural ordering and gives the WRONG information about actual >> distribution of memory among processors. > > > The natural ordering is different from the PETSc ordering. If you want to > view the Vec in the order it is stored in, use > > PetscViewerPushFormat(viewer,PETSC_VIEWER_NATIVE); > VecView(X,viewer); > PetscViewerPopFormat(viewer); > -------------- next part -------------- An HTML attachment was scrubbed... URL: From m.skates82 at gmail.com Wed Mar 23 07:48:45 2011 From: m.skates82 at gmail.com (Nun ion) Date: Wed, 23 Mar 2011 07:48:45 -0500 Subject: [petsc-users] AMG for PETSc Message-ID: Hello all, I am interested in using AMG for PETSc, where my matrices are complex. I looked at the documentation for Hypre and I'm not sure that it supports complex arithmetic. I also looked at ML from trilinos, however the only work around would be to recast my problem as a real system (doubling the MATVEcs). Any suggestions? Thanks Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Wed Mar 23 07:58:52 2011 From: jed at 59A2.org (Jed Brown) Date: Wed, 23 Mar 2011 13:58:52 +0100 Subject: [petsc-users] AMG for PETSc In-Reply-To: References: Message-ID: On Wed, Mar 23, 2011 at 13:48, Nun ion wrote: > I am interested in using AMG for PETSc, where my matrices are complex. I > looked at the documentation for Hypre and I'm not sure that it supports > complex arithmetic. I also looked at ML from trilinos, however the only > work around would be to recast my problem as a real system (doubling the > MATVEcs). Any suggestions? Hypre and ML do not support native complex. I believe Prometheus does, but I have not built it that way and --download-prometheus does not work for complex (this might be easy to fix, I haven't tried). -------------- next part -------------- An HTML attachment was scrubbed... URL: From gdiso at ustc.edu Wed Mar 23 09:36:20 2011 From: gdiso at ustc.edu (Gong Ding) Date: Wed, 23 Mar 2011 22:36:20 +0800 (CST) Subject: [petsc-users] Howto use pARMS Message-ID: <12646685.15811300890980690.JavaMail.coremail@mail.ustc.edu> Dear all, Where can I find some examples about how to use pARMS within petsc framework? ilu is powerful preconditioner, especially for my problem. However, petsc only has a simple serial version. Hypre parallel ilu does not scalar well. I hole pARMS works fine. BTW, pastix announced it support parallel ilu (kass module), is it possible to call pastix kass in petsc? Gong Ding From austin at txcorp.com Wed Mar 23 09:48:36 2011 From: austin at txcorp.com (Travis Austin) Date: Wed, 23 Mar 2011 07:48:36 -0700 Subject: [petsc-users] AMG for PETSc In-Reply-To: References: Message-ID: <5F522E45-431F-4056-86CE-38F0CE946C99@txcorp.com> Hi Mark, AMG for complex systems is an active research area. Neither Hypre or ML support directly solving the complex systems. Both at this point would require recasting the system as a real system but, at least for Hypre, building PETSc with HYPRE with complex turned on causes problems as you cannot build Hypre with complex types. We have been exploring hooking a real build of HYPRE into a complex build of PETSC but our progress lately has stalled due to other commitments. In any case, I'd suggest the reference comparing real equivalent versus complex. MacLachlan and Oosterlee. Algebraic multigrid solvers for complex-valued matrices. SIAM Journal on Scientific Computing (2008) vol. 30 (3) pp. 1548-1571 Travis ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Travis Austin, Ph.D. Tech-X Corporation 5621 Arapahoe Ave, Suite A Boulder, CO 80303 austin at txcorp.com ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ On Mar 23, 2011, at 5:48 AM, Nun ion wrote: > Hello all, > > I am interested in using AMG for PETSc, where my matrices are complex. I looked at the documentation for Hypre and I'm not sure that it supports complex arithmetic. I also looked at ML from trilinos, however the only work around would be to recast my problem as a real system (doubling the MATVEcs). Any suggestions? > > Thanks > > Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Wed Mar 23 09:49:11 2011 From: jed at 59A2.org (Jed Brown) Date: Wed, 23 Mar 2011 15:49:11 +0100 Subject: [petsc-users] Howto use pARMS In-Reply-To: <12646685.15811300890980690.JavaMail.coremail@mail.ustc.edu> References: <12646685.15811300890980690.JavaMail.coremail@mail.ustc.edu> Message-ID: 2011/3/23 Gong Ding > ilu is powerful preconditioner, especially for my problem. > However, petsc only has a simple serial version. > Hypre parallel ilu does not scalar well. > I hole pARMS works fine. > Note that pARMS is not doing parallel ILU. To see the options, run with -pc_type parms -help | grep parms This one is especially relevant: -pc_parms_global (choose one of) RAS SCHUR BJ (PCPARMSSetGlobal) note that SCHUR is nonlinear by default so you should use -ksp_type fgmres > BTW, pastix announced it support parallel ilu (kass module), > is it possible to call pastix kass in petsc? > That is not implemented, but someone could update the module. Note that -pc_type hypre -pc_hypre_type euclid is also a parallel ILU. -------------- next part -------------- An HTML attachment was scrubbed... URL: From xy2102 at columbia.edu Wed Mar 23 11:46:51 2011 From: xy2102 at columbia.edu (Rebecca Xuefei Yuan) Date: Wed, 23 Mar 2011 12:46:51 -0400 Subject: [petsc-users] Can I use the direct solver without providing the analytical Jacobian? Message-ID: <20110323124651.eb4r8kpe880800g8@cubmail.cc.columbia.edu> Hi, If the nonlinear system is evaluated at F(u) = 0 in "FormFunction()", but the symbolic Jacobian is not provided. Is there any options that I can call to solve the corresponding linear system by direct solver? Thanks very much! Best regards, Rebecca From knepley at gmail.com Wed Mar 23 11:48:26 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 23 Mar 2011 11:48:26 -0500 Subject: [petsc-users] Can I use the direct solver without providing the analytical Jacobian? In-Reply-To: <20110323124651.eb4r8kpe880800g8@cubmail.cc.columbia.edu> References: <20110323124651.eb4r8kpe880800g8@cubmail.cc.columbia.edu> Message-ID: On Wed, Mar 23, 2011 at 11:46 AM, Rebecca Xuefei Yuan wrote: > Hi, > > If the nonlinear system is evaluated at F(u) = 0 in "FormFunction()", but > the symbolic Jacobian is not provided. Is there any options that I can call > to solve the corresponding linear system by direct solver? > -snes_fd, which is very slow. Matt > Thanks very much! > > Best regards, > > Rebecca > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From xy2102 at columbia.edu Wed Mar 23 12:04:13 2011 From: xy2102 at columbia.edu (Rebecca Xuefei Yuan) Date: Wed, 23 Mar 2011 13:04:13 -0400 Subject: [petsc-users] Can I use the direct solver without providing the analytical Jacobian? In-Reply-To: References: <20110323124651.eb4r8kpe880800g8@cubmail.cc.columbia.edu> Message-ID: <20110323130413.4mu0w18qgwggos8o@cubmail.cc.columbia.edu> Dear Matt, Thanks very much for your reply. I add "snes_fd" in the option, but get ********************************************* 0 SNES Function norm 6.064004119408e-01 Linear solve converged due to CONVERGED_RTOL iterations 2 Nonlinear solve did not converge due to DIVERGED_FUNCTION_COUNT ********************************************* which is worse than the default Newton-Krylov solver as ********************************************* 0 SNES Function norm 6.064004119408e-01 Linear solve converged due to CONVERGED_RTOL iterations 2 1 SNES Function norm 7.917252729980e-02 Linear solve converged due to CONVERGED_RTOL iterations 2 2 SNES Function norm 6.967390244191e-03 Linear solve converged due to CONVERGED_RTOL iterations 10 3 SNES Function norm 4.863622639964e-03 Linear solve converged due to CONVERGED_RTOL iterations 5 4 SNES Function norm 3.696117297933e-03 Linear solve converged due to CONVERGED_RTOL iterations 1 5 SNES Function norm 1.755926765314e-03 Linear solve converged due to CONVERGED_RTOL iterations 4 6 SNES Function norm 1.206810772164e-03 Linear solve converged due to CONVERGED_RTOL iterations 1 7 SNES Function norm 2.339140575778e-04 Linear solve converged due to CONVERGED_RTOL iterations 7 8 SNES Function norm 1.676689222401e-04 Linear solve converged due to CONVERGED_RTOL iterations 1 9 SNES Function norm 6.032645773400e-05 Linear solve converged due to CONVERGED_RTOL iterations 5 10 SNES Function norm 4.747680518253e-05 Linear solve converged due to CONVERGED_RTOL iterations 1 11 SNES Function norm 1.650301627011e-05 Linear solve converged due to CONVERGED_RTOL iterations 2 12 SNES Function norm 6.030736960845e-06 Linear solve converged due to CONVERGED_RTOL iterations 5 13 SNES Function norm 3.278948607849e-06 Linear solve converged due to CONVERGED_RTOL iterations 5 14 SNES Function norm 2.488486141044e-06 Linear solve converged due to CONVERGED_RTOL iterations 1 15 SNES Function norm 6.689684826529e-07 Linear solve converged due to CONVERGED_RTOL iterations 3 16 SNES Function norm 2.543676285453e-07 Linear solve converged due to CONVERGED_RTOL iterations 7 17 SNES Function norm 1.671616367100e-07 Linear solve converged due to CONVERGED_RTOL iterations 1 18 SNES Function norm 7.228647551356e-08 Linear solve converged due to CONVERGED_RTOL iterations 7 19 SNES Function norm 5.812218130475e-08 Linear solve converged due to CONVERGED_RTOL iterations 1 20 SNES Function norm 2.883628165794e-08 Linear solve converged due to CONVERGED_RTOL iterations 3 21 SNES Function norm 2.291142576313e-08 Linear solve converged due to CONVERGED_RTOL iterations 2 22 SNES Function norm 2.291142573122e-08 Linear solve converged due to CONVERGED_RTOL iterations 2 23 SNES Function norm 2.291142569390e-08 Linear solve converged due to CONVERGED_RTOL iterations 2 24 SNES Function norm 2.291142565808e-08 Linear solve converged due to CONVERGED_RTOL iterations 2 25 SNES Function norm 2.291142562249e-08 Linear solve converged due to CONVERGED_RTOL iterations 2 26 SNES Function norm 2.291142558929e-08 Linear solve converged due to CONVERGED_RTOL iterations 2 27 SNES Function norm 2.291142555415e-08 Linear solve converged due to CONVERGED_RTOL iterations 2 28 SNES Function norm 2.291142554381e-08 Linear solve converged due to CONVERGED_RTOL iterations 2 29 SNES Function norm 2.291142550392e-08 Linear solve converged due to CONVERGED_RTOL iterations 2 30 SNES Function norm 2.291142549609e-08 Linear solve converged due to CONVERGED_RTOL iterations 2 31 SNES Function norm 2.291142546087e-08 Linear solve converged due to CONVERGED_RTOL iterations 2 32 SNES Function norm 2.291142544510e-08 Linear solve converged due to CONVERGED_RTOL iterations 2 33 SNES Function norm 2.291142542335e-08 Linear solve converged due to CONVERGED_RTOL iterations 2 34 SNES Function norm 2.291142541222e-08 Linear solve converged due to CONVERGED_RTOL iterations 2 35 SNES Function norm 2.291142540142e-08 Linear solve converged due to CONVERGED_RTOL iterations 2 36 SNES Function norm 2.291142539388e-08 Linear solve converged due to CONVERGED_RTOL iterations 2 37 SNES Function norm 2.291142535647e-08 Linear solve converged due to CONVERGED_RTOL iterations 2 38 SNES Function norm 2.291142532336e-08 Linear solve converged due to CONVERGED_RTOL iterations 2 39 SNES Function norm 2.291142528666e-08 Linear solve converged due to CONVERGED_RTOL iterations 2 40 SNES Function norm 2.291142526471e-08 Linear solve converged due to CONVERGED_RTOL iterations 2 41 SNES Function norm 2.291142526200e-08 Linear solve converged due to CONVERGED_RTOL iterations 2 42 SNES Function norm 2.291142526187e-08 Linear solve converged due to CONVERGED_RTOL iterations 2 43 SNES Function norm 2.291142522689e-08 Linear solve converged due to CONVERGED_RTOL iterations 2 44 SNES Function norm 2.291142518799e-08 Linear solve converged due to CONVERGED_RTOL iterations 2 45 SNES Function norm 2.291142517718e-08 Linear solve converged due to CONVERGED_RTOL iterations 2 46 SNES Function norm 2.291142517238e-08 Linear solve converged due to CONVERGED_RTOL iterations 2 47 SNES Function norm 2.291142515991e-08 Linear solve converged due to CONVERGED_RTOL iterations 2 48 SNES Function norm 2.291142512095e-08 Linear solve converged due to CONVERGED_RTOL iterations 2 49 SNES Function norm 2.291142508630e-08 Linear solve converged due to CONVERGED_RTOL iterations 2 50 SNES Function norm 2.291142507169e-08 Nonlinear solve did not converge due to DIVERGED_MAX_IT ********************************************* When I see this residual history, I would like to make some improvement on the solver, any suggestions on it? The options for the Newton-Krylov run are -pc_type asm -snes_ksp_ew true for np=1. Thanks again for your kind help! Cheers, Rebecca Quoting Matthew Knepley : > On Wed, Mar 23, 2011 at 11:46 AM, Rebecca Xuefei Yuan > wrote: > >> Hi, >> >> If the nonlinear system is evaluated at F(u) = 0 in "FormFunction()", but >> the symbolic Jacobian is not provided. Is there any options that I can call >> to solve the corresponding linear system by direct solver? >> > > -snes_fd, which is very slow. > > Matt > > >> Thanks very much! >> >> Best regards, >> >> Rebecca >> >> >> > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener > Rebecca Xuefei YUAN Department of Applied Physics and Applied Mathematics Columbia University Tel:917-399-8032 www.columbia.edu/~xy2102 From knepley at gmail.com Wed Mar 23 12:07:26 2011 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 23 Mar 2011 12:07:26 -0500 Subject: [petsc-users] Can I use the direct solver without providing the analytical Jacobian? In-Reply-To: <20110323130413.4mu0w18qgwggos8o@cubmail.cc.columbia.edu> References: <20110323124651.eb4r8kpe880800g8@cubmail.cc.columbia.edu> <20110323130413.4mu0w18qgwggos8o@cubmail.cc.columbia.edu> Message-ID: On Wed, Mar 23, 2011 at 12:04 PM, Rebecca Xuefei Yuan wrote: > Dear Matt, > > Thanks very much for your reply. > > I add "snes_fd" in the option, but get > > ********************************************* > 0 SNES Function norm 6.064004119408e-01 > Linear solve converged due to CONVERGED_RTOL iterations 2 > Nonlinear solve did not converge due to DIVERGED_FUNCTION_COUNT > ********************************************* > It looks like this FD derivative did not give any descent directions when factored. Possibly the FD Jacobian is too inaccurate, or the factorization was. These both mean your problem is very ill-conditioned. You should try to reformulate it. Matt > which is worse than the default Newton-Krylov solver as > > > > ********************************************* > 0 SNES Function norm 6.064004119408e-01 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 1 SNES Function norm 7.917252729980e-02 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 2 SNES Function norm 6.967390244191e-03 > Linear solve converged due to CONVERGED_RTOL iterations 10 > 3 SNES Function norm 4.863622639964e-03 > Linear solve converged due to CONVERGED_RTOL iterations 5 > 4 SNES Function norm 3.696117297933e-03 > Linear solve converged due to CONVERGED_RTOL iterations 1 > 5 SNES Function norm 1.755926765314e-03 > Linear solve converged due to CONVERGED_RTOL iterations 4 > 6 SNES Function norm 1.206810772164e-03 > Linear solve converged due to CONVERGED_RTOL iterations 1 > 7 SNES Function norm 2.339140575778e-04 > Linear solve converged due to CONVERGED_RTOL iterations 7 > 8 SNES Function norm 1.676689222401e-04 > Linear solve converged due to CONVERGED_RTOL iterations 1 > 9 SNES Function norm 6.032645773400e-05 > Linear solve converged due to CONVERGED_RTOL iterations 5 > 10 SNES Function norm 4.747680518253e-05 > Linear solve converged due to CONVERGED_RTOL iterations 1 > 11 SNES Function norm 1.650301627011e-05 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 12 SNES Function norm 6.030736960845e-06 > Linear solve converged due to CONVERGED_RTOL iterations 5 > 13 SNES Function norm 3.278948607849e-06 > Linear solve converged due to CONVERGED_RTOL iterations 5 > 14 SNES Function norm 2.488486141044e-06 > Linear solve converged due to CONVERGED_RTOL iterations 1 > 15 SNES Function norm 6.689684826529e-07 > Linear solve converged due to CONVERGED_RTOL iterations 3 > 16 SNES Function norm 2.543676285453e-07 > Linear solve converged due to CONVERGED_RTOL iterations 7 > 17 SNES Function norm 1.671616367100e-07 > Linear solve converged due to CONVERGED_RTOL iterations 1 > 18 SNES Function norm 7.228647551356e-08 > Linear solve converged due to CONVERGED_RTOL iterations 7 > 19 SNES Function norm 5.812218130475e-08 > Linear solve converged due to CONVERGED_RTOL iterations 1 > 20 SNES Function norm 2.883628165794e-08 > Linear solve converged due to CONVERGED_RTOL iterations 3 > 21 SNES Function norm 2.291142576313e-08 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 22 SNES Function norm 2.291142573122e-08 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 23 SNES Function norm 2.291142569390e-08 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 24 SNES Function norm 2.291142565808e-08 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 25 SNES Function norm 2.291142562249e-08 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 26 SNES Function norm 2.291142558929e-08 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 27 SNES Function norm 2.291142555415e-08 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 28 SNES Function norm 2.291142554381e-08 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 29 SNES Function norm 2.291142550392e-08 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 30 SNES Function norm 2.291142549609e-08 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 31 SNES Function norm 2.291142546087e-08 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 32 SNES Function norm 2.291142544510e-08 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 33 SNES Function norm 2.291142542335e-08 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 34 SNES Function norm 2.291142541222e-08 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 35 SNES Function norm 2.291142540142e-08 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 36 SNES Function norm 2.291142539388e-08 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 37 SNES Function norm 2.291142535647e-08 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 38 SNES Function norm 2.291142532336e-08 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 39 SNES Function norm 2.291142528666e-08 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 40 SNES Function norm 2.291142526471e-08 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 41 SNES Function norm 2.291142526200e-08 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 42 SNES Function norm 2.291142526187e-08 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 43 SNES Function norm 2.291142522689e-08 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 44 SNES Function norm 2.291142518799e-08 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 45 SNES Function norm 2.291142517718e-08 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 46 SNES Function norm 2.291142517238e-08 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 47 SNES Function norm 2.291142515991e-08 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 48 SNES Function norm 2.291142512095e-08 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 49 SNES Function norm 2.291142508630e-08 > Linear solve converged due to CONVERGED_RTOL iterations 2 > 50 SNES Function norm 2.291142507169e-08 > Nonlinear solve did not converge due to DIVERGED_MAX_IT > ********************************************* > > When I see this residual history, I would like to make some improvement on > the solver, any suggestions on it? > > The options for the Newton-Krylov run are > > -pc_type asm > -snes_ksp_ew true > > for np=1. > > Thanks again for your kind help! > > Cheers, > > Rebecca > > > > > > > > > Quoting Matthew Knepley : > > On Wed, Mar 23, 2011 at 11:46 AM, Rebecca Xuefei Yuan >> wrote: >> >> Hi, >>> >>> If the nonlinear system is evaluated at F(u) = 0 in "FormFunction()", but >>> the symbolic Jacobian is not provided. Is there any options that I can >>> call >>> to solve the corresponding linear system by direct solver? >>> >>> >> -snes_fd, which is very slow. >> >> Matt >> >> >> Thanks very much! >>> >>> Best regards, >>> >>> Rebecca >>> >>> >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments >> is infinitely more interesting than any results to which their experiments >> lead. >> -- Norbert Wiener >> >> > > > Rebecca Xuefei YUAN > Department of Applied Physics and Applied Mathematics > Columbia University > Tel:917-399-8032 > www.columbia.edu/~xy2102 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From ckontzialis at lycos.com Wed Mar 23 20:42:09 2011 From: ckontzialis at lycos.com (Kontsantinos Kontzialis) Date: Thu, 24 Mar 2011 03:42:09 +0200 Subject: [petsc-users] petsc-users Digest, Vol 27, Issue 47 In-Reply-To: References: Message-ID: <4D8AA171.8020804@lycos.com> On 03/23/2011 02:48 PM, petsc-users-request at mcs.anl.gov wrote: > Send petsc-users mailing list submissions to > petsc-users at mcs.anl.gov > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.mcs.anl.gov/mailman/listinfo/petsc-users > or, via email, send a message with subject or body 'help' to > petsc-users-request at mcs.anl.gov > > You can reach the person managing the list at > petsc-users-owner at mcs.anl.gov > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of petsc-users digest..." > > > Today's Topics: > > 1. Re: matrix free and preconditioner (Barry Smith) > 2. MatCreateMPIAIJWithSplitArrays (Alejandro Marcos Arag?n) > 3. Re: MatCreateMPIAIJWithSplitArrays (Jed Brown) > 4. Re: VecView doesn't work properly with DA global vectors > (??????? ???????) > 5. AMG for PETSc (Nun ion) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 22 Mar 2011 22:29:50 -0500 > From: Barry Smith > Subject: Re: [petsc-users] matrix free and preconditioner > To: PETSc users list > Message-ID: > Content-Type: text/plain; charset=us-ascii > > > Call MatMFFDSetFromOptions(sys.J); after the MatCreateSNESMF(). > > Your jacobian_matrix() HAS to call MatAssemblyBegin/End() on the first matrix passed into it (this one passed in will be sys.J) as well as whatever it does for the sys.P matrix. > > If this does not resolve the problem please let us know. > > Barry > > > On Mar 22, 2011, at 9:46 PM, Kontsantinos Kontzialis wrote: > >> Dear all, >> >> I'm using a matrix free formulation of the Newton-Krylov method. >> My code is running when I use a constant matrix as a preconditioner. >> However, when I want to use another matrix (problem specific) I read to the >> manual that I have to use the -snes_mf_operator option, with >> appropriately setting >> snessetjacobian function. I do that in my code as follows: >> ierr = MatCreateSNESMF(sys.snes,&sys.J); >> CHKERRQ(ierr); >> ierr = SNESSetJacobian(sys.snes, sys.J, sys.P, jacobian_matrix,&sys); >> CHKERRQ(ierr); >> I have preallocated space for matrix sys.P, but when I start running I >> get the following error: >> [1]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [1]PETSC ERROR: Null argument, when expecting valid pointer! >> [1]PETSC ERROR: Null Object: Parameter # 1! >> [1]PETSC ERROR: >> ------------------------------------------------------------------------ >> [1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 >> >> - Ignored: >> 14:26:37 CST 2010 >> [1]PETSC ERROR: See docs/changes/index.html for recent updates. >> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [1]PETSC ERROR: See docs/index.html for manual pages. >> [1]PETSC ERROR: >> ------------------------------------------------------------------------ >> [1]PETSC ERROR: ./hoac_forth on a linux-gnu named localhost by >> kontzialis Wed Mar 23 04:42:52 2011 >> [1]PETSC ERROR: Libraries linked from >> /home/kontzialis/PETSC/petsc-3.1-p7/linux-gnu-c-debug/lib >> [1]PETSC ERROR: Configure run at Mon Feb 21 14:55:24 2011 >> [1]PETSC ERROR: Configure options --with-debugging=1 --with-cc=mpicc >> --with-fc=mpif90 --with-shared=1 --with-shared-libraries --wi[0]PETSC >> ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Null argument, when expecting valid pointer! >> [0]PETSC ERROR: Null Object: Parameter # 1! >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 >> 14:26:37 CST 2010 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: ./hoac_forth on a linux-gnu named localhost by >> kontzialis Wed Mar 23 04:42:52 2011 >> [0]PETSC ERROR: Libraries linked from >> /home/kontzialis/PETSC/petsc-3.1-p7/linux-gnu-c-debug/lib >> [0]PETSC ERROR: Configure run at Mon Feb 21 14:55:24 2011 >> [0]PETSC ERROR: Configure options --with-debugging=1 --with-cc=mpicc >> --with-fc=mpif90 --with-shared=1 --with-shared-libraries >> --with-large-file-io=1 --with-precision=double --with-blacs=1 >> --download-blacs=yes --download-f-blas-lapack=yes --with-plapack=1 >> --download-plapack=yes --with-scalapack=1 --download-scalapack=yes >> --with-superlu=1 --download-superlu=yes --with-superlu_dist=1 >> --download-superlu_dist=yes --with-ml=1 --download-ml=yes >> --with-umfpack=1 --download-umfpack=yes --with-mpi=1 --download-mpich=1 >> --with-sundials=1 --download-sundials=1 --with-parmetis=1 >> --download-parmetis=1 --with-hypre=1 --download-hypre=1 >> [1]PETSC ERROR: >> ------------------------------------------------------------------------ >> [1]PETSC ERROR: PetscObjectGetComm() line 34 in src/sys/objects/gcomm.c >> [1]PETSC ERROR: VecNormBegin() line 495 in src/vec/vec/utils/comb.c >> [1]PETSC ERROR: MatMFFDCompute_WP() line 73 in src/mat/impls/mffd/wp.c >> [1]PETSC ERROR: MatMult_MFFD() line 315 in src/mat/impls/mffd/mffd.c >> [1]PETSC ERROR: MatMult() line 1899 in src/mat/interface/matrix.c >> [1]PETSC ERROR: PCApplyBAorAB() line 585 in src/ksp/pc/interface/precon.c >> [1]PETSC ERROR: Gth-large-file-io=1 --with-precision=double >> --with-blacs=1 --download-blacs=yes --download-f-blas-lapack=yes >> --with-plapack=1 --download-plapack=yes --with-scalapack=1 >> --download-scalapack=yes --with-superlu=1 --download-superlu=yes >> --with-superlu_dist=1 --download-superlu_dist=yes --with-ml=1 >> --download-ml=yes --with-umfpack=1 --download-umfpack=yes --with-mpi=1 >> --download-mpich=1 --with-sundials=1 --download-sundials=1 >> --with-parmetis=1 --download-parmetis=1 --with-hypre=1 --download-hypre=1 >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: PetscObjectGetComm() line 34 in src/sys/objects/gcomm.c >> [0]PETSC ERROR: VecNormBegin() line 495 in src/vec/vec/utils/comb.c >> [0]PETSC ERROR: MatMFFDCompute_WP() line 73 in src/mat/impls/mffd/wp.c >> [0]PETSC ERROR: MatMult_MFFD() line 315 in src/mat/impls/mffd/mffd.c >> [0]PETSC ERROR: MatMult() line 1899 in src/mat/interface/matrix.c >> [0]PETSC ERROR: PCApplyBAorAB() line 585 in src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: GMREScycle() line 161 in src/ksp/ksp/impls/gmres/gmres.c >> [1]PETSC ERROR: KSPSolve_GMRES() line 241 in src/ksp/ksp/impls/gmres/gmres.c >> [1]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c >> [1]PETSC ERROR: SNES_KSPSolve() line 2944 in src/snes/interface/snes.c >> [1]PETSC ERROR: SNESSolve_LS() line 191 in src/snes/impls/ls/ls.c >> [1]PETSC ERROR: SNESSolve() line 2255 in src/snes/interface/snes.c >> [1]PETSC ERROR: User provided function() line 33 in >> "unknowndirectory/"../crank_nickolson.c >> [1]PETSC ERROR: User provided function() line 84 in >> "unknowndirectory/"../implicit_time.c >> [1]PETSC ERROR: User provided function() line 1158 in >> "unknowndirectory/"../hoac.c >> MREScycle() line 161 in src/ksp/ksp/impls/gmres/gmres.c >> [0]PETSC ERROR: KSPSolve_GMRES() line 241 in src/ksp/ksp/impls/gmres/gmres.c >> [0]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: SNES_KSPSolve() line 2944 in src/snes/interface/snes.c >> [0]PETSC ERROR: SNESSolve_LS() line 191 in src/snes/impls/ls/ls.c >> [0]PETSC ERROR: SNESSolve() line 2255 in src/snes/interface/snes.c >> [0]PETSC ERROR: User provided function() line 33 in >> "unknowndirectory/"../crank_nickolson.c >> [0]PETSC ERROR: User provided function() line 84 in >> "unknowndirectory/"../implicit_time.c >> [0]PETSC ERROR: User provided function() line 1158 in >> "unknowndirectory/"../hoac.c >> >> Please help. >> >> Costas >> >> >> - Done. >> >> >> >> matrix free and preconditioner.eml >> Subject: matrix free and preconditioner >> From: Kontsantinos Kontzialis >> Date: Wed, 23 Mar 2011 04:43:43 +0200 >> To: petsc-users-request at mcs.anl.gov >> Dear Petsc team, >> >> I'm using a matrix free formulation of the Newton-Krylov method. >> My code is running when I use a constant matrix as a preconditioner. >> However, when I want to use another matrix (problem specific) I read to the >> manual that I have to use the -snes_mf_operator option, with appropriately setting >> snessetjacobian function. I do that in my code as follows: >> >> ierr = MatCreateSNESMF(sys.snes,&sys.J); >> CHKERRQ(ierr); >> >> ierr = SNESSetJacobian(sys.snes, sys.J, sys.P, jacobian_matrix,&sys); >> CHKERRQ(ierr); >> >> I have preallocated space for matrix sys.P, but when I start running I get the following error: >> >> [1]PETSC ERROR: --------------------- Error Message ------------------------------------ >> [1]PETSC ERROR: Null argument, when expecting valid pointer! >> [1]PETSC ERROR: Null Object: Parameter # 1! >> [1]PETSC ERROR: ------------------------------------------------------------------------ >> [1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 14:26:37 CST 2010 >> [1]PETSC ERROR: See docs/changes/index.html for recent updates. >> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [1]PETSC ERROR: See docs/index.html for manual pages. >> [1]PETSC ERROR: ------------------------------------------------------------------------ >> [1]PETSC ERROR: ./hoac_forth on a linux-gnu named localhost by kontzialis Wed Mar 23 04:42:52 2011 >> [1]PETSC ERROR: Libraries linked from /home/kontzialis/PETSC/petsc-3.1-p7/linux-gnu-c-debug/lib >> [1]PETSC ERROR: Configure run at Mon Feb 21 14:55:24 2011 >> [1]PETSC ERROR: Configure options --with-debugging=1 --with-cc=mpicc --with-fc=mpif90 --with-shared=1 --with-shared-libraries --wi[0]PETSC ERROR: --------------------- Error Message ------------------------------------ >> [0]PETSC ERROR: Null argument, when expecting valid pointer! >> [0]PETSC ERROR: Null Object: Parameter # 1! >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 14:26:37 CST 2010 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: ./hoac_forth on a linux-gnu named localhost by kontzialis Wed Mar 23 04:42:52 2011 >> [0]PETSC ERROR: Libraries linked from /home/kontzialis/PETSC/petsc-3.1-p7/linux-gnu-c-debug/lib >> [0]PETSC ERROR: Configure run at Mon Feb 21 14:55:24 2011 >> [0]PETSC ERROR: Configure options --with-debugging=1 --with-cc=mpicc --with-fc=mpif90 --with-shared=1 --with-shared-libraries --with-large-file-io=1 --with-precision=double --with-blacs=1 --download-blacs=yes --download-f-blas-lapack=yes --with-plapack=1 --download-plapack=yes --with-scalapack=1 --download-scalapack=yes --with-superlu=1 --download-superlu=yes --with-superlu_dist=1 --download-superlu_dist=yes --with-ml=1 --download-ml=yes --with-umfpack=1 --download-umfpack=yes --with-mpi=1 --download-mpich=1 --with-sundials=1 --download-sundials=1 --with-parmetis=1 --download-parmetis=1 --with-hypre=1 --download-hypre=1 >> [1]PETSC ERROR: ------------------------------------------------------------------------ >> [1]PETSC ERROR: PetscObjectGetComm() line 34 in src/sys/objects/gcomm.c >> [1]PETSC ERROR: VecNormBegin() line 495 in src/vec/vec/utils/comb.c >> [1]PETSC ERROR: MatMFFDCompute_WP() line 73 in src/mat/impls/mffd/wp.c >> [1]PETSC ERROR: MatMult_MFFD() line 315 in src/mat/impls/mffd/mffd.c >> [1]PETSC ERROR: MatMult() line 1899 in src/mat/interface/matrix.c >> [1]PETSC ERROR: PCApplyBAorAB() line 585 in src/ksp/pc/interface/precon.c >> [1]PETSC ERROR: Gth-large-file-io=1 --with-precision=double --with-blacs=1 --download-blacs=yes --download-f-blas-lapack=yes --with-plapack=1 --download-plapack=yes --with-scalapack=1 --download-scalapack=yes --with-superlu=1 --download-superlu=yes --with-superlu_dist=1 --download-superlu_dist=yes --with-ml=1 --download-ml=yes --with-umfpack=1 --download-umfpack=yes --with-mpi=1 --download-mpich=1 --with-sundials=1 --download-sundials=1 --with-parmetis=1 --download-parmetis=1 --with-hypre=1 --download-hypre=1 >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: PetscObjectGetComm() line 34 in src/sys/objects/gcomm.c >> [0]PETSC ERROR: VecNormBegin() line 495 in src/vec/vec/utils/comb.c >> [0]PETSC ERROR: MatMFFDCompute_WP() line 73 in src/mat/impls/mffd/wp.c >> [0]PETSC ERROR: MatMult_MFFD() line 315 in src/mat/impls/mffd/mffd.c >> [0]PETSC ERROR: MatMult() line 1899 in src/mat/interface/matrix.c >> [0]PETSC ERROR: PCApplyBAorAB() line 585 in src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: GMREScycle() line 161 in src/ksp/ksp/impls/gmres/gmres.c >> [1]PETSC ERROR: KSPSolve_GMRES() line 241 in src/ksp/ksp/impls/gmres/gmres.c >> [1]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c >> [1]PETSC ERROR: SNES_KSPSolve() line 2944 in src/snes/interface/snes.c >> [1]PETSC ERROR: SNESSolve_LS() line 191 in src/snes/impls/ls/ls.c >> [1]PETSC ERROR: SNESSolve() line 2255 in src/snes/interface/snes.c >> [1]PETSC ERROR: User provided function() line 33 in "unknowndirectory/"../crank_nickolson.c >> [1]PETSC ERROR: User provided function() line 84 in "unknowndirectory/"../implicit_time.c >> [1]PETSC ERROR: User provided function() line 1158 in "unknowndirectory/"../hoac.c >> MREScycle() line 161 in src/ksp/ksp/impls/gmres/gmres.c >> [0]PETSC ERROR: KSPSolve_GMRES() line 241 in src/ksp/ksp/impls/gmres/gmres.c >> [0]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: SNES_KSPSolve() line 2944 in src/snes/interface/snes.c >> [0]PETSC ERROR: SNESSolve_LS() line 191 in src/snes/impls/ls/ls.c >> [0]PETSC ERROR: SNESSolve() line 2255 in src/snes/interface/snes.c >> [0]PETSC ERROR: User provided function() line 33 in "unknowndirectory/"../crank_nickolson.c >> [0]PETSC ERROR: User provided function() line 84 in "unknowndirectory/"../implicit_time.c >> [0]PETSC ERROR: User provided function() line 1158 in "unknowndirectory/"../hoac.c >> >> Please help. >> >> Costas > > > ------------------------------ > > Message: 2 > Date: Wed, 23 Mar 2011 10:59:11 +0100 > From: Alejandro Marcos Arag?n > Subject: [petsc-users] MatCreateMPIAIJWithSplitArrays > To: petsc-users at mcs.anl.gov > Message-ID: > Content-Type: text/plain; charset=iso-8859-1 > > Hi PETSc users, > > My code uses a custom sparse matrix format that uses a hashed container for the entries of the matrix. The interface can then create a matrix in compressed row (or compressed column) storage format. That is, it can give me the three arrays needed to represent the sparse matrix in these formats. I would like to use the PETSc parallel solver, so then I thought that it would be good to try the MatCreateMPIAIJWithSplitArrays function so that I don't have to copy the values to the PETSc matrix again. > > Now my question to you is that I really don't get the point of having a diagonal and off-diagonal blocks of the sparse matrix. In the compressed row storage format, there is no distinction between these two blocks. Besides, I don't think there is a clear way to determine which is the boundary between these two blocks. Can someone point me how I should use this function, or if there is a better function that can take the three arrays that I have at this point? > > Also, since the sparse matrix in each process is the result of a finite element assembly routine, some rows are overlapped among the processes (there are several finite element nodes shared among the processes). At this point using the MatSetValues with the ADD_VALUES flag works fine, but I want to make sure that if I use the MatCreateMPIAIJWithSplitArrays (where I need to set the number of local rows) I can still get this behavior. In other words, if I sum the number of local rows in each process, I get a total number of rows that is greater than the number of global rows because of superposition. > > Thank you all, > > Alejandro M. Arag?n, Ph.D. > > ------------------------------ > > Message: 3 > Date: Wed, 23 Mar 2011 12:34:36 +0100 > From: Jed Brown > Subject: Re: [petsc-users] MatCreateMPIAIJWithSplitArrays > To: PETSc users list > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > 2011/3/23 Alejandro Marcos Arag?n > >> My code uses a custom sparse matrix format that uses a hashed container for >> the entries of the matrix. The interface can then create a matrix in >> compressed row (or compressed column) storage format. That is, it can give >> me the three arrays needed to represent the sparse matrix in these formats. >> I would like to use the PETSc parallel solver, so then I thought that it >> would be good to try the MatCreateMPIAIJWithSplitArrays function so that I >> don't have to copy the values to the PETSc matrix again. >> > Have your hashed format generate CSR rows and insert those one row at a time > using MatSetValues(). It will be very fast and avoid the copy of using > MatCreateMPIAIJWithArrays. > > >> Now my question to you is that I really don't get the point of having a >> diagonal and off-diagonal blocks of the sparse matrix. >> > Storing the diagonal and off-diagonal part separately makes MatMult more > tolerant to network latency and permits block Jacobi preconditioning without > needing to copy out the local blocks. > > >> In the compressed row storage format, there is no distinction between these >> two blocks. Besides, I don't think there is a clear way to determine which >> is the boundary between these two blocks. Can someone point me how I should >> use this function, or if there is a better function that can take the three >> arrays that I have at this point? >> >> Also, since the sparse matrix in each process is the result of a finite >> element assembly routine, some rows are overlapped among the processes >> (there are several finite element nodes shared among the processes). At this >> point using the MatSetValues with the ADD_VALUES flag works fine, but I want >> to make sure that if I use the MatCreateMPIAIJWithSplitArrays (where I need >> to set the number of local rows) I can still get this behavior. In other >> words, if I sum the number of local rows in each process, I get a total >> number of rows that is greater than the number of global rows because of >> superposition. >> > Just use MatSetValues(), it will take care of parallel assembly. > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > Message: 4 > Date: Wed, 23 Mar 2011 15:10:18 +0300 > From: ??????? ??????? > Subject: Re: [petsc-users] VecView doesn't work properly with DA > global vectors > To: Jed Brown > Cc: PETSc users list > Message-ID: > > Content-Type: text/plain; charset="koi8-r" > > Thank You, Jed! I'll follow your advice. Now im having some problems with > using nondefault viewers, but i think i can cope with it. Thanks > > 2011/3/22 Jed Brown > >> On Tue, Mar 22, 2011 at 10:41, ??????? ??????? wrote: >> >>> Thank You for you answer! Yes now I can see from the output, that VecView >>> uses natural ordering, but its also very importaint to understand where (on >>> which processor) elements are finaly stored. If I got it in a right way, >>> VecView uses natural ordering and gives the WRONG information about actual >>> distribution of memory among processors. >> >> The natural ordering is different from the PETSc ordering. If you want to >> view the Vec in the order it is stored in, use >> >> PetscViewerPushFormat(viewer,PETSC_VIEWER_NATIVE); >> VecView(X,viewer); >> PetscViewerPopFormat(viewer); >> > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > Message: 5 > Date: Wed, 23 Mar 2011 07:48:45 -0500 > From: Nun ion > Subject: [petsc-users] AMG for PETSc > To: petsc-users at mcs.anl.gov > Message-ID: > > Content-Type: text/plain; charset="iso-8859-1" > > Hello all, > > I am interested in using AMG for PETSc, where my matrices are complex. I > looked at the documentation for Hypre and I'm not sure that it supports > complex arithmetic. I also looked at ML from trilinos, however the only > work around would be to recast my problem as a real system (doubling the > MATVEcs). Any suggestions? > > Thanks > > Mark > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > _______________________________________________ > petsc-users mailing list > petsc-users at mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/petsc-users > > > End of petsc-users Digest, Vol 27, Issue 47 > ******************************************* > Barry, It works now. Thank you! Costas From bsmith at mcs.anl.gov Wed Mar 23 20:47:52 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 23 Mar 2011 20:47:52 -0500 Subject: [petsc-users] petsc-users Digest, Vol 27, Issue 47 In-Reply-To: <4D8AA171.8020804@lycos.com> References: <4D8AA171.8020804@lycos.com> Message-ID: <124245FA-0913-4C89-AB51-FAFB64D8F36C@mcs.anl.gov> I checked petsc-dev and it now prints out a useful error message in this case indicating the solution. Barry On Mar 23, 2011, at 8:42 PM, Kontsantinos Kontzialis wrote: > On 03/23/2011 02:48 PM, petsc-users-request at mcs.anl.gov wrote: >> Send petsc-users mailing list submissions to >> petsc-users at mcs.anl.gov >> >> To subscribe or unsubscribe via the World Wide Web, visit >> https://lists.mcs.anl.gov/mailman/listinfo/petsc-users >> or, via email, send a message with subject or body 'help' to >> petsc-users-request at mcs.anl.gov >> >> You can reach the person managing the list at >> petsc-users-owner at mcs.anl.gov >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of petsc-users digest..." >> >> >> Today's Topics: >> >> 1. Re: matrix free and preconditioner (Barry Smith) >> 2. MatCreateMPIAIJWithSplitArrays (Alejandro Marcos Arag?n) >> 3. Re: MatCreateMPIAIJWithSplitArrays (Jed Brown) >> 4. Re: VecView doesn't work properly with DA global vectors >> (??????? ???????) >> 5. AMG for PETSc (Nun ion) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Tue, 22 Mar 2011 22:29:50 -0500 >> From: Barry Smith >> Subject: Re: [petsc-users] matrix free and preconditioner >> To: PETSc users list >> Message-ID: >> Content-Type: text/plain; charset=us-ascii >> >> >> Call MatMFFDSetFromOptions(sys.J); after the MatCreateSNESMF(). >> >> Your jacobian_matrix() HAS to call MatAssemblyBegin/End() on the first matrix passed into it (this one passed in will be sys.J) as well as whatever it does for the sys.P matrix. >> >> If this does not resolve the problem please let us know. >> >> Barry >> >> >> On Mar 22, 2011, at 9:46 PM, Kontsantinos Kontzialis wrote: >> >>> Dear all, >>> >>> I'm using a matrix free formulation of the Newton-Krylov method. >>> My code is running when I use a constant matrix as a preconditioner. >>> However, when I want to use another matrix (problem specific) I read to the >>> manual that I have to use the -snes_mf_operator option, with >>> appropriately setting >>> snessetjacobian function. I do that in my code as follows: >>> ierr = MatCreateSNESMF(sys.snes,&sys.J); >>> CHKERRQ(ierr); >>> ierr = SNESSetJacobian(sys.snes, sys.J, sys.P, jacobian_matrix,&sys); >>> CHKERRQ(ierr); >>> I have preallocated space for matrix sys.P, but when I start running I >>> get the following error: >>> [1]PETSC ERROR: --------------------- Error Message >>> ------------------------------------ >>> [1]PETSC ERROR: Null argument, when expecting valid pointer! >>> [1]PETSC ERROR: Null Object: Parameter # 1! >>> [1]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 >>> >>> - Ignored: >>> 14:26:37 CST 2010 >>> [1]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [1]PETSC ERROR: See docs/index.html for manual pages. >>> [1]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [1]PETSC ERROR: ./hoac_forth on a linux-gnu named localhost by >>> kontzialis Wed Mar 23 04:42:52 2011 >>> [1]PETSC ERROR: Libraries linked from >>> /home/kontzialis/PETSC/petsc-3.1-p7/linux-gnu-c-debug/lib >>> [1]PETSC ERROR: Configure run at Mon Feb 21 14:55:24 2011 >>> [1]PETSC ERROR: Configure options --with-debugging=1 --with-cc=mpicc >>> --with-fc=mpif90 --with-shared=1 --with-shared-libraries --wi[0]PETSC >>> ERROR: --------------------- Error Message >>> ------------------------------------ >>> [0]PETSC ERROR: Null argument, when expecting valid pointer! >>> [0]PETSC ERROR: Null Object: Parameter # 1! >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 >>> 14:26:37 CST 2010 >>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [0]PETSC ERROR: See docs/index.html for manual pages. >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: ./hoac_forth on a linux-gnu named localhost by >>> kontzialis Wed Mar 23 04:42:52 2011 >>> [0]PETSC ERROR: Libraries linked from >>> /home/kontzialis/PETSC/petsc-3.1-p7/linux-gnu-c-debug/lib >>> [0]PETSC ERROR: Configure run at Mon Feb 21 14:55:24 2011 >>> [0]PETSC ERROR: Configure options --with-debugging=1 --with-cc=mpicc >>> --with-fc=mpif90 --with-shared=1 --with-shared-libraries >>> --with-large-file-io=1 --with-precision=double --with-blacs=1 >>> --download-blacs=yes --download-f-blas-lapack=yes --with-plapack=1 >>> --download-plapack=yes --with-scalapack=1 --download-scalapack=yes >>> --with-superlu=1 --download-superlu=yes --with-superlu_dist=1 >>> --download-superlu_dist=yes --with-ml=1 --download-ml=yes >>> --with-umfpack=1 --download-umfpack=yes --with-mpi=1 --download-mpich=1 >>> --with-sundials=1 --download-sundials=1 --with-parmetis=1 >>> --download-parmetis=1 --with-hypre=1 --download-hypre=1 >>> [1]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [1]PETSC ERROR: PetscObjectGetComm() line 34 in src/sys/objects/gcomm.c >>> [1]PETSC ERROR: VecNormBegin() line 495 in src/vec/vec/utils/comb.c >>> [1]PETSC ERROR: MatMFFDCompute_WP() line 73 in src/mat/impls/mffd/wp.c >>> [1]PETSC ERROR: MatMult_MFFD() line 315 in src/mat/impls/mffd/mffd.c >>> [1]PETSC ERROR: MatMult() line 1899 in src/mat/interface/matrix.c >>> [1]PETSC ERROR: PCApplyBAorAB() line 585 in src/ksp/pc/interface/precon.c >>> [1]PETSC ERROR: Gth-large-file-io=1 --with-precision=double >>> --with-blacs=1 --download-blacs=yes --download-f-blas-lapack=yes >>> --with-plapack=1 --download-plapack=yes --with-scalapack=1 >>> --download-scalapack=yes --with-superlu=1 --download-superlu=yes >>> --with-superlu_dist=1 --download-superlu_dist=yes --with-ml=1 >>> --download-ml=yes --with-umfpack=1 --download-umfpack=yes --with-mpi=1 >>> --download-mpich=1 --with-sundials=1 --download-sundials=1 >>> --with-parmetis=1 --download-parmetis=1 --with-hypre=1 --download-hypre=1 >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: PetscObjectGetComm() line 34 in src/sys/objects/gcomm.c >>> [0]PETSC ERROR: VecNormBegin() line 495 in src/vec/vec/utils/comb.c >>> [0]PETSC ERROR: MatMFFDCompute_WP() line 73 in src/mat/impls/mffd/wp.c >>> [0]PETSC ERROR: MatMult_MFFD() line 315 in src/mat/impls/mffd/mffd.c >>> [0]PETSC ERROR: MatMult() line 1899 in src/mat/interface/matrix.c >>> [0]PETSC ERROR: PCApplyBAorAB() line 585 in src/ksp/pc/interface/precon.c >>> [0]PETSC ERROR: GMREScycle() line 161 in src/ksp/ksp/impls/gmres/gmres.c >>> [1]PETSC ERROR: KSPSolve_GMRES() line 241 in src/ksp/ksp/impls/gmres/gmres.c >>> [1]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c >>> [1]PETSC ERROR: SNES_KSPSolve() line 2944 in src/snes/interface/snes.c >>> [1]PETSC ERROR: SNESSolve_LS() line 191 in src/snes/impls/ls/ls.c >>> [1]PETSC ERROR: SNESSolve() line 2255 in src/snes/interface/snes.c >>> [1]PETSC ERROR: User provided function() line 33 in >>> "unknowndirectory/"../crank_nickolson.c >>> [1]PETSC ERROR: User provided function() line 84 in >>> "unknowndirectory/"../implicit_time.c >>> [1]PETSC ERROR: User provided function() line 1158 in >>> "unknowndirectory/"../hoac.c >>> MREScycle() line 161 in src/ksp/ksp/impls/gmres/gmres.c >>> [0]PETSC ERROR: KSPSolve_GMRES() line 241 in src/ksp/ksp/impls/gmres/gmres.c >>> [0]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c >>> [0]PETSC ERROR: SNES_KSPSolve() line 2944 in src/snes/interface/snes.c >>> [0]PETSC ERROR: SNESSolve_LS() line 191 in src/snes/impls/ls/ls.c >>> [0]PETSC ERROR: SNESSolve() line 2255 in src/snes/interface/snes.c >>> [0]PETSC ERROR: User provided function() line 33 in >>> "unknowndirectory/"../crank_nickolson.c >>> [0]PETSC ERROR: User provided function() line 84 in >>> "unknowndirectory/"../implicit_time.c >>> [0]PETSC ERROR: User provided function() line 1158 in >>> "unknowndirectory/"../hoac.c >>> >>> Please help. >>> >>> Costas >>> >>> >>> - Done. >>> >>> >>> >>> matrix free and preconditioner.eml >>> Subject: matrix free and preconditioner >>> From: Kontsantinos Kontzialis >>> Date: Wed, 23 Mar 2011 04:43:43 +0200 >>> To: petsc-users-request at mcs.anl.gov >>> Dear Petsc team, >>> >>> I'm using a matrix free formulation of the Newton-Krylov method. >>> My code is running when I use a constant matrix as a preconditioner. >>> However, when I want to use another matrix (problem specific) I read to the >>> manual that I have to use the -snes_mf_operator option, with appropriately setting >>> snessetjacobian function. I do that in my code as follows: >>> >>> ierr = MatCreateSNESMF(sys.snes,&sys.J); >>> CHKERRQ(ierr); >>> >>> ierr = SNESSetJacobian(sys.snes, sys.J, sys.P, jacobian_matrix,&sys); >>> CHKERRQ(ierr); >>> >>> I have preallocated space for matrix sys.P, but when I start running I get the following error: >>> >>> [1]PETSC ERROR: --------------------- Error Message ------------------------------------ >>> [1]PETSC ERROR: Null argument, when expecting valid pointer! >>> [1]PETSC ERROR: Null Object: Parameter # 1! >>> [1]PETSC ERROR: ------------------------------------------------------------------------ >>> [1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 14:26:37 CST 2010 >>> [1]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [1]PETSC ERROR: See docs/index.html for manual pages. >>> [1]PETSC ERROR: ------------------------------------------------------------------------ >>> [1]PETSC ERROR: ./hoac_forth on a linux-gnu named localhost by kontzialis Wed Mar 23 04:42:52 2011 >>> [1]PETSC ERROR: Libraries linked from /home/kontzialis/PETSC/petsc-3.1-p7/linux-gnu-c-debug/lib >>> [1]PETSC ERROR: Configure run at Mon Feb 21 14:55:24 2011 >>> [1]PETSC ERROR: Configure options --with-debugging=1 --with-cc=mpicc --with-fc=mpif90 --with-shared=1 --with-shared-libraries --wi[0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>> [0]PETSC ERROR: Null argument, when expecting valid pointer! >>> [0]PETSC ERROR: Null Object: Parameter # 1! >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 14:26:37 CST 2010 >>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [0]PETSC ERROR: See docs/index.html for manual pages. >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: ./hoac_forth on a linux-gnu named localhost by kontzialis Wed Mar 23 04:42:52 2011 >>> [0]PETSC ERROR: Libraries linked from /home/kontzialis/PETSC/petsc-3.1-p7/linux-gnu-c-debug/lib >>> [0]PETSC ERROR: Configure run at Mon Feb 21 14:55:24 2011 >>> [0]PETSC ERROR: Configure options --with-debugging=1 --with-cc=mpicc --with-fc=mpif90 --with-shared=1 --with-shared-libraries --with-large-file-io=1 --with-precision=double --with-blacs=1 --download-blacs=yes --download-f-blas-lapack=yes --with-plapack=1 --download-plapack=yes --with-scalapack=1 --download-scalapack=yes --with-superlu=1 --download-superlu=yes --with-superlu_dist=1 --download-superlu_dist=yes --with-ml=1 --download-ml=yes --with-umfpack=1 --download-umfpack=yes --with-mpi=1 --download-mpich=1 --with-sundials=1 --download-sundials=1 --with-parmetis=1 --download-parmetis=1 --with-hypre=1 --download-hypre=1 >>> [1]PETSC ERROR: ------------------------------------------------------------------------ >>> [1]PETSC ERROR: PetscObjectGetComm() line 34 in src/sys/objects/gcomm.c >>> [1]PETSC ERROR: VecNormBegin() line 495 in src/vec/vec/utils/comb.c >>> [1]PETSC ERROR: MatMFFDCompute_WP() line 73 in src/mat/impls/mffd/wp.c >>> [1]PETSC ERROR: MatMult_MFFD() line 315 in src/mat/impls/mffd/mffd.c >>> [1]PETSC ERROR: MatMult() line 1899 in src/mat/interface/matrix.c >>> [1]PETSC ERROR: PCApplyBAorAB() line 585 in src/ksp/pc/interface/precon.c >>> [1]PETSC ERROR: Gth-large-file-io=1 --with-precision=double --with-blacs=1 --download-blacs=yes --download-f-blas-lapack=yes --with-plapack=1 --download-plapack=yes --with-scalapack=1 --download-scalapack=yes --with-superlu=1 --download-superlu=yes --with-superlu_dist=1 --download-superlu_dist=yes --with-ml=1 --download-ml=yes --with-umfpack=1 --download-umfpack=yes --with-mpi=1 --download-mpich=1 --with-sundials=1 --download-sundials=1 --with-parmetis=1 --download-parmetis=1 --with-hypre=1 --download-hypre=1 >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: PetscObjectGetComm() line 34 in src/sys/objects/gcomm.c >>> [0]PETSC ERROR: VecNormBegin() line 495 in src/vec/vec/utils/comb.c >>> [0]PETSC ERROR: MatMFFDCompute_WP() line 73 in src/mat/impls/mffd/wp.c >>> [0]PETSC ERROR: MatMult_MFFD() line 315 in src/mat/impls/mffd/mffd.c >>> [0]PETSC ERROR: MatMult() line 1899 in src/mat/interface/matrix.c >>> [0]PETSC ERROR: PCApplyBAorAB() line 585 in src/ksp/pc/interface/precon.c >>> [0]PETSC ERROR: GMREScycle() line 161 in src/ksp/ksp/impls/gmres/gmres.c >>> [1]PETSC ERROR: KSPSolve_GMRES() line 241 in src/ksp/ksp/impls/gmres/gmres.c >>> [1]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c >>> [1]PETSC ERROR: SNES_KSPSolve() line 2944 in src/snes/interface/snes.c >>> [1]PETSC ERROR: SNESSolve_LS() line 191 in src/snes/impls/ls/ls.c >>> [1]PETSC ERROR: SNESSolve() line 2255 in src/snes/interface/snes.c >>> [1]PETSC ERROR: User provided function() line 33 in "unknowndirectory/"../crank_nickolson.c >>> [1]PETSC ERROR: User provided function() line 84 in "unknowndirectory/"../implicit_time.c >>> [1]PETSC ERROR: User provided function() line 1158 in "unknowndirectory/"../hoac.c >>> MREScycle() line 161 in src/ksp/ksp/impls/gmres/gmres.c >>> [0]PETSC ERROR: KSPSolve_GMRES() line 241 in src/ksp/ksp/impls/gmres/gmres.c >>> [0]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c >>> [0]PETSC ERROR: SNES_KSPSolve() line 2944 in src/snes/interface/snes.c >>> [0]PETSC ERROR: SNESSolve_LS() line 191 in src/snes/impls/ls/ls.c >>> [0]PETSC ERROR: SNESSolve() line 2255 in src/snes/interface/snes.c >>> [0]PETSC ERROR: User provided function() line 33 in "unknowndirectory/"../crank_nickolson.c >>> [0]PETSC ERROR: User provided function() line 84 in "unknowndirectory/"../implicit_time.c >>> [0]PETSC ERROR: User provided function() line 1158 in "unknowndirectory/"../hoac.c >>> >>> Please help. >>> >>> Costas >> >> >> ------------------------------ >> >> Message: 2 >> Date: Wed, 23 Mar 2011 10:59:11 +0100 >> From: Alejandro Marcos Arag?n >> Subject: [petsc-users] MatCreateMPIAIJWithSplitArrays >> To: petsc-users at mcs.anl.gov >> Message-ID: >> Content-Type: text/plain; charset=iso-8859-1 >> >> Hi PETSc users, >> >> My code uses a custom sparse matrix format that uses a hashed container for the entries of the matrix. The interface can then create a matrix in compressed row (or compressed column) storage format. That is, it can give me the three arrays needed to represent the sparse matrix in these formats. I would like to use the PETSc parallel solver, so then I thought that it would be good to try the MatCreateMPIAIJWithSplitArrays function so that I don't have to copy the values to the PETSc matrix again. >> >> Now my question to you is that I really don't get the point of having a diagonal and off-diagonal blocks of the sparse matrix. In the compressed row storage format, there is no distinction between these two blocks. Besides, I don't think there is a clear way to determine which is the boundary between these two blocks. Can someone point me how I should use this function, or if there is a better function that can take the three arrays that I have at this point? >> >> Also, since the sparse matrix in each process is the result of a finite element assembly routine, some rows are overlapped among the processes (there are several finite element nodes shared among the processes). At this point using the MatSetValues with the ADD_VALUES flag works fine, but I want to make sure that if I use the MatCreateMPIAIJWithSplitArrays (where I need to set the number of local rows) I can still get this behavior. In other words, if I sum the number of local rows in each process, I get a total number of rows that is greater than the number of global rows because of superposition. >> >> Thank you all, >> >> Alejandro M. Arag?n, Ph.D. >> >> ------------------------------ >> >> Message: 3 >> Date: Wed, 23 Mar 2011 12:34:36 +0100 >> From: Jed Brown >> Subject: Re: [petsc-users] MatCreateMPIAIJWithSplitArrays >> To: PETSc users list >> Message-ID: >> >> Content-Type: text/plain; charset="utf-8" >> >> 2011/3/23 Alejandro Marcos Arag?n >> >>> My code uses a custom sparse matrix format that uses a hashed container for >>> the entries of the matrix. The interface can then create a matrix in >>> compressed row (or compressed column) storage format. That is, it can give >>> me the three arrays needed to represent the sparse matrix in these formats. >>> I would like to use the PETSc parallel solver, so then I thought that it >>> would be good to try the MatCreateMPIAIJWithSplitArrays function so that I >>> don't have to copy the values to the PETSc matrix again. >>> >> Have your hashed format generate CSR rows and insert those one row at a time >> using MatSetValues(). It will be very fast and avoid the copy of using >> MatCreateMPIAIJWithArrays. >> >> >>> Now my question to you is that I really don't get the point of having a >>> diagonal and off-diagonal blocks of the sparse matrix. >>> >> Storing the diagonal and off-diagonal part separately makes MatMult more >> tolerant to network latency and permits block Jacobi preconditioning without >> needing to copy out the local blocks. >> >> >>> In the compressed row storage format, there is no distinction between these >>> two blocks. Besides, I don't think there is a clear way to determine which >>> is the boundary between these two blocks. Can someone point me how I should >>> use this function, or if there is a better function that can take the three >>> arrays that I have at this point? >>> >>> Also, since the sparse matrix in each process is the result of a finite >>> element assembly routine, some rows are overlapped among the processes >>> (there are several finite element nodes shared among the processes). At this >>> point using the MatSetValues with the ADD_VALUES flag works fine, but I want >>> to make sure that if I use the MatCreateMPIAIJWithSplitArrays (where I need >>> to set the number of local rows) I can still get this behavior. In other >>> words, if I sum the number of local rows in each process, I get a total >>> number of rows that is greater than the number of global rows because of >>> superposition. >>> >> Just use MatSetValues(), it will take care of parallel assembly. >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: >> >> ------------------------------ >> >> Message: 4 >> Date: Wed, 23 Mar 2011 15:10:18 +0300 >> From: ??????? ??????? >> Subject: Re: [petsc-users] VecView doesn't work properly with DA >> global vectors >> To: Jed Brown >> Cc: PETSc users list >> Message-ID: >> >> Content-Type: text/plain; charset="koi8-r" >> >> Thank You, Jed! I'll follow your advice. Now im having some problems with >> using nondefault viewers, but i think i can cope with it. Thanks >> >> 2011/3/22 Jed Brown >> >>> On Tue, Mar 22, 2011 at 10:41, ??????? ??????? wrote: >>> >>>> Thank You for you answer! Yes now I can see from the output, that VecView >>>> uses natural ordering, but its also very importaint to understand where (on >>>> which processor) elements are finaly stored. If I got it in a right way, >>>> VecView uses natural ordering and gives the WRONG information about actual >>>> distribution of memory among processors. >>> >>> The natural ordering is different from the PETSc ordering. If you want to >>> view the Vec in the order it is stored in, use >>> >>> PetscViewerPushFormat(viewer,PETSC_VIEWER_NATIVE); >>> VecView(X,viewer); >>> PetscViewerPopFormat(viewer); >>> >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: >> >> ------------------------------ >> >> Message: 5 >> Date: Wed, 23 Mar 2011 07:48:45 -0500 >> From: Nun ion >> Subject: [petsc-users] AMG for PETSc >> To: petsc-users at mcs.anl.gov >> Message-ID: >> >> Content-Type: text/plain; charset="iso-8859-1" >> >> Hello all, >> >> I am interested in using AMG for PETSc, where my matrices are complex. I >> looked at the documentation for Hypre and I'm not sure that it supports >> complex arithmetic. I also looked at ML from trilinos, however the only >> work around would be to recast my problem as a real system (doubling the >> MATVEcs). Any suggestions? >> >> Thanks >> >> Mark >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: >> >> ------------------------------ >> >> _______________________________________________ >> petsc-users mailing list >> petsc-users at mcs.anl.gov >> https://lists.mcs.anl.gov/mailman/listinfo/petsc-users >> >> >> End of petsc-users Digest, Vol 27, Issue 47 >> ******************************************* >> > Barry, > > It works now. Thank you! > > Costas From jed at 59A2.org Wed Mar 23 22:17:50 2011 From: jed at 59A2.org (Jed Brown) Date: Thu, 24 Mar 2011 06:17:50 +0300 Subject: [petsc-users] Howto use pARMS In-Reply-To: <31311199.16591300935882286.JavaMail.coremail@mail.ustc.edu> References: <12646685.15811300890980690.JavaMail.coremail@mail.ustc.edu> <31311199.16591300935882286.JavaMail.coremail@mail.ustc.edu> Message-ID: On Thu, Mar 24, 2011 at 06:04, Gong Ding wrote: > > I see. pARMS first do domain decomposition and local ilu in each domain. > Thanks. > > And how to set PC to pARMS in code instead of command line argument? > Just like any other PC: PCSetType(pc,PCPARMS); > > The scaling of hypre parallel ilu seems bad. > I think ilu from pastix can reuse many components of sparse direct solver, > such as elimination tree, supernode structure, parallel task schedule. > It may have better performance. > Possible, but ILU "stays sparse" compared to full LU that can take advantage of dense operations and more computation per communication. > > > 2011/3/23 Gong Ding > >> ilu is powerful preconditioner, especially for my problem. >> However, petsc only has a simple serial version. >> Hypre parallel ilu does not scalar well. >> I hole pARMS works fine. >> > > Note that pARMS is not doing parallel ILU. To see the options, run with > > -pc_type parms -help | grep parms > > This one is especially relevant: > > -pc_parms_global (choose one of) RAS SCHUR BJ (PCPARMSSetGlobal) > > note that SCHUR is nonlinear by default so you should use -ksp_type fgmres > > > >> BTW, pastix announced it support parallel ilu (kass module), >> is it possible to call pastix kass in petsc? >> > > That is not implemented, but someone could update the module. > > Note that -pc_type hypre -pc_hypre_type euclid is also a parallel ILU. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Wed Mar 23 22:22:15 2011 From: jed at 59A2.org (Jed Brown) Date: Thu, 24 Mar 2011 06:22:15 +0300 Subject: [petsc-users] Can I use the direct solver without providing the analytical Jacobian? In-Reply-To: References: <20110323124651.eb4r8kpe880800g8@cubmail.cc.columbia.edu> <20110323130413.4mu0w18qgwggos8o@cubmail.cc.columbia.edu> Message-ID: On Wed, Mar 23, 2011 at 20:07, Matthew Knepley wrote: > It looks like this FD derivative did not give any descent directions when > factored. Possibly > the FD Jacobian is too inaccurate, or the factorization was. These both > mean your problem > is very ill-conditioned. You should try to reformulate it. > Yes, it is often possible to reformulate to make the system better scaled. You can also run with -snes_fd_type ds which uses a more robust method of computing the step, sometimes this make a difference. If you know the sparsity pattern of the Jacobian, you can use coloring to "efficiently" assemble a sparse Jacobian using only residual evaluation. There are many examples that do this. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ckontzialis at lycos.com Thu Mar 24 00:25:29 2011 From: ckontzialis at lycos.com (Kontsantinos Kontzialis) Date: Thu, 24 Mar 2011 07:25:29 +0200 Subject: [petsc-users] how to get the differencing paramter in a matrix free formulation Message-ID: <4D8AD5C9.7040009@lycos.com> Dear all, I'm using the following function: ierr = MatMFFDGetH(sys.J, &h); CHKERRQ(ierr); to get the parameter h in the formula for the jacobian-vector product, but it returns a value of 0. But how can that be, otherwise a division by zero would result. Thank you, Costas From hung.thanh.nguyen at petrell.no Thu Mar 24 04:20:39 2011 From: hung.thanh.nguyen at petrell.no (Hung Thanh Nguyen) Date: Thu, 24 Mar 2011 10:20:39 +0100 Subject: [petsc-users] It is posible to install petcs-3.1-p7 on window-intel-mkl plattform vers. 11.1-067 Message-ID: Hi Petsc-use I've spent many hours for to install ( ./config/configure) Petcs petsc-3.1-p7 on window Intel mkl platform (Intel-Compiler-11.1-067-ia32). I try different ways to configure Petcs ( see the last email) and I got some value tips from PETSC- used. However, The configure option is still not working . Therefore, I am wondering Is it possible to run PETSc on Window Intel Mkl platform? If the answer is yes, so how and/or any where can I addressed myself for to get help. Thanks and Kind regards Hung T. Nguyen Research in Numerical Analys Norway -------------- next part -------------- An HTML attachment was scrubbed... URL: From longmin.ran at gmail.com Thu Mar 24 05:26:48 2011 From: longmin.ran at gmail.com (Longmin RAN) Date: Thu, 24 Mar 2011 11:26:48 +0100 Subject: [petsc-users] Solving a linear system with symmetrical matrix Message-ID: Hello everyone, I use superlu inside petsc to solve linear systems with symmetrical matrix. The solving time is at least twice longer than using superlu directly. Since I'm new with Petsc, I didn't fine-tune much the petsc solver, as I had done when using superlu directly. For example, in superlu there are options like diagonal pivot threshold(superlu_options_t.DiagPivotThresh), symmetric mode (superlu_options_t.SymmetricMode), and column permutation(superlu_options_t.ColPerm). Could anybody inform me about the corresponding options in petsc? Thanks in advance. BTW, I tried the following options: MatSetOption(A, MAT_SYMMETRIC, PETSC_TRUE); MatSetOption(A, MAT_SYMMETRY_ETERNAL, PETSC_TRUE); But it seems that these options weren't enough to reduce the solving time sufficiently. Best regards, - Longmin Ran From agrayver at gfz-potsdam.de Thu Mar 24 05:27:46 2011 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Thu, 24 Mar 2011 11:27:46 +0100 Subject: [petsc-users] It is posible to install petcs-3.1-p7 on window-intel-mkl plattform vers. 11.1-067 In-Reply-To: References: Message-ID: <4D8B1CA2.2010000@gfz-potsdam.de> Hi Hung, Are you able to compile (using your Intel MKL platform) and run simple programs within cygwin? On 24.03.2011 10:20, Hung Thanh Nguyen wrote: > > Hi Petsc-use > > I've spent many hours for to install ( ./config/configure) Petcs > petsc-3.1-p7 on window Intel mkl platform > (Intel-Compiler-11.1-067-ia32). I try different ways to configure > Petcs ( see the last email) and I got some value tips from PETSC- > used. However, The configure option is still not working . Therefore, > I am wondering Is it possible to run PETSc on Window Intel Mkl > platform? If the answer is yes, so how and/or any where can I > addressed myself for to get help. Thanks and > > Kind regards > > Hung T. Nguyen > > Research in Numerical Analys > > Norway > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Thu Mar 24 06:23:50 2011 From: jed at 59A2.org (Jed Brown) Date: Thu, 24 Mar 2011 14:23:50 +0300 Subject: [petsc-users] how to get the differencing paramter in a matrix free formulation In-Reply-To: <4D8AD5C9.7040009@lycos.com> References: <4D8AD5C9.7040009@lycos.com> Message-ID: On Thu, Mar 24, 2011 at 08:25, Kontsantinos Kontzialis < ckontzialis at lycos.com> wrote: > Dear all, > > I'm using the following function: > > ierr = MatMFFDGetH(sys.J, &h); > CHKERRQ(ierr); > > to get the parameter h in the formula for the jacobian-vector product, but > it returns a value of 0. But how can that be, otherwise a division by zero > would result. > "Gets the last value that was used as the differencing parameter." http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatMFFDGetH.html Sounds like you haven't done a multiply yet. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Mar 24 07:30:31 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 24 Mar 2011 07:30:31 -0500 Subject: [petsc-users] Solving a linear system with symmetrical matrix In-Reply-To: References: Message-ID: <0BC32278-A255-4684-8150-27946524051C@mcs.anl.gov> On Mar 24, 2011, at 5:26 AM, Longmin RAN wrote: > Hello everyone, > > > I use superlu inside petsc to solve linear systems with symmetrical > matrix. The solving time is at least twice longer than using superlu > directly. How much of that time is in other parts of the code, not the actual superlu solver? Run the PETSc version with -log_summary; the times given in the lines MatFactorSymbolic, MatFactorNumerical, MatSolve (or some similar names) are the amount of time in Superlu. Compare those times to the superlu standalone solution time. Did you ./configure PETSc with the option --with-debugging=0? Barry > Since I'm new with Petsc, I didn't fine-tune much the petsc > solver, as I had done when using superlu directly. For example, in > superlu there are options like diagonal pivot > threshold(superlu_options_t.DiagPivotThresh), symmetric mode > (superlu_options_t.SymmetricMode), and column > permutation(superlu_options_t.ColPerm). Could anybody inform me about > the corresponding options in petsc? Thanks in advance. > > BTW, I tried the following options: > MatSetOption(A, MAT_SYMMETRIC, PETSC_TRUE); > MatSetOption(A, MAT_SYMMETRY_ETERNAL, PETSC_TRUE); > But it seems that these options weren't enough to reduce the solving > time sufficiently. > > > Best regards, > > - Longmin Ran From bsmith at mcs.anl.gov Thu Mar 24 07:33:20 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 24 Mar 2011 07:33:20 -0500 Subject: [petsc-users] It is posible to install petcs-3.1-p7 on window-intel-mkl plattform vers. 11.1-067 In-Reply-To: References: Message-ID: <2B8C3818-042B-486B-B0A9-3DB32F91B63F@mcs.anl.gov> Don't spend hours trying random combinations of things. Instead run what you think you should work, then send the failed configure.log to petsc-maint at mcs.anl.gov We will suggest a new thing to try, run that again sending failed output to petsc-maint at mcs.anl.gov etc. But you must follow our suggestions for the "next try" or you will end up going around in circles and getting very frustrated. Barry On Mar 24, 2011, at 4:20 AM, Hung Thanh Nguyen wrote: > Hi Petsc-use > I?ve spent many hours for to install ( ./config/configure) Petcs petsc-3.1-p7 on window Intel mkl platform (Intel-Compiler-11.1-067-ia32). I try different ways to configure Petcs ( see the last email) and I got some value tips from PETSC- used. However, The configure option is still not working . Therefore, I am wondering Is it possible to run PETSc on Window Intel Mkl platform? If the answer is yes, so how and/or any where can I addressed myself for to get help. Thanks and > Kind regards > Hung T. Nguyen > Research in Numerical Analys > Norway > > From panourg at mech.upatras.gr Thu Mar 24 03:10:16 2011 From: panourg at mech.upatras.gr (panourg at mech.upatras.gr) Date: Thu, 24 Mar 2011 10:10:16 +0200 (EET) Subject: [petsc-users] Problem: Preconditioning of non-daigonally dominant matrix Message-ID: <13039.94.66.84.50.1300954216.squirrel@mail.mech.upatras.gr> Dir All I have a problem with converging of large block sparse non-diagonally dominant matrix. Which preconditioner is the best choice for these systems? When the diagonal blocks of matrix have more large values by off-diagonal(diagonally dominant) then the ILU prec gives good results but when the diagonal blocks have values smaller than off diag. then i cant take convergence with ILU or other classic preconditioner. Have anyone the same problem and what did? K.Panourgias From longmin.ran at gmail.com Thu Mar 24 09:29:08 2011 From: longmin.ran at gmail.com (Longmin RAN) Date: Thu, 24 Mar 2011 15:29:08 +0100 Subject: [petsc-users] Solving a linear system with symmetrical matrix In-Reply-To: <0BC32278-A255-4684-8150-27946524051C@mcs.anl.gov> References: <0BC32278-A255-4684-8150-27946524051C@mcs.anl.gov> Message-ID: Thanks for the tips, Barry. I didn't specify --with-debugging=0 during the configuration, I think this is the problem. I'll recompile petsc accordingly to check on this. Longmin On Thu, Mar 24, 2011 at 1:30 PM, Barry Smith wrote: > > On Mar 24, 2011, at 5:26 AM, Longmin RAN wrote: > >> Hello everyone, >> >> >> I use superlu inside petsc to solve linear systems with symmetrical >> matrix. The solving time is at least twice longer than using superlu >> directly. > > ? How much of that time is in other parts of the code, not the actual superlu solver? Run the PETSc version with -log_summary; the times given in the lines > MatFactorSymbolic, MatFactorNumerical, MatSolve (or some similar names) are the amount of time in Superlu. Compare those times to the superlu standalone solution time. > > ? ?Did you ./configure PETSc with the option --with-debugging=0? > > ? Barry > > > >> Since I'm new with Petsc, I didn't fine-tune much the petsc >> solver, as I had done when using superlu directly. For example, in >> superlu there are options like diagonal pivot >> threshold(superlu_options_t.DiagPivotThresh), symmetric mode >> (superlu_options_t.SymmetricMode), and column >> permutation(superlu_options_t.ColPerm). Could anybody inform me about >> the corresponding options in petsc? Thanks in advance. >> >> BTW, I tried the following options: >> ? ? ? MatSetOption(A, MAT_SYMMETRIC, PETSC_TRUE); >> ? ? ? MatSetOption(A, MAT_SYMMETRY_ETERNAL, PETSC_TRUE); >> But it seems that these options weren't enough to reduce the solving >> time sufficiently. >> >> >> Best regards, >> >> - Longmin Ran > > From vkuhlem at emory.edu Thu Mar 24 15:23:25 2011 From: vkuhlem at emory.edu (Verena Kuhlemann) Date: Thu, 24 Mar 2011 16:23:25 -0400 Subject: [petsc-users] PetscOptionsSetValue In-Reply-To: <13039.94.66.84.50.1300954216.squirrel@mail.mech.upatras.gr> References: <13039.94.66.84.50.1300954216.squirrel@mail.mech.upatras.gr> Message-ID: <9A0012E0-1C33-4E2C-90A3-33DE85D20575@emory.edu> Hello! I am trying to set some options with PetscOptionsSetValue. In particular, I want to set some options for the pilut preconditioner in hypre. Here is how I trying to set the factor row size: ierr = PCSetType(subpc, PCHYPRE);CHKERRQ(ierr); ierr = PCHYPRESetType(subpc,"pilut");CHKERRQ(ierr); ierr = PetscOptionsSetValue("- pc_hypre_pilut_factorrowsize","10");CHKERRQ(ierr); ierr = PCSetFromOptions(subpc);CHKERRQ(ierr); Unfortunately, this does not seem to work. I get a warning that the option -pc_hypre_pilut_factorrowsize was not used. I would be thankful for any help. Thanks, Verena From knepley at gmail.com Thu Mar 24 15:35:07 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 24 Mar 2011 15:35:07 -0500 Subject: [petsc-users] PetscOptionsSetValue In-Reply-To: <9A0012E0-1C33-4E2C-90A3-33DE85D20575@emory.edu> References: <13039.94.66.84.50.1300954216.squirrel@mail.mech.upatras.gr> <9A0012E0-1C33-4E2C-90A3-33DE85D20575@emory.edu> Message-ID: On Thu, Mar 24, 2011 at 3:23 PM, Verena Kuhlemann wrote: > Hello! > > I am trying to set some options with PetscOptionsSetValue. In particular, > I want to set some options for the pilut preconditioner in hypre. > Here is how I trying to set the factor row size: > > ierr = PCSetType(subpc, PCHYPRE);CHKERRQ(ierr); > ierr = PCHYPRESetType(subpc,"pilut");CHKERRQ(ierr); > ierr = > PetscOptionsSetValue("-pc_hypre_pilut_factorrowsize","10");CHKERRQ(ierr); > Your code suggests it should be -sub_pc_hypre_pilut_factorrowsize. Why would you use pilut on a subpc? Thanks, Matt > ierr = PCSetFromOptions(subpc);CHKERRQ(ierr); > > Unfortunately, this does not seem to work. I get a warning that the option > -pc_hypre_pilut_factorrowsize > was not used. > > I would be thankful for any help. > > Thanks, > Verena > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From gdiso at ustc.edu Fri Mar 25 01:57:05 2011 From: gdiso at ustc.edu (Gong Ding) Date: Fri, 25 Mar 2011 14:57:05 +0800 (CST) Subject: [petsc-users] compile petsc error about mmintrin.h Message-ID: <14143330.19431301036225001.JavaMail.coremail@mail.ustc.edu> On centos5 with icc version 11.1, factimpl.c produced many errors such as /usr/lib/gcc/x86_64-redhat-linux/4.1.2/include/mmintrin.h(78): error: cast to type "__m64" is not allowed return (__m64) __i; This problem exists for a long time. In the petsc version of 3.1, I just comment out #ifndef PETSC_HAVE_XMMINTRIN_H #define PETSC_HAVE_XMMINTRIN_H 1 #endif in the petscconf.h, and compile it. However, petsc-dev has other dependence on mmintrin.h, comment out PETSC_HAVE_XMMINTRIN_H will product other problems. How can I solve this problem? BTW, it seems gcc works well. Gong Ding From jed at 59A2.org Fri Mar 25 03:22:06 2011 From: jed at 59A2.org (Jed Brown) Date: Fri, 25 Mar 2011 11:22:06 +0300 Subject: [petsc-users] compile petsc error about mmintrin.h In-Reply-To: <14143330.19431301036225001.JavaMail.coremail@mail.ustc.edu> References: <14143330.19431301036225001.JavaMail.coremail@mail.ustc.edu> Message-ID: 2011/3/25 Gong Ding > On centos5 with icc version 11.1, factimpl.c produced many errors such as > > /usr/lib/gcc/x86_64-redhat-linux/4.1.2/include/mmintrin.h(78): error: cast > to type "__m64" is not allowed > return (__m64) __i; > Where does this include path come from? The Intel compiler should find its own copy of mmintrin.h, not gcc's copy. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gdiso at ustc.edu Fri Mar 25 06:11:05 2011 From: gdiso at ustc.edu (Gong Ding) Date: Fri, 25 Mar 2011 19:11:05 +0800 (CST) Subject: [petsc-users] compile petsc-dev error Message-ID: <29359373.20011301051465583.JavaMail.coremail@mail.ustc.edu> It seems pastix and parms has function name conflict. /usr/local/petsc-dev/linux-icc/lib/libpastix.a(sparRow.o): In function `cleanCS' : kass/src/sparRow.c:(.text+0x0): multiple definition of `cleanCS' /usr/local/petsc-dev/linux-icc/lib/libparms.a(sets.o):src/DDPQ/sets.c:(.text+0x6 60): first defined here ld: Warning: size of symbol `cleanCS' changed from 160 in /usr/local/petsc-dev/l inux-icc/lib/libparms.a(sets.o) to 288 in /usr/local/petsc-dev/linux-icc/lib/lib pastix.a(sparRow.o) /usr/local/petsc-dev/linux-icc/lib/libpastix.a(sparRow.o): In function `cleanCS' : kass/src/sparRow.c:(.text+0x0): multiple definition of `cleanCS' /usr/local/petsc-dev/linux-icc/lib/libparms.a(sets.o):src/DDPQ/sets.c:(.text+0x6 60): first defined here ld: Warning: size of symbol `cleanCS' changed from 160 in /usr/local/petsc-dev/l inux-icc/lib/libparms.a(sets.o) to 288 in /usr/local/petsc-dev/linux-icc/lib/lib pastix.a(sparRow.o) The function ISCreateGeneral now takes an extra parameter PetscCopyMode. Which value is compatible to previous version of ISCreateGeneral? MatZeroRows takes two extra Vec. Is it possible to keep the old API unchanged but use a new API? And what will the version of next petsc release? Is it will be 3.2 or 4.0? From bsmith at mcs.anl.gov Fri Mar 25 08:47:36 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 25 Mar 2011 08:47:36 -0500 Subject: [petsc-users] compile petsc error about mmintrin.h In-Reply-To: <14143330.19431301036225001.JavaMail.coremail@mail.ustc.edu> References: <14143330.19431301036225001.JavaMail.coremail@mail.ustc.edu> Message-ID: <47C31857-BA68-46A5-ABB7-99556FE9E22A@mcs.anl.gov> We need configure.log and make.log sent to petsc-maint at mcs.anl.gov Always error on the side of sending too much information to petsc-maint at mcs.anl.gov rather than too few. Barry On Mar 25, 2011, at 1:57 AM, Gong Ding wrote: > On centos5 with icc version 11.1, factimpl.c produced many errors such as > > /usr/lib/gcc/x86_64-redhat-linux/4.1.2/include/mmintrin.h(78): error: cast to type "__m64" is not allowed > return (__m64) __i; > > This problem exists for a long time. In the petsc version of 3.1, I just comment out > #ifndef PETSC_HAVE_XMMINTRIN_H > #define PETSC_HAVE_XMMINTRIN_H 1 > #endif > in the petscconf.h, and compile it. > > However, petsc-dev has other dependence on mmintrin.h, comment out PETSC_HAVE_XMMINTRIN_H will product other problems. > How can I solve this problem? > > > BTW, it seems gcc works well. > > Gong Ding > > > From jed at 59A2.org Fri Mar 25 08:48:52 2011 From: jed at 59A2.org (Jed Brown) Date: Fri, 25 Mar 2011 16:48:52 +0300 Subject: [petsc-users] compile petsc-dev error In-Reply-To: <29359373.20011301051465583.JavaMail.coremail@mail.ustc.edu> References: <29359373.20011301051465583.JavaMail.coremail@mail.ustc.edu> Message-ID: 2011/3/25 Gong Ding > It seems pastix and parms has function name conflict. > Indeed, this is the problem with forgetting to namespace. We can roll a version with just this symbol namespaced, it would be better if the developers of these packages would namespace their symbols. > The function ISCreateGeneral now takes an extra parameter PetscCopyMode. > Which value is compatible to previous version of ISCreateGeneral? > PETSC_COPY_VALUES > > MatZeroRows takes two extra Vec. Is it possible to keep the old API > unchanged but > use a new API? > That becomes an untenable maintenance burden (not so much in this particular case, but code changes are already required for a few other cases). You can check the PETSc version and provide a wrapper with the old API that sends NULL for the last two parameters. A language with keyword arguments would be better in this regard, but no such language is a presently viable alternative. > And what will the version of next petsc release? > Is it will be 3.2 or 4.0? > 3.2 -------------- next part -------------- An HTML attachment was scrubbed... URL: From longmin.ran at gmail.com Fri Mar 25 10:16:21 2011 From: longmin.ran at gmail.com (Longmin RAN) Date: Fri, 25 Mar 2011 16:16:21 +0100 Subject: [petsc-users] Solving a linear system with symmetrical matrix In-Reply-To: References: <0BC32278-A255-4684-8150-27946524051C@mcs.anl.gov> Message-ID: Hello again, I recompiled petsc using --with-debugging=0 but it doesn't improve the situation much. What's more, when running the program with "-info", I saw "Option SYMMETRIC ignored" and "Option SYMMETRY_ETERNAL ignored". Why are these options ignored? For superlu to specify the symmetric mode with MMD_AT_PLUS_A ordering will significantly shorten the solution time. I didn't find corresponding ordering type in petsc. Longmin On Thu, Mar 24, 2011 at 3:29 PM, Longmin RAN wrote: > Thanks for the tips, Barry. I didn't specify --with-debugging=0 during > the configuration, I think this is the problem. I'll recompile petsc > accordingly to check on this. > > > Longmin > > On Thu, Mar 24, 2011 at 1:30 PM, Barry Smith wrote: >> >> On Mar 24, 2011, at 5:26 AM, Longmin RAN wrote: >> >>> Hello everyone, >>> >>> >>> I use superlu inside petsc to solve linear systems with symmetrical >>> matrix. The solving time is at least twice longer than using superlu >>> directly. >> >> ? How much of that time is in other parts of the code, not the actual superlu solver? Run the PETSc version with -log_summary; the times given in the lines >> MatFactorSymbolic, MatFactorNumerical, MatSolve (or some similar names) are the amount of time in Superlu. Compare those times to the superlu standalone solution time. >> >> ? ?Did you ./configure PETSc with the option --with-debugging=0? >> >> ? Barry >> >> >> >>> Since I'm new with Petsc, I didn't fine-tune much the petsc >>> solver, as I had done when using superlu directly. For example, in >>> superlu there are options like diagonal pivot >>> threshold(superlu_options_t.DiagPivotThresh), symmetric mode >>> (superlu_options_t.SymmetricMode), and column >>> permutation(superlu_options_t.ColPerm). Could anybody inform me about >>> the corresponding options in petsc? Thanks in advance. >>> >>> BTW, I tried the following options: >>> ? ? ? MatSetOption(A, MAT_SYMMETRIC, PETSC_TRUE); >>> ? ? ? MatSetOption(A, MAT_SYMMETRY_ETERNAL, PETSC_TRUE); >>> But it seems that these options weren't enough to reduce the solving >>> time sufficiently. >>> >>> >>> Best regards, >>> >>> - Longmin Ran >> >> > From bsmith at mcs.anl.gov Fri Mar 25 11:08:30 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 25 Mar 2011 11:08:30 -0500 Subject: [petsc-users] Solving a linear system with symmetrical matrix In-Reply-To: References: <0BC32278-A255-4684-8150-27946524051C@mcs.anl.gov> Message-ID: Since Superlu is its own self-contained package you set its various options using the interfaces below, listed in the manual page for MATSOLVERSUPERLU + -mat_superlu_equil: Equil (None) . -mat_superlu_colperm (choose one of) NATURAL MMD_ATA MMD_AT_PLUS_A COLAMD . -mat_superlu_iterrefine (choose one of) NOREFINE SINGLE DOUBLE EXTRA . -mat_superlu_symmetricmode: SymmetricMode (None) . -mat_superlu_diagpivotthresh <1>: DiagPivotThresh (None) . -mat_superlu_pivotgrowth: PivotGrowth (None) . -mat_superlu_conditionnumber: ConditionNumber (None) . -mat_superlu_rowperm (choose one of) NOROWPERM LargeDiag . -mat_superlu_replacetinypivot: ReplaceTinyPivot (None) . -mat_superlu_printstat: PrintStat (None) . -mat_superlu_lwork <0>: size of work array in bytes used by factorization (None) . -mat_superlu_ilu_droptol <0>: ILU_DropTol (None) . -mat_superlu_ilu_filltol <0>: ILU_FillTol (None) . -mat_superlu_ilu_fillfactor <0>: ILU_FillFactor (None) . -mat_superlu_ilu_droprull <0>: ILU_DropRule (None) . -mat_superlu_ilu_norm <0>: ILU_Norm (None) - -mat_superlu_ilu_milu <0>: ILU_MILU (None) On Mar 25, 2011, at 10:16 AM, Longmin RAN wrote: > Hello again, > > I recompiled petsc using --with-debugging=0 but it doesn't improve the > situation much. What's more, when running the program with "-info", I > saw "Option SYMMETRIC ignored" and "Option SYMMETRY_ETERNAL ignored". > Why are these options ignored? For superlu to specify the symmetric > mode with MMD_AT_PLUS_A ordering will significantly shorten the > solution time. I didn't find corresponding ordering type in petsc. > > > Longmin > > On Thu, Mar 24, 2011 at 3:29 PM, Longmin RAN wrote: >> Thanks for the tips, Barry. I didn't specify --with-debugging=0 during >> the configuration, I think this is the problem. I'll recompile petsc >> accordingly to check on this. >> >> >> Longmin >> >> On Thu, Mar 24, 2011 at 1:30 PM, Barry Smith wrote: >>> >>> On Mar 24, 2011, at 5:26 AM, Longmin RAN wrote: >>> >>>> Hello everyone, >>>> >>>> >>>> I use superlu inside petsc to solve linear systems with symmetrical >>>> matrix. The solving time is at least twice longer than using superlu >>>> directly. >>> >>> How much of that time is in other parts of the code, not the actual superlu solver? Run the PETSc version with -log_summary; the times given in the lines >>> MatFactorSymbolic, MatFactorNumerical, MatSolve (or some similar names) are the amount of time in Superlu. Compare those times to the superlu standalone solution time. >>> >>> Did you ./configure PETSc with the option --with-debugging=0? >>> >>> Barry >>> >>> >>> >>>> Since I'm new with Petsc, I didn't fine-tune much the petsc >>>> solver, as I had done when using superlu directly. For example, in >>>> superlu there are options like diagonal pivot >>>> threshold(superlu_options_t.DiagPivotThresh), symmetric mode >>>> (superlu_options_t.SymmetricMode), and column >>>> permutation(superlu_options_t.ColPerm). Could anybody inform me about >>>> the corresponding options in petsc? Thanks in advance. >>>> >>>> BTW, I tried the following options: >>>> MatSetOption(A, MAT_SYMMETRIC, PETSC_TRUE); >>>> MatSetOption(A, MAT_SYMMETRY_ETERNAL, PETSC_TRUE); >>>> But it seems that these options weren't enough to reduce the solving >>>> time sufficiently. >>>> >>>> >>>> Best regards, >>>> >>>> - Longmin Ran >>> >>> >> From gdiso at ustc.edu Sun Mar 27 01:12:00 2011 From: gdiso at ustc.edu (Gong Ding) Date: Sun, 27 Mar 2011 14:12:00 +0800 (CST) Subject: [petsc-users] makefile of pastix on petsc-dev seems lack of flag -DWITH_SCOTCH Message-ID: <30411755.21791301206320013.JavaMail.coremail@mail.ustc.edu> Hi, The configure and compileing pastix with petsc-dev is ok. However, run time error occur: ERROR: Scotch ordering needs to compile PaStiX with -DWITH_SCOTCH Gong Ding From gdiso at ustc.edu Sun Mar 27 01:00:40 2011 From: gdiso at ustc.edu (Gong Ding) Date: Sun, 27 Mar 2011 14:00:40 +0800 (CST) Subject: [petsc-users] superlu ILUT preconditioner break Message-ID: <25324380.21751301205640439.JavaMail.coremail@mail.ustc.edu> Hi, I compiled latest petsc-dev with superlu and try the ILUT preconditioner. However, it raising segmentation violation error. The valgrind reported: Poisson Solver init... Using GMRES linear solver... Using SuperLU ILUT preconditioner... its 0 |residual|_2 = 6.216736e+00 linear iter = 0 ==18820== Invalid read of size 4 ==18820== at 0x269AF7F: dgsisx (dgsisx.c:692) ==18820== by 0x163E5D6: MatSolve_SuperLU_Private (superlu.c:321) ==18820== by 0x163F162: MatSolve_SuperLU (superlu.c:366) ==18820== by 0x151866A: MatSolve (matrix.c:3055) ==18820== by 0x1B56197: PCApply_ILU (ilu.c:270) ==18820== by 0x1F95DBF: PCApply (precon.c:384) ==18820== by 0x1FBFE83: KSPInitialResidual (itres.c:64) ==18820== by 0x1C6CF31: KSPSolve_GMRES (gmres.c:243) ==18820== by 0x1BEC8B5: KSPSolve (itfunc.c:426) ==18820== by 0x1D2433F: SNES_KSPSolve (snes.c:3060) ==18820== by 0x1D3DE74: SNESSolve_LS (ls.c:190) ==18820== by 0x1D1AA5B: SNESSolve (snes.c:2360) ==18820== Address 0x0 is not stack'd, malloc'd or (recently) free'd ==18820== [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] MatSolve_SuperLU_Private line 275 src/mat/impls/aij/seq/superlu/superlu.c [0]PETSC ERROR: [0] MatSolve_SuperLU line 364 src/mat/impls/aij/seq/superlu/superlu.c [0]PETSC ERROR: [0] MatSolve line 3038 src/mat/interface/matrix.c [0]PETSC ERROR: [0] PCApply_ILU line 269 src/ksp/pc/impls/factor/ilu/ilu.c [0]PETSC ERROR: [0] PCApply line 374 src/ksp/pc/interface/precon.c [0]PETSC ERROR: [0] KSPInitialResidual line 45 src/ksp/ksp/interface/itres.c [0]PETSC ERROR: [0] KSPSolve_GMRES line 224 src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: [0] KSPSolve line 332 src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: [0] SNES_KSPSolve line 3058 src/snes/interface/snes.c [0]PETSC ERROR: [0] SNESSolve_LS line 142 src/snes/impls/ls/ls.c [0]PETSC ERROR: [0] SNESSolve line 2334 src/snes/interface/snes.c --------------------- Error Message ------------------------------------ Fatal Error: at line 0 in unknown file ------------------------------------------------------------------------ The PC configureation is listed as follows: ierr = PCSetType (pc, (char*) PCILU); genius_assert(!ierr); ierr = PCFactorSetMatSolverPackage (pc, "superlu"); genius_assert(!ierr); ierr = PetscOptionsSetValue("-mat_superlu_ilu_droptol","1e-7"); genius_assert(!ierr); ierr = PetscOptionsSetValue("-mat_superlu_ilu_filltol","1e-2"); genius_assert(!ierr); ierr = PetscOptionsSetValue("-mat_superlu_ilu_fillfactor","200"); genius_assert(!ierr); Or I tested in a much simple code (attached) with command line argument /petsc_solver -fmat poisson.mat -fvec poisson.rhs -pc_type ilu -pc_factor_mat_solver_package superlu -ksp_monitor which has the same problem. Is it the problem of superlu itself or petsc interface? -------------- next part -------------- A non-text attachment was scrubbed... Name: poisson.mat Type: application/octet-stream Size: 58656 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: poisson.rhs Type: application/octet-stream Size: 5640 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: petsc_solver.c Type: text/x-csrc Size: 2298 bytes Desc: not available URL: From bsmith at mcs.anl.gov Sun Mar 27 19:16:02 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 27 Mar 2011 19:16:02 -0500 Subject: [petsc-users] makefile of pastix on petsc-dev seems lack of flag -DWITH_SCOTCH In-Reply-To: <30411755.21791301206320013.JavaMail.coremail@mail.ustc.edu> References: <30411755.21791301206320013.JavaMail.coremail@mail.ustc.edu> Message-ID: <2A307169-5C42-406A-A7E3-764290D033EB@mcs.anl.gov> Start rant: We're going to need some help with this. Between MUMPS, Pastix, Scotch, and ptscotch their complicated configurations are going to drive us insane. Just look at all the flags, strings and counter flags that have to be set correctly in config/PETSc/packages/PaStiX.py, MUMPS.py and Scotch.py to get builds that work with each other and work with each others optional features. It is a full time job to understand and keep up with this stuff and we don't have full time. They've got to simplify, simplify and simplify, this much complication is a problem. We can try to hack together the flags based on partial understanding but it will break the next time they muck with required stuff for their makefiles. I'd really like PETSc to use all these packages easily but am really frustrated with the complexity of what should be simple installs. Please feel free to forward this to the scotch, pastix and mumps developers so they can help us. End rant: Barry On Mar 27, 2011, at 1:12 AM, Gong Ding wrote: > Hi, > > The configure and compileing pastix with petsc-dev is ok. > > However, run time error occur: > ERROR: Scotch ordering needs to compile PaStiX with -DWITH_SCOTCH > > Gong Ding > From fd.kong at siat.ac.cn Mon Mar 28 01:06:13 2011 From: fd.kong at siat.ac.cn (=?ISO-8859-1?B?ZmRrb25n?=) Date: Mon, 28 Mar 2011 14:06:13 +0800 Subject: [petsc-users] Generation, refinement of the mesh (Sieve mesh) is very slow! Message-ID: Hi everyone I have developed my application based on the sieve mesh object in the Pestc. And now, I encountered some serious problems. 1. The generation of mesh takes a lot of time, run very slowly. The following code is used: double lower[2] = {-1.0, -1.0}; double upper[2] = {1.0, 1.0}; int edges[2] = {256,256}; mB = ALE::MeshBuilder::createSquareBoundary(comm, lower, upper, edges, debug); ALE::ISieveConverter::convertMesh(*mB, *meshBd, renumbering, false); ierr = PetscPrintf(PETSC_COMM_WORLD," End build convertMesh \n");CHKERRQ(ierr); ierr = MeshSetMesh(boundary, meshBd);CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD," Begin build MeshGenerate \n");CHKERRQ(ierr); ierr = MeshGenerate(boundary,interpolate, &mesh);CHKERRQ(ierr); 2. The refinement of mesh is also very slow. Th code: refinementLimit=0.0001; if (refinementLimit > 0.0) { Mesh refinedMesh; ierr = MeshRefine(mesh, refinementLimit,interpolate, &refinedMesh);CHKERRQ(ierr); ierr = MeshDestroy(mesh);CHKERRQ(ierr); mesh = refinedMesh; } 3. The distribution of mesh is also very slow. The code: if (size > 1) { Mesh parallelMesh; //ierr = DistributeMeshnew(mesh, "chao", ¶llelMesh);CHKERRQ(ierr); ierr = DistributeMeshnew(mesh, "parmetis", ¶llelMesh);CHKERRQ(ierr); ierr = MeshDestroy(mesh);CHKERRQ(ierr); mesh = parallelMesh; }. Does anyone encounter these similar problem? If anyone can help, thank you very much! And I wonder to consult which parallel mesh can work with Petsc very well, when we develop some complex problem? ------------------ Fande Kong ShenZhen Institutes of Advanced Technology Chinese Academy of Sciences -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Mar 28 01:19:50 2011 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 28 Mar 2011 01:19:50 -0500 Subject: [petsc-users] Generation, refinement of the mesh (Sieve mesh) is very slow! In-Reply-To: References: Message-ID: 1) Always send the output of -log_summary when asking a performance question 2) There are implementations that are optimized for different things. Its possible to optimize mesh handling for a cells-vertices mesh, but not if you need edges and faces generated. 3) I am out of the country. I can look at the performance when I get back. Matt On Mon, Mar 28, 2011 at 1:06 AM, fdkong wrote: > Hi everyone > I have developed my application based on the sieve mesh object in the > Pestc. And now, I encountered some serious problems. > 1. The generation of mesh takes a lot of time, run very slowly. The > following code is used: > double lower[2] = {-1.0, -1.0}; > double upper[2] = {1.0, 1.0}; > int edges[2] = {256,256}; > mB = ALE::MeshBuilder::createSquareBoundary(comm, lower, > upper, edges, debug); > ALE::ISieveConverter::convertMesh(*mB, *meshBd, renumbering, false); > ierr = PetscPrintf(PETSC_COMM_WORLD," End build convertMesh > \n");CHKERRQ(ierr); > ierr = MeshSetMesh(boundary, meshBd);CHKERRQ(ierr); > ierr = PetscPrintf(PETSC_COMM_WORLD," Begin build MeshGenerate > \n");CHKERRQ(ierr); > > ierr = MeshGenerate(boundary,interpolate, &mesh);CHKERRQ(ierr); > > 2. The refinement of mesh is also very slow. Th code: > refinementLimit=0.0001; > if (refinementLimit > 0.0) > { > Mesh refinedMesh; > > ierr = MeshRefine(mesh, refinementLimit,interpolate, > &refinedMesh);CHKERRQ(ierr); > ierr = MeshDestroy(mesh);CHKERRQ(ierr); > mesh = refinedMesh; > } > > 3. The distribution of mesh is also very slow. The code: > if (size > 1) > { > Mesh parallelMesh; > > //ierr = DistributeMeshnew(mesh, "chao", > ¶llelMesh);CHKERRQ(ierr); > ierr = DistributeMeshnew(mesh, "parmetis", > ¶llelMesh);CHKERRQ(ierr); > ierr = MeshDestroy(mesh);CHKERRQ(ierr); > mesh = parallelMesh; > }. > > Does anyone encounter these similar problem? If anyone can help, thank > you very much! > > And I wonder to consult which parallel mesh can work with Petsc very > well, when we develop some complex problem? > > > ------------------ > Fande Kong > ShenZhen Institutes of Advanced Technology > Chinese Academy of Sciences > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From fd.kong at siat.ac.cn Mon Mar 28 01:33:42 2011 From: fd.kong at siat.ac.cn (=?gbk?B?ZmRrb25n?=) Date: Mon, 28 Mar 2011 14:33:42 +0800 Subject: [petsc-users] Generation, refinement of the mesh (Sieve mesh) isvery slow! Message-ID: Thank you very much for your reply. The -log_summary is shown as follows: ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./linearElasticity on a linux-gnu named c0409 with 64 processors, by fdkong Sat Mar 26 12:44:53 2011 Using Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 14:26:37 CST 2010 Max Max/Min Avg Total Time (sec): 3.459e+02 1.00012 3.459e+02 Objects: 2.120e+02 1.00000 2.120e+02 Flops: 3.205e+08 1.40451 2.697e+08 1.726e+10 Flops/sec: 9.268e+05 1.40451 7.799e+05 4.991e+07 Memory: 3.065e+06 1.25099 1.721e+08 MPI Messages: 1.182e+04 3.60692 8.252e+03 5.281e+05 MPI Message Lengths: 1.066e+07 5.78633 4.086e+02 2.158e+08 MPI Reductions: 4.365e+03 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 3.4587e+02 100.0% 1.7263e+10 100.0% 5.281e+05 100.0% 4.086e+02 100.0% 4.278e+03 98.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ ########################################################## # # # WARNING!!! # # # # This code was compiled with a debugging option, # # To get timing results run config/configure.py # # using --with-debugging=no, the performance will # # be generally two or three times faster. # # # ########################################################## Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage VecMDot 3195 1.0 8.1583e-01 2.7 2.23e+07 1.4 0.0e+00 0.0e+00 3.6e+02 0 7 0 0 8 0 7 0 0 8 1480 VecNorm 4335 1.0 1.2828e+00 2.0 1.13e+07 1.4 0.0e+00 0.0e+00 7.8e+02 0 4 0 0 18 0 4 0 0 18 475 VecScale 4192 1.0 2.3778e-02 1.3 5.47e+06 1.4 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 12415 VecCopy 995 1.0 5.0113e-03 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 2986 1.0 6.1189e-03 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 1140 1.0 8.8003e-03 1.3 2.78e+06 1.3 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 17310 VecAYPX 71 1.0 8.8720e-04 1.7 8.09e+04 1.2 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 5286 VecWAXPY 2 1.0 2.3029e-05 1.7 2.28e+03 1.2 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 5736 VecMAXPY 4192 1.0 5.6833e-02 1.4 3.08e+07 1.4 0.0e+00 0.0e+00 0.0e+00 0 10 0 0 0 0 10 0 0 0 29299 VecAssemblyBegin 3 1.0 5.8301e-03 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 9.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd 3 1.0 1.3023e-05 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 2279 1.0 5.6063e-02 3.6 0.00e+00 0.0 5.1e+05 3.7e+02 0.0e+00 0 0 96 88 0 0 0 96 88 0 0 VecScatterEnd 2279 1.0 4.8437e-0122.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 4118 1.0 8.3514e-01 1.9 1.61e+07 1.4 0.0e+00 0.0e+00 5.7e+02 0 5 0 0 13 0 5 0 0 13 1043 MatMult 3482 1.0 1.0133e+00 2.1 1.18e+08 1.4 2.1e+05 2.8e+02 0.0e+00 0 37 39 27 0 0 37 39 27 0 6271 MatMultAdd 71 1.0 9.5340e-02 3.9 1.04e+06 1.3 2.3e+04 1.8e+02 0.0e+00 0 0 4 2 0 0 0 4 2 0 611 MatMultTranspose 142 1.0 2.2453e-01 1.6 2.09e+06 1.3 4.6e+04 1.8e+02 2.8e+02 0 1 9 4 7 0 1 9 4 7 519 MatSolve 3550 1.0 5.7862e-01 1.4 1.26e+08 1.4 0.0e+00 0.0e+00 0.0e+00 0 39 0 0 0 0 39 0 0 0 11693 MatLUFactorNum 2 1.0 4.7321e-03 1.5 3.25e+05 1.5 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 3655 MatILUFactorSym 2 1.0 1.1258e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 5 1.0 1.6813e-0120.3 0.00e+00 0.0 1.4e+03 2.3e+03 6.0e+00 0 0 0 2 0 0 0 0 2 0 0 MatAssemblyEnd 5 1.0 2.3137e-02 1.3 0.00e+00 0.0 1.9e+03 6.1e+01 2.8e+01 0 0 0 0 1 0 0 0 0 1 0 MatGetRowIJ 2 1.0 4.9662e-06 4.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetSubMatrice 2 1.0 2.5637e-0132.3 0.00e+00 0.0 3.2e+03 2.1e+03 1.0e+01 0 0 1 3 0 0 0 1 3 0 0 MatGetOrdering 2 1.0 1.2449e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatIncreaseOvrlp 2 1.0 9.9950e-03 1.1 0.00e+00 0.0 1.3e+03 1.8e+02 6.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatZeroEntries 2 1.0 8.5980e-05 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MeshView 4 1.0 5.9615e+00 1.0 0.00e+00 0.0 1.8e+03 3.1e+03 0.0e+00 2 0 0 3 0 2 0 0 3 0 0 MeshGetGlobalScatter 3 1.0 4.1654e-02 1.2 0.00e+00 0.0 9.7e+02 6.0e+01 1.8e+01 0 0 0 0 0 0 0 0 0 0 0 MeshAssembleMatrix 1606 1.1 6.7121e-02 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MeshUpdateOperator 2168 1.1 2.7389e-01 4.6 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 0 0 0 0 0 0 0 SectionRealView 2 1.0 5.9061e-01199.5 0.00e+00 0.0 2.5e+02 4.1e+03 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 5 1.0 2.8859e-01 7.4 3.25e+05 1.5 5.8e+03 1.2e+03 4.6e+01 0 0 1 3 1 0 0 1 3 1 60 PCSetUpOnBlocks 284 1.0 8.3234e-03 1.3 3.25e+05 1.5 0.0e+00 0.0e+00 2.6e+01 0 0 0 0 1 0 0 0 0 1 2078 PCApply 71 1.0 4.8040e+00 1.0 3.13e+08 1.4 4.8e+05 3.8e+02 4.0e+03 1 97 91 84 92 1 97 91 84 94 3503 KSPGMRESOrthog 3195 1.0 8.5857e-01 2.5 4.46e+07 1.4 0.0e+00 0.0e+00 3.6e+02 0 14 0 0 8 0 14 0 0 8 2814 KSPSetup 6 1.0 2.9785e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 1 1.0 5.0004e+00 1.0 3.20e+08 1.4 5.1e+05 3.7e+02 4.2e+03 1100 96 87 95 1100 96 87 97 3449 MeshDestroy 5 1.0 3.1958e-011357.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 DistributeMesh 1 1.0 4.5183e+00 1.1 0.00e+00 0.0 5.0e+02 9.5e+03 0.0e+00 1 0 0 2 0 1 0 0 2 0 0 PartitionCreate 2 1.0 3.5427e-01 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PartitionClosure 2 1.0 1.2162e+0011594.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 DistributeCoords 2 1.0 8.2849e-01 2.8 0.00e+00 0.0 5.0e+02 3.0e+03 0.0e+00 0 0 0 1 0 0 0 0 1 0 0 DistributeLabels 2 1.0 1.6425e+00 1.0 0.00e+00 0.0 3.8e+02 6.9e+02 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 CreateOverlap 2 1.0 1.2166e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 1 0 0 0 0 1 0 0 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Viewer 4 4 1344 0 Index Set 29 29 89664 0 Vec 132 131 1098884 0 Vec Scatter 8 8 4320 0 Matrix 13 13 1315884 0 Mesh 5 5 1680 0 SectionReal 7 5 1320 0 Preconditioner 7 7 3132 0 Krylov Solver 7 7 88364 0 ======================================================================================================================== Average time to get PetscTime(): 1.90735e-07 Average time for MPI_Barrier(): 0.000211 Average time for zero size MPI_Send(): 1.4998e-05 #PETSc Option Table entries: -log_summary #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 4 sizeof(void*) 4 sizeof(PetscScalar) 8 Configure run at: Wed Mar 9 20:22:08 2011 Configure options: --with-clanguage=cxx --with-shared=1 --with-dynamic=1 --download-f-blas-lapack=1 --with-mpi-dir=/bwfs/software/ictce3.2/impi/3.2.0.011 --download-boost=1 --download-fiat=1 --download-generator=1 --download-triangle=1 --download-tetgen=1 --download-chaco=1 --download-parmetis=1 --download-zoltan=1 --with-sieve=1 --with-opt-sieve=1 --with-exodusii-dir=/bwfs/home/fdkong/petsc/petsc-3.1-p7/externalpackages/exodusii-4.75 --with-netcdf-dir=/bwfs/home/fdkong/petsc/petsc-3.1-p7/externalpackages/netcdf-4.1.1 ----------------------------------------- Libraries compiled on Wed Mar 9 20:22:27 CST 2011 on console Machine characteristics: Linux console 2.6.18-128.el5 #1 SMP Wed Dec 17 11:41:38 EST 2008 x86_64 x86_64 x86_64 GNU/Linux Using PETSc directory: /bwfs/home/fdkong/petsc/petsc-3.1-p7 Using PETSc arch: linux-gnu-c-debug ----------------------------------------- Using C compiler: /bwfs/software/ictce3.2/impi/3.2.0.011/bin/mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -g -fPIC Using Fortran compiler: /bwfs/software/ictce3.2/impi/3.2.0.011/bin/mpif90 -fPIC -Wall -Wno-unused-variable -g ----------------------------------------- Using include paths: -I/bwfs/home/fdkong/petsc/petsc-3.1-p7/linux-gnu-c-debug/include -I/bwfs/home/fdkong/petsc/petsc-3.1-p7/include -I/bwfs/home/fdkong/petsc/petsc-3.1-p7/linux-gnu-c-debug/include -I/export/ictce3.2/impi/3.2.0.011/include/gfortran/4.1.0 -I/export/ictce3.2/impi/3.2.0.011/include -I/bwfs/home/fdkong/petsc/petsc-3.1-p7/include/sieve -I/bwfs/home/fdkong/petsc/petsc-3.1-p7/externalpackages/Boost/ -I/bwfs/home/fdkong/petsc/petsc-3.1-p7/externalpackages/exodusii-4.75/include -I/bwfs/home/fdkong/petsc/petsc-3.1-p7/externalpackages/netcdf-4.1.1/include -I/bwfs/software/ictce3.2/impi/3.2.0.011/include ------------------------------------------ Using C linker: /bwfs/software/ictce3.2/impi/3.2.0.011/bin/mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -g Using Fortran linker: /bwfs/software/ictce3.2/impi/3.2.0.011/bin/mpif90 -fPIC -Wall -Wno-unused-variable -g Using libraries: -Wl,-rpath,/bwfs/home/fdkong/petsc/petsc-3.1-p7/linux-gnu-c-debug/lib -L/bwfs/home/fdkong/petsc/petsc-3.1-p7/linux-gnu-c-debug/lib -lpetsc -Wl,-rpath,/bwfs/home/fdkong/petsc/petsc-3.1-p7/linux-gnu-c-debug/lib -L/bwfs/home/fdkong/petsc/petsc-3.1-p7/linux-gnu-c-debug/lib -lzoltan -ltriangle -lX11 -lchaco -lparmetis -lmetis -ltetgen -lflapack -lfblas -Wl,-rpath,/bwfs/home/fdkong/petsc/petsc-3.1-p7/externalpackages/exodusii-4.75/lib -L/bwfs/home/fdkong/petsc/petsc-3.1-p7/externalpackages/exodusii-4.75/lib -lexoIIv2for -lexoIIv2c -Wl,-rpath,/bwfs/home/fdkong/petsc/petsc-3.1-p7/externalpackages/netcdf-4.1.1/lib -L/bwfs/home/fdkong/petsc/petsc-3.1-p7/externalpackages/netcdf-4.1.1/lib -lnetcdf -Wl,-rpath,/export/ictce3.2/impi/3.2.0.011/lib -L/export/ictce3.2/impi/3.2.0.011/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.1.2/32 -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2/32 -ldl -lmpi -lmpigf -lmpigi -lrt -lpthread -lgcc_s -Wl,-rpath,/bwfs/home/fdkong/petsc/petsc-3.1-p7/-Xlinker -lmpi_dbg -lgfortran -lm -Wl,-rpath,/opt/intel/mpi-rt/3.2 -lm -lmpigc4 -lmpi_dbg -lstdc++ -lmpigc4 -lmpi_dbg -lstdc++ -ldl -lmpi -lmpigf -lmpigi -lrt -lpthread -lgcc_s -ldl ------------------------------------------ ------------------ Fande Kong ShenZhen Institutes of Advanced Technology Chinese Academy of Sciences ------------------ Original ------------------ From: "knepley"; Date: Mon, Mar 28, 2011 02:19 PM To: "PETSc users list"; Cc: "fdkong"; Subject: Re: [petsc-users] Generation, refinement of the mesh (Sieve mesh) isvery slow! 1) Always send the output of -log_summary when asking a performance question 2) There are implementations that are optimized for different things. Its possible to ?1?7?1?7 ?1?7optimize mesh handling for a cells-vertices mesh, but not if you need edges and ?1?7?1?7 ?1?7faces generated. 3) I am out of the country. I can look at the performance when I get back. ?1?7?1?7 Matt On Mon, Mar 28, 2011 at 1:06 AM, fdkong wrote: Hi everyone ?1?7?1?7 I have developed my application based on the sieve mesh object in the Pestc. And now,?1?7I encountered some ?1?7serious problems.?1?7 ?1?71. The generation of mesh?1?7takes a lot of time, run very slowly. The following code is used: ?1?7?1?7?1?7?1?7?1?7 ?1?7double lower[2] = {-1.0, -1.0}; ?1?7?1?7 ?1?7 ?1?7 ?1?7double upper[2] = {1.0, 1.0}; ?1?7?1?7 ?1?7 ?1?7 ?1?7int ?1?7 ?1?7edges[2] = {256,256}; ?1?7?1?7 ?1?7 ?1?7 ?1?7mB = ALE::MeshBuilder::createSquareBoundary(comm, lower, upper, edges, debug); ?1?7?1?7ALE::ISieveConverter::convertMesh(*mB, *meshBd, renumbering, false); ?1?7?1?7 ?1?7 ?1?7ierr = PetscPrintf(PETSC_COMM_WORLD," End build convertMesh ?1?7\n");CHKERRQ(ierr); ?1?7?1?7 ?1?7 ?1?7ierr = MeshSetMesh(boundary, meshBd);CHKERRQ(ierr); ?1?7?1?7 ?1?7 ?1?7ierr = PetscPrintf(PETSC_COMM_WORLD," Begin build MeshGenerate ?1?7\n");CHKERRQ(ierr); ?1?7?1?7 ?1?7 ?1?7ierr = MeshGenerate(boundary,interpolate, &mesh);CHKERRQ(ierr); ?1?7?1?7?1?7 ?1?7?1?7 2. The refinement of mesh is also very slow. Th code: ?1?7?1?7 ?1?7?1?7refinementLimit=0.0001; ?1?7?1?7 ?1?7if (refinementLimit > 0.0)?1?7 ?1?7?1?7 ?1?7{ ?1?7?1?7 ?1?7 ?1?7Mesh refinedMesh; ?1?7?1?7 ?1?7 ?1?7ierr = MeshRefine(mesh, refinementLimit,interpolate, &refinedMesh);CHKERRQ(ierr); ?1?7?1?7 ?1?7 ?1?7ierr = MeshDestroy(mesh);CHKERRQ(ierr); ?1?7?1?7 ?1?7 ?1?7mesh = refinedMesh; ?1?7?1?7 ?1?7} ?1?7?1?7 ?1?73. The distribution of mesh is also very slow. The code: ?1?7?1?7 ?1?7?1?7if (size > 1)?1?7 ?1?7?1?7 ?1?7{ ?1?7?1?7 ?1?7 ?1?7Mesh parallelMesh; ?1?7?1?7 ?1?7 ?1?7//ierr = DistributeMeshnew(mesh, "chao", ¶llelMesh);CHKERRQ(ierr); ?1?7?1?7 ?1?7 ?1?7ierr = DistributeMeshnew(mesh, "parmetis", ¶llelMesh);CHKERRQ(ierr); ?1?7?1?7 ?1?7 ?1?7ierr = MeshDestroy(mesh);CHKERRQ(ierr); ?1?7?1?7 ?1?7 ?1?7mesh = parallelMesh; ?1?7?1?7 ?1?7}. ?1?7?1?7?1?7 ?1?7?1?7 ?1?7Does anyone encounter these similar problem? If anyone can help, thank you very much! ?1?7 ?1?7?1?7 ?1?7?1?7 And I wonder to consult which parallel mesh can work ?1?7with Petsc very well, when we develop some complex problem??1?7 ?1?7?1?7?1?7 ?1?7?1?7 ?1?7?1?7 ------------------ Fande Kong ShenZhen Institutes of Advanced Technology Chinese Academy of Sciences ?1?7 -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From austin at txcorp.com Mon Mar 28 10:47:18 2011 From: austin at txcorp.com (Travis Austin) Date: Mon, 28 Mar 2011 08:47:18 -0700 Subject: [petsc-users] Generation, refinement of the mesh (Sieve mesh) isvery slow! In-Reply-To: References: Message-ID: <3BFE26BC-D9C8-4B6B-AADF-4E4ACD925BB6@txcorp.com> I'm no expert on Sieve but I would probably rerun these simulations with the code compiled without '-g' which is the debugging option. See the following: > > ########################################################## > # # > # WARNING!!! # > # # > # This code was compiled with a debugging option, # > # To get timing results run config/configure.py # > # using --with-debugging=no, the performance will # > # be generally two or three times faster. # > # # > ########################################################## From my experience it is good to run with no debugging to get a true timing of the simulation and much better than just trying to estimate that no debugging will be 2-3x faster. Travis > > > Event Count Time (sec) Flops --- Global --- --- Stage --- Total > Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s > ------------------------------------------------------------------------------------------------------------------------ > > --- Event Stage 0: Main Stage > > VecMDot 3195 1.0 8.1583e-01 2.7 2.23e+07 1.4 0.0e+00 0.0e+00 3.6e+02 0 7 0 0 8 0 7 0 0 8 1480 > VecNorm 4335 1.0 1.2828e+00 2.0 1.13e+07 1.4 0.0e+00 0.0e+00 7.8e+02 0 4 0 0 18 0 4 0 0 18 475 > VecScale 4192 1.0 2.3778e-02 1.3 5.47e+06 1.4 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 12415 > VecCopy 995 1.0 5.0113e-03 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecSet 2986 1.0 6.1189e-03 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecAXPY 1140 1.0 8.8003e-03 1.3 2.78e+06 1.3 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 17310 > VecAYPX 71 1.0 8.8720e-04 1.7 8.09e+04 1.2 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 5286 > VecWAXPY 2 1.0 2.3029e-05 1.7 2.28e+03 1.2 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 5736 > VecMAXPY 4192 1.0 5.6833e-02 1.4 3.08e+07 1.4 0.0e+00 0.0e+00 0.0e+00 0 10 0 0 0 0 10 0 0 0 29299 > VecAssemblyBegin 3 1.0 5.8301e-03 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 9.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecAssemblyEnd 3 1.0 1.3023e-05 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecScatterBegin 2279 1.0 5.6063e-02 3.6 0.00e+00 0.0 5.1e+05 3.7e+02 0.0e+00 0 0 96 88 0 0 0 96 88 0 0 > VecScatterEnd 2279 1.0 4.8437e-0122.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecNormalize 4118 1.0 8.3514e-01 1.9 1.61e+07 1.4 0.0e+00 0.0e+00 5.7e+02 0 5 0 0 13 0 5 0 0 13 1043 > MatMult 3482 1.0 1.0133e+00 2.1 1.18e+08 1.4 2.1e+05 2.8e+02 0.0e+00 0 37 39 27 0 0 37 39 27 0 6271 > MatMultAdd 71 1.0 9.5340e-02 3.9 1.04e+06 1.3 2.3e+04 1.8e+02 0.0e+00 0 0 4 2 0 0 0 4 2 0 611 > MatMultTranspose 142 1.0 2.2453e-01 1.6 2.09e+06 1.3 4.6e+04 1.8e+02 2.8e+02 0 1 9 4 7 0 1 9 4 7 519 > MatSolve 3550 1.0 5.7862e-01 1.4 1.26e+08 1.4 0.0e+00 0.0e+00 0.0e+00 0 39 0 0 0 0 39 0 0 0 11693 > MatLUFactorNum 2 1.0 4.7321e-03 1.5 3.25e+05 1.5 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 3655 > MatILUFactorSym 2 1.0 1.1258e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyBegin 5 1.0 1.6813e-0120.3 0.00e+00 0.0 1.4e+03 2.3e+03 6.0e+00 0 0 0 2 0 0 0 0 2 0 0 > MatAssemblyEnd 5 1.0 2.3137e-02 1.3 0.00e+00 0.0 1.9e+03 6.1e+01 2.8e+01 0 0 0 0 1 0 0 0 0 1 0 > MatGetRowIJ 2 1.0 4.9662e-06 4.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatGetSubMatrice 2 1.0 2.5637e-0132.3 0.00e+00 0.0 3.2e+03 2.1e+03 1.0e+01 0 0 1 3 0 0 0 1 3 0 0 > MatGetOrdering 2 1.0 1.2449e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatIncreaseOvrlp 2 1.0 9.9950e-03 1.1 0.00e+00 0.0 1.3e+03 1.8e+02 6.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatZeroEntries 2 1.0 8.5980e-05 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MeshView 4 1.0 5.9615e+00 1.0 0.00e+00 0.0 1.8e+03 3.1e+03 0.0e+00 2 0 0 3 0 2 0 0 3 0 0 > MeshGetGlobalScatter 3 1.0 4.1654e-02 1.2 0.00e+00 0.0 9.7e+02 6.0e+01 1.8e+01 0 0 0 0 0 0 0 0 0 0 0 > MeshAssembleMatrix 1606 1.1 6.7121e-02 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MeshUpdateOperator 2168 1.1 2.7389e-01 4.6 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 0 0 0 0 0 0 0 > SectionRealView 2 1.0 5.9061e-01199.5 0.00e+00 0.0 2.5e+02 4.1e+03 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > PCSetUp 5 1.0 2.8859e-01 7.4 3.25e+05 1.5 5.8e+03 1.2e+03 4.6e+01 0 0 1 3 1 0 0 1 3 1 60 > PCSetUpOnBlocks 284 1.0 8.3234e-03 1.3 3.25e+05 1.5 0.0e+00 0.0e+00 2.6e+01 0 0 0 0 1 0 0 0 0 1 2078 > PCApply 71 1.0 4.8040e+00 1.0 3.13e+08 1.4 4.8e+05 3.8e+02 4.0e+03 1 97 91 84 92 1 97 91 84 94 3503 > KSPGMRESOrthog 3195 1.0 8.5857e-01 2.5 4.46e+07 1.4 0.0e+00 0.0e+00 3.6e+02 0 14 0 0 8 0 14 0 0 8 2814 > KSPSetup 6 1.0 2.9785e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01 0 0 0 0 0 0 0 0 0 0 0 > KSPSolve 1 1.0 5.0004e+00 1.0 3.20e+08 1.4 5.1e+05 3.7e+02 4.2e+03 1100 96 87 95 1100 96 87 97 3449 > MeshDestroy 5 1.0 3.1958e-011357.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > DistributeMesh 1 1.0 4.5183e+00 1.1 0.00e+00 0.0 5.0e+02 9.5e+03 0.0e+00 1 0 0 2 0 1 0 0 2 0 0 > PartitionCreate 2 1.0 3.5427e-01 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > PartitionClosure 2 1.0 1.2162e+0011594.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > DistributeCoords 2 1.0 8.2849e-01 2.8 0.00e+00 0.0 5.0e+02 3.0e+03 0.0e+00 0 0 0 1 0 0 0 0 1 0 0 > DistributeLabels 2 1.0 1.6425e+00 1.0 0.00e+00 0.0 3.8e+02 6.9e+02 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > CreateOverlap 2 1.0 1.2166e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 1 0 0 0 0 1 0 0 > ------------------------------------------------------------------------------------------------------------------------ > > Memory usage is given in bytes: > > Object Type Creations Destructions Memory Descendants' Mem. > Reports information only for process 0. > > --- Event Stage 0: Main Stage > > Viewer 4 4 1344 0 > Index Set 29 29 89664 0 > Vec 132 131 1098884 0 > Vec Scatter 8 8 4320 0 > Matrix 13 13 1315884 0 > Mesh 5 5 1680 0 > SectionReal 7 5 1320 0 > Preconditioner 7 7 3132 0 > Krylov Solver 7 7 88364 0 > ======================================================================================================================== > Average time to get PetscTime(): 1.90735e-07 > Average time for MPI_Barrier(): 0.000211 > Average time for zero size MPI_Send(): 1.4998e-05 > #PETSc Option Table entries: > -log_summary > #End of PETSc Option Table entries > Compiled without FORTRAN kernels > Compiled with full precision matrices (default) > sizeof(short) 2 sizeof(int) 4 sizeof(long) 4 sizeof(void*) 4 sizeof(PetscScalar) 8 > Configure run at: Wed Mar 9 20:22:08 2011 > Configure options: --with-clanguage=cxx --with-shared=1 --with-dynamic=1 --download-f-blas-lapack=1 --with-mpi-dir=/bwfs/software/ictce3.2/impi/3.2.0.011 --download-boost=1 --download-fiat=1 --download-generator=1 --download-triangle=1 --download-tetgen=1 --download-chaco=1 --download-parmetis=1 --download-zoltan=1 --with-sieve=1 --with-opt-sieve=1 --with-exodusii-dir=/bwfs/home/fdkong/petsc/petsc-3.1-p7/externalpackages/exodusii-4.75 --with-netcdf-dir=/bwfs/home/fdkong/petsc/petsc-3.1-p7/externalpackages/netcdf-4.1.1 > ----------------------------------------- > Libraries compiled on Wed Mar 9 20:22:27 CST 2011 on console > Machine characteristics: Linux console 2.6.18-128.el5 #1 SMP Wed Dec 17 11:41:38 EST 2008 x86_64 x86_64 x86_64 GNU/Linux > Using PETSc directory: /bwfs/home/fdkong/petsc/petsc-3.1-p7 > Using PETSc arch: linux-gnu-c-debug > ----------------------------------------- > Using C compiler: /bwfs/software/ictce3.2/impi/3.2.0.011/bin/mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -g -fPIC > Using Fortran compiler: /bwfs/software/ictce3.2/impi/3.2.0.011/bin/mpif90 -fPIC -Wall -Wno-unused-variable -g > ----------------------------------------- > Using include paths: -I/bwfs/home/fdkong/petsc/petsc-3.1-p7/linux-gnu-c-debug/include -I/bwfs/home/fdkong/petsc/petsc-3.1-p7/include -I/bwfs/home/fdkong/petsc/petsc-3.1-p7/linux-gnu-c-debug/include -I/export/ictce3.2/impi/3.2.0.011/include/gfortran/4.1.0 -I/export/ictce3.2/impi/3.2.0.011/include -I/bwfs/home/fdkong/petsc/petsc-3.1-p7/include/sieve -I/bwfs/home/fdkong/petsc/petsc-3.1-p7/externalpackages/Boost/ -I/bwfs/home/fdkong/petsc/petsc-3.1-p7/externalpackages/exodusii-4.75/include -I/bwfs/home/fdkong/petsc/petsc-3.1-p7/externalpackages/netcdf-4.1.1/include -I/bwfs/software/ictce3.2/impi/3.2.0.011/include > ------------------------------------------ > Using C linker: /bwfs/software/ictce3.2/impi/3.2.0.011/bin/mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -g > Using Fortran linker: /bwfs/software/ictce3.2/impi/3.2.0.011/bin/mpif90 -fPIC -Wall -Wno-unused-variable -g > Using libraries: -Wl,-rpath,/bwfs/home/fdkong/petsc/petsc-3.1-p7/linux-gnu-c-debug/lib -L/bwfs/home/fdkong/petsc/petsc-3.1-p7/linux-gnu-c-debug/lib -lpetsc -Wl,-rpath,/bwfs/home/fdkong/petsc/petsc-3.1-p7/linux-gnu-c-debug/lib -L/bwfs/home/fdkong/petsc/petsc-3.1-p7/linux-gnu-c-debug/lib -lzoltan -ltriangle -lX11 -lchaco -lparmetis -lmetis -ltetgen -lflapack -lfblas -Wl,-rpath,/bwfs/home/fdkong/petsc/petsc-3.1-p7/externalpackages/exodusii-4.75/lib -L/bwfs/home/fdkong/petsc/petsc-3.1-p7/externalpackages/exodusii-4.75/lib -lexoIIv2for -lexoIIv2c -Wl,-rpath,/bwfs/home/fdkong/petsc/petsc-3.1-p7/externalpackages/netcdf-4.1.1/lib -L/bwfs/home/fdkong/petsc/petsc-3.1-p7/externalpackages/netcdf-4.1.1/lib -lnetcdf -Wl,-rpath,/export/ictce3.2/impi/3.2.0.011/lib -L/export/ictce3.2/impi/3.2.0.011/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.1.2/32 -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2/32 -ldl -lmpi -lmpigf -lmpigi -lrt -lpthread -lgcc_s -Wl,-rpath,/bwfs/home/fdkong/petsc/petsc-3.1-p7/-Xlinker -lmpi_dbg -lgfortran -lm -Wl,-rpath,/opt/intel/mpi-rt/3.2 -lm -lmpigc4 -lmpi_dbg -lstdc++ -lmpigc4 -lmpi_dbg -lstdc++ -ldl -lmpi -lmpigf -lmpigi -lrt -lpthread -lgcc_s -ldl > ------------------------------------------ > > > > ------------------ > Fande Kong > ShenZhen Institutes of Advanced Technology > Chinese Academy of Sciences > > > > ------------------ Original ------------------ > From: "knepley"; > Date: Mon, Mar 28, 2011 02:19 PM > To: "PETSc users list"; > Cc: "fdkong"; > Subject: Re: [petsc-users] Generation, refinement of the mesh (Sieve mesh) isvery slow! > > 1) Always send the output of -log_summary when asking a performance question > > 2) There are implementations that are optimized for different things. Its possible to > ?1?7?1?7 ?1?7optimize mesh handling for a cells-vertices mesh, but not if you need edges and > ?1?7?1?7 ?1?7faces generated. > > 3) I am out of the country. I can look at the performance when I get back. > > ?1?7?1?7 Matt > > On Mon, Mar 28, 2011 at 1:06 AM, fdkong wrote: > Hi everyone > ?1?7?1?7 I have developed my application based on the sieve mesh object in the Pestc. And now,?1?7I encountered some ?1?7serious problems.?1?7 > ?1?71. The generation of mesh?1?7takes a lot of time, run very slowly. The following code is used: > ?1?7?1?7?1?7?1?7?1?7 ?1?7double lower[2] = {-1.0, -1.0}; > ?1?7?1?7 ?1?7 ?1?7 ?1?7double upper[2] = {1.0, 1.0}; > ?1?7?1?7 ?1?7 ?1?7 ?1?7int ?1?7 ?1?7edges[2] = {256,256}; > ?1?7?1?7 ?1?7 ?1?7 ?1?7mB = ALE::MeshBuilder::createSquareBoundary(comm, lower, upper, edges, debug); > ?1?7?1?7ALE::ISieveConverter::convertMesh(*mB, *meshBd, renumbering, false); > ?1?7?1?7 ?1?7 ?1?7ierr = PetscPrintf(PETSC_COMM_WORLD," End build convertMesh ?1?7\n");CHKERRQ(ierr); > ?1?7?1?7 ?1?7 ?1?7ierr = MeshSetMesh(boundary, meshBd);CHKERRQ(ierr); > ?1?7?1?7 ?1?7 ?1?7ierr = PetscPrintf(PETSC_COMM_WORLD," Begin build MeshGenerate ?1?7\n");CHKERRQ(ierr); > > ?1?7?1?7 ?1?7 ?1?7ierr = MeshGenerate(boundary,interpolate, &mesh);CHKERRQ(ierr); > ?1?7?1?7?1?7 > ?1?7?1?7 2. The refinement of mesh is also very slow. Th code: > ?1?7?1?7 ?1?7?1?7refinementLimit=0.0001; > ?1?7?1?7 ?1?7if (refinementLimit > 0.0)?1?7 > ?1?7?1?7 ?1?7{ > ?1?7?1?7 ?1?7 ?1?7Mesh refinedMesh; > > ?1?7?1?7 ?1?7 ?1?7ierr = MeshRefine(mesh, refinementLimit,interpolate, &refinedMesh);CHKERRQ(ierr); > ?1?7?1?7 ?1?7 ?1?7ierr = MeshDestroy(mesh);CHKERRQ(ierr); > ?1?7?1?7 ?1?7 ?1?7mesh = refinedMesh; > ?1?7?1?7 ?1?7} > > ?1?7?1?7 ?1?73. The distribution of mesh is also very slow. The code: > ?1?7?1?7 ?1?7?1?7if (size > 1)?1?7 > ?1?7?1?7 ?1?7{ > ?1?7?1?7 ?1?7 ?1?7Mesh parallelMesh; > > ?1?7?1?7 ?1?7 ?1?7//ierr = DistributeMeshnew(mesh, "chao", ¶llelMesh);CHKERRQ(ierr); > ?1?7?1?7 ?1?7 ?1?7ierr = DistributeMeshnew(mesh, "parmetis", ¶llelMesh);CHKERRQ(ierr); > ?1?7?1?7 ?1?7 ?1?7ierr = MeshDestroy(mesh);CHKERRQ(ierr); > ?1?7?1?7 ?1?7 ?1?7mesh = parallelMesh; > ?1?7?1?7 ?1?7}. > ?1?7?1?7?1?7 > ?1?7?1?7 ?1?7Does anyone encounter these similar problem? If anyone can help, thank you very much! ?1?7 > ?1?7?1?7 > ?1?7?1?7 And I wonder to consult which parallel mesh can work ?1?7with Petsc very well, when we develop some complex problem??1?7 > ?1?7?1?7?1?7 > ?1?7?1?7 ?1?7?1?7 > ------------------ > Fande Kong > ShenZhen Institutes of Advanced Technology > Chinese Academy of Sciences > ?1?7 > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From gaurish108 at gmail.com Mon Mar 28 12:04:18 2011 From: gaurish108 at gmail.com (Gaurish Telang) Date: Mon, 28 Mar 2011 13:04:18 -0400 Subject: [petsc-users] Typecasting PetscReal and PetscScalar before doing arithmetic? Message-ID: Hi, I have some variables in a program some of which are of type PetscReal and some of the type PetscScalar. Is it safe doing arithmetic with these variables without doing any explicit type casting? Thanks, Gaurish -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Mon Mar 28 12:14:45 2011 From: jed at 59A2.org (Jed Brown) Date: Mon, 28 Mar 2011 19:14:45 +0200 Subject: [petsc-users] Typecasting PetscReal and PetscScalar before doing arithmetic? In-Reply-To: References: Message-ID: On Mon, Mar 28, 2011 at 19:04, Gaurish Telang wrote: > I have some variables in a program some of which are of type PetscReal and > some of the type PetscScalar. Is it safe doing arithmetic with these > variables without doing any explicit type casting? Yes, but the result will have type PetscScalar unless you use PetscRealPart() or similar. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fd.kong at siat.ac.cn Mon Mar 28 20:18:50 2011 From: fd.kong at siat.ac.cn (=?ISO-8859-1?B?ZmRrb25n?=) Date: Tue, 29 Mar 2011 09:18:50 +0800 Subject: [petsc-users] Generation, refinement of the mesh (Sieve mesh) is very slow! Message-ID: Thank you Travis. I means that if we run the program with debugging, the generation and refinement of the sieve mesh are always very slow, compared with the action of other objects (e.g. KSPsolve, MatMult ). Fande ------------------ Fande Kong ShenZhen Institutes of Advanced Technology Chinese Academy of Sciences -------------- next part -------------- An HTML attachment was scrubbed... URL: From ckontzialis at lycos.com Tue Mar 29 01:07:19 2011 From: ckontzialis at lycos.com (Kontsantinos Kontzialis) Date: Tue, 29 Mar 2011 09:07:19 +0300 Subject: [petsc-users] Differencing parameter Message-ID: <4D917717.5050404@lycos.com> Dear all, When should I call MatMFFDGeth in order to get the differencing parameter? I call it after SNESSolve but I get zero for h. Thank you, Costas From jed at 59A2.org Tue Mar 29 02:35:14 2011 From: jed at 59A2.org (Jed Brown) Date: Tue, 29 Mar 2011 09:35:14 +0200 Subject: [petsc-users] Differencing parameter In-Reply-To: <4D917717.5050404@lycos.com> References: <4D917717.5050404@lycos.com> Message-ID: On Tue, Mar 29, 2011 at 08:07, Kontsantinos Kontzialis < ckontzialis at lycos.com> wrote: > When should I call MatMFFDGeth in order to get the differencing parameter? > I call it after SNESSolve but I get zero for h. > I suspect you are calling this function on a matrix of the wrong type (and, because this function does not check types properly, returning nonsense bytes that happen to represent a very small floating point value). Try calling ierr = SNESGetJacobian(snes,&mat,0,0,0);CHKERRQ(ierr); ierr = MatMFFDGetH(mat,&h);CHKERRQ(ierr); -------------- next part -------------- An HTML attachment was scrubbed... URL: From ckontzialis at lycos.com Tue Mar 29 02:49:47 2011 From: ckontzialis at lycos.com (Kontsantinos Kontzialis) Date: Tue, 29 Mar 2011 10:49:47 +0300 Subject: [petsc-users] Differencing parameter works now Message-ID: <4D918F1B.4010409@lycos.com> Jed, It is ok now! I have another question. I use the Discontinoues Galerkin discretization for the equations of gas dynamics. I use a matrix free formulation and that's why I need a very good preconditioner, which I compute it numerically and untill now it works ok, but it is a bit slow The preconditioner is just the Jacobian of the system. I wonder if PETSc has a built in capability to compute its entries. Costas From jed at 59A2.org Tue Mar 29 05:55:55 2011 From: jed at 59A2.org (Jed Brown) Date: Tue, 29 Mar 2011 12:55:55 +0200 Subject: [petsc-users] Differencing parameter works now In-Reply-To: <4D918F1B.4010409@lycos.com> References: <4D918F1B.4010409@lycos.com> Message-ID: On Tue, Mar 29, 2011 at 09:49, Kontsantinos Kontzialis < ckontzialis at lycos.com> wrote: > I have another question. I use the Discontinoues Galerkin discretization > for the equations of gas dynamics. I use a matrix free formulation and > that's why I need a very good preconditioner, which I compute it numerically > and untill now it works ok, but it is a bit slow The preconditioner is just > the Jacobian of the system. I wonder if PETSc has a built in capability to > compute its entries. You can use coloring, see snes/examples/tutorials/ex5.c for an example. It is more expensive in 3D and if the stencil is large. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Mar 29 07:50:53 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 29 Mar 2011 07:50:53 -0500 Subject: [petsc-users] Differencing parameter In-Reply-To: References: <4D917717.5050404@lycos.com> Message-ID: <20C97BC7-A387-45CC-BF55-8F10433CCF16@mcs.anl.gov> I have fixed petsc-dev to never return garbage in this call. Also fixed the other MatMFFD methods to use the compose/query function paradigm Barry On Mar 29, 2011, at 2:35 AM, Jed Brown wrote: > On Tue, Mar 29, 2011 at 08:07, Kontsantinos Kontzialis wrote: > When should I call MatMFFDGeth in order to get the differencing parameter? > I call it after SNESSolve but I get zero for h. > > I suspect you are calling this function on a matrix of the wrong type (and, because this function does not check types properly, returning nonsense bytes that happen to represent a very small floating point value). Try calling > > ierr = SNESGetJacobian(snes,&mat,0,0,0);CHKERRQ(ierr); > ierr = MatMFFDGetH(mat,&h);CHKERRQ(ierr); > From domenico.borzacchiello at univ-st-etienne.fr Wed Mar 30 08:04:44 2011 From: domenico.borzacchiello at univ-st-etienne.fr (domenico.borzacchiello at univ-st-etienne.fr) Date: Wed, 30 Mar 2011 15:04:44 +0200 (CEST) Subject: [petsc-users] Petsc and Matlab R2010b Message-ID: <95018ee7d2a18b84236102689f63d6b5.squirrel@arcon.univ-st-etienne.fr> Hi, I'm trying to build Petsc (current release) with Matlab R2010b with the following command line ./config/configure.py --with-cc=icc --with-fc=ifort --download-f-blas-lapack=1 --download-mpich=1 --with-matlab-dir=/opt/MATLAB/R2010b --with-matlab-engine The configuration stage is successfully completed. then If I build the libraries I get the following warning messages: BEGINNING TO COMPILE MATLAB INTERFACE sread.c(35): warning #188: enumerated type mixed with another type ierr = PetscBinaryRead(fd,mxGetPr(plhs[0]),cnt,dt);if (ierr) PETSC_MEX_ERROR("Unable to receive double items."); ^ sread.c(38): warning #188: enumerated type mixed with another type ierr = PetscBinaryRead(fd,mxGetPr(plhs[0]),cnt,dt);if (ierr) PETSC_MEX_ERROR("Unable to receive int items."); ^ sread.c(41): warning #188: enumerated type mixed with another type ierr = PetscBinaryRead(fd,tmp,cnt,dt);if (ierr) PETSC_MEX_ERROR("Unable to receive char items."); ^ ld: sread.o: relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC sread.o: could not read symbols: Bad value mex: link of ' "sread.mexa64"' failed. make[10]: [sread] Errore 1 (ignorato) /bin/mv: impossibile eseguire stat di "sread.mex*": File o directory non esistente make[10]: [sread] Errore 1 (ignorato) ld: sopen.o: relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC sopen.o: could not read symbols: Bad value mex: link of ' "sopen.mexa64"' failed. make[10]: [sopen] Errore 1 (ignorato) /bin/mv: impossibile eseguire stat di "sopen.mex*": File o directory non esistente make[10]: [sopen] Errore 1 (ignorato) ld: sclose.o: relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC sclose.o: could not read symbols: Bad value mex: link of ' "sclose.mexa64"' failed. make[10]: [sclose] Errore 1 (ignorato) /bin/mv: impossibile eseguire stat di "sclose.mex*": File o directory non esistente make[10]: [sclose] Errore 1 (ignorato) swrite.c(32): warning #188: enumerated type mixed with another type ierr = PetscBinaryWrite(fd,mxGetPr(prhs[1]),cnt,dt,PETSC_FALSE);if (ierr) PETSC_MEX_ERROR("Unable to send double items."); ^ swrite.c(37): warning #188: enumerated type mixed with another type ierr = PetscBinaryWrite(fd,tmp,cnt,dt,PETSC_FALSE);if (ierr) PETSC_MEX_ERROR("Unable to send int items."); ^ swrite.c(42): warning #188: enumerated type mixed with another type ierr = PetscBinaryWrite(fd,tmp,cnt,dt,PETSC_FALSE);if (ierr) PETSC_MEX_ERROR("Unable to send char items."); ^ ld: swrite.o: relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC swrite.o: could not read symbols: Bad value mex: link of ' "swrite.mexa64"' failed. Then if I try to run the tests i get the error: ld: cannot find -licudata ld: cannot find -licui18n ld: cannot find -licuuc ld: cannot find -lustdio What am I doing wrong? It worked perfectly without the matlab engine? Thank you in advance, Domenico From bartlomiej.wach at yahoo.pl Wed Mar 30 08:31:19 2011 From: bartlomiej.wach at yahoo.pl (=?utf-8?B?QmFydMWCb21pZWogVw==?=) Date: Wed, 30 Mar 2011 14:31:19 +0100 (BST) Subject: [petsc-users] Matrix inverse and LU decomposition Message-ID: <158063.63396.qm@web28305.mail.ukl.yahoo.com> Hello, I want to obtain the inverse of a matrix, even though it's not advised by the FAQ ;-) and I have a problem that earlier responses didn't solve for me. Simply speaking : the inverse matrix that is a result of? DI=Dinv as told by the FAQ, is empty for me. Here is my code(the significant part) : ? ierr = MatCreate(PETSC_COMM_WORLD,&D);CHKERRQ(ierr); ? ierr = MatSetSizes(D,PETSC_DECIDE,PETSC_DECIDE,n,n);CHKERRQ(ierr); ? ierr = MatSetFromOptions(D);CHKERRQ(ierr); ? ierr = MatCreate(PETSC_COMM_WORLD,&Dinv);CHKERRQ(ierr); ? ierr = MatSetSizes(Dinv,PETSC_DECIDE,PETSC_DECIDE,n,n);CHKERRQ(ierr); ? ierr = MatSetFromOptions(Dinv);CHKERRQ(ierr); ? ierr = MatCreate(PETSC_COMM_WORLD,&I);CHKERRQ(ierr); ? ierr = MatSetSizes(I,PETSC_DECIDE,PETSC_DECIDE,n,n);CHKERRQ(ierr); ? ierr = MatSetFromOptions(I);CHKERRQ(ierr); (...) ? ierr = MatAssemblyEnd(D,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); ? ierr = MatAssemblyEnd(I,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); ? ierr = MatAssemblyEnd(Dinv,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); ? MatView(D,PETSC_VIEWER_STDOUT_SELF); ??? ? ierr = MatGetOrdering(D,? MATORDERING_NATURAL,? &perm,? &iperm); CHKERRQ(ierr); ??? ? ierr = MatFactorInfoInitialize(&info); CHKERRQ(ierr); ? ierr = MatLUFactor(D, perm, iperm, &info); CHKERRQ(ierr); ? MatView(D,PETSC_VIEWER_STDOUT_SELF); ? MatView(I,PETSC_VIEWER_STDOUT_SELF); ??? ? ierr = MatMatSolve(D,I,Dinv); CHKERRQ(ierr); ? MatView(Dinv,PETSC_VIEWER_STDOUT_SELF); // Results in such output: //==== MATRIX D row 0: (0, 2) row 1: (1, 1) row 2: (2, 1) row 3: (3, 1) //==== MATRIX D after LU decomposition row 0: (0, 0.5) row 1: (1, 1) row 2: (2, 1) row 3: (3, 1) //==== Identity matrix row 0: (0, 1) row 1: (1, 1) row 2: (2, 1) row 3: (3, 1) //==== inverse of D? (result of MatMatSolve(D,I,Dinv)) row 0: row 1: row 2: row 3: The approach with : ??? ierr = MatGetOrdering(D,? MATORDERING_NATURAL,? &perm,? &iperm); CHKERRQ(ierr); ??? ??? ierr = MatGetFactor(D, MAT_SOLVER_PETSC, MAT_FACTOR_LU,&F); ??? ierr = MatFactorInfoInitialize(&info); CHKERRQ(ierr); ??? ierr = MatLUFactorSymbolic(F,D,perm,iperm,&info);CHKERRQ(ierr); ??? ierr = MatLUFactorNumeric(F,D,&info);CHKERRQ(ierr); ??? ierr = MatLUFactor(D, perm, iperm, &info); CHKERRQ(ierr); Provides same output Thank You for Your help. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Wed Mar 30 10:07:54 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Wed, 30 Mar 2011 10:07:54 -0500 Subject: [petsc-users] Matrix inverse and LU decomposition In-Reply-To: <158063.63396.qm@web28305.mail.ukl.yahoo.com> References: <158063.63396.qm@web28305.mail.ukl.yahoo.com> Message-ID: You need call MatSetType(I,MATDENSE); MatSetType(Dinv,MATDENSE); See petsc-dev/src/mat/examples/tests/ex1.c (attached) on how to use MatMatSolve(). Can you send me your complete code so I can check why Dinv is empty? Hong On Wed, Mar 30, 2011 at 8:31 AM, Bart?omiej W wrote: > Hello, > > I want to obtain the inverse of a matrix, even though it's not advised by > the FAQ ;-) and I have a problem that earlier responses didn't solve for me. > Simply speaking : the inverse matrix that is a result of DI=Dinv as told by > the FAQ, is empty for me. Here is my code(the significant part) : > > ierr = MatCreate(PETSC_COMM_WORLD,&D);CHKERRQ(ierr); > ierr = MatSetSizes(D,PETSC_DECIDE,PETSC_DECIDE,n,n);CHKERRQ(ierr); > ierr = MatSetFromOptions(D);CHKERRQ(ierr); > > ierr = MatCreate(PETSC_COMM_WORLD,&Dinv);CHKERRQ(ierr); > ierr = MatSetSizes(Dinv,PETSC_DECIDE,PETSC_DECIDE,n,n);CHKERRQ(ierr); > ierr = MatSetFromOptions(Dinv);CHKERRQ(ierr); > > ierr = MatCreate(PETSC_COMM_WORLD,&I);CHKERRQ(ierr); > ierr = MatSetSizes(I,PETSC_DECIDE,PETSC_DECIDE,n,n);CHKERRQ(ierr); > ierr = MatSetFromOptions(I);CHKERRQ(ierr); > > (...) > > ierr = MatAssemblyEnd(D,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = MatAssemblyEnd(I,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = MatAssemblyEnd(Dinv,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > > MatView(D,PETSC_VIEWER_STDOUT_SELF); > > ierr = MatGetOrdering(D, MATORDERING_NATURAL, &perm, &iperm); > CHKERRQ(ierr); > ierr = MatFactorInfoInitialize(&info); CHKERRQ(ierr); > ierr = MatLUFactor(D, perm, iperm, &info); CHKERRQ(ierr); > > MatView(D,PETSC_VIEWER_STDOUT_SELF); > MatView(I,PETSC_VIEWER_STDOUT_SELF); > > ierr = MatMatSolve(D,I,Dinv); CHKERRQ(ierr); > > MatView(Dinv,PETSC_VIEWER_STDOUT_SELF); > > // Results in such output: > > //==== MATRIX D > row 0: (0, 2) > row 1: (1, 1) > row 2: (2, 1) > row 3: (3, 1) > > //==== MATRIX D after LU decomposition > row 0: (0, 0.5) > row 1: (1, 1) > row 2: (2, 1) > row 3: (3, 1) > > //==== Identity matrix > row 0: (0, 1) > row 1: (1, 1) > row 2: (2, 1) > row 3: (3, 1) > > //==== inverse of D (result of MatMatSolve(D,I,Dinv)) > row 0: > row 1: > row 2: > row 3: > > > The approach with : > > ierr = MatGetOrdering(D, MATORDERING_NATURAL, &perm, &iperm); > CHKERRQ(ierr); > ierr = MatGetFactor(D, MAT_SOLVER_PETSC, MAT_FACTOR_LU,&F); > ierr = MatFactorInfoInitialize(&info); CHKERRQ(ierr); > ierr = MatLUFactorSymbolic(F,D,perm,iperm,&info);CHKERRQ(ierr); > ierr = MatLUFactorNumeric(F,D,&info);CHKERRQ(ierr); > ierr = MatLUFactor(D, perm, iperm, &info); CHKERRQ(ierr); > > Provides same output > > Thank You for Your help. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex1.c Type: text/x-csrc Size: 6160 bytes Desc: not available URL: From dmitry.s.kamenetskiy at boeing.com Wed Mar 30 10:43:37 2011 From: dmitry.s.kamenetskiy at boeing.com (I-Kamenetskiy, Dmitry S) Date: Wed, 30 Mar 2011 08:43:37 -0700 Subject: [petsc-users] Ghosted vectors in KSP via SNES Message-ID: Hello, I have run in to the following issue with the PETSc SNES usage. The logic of the SNESSolve() routine in my code is that the solution vector x should include ghost points at the boundary of subdomains in the distributed case. This is the 'current solution' vector I provide to the custom EvaluateResidual() function, a pointer to which is passed to the SNES object when calling SNESSetFunction(). However, I would strongly like to avoid using ghosted vectors in the internal SNES's KSP object as, multiplied by a large number of Krylov directions, this leads to a very significant memory overhead in my case. Now: here I am not 100 percent sure that this is what happens, but at least my memory measurements with -ksp_gmres_preallocate option support this hypothesis. It seems that there is no reason to store/update the ghosted vector components when applying the Matrix-vector multiplication needed to evaluate the next Krylov direction. My question is: does one have any handle to provide alternative Vec type to the KSP, when it is used from inside the SNES object? Sorry if I am overly complicating matters, or have invented the problem that is non-existent. Any comments will be much appreciated. Thank you, Dmitry Kamenetskiy. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bartlomiej.wach at yahoo.pl Wed Mar 30 11:11:50 2011 From: bartlomiej.wach at yahoo.pl (=?utf-8?B?QmFydMWCb21pZWogVw==?=) Date: Wed, 30 Mar 2011 17:11:50 +0100 (BST) Subject: [petsc-users] Matrix inverse and LU decomposition In-Reply-To: Message-ID: <428295.71575.qm@web28305.mail.ukl.yahoo.com> That solved the problem, thank you very much! Afterall, it ocurred that I don't need this procedure since I will only use diagonal matrices and inverse of such are trivial. It will come in handy someday though, I am sure. Thanks again. --- 30.3.11 (?r), Hong Zhang napisa?(a): Od: Hong Zhang Temat: Re: [petsc-users] Matrix inverse and LU decomposition Do: "PETSc users list" Data: 30 Marzec 2011 (?roda), 15:07 You need call MatSetType(I,MATDENSE);MatSetType(Dinv,MATDENSE); See?petsc-dev/src/mat/examples/tests/ex1.c (attached) on how to useMatMatSolve().Can you send me your complete code so I can check why Dinv is empty? Hong On Wed, Mar 30, 2011 at 8:31 AM, Bart?omiej W wrote: Hello, I want to obtain the inverse of a matrix, even though it's not advised by the FAQ ;-) and I have a problem that earlier responses didn't solve for me. Simply speaking : the inverse matrix that is a result of? DI=Dinv as told by the FAQ, is empty for me. Here is my code(the significant part) : ? ierr = MatCreate(PETSC_COMM_WORLD,&D);CHKERRQ(ierr); ? ierr = MatSetSizes(D,PETSC_DECIDE,PETSC_DECIDE,n,n);CHKERRQ(ierr); ? ierr = MatSetFromOptions(D);CHKERRQ(ierr); ? ierr = MatCreate(PETSC_COMM_WORLD,&Dinv);CHKERRQ(ierr); ? ierr = MatSetSizes(Dinv,PETSC_DECIDE,PETSC_DECIDE,n,n);CHKERRQ(ierr); ? ierr = MatSetFromOptions(Dinv);CHKERRQ(ierr); ? ierr = MatCreate(PETSC_COMM_WORLD,&I);CHKERRQ(ierr); ? ierr = MatSetSizes(I,PETSC_DECIDE,PETSC_DECIDE,n,n);CHKERRQ(ierr); ? ierr = MatSetFromOptions(I);CHKERRQ(ierr); (...) ? ierr = MatAssemblyEnd(D,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); ? ierr = MatAssemblyEnd(I,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); ? ierr = MatAssemblyEnd(Dinv,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); ? MatView(D,PETSC_VIEWER_STDOUT_SELF); ??? ? ierr = MatGetOrdering(D,? MATORDERING_NATURAL,? &perm,? &iperm); CHKERRQ(ierr); ??? ? ierr = MatFactorInfoInitialize(&info); CHKERRQ(ierr); ? ierr = MatLUFactor(D, perm, iperm, &info); CHKERRQ(ierr); ? MatView(D,PETSC_VIEWER_STDOUT_SELF); ? MatView(I,PETSC_VIEWER_STDOUT_SELF); ??? ? ierr = MatMatSolve(D,I,Dinv); CHKERRQ(ierr); ? MatView(Dinv,PETSC_VIEWER_STDOUT_SELF); // Results in such output: //==== MATRIX D row 0: (0, 2) row 1: (1, 1) row 2: (2, 1) row 3: (3, 1) //==== MATRIX D after LU decomposition row 0: (0, 0.5) row 1: (1, 1) row 2: (2, 1) row 3: (3, 1) //==== Identity matrix row 0: (0, 1) row 1: (1, 1) row 2: (2, 1) row 3: (3, 1) //==== inverse of D? (result of MatMatSolve(D,I,Dinv)) row 0: row 1: row 2: row 3: The approach with : ??? ierr = MatGetOrdering(D,? MATORDERING_NATURAL,? &perm,? &iperm); CHKERRQ(ierr); ??? ??? ierr = MatGetFactor(D, MAT_SOLVER_PETSC, MAT_FACTOR_LU,&F); ??? ierr = MatFactorInfoInitialize(&info); CHKERRQ(ierr); ??? ierr = MatLUFactorSymbolic(F,D,perm,iperm,&info);CHKERRQ(ierr); ??? ierr = MatLUFactorNumeric(F,D,&info);CHKERRQ(ierr); ??? ierr = MatLUFactor(D, perm, iperm, &info); CHKERRQ(ierr); Provides same output Thank You for Your help. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Mar 30 11:54:23 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 30 Mar 2011 11:54:23 -0500 Subject: [petsc-users] Petsc and Matlab R2010b In-Reply-To: <95018ee7d2a18b84236102689f63d6b5.squirrel@arcon.univ-st-etienne.fr> References: <95018ee7d2a18b84236102689f63d6b5.squirrel@arcon.univ-st-etienne.fr> Message-ID: <1CC5CDB7-D341-46CD-9124-8B24B00A7CED@mcs.anl.gov> Please switch to petsc-dev http://www.mcs.anl.gov/petsc/petsc-as/developers/index.html to build with Matlab R2010b. This version of Matlab came out long after petsc-3.1 and hence we needed to update PETSc for it. Barry On Mar 30, 2011, at 8:04 AM, domenico.borzacchiello at univ-st-etienne.fr wrote: > Hi, > I'm trying to build Petsc (current release) with Matlab R2010b with the > following command line > > ./config/configure.py --with-cc=icc --with-fc=ifort > --download-f-blas-lapack=1 --download-mpich=1 > --with-matlab-dir=/opt/MATLAB/R2010b --with-matlab-engine > > The configuration stage is successfully completed. then If I build the > libraries I get the following warning messages: > > BEGINNING TO COMPILE MATLAB INTERFACE > sread.c(35): warning #188: enumerated type mixed with another type > ierr = PetscBinaryRead(fd,mxGetPr(plhs[0]),cnt,dt);if (ierr) > PETSC_MEX_ERROR("Unable to receive double items."); > ^ > > sread.c(38): warning #188: enumerated type mixed with another type > ierr = PetscBinaryRead(fd,mxGetPr(plhs[0]),cnt,dt);if (ierr) > PETSC_MEX_ERROR("Unable to receive int items."); > ^ > > sread.c(41): warning #188: enumerated type mixed with another type > ierr = PetscBinaryRead(fd,tmp,cnt,dt);if (ierr) > PETSC_MEX_ERROR("Unable to receive char items."); > ^ > > ld: sread.o: relocation R_X86_64_32 against `.rodata' can not be used when > making a shared object; recompile with -fPIC > sread.o: could not read symbols: Bad value > > mex: link of ' "sread.mexa64"' failed. > > make[10]: [sread] Errore 1 (ignorato) > /bin/mv: impossibile eseguire stat di "sread.mex*": File o directory non > esistente > make[10]: [sread] Errore 1 (ignorato) > ld: sopen.o: relocation R_X86_64_32 against `.rodata' can not be used when > making a shared object; recompile with -fPIC > sopen.o: could not read symbols: Bad value > > mex: link of ' "sopen.mexa64"' failed. > > make[10]: [sopen] Errore 1 (ignorato) > /bin/mv: impossibile eseguire stat di "sopen.mex*": File o directory non > esistente > make[10]: [sopen] Errore 1 (ignorato) > ld: sclose.o: relocation R_X86_64_32 against `.rodata' can not be used > when making a shared object; recompile with -fPIC > sclose.o: could not read symbols: Bad value > > mex: link of ' "sclose.mexa64"' failed. > > make[10]: [sclose] Errore 1 (ignorato) > /bin/mv: impossibile eseguire stat di "sclose.mex*": File o directory non > esistente > make[10]: [sclose] Errore 1 (ignorato) > swrite.c(32): warning #188: enumerated type mixed with another type > ierr = PetscBinaryWrite(fd,mxGetPr(prhs[1]),cnt,dt,PETSC_FALSE);if > (ierr) PETSC_MEX_ERROR("Unable to send double items."); > ^ > > swrite.c(37): warning #188: enumerated type mixed with another type > ierr = PetscBinaryWrite(fd,tmp,cnt,dt,PETSC_FALSE);if (ierr) > PETSC_MEX_ERROR("Unable to send int items."); > ^ > > swrite.c(42): warning #188: enumerated type mixed with another type > ierr = PetscBinaryWrite(fd,tmp,cnt,dt,PETSC_FALSE);if (ierr) > PETSC_MEX_ERROR("Unable to send char items."); > ^ > > ld: swrite.o: relocation R_X86_64_32 against `.rodata' can not be used > when making a shared object; recompile with -fPIC > swrite.o: could not read symbols: Bad value > > mex: link of ' "swrite.mexa64"' failed. > > Then if I try to run the tests i get the error: > > ld: cannot find -licudata > ld: cannot find -licui18n > ld: cannot find -licuuc > ld: cannot find -lustdio > > What am I doing wrong? It worked perfectly without the matlab engine? > > Thank you in advance, > Domenico > From gdiso at ustc.edu Wed Mar 30 12:06:27 2011 From: gdiso at ustc.edu (Gong Ding) Date: Thu, 31 Mar 2011 01:06:27 +0800 Subject: [petsc-users] schur complement and condition number Message-ID: Hi all, These days I am dealing with a very bad conditioned problem. ksp solver can not convergence. Although direct linear solver such as mumps works, memory is a limitation to me. So I am still seeking a method to make iterative method work. I know the bad condition number comes from a lumped boundary. For this boundary, the sum of flux comes from all the boundary elements shoule equals to a fixed given flux. If I set the value of each boundary elements equals to a const value (first type boundary condition) the condition number of final matrix is in moderate range. I image if I can do Gaussian elimination first to the boundary elements, (create schur complement), is it possible to reduce condition number of origin matrix? Then I can solve the problem by ksp method. From amesga1 at tigers.lsu.edu Wed Mar 30 12:49:32 2011 From: amesga1 at tigers.lsu.edu (Ataollah Mesgarnejad) Date: Wed, 30 Mar 2011 12:49:32 -0500 Subject: [petsc-users] reading text files Message-ID: Dear all, I have a small file that need to read. I use PetscFOpen and PetscFPrintf to write it,so I wonder if there is a PetscFScanf too? Cheers, Ata From bsmith at mcs.anl.gov Wed Mar 30 15:10:02 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 30 Mar 2011 15:10:02 -0500 Subject: [petsc-users] reading text files In-Reply-To: References: Message-ID: <815F3A2E-197F-4674-9288-FCA189270C12@mcs.anl.gov> No, the closest thing is PetscSynchronizedFGets(). Feel free to contribute: http://www.mcs.anl.gov/petsc/petsc-as/developers/index.html#Sending_patches_to_update_the_ Barry On Mar 30, 2011, at 12:49 PM, Ataollah Mesgarnejad wrote: > Dear all, > > I have a small file that need to read. I use PetscFOpen and PetscFPrintf to write it,so I wonder if there is a PetscFScanf too? > > Cheers, > Ata > From jed at 59A2.org Wed Mar 30 17:15:54 2011 From: jed at 59A2.org (Jed Brown) Date: Thu, 31 Mar 2011 01:15:54 +0300 Subject: [petsc-users] Ghosted vectors in KSP via SNES In-Reply-To: References: Message-ID: On Wed, Mar 30, 2011 at 18:43, I-Kamenetskiy, Dmitry S < dmitry.s.kamenetskiy at boeing.com> wrote: > I have run in to the following issue with the PETSc SNES usage. The logic > of the SNESSolve() routine in my code is that the solution vector x should > include ghost points at the boundary of subdomains in the distributed case. Not true, the solvers take "global" vectors, each degree of freedom appears exactly once on exactly one process. Ghost points are typically needed inside of residual evaluation and Jacobian assembly, but they should not be present in the vector you give to the solver. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Wed Mar 30 17:25:38 2011 From: jed at 59A2.org (Jed Brown) Date: Thu, 31 Mar 2011 01:25:38 +0300 Subject: [petsc-users] schur complement and condition number In-Reply-To: References: Message-ID: 2011/3/30 Gong Ding > I know the bad condition number comes from a lumped boundary. > For this boundary, the sum of flux comes from all the boundary elements > shoule equals to a fixed given flux. > Sounds like your problem has a one dimensional null space, have you tried setting the null space? (See the user's manual section on solving singular systems or docs for KSPSetNullSpace().) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Thu Mar 31 01:26:06 2011 From: jed at 59A2.org (Jed Brown) Date: Thu, 31 Mar 2011 09:26:06 +0300 Subject: [petsc-users] schur complement and condition number In-Reply-To: <21436891.30111301547569847.JavaMail.coremail@mail.ustc.edu> References: <21436891.30111301547569847.JavaMail.coremail@mail.ustc.edu> Message-ID: On Thu, Mar 31, 2011 at 07:59, Gong Ding wrote: > No, not a singular system. Just bad conditioned. > I mean if I can eliminate the bad conditioned rows by building schur > complement? > How are you enforcing the integral constraint in the boundary conditions? There is probably a way to write your boundary condition so that it is well scaled. The Schur complement of your strip is an awkward thing to compute and will significantly affect sparsity unless that part of the matrix is already diagonal in which case there are good ways to scale correctly during assembly or eliminate a-priori, or have PCREDISTRIBUTE do it automatically. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at 59A2.org Thu Mar 31 06:15:21 2011 From: jed at 59A2.org (Jed Brown) Date: Thu, 31 Mar 2011 14:15:21 +0300 Subject: [petsc-users] schur complement and condition number In-Reply-To: <27252330.30911301568255813.JavaMail.coremail@mail.ustc.edu> References: <21436891.30111301547569847.JavaMail.coremail@mail.ustc.edu> <27252330.30911301568255813.JavaMail.coremail@mail.ustc.edu> Message-ID: On Thu, Mar 31, 2011 at 13:44, Gong Ding wrote: > This afternoon, I installed slepc for computing the singualr value > (condition number) of the jacobian matrix. > However, slepc SVD solver broken for smallest singualr value. > Fortunately, superlu gives an approximate condition number, which in the > order of 1e22. > If I remove the connection, condition number is in the order of 1e10. > A condition number larger than 1e12 or so usually indicates that the matrix is singular. You could compile PETSc (dev) using gcc-4.6 and configuring --with-precision=__float128. External packages won't work this way, but you can use KSP to estimate your singular values more accurately and to solve a nearly singular system. > > Is it possible to use Tikhonov regularization (A^TA + sigma I)x = A^Tb with > this problem? > Of course, I need to do it during the nonlinear iteration. That is a > Newton-Tikhonov method. > Sure, you can regularize the Jacobian however you like when using SNES. > Or I have to use direct solver without other choice? > Was the direct solver producing a good solution? With such an ill-conditioned system, the solution could be incorrect in the first digit. I would at least check with __float128. -------------- next part -------------- An HTML attachment was scrubbed... URL: From SJ_Ormiston at UManitoba.ca Thu Mar 31 06:57:27 2011 From: SJ_Ormiston at UManitoba.ca (Ormiston, Scott J.) Date: Thu, 31 Mar 2011 06:57:27 -0500 Subject: [petsc-users] Parallel direct solver Message-ID: <4D946C27.5090704@UManitoba.ca> I need to be able to run a parallel version of a direct solver. Is there a parallel direct solver in PETSc? My understanding is that the direct solver in PETSc is only sequential. If there is no parallel direct solver in PETSc, what should I use? I have seen some people recommend SuperLU. This seems to be a totally separate package from PETSc with its own syntax and setup. Therefore, I must start from scratch with the SuperLU setup and I cannot take advantage of the parallel matrix setup that I have already done to solve my problem with a ksp solver with PETSc. Is this my only option? Scott Ormiston -------------- next part -------------- A non-text attachment was scrubbed... Name: SJ_Ormiston.vcf Type: text/x-vcard Size: 321 bytes Desc: not available URL: From jed at 59A2.org Thu Mar 31 07:09:20 2011 From: jed at 59A2.org (Jed Brown) Date: Thu, 31 Mar 2011 15:09:20 +0300 Subject: [petsc-users] Parallel direct solver In-Reply-To: <4D946C27.5090704@UManitoba.ca> References: <4D946C27.5090704@UManitoba.ca> Message-ID: On Thu, Mar 31, 2011 at 14:57, Ormiston, Scott J. wrote: > If there is no parallel direct solver in PETSc, what should I use? I have > seen some people recommend SuperLU. This seems to be a totally separate > package from PETSc with its own syntax and setup. Therefore, I must start > from scratch with the SuperLU setup and I cannot take advantage of the > parallel matrix setup that I have already done to solve my problem with a > ksp solver with PETSc. Hey now, we're not sadists. ;-) First configure PETSc with --download-superlu_dist or --download-mumps (or install these packages yourself and use --with-superlu_dist-dir=, see the output from ./configure --help for details). Then just run your program with -pc_type lu -pc_factor_mat_solver_type superlu_dist, no need to modify any code. -------------- next part -------------- An HTML attachment was scrubbed... URL: From loic.gouarin at math.u-psud.fr Thu Mar 31 08:56:28 2011 From: loic.gouarin at math.u-psud.fr (gouarin) Date: Thu, 31 Mar 2011 15:56:28 +0200 Subject: [petsc-users] extract Mat data Message-ID: <4D94880C.4000102@math.u-psud.fr> Hello, I need to extract a pseudo CSR format from a PETSc Mat. It's easy with MATSEQ, I use: MatGetRowIJ(A, 1, PETSC_FALSE, PETSC_FALSE, &(Acsr.n), &(Acsr.ia), &(Acsr.ja), &done); MatGetArray(A, &(Acsr.a)); I want to do the same for MATMPI. This is what I do: MatGetLocalMat(A,MAT_INITIAL_MATRIX, &Aloc); MatGetRowIJ(Aloc, 1, PETSC_FALSE, PETSC_FALSE, &(Acsr.n), &(Acsr.ia), &(Acsr.ja), &done); MatGetArray(Aloc, &(Acsr.a)); Now, I need to know on what process the non local points are. Is there a petsc function that gives this partition ? Best regards, Loic From hzhang at mcs.anl.gov Thu Mar 31 09:48:14 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Thu, 31 Mar 2011 09:48:14 -0500 Subject: [petsc-users] extract Mat data In-Reply-To: <4D94880C.4000102@math.u-psud.fr> References: <4D94880C.4000102@math.u-psud.fr> Message-ID: Loic, > > I want to do the same for MATMPI. This is what I do: > > MatGetLocalMat(A,MAT_INITIAL_MATRIX, &Aloc); > MatGetRowIJ(Aloc, 1, PETSC_FALSE, PETSC_FALSE, &(Acsr.n), &(Acsr.ia), > &(Acsr.ja), &done); > MatGetArray(Aloc, &(Acsr.a)); > > Now, I need to know on what process the non local points are. Each process get its own local submatrix. See http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatCreateMPIAIJ.html on data structure of MPIAIJ format. Applying your statements above to the example listed on the page (a 8x8 matrix), Proc0 will get Aloc=A(0:2,:), ..., Porc2 gets Aloc=A(6:7,:). > > Is there a petsc function that gives this partition ? MatGetOwnershipRange() Hong > From loic.gouarin at math.u-psud.fr Thu Mar 31 10:10:57 2011 From: loic.gouarin at math.u-psud.fr (gouarin) Date: Thu, 31 Mar 2011 17:10:57 +0200 Subject: [petsc-users] extract Mat data In-Reply-To: References: <4D94880C.4000102@math.u-psud.fr> Message-ID: <4D949981.5070206@math.u-psud.fr> Thanks for the reply. On 31/03/2011 16:48, Hong Zhang wrote: > Loic, > >> I want to do the same for MATMPI. This is what I do: >> >> MatGetLocalMat(A,MAT_INITIAL_MATRIX,&Aloc); >> MatGetRowIJ(Aloc, 1, PETSC_FALSE, PETSC_FALSE,&(Acsr.n),&(Acsr.ia), >> &(Acsr.ja),&done); >> MatGetArray(Aloc,&(Acsr.a)); >> >> Now, I need to know on what process the non local points are. > Each process get its own local submatrix. See > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatCreateMPIAIJ.html > on data structure of MPIAIJ format. Applying your statements above to > the example listed on the page (a 8x8 matrix), Proc0 will get > Aloc=A(0:2,:), ..., Porc2 gets Aloc=A(6:7,:). >> Is there a petsc function that gives this partition ? > MatGetOwnershipRange() I use this function to know the local points index. In the same way, with this function I know the non local index. But this function doesn't give the process. I give you an example. I have this matrix: 1 0 | 0 1 1 2 | 1 0 ----------- 0 0 | 1 0 0 2 | 0 1 I want this kind of format for proc 0 a = {1,1,1,2,1} ia = {0,2,5} ja = {0,3,0,1,2} rank = {0,0,1,1} for proc 1 a = {1,2,1} ia = {0,1,3} ja = {0,2,1} rank = {1,1,0} rank give me the process for each non zero column. Local rows should have numbers 1,...N and offdiagonal entries with j>N. I suppose that I have no information about the structure of MPIAIJ. Loic From alejandro.aragon at gmail.com Thu Mar 31 10:19:26 2011 From: alejandro.aragon at gmail.com (=?iso-8859-1?Q?Alejandro_Marcos_Arag=F3n?=) Date: Thu, 31 Mar 2011 17:19:26 +0200 Subject: [petsc-users] strange and slow behavior when using KSPSolve Message-ID: <655FF089-32C9-49F2-900E-8F9855F4FEDB@gmail.com> Hi all, I am currently using PETSc to solve a system of linear equations in parallel, so I created a wrapper class that interfaces my C++ code with the PETSc library. Since the problem I'm trying to solve is nonlinear, I assemble the matrix once and I reuse the structure (the sparsity pattern is the same). The problem is that after a few times of solving the system very fast, it suddenly takes a considerable amount of time and I really don't understand why this is happening. This is an extract of the code showing the timing issue that I have just before solving the system with 2 processes: [0] Inside PETSc solver:7.15256e-06 s [0] Got ownership range... 3.21865e-05 s Inside PETSc solver:1.5974e-05 s [1] Got ownership range... 4.00543e-05 s [1] Zeroed-out sparse matrix entries... [0] Zeroed-out sparse matrix entries... 0.000444174 s 0.000442982 s [1] Filled sparse matrix... 0.0772262 s [0] Filled sparse matrix... 0.081254 s [1] Sparse matrix assembled... 0.0897892 s [0] Sparse matrix assembled... 0.0954142 s [0] Vectors created... 0.0955811 s[1] Vectors created... 0.095592 s [0] Right hand side set... 0.0965121 s [1] Right hand side set... 0.0965312 s [0] Right-hand-side vector assembled... 0.096647 s [1] Right-hand-side vector assembled... 0.096657 s [0] System solved... 32.6188 s [1] System solved... 32.6188 s [0] Solution vector scattered... [1] Solution vector scattered... 32.6192 s 32.6192 s [1] Solution vector copied... 32.6215[0] Solution vector copied... 32.6215 s s [1] Temporary data structures destroyed... 32.6216 s[0] Temporary data structures destroyed... 32.6216 s So you can see that it takes a considerable amount of time to solve the system at this point, when usually this is what I get: [0] Right-hand-side vector assembled... 0.089427 s [1] Right-hand-side vector assembled... 0.089438 s [0] System solved... 0.383509 s [1] System solved... 0.38352 s Does anyone know why this is happening? Thanks in advance for any insight on this. Alejandro M. Arag?n P.S.: The code I use is given below: struct PETSc_solver { typedef doVERBOSEuble value_type; typedef sparse_vector sparse_vector_type; typedef sparse_matrix sparse_matrix_type; int argc_; char **args_; Mat A_; //!< linear system matrix Vec x_; //!< Solution vector KSP ksp_; //!< linear solver context bool allocated_; PETSc_solver() : argc_(), args_(NULL), allocated_(false) { PetscInitialize(&argc_,&args_,NULL,NULL); PetscErrorCode ierr = MatCreate(PETSC_COMM_WORLD,&A_);CHKERR(ierr); ierr = KSPCreate(PETSC_COMM_WORLD,&ksp_);CHKERR(ierr); } PETSc_solver(int argc, char** args) : argc_(argc), args_(args) { PetscInitialize(&argc_,&args_,NULL,NULL); PetscErrorCode ierr = MatCreate(PETSC_COMM_WORLD,&A_);CHKERR(ierr); ierr = KSPCreate(PETSC_COMM_WORLD,&ksp_);CHKERR(ierr); } //! Overload operator() to solve system of linear equations sparse_vector_type operator()(const sparse_matrix_type& AA, const sparse_vector_type& bb); ~PETSc_solver() { PetscErrorCode ierr = MatDestroy(A_);CHKERR(ierr); ierr = KSPDestroy(ksp_);CHKERR(ierr); ierr = VecDestroy(x_);CHKERR(ierr); } }; PETSc_solver::sparse_vector_type PETSc_solver::operator()(const PETSc_solver::sparse_matrix_type& AA, const PETSc_solver::sparse_vector_type& bb) { assert(AA.rows() == bb.size()); Vec x,b; /* approx solution, RHS */ PetscInt Istart,Iend; PetscErrorCode ierr; #ifdef VERBOSE cpputils::ctimer timer; MPIPRINT("Inside PETSc solver:" ); cout< nz = AA.non_zero_off_diagonal(); ierr = MatMPIAIJSetPreallocation(A_, nz.first, PETSC_NULL, nz.second, PETSC_NULL); CHKERR(ierr); ierr = MatSeqAIJSetPreallocation(A_, nz.first, PETSC_NULL); CHKERR(ierr); #ifdef VERBOSE cout<<"Ok! "< subs = AA.unhash(it->first); int row = subs.first; int col = subs.second; ierr = MatSetValues(A_, 1, &row, 1, &col, &it->second, ADD_VALUES);CHKERR(ierr); } #ifdef VERBOSE MPIPRINT(" Filled sparse matrix... "); cout<first; ierr = VecSetValues(b, 1, &row, &it->second, ADD_VALUES); } #ifdef VERBOSE MPIPRINT(" Right hand side set... "); cout<first; ierr = VecGetValues(x_all, 1, &row, &xx[row]); } #ifdef VERBOSE MPIPRINT(" Solution vector copied... "); cout< From bsmith at mcs.anl.gov Thu Mar 31 11:04:45 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 31 Mar 2011 11:04:45 -0500 Subject: [petsc-users] strange and slow behavior when using KSPSolve In-Reply-To: <655FF089-32C9-49F2-900E-8F9855F4FEDB@gmail.com> References: <655FF089-32C9-49F2-900E-8F9855F4FEDB@gmail.com> Message-ID: One thing that pops up immediately (though it shouldn't normally be a big deal). Create the VecScatterCreateToAll() only once up at the top, like the KSP. You can run with -malloc and add PetscMallocDump() at the end of the each outer loop to see if more and more memory is being used. After the KSPSolve(), call KSPGetIterationNumber() to see if the number of iterations required is getting much worse with later solves. Also call KSPGetConvergedReason() and KSPGetResidualNorm() to see if it is oversolving or undersolving the linear system relative to earlier solves. Barry Barry On Mar 31, 2011, at 10:19 AM, Alejandro Marcos Arag?n wrote: > Hi all, > > I am currently using PETSc to solve a system of linear equations in parallel, so I created a wrapper class that interfaces my C++ code with the PETSc library. > > Since the problem I'm trying to solve is nonlinear, I assemble the matrix once and I reuse the structure (the sparsity pattern is the same). The problem is that after a few times of solving the system very fast, it suddenly takes a considerable amount of time and I really don't understand why this is happening. > > This is an extract of the code showing the timing issue that I have just before solving the system with 2 processes: > > > [0] Inside PETSc solver:7.15256e-06 s > [0] Got ownership range... 3.21865e-05 s > Inside PETSc solver:1.5974e-05 s > [1] Got ownership range... 4.00543e-05 s > [1] Zeroed-out sparse matrix entries... [0] Zeroed-out sparse matrix entries... 0.000444174 s > 0.000442982 s > [1] Filled sparse matrix... 0.0772262 s > [0] Filled sparse matrix... 0.081254 s > [1] Sparse matrix assembled... 0.0897892 s > [0] Sparse matrix assembled... 0.0954142 s > [0] Vectors created... 0.0955811 s[1] Vectors created... 0.095592 s > > [0] Right hand side set... 0.0965121 s > [1] Right hand side set... 0.0965312 s > [0] Right-hand-side vector assembled... 0.096647 s > [1] Right-hand-side vector assembled... 0.096657 s > [0] System solved... 32.6188 s > [1] System solved... 32.6188 s > [0] Solution vector scattered... [1] Solution vector scattered... 32.6192 s > 32.6192 s > [1] Solution vector copied... 32.6215[0] Solution vector copied... 32.6215 s > s > [1] Temporary data structures destroyed... 32.6216 s[0] Temporary data structures destroyed... 32.6216 s > > > > So you can see that it takes a considerable amount of time to solve the system at this point, when usually this is what I get: > > [0] Right-hand-side vector assembled... 0.089427 s > [1] Right-hand-side vector assembled... 0.089438 s > [0] System solved... 0.383509 s > [1] System solved... 0.38352 s > > > Does anyone know why this is happening? Thanks in advance for any insight on this. > > Alejandro M. Arag?n > > P.S.: The code I use is given below: > > struct PETSc_solver { > > typedef doVERBOSEuble value_type; > typedef sparse_vector sparse_vector_type; > typedef sparse_matrix sparse_matrix_type; > > int argc_; > char **args_; > Mat A_; //!< linear system matrix > Vec x_; //!< Solution vector > KSP ksp_; //!< linear solver context > bool allocated_; > > PETSc_solver() : argc_(), args_(NULL), allocated_(false) { > > PetscInitialize(&argc_,&args_,NULL,NULL); > PetscErrorCode ierr = MatCreate(PETSC_COMM_WORLD,&A_);CHKERR(ierr); > ierr = KSPCreate(PETSC_COMM_WORLD,&ksp_);CHKERR(ierr); > } > PETSc_solver(int argc, char** args) : argc_(argc), args_(args) { > > PetscInitialize(&argc_,&args_,NULL,NULL); > PetscErrorCode ierr = MatCreate(PETSC_COMM_WORLD,&A_);CHKERR(ierr); > ierr = KSPCreate(PETSC_COMM_WORLD,&ksp_);CHKERR(ierr); > } > > //! Overload operator() to solve system of linear equations > sparse_vector_type operator()(const sparse_matrix_type& AA, const sparse_vector_type& bb); > > ~PETSc_solver() { > > PetscErrorCode ierr = MatDestroy(A_);CHKERR(ierr); > ierr = KSPDestroy(ksp_);CHKERR(ierr); > ierr = VecDestroy(x_);CHKERR(ierr); > } > }; > > > > PETSc_solver::sparse_vector_type PETSc_solver::operator()(const PETSc_solver::sparse_matrix_type& AA, const PETSc_solver::sparse_vector_type& bb) { > > assert(AA.rows() == bb.size()); > > Vec x,b; /* approx solution, RHS */ > PetscInt Istart,Iend; > PetscErrorCode ierr; > > > #ifdef VERBOSE > cpputils::ctimer timer; > MPIPRINT("Inside PETSc solver:" ); > cout< #endif > > ierr = MatSetFromOptions(A_);CHKERR(ierr); > > if (!allocated_) { > > #ifdef VERBOSE > MPIPRINT(" Allocating memory... "); > cout< #endif > > > ierr = MatSetSizes(A_,PETSC_DECIDE,PETSC_DECIDE, AA.rows(), AA.columns());CHKERR(ierr); > > // get number of non-zeros in both diagonal and non-diagonal portions of the matrix > std::pair nz = AA.non_zero_off_diagonal(); > ierr = MatMPIAIJSetPreallocation(A_, nz.first, PETSC_NULL, nz.second, PETSC_NULL); CHKERR(ierr); > ierr = MatSeqAIJSetPreallocation(A_, nz.first, PETSC_NULL); CHKERR(ierr); > > #ifdef VERBOSE > cout<<"Ok! "< #endif > } > > /* > Currently, all PETSc parallel matrix formats are partitioned by > contiguous chunks of rows across the processors. Determine which > rows of the matrix are locally owned. > */ > ierr = MatGetOwnershipRange(A_,&Istart,&Iend);CHKERR(ierr); > > #ifdef VERBOSE > MPIPRINT(" Got ownership range... "); > cout< #endif > > // zero-out matrix > MatZeroEntries(A_); > > #ifdef VERBOSE > MPIPRINT(" Zeroed-out sparse matrix entries... "); > cout< #endif > > for (sparse_matrix_type::const_hash_iterator it = AA.map_.begin(); it != AA.map_.end(); ++it) { > > // get subscripts > std::pair subs = AA.unhash(it->first); > int row = subs.first; > int col = subs.second; > ierr = MatSetValues(A_, 1, &row, 1, &col, &it->second, ADD_VALUES);CHKERR(ierr); > } > > #ifdef VERBOSE > MPIPRINT(" Filled sparse matrix... "); > cout< #endif > > /* > Assemble matrix, using the 2-step process: > MatAssemblyBegin(), MatAssemblyEnd() > Computations can be done while messages are in transition > by placing code between these two statements. > */ > ierr = MatAssemblyBegin(A_,MAT_FINAL_ASSEMBLY);CHKERR(ierr); > ierr = MatAssemblyEnd(A_,MAT_FINAL_ASSEMBLY);CHKERR(ierr); > > #ifdef VERBOSE > MPIPRINT(" Sparse matrix assembled... "); > cout< #endif > > ierr = MatSetOption(A_, MAT_NEW_NONZERO_LOCATIONS, PETSC_FALSE);CHKERR(ierr); > > > /* Create parallel vectors. */ > ierr = VecCreate(PETSC_COMM_WORLD,&b);CHKERR(ierr); > ierr = VecSetSizes(b,PETSC_DECIDE, bb.size());CHKERR(ierr); > ierr = VecSetFromOptions(b);CHKERR(ierr); > > if (!allocated_) { > > ierr = VecDuplicate(b,&x_);CHKERR(ierr); > > // set flag to true > allocated_ = true; > } > > VecZeroEntries(x_); > > #ifdef VERBOSE > MPIPRINT(" Vectors created... "); > cout< #endif > > /* Set hight-hand-side vector */ > for (sparse_vector_type::const_hash_iterator it = bb.map_.begin(); it != bb.map_.end(); ++it) { > int row = it->first; > ierr = VecSetValues(b, 1, &row, &it->second, ADD_VALUES); > } > > #ifdef VERBOSE > MPIPRINT(" Right hand side set... "); > cout< #endif > > /* > Assemble vector, using the 2-step process: > VecAssemblyBegin(), VecAssemblyEnd() > Computations can be done while messages are in transition > by placing code between these two statements. > */ > ierr = VecAssemblyBegin(b);CHKERR(ierr); > ierr = VecAssemblyEnd(b);CHKERR(ierr); > > #ifdef VERBOSE > MPIPRINT(" Right-hand-side vector assembled... "); > cout< #endif > > /* > Set operators. Here the matrix that defines the linear system > also serves as the preconditioning matrix. > */ > ierr = KSPSetOperators(ksp_,A_,A_,SAME_NONZERO_PATTERN);CHKERR(ierr); > > > /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > Solve the linear system > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */ > ierr = KSPSolve(ksp_,b,x_);CHKERR(ierr); > > #ifdef VERBOSE > MPIPRINT(" System solved... "); > cout< #endif > > > /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > Check solution and clean up > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */ > > VecScatter ctx; > Vec x_all; > > ierr = VecScatterCreateToAll(x_, &ctx, &x_all);CHKERR(ierr); > ierr = VecScatterBegin(ctx,x_,x_all,INSERT_VALUES,SCATTER_FORWARD);CHKERR(ierr); > ierr = VecScatterEnd(ctx,x_,x_all,INSERT_VALUES,SCATTER_FORWARD);CHKERR(ierr); > > #ifdef VERBOSE > MPIPRINT(" Solution vector scattered... "); > cout< #endif > > /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > Get values from solution and store them in the object that will be > returned > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */ > > sparse_vector_type xx(bb.size()); > > /* Set solution vector */ > for (sparse_vector_type::const_hash_iterator it = bb.map_.begin(); it != bb.map_.end(); ++it) { > int row = it->first; > ierr = VecGetValues(x_all, 1, &row, &xx[row]); > } > > #ifdef VERBOSE > MPIPRINT(" Solution vector copied... "); > cout< #endif > > ierr = VecDestroy(b);CHKERR(ierr); > ierr = VecDestroy(x_all);CHKERR(ierr); > > #ifdef VERBOSE > MPIPRINT(" Temporary data structures destroyed... "); > cout< #endif > > return xx; > } > From knepley at gmail.com Thu Mar 31 12:06:49 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 31 Mar 2011 12:06:49 -0500 Subject: [petsc-users] extract Mat data In-Reply-To: <4D949981.5070206@math.u-psud.fr> References: <4D94880C.4000102@math.u-psud.fr> <4D949981.5070206@math.u-psud.fr> Message-ID: On Thu, Mar 31, 2011 at 10:10 AM, gouarin wrote: > Thanks for the reply. > > > On 31/03/2011 16:48, Hong Zhang wrote: > >> Loic, >> >> I want to do the same for MATMPI. This is what I do: >>> >>> MatGetLocalMat(A,MAT_INITIAL_MATRIX,&Aloc); >>> MatGetRowIJ(Aloc, 1, PETSC_FALSE, PETSC_FALSE,&(Acsr.n),&(Acsr.ia), >>> &(Acsr.ja),&done); >>> MatGetArray(Aloc,&(Acsr.a)); >>> >>> Now, I need to know on what process the non local points are. >>> >> Each process get its own local submatrix. See >> >> http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatCreateMPIAIJ.html >> on data structure of MPIAIJ format. Applying your statements above to >> the example listed on the page (a 8x8 matrix), Proc0 will get >> Aloc=A(0:2,:), ..., Porc2 gets Aloc=A(6:7,:). >> >>> Is there a petsc function that gives this partition ? >>> >> MatGetOwnershipRange() >> > I use this function to know the local points index. In the same way, with > this function I know the non local index. > > But this function doesn't give the process. I give you an example. I have > this matrix: > > 1 0 | 0 1 > 1 2 | 1 0 > ----------- > 0 0 | 1 0 > 0 2 | 0 1 > > I want this kind of format > > for proc 0 > a = {1,1,1,2,1} > ia = {0,2,5} > ja = {0,3,0,1,2} > rank = {0,0,1,1} > > for proc 1 > a = {1,2,1} > ia = {0,1,3} > ja = {0,2,1} > rank = {1,1,0} > > rank give me the process for each non zero column. Local rows should have > numbers 1,...N and offdiagonal entries with j>N. > The ownership range is the row partition. However, MPIAIJ matrices are not stored as you indicate above. Different storage formats make the process you describe fragile. WHy do you need the CSR form of the matrix? If you really do, use MatGetSubMatrix() to get a SEQAIJ matrix on each process and pull out the values. Matt > I suppose that I have no information about the structure of MPIAIJ. > > Loic > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From loic.gouarin at math.u-psud.fr Thu Mar 31 12:38:12 2011 From: loic.gouarin at math.u-psud.fr (gouarin) Date: Thu, 31 Mar 2011 19:38:12 +0200 Subject: [petsc-users] extract Mat data In-Reply-To: References: <4D94880C.4000102@math.u-psud.fr> <4D949981.5070206@math.u-psud.fr> Message-ID: <4D94BC04.7060301@math.u-psud.fr> On 31/03/2011 19:06, Matthew Knepley wrote: > On Thu, Mar 31, 2011 at 10:10 AM, gouarin > wrote: > > Thanks for the reply. > > > On 31/03/2011 16 :48, Hong Zhang wrote: > > Loic, > > I want to do the same for MATMPI. This is what I do: > > MatGetLocalMat(A,MAT_INITIAL_MATRIX,&Aloc); > MatGetRowIJ(Aloc, 1, PETSC_FALSE, > PETSC_FALSE,&(Acsr.n),&(Acsr.ia), > &(Acsr.ja),&done); > MatGetArray(Aloc,&(Acsr.a)); > > Now, I need to know on what process the non local points are. > > Each process get its own local submatrix. See > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatCreateMPIAIJ.html > on data structure of MPIAIJ format. Applying your statements > above to > the example listed on the page (a 8x8 matrix), Proc0 will get > Aloc=A(0:2,:), ..., Porc2 gets Aloc=A(6:7,:). > > Is there a petsc function that gives this partition ? > > MatGetOwnershipRange() > > I use this function to know the local points index. In the same > way, with this function I know the non local index. > > But this function doesn't give the process. I give you an example. > I have this matrix: > > 1 0 | 0 1 > 1 2 | 1 0 > ----------- > 0 0 | 1 0 > 0 2 | 0 1 > > I want this kind of format > > for proc 0 > a = {1,1,1,2,1} > ia = {0,2,5} > ja = {0,3,0,1,2} > rank = {0,0,1,1} > > for proc 1 > a = {1,2,1} > ia = {0,1,3} > ja = {0,2,1} > rank = {1,1,0} > > rank give me the process for each non zero column. Local rows > should have numbers 1,...N and offdiagonal entries with j>N. > > > The ownership range is the row partition. However, MPIAIJ matrices are > not stored as you indicate above. > > Different storage formats make the process you describe fragile. WHy > do you need the CSR form of the matrix? If you really > do, use MatGetSubMatrix() to get a SEQAIJ matrix on each process and > pull out the values. > I want to use this format because I try to create a new multigrid PC with this method: http://homepages.ulb.ac.be/~ynotay/AGMG/userguide/index.html I don't understand why it's not possible to have the process partition. When you create a MPIAIJ matrix and you use a KSP to solve the system, you know what values to exchange. No? Where is this information ? I can also use MatGetOwnershipRange() and MPI_Allgather to have this information. Loic > Matt > > I suppose that I have no information about the structure of MPIAIJ. > > Loic > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -- Loic Gouarin Laboratoire de Math?matiques Universit? Paris-Sud B?timent 425 91405 Orsay Cedex France Tel: (+33) 1 69 15 60 14 Fax: (+33) 1 69 15 67 18 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Mar 31 12:59:15 2011 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 31 Mar 2011 12:59:15 -0500 Subject: [petsc-users] extract Mat data In-Reply-To: <4D94BC04.7060301@math.u-psud.fr> References: <4D94880C.4000102@math.u-psud.fr> <4D949981.5070206@math.u-psud.fr> <4D94BC04.7060301@math.u-psud.fr> Message-ID: On Thu, Mar 31, 2011 at 12:38 PM, gouarin wrote: > On 31/03/2011 19:06, Matthew Knepley wrote: > > On Thu, Mar 31, 2011 at 10:10 AM, gouarin wrote: > > Thanks for the reply. > > > On 31/03/2011 16:48, Hong Zhang wrote: > > Loic, > > I want to do the same for MATMPI. This is what I do: > > MatGetLocalMat(A,MAT_INITIAL_MATRIX,&Aloc); > MatGetRowIJ(Aloc, 1, PETSC_FALSE, PETSC_FALSE,&(Acsr.n),&(Acsr.ia), > &(Acsr.ja),&done); > MatGetArray(Aloc,&(Acsr.a)); > > Now, I need to know on what process the non local points are. > > Each process get its own local submatrix. See > > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatCreateMPIAIJ.html > on data structure of MPIAIJ format. Applying your statements above to > the example listed on the page (a 8x8 matrix), Proc0 will get > Aloc=A(0:2,:), ..., Porc2 gets Aloc=A(6:7,:). > > Is there a petsc function that gives this partition ? > > MatGetOwnershipRange() > > I use this function to know the local points index. In the same way, with > this function I know the non local index. > > But this function doesn't give the process. I give you an example. I have > this matrix: > > 1 0 | 0 1 > 1 2 | 1 0 > ----------- > 0 0 | 1 0 > 0 2 | 0 1 > > I want this kind of format > > for proc 0 > a = {1,1,1,2,1} > ia = {0,2,5} > ja = {0,3,0,1,2} > rank = {0,0,1,1} > > for proc 1 > a = {1,2,1} > ia = {0,1,3} > ja = {0,2,1} > rank = {1,1,0} > > rank give me the process for each non zero column. Local rows should have > numbers 1,...N and offdiagonal entries with j>N. > > > The ownership range is the row partition. However, MPIAIJ matrices are > not stored as you indicate above. > > Different storage formats make the process you describe fragile. WHy do > you need the CSR form of the matrix? If you really > do, use MatGetSubMatrix() to get a SEQAIJ matrix on each process and pull > out the values. > > I want to use this format because I try to create a new multigrid PC > with this method: > > http://homepages.ulb.ac.be/~ynotay/AGMG/userguide/index.html > > I don't understand why it's not possible to have the process partition. > When you create a MPIAIJ matrix and you use a KSP to solve the system, you > know what values to exchange. No? Where is this information ? > 1) It is possible to get the partition of rows across processes. This is EXACTLY what MatGetOwnershipRange() returns, as I said in the last email. 2) As I explained, matrices can be stored in a variety of formats. The MPIAIJ format stores 2 local matrices per partition in order to overlap communication with computation. I can also use MatGetOwnershipRange() and MPI_Allgather to have this > information. > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatGetOwnershipRanges.html Matt > Loic > > > > > Matt > > > I suppose that I have no information about the structure of MPIAIJ. > > Loic > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > -- > Loic Gouarin > Laboratoire de Math?matiques > Universit? Paris-Sud > B?timent 425 > 91405 Orsay Cedex > France > Tel: (+33) 1 69 15 60 14 > Fax: (+33) 1 69 15 67 18 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Mar 31 13:05:01 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 31 Mar 2011 13:05:01 -0500 Subject: [petsc-users] extract Mat data In-Reply-To: <4D94BC04.7060301@math.u-psud.fr> References: <4D94880C.4000102@math.u-psud.fr> <4D949981.5070206@math.u-psud.fr> <4D94BC04.7060301@math.u-psud.fr> Message-ID: <94CAE4FB-389D-4891-8165-AB7189C8239B@mcs.anl.gov> On Mar 31, 2011, at 12:38 PM, gouarin wrote: > On 31/03/2011 19:06, Matthew Knepley wrote: >> On Thu, Mar 31, 2011 at 10:10 AM, gouarin wrote: >> Thanks for the reply. >> >> >> On 31/03/2011 16:48, Hong Zhang wrote: >> Loic, >> >> I want to do the same for MATMPI. This is what I do: >> >> MatGetLocalMat(A,MAT_INITIAL_MATRIX,&Aloc); >> MatGetRowIJ(Aloc, 1, PETSC_FALSE, PETSC_FALSE,&(Acsr.n),&(Acsr.ia), >> &(Acsr.ja),&done); >> MatGetArray(Aloc,&(Acsr.a)); >> >> Now, I need to know on what process the non local points are. >> Each process get its own local submatrix. See >> http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatCreateMPIAIJ.html >> on data structure of MPIAIJ format. Applying your statements above to >> the example listed on the page (a 8x8 matrix), Proc0 will get >> Aloc=A(0:2,:), ..., Porc2 gets Aloc=A(6:7,:). >> Is there a petsc function that gives this partition ? >> MatGetOwnershipRange() >> I use this function to know the local points index. In the same way, with this function I know the non local index. >> >> But this function doesn't give the process. I give you an example. I have this matrix: >> >> 1 0 | 0 1 >> 1 2 | 1 0 >> ----------- >> 0 0 | 1 0 >> 0 2 | 0 1 >> >> I want this kind of format >> >> for proc 0 >> a = {1,1,1,2,1} >> ia = {0,2,5} >> ja = {0,3,0,1,2} >> rank = {0,0,1,1} >> >> for proc 1 >> a = {1,2,1} >> ia = {0,1,3} >> ja = {0,2,1} >> rank = {1,1,0} >> >> rank give me the process for each non zero column. Local rows should have numbers 1,...N and offdiagonal entries with j>N. >> >> The ownership range is the row partition. However, MPIAIJ matrices are not stored as you indicate above. >> >> Different storage formats make the process you describe fragile. WHy do you need the CSR form of the matrix? If you really >> do, use MatGetSubMatrix() to get a SEQAIJ matrix on each process and pull out the values. >> > I want to use this format because I try to create a new multigrid PC with this method: > > http://homepages.ulb.ac.be/~ynotay/AGMG/userguide/index.html > > I don't understand why it's not possible to have the process partition. When you create a MPIAIJ matrix and you use a KSP to solve the system, you know what values to exchange. No? Where is this information ? The point is not that the information is not available, it is just that it is not trivially available since we don't use the data in that format. Here is what you have to do. Include "src/mat/impls/aij/mpi/mpiaij.h" directly in your interface files and access the data directly, do not try to find some PETSc function that provides exactly what you want directly, that cannot exist because we cannot know what format each other package wants. In particular inside Mat_MPIAJ are the two submatrices called A and B which are Mat_SeqAIJ matrices. The data in #if defined (PETSC_USE_CTABLE) PetscTable colmap; #else PetscInt *colmap; /* local col number of off-diag col */ #endif PetscInt *garray; /* global index of all off-processor columns */ provides the information about the "off process" columns for the matrices stored in B. But you will have to understand the data structures somewhat in order to get the data in the form you want. Barry > > I can also use MatGetOwnershipRange() and MPI_Allgather to have this information. > > Loic > > > > >> Matt >> >> I suppose that I have no information about the structure of MPIAIJ. >> >> Loic >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener > > > -- > Loic Gouarin > Laboratoire de Math?matiques > Universit? Paris-Sud > B?timent 425 > 91405 Orsay Cedex > France > Tel: (+33) 1 69 15 60 14 > Fax: (+33) 1 69 15 67 18 > From loic.gouarin at math.u-psud.fr Thu Mar 31 13:15:23 2011 From: loic.gouarin at math.u-psud.fr (gouarin) Date: Thu, 31 Mar 2011 20:15:23 +0200 Subject: [petsc-users] extract Mat data In-Reply-To: <94CAE4FB-389D-4891-8165-AB7189C8239B@mcs.anl.gov> References: <4D94880C.4000102@math.u-psud.fr> <4D949981.5070206@math.u-psud.fr> <4D94BC04.7060301@math.u-psud.fr> <94CAE4FB-389D-4891-8165-AB7189C8239B@mcs.anl.gov> Message-ID: <4D94C4BB.1060009@math.u-psud.fr> On 31/03/2011 20:05, Barry Smith wrote: > On Mar 31, 2011, at 12:38 PM, gouarin wrote: > >> On 31/03/2011 19:06, Matthew Knepley wrote: >>> On Thu, Mar 31, 2011 at 10:10 AM, gouarin wrote: >>> Thanks for the reply. >>> >>> >>> On 31/03/2011 16:48, Hong Zhang wrote: >>> Loic, >>> >>> I want to do the same for MATMPI. This is what I do: >>> >>> MatGetLocalMat(A,MAT_INITIAL_MATRIX,&Aloc); >>> MatGetRowIJ(Aloc, 1, PETSC_FALSE, PETSC_FALSE,&(Acsr.n),&(Acsr.ia), >>> &(Acsr.ja),&done); >>> MatGetArray(Aloc,&(Acsr.a)); >>> >>> Now, I need to know on what process the non local points are. >>> Each process get its own local submatrix. See >>> http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatCreateMPIAIJ.html >>> on data structure of MPIAIJ format. Applying your statements above to >>> the example listed on the page (a 8x8 matrix), Proc0 will get >>> Aloc=A(0:2,:), ..., Porc2 gets Aloc=A(6:7,:). >>> Is there a petsc function that gives this partition ? >>> MatGetOwnershipRange() >>> I use this function to know the local points index. In the same way, with this function I know the non local index. >>> >>> But this function doesn't give the process. I give you an example. I have this matrix: >>> >>> 1 0 | 0 1 >>> 1 2 | 1 0 >>> ----------- >>> 0 0 | 1 0 >>> 0 2 | 0 1 >>> >>> I want this kind of format >>> >>> for proc 0 >>> a = {1,1,1,2,1} >>> ia = {0,2,5} >>> ja = {0,3,0,1,2} >>> rank = {0,0,1,1} >>> >>> for proc 1 >>> a = {1,2,1} >>> ia = {0,1,3} >>> ja = {0,2,1} >>> rank = {1,1,0} >>> >>> rank give me the process for each non zero column. Local rows should have numbers 1,...N and offdiagonal entries with j>N. >>> >>> The ownership range is the row partition. However, MPIAIJ matrices are not stored as you indicate above. >>> >>> Different storage formats make the process you describe fragile. WHy do you need the CSR form of the matrix? If you really >>> do, use MatGetSubMatrix() to get a SEQAIJ matrix on each process and pull out the values. >>> >> I want to use this format because I try to create a new multigrid PC with this method: >> >> http://homepages.ulb.ac.be/~ynotay/AGMG/userguide/index.html >> >> I don't understand why it's not possible to have the process partition. When you create a MPIAIJ matrix and you use a KSP to solve the system, you know what values to exchange. No? Where is this information ? > The point is not that the information is not available, it is just that it is not trivially available since we don't use the data in that format. > > Here is what you have to do. Include "src/mat/impls/aij/mpi/mpiaij.h" directly in your interface files and access the data directly, do not try to find some PETSc function that provides exactly what you want directly, that cannot exist because we cannot know what format each other package wants. In particular inside Mat_MPIAJ are the two submatrices called A and B which are Mat_SeqAIJ matrices. The data in > #if defined (PETSC_USE_CTABLE) > PetscTable colmap; > #else > PetscInt *colmap; /* local col number of off-diag col */ > #endif > PetscInt *garray; /* global index of all off-processor columns */ > provides the information about the "off process" columns for the matrices stored in B. > > But you will have to understand the data structures somewhat in order to get the data in the form you want. > Thanks Matt and Barry. I looked at the interface for hypre to see what you have done and I saw this include file. I'll try to do that in a simple way with MatGetOwnershipRanges for my first PC version. There are a lot of functions in PETSc and it's sometime difficult to find the good one. Thanks again. Loic > Barry > > > > > >> I can also use MatGetOwnershipRange() and MPI_Allgather to have this information. >> >> Loic >> >> >> >> >>> Matt >>> >>> I suppose that I have no information about the structure of MPIAIJ. >>> >>> Loic >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener From bsmith at mcs.anl.gov Thu Mar 31 13:20:34 2011 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 31 Mar 2011 13:20:34 -0500 Subject: [petsc-users] extract Mat data In-Reply-To: <4D94C4BB.1060009@math.u-psud.fr> References: <4D94880C.4000102@math.u-psud.fr> <4D949981.5070206@math.u-psud.fr> <4D94BC04.7060301@math.u-psud.fr> <94CAE4FB-389D-4891-8165-AB7189C8239B@mcs.anl.gov> <4D94C4BB.1060009@math.u-psud.fr> Message-ID: On Mar 31, 2011, at 1:15 PM, gouarin wrote: > On 31/03/2011 20:05, Barry Smith wrote: >> On Mar 31, 2011, at 12:38 PM, gouarin wrote: >> >>> On 31/03/2011 19:06, Matthew Knepley wrote: >>>> On Thu, Mar 31, 2011 at 10:10 AM, gouarin wrote: >>>> u want. >> > Thanks Matt and Barry. I looked at the interface for hypre to see what you have done and I saw this include file. > > I'll try to do that in a simple way with MatGetOwnershipRanges for my first PC version. > > There are a lot of functions in PETSc and it's sometime difficult to find the good one. The particular data structure you need is not something we provide natively so, sorry, it will require that you poke around in order to extract it to the form you want. Barry > > Thanks again. > > Loic > > >> Barry >> >> >> >> >> >>> I can also use MatGetOwnershipRange() and MPI_Allgather to have this information. >>> >>> Loic >>> >>> >>> >>> >>>> Matt >>>> >>>> I suppose that I have no information about the structure of MPIAIJ. >>>> >>>> Loic >>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>> -- Norbert Wiener > From khalid_eee at yahoo.com Thu Mar 31 21:51:24 2011 From: khalid_eee at yahoo.com (khalid ashraf) Date: Thu, 31 Mar 2011 19:51:24 -0700 (PDT) Subject: [petsc-users] FFT with PETSC Message-ID: <311699.28796.qm@web112607.mail.gq1.yahoo.com> Hi, Could you please suggest what would be the best way to perform forward and inverse discrete Fourier Transform with PETSC ? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Thu Mar 31 21:57:27 2011 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Thu, 31 Mar 2011 21:57:27 -0500 Subject: [petsc-users] FFT with PETSC In-Reply-To: <311699.28796.qm@web112607.mail.gq1.yahoo.com> References: <311699.28796.qm@web112607.mail.gq1.yahoo.com> Message-ID: You can use FFTW through petsc interface. Suggest using petsc-dev (see http://www.mcs.anl.gov/petsc/petsc-as/developers/index.html). Examples can be found at petsc-dev/src/mat/examples/tests/ex112.c, ex121.c, ex142.c and ex143.c Hong On Thu, Mar 31, 2011 at 9:51 PM, khalid ashraf wrote: > Hi, > Could you please suggest what would be the best way to perform forward and > inverse discrete > Fourier ?Transform with PETSC ? > Thanks.