From michael.afanasiev at erdw.ethz.ch Mon Feb 1 03:30:50 2016 From: michael.afanasiev at erdw.ethz.ch (Afanasiev Michael) Date: Mon, 1 Feb 2016 09:30:50 +0000 Subject: [petsc-users] General DOF ordering in DMPlex. Message-ID: <1E8B8A51-E6A4-4C31-8010-EC753A129EF5@erdw.ethz.ch> Hi everyone, A few of us here at ETH have been working on some finite-element stuff using DMPLEX, which has been wildly useful. Is there anywhere that we can get general rules for the degree of freedom ordering returned by DMPlexVecGetClosure? Something akin to the attached PDF (provided by Dave May), but for Hexs, Tris, and Tets? Thanks for any help, Mike. -- Michael Afanasiev Ph.D. Candidate Computational Seismology Institut f?r Geophysik ETH Z?rich Sonneggstrasse 5, NO H 39.2 CH 8092 Z?rich michael.afanasiev at erdw.ethz.ch -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: plex2d-quad.pdf Type: application/pdf Size: 43302 bytes Desc: plex2d-quad.pdf URL: From knepley at gmail.com Mon Feb 1 07:17:12 2016 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 1 Feb 2016 07:17:12 -0600 Subject: [petsc-users] General DOF ordering in DMPlex. In-Reply-To: <1E8B8A51-E6A4-4C31-8010-EC753A129EF5@erdw.ethz.ch> References: <1E8B8A51-E6A4-4C31-8010-EC753A129EF5@erdw.ethz.ch> Message-ID: On Mon, Feb 1, 2016 at 3:30 AM, Afanasiev Michael < michael.afanasiev at erdw.ethz.ch> wrote: > Hi everyone, > > A few of us here at ETH have been working on some finite-element stuff > using DMPLEX, which has been wildly useful. Is there anywhere that we can > get general rules for the degree of freedom ordering returned by > DMPlexVecGetClosure? Something akin to the attached PDF (provided by Dave > May), but for Hexs, Tris, and Tets? > I will make those, but I am honestly very pressed right now with a hard deadline. I will just describe the process quickly so that maybe you will not be held up until I can get to it. The order of dofs is based upon the topological order of the closure. Thus, you can see in the quad example that it always proceeds {cell, faces, edges, vertices} So the first question is: what is the order of the boundary around any mesh point? This is specified here: https://bitbucket.org/petsc/petsc/src/e03cc9b84019b00a431ca4cfdfeb76830ce7201c/src/dm/impls/plex/plexinterpolate.c?at=master&fileviewer=file-view-default#plexinterpolate.c-43 which clearly would benefit from being written down somewhere. Once you have the face order, the only other element is that faces may be oriented when extracted. You can see this orientation in the order of vertices in the code above. I orient the cell so that you get a consistent normal (in 3D its the generalization that the iterated boundary operator is null), as you can see from the quad example. Finally, each point only appears the first time it is considered in the closure. The additional complications with dofs are that - Dofs are ordered according to the orientation of the point they live on. Basically you have a representation of the symmetric group attached to each orientation (strictly its an order I think). This is probably not general enough but it works for now. - I further partition dofs by field, so that all dofs of each field are contiguous. Thanks, Matt > Thanks for any help, > Mike. > > > -- > Michael Afanasiev > Ph.D. Candidate > Computational Seismology > Institut f?r Geophysik > ETH Z?rich > > Sonneggstrasse 5, NO H 39.2 > CH 8092 Z?rich > michael.afanasiev at erdw.ethz.ch > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From epscodes at gmail.com Mon Feb 1 13:46:40 2016 From: epscodes at gmail.com (Xiangdong) Date: Mon, 1 Feb 2016 14:46:40 -0500 Subject: [petsc-users] snes linesearch bt vs basic? Message-ID: Hello everyone, I have a question on snes with line search. Under what circumstance, applying the (globalized) line search can perform WORSE than without line search (i.e. full newton step ls type basic)? When I apply snes with my problem, I found that at some time steps, snes diverges with linesearch_bt, but can converge with linesearch_basic (using same initial guess). For the same problem, there are also some steps, at which linearsearch_basic fails, but linesearch_bt converges. Is there a general guide line to decide whether linesearch should be used? Thanks. Xiangdong -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Feb 1 13:59:40 2016 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 1 Feb 2016 13:59:40 -0600 Subject: [petsc-users] snes linesearch bt vs basic? In-Reply-To: References: Message-ID: On Mon, Feb 1, 2016 at 1:46 PM, Xiangdong wrote: > Hello everyone, > > I have a question on snes with line search. Under what circumstance, > applying the (globalized) line search can perform WORSE than without line > search (i.e. full newton step ls type basic)? > > When I apply snes with my problem, I found that at some time steps, snes > diverges with linesearch_bt, but can converge with linesearch_basic (using > same initial guess). For the same problem, there are also some steps, at > which linearsearch_basic fails, but linesearch_bt converges. Is there a > general guide line to decide whether linesearch should be used? > No, and it is likely impossible to give one. 'basic' can succeed by taking some non-decreasing steps which happen to fall into a basin of convergence, whereas 'bt' will succeed in many circumstances where basic diverges. If we knew how to do this, we could retire. Matt > Thanks. > > Xiangdong > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Feb 1 14:16:56 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 1 Feb 2016 14:16:56 -0600 Subject: [petsc-users] snes linesearch bt vs basic? In-Reply-To: References: Message-ID: > On Feb 1, 2016, at 1:46 PM, Xiangdong wrote: > > Hello everyone, > > I have a question on snes with line search. Under what circumstance, applying the (globalized) line search can perform WORSE than without line search (i.e. full newton step ls type basic)? > > When I apply snes with my problem, I found that at some time steps, snes diverges with linesearch_bt, but can converge with linesearch_basic (using same initial guess). For the same problem, there are also some steps, at which linearsearch_basic fails, but linesearch_bt converges. Here are you referring to "time steps"? When doing time stepping, depending on the evolution of the equations, the nonlinear systems can become more difficult to solver or easier to solve. One can often adjust the time-step size to make sure the nonlinear equations are never to difficult. So for example if the nonlinear solver is taking more than, say, 10 Newton steps, or is failing, decrease the timestep by some factor and try again. If the nonlinear solver needs only 1 or 2 Newton steps one can try increasing the time-step. Ideally this is also done in conjunction with an ODE error estimator to control the error by adjusting the timestep. Some of the PETSc implicit TS ODE integrators do this automatically. Generally adjusting the timestep is a more reliable way to insure convergence than trying to produce a completely robust nonlinear solver (as Matt points out if you did develop a completely robust nonlinear solver you could retire on your fame). Barry > Is there a general guide line to decide whether linesearch should be used? > > Thanks. > > Xiangdong From epscodes at gmail.com Mon Feb 1 15:43:29 2016 From: epscodes at gmail.com (Xiangdong) Date: Mon, 1 Feb 2016 16:43:29 -0500 Subject: [petsc-users] snes linesearch bt vs basic? In-Reply-To: References: Message-ID: I did not use the TS, but I did adjust my time step based on the number of snes iterations as well as the solution changes in the current time step. We want to minimize the time cuts (failed snes) as well as not using too small dt. If I want to know how petsc computes the dt_next based on dt_current, namely, dt_next = dt_current*func(snesits, etc), which function/file should I look into? Thanks. Xiangdong On Mon, Feb 1, 2016 at 3:16 PM, Barry Smith wrote: > > > On Feb 1, 2016, at 1:46 PM, Xiangdong wrote: > > > > Hello everyone, > > > > I have a question on snes with line search. Under what circumstance, > applying the (globalized) line search can perform WORSE than without line > search (i.e. full newton step ls type basic)? > > > > When I apply snes with my problem, I found that at some time steps, snes > diverges with linesearch_bt, but can converge with linesearch_basic (using > same initial guess). For the same problem, there are also some steps, at > which linearsearch_basic fails, but linesearch_bt converges. > > Here are you referring to "time steps"? When doing time stepping, > depending on the evolution of the equations, the nonlinear systems can > become more difficult to solver or easier to solve. One can often adjust > the time-step size to make sure the nonlinear equations are never to > difficult. So for example if the nonlinear solver is taking more than, say, > 10 Newton steps, or is failing, decrease the timestep by some factor and > try again. If the nonlinear solver needs only 1 or 2 Newton steps one can > try increasing the time-step. Ideally this is also done in conjunction with > an ODE error estimator to control the error by adjusting the timestep. Some > of the PETSc implicit TS ODE integrators do this automatically. Generally > adjusting the timestep is a more reliable way to insure convergence than > trying to produce a completely robust nonlinear solver (as Matt points out > if you did develop a completely robust nonlinear solver you could retire on > your fame). > > Barry > > > > Is there a general guide line to decide whether linesearch should be > used? > > > > Thanks. > > > > Xiangdong > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Mon Feb 1 15:52:41 2016 From: jed at jedbrown.org (Jed Brown) Date: Mon, 01 Feb 2016 14:52:41 -0700 Subject: [petsc-users] snes linesearch bt vs basic? In-Reply-To: References: Message-ID: <87mvrk9jyu.fsf@jedbrown.org> Xiangdong writes: > If I want to know how petsc computes the dt_next based on dt_current, > namely, dt_next = dt_current*func(snesits, etc), which function/file should > I look into? Check out TSAdapt, especially TSAdaptChoose_Basic and TSAdaptCheckStage. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From bsmith at mcs.anl.gov Mon Feb 1 16:26:56 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 1 Feb 2016 16:26:56 -0600 Subject: [petsc-users] snes linesearch bt vs basic? In-Reply-To: References: Message-ID: > On Feb 1, 2016, at 2:16 PM, Barry Smith wrote: > > >> On Feb 1, 2016, at 1:46 PM, Xiangdong wrote: >> >> Hello everyone, >> >> I have a question on snes with line search. Under what circumstance, applying the (globalized) line search can perform WORSE than without line search (i.e. full newton step ls type basic)? >> >> When I apply snes with my problem, I found that at some time steps, snes diverges with linesearch_bt, but can converge with linesearch_basic (using same initial guess). For the same problem, there are also some steps, at which linearsearch_basic fails, but linesearch_bt converges. > > Here are you referring to "time steps"? When doing time stepping, depending on the evolution of the equations, the nonlinear systems can become more difficult to solver or easier to solve. One can often adjust the time-step size to make sure the nonlinear equations are never to difficult. So for example if the nonlinear solver is taking more than, say, 10 Newton steps, or is failing, decrease the timestep by some factor and try again. If the nonlinear solver needs only 1 or 2 Newton steps one can try increasing the time-step. Ideally this is also done in conjunction with an ODE error estimator to control the error by adjusting the timestep. Some of the PETSc implicit TS ODE integrators do this automatically. Generally adjusting the timestep is a more reliable way to insure convergence than trying to produce a completely robust nonlinear solver (as Matt points out if you did develop a completely robust nonlinear solver you could retire on your fame). Of course this will not always work, if the solution develops a singularity in finite time or is "really nasty" the time-step could be driven to zero; but having a more robust nonlinear solver won't help in that situation either. Barry > > Barry > > >> Is there a general guide line to decide whether linesearch should be used? >> >> Thanks. >> >> Xiangdong > From epscodes at gmail.com Mon Feb 1 16:29:59 2016 From: epscodes at gmail.com (Xiangdong) Date: Mon, 1 Feb 2016 17:29:59 -0500 Subject: [petsc-users] snes linesearch bt vs basic? In-Reply-To: <87mvrk9jyu.fsf@jedbrown.org> References: <87mvrk9jyu.fsf@jedbrown.org> Message-ID: It seems that TSAdaptChoose_Basic only uses the local truncation error to predict the next time step. Do you also include the number of snes iterations in computing the next time step in some adapt option? Thanks. Xiangdong On Mon, Feb 1, 2016 at 4:52 PM, Jed Brown wrote: > Xiangdong writes: > > If I want to know how petsc computes the dt_next based on dt_current, > > namely, dt_next = dt_current*func(snesits, etc), which function/file > should > > I look into? > > Check out TSAdapt, especially TSAdaptChoose_Basic and TSAdaptCheckStage. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Mon Feb 1 16:36:13 2016 From: jed at jedbrown.org (Jed Brown) Date: Mon, 01 Feb 2016 15:36:13 -0700 Subject: [petsc-users] snes linesearch bt vs basic? In-Reply-To: References: <87mvrk9jyu.fsf@jedbrown.org> Message-ID: <87h9hs9hya.fsf@jedbrown.org> Xiangdong writes: > It seems that TSAdaptChoose_Basic only uses the local truncation error to > predict the next time step. Do you also include the number of snes > iterations in computing the next time step in some adapt option? TSAdaptChoose_Basic doesn't use the number of iterations (you can set your own callback for whatever policy you want), but TSAdaptCheckStage handles SNES failures and you can set a max number of SNES iterations. (This could be less efficient than a progressive function that gradually reduces time step as the number of iterations increases.) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From epscodes at gmail.com Mon Feb 1 16:45:52 2016 From: epscodes at gmail.com (Xiangdong) Date: Mon, 1 Feb 2016 17:45:52 -0500 Subject: [petsc-users] snes linesearch bt vs basic? In-Reply-To: References: Message-ID: On Mon, Feb 1, 2016 at 5:26 PM, Barry Smith wrote: > > > On Feb 1, 2016, at 2:16 PM, Barry Smith wrote: > > > > > >> On Feb 1, 2016, at 1:46 PM, Xiangdong wrote: > >> > >> Hello everyone, > >> > >> I have a question on snes with line search. Under what circumstance, > applying the (globalized) line search can perform WORSE than without line > search (i.e. full newton step ls type basic)? > >> > >> When I apply snes with my problem, I found that at some time steps, > snes diverges with linesearch_bt, but can converge with linesearch_basic > (using same initial guess). For the same problem, there are also some > steps, at which linearsearch_basic fails, but linesearch_bt converges. > > > > Here are you referring to "time steps"? When doing time stepping, > depending on the evolution of the equations, the nonlinear systems can > become more difficult to solver or easier to solve. One can often adjust > the time-step size to make sure the nonlinear equations are never to > difficult. So for example if the nonlinear solver is taking more than, say, > 10 Newton steps, or is failing, decrease the timestep by some factor and > try again. If the nonlinear solver needs only 1 or 2 Newton steps one can > try increasing the time-step. Ideally this is also done in conjunction with > an ODE error estimator to control the error by adjusting the timestep. Some > of the PETSc implicit TS ODE integrators do this automatically. Generally > adjusting the timestep is a more reliable way to insure convergence than > trying to produce a completely robust nonlinear solver (as Matt points out > if you did develop a completely robust nonlinear solver you could retire on > your fame). > > Of course this will not always work, if the solution develops a > singularity in finite time or is "really nasty" the time-step could be > driven to zero; but having a more robust nonlinear solver won't help in > that situation either. > Actually my problem is not that ill-conditioned. Here is my situation. Let us say at time step n the snes_linesearch_basic failed. I have at least two options now: i) cut the dt immediately, ii) switch to snes_linesearch_bt with the same dt. Sometimes, switching to a different linesearch method without cutting dt will make snes converge, but sometimes not (then I need to cut dt, which is bad since more nonlinear iterations were wasted). Based on what you mentioned, it is impossible to decide which option is good and I just need to simply pick one. Thanks. Xiangdong > > Barry > > > > > Barry > > > > > >> Is there a general guide line to decide whether linesearch should be > used? > >> > >> Thanks. > >> > >> Xiangdong > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Mon Feb 1 16:57:26 2016 From: jed at jedbrown.org (Jed Brown) Date: Mon, 01 Feb 2016 15:57:26 -0700 Subject: [petsc-users] snes linesearch bt vs basic? In-Reply-To: References: Message-ID: <878u349gyx.fsf@jedbrown.org> Xiangdong writes: > Actually my problem is not that ill-conditioned. Here is my situation. Let > us say at time step n the snes_linesearch_basic failed. I have at least two > options now: i) cut the dt immediately, ii) switch to snes_linesearch_bt > with the same dt. Sometimes, switching to a different linesearch method > without cutting dt will make snes converge, but sometimes not (then I need > to cut dt, which is bad since more nonlinear iterations were wasted). Based > on what you mentioned, it is impossible to decide which option is good and > I just need to simply pick one. There's no automatic rule to do the right thing. If you think you can be smart about it (perhaps using problem-specific information), you can implement your own SNESLineSearch (it's a plugin system meant to do this) that calls the BT versus Basic implementations (or your own, etc.). If you develop a new line search that is better, you can publish a paper and contribute your code so that others can use it. Have fun. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From bsmith at mcs.anl.gov Mon Feb 1 18:22:35 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 1 Feb 2016 18:22:35 -0600 Subject: [petsc-users] snes linesearch bt vs basic? In-Reply-To: References: Message-ID: > On Feb 1, 2016, at 4:45 PM, Xiangdong wrote: > > > > On Mon, Feb 1, 2016 at 5:26 PM, Barry Smith wrote: > > > On Feb 1, 2016, at 2:16 PM, Barry Smith wrote: > > > > > >> On Feb 1, 2016, at 1:46 PM, Xiangdong wrote: > >> > >> Hello everyone, > >> > >> I have a question on snes with line search. Under what circumstance, applying the (globalized) line search can perform WORSE than without line search (i.e. full newton step ls type basic)? > >> > >> When I apply snes with my problem, I found that at some time steps, snes diverges with linesearch_bt, but can converge with linesearch_basic (using same initial guess). For the same problem, there are also some steps, at which linearsearch_basic fails, but linesearch_bt converges. > > > > Here are you referring to "time steps"? When doing time stepping, depending on the evolution of the equations, the nonlinear systems can become more difficult to solver or easier to solve. One can often adjust the time-step size to make sure the nonlinear equations are never to difficult. So for example if the nonlinear solver is taking more than, say, 10 Newton steps, or is failing, decrease the timestep by some factor and try again. If the nonlinear solver needs only 1 or 2 Newton steps one can try increasing the time-step. Ideally this is also done in conjunction with an ODE error estimator to control the error by adjusting the timestep. Some of the PETSc implicit TS ODE integrators do this automatically. Generally adjusting the timestep is a more reliable way to insure convergence than trying to produce a completely robust nonlinear solver (as Matt points out if you did develop a completely robust nonlinear solver you could retire on your fame). > > Of course this will not always work, if the solution develops a singularity in finite time or is "really nasty" the time-step could be driven to zero; but having a more robust nonlinear solver won't help in that situation either. > > Actually my problem is not that ill-conditioned. Here is my situation. Let us say at time step n the snes_linesearch_basic failed. I have at least two options now: i) cut the dt immediately, ii) switch to snes_linesearch_bt with the same dt. Sometimes, switching to a different linesearch method without cutting dt will make snes converge, but sometimes not (then I need to cut dt, which is bad since more nonlinear iterations were wasted). Based on what you mentioned, it is impossible to decide which option is good and I just need to simply pick one. SNESLineSearch_Basic simply does the full Newton step regardless of whether it decreases or increases the function value. IMHO if basic converges while bt does not then this is just blind luck; not that basic was "better" for that configuration. I would just stick to bt and adjust the dt accordingly. Barry > > Thanks. > > Xiangdong > > > > Barry > > > > > Barry > > > > > >> Is there a general guide line to decide whether linesearch should be used? > >> > >> Thanks. > >> > >> Xiangdong > > > > From knepley at gmail.com Mon Feb 1 19:12:22 2016 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 1 Feb 2016 19:12:22 -0600 Subject: [petsc-users] snes linesearch bt vs basic? In-Reply-To: References: Message-ID: On Mon, Feb 1, 2016 at 6:22 PM, Barry Smith wrote: > > > On Feb 1, 2016, at 4:45 PM, Xiangdong wrote: > > > > > > > > On Mon, Feb 1, 2016 at 5:26 PM, Barry Smith wrote: > > > > > On Feb 1, 2016, at 2:16 PM, Barry Smith wrote: > > > > > > > > >> On Feb 1, 2016, at 1:46 PM, Xiangdong wrote: > > >> > > >> Hello everyone, > > >> > > >> I have a question on snes with line search. Under what circumstance, > applying the (globalized) line search can perform WORSE than without line > search (i.e. full newton step ls type basic)? > > >> > > >> When I apply snes with my problem, I found that at some time steps, > snes diverges with linesearch_bt, but can converge with linesearch_basic > (using same initial guess). For the same problem, there are also some > steps, at which linearsearch_basic fails, but linesearch_bt converges. > > > > > > Here are you referring to "time steps"? When doing time stepping, > depending on the evolution of the equations, the nonlinear systems can > become more difficult to solver or easier to solve. One can often adjust > the time-step size to make sure the nonlinear equations are never to > difficult. So for example if the nonlinear solver is taking more than, say, > 10 Newton steps, or is failing, decrease the timestep by some factor and > try again. If the nonlinear solver needs only 1 or 2 Newton steps one can > try increasing the time-step. Ideally this is also done in conjunction with > an ODE error estimator to control the error by adjusting the timestep. Some > of the PETSc implicit TS ODE integrators do this automatically. Generally > adjusting the timestep is a more reliable way to insure convergence than > trying to produce a completely robust nonlinear solver (as Matt points out > if you did develop a completely robust nonlinear solver you could retire on > your fame). > > > > Of course this will not always work, if the solution develops a > singularity in finite time or is "really nasty" the time-step could be > driven to zero; but having a more robust nonlinear solver won't help in > that situation either. > > > > Actually my problem is not that ill-conditioned. Here is my situation. > Let us say at time step n the snes_linesearch_basic failed. I have at least > two options now: i) cut the dt immediately, ii) switch to > snes_linesearch_bt with the same dt. Sometimes, switching to a different > linesearch method without cutting dt will make snes converge, but sometimes > not (then I need to cut dt, which is bad since more nonlinear iterations > were wasted). Based on what you mentioned, it is impossible to decide which > option is good and I just need to simply pick one. > > SNESLineSearch_Basic simply does the full Newton step regardless of > whether it decreases or increases the function value. IMHO if basic > converges while bt does not then this is just blind luck; not that basic > was "better" for that configuration. I would just stick to bt and adjust > the dt accordingly. > I will note that 'nleqerr' is a new line search that performs considerably better than 'bt' which was added by Patrick Farrell of Oxford. I would give this a try if bt is failing. Thanks, Matt > Barry > > > > > Thanks. > > > > Xiangdong > > > > > > > > Barry > > > > > > > > Barry > > > > > > > > >> Is there a general guide line to decide whether linesearch should be > used? > > >> > > >> Thanks. > > >> > > >> Xiangdong > > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From orxan.shibli at gmail.com Tue Feb 2 08:35:14 2016 From: orxan.shibli at gmail.com (Orxan Shibliyev) Date: Tue, 2 Feb 2016 16:35:14 +0200 Subject: [petsc-users] Solving only for certain rows Message-ID: After hole cutting my unstructured mesh has some active and inactive/fringe cells. I solve equations on active/computational/field cells only. Some of the active cells are neighbors of inactive cells. If I prepare petsc matrix including all cells then the solver does not converge due unchanging values of inactive cells. How can I tell Petsc not to solve for certain rows? If I make all cells active then the solver converge so I am sure that there is no problem related to solver. -------------- next part -------------- An HTML attachment was scrubbed... URL: From epscodes at gmail.com Tue Feb 2 08:46:35 2016 From: epscodes at gmail.com (Xiangdong) Date: Tue, 2 Feb 2016 09:46:35 -0500 Subject: [petsc-users] petsc externalpackage directory Message-ID: Hello everyone, If I use the option -download-pastix=1, which directory is the package pastix_5.2.2.20.tar.bz2 downloaded to? It was compiled and installed in arch/externalpackages, but I want to know where the tar file is. I want to copy these tar files into a computer without internet connection and install petsc there. Thanks. Xiangdong -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Feb 2 08:58:16 2016 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 2 Feb 2016 08:58:16 -0600 Subject: [petsc-users] Solving only for certain rows In-Reply-To: References: Message-ID: On Tue, Feb 2, 2016 at 8:35 AM, Orxan Shibliyev wrote: > After hole cutting my unstructured mesh has some active and > inactive/fringe cells. I solve equations on active/computational/field > cells only. Some of the active cells are neighbors of inactive cells. If I > prepare petsc matrix including all cells then the solver does not converge > due unchanging values of inactive cells. How can I tell Petsc not to solve > for certain rows? If I make all cells active then the solver converge so I > am sure that there is no problem related to solver. > Leave them out of the matrix? Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Feb 2 08:59:30 2016 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 2 Feb 2016 08:59:30 -0600 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: References: Message-ID: On Tue, Feb 2, 2016 at 8:46 AM, Xiangdong wrote: > Hello everyone, > > If I use the option -download-pastix=1, which directory is the package > pastix_5.2.2.20.tar.bz2 downloaded to? It was compiled and installed in > arch/externalpackages, but I want to know where the tar file is. I want to > copy these tar files into a computer without internet connection and > install petsc there. > If I understand your question correctly, you can just give --download-pastix= and it will do what you want. Thanks, Matt > Thanks. > > Xiangdong > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From epscodes at gmail.com Tue Feb 2 09:15:05 2016 From: epscodes at gmail.com (Xiangdong) Date: Tue, 2 Feb 2016 10:15:05 -0500 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: References: Message-ID: Not really. My question is if I use download-pastix=1, which directory is the tarball (package pastix_5.2.2.20.tar.bz2) downloaded into? Thanks. Xiangdong On Tue, Feb 2, 2016 at 9:59 AM, Matthew Knepley wrote: > On Tue, Feb 2, 2016 at 8:46 AM, Xiangdong wrote: > >> Hello everyone, >> >> If I use the option -download-pastix=1, which directory is the package >> pastix_5.2.2.20.tar.bz2 downloaded to? It was compiled and installed in >> arch/externalpackages, but I want to know where the tar file is. I want to >> copy these tar files into a computer without internet connection and >> install petsc there. >> > > If I understand your question correctly, you can just give > > --download-pastix= > > and it will do what you want. > > Thanks, > > Matt > > >> Thanks. >> >> Xiangdong >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Feb 2 09:15:39 2016 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 2 Feb 2016 09:15:39 -0600 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: References: Message-ID: On Tue, Feb 2, 2016 at 9:15 AM, Xiangdong wrote: > Not really. My question is if I use download-pastix=1, which directory is > the tarball (package pastix_5.2.2.20.tar.bz2) downloaded into? > $PETSC_ARCH/externalpackages Matt > Thanks. > > Xiangdong > > On Tue, Feb 2, 2016 at 9:59 AM, Matthew Knepley wrote: > >> On Tue, Feb 2, 2016 at 8:46 AM, Xiangdong wrote: >> >>> Hello everyone, >>> >>> If I use the option -download-pastix=1, which directory is the package >>> pastix_5.2.2.20.tar.bz2 downloaded to? It was compiled and installed in >>> arch/externalpackages, but I want to know where the tar file is. I want to >>> copy these tar files into a computer without internet connection and >>> install petsc there. >>> >> >> If I understand your question correctly, you can just give >> >> --download-pastix= >> >> and it will do what you want. >> >> Thanks, >> >> Matt >> >> >>> Thanks. >>> >>> Xiangdong >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From epscodes at gmail.com Tue Feb 2 09:21:52 2016 From: epscodes at gmail.com (Xiangdong) Date: Tue, 2 Feb 2016 10:21:52 -0500 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: References: Message-ID: the package is untar and compiled in $PETSC_ARCH/externalpackages, but the original tarball .tar file is deleted. Is it correct? Xiangdong On Tue, Feb 2, 2016 at 10:15 AM, Matthew Knepley wrote: > On Tue, Feb 2, 2016 at 9:15 AM, Xiangdong wrote: > >> Not really. My question is if I use download-pastix=1, which directory is >> the tarball (package pastix_5.2.2.20.tar.bz2) downloaded into? >> > > $PETSC_ARCH/externalpackages > > Matt > > >> Thanks. >> >> Xiangdong >> >> On Tue, Feb 2, 2016 at 9:59 AM, Matthew Knepley >> wrote: >> >>> On Tue, Feb 2, 2016 at 8:46 AM, Xiangdong wrote: >>> >>>> Hello everyone, >>>> >>>> If I use the option -download-pastix=1, which directory is the package >>>> pastix_5.2.2.20.tar.bz2 downloaded to? It was compiled and installed in >>>> arch/externalpackages, but I want to know where the tar file is. I want to >>>> copy these tar files into a computer without internet connection and >>>> install petsc there. >>>> >>> >>> If I understand your question correctly, you can just give >>> >>> --download-pastix= >>> >>> and it will do what you want. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thanks. >>>> >>>> Xiangdong >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Feb 2 09:46:20 2016 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 2 Feb 2016 09:46:20 -0600 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: References: Message-ID: On Tue, Feb 2, 2016 at 9:21 AM, Xiangdong wrote: > the package is untar and compiled in $PETSC_ARCH/externalpackages, but the > original tarball .tar file is deleted. Is it correct? > If you want the tarball URL, you can see it in the Python; https://bitbucket.org/petsc/petsc/src/e03cc9b84019b00a431ca4cfdfeb76830ce7201c/config/BuildSystem/config/packages/PaStiX.py?at=master&fileviewer=file-view-default#PaStiX.py-6 Matt > Xiangdong > > On Tue, Feb 2, 2016 at 10:15 AM, Matthew Knepley > wrote: > >> On Tue, Feb 2, 2016 at 9:15 AM, Xiangdong wrote: >> >>> Not really. My question is if I use download-pastix=1, which directory >>> is the tarball (package pastix_5.2.2.20.tar.bz2) downloaded into? >>> >> >> $PETSC_ARCH/externalpackages >> >> Matt >> >> >>> Thanks. >>> >>> Xiangdong >>> >>> On Tue, Feb 2, 2016 at 9:59 AM, Matthew Knepley >>> wrote: >>> >>>> On Tue, Feb 2, 2016 at 8:46 AM, Xiangdong wrote: >>>> >>>>> Hello everyone, >>>>> >>>>> If I use the option -download-pastix=1, which directory is the package >>>>> pastix_5.2.2.20.tar.bz2 downloaded to? It was compiled and installed in >>>>> arch/externalpackages, but I want to know where the tar file is. I want to >>>>> copy these tar files into a computer without internet connection and >>>>> install petsc there. >>>>> >>>> >>>> If I understand your question correctly, you can just give >>>> >>>> --download-pastix= >>>> >>>> and it will do what you want. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Thanks. >>>>> >>>>> Xiangdong >>>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From s_g at berkeley.edu Tue Feb 2 11:43:25 2016 From: s_g at berkeley.edu (Sanjay Govindjee) Date: Tue, 2 Feb 2016 09:43:25 -0800 Subject: [petsc-users] Solving only for certain rows In-Reply-To: References: Message-ID: <56B0EABD.6000604@berkeley.edu> Alternately, assemble zeros in the corresponding rows and columns with a 1 on the diagonal. -sg On 2/2/16 6:58 AM, Matthew Knepley wrote: > On Tue, Feb 2, 2016 at 8:35 AM, Orxan Shibliyev > > wrote: > > After hole cutting my unstructured mesh has some active and > inactive/fringe cells. I solve equations on > active/computational/field cells only. Some of the active cells > are neighbors of inactive cells. If I prepare petsc matrix > including all cells then the solver does not converge due > unchanging values of inactive cells. How can I tell Petsc not to > solve for certain rows? If I make all cells active then the solver > converge so I am sure that there is no problem related to solver. > > > Leave them out of the matrix? > > Matt > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -- ----------------------------------------------- Sanjay Govindjee, PhD, PE Professor of Civil Engineering 779 Davis Hall University of California Berkeley, CA 94720-1710 Voice: +1 510 642 6060 FAX: +1 510 643 5264 s_g at berkeley.edu http://www.ce.berkeley.edu/~sanjay ----------------------------------------------- Books: Engineering Mechanics of Deformable Solids: A Presentation with Exercises http://www.oup.com/us/catalog/general/subject/Physics/MaterialsScience/?view=usa&ci=9780199651641 http://ukcatalogue.oup.com/product/9780199651641.do http://amzn.com/0199651647 Engineering Mechanics 3 (Dynamics) 2nd Edition http://www.springer.com/978-3-642-53711-0 http://amzn.com/3642537111 Engineering Mechanics 3, Supplementary Problems: Dynamics http://www.amzn.com/B00SOXN8JU ----------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Feb 2 13:25:27 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 2 Feb 2016 13:25:27 -0600 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: References: Message-ID: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> > On Feb 2, 2016, at 9:21 AM, Xiangdong wrote: > > the package is untar and compiled in $PETSC_ARCH/externalpackages, but the original tarball .tar file is deleted. Is it correct? Yes When you ran the successful configure it printed to the screen where the tar file came from. You should use curl or wget to get the tarball and then copy it over to the machine you want it on. Barry > > Xiangdong > > On Tue, Feb 2, 2016 at 10:15 AM, Matthew Knepley wrote: > On Tue, Feb 2, 2016 at 9:15 AM, Xiangdong wrote: > Not really. My question is if I use download-pastix=1, which directory is the tarball (package pastix_5.2.2.20.tar.bz2) downloaded into? > > $PETSC_ARCH/externalpackages > > Matt > > Thanks. > > Xiangdong > > On Tue, Feb 2, 2016 at 9:59 AM, Matthew Knepley wrote: > On Tue, Feb 2, 2016 at 8:46 AM, Xiangdong wrote: > Hello everyone, > > If I use the option -download-pastix=1, which directory is the package pastix_5.2.2.20.tar.bz2 downloaded to? It was compiled and installed in arch/externalpackages, but I want to know where the tar file is. I want to copy these tar files into a computer without internet connection and install petsc there. > > If I understand your question correctly, you can just give > > --download-pastix= > > and it will do what you want. > > Thanks, > > Matt > > Thanks. > > Xiangdong > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > From balay at mcs.anl.gov Tue Feb 2 13:27:17 2016 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 2 Feb 2016 13:27:17 -0600 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> Message-ID: On Tue, 2 Feb 2016, Barry Smith wrote: > > > On Feb 2, 2016, at 9:21 AM, Xiangdong wrote: > > > > the package is untar and compiled in $PETSC_ARCH/externalpackages, but the original tarball .tar file is deleted. Is it correct? > > Yes > > When you ran the successful configure it printed to the screen where the tar file came from. You should use curl or wget to get the tarball and then copy it over to the machine you want it on. Or you can copy the whole '$PETSC_ARCH/externalpackages' dir over to this other machine [and place in the same location - for any PETSC_ARCH you plan to use..] Satish From epscodes at gmail.com Tue Feb 2 14:06:58 2016 From: epscodes at gmail.com (Xiangdong) Date: Tue, 2 Feb 2016 15:06:58 -0500 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> Message-ID: On Tue, Feb 2, 2016 at 2:27 PM, Satish Balay wrote: > On Tue, 2 Feb 2016, Barry Smith wrote: > > > > > > On Feb 2, 2016, at 9:21 AM, Xiangdong wrote: > > > > > > the package is untar and compiled in $PETSC_ARCH/externalpackages, but > the original tarball .tar file is deleted. Is it correct? > > > > Yes > > > > When you ran the successful configure it printed to the screen where > the tar file came from. You should use curl or wget to get the tarball and > then copy it over to the machine you want it on. > > > Or you can copy the whole '$PETSC_ARCH/externalpackages' dir over to > this other machine [and place in the same location - for any > PETSC_ARCH you plan to use..] > If I have 15 packages to install, this method would be easy. Once I copied this folder to the new machine, do I use --with-pastix-dir=$PETSC_ARCH/externalpackages/pastix_5.2 or --download-pastix=$PETSC_ARCH/externalpackages/pastix_5.2? The first may fail due to the different libraries installed on that machine. For the second one, can --download-pastix take the directory path instead of the tarball file? Thanks. Xiangdong > > Satish > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Feb 2 14:08:47 2016 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 2 Feb 2016 14:08:47 -0600 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> Message-ID: On Tue, Feb 2, 2016 at 2:06 PM, Xiangdong wrote: > > On Tue, Feb 2, 2016 at 2:27 PM, Satish Balay wrote: > >> On Tue, 2 Feb 2016, Barry Smith wrote: >> >> > >> > > On Feb 2, 2016, at 9:21 AM, Xiangdong wrote: >> > > >> > > the package is untar and compiled in $PETSC_ARCH/externalpackages, >> but the original tarball .tar file is deleted. Is it correct? >> > >> > Yes >> > >> > When you ran the successful configure it printed to the screen where >> the tar file came from. You should use curl or wget to get the tarball and >> then copy it over to the machine you want it on. >> >> >> Or you can copy the whole '$PETSC_ARCH/externalpackages' dir over to >> this other machine [and place in the same location - for any >> PETSC_ARCH you plan to use..] >> > > If I have 15 packages to install, this method would be easy. Once I copied > this folder to the new machine, do I use > --with-pastix-dir=$PETSC_ARCH/externalpackages/pastix_5.2 or > --download-pastix=$PETSC_ARCH/externalpackages/pastix_5.2? The first may > fail due to the different libraries installed on that machine. For the > second one, can --download-pastix take the directory path instead of the > tarball file? > Just use the regular --download-* options. It will find the downloaded packages and rebuild them. Matt > Thanks. > > Xiangdong > >> >> Satish >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Feb 2 14:11:08 2016 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 2 Feb 2016 14:11:08 -0600 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> Message-ID: On Tue, 2 Feb 2016, Xiangdong wrote: > On Tue, Feb 2, 2016 at 2:27 PM, Satish Balay wrote: > > > On Tue, 2 Feb 2016, Barry Smith wrote: > > > > > > > > > On Feb 2, 2016, at 9:21 AM, Xiangdong wrote: > > > > > > > > the package is untar and compiled in $PETSC_ARCH/externalpackages, but > > the original tarball .tar file is deleted. Is it correct? > > > > > > Yes > > > > > > When you ran the successful configure it printed to the screen where > > the tar file came from. You should use curl or wget to get the tarball and > > then copy it over to the machine you want it on. > > > > > > Or you can copy the whole '$PETSC_ARCH/externalpackages' dir over to > > this other machine [and place in the same location - for any > > PETSC_ARCH you plan to use..] > > > > If I have 15 packages to install, this method would be easy. Once I copied > this folder to the new machine, do I use > --with-pastix-dir=$PETSC_ARCH/externalpackages/pastix_5.2 or > --download-pastix=$PETSC_ARCH/externalpackages/pastix_5.2? The first may > fail due to the different libraries installed on that machine. For the > second one, can --download-pastix take the directory path instead of the > tarball file? If you already have $PETSC_ARCH/externalpackages/pastix_5.2 - then you would use: --download-pastix=1 configure first checks if the package is already downloaded or not: - if found [in $PETSC_ARCH/externalpackages/] - it goes ahead and uses it. - if not found - it attempts to downlaod and extract in $PETSC_ARCH/externalpackages/ Satish From hng.email at gmail.com Tue Feb 2 14:20:58 2016 From: hng.email at gmail.com (Hom Nath Gharti) Date: Tue, 2 Feb 2016 15:20:58 -0500 Subject: [petsc-users] PCFIELDSPLIT question In-Reply-To: References: Message-ID: Hi Matt, hi all, I am trying to use PCFIELDSPLIT for my solver which consists of 4 different variables, namely, u (displacement vector), \chi (displacement potential), p(pressure), and \phi (gravity potential). My code segment looks like the following: ====================================================== call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) call KSPGetPC(ksp,pc,ierr) nsplit=4 call PCFieldSplitSetBlockSize(pc,nsplit,ierr); call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); call ISDestroy(gdofu_is,ierr) call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); call ISDestroy(gdofchi_is,ierr) call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); call ISDestroy(gdofp_is,ierr) call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); call ISDestroy(gdofphi_is,ierr) call PCFieldSplitGetSubKSP(pc,nsplit,subksp,ierr); call KSPSetOperators(subksp(1),Amat,Amat,ierr); call KSPSetOperators(subksp(2),Amat,Amat,ierr); call KSPSetOperators(subksp(3),Amat,Amat,ierr); call KSPSetOperators(subksp(4),Amat,Amat,ierr); call KSPSolve(ksp,bvec,xvec,ierr) ====================================================== But I am getting the following error: [79]PETSC ERROR: Null argument, when expecting valid pointer [79]PETSC ERROR: Null Object: Parameter # 1 [79]PETSC ERROR: #1 KSPSetOperators() line 536 in /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interf It looks like I am doing something wrong in "call KSPSetOperators" but I could not figure that out. Could anybody help me to fix this problem? I looked into almost all related examples but I could not really figure out the correct steps after "call PCFieldSplitSetIS". Any help will be greatly appreciated. Best, Hom nath On Sun, Jan 24, 2016 at 7:14 PM, Hom Nath Gharti wrote: > Thank you so much Matt! I will try. > > Hom Nath > > On Sun, Jan 24, 2016 at 6:26 AM, Matthew Knepley wrote: >> On Fri, Jan 22, 2016 at 2:19 PM, Hom Nath Gharti >> wrote: >>> >>> Dear all, >>> >>> I am new to PcFieldSplit. >>> >>> I have a matrix formed using MATMPIAIJ. Is it possible to use >>> PCFIELDSPLIT operations in this type of matrix? Or does it have to be >>> MATMPIBIJ or MATNEST format? >> >> >> Yes, you can split AIJ. >> >>> >>> If possible for MATMPIAIJ, could anybody provide me a simple example >>> or few steps? Variables in the equations are displacement vector, >>> scalar potential and pressure. >> >> >> If you do not have a collocated discretization, then you have to use >> >> >> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html >> >> Thanks, >> >> Matt >> >>> >>> Thanks for help. >>> >>> Hom Nath >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments >> is infinitely more interesting than any results to which their experiments >> lead. >> -- Norbert Wiener From knepley at gmail.com Tue Feb 2 14:24:35 2016 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 2 Feb 2016 14:24:35 -0600 Subject: [petsc-users] PCFIELDSPLIT question In-Reply-To: References: Message-ID: On Tue, Feb 2, 2016 at 2:20 PM, Hom Nath Gharti wrote: > Hi Matt, hi all, > > I am trying to use PCFIELDSPLIT for my solver which consists of 4 > different variables, namely, u (displacement vector), \chi > (displacement potential), p(pressure), and \phi (gravity potential). > > My code segment looks like the following: > ====================================================== > call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) > call KSPGetPC(ksp,pc,ierr) > nsplit=4 > call PCFieldSplitSetBlockSize(pc,nsplit,ierr); > You do not need the statement above. > call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); > call ISDestroy(gdofu_is,ierr) > call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); > call ISDestroy(gdofchi_is,ierr) > call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); > call ISDestroy(gdofp_is,ierr) > call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); > call ISDestroy(gdofphi_is,ierr) > > call PCFieldSplitGetSubKSP(pc,nsplit,subksp,ierr); > These SetOperators() calls are wrong. I am not sure why you want them here. Also, that means you do not need the call above. Thanks, Matt > call KSPSetOperators(subksp(1),Amat,Amat,ierr); > call KSPSetOperators(subksp(2),Amat,Amat,ierr); > call KSPSetOperators(subksp(3),Amat,Amat,ierr); > call KSPSetOperators(subksp(4),Amat,Amat,ierr); > > call KSPSolve(ksp,bvec,xvec,ierr) > ====================================================== > > But I am getting the following error: > [79]PETSC ERROR: Null argument, when expecting valid pointer > [79]PETSC ERROR: Null Object: Parameter # 1 > [79]PETSC ERROR: #1 KSPSetOperators() line 536 in > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interf > > It looks like I am doing something wrong in "call KSPSetOperators" but > I could not figure that out. > > Could anybody help me to fix this problem? I looked into almost all > related examples but I could not really figure out the correct steps > after "call PCFieldSplitSetIS". > > Any help will be greatly appreciated. > > Best, > Hom nath > > On Sun, Jan 24, 2016 at 7:14 PM, Hom Nath Gharti > wrote: > > Thank you so much Matt! I will try. > > > > Hom Nath > > > > On Sun, Jan 24, 2016 at 6:26 AM, Matthew Knepley > wrote: > >> On Fri, Jan 22, 2016 at 2:19 PM, Hom Nath Gharti > >> wrote: > >>> > >>> Dear all, > >>> > >>> I am new to PcFieldSplit. > >>> > >>> I have a matrix formed using MATMPIAIJ. Is it possible to use > >>> PCFIELDSPLIT operations in this type of matrix? Or does it have to be > >>> MATMPIBIJ or MATNEST format? > >> > >> > >> Yes, you can split AIJ. > >> > >>> > >>> If possible for MATMPIAIJ, could anybody provide me a simple example > >>> or few steps? Variables in the equations are displacement vector, > >>> scalar potential and pressure. > >> > >> > >> If you do not have a collocated discretization, then you have to use > >> > >> > >> > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html > >> > >> Thanks, > >> > >> Matt > >> > >>> > >>> Thanks for help. > >>> > >>> Hom Nath > >> > >> > >> > >> > >> -- > >> What most experimenters take for granted before they begin their > experiments > >> is infinitely more interesting than any results to which their > experiments > >> lead. > >> -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hng.email at gmail.com Tue Feb 2 14:35:34 2016 From: hng.email at gmail.com (Hom Nath Gharti) Date: Tue, 2 Feb 2016 15:35:34 -0500 Subject: [petsc-users] PCFIELDSPLIT question In-Reply-To: References: Message-ID: Thank you so much Matt. I now tried the following: ====================================================== call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) call KSPGetPC(ksp,pc,ierr) call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); call ISDestroy(gdofu_is,ierr) call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); call ISDestroy(gdofchi_is,ierr) call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); call ISDestroy(gdofp_is,ierr) call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); call ISDestroy(gdofphi_is,ierr) ! Amat changes in each time steps call KSPSetOperators(ksp,Amat,Amat,ierr) !version >= 3.5.0 call KSPSolve(ksp,bvec,xvec,ierr) ====================================================== But I get the following error: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Petsc has generated inconsistent data [0]PETSC ERROR: Unhandled case, must have at least two fields, not 1 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [0]PETSC ERROR: /tigress/hgharti/gitwork/SPECFEM3D_GLOBEVSPP/./bin/xspecfem3D on a arch-linux2-c-debug named tiger-r3c1n7 by hgharti Tue Feb 2 15: 29:30 2016 [0]PETSC ERROR: Configure options --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64/ --with-cc=mpicc --with-cxx=mpicxx --wit h-fc=mpif90 --with-mpiexec=mpiexec --with-debugging=1 --download-scalapack --download-mumps --download-pastix --download-superlu --download-superlu_dist --download-metis --download-parmetis --download-ptscotch --download-hypre [0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 469 in /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 486 in /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: #3 PCSetUp() line 983 in /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #4 KSPSetUp() line 332 in /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #5 KSPSolve() line 546 in /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c forrtl: error (76): Abort trap signal Am I missing something? Thanks, Hom Nath On Tue, Feb 2, 2016 at 3:24 PM, Matthew Knepley wrote: > On Tue, Feb 2, 2016 at 2:20 PM, Hom Nath Gharti wrote: >> >> Hi Matt, hi all, >> >> I am trying to use PCFIELDSPLIT for my solver which consists of 4 >> different variables, namely, u (displacement vector), \chi >> (displacement potential), p(pressure), and \phi (gravity potential). >> >> My code segment looks like the following: >> ====================================================== >> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >> call KSPGetPC(ksp,pc,ierr) >> nsplit=4 >> call PCFieldSplitSetBlockSize(pc,nsplit,ierr); > > > You do not need the statement above. > >> >> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); >> call ISDestroy(gdofu_is,ierr) >> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); >> call ISDestroy(gdofchi_is,ierr) >> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); >> call ISDestroy(gdofp_is,ierr) >> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); >> call ISDestroy(gdofphi_is,ierr) >> >> call PCFieldSplitGetSubKSP(pc,nsplit,subksp,ierr); > > > These SetOperators() calls are wrong. I am not sure why you want them here. > Also, that means you do not need the call above. > > Thanks, > > Matt > >> >> call KSPSetOperators(subksp(1),Amat,Amat,ierr); >> call KSPSetOperators(subksp(2),Amat,Amat,ierr); >> call KSPSetOperators(subksp(3),Amat,Amat,ierr); >> call KSPSetOperators(subksp(4),Amat,Amat,ierr); >> >> call KSPSolve(ksp,bvec,xvec,ierr) >> ====================================================== >> >> But I am getting the following error: >> [79]PETSC ERROR: Null argument, when expecting valid pointer >> [79]PETSC ERROR: Null Object: Parameter # 1 >> [79]PETSC ERROR: #1 KSPSetOperators() line 536 in >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interf >> >> It looks like I am doing something wrong in "call KSPSetOperators" but >> I could not figure that out. >> >> Could anybody help me to fix this problem? I looked into almost all >> related examples but I could not really figure out the correct steps >> after "call PCFieldSplitSetIS". >> >> Any help will be greatly appreciated. >> >> Best, >> Hom nath >> >> On Sun, Jan 24, 2016 at 7:14 PM, Hom Nath Gharti >> wrote: >> > Thank you so much Matt! I will try. >> > >> > Hom Nath >> > >> > On Sun, Jan 24, 2016 at 6:26 AM, Matthew Knepley >> > wrote: >> >> On Fri, Jan 22, 2016 at 2:19 PM, Hom Nath Gharti >> >> wrote: >> >>> >> >>> Dear all, >> >>> >> >>> I am new to PcFieldSplit. >> >>> >> >>> I have a matrix formed using MATMPIAIJ. Is it possible to use >> >>> PCFIELDSPLIT operations in this type of matrix? Or does it have to be >> >>> MATMPIBIJ or MATNEST format? >> >> >> >> >> >> Yes, you can split AIJ. >> >> >> >>> >> >>> If possible for MATMPIAIJ, could anybody provide me a simple example >> >>> or few steps? Variables in the equations are displacement vector, >> >>> scalar potential and pressure. >> >> >> >> >> >> If you do not have a collocated discretization, then you have to use >> >> >> >> >> >> >> >> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html >> >> >> >> Thanks, >> >> >> >> Matt >> >> >> >>> >> >>> Thanks for help. >> >>> >> >>> Hom Nath >> >> >> >> >> >> >> >> >> >> -- >> >> What most experimenters take for granted before they begin their >> >> experiments >> >> is infinitely more interesting than any results to which their >> >> experiments >> >> lead. >> >> -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener From knepley at gmail.com Tue Feb 2 14:42:16 2016 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 2 Feb 2016 14:42:16 -0600 Subject: [petsc-users] PCFIELDSPLIT question In-Reply-To: References: Message-ID: On Tue, Feb 2, 2016 at 2:35 PM, Hom Nath Gharti wrote: > Thank you so much Matt. > > I now tried the following: > > ====================================================== > call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) > call KSPGetPC(ksp,pc,ierr) > > call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); > call ISDestroy(gdofu_is,ierr) > call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); > call ISDestroy(gdofchi_is,ierr) > call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); > call ISDestroy(gdofp_is,ierr) > call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); > call ISDestroy(gdofphi_is,ierr) > > ! Amat changes in each time steps > call KSPSetOperators(ksp,Amat,Amat,ierr) !version >= 3.5.0 > > call KSPSolve(ksp,bvec,xvec,ierr) > ====================================================== > I am guessing that "0" is not a valid string for your Fortran compiler. Are you sure its not single quotes '0'? Matt > But I get the following error: > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Petsc has generated inconsistent data > [0]PETSC ERROR: Unhandled case, must have at least two fields, not 1 > [0]PETSC ERROR: See > http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > [0]PETSC ERROR: > /tigress/hgharti/gitwork/SPECFEM3D_GLOBEVSPP/./bin/xspecfem3D on a > arch-linux2-c-debug named tiger-r3c1n7 by hgharti Tue Feb 2 15: > 29:30 2016 > [0]PETSC ERROR: Configure options > --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64/ > --with-cc=mpicc --with-cxx=mpicxx --wit > h-fc=mpif90 --with-mpiexec=mpiexec --with-debugging=1 > --download-scalapack --download-mumps --download-pastix > --download-superlu --download-superlu_dist --download-metis > --download-parmetis --download-ptscotch --download-hypre > [0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 469 in > > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 486 in > > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > [0]PETSC ERROR: #3 PCSetUp() line 983 in > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #4 KSPSetUp() line 332 in > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #5 KSPSolve() line 546 in > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c > forrtl: error (76): Abort trap signal > > Am I missing something? > > Thanks, > Hom Nath > > On Tue, Feb 2, 2016 at 3:24 PM, Matthew Knepley wrote: > > On Tue, Feb 2, 2016 at 2:20 PM, Hom Nath Gharti > wrote: > >> > >> Hi Matt, hi all, > >> > >> I am trying to use PCFIELDSPLIT for my solver which consists of 4 > >> different variables, namely, u (displacement vector), \chi > >> (displacement potential), p(pressure), and \phi (gravity potential). > >> > >> My code segment looks like the following: > >> ====================================================== > >> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) > >> call KSPGetPC(ksp,pc,ierr) > >> nsplit=4 > >> call PCFieldSplitSetBlockSize(pc,nsplit,ierr); > > > > > > You do not need the statement above. > > > >> > >> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); > >> call ISDestroy(gdofu_is,ierr) > >> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); > >> call ISDestroy(gdofchi_is,ierr) > >> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); > >> call ISDestroy(gdofp_is,ierr) > >> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); > >> call ISDestroy(gdofphi_is,ierr) > >> > >> call PCFieldSplitGetSubKSP(pc,nsplit,subksp,ierr); > > > > > > These SetOperators() calls are wrong. I am not sure why you want them > here. > > Also, that means you do not need the call above. > > > > Thanks, > > > > Matt > > > >> > >> call KSPSetOperators(subksp(1),Amat,Amat,ierr); > >> call KSPSetOperators(subksp(2),Amat,Amat,ierr); > >> call KSPSetOperators(subksp(3),Amat,Amat,ierr); > >> call KSPSetOperators(subksp(4),Amat,Amat,ierr); > >> > >> call KSPSolve(ksp,bvec,xvec,ierr) > >> ====================================================== > >> > >> But I am getting the following error: > >> [79]PETSC ERROR: Null argument, when expecting valid pointer > >> [79]PETSC ERROR: Null Object: Parameter # 1 > >> [79]PETSC ERROR: #1 KSPSetOperators() line 536 in > >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interf > >> > >> It looks like I am doing something wrong in "call KSPSetOperators" but > >> I could not figure that out. > >> > >> Could anybody help me to fix this problem? I looked into almost all > >> related examples but I could not really figure out the correct steps > >> after "call PCFieldSplitSetIS". > >> > >> Any help will be greatly appreciated. > >> > >> Best, > >> Hom nath > >> > >> On Sun, Jan 24, 2016 at 7:14 PM, Hom Nath Gharti > >> wrote: > >> > Thank you so much Matt! I will try. > >> > > >> > Hom Nath > >> > > >> > On Sun, Jan 24, 2016 at 6:26 AM, Matthew Knepley > >> > wrote: > >> >> On Fri, Jan 22, 2016 at 2:19 PM, Hom Nath Gharti < > hng.email at gmail.com> > >> >> wrote: > >> >>> > >> >>> Dear all, > >> >>> > >> >>> I am new to PcFieldSplit. > >> >>> > >> >>> I have a matrix formed using MATMPIAIJ. Is it possible to use > >> >>> PCFIELDSPLIT operations in this type of matrix? Or does it have to > be > >> >>> MATMPIBIJ or MATNEST format? > >> >> > >> >> > >> >> Yes, you can split AIJ. > >> >> > >> >>> > >> >>> If possible for MATMPIAIJ, could anybody provide me a simple example > >> >>> or few steps? Variables in the equations are displacement vector, > >> >>> scalar potential and pressure. > >> >> > >> >> > >> >> If you do not have a collocated discretization, then you have to use > >> >> > >> >> > >> >> > >> >> > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html > >> >> > >> >> Thanks, > >> >> > >> >> Matt > >> >> > >> >>> > >> >>> Thanks for help. > >> >>> > >> >>> Hom Nath > >> >> > >> >> > >> >> > >> >> > >> >> -- > >> >> What most experimenters take for granted before they begin their > >> >> experiments > >> >> is infinitely more interesting than any results to which their > >> >> experiments > >> >> lead. > >> >> -- Norbert Wiener > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments > > is infinitely more interesting than any results to which their > experiments > > lead. > > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hng.email at gmail.com Tue Feb 2 15:01:23 2016 From: hng.email at gmail.com (Hom Nath Gharti) Date: Tue, 2 Feb 2016 16:01:23 -0500 Subject: [petsc-users] PCFIELDSPLIT question In-Reply-To: References: Message-ID: Thanks again Matt. Unfortunately got the same errors with '0'. I think both are valid in Fortran. Hom Nath On Tue, Feb 2, 2016 at 3:42 PM, Matthew Knepley wrote: > On Tue, Feb 2, 2016 at 2:35 PM, Hom Nath Gharti wrote: >> >> Thank you so much Matt. >> >> I now tried the following: >> >> ====================================================== >> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >> call KSPGetPC(ksp,pc,ierr) >> >> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); >> call ISDestroy(gdofu_is,ierr) >> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); >> call ISDestroy(gdofchi_is,ierr) >> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); >> call ISDestroy(gdofp_is,ierr) >> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); >> call ISDestroy(gdofphi_is,ierr) >> >> ! Amat changes in each time steps >> call KSPSetOperators(ksp,Amat,Amat,ierr) !version >= 3.5.0 >> >> call KSPSolve(ksp,bvec,xvec,ierr) >> ====================================================== > > > I am guessing that "0" is not a valid string for your Fortran compiler. Are > you sure > its not single quotes '0'? > > Matt > >> >> But I get the following error: >> >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Petsc has generated inconsistent data >> [0]PETSC ERROR: Unhandled case, must have at least two fields, not 1 >> [0]PETSC ERROR: See >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >> shooting. >> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >> [0]PETSC ERROR: >> /tigress/hgharti/gitwork/SPECFEM3D_GLOBEVSPP/./bin/xspecfem3D on a >> arch-linux2-c-debug named tiger-r3c1n7 by hgharti Tue Feb 2 15: >> 29:30 2016 >> [0]PETSC ERROR: Configure options >> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64/ >> --with-cc=mpicc --with-cxx=mpicxx --wit >> h-fc=mpif90 --with-mpiexec=mpiexec --with-debugging=1 >> --download-scalapack --download-mumps --download-pastix >> --download-superlu --download-superlu_dist --download-metis >> --download-parmetis --download-ptscotch --download-hypre >> [0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 469 in >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 486 in >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> [0]PETSC ERROR: #3 PCSetUp() line 983 in >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: #4 KSPSetUp() line 332 in >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: #5 KSPSolve() line 546 in >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >> forrtl: error (76): Abort trap signal >> >> Am I missing something? >> >> Thanks, >> Hom Nath >> >> On Tue, Feb 2, 2016 at 3:24 PM, Matthew Knepley wrote: >> > On Tue, Feb 2, 2016 at 2:20 PM, Hom Nath Gharti >> > wrote: >> >> >> >> Hi Matt, hi all, >> >> >> >> I am trying to use PCFIELDSPLIT for my solver which consists of 4 >> >> different variables, namely, u (displacement vector), \chi >> >> (displacement potential), p(pressure), and \phi (gravity potential). >> >> >> >> My code segment looks like the following: >> >> ====================================================== >> >> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >> >> call KSPGetPC(ksp,pc,ierr) >> >> nsplit=4 >> >> call PCFieldSplitSetBlockSize(pc,nsplit,ierr); >> > >> > >> > You do not need the statement above. >> > >> >> >> >> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); >> >> call ISDestroy(gdofu_is,ierr) >> >> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); >> >> call ISDestroy(gdofchi_is,ierr) >> >> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); >> >> call ISDestroy(gdofp_is,ierr) >> >> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); >> >> call ISDestroy(gdofphi_is,ierr) >> >> >> >> call PCFieldSplitGetSubKSP(pc,nsplit,subksp,ierr); >> > >> > >> > These SetOperators() calls are wrong. I am not sure why you want them >> > here. >> > Also, that means you do not need the call above. >> > >> > Thanks, >> > >> > Matt >> > >> >> >> >> call KSPSetOperators(subksp(1),Amat,Amat,ierr); >> >> call KSPSetOperators(subksp(2),Amat,Amat,ierr); >> >> call KSPSetOperators(subksp(3),Amat,Amat,ierr); >> >> call KSPSetOperators(subksp(4),Amat,Amat,ierr); >> >> >> >> call KSPSolve(ksp,bvec,xvec,ierr) >> >> ====================================================== >> >> >> >> But I am getting the following error: >> >> [79]PETSC ERROR: Null argument, when expecting valid pointer >> >> [79]PETSC ERROR: Null Object: Parameter # 1 >> >> [79]PETSC ERROR: #1 KSPSetOperators() line 536 in >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interf >> >> >> >> It looks like I am doing something wrong in "call KSPSetOperators" but >> >> I could not figure that out. >> >> >> >> Could anybody help me to fix this problem? I looked into almost all >> >> related examples but I could not really figure out the correct steps >> >> after "call PCFieldSplitSetIS". >> >> >> >> Any help will be greatly appreciated. >> >> >> >> Best, >> >> Hom nath >> >> >> >> On Sun, Jan 24, 2016 at 7:14 PM, Hom Nath Gharti >> >> wrote: >> >> > Thank you so much Matt! I will try. >> >> > >> >> > Hom Nath >> >> > >> >> > On Sun, Jan 24, 2016 at 6:26 AM, Matthew Knepley >> >> > wrote: >> >> >> On Fri, Jan 22, 2016 at 2:19 PM, Hom Nath Gharti >> >> >> >> >> >> wrote: >> >> >>> >> >> >>> Dear all, >> >> >>> >> >> >>> I am new to PcFieldSplit. >> >> >>> >> >> >>> I have a matrix formed using MATMPIAIJ. Is it possible to use >> >> >>> PCFIELDSPLIT operations in this type of matrix? Or does it have to >> >> >>> be >> >> >>> MATMPIBIJ or MATNEST format? >> >> >> >> >> >> >> >> >> Yes, you can split AIJ. >> >> >> >> >> >>> >> >> >>> If possible for MATMPIAIJ, could anybody provide me a simple >> >> >>> example >> >> >>> or few steps? Variables in the equations are displacement vector, >> >> >>> scalar potential and pressure. >> >> >> >> >> >> >> >> >> If you do not have a collocated discretization, then you have to use >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html >> >> >> >> >> >> Thanks, >> >> >> >> >> >> Matt >> >> >> >> >> >>> >> >> >>> Thanks for help. >> >> >>> >> >> >>> Hom Nath >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> What most experimenters take for granted before they begin their >> >> >> experiments >> >> >> is infinitely more interesting than any results to which their >> >> >> experiments >> >> >> lead. >> >> >> -- Norbert Wiener >> > >> > >> > >> > >> > -- >> > What most experimenters take for granted before they begin their >> > experiments >> > is infinitely more interesting than any results to which their >> > experiments >> > lead. >> > -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener From knepley at gmail.com Tue Feb 2 15:02:56 2016 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 2 Feb 2016 15:02:56 -0600 Subject: [petsc-users] PCFIELDSPLIT question In-Reply-To: References: Message-ID: On Tue, Feb 2, 2016 at 3:01 PM, Hom Nath Gharti wrote: > Thanks again Matt. Unfortunately got the same errors with '0'. I think > both are valid in Fortran. > Then you will have to go in the debugger and see why its not creating 4 splits, since this is exactly what it does in our tests. This is how all the SNES tests that I use work. I am sure its a Fortran problem. Thanks, Matt > Hom Nath > > On Tue, Feb 2, 2016 at 3:42 PM, Matthew Knepley wrote: > > On Tue, Feb 2, 2016 at 2:35 PM, Hom Nath Gharti > wrote: > >> > >> Thank you so much Matt. > >> > >> I now tried the following: > >> > >> ====================================================== > >> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) > >> call KSPGetPC(ksp,pc,ierr) > >> > >> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); > >> call ISDestroy(gdofu_is,ierr) > >> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); > >> call ISDestroy(gdofchi_is,ierr) > >> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); > >> call ISDestroy(gdofp_is,ierr) > >> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); > >> call ISDestroy(gdofphi_is,ierr) > >> > >> ! Amat changes in each time steps > >> call KSPSetOperators(ksp,Amat,Amat,ierr) !version >= 3.5.0 > >> > >> call KSPSolve(ksp,bvec,xvec,ierr) > >> ====================================================== > > > > > > I am guessing that "0" is not a valid string for your Fortran compiler. > Are > > you sure > > its not single quotes '0'? > > > > Matt > > > >> > >> But I get the following error: > >> > >> [0]PETSC ERROR: --------------------- Error Message > >> -------------------------------------------------------------- > >> [0]PETSC ERROR: Petsc has generated inconsistent data > >> [0]PETSC ERROR: Unhandled case, must have at least two fields, not 1 > >> [0]PETSC ERROR: See > >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > >> shooting. > >> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > >> [0]PETSC ERROR: > >> /tigress/hgharti/gitwork/SPECFEM3D_GLOBEVSPP/./bin/xspecfem3D on a > >> arch-linux2-c-debug named tiger-r3c1n7 by hgharti Tue Feb 2 15: > >> 29:30 2016 > >> [0]PETSC ERROR: Configure options > >> > --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64/ > >> --with-cc=mpicc --with-cxx=mpicxx --wit > >> h-fc=mpif90 --with-mpiexec=mpiexec --with-debugging=1 > >> --download-scalapack --download-mumps --download-pastix > >> --download-superlu --download-superlu_dist --download-metis > >> --download-parmetis --download-ptscotch --download-hypre > >> [0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 469 in > >> > >> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > >> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 486 in > >> > >> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > >> [0]PETSC ERROR: #3 PCSetUp() line 983 in > >> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/interface/precon.c > >> [0]PETSC ERROR: #4 KSPSetUp() line 332 in > >> > >> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c > >> [0]PETSC ERROR: #5 KSPSolve() line 546 in > >> > >> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c > >> forrtl: error (76): Abort trap signal > >> > >> Am I missing something? > >> > >> Thanks, > >> Hom Nath > >> > >> On Tue, Feb 2, 2016 at 3:24 PM, Matthew Knepley > wrote: > >> > On Tue, Feb 2, 2016 at 2:20 PM, Hom Nath Gharti > >> > wrote: > >> >> > >> >> Hi Matt, hi all, > >> >> > >> >> I am trying to use PCFIELDSPLIT for my solver which consists of 4 > >> >> different variables, namely, u (displacement vector), \chi > >> >> (displacement potential), p(pressure), and \phi (gravity potential). > >> >> > >> >> My code segment looks like the following: > >> >> ====================================================== > >> >> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) > >> >> call KSPGetPC(ksp,pc,ierr) > >> >> nsplit=4 > >> >> call PCFieldSplitSetBlockSize(pc,nsplit,ierr); > >> > > >> > > >> > You do not need the statement above. > >> > > >> >> > >> >> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); > >> >> call ISDestroy(gdofu_is,ierr) > >> >> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); > >> >> call ISDestroy(gdofchi_is,ierr) > >> >> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); > >> >> call ISDestroy(gdofp_is,ierr) > >> >> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); > >> >> call ISDestroy(gdofphi_is,ierr) > >> >> > >> >> call PCFieldSplitGetSubKSP(pc,nsplit,subksp,ierr); > >> > > >> > > >> > These SetOperators() calls are wrong. I am not sure why you want them > >> > here. > >> > Also, that means you do not need the call above. > >> > > >> > Thanks, > >> > > >> > Matt > >> > > >> >> > >> >> call KSPSetOperators(subksp(1),Amat,Amat,ierr); > >> >> call KSPSetOperators(subksp(2),Amat,Amat,ierr); > >> >> call KSPSetOperators(subksp(3),Amat,Amat,ierr); > >> >> call KSPSetOperators(subksp(4),Amat,Amat,ierr); > >> >> > >> >> call KSPSolve(ksp,bvec,xvec,ierr) > >> >> ====================================================== > >> >> > >> >> But I am getting the following error: > >> >> [79]PETSC ERROR: Null argument, when expecting valid pointer > >> >> [79]PETSC ERROR: Null Object: Parameter # 1 > >> >> [79]PETSC ERROR: #1 KSPSetOperators() line 536 in > >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interf > >> >> > >> >> It looks like I am doing something wrong in "call KSPSetOperators" > but > >> >> I could not figure that out. > >> >> > >> >> Could anybody help me to fix this problem? I looked into almost all > >> >> related examples but I could not really figure out the correct steps > >> >> after "call PCFieldSplitSetIS". > >> >> > >> >> Any help will be greatly appreciated. > >> >> > >> >> Best, > >> >> Hom nath > >> >> > >> >> On Sun, Jan 24, 2016 at 7:14 PM, Hom Nath Gharti < > hng.email at gmail.com> > >> >> wrote: > >> >> > Thank you so much Matt! I will try. > >> >> > > >> >> > Hom Nath > >> >> > > >> >> > On Sun, Jan 24, 2016 at 6:26 AM, Matthew Knepley < > knepley at gmail.com> > >> >> > wrote: > >> >> >> On Fri, Jan 22, 2016 at 2:19 PM, Hom Nath Gharti > >> >> >> > >> >> >> wrote: > >> >> >>> > >> >> >>> Dear all, > >> >> >>> > >> >> >>> I am new to PcFieldSplit. > >> >> >>> > >> >> >>> I have a matrix formed using MATMPIAIJ. Is it possible to use > >> >> >>> PCFIELDSPLIT operations in this type of matrix? Or does it have > to > >> >> >>> be > >> >> >>> MATMPIBIJ or MATNEST format? > >> >> >> > >> >> >> > >> >> >> Yes, you can split AIJ. > >> >> >> > >> >> >>> > >> >> >>> If possible for MATMPIAIJ, could anybody provide me a simple > >> >> >>> example > >> >> >>> or few steps? Variables in the equations are displacement vector, > >> >> >>> scalar potential and pressure. > >> >> >> > >> >> >> > >> >> >> If you do not have a collocated discretization, then you have to > use > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html > >> >> >> > >> >> >> Thanks, > >> >> >> > >> >> >> Matt > >> >> >> > >> >> >>> > >> >> >>> Thanks for help. > >> >> >>> > >> >> >>> Hom Nath > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> -- > >> >> >> What most experimenters take for granted before they begin their > >> >> >> experiments > >> >> >> is infinitely more interesting than any results to which their > >> >> >> experiments > >> >> >> lead. > >> >> >> -- Norbert Wiener > >> > > >> > > >> > > >> > > >> > -- > >> > What most experimenters take for granted before they begin their > >> > experiments > >> > is infinitely more interesting than any results to which their > >> > experiments > >> > lead. > >> > -- Norbert Wiener > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments > > is infinitely more interesting than any results to which their > experiments > > lead. > > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hng.email at gmail.com Tue Feb 2 15:11:00 2016 From: hng.email at gmail.com (Hom Nath Gharti) Date: Tue, 2 Feb 2016 16:11:00 -0500 Subject: [petsc-users] PCFIELDSPLIT question In-Reply-To: References: Message-ID: Thanks a lot. I will try. Hom Nath On Tue, Feb 2, 2016 at 4:02 PM, Matthew Knepley wrote: > On Tue, Feb 2, 2016 at 3:01 PM, Hom Nath Gharti wrote: >> >> Thanks again Matt. Unfortunately got the same errors with '0'. I think >> both are valid in Fortran. > > > Then you will have to go in the debugger and see why its not creating 4 > splits, since this is exactly > what it does in our tests. This is how all the SNES tests that I use work. I > am sure its a Fortran > problem. > > Thanks, > > Matt > >> >> Hom Nath >> >> On Tue, Feb 2, 2016 at 3:42 PM, Matthew Knepley wrote: >> > On Tue, Feb 2, 2016 at 2:35 PM, Hom Nath Gharti >> > wrote: >> >> >> >> Thank you so much Matt. >> >> >> >> I now tried the following: >> >> >> >> ====================================================== >> >> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >> >> call KSPGetPC(ksp,pc,ierr) >> >> >> >> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); >> >> call ISDestroy(gdofu_is,ierr) >> >> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); >> >> call ISDestroy(gdofchi_is,ierr) >> >> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); >> >> call ISDestroy(gdofp_is,ierr) >> >> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); >> >> call ISDestroy(gdofphi_is,ierr) >> >> >> >> ! Amat changes in each time steps >> >> call KSPSetOperators(ksp,Amat,Amat,ierr) !version >= 3.5.0 >> >> >> >> call KSPSolve(ksp,bvec,xvec,ierr) >> >> ====================================================== >> > >> > >> > I am guessing that "0" is not a valid string for your Fortran compiler. >> > Are >> > you sure >> > its not single quotes '0'? >> > >> > Matt >> > >> >> >> >> But I get the following error: >> >> >> >> [0]PETSC ERROR: --------------------- Error Message >> >> -------------------------------------------------------------- >> >> [0]PETSC ERROR: Petsc has generated inconsistent data >> >> [0]PETSC ERROR: Unhandled case, must have at least two fields, not 1 >> >> [0]PETSC ERROR: See >> >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >> >> shooting. >> >> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >> >> [0]PETSC ERROR: >> >> /tigress/hgharti/gitwork/SPECFEM3D_GLOBEVSPP/./bin/xspecfem3D on a >> >> arch-linux2-c-debug named tiger-r3c1n7 by hgharti Tue Feb 2 15: >> >> 29:30 2016 >> >> [0]PETSC ERROR: Configure options >> >> >> >> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64/ >> >> --with-cc=mpicc --with-cxx=mpicxx --wit >> >> h-fc=mpif90 --with-mpiexec=mpiexec --with-debugging=1 >> >> --download-scalapack --download-mumps --download-pastix >> >> --download-superlu --download-superlu_dist --download-metis >> >> --download-parmetis --download-ptscotch --download-hypre >> >> [0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 469 in >> >> >> >> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> >> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 486 in >> >> >> >> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> >> [0]PETSC ERROR: #3 PCSetUp() line 983 in >> >> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/interface/precon.c >> >> [0]PETSC ERROR: #4 KSPSetUp() line 332 in >> >> >> >> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >> >> [0]PETSC ERROR: #5 KSPSolve() line 546 in >> >> >> >> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >> >> forrtl: error (76): Abort trap signal >> >> >> >> Am I missing something? >> >> >> >> Thanks, >> >> Hom Nath >> >> >> >> On Tue, Feb 2, 2016 at 3:24 PM, Matthew Knepley >> >> wrote: >> >> > On Tue, Feb 2, 2016 at 2:20 PM, Hom Nath Gharti >> >> > wrote: >> >> >> >> >> >> Hi Matt, hi all, >> >> >> >> >> >> I am trying to use PCFIELDSPLIT for my solver which consists of 4 >> >> >> different variables, namely, u (displacement vector), \chi >> >> >> (displacement potential), p(pressure), and \phi (gravity potential). >> >> >> >> >> >> My code segment looks like the following: >> >> >> ====================================================== >> >> >> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >> >> >> call KSPGetPC(ksp,pc,ierr) >> >> >> nsplit=4 >> >> >> call PCFieldSplitSetBlockSize(pc,nsplit,ierr); >> >> > >> >> > >> >> > You do not need the statement above. >> >> > >> >> >> >> >> >> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); >> >> >> call ISDestroy(gdofu_is,ierr) >> >> >> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); >> >> >> call ISDestroy(gdofchi_is,ierr) >> >> >> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); >> >> >> call ISDestroy(gdofp_is,ierr) >> >> >> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); >> >> >> call ISDestroy(gdofphi_is,ierr) >> >> >> >> >> >> call PCFieldSplitGetSubKSP(pc,nsplit,subksp,ierr); >> >> > >> >> > >> >> > These SetOperators() calls are wrong. I am not sure why you want them >> >> > here. >> >> > Also, that means you do not need the call above. >> >> > >> >> > Thanks, >> >> > >> >> > Matt >> >> > >> >> >> >> >> >> call KSPSetOperators(subksp(1),Amat,Amat,ierr); >> >> >> call KSPSetOperators(subksp(2),Amat,Amat,ierr); >> >> >> call KSPSetOperators(subksp(3),Amat,Amat,ierr); >> >> >> call KSPSetOperators(subksp(4),Amat,Amat,ierr); >> >> >> >> >> >> call KSPSolve(ksp,bvec,xvec,ierr) >> >> >> ====================================================== >> >> >> >> >> >> But I am getting the following error: >> >> >> [79]PETSC ERROR: Null argument, when expecting valid pointer >> >> >> [79]PETSC ERROR: Null Object: Parameter # 1 >> >> >> [79]PETSC ERROR: #1 KSPSetOperators() line 536 in >> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interf >> >> >> >> >> >> It looks like I am doing something wrong in "call KSPSetOperators" >> >> >> but >> >> >> I could not figure that out. >> >> >> >> >> >> Could anybody help me to fix this problem? I looked into almost all >> >> >> related examples but I could not really figure out the correct steps >> >> >> after "call PCFieldSplitSetIS". >> >> >> >> >> >> Any help will be greatly appreciated. >> >> >> >> >> >> Best, >> >> >> Hom nath >> >> >> >> >> >> On Sun, Jan 24, 2016 at 7:14 PM, Hom Nath Gharti >> >> >> >> >> >> wrote: >> >> >> > Thank you so much Matt! I will try. >> >> >> > >> >> >> > Hom Nath >> >> >> > >> >> >> > On Sun, Jan 24, 2016 at 6:26 AM, Matthew Knepley >> >> >> > >> >> >> > wrote: >> >> >> >> On Fri, Jan 22, 2016 at 2:19 PM, Hom Nath Gharti >> >> >> >> >> >> >> >> wrote: >> >> >> >>> >> >> >> >>> Dear all, >> >> >> >>> >> >> >> >>> I am new to PcFieldSplit. >> >> >> >>> >> >> >> >>> I have a matrix formed using MATMPIAIJ. Is it possible to use >> >> >> >>> PCFIELDSPLIT operations in this type of matrix? Or does it have >> >> >> >>> to >> >> >> >>> be >> >> >> >>> MATMPIBIJ or MATNEST format? >> >> >> >> >> >> >> >> >> >> >> >> Yes, you can split AIJ. >> >> >> >> >> >> >> >>> >> >> >> >>> If possible for MATMPIAIJ, could anybody provide me a simple >> >> >> >>> example >> >> >> >>> or few steps? Variables in the equations are displacement >> >> >> >>> vector, >> >> >> >>> scalar potential and pressure. >> >> >> >> >> >> >> >> >> >> >> >> If you do not have a collocated discretization, then you have to >> >> >> >> use >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html >> >> >> >> >> >> >> >> Thanks, >> >> >> >> >> >> >> >> Matt >> >> >> >> >> >> >> >>> >> >> >> >>> Thanks for help. >> >> >> >>> >> >> >> >>> Hom Nath >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> >> What most experimenters take for granted before they begin their >> >> >> >> experiments >> >> >> >> is infinitely more interesting than any results to which their >> >> >> >> experiments >> >> >> >> lead. >> >> >> >> -- Norbert Wiener >> >> > >> >> > >> >> > >> >> > >> >> > -- >> >> > What most experimenters take for granted before they begin their >> >> > experiments >> >> > is infinitely more interesting than any results to which their >> >> > experiments >> >> > lead. >> >> > -- Norbert Wiener >> > >> > >> > >> > >> > -- >> > What most experimenters take for granted before they begin their >> > experiments >> > is infinitely more interesting than any results to which their >> > experiments >> > lead. >> > -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener From knepley at gmail.com Tue Feb 2 15:54:31 2016 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 2 Feb 2016 15:54:31 -0600 Subject: [petsc-users] PCFIELDSPLIT question In-Reply-To: References: Message-ID: On Tue, Feb 2, 2016 at 3:11 PM, Hom Nath Gharti wrote: > Thanks a lot. I will try. > Also, if you send a small test case, we can run it and tell you the problem. Matt > Hom Nath > > On Tue, Feb 2, 2016 at 4:02 PM, Matthew Knepley wrote: > > On Tue, Feb 2, 2016 at 3:01 PM, Hom Nath Gharti > wrote: > >> > >> Thanks again Matt. Unfortunately got the same errors with '0'. I think > >> both are valid in Fortran. > > > > > > Then you will have to go in the debugger and see why its not creating 4 > > splits, since this is exactly > > what it does in our tests. This is how all the SNES tests that I use > work. I > > am sure its a Fortran > > problem. > > > > Thanks, > > > > Matt > > > >> > >> Hom Nath > >> > >> On Tue, Feb 2, 2016 at 3:42 PM, Matthew Knepley > wrote: > >> > On Tue, Feb 2, 2016 at 2:35 PM, Hom Nath Gharti > >> > wrote: > >> >> > >> >> Thank you so much Matt. > >> >> > >> >> I now tried the following: > >> >> > >> >> ====================================================== > >> >> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) > >> >> call KSPGetPC(ksp,pc,ierr) > >> >> > >> >> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); > >> >> call ISDestroy(gdofu_is,ierr) > >> >> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); > >> >> call ISDestroy(gdofchi_is,ierr) > >> >> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); > >> >> call ISDestroy(gdofp_is,ierr) > >> >> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); > >> >> call ISDestroy(gdofphi_is,ierr) > >> >> > >> >> ! Amat changes in each time steps > >> >> call KSPSetOperators(ksp,Amat,Amat,ierr) !version >= 3.5.0 > >> >> > >> >> call KSPSolve(ksp,bvec,xvec,ierr) > >> >> ====================================================== > >> > > >> > > >> > I am guessing that "0" is not a valid string for your Fortran > compiler. > >> > Are > >> > you sure > >> > its not single quotes '0'? > >> > > >> > Matt > >> > > >> >> > >> >> But I get the following error: > >> >> > >> >> [0]PETSC ERROR: --------------------- Error Message > >> >> -------------------------------------------------------------- > >> >> [0]PETSC ERROR: Petsc has generated inconsistent data > >> >> [0]PETSC ERROR: Unhandled case, must have at least two fields, not 1 > >> >> [0]PETSC ERROR: See > >> >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > >> >> shooting. > >> >> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > >> >> [0]PETSC ERROR: > >> >> /tigress/hgharti/gitwork/SPECFEM3D_GLOBEVSPP/./bin/xspecfem3D on a > >> >> arch-linux2-c-debug named tiger-r3c1n7 by hgharti Tue Feb 2 15: > >> >> 29:30 2016 > >> >> [0]PETSC ERROR: Configure options > >> >> > >> >> > --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64/ > >> >> --with-cc=mpicc --with-cxx=mpicxx --wit > >> >> h-fc=mpif90 --with-mpiexec=mpiexec --with-debugging=1 > >> >> --download-scalapack --download-mumps --download-pastix > >> >> --download-superlu --download-superlu_dist --download-metis > >> >> --download-parmetis --download-ptscotch --download-hypre > >> >> [0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 469 in > >> >> > >> >> > >> >> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > >> >> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 486 in > >> >> > >> >> > >> >> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > >> >> [0]PETSC ERROR: #3 PCSetUp() line 983 in > >> >> > >> >> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/interface/precon.c > >> >> [0]PETSC ERROR: #4 KSPSetUp() line 332 in > >> >> > >> >> > >> >> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c > >> >> [0]PETSC ERROR: #5 KSPSolve() line 546 in > >> >> > >> >> > >> >> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c > >> >> forrtl: error (76): Abort trap signal > >> >> > >> >> Am I missing something? > >> >> > >> >> Thanks, > >> >> Hom Nath > >> >> > >> >> On Tue, Feb 2, 2016 at 3:24 PM, Matthew Knepley > >> >> wrote: > >> >> > On Tue, Feb 2, 2016 at 2:20 PM, Hom Nath Gharti < > hng.email at gmail.com> > >> >> > wrote: > >> >> >> > >> >> >> Hi Matt, hi all, > >> >> >> > >> >> >> I am trying to use PCFIELDSPLIT for my solver which consists of 4 > >> >> >> different variables, namely, u (displacement vector), \chi > >> >> >> (displacement potential), p(pressure), and \phi (gravity > potential). > >> >> >> > >> >> >> My code segment looks like the following: > >> >> >> ====================================================== > >> >> >> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) > >> >> >> call KSPGetPC(ksp,pc,ierr) > >> >> >> nsplit=4 > >> >> >> call PCFieldSplitSetBlockSize(pc,nsplit,ierr); > >> >> > > >> >> > > >> >> > You do not need the statement above. > >> >> > > >> >> >> > >> >> >> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); > >> >> >> call ISDestroy(gdofu_is,ierr) > >> >> >> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); > >> >> >> call ISDestroy(gdofchi_is,ierr) > >> >> >> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); > >> >> >> call ISDestroy(gdofp_is,ierr) > >> >> >> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); > >> >> >> call ISDestroy(gdofphi_is,ierr) > >> >> >> > >> >> >> call PCFieldSplitGetSubKSP(pc,nsplit,subksp,ierr); > >> >> > > >> >> > > >> >> > These SetOperators() calls are wrong. I am not sure why you want > them > >> >> > here. > >> >> > Also, that means you do not need the call above. > >> >> > > >> >> > Thanks, > >> >> > > >> >> > Matt > >> >> > > >> >> >> > >> >> >> call KSPSetOperators(subksp(1),Amat,Amat,ierr); > >> >> >> call KSPSetOperators(subksp(2),Amat,Amat,ierr); > >> >> >> call KSPSetOperators(subksp(3),Amat,Amat,ierr); > >> >> >> call KSPSetOperators(subksp(4),Amat,Amat,ierr); > >> >> >> > >> >> >> call KSPSolve(ksp,bvec,xvec,ierr) > >> >> >> ====================================================== > >> >> >> > >> >> >> But I am getting the following error: > >> >> >> [79]PETSC ERROR: Null argument, when expecting valid pointer > >> >> >> [79]PETSC ERROR: Null Object: Parameter # 1 > >> >> >> [79]PETSC ERROR: #1 KSPSetOperators() line 536 in > >> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interf > >> >> >> > >> >> >> It looks like I am doing something wrong in "call KSPSetOperators" > >> >> >> but > >> >> >> I could not figure that out. > >> >> >> > >> >> >> Could anybody help me to fix this problem? I looked into almost > all > >> >> >> related examples but I could not really figure out the correct > steps > >> >> >> after "call PCFieldSplitSetIS". > >> >> >> > >> >> >> Any help will be greatly appreciated. > >> >> >> > >> >> >> Best, > >> >> >> Hom nath > >> >> >> > >> >> >> On Sun, Jan 24, 2016 at 7:14 PM, Hom Nath Gharti > >> >> >> > >> >> >> wrote: > >> >> >> > Thank you so much Matt! I will try. > >> >> >> > > >> >> >> > Hom Nath > >> >> >> > > >> >> >> > On Sun, Jan 24, 2016 at 6:26 AM, Matthew Knepley > >> >> >> > > >> >> >> > wrote: > >> >> >> >> On Fri, Jan 22, 2016 at 2:19 PM, Hom Nath Gharti > >> >> >> >> > >> >> >> >> wrote: > >> >> >> >>> > >> >> >> >>> Dear all, > >> >> >> >>> > >> >> >> >>> I am new to PcFieldSplit. > >> >> >> >>> > >> >> >> >>> I have a matrix formed using MATMPIAIJ. Is it possible to use > >> >> >> >>> PCFIELDSPLIT operations in this type of matrix? Or does it > have > >> >> >> >>> to > >> >> >> >>> be > >> >> >> >>> MATMPIBIJ or MATNEST format? > >> >> >> >> > >> >> >> >> > >> >> >> >> Yes, you can split AIJ. > >> >> >> >> > >> >> >> >>> > >> >> >> >>> If possible for MATMPIAIJ, could anybody provide me a simple > >> >> >> >>> example > >> >> >> >>> or few steps? Variables in the equations are displacement > >> >> >> >>> vector, > >> >> >> >>> scalar potential and pressure. > >> >> >> >> > >> >> >> >> > >> >> >> >> If you do not have a collocated discretization, then you have > to > >> >> >> >> use > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html > >> >> >> >> > >> >> >> >> Thanks, > >> >> >> >> > >> >> >> >> Matt > >> >> >> >> > >> >> >> >>> > >> >> >> >>> Thanks for help. > >> >> >> >>> > >> >> >> >>> Hom Nath > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> -- > >> >> >> >> What most experimenters take for granted before they begin > their > >> >> >> >> experiments > >> >> >> >> is infinitely more interesting than any results to which their > >> >> >> >> experiments > >> >> >> >> lead. > >> >> >> >> -- Norbert Wiener > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > -- > >> >> > What most experimenters take for granted before they begin their > >> >> > experiments > >> >> > is infinitely more interesting than any results to which their > >> >> > experiments > >> >> > lead. > >> >> > -- Norbert Wiener > >> > > >> > > >> > > >> > > >> > -- > >> > What most experimenters take for granted before they begin their > >> > experiments > >> > is infinitely more interesting than any results to which their > >> > experiments > >> > lead. > >> > -- Norbert Wiener > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments > > is infinitely more interesting than any results to which their > experiments > > lead. > > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue Feb 2 19:00:48 2016 From: jed at jedbrown.org (Jed Brown) Date: Tue, 02 Feb 2016 18:00:48 -0700 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> Message-ID: <878u328v5r.fsf@jedbrown.org> Satish Balay writes: > Or you can copy the whole '$PETSC_ARCH/externalpackages' dir over to > this other machine [and place in the same location - for any > PETSC_ARCH you plan to use..] This sounds unreliable. We used to have a unified externalpackages directory, but made it per-PETSC_ARCH partly to avoid dirty source trees causing incorrect builds (and so that configure could run in parallel for different PETSC_ARCH). -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From c.khroulev at gmail.com Tue Feb 2 19:17:52 2016 From: c.khroulev at gmail.com (Constantine Khroulev) Date: Tue, 02 Feb 2016 16:17:52 -0900 Subject: [petsc-users] Understanding DMDAGetElements() Message-ID: Hello PETSc developers, I recently discovered DMDAGetElements()[1] and got excited about using it... but then I realized that I don't understand what it does well enough to use it. So, here is a question. Consider the four by four grid pictured below, split across 2 processes. There are 16 nodes (marked with MPI ranks that own them) and 9 elements. #+BEGIN_SRC artist 1---------1---------1---------1 | | | | | 6 | 7 | 8 | | | | | 1---------1---------1---------1 | | | | | 3 | 4 | 5 | | | | | 0---------0---------0---------0 | | | | | 0 | 1 | 2 | | | | | 0---------0---------0---------0 #+END_SRC I expect DMDAGetElements(...) to return 6 elements (0, 1, 2, 3, 4, 5) on rank 0, because those are the elements I would need to integrate over to add up contributions to nodes owned by this rank. Similarly, on rank 1 I also expect to get 6 elements (3, 4, 5, 6, 7, 8). Instead I see 3 elements on rank 0 and 6 elements on rank 1. Please see the minimal setup in dm-get-elements.c; running it on 2 cores produces the following. $ mpiexec -n 2 ./dm-get-elements rank 0: 3 elements element 0: [ 0 1 5 4 ] element 1: [ 1 2 6 5 ] element 2: [ 2 3 7 6 ] rank 1: 6 elements element 0: [ 0 1 5 4 ] element 1: [ 1 2 6 5 ] element 2: [ 2 3 7 6 ] element 3: [ 4 5 9 8 ] element 4: [ 5 6 10 9 ] element 5: [ 6 7 11 10 ] What am I doing wrong here? Thank you! [1]: http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMDAGetElements.html -- Constantine -------------- next part -------------- A non-text attachment was scrubbed... Name: dm-get-elements.c Type: application/octet-stream Size: 2026 bytes Desc: A minimal test program URL: From bsmith at mcs.anl.gov Tue Feb 2 19:32:57 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 2 Feb 2016 19:32:57 -0600 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: <878u328v5r.fsf@jedbrown.org> References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> <878u328v5r.fsf@jedbrown.org> Message-ID: > On Feb 2, 2016, at 7:00 PM, Jed Brown wrote: > > Satish Balay writes: >> Or you can copy the whole '$PETSC_ARCH/externalpackages' dir over to >> this other machine [and place in the same location - for any >> PETSC_ARCH you plan to use..] > > This sounds unreliable. In what way? > We used to have a unified externalpackages > directory, but made it per-PETSC_ARCH partly to avoid dirty source trees > causing incorrect builds (and so that configure could run in parallel > for different PETSC_ARCH). From jed at jedbrown.org Tue Feb 2 21:51:22 2016 From: jed at jedbrown.org (Jed Brown) Date: Tue, 02 Feb 2016 20:51:22 -0700 Subject: [petsc-users] Understanding DMDAGetElements() In-Reply-To: References: Message-ID: <8737ta8n9h.fsf@jedbrown.org> Constantine Khroulev writes: > Hello PETSc developers, I recently discovered > DMDAGetElements()[1] and got excited about using it... but then I > realized that I don't understand what it does well enough to use > it. So, here is a question. Consider the four by four grid > pictured below, split across 2 processes. There are 16 nodes > (marked with MPI ranks that own them) and 9 elements. > #+BEGIN_SRC artist > 1---------1---------1---------1 | | | | > | 6 | 7 | 8 | | | | | > 1---------1---------1---------1 | | | | > | 3 | 4 | 5 | | | | | > 0---------0---------0---------0 | | | | > | 0 | 1 | 2 | | | | | > 0---------0---------0---------0 > #+END_SRC I expect DMDAGetElements(...) to return 6 elements (0, > 1, 2, 3, 4, 5) on rank 0, because those are the elements I would > need to integrate over to add up contributions to nodes owned by > this rank. Similarly, on rank 1 I also expect to get 6 elements > (3, 4, 5, 6, 7, 8). Instead I see 3 elements on rank 0 and 6 > elements on rank 1. The function is returning a non-overlapping partition of the elements. If you integrate over those elements and use ADD_VALUES, you'll get the correct result. It would be more subtle if you were to get the closure of the owned vertices, both at the insert stage and because some of those vertices aren't in the local vector, so you wouldn't have the state required to evaluate nonlinearities. If you want different behavior, you should either do it in your code or add a new variant to PETSc. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From jed at jedbrown.org Tue Feb 2 22:01:42 2016 From: jed at jedbrown.org (Jed Brown) Date: Tue, 02 Feb 2016 21:01:42 -0700 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> <878u328v5r.fsf@jedbrown.org> Message-ID: <87wpqm787t.fsf@jedbrown.org> Barry Smith writes: >> On Feb 2, 2016, at 7:00 PM, Jed Brown wrote: >> >> Satish Balay writes: >>> Or you can copy the whole '$PETSC_ARCH/externalpackages' dir over to >>> this other machine [and place in the same location - for any >>> PETSC_ARCH you plan to use..] >> >> This sounds unreliable. > > In what way? You'll be building in a dirty directory (one that was used for a different PETSC_ARCH). If the project doesn't manage their dependencies perfectly, it could use a build artifact from a different PETSC_ARCH, which would result in linking errors (for example). I'd rather cache the tarballs (with checksum) or use git repositories (which can be reliably cleaned) than simply build in dirty directories that cannot be cleaned. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From bsmith at mcs.anl.gov Tue Feb 2 22:13:05 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 2 Feb 2016 22:13:05 -0600 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: <87wpqm787t.fsf@jedbrown.org> References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> <878u328v5r.fsf@jedbrown.org> <87wpqm787t.fsf@jedbrown.org> Message-ID: <0D52BE64-DE48-4921-8CCD-3AA1C96BE863@mcs.anl.gov> > On Feb 2, 2016, at 10:01 PM, Jed Brown wrote: > > Barry Smith writes: > >>> On Feb 2, 2016, at 7:00 PM, Jed Brown wrote: >>> >>> Satish Balay writes: >>>> Or you can copy the whole '$PETSC_ARCH/externalpackages' dir over to >>>> this other machine [and place in the same location - for any >>>> PETSC_ARCH you plan to use..] >>> >>> This sounds unreliable. >> >> In what way? > > You'll be building in a dirty directory (one that was used for a > different PETSC_ARCH). If the project doesn't manage their dependencies > perfectly, it could use a build artifact from a different PETSC_ARCH, > which would result in linking errors (for example). You are right, this could be an issue > I'd rather cache > the tarballs (with checksum) or use git repositories (which can be > reliably cleaned) Ok then you write the code to do this. It doesn't currently exist (and I am not sure we want to be in the business of writing a full package management system :-). Anybody who thinks that a computer system that doesn't allow direct downloads of open source tarballs or git repositories (because of security considerations) but allows people to indirectly download those exact same tarballs or git repositories (as if that indirect download somehow magically cleanses the tarballs or repositories) is a damn fool anyways. Barry > than simply build in dirty directories that cannot be > cleaned. From jed at jedbrown.org Tue Feb 2 22:29:01 2016 From: jed at jedbrown.org (Jed Brown) Date: Tue, 02 Feb 2016 21:29:01 -0700 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: <0D52BE64-DE48-4921-8CCD-3AA1C96BE863@mcs.anl.gov> References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> <878u328v5r.fsf@jedbrown.org> <87wpqm787t.fsf@jedbrown.org> <0D52BE64-DE48-4921-8CCD-3AA1C96BE863@mcs.anl.gov> Message-ID: <87twlq76ya.fsf@jedbrown.org> Barry Smith writes: >> I'd rather cache >> the tarballs (with checksum) or use git repositories (which can be >> reliably cleaned) > > Ok then you write the code to do this. It doesn't currently exist > (and I am not sure we want to be in the business of writing a full > package management system :-). Aren't we deleting the tarballs now "to save storage"? I don't want to add yet another configuration option for users to read (it's irrelevant for 99% of them anyway). The problem with too many configuration options is that people give up on reading --help before finding the part that documents something they want. > Anybody who thinks that a computer system that doesn't allow direct > downloads of open source tarballs or git repositories (because of > security considerations) but allows people to indirectly download > those exact same tarballs or git repositories (as if that indirect > download somehow magically cleanses the tarballs or repositories) > is a damn fool anyways. Which is why I'm not motivated to add complicated features to facilitate. Employees need to complain about asinine policies they are forced to work around. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From balay at mcs.anl.gov Tue Feb 2 22:30:59 2016 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 2 Feb 2016 22:30:59 -0600 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: <0D52BE64-DE48-4921-8CCD-3AA1C96BE863@mcs.anl.gov> References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> <878u328v5r.fsf@jedbrown.org> <87wpqm787t.fsf@jedbrown.org> <0D52BE64-DE48-4921-8CCD-3AA1C96BE863@mcs.anl.gov> Message-ID: On Tue, 2 Feb 2016, Barry Smith wrote: > > > On Feb 2, 2016, at 10:01 PM, Jed Brown wrote: > > > > Barry Smith writes: > > > >>> On Feb 2, 2016, at 7:00 PM, Jed Brown wrote: > >>> > >>> Satish Balay writes: > >>>> Or you can copy the whole '$PETSC_ARCH/externalpackages' dir over to > >>>> this other machine [and place in the same location - for any > >>>> PETSC_ARCH you plan to use..] > >>> > >>> This sounds unreliable. > >> > >> In what way? > > > > You'll be building in a dirty directory (one that was used for a > > different PETSC_ARCH). If the project doesn't manage their dependencies > > perfectly, it could use a build artifact from a different PETSC_ARCH, > > which would result in linking errors (for example). > > You are right, this could be an issue Well my recommendation was for this use case of "same install on a different machine" And currently we do kinda support inplace repeated install of the same package [otherwise configure should be deleting all sources after build - and redownload each time. We don't do that] And even if the PETSC_ARCH1/externalpackages is copied over to PETSC_ARCH2/externalpackages - builds should work. the primary reason we moved externalpackages from PETSC_DIR/externalpackages to PETSC_DIR/PETSC_ARCH/externalpackages is to support concurrent builds in these locations. The issues you raise come about with using incompatible versions of packages. This can be due to switching git branches - or rebuilding after 'git pull' - that might have changed package dependencies.. But thats not the use case - as I understood it.. Sure - keeping track of all the tarballs for a given petsc version is a better approach. > > > I'd rather cache > > the tarballs (with checksum) or use git repositories (which can be > > reliably cleaned) I sure would like a common cache [for any build/arch/version] for both tarballs and git repos - so network traffic can be cut down. [say in ~/.local/petsc/cache]. > > Ok then you write the code to do this. It doesn't currently exist (and I am not sure we want to be in the business of writing a full package management system :-). > > Anybody who thinks that a computer system that doesn't allow direct downloads of open source tarballs or git repositories (because of security considerations) but allows people to indirectly download those exact same tarballs or git repositories (as if that indirect download somehow magically cleanses the tarballs or repositories) is a damn fool anyways. This is an easy way to cut-down syadmin overhead [or system complexity/downtime] for special-use systems. For ex; JLSE has various types of hardware. Its not easy to make sure they are secure enough for all network traffic. So all of them [except front-end general-use linux boxes] are firewalled off. Other systems provide proxies [to monitor all such traffic] - This is not easy to use via configure. [even though python urllib supports proxies..] Satish > > Barry > > > than simply build in dirty directories that cannot be > > cleaned. > > From bsmith at mcs.anl.gov Tue Feb 2 22:36:00 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 2 Feb 2016 22:36:00 -0600 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> <878u328v5r.fsf@jedbrown.org> <87wpqm787t.fsf@jedbrown.org> <0D52BE64-DE48-4921-8CCD-3AA1C96BE863@mcs.anl.gov> Message-ID: > On Feb 2, 2016, at 10:30 PM, Satish Balay wrote: > > On Tue, 2 Feb 2016, Barry Smith wrote: > >> >>> On Feb 2, 2016, at 10:01 PM, Jed Brown wrote: >>> >>> Barry Smith writes: >>> >>>>> On Feb 2, 2016, at 7:00 PM, Jed Brown wrote: >>>>> >>>>> Satish Balay writes: >>>>>> Or you can copy the whole '$PETSC_ARCH/externalpackages' dir over to >>>>>> this other machine [and place in the same location - for any >>>>>> PETSC_ARCH you plan to use..] >>>>> >>>>> This sounds unreliable. >>>> >>>> In what way? >>> >>> You'll be building in a dirty directory (one that was used for a >>> different PETSC_ARCH). If the project doesn't manage their dependencies >>> perfectly, it could use a build artifact from a different PETSC_ARCH, >>> which would result in linking errors (for example). >> >> You are right, this could be an issue > > Well my recommendation was for this use case of "same install on a different machine" > > And currently we do kinda support inplace repeated install of the same > package > > [otherwise configure should be deleting all sources after build - and > redownload each time. We don't do that] > > And even if the PETSC_ARCH1/externalpackages is copied over to > PETSC_ARCH2/externalpackages - builds should work. the primary reason > we moved externalpackages from PETSC_DIR/externalpackages to > PETSC_DIR/PETSC_ARCH/externalpackages is to support concurrent builds > in these locations. > > The issues you raise come about with using incompatible versions of > packages. This can be due to switching git branches - or rebuilding > after 'git pull' - that might have changed package dependencies.. But > thats not the use case - as I understood it.. > > Sure - keeping track of all the tarballs for a given petsc version is > a better approach. > >> >>> I'd rather cache >>> the tarballs (with checksum) or use git repositories (which can be >>> reliably cleaned) > > I sure would like a common cache [for any build/arch/version] for both > tarballs and git repos - so network traffic can be cut down. [say in > ~/.local/petsc/cache]. > >> >> Ok then you write the code to do this. It doesn't currently exist (and I am not sure we want to be in the business of writing a full package management system :-). >> >> Anybody who thinks that a computer system that doesn't allow direct downloads of open source tarballs or git repositories (because of security considerations) but allows people to indirectly download those exact same tarballs or git repositories (as if that indirect download somehow magically cleanses the tarballs or repositories) is a damn fool anyways. > > This is an easy way to cut-down syadmin overhead [or system > complexity/downtime] for special-use systems. For ex; JLSE has various > types of hardware. Its not easy to make sure they are secure enough > for all network traffic. So all of them [except front-end general-use > linux boxes] are firewalled off. > > Other systems provide proxies [to monitor all such traffic] - This is > not easy to use via configure. [even though python urllib supports > proxies..] How do the "firewalls" help if users scp over the EXACT same files that the firewall blocked? > > Satish > >> >> Barry >> >>> than simply build in dirty directories that cannot be >>> cleaned. >> >> > From bsmith at mcs.anl.gov Tue Feb 2 22:38:32 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 2 Feb 2016 22:38:32 -0600 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: <87twlq76ya.fsf@jedbrown.org> References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> <878u328v5r.fsf@jedbrown.org> <87wpqm787t.fsf@jedbrown.org> <0D52BE64-DE48-4921-8CCD-3AA1C96BE863@mcs.anl.gov> <87twlq76ya.fsf@jedbrown.org> Message-ID: <04590BF1-B737-43C3-B69D-465B01943F53@mcs.anl.gov> > On Feb 2, 2016, at 10:29 PM, Jed Brown wrote: > > Barry Smith writes: >>> I'd rather cache >>> the tarballs (with checksum) or use git repositories (which can be >>> reliably cleaned) >> >> Ok then you write the code to do this. It doesn't currently exist >> (and I am not sure we want to be in the business of writing a full >> package management system :-). > > Aren't we deleting the tarballs now "to save storage"? I don't want to > add yet another configuration option for users to read (it's irrelevant > for 99% of them anyway). The problem with too many configuration > options is that people give up on reading --help before finding the part > that documents something they want. > >> Anybody who thinks that a computer system that doesn't allow direct >> downloads of open source tarballs or git repositories (because of >> security considerations) but allows people to indirectly download >> those exact same tarballs or git repositories (as if that indirect >> download somehow magically cleanses the tarballs or repositories) >> is a damn fool anyways. > > Which is why I'm not motivated to add complicated features to > facilitate. Employees need to complain about asinine policies they are > forced to work around. But in your previous email you seemed to advocate exactly that. You said I'd rather cache the tarballs (with checksum) or use git repositories (which can be reliably cleaned) isn't that adding "complicated features to facilitate" From jed at jedbrown.org Tue Feb 2 22:38:40 2016 From: jed at jedbrown.org (Jed Brown) Date: Tue, 02 Feb 2016 21:38:40 -0700 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> <878u328v5r.fsf@jedbrown.org> <87wpqm787t.fsf@jedbrown.org> <0D52BE64-DE48-4921-8CCD-3AA1C96BE863@mcs.anl.gov> Message-ID: <87r3gu76i7.fsf@jedbrown.org> Satish Balay writes: > Well my recommendation was for this use case of "same install on a different machine" If you're installing to the same location on equivalently-configured machines, you can just copy $prefix (optionally by using DESTDIR to install to a temporary directory that you tar up). If you need to rebuild, then the old build artifacts are flawed. > And currently we do kinda support inplace repeated install of the same > package And it breaks _constantly_. It must be like 30% of petsc-maint traffic in the past year; advice being "delete PETSC_ARCH and reconfigure". > I sure would like a common cache [for any build/arch/version] for both > tarballs and git repos - so network traffic can be cut down. [say in > ~/.local/petsc/cache]. We need checksums for this. (We should always have checksums anyway.) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From coyigg at hotmail.com Tue Feb 2 22:40:54 2016 From: coyigg at hotmail.com (keguoyi) Date: Wed, 3 Feb 2016 12:40:54 +0800 Subject: [petsc-users] Problem with least squares commutators(LSC) Message-ID: Dear PETSc users, This is Guoyi Ke, a graduate student of Texas Tech University. I have a 2D Navier Stokes problem that has block matrices: J=[F B^T; B 0]. I use Schur complement preconditioner for block J. The code was set as: PCFieldSplitSetType( pc, PC_COMPOSITE_SCHUR ); PCFieldSplitSetSchurFactType(pc, PC_FIELDSPLIT_SCHUR_FACT_LOWER); PCFieldSplitSetSchurPre(pc,PC_FIELDSPLIT_SCHUR_PRE_SELFP,NULL); For both velocity block (fieldsplit: 0) and pressure block (fieldsplit: 1) , we set the KSP as KSPPREONLY and PC as PCILU. For this case, it works fine for us. However, in order to use LSC we change the code as: PCFieldSplitSetType( pc, PC_COMPOSITE_SCHUR ); PCFieldSplitSetSchurFactType(pc, PC_FIELDSPLIT_SCHUR_FACT_LOWER); PCFieldSplitSetSchurPre(pc,PC_FIELDSPLIT_SCHUR_PRE_SELF,NULL); We keep the same setup for velocity block (fieldsplit: 0) as above. For pressure block (fieldsplit: 1), we set the KSP as KSPGMRES and PC as PCLSU. I got the following errors: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Matrix is missing diagonal entry 0 Any suggestion and help will be highly appreciated. Thank you so much! Best, Guoyi -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Feb 2 22:43:41 2016 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 2 Feb 2016 22:43:41 -0600 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> <878u328v5r.fsf@jedbrown.org> <87wpqm787t.fsf@jedbrown.org> <0D52BE64-DE48-4921-8CCD-3AA1C96BE863@mcs.anl.gov> Message-ID: On Tue, 2 Feb 2016, Barry Smith wrote: > How do the "firewalls" help if users scp over the EXACT same files that the firewall blocked? I'm not sure how things are setup - but I suspect: - one can ssh on to the net [perhaps using a securecard] - one can scp file to the net [again using the same mechanism] But once you are logged on to such a machine - I don't think you can scp or wget or anything [as all traffic in that direction might be blocked - except through a proxy..] Satish From balay at mcs.anl.gov Tue Feb 2 22:51:51 2016 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 2 Feb 2016 22:51:51 -0600 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: <87r3gu76i7.fsf@jedbrown.org> References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> <878u328v5r.fsf@jedbrown.org> <87wpqm787t.fsf@jedbrown.org> <0D52BE64-DE48-4921-8CCD-3AA1C96BE863@mcs.anl.gov> <87r3gu76i7.fsf@jedbrown.org> Message-ID: On Tue, 2 Feb 2016, Jed Brown wrote: > Satish Balay writes: > > > Well my recommendation was for this use case of "same install on a different machine" > > If you're installing to the same location on equivalently-configured > machines, I presumed the copy is to a different machine - perhaps with different file system/organization. > you can just copy $prefix (optionally by using DESTDIR to > install to a temporary directory that you tar up). If you need to > rebuild, then the old build artifacts are flawed. > > > And currently we do kinda support inplace repeated install of the same > > package > > And it breaks _constantly_. It must be like 30% of petsc-maint traffic > in the past year; advice being "delete PETSC_ARCH and reconfigure". All of these breakages are usually due to 2 different cases. - [as mentioned before] - due to 'git pull' changing dependencies - [without git pull or equivalent] conflict between switching from say --download-mpich to --download-openmpi - in install location [PETSC_ARCH/include] and not sources in PETSC_ARCH/externalpackages I agree starting from tarballs is the recommended way - but this copy [of petsc+externalpackages together] will also work. Satish > > > I sure would like a common cache [for any build/arch/version] for both > > tarballs and git repos - so network traffic can be cut down. [say in > > ~/.local/petsc/cache]. > > We need checksums for this. (We should always have checksums anyway.) > From balay at mcs.anl.gov Tue Feb 2 22:57:44 2016 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 2 Feb 2016 22:57:44 -0600 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: <87r3gu76i7.fsf@jedbrown.org> References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> <878u328v5r.fsf@jedbrown.org> <87wpqm787t.fsf@jedbrown.org> <0D52BE64-DE48-4921-8CCD-3AA1C96BE863@mcs.anl.gov> <87r3gu76i7.fsf@jedbrown.org> Message-ID: On Tue, 2 Feb 2016, Jed Brown wrote: > > I sure would like a common cache [for any build/arch/version] for both > > tarballs and git repos - so network traffic can be cut down. [say in > > ~/.local/petsc/cache]. > > We need checksums for this. (We should always have checksums anyway.) Manually updating checksums in externalpackage.py [for any externapackage] - each time the tarball is updated - I think is error prone.. Is there a better mechanism to handle this? Satish From jed at jedbrown.org Tue Feb 2 23:02:50 2016 From: jed at jedbrown.org (Jed Brown) Date: Tue, 02 Feb 2016 22:02:50 -0700 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> <878u328v5r.fsf@jedbrown.org> <87wpqm787t.fsf@jedbrown.org> <0D52BE64-DE48-4921-8CCD-3AA1C96BE863@mcs.anl.gov> <87r3gu76i7.fsf@jedbrown.org> Message-ID: <87oaby75dx.fsf@jedbrown.org> Satish Balay writes: > On Tue, 2 Feb 2016, Jed Brown wrote: > >> Satish Balay writes: >> >> > Well my recommendation was for this use case of "same install on a different machine" >> >> If you're installing to the same location on equivalently-configured >> machines, > > I presumed the copy is to a different machine - perhaps with different file system/organization. Like different versions of compilers or different library versions? Then the failure mode I mentioned is possible. >> And it breaks _constantly_. It must be like 30% of petsc-maint traffic >> in the past year; advice being "delete PETSC_ARCH and reconfigure". > > All of these breakages are usually due to 2 different cases. > - [as mentioned before] - due to 'git pull' changing dependencies > - [without git pull or equivalent] conflict between switching from say > --download-mpich to --download-openmpi - in install location > [PETSC_ARCH/include] and not sources in PETSC_ARCH/externalpackages > > I agree starting from tarballs is the recommended way - but this copy > [of petsc+externalpackages together] will also work. I said it was unreliable, not that it would necessarily fail. I don't think we can claim this is supported behavior without a way to ensure clean build directories. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From jed at jedbrown.org Tue Feb 2 23:05:17 2016 From: jed at jedbrown.org (Jed Brown) Date: Tue, 02 Feb 2016 22:05:17 -0700 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: <04590BF1-B737-43C3-B69D-465B01943F53@mcs.anl.gov> References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> <878u328v5r.fsf@jedbrown.org> <87wpqm787t.fsf@jedbrown.org> <0D52BE64-DE48-4921-8CCD-3AA1C96BE863@mcs.anl.gov> <87twlq76ya.fsf@jedbrown.org> <04590BF1-B737-43C3-B69D-465B01943F53@mcs.anl.gov> Message-ID: <87lh72759u.fsf@jedbrown.org> Barry Smith writes: > But in your previous email you seemed to advocate exactly that. You said > > I'd rather cache > the tarballs (with checksum) or use git repositories (which can be > reliably cleaned) > > isn't that adding "complicated features to facilitate" I'd rather cache tarballs than support people copying dirty build directories around. I don't particularly want either of these, but caching tarballs and verifying them via a hash can at least be made reliable. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From balay at mcs.anl.gov Tue Feb 2 23:06:58 2016 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 2 Feb 2016 23:06:58 -0600 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: <87oaby75dx.fsf@jedbrown.org> References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> <878u328v5r.fsf@jedbrown.org> <87wpqm787t.fsf@jedbrown.org> <0D52BE64-DE48-4921-8CCD-3AA1C96BE863@mcs.anl.gov> <87r3gu76i7.fsf@jedbrown.org> <87oaby75dx.fsf@jedbrown.org> Message-ID: On Tue, 2 Feb 2016, Jed Brown wrote: > Satish Balay writes: > > > On Tue, 2 Feb 2016, Jed Brown wrote: > > > >> Satish Balay writes: > >> > >> > Well my recommendation was for this use case of "same install on a different machine" > >> > >> If you're installing to the same location on equivalently-configured > >> machines, > > > > I presumed the copy is to a different machine - perhaps with different file system/organization. > > Like different versions of compilers or different library versions? > Then the failure mode I mentioned is possible. This has nothing to do with sources in PETSC_ARCH/externalpackages. They are equivalent to tarballs. [stuff in PETSC_ARCH/include can conflict] > > >> And it breaks _constantly_. It must be like 30% of petsc-maint traffic > >> in the past year; advice being "delete PETSC_ARCH and reconfigure". > > > > All of these breakages are usually due to 2 different cases. > > - [as mentioned before] - due to 'git pull' changing dependencies > > - [without git pull or equivalent] conflict between switching from say > > --download-mpich to --download-openmpi - in install location > > [PETSC_ARCH/include] and not sources in PETSC_ARCH/externalpackages > > > > I agree starting from tarballs is the recommended way - but this copy > > [of petsc+externalpackages together] will also work. > > I said it was unreliable, not that it would necessarily fail. I don't > think we can claim this is supported behavior without a way to ensure > clean build directories. Each time configure is invoked with --download-package - it does a 'make clean' or equivalent in each of these packages/build dirs - and then rebuilds. Satish From jed at jedbrown.org Tue Feb 2 23:08:34 2016 From: jed at jedbrown.org (Jed Brown) Date: Tue, 02 Feb 2016 22:08:34 -0700 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> <878u328v5r.fsf@jedbrown.org> <87wpqm787t.fsf@jedbrown.org> <0D52BE64-DE48-4921-8CCD-3AA1C96BE863@mcs.anl.gov> Message-ID: <87io26754d.fsf@jedbrown.org> Satish Balay writes: > On Tue, 2 Feb 2016, Barry Smith wrote: > >> How do the "firewalls" help if users scp over the EXACT same files that the firewall blocked? > > I'm not sure how things are setup - but I suspect: > > - one can ssh on to the net [perhaps using a securecard] If you ssh with RemoteForward, then that host can ssh back to the client machine (without needing a password). This is very handy and also defeats the firewall. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From bsmith at mcs.anl.gov Tue Feb 2 23:12:56 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 2 Feb 2016 23:12:56 -0600 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: <87lh72759u.fsf@jedbrown.org> References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> <878u328v5r.fsf@jedbrown.org> <87wpqm787t.fsf@jedbrown.org> <0D52BE64-DE48-4921-8CCD-3AA1C96BE863@mcs.anl.gov> <87twlq76ya.fsf@jedbrown.org> <04590BF1-B737-43C3-B69D-465B01943F53@mcs.anl.gov> <87lh72759u.fsf@jedbrown.org> Message-ID: <9FEF3045-C536-401A-8BE3-5A5E3645051E@mcs.anl.gov> > On Feb 2, 2016, at 11:05 PM, Jed Brown wrote: > > Barry Smith writes: >> But in your previous email you seemed to advocate exactly that. You said >> >> I'd rather cache >> the tarballs (with checksum) or use git repositories (which can be >> reliably cleaned) >> >> isn't that adding "complicated features to facilitate" > > I'd rather cache tarballs than support people copying dirty build > directories around. I don't particularly want either of these, but > caching tarballs and verifying them via a hash can at least be made > reliable. Hey I never particularly advocating taring up externalpackage directories, that was Satish! I just asked what the flaw was with it which you correctly pointed out. My model is just read the damn list of tarballs that is displayed when you run ./configure on the "public" machine and use those same locations to get the tarballs needed for the "private" machine. It is goofy for us to maintain a proper cache of downloaded tarballs when people can just get them themselves anyway. From bsmith at mcs.anl.gov Tue Feb 2 23:15:10 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 2 Feb 2016 23:15:10 -0600 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> <878u328v5r.fsf@jedbrown.org> <87wpqm787t.fsf@jedbrown.org> <0D52BE64-DE48-4921-8CCD-3AA1C96BE863@mcs.anl.gov> <87r3gu76i7.fsf@jedbrown.org> <87oaby75dx.fsf@jedbrown.org> Message-ID: > On Feb 2, 2016, at 11:06 PM, Satish Balay wrote: > > On Tue, 2 Feb 2016, Jed Brown wrote: > >> Satish Balay writes: >> >>> On Tue, 2 Feb 2016, Jed Brown wrote: >>> >>>> Satish Balay writes: >>>> >>>>> Well my recommendation was for this use case of "same install on a different machine" >>>> >>>> If you're installing to the same location on equivalently-configured >>>> machines, >>> >>> I presumed the copy is to a different machine - perhaps with different file system/organization. >> >> Like different versions of compilers or different library versions? >> Then the failure mode I mentioned is possible. > > This has nothing to do with sources in PETSC_ARCH/externalpackages. They are equivalent to tarballs. > > [stuff in PETSC_ARCH/include can conflict] > >> >>>> And it breaks _constantly_. It must be like 30% of petsc-maint traffic >>>> in the past year; advice being "delete PETSC_ARCH and reconfigure". >>> >>> All of these breakages are usually due to 2 different cases. >>> - [as mentioned before] - due to 'git pull' changing dependencies >>> - [without git pull or equivalent] conflict between switching from say >>> --download-mpich to --download-openmpi - in install location >>> [PETSC_ARCH/include] and not sources in PETSC_ARCH/externalpackages >>> >>> I agree starting from tarballs is the recommended way - but this copy >>> [of petsc+externalpackages together] will also work. >> >> I said it was unreliable, not that it would necessarily fail. I don't >> think we can claim this is supported behavior without a way to ensure >> clean build directories. > > Each time configure is invoked with --download-package - it does a > 'make clean' or equivalent in each of these packages/build dirs - and > then rebuilds. Satish, Jed's point is most external packages (or at least some) do not properly implement make clean so you will often be dragging along a bunch of already generated (and likely wrong) files. You could say, "well that is not our fault, but it can still cause problems so the fact that is not our fault is not relevant." Barry > > Satish > From balay at mcs.anl.gov Tue Feb 2 23:16:29 2016 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 2 Feb 2016 23:16:29 -0600 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: <87io26754d.fsf@jedbrown.org> References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> <878u328v5r.fsf@jedbrown.org> <87wpqm787t.fsf@jedbrown.org> <0D52BE64-DE48-4921-8CCD-3AA1C96BE863@mcs.anl.gov> <87io26754d.fsf@jedbrown.org> Message-ID: On Tue, 2 Feb 2016, Jed Brown wrote: > Satish Balay writes: > > > On Tue, 2 Feb 2016, Barry Smith wrote: > > > >> How do the "firewalls" help if users scp over the EXACT same files that the firewall blocked? > > > > I'm not sure how things are setup - but I suspect: > > > > - one can ssh on to the net [perhaps using a securecard] > > If you ssh with RemoteForward, then that host can ssh back to the client > machine (without needing a password). This is very handy and also > defeats the firewall. I believe some of this stuff can be disabled in sshd config.. Satish From bsmith at mcs.anl.gov Tue Feb 2 23:17:51 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 2 Feb 2016 23:17:51 -0600 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: <87io26754d.fsf@jedbrown.org> References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> <878u328v5r.fsf@jedbrown.org> <87wpqm787t.fsf@jedbrown.org> <0D52BE64-DE48-4921-8CCD-3AA1C96BE863@mcs.anl.gov> <87io26754d.fsf@jedbrown.org> Message-ID: <02671BBF-E3D8-41B3-97F8-2761D39572C5@mcs.anl.gov> > On Feb 2, 2016, at 11:08 PM, Jed Brown wrote: > > Satish Balay writes: > >> On Tue, 2 Feb 2016, Barry Smith wrote: >> >>> How do the "firewalls" help if users scp over the EXACT same files that the firewall blocked? >> >> I'm not sure how things are setup - but I suspect: >> >> - one can ssh on to the net [perhaps using a securecard] > > If you ssh with RemoteForward, then that host can ssh back to the client > machine (without needing a password). This is very handy and also > defeats the firewall. I think Satish has it backwards; can can ssh and scp INTO the machine from outside, thus manually copy in tarballs and other sources of infection but you cannot ssh, scp, curl, wget or anything OUT of the machine to GET infected tarballs. Of course the end result is still that you have infected tarballs on your machine but now the sys admin can say it is your fault and not his or hers. Barry From balay at mcs.anl.gov Tue Feb 2 23:18:17 2016 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 2 Feb 2016 23:18:17 -0600 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> <878u328v5r.fsf@jedbrown.org> <87wpqm787t.fsf@jedbrown.org> <0D52BE64-DE48-4921-8CCD-3AA1C96BE863@mcs.anl.gov> <87r3gu76i7.fsf@jedbrown.org> <87oaby75dx.fsf@jedbrown.org> Message-ID: On Tue, 2 Feb 2016, Barry Smith wrote: > > > On Feb 2, 2016, at 11:06 PM, Satish Balay wrote: > > > > On Tue, 2 Feb 2016, Jed Brown wrote: > > > >> Satish Balay writes: > >> > >>> On Tue, 2 Feb 2016, Jed Brown wrote: > >>> > >>>> Satish Balay writes: > >>>> > >>>>> Well my recommendation was for this use case of "same install on a different machine" > >>>> > >>>> If you're installing to the same location on equivalently-configured > >>>> machines, > >>> > >>> I presumed the copy is to a different machine - perhaps with different file system/organization. > >> > >> Like different versions of compilers or different library versions? > >> Then the failure mode I mentioned is possible. > > > > This has nothing to do with sources in PETSC_ARCH/externalpackages. They are equivalent to tarballs. > > > > [stuff in PETSC_ARCH/include can conflict] > > > >> > >>>> And it breaks _constantly_. It must be like 30% of petsc-maint traffic > >>>> in the past year; advice being "delete PETSC_ARCH and reconfigure". > >>> > >>> All of these breakages are usually due to 2 different cases. > >>> - [as mentioned before] - due to 'git pull' changing dependencies > >>> - [without git pull or equivalent] conflict between switching from say > >>> --download-mpich to --download-openmpi - in install location > >>> [PETSC_ARCH/include] and not sources in PETSC_ARCH/externalpackages > >>> > >>> I agree starting from tarballs is the recommended way - but this copy > >>> [of petsc+externalpackages together] will also work. > >> > >> I said it was unreliable, not that it would necessarily fail. I don't > >> think we can claim this is supported behavior without a way to ensure > >> clean build directories. > > > > Each time configure is invoked with --download-package - it does a > > 'make clean' or equivalent in each of these packages/build dirs - and > > then rebuilds. > > Satish, > > Jed's point is most external packages (or at least some) do not properly implement make clean so you will often be dragging along a bunch of already generated (and likely wrong) files. You could say, "well that is not our fault, but it can still cause problems so the fact that is not our fault is not relevant." > My point here is - we haven't have problems with it. [if we had - then you would be doing 'rm -rf PETSC_ARCH' 20 times every day..] Satish From balay at mcs.anl.gov Tue Feb 2 23:25:26 2016 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 2 Feb 2016 23:25:26 -0600 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: <02671BBF-E3D8-41B3-97F8-2761D39572C5@mcs.anl.gov> References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> <878u328v5r.fsf@jedbrown.org> <87wpqm787t.fsf@jedbrown.org> <0D52BE64-DE48-4921-8CCD-3AA1C96BE863@mcs.anl.gov> <87io26754d.fsf@jedbrown.org> <02671BBF-E3D8-41B3-97F8-2761D39572C5@mcs.anl.gov> Message-ID: On Tue, 2 Feb 2016, Barry Smith wrote: > > > On Feb 2, 2016, at 11:08 PM, Jed Brown wrote: > > > > Satish Balay writes: > > > >> On Tue, 2 Feb 2016, Barry Smith wrote: > >> > >>> How do the "firewalls" help if users scp over the EXACT same files that the firewall blocked? > >> > >> I'm not sure how things are setup - but I suspect: > >> > >> - one can ssh on to the net [perhaps using a securecard] > > > > If you ssh with RemoteForward, then that host can ssh back to the client > > machine (without needing a password). This is very handy and also > > defeats the firewall. > > I think Satish has it backwards; can can ssh and scp INTO the machine from outside, Yeah - thats what I tried to say. Bad choice of words [should have said 'secure net' aka 'firewalled network' or the remote-machine-with-firewall] > thus manually copy in tarballs and other sources of infection but you cannot ssh, scp, curl, wget or anything OUT of the machine to GET infected tarballs. Of course the end result is still that you have infected tarballs on your machine but now the sys admin can say it is your fault and not his or hers. I think the primary concern here is open network paths that can be exploited by other means [not the stuff 'users' copy over] And even in case of 'infected' tarballs - one usual thingy that such 'inections' do is - get more (damaging/latest) stuff to run using wget (or equivalent).. Without the 'network path' such things get blocked.. Satish From bsmith at mcs.anl.gov Tue Feb 2 23:29:05 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 2 Feb 2016 23:29:05 -0600 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> <878u328v5r.fsf@jedbrown.org> <87wpqm787t.fsf@jedbrown.org> <0D52BE64-DE48-4921-8CCD-3AA1C96BE863@mcs.anl.gov> <87r3gu76i7.fsf@jedbrown.org> <87oaby75dx.fsf@jedbrown.org> Message-ID: > On Feb 2, 2016, at 11:18 PM, Satish Balay wrote: > > On Tue, 2 Feb 2016, Barry Smith wrote: > >> >>> On Feb 2, 2016, at 11:06 PM, Satish Balay wrote: >>> >>> On Tue, 2 Feb 2016, Jed Brown wrote: >>> >>>> Satish Balay writes: >>>> >>>>> On Tue, 2 Feb 2016, Jed Brown wrote: >>>>> >>>>>> Satish Balay writes: >>>>>> >>>>>>> Well my recommendation was for this use case of "same install on a different machine" >>>>>> >>>>>> If you're installing to the same location on equivalently-configured >>>>>> machines, >>>>> >>>>> I presumed the copy is to a different machine - perhaps with different file system/organization. >>>> >>>> Like different versions of compilers or different library versions? >>>> Then the failure mode I mentioned is possible. >>> >>> This has nothing to do with sources in PETSC_ARCH/externalpackages. They are equivalent to tarballs. >>> >>> [stuff in PETSC_ARCH/include can conflict] >>> >>>> >>>>>> And it breaks _constantly_. It must be like 30% of petsc-maint traffic >>>>>> in the past year; advice being "delete PETSC_ARCH and reconfigure". >>>>> >>>>> All of these breakages are usually due to 2 different cases. >>>>> - [as mentioned before] - due to 'git pull' changing dependencies >>>>> - [without git pull or equivalent] conflict between switching from say >>>>> --download-mpich to --download-openmpi - in install location >>>>> [PETSC_ARCH/include] and not sources in PETSC_ARCH/externalpackages >>>>> >>>>> I agree starting from tarballs is the recommended way - but this copy >>>>> [of petsc+externalpackages together] will also work. >>>> >>>> I said it was unreliable, not that it would necessarily fail. I don't >>>> think we can claim this is supported behavior without a way to ensure >>>> clean build directories. >>> >>> Each time configure is invoked with --download-package - it does a >>> 'make clean' or equivalent in each of these packages/build dirs - and >>> then rebuilds. >> >> Satish, >> >> Jed's point is most external packages (or at least some) do not properly implement make clean so you will often be dragging along a bunch of already generated (and likely wrong) files. You could say, "well that is not our fault, but it can still cause problems so the fact that is not our fault is not relevant." >> > > My point here is - we haven't have problems with it. [if we had - then you would be doing 'rm -rf PETSC_ARCH' 20 times every day..] But people who "think" machine 1 (outside the firewall) has the same configuration at machine 2 (inside the firewall) are almost always wrong!. The two machines will likely have different versions of all kinds of things that "may" cause the inappropriate already generated files to break something. Barry > > Satish From bsmith at mcs.anl.gov Tue Feb 2 23:31:32 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 2 Feb 2016 23:31:32 -0600 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> <878u328v5r.fsf@jedbrown.org> <87wpqm787t.fsf@jedbrown.org> <0D52BE64-DE48-4921-8CCD-3AA1C96BE863@mcs.anl.gov> <87io26754d.fsf@jedbrown.org> <02671BBF-E3D8-41B3-97F8-2761D39572C5@mcs.anl.gov> Message-ID: <30A40240-E845-401F-BD3D-AC93DFD6C5E0@mcs.anl.gov> > On Feb 2, 2016, at 11:25 PM, Satish Balay wrote: > > On Tue, 2 Feb 2016, Barry Smith wrote: > >> >>> On Feb 2, 2016, at 11:08 PM, Jed Brown wrote: >>> >>> Satish Balay writes: >>> >>>> On Tue, 2 Feb 2016, Barry Smith wrote: >>>> >>>>> How do the "firewalls" help if users scp over the EXACT same files that the firewall blocked? >>>> >>>> I'm not sure how things are setup - but I suspect: >>>> >>>> - one can ssh on to the net [perhaps using a securecard] >>> >>> If you ssh with RemoteForward, then that host can ssh back to the client >>> machine (without needing a password). This is very handy and also >>> defeats the firewall. >> >> I think Satish has it backwards; can can ssh and scp INTO the machine from outside, > > Yeah - thats what I tried to say. Bad choice of words [should have said 'secure net' aka 'firewalled network' or the remote-machine-with-firewall] > >> thus manually copy in tarballs and other sources of infection but you cannot ssh, scp, curl, wget or anything OUT of the machine to GET infected tarballs. Of course the end result is still that you have infected tarballs on your machine but now the sys admin can say it is your fault and not his or hers. > > I think the primary concern here is open network paths that can be exploited by other means [not the stuff 'users' copy over] > And even in case of 'infected' tarballs - one usual thingy that such 'inections' do is - get more (damaging/latest) stuff to run using wget (or equivalent).. > Without the 'network path' such things get blocked.. Yes that is the (poor) logic of these "firewalls". > > Satish From balay at mcs.anl.gov Tue Feb 2 23:37:44 2016 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 2 Feb 2016 23:37:44 -0600 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: <30A40240-E845-401F-BD3D-AC93DFD6C5E0@mcs.anl.gov> References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> <878u328v5r.fsf@jedbrown.org> <87wpqm787t.fsf@jedbrown.org> <0D52BE64-DE48-4921-8CCD-3AA1C96BE863@mcs.anl.gov> <87io26754d.fsf@jedbrown.org> <02671BBF-E3D8-41B3-97F8-2761D39572C5@mcs.anl.gov> <30A40240-E845-401F-BD3D-AC93DFD6C5E0@mcs.anl.gov> Message-ID: On Tue, 2 Feb 2016, Barry Smith wrote: > >> > >> I think Satish has it backwards; can can ssh and scp INTO the machine from outside, > > > > Yeah - thats what I tried to say. Bad choice of words [should have said 'secure net' aka 'firewalled network' or the remote-machine-with-firewall] > > > >> thus manually copy in tarballs and other sources of infection but you cannot ssh, scp, curl, wget or anything OUT of the machine to GET infected tarballs. Of course the end result is still that you have infected tarballs on your machine but now the sys admin can say it is your fault and not his or hers. > > > > I think the primary concern here is open network paths that can be exploited by other means [not the stuff 'users' copy over] > > And even in case of 'infected' tarballs - one usual thingy that such 'inections' do is - get more (damaging/latest) stuff to run using wget (or equivalent).. > > Without the 'network path' such things get blocked.. > > Yes that is the (poor) logic of these "firewalls". Something I missed saying is - usually if these infections are successful - they just copy lots of data [passwd files, ssh key files or other important data] over to the haker servers. The folks that setup and run these stuff might offer better explanations than what I can.. Satish From epscodes at gmail.com Wed Feb 3 20:27:37 2016 From: epscodes at gmail.com (Xiangdong) Date: Wed, 3 Feb 2016 21:27:37 -0500 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> Message-ID: On Tue, Feb 2, 2016 at 3:11 PM, Satish Balay wrote: > On Tue, 2 Feb 2016, Xiangdong wrote: > > > On Tue, Feb 2, 2016 at 2:27 PM, Satish Balay wrote: > > > > > On Tue, 2 Feb 2016, Barry Smith wrote: > > > > > > > > > > > > On Feb 2, 2016, at 9:21 AM, Xiangdong wrote: > > > > > > > > > > the package is untar and compiled in $PETSC_ARCH/externalpackages, > but > > > the original tarball .tar file is deleted. Is it correct? > > > > > > > > Yes > > > > > > > > When you ran the successful configure it printed to the screen > where > > > the tar file came from. You should use curl or wget to get the tarball > and > > > then copy it over to the machine you want it on. > > > > > > > > > Or you can copy the whole '$PETSC_ARCH/externalpackages' dir over to > > > this other machine [and place in the same location - for any > > > PETSC_ARCH you plan to use..] > > > > > > > If I have 15 packages to install, this method would be easy. Once I > copied > > this folder to the new machine, do I use > > --with-pastix-dir=$PETSC_ARCH/externalpackages/pastix_5.2 or > > --download-pastix=$PETSC_ARCH/externalpackages/pastix_5.2? The first may > > fail due to the different libraries installed on that machine. For the > > second one, can --download-pastix take the directory path instead of the > > tarball file? > > If you already have $PETSC_ARCH/externalpackages/pastix_5.2 - then you > would use: > > --download-pastix=1 > > configure first checks if the package is already downloaded or not: > - if found [in $PETSC_ARCH/externalpackages/] - it goes ahead and uses it. > - if not found - it attempts to downlaod and extract in > $PETSC_ARCH/externalpackages/ > > Yes, it works for my case! I did not realized that this question has generated a long discussion on firewall....... Thanks. Xiangdong > Satish > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Wed Feb 3 20:48:50 2016 From: jed at jedbrown.org (Jed Brown) Date: Wed, 03 Feb 2016 19:48:50 -0700 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> <878u328v5r.fsf@jedbrown.org> <87wpqm787t.fsf@jedbrown.org> <0D52BE64-DE48-4921-8CCD-3AA1C96BE863@mcs.anl.gov> <87r3gu76i7.fsf@jedbrown.org> <87oaby75dx.fsf@jedbrown.org> Message-ID: <87io255gx9.fsf@jedbrown.org> Satish Balay writes: > My point here is - we haven't have problems with it. [if we had - then you would be doing 'rm -rf PETSC_ARCH' 20 times every day..] I don't understand this logic. We tell users to do this several times every day. We might attribute it to having something "old" (like from a different version of PETSc), but we never really know. I don't consider the unpacked, dirty source tree in PETSC_ARCH/externalpackages to be in any way equivalent to a tarball. We could insist on out-of-source builds, though not all packages support that. I just don't think it's a good idea to advise people to copy the externalpackages directory around. Download the tarballs and copy them wherever you need them. Or open a proxy (easy with ssh) so you don't have to waste effort on the silly firewall. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From bsmith at mcs.anl.gov Wed Feb 3 22:26:42 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 3 Feb 2016 22:26:42 -0600 Subject: [petsc-users] Problem with least squares commutators(LSC) In-Reply-To: References: Message-ID: <601DA012-D197-49AF-9191-C76E1F1150FB@mcs.anl.gov> We'd need the full error message to determine what is happening. But note that since the lower right block is all 0 any attempt to use a factorization of that inside a preconditioner will fail with zero pivot. > On Feb 2, 2016, at 10:40 PM, keguoyi wrote: > > Dear PETSc users, > > This is Guoyi Ke, a graduate student of Texas Tech University. I have a 2D Navier Stokes problem that has block matrices: J=[F B^T; B 0]. I use Schur complement preconditioner for block J. The code was set as: > > PCFieldSplitSetType( pc, PC_COMPOSITE_SCHUR ); > PCFieldSplitSetSchurFactType(pc, PC_FIELDSPLIT_SCHUR_FACT_LOWER); > PCFieldSplitSetSchurPre(pc,PC_FIELDSPLIT_SCHUR_PRE_SELFP,NULL); > > For both velocity block (fieldsplit: 0) and pressure block (fieldsplit: 1) , we set the KSP as KSPPREONLY and PC as PCILU. For this case, it works fine for us. > However, in order to use LSC we change the code as: > > PCFieldSplitSetType( pc, PC_COMPOSITE_SCHUR ); > PCFieldSplitSetSchurFactType(pc, PC_FIELDSPLIT_SCHUR_FACT_LOWER); > PCFieldSplitSetSchurPre(pc,PC_FIELDSPLIT_SCHUR_PRE_SELF,NULL); > > We keep the same setup for velocity block (fieldsplit: 0) as above. For pressure block (fieldsplit: 1), we set the KSP as KSPGMRES and PC as PCLSU. I got the following errors: > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: Matrix is missing diagonal entry 0 > > > Any suggestion and help will be highly appreciated. Thank you so much! > > Best, > Guoyi From balay at mcs.anl.gov Wed Feb 3 23:25:26 2016 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 3 Feb 2016 23:25:26 -0600 Subject: [petsc-users] petsc externalpackage directory In-Reply-To: <87io255gx9.fsf@jedbrown.org> References: <42F6CECE-9820-4F4A-97DA-B0AC05D1A111@mcs.anl.gov> <878u328v5r.fsf@jedbrown.org> <87wpqm787t.fsf@jedbrown.org> <0D52BE64-DE48-4921-8CCD-3AA1C96BE863@mcs.anl.gov> <87r3gu76i7.fsf@jedbrown.org> <87oaby75dx.fsf@jedbrown.org> <87io255gx9.fsf@jedbrown.org> Message-ID: On Wed, 3 Feb 2016, Jed Brown wrote: > Satish Balay writes: > > My point here is - we haven't have problems with it. [if we had - then you would be doing 'rm -rf PETSC_ARCH' 20 times every day..] > > I don't understand this logic. We tell users to do this several times > every day. We might attribute it to having something "old" (like from a > different version of PETSc), but we never really know. I don't consider > the unpacked, dirty source tree in PETSC_ARCH/externalpackages to be in > any way equivalent to a tarball. We could insist on out-of-source > builds, though not all packages support that. I just don't think it's a > good idea to advise people to copy the externalpackages directory > around. Download the tarballs and copy them wherever you need them. Or > open a proxy (easy with ssh) so you don't have to waste effort on the > silly firewall. Can you list the externalpackages [with inplace builds] that currently break with the following? ./configure [or make.inc] && make clean && make [lib] && make install [or cp] ./configure [or make.inc] {with different option} && make clean && make [lib] && make install [or cp] Or do you recall any bugreports where we identified this as the issue? Note: doing the following with petsc configure does not bypass the above usage internally within petsc configure. ./configure --download-package=foo.tar.gz ./configure --download-package=foo.tar.gz {with different options} As mentioned before - most of the petsc-maint issues you refere to are due to 'git pull' causing version conflicts or conflicts in PETSC_ARCH/include in between the above configure runs with different options - and not the extracted externalpackage sources. The current system is not ideal - but to me the above 2 usages are equivalent. Sure you can say one should always do: ./configure --download-package=foo.tar.gz rm -rf PETSC_ARCH ./configure --download-package=foo.tar.gz {with different options} Or configure should do this automatically. But We've always reused externalpakcages and avoided redownling for all these years - and suggest 'rm -rf PETSC_ARCH' only when one sees broken builds when conflicts come up. In petsc 'master' - the support for git repos is almost ideal [for any changes with git pull or switch branch or any such thing] - where a 'rm externalpackages' is never necessary [rm PETSC_ARCH/include might still be needed - i.e 'git clean -f -d -x' could suffice]. So One way to improve current infrastructure is to have every externalpackage hosted in a git repo. Alternative - as you suggest is to add the extra infrastructure needed for tarballs/checksums and stash them them as primary source. [somehow have to manage simultaneous download of the same tarball potentially invoked by different simultaneous configure runs - so that a common taball cache can be used by all petsc versions/builds]. Then we can discard the extracted sources [along with the corresponding build files] - with each configure run - that might result in a rebuild of the externalpackage. Satish From coyigg at hotmail.com Wed Feb 3 23:30:34 2016 From: coyigg at hotmail.com (keguoyi) Date: Thu, 4 Feb 2016 13:30:34 +0800 Subject: [petsc-users] Problem with least squares commutators(LSC) In-Reply-To: <601DA012-D197-49AF-9191-C76E1F1150FB@mcs.anl.gov> References: , <601DA012-D197-49AF-9191-C76E1F1150FB@mcs.anl.gov> Message-ID: Dear Users, The following is the full error message. Thank you so much! *************** Linear iteration 1 *********** [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Matrix is missing diagonal entry 0 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.1, unknown [0]PETSC ERROR: ./PCFieldSplit_ex13 on a arch-linux2-cxx-opt named linux-394h by coyigg Wed Feb 3 14:57:21 2016 [0]PETSC ERROR: Configure options --with-clanguage=cxx --with-debugging=0 --download-fblaslapack=1 --download-hdf5=1 --download-openmpi=1 --download-metis=1 --download-parmetis=1 --with-shared-libraries=1 --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --download-blacs=1 --download-scalapack=1 --download-mumps=1 [0]PETSC ERROR: #1 MatILUFactorSymbolic_SeqAIJ() line 1729 in /home/eaulisa/software/petsc/src/mat/impls/aij/seq/aijfact.c [0]PETSC ERROR: #2 MatILUFactorSymbolic() line 6457 in /home/eaulisa/software/petsc/src/mat/interface/matrix.c [0]PETSC ERROR: #3 PCSetUp_ILU() line 204 in /home/eaulisa/software/petsc/src/ksp/pc/impls/factor/ilu/ilu.c [0]PETSC ERROR: #4 PCSetUp() line 983 in /home/eaulisa/software/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #5 KSPSetUp() line 332 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #6 KSPSolve() line 546 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #7 PCApply_LSC() line 83 in /home/eaulisa/software/petsc/src/ksp/pc/impls/lsc/lsc.c [0]PETSC ERROR: #8 PCApply() line 483 in /home/eaulisa/software/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #9 KSP_PCApply() line 242 in /home/eaulisa/software/petsc/include/petsc/private/kspimpl.h [0]PETSC ERROR: #10 KSPSolve_PREONLY() line 26 in /home/eaulisa/software/petsc/src/ksp/ksp/impls/preonly/preonly.c [0]PETSC ERROR: #11 KSPSolve() line 604 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #12 PCApply_FieldSplit_Schur() line 874 in /home/eaulisa/software/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: #13 PCApply() line 483 in /home/eaulisa/software/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #14 KSP_PCApply() line 242 in /home/eaulisa/software/petsc/include/petsc/private/kspimpl.h [0]PETSC ERROR: #15 KSPInitialResidual() line 63 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itres.c [0]PETSC ERROR: #16 KSPSolve_GMRES() line 235 in /home/eaulisa/software/petsc/src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: #17 KSPSolve() line 604 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #18 PCMGMCycle_Private() line 19 in /home/eaulisa/software/petsc/src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: #19 PCApply_MG() line 340 in /home/eaulisa/software/petsc/src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: #20 PCApply() line 483 in /home/eaulisa/software/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #21 KSPSolve() line 582 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c *************** MG linear solver time: 0.002 *************** Number of outer ksp solver iterations = 0 *************** Convergence reason = 0 *************** Residual norm = 0 *************** Level Max 2 Linear Res L2norm U = 4.033131e-02 *************** Level Max 2 Linear Res L2norm V = 5.330749e-02 *************** Level Max 2 Linear Res L2norm P = 1.396813e-01 > Subject: Re: [petsc-users] Problem with least squares commutators(LSC) > From: bsmith at mcs.anl.gov > Date: Wed, 3 Feb 2016 22:26:42 -0600 > CC: petsc-users at mcs.anl.gov > To: coyigg at hotmail.com > > > We'd need the full error message to determine what is happening. > > But note that since the lower right block is all 0 any attempt to use a factorization of that inside a preconditioner will fail with zero pivot. > > > > On Feb 2, 2016, at 10:40 PM, keguoyi wrote: > > > > Dear PETSc users, > > > > This is Guoyi Ke, a graduate student of Texas Tech University. I have a 2D Navier Stokes problem that has block matrices: J=[F B^T; B 0]. I use Schur complement preconditioner for block J. The code was set as: > > > > PCFieldSplitSetType( pc, PC_COMPOSITE_SCHUR ); > > PCFieldSplitSetSchurFactType(pc, PC_FIELDSPLIT_SCHUR_FACT_LOWER); > > PCFieldSplitSetSchurPre(pc,PC_FIELDSPLIT_SCHUR_PRE_SELFP,NULL); > > > > For both velocity block (fieldsplit: 0) and pressure block (fieldsplit: 1) , we set the KSP as KSPPREONLY and PC as PCILU. For this case, it works fine for us. > > However, in order to use LSC we change the code as: > > > > PCFieldSplitSetType( pc, PC_COMPOSITE_SCHUR ); > > PCFieldSplitSetSchurFactType(pc, PC_FIELDSPLIT_SCHUR_FACT_LOWER); > > PCFieldSplitSetSchurPre(pc,PC_FIELDSPLIT_SCHUR_PRE_SELF,NULL); > > > > We keep the same setup for velocity block (fieldsplit: 0) as above. For pressure block (fieldsplit: 1), we set the KSP as KSPGMRES and PC as PCLSU. I got the following errors: > > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [0]PETSC ERROR: Object is in wrong state > > [0]PETSC ERROR: Matrix is missing diagonal entry 0 > > > > > > Any suggestion and help will be highly appreciated. Thank you so much! > > > > Best, > > Guoyi > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Feb 4 00:09:57 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 4 Feb 2016 00:09:57 -0600 Subject: [petsc-users] Problem with least squares commutators(LSC) In-Reply-To: References: <601DA012-D197-49AF-9191-C76E1F1150FB@mcs.anl.gov> Message-ID: <07E37232-9385-42C3-AF2B-5C5E07F68AEB@mcs.anl.gov> It is failing due to zero diagonal entry in factoring the matrix Lp which is A10*A01 which I think in your case is B * B^T Do you use code like? PetscObjectCompose((PetscObject)Sp,"LSC_L",(PetscObject)L); PetscObjectCompose((PetscObject)Sp,"LSC_Lp",(PetscObject)Lp); And then your Jacobian assembly would look like PetscObjectQuery((PetscObject)Sp,"LSC_L",(PetscObject*)&L); PetscObjectQuery((PetscObject)Sp,"LSC_Lp",(PetscObject*)&Lp); if (L) { assembly L } if (Lp) { assemble Lp } That is are you providing a matrix that contains B * B^T already computed or are you just letting PETSc compute it from the blocks of the full matrix you provide? (I am guessing you let PETSc compute it). PETSc computes it in the routine PCSetUp_LSC() ierr = MatSchurComplementGetSubMatrices(pc->mat,NULL,NULL,&B,&C,NULL);CHKERRQ(ierr); if (!lsc->L) { ierr = MatMatMult(C,B,MAT_INITIAL_MATRIX,PETSC_DEFAULT,&lsc->L);CHKERRQ(ierr); } else { ierr = MatMatMult(C,B,MAT_REUSE_MATRIX,PETSC_DEFAULT,&lsc->L);CHKERRQ(ierr); } Lp = L = lsc->L; It would be useful to see what Lp is. You can run with -start_in_debugger put a breakpoint in PCSetUp_LSC then at line Lp = L = lsc->L; and call MatView(Lp,0) and MatView(B,0) and MatView(C,0) at that line to see what the matrices look like. Of course run on 1 process with a small size for the matrices. Do you maybe have a periodic domain or Neumann boundary conditions so that B * B^T is singular? Barry > On Feb 3, 2016, at 11:30 PM, keguoyi wrote: > > Dear Users, > > The following is the full error message. Thank you so much! > > *************** Linear iteration 1 *********** > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: Matrix is missing diagonal entry 0 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.1, unknown > [0]PETSC ERROR: ./PCFieldSplit_ex13 on a arch-linux2-cxx-opt named linux-394h by coyigg Wed Feb 3 14:57:21 2016 > [0]PETSC ERROR: Configure options --with-clanguage=cxx --with-debugging=0 --download-fblaslapack=1 --download-hdf5=1 --download-openmpi=1 --download-metis=1 --download-parmetis=1 --with-shared-libraries=1 --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --download-blacs=1 --download-scalapack=1 --download-mumps=1 > [0]PETSC ERROR: #1 MatILUFactorSymbolic_SeqAIJ() line 1729 in /home/eaulisa/software/petsc/src/mat/impls/aij/seq/aijfact.c > [0]PETSC ERROR: #2 MatILUFactorSymbolic() line 6457 in /home/eaulisa/software/petsc/src/mat/interface/matrix.c > [0]PETSC ERROR: #3 PCSetUp_ILU() line 204 in /home/eaulisa/software/petsc/src/ksp/pc/impls/factor/ilu/ilu.c > [0]PETSC ERROR: #4 PCSetUp() line 983 in /home/eaulisa/software/petsc/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #5 KSPSetUp() line 332 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #6 KSPSolve() line 546 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #7 PCApply_LSC() line 83 in /home/eaulisa/software/petsc/src/ksp/pc/impls/lsc/lsc.c > [0]PETSC ERROR: #8 PCApply() line 483 in /home/eaulisa/software/petsc/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #9 KSP_PCApply() line 242 in /home/eaulisa/software/petsc/include/petsc/private/kspimpl.h > [0]PETSC ERROR: #10 KSPSolve_PREONLY() line 26 in /home/eaulisa/software/petsc/src/ksp/ksp/impls/preonly/preonly.c > [0]PETSC ERROR: #11 KSPSolve() line 604 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #12 PCApply_FieldSplit_Schur() line 874 in /home/eaulisa/software/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c > [0]PETSC ERROR: #13 PCApply() line 483 in /home/eaulisa/software/petsc/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #14 KSP_PCApply() line 242 in /home/eaulisa/software/petsc/include/petsc/private/kspimpl.h > [0]PETSC ERROR: #15 KSPInitialResidual() line 63 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: #16 KSPSolve_GMRES() line 235 in /home/eaulisa/software/petsc/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: #17 KSPSolve() line 604 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #18 PCMGMCycle_Private() line 19 in /home/eaulisa/software/petsc/src/ksp/pc/impls/mg/mg.c > [0]PETSC ERROR: #19 PCApply_MG() line 340 in /home/eaulisa/software/petsc/src/ksp/pc/impls/mg/mg.c > [0]PETSC ERROR: #20 PCApply() line 483 in /home/eaulisa/software/petsc/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #21 KSPSolve() line 582 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c > *************** MG linear solver time: 0.002 > *************** Number of outer ksp solver iterations = 0 > *************** Convergence reason = 0 > *************** Residual norm = 0 > *************** Level Max 2 Linear Res L2norm U = 4.033131e-02 > *************** Level Max 2 Linear Res L2norm V = 5.330749e-02 > *************** Level Max 2 Linear Res L2norm P = 1.396813e-01 > > > > Subject: Re: [petsc-users] Problem with least squares commutators(LSC) > > From: bsmith at mcs.anl.gov > > Date: Wed, 3 Feb 2016 22:26:42 -0600 > > CC: petsc-users at mcs.anl.gov > > To: coyigg at hotmail.com > > > > > > We'd need the full error message to determine what is happening. > > > > But note that since the lower right block is all 0 any attempt to use a factorization of that inside a preconditioner will fail with zero pivot. > > > > > > > On Feb 2, 2016, at 10:40 PM, keguoyi wrote: > > > > > > Dear PETSc users, > > > > > > This is Guoyi Ke, a graduate student of Texas Tech University. I have a 2D Navier Stokes problem that has block matrices: J=[F B^T; B 0]. I use Schur complement preconditioner for block J. The code was set as: > > > > > > PCFieldSplitSetType( pc, PC_COMPOSITE_SCHUR ); > > > PCFieldSplitSetSchurFactType(pc, PC_FIELDSPLIT_SCHUR_FACT_LOWER); > > > PCFieldSplitSetSchurPre(pc,PC_FIELDSPLIT_SCHUR_PRE_SELFP,NULL); > > > > > > For both velocity block (fieldsplit: 0) and pressure block (fieldsplit: 1) , we set the KSP as KSPPREONLY and PC as PCILU. For this case, it works fine for us. > > > However, in order to use LSC we change the code as: > > > > > > PCFieldSplitSetType( pc, PC_COMPOSITE_SCHUR ); > > > PCFieldSplitSetSchurFactType(pc, PC_FIELDSPLIT_SCHUR_FACT_LOWER); > > > PCFieldSplitSetSchurPre(pc,PC_FIELDSPLIT_SCHUR_PRE_SELF,NULL); > > > > > > We keep the same setup for velocity block (fieldsplit: 0) as above. For pressure block (fieldsplit: 1), we set the KSP as KSPGMRES and PC as PCLSU. I got the following errors: > > > > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > > [0]PETSC ERROR: Object is in wrong state > > > [0]PETSC ERROR: Matrix is missing diagonal entry 0 > > > > > > > > > Any suggestion and help will be highly appreciated. Thank you so much! > > > > > > Best, > > > Guoyi > > From coyigg at hotmail.com Thu Feb 4 09:45:50 2016 From: coyigg at hotmail.com (keguoyi) Date: Thu, 4 Feb 2016 23:45:50 +0800 Subject: [petsc-users] Problem with least squares commutators(LSC) In-Reply-To: <07E37232-9385-42C3-AF2B-5C5E07F68AEB@mcs.anl.gov> References: <601DA012-D197-49AF-9191-C76E1F1150FB@mcs.anl.gov> , <07E37232-9385-42C3-AF2B-5C5E07F68AEB@mcs.anl.gov> Message-ID: Dear Users, Thanks a lot for the reply. It really helps a lot. The code for -fieldsplit-pressure is here, and we just let PETSc compute it from the blocks of the full matrix we provide. if( _preconditioner == LSC_PRECOND ) { _rtol = 1.e-3; _abstol = 1.e-20; _dtol = 1.e+50; _maxits = 1; SetPetscSolverType(ksp); PC pc; KSPGetPC( ksp, &pc ); KSPSetTolerances( ksp, _rtol, _abstol, _dtol, _maxits ); KSPSetFromOptions( ksp ); PCSetType( pc, PCLSC ); } We think what our problem is This is a NavierStokes problem with Dirchhlet Boundary conditions everywhere but one point (to fix the pressure). We impose BC using penalty method, thus changing the diagonal of A00, but we leave A01 unchanged. So in evaluating A10*A01 it is like we are having a Neumann Problem everywhere. We have to rethink how to impose the BC. If you have other suggestions, please let us know. Thank you so much!! Best, Guoyi > Subject: Re: [petsc-users] Problem with least squares commutators(LSC) > From: bsmith at mcs.anl.gov > Date: Thu, 4 Feb 2016 00:09:57 -0600 > CC: petsc-users at mcs.anl.gov > To: coyigg at hotmail.com > > > It is failing due to zero diagonal entry in factoring the matrix Lp which is A10*A01 which I think in your case is B * B^T > > Do you use code like? > > PetscObjectCompose((PetscObject)Sp,"LSC_L",(PetscObject)L); > PetscObjectCompose((PetscObject)Sp,"LSC_Lp",(PetscObject)Lp); > > And then your Jacobian assembly would look like > PetscObjectQuery((PetscObject)Sp,"LSC_L",(PetscObject*)&L); > PetscObjectQuery((PetscObject)Sp,"LSC_Lp",(PetscObject*)&Lp); > if (L) { assembly L } > if (Lp) { assemble Lp } > > That is are you providing a matrix that contains B * B^T already computed or are you just letting PETSc compute it from the blocks of the > full matrix you provide? (I am guessing you let PETSc compute it). > > PETSc computes it in the routine PCSetUp_LSC() > > ierr = MatSchurComplementGetSubMatrices(pc->mat,NULL,NULL,&B,&C,NULL);CHKERRQ(ierr); > if (!lsc->L) { > ierr = MatMatMult(C,B,MAT_INITIAL_MATRIX,PETSC_DEFAULT,&lsc->L);CHKERRQ(ierr); > } else { > ierr = MatMatMult(C,B,MAT_REUSE_MATRIX,PETSC_DEFAULT,&lsc->L);CHKERRQ(ierr); > } > Lp = L = lsc->L; > > It would be useful to see what Lp is. You can run with -start_in_debugger put a breakpoint in PCSetUp_LSC then at line Lp = L = lsc->L; > and call MatView(Lp,0) and MatView(B,0) and MatView(C,0) at that line to see what the matrices look like. Of course run on 1 process with a small size for the matrices. > > Do you maybe have a periodic domain or Neumann boundary conditions so that B * B^T is singular? > > Barry > > > > > > On Feb 3, 2016, at 11:30 PM, keguoyi wrote: > > > > Dear Users, > > > > The following is the full error message. Thank you so much! > > > > *************** Linear iteration 1 *********** > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [0]PETSC ERROR: Object is in wrong state > > [0]PETSC ERROR: Matrix is missing diagonal entry 0 > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.6.1, unknown > > [0]PETSC ERROR: ./PCFieldSplit_ex13 on a arch-linux2-cxx-opt named linux-394h by coyigg Wed Feb 3 14:57:21 2016 > > [0]PETSC ERROR: Configure options --with-clanguage=cxx --with-debugging=0 --download-fblaslapack=1 --download-hdf5=1 --download-openmpi=1 --download-metis=1 --download-parmetis=1 --with-shared-libraries=1 --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --download-blacs=1 --download-scalapack=1 --download-mumps=1 > > [0]PETSC ERROR: #1 MatILUFactorSymbolic_SeqAIJ() line 1729 in /home/eaulisa/software/petsc/src/mat/impls/aij/seq/aijfact.c > > [0]PETSC ERROR: #2 MatILUFactorSymbolic() line 6457 in /home/eaulisa/software/petsc/src/mat/interface/matrix.c > > [0]PETSC ERROR: #3 PCSetUp_ILU() line 204 in /home/eaulisa/software/petsc/src/ksp/pc/impls/factor/ilu/ilu.c > > [0]PETSC ERROR: #4 PCSetUp() line 983 in /home/eaulisa/software/petsc/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #5 KSPSetUp() line 332 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: #6 KSPSolve() line 546 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: #7 PCApply_LSC() line 83 in /home/eaulisa/software/petsc/src/ksp/pc/impls/lsc/lsc.c > > [0]PETSC ERROR: #8 PCApply() line 483 in /home/eaulisa/software/petsc/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #9 KSP_PCApply() line 242 in /home/eaulisa/software/petsc/include/petsc/private/kspimpl.h > > [0]PETSC ERROR: #10 KSPSolve_PREONLY() line 26 in /home/eaulisa/software/petsc/src/ksp/ksp/impls/preonly/preonly.c > > [0]PETSC ERROR: #11 KSPSolve() line 604 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: #12 PCApply_FieldSplit_Schur() line 874 in /home/eaulisa/software/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c > > [0]PETSC ERROR: #13 PCApply() line 483 in /home/eaulisa/software/petsc/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #14 KSP_PCApply() line 242 in /home/eaulisa/software/petsc/include/petsc/private/kspimpl.h > > [0]PETSC ERROR: #15 KSPInitialResidual() line 63 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itres.c > > [0]PETSC ERROR: #16 KSPSolve_GMRES() line 235 in /home/eaulisa/software/petsc/src/ksp/ksp/impls/gmres/gmres.c > > [0]PETSC ERROR: #17 KSPSolve() line 604 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: #18 PCMGMCycle_Private() line 19 in /home/eaulisa/software/petsc/src/ksp/pc/impls/mg/mg.c > > [0]PETSC ERROR: #19 PCApply_MG() line 340 in /home/eaulisa/software/petsc/src/ksp/pc/impls/mg/mg.c > > [0]PETSC ERROR: #20 PCApply() line 483 in /home/eaulisa/software/petsc/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #21 KSPSolve() line 582 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c > > *************** MG linear solver time: 0.002 > > *************** Number of outer ksp solver iterations = 0 > > *************** Convergence reason = 0 > > *************** Residual norm = 0 > > *************** Level Max 2 Linear Res L2norm U = 4.033131e-02 > > *************** Level Max 2 Linear Res L2norm V = 5.330749e-02 > > *************** Level Max 2 Linear Res L2norm P = 1.396813e-01 > > > > > > > Subject: Re: [petsc-users] Problem with least squares commutators(LSC) > > > From: bsmith at mcs.anl.gov > > > Date: Wed, 3 Feb 2016 22:26:42 -0600 > > > CC: petsc-users at mcs.anl.gov > > > To: coyigg at hotmail.com > > > > > > > > > We'd need the full error message to determine what is happening. > > > > > > But note that since the lower right block is all 0 any attempt to use a factorization of that inside a preconditioner will fail with zero pivot. > > > > > > > > > > On Feb 2, 2016, at 10:40 PM, keguoyi wrote: > > > > > > > > Dear PETSc users, > > > > > > > > This is Guoyi Ke, a graduate student of Texas Tech University. I have a 2D Navier Stokes problem that has block matrices: J=[F B^T; B 0]. I use Schur complement preconditioner for block J. The code was set as: > > > > > > > > PCFieldSplitSetType( pc, PC_COMPOSITE_SCHUR ); > > > > PCFieldSplitSetSchurFactType(pc, PC_FIELDSPLIT_SCHUR_FACT_LOWER); > > > > PCFieldSplitSetSchurPre(pc,PC_FIELDSPLIT_SCHUR_PRE_SELFP,NULL); > > > > > > > > For both velocity block (fieldsplit: 0) and pressure block (fieldsplit: 1) , we set the KSP as KSPPREONLY and PC as PCILU. For this case, it works fine for us. > > > > However, in order to use LSC we change the code as: > > > > > > > > PCFieldSplitSetType( pc, PC_COMPOSITE_SCHUR ); > > > > PCFieldSplitSetSchurFactType(pc, PC_FIELDSPLIT_SCHUR_FACT_LOWER); > > > > PCFieldSplitSetSchurPre(pc,PC_FIELDSPLIT_SCHUR_PRE_SELF,NULL); > > > > > > > > We keep the same setup for velocity block (fieldsplit: 0) as above. For pressure block (fieldsplit: 1), we set the KSP as KSPGMRES and PC as PCLSU. I got the following errors: > > > > > > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > > > [0]PETSC ERROR: Object is in wrong state > > > > [0]PETSC ERROR: Matrix is missing diagonal entry 0 > > > > > > > > > > > > Any suggestion and help will be highly appreciated. Thank you so much! > > > > > > > > Best, > > > > Guoyi > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rupp at iue.tuwien.ac.at Fri Feb 5 00:21:30 2016 From: rupp at iue.tuwien.ac.at (Karl Rupp) Date: Fri, 5 Feb 2016 07:21:30 +0100 Subject: [petsc-users] PETSc User Meeting 2016 Message-ID: <56B43F6A.6010708@iue.tuwien.ac.at> Dear PETSc users, we would like to invite you to join us for a PETSc User Meeting held in Vienna, Austria, on June 28-30, 2016. The first day consists of tutorials on various aspects and features of PETSc. The second and third day are devoted to exchange, discussions, and a refinement of strategies for the future with our users. We encourage you to present work illustrating your own use of PETSc, for example in applications or in libraries built on top of PETSc. Registration for the PETSc User Meeting 2016 is free of charge. We can host a maximum of 120 participants, so register soon (but no later than May 1). We are still in the process of acquiring money for providing student travel grants; prospective sponsors, please contact us. Website with full information: http://www.mcs.anl.gov/petsc/meetings/2016/ We are looking forward to welcoming you in Vienna! Karl Rupp and the PETSc team From kaushikggg at gmail.com Fri Feb 5 02:57:49 2016 From: kaushikggg at gmail.com (Kaushik Kulkarni) Date: Fri, 5 Feb 2016 14:27:49 +0530 Subject: [petsc-users] Editor for PETSc Message-ID: Hello, I am a newbie to PETSc. I am going thorugh the tutorials right now. But I am struggling a bit in finding the correct editor for me. Till now I was fine with VIM, as it gave me a bunch of completions options in the form of a drop down box. And using the Clang support feature it would also highlight the errors hence drastically decreasing my errors. Is there any way I could continue working on such environment. I was using the "YouCompleteMe" plugin with VIM which had all of these features. Thanks, *Kaushik* -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothee.nicolas at gmail.com Fri Feb 5 03:06:25 2016 From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=) Date: Fri, 5 Feb 2016 18:06:25 +0900 Subject: [petsc-users] Editor for PETSc In-Reply-To: References: Message-ID: Hi, You don't really specify your problem with this editor, do you ? But in any case, why not try emacs ? As you can see in the FAQ, this is what they use to develop it : http://www.mcs.anl.gov/petsc/documentation/faq.html Best Timothee 2016-02-05 17:57 GMT+09:00 Kaushik Kulkarni : > Hello, > I am a newbie to PETSc. I am going thorugh the tutorials right now. But I > am struggling a bit in finding the correct editor for me. Till now I was > fine with VIM, as it gave me a bunch of completions options in the form of > a drop down box. And using the Clang support feature it would also > highlight the errors hence drastically decreasing my errors. Is there any > way I could continue working on such environment. > > I was using the "YouCompleteMe" plugin with VIM which had all of these > features. > > Thanks, > *Kaushik* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kaushikggg at gmail.com Fri Feb 5 08:14:53 2016 From: kaushikggg at gmail.com (Kaushik Kulkarni) Date: Fri, 5 Feb 2016 19:44:53 +0530 Subject: [petsc-users] Editor for PETSc In-Reply-To: References: Message-ID: The problem is not exactly with VIM. It is the plugins, I am using which gives me suggestions( or the auto-complete feature ) as well as checks the syntax. But, as it can be seen in the image attached it keeps on popping an error. The thing would be mostly to provide the correct flags to the plugin, which currently is : *===============================================================flags = [ '-Wall', '-std=c++11', '-stdlib=libc++', '-x', 'c++', '-I', '.',]================================================================* Is there anyway, I could modify it so that my problem is solved. Thanks, Kaushik ?. ? On Fri, Feb 5, 2016 at 2:36 PM, Timoth?e Nicolas wrote: > Hi, > > You don't really specify your problem with this editor, do you ? But in > any case, why not try emacs ? As you can see in the FAQ, this is what they > use to develop it : > > http://www.mcs.anl.gov/petsc/documentation/faq.html > > Best > > Timothee > > 2016-02-05 17:57 GMT+09:00 Kaushik Kulkarni : > >> Hello, >> I am a newbie to PETSc. I am going thorugh the tutorials right now. But I >> am struggling a bit in finding the correct editor for me. Till now I was >> fine with VIM, as it gave me a bunch of completions options in the form of >> a drop down box. And using the Clang support feature it would also >> highlight the errors hence drastically decreasing my errors. Is there any >> way I could continue working on such environment. >> >> I was using the "YouCompleteMe" plugin with VIM which had all of these >> features. >> >> Thanks, >> *Kaushik* >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot from 2016-02-05 14:05:58.png Type: image/png Size: 69458 bytes Desc: not available URL: From jed at jedbrown.org Fri Feb 5 10:06:43 2016 From: jed at jedbrown.org (Jed Brown) Date: Fri, 05 Feb 2016 09:06:43 -0700 Subject: [petsc-users] Editor for PETSc In-Reply-To: References: Message-ID: <87zivfdtv0.fsf@jedbrown.org> Kaushik Kulkarni writes: > The problem is not exactly with VIM. It is the plugins, I am using which > gives me suggestions( or the auto-complete feature ) as well as checks the > syntax. But, as it can be seen in the image attached it keeps on popping an > error. > > The thing would be mostly to provide the correct flags to the plugin, which > currently is : > > *===============================================================flags = > [ '-Wall', '-std=c++11', '-stdlib=libc++', '-x', 'c++', > '-I', > '.',]================================================================* > Is there anyway, I could modify it so that my problem is solved. -I$PETSC_DIR/include -I$PETSC_DIR/$PETSC_ARCH/include -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From patrick.sanan at gmail.com Fri Feb 5 10:31:08 2016 From: patrick.sanan at gmail.com (Patrick Sanan) Date: Fri, 5 Feb 2016 17:31:08 +0100 Subject: [petsc-users] Editor for PETSc In-Reply-To: References: Message-ID: <20160205163108.GG39386@Patricks-MacBook-Pro-4.local> If you're using vim, using ctags (you want "exuberant ctags") is a quick and handy way to look around in the PETSc source (with its included man pages). See the manual, http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf, Section 15.9 . (and if you use emacs, see the preceding Section 15.8 for instructions on tags and more). On Fri, Feb 05, 2016 at 02:27:49PM +0530, Kaushik Kulkarni wrote: > Hello, > I am a newbie to PETSc. I am going thorugh the tutorials right now. But I > am struggling a bit in finding the correct editor for me. Till now I was > fine with VIM, as it gave me a bunch of completions options in the form of > a drop down box. And using the Clang support feature it would also > highlight the errors hence drastically decreasing my errors. Is there any > way I could continue working on such environment. > > I was using the "YouCompleteMe" plugin with VIM which had all of these > features. > > Thanks, > *Kaushik* -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 473 bytes Desc: not available URL: From kaushikggg at gmail.com Fri Feb 5 10:59:24 2016 From: kaushikggg at gmail.com (Kaushik Kulkarni) Date: Fri, 5 Feb 2016 22:29:24 +0530 Subject: [petsc-users] Editor for PETSc In-Reply-To: <20160205163108.GG39386@Patricks-MacBook-Pro-4.local> References: <20160205163108.GG39386@Patricks-MacBook-Pro-4.local> Message-ID: Hey Jed, Thanks for your response. But a small doubt. Am I supposed to add these two lines to the list of flags. '-I$PETSC_DIR/include', '-I$PETSC_DIR/$PETSC_ARCH/include', Because I did so, and still it was not working. Thanks, Kaushik. *Kaushik KulkarniThird Year Undergraduate* *Department of Mechanical EngineeringIndian Institute of Technology Bombay* *Mumbai, Indiahttp://home.iitb.ac.in/~kaushikkulkarni/ * *Contact: +91-9967687150* On Fri, Feb 5, 2016 at 10:01 PM, Patrick Sanan wrote: > If you're using vim, using ctags (you want "exuberant ctags") is a quick > and handy way to > look around in the PETSc source (with its included man pages). > See the manual, http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf, > Section 15.9 . > (and if you use emacs, see the preceding Section 15.8 for instructions on > tags and more). > On Fri, Feb 05, 2016 at 02:27:49PM +0530, Kaushik Kulkarni wrote: > > Hello, > > I am a newbie to PETSc. I am going thorugh the tutorials right now. But I > > am struggling a bit in finding the correct editor for me. Till now I was > > fine with VIM, as it gave me a bunch of completions options in the form > of > > a drop down box. And using the Clang support feature it would also > > highlight the errors hence drastically decreasing my errors. Is there any > > way I could continue working on such environment. > > > > I was using the "YouCompleteMe" plugin with VIM which had all of these > > features. > > > > Thanks, > > *Kaushik* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From epscodes at gmail.com Fri Feb 5 11:12:03 2016 From: epscodes at gmail.com (Xiangdong) Date: Fri, 5 Feb 2016 12:12:03 -0500 Subject: [petsc-users] log_summary time ratio and flops ratio Message-ID: Hello everyone, When I looked at the log_summary, I found that flops ratio of VecDot is 1.1, but the time ratio is 1.8. Also the flops ratio of MatMult is 1.2, but the time ratio is 1.6. I am using the simple SOR preconditioner on 256 procs. Since each of them consumes about 20% of total time, is it worth looking into these imbalance issue? If yes, how do I know which procs are causing the issue? What tools should I start with? Thanks. Xiangdong ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage VecView 44 1.0 3.9026e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.8e+01 1 0 0 0 0 1 0 0 0 0 0 VecDot 817452 1.0 1.5972e+02 1.8 1.51e+10 1.1 0.0e+00 0.0e+00 8.2e+05 18 3 0 0 40 18 3 0 0 40 22969 VecMDot 613089 1.0 5.8918e+01 1.1 2.27e+10 1.1 0.0e+00 0.0e+00 6.1e+05 8 4 0 0 30 8 4 0 0 30 93397 VecNorm 618619 1.0 1.4426e+02 2.2 1.14e+10 1.1 0.0e+00 0.0e+00 6.2e+05 15 2 0 0 30 15 2 0 0 30 19145 VecScale 938 1.0 8.3907e-03 1.2 8.67e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 250859 VecCopy 8159 1.0 1.0394e-01 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 1884 1.0 3.9701e-02 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 1021815 1.0 2.2148e+01 2.1 1.89e+10 1.1 0.0e+00 0.0e+00 0.0e+00 2 4 0 0 0 2 4 0 0 0 207057 VecAYPX 613089 1.0 6.5360e+00 1.9 1.13e+10 1.1 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 420336 VecAXPBYCZ 191 1.0 3.7365e-03 1.4 7.06e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 458830 VecWAXPY 938 1.0 3.8502e-02 1.4 8.67e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 54669 VecMAXPY 613089 1.0 1.3276e+01 2.2 2.27e+10 1.1 0.0e+00 0.0e+00 0.0e+00 1 4 0 0 0 1 4 0 0 0 414499 VecLoad 5 1.0 2.0913e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 820695 1.0 1.8959e+01 2.8 0.00e+00 0.0 1.1e+09 4.6e+03 0.0e+00 2 0100100 0 2 0100100 0 0 VecScatterEnd 820695 1.0 5.2002e+01 4.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 4 0 0 0 0 4 0 0 0 0 0 VecReduceArith 2814 1.0 1.8517e-02 1.2 5.20e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 681995 VecReduceComm 938 1.0 2.6041e+00 3.4 0.00e+00 0.0 0.0e+00 0.0e+00 9.4e+02 0 0 0 0 0 0 0 0 0 0 0 MatMult 817452 1.0 1.9379e+02 1.6 2.04e+11 1.2 1.0e+09 4.6e+03 0.0e+00 23 40100100 0 23 40100100 0 253083 MatSOR 818390 1.0 1.9608e+02 1.5 2.00e+11 1.1 0.0e+00 0.0e+00 0.0e+00 22 40 0 0 0 22 40 0 0 0 247472 MatAssemblyBegin 939 1.0 7.9671e+00 6.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.9e+03 1 0 0 0 0 1 0 0 0 0 0 MatAssemblyEnd 939 1.0 2.0570e-01 1.2 0.00e+00 0.0 2.6e+03 1.2e+03 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatZeroEntries 938 1.0 1.8202e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SNESSolve 210 1.0 6.8585e+02 1.0 5.06e+11 1.2 1.1e+09 4.6e+03 2.1e+06 98100100100100 98100100100100 178411 SNESFunctionEval 2296 1.0 2.0512e+01 1.3 0.00e+00 0.0 2.9e+06 3.5e+03 0.0e+00 3 0 0 0 0 3 0 0 0 0 0 SNESJacobianEval 938 1.0 2.7680e+01 1.0 0.00e+00 0.0 1.2e+06 3.5e+03 1.9e+03 4 0 0 0 0 4 0 0 0 0 0 SNESLineSearch 938 1.0 9.2491e+00 1.0 6.07e+07 1.1 1.2e+06 3.5e+03 9.4e+02 1 0 0 0 0 1 0 0 0 0 1593 KSPSetUp 938 1.0 2.7661e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+01 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 938 1.0 6.3433e+02 1.0 5.06e+11 1.2 1.0e+09 4.6e+03 2.0e+06 91100100100 99 91100100100 99 192852 PCSetUp 938 1.0 2.1005e-04 3.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCApply 818390 1.0 1.9675e+02 1.5 2.00e+11 1.1 0.0e+00 0.0e+00 0.0e+00 22 40 0 0 0 22 40 0 0 0 246626 ------------------------------------------------------------------------------------------------------------------------ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Feb 5 11:13:46 2016 From: jed at jedbrown.org (Jed Brown) Date: Fri, 05 Feb 2016 10:13:46 -0700 Subject: [petsc-users] Editor for PETSc In-Reply-To: References: <20160205163108.GG39386@Patricks-MacBook-Pro-4.local> Message-ID: <87oabvdqr9.fsf@jedbrown.org> Kaushik Kulkarni writes: > Hey Jed, > Thanks for your response. But a small doubt. > Am I supposed to add these two lines to the list of flags. > '-I$PETSC_DIR/include', > '-I$PETSC_DIR/$PETSC_ARCH/include', > > Because I did so, and still it was not working. The above probably doesn't expand those variables and they might not be set in the editor's environment anyway. Expand them manually if you need. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From knepley at gmail.com Fri Feb 5 11:18:18 2016 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 5 Feb 2016 11:18:18 -0600 Subject: [petsc-users] log_summary time ratio and flops ratio In-Reply-To: References: Message-ID: On Fri, Feb 5, 2016 at 11:12 AM, Xiangdong wrote: > Hello everyone, > > When I looked at the log_summary, I found that flops ratio of VecDot is > 1.1, but the time ratio is 1.8. Also the flops ratio of MatMult is 1.2, but > the time ratio is 1.6. I am using the simple SOR preconditioner on 256 > procs. Since each of them consumes about 20% of total time, is it worth > looking into these imbalance issue? If yes, how do I know which procs are > causing the issue? What tools should I start with? > 1) Always send all the output 2) You have a large number of dot products, and I am guessing you have a slow reduce on this machine (we could see the data at the end of log_summary). The waiting time comes from slow collective communication, load imbalance, system noise, etc. You can see this by looking at the difference between VecDot() and VecMDot(). For the latter there is enough work to cover up the latency, and its imbalance is only 1.1, not 1.8. Thus I think it is clear that your problem is due to a slow communication network rather than load imbalance. 3) Thus you would be better served with a stronger preconditioner (more flops) which resulted in fewer iterations (less synchronization). Thanks, Matt > Thanks. > > Xiangdong > > > ------------------------------------------------------------------------------------------------------------------------ > Event Count Time (sec) Flops > --- Global --- --- Stage --- Total > Max Ratio Max Ratio Max Ratio Mess Avg len > Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s > > ------------------------------------------------------------------------------------------------------------------------ > > --- Event Stage 0: Main Stage > > VecView 44 1.0 3.9026e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 4.8e+01 1 0 0 0 0 1 0 0 0 0 0 > VecDot 817452 1.0 1.5972e+02 1.8 1.51e+10 1.1 0.0e+00 0.0e+00 > 8.2e+05 18 3 0 0 40 18 3 0 0 40 22969 > VecMDot 613089 1.0 5.8918e+01 1.1 2.27e+10 1.1 0.0e+00 0.0e+00 > 6.1e+05 8 4 0 0 30 8 4 0 0 30 93397 > VecNorm 618619 1.0 1.4426e+02 2.2 1.14e+10 1.1 0.0e+00 0.0e+00 > 6.2e+05 15 2 0 0 30 15 2 0 0 30 19145 > VecScale 938 1.0 8.3907e-03 1.2 8.67e+06 1.1 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 250859 > VecCopy 8159 1.0 1.0394e-01 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecSet 1884 1.0 3.9701e-02 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecAXPY 1021815 1.0 2.2148e+01 2.1 1.89e+10 1.1 0.0e+00 0.0e+00 > 0.0e+00 2 4 0 0 0 2 4 0 0 0 207057 > VecAYPX 613089 1.0 6.5360e+00 1.9 1.13e+10 1.1 0.0e+00 0.0e+00 > 0.0e+00 1 2 0 0 0 1 2 0 0 0 420336 > VecAXPBYCZ 191 1.0 3.7365e-03 1.4 7.06e+06 1.1 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 458830 > VecWAXPY 938 1.0 3.8502e-02 1.4 8.67e+06 1.1 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 54669 > VecMAXPY 613089 1.0 1.3276e+01 2.2 2.27e+10 1.1 0.0e+00 0.0e+00 > 0.0e+00 1 4 0 0 0 1 4 0 0 0 414499 > VecLoad 5 1.0 2.0913e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecScatterBegin 820695 1.0 1.8959e+01 2.8 0.00e+00 0.0 1.1e+09 4.6e+03 > 0.0e+00 2 0100100 0 2 0100100 0 0 > VecScatterEnd 820695 1.0 5.2002e+01 4.2 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 4 0 0 0 0 4 0 0 0 0 0 > VecReduceArith 2814 1.0 1.8517e-02 1.2 5.20e+07 1.1 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 681995 > VecReduceComm 938 1.0 2.6041e+00 3.4 0.00e+00 0.0 0.0e+00 0.0e+00 > 9.4e+02 0 0 0 0 0 0 0 0 0 0 0 > MatMult 817452 1.0 1.9379e+02 1.6 2.04e+11 1.2 1.0e+09 4.6e+03 > 0.0e+00 23 40100100 0 23 40100100 0 253083 > MatSOR 818390 1.0 1.9608e+02 1.5 2.00e+11 1.1 0.0e+00 0.0e+00 > 0.0e+00 22 40 0 0 0 22 40 0 0 0 247472 > MatAssemblyBegin 939 1.0 7.9671e+00 6.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 1.9e+03 1 0 0 0 0 1 0 0 0 0 0 > MatAssemblyEnd 939 1.0 2.0570e-01 1.2 0.00e+00 0.0 2.6e+03 1.2e+03 > 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatZeroEntries 938 1.0 1.8202e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > SNESSolve 210 1.0 6.8585e+02 1.0 5.06e+11 1.2 1.1e+09 4.6e+03 > 2.1e+06 98100100100100 98100100100100 178411 > SNESFunctionEval 2296 1.0 2.0512e+01 1.3 0.00e+00 0.0 2.9e+06 3.5e+03 > 0.0e+00 3 0 0 0 0 3 0 0 0 0 0 > SNESJacobianEval 938 1.0 2.7680e+01 1.0 0.00e+00 0.0 1.2e+06 3.5e+03 > 1.9e+03 4 0 0 0 0 4 0 0 0 0 0 > SNESLineSearch 938 1.0 9.2491e+00 1.0 6.07e+07 1.1 1.2e+06 3.5e+03 > 9.4e+02 1 0 0 0 0 1 0 0 0 0 1593 > KSPSetUp 938 1.0 2.7661e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 > 2.0e+01 0 0 0 0 0 0 0 0 0 0 0 > KSPSolve 938 1.0 6.3433e+02 1.0 5.06e+11 1.2 1.0e+09 4.6e+03 > 2.0e+06 91100100100 99 91100100100 99 192852 > PCSetUp 938 1.0 2.1005e-04 3.1 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > PCApply 818390 1.0 1.9675e+02 1.5 2.00e+11 1.1 0.0e+00 0.0e+00 > 0.0e+00 22 40 0 0 0 22 40 0 0 0 246626 > > ------------------------------------------------------------------------------------------------------------------------ > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From kaushikggg at gmail.com Fri Feb 5 11:27:42 2016 From: kaushikggg at gmail.com (Kaushik Kulkarni) Date: Fri, 5 Feb 2016 22:57:42 +0530 Subject: [petsc-users] Editor for PETSc In-Reply-To: <87oabvdqr9.fsf@jedbrown.org> References: <20160205163108.GG39386@Patricks-MacBook-Pro-4.local> <87oabvdqr9.fsf@jedbrown.org> Message-ID: Thank you very much it works, and is even giving auto-completion options.!! *Kaushik KulkarniThird Year Undergraduate* *Department of Mechanical EngineeringIndian Institute of Technology Bombay* *Mumbai, Indiahttp://home.iitb.ac.in/~kaushikkulkarni/ * *Contact: +91-9967687150* On Fri, Feb 5, 2016 at 10:43 PM, Jed Brown wrote: > Kaushik Kulkarni writes: > > > Hey Jed, > > Thanks for your response. But a small doubt. > > Am I supposed to add these two lines to the list of flags. > > '-I$PETSC_DIR/include', > > '-I$PETSC_DIR/$PETSC_ARCH/include', > > > > Because I did so, and still it was not working. > > The above probably doesn't expand those variables and they might not be > set in the editor's environment anyway. Expand them manually if you need. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From epscodes at gmail.com Fri Feb 5 11:32:48 2016 From: epscodes at gmail.com (Xiangdong) Date: Fri, 5 Feb 2016 12:32:48 -0500 Subject: [petsc-users] log_summary time ratio and flops ratio In-Reply-To: References: Message-ID: On Fri, Feb 5, 2016 at 12:18 PM, Matthew Knepley wrote: > On Fri, Feb 5, 2016 at 11:12 AM, Xiangdong wrote: > >> Hello everyone, >> >> When I looked at the log_summary, I found that flops ratio of VecDot is >> 1.1, but the time ratio is 1.8. Also the flops ratio of MatMult is 1.2, but >> the time ratio is 1.6. I am using the simple SOR preconditioner on 256 >> procs. Since each of them consumes about 20% of total time, is it worth >> looking into these imbalance issue? If yes, how do I know which procs are >> causing the issue? What tools should I start with? >> > > 1) Always send all the output > > 2) You have a large number of dot products, and I am guessing you have a > slow reduce on this machine (we could see the data at the end of > log_summary). > The waiting time comes from slow collective communication, load > imbalance, system noise, etc. You can see this by looking at the difference > between > VecDot() and VecMDot(). For the latter there is enough work to cover > up the latency, and its imbalance is only 1.1, not 1.8. Thus I think it is > clear that your > problem is due to a slow communication network rather than load > imbalance. > I attached the full summary. At the end, it has Average time to get PetscTime(): 0 Average time for MPI_Barrier(): 8.3971e-05 Average time for zero size MPI_Send(): 7.16746e-06 Is it an indication of slow network? > > 3) Thus you would be better served with a stronger preconditioner (more > flops) which resulted in fewer iterations (less synchronization). > Yes, the preconditioner I am using is simple (just sor). I am also trying other advanced preconditioners. > > Thanks, > > Matt > > >> Thanks. >> >> Xiangdong >> >> >> ------------------------------------------------------------------------------------------------------------------------ >> Event Count Time (sec) Flops >> --- Global --- --- Stage --- Total >> Max Ratio Max Ratio Max Ratio Mess Avg len >> Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s >> >> ------------------------------------------------------------------------------------------------------------------------ >> >> --- Event Stage 0: Main Stage >> >> VecView 44 1.0 3.9026e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >> 4.8e+01 1 0 0 0 0 1 0 0 0 0 0 >> VecDot 817452 1.0 1.5972e+02 1.8 1.51e+10 1.1 0.0e+00 0.0e+00 >> 8.2e+05 18 3 0 0 40 18 3 0 0 40 22969 >> VecMDot 613089 1.0 5.8918e+01 1.1 2.27e+10 1.1 0.0e+00 0.0e+00 >> 6.1e+05 8 4 0 0 30 8 4 0 0 30 93397 >> VecNorm 618619 1.0 1.4426e+02 2.2 1.14e+10 1.1 0.0e+00 0.0e+00 >> 6.2e+05 15 2 0 0 30 15 2 0 0 30 19145 >> VecScale 938 1.0 8.3907e-03 1.2 8.67e+06 1.1 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 250859 >> VecCopy 8159 1.0 1.0394e-01 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> VecSet 1884 1.0 3.9701e-02 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> VecAXPY 1021815 1.0 2.2148e+01 2.1 1.89e+10 1.1 0.0e+00 0.0e+00 >> 0.0e+00 2 4 0 0 0 2 4 0 0 0 207057 >> VecAYPX 613089 1.0 6.5360e+00 1.9 1.13e+10 1.1 0.0e+00 0.0e+00 >> 0.0e+00 1 2 0 0 0 1 2 0 0 0 420336 >> VecAXPBYCZ 191 1.0 3.7365e-03 1.4 7.06e+06 1.1 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 458830 >> VecWAXPY 938 1.0 3.8502e-02 1.4 8.67e+06 1.1 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 54669 >> VecMAXPY 613089 1.0 1.3276e+01 2.2 2.27e+10 1.1 0.0e+00 0.0e+00 >> 0.0e+00 1 4 0 0 0 1 4 0 0 0 414499 >> VecLoad 5 1.0 2.0913e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> VecScatterBegin 820695 1.0 1.8959e+01 2.8 0.00e+00 0.0 1.1e+09 4.6e+03 >> 0.0e+00 2 0100100 0 2 0100100 0 0 >> VecScatterEnd 820695 1.0 5.2002e+01 4.2 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 4 0 0 0 0 4 0 0 0 0 0 >> VecReduceArith 2814 1.0 1.8517e-02 1.2 5.20e+07 1.1 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 681995 >> VecReduceComm 938 1.0 2.6041e+00 3.4 0.00e+00 0.0 0.0e+00 0.0e+00 >> 9.4e+02 0 0 0 0 0 0 0 0 0 0 0 >> MatMult 817452 1.0 1.9379e+02 1.6 2.04e+11 1.2 1.0e+09 4.6e+03 >> 0.0e+00 23 40100100 0 23 40100100 0 253083 >> MatSOR 818390 1.0 1.9608e+02 1.5 2.00e+11 1.1 0.0e+00 0.0e+00 >> 0.0e+00 22 40 0 0 0 22 40 0 0 0 247472 >> MatAssemblyBegin 939 1.0 7.9671e+00 6.0 0.00e+00 0.0 0.0e+00 0.0e+00 >> 1.9e+03 1 0 0 0 0 1 0 0 0 0 0 >> MatAssemblyEnd 939 1.0 2.0570e-01 1.2 0.00e+00 0.0 2.6e+03 1.2e+03 >> 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> MatZeroEntries 938 1.0 1.8202e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> SNESSolve 210 1.0 6.8585e+02 1.0 5.06e+11 1.2 1.1e+09 4.6e+03 >> 2.1e+06 98100100100100 98100100100100 178411 >> SNESFunctionEval 2296 1.0 2.0512e+01 1.3 0.00e+00 0.0 2.9e+06 3.5e+03 >> 0.0e+00 3 0 0 0 0 3 0 0 0 0 0 >> SNESJacobianEval 938 1.0 2.7680e+01 1.0 0.00e+00 0.0 1.2e+06 3.5e+03 >> 1.9e+03 4 0 0 0 0 4 0 0 0 0 0 >> SNESLineSearch 938 1.0 9.2491e+00 1.0 6.07e+07 1.1 1.2e+06 3.5e+03 >> 9.4e+02 1 0 0 0 0 1 0 0 0 0 1593 >> KSPSetUp 938 1.0 2.7661e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 >> 2.0e+01 0 0 0 0 0 0 0 0 0 0 0 >> KSPSolve 938 1.0 6.3433e+02 1.0 5.06e+11 1.2 1.0e+09 4.6e+03 >> 2.0e+06 91100100100 99 91100100100 99 192852 >> PCSetUp 938 1.0 2.1005e-04 3.1 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> PCApply 818390 1.0 1.9675e+02 1.5 2.00e+11 1.1 0.0e+00 0.0e+00 >> 0.0e+00 22 40 0 0 0 22 40 0 0 0 246626 >> >> ------------------------------------------------------------------------------------------------------------------------ >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- Using Petsc Release Version 3.6.3, unknown Max Max/Min Avg Total Time (sec): 6.981e+02 1.00002 6.981e+02 Objects: 2.223e+03 1.00000 2.223e+03 Flops: 5.061e+11 1.15624 4.780e+11 1.224e+14 Flops/sec: 7.250e+08 1.15625 6.847e+08 1.753e+11 MPI Messages: 4.924e+06 2.00000 4.104e+06 1.050e+09 MPI Message Lengths: 2.343e+10 2.16478 4.601e+03 4.833e+12 MPI Reductions: 2.065e+06 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 6.9805e+02 100.0% 1.2236e+14 100.0% 1.050e+09 100.0% 4.601e+03 100.0% 2.065e+06 100.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage VecView 44 1.0 3.9026e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.8e+01 1 0 0 0 0 1 0 0 0 0 0 VecDot 817452 1.0 1.5972e+02 1.8 1.51e+10 1.1 0.0e+00 0.0e+00 8.2e+05 18 3 0 0 40 18 3 0 0 40 22969 VecMDot 613089 1.0 5.8918e+01 1.1 2.27e+10 1.1 0.0e+00 0.0e+00 6.1e+05 8 4 0 0 30 8 4 0 0 30 93397 VecNorm 618619 1.0 1.4426e+02 2.2 1.14e+10 1.1 0.0e+00 0.0e+00 6.2e+05 15 2 0 0 30 15 2 0 0 30 19145 VecScale 938 1.0 8.3907e-03 1.2 8.67e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 250859 VecCopy 8159 1.0 1.0394e-01 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 1884 1.0 3.9701e-02 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 1021815 1.0 2.2148e+01 2.1 1.89e+10 1.1 0.0e+00 0.0e+00 0.0e+00 2 4 0 0 0 2 4 0 0 0 207057 VecAYPX 613089 1.0 6.5360e+00 1.9 1.13e+10 1.1 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 420336 VecAXPBYCZ 191 1.0 3.7365e-03 1.4 7.06e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 458830 VecWAXPY 938 1.0 3.8502e-02 1.4 8.67e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 54669 VecMAXPY 613089 1.0 1.3276e+01 2.2 2.27e+10 1.1 0.0e+00 0.0e+00 0.0e+00 1 4 0 0 0 1 4 0 0 0 414499 VecLoad 5 1.0 2.0913e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 820695 1.0 1.8959e+01 2.8 0.00e+00 0.0 1.1e+09 4.6e+03 0.0e+00 2 0100100 0 2 0100100 0 0 VecScatterEnd 820695 1.0 5.2002e+01 4.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 4 0 0 0 0 4 0 0 0 0 0 VecReduceArith 2814 1.0 1.8517e-02 1.2 5.20e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 681995 VecReduceComm 938 1.0 2.6041e+00 3.4 0.00e+00 0.0 0.0e+00 0.0e+00 9.4e+02 0 0 0 0 0 0 0 0 0 0 0 MatMult 817452 1.0 1.9379e+02 1.6 2.04e+11 1.2 1.0e+09 4.6e+03 0.0e+00 23 40100100 0 23 40100100 0 253083 MatSOR 818390 1.0 1.9608e+02 1.5 2.00e+11 1.1 0.0e+00 0.0e+00 0.0e+00 22 40 0 0 0 22 40 0 0 0 247472 MatAssemblyBegin 939 1.0 7.9671e+00 6.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.9e+03 1 0 0 0 0 1 0 0 0 0 0 MatAssemblyEnd 939 1.0 2.0570e-01 1.2 0.00e+00 0.0 2.6e+03 1.2e+03 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatZeroEntries 938 1.0 1.8202e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SNESSolve 210 1.0 6.8585e+02 1.0 5.06e+11 1.2 1.1e+09 4.6e+03 2.1e+06 98100100100100 98100100100100 178411 SNESFunctionEval 2296 1.0 2.0512e+01 1.3 0.00e+00 0.0 2.9e+06 3.5e+03 0.0e+00 3 0 0 0 0 3 0 0 0 0 0 SNESJacobianEval 938 1.0 2.7680e+01 1.0 0.00e+00 0.0 1.2e+06 3.5e+03 1.9e+03 4 0 0 0 0 4 0 0 0 0 0 SNESLineSearch 938 1.0 9.2491e+00 1.0 6.07e+07 1.1 1.2e+06 3.5e+03 9.4e+02 1 0 0 0 0 1 0 0 0 0 1593 KSPSetUp 938 1.0 2.7661e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+01 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 938 1.0 6.3433e+02 1.0 5.06e+11 1.2 1.0e+09 4.6e+03 2.0e+06 91100100100 99 91100100100 99 192852 PCSetUp 938 1.0 2.1005e-04 3.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCApply 818390 1.0 1.9675e+02 1.5 2.00e+11 1.1 0.0e+00 0.0e+00 0.0e+00 22 40 0 0 0 22 40 0 0 0 246626 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Vector 2150 2150 161353280 0 Vector Scatter 3 2 39136 0 Matrix 3 3 1820156 0 Distributed Mesh 2 1 5040 0 Star Forest Bipartite Graph 4 2 1712 0 Discrete System 2 1 848 0 Index Set 6 6 55904 0 IS L to G Mapping 2 1 22896 0 Viewer 45 44 32032 0 SNES 1 1 1332 0 SNESLineSearch 1 1 864 0 DMSNES 1 1 664 0 Krylov Solver 1 1 1464 0 DMKSP interface 1 1 648 0 Preconditioner 1 1 1024 0 ======================================================================================================================== Average time to get PetscTime(): 0 Average time for MPI_Barrier(): 8.3971e-05 Average time for zero size MPI_Send(): 7.16746e-06 From richardtmills at gmail.com Fri Feb 5 11:39:09 2016 From: richardtmills at gmail.com (Richard Mills) Date: Fri, 5 Feb 2016 09:39:09 -0800 Subject: [petsc-users] Editor for PETSc In-Reply-To: <20160205163108.GG39386@Patricks-MacBook-Pro-4.local> References: <20160205163108.GG39386@Patricks-MacBook-Pro-4.local> Message-ID: +1. I've always edited my PETSc contributions with Vim and used Exuberant Ctags. The combination has always worked fine. (I switched from Emacs to Vim about 15 years ago and I've found that Vim works well enough for me--given a few crucial plugins--that I don't feel bad about never having been able to figure out all the Lisp required to make Emacs do cool stuff.) On Fri, Feb 5, 2016 at 8:31 AM, Patrick Sanan wrote: > If you're using vim, using ctags (you want "exuberant ctags") is a quick > and handy way to > look around in the PETSc source (with its included man pages). > See the manual, http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf, > Section 15.9 . > (and if you use emacs, see the preceding Section 15.8 for instructions on > tags and more). > On Fri, Feb 05, 2016 at 02:27:49PM +0530, Kaushik Kulkarni wrote: > > Hello, > > I am a newbie to PETSc. I am going thorugh the tutorials right now. But I > > am struggling a bit in finding the correct editor for me. Till now I was > > fine with VIM, as it gave me a bunch of completions options in the form > of > > a drop down box. And using the Clang support feature it would also > > highlight the errors hence drastically decreasing my errors. Is there any > > way I could continue working on such environment. > > > > I was using the "YouCompleteMe" plugin with VIM which had all of these > > features. > > > > Thanks, > > *Kaushik* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Feb 5 16:05:53 2016 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 5 Feb 2016 16:05:53 -0600 Subject: [petsc-users] log_summary time ratio and flops ratio In-Reply-To: References: Message-ID: On Fri, Feb 5, 2016 at 11:32 AM, Xiangdong wrote: > > > On Fri, Feb 5, 2016 at 12:18 PM, Matthew Knepley > wrote: > >> On Fri, Feb 5, 2016 at 11:12 AM, Xiangdong wrote: >> >>> Hello everyone, >>> >>> When I looked at the log_summary, I found that flops ratio of VecDot is >>> 1.1, but the time ratio is 1.8. Also the flops ratio of MatMult is 1.2, but >>> the time ratio is 1.6. I am using the simple SOR preconditioner on 256 >>> procs. Since each of them consumes about 20% of total time, is it worth >>> looking into these imbalance issue? If yes, how do I know which procs are >>> causing the issue? What tools should I start with? >>> >> >> 1) Always send all the output >> >> 2) You have a large number of dot products, and I am guessing you have a >> slow reduce on this machine (we could see the data at the end of >> log_summary). >> The waiting time comes from slow collective communication, load >> imbalance, system noise, etc. You can see this by looking at the difference >> between >> VecDot() and VecMDot(). For the latter there is enough work to cover >> up the latency, and its imbalance is only 1.1, not 1.8. Thus I think it is >> clear that your >> problem is due to a slow communication network rather than load >> imbalance. >> > > I attached the full summary. At the end, it has > > Average time to get PetscTime(): 0 > Average time for MPI_Barrier(): 8.3971e-05 > Average time for zero size MPI_Send(): 7.16746e-06 > > Is it an indication of slow network? > I think so. It takes nearly 100 microseconds to synchronize processes. Thanks, Matt > >> 3) Thus you would be better served with a stronger preconditioner (more >> flops) which resulted in fewer iterations (less synchronization). >> > > Yes, the preconditioner I am using is simple (just sor). I am also trying > other advanced preconditioners. > >> >> Thanks, >> >> Matt >> >> >>> Thanks. >>> >>> Xiangdong >>> >>> >>> ------------------------------------------------------------------------------------------------------------------------ >>> Event Count Time (sec) Flops >>> --- Global --- --- Stage --- Total >>> Max Ratio Max Ratio Max Ratio Mess Avg len >>> Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s >>> >>> ------------------------------------------------------------------------------------------------------------------------ >>> >>> --- Event Stage 0: Main Stage >>> >>> VecView 44 1.0 3.9026e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 4.8e+01 1 0 0 0 0 1 0 0 0 0 0 >>> VecDot 817452 1.0 1.5972e+02 1.8 1.51e+10 1.1 0.0e+00 0.0e+00 >>> 8.2e+05 18 3 0 0 40 18 3 0 0 40 22969 >>> VecMDot 613089 1.0 5.8918e+01 1.1 2.27e+10 1.1 0.0e+00 0.0e+00 >>> 6.1e+05 8 4 0 0 30 8 4 0 0 30 93397 >>> VecNorm 618619 1.0 1.4426e+02 2.2 1.14e+10 1.1 0.0e+00 0.0e+00 >>> 6.2e+05 15 2 0 0 30 15 2 0 0 30 19145 >>> VecScale 938 1.0 8.3907e-03 1.2 8.67e+06 1.1 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 250859 >>> VecCopy 8159 1.0 1.0394e-01 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> VecSet 1884 1.0 3.9701e-02 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> VecAXPY 1021815 1.0 2.2148e+01 2.1 1.89e+10 1.1 0.0e+00 0.0e+00 >>> 0.0e+00 2 4 0 0 0 2 4 0 0 0 207057 >>> VecAYPX 613089 1.0 6.5360e+00 1.9 1.13e+10 1.1 0.0e+00 0.0e+00 >>> 0.0e+00 1 2 0 0 0 1 2 0 0 0 420336 >>> VecAXPBYCZ 191 1.0 3.7365e-03 1.4 7.06e+06 1.1 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 458830 >>> VecWAXPY 938 1.0 3.8502e-02 1.4 8.67e+06 1.1 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 54669 >>> VecMAXPY 613089 1.0 1.3276e+01 2.2 2.27e+10 1.1 0.0e+00 0.0e+00 >>> 0.0e+00 1 4 0 0 0 1 4 0 0 0 414499 >>> VecLoad 5 1.0 2.0913e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> VecScatterBegin 820695 1.0 1.8959e+01 2.8 0.00e+00 0.0 1.1e+09 4.6e+03 >>> 0.0e+00 2 0100100 0 2 0100100 0 0 >>> VecScatterEnd 820695 1.0 5.2002e+01 4.2 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 4 0 0 0 0 4 0 0 0 0 0 >>> VecReduceArith 2814 1.0 1.8517e-02 1.2 5.20e+07 1.1 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 681995 >>> VecReduceComm 938 1.0 2.6041e+00 3.4 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 9.4e+02 0 0 0 0 0 0 0 0 0 0 0 >>> MatMult 817452 1.0 1.9379e+02 1.6 2.04e+11 1.2 1.0e+09 4.6e+03 >>> 0.0e+00 23 40100100 0 23 40100100 0 253083 >>> MatSOR 818390 1.0 1.9608e+02 1.5 2.00e+11 1.1 0.0e+00 0.0e+00 >>> 0.0e+00 22 40 0 0 0 22 40 0 0 0 247472 >>> MatAssemblyBegin 939 1.0 7.9671e+00 6.0 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 1.9e+03 1 0 0 0 0 1 0 0 0 0 0 >>> MatAssemblyEnd 939 1.0 2.0570e-01 1.2 0.00e+00 0.0 2.6e+03 1.2e+03 >>> 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> MatZeroEntries 938 1.0 1.8202e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> SNESSolve 210 1.0 6.8585e+02 1.0 5.06e+11 1.2 1.1e+09 4.6e+03 >>> 2.1e+06 98100100100100 98100100100100 178411 >>> SNESFunctionEval 2296 1.0 2.0512e+01 1.3 0.00e+00 0.0 2.9e+06 3.5e+03 >>> 0.0e+00 3 0 0 0 0 3 0 0 0 0 0 >>> SNESJacobianEval 938 1.0 2.7680e+01 1.0 0.00e+00 0.0 1.2e+06 3.5e+03 >>> 1.9e+03 4 0 0 0 0 4 0 0 0 0 0 >>> SNESLineSearch 938 1.0 9.2491e+00 1.0 6.07e+07 1.1 1.2e+06 3.5e+03 >>> 9.4e+02 1 0 0 0 0 1 0 0 0 0 1593 >>> KSPSetUp 938 1.0 2.7661e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 2.0e+01 0 0 0 0 0 0 0 0 0 0 0 >>> KSPSolve 938 1.0 6.3433e+02 1.0 5.06e+11 1.2 1.0e+09 4.6e+03 >>> 2.0e+06 91100100100 99 91100100100 99 192852 >>> PCSetUp 938 1.0 2.1005e-04 3.1 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> PCApply 818390 1.0 1.9675e+02 1.5 2.00e+11 1.1 0.0e+00 0.0e+00 >>> 0.0e+00 22 40 0 0 0 22 40 0 0 0 246626 >>> >>> ------------------------------------------------------------------------------------------------------------------------ >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Feb 5 16:27:29 2016 From: jed at jedbrown.org (Jed Brown) Date: Fri, 05 Feb 2016 15:27:29 -0700 Subject: [petsc-users] log_summary time ratio and flops ratio In-Reply-To: References: Message-ID: <874mdmeqsu.fsf@jedbrown.org> Matthew Knepley writes: >> I attached the full summary. At the end, it has >> >> Average time to get PetscTime(): 0 >> Average time for MPI_Barrier(): 8.3971e-05 >> Average time for zero size MPI_Send(): 7.16746e-06 >> >> Is it an indication of slow network? >> > > I think so. It takes nearly 100 microseconds to synchronize processes. Edison with 65536 processes: Average time for MPI_Barrier(): 4.23908e-05 Average time for zero size MPI_Send(): 2.46466e-06 Mira with 16384 processes: Average time for MPI_Barrier(): 5.7075e-06 Average time for zero size MPI_Send(): 1.33179e-05 Titan with 131072 processes: Average time for MPI_Barrier(): 0.000368595 Average time for zero size MPI_Send(): 1.71567e-05 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From epscodes at gmail.com Fri Feb 5 20:59:25 2016 From: epscodes at gmail.com (Xiangdong) Date: Fri, 5 Feb 2016 21:59:25 -0500 Subject: [petsc-users] log_summary time ratio and flops ratio In-Reply-To: <874mdmeqsu.fsf@jedbrown.org> References: <874mdmeqsu.fsf@jedbrown.org> Message-ID: If I want to know whether only rank 0 is slow (since it may has more io) or actually a portion of cores are slow, what tools can I start with? Thanks. Xiangdong On Fri, Feb 5, 2016 at 5:27 PM, Jed Brown wrote: > Matthew Knepley writes: > >> I attached the full summary. At the end, it has > >> > >> Average time to get PetscTime(): 0 > >> Average time for MPI_Barrier(): 8.3971e-05 > >> Average time for zero size MPI_Send(): 7.16746e-06 > >> > >> Is it an indication of slow network? > >> > > > > I think so. It takes nearly 100 microseconds to synchronize processes. > > Edison with 65536 processes: > Average time for MPI_Barrier(): 4.23908e-05 > Average time for zero size MPI_Send(): 2.46466e-06 > > Mira with 16384 processes: > Average time for MPI_Barrier(): 5.7075e-06 > Average time for zero size MPI_Send(): 1.33179e-05 > > Titan with 131072 processes: > Average time for MPI_Barrier(): 0.000368595 > Average time for zero size MPI_Send(): 1.71567e-05 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Feb 5 21:34:08 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 5 Feb 2016 21:34:08 -0600 Subject: [petsc-users] log_summary time ratio and flops ratio In-Reply-To: References: <874mdmeqsu.fsf@jedbrown.org> Message-ID: Make the same run with no IO and see if the numbers are much better and if the load balance is better. > On Feb 5, 2016, at 8:59 PM, Xiangdong wrote: > > If I want to know whether only rank 0 is slow (since it may has more io) or actually a portion of cores are slow, what tools can I start with? > > Thanks. > > Xiangdong > > On Fri, Feb 5, 2016 at 5:27 PM, Jed Brown wrote: > Matthew Knepley writes: > >> I attached the full summary. At the end, it has > >> > >> Average time to get PetscTime(): 0 > >> Average time for MPI_Barrier(): 8.3971e-05 > >> Average time for zero size MPI_Send(): 7.16746e-06 > >> > >> Is it an indication of slow network? > >> > > > > I think so. It takes nearly 100 microseconds to synchronize processes. > > Edison with 65536 processes: > Average time for MPI_Barrier(): 4.23908e-05 > Average time for zero size MPI_Send(): 2.46466e-06 > > Mira with 16384 processes: > Average time for MPI_Barrier(): 5.7075e-06 > Average time for zero size MPI_Send(): 1.33179e-05 > > Titan with 131072 processes: > Average time for MPI_Barrier(): 0.000368595 > Average time for zero size MPI_Send(): 1.71567e-05 > From adrianj at epcc.ed.ac.uk Sun Feb 7 03:51:53 2016 From: adrianj at epcc.ed.ac.uk (Adrian Jackson) Date: Sun, 7 Feb 2016 09:51:53 +0000 Subject: [petsc-users] fstack-array issue Message-ID: <56B713B9.20708@epcc.ed.ac.uk> Hi, We're using PETSc in a fortran code and get an error if we compile the code using the -fstack-arrays flag when using the GNU fortran compilers. This is the error we are getting: [22]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [22]PETSC ERROR: Arguments are incompatible [22]PETSC ERROR: Zero diagonal on row 23003 .... [22]PETSC ERROR: #1 MatInvertDiagonal_SeqAIJ() line 1841 in /b/cray-petsc/.cray-build/GNU/49/sandybridge/cray-petsc-base-dynamic/petsc-3.5.2/src/mat/impls/aij/seq/aij.c [22]PETSC ERROR: #2 MatSOR_SeqAIJ() line 1873 in /b/cray-petsc/.cray-build/GNU/49/sandybridge/cray-petsc-base-dynamic/petsc-3.5.2/src/mat/impls/aij/seq/aij.c .... [22]PETSC ERROR: #14 KSPSolve() line 459 in /b/cray-petsc/.cray-build/GNU/49/sandybridge/cray-petsc-base-dynamic/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c WARNING: Failed to converge. PETSc did not converge for matrix solve of: DeltaU Reason for non-convergence is undefined: 0 Number of iterations: 0 Sending signal to dump and finish I've checked the right hand side vector passed to KSPSolve in both cases and it's the same. I've not fully verified the ksp context is exactly the same but it doesn't seem to be corrupted when using -fstack-arrays. Is this a problem people have encountered before? thanks adrianj -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. From kaushikggg at gmail.com Sun Feb 7 05:29:01 2016 From: kaushikggg at gmail.com (Kaushik Kulkarni) Date: Sun, 7 Feb 2016 16:59:01 +0530 Subject: [petsc-users] VecGetValues not working. Message-ID: Hello, I am a beginner at PETSc, so please excuse for such a trivial doubt. I am writing a program to learn about PETSc vectors. So in the process I thought to write a small program to learn about initializing and accessing vectors, which creates vectors with size `rank+1' where rank is initialized using `MPI_Comm_rank (PETSC_COMM_WORLD, &rank);`. And, then simply I looped thorugh all the elements to get the sum. But as can seen in the output(attached), it is returning "junk" values. Source Code:- http://pastebin.com/Swku7A3X Output:- http://pastebin.com/paSWnFwz Thanks, *Kaushik* -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Sun Feb 7 06:58:38 2016 From: dave.mayhem23 at gmail.com (Dave May) Date: Sun, 7 Feb 2016 13:58:38 +0100 Subject: [petsc-users] VecGetValues not working. In-Reply-To: References: Message-ID: On Sunday, 7 February 2016, Kaushik Kulkarni wrote: > Hello, > I am a beginner at PETSc, so please excuse for such a trivial doubt. I am > writing a program to learn about PETSc vectors. So in the process I thought > to write a small program to learn about initializing and accessing vectors, > which creates vectors with size `rank+1' where rank is initialized using > `MPI_Comm_rank (PETSC_COMM_WORLD, &rank);`. And, then simply I looped > thorugh all the elements to get the sum. But as can seen in the > output(attached), it is returning "junk" values. > Your call to printf is treating Y[i] as an integer (%d) when it should be be using a specifier for a floating point number , eg %1.4e > Source Code:- http://pastebin.com/Swku7A3X > Output:- http://pastebin.com/paSWnFwz > > > Thanks, > > *Kaushik* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kaushikggg at gmail.com Sun Feb 7 07:08:10 2016 From: kaushikggg at gmail.com (Kaushik Kulkarni) Date: Sun, 7 Feb 2016 18:38:10 +0530 Subject: [petsc-users] VecGetValues not working. In-Reply-To: References: Message-ID: Thanks Dave, No doubt I am a noob. *?~?Kaushik* On Sun, Feb 7, 2016 at 6:28 PM, Dave May wrote: > > > On Sunday, 7 February 2016, Kaushik Kulkarni wrote: > >> Hello, >> I am a beginner at PETSc, so please excuse for such a trivial doubt. I am >> writing a program to learn about PETSc vectors. So in the process I thought >> to write a small program to learn about initializing and accessing vectors, >> which creates vectors with size `rank+1' where rank is initialized using >> `MPI_Comm_rank (PETSC_COMM_WORLD, &rank);`. And, then simply I looped >> thorugh all the elements to get the sum. But as can seen in the >> output(attached), it is returning "junk" values. >> > > Your call to printf is treating Y[i] as an integer (%d) when it should be > be using a specifier for a floating point number , eg %1.4e > > > > > >> Source Code:- http://pastebin.com/Swku7A3X >> Output:- http://pastebin.com/paSWnFwz >> >> >> Thanks, >> >> *Kaushik* >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Feb 7 10:13:02 2016 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 7 Feb 2016 10:13:02 -0600 Subject: [petsc-users] fstack-array issue In-Reply-To: <56B713B9.20708@epcc.ed.ac.uk> References: <56B713B9.20708@epcc.ed.ac.uk> Message-ID: It seems like that you have exceeded stack space with your local array. Did you run under valgrind? Matt On Sun, Feb 7, 2016 at 3:51 AM, Adrian Jackson wrote: > Hi, > > We're using PETSc in a fortran code and get an error if we compile the > code using the -fstack-arrays flag when using the GNU fortran compilers. > This is the error we are getting: > > [22]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [22]PETSC ERROR: Arguments are incompatible > [22]PETSC ERROR: Zero diagonal on row 23003 > .... > [22]PETSC ERROR: #1 MatInvertDiagonal_SeqAIJ() line 1841 in > > /b/cray-petsc/.cray-build/GNU/49/sandybridge/cray-petsc-base-dynamic/petsc-3.5.2/src/mat/impls/aij/seq/aij.c > [22]PETSC ERROR: #2 MatSOR_SeqAIJ() line 1873 in > > /b/cray-petsc/.cray-build/GNU/49/sandybridge/cray-petsc-base-dynamic/petsc-3.5.2/src/mat/impls/aij/seq/aij.c > .... > [22]PETSC ERROR: #14 KSPSolve() line 459 in > > /b/cray-petsc/.cray-build/GNU/49/sandybridge/cray-petsc-base-dynamic/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c > WARNING: Failed to converge. > PETSc did not converge for matrix solve of: DeltaU > Reason for non-convergence is undefined: 0 > Number of iterations: 0 > Sending signal to dump and finish > > > I've checked the right hand side vector passed to KSPSolve in both cases > and it's the same. I've not fully verified the ksp context is exactly > the same but it doesn't seem to be corrupted when using -fstack-arrays. > Is this a problem people have encountered before? > > thanks > > adrianj > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From adrianj at epcc.ed.ac.uk Sun Feb 7 10:52:54 2016 From: adrianj at epcc.ed.ac.uk (Adrian Jackson) Date: Sun, 7 Feb 2016 16:52:54 +0000 Subject: [petsc-users] fstack-array issue In-Reply-To: References: <56B713B9.20708@epcc.ed.ac.uk> Message-ID: <56B77666.6010100@epcc.ed.ac.uk> Hi, I have tried with -fstack-protector-all and that didn't pick up anything, but not with valgrind yet. cheers adrianj On 07/02/2016 16:13, Matthew Knepley wrote: > It seems like that you have exceeded stack space with your local array. > Did you run under valgrind? > > Matt > > On Sun, Feb 7, 2016 at 3:51 AM, Adrian Jackson > wrote: > > Hi, > > We're using PETSc in a fortran code and get an error if we compile the > code using the -fstack-arrays flag when using the GNU fortran compilers. > This is the error we are getting: > > [22]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [22]PETSC ERROR: Arguments are incompatible > [22]PETSC ERROR: Zero diagonal on row 23003 > .... > [22]PETSC ERROR: #1 MatInvertDiagonal_SeqAIJ() line 1841 in > /b/cray-petsc/.cray-build/GNU/49/sandybridge/cray-petsc-base-dynamic/petsc-3.5.2/src/mat/impls/aij/seq/aij.c > [22]PETSC ERROR: #2 MatSOR_SeqAIJ() line 1873 in > /b/cray-petsc/.cray-build/GNU/49/sandybridge/cray-petsc-base-dynamic/petsc-3.5.2/src/mat/impls/aij/seq/aij.c > .... > [22]PETSC ERROR: #14 KSPSolve() line 459 in > /b/cray-petsc/.cray-build/GNU/49/sandybridge/cray-petsc-base-dynamic/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c > WARNING: Failed to converge. > PETSc did not converge for matrix solve of: DeltaU > Reason for non-convergence is undefined: 0 > Number of iterations: 0 > Sending signal to dump and finish > > > I've checked the right hand side vector passed to KSPSolve in both cases > and it's the same. I've not fully verified the ksp context is exactly > the same but it doesn't seem to be corrupted when using -fstack-arrays. > Is this a problem people have encountered before? > > thanks > > adrianj > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. From ncreati at inogs.it Mon Feb 8 02:41:35 2016 From: ncreati at inogs.it (Nicola Creati) Date: Mon, 8 Feb 2016 09:41:35 +0100 Subject: [petsc-users] petsc4py snes behavior Message-ID: <56B854BF.1030306@inogs.it> Hello, I'm trying to convert some PETSc examples from C to Python. I noted a strange behavior of the snes solver. It seems that the same example in Python and C are characterized by a different number of function evaluations. I converted ex25.c example in the " snes/examples/tutorials" folder. The number of function evaluations is always bigger in Python and increases with the number of grid nodes. The increased number of function evaluations affects the performance and in other examples can drastically increase the execution time (for example in ex30.c " Steady-state 2D subduction..."). This is my Python version of the simple ex25.c: ## # SNES/Tutorials/ex25 import sys import petsc4py petsc4py.init(sys.argv) from petsc4py import PETSc import numpy as np class UserContext(object): def __init__(self, da): self.da = da self.localX = da.createLocalVec() def formFunction(self, snes, X, F): self.da.globalToLocal(X, self.localX) x = self.da.getVecArray(self.localX) f = self.da.getVecArray(F) (x0, x1), (y0, y1)= self.da.getRanges() hx = 1/float(x1-1) hy = 1/float(y1-1) for j in range(y0, y1): for i in range(x0, x1): if i == 0 or i == (x1-1) or j == 0 or j == (y1-1 ): f[j,i] = x[j,i] - (1.0 - (2.0 * hx * i - 1.0) * (2.0 * hx * i -1.0)) else: gradup = (x[j+1, i] -x[j, i])/hy graddown = (x[j, i] -x[j-1, i])/hy gradright = (x[j, i+1] -x[j, i])/hx gradleft = (x[j, i] -x[j, i-1])/hx gradx = 0.5 * (x[j, i+1] - x[j, i-1])/hx grady = 0.5 * (x[j+1, i] - x[j-1, i])/hy coeffup = 1.0/np.sqrt(1.0 + gradup * gradup + gradx * gradx) coeffdown = 1.0/np.sqrt(1.0 + graddown * graddown + gradx * gradx) coeffleft = 1.0/np.sqrt(1.0 + gradleft * gradleft + grady * grady) coeffright = 1.0/np.sqrt(1.0 + gradright * gradright + grady * grady) f[j, i] = (coeffup * gradup - coeffdown * graddown) * hx + (coeffright * gradright - coeffleft * gradleft) * hy snes = PETSc.SNES().create() da = PETSc.DMDA().create(dim=(-5, -5), stencil_type = PETSc.DMDA.StencilType.STAR, stencil_width =1, setup=False) da.setFromOptions() da.setUp() snes.setDM(da) ctx = UserContext(da) da.setAppCtx(ctx) F = da.createGlobalVec() snes.setFunction(ctx.formFunction, F) snes.setFromOptions() x = da.createGlobalVector() snes.solve(None, x) its = snes.getIterationNumber() lits = snes.getLinearSolveIterations() print "Number of SNES iterations = %d" % its print "Number of Linear iterations = %d" % lits litspit = lits/float(its) print "Average Linear its / SNES = %e" % float(litspit) ##################################### I executed the code with the following options: python ex25.py -snes_view -snes_monitor -da_grid_x 50 -da_grid_y 50 This is the output from Python: Konsole output 0 SNES Function norm 7.229568285876e+00 1 SNES Function norm 1.970854029907e-02 2 SNES Function norm 5.721344031051e-03 3 SNES Function norm 1.179738644817e-03 4 SNES Function norm 1.028879149924e-04 5 SNES Function norm 7.472012969660e-07 6 SNES Function norm 4.744184684658e-11 SNES Object: 1 MPI processes type: newtonls maximum iterations=50, maximum function evaluations=10000 tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 total number of linear solver iterations=223 total number of function evaluations=37 norm schedule ALWAYS SNESLineSearch Object: 1 MPI processes type: bt interpolation: cubic alpha=1.000000e-04 maxstep=1.000000e+08, minlambda=1.000000e-12 tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 maximum iterations=40 KSP Object: 1 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: ilu ILU: out-of-place factorization 0 levels of fill tolerance for zero pivot 2.22045e-14 matrix ordering: natural factor fill ratio given 1, needed 1 Factored matrix follows: Mat Object: 1 MPI processes type: seqaij rows=2500, cols=2500 package used to perform factorization: petsc total: nonzeros=12300, allocated nonzeros=12300 total number of mallocs used during MatSetValues calls =0 not using I-node routines linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=2500, cols=2500 total: nonzeros=12300, allocated nonzeros=12300 total number of mallocs used during MatSetValues calls =0 not using I-node routines Number of SNES iterations = 6 Number of Linear iterations = 223 Average Linear its / SNES = 3.716667e+01 This the output from the original C code with the same options (./ex25 -snes_view -snes_monitor -da_grid_x 50 -da_grid_y 50): Konsole output 0 SNES Function norm 7.229568285876e+00 1 SNES Function norm 1.970854029907e-02 2 SNES Function norm 5.721344035538e-03 3 SNES Function norm 1.179738636852e-03 4 SNES Function norm 1.028879138425e-04 5 SNES Function norm 7.472012249912e-07 6 SNES Function norm 4.743884624164e-11 SNES Object: 1 MPI processes type: newtonls maximum iterations=50, maximum function evaluations=10000 tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 total number of linear solver iterations=223 total number of function evaluations=7 norm schedule ALWAYS SNESLineSearch Object: 1 MPI processes type: bt interpolation: cubic alpha=1.000000e-04 maxstep=1.000000e+08, minlambda=1.000000e-12 tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 maximum iterations=40 KSP Object: 1 MPI processes type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: ilu ILU: out-of-place factorization 0 levels of fill tolerance for zero pivot 2.22045e-14 matrix ordering: natural factor fill ratio given 1, needed 1 Factored matrix follows: Mat Object: 1 MPI processes type: seqaij rows=2500, cols=2500 package used to perform factorization: petsc total: nonzeros=12300, allocated nonzeros=12300 total number of mallocs used during MatSetValues calls =0 not using I-node routines linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=2500, cols=2500 total: nonzeros=12300, allocated nonzeros=12300 total number of mallocs used during MatSetValues calls =0 not using I-node routines Number of SNES iterations = 6 Number of Linear iterations = 223 Average Linear its / SNES = 3.716667e+01 Why the number of function evaluations is so different? I'm using last version of PETSc and petsc4py. Thank you. Nicola -- _____________________________________________________________________ Nicola Creati Istituto Nazionale di Oceanografia e di Geofisica Sperimentale - OGS IRI (Ricerca Teconologica e Infrastrutture) Department B.go Grotta Gigante - Brisciki 42/c 34010 Sgonico - Zgonik (TS) - Italy Tel. +39-0402140213 Fax +39-040327307 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Feb 8 05:02:44 2016 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 8 Feb 2016 05:02:44 -0600 Subject: [petsc-users] petsc4py snes behavior In-Reply-To: <56B854BF.1030306@inogs.it> References: <56B854BF.1030306@inogs.it> Message-ID: On Mon, Feb 8, 2016 at 2:41 AM, Nicola Creati wrote: > Hello, > I'm trying to convert some PETSc examples from C to Python. I noted a > strange behavior of the snes solver. It seems that the same example in > Python and C are characterized by a different number of function > evaluations. I converted ex25.c example in the " snes/examples/tutorials" > folder. The number of function evaluations is always bigger in Python and > increases with the number of grid nodes. The increased number of function > evaluations affects the performance and in other examples can drastically > increase the execution time (for example in ex30.c " Steady-state 2D > subduction..."). > This is my Python version of the simple ex25.c: > You did not provide the Jacobian function, resulting in a finite difference approximation that uses more function evaluations. Matt > ## # SNES/Tutorials/ex25 > import sys > import petsc4py > petsc4py.init(sys.argv) > from petsc4py import PETSc > import numpy as np > > class UserContext(object): > def __init__(self, da): > self.da = da > self.localX = da.createLocalVec() > > def formFunction(self, snes, X, F): > > self.da.globalToLocal(X, self.localX) > x = self.da.getVecArray(self.localX) > f = self.da.getVecArray(F) > > (x0, x1), (y0, y1)= self.da.getRanges() > hx = 1/float(x1-1) > hy = 1/float(y1-1) > > for j in range(y0, y1): > for i in range(x0, x1): > if i == 0 or i == (x1-1) or j == 0 or j == (y1-1 ): > f[j,i] = x[j,i] - (1.0 - (2.0 * hx * i - 1.0) * (2.0 * > hx * i -1.0)) > else: > gradup = (x[j+1, i] -x[j, i])/hy > graddown = (x[j, i] -x[j-1, i])/hy > gradright = (x[j, i+1] -x[j, i])/hx > gradleft = (x[j, i] -x[j, i-1])/hx > > gradx = 0.5 * (x[j, i+1] - x[j, i-1])/hx > grady = 0.5 * (x[j+1, i] - x[j-1, i])/hy > > coeffup = 1.0/np.sqrt(1.0 + gradup * gradup + gradx * > gradx) > coeffdown = 1.0/np.sqrt(1.0 + graddown * graddown + > gradx * gradx) > > coeffleft = 1.0/np.sqrt(1.0 + gradleft * gradleft + > grady * grady) > coeffright = 1.0/np.sqrt(1.0 + gradright * gradright + > grady * grady) > > f[j, i] = (coeffup * gradup - coeffdown * graddown) * > hx + (coeffright * gradright - coeffleft * gradleft) * hy > > > snes = PETSc.SNES().create() > da = PETSc.DMDA().create(dim=(-5, -5), > stencil_type = PETSc.DMDA.StencilType.STAR, > stencil_width =1, > setup=False) > da.setFromOptions() > da.setUp() > > snes.setDM(da) > > ctx = UserContext(da) > da.setAppCtx(ctx) > > F = da.createGlobalVec() > snes.setFunction(ctx.formFunction, F) > > snes.setFromOptions() > > x = da.createGlobalVector() > > snes.solve(None, x) > its = snes.getIterationNumber() > lits = snes.getLinearSolveIterations() > > print "Number of SNES iterations = %d" % its > print "Number of Linear iterations = %d" % lits > > litspit = lits/float(its) > print "Average Linear its / SNES = %e" % float(litspit) > > ##################################### > > I executed the code with the following options: > > python ex25.py -snes_view -snes_monitor -da_grid_x 50 -da_grid_y 50 > > This is the output from Python: > > 0 SNES Function norm 7.229568285876e+00 > 1 SNES Function norm 1.970854029907e-02 > 2 SNES Function norm 5.721344031051e-03 > 3 SNES Function norm 1.179738644817e-03 > 4 SNES Function norm 1.028879149924e-04 > 5 SNES Function norm 7.472012969660e-07 > 6 SNES Function norm 4.744184684658e-11 > SNES Object: 1 MPI processes > type: newtonls > maximum iterations=50, maximum function evaluations=10000 > tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 > total number of linear solver iterations=223 > total number of function evaluations=37 > norm schedule ALWAYS > SNESLineSearch Object: 1 MPI processes > type: bt > interpolation: cubic > alpha=1.000000e-04 > maxstep=1.000000e+08, minlambda=1.000000e-12 > tolerances: relative=1.000000e-08, absolute=1.000000e-15, > lambda=1.000000e-08 > maximum iterations=40 > KSP Object: 1 MPI processes > type: gmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: ilu > ILU: out-of-place factorization > 0 levels of fill > tolerance for zero pivot 2.22045e-14 > matrix ordering: natural > factor fill ratio given 1, needed 1 > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqaij > rows=2500, cols=2500 > package used to perform factorization: petsc > total: nonzeros=12300, allocated nonzeros=12300 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=2500, cols=2500 > total: nonzeros=12300, allocated nonzeros=12300 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > Number of SNES iterations = 6 > Number of Linear iterations = 223 > Average Linear its / SNES = 3.716667e+01 > > This the output from the original C code with the same options (./ex25 -snes_view > -snes_monitor -da_grid_x 50 -da_grid_y 50): > > 0 SNES Function norm 7.229568285876e+00 > 1 SNES Function norm 1.970854029907e-02 > 2 SNES Function norm 5.721344035538e-03 > 3 SNES Function norm 1.179738636852e-03 > 4 SNES Function norm 1.028879138425e-04 > 5 SNES Function norm 7.472012249912e-07 > 6 SNES Function norm 4.743884624164e-11 > SNES Object: 1 MPI processes > type: newtonls > maximum iterations=50, maximum function evaluations=10000 > tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 > total number of linear solver iterations=223 > total number of function evaluations=7 > norm schedule ALWAYS > SNESLineSearch Object: 1 MPI processes > type: bt > interpolation: cubic > alpha=1.000000e-04 > maxstep=1.000000e+08, minlambda=1.000000e-12 > tolerances: relative=1.000000e-08, absolute=1.000000e-15, > lambda=1.000000e-08 > maximum iterations=40 > KSP Object: 1 MPI processes > type: gmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: ilu > ILU: out-of-place factorization > 0 levels of fill > tolerance for zero pivot 2.22045e-14 > matrix ordering: natural > factor fill ratio given 1, needed 1 > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqaij > rows=2500, cols=2500 > package used to perform factorization: petsc > total: nonzeros=12300, allocated nonzeros=12300 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=2500, cols=2500 > total: nonzeros=12300, allocated nonzeros=12300 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > Number of SNES iterations = 6 > Number of Linear iterations = 223 > Average Linear its / SNES = 3.716667e+01 > > Why the number of function evaluations is so different? > I'm using last version of PETSc and petsc4py. > Thank you. > > Nicola > > -- > _____________________________________________________________________ > Nicola Creati > Istituto Nazionale di Oceanografia e di Geofisica Sperimentale - OGS > IRI (Ricerca Teconologica e Infrastrutture) Department > B.go Grotta Gigante - Brisciki 42/c > 34010 Sgonico - Zgonik (TS) - Italy > Tel. +39-0402140213 > Fax +39-040327307 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jacek.miloszewski at gmail.com Mon Feb 8 05:31:44 2016 From: jacek.miloszewski at gmail.com (Jacek Miloszewski) Date: Mon, 8 Feb 2016 11:31:44 +0000 Subject: [petsc-users] MatSetValues timing Message-ID: Dear PETSc users, I use PETSc to assembly a square matrix (in the attached example it is n = 4356) which has around 12% of non-zero entries. I timed my code using various number of process (data in table). Now I have 2 questions: 1. Why with doubling number of processes the speed-up is 4x? I would expect 2x at the most. 2. Is there a way to speed-up matrix construction in general? I attach piece of my fortran code at the bottom. At compilation time I have the following knowledge about the matrix: the total number of non-zero matrix elements, all diagonal elements are non-zero. Timing data: number of proc time [s] speedup ratio 1 2044.941 2 504.692 4.051859352 4 149.678 3.371851575 8 64.102 2.334997348 16 17.296 3.706174838 32 4.43 3.904288939 64 1.096 4.041970803 Code: call MatCreate(PETSC_COMM_WORLD, a, ierr) call MatSetSizes(a, PETSC_DECIDE, PETSC_DECIDE, n, n, ierr) call MatSetFromOptions(a, ierr) call MatSetUp(a, ierr) call MatGetOwnershipRange(a, Istart, Iend, ierr) call system_clock(t1) t_dim = h_dim*e_dim do row = Istart, Iend - 1 ! row do col = 0, t_dim - 1 ! col call h_ij(row + 1, col + 1, n_h, n_e, b_h, b_e, h_dim, e_dim, e_sp, v, basis, ht, hs, info_1) if (hs) then hh = ht call MatSetValues(a, 1, row, 1, col, hh, INSERT_VALUES, ierr) end if end do end do call system_clock(t2, ct) if (rank == 0) then write(*, '(a, f0.3)') 'Matrix assembly time: ', real((t2 - t1), r8)/real(ct, r8) end if -- Best Wishes Jacek Miloszewski -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Mon Feb 8 05:37:41 2016 From: dave.mayhem23 at gmail.com (Dave May) Date: Mon, 8 Feb 2016 12:37:41 +0100 Subject: [petsc-users] MatSetValues timing In-Reply-To: References: Message-ID: On 8 February 2016 at 12:31, Jacek Miloszewski wrote: > Dear PETSc users, > > I use PETSc to assembly a square matrix (in the attached example it is n = > 4356) which has around 12% of non-zero entries. I timed my code using > various number of process (data in table). Now I have 2 questions: > > 1. Why with doubling number of processes the speed-up is 4x? I would > expect 2x at the most. > Your timing doesn't appear to include the time required to scatter off-processor values. You should move the timer to be after the following calls MatAssemblyBegin(a,MAT_FINAL_ASSEMBLY); MatAssemblyEnd(a,MAT_FINAL_ASSEMBLY); > > 2. Is there a way to speed-up matrix construction in general? I attach > piece of my fortran code at the bottom. At compilation time I have the > following knowledge about the matrix: the total number of non-zero matrix > elements, all diagonal elements are non-zero. > > Timing data: > > number of proc time [s] speedup ratio > 1 2044.941 > 2 504.692 4.051859352 > 4 149.678 3.371851575 > 8 64.102 2.334997348 > 16 17.296 3.706174838 > 32 4.43 3.904288939 > 64 1.096 4.041970803 > > Code: > > call MatCreate(PETSC_COMM_WORLD, a, ierr) > call MatSetSizes(a, PETSC_DECIDE, PETSC_DECIDE, n, n, ierr) > call MatSetFromOptions(a, ierr) > call MatSetUp(a, ierr) > > call MatGetOwnershipRange(a, Istart, Iend, ierr) > > call system_clock(t1) > t_dim = h_dim*e_dim > do row = Istart, Iend - 1 ! row > do col = 0, t_dim - 1 ! col > call h_ij(row + 1, col + 1, n_h, n_e, b_h, b_e, h_dim, e_dim, e_sp, > v, basis, ht, hs, info_1) > if (hs) then > hh = ht > call MatSetValues(a, 1, row, 1, col, hh, INSERT_VALUES, ierr) > end if > end do > end do > call system_clock(t2, ct) > if (rank == 0) then > write(*, '(a, f0.3)') 'Matrix assembly time: ', real((t2 - t1), > r8)/real(ct, r8) > end if > > -- > Best Wishes > Jacek Miloszewski > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Feb 8 05:39:00 2016 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 8 Feb 2016 05:39:00 -0600 Subject: [petsc-users] MatSetValues timing In-Reply-To: References: Message-ID: On Mon, Feb 8, 2016 at 5:37 AM, Dave May wrote: > > > On 8 February 2016 at 12:31, Jacek Miloszewski < > jacek.miloszewski at gmail.com> wrote: > >> Dear PETSc users, >> >> I use PETSc to assembly a square matrix (in the attached example it is n >> = 4356) which has around 12% of non-zero entries. I timed my code using >> various number of process (data in table). Now I have 2 questions: >> >> 1. Why with doubling number of processes the speed-up is 4x? I would >> expect 2x at the most. >> > > Your timing doesn't appear to include the time required to scatter > off-processor values. > You should move the timer to be after the following calls > > MatAssemblyBegin(a,MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(a,MAT_FINAL_ASSEMBLY); > Also, I would guess that you are not preallocating the matrix: http://www.mcs.anl.gov/petsc/documentation/faq.html#efficient-assembly Matt > > >> >> 2. Is there a way to speed-up matrix construction in general? I attach >> piece of my fortran code at the bottom. At compilation time I have the >> following knowledge about the matrix: the total number of non-zero matrix >> elements, all diagonal elements are non-zero. >> >> Timing data: >> >> number of proc time [s] speedup ratio >> 1 2044.941 >> 2 504.692 4.051859352 >> 4 149.678 3.371851575 >> 8 64.102 2.334997348 >> 16 17.296 3.706174838 >> 32 4.43 3.904288939 >> 64 1.096 4.041970803 >> >> Code: >> >> call MatCreate(PETSC_COMM_WORLD, a, ierr) >> call MatSetSizes(a, PETSC_DECIDE, PETSC_DECIDE, n, n, ierr) >> call MatSetFromOptions(a, ierr) >> call MatSetUp(a, ierr) >> >> call MatGetOwnershipRange(a, Istart, Iend, ierr) >> >> call system_clock(t1) >> t_dim = h_dim*e_dim >> do row = Istart, Iend - 1 ! row >> do col = 0, t_dim - 1 ! col >> call h_ij(row + 1, col + 1, n_h, n_e, b_h, b_e, h_dim, e_dim, e_sp, >> v, basis, ht, hs, info_1) >> if (hs) then >> hh = ht >> call MatSetValues(a, 1, row, 1, col, hh, INSERT_VALUES, ierr) >> end if >> end do >> end do >> call system_clock(t2, ct) >> if (rank == 0) then >> write(*, '(a, f0.3)') 'Matrix assembly time: ', real((t2 - t1), >> r8)/real(ct, r8) >> end if >> >> -- >> Best Wishes >> Jacek Miloszewski >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Mon Feb 8 05:41:06 2016 From: dave.mayhem23 at gmail.com (Dave May) Date: Mon, 8 Feb 2016 12:41:06 +0100 Subject: [petsc-users] MatSetValues timing In-Reply-To: References: Message-ID: On 8 February 2016 at 12:31, Jacek Miloszewski wrote: > Dear PETSc users, > > I use PETSc to assembly a square matrix (in the attached example it is n = > 4356) which has around 12% of non-zero entries. I timed my code using > various number of process (data in table). Now I have 2 questions: > > 1. Why with doubling number of processes the speed-up is 4x? I would > expect 2x at the most. > > 2. Is there a way to speed-up matrix construction in general? I attach > piece of my fortran code at the bottom. At compilation time I have the > following knowledge about the matrix: the total number of non-zero matrix > elements, all diagonal elements are non-zero. > Performance improvements of the assembly require the preallocation of the matrix be specified. Check out these pages for further information http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSeqAIJSetPreallocation.html http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatMPIAIJSetPreallocation.html also note the FAQ item "Assembling large sparse matrices takes a long time. What can I do make this process faster? or MatSetValues() is so slow, what can I do to speed it up?" http://www.mcs.anl.gov/petsc/documentation/faq.html Thanks, Dave > Timing data: > > number of proc time [s] speedup ratio > 1 2044.941 > 2 504.692 4.051859352 > 4 149.678 3.371851575 > 8 64.102 2.334997348 > 16 17.296 3.706174838 > 32 4.43 3.904288939 > 64 1.096 4.041970803 > > Code: > > call MatCreate(PETSC_COMM_WORLD, a, ierr) > call MatSetSizes(a, PETSC_DECIDE, PETSC_DECIDE, n, n, ierr) > call MatSetFromOptions(a, ierr) > call MatSetUp(a, ierr) > > call MatGetOwnershipRange(a, Istart, Iend, ierr) > > call system_clock(t1) > t_dim = h_dim*e_dim > do row = Istart, Iend - 1 ! row > do col = 0, t_dim - 1 ! col > call h_ij(row + 1, col + 1, n_h, n_e, b_h, b_e, h_dim, e_dim, e_sp, > v, basis, ht, hs, info_1) > if (hs) then > hh = ht > call MatSetValues(a, 1, row, 1, col, hh, INSERT_VALUES, ierr) > end if > end do > end do > call system_clock(t2, ct) > if (rank == 0) then > write(*, '(a, f0.3)') 'Matrix assembly time: ', real((t2 - t1), > r8)/real(ct, r8) > end if > > -- > Best Wishes > Jacek Miloszewski > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hng.email at gmail.com Mon Feb 8 11:20:08 2016 From: hng.email at gmail.com (Hom Nath Gharti) Date: Mon, 8 Feb 2016 12:20:08 -0500 Subject: [petsc-users] PCFIELDSPLIT question In-Reply-To: References: Message-ID: Hi Matt, Hi all, I am trying to have some feel for PCFIELDSPLIT testing on a very small matrix (attached below). I have 4 degrees of freedoms. I use 4 processors. I want to split 4 dofs into two fields each having two dofs. I don't know whether this my be a problem for petsc. When I use the command: srun -n 4 ./xtestfs -ksp_monitor -ksp_view It runs fine. But when I want to use field split options using the command: srun -n 4 ./xtestfs -ksp_monitor -ksp_type fgmres -pc_type fieldsplit -fieldsplit_0_ksp_type gmres -fieldsplit_0_pc_type bjacobi -fieldsplit_1_pc_type jacobi I get the following error message: [0]PETSC ERROR: Petsc has generated inconsistent data [0]PETSC ERROR: Unhandled case, must have at least two fields, not 1 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [0]PETSC ERROR: /tigress/hgharti/test/testfs/./xtestfs on a arch-linux2-c-debug named tiger-r11n1 by hgharti Mon Feb 8 11:40:03 2016 [0]PETSC ERROR: Configure options --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64/ --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --with-mpiexec=mpiexec --with-debugging=1 --download-scalapack --download-mumps --download-pastix --download-superlu --download-superlu_dist --download-metis --download-parmetis --download-ptscotch --download-hypre [0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 469 in /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 486 in /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: #3 PCSetUp() line 983 in /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #4 KSPSetUp() line 332 in /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #5 KSPSolve() line 546 in /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-fieldsplit_0_ksp_type value: gmres Option left: name:-fieldsplit_0_pc_type value: bjacobi Option left: name:-fieldsplit_1_pc_type value: jacobi forrtl: error (76): Abort trap signal I tried several trials but could not fix the problem. Is it the FORTRAN problem or am I doing something wrong? Any suggestions would be greatly appreciated. For your information I use: PETSc-3.6.3 intel-16.0 compiler intel-mpi-5.1.1 Program Best, Hom Nath ! simple test case for PCFIELDSPLIT ! ----------------------------------------------------------------------- program testfs implicit none #include "petsc/finclude/petscsys.h" #include "petsc/finclude/petscvec.h" #include "petsc/finclude/petscvec.h90" #include "petsc/finclude/petscmat.h" #include "petsc/finclude/petscksp.h" #include "petsc/finclude/petscpc.h" #include "petsc/finclude/petscviewer.h" #include "petsc/finclude/petscviewer.h90" Vec x,b,u Mat A KSP ksp PC pc PetscErrorCode ierr PetscBool flg PetscInt errcode,its,maxitr,myrank,nproc PetscInt nedof,nelmt,ndof,nenod,ngdof,fixdof,nsparse,neq PetscInt,allocatable :: krow_sparse(:),ggdof_elmt(:,:), & inode_elmt(:,:) PetscReal,allocatable :: storef(:,:),storekmat(:,:,:) PetscInt gdof0(2),gdof1(2) ! initialize MPI call MPI_INIT(errcode) if(errcode /= 0)stop 'ERROR: cannot initialize MPI!' call MPI_COMM_RANK(MPI_COMM_WORLD,myrank,errcode) if(errcode /= 0)stop 'ERROR: cannot find processor ID' call MPI_COMM_SIZE(MPI_COMM_WORLD,nproc,errcode) if(errcode /= 0)stop 'ERROR: cannot find number of processors!' ! define some parameters ! 1D model with 4 elements. Each elements consits of 2 nodes. Node 0 is fixed. ! 1-----------2-----------3-----------4-----------5 nelmt=1 ! per processor nenod=2 ! number of nodes per element nedof=2 ! number of DOFs per element ndof=2 ! number of DOFs ngdof=4 ! total number of global DOFs fixdof=1 if(myrank==0)then neq=1 ! number of equations nsparse=1 else neq=2 ! number of equations nsparse=4 endif allocate(storef(nedof,nelmt),storekmat(nedof,nedof,nelmt), & ggdof_elmt(nedof,nelmt),inode_elmt(nenod,nelmt),krow_sparse(nsparse)) storef=0.0_8 ! local RHS vector storekmat=0.0_8 ! local stiffness matrix if(myrank==0)then krow_sparse(:)=1 storef(:,1)=(/12.5000_8, 12.5000_8/) storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & 0.0166666667_8, 1.2333333332_8/),(/nedof,nedof/)) inode_elmt(:,1) = (/1,2/) ggdof_elmt(:,1) = (/0,1/) ! 0 because left node is fixed elseif(myrank==1)then krow_sparse(:)=(/1,1,2,2/) storef(:,1)=(/12.5000_8, 12.5000_8/) storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & 0.0166666667_8, 1.2333333332_8/),(/nedof,nedof/)) inode_elmt(:,1) = (/1,2/) ggdof_elmt(:,1) = (/1,2/) elseif(myrank==2)then krow_sparse(:)=(/1,1,2,2/) storef(:,1)=(/12.5000_8, 12.5000_8/) storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & 0.0166666667_8, 1.2333333332_8/),(/nedof,nedof/)) inode_elmt(:,1) = (/1,2/) ggdof_elmt(:,1) = (/2,3/) elseif(myrank==3)then krow_sparse(:)=(/1,1,2,2/) storef(:,1)=(/12.5000_8, 22.5000_8/) storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & 0.0166666667_8, 1.2333333332_8/),(/nedof,nedof/)) inode_elmt(:,1) = (/1,2/) ggdof_elmt(:,1) = (/3,4/) endif call petsc_initialize call petsc_set_matrix call petsc_set_vector call petsc_solve call petsc_finalize ! deallocate deallocate(storef,storekmat,ggdof_elmt,inode_elmt) deallocate(krow_sparse) contains !------------------------------------------------------------------------------- subroutine petsc_initialize() implicit none PetscInt :: istart,iend PetscInt :: nzeros_max,nzeros_min PetscReal :: atol,dtol,rtol PetscInt,allocatable :: nzeros(:) IS gdof0_is,gdof1_is call PetscInitialize(PETSC_NULL_CHARACTER,ierr) call PetscOptionsGetInt(PETSC_NULL_CHARACTER,'-n',ngdof,flg,ierr) ! count number of nonzeros per row allocate(nzeros(neq)) nzeros=0 nzeros(krow_sparse)=nzeros(krow_sparse)+1 nzeros_max=maxval(nzeros) nzeros_min=minval(nzeros) ! create matrix call MatCreate(PETSC_COMM_WORLD,A,ierr) call MatSetType(A,MATMPIAIJ,ierr) CHKERRQ(ierr) call MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,ngdof,ngdof,ierr) CHKERRQ(ierr) ! preallocation call MatMPIAIJSetPreallocation(A,nzeros_max,PETSC_NULL_INTEGER,nzeros_max, & PETSC_NULL_INTEGER,ierr) CHKERRQ(ierr) call MatGetOwnershipRange(A,istart,iend,ierr) CHKERRQ(ierr) print*,'ownership:',myrank,istart,iend ! create vector call VecCreate(PETSC_COMM_WORLD,x,ierr) call VecSetSizes(x,PETSC_DECIDE,ngdof,ierr) call VecSetType(x,VECMPI,ierr) call VecDuplicate(x,b,ierr) call VecDuplicate(x,u,ierr) !******************************************************************************* ! get IS for split fields gdof0=(/0,1/) gdof1=(/2,3/) call ISCreateGeneral(PETSC_COMM_WORLD,2,gdof0,PETSC_COPY_VALUES,gdof0_is,ierr) CHKERRQ(ierr) call ISCreateGeneral(PETSC_COMM_WORLD,2,gdof1,PETSC_COPY_VALUES,gdof1_is,ierr) CHKERRQ(ierr) !******************************************************************************* ! Create linear solver context call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) call KSPSetOperators(ksp,A,A,ierr) call KSPGetPC(ksp,pc,ierr) CHKERRQ(ierr) !******************************************************************************* ! split PC call PCFieldSplitSetIS(pc,"0",gdof0_is,ierr); CHKERRQ(ierr) call ISDestroy(gdof0_is,ierr) CHKERRQ(ierr) call PCFieldSplitSetIS(pc,"1",gdof1_is,ierr); CHKERRQ(ierr) call ISDestroy(gdof1_is,ierr) CHKERRQ(ierr) !******************************************************************************* rtol = 1.0d-6 atol = 1.0d-6 dtol = 1.0d10 maxitr = 1000 call KSPSetTolerances(ksp,rtol,atol,dtol,maxitr,ierr) CHKERRQ(ierr) call KSPSetFromOptions(ksp,ierr) end subroutine petsc_initialize !------------------------------------------------------------------------------- subroutine petsc_set_matrix() implicit none integer :: i,i_elmt,j,ncount integer :: ggdof(NEDOF),idof(NEDOF),igdof(NEDOF) ! Set and assemble matrix. call MatZeroEntries(A,ierr) CHKERRQ(ierr) do i_elmt=1,nelmt ggdof(:)=ggdof_elmt(:,i_elmt) ggdof(:)=ggdof(:)-1 ! petsc index starts from 0 ncount=0; idof=-1; igdof=-1 do i=1,NEDOF do j=1,NEDOF if(ggdof(i).ge.0.and.ggdof(j).ge.0)then call MatSetValues(A,1,ggdof(i),1,ggdof(j),storekmat(i,j,i_elmt), & ADD_VALUES,ierr) CHKERRQ(ierr) endif enddo enddo enddo call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) if(myrank==0)print*,'matrix setting & assembly complete!' end subroutine petsc_set_matrix !------------------------------------------------------------------------------- subroutine petsc_set_vector() implicit none PetscScalar zero integer :: i,i_elmt,ncount integer :: ggdof(NEDOF),idof(NEDOF),igdof(NEDOF) ! set vector zero=0.0_8 call VecSet(b,zero,ierr) do i_elmt=1,nelmt ggdof(:)=ggdof_elmt(:,i_elmt) ggdof(:)=ggdof(:)-1 ! petsc index starts from 0 ncount=0; idof=-1; igdof=-1 do i=1,NEDOF if(ggdof(i).ge.0)then call VecSetValues(b,1,ggdof(i),storef(i,i_elmt),ADD_VALUES,ierr); CHKERRQ(ierr) endif enddo enddo ! assemble vector call VecAssemblyBegin(b,ierr) call VecAssemblyEnd(b,ierr) if(myrank==0)print*,'vector setting & assembly complete!' end subroutine petsc_set_vector !------------------------------------------------------------------------------- subroutine petsc_solve() implicit none PetscInt ireason PetscScalar a_v(1) PetscOffset a_i PetscInt n PetscReal,allocatable :: sdata(:) call VecGetSize(b,n,ierr) CHKERRQ(ierr) allocate(sdata(n)) sdata=0.0_8 call VecGetArray(b,a_v,a_i,ierr) CHKERRQ(ierr) sdata(1:n)=a_v(a_i+1:a_i+n) call VecRestoreArray(b,a_v,a_i,ierr) CHKERRQ(ierr) print*,'rhs:',sdata call KSPSolve(ksp,b,x,ierr) sdata=0.0_8 call VecGetArray(x,a_v,a_i,ierr) CHKERRQ(ierr) sdata(1:n)=a_v(a_i+1:a_i+n) call VecRestoreArray(b,a_v,a_i,ierr) CHKERRQ(ierr) print*,'solution:',sdata ! Check convergence call KSPGetConvergedReason(ksp,ireason,ierr) if(myrank==0)print*,'converges reason',myrank,ireason call KSPGetIterationNumber(ksp,its,ierr) if(myrank==0)print*,'Iterations:',its deallocate(sdata) end subroutine petsc_solve !------------------------------------------------------------------------------- subroutine petsc_finalize() implicit none ! Free work space. call VecDestroy(x,ierr) call VecDestroy(u,ierr) call VecDestroy(b,ierr) call MatDestroy(A,ierr) call KSPDestroy(ksp,ierr) call PetscFinalize(ierr) CHKERRQ(ierr) end subroutine petsc_finalize !------------------------------------------------------------------------------- end program testfs On Tue, Feb 2, 2016 at 4:54 PM, Matthew Knepley wrote: > On Tue, Feb 2, 2016 at 3:11 PM, Hom Nath Gharti wrote: >> >> Thanks a lot. I will try. > > > Also, if you send a small test case, we can run it and tell you the problem. > > Matt > >> >> Hom Nath >> >> On Tue, Feb 2, 2016 at 4:02 PM, Matthew Knepley wrote: >> > On Tue, Feb 2, 2016 at 3:01 PM, Hom Nath Gharti >> > wrote: >> >> >> >> Thanks again Matt. Unfortunately got the same errors with '0'. I think >> >> both are valid in Fortran. >> > >> > >> > Then you will have to go in the debugger and see why its not creating 4 >> > splits, since this is exactly >> > what it does in our tests. This is how all the SNES tests that I use >> > work. I >> > am sure its a Fortran >> > problem. >> > >> > Thanks, >> > >> > Matt >> > >> >> >> >> Hom Nath >> >> >> >> On Tue, Feb 2, 2016 at 3:42 PM, Matthew Knepley >> >> wrote: >> >> > On Tue, Feb 2, 2016 at 2:35 PM, Hom Nath Gharti >> >> > wrote: >> >> >> >> >> >> Thank you so much Matt. >> >> >> >> >> >> I now tried the following: >> >> >> >> >> >> ====================================================== >> >> >> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >> >> >> call KSPGetPC(ksp,pc,ierr) >> >> >> >> >> >> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); >> >> >> call ISDestroy(gdofu_is,ierr) >> >> >> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); >> >> >> call ISDestroy(gdofchi_is,ierr) >> >> >> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); >> >> >> call ISDestroy(gdofp_is,ierr) >> >> >> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); >> >> >> call ISDestroy(gdofphi_is,ierr) >> >> >> >> >> >> ! Amat changes in each time steps >> >> >> call KSPSetOperators(ksp,Amat,Amat,ierr) !version >= 3.5.0 >> >> >> >> >> >> call KSPSolve(ksp,bvec,xvec,ierr) >> >> >> ====================================================== >> >> > >> >> > >> >> > I am guessing that "0" is not a valid string for your Fortran >> >> > compiler. >> >> > Are >> >> > you sure >> >> > its not single quotes '0'? >> >> > >> >> > Matt >> >> > >> >> >> >> >> >> But I get the following error: >> >> >> >> >> >> [0]PETSC ERROR: --------------------- Error Message >> >> >> -------------------------------------------------------------- >> >> >> [0]PETSC ERROR: Petsc has generated inconsistent data >> >> >> [0]PETSC ERROR: Unhandled case, must have at least two fields, not 1 >> >> >> [0]PETSC ERROR: See >> >> >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >> >> >> shooting. >> >> >> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >> >> >> [0]PETSC ERROR: >> >> >> /tigress/hgharti/gitwork/SPECFEM3D_GLOBEVSPP/./bin/xspecfem3D on a >> >> >> arch-linux2-c-debug named tiger-r3c1n7 by hgharti Tue Feb 2 15: >> >> >> 29:30 2016 >> >> >> [0]PETSC ERROR: Configure options >> >> >> >> >> >> >> >> >> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64/ >> >> >> --with-cc=mpicc --with-cxx=mpicxx --wit >> >> >> h-fc=mpif90 --with-mpiexec=mpiexec --with-debugging=1 >> >> >> --download-scalapack --download-mumps --download-pastix >> >> >> --download-superlu --download-superlu_dist --download-metis >> >> >> --download-parmetis --download-ptscotch --download-hypre >> >> >> [0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 469 in >> >> >> >> >> >> >> >> >> >> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> >> >> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 486 in >> >> >> >> >> >> >> >> >> >> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> >> >> [0]PETSC ERROR: #3 PCSetUp() line 983 in >> >> >> >> >> >> >> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/interface/precon.c >> >> >> [0]PETSC ERROR: #4 KSPSetUp() line 332 in >> >> >> >> >> >> >> >> >> >> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >> >> >> [0]PETSC ERROR: #5 KSPSolve() line 546 in >> >> >> >> >> >> >> >> >> >> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >> >> >> forrtl: error (76): Abort trap signal >> >> >> >> >> >> Am I missing something? >> >> >> >> >> >> Thanks, >> >> >> Hom Nath >> >> >> >> >> >> On Tue, Feb 2, 2016 at 3:24 PM, Matthew Knepley >> >> >> wrote: >> >> >> > On Tue, Feb 2, 2016 at 2:20 PM, Hom Nath Gharti >> >> >> > >> >> >> > wrote: >> >> >> >> >> >> >> >> Hi Matt, hi all, >> >> >> >> >> >> >> >> I am trying to use PCFIELDSPLIT for my solver which consists of 4 >> >> >> >> different variables, namely, u (displacement vector), \chi >> >> >> >> (displacement potential), p(pressure), and \phi (gravity >> >> >> >> potential). >> >> >> >> >> >> >> >> My code segment looks like the following: >> >> >> >> ====================================================== >> >> >> >> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >> >> >> >> call KSPGetPC(ksp,pc,ierr) >> >> >> >> nsplit=4 >> >> >> >> call PCFieldSplitSetBlockSize(pc,nsplit,ierr); >> >> >> > >> >> >> > >> >> >> > You do not need the statement above. >> >> >> > >> >> >> >> >> >> >> >> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); >> >> >> >> call ISDestroy(gdofu_is,ierr) >> >> >> >> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); >> >> >> >> call ISDestroy(gdofchi_is,ierr) >> >> >> >> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); >> >> >> >> call ISDestroy(gdofp_is,ierr) >> >> >> >> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); >> >> >> >> call ISDestroy(gdofphi_is,ierr) >> >> >> >> >> >> >> >> call PCFieldSplitGetSubKSP(pc,nsplit,subksp,ierr); >> >> >> > >> >> >> > >> >> >> > These SetOperators() calls are wrong. I am not sure why you want >> >> >> > them >> >> >> > here. >> >> >> > Also, that means you do not need the call above. >> >> >> > >> >> >> > Thanks, >> >> >> > >> >> >> > Matt >> >> >> > >> >> >> >> >> >> >> >> call KSPSetOperators(subksp(1),Amat,Amat,ierr); >> >> >> >> call KSPSetOperators(subksp(2),Amat,Amat,ierr); >> >> >> >> call KSPSetOperators(subksp(3),Amat,Amat,ierr); >> >> >> >> call KSPSetOperators(subksp(4),Amat,Amat,ierr); >> >> >> >> >> >> >> >> call KSPSolve(ksp,bvec,xvec,ierr) >> >> >> >> ====================================================== >> >> >> >> >> >> >> >> But I am getting the following error: >> >> >> >> [79]PETSC ERROR: Null argument, when expecting valid pointer >> >> >> >> [79]PETSC ERROR: Null Object: Parameter # 1 >> >> >> >> [79]PETSC ERROR: #1 KSPSetOperators() line 536 in >> >> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interf >> >> >> >> >> >> >> >> It looks like I am doing something wrong in "call >> >> >> >> KSPSetOperators" >> >> >> >> but >> >> >> >> I could not figure that out. >> >> >> >> >> >> >> >> Could anybody help me to fix this problem? I looked into almost >> >> >> >> all >> >> >> >> related examples but I could not really figure out the correct >> >> >> >> steps >> >> >> >> after "call PCFieldSplitSetIS". >> >> >> >> >> >> >> >> Any help will be greatly appreciated. >> >> >> >> >> >> >> >> Best, >> >> >> >> Hom nath >> >> >> >> >> >> >> >> On Sun, Jan 24, 2016 at 7:14 PM, Hom Nath Gharti >> >> >> >> >> >> >> >> wrote: >> >> >> >> > Thank you so much Matt! I will try. >> >> >> >> > >> >> >> >> > Hom Nath >> >> >> >> > >> >> >> >> > On Sun, Jan 24, 2016 at 6:26 AM, Matthew Knepley >> >> >> >> > >> >> >> >> > wrote: >> >> >> >> >> On Fri, Jan 22, 2016 at 2:19 PM, Hom Nath Gharti >> >> >> >> >> >> >> >> >> >> wrote: >> >> >> >> >>> >> >> >> >> >>> Dear all, >> >> >> >> >>> >> >> >> >> >>> I am new to PcFieldSplit. >> >> >> >> >>> >> >> >> >> >>> I have a matrix formed using MATMPIAIJ. Is it possible to use >> >> >> >> >>> PCFIELDSPLIT operations in this type of matrix? Or does it >> >> >> >> >>> have >> >> >> >> >>> to >> >> >> >> >>> be >> >> >> >> >>> MATMPIBIJ or MATNEST format? >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Yes, you can split AIJ. >> >> >> >> >> >> >> >> >> >>> >> >> >> >> >>> If possible for MATMPIAIJ, could anybody provide me a simple >> >> >> >> >>> example >> >> >> >> >>> or few steps? Variables in the equations are displacement >> >> >> >> >>> vector, >> >> >> >> >>> scalar potential and pressure. >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> If you do not have a collocated discretization, then you have >> >> >> >> >> to >> >> >> >> >> use >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html >> >> >> >> >> >> >> >> >> >> Thanks, >> >> >> >> >> >> >> >> >> >> Matt >> >> >> >> >> >> >> >> >> >>> >> >> >> >> >>> Thanks for help. >> >> >> >> >>> >> >> >> >> >>> Hom Nath >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> >> >> What most experimenters take for granted before they begin >> >> >> >> >> their >> >> >> >> >> experiments >> >> >> >> >> is infinitely more interesting than any results to which their >> >> >> >> >> experiments >> >> >> >> >> lead. >> >> >> >> >> -- Norbert Wiener >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > -- >> >> >> > What most experimenters take for granted before they begin their >> >> >> > experiments >> >> >> > is infinitely more interesting than any results to which their >> >> >> > experiments >> >> >> > lead. >> >> >> > -- Norbert Wiener >> >> > >> >> > >> >> > >> >> > >> >> > -- >> >> > What most experimenters take for granted before they begin their >> >> > experiments >> >> > is infinitely more interesting than any results to which their >> >> > experiments >> >> > lead. >> >> > -- Norbert Wiener >> > >> > >> > >> > >> > -- >> > What most experimenters take for granted before they begin their >> > experiments >> > is infinitely more interesting than any results to which their >> > experiments >> > lead. >> > -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener From bsmith at mcs.anl.gov Mon Feb 8 13:27:10 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 8 Feb 2016 13:27:10 -0600 Subject: [petsc-users] PCFIELDSPLIT question In-Reply-To: References: Message-ID: In this case you need to provide two pieces of information to the PCFIELDSPLIT. What we call the "block size" or bs which is the number of "basic fields" in the problem. For example if at each grid point you have x-velocity, y-velocity, and pressure the block size is 3. The second is you need to tell PCFIELDSPLIT what "basic fields" are in each split you want to use. In your case you can do this with -pc_fieldsplit_block_size 4. (Note if you use a DM to define your problem the PCFIELDSPLIT will automatically get the bs from the DM so you do not need to provide it. Similarly if you set a block size on your Mat the PCFIELDSPLIT will use that so you don't need to set it). Then use -pc_fieldsplit_0_fields 0,1 to indicate your first split is the 0 and 1 basic fields and -pc_fieldsplit_1_fields 2,3 to indicate the second split is the 2 and 3 basic fields. (By default if you do not provide this then PCFIELDSPLIT will use bs splits (4 in your case) one for each basic field. Barry > On Feb 8, 2016, at 11:20 AM, Hom Nath Gharti wrote: > > Hi Matt, Hi all, > > I am trying to have some feel for PCFIELDSPLIT testing on a very small > matrix (attached below). I have 4 degrees of freedoms. I use 4 > processors. I want to split 4 dofs into two fields each having two > dofs. I don't know whether this my be a problem for petsc. When I use > the command: > srun -n 4 ./xtestfs -ksp_monitor -ksp_view > > It runs fine. > > But when I want to use field split options using the command: > srun -n 4 ./xtestfs -ksp_monitor -ksp_type fgmres -pc_type fieldsplit > -fieldsplit_0_ksp_type gmres -fieldsplit_0_pc_type bjacobi > -fieldsplit_1_pc_type jacobi > > I get the following error message: > [0]PETSC ERROR: Petsc has generated inconsistent data > [0]PETSC ERROR: Unhandled case, must have at least two fields, not 1 > [0]PETSC ERROR: See > http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > [0]PETSC ERROR: /tigress/hgharti/test/testfs/./xtestfs on a > arch-linux2-c-debug named tiger-r11n1 by hgharti Mon Feb 8 11:40:03 > 2016 > [0]PETSC ERROR: Configure options > --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64/ > --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > --with-mpiexec=mpiexec --with-debugging=1 --download-scalapack > --download-mumps --download-pastix --download-superlu > --download-superlu_dist --download-metis --download-parmetis > --download-ptscotch --download-hypre > [0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 469 in > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 486 in > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > [0]PETSC ERROR: #3 PCSetUp() line 983 in > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #4 KSPSetUp() line 332 in > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #5 KSPSolve() line 546 in > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c > WARNING! There are options you set that were not used! > WARNING! could be spelling mistake, etc! > Option left: name:-fieldsplit_0_ksp_type value: gmres > Option left: name:-fieldsplit_0_pc_type value: bjacobi > Option left: name:-fieldsplit_1_pc_type value: jacobi > forrtl: error (76): Abort trap signal > > I tried several trials but could not fix the problem. Is it the > FORTRAN problem or am I doing something wrong? Any suggestions would > be greatly appreciated. > For your information I use: > PETSc-3.6.3 > intel-16.0 compiler > intel-mpi-5.1.1 > > Program > > > Best, > Hom Nath > > ! simple test case for PCFIELDSPLIT > ! ----------------------------------------------------------------------- > program testfs > implicit none > #include "petsc/finclude/petscsys.h" > #include "petsc/finclude/petscvec.h" > #include "petsc/finclude/petscvec.h90" > #include "petsc/finclude/petscmat.h" > #include "petsc/finclude/petscksp.h" > #include "petsc/finclude/petscpc.h" > #include "petsc/finclude/petscviewer.h" > #include "petsc/finclude/petscviewer.h90" > > Vec x,b,u > Mat A > KSP ksp > PC pc > PetscErrorCode ierr > PetscBool flg > PetscInt errcode,its,maxitr,myrank,nproc > PetscInt nedof,nelmt,ndof,nenod,ngdof,fixdof,nsparse,neq > PetscInt,allocatable :: krow_sparse(:),ggdof_elmt(:,:), & > inode_elmt(:,:) > > PetscReal,allocatable :: storef(:,:),storekmat(:,:,:) > PetscInt gdof0(2),gdof1(2) > > ! initialize MPI > > call MPI_INIT(errcode) > if(errcode /= 0)stop 'ERROR: cannot initialize MPI!' > call MPI_COMM_RANK(MPI_COMM_WORLD,myrank,errcode) > if(errcode /= 0)stop 'ERROR: cannot find processor ID' > call MPI_COMM_SIZE(MPI_COMM_WORLD,nproc,errcode) > if(errcode /= 0)stop 'ERROR: cannot find number of processors!' > > ! define some parameters > ! 1D model with 4 elements. Each elements consits of 2 nodes. Node 0 is fixed. > ! 1-----------2-----------3-----------4-----------5 > nelmt=1 ! per processor > nenod=2 ! number of nodes per element > nedof=2 ! number of DOFs per element > ndof=2 ! number of DOFs > ngdof=4 ! total number of global DOFs > fixdof=1 > > if(myrank==0)then > neq=1 ! number of equations > nsparse=1 > else > neq=2 ! number of equations > nsparse=4 > endif > > allocate(storef(nedof,nelmt),storekmat(nedof,nedof,nelmt), & > ggdof_elmt(nedof,nelmt),inode_elmt(nenod,nelmt),krow_sparse(nsparse)) > > storef=0.0_8 ! local RHS vector > storekmat=0.0_8 ! local stiffness matrix > if(myrank==0)then > krow_sparse(:)=1 > storef(:,1)=(/12.5000_8, 12.5000_8/) > storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & > 0.0166666667_8, 1.2333333332_8/),(/nedof,nedof/)) > inode_elmt(:,1) = (/1,2/) > ggdof_elmt(:,1) = (/0,1/) ! 0 because left node is fixed > elseif(myrank==1)then > krow_sparse(:)=(/1,1,2,2/) > storef(:,1)=(/12.5000_8, 12.5000_8/) > storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & > 0.0166666667_8, 1.2333333332_8/),(/nedof,nedof/)) > inode_elmt(:,1) = (/1,2/) > ggdof_elmt(:,1) = (/1,2/) > elseif(myrank==2)then > krow_sparse(:)=(/1,1,2,2/) > storef(:,1)=(/12.5000_8, 12.5000_8/) > storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & > 0.0166666667_8, 1.2333333332_8/),(/nedof,nedof/)) > inode_elmt(:,1) = (/1,2/) > ggdof_elmt(:,1) = (/2,3/) > elseif(myrank==3)then > krow_sparse(:)=(/1,1,2,2/) > storef(:,1)=(/12.5000_8, 22.5000_8/) > storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & > 0.0166666667_8, 1.2333333332_8/),(/nedof,nedof/)) > inode_elmt(:,1) = (/1,2/) > ggdof_elmt(:,1) = (/3,4/) > endif > > call petsc_initialize > call petsc_set_matrix > call petsc_set_vector > call petsc_solve > call petsc_finalize > > ! deallocate > deallocate(storef,storekmat,ggdof_elmt,inode_elmt) > deallocate(krow_sparse) > contains > !------------------------------------------------------------------------------- > subroutine petsc_initialize() > implicit none > PetscInt :: istart,iend > PetscInt :: nzeros_max,nzeros_min > PetscReal :: atol,dtol,rtol > PetscInt,allocatable :: nzeros(:) > IS gdof0_is,gdof1_is > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > call PetscOptionsGetInt(PETSC_NULL_CHARACTER,'-n',ngdof,flg,ierr) > > ! count number of nonzeros per row > allocate(nzeros(neq)) > nzeros=0 > nzeros(krow_sparse)=nzeros(krow_sparse)+1 > nzeros_max=maxval(nzeros) > nzeros_min=minval(nzeros) > > ! create matrix > call MatCreate(PETSC_COMM_WORLD,A,ierr) > call MatSetType(A,MATMPIAIJ,ierr) > CHKERRQ(ierr) > call MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,ngdof,ngdof,ierr) > CHKERRQ(ierr) > ! preallocation > call MatMPIAIJSetPreallocation(A,nzeros_max,PETSC_NULL_INTEGER,nzeros_max, & > PETSC_NULL_INTEGER,ierr) > CHKERRQ(ierr) > > call MatGetOwnershipRange(A,istart,iend,ierr) > CHKERRQ(ierr) > print*,'ownership:',myrank,istart,iend > ! create vector > call VecCreate(PETSC_COMM_WORLD,x,ierr) > call VecSetSizes(x,PETSC_DECIDE,ngdof,ierr) > call VecSetType(x,VECMPI,ierr) > call VecDuplicate(x,b,ierr) > call VecDuplicate(x,u,ierr) > > !******************************************************************************* > ! get IS for split fields > gdof0=(/0,1/) > gdof1=(/2,3/) > call ISCreateGeneral(PETSC_COMM_WORLD,2,gdof0,PETSC_COPY_VALUES,gdof0_is,ierr) > CHKERRQ(ierr) > call ISCreateGeneral(PETSC_COMM_WORLD,2,gdof1,PETSC_COPY_VALUES,gdof1_is,ierr) > CHKERRQ(ierr) > !******************************************************************************* > > ! Create linear solver context > call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) > call KSPSetOperators(ksp,A,A,ierr) > > call KSPGetPC(ksp,pc,ierr) > CHKERRQ(ierr) > > !******************************************************************************* > ! split PC > call PCFieldSplitSetIS(pc,"0",gdof0_is,ierr); > CHKERRQ(ierr) > call ISDestroy(gdof0_is,ierr) > CHKERRQ(ierr) > call PCFieldSplitSetIS(pc,"1",gdof1_is,ierr); > CHKERRQ(ierr) > call ISDestroy(gdof1_is,ierr) > CHKERRQ(ierr) > !******************************************************************************* > > rtol = 1.0d-6 > atol = 1.0d-6 > dtol = 1.0d10 > maxitr = 1000 > call KSPSetTolerances(ksp,rtol,atol,dtol,maxitr,ierr) > CHKERRQ(ierr) > call KSPSetFromOptions(ksp,ierr) > end subroutine petsc_initialize > !------------------------------------------------------------------------------- > > subroutine petsc_set_matrix() > > implicit none > integer :: i,i_elmt,j,ncount > integer :: ggdof(NEDOF),idof(NEDOF),igdof(NEDOF) > > ! Set and assemble matrix. > call MatZeroEntries(A,ierr) > CHKERRQ(ierr) > do i_elmt=1,nelmt > ggdof(:)=ggdof_elmt(:,i_elmt) > ggdof(:)=ggdof(:)-1 ! petsc index starts from 0 > ncount=0; idof=-1; igdof=-1 > do i=1,NEDOF > do j=1,NEDOF > if(ggdof(i).ge.0.and.ggdof(j).ge.0)then > call MatSetValues(A,1,ggdof(i),1,ggdof(j),storekmat(i,j,i_elmt), > & > ADD_VALUES,ierr) > CHKERRQ(ierr) > endif > enddo > enddo > enddo > call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) > call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) > if(myrank==0)print*,'matrix setting & assembly complete!' > > end subroutine petsc_set_matrix > !------------------------------------------------------------------------------- > > subroutine petsc_set_vector() > implicit none > PetscScalar zero > integer :: i,i_elmt,ncount > integer :: ggdof(NEDOF),idof(NEDOF),igdof(NEDOF) > > ! set vector > zero=0.0_8 > call VecSet(b,zero,ierr) > do i_elmt=1,nelmt > ggdof(:)=ggdof_elmt(:,i_elmt) > ggdof(:)=ggdof(:)-1 ! petsc index starts from 0 > ncount=0; idof=-1; igdof=-1 > do i=1,NEDOF > if(ggdof(i).ge.0)then > call VecSetValues(b,1,ggdof(i),storef(i,i_elmt),ADD_VALUES,ierr); > CHKERRQ(ierr) > endif > enddo > enddo > > ! assemble vector > call VecAssemblyBegin(b,ierr) > call VecAssemblyEnd(b,ierr) > if(myrank==0)print*,'vector setting & assembly complete!' > > end subroutine petsc_set_vector > !------------------------------------------------------------------------------- > > subroutine petsc_solve() > implicit none > PetscInt ireason > PetscScalar a_v(1) > PetscOffset a_i > PetscInt n > PetscReal,allocatable :: sdata(:) > > call VecGetSize(b,n,ierr) > CHKERRQ(ierr) > allocate(sdata(n)) > sdata=0.0_8 > call VecGetArray(b,a_v,a_i,ierr) > CHKERRQ(ierr) > sdata(1:n)=a_v(a_i+1:a_i+n) > call VecRestoreArray(b,a_v,a_i,ierr) > CHKERRQ(ierr) > print*,'rhs:',sdata > > call KSPSolve(ksp,b,x,ierr) > sdata=0.0_8 > call VecGetArray(x,a_v,a_i,ierr) > CHKERRQ(ierr) > sdata(1:n)=a_v(a_i+1:a_i+n) > call VecRestoreArray(b,a_v,a_i,ierr) > CHKERRQ(ierr) > print*,'solution:',sdata > > ! Check convergence > call KSPGetConvergedReason(ksp,ireason,ierr) > if(myrank==0)print*,'converges reason',myrank,ireason > call KSPGetIterationNumber(ksp,its,ierr) > if(myrank==0)print*,'Iterations:',its > deallocate(sdata) > end subroutine petsc_solve > !------------------------------------------------------------------------------- > > subroutine petsc_finalize() > implicit none > > ! Free work space. > call VecDestroy(x,ierr) > call VecDestroy(u,ierr) > call VecDestroy(b,ierr) > call MatDestroy(A,ierr) > call KSPDestroy(ksp,ierr) > call PetscFinalize(ierr) > CHKERRQ(ierr) > > end subroutine petsc_finalize > !------------------------------------------------------------------------------- > > end program testfs > > On Tue, Feb 2, 2016 at 4:54 PM, Matthew Knepley wrote: >> On Tue, Feb 2, 2016 at 3:11 PM, Hom Nath Gharti wrote: >>> >>> Thanks a lot. I will try. >> >> >> Also, if you send a small test case, we can run it and tell you the problem. >> >> Matt >> >>> >>> Hom Nath >>> >>> On Tue, Feb 2, 2016 at 4:02 PM, Matthew Knepley wrote: >>>> On Tue, Feb 2, 2016 at 3:01 PM, Hom Nath Gharti >>>> wrote: >>>>> >>>>> Thanks again Matt. Unfortunately got the same errors with '0'. I think >>>>> both are valid in Fortran. >>>> >>>> >>>> Then you will have to go in the debugger and see why its not creating 4 >>>> splits, since this is exactly >>>> what it does in our tests. This is how all the SNES tests that I use >>>> work. I >>>> am sure its a Fortran >>>> problem. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>>> >>>>> Hom Nath >>>>> >>>>> On Tue, Feb 2, 2016 at 3:42 PM, Matthew Knepley >>>>> wrote: >>>>>> On Tue, Feb 2, 2016 at 2:35 PM, Hom Nath Gharti >>>>>> wrote: >>>>>>> >>>>>>> Thank you so much Matt. >>>>>>> >>>>>>> I now tried the following: >>>>>>> >>>>>>> ====================================================== >>>>>>> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >>>>>>> call KSPGetPC(ksp,pc,ierr) >>>>>>> >>>>>>> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); >>>>>>> call ISDestroy(gdofu_is,ierr) >>>>>>> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); >>>>>>> call ISDestroy(gdofchi_is,ierr) >>>>>>> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); >>>>>>> call ISDestroy(gdofp_is,ierr) >>>>>>> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); >>>>>>> call ISDestroy(gdofphi_is,ierr) >>>>>>> >>>>>>> ! Amat changes in each time steps >>>>>>> call KSPSetOperators(ksp,Amat,Amat,ierr) !version >= 3.5.0 >>>>>>> >>>>>>> call KSPSolve(ksp,bvec,xvec,ierr) >>>>>>> ====================================================== >>>>>> >>>>>> >>>>>> I am guessing that "0" is not a valid string for your Fortran >>>>>> compiler. >>>>>> Are >>>>>> you sure >>>>>> its not single quotes '0'? >>>>>> >>>>>> Matt >>>>>> >>>>>>> >>>>>>> But I get the following error: >>>>>>> >>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>> -------------------------------------------------------------- >>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>>>> [0]PETSC ERROR: Unhandled case, must have at least two fields, not 1 >>>>>>> [0]PETSC ERROR: See >>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >>>>>>> shooting. >>>>>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>>>>>> [0]PETSC ERROR: >>>>>>> /tigress/hgharti/gitwork/SPECFEM3D_GLOBEVSPP/./bin/xspecfem3D on a >>>>>>> arch-linux2-c-debug named tiger-r3c1n7 by hgharti Tue Feb 2 15: >>>>>>> 29:30 2016 >>>>>>> [0]PETSC ERROR: Configure options >>>>>>> >>>>>>> >>>>>>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64/ >>>>>>> --with-cc=mpicc --with-cxx=mpicxx --wit >>>>>>> h-fc=mpif90 --with-mpiexec=mpiexec --with-debugging=1 >>>>>>> --download-scalapack --download-mumps --download-pastix >>>>>>> --download-superlu --download-superlu_dist --download-metis >>>>>>> --download-parmetis --download-ptscotch --download-hypre >>>>>>> [0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 469 in >>>>>>> >>>>>>> >>>>>>> >>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >>>>>>> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 486 in >>>>>>> >>>>>>> >>>>>>> >>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >>>>>>> [0]PETSC ERROR: #3 PCSetUp() line 983 in >>>>>>> >>>>>>> >>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/interface/precon.c >>>>>>> [0]PETSC ERROR: #4 KSPSetUp() line 332 in >>>>>>> >>>>>>> >>>>>>> >>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >>>>>>> [0]PETSC ERROR: #5 KSPSolve() line 546 in >>>>>>> >>>>>>> >>>>>>> >>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >>>>>>> forrtl: error (76): Abort trap signal >>>>>>> >>>>>>> Am I missing something? >>>>>>> >>>>>>> Thanks, >>>>>>> Hom Nath >>>>>>> >>>>>>> On Tue, Feb 2, 2016 at 3:24 PM, Matthew Knepley >>>>>>> wrote: >>>>>>>> On Tue, Feb 2, 2016 at 2:20 PM, Hom Nath Gharti >>>>>>>> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hi Matt, hi all, >>>>>>>>> >>>>>>>>> I am trying to use PCFIELDSPLIT for my solver which consists of 4 >>>>>>>>> different variables, namely, u (displacement vector), \chi >>>>>>>>> (displacement potential), p(pressure), and \phi (gravity >>>>>>>>> potential). >>>>>>>>> >>>>>>>>> My code segment looks like the following: >>>>>>>>> ====================================================== >>>>>>>>> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >>>>>>>>> call KSPGetPC(ksp,pc,ierr) >>>>>>>>> nsplit=4 >>>>>>>>> call PCFieldSplitSetBlockSize(pc,nsplit,ierr); >>>>>>>> >>>>>>>> >>>>>>>> You do not need the statement above. >>>>>>>> >>>>>>>>> >>>>>>>>> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); >>>>>>>>> call ISDestroy(gdofu_is,ierr) >>>>>>>>> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); >>>>>>>>> call ISDestroy(gdofchi_is,ierr) >>>>>>>>> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); >>>>>>>>> call ISDestroy(gdofp_is,ierr) >>>>>>>>> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); >>>>>>>>> call ISDestroy(gdofphi_is,ierr) >>>>>>>>> >>>>>>>>> call PCFieldSplitGetSubKSP(pc,nsplit,subksp,ierr); >>>>>>>> >>>>>>>> >>>>>>>> These SetOperators() calls are wrong. I am not sure why you want >>>>>>>> them >>>>>>>> here. >>>>>>>> Also, that means you do not need the call above. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>>> >>>>>>>>> call KSPSetOperators(subksp(1),Amat,Amat,ierr); >>>>>>>>> call KSPSetOperators(subksp(2),Amat,Amat,ierr); >>>>>>>>> call KSPSetOperators(subksp(3),Amat,Amat,ierr); >>>>>>>>> call KSPSetOperators(subksp(4),Amat,Amat,ierr); >>>>>>>>> >>>>>>>>> call KSPSolve(ksp,bvec,xvec,ierr) >>>>>>>>> ====================================================== >>>>>>>>> >>>>>>>>> But I am getting the following error: >>>>>>>>> [79]PETSC ERROR: Null argument, when expecting valid pointer >>>>>>>>> [79]PETSC ERROR: Null Object: Parameter # 1 >>>>>>>>> [79]PETSC ERROR: #1 KSPSetOperators() line 536 in >>>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interf >>>>>>>>> >>>>>>>>> It looks like I am doing something wrong in "call >>>>>>>>> KSPSetOperators" >>>>>>>>> but >>>>>>>>> I could not figure that out. >>>>>>>>> >>>>>>>>> Could anybody help me to fix this problem? I looked into almost >>>>>>>>> all >>>>>>>>> related examples but I could not really figure out the correct >>>>>>>>> steps >>>>>>>>> after "call PCFieldSplitSetIS". >>>>>>>>> >>>>>>>>> Any help will be greatly appreciated. >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Hom nath >>>>>>>>> >>>>>>>>> On Sun, Jan 24, 2016 at 7:14 PM, Hom Nath Gharti >>>>>>>>> >>>>>>>>> wrote: >>>>>>>>>> Thank you so much Matt! I will try. >>>>>>>>>> >>>>>>>>>> Hom Nath >>>>>>>>>> >>>>>>>>>> On Sun, Jan 24, 2016 at 6:26 AM, Matthew Knepley >>>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>>> On Fri, Jan 22, 2016 at 2:19 PM, Hom Nath Gharti >>>>>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Dear all, >>>>>>>>>>>> >>>>>>>>>>>> I am new to PcFieldSplit. >>>>>>>>>>>> >>>>>>>>>>>> I have a matrix formed using MATMPIAIJ. Is it possible to use >>>>>>>>>>>> PCFIELDSPLIT operations in this type of matrix? Or does it >>>>>>>>>>>> have >>>>>>>>>>>> to >>>>>>>>>>>> be >>>>>>>>>>>> MATMPIBIJ or MATNEST format? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Yes, you can split AIJ. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> If possible for MATMPIAIJ, could anybody provide me a simple >>>>>>>>>>>> example >>>>>>>>>>>> or few steps? Variables in the equations are displacement >>>>>>>>>>>> vector, >>>>>>>>>>>> scalar potential and pressure. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> If you do not have a collocated discretization, then you have >>>>>>>>>>> to >>>>>>>>>>> use >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Matt >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Thanks for help. >>>>>>>>>>>> >>>>>>>>>>>> Hom Nath >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>> their >>>>>>>>>>> experiments >>>>>>>>>>> is infinitely more interesting than any results to which their >>>>>>>>>>> experiments >>>>>>>>>>> lead. >>>>>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their >>>>>>>> experiments >>>>>>>> is infinitely more interesting than any results to which their >>>>>>>> experiments >>>>>>>> lead. >>>>>>>> -- Norbert Wiener >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments >>>>>> is infinitely more interesting than any results to which their >>>>>> experiments >>>>>> lead. >>>>>> -- Norbert Wiener >>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments >>>> is infinitely more interesting than any results to which their >>>> experiments >>>> lead. >>>> -- Norbert Wiener >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments >> is infinitely more interesting than any results to which their experiments >> lead. >> -- Norbert Wiener From epscodes at gmail.com Mon Feb 8 16:21:22 2016 From: epscodes at gmail.com (Xiangdong) Date: Mon, 8 Feb 2016 17:21:22 -0500 Subject: [petsc-users] log_summary time ratio and flops ratio In-Reply-To: References: <874mdmeqsu.fsf@jedbrown.org> Message-ID: Based on what you suggested, I have done the following: i) rerun the same problem without output. The ratios are still roughly the same. So it is not the problem of IO. ii) rerun the program on a supercomputer (Stampede), instead of group cluster. the MPI_Barrier time got better: Average time to get PetscTime(): 0 Average time for MPI_Barrier(): 1.27792e-05 Average time for zero size MPI_Send(): 3.94508e-06 the full petsc logsummary is here: https://googledrive.com/host/0BxEfb1tasJxhTjNTVXh4bmJmWlk iii) since the time ratios of VecDot (2.5) and MatMult (1.5) are still high, I rerun the program with ipm module. The IPM summary is here: https://drive.google.com/file/d/0BxEfb1tasJxhYXI0VkV0cjlLWUU/view?usp=sharing. >From this IPM reuslts, MPI_Allreduce takes 74% of MPI time. The communication by task figure (1st figure in p4) in above link showed that it is not well-balanced. Is this related to the hardware and network (which the users cannot control) or can I do something on my codes to improve? Thank you. Best, Xiangdong On Fri, Feb 5, 2016 at 10:34 PM, Barry Smith wrote: > > Make the same run with no IO and see if the numbers are much better and > if the load balance is better. > > > On Feb 5, 2016, at 8:59 PM, Xiangdong wrote: > > > > If I want to know whether only rank 0 is slow (since it may has more io) > or actually a portion of cores are slow, what tools can I start with? > > > > Thanks. > > > > Xiangdong > > > > On Fri, Feb 5, 2016 at 5:27 PM, Jed Brown wrote: > > Matthew Knepley writes: > > >> I attached the full summary. At the end, it has > > >> > > >> Average time to get PetscTime(): 0 > > >> Average time for MPI_Barrier(): 8.3971e-05 > > >> Average time for zero size MPI_Send(): 7.16746e-06 > > >> > > >> Is it an indication of slow network? > > >> > > > > > > I think so. It takes nearly 100 microseconds to synchronize processes. > > > > Edison with 65536 processes: > > Average time for MPI_Barrier(): 4.23908e-05 > > Average time for zero size MPI_Send(): 2.46466e-06 > > > > Mira with 16384 processes: > > Average time for MPI_Barrier(): 5.7075e-06 > > Average time for zero size MPI_Send(): 1.33179e-05 > > > > Titan with 131072 processes: > > Average time for MPI_Barrier(): 0.000368595 > > Average time for zero size MPI_Send(): 1.71567e-05 > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Feb 8 17:19:33 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 8 Feb 2016 17:19:33 -0600 Subject: [petsc-users] log_summary time ratio and flops ratio In-Reply-To: References: <874mdmeqsu.fsf@jedbrown.org> Message-ID: <2AAFE49C-FA38-41BA-BA10-3B10D4DEF4BB@mcs.anl.gov> The following routines are all embarrassingly parallel. VecAXPY 1001160 1.0 2.0483e+01 2.7 1.85e+10 1.1 0.0e+00 0.0e+00 0.0e+00 3 4 0 0 0 3 4 0 0 0 219358 VecAYPX 600696 1.0 6.6270e+00 2.0 1.11e+10 1.1 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 406161 VecAXPBYCZ 194 1.0 4.9155e-03 1.7 7.17e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 354257 VecWAXPY 954 1.0 4.3450e-02 1.5 8.81e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 49269 VecMAXPY 600696 1.0 1.2831e+01 2.0 2.22e+10 1.1 0.0e+00 0.0e+00 0.0e+00 2 4 0 0 0 2 4 0 0 0 420212 Note the ratio of time is between 1.5 and 2.7 this indicates that there is very likely an imbalance in amount of work for different processes. So your load balancing is highly suspicious. Barry > On Feb 8, 2016, at 4:21 PM, Xiangdong wrote: > > Based on what you suggested, I have done the following: > > i) rerun the same problem without output. The ratios are still roughly the same. So it is not the problem of IO. > > ii) rerun the program on a supercomputer (Stampede), instead of group cluster. the MPI_Barrier time got better: > > Average time to get PetscTime(): 0 > Average time for MPI_Barrier(): 1.27792e-05 > Average time for zero size MPI_Send(): 3.94508e-06 > > the full petsc logsummary is here: https://googledrive.com/host/0BxEfb1tasJxhTjNTVXh4bmJmWlk > > iii) since the time ratios of VecDot (2.5) and MatMult (1.5) are still high, I rerun the program with ipm module. The IPM summary is here: https://drive.google.com/file/d/0BxEfb1tasJxhYXI0VkV0cjlLWUU/view?usp=sharing. From this IPM reuslts, MPI_Allreduce takes 74% of MPI time. The communication by task figure (1st figure in p4) in above link showed that it is not well-balanced. Is this related to the hardware and network (which the users cannot control) or can I do something on my codes to improve? > > Thank you. > > Best, > Xiangdong > > On Fri, Feb 5, 2016 at 10:34 PM, Barry Smith wrote: > > Make the same run with no IO and see if the numbers are much better and if the load balance is better. > > > On Feb 5, 2016, at 8:59 PM, Xiangdong wrote: > > > > If I want to know whether only rank 0 is slow (since it may has more io) or actually a portion of cores are slow, what tools can I start with? > > > > Thanks. > > > > Xiangdong > > > > On Fri, Feb 5, 2016 at 5:27 PM, Jed Brown wrote: > > Matthew Knepley writes: > > >> I attached the full summary. At the end, it has > > >> > > >> Average time to get PetscTime(): 0 > > >> Average time for MPI_Barrier(): 8.3971e-05 > > >> Average time for zero size MPI_Send(): 7.16746e-06 > > >> > > >> Is it an indication of slow network? > > >> > > > > > > I think so. It takes nearly 100 microseconds to synchronize processes. > > > > Edison with 65536 processes: > > Average time for MPI_Barrier(): 4.23908e-05 > > Average time for zero size MPI_Send(): 2.46466e-06 > > > > Mira with 16384 processes: > > Average time for MPI_Barrier(): 5.7075e-06 > > Average time for zero size MPI_Send(): 1.33179e-05 > > > > Titan with 131072 processes: > > Average time for MPI_Barrier(): 0.000368595 > > Average time for zero size MPI_Send(): 1.71567e-05 > > > > From jed at jedbrown.org Mon Feb 8 17:45:29 2016 From: jed at jedbrown.org (Jed Brown) Date: Mon, 08 Feb 2016 16:45:29 -0700 Subject: [petsc-users] log_summary time ratio and flops ratio In-Reply-To: References: <874mdmeqsu.fsf@jedbrown.org> Message-ID: <87h9hiahra.fsf@jedbrown.org> Xiangdong writes: > iii) since the time ratios of VecDot (2.5) and MatMult (1.5) are still > high, I rerun the program with ipm module. The IPM summary is here: > https://drive.google.com/file/d/0BxEfb1tasJxhYXI0VkV0cjlLWUU/view?usp=sharing. > From this IPM reuslts, MPI_Allreduce takes 74% of MPI time. The > communication by task figure (1st figure in p4) in above link showed that > it is not well-balanced. Is this related to the hardware and network (which > the users cannot control) or can I do something on my codes to improve? Here are a few functions that don't have any communication, but still have significant load imbalance. VecAXPY 1021815 1.0 2.2148e+01 2.1 1.89e+10 1.1 0.0e+00 0.0e+00 0.0e+00 2 4 0 0 0 2 4 0 0 0 207057 VecMAXPY 613089 1.0 1.3276e+01 2.2 2.27e+10 1.1 0.0e+00 0.0e+00 0.0e+00 1 4 0 0 0 1 4 0 0 0 414499 MatSOR 818390 1.0 1.9608e+02 1.5 2.00e+11 1.1 0.0e+00 0.0e+00 0.0e+00 22 40 0 0 0 22 40 0 0 0 247472 You can and should improve load balance before stressing about network costs. This could be that the nodes aren't clean (running at different speeds) or that the partition is not balancing data. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From epscodes at gmail.com Mon Feb 8 21:30:45 2016 From: epscodes at gmail.com (Xiangdong) Date: Mon, 8 Feb 2016 22:30:45 -0500 Subject: [petsc-users] log_summary time ratio and flops ratio In-Reply-To: <87h9hiahra.fsf@jedbrown.org> References: <874mdmeqsu.fsf@jedbrown.org> <87h9hiahra.fsf@jedbrown.org> Message-ID: On Mon, Feb 8, 2016 at 6:45 PM, Jed Brown wrote: > Xiangdong writes: > > > iii) since the time ratios of VecDot (2.5) and MatMult (1.5) are still > > high, I rerun the program with ipm module. The IPM summary is here: > > > https://drive.google.com/file/d/0BxEfb1tasJxhYXI0VkV0cjlLWUU/view?usp=sharing > . > > From this IPM reuslts, MPI_Allreduce takes 74% of MPI time. The > > communication by task figure (1st figure in p4) in above link showed that > > it is not well-balanced. Is this related to the hardware and network > (which > > the users cannot control) or can I do something on my codes to improve? > > Here are a few functions that don't have any communication, but still > have significant load imbalance. > > VecAXPY 1021815 1.0 2.2148e+01 2.1 1.89e+10 1.1 0.0e+00 0.0e+00 > 0.0e+00 2 4 0 0 0 2 4 0 0 0 207057 > VecMAXPY 613089 1.0 1.3276e+01 2.2 2.27e+10 1.1 0.0e+00 0.0e+00 > 0.0e+00 1 4 0 0 0 1 4 0 0 0 414499 > MatSOR 818390 1.0 1.9608e+02 1.5 2.00e+11 1.1 0.0e+00 0.0e+00 > 0.0e+00 22 40 0 0 0 22 40 0 0 0 247472 > > For these functions, the flop ratios are all 1.1, while the time ratio are 1.5-2.2. So the amount of work are sort of balanced for each processes. Both runs on Stampede and my group cluster gave similar behaviors. Given that I only use 256 cores, do you think it is likely that my job was assigned cores with different speeds? How can I test/measure this since each time the job was assigned to different nodes? Are there any other factors I should also look into for the behavior that flops ratio 1.1 but time ratio 1.5-2.1 for non-communicating functions? > You can and should improve load balance before stressing about network > costs. This could be that the nodes aren't clean (running at different > speeds) or that the partition is not balancing data. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Mon Feb 8 21:41:41 2016 From: jed at jedbrown.org (Jed Brown) Date: Mon, 08 Feb 2016 20:41:41 -0700 Subject: [petsc-users] log_summary time ratio and flops ratio In-Reply-To: References: <874mdmeqsu.fsf@jedbrown.org> <87h9hiahra.fsf@jedbrown.org> Message-ID: <871t8ma6tm.fsf@jedbrown.org> Xiangdong writes: > For these functions, the flop ratios are all 1.1, while the time ratio are > 1.5-2.2. So the amount of work are sort of balanced for each processes. > Both runs on Stampede and my group cluster gave similar behaviors. Given > that I only use 256 cores, do you think it is likely that my job was > assigned cores with different speeds? How can I test/measure this since > each time the job was assigned to different nodes? > > Are there any other factors I should also look into for the behavior that > flops ratio 1.1 but time ratio 1.5-2.1 for non-communicating functions? Memory bandwidth can be an issue. Like some nodes could have slower memory installed. Or, like happened to Dave and me at ETH, an old, lopsided ramdisk partition could have been left behind by a previous job, causing slow bandwidth after all the memory ends up faulted onto a single memory channel. You can investigate such issues with numastat and third-party profilers. I would start by seeing if you can reproduce with simpler PETSc examples, then distinguish performance of a flops-limited local operation versus bandwidth-limited operation. It might be simple to figure out, but it might also take a lot of work. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From mailinglists at xgm.de Tue Feb 9 07:06:01 2016 From: mailinglists at xgm.de (Florian Lindner) Date: Tue, 9 Feb 2016 14:06:01 +0100 Subject: [petsc-users] Different results with different distribution of rows Message-ID: <20160209140601.62db2162@asaru> Hello, I use PETSc with 4 MPI processes and I experience different results when using different distribution of rows amoung ranks. The code looks like that: KSPSetOperators(_solver, _matrixC.matrix, _matrixC.matrix); // _solverRtol = 1e-9 KSPSetTolerances(_solver, _solverRtol, PETSC_DEFAULT, PETSC_DEFAULT, PETSC_DEFAULT); KSPSetFromOptions(_solver); // means: MatGetVecs(matrix, nullptr, &vector); petsc::Vector Au(_matrixC, "Au"); petsc::Vector out(_matrixC, "out"); petsc::Vector in(_matrixA, "in"); // is an identity mapping here ierr = VecSetLocalToGlobalMapping(in.vector, _ISmapping); // fill and assemble vector in in.view(); MatMultTranspose(_matrixA.matrix, in.vector, Au.vector); Au.view(); KSPSolve(_solver, Au.vector, out.vector); out.view(); I have experimented with two variants. The first one, non-working has MatrixC rows distributed like that: 3, 4, 2, 2. The other, working, one has 4, 3, 2, 2. All input values: _matrixA, _matrixC, in are identival, Au is identical too, but out differs. See the results from object::view below. Could the bad condition of matrixC be a problem? I'm not using any special KSP options. matrixC is of type MATSBAIJ, matrixA is MATAIJ. Thanks, Florian WRONG ================= Vec Object:in 4 MPI processes type: mpi Process [0] 1 2 Process [1] 3 4 Process [2] 5 6 Process [3] 7 8 Vec Object:Au 4 MPI processes type: mpi Process [0] 36 74 20 Process [1] 1.09292 2.09259 3.18584 4.20349 Process [2] 5.29708 6.31472 Process [3] 7.24012 8.23978 // should not be result Vec Object:out 4 MPI processes type: mpi Process [0] -1.10633e+07 618058 9.01497e+06 Process [1] 0.996195 1.98711 3.01111 4.00203 Process [2] 5.00736 6.01644 Process [3] 6.98534 7.99442 Mat Object:C 4 MPI processes type: mpisbaij row 0: (0, 0) (3, 1) (4, 1) (5, 1) (6, 1) (7, 1) (8, 1) (9, 1) (10, 1) row 1: (1, 0) (3, 0) (4, 0) (5, 1) (6, 1) (7, 2) (8, 2) (9, 3) (10, 3) row 2: (2, 0) (3, 0) (4, 1) (5, 0) (6, 1) (7, 0) (8, 1) (9, 0) (10, 1) row 3: (3, 1) (4, 0.0183156) (5, 0.0183156) (6, 0.000335463) (7, 1.12535e-07) (8, 2.06115e-09) row 4: (4, 1) (5, 0.000335463) (6, 0.0183156) (7, 2.06115e-09) (8, 1.12535e-07) row 5: (5, 1) (6, 0.0183156) (7, 0.0183156) (8, 0.000335463) (9, 1.12535e-07) (10, 2.06115e-09) row 6: (6, 1) (7, 0.000335463) (8, 0.0183156) (9, 2.06115e-09) (10, 1.12535e-07) row 7: (7, 1) (8, 0.0183156) (9, 0.0183156) (10, 0.000335463) row 8: (8, 1) (9, 0.000335463) (10, 0.0183156) row 9: (9, 1) (10, 0.0183156) row 10: (10, 1) Mat Object:A 4 MPI processes type: mpiaij 1.00000e+00 0.00000e+00 0.00000e+00 1.00000e+00 1.83156e-02 1.83156e-02 3.35463e-04 1.12535e-07 2.06115e-09 0.00000e+00 0.00000e+00 1.00000e+00 0.00000e+00 1.00000e+00 1.83156e-02 1.00000e+00 3.35463e-04 1.83156e-02 2.06115e-09 1.12535e-07 0.00000e+00 0.00000e+00 1.00000e+00 1.00000e+00 0.00000e+00 1.83156e-02 3.35463e-04 1.00000e+00 1.83156e-02 1.83156e-02 3.35463e-04 1.12535e-07 2.06115e-09 1.00000e+00 1.00000e+00 1.00000e+00 3.35463e-04 1.83156e-02 1.83156e-02 1.00000e+00 3.35463e-04 1.83156e-02 2.06115e-09 1.12535e-07 1.00000e+00 2.00000e+00 0.00000e+00 1.12535e-07 2.06115e-09 1.83156e-02 3.35463e-04 1.00000e+00 1.83156e-02 1.83156e-02 3.35463e-04 1.00000e+00 2.00000e+00 1.00000e+00 2.06115e-09 1.12535e-07 3.35463e-04 1.83156e-02 1.83156e-02 1.00000e+00 3.35463e-04 1.83156e-02 1.00000e+00 3.00000e+00 0.00000e+00 0.00000e+00 0.00000e+00 1.12535e-07 2.06115e-09 1.83156e-02 3.35463e-04 1.00000e+00 1.83156e-02 1.00000e+00 3.00000e+00 1.00000e+00 0.00000e+00 0.00000e+00 2.06115e-09 1.12535e-07 3.35463e-04 1.83156e-02 1.83156e-02 1.00000e+00 RIGHT ================= Vec Object:in 4 MPI processes type: mpi Process [0] 1 2 Process [1] 3 4 Process [2] 5 6 Process [3] 7 8 Vec Object:Au 4 MPI processes type: mpi Process [0] 36 74 20 1.09292 Process [1] 2.09259 3.18584 4.20349 Process [2] 5.29708 6.31472 Process [3] 7.24012 8.23978 // should be result Vec Object:out 4 MPI processes type: mpi Process [0] 0 0 0 1 Process [1] 2 3 4 Process [2] 5 6 Process [3] 7 8 Mat Object:C 4 MPI processes type: mpisbaij row 0: (0, 0) (3, 1) (4, 1) (5, 1) (6, 1) (7, 1) (8, 1) (9, 1) (10, 1) row 1: (1, 0) (3, 0) (4, 0) (5, 1) (6, 1) (7, 2) (8, 2) (9, 3) (10, 3) row 2: (2, 0) (3, 0) (4, 1) (5, 0) (6, 1) (7, 0) (8, 1) (9, 0) (10, 1) row 3: (3, 1) (4, 0.0183156) (5, 0.0183156) (6, 0.000335463) (7, 1.12535e-07) (8, 2.06115e-09) row 4: (4, 1) (5, 0.000335463) (6, 0.0183156) (7, 2.06115e-09) (8, 1.12535e-07) row 5: (5, 1) (6, 0.0183156) (7, 0.0183156) (8, 0.000335463) (9, 1.12535e-07) (10, 2.06115e-09) row 6: (6, 1) (7, 0.000335463) (8, 0.0183156) (9, 2.06115e-09) (10, 1.12535e-07) row 7: (7, 1) (8, 0.0183156) (9, 0.0183156) (10, 0.000335463) row 8: (8, 1) (9, 0.000335463) (10, 0.0183156) row 9: (9, 1) (10, 0.0183156) row 10: (10, 1) Mat Object:A 4 MPI processes type: mpiaij 1.00000e+00 0.00000e+00 0.00000e+00 1.00000e+00 1.83156e-02 1.83156e-02 3.35463e-04 1.12535e-07 2.06115e-09 0.00000e+00 0.00000e+00 1.00000e+00 0.00000e+00 1.00000e+00 1.83156e-02 1.00000e+00 3.35463e-04 1.83156e-02 2.06115e-09 1.12535e-07 0.00000e+00 0.00000e+00 1.00000e+00 1.00000e+00 0.00000e+00 1.83156e-02 3.35463e-04 1.00000e+00 1.83156e-02 1.83156e-02 3.35463e-04 1.12535e-07 2.06115e-09 1.00000e+00 1.00000e+00 1.00000e+00 3.35463e-04 1.83156e-02 1.83156e-02 1.00000e+00 3.35463e-04 1.83156e-02 2.06115e-09 1.12535e-07 1.00000e+00 2.00000e+00 0.00000e+00 1.12535e-07 2.06115e-09 1.83156e-02 3.35463e-04 1.00000e+00 1.83156e-02 1.83156e-02 3.35463e-04 1.00000e+00 2.00000e+00 1.00000e+00 2.06115e-09 1.12535e-07 3.35463e-04 1.83156e-02 1.83156e-02 1.00000e+00 3.35463e-04 1.83156e-02 1.00000e+00 3.00000e+00 0.00000e+00 0.00000e+00 0.00000e+00 1.12535e-07 2.06115e-09 1.83156e-02 3.35463e-04 1.00000e+00 1.83156e-02 1.00000e+00 3.00000e+00 1.00000e+00 0.00000e+00 0.00000e+00 2.06115e-09 1.12535e-07 3.35463e-04 1.83156e-02 1.83156e-02 1.00000e+00 From mailinglists at xgm.de Tue Feb 9 07:39:10 2016 From: mailinglists at xgm.de (Florian Lindner) Date: Tue, 9 Feb 2016 14:39:10 +0100 Subject: [petsc-users] Different results with different distribution of rows In-Reply-To: <20160209140601.62db2162@asaru> References: <20160209140601.62db2162@asaru> Message-ID: <20160209143910.1684381c@asaru> Addition. The KSP Solver shows very different convergence: WRONG: [0] KSPConvergedDefault(): Linear solver has converged. Residual norm 6.832362172732e+06 is less than relative tolerance 1.000000000000e-09 times initial right hand side norm 6.934533099989e+15 at iteration 8447 RIGHT: [0] KSPConvergedDefault(): Linear solver has converged. Residual norm 7.959757133341e-08 is less than relative tolerance 1.000000000000e-09 times initial right hand side norm 1.731788191624e+02 at iteration 9 Best, Florian On Tue, 9 Feb 2016 14:06:01 +0100 Florian Lindner wrote: > Hello, > > I use PETSc with 4 MPI processes and I experience different results > when using different distribution of rows amoung ranks. The code looks > like that: > > > KSPSetOperators(_solver, _matrixC.matrix, _matrixC.matrix); > // _solverRtol = 1e-9 > KSPSetTolerances(_solver, _solverRtol, PETSC_DEFAULT, PETSC_DEFAULT, > PETSC_DEFAULT); > KSPSetFromOptions(_solver); > > // means: MatGetVecs(matrix, nullptr, &vector); > petsc::Vector Au(_matrixC, "Au"); > petsc::Vector out(_matrixC, "out"); > petsc::Vector in(_matrixA, "in"); > > // is an identity mapping here > ierr = VecSetLocalToGlobalMapping(in.vector, _ISmapping); > > // fill and assemble vector in > > in.view(); > MatMultTranspose(_matrixA.matrix, in.vector, Au.vector); > Au.view(); > KSPSolve(_solver, Au.vector, out.vector); > > out.view(); > > I have experimented with two variants. The first one, non-working has > MatrixC rows distributed like that: 3, 4, 2, 2. The other, working, > one has 4, 3, 2, 2. > > All input values: _matrixA, _matrixC, in are identival, Au is > identical too, but out differs. See the results from object::view > below. > > Could the bad condition of matrixC be a problem? I'm not using any > special KSP options. matrixC is of type MATSBAIJ, matrixA is MATAIJ. > > Thanks, > Florian > > WRONG > ================= > > Vec Object:in 4 MPI processes > type: mpi > Process [0] > 1 > 2 > Process [1] > 3 > 4 > Process [2] > 5 > 6 > Process [3] > 7 > 8 > > Vec Object:Au 4 MPI processes > type: mpi > Process [0] > 36 > 74 > 20 > Process [1] > 1.09292 > 2.09259 > 3.18584 > 4.20349 > Process [2] > 5.29708 > 6.31472 > Process [3] > 7.24012 > 8.23978 > > // should not be result > Vec Object:out 4 MPI processes > type: mpi > Process [0] > -1.10633e+07 > 618058 > 9.01497e+06 > Process [1] > 0.996195 > 1.98711 > 3.01111 > 4.00203 > Process [2] > 5.00736 > 6.01644 > Process [3] > 6.98534 > 7.99442 > > Mat Object:C 4 MPI processes > type: mpisbaij > row 0: (0, 0) (3, 1) (4, 1) (5, 1) (6, 1) (7, 1) (8, 1) (9, > 1) (10, 1) row 1: (1, 0) (3, 0) (4, 0) (5, 1) (6, 1) (7, 2) > (8, 2) (9, 3) (10, 3) row 2: (2, 0) (3, 0) (4, 1) (5, 0) (6, > 1) (7, 0) (8, 1) (9, 0) (10, 1) row 3: (3, 1) (4, 0.0183156) > (5, 0.0183156) (6, 0.000335463) (7, 1.12535e-07) (8, 2.06115e-09) > row 4: (4, 1) (5, 0.000335463) (6, 0.0183156) (7, 2.06115e-09) > (8, 1.12535e-07) row 5: (5, 1) (6, 0.0183156) (7, 0.0183156) (8, > 0.000335463) (9, 1.12535e-07) (10, 2.06115e-09) row 6: (6, 1) (7, > 0.000335463) (8, 0.0183156) (9, 2.06115e-09) (10, 1.12535e-07) row > 7: (7, 1) (8, 0.0183156) (9, 0.0183156) (10, 0.000335463) row 8: > (8, 1) (9, 0.000335463) (10, 0.0183156) row 9: (9, 1) (10, > 0.0183156) row 10: (10, 1) > > Mat Object:A 4 MPI processes > type: mpiaij > 1.00000e+00 0.00000e+00 0.00000e+00 1.00000e+00 1.83156e-02 > 1.83156e-02 3.35463e-04 1.12535e-07 2.06115e-09 0.00000e+00 > 0.00000e+00 1.00000e+00 0.00000e+00 1.00000e+00 1.83156e-02 > 1.00000e+00 3.35463e-04 1.83156e-02 2.06115e-09 1.12535e-07 > 0.00000e+00 0.00000e+00 1.00000e+00 1.00000e+00 0.00000e+00 > 1.83156e-02 3.35463e-04 1.00000e+00 1.83156e-02 1.83156e-02 > 3.35463e-04 1.12535e-07 2.06115e-09 1.00000e+00 1.00000e+00 > 1.00000e+00 3.35463e-04 1.83156e-02 1.83156e-02 1.00000e+00 > 3.35463e-04 1.83156e-02 2.06115e-09 1.12535e-07 1.00000e+00 > 2.00000e+00 0.00000e+00 1.12535e-07 2.06115e-09 1.83156e-02 > 3.35463e-04 1.00000e+00 1.83156e-02 1.83156e-02 3.35463e-04 > 1.00000e+00 2.00000e+00 1.00000e+00 2.06115e-09 1.12535e-07 > 3.35463e-04 1.83156e-02 1.83156e-02 1.00000e+00 3.35463e-04 > 1.83156e-02 1.00000e+00 3.00000e+00 0.00000e+00 0.00000e+00 > 0.00000e+00 1.12535e-07 2.06115e-09 1.83156e-02 3.35463e-04 > 1.00000e+00 1.83156e-02 1.00000e+00 3.00000e+00 1.00000e+00 > 0.00000e+00 0.00000e+00 2.06115e-09 1.12535e-07 3.35463e-04 > 1.83156e-02 1.83156e-02 1.00000e+00 > > > > RIGHT > ================= > > Vec Object:in 4 MPI processes > type: mpi > Process [0] > 1 > 2 > Process [1] > 3 > 4 > Process [2] > 5 > 6 > Process [3] > 7 > 8 > > Vec Object:Au 4 MPI processes > type: mpi > Process [0] > 36 > 74 > 20 > 1.09292 > Process [1] > 2.09259 > 3.18584 > 4.20349 > Process [2] > 5.29708 > 6.31472 > Process [3] > 7.24012 > 8.23978 > > // should be result > Vec Object:out 4 MPI processes > type: mpi > Process [0] > 0 > 0 > 0 > 1 > Process [1] > 2 > 3 > 4 > Process [2] > 5 > 6 > Process [3] > 7 > 8 > > > Mat Object:C 4 MPI processes > type: mpisbaij > row 0: (0, 0) (3, 1) (4, 1) (5, 1) (6, 1) (7, 1) (8, 1) (9, > 1) (10, 1) row 1: (1, 0) (3, 0) (4, 0) (5, 1) (6, 1) (7, 2) > (8, 2) (9, 3) (10, 3) row 2: (2, 0) (3, 0) (4, 1) (5, 0) (6, > 1) (7, 0) (8, 1) (9, 0) (10, 1) row 3: (3, 1) (4, 0.0183156) > (5, 0.0183156) (6, 0.000335463) (7, 1.12535e-07) (8, 2.06115e-09) > row 4: (4, 1) (5, 0.000335463) (6, 0.0183156) (7, 2.06115e-09) > (8, 1.12535e-07) row 5: (5, 1) (6, 0.0183156) (7, 0.0183156) (8, > 0.000335463) (9, 1.12535e-07) (10, 2.06115e-09) row 6: (6, 1) (7, > 0.000335463) (8, 0.0183156) (9, 2.06115e-09) (10, 1.12535e-07) row > 7: (7, 1) (8, 0.0183156) (9, 0.0183156) (10, 0.000335463) row 8: > (8, 1) (9, 0.000335463) (10, 0.0183156) row 9: (9, 1) (10, > 0.0183156) row 10: (10, 1) > > Mat Object:A 4 MPI processes > type: mpiaij > 1.00000e+00 0.00000e+00 0.00000e+00 1.00000e+00 1.83156e-02 > 1.83156e-02 3.35463e-04 1.12535e-07 2.06115e-09 0.00000e+00 > 0.00000e+00 1.00000e+00 0.00000e+00 1.00000e+00 1.83156e-02 > 1.00000e+00 3.35463e-04 1.83156e-02 2.06115e-09 1.12535e-07 > 0.00000e+00 0.00000e+00 1.00000e+00 1.00000e+00 0.00000e+00 > 1.83156e-02 3.35463e-04 1.00000e+00 1.83156e-02 1.83156e-02 > 3.35463e-04 1.12535e-07 2.06115e-09 1.00000e+00 1.00000e+00 > 1.00000e+00 3.35463e-04 1.83156e-02 1.83156e-02 1.00000e+00 > 3.35463e-04 1.83156e-02 2.06115e-09 1.12535e-07 1.00000e+00 > 2.00000e+00 0.00000e+00 1.12535e-07 2.06115e-09 1.83156e-02 > 3.35463e-04 1.00000e+00 1.83156e-02 1.83156e-02 3.35463e-04 > 1.00000e+00 2.00000e+00 1.00000e+00 2.06115e-09 1.12535e-07 > 3.35463e-04 1.83156e-02 1.83156e-02 1.00000e+00 3.35463e-04 > 1.83156e-02 1.00000e+00 3.00000e+00 0.00000e+00 0.00000e+00 > 0.00000e+00 1.12535e-07 2.06115e-09 1.83156e-02 3.35463e-04 > 1.00000e+00 1.83156e-02 1.00000e+00 3.00000e+00 1.00000e+00 > 0.00000e+00 0.00000e+00 2.06115e-09 1.12535e-07 3.35463e-04 > 1.83156e-02 1.83156e-02 1.00000e+00 From knepley at gmail.com Tue Feb 9 08:00:54 2016 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 9 Feb 2016 08:00:54 -0600 Subject: [petsc-users] Different results with different distribution of rows In-Reply-To: <20160209140601.62db2162@asaru> References: <20160209140601.62db2162@asaru> Message-ID: On Tue, Feb 9, 2016 at 7:06 AM, Florian Lindner wrote: > Hello, > > I use PETSc with 4 MPI processes and I experience different results > when using different distribution of rows amoung ranks. The code looks > like that: > The default PC is BJacobi/ILU. This depends on the parallel layout because each subdomain block is approximately factored. ILU is not very reliable, so it can fail as you see below. If you change to LU, you will see the problem -sub_pc_type lu Thanks, Matt > KSPSetOperators(_solver, _matrixC.matrix, _matrixC.matrix); > // _solverRtol = 1e-9 > KSPSetTolerances(_solver, _solverRtol, PETSC_DEFAULT, PETSC_DEFAULT, > PETSC_DEFAULT); > KSPSetFromOptions(_solver); > > // means: MatGetVecs(matrix, nullptr, &vector); > petsc::Vector Au(_matrixC, "Au"); > petsc::Vector out(_matrixC, "out"); > petsc::Vector in(_matrixA, "in"); > > // is an identity mapping here > ierr = VecSetLocalToGlobalMapping(in.vector, _ISmapping); > > // fill and assemble vector in > > in.view(); > MatMultTranspose(_matrixA.matrix, in.vector, Au.vector); > Au.view(); > KSPSolve(_solver, Au.vector, out.vector); > > out.view(); > > I have experimented with two variants. The first one, non-working has > MatrixC rows distributed like that: 3, 4, 2, 2. The other, working, one > has 4, 3, 2, 2. > > All input values: _matrixA, _matrixC, in are identival, Au is identical > too, but out differs. See the results from object::view below. > > Could the bad condition of matrixC be a problem? I'm not using any > special KSP options. matrixC is of type MATSBAIJ, matrixA is MATAIJ. > > Thanks, > Florian > > WRONG > ================= > > Vec Object:in 4 MPI processes > type: mpi > Process [0] > 1 > 2 > Process [1] > 3 > 4 > Process [2] > 5 > 6 > Process [3] > 7 > 8 > > Vec Object:Au 4 MPI processes > type: mpi > Process [0] > 36 > 74 > 20 > Process [1] > 1.09292 > 2.09259 > 3.18584 > 4.20349 > Process [2] > 5.29708 > 6.31472 > Process [3] > 7.24012 > 8.23978 > > // should not be result > Vec Object:out 4 MPI processes > type: mpi > Process [0] > -1.10633e+07 > 618058 > 9.01497e+06 > Process [1] > 0.996195 > 1.98711 > 3.01111 > 4.00203 > Process [2] > 5.00736 > 6.01644 > Process [3] > 6.98534 > 7.99442 > > Mat Object:C 4 MPI processes > type: mpisbaij > row 0: (0, 0) (3, 1) (4, 1) (5, 1) (6, 1) (7, 1) (8, 1) (9, 1) > (10, 1) > row 1: (1, 0) (3, 0) (4, 0) (5, 1) (6, 1) (7, 2) (8, 2) (9, 3) > (10, 3) > row 2: (2, 0) (3, 0) (4, 1) (5, 0) (6, 1) (7, 0) (8, 1) (9, 0) > (10, 1) > row 3: (3, 1) (4, 0.0183156) (5, 0.0183156) (6, 0.000335463) (7, > 1.12535e-07) (8, 2.06115e-09) > row 4: (4, 1) (5, 0.000335463) (6, 0.0183156) (7, 2.06115e-09) (8, > 1.12535e-07) > row 5: (5, 1) (6, 0.0183156) (7, 0.0183156) (8, 0.000335463) (9, > 1.12535e-07) (10, 2.06115e-09) > row 6: (6, 1) (7, 0.000335463) (8, 0.0183156) (9, 2.06115e-09) (10, > 1.12535e-07) > row 7: (7, 1) (8, 0.0183156) (9, 0.0183156) (10, 0.000335463) > row 8: (8, 1) (9, 0.000335463) (10, 0.0183156) > row 9: (9, 1) (10, 0.0183156) > row 10: (10, 1) > > Mat Object:A 4 MPI processes > type: mpiaij > 1.00000e+00 0.00000e+00 0.00000e+00 1.00000e+00 1.83156e-02 > 1.83156e-02 3.35463e-04 1.12535e-07 2.06115e-09 0.00000e+00 0.00000e+00 > 1.00000e+00 0.00000e+00 1.00000e+00 1.83156e-02 1.00000e+00 > 3.35463e-04 1.83156e-02 2.06115e-09 1.12535e-07 0.00000e+00 0.00000e+00 > 1.00000e+00 1.00000e+00 0.00000e+00 1.83156e-02 3.35463e-04 > 1.00000e+00 1.83156e-02 1.83156e-02 3.35463e-04 1.12535e-07 2.06115e-09 > 1.00000e+00 1.00000e+00 1.00000e+00 3.35463e-04 1.83156e-02 > 1.83156e-02 1.00000e+00 3.35463e-04 1.83156e-02 2.06115e-09 1.12535e-07 > 1.00000e+00 2.00000e+00 0.00000e+00 1.12535e-07 2.06115e-09 > 1.83156e-02 3.35463e-04 1.00000e+00 1.83156e-02 1.83156e-02 3.35463e-04 > 1.00000e+00 2.00000e+00 1.00000e+00 2.06115e-09 1.12535e-07 > 3.35463e-04 1.83156e-02 1.83156e-02 1.00000e+00 3.35463e-04 1.83156e-02 > 1.00000e+00 3.00000e+00 0.00000e+00 0.00000e+00 0.00000e+00 > 1.12535e-07 2.06115e-09 1.83156e-02 3.35463e-04 1.00000e+00 1.83156e-02 > 1.00000e+00 3.00000e+00 1.00000e+00 0.00000e+00 0.00000e+00 > 2.06115e-09 1.12535e-07 3.35463e-04 1.83156e-02 1.83156e-02 1.00000e+00 > > > > RIGHT > ================= > > Vec Object:in 4 MPI processes > type: mpi > Process [0] > 1 > 2 > Process [1] > 3 > 4 > Process [2] > 5 > 6 > Process [3] > 7 > 8 > > Vec Object:Au 4 MPI processes > type: mpi > Process [0] > 36 > 74 > 20 > 1.09292 > Process [1] > 2.09259 > 3.18584 > 4.20349 > Process [2] > 5.29708 > 6.31472 > Process [3] > 7.24012 > 8.23978 > > // should be result > Vec Object:out 4 MPI processes > type: mpi > Process [0] > 0 > 0 > 0 > 1 > Process [1] > 2 > 3 > 4 > Process [2] > 5 > 6 > Process [3] > 7 > 8 > > > Mat Object:C 4 MPI processes > type: mpisbaij > row 0: (0, 0) (3, 1) (4, 1) (5, 1) (6, 1) (7, 1) (8, 1) (9, 1) > (10, 1) > row 1: (1, 0) (3, 0) (4, 0) (5, 1) (6, 1) (7, 2) (8, 2) (9, 3) > (10, 3) > row 2: (2, 0) (3, 0) (4, 1) (5, 0) (6, 1) (7, 0) (8, 1) (9, 0) > (10, 1) > row 3: (3, 1) (4, 0.0183156) (5, 0.0183156) (6, 0.000335463) (7, > 1.12535e-07) (8, 2.06115e-09) > row 4: (4, 1) (5, 0.000335463) (6, 0.0183156) (7, 2.06115e-09) (8, > 1.12535e-07) > row 5: (5, 1) (6, 0.0183156) (7, 0.0183156) (8, 0.000335463) (9, > 1.12535e-07) (10, 2.06115e-09) > row 6: (6, 1) (7, 0.000335463) (8, 0.0183156) (9, 2.06115e-09) (10, > 1.12535e-07) > row 7: (7, 1) (8, 0.0183156) (9, 0.0183156) (10, 0.000335463) > row 8: (8, 1) (9, 0.000335463) (10, 0.0183156) > row 9: (9, 1) (10, 0.0183156) > row 10: (10, 1) > > Mat Object:A 4 MPI processes > type: mpiaij > 1.00000e+00 0.00000e+00 0.00000e+00 1.00000e+00 1.83156e-02 > 1.83156e-02 3.35463e-04 1.12535e-07 2.06115e-09 0.00000e+00 0.00000e+00 > 1.00000e+00 0.00000e+00 1.00000e+00 1.83156e-02 1.00000e+00 > 3.35463e-04 1.83156e-02 2.06115e-09 1.12535e-07 0.00000e+00 0.00000e+00 > 1.00000e+00 1.00000e+00 0.00000e+00 1.83156e-02 3.35463e-04 > 1.00000e+00 1.83156e-02 1.83156e-02 3.35463e-04 1.12535e-07 2.06115e-09 > 1.00000e+00 1.00000e+00 1.00000e+00 3.35463e-04 1.83156e-02 > 1.83156e-02 1.00000e+00 3.35463e-04 1.83156e-02 2.06115e-09 1.12535e-07 > 1.00000e+00 2.00000e+00 0.00000e+00 1.12535e-07 2.06115e-09 > 1.83156e-02 3.35463e-04 1.00000e+00 1.83156e-02 1.83156e-02 3.35463e-04 > 1.00000e+00 2.00000e+00 1.00000e+00 2.06115e-09 1.12535e-07 > 3.35463e-04 1.83156e-02 1.83156e-02 1.00000e+00 3.35463e-04 1.83156e-02 > 1.00000e+00 3.00000e+00 0.00000e+00 0.00000e+00 0.00000e+00 > 1.12535e-07 2.06115e-09 1.83156e-02 3.35463e-04 1.00000e+00 1.83156e-02 > 1.00000e+00 3.00000e+00 1.00000e+00 0.00000e+00 0.00000e+00 > 2.06115e-09 1.12535e-07 3.35463e-04 1.83156e-02 1.83156e-02 1.00000e+00 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Tue Feb 9 08:06:32 2016 From: dave.mayhem23 at gmail.com (Dave May) Date: Tue, 9 Feb 2016 15:06:32 +0100 Subject: [petsc-users] Different results with different distribution of rows In-Reply-To: <20160209143910.1684381c@asaru> References: <20160209140601.62db2162@asaru> <20160209143910.1684381c@asaru> Message-ID: If you don't specify preconditioner via -pc_type XXX, the default being used is BJacobi-ILU. This preconditioner will yield different results on different numbers of MPI-processes, and will yield different results for a fixed number of different MPI-processes, but with a different matrix partitioning. If you operator is singular (or close to singular), ILU is likely to fail. To be sure your code is working correctly, test it using a preconditioner which isn't dependent on the partitioning of the matrix. I would use these options: -pc_type jacobi -ksp_monitor_true_residual The last option is useful as it will report both the preconditioned residual and the true residual. If the operator is singular, or close to singular, gmres-ILU or gmres-BJacobi-ILU can report a preconditioned residual which is small, but is orders of magnitude different from the true residual. Thanks, Dave On 9 February 2016 at 14:39, Florian Lindner wrote: > Addition. The KSP Solver shows very different convergence: > > > WRONG: > > [0] KSPConvergedDefault(): Linear solver has converged. Residual norm > 6.832362172732e+06 is less than relative tolerance 1.000000000000e-09 > times initial right hand side norm 6.934533099989e+15 at iteration 8447 > > RIGHT: > > [0] KSPConvergedDefault(): Linear solver has converged. Residual norm > 7.959757133341e-08 is less than relative tolerance 1.000000000000e-09 > times initial right hand side norm 1.731788191624e+02 at iteration 9 > > Best, > Florian > > On Tue, 9 Feb 2016 14:06:01 +0100 > Florian Lindner wrote: > > > Hello, > > > > I use PETSc with 4 MPI processes and I experience different results > > when using different distribution of rows amoung ranks. The code looks > > like that: > > > > > > KSPSetOperators(_solver, _matrixC.matrix, _matrixC.matrix); > > // _solverRtol = 1e-9 > > KSPSetTolerances(_solver, _solverRtol, PETSC_DEFAULT, PETSC_DEFAULT, > > PETSC_DEFAULT); > > KSPSetFromOptions(_solver); > > > > // means: MatGetVecs(matrix, nullptr, &vector); > > petsc::Vector Au(_matrixC, "Au"); > > petsc::Vector out(_matrixC, "out"); > > petsc::Vector in(_matrixA, "in"); > > > > // is an identity mapping here > > ierr = VecSetLocalToGlobalMapping(in.vector, _ISmapping); > > > > // fill and assemble vector in > > > > in.view(); > > MatMultTranspose(_matrixA.matrix, in.vector, Au.vector); > > Au.view(); > > KSPSolve(_solver, Au.vector, out.vector); > > > > out.view(); > > > > I have experimented with two variants. The first one, non-working has > > MatrixC rows distributed like that: 3, 4, 2, 2. The other, working, > > one has 4, 3, 2, 2. > > > > All input values: _matrixA, _matrixC, in are identival, Au is > > identical too, but out differs. See the results from object::view > > below. > > > > Could the bad condition of matrixC be a problem? I'm not using any > > special KSP options. matrixC is of type MATSBAIJ, matrixA is MATAIJ. > > > > Thanks, > > Florian > > > > WRONG > > ================= > > > > Vec Object:in 4 MPI processes > > type: mpi > > Process [0] > > 1 > > 2 > > Process [1] > > 3 > > 4 > > Process [2] > > 5 > > 6 > > Process [3] > > 7 > > 8 > > > > Vec Object:Au 4 MPI processes > > type: mpi > > Process [0] > > 36 > > 74 > > 20 > > Process [1] > > 1.09292 > > 2.09259 > > 3.18584 > > 4.20349 > > Process [2] > > 5.29708 > > 6.31472 > > Process [3] > > 7.24012 > > 8.23978 > > > > // should not be result > > Vec Object:out 4 MPI processes > > type: mpi > > Process [0] > > -1.10633e+07 > > 618058 > > 9.01497e+06 > > Process [1] > > 0.996195 > > 1.98711 > > 3.01111 > > 4.00203 > > Process [2] > > 5.00736 > > 6.01644 > > Process [3] > > 6.98534 > > 7.99442 > > > > Mat Object:C 4 MPI processes > > type: mpisbaij > > row 0: (0, 0) (3, 1) (4, 1) (5, 1) (6, 1) (7, 1) (8, 1) (9, > > 1) (10, 1) row 1: (1, 0) (3, 0) (4, 0) (5, 1) (6, 1) (7, 2) > > (8, 2) (9, 3) (10, 3) row 2: (2, 0) (3, 0) (4, 1) (5, 0) (6, > > 1) (7, 0) (8, 1) (9, 0) (10, 1) row 3: (3, 1) (4, 0.0183156) > > (5, 0.0183156) (6, 0.000335463) (7, 1.12535e-07) (8, 2.06115e-09) > > row 4: (4, 1) (5, 0.000335463) (6, 0.0183156) (7, 2.06115e-09) > > (8, 1.12535e-07) row 5: (5, 1) (6, 0.0183156) (7, 0.0183156) (8, > > 0.000335463) (9, 1.12535e-07) (10, 2.06115e-09) row 6: (6, 1) (7, > > 0.000335463) (8, 0.0183156) (9, 2.06115e-09) (10, 1.12535e-07) row > > 7: (7, 1) (8, 0.0183156) (9, 0.0183156) (10, 0.000335463) row 8: > > (8, 1) (9, 0.000335463) (10, 0.0183156) row 9: (9, 1) (10, > > 0.0183156) row 10: (10, 1) > > > > Mat Object:A 4 MPI processes > > type: mpiaij > > 1.00000e+00 0.00000e+00 0.00000e+00 1.00000e+00 1.83156e-02 > > 1.83156e-02 3.35463e-04 1.12535e-07 2.06115e-09 0.00000e+00 > > 0.00000e+00 1.00000e+00 0.00000e+00 1.00000e+00 1.83156e-02 > > 1.00000e+00 3.35463e-04 1.83156e-02 2.06115e-09 1.12535e-07 > > 0.00000e+00 0.00000e+00 1.00000e+00 1.00000e+00 0.00000e+00 > > 1.83156e-02 3.35463e-04 1.00000e+00 1.83156e-02 1.83156e-02 > > 3.35463e-04 1.12535e-07 2.06115e-09 1.00000e+00 1.00000e+00 > > 1.00000e+00 3.35463e-04 1.83156e-02 1.83156e-02 1.00000e+00 > > 3.35463e-04 1.83156e-02 2.06115e-09 1.12535e-07 1.00000e+00 > > 2.00000e+00 0.00000e+00 1.12535e-07 2.06115e-09 1.83156e-02 > > 3.35463e-04 1.00000e+00 1.83156e-02 1.83156e-02 3.35463e-04 > > 1.00000e+00 2.00000e+00 1.00000e+00 2.06115e-09 1.12535e-07 > > 3.35463e-04 1.83156e-02 1.83156e-02 1.00000e+00 3.35463e-04 > > 1.83156e-02 1.00000e+00 3.00000e+00 0.00000e+00 0.00000e+00 > > 0.00000e+00 1.12535e-07 2.06115e-09 1.83156e-02 3.35463e-04 > > 1.00000e+00 1.83156e-02 1.00000e+00 3.00000e+00 1.00000e+00 > > 0.00000e+00 0.00000e+00 2.06115e-09 1.12535e-07 3.35463e-04 > > 1.83156e-02 1.83156e-02 1.00000e+00 > > > > > > > > RIGHT > > ================= > > > > Vec Object:in 4 MPI processes > > type: mpi > > Process [0] > > 1 > > 2 > > Process [1] > > 3 > > 4 > > Process [2] > > 5 > > 6 > > Process [3] > > 7 > > 8 > > > > Vec Object:Au 4 MPI processes > > type: mpi > > Process [0] > > 36 > > 74 > > 20 > > 1.09292 > > Process [1] > > 2.09259 > > 3.18584 > > 4.20349 > > Process [2] > > 5.29708 > > 6.31472 > > Process [3] > > 7.24012 > > 8.23978 > > > > // should be result > > Vec Object:out 4 MPI processes > > type: mpi > > Process [0] > > 0 > > 0 > > 0 > > 1 > > Process [1] > > 2 > > 3 > > 4 > > Process [2] > > 5 > > 6 > > Process [3] > > 7 > > 8 > > > > > > Mat Object:C 4 MPI processes > > type: mpisbaij > > row 0: (0, 0) (3, 1) (4, 1) (5, 1) (6, 1) (7, 1) (8, 1) (9, > > 1) (10, 1) row 1: (1, 0) (3, 0) (4, 0) (5, 1) (6, 1) (7, 2) > > (8, 2) (9, 3) (10, 3) row 2: (2, 0) (3, 0) (4, 1) (5, 0) (6, > > 1) (7, 0) (8, 1) (9, 0) (10, 1) row 3: (3, 1) (4, 0.0183156) > > (5, 0.0183156) (6, 0.000335463) (7, 1.12535e-07) (8, 2.06115e-09) > > row 4: (4, 1) (5, 0.000335463) (6, 0.0183156) (7, 2.06115e-09) > > (8, 1.12535e-07) row 5: (5, 1) (6, 0.0183156) (7, 0.0183156) (8, > > 0.000335463) (9, 1.12535e-07) (10, 2.06115e-09) row 6: (6, 1) (7, > > 0.000335463) (8, 0.0183156) (9, 2.06115e-09) (10, 1.12535e-07) row > > 7: (7, 1) (8, 0.0183156) (9, 0.0183156) (10, 0.000335463) row 8: > > (8, 1) (9, 0.000335463) (10, 0.0183156) row 9: (9, 1) (10, > > 0.0183156) row 10: (10, 1) > > > > Mat Object:A 4 MPI processes > > type: mpiaij > > 1.00000e+00 0.00000e+00 0.00000e+00 1.00000e+00 1.83156e-02 > > 1.83156e-02 3.35463e-04 1.12535e-07 2.06115e-09 0.00000e+00 > > 0.00000e+00 1.00000e+00 0.00000e+00 1.00000e+00 1.83156e-02 > > 1.00000e+00 3.35463e-04 1.83156e-02 2.06115e-09 1.12535e-07 > > 0.00000e+00 0.00000e+00 1.00000e+00 1.00000e+00 0.00000e+00 > > 1.83156e-02 3.35463e-04 1.00000e+00 1.83156e-02 1.83156e-02 > > 3.35463e-04 1.12535e-07 2.06115e-09 1.00000e+00 1.00000e+00 > > 1.00000e+00 3.35463e-04 1.83156e-02 1.83156e-02 1.00000e+00 > > 3.35463e-04 1.83156e-02 2.06115e-09 1.12535e-07 1.00000e+00 > > 2.00000e+00 0.00000e+00 1.12535e-07 2.06115e-09 1.83156e-02 > > 3.35463e-04 1.00000e+00 1.83156e-02 1.83156e-02 3.35463e-04 > > 1.00000e+00 2.00000e+00 1.00000e+00 2.06115e-09 1.12535e-07 > > 3.35463e-04 1.83156e-02 1.83156e-02 1.00000e+00 3.35463e-04 > > 1.83156e-02 1.00000e+00 3.00000e+00 0.00000e+00 0.00000e+00 > > 0.00000e+00 1.12535e-07 2.06115e-09 1.83156e-02 3.35463e-04 > > 1.00000e+00 1.83156e-02 1.00000e+00 3.00000e+00 1.00000e+00 > > 0.00000e+00 0.00000e+00 2.06115e-09 1.12535e-07 3.35463e-04 > > 1.83156e-02 1.83156e-02 1.00000e+00 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From albert.gutierrez at bsc.es Tue Feb 9 08:20:44 2016 From: albert.gutierrez at bsc.es (=?UTF-8?Q?albert_guti=C3=A9rrez?=) Date: Tue, 9 Feb 2016 15:20:44 +0100 Subject: [petsc-users] combine single and double precision Message-ID: Hello everyone, At times it can be useful to combine different data types in the same problem. For example, there are parts of an algorithm which may be memory bounded (requiring float) and operations where more precision is needed (requiring double or float_128). Can single and double precision be combined in PETSc? I know the data type is set in the configuration process using the parameter "--with-precision=TYPE". It looks like having single and double precision at the same time will require the modification of the source code of the library. Is there an easy way to implement this feature? Regards. WARNING / LEGAL TEXT: This message is intended only for the use of the individual or entity to which it is addressed and may contain information which is privileged, confidential, proprietary, or exempt from disclosure under applicable law. If you are not the intended recipient or the person responsible for delivering the message to the intended recipient, you are strictly prohibited from disclosing, distributing, copying, or in any way using this message. If you have received this communication in error, please notify the sender and destroy and delete any copies you may have received. http://www.bsc.es/disclaimer -------------- next part -------------- An HTML attachment was scrubbed... URL: From hng.email at gmail.com Tue Feb 9 09:10:10 2016 From: hng.email at gmail.com (Hom Nath Gharti) Date: Tue, 9 Feb 2016 10:10:10 -0500 Subject: [petsc-users] PCFIELDSPLIT question In-Reply-To: References: Message-ID: Thank you so much Barry! For my small test case, with -pc_fieldsplit_block_size 4, the program runs, although the answer was not correct. At least now I get something to look upon. I am using PCFieldSplitSetIS to set the fields. Do I still need to use -pc_fieldsplit_block_size? In my case each grid point may have different variable sets. For example, the grid point in the solid region has displacement and gravity potential: ux, uy, uz, and \phi. Whereas the grid point in the fluid region has scalar potential, pressure and gravity potential: \chi, p, and \phi. And the solid-fluid interface has all of them. Do you think I can still use PCFIELDSPLIT in this situation? Best, Hom Nath On Mon, Feb 8, 2016 at 2:27 PM, Barry Smith wrote: > > > In this case you need to provide two pieces of information to the PCFIELDSPLIT. What we call the "block size" or bs which is the number of "basic fields" in the problem. For example if at each grid point you have x-velocity, y-velocity, and pressure the block size is 3. The second is you need to tell PCFIELDSPLIT what "basic fields" are in each split you want to use. > > In your case you can do this with -pc_fieldsplit_block_size 4. (Note if you use a DM to define your problem the PCFIELDSPLIT will automatically get the bs from the DM so you do not need to provide it. Similarly if you set a block size on your Mat the PCFIELDSPLIT will use that so you don't need to set it). > > Then use -pc_fieldsplit_0_fields 0,1 to indicate your first split is the 0 and 1 basic fields and -pc_fieldsplit_1_fields 2,3 to indicate the second split is the 2 and 3 basic fields. > (By default if you do not provide this then PCFIELDSPLIT will use bs splits (4 in your case) one for each basic field. > > Barry > >> On Feb 8, 2016, at 11:20 AM, Hom Nath Gharti wrote: >> >> Hi Matt, Hi all, >> >> I am trying to have some feel for PCFIELDSPLIT testing on a very small >> matrix (attached below). I have 4 degrees of freedoms. I use 4 >> processors. I want to split 4 dofs into two fields each having two >> dofs. I don't know whether this my be a problem for petsc. When I use >> the command: >> srun -n 4 ./xtestfs -ksp_monitor -ksp_view >> >> It runs fine. >> >> But when I want to use field split options using the command: >> srun -n 4 ./xtestfs -ksp_monitor -ksp_type fgmres -pc_type fieldsplit >> -fieldsplit_0_ksp_type gmres -fieldsplit_0_pc_type bjacobi >> -fieldsplit_1_pc_type jacobi >> >> I get the following error message: >> [0]PETSC ERROR: Petsc has generated inconsistent data >> [0]PETSC ERROR: Unhandled case, must have at least two fields, not 1 >> [0]PETSC ERROR: See >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >> shooting. >> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >> [0]PETSC ERROR: /tigress/hgharti/test/testfs/./xtestfs on a >> arch-linux2-c-debug named tiger-r11n1 by hgharti Mon Feb 8 11:40:03 >> 2016 >> [0]PETSC ERROR: Configure options >> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64/ >> --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 >> --with-mpiexec=mpiexec --with-debugging=1 --download-scalapack >> --download-mumps --download-pastix --download-superlu >> --download-superlu_dist --download-metis --download-parmetis >> --download-ptscotch --download-hypre >> [0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 469 in >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 486 in >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> [0]PETSC ERROR: #3 PCSetUp() line 983 in >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: #4 KSPSetUp() line 332 in >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: #5 KSPSolve() line 546 in >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >> WARNING! There are options you set that were not used! >> WARNING! could be spelling mistake, etc! >> Option left: name:-fieldsplit_0_ksp_type value: gmres >> Option left: name:-fieldsplit_0_pc_type value: bjacobi >> Option left: name:-fieldsplit_1_pc_type value: jacobi >> forrtl: error (76): Abort trap signal >> >> I tried several trials but could not fix the problem. Is it the >> FORTRAN problem or am I doing something wrong? Any suggestions would >> be greatly appreciated. >> For your information I use: >> PETSc-3.6.3 >> intel-16.0 compiler >> intel-mpi-5.1.1 >> >> Program >> >> >> Best, >> Hom Nath >> >> ! simple test case for PCFIELDSPLIT >> ! ----------------------------------------------------------------------- >> program testfs >> implicit none >> #include "petsc/finclude/petscsys.h" >> #include "petsc/finclude/petscvec.h" >> #include "petsc/finclude/petscvec.h90" >> #include "petsc/finclude/petscmat.h" >> #include "petsc/finclude/petscksp.h" >> #include "petsc/finclude/petscpc.h" >> #include "petsc/finclude/petscviewer.h" >> #include "petsc/finclude/petscviewer.h90" >> >> Vec x,b,u >> Mat A >> KSP ksp >> PC pc >> PetscErrorCode ierr >> PetscBool flg >> PetscInt errcode,its,maxitr,myrank,nproc >> PetscInt nedof,nelmt,ndof,nenod,ngdof,fixdof,nsparse,neq >> PetscInt,allocatable :: krow_sparse(:),ggdof_elmt(:,:), & >> inode_elmt(:,:) >> >> PetscReal,allocatable :: storef(:,:),storekmat(:,:,:) >> PetscInt gdof0(2),gdof1(2) >> >> ! initialize MPI >> >> call MPI_INIT(errcode) >> if(errcode /= 0)stop 'ERROR: cannot initialize MPI!' >> call MPI_COMM_RANK(MPI_COMM_WORLD,myrank,errcode) >> if(errcode /= 0)stop 'ERROR: cannot find processor ID' >> call MPI_COMM_SIZE(MPI_COMM_WORLD,nproc,errcode) >> if(errcode /= 0)stop 'ERROR: cannot find number of processors!' >> >> ! define some parameters >> ! 1D model with 4 elements. Each elements consits of 2 nodes. Node 0 is fixed. >> ! 1-----------2-----------3-----------4-----------5 >> nelmt=1 ! per processor >> nenod=2 ! number of nodes per element >> nedof=2 ! number of DOFs per element >> ndof=2 ! number of DOFs >> ngdof=4 ! total number of global DOFs >> fixdof=1 >> >> if(myrank==0)then >> neq=1 ! number of equations >> nsparse=1 >> else >> neq=2 ! number of equations >> nsparse=4 >> endif >> >> allocate(storef(nedof,nelmt),storekmat(nedof,nedof,nelmt), & >> ggdof_elmt(nedof,nelmt),inode_elmt(nenod,nelmt),krow_sparse(nsparse)) >> >> storef=0.0_8 ! local RHS vector >> storekmat=0.0_8 ! local stiffness matrix >> if(myrank==0)then >> krow_sparse(:)=1 >> storef(:,1)=(/12.5000_8, 12.5000_8/) >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & >> 0.0166666667_8, 1.2333333332_8/),(/nedof,nedof/)) >> inode_elmt(:,1) = (/1,2/) >> ggdof_elmt(:,1) = (/0,1/) ! 0 because left node is fixed >> elseif(myrank==1)then >> krow_sparse(:)=(/1,1,2,2/) >> storef(:,1)=(/12.5000_8, 12.5000_8/) >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & >> 0.0166666667_8, 1.2333333332_8/),(/nedof,nedof/)) >> inode_elmt(:,1) = (/1,2/) >> ggdof_elmt(:,1) = (/1,2/) >> elseif(myrank==2)then >> krow_sparse(:)=(/1,1,2,2/) >> storef(:,1)=(/12.5000_8, 12.5000_8/) >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & >> 0.0166666667_8, 1.2333333332_8/),(/nedof,nedof/)) >> inode_elmt(:,1) = (/1,2/) >> ggdof_elmt(:,1) = (/2,3/) >> elseif(myrank==3)then >> krow_sparse(:)=(/1,1,2,2/) >> storef(:,1)=(/12.5000_8, 22.5000_8/) >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & >> 0.0166666667_8, 1.2333333332_8/),(/nedof,nedof/)) >> inode_elmt(:,1) = (/1,2/) >> ggdof_elmt(:,1) = (/3,4/) >> endif >> >> call petsc_initialize >> call petsc_set_matrix >> call petsc_set_vector >> call petsc_solve >> call petsc_finalize >> >> ! deallocate >> deallocate(storef,storekmat,ggdof_elmt,inode_elmt) >> deallocate(krow_sparse) >> contains >> !------------------------------------------------------------------------------- >> subroutine petsc_initialize() >> implicit none >> PetscInt :: istart,iend >> PetscInt :: nzeros_max,nzeros_min >> PetscReal :: atol,dtol,rtol >> PetscInt,allocatable :: nzeros(:) >> IS gdof0_is,gdof1_is >> >> call PetscInitialize(PETSC_NULL_CHARACTER,ierr) >> call PetscOptionsGetInt(PETSC_NULL_CHARACTER,'-n',ngdof,flg,ierr) >> >> ! count number of nonzeros per row >> allocate(nzeros(neq)) >> nzeros=0 >> nzeros(krow_sparse)=nzeros(krow_sparse)+1 >> nzeros_max=maxval(nzeros) >> nzeros_min=minval(nzeros) >> >> ! create matrix >> call MatCreate(PETSC_COMM_WORLD,A,ierr) >> call MatSetType(A,MATMPIAIJ,ierr) >> CHKERRQ(ierr) >> call MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,ngdof,ngdof,ierr) >> CHKERRQ(ierr) >> ! preallocation >> call MatMPIAIJSetPreallocation(A,nzeros_max,PETSC_NULL_INTEGER,nzeros_max, & >> PETSC_NULL_INTEGER,ierr) >> CHKERRQ(ierr) >> >> call MatGetOwnershipRange(A,istart,iend,ierr) >> CHKERRQ(ierr) >> print*,'ownership:',myrank,istart,iend >> ! create vector >> call VecCreate(PETSC_COMM_WORLD,x,ierr) >> call VecSetSizes(x,PETSC_DECIDE,ngdof,ierr) >> call VecSetType(x,VECMPI,ierr) >> call VecDuplicate(x,b,ierr) >> call VecDuplicate(x,u,ierr) >> >> !******************************************************************************* >> ! get IS for split fields >> gdof0=(/0,1/) >> gdof1=(/2,3/) >> call ISCreateGeneral(PETSC_COMM_WORLD,2,gdof0,PETSC_COPY_VALUES,gdof0_is,ierr) >> CHKERRQ(ierr) >> call ISCreateGeneral(PETSC_COMM_WORLD,2,gdof1,PETSC_COPY_VALUES,gdof1_is,ierr) >> CHKERRQ(ierr) >> !******************************************************************************* >> >> ! Create linear solver context >> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >> call KSPSetOperators(ksp,A,A,ierr) >> >> call KSPGetPC(ksp,pc,ierr) >> CHKERRQ(ierr) >> >> !******************************************************************************* >> ! split PC >> call PCFieldSplitSetIS(pc,"0",gdof0_is,ierr); >> CHKERRQ(ierr) >> call ISDestroy(gdof0_is,ierr) >> CHKERRQ(ierr) >> call PCFieldSplitSetIS(pc,"1",gdof1_is,ierr); >> CHKERRQ(ierr) >> call ISDestroy(gdof1_is,ierr) >> CHKERRQ(ierr) >> !******************************************************************************* >> >> rtol = 1.0d-6 >> atol = 1.0d-6 >> dtol = 1.0d10 >> maxitr = 1000 >> call KSPSetTolerances(ksp,rtol,atol,dtol,maxitr,ierr) >> CHKERRQ(ierr) >> call KSPSetFromOptions(ksp,ierr) >> end subroutine petsc_initialize >> !------------------------------------------------------------------------------- >> >> subroutine petsc_set_matrix() >> >> implicit none >> integer :: i,i_elmt,j,ncount >> integer :: ggdof(NEDOF),idof(NEDOF),igdof(NEDOF) >> >> ! Set and assemble matrix. >> call MatZeroEntries(A,ierr) >> CHKERRQ(ierr) >> do i_elmt=1,nelmt >> ggdof(:)=ggdof_elmt(:,i_elmt) >> ggdof(:)=ggdof(:)-1 ! petsc index starts from 0 >> ncount=0; idof=-1; igdof=-1 >> do i=1,NEDOF >> do j=1,NEDOF >> if(ggdof(i).ge.0.and.ggdof(j).ge.0)then >> call MatSetValues(A,1,ggdof(i),1,ggdof(j),storekmat(i,j,i_elmt), >> & >> ADD_VALUES,ierr) >> CHKERRQ(ierr) >> endif >> enddo >> enddo >> enddo >> call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) >> call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) >> if(myrank==0)print*,'matrix setting & assembly complete!' >> >> end subroutine petsc_set_matrix >> !------------------------------------------------------------------------------- >> >> subroutine petsc_set_vector() >> implicit none >> PetscScalar zero >> integer :: i,i_elmt,ncount >> integer :: ggdof(NEDOF),idof(NEDOF),igdof(NEDOF) >> >> ! set vector >> zero=0.0_8 >> call VecSet(b,zero,ierr) >> do i_elmt=1,nelmt >> ggdof(:)=ggdof_elmt(:,i_elmt) >> ggdof(:)=ggdof(:)-1 ! petsc index starts from 0 >> ncount=0; idof=-1; igdof=-1 >> do i=1,NEDOF >> if(ggdof(i).ge.0)then >> call VecSetValues(b,1,ggdof(i),storef(i,i_elmt),ADD_VALUES,ierr); >> CHKERRQ(ierr) >> endif >> enddo >> enddo >> >> ! assemble vector >> call VecAssemblyBegin(b,ierr) >> call VecAssemblyEnd(b,ierr) >> if(myrank==0)print*,'vector setting & assembly complete!' >> >> end subroutine petsc_set_vector >> !------------------------------------------------------------------------------- >> >> subroutine petsc_solve() >> implicit none >> PetscInt ireason >> PetscScalar a_v(1) >> PetscOffset a_i >> PetscInt n >> PetscReal,allocatable :: sdata(:) >> >> call VecGetSize(b,n,ierr) >> CHKERRQ(ierr) >> allocate(sdata(n)) >> sdata=0.0_8 >> call VecGetArray(b,a_v,a_i,ierr) >> CHKERRQ(ierr) >> sdata(1:n)=a_v(a_i+1:a_i+n) >> call VecRestoreArray(b,a_v,a_i,ierr) >> CHKERRQ(ierr) >> print*,'rhs:',sdata >> >> call KSPSolve(ksp,b,x,ierr) >> sdata=0.0_8 >> call VecGetArray(x,a_v,a_i,ierr) >> CHKERRQ(ierr) >> sdata(1:n)=a_v(a_i+1:a_i+n) >> call VecRestoreArray(b,a_v,a_i,ierr) >> CHKERRQ(ierr) >> print*,'solution:',sdata >> >> ! Check convergence >> call KSPGetConvergedReason(ksp,ireason,ierr) >> if(myrank==0)print*,'converges reason',myrank,ireason >> call KSPGetIterationNumber(ksp,its,ierr) >> if(myrank==0)print*,'Iterations:',its >> deallocate(sdata) >> end subroutine petsc_solve >> !------------------------------------------------------------------------------- >> >> subroutine petsc_finalize() >> implicit none >> >> ! Free work space. >> call VecDestroy(x,ierr) >> call VecDestroy(u,ierr) >> call VecDestroy(b,ierr) >> call MatDestroy(A,ierr) >> call KSPDestroy(ksp,ierr) >> call PetscFinalize(ierr) >> CHKERRQ(ierr) >> >> end subroutine petsc_finalize >> !------------------------------------------------------------------------------- >> >> end program testfs >> >> On Tue, Feb 2, 2016 at 4:54 PM, Matthew Knepley wrote: >>> On Tue, Feb 2, 2016 at 3:11 PM, Hom Nath Gharti wrote: >>>> >>>> Thanks a lot. I will try. >>> >>> >>> Also, if you send a small test case, we can run it and tell you the problem. >>> >>> Matt >>> >>>> >>>> Hom Nath >>>> >>>> On Tue, Feb 2, 2016 at 4:02 PM, Matthew Knepley wrote: >>>>> On Tue, Feb 2, 2016 at 3:01 PM, Hom Nath Gharti >>>>> wrote: >>>>>> >>>>>> Thanks again Matt. Unfortunately got the same errors with '0'. I think >>>>>> both are valid in Fortran. >>>>> >>>>> >>>>> Then you will have to go in the debugger and see why its not creating 4 >>>>> splits, since this is exactly >>>>> what it does in our tests. This is how all the SNES tests that I use >>>>> work. I >>>>> am sure its a Fortran >>>>> problem. >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>>> >>>>>> Hom Nath >>>>>> >>>>>> On Tue, Feb 2, 2016 at 3:42 PM, Matthew Knepley >>>>>> wrote: >>>>>>> On Tue, Feb 2, 2016 at 2:35 PM, Hom Nath Gharti >>>>>>> wrote: >>>>>>>> >>>>>>>> Thank you so much Matt. >>>>>>>> >>>>>>>> I now tried the following: >>>>>>>> >>>>>>>> ====================================================== >>>>>>>> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >>>>>>>> call KSPGetPC(ksp,pc,ierr) >>>>>>>> >>>>>>>> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); >>>>>>>> call ISDestroy(gdofu_is,ierr) >>>>>>>> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); >>>>>>>> call ISDestroy(gdofchi_is,ierr) >>>>>>>> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); >>>>>>>> call ISDestroy(gdofp_is,ierr) >>>>>>>> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); >>>>>>>> call ISDestroy(gdofphi_is,ierr) >>>>>>>> >>>>>>>> ! Amat changes in each time steps >>>>>>>> call KSPSetOperators(ksp,Amat,Amat,ierr) !version >= 3.5.0 >>>>>>>> >>>>>>>> call KSPSolve(ksp,bvec,xvec,ierr) >>>>>>>> ====================================================== >>>>>>> >>>>>>> >>>>>>> I am guessing that "0" is not a valid string for your Fortran >>>>>>> compiler. >>>>>>> Are >>>>>>> you sure >>>>>>> its not single quotes '0'? >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>>> >>>>>>>> But I get the following error: >>>>>>>> >>>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>>> -------------------------------------------------------------- >>>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>>>>> [0]PETSC ERROR: Unhandled case, must have at least two fields, not 1 >>>>>>>> [0]PETSC ERROR: See >>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >>>>>>>> shooting. >>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>>>>>>> [0]PETSC ERROR: >>>>>>>> /tigress/hgharti/gitwork/SPECFEM3D_GLOBEVSPP/./bin/xspecfem3D on a >>>>>>>> arch-linux2-c-debug named tiger-r3c1n7 by hgharti Tue Feb 2 15: >>>>>>>> 29:30 2016 >>>>>>>> [0]PETSC ERROR: Configure options >>>>>>>> >>>>>>>> >>>>>>>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64/ >>>>>>>> --with-cc=mpicc --with-cxx=mpicxx --wit >>>>>>>> h-fc=mpif90 --with-mpiexec=mpiexec --with-debugging=1 >>>>>>>> --download-scalapack --download-mumps --download-pastix >>>>>>>> --download-superlu --download-superlu_dist --download-metis >>>>>>>> --download-parmetis --download-ptscotch --download-hypre >>>>>>>> [0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 469 in >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >>>>>>>> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 486 in >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >>>>>>>> [0]PETSC ERROR: #3 PCSetUp() line 983 in >>>>>>>> >>>>>>>> >>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/interface/precon.c >>>>>>>> [0]PETSC ERROR: #4 KSPSetUp() line 332 in >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >>>>>>>> [0]PETSC ERROR: #5 KSPSolve() line 546 in >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >>>>>>>> forrtl: error (76): Abort trap signal >>>>>>>> >>>>>>>> Am I missing something? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Hom Nath >>>>>>>> >>>>>>>> On Tue, Feb 2, 2016 at 3:24 PM, Matthew Knepley >>>>>>>> wrote: >>>>>>>>> On Tue, Feb 2, 2016 at 2:20 PM, Hom Nath Gharti >>>>>>>>> >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Hi Matt, hi all, >>>>>>>>>> >>>>>>>>>> I am trying to use PCFIELDSPLIT for my solver which consists of 4 >>>>>>>>>> different variables, namely, u (displacement vector), \chi >>>>>>>>>> (displacement potential), p(pressure), and \phi (gravity >>>>>>>>>> potential). >>>>>>>>>> >>>>>>>>>> My code segment looks like the following: >>>>>>>>>> ====================================================== >>>>>>>>>> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >>>>>>>>>> call KSPGetPC(ksp,pc,ierr) >>>>>>>>>> nsplit=4 >>>>>>>>>> call PCFieldSplitSetBlockSize(pc,nsplit,ierr); >>>>>>>>> >>>>>>>>> >>>>>>>>> You do not need the statement above. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); >>>>>>>>>> call ISDestroy(gdofu_is,ierr) >>>>>>>>>> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); >>>>>>>>>> call ISDestroy(gdofchi_is,ierr) >>>>>>>>>> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); >>>>>>>>>> call ISDestroy(gdofp_is,ierr) >>>>>>>>>> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); >>>>>>>>>> call ISDestroy(gdofphi_is,ierr) >>>>>>>>>> >>>>>>>>>> call PCFieldSplitGetSubKSP(pc,nsplit,subksp,ierr); >>>>>>>>> >>>>>>>>> >>>>>>>>> These SetOperators() calls are wrong. I am not sure why you want >>>>>>>>> them >>>>>>>>> here. >>>>>>>>> Also, that means you do not need the call above. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Matt >>>>>>>>> >>>>>>>>>> >>>>>>>>>> call KSPSetOperators(subksp(1),Amat,Amat,ierr); >>>>>>>>>> call KSPSetOperators(subksp(2),Amat,Amat,ierr); >>>>>>>>>> call KSPSetOperators(subksp(3),Amat,Amat,ierr); >>>>>>>>>> call KSPSetOperators(subksp(4),Amat,Amat,ierr); >>>>>>>>>> >>>>>>>>>> call KSPSolve(ksp,bvec,xvec,ierr) >>>>>>>>>> ====================================================== >>>>>>>>>> >>>>>>>>>> But I am getting the following error: >>>>>>>>>> [79]PETSC ERROR: Null argument, when expecting valid pointer >>>>>>>>>> [79]PETSC ERROR: Null Object: Parameter # 1 >>>>>>>>>> [79]PETSC ERROR: #1 KSPSetOperators() line 536 in >>>>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interf >>>>>>>>>> >>>>>>>>>> It looks like I am doing something wrong in "call >>>>>>>>>> KSPSetOperators" >>>>>>>>>> but >>>>>>>>>> I could not figure that out. >>>>>>>>>> >>>>>>>>>> Could anybody help me to fix this problem? I looked into almost >>>>>>>>>> all >>>>>>>>>> related examples but I could not really figure out the correct >>>>>>>>>> steps >>>>>>>>>> after "call PCFieldSplitSetIS". >>>>>>>>>> >>>>>>>>>> Any help will be greatly appreciated. >>>>>>>>>> >>>>>>>>>> Best, >>>>>>>>>> Hom nath >>>>>>>>>> >>>>>>>>>> On Sun, Jan 24, 2016 at 7:14 PM, Hom Nath Gharti >>>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>>> Thank you so much Matt! I will try. >>>>>>>>>>> >>>>>>>>>>> Hom Nath >>>>>>>>>>> >>>>>>>>>>> On Sun, Jan 24, 2016 at 6:26 AM, Matthew Knepley >>>>>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>>> On Fri, Jan 22, 2016 at 2:19 PM, Hom Nath Gharti >>>>>>>>>>>> >>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Dear all, >>>>>>>>>>>>> >>>>>>>>>>>>> I am new to PcFieldSplit. >>>>>>>>>>>>> >>>>>>>>>>>>> I have a matrix formed using MATMPIAIJ. Is it possible to use >>>>>>>>>>>>> PCFIELDSPLIT operations in this type of matrix? Or does it >>>>>>>>>>>>> have >>>>>>>>>>>>> to >>>>>>>>>>>>> be >>>>>>>>>>>>> MATMPIBIJ or MATNEST format? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Yes, you can split AIJ. >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> If possible for MATMPIAIJ, could anybody provide me a simple >>>>>>>>>>>>> example >>>>>>>>>>>>> or few steps? Variables in the equations are displacement >>>>>>>>>>>>> vector, >>>>>>>>>>>>> scalar potential and pressure. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> If you do not have a collocated discretization, then you have >>>>>>>>>>>> to >>>>>>>>>>>> use >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> >>>>>>>>>>>> Matt >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks for help. >>>>>>>>>>>>> >>>>>>>>>>>>> Hom Nath >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>>> their >>>>>>>>>>>> experiments >>>>>>>>>>>> is infinitely more interesting than any results to which their >>>>>>>>>>>> experiments >>>>>>>>>>>> lead. >>>>>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>> experiments >>>>>>>>> is infinitely more interesting than any results to which their >>>>>>>>> experiments >>>>>>>>> lead. >>>>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments >>>>>>> is infinitely more interesting than any results to which their >>>>>>> experiments >>>>>>> lead. >>>>>>> -- Norbert Wiener >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments >>>>> is infinitely more interesting than any results to which their >>>>> experiments >>>>> lead. >>>>> -- Norbert Wiener >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments >>> is infinitely more interesting than any results to which their experiments >>> lead. >>> -- Norbert Wiener > From knepley at gmail.com Tue Feb 9 09:19:31 2016 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 9 Feb 2016 09:19:31 -0600 Subject: [petsc-users] PCFIELDSPLIT question In-Reply-To: References: Message-ID: On Tue, Feb 9, 2016 at 9:10 AM, Hom Nath Gharti wrote: > Thank you so much Barry! > > For my small test case, with -pc_fieldsplit_block_size 4, the program > runs, although the answer was not correct. At least now I get > something to look upon. I am using PCFieldSplitSetIS to set the > fields. Do I still need to use -pc_fieldsplit_block_size? > No, this is only for co-located discretizations. > In my case each grid point may have different variable sets. For > example, the grid point in the solid region has displacement and > gravity potential: ux, uy, uz, and \phi. Whereas the grid point in the > fluid region has scalar potential, pressure and gravity potential: > \chi, p, and \phi. And the solid-fluid interface has all of them. Do > you think I can still use PCFIELDSPLIT in this situation? > We have examples, like SNES ex62, which do exactly this. Thanks, Matt > Best, > Hom Nath > > > > On Mon, Feb 8, 2016 at 2:27 PM, Barry Smith wrote: > > > > > > In this case you need to provide two pieces of information to the > PCFIELDSPLIT. What we call the "block size" or bs which is the number of > "basic fields" in the problem. For example if at each grid point you have > x-velocity, y-velocity, and pressure the block size is 3. The second is you > need to tell PCFIELDSPLIT what "basic fields" are in each split you want to > use. > > > > In your case you can do this with -pc_fieldsplit_block_size 4. > (Note if you use a DM to define your problem the PCFIELDSPLIT will > automatically get the bs from the DM so you do not need to provide it. > Similarly if you set a block size on your Mat the PCFIELDSPLIT will use > that so you don't need to set it). > > > > Then use -pc_fieldsplit_0_fields 0,1 to indicate your first split is > the 0 and 1 basic fields and -pc_fieldsplit_1_fields 2,3 to indicate the > second split is the 2 and 3 basic fields. > > (By default if you do not provide this then PCFIELDSPLIT will use bs > splits (4 in your case) one for each basic field. > > > > Barry > > > >> On Feb 8, 2016, at 11:20 AM, Hom Nath Gharti > wrote: > >> > >> Hi Matt, Hi all, > >> > >> I am trying to have some feel for PCFIELDSPLIT testing on a very small > >> matrix (attached below). I have 4 degrees of freedoms. I use 4 > >> processors. I want to split 4 dofs into two fields each having two > >> dofs. I don't know whether this my be a problem for petsc. When I use > >> the command: > >> srun -n 4 ./xtestfs -ksp_monitor -ksp_view > >> > >> It runs fine. > >> > >> But when I want to use field split options using the command: > >> srun -n 4 ./xtestfs -ksp_monitor -ksp_type fgmres -pc_type fieldsplit > >> -fieldsplit_0_ksp_type gmres -fieldsplit_0_pc_type bjacobi > >> -fieldsplit_1_pc_type jacobi > >> > >> I get the following error message: > >> [0]PETSC ERROR: Petsc has generated inconsistent data > >> [0]PETSC ERROR: Unhandled case, must have at least two fields, not 1 > >> [0]PETSC ERROR: See > >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > >> shooting. > >> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > >> [0]PETSC ERROR: /tigress/hgharti/test/testfs/./xtestfs on a > >> arch-linux2-c-debug named tiger-r11n1 by hgharti Mon Feb 8 11:40:03 > >> 2016 > >> [0]PETSC ERROR: Configure options > >> > --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64/ > >> --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > >> --with-mpiexec=mpiexec --with-debugging=1 --download-scalapack > >> --download-mumps --download-pastix --download-superlu > >> --download-superlu_dist --download-metis --download-parmetis > >> --download-ptscotch --download-hypre > >> [0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 469 in > >> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > >> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 486 in > >> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > >> [0]PETSC ERROR: #3 PCSetUp() line 983 in > >> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/interface/precon.c > >> [0]PETSC ERROR: #4 KSPSetUp() line 332 in > >> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c > >> [0]PETSC ERROR: #5 KSPSolve() line 546 in > >> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c > >> WARNING! There are options you set that were not used! > >> WARNING! could be spelling mistake, etc! > >> Option left: name:-fieldsplit_0_ksp_type value: gmres > >> Option left: name:-fieldsplit_0_pc_type value: bjacobi > >> Option left: name:-fieldsplit_1_pc_type value: jacobi > >> forrtl: error (76): Abort trap signal > >> > >> I tried several trials but could not fix the problem. Is it the > >> FORTRAN problem or am I doing something wrong? Any suggestions would > >> be greatly appreciated. > >> For your information I use: > >> PETSc-3.6.3 > >> intel-16.0 compiler > >> intel-mpi-5.1.1 > >> > >> Program > >> > >> > >> Best, > >> Hom Nath > >> > >> ! simple test case for PCFIELDSPLIT > >> ! > ----------------------------------------------------------------------- > >> program testfs > >> implicit none > >> #include "petsc/finclude/petscsys.h" > >> #include "petsc/finclude/petscvec.h" > >> #include "petsc/finclude/petscvec.h90" > >> #include "petsc/finclude/petscmat.h" > >> #include "petsc/finclude/petscksp.h" > >> #include "petsc/finclude/petscpc.h" > >> #include "petsc/finclude/petscviewer.h" > >> #include "petsc/finclude/petscviewer.h90" > >> > >> Vec x,b,u > >> Mat A > >> KSP ksp > >> PC pc > >> PetscErrorCode ierr > >> PetscBool flg > >> PetscInt errcode,its,maxitr,myrank,nproc > >> PetscInt nedof,nelmt,ndof,nenod,ngdof,fixdof,nsparse,neq > >> PetscInt,allocatable :: krow_sparse(:),ggdof_elmt(:,:), & > >> inode_elmt(:,:) > >> > >> PetscReal,allocatable :: storef(:,:),storekmat(:,:,:) > >> PetscInt gdof0(2),gdof1(2) > >> > >> ! initialize MPI > >> > >> call MPI_INIT(errcode) > >> if(errcode /= 0)stop 'ERROR: cannot initialize MPI!' > >> call MPI_COMM_RANK(MPI_COMM_WORLD,myrank,errcode) > >> if(errcode /= 0)stop 'ERROR: cannot find processor ID' > >> call MPI_COMM_SIZE(MPI_COMM_WORLD,nproc,errcode) > >> if(errcode /= 0)stop 'ERROR: cannot find number of processors!' > >> > >> ! define some parameters > >> ! 1D model with 4 elements. Each elements consits of 2 nodes. Node 0 is > fixed. > >> ! 1-----------2-----------3-----------4-----------5 > >> nelmt=1 ! per processor > >> nenod=2 ! number of nodes per element > >> nedof=2 ! number of DOFs per element > >> ndof=2 ! number of DOFs > >> ngdof=4 ! total number of global DOFs > >> fixdof=1 > >> > >> if(myrank==0)then > >> neq=1 ! number of equations > >> nsparse=1 > >> else > >> neq=2 ! number of equations > >> nsparse=4 > >> endif > >> > >> allocate(storef(nedof,nelmt),storekmat(nedof,nedof,nelmt), > & > >> ggdof_elmt(nedof,nelmt),inode_elmt(nenod,nelmt),krow_sparse(nsparse)) > >> > >> storef=0.0_8 ! local RHS vector > >> storekmat=0.0_8 ! local stiffness matrix > >> if(myrank==0)then > >> krow_sparse(:)=1 > >> storef(:,1)=(/12.5000_8, 12.5000_8/) > >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & > >> 0.0166666667_8, > 1.2333333332_8/),(/nedof,nedof/)) > >> inode_elmt(:,1) = (/1,2/) > >> ggdof_elmt(:,1) = (/0,1/) ! 0 because left node is fixed > >> elseif(myrank==1)then > >> krow_sparse(:)=(/1,1,2,2/) > >> storef(:,1)=(/12.5000_8, 12.5000_8/) > >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & > >> 0.0166666667_8, > 1.2333333332_8/),(/nedof,nedof/)) > >> inode_elmt(:,1) = (/1,2/) > >> ggdof_elmt(:,1) = (/1,2/) > >> elseif(myrank==2)then > >> krow_sparse(:)=(/1,1,2,2/) > >> storef(:,1)=(/12.5000_8, 12.5000_8/) > >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & > >> 0.0166666667_8, > 1.2333333332_8/),(/nedof,nedof/)) > >> inode_elmt(:,1) = (/1,2/) > >> ggdof_elmt(:,1) = (/2,3/) > >> elseif(myrank==3)then > >> krow_sparse(:)=(/1,1,2,2/) > >> storef(:,1)=(/12.5000_8, 22.5000_8/) > >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & > >> 0.0166666667_8, > 1.2333333332_8/),(/nedof,nedof/)) > >> inode_elmt(:,1) = (/1,2/) > >> ggdof_elmt(:,1) = (/3,4/) > >> endif > >> > >> call petsc_initialize > >> call petsc_set_matrix > >> call petsc_set_vector > >> call petsc_solve > >> call petsc_finalize > >> > >> ! deallocate > >> deallocate(storef,storekmat,ggdof_elmt,inode_elmt) > >> deallocate(krow_sparse) > >> contains > >> > !------------------------------------------------------------------------------- > >> subroutine petsc_initialize() > >> implicit none > >> PetscInt :: istart,iend > >> PetscInt :: nzeros_max,nzeros_min > >> PetscReal :: atol,dtol,rtol > >> PetscInt,allocatable :: nzeros(:) > >> IS gdof0_is,gdof1_is > >> > >> call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > >> call PetscOptionsGetInt(PETSC_NULL_CHARACTER,'-n',ngdof,flg,ierr) > >> > >> ! count number of nonzeros per row > >> allocate(nzeros(neq)) > >> nzeros=0 > >> nzeros(krow_sparse)=nzeros(krow_sparse)+1 > >> nzeros_max=maxval(nzeros) > >> nzeros_min=minval(nzeros) > >> > >> ! create matrix > >> call MatCreate(PETSC_COMM_WORLD,A,ierr) > >> call MatSetType(A,MATMPIAIJ,ierr) > >> CHKERRQ(ierr) > >> call MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,ngdof,ngdof,ierr) > >> CHKERRQ(ierr) > >> ! preallocation > >> call > MatMPIAIJSetPreallocation(A,nzeros_max,PETSC_NULL_INTEGER,nzeros_max, & > >> PETSC_NULL_INTEGER,ierr) > >> CHKERRQ(ierr) > >> > >> call MatGetOwnershipRange(A,istart,iend,ierr) > >> CHKERRQ(ierr) > >> print*,'ownership:',myrank,istart,iend > >> ! create vector > >> call VecCreate(PETSC_COMM_WORLD,x,ierr) > >> call VecSetSizes(x,PETSC_DECIDE,ngdof,ierr) > >> call VecSetType(x,VECMPI,ierr) > >> call VecDuplicate(x,b,ierr) > >> call VecDuplicate(x,u,ierr) > >> > >> > !******************************************************************************* > >> ! get IS for split fields > >> gdof0=(/0,1/) > >> gdof1=(/2,3/) > >> call > ISCreateGeneral(PETSC_COMM_WORLD,2,gdof0,PETSC_COPY_VALUES,gdof0_is,ierr) > >> CHKERRQ(ierr) > >> call > ISCreateGeneral(PETSC_COMM_WORLD,2,gdof1,PETSC_COPY_VALUES,gdof1_is,ierr) > >> CHKERRQ(ierr) > >> > !******************************************************************************* > >> > >> ! Create linear solver context > >> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) > >> call KSPSetOperators(ksp,A,A,ierr) > >> > >> call KSPGetPC(ksp,pc,ierr) > >> CHKERRQ(ierr) > >> > >> > !******************************************************************************* > >> ! split PC > >> call PCFieldSplitSetIS(pc,"0",gdof0_is,ierr); > >> CHKERRQ(ierr) > >> call ISDestroy(gdof0_is,ierr) > >> CHKERRQ(ierr) > >> call PCFieldSplitSetIS(pc,"1",gdof1_is,ierr); > >> CHKERRQ(ierr) > >> call ISDestroy(gdof1_is,ierr) > >> CHKERRQ(ierr) > >> > !******************************************************************************* > >> > >> rtol = 1.0d-6 > >> atol = 1.0d-6 > >> dtol = 1.0d10 > >> maxitr = 1000 > >> call KSPSetTolerances(ksp,rtol,atol,dtol,maxitr,ierr) > >> CHKERRQ(ierr) > >> call KSPSetFromOptions(ksp,ierr) > >> end subroutine petsc_initialize > >> > !------------------------------------------------------------------------------- > >> > >> subroutine petsc_set_matrix() > >> > >> implicit none > >> integer :: i,i_elmt,j,ncount > >> integer :: ggdof(NEDOF),idof(NEDOF),igdof(NEDOF) > >> > >> ! Set and assemble matrix. > >> call MatZeroEntries(A,ierr) > >> CHKERRQ(ierr) > >> do i_elmt=1,nelmt > >> ggdof(:)=ggdof_elmt(:,i_elmt) > >> ggdof(:)=ggdof(:)-1 ! petsc index starts from 0 > >> ncount=0; idof=-1; igdof=-1 > >> do i=1,NEDOF > >> do j=1,NEDOF > >> if(ggdof(i).ge.0.and.ggdof(j).ge.0)then > >> call MatSetValues(A,1,ggdof(i),1,ggdof(j),storekmat(i,j,i_elmt), > >> & > >> ADD_VALUES,ierr) > >> CHKERRQ(ierr) > >> endif > >> enddo > >> enddo > >> enddo > >> call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) > >> call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) > >> if(myrank==0)print*,'matrix setting & assembly complete!' > >> > >> end subroutine petsc_set_matrix > >> > !------------------------------------------------------------------------------- > >> > >> subroutine petsc_set_vector() > >> implicit none > >> PetscScalar zero > >> integer :: i,i_elmt,ncount > >> integer :: ggdof(NEDOF),idof(NEDOF),igdof(NEDOF) > >> > >> ! set vector > >> zero=0.0_8 > >> call VecSet(b,zero,ierr) > >> do i_elmt=1,nelmt > >> ggdof(:)=ggdof_elmt(:,i_elmt) > >> ggdof(:)=ggdof(:)-1 ! petsc index starts from 0 > >> ncount=0; idof=-1; igdof=-1 > >> do i=1,NEDOF > >> if(ggdof(i).ge.0)then > >> call VecSetValues(b,1,ggdof(i),storef(i,i_elmt),ADD_VALUES,ierr); > >> CHKERRQ(ierr) > >> endif > >> enddo > >> enddo > >> > >> ! assemble vector > >> call VecAssemblyBegin(b,ierr) > >> call VecAssemblyEnd(b,ierr) > >> if(myrank==0)print*,'vector setting & assembly complete!' > >> > >> end subroutine petsc_set_vector > >> > !------------------------------------------------------------------------------- > >> > >> subroutine petsc_solve() > >> implicit none > >> PetscInt ireason > >> PetscScalar a_v(1) > >> PetscOffset a_i > >> PetscInt n > >> PetscReal,allocatable :: sdata(:) > >> > >> call VecGetSize(b,n,ierr) > >> CHKERRQ(ierr) > >> allocate(sdata(n)) > >> sdata=0.0_8 > >> call VecGetArray(b,a_v,a_i,ierr) > >> CHKERRQ(ierr) > >> sdata(1:n)=a_v(a_i+1:a_i+n) > >> call VecRestoreArray(b,a_v,a_i,ierr) > >> CHKERRQ(ierr) > >> print*,'rhs:',sdata > >> > >> call KSPSolve(ksp,b,x,ierr) > >> sdata=0.0_8 > >> call VecGetArray(x,a_v,a_i,ierr) > >> CHKERRQ(ierr) > >> sdata(1:n)=a_v(a_i+1:a_i+n) > >> call VecRestoreArray(b,a_v,a_i,ierr) > >> CHKERRQ(ierr) > >> print*,'solution:',sdata > >> > >> ! Check convergence > >> call KSPGetConvergedReason(ksp,ireason,ierr) > >> if(myrank==0)print*,'converges reason',myrank,ireason > >> call KSPGetIterationNumber(ksp,its,ierr) > >> if(myrank==0)print*,'Iterations:',its > >> deallocate(sdata) > >> end subroutine petsc_solve > >> > !------------------------------------------------------------------------------- > >> > >> subroutine petsc_finalize() > >> implicit none > >> > >> ! Free work space. > >> call VecDestroy(x,ierr) > >> call VecDestroy(u,ierr) > >> call VecDestroy(b,ierr) > >> call MatDestroy(A,ierr) > >> call KSPDestroy(ksp,ierr) > >> call PetscFinalize(ierr) > >> CHKERRQ(ierr) > >> > >> end subroutine petsc_finalize > >> > !------------------------------------------------------------------------------- > >> > >> end program testfs > >> > >> On Tue, Feb 2, 2016 at 4:54 PM, Matthew Knepley > wrote: > >>> On Tue, Feb 2, 2016 at 3:11 PM, Hom Nath Gharti > wrote: > >>>> > >>>> Thanks a lot. I will try. > >>> > >>> > >>> Also, if you send a small test case, we can run it and tell you the > problem. > >>> > >>> Matt > >>> > >>>> > >>>> Hom Nath > >>>> > >>>> On Tue, Feb 2, 2016 at 4:02 PM, Matthew Knepley > wrote: > >>>>> On Tue, Feb 2, 2016 at 3:01 PM, Hom Nath Gharti > > >>>>> wrote: > >>>>>> > >>>>>> Thanks again Matt. Unfortunately got the same errors with '0'. I > think > >>>>>> both are valid in Fortran. > >>>>> > >>>>> > >>>>> Then you will have to go in the debugger and see why its not > creating 4 > >>>>> splits, since this is exactly > >>>>> what it does in our tests. This is how all the SNES tests that I use > >>>>> work. I > >>>>> am sure its a Fortran > >>>>> problem. > >>>>> > >>>>> Thanks, > >>>>> > >>>>> Matt > >>>>> > >>>>>> > >>>>>> Hom Nath > >>>>>> > >>>>>> On Tue, Feb 2, 2016 at 3:42 PM, Matthew Knepley > >>>>>> wrote: > >>>>>>> On Tue, Feb 2, 2016 at 2:35 PM, Hom Nath Gharti < > hng.email at gmail.com> > >>>>>>> wrote: > >>>>>>>> > >>>>>>>> Thank you so much Matt. > >>>>>>>> > >>>>>>>> I now tried the following: > >>>>>>>> > >>>>>>>> ====================================================== > >>>>>>>> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) > >>>>>>>> call KSPGetPC(ksp,pc,ierr) > >>>>>>>> > >>>>>>>> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); > >>>>>>>> call ISDestroy(gdofu_is,ierr) > >>>>>>>> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); > >>>>>>>> call ISDestroy(gdofchi_is,ierr) > >>>>>>>> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); > >>>>>>>> call ISDestroy(gdofp_is,ierr) > >>>>>>>> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); > >>>>>>>> call ISDestroy(gdofphi_is,ierr) > >>>>>>>> > >>>>>>>> ! Amat changes in each time steps > >>>>>>>> call KSPSetOperators(ksp,Amat,Amat,ierr) !version >= 3.5.0 > >>>>>>>> > >>>>>>>> call KSPSolve(ksp,bvec,xvec,ierr) > >>>>>>>> ====================================================== > >>>>>>> > >>>>>>> > >>>>>>> I am guessing that "0" is not a valid string for your Fortran > >>>>>>> compiler. > >>>>>>> Are > >>>>>>> you sure > >>>>>>> its not single quotes '0'? > >>>>>>> > >>>>>>> Matt > >>>>>>> > >>>>>>>> > >>>>>>>> But I get the following error: > >>>>>>>> > >>>>>>>> [0]PETSC ERROR: --------------------- Error Message > >>>>>>>> -------------------------------------------------------------- > >>>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data > >>>>>>>> [0]PETSC ERROR: Unhandled case, must have at least two fields, > not 1 > >>>>>>>> [0]PETSC ERROR: See > >>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > >>>>>>>> shooting. > >>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > >>>>>>>> [0]PETSC ERROR: > >>>>>>>> /tigress/hgharti/gitwork/SPECFEM3D_GLOBEVSPP/./bin/xspecfem3D on a > >>>>>>>> arch-linux2-c-debug named tiger-r3c1n7 by hgharti Tue Feb 2 15: > >>>>>>>> 29:30 2016 > >>>>>>>> [0]PETSC ERROR: Configure options > >>>>>>>> > >>>>>>>> > >>>>>>>> > --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64/ > >>>>>>>> --with-cc=mpicc --with-cxx=mpicxx --wit > >>>>>>>> h-fc=mpif90 --with-mpiexec=mpiexec --with-debugging=1 > >>>>>>>> --download-scalapack --download-mumps --download-pastix > >>>>>>>> --download-superlu --download-superlu_dist --download-metis > >>>>>>>> --download-parmetis --download-ptscotch --download-hypre > >>>>>>>> [0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 469 in > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > >>>>>>>> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 486 in > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > >>>>>>>> [0]PETSC ERROR: #3 PCSetUp() line 983 in > >>>>>>>> > >>>>>>>> > >>>>>>>> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/interface/precon.c > >>>>>>>> [0]PETSC ERROR: #4 KSPSetUp() line 332 in > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c > >>>>>>>> [0]PETSC ERROR: #5 KSPSolve() line 546 in > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c > >>>>>>>> forrtl: error (76): Abort trap signal > >>>>>>>> > >>>>>>>> Am I missing something? > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Hom Nath > >>>>>>>> > >>>>>>>> On Tue, Feb 2, 2016 at 3:24 PM, Matthew Knepley < > knepley at gmail.com> > >>>>>>>> wrote: > >>>>>>>>> On Tue, Feb 2, 2016 at 2:20 PM, Hom Nath Gharti > >>>>>>>>> > >>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>> Hi Matt, hi all, > >>>>>>>>>> > >>>>>>>>>> I am trying to use PCFIELDSPLIT for my solver which consists of > 4 > >>>>>>>>>> different variables, namely, u (displacement vector), \chi > >>>>>>>>>> (displacement potential), p(pressure), and \phi (gravity > >>>>>>>>>> potential). > >>>>>>>>>> > >>>>>>>>>> My code segment looks like the following: > >>>>>>>>>> ====================================================== > >>>>>>>>>> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) > >>>>>>>>>> call KSPGetPC(ksp,pc,ierr) > >>>>>>>>>> nsplit=4 > >>>>>>>>>> call PCFieldSplitSetBlockSize(pc,nsplit,ierr); > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> You do not need the statement above. > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); > >>>>>>>>>> call ISDestroy(gdofu_is,ierr) > >>>>>>>>>> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); > >>>>>>>>>> call ISDestroy(gdofchi_is,ierr) > >>>>>>>>>> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); > >>>>>>>>>> call ISDestroy(gdofp_is,ierr) > >>>>>>>>>> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); > >>>>>>>>>> call ISDestroy(gdofphi_is,ierr) > >>>>>>>>>> > >>>>>>>>>> call PCFieldSplitGetSubKSP(pc,nsplit,subksp,ierr); > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> These SetOperators() calls are wrong. I am not sure why you want > >>>>>>>>> them > >>>>>>>>> here. > >>>>>>>>> Also, that means you do not need the call above. > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> > >>>>>>>>> Matt > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> call KSPSetOperators(subksp(1),Amat,Amat,ierr); > >>>>>>>>>> call KSPSetOperators(subksp(2),Amat,Amat,ierr); > >>>>>>>>>> call KSPSetOperators(subksp(3),Amat,Amat,ierr); > >>>>>>>>>> call KSPSetOperators(subksp(4),Amat,Amat,ierr); > >>>>>>>>>> > >>>>>>>>>> call KSPSolve(ksp,bvec,xvec,ierr) > >>>>>>>>>> ====================================================== > >>>>>>>>>> > >>>>>>>>>> But I am getting the following error: > >>>>>>>>>> [79]PETSC ERROR: Null argument, when expecting valid pointer > >>>>>>>>>> [79]PETSC ERROR: Null Object: Parameter # 1 > >>>>>>>>>> [79]PETSC ERROR: #1 KSPSetOperators() line 536 in > >>>>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interf > >>>>>>>>>> > >>>>>>>>>> It looks like I am doing something wrong in "call > >>>>>>>>>> KSPSetOperators" > >>>>>>>>>> but > >>>>>>>>>> I could not figure that out. > >>>>>>>>>> > >>>>>>>>>> Could anybody help me to fix this problem? I looked into almost > >>>>>>>>>> all > >>>>>>>>>> related examples but I could not really figure out the correct > >>>>>>>>>> steps > >>>>>>>>>> after "call PCFieldSplitSetIS". > >>>>>>>>>> > >>>>>>>>>> Any help will be greatly appreciated. > >>>>>>>>>> > >>>>>>>>>> Best, > >>>>>>>>>> Hom nath > >>>>>>>>>> > >>>>>>>>>> On Sun, Jan 24, 2016 at 7:14 PM, Hom Nath Gharti > >>>>>>>>>> > >>>>>>>>>> wrote: > >>>>>>>>>>> Thank you so much Matt! I will try. > >>>>>>>>>>> > >>>>>>>>>>> Hom Nath > >>>>>>>>>>> > >>>>>>>>>>> On Sun, Jan 24, 2016 at 6:26 AM, Matthew Knepley > >>>>>>>>>>> > >>>>>>>>>>> wrote: > >>>>>>>>>>>> On Fri, Jan 22, 2016 at 2:19 PM, Hom Nath Gharti > >>>>>>>>>>>> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> Dear all, > >>>>>>>>>>>>> > >>>>>>>>>>>>> I am new to PcFieldSplit. > >>>>>>>>>>>>> > >>>>>>>>>>>>> I have a matrix formed using MATMPIAIJ. Is it possible to use > >>>>>>>>>>>>> PCFIELDSPLIT operations in this type of matrix? Or does it > >>>>>>>>>>>>> have > >>>>>>>>>>>>> to > >>>>>>>>>>>>> be > >>>>>>>>>>>>> MATMPIBIJ or MATNEST format? > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Yes, you can split AIJ. > >>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> If possible for MATMPIAIJ, could anybody provide me a simple > >>>>>>>>>>>>> example > >>>>>>>>>>>>> or few steps? Variables in the equations are displacement > >>>>>>>>>>>>> vector, > >>>>>>>>>>>>> scalar potential and pressure. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> If you do not have a collocated discretization, then you have > >>>>>>>>>>>> to > >>>>>>>>>>>> use > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks, > >>>>>>>>>>>> > >>>>>>>>>>>> Matt > >>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks for help. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Hom Nath > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> -- > >>>>>>>>>>>> What most experimenters take for granted before they begin > >>>>>>>>>>>> their > >>>>>>>>>>>> experiments > >>>>>>>>>>>> is infinitely more interesting than any results to which their > >>>>>>>>>>>> experiments > >>>>>>>>>>>> lead. > >>>>>>>>>>>> -- Norbert Wiener > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> What most experimenters take for granted before they begin their > >>>>>>>>> experiments > >>>>>>>>> is infinitely more interesting than any results to which their > >>>>>>>>> experiments > >>>>>>>>> lead. > >>>>>>>>> -- Norbert Wiener > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> What most experimenters take for granted before they begin their > >>>>>>> experiments > >>>>>>> is infinitely more interesting than any results to which their > >>>>>>> experiments > >>>>>>> lead. > >>>>>>> -- Norbert Wiener > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> What most experimenters take for granted before they begin their > >>>>> experiments > >>>>> is infinitely more interesting than any results to which their > >>>>> experiments > >>>>> lead. > >>>>> -- Norbert Wiener > >>> > >>> > >>> > >>> > >>> -- > >>> What most experimenters take for granted before they begin their > experiments > >>> is infinitely more interesting than any results to which their > experiments > >>> lead. > >>> -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue Feb 9 09:30:58 2016 From: jed at jedbrown.org (Jed Brown) Date: Tue, 09 Feb 2016 08:30:58 -0700 Subject: [petsc-users] combine single and double precision In-Reply-To: References: Message-ID: <87k2md99zh.fsf@jedbrown.org> albert guti?rrez writes: > Hello everyone, > > At times it can be useful to combine different data types in the same > problem. For example, there are parts of an algorithm which may be memory > bounded (requiring float) and operations where more precision is needed > (requiring double or float_128). > > Can single and double precision be combined in PETSc? Unfortunately not in a supported way. (It's possible to link different versions of the library into the same application, but it's a mess and not supported.) > I know the data type is set in the configuration process using the > parameter "--with-precision=TYPE". It looks like having single and double > precision at the same time will require the modification of the source code > of the library. Is there an easy way to implement this feature? Current languages are really bad at this and the semantics of composing objects of different types are ambiguous unless types are always promoted to the largest, which erases much of the memory benefit in many algorithms. But if we knew a clean way to do it, we would. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From hng.email at gmail.com Tue Feb 9 09:31:36 2016 From: hng.email at gmail.com (Hom Nath Gharti) Date: Tue, 9 Feb 2016 10:31:36 -0500 Subject: [petsc-users] PCFIELDSPLIT question In-Reply-To: References: Message-ID: Thanks a lot Matt! Were you referring to http://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex62.c.html ? I do not see any statements related to PCFieldSplit there. Am I missing something here? Thanks, Hom Nath On Tue, Feb 9, 2016 at 10:19 AM, Matthew Knepley wrote: > On Tue, Feb 9, 2016 at 9:10 AM, Hom Nath Gharti wrote: >> >> Thank you so much Barry! >> >> For my small test case, with -pc_fieldsplit_block_size 4, the program >> runs, although the answer was not correct. At least now I get >> something to look upon. I am using PCFieldSplitSetIS to set the >> fields. Do I still need to use -pc_fieldsplit_block_size? > > > No, this is only for co-located discretizations. > >> >> In my case each grid point may have different variable sets. For >> example, the grid point in the solid region has displacement and >> gravity potential: ux, uy, uz, and \phi. Whereas the grid point in the >> fluid region has scalar potential, pressure and gravity potential: >> \chi, p, and \phi. And the solid-fluid interface has all of them. Do >> you think I can still use PCFIELDSPLIT in this situation? > > > We have examples, like SNES ex62, which do exactly this. > > Thanks, > > Matt > >> >> Best, >> Hom Nath >> >> >> >> On Mon, Feb 8, 2016 at 2:27 PM, Barry Smith wrote: >> > >> > >> > In this case you need to provide two pieces of information to the >> > PCFIELDSPLIT. What we call the "block size" or bs which is the number of >> > "basic fields" in the problem. For example if at each grid point you have >> > x-velocity, y-velocity, and pressure the block size is 3. The second is you >> > need to tell PCFIELDSPLIT what "basic fields" are in each split you want to >> > use. >> > >> > In your case you can do this with -pc_fieldsplit_block_size 4. >> > (Note if you use a DM to define your problem the PCFIELDSPLIT will >> > automatically get the bs from the DM so you do not need to provide it. >> > Similarly if you set a block size on your Mat the PCFIELDSPLIT will use that >> > so you don't need to set it). >> > >> > Then use -pc_fieldsplit_0_fields 0,1 to indicate your first split is >> > the 0 and 1 basic fields and -pc_fieldsplit_1_fields 2,3 to indicate the >> > second split is the 2 and 3 basic fields. >> > (By default if you do not provide this then PCFIELDSPLIT will use bs >> > splits (4 in your case) one for each basic field. >> > >> > Barry >> > >> >> On Feb 8, 2016, at 11:20 AM, Hom Nath Gharti >> >> wrote: >> >> >> >> Hi Matt, Hi all, >> >> >> >> I am trying to have some feel for PCFIELDSPLIT testing on a very small >> >> matrix (attached below). I have 4 degrees of freedoms. I use 4 >> >> processors. I want to split 4 dofs into two fields each having two >> >> dofs. I don't know whether this my be a problem for petsc. When I use >> >> the command: >> >> srun -n 4 ./xtestfs -ksp_monitor -ksp_view >> >> >> >> It runs fine. >> >> >> >> But when I want to use field split options using the command: >> >> srun -n 4 ./xtestfs -ksp_monitor -ksp_type fgmres -pc_type fieldsplit >> >> -fieldsplit_0_ksp_type gmres -fieldsplit_0_pc_type bjacobi >> >> -fieldsplit_1_pc_type jacobi >> >> >> >> I get the following error message: >> >> [0]PETSC ERROR: Petsc has generated inconsistent data >> >> [0]PETSC ERROR: Unhandled case, must have at least two fields, not 1 >> >> [0]PETSC ERROR: See >> >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >> >> shooting. >> >> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >> >> [0]PETSC ERROR: /tigress/hgharti/test/testfs/./xtestfs on a >> >> arch-linux2-c-debug named tiger-r11n1 by hgharti Mon Feb 8 11:40:03 >> >> 2016 >> >> [0]PETSC ERROR: Configure options >> >> >> >> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64/ >> >> --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 >> >> --with-mpiexec=mpiexec --with-debugging=1 --download-scalapack >> >> --download-mumps --download-pastix --download-superlu >> >> --download-superlu_dist --download-metis --download-parmetis >> >> --download-ptscotch --download-hypre >> >> [0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 469 in >> >> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> >> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 486 in >> >> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> >> [0]PETSC ERROR: #3 PCSetUp() line 983 in >> >> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/interface/precon.c >> >> [0]PETSC ERROR: #4 KSPSetUp() line 332 in >> >> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >> >> [0]PETSC ERROR: #5 KSPSolve() line 546 in >> >> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >> >> WARNING! There are options you set that were not used! >> >> WARNING! could be spelling mistake, etc! >> >> Option left: name:-fieldsplit_0_ksp_type value: gmres >> >> Option left: name:-fieldsplit_0_pc_type value: bjacobi >> >> Option left: name:-fieldsplit_1_pc_type value: jacobi >> >> forrtl: error (76): Abort trap signal >> >> >> >> I tried several trials but could not fix the problem. Is it the >> >> FORTRAN problem or am I doing something wrong? Any suggestions would >> >> be greatly appreciated. >> >> For your information I use: >> >> PETSc-3.6.3 >> >> intel-16.0 compiler >> >> intel-mpi-5.1.1 >> >> >> >> Program >> >> >> >> >> >> Best, >> >> Hom Nath >> >> >> >> ! simple test case for PCFIELDSPLIT >> >> ! >> >> ----------------------------------------------------------------------- >> >> program testfs >> >> implicit none >> >> #include "petsc/finclude/petscsys.h" >> >> #include "petsc/finclude/petscvec.h" >> >> #include "petsc/finclude/petscvec.h90" >> >> #include "petsc/finclude/petscmat.h" >> >> #include "petsc/finclude/petscksp.h" >> >> #include "petsc/finclude/petscpc.h" >> >> #include "petsc/finclude/petscviewer.h" >> >> #include "petsc/finclude/petscviewer.h90" >> >> >> >> Vec x,b,u >> >> Mat A >> >> KSP ksp >> >> PC pc >> >> PetscErrorCode ierr >> >> PetscBool flg >> >> PetscInt errcode,its,maxitr,myrank,nproc >> >> PetscInt nedof,nelmt,ndof,nenod,ngdof,fixdof,nsparse,neq >> >> PetscInt,allocatable :: krow_sparse(:),ggdof_elmt(:,:), & >> >> inode_elmt(:,:) >> >> >> >> PetscReal,allocatable :: storef(:,:),storekmat(:,:,:) >> >> PetscInt gdof0(2),gdof1(2) >> >> >> >> ! initialize MPI >> >> >> >> call MPI_INIT(errcode) >> >> if(errcode /= 0)stop 'ERROR: cannot initialize MPI!' >> >> call MPI_COMM_RANK(MPI_COMM_WORLD,myrank,errcode) >> >> if(errcode /= 0)stop 'ERROR: cannot find processor ID' >> >> call MPI_COMM_SIZE(MPI_COMM_WORLD,nproc,errcode) >> >> if(errcode /= 0)stop 'ERROR: cannot find number of processors!' >> >> >> >> ! define some parameters >> >> ! 1D model with 4 elements. Each elements consits of 2 nodes. Node 0 is >> >> fixed. >> >> ! 1-----------2-----------3-----------4-----------5 >> >> nelmt=1 ! per processor >> >> nenod=2 ! number of nodes per element >> >> nedof=2 ! number of DOFs per element >> >> ndof=2 ! number of DOFs >> >> ngdof=4 ! total number of global DOFs >> >> fixdof=1 >> >> >> >> if(myrank==0)then >> >> neq=1 ! number of equations >> >> nsparse=1 >> >> else >> >> neq=2 ! number of equations >> >> nsparse=4 >> >> endif >> >> >> >> allocate(storef(nedof,nelmt),storekmat(nedof,nedof,nelmt), >> >> & >> >> ggdof_elmt(nedof,nelmt),inode_elmt(nenod,nelmt),krow_sparse(nsparse)) >> >> >> >> storef=0.0_8 ! local RHS vector >> >> storekmat=0.0_8 ! local stiffness matrix >> >> if(myrank==0)then >> >> krow_sparse(:)=1 >> >> storef(:,1)=(/12.5000_8, 12.5000_8/) >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & >> >> 0.0166666667_8, >> >> 1.2333333332_8/),(/nedof,nedof/)) >> >> inode_elmt(:,1) = (/1,2/) >> >> ggdof_elmt(:,1) = (/0,1/) ! 0 because left node is fixed >> >> elseif(myrank==1)then >> >> krow_sparse(:)=(/1,1,2,2/) >> >> storef(:,1)=(/12.5000_8, 12.5000_8/) >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & >> >> 0.0166666667_8, >> >> 1.2333333332_8/),(/nedof,nedof/)) >> >> inode_elmt(:,1) = (/1,2/) >> >> ggdof_elmt(:,1) = (/1,2/) >> >> elseif(myrank==2)then >> >> krow_sparse(:)=(/1,1,2,2/) >> >> storef(:,1)=(/12.5000_8, 12.5000_8/) >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & >> >> 0.0166666667_8, >> >> 1.2333333332_8/),(/nedof,nedof/)) >> >> inode_elmt(:,1) = (/1,2/) >> >> ggdof_elmt(:,1) = (/2,3/) >> >> elseif(myrank==3)then >> >> krow_sparse(:)=(/1,1,2,2/) >> >> storef(:,1)=(/12.5000_8, 22.5000_8/) >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & >> >> 0.0166666667_8, >> >> 1.2333333332_8/),(/nedof,nedof/)) >> >> inode_elmt(:,1) = (/1,2/) >> >> ggdof_elmt(:,1) = (/3,4/) >> >> endif >> >> >> >> call petsc_initialize >> >> call petsc_set_matrix >> >> call petsc_set_vector >> >> call petsc_solve >> >> call petsc_finalize >> >> >> >> ! deallocate >> >> deallocate(storef,storekmat,ggdof_elmt,inode_elmt) >> >> deallocate(krow_sparse) >> >> contains >> >> >> >> !------------------------------------------------------------------------------- >> >> subroutine petsc_initialize() >> >> implicit none >> >> PetscInt :: istart,iend >> >> PetscInt :: nzeros_max,nzeros_min >> >> PetscReal :: atol,dtol,rtol >> >> PetscInt,allocatable :: nzeros(:) >> >> IS gdof0_is,gdof1_is >> >> >> >> call PetscInitialize(PETSC_NULL_CHARACTER,ierr) >> >> call PetscOptionsGetInt(PETSC_NULL_CHARACTER,'-n',ngdof,flg,ierr) >> >> >> >> ! count number of nonzeros per row >> >> allocate(nzeros(neq)) >> >> nzeros=0 >> >> nzeros(krow_sparse)=nzeros(krow_sparse)+1 >> >> nzeros_max=maxval(nzeros) >> >> nzeros_min=minval(nzeros) >> >> >> >> ! create matrix >> >> call MatCreate(PETSC_COMM_WORLD,A,ierr) >> >> call MatSetType(A,MATMPIAIJ,ierr) >> >> CHKERRQ(ierr) >> >> call MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,ngdof,ngdof,ierr) >> >> CHKERRQ(ierr) >> >> ! preallocation >> >> call >> >> MatMPIAIJSetPreallocation(A,nzeros_max,PETSC_NULL_INTEGER,nzeros_max, & >> >> PETSC_NULL_INTEGER,ierr) >> >> CHKERRQ(ierr) >> >> >> >> call MatGetOwnershipRange(A,istart,iend,ierr) >> >> CHKERRQ(ierr) >> >> print*,'ownership:',myrank,istart,iend >> >> ! create vector >> >> call VecCreate(PETSC_COMM_WORLD,x,ierr) >> >> call VecSetSizes(x,PETSC_DECIDE,ngdof,ierr) >> >> call VecSetType(x,VECMPI,ierr) >> >> call VecDuplicate(x,b,ierr) >> >> call VecDuplicate(x,u,ierr) >> >> >> >> >> >> !******************************************************************************* >> >> ! get IS for split fields >> >> gdof0=(/0,1/) >> >> gdof1=(/2,3/) >> >> call >> >> ISCreateGeneral(PETSC_COMM_WORLD,2,gdof0,PETSC_COPY_VALUES,gdof0_is,ierr) >> >> CHKERRQ(ierr) >> >> call >> >> ISCreateGeneral(PETSC_COMM_WORLD,2,gdof1,PETSC_COPY_VALUES,gdof1_is,ierr) >> >> CHKERRQ(ierr) >> >> >> >> !******************************************************************************* >> >> >> >> ! Create linear solver context >> >> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >> >> call KSPSetOperators(ksp,A,A,ierr) >> >> >> >> call KSPGetPC(ksp,pc,ierr) >> >> CHKERRQ(ierr) >> >> >> >> >> >> !******************************************************************************* >> >> ! split PC >> >> call PCFieldSplitSetIS(pc,"0",gdof0_is,ierr); >> >> CHKERRQ(ierr) >> >> call ISDestroy(gdof0_is,ierr) >> >> CHKERRQ(ierr) >> >> call PCFieldSplitSetIS(pc,"1",gdof1_is,ierr); >> >> CHKERRQ(ierr) >> >> call ISDestroy(gdof1_is,ierr) >> >> CHKERRQ(ierr) >> >> >> >> !******************************************************************************* >> >> >> >> rtol = 1.0d-6 >> >> atol = 1.0d-6 >> >> dtol = 1.0d10 >> >> maxitr = 1000 >> >> call KSPSetTolerances(ksp,rtol,atol,dtol,maxitr,ierr) >> >> CHKERRQ(ierr) >> >> call KSPSetFromOptions(ksp,ierr) >> >> end subroutine petsc_initialize >> >> >> >> !------------------------------------------------------------------------------- >> >> >> >> subroutine petsc_set_matrix() >> >> >> >> implicit none >> >> integer :: i,i_elmt,j,ncount >> >> integer :: ggdof(NEDOF),idof(NEDOF),igdof(NEDOF) >> >> >> >> ! Set and assemble matrix. >> >> call MatZeroEntries(A,ierr) >> >> CHKERRQ(ierr) >> >> do i_elmt=1,nelmt >> >> ggdof(:)=ggdof_elmt(:,i_elmt) >> >> ggdof(:)=ggdof(:)-1 ! petsc index starts from 0 >> >> ncount=0; idof=-1; igdof=-1 >> >> do i=1,NEDOF >> >> do j=1,NEDOF >> >> if(ggdof(i).ge.0.and.ggdof(j).ge.0)then >> >> call MatSetValues(A,1,ggdof(i),1,ggdof(j),storekmat(i,j,i_elmt), >> >> & >> >> ADD_VALUES,ierr) >> >> CHKERRQ(ierr) >> >> endif >> >> enddo >> >> enddo >> >> enddo >> >> call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) >> >> call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) >> >> if(myrank==0)print*,'matrix setting & assembly complete!' >> >> >> >> end subroutine petsc_set_matrix >> >> >> >> !------------------------------------------------------------------------------- >> >> >> >> subroutine petsc_set_vector() >> >> implicit none >> >> PetscScalar zero >> >> integer :: i,i_elmt,ncount >> >> integer :: ggdof(NEDOF),idof(NEDOF),igdof(NEDOF) >> >> >> >> ! set vector >> >> zero=0.0_8 >> >> call VecSet(b,zero,ierr) >> >> do i_elmt=1,nelmt >> >> ggdof(:)=ggdof_elmt(:,i_elmt) >> >> ggdof(:)=ggdof(:)-1 ! petsc index starts from 0 >> >> ncount=0; idof=-1; igdof=-1 >> >> do i=1,NEDOF >> >> if(ggdof(i).ge.0)then >> >> call VecSetValues(b,1,ggdof(i),storef(i,i_elmt),ADD_VALUES,ierr); >> >> CHKERRQ(ierr) >> >> endif >> >> enddo >> >> enddo >> >> >> >> ! assemble vector >> >> call VecAssemblyBegin(b,ierr) >> >> call VecAssemblyEnd(b,ierr) >> >> if(myrank==0)print*,'vector setting & assembly complete!' >> >> >> >> end subroutine petsc_set_vector >> >> >> >> !------------------------------------------------------------------------------- >> >> >> >> subroutine petsc_solve() >> >> implicit none >> >> PetscInt ireason >> >> PetscScalar a_v(1) >> >> PetscOffset a_i >> >> PetscInt n >> >> PetscReal,allocatable :: sdata(:) >> >> >> >> call VecGetSize(b,n,ierr) >> >> CHKERRQ(ierr) >> >> allocate(sdata(n)) >> >> sdata=0.0_8 >> >> call VecGetArray(b,a_v,a_i,ierr) >> >> CHKERRQ(ierr) >> >> sdata(1:n)=a_v(a_i+1:a_i+n) >> >> call VecRestoreArray(b,a_v,a_i,ierr) >> >> CHKERRQ(ierr) >> >> print*,'rhs:',sdata >> >> >> >> call KSPSolve(ksp,b,x,ierr) >> >> sdata=0.0_8 >> >> call VecGetArray(x,a_v,a_i,ierr) >> >> CHKERRQ(ierr) >> >> sdata(1:n)=a_v(a_i+1:a_i+n) >> >> call VecRestoreArray(b,a_v,a_i,ierr) >> >> CHKERRQ(ierr) >> >> print*,'solution:',sdata >> >> >> >> ! Check convergence >> >> call KSPGetConvergedReason(ksp,ireason,ierr) >> >> if(myrank==0)print*,'converges reason',myrank,ireason >> >> call KSPGetIterationNumber(ksp,its,ierr) >> >> if(myrank==0)print*,'Iterations:',its >> >> deallocate(sdata) >> >> end subroutine petsc_solve >> >> >> >> !------------------------------------------------------------------------------- >> >> >> >> subroutine petsc_finalize() >> >> implicit none >> >> >> >> ! Free work space. >> >> call VecDestroy(x,ierr) >> >> call VecDestroy(u,ierr) >> >> call VecDestroy(b,ierr) >> >> call MatDestroy(A,ierr) >> >> call KSPDestroy(ksp,ierr) >> >> call PetscFinalize(ierr) >> >> CHKERRQ(ierr) >> >> >> >> end subroutine petsc_finalize >> >> >> >> !------------------------------------------------------------------------------- >> >> >> >> end program testfs >> >> >> >> On Tue, Feb 2, 2016 at 4:54 PM, Matthew Knepley >> >> wrote: >> >>> On Tue, Feb 2, 2016 at 3:11 PM, Hom Nath Gharti >> >>> wrote: >> >>>> >> >>>> Thanks a lot. I will try. >> >>> >> >>> >> >>> Also, if you send a small test case, we can run it and tell you the >> >>> problem. >> >>> >> >>> Matt >> >>> >> >>>> >> >>>> Hom Nath >> >>>> >> >>>> On Tue, Feb 2, 2016 at 4:02 PM, Matthew Knepley >> >>>> wrote: >> >>>>> On Tue, Feb 2, 2016 at 3:01 PM, Hom Nath Gharti >> >>>>> >> >>>>> wrote: >> >>>>>> >> >>>>>> Thanks again Matt. Unfortunately got the same errors with '0'. I >> >>>>>> think >> >>>>>> both are valid in Fortran. >> >>>>> >> >>>>> >> >>>>> Then you will have to go in the debugger and see why its not >> >>>>> creating 4 >> >>>>> splits, since this is exactly >> >>>>> what it does in our tests. This is how all the SNES tests that I use >> >>>>> work. I >> >>>>> am sure its a Fortran >> >>>>> problem. >> >>>>> >> >>>>> Thanks, >> >>>>> >> >>>>> Matt >> >>>>> >> >>>>>> >> >>>>>> Hom Nath >> >>>>>> >> >>>>>> On Tue, Feb 2, 2016 at 3:42 PM, Matthew Knepley >> >>>>>> wrote: >> >>>>>>> On Tue, Feb 2, 2016 at 2:35 PM, Hom Nath Gharti >> >>>>>>> >> >>>>>>> wrote: >> >>>>>>>> >> >>>>>>>> Thank you so much Matt. >> >>>>>>>> >> >>>>>>>> I now tried the following: >> >>>>>>>> >> >>>>>>>> ====================================================== >> >>>>>>>> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >> >>>>>>>> call KSPGetPC(ksp,pc,ierr) >> >>>>>>>> >> >>>>>>>> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); >> >>>>>>>> call ISDestroy(gdofu_is,ierr) >> >>>>>>>> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); >> >>>>>>>> call ISDestroy(gdofchi_is,ierr) >> >>>>>>>> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); >> >>>>>>>> call ISDestroy(gdofp_is,ierr) >> >>>>>>>> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); >> >>>>>>>> call ISDestroy(gdofphi_is,ierr) >> >>>>>>>> >> >>>>>>>> ! Amat changes in each time steps >> >>>>>>>> call KSPSetOperators(ksp,Amat,Amat,ierr) !version >= 3.5.0 >> >>>>>>>> >> >>>>>>>> call KSPSolve(ksp,bvec,xvec,ierr) >> >>>>>>>> ====================================================== >> >>>>>>> >> >>>>>>> >> >>>>>>> I am guessing that "0" is not a valid string for your Fortran >> >>>>>>> compiler. >> >>>>>>> Are >> >>>>>>> you sure >> >>>>>>> its not single quotes '0'? >> >>>>>>> >> >>>>>>> Matt >> >>>>>>> >> >>>>>>>> >> >>>>>>>> But I get the following error: >> >>>>>>>> >> >>>>>>>> [0]PETSC ERROR: --------------------- Error Message >> >>>>>>>> -------------------------------------------------------------- >> >>>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >> >>>>>>>> [0]PETSC ERROR: Unhandled case, must have at least two fields, >> >>>>>>>> not 1 >> >>>>>>>> [0]PETSC ERROR: See >> >>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >> >>>>>>>> shooting. >> >>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >> >>>>>>>> [0]PETSC ERROR: >> >>>>>>>> /tigress/hgharti/gitwork/SPECFEM3D_GLOBEVSPP/./bin/xspecfem3D on >> >>>>>>>> a >> >>>>>>>> arch-linux2-c-debug named tiger-r3c1n7 by hgharti Tue Feb 2 15: >> >>>>>>>> 29:30 2016 >> >>>>>>>> [0]PETSC ERROR: Configure options >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64/ >> >>>>>>>> --with-cc=mpicc --with-cxx=mpicxx --wit >> >>>>>>>> h-fc=mpif90 --with-mpiexec=mpiexec --with-debugging=1 >> >>>>>>>> --download-scalapack --download-mumps --download-pastix >> >>>>>>>> --download-superlu --download-superlu_dist --download-metis >> >>>>>>>> --download-parmetis --download-ptscotch --download-hypre >> >>>>>>>> [0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 469 in >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> >>>>>>>> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 486 in >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> >>>>>>>> [0]PETSC ERROR: #3 PCSetUp() line 983 in >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/interface/precon.c >> >>>>>>>> [0]PETSC ERROR: #4 KSPSetUp() line 332 in >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >> >>>>>>>> [0]PETSC ERROR: #5 KSPSolve() line 546 in >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >> >>>>>>>> forrtl: error (76): Abort trap signal >> >>>>>>>> >> >>>>>>>> Am I missing something? >> >>>>>>>> >> >>>>>>>> Thanks, >> >>>>>>>> Hom Nath >> >>>>>>>> >> >>>>>>>> On Tue, Feb 2, 2016 at 3:24 PM, Matthew Knepley >> >>>>>>>> >> >>>>>>>> wrote: >> >>>>>>>>> On Tue, Feb 2, 2016 at 2:20 PM, Hom Nath Gharti >> >>>>>>>>> >> >>>>>>>>> wrote: >> >>>>>>>>>> >> >>>>>>>>>> Hi Matt, hi all, >> >>>>>>>>>> >> >>>>>>>>>> I am trying to use PCFIELDSPLIT for my solver which consists of >> >>>>>>>>>> 4 >> >>>>>>>>>> different variables, namely, u (displacement vector), \chi >> >>>>>>>>>> (displacement potential), p(pressure), and \phi (gravity >> >>>>>>>>>> potential). >> >>>>>>>>>> >> >>>>>>>>>> My code segment looks like the following: >> >>>>>>>>>> ====================================================== >> >>>>>>>>>> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >> >>>>>>>>>> call KSPGetPC(ksp,pc,ierr) >> >>>>>>>>>> nsplit=4 >> >>>>>>>>>> call PCFieldSplitSetBlockSize(pc,nsplit,ierr); >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> You do not need the statement above. >> >>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); >> >>>>>>>>>> call ISDestroy(gdofu_is,ierr) >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); >> >>>>>>>>>> call ISDestroy(gdofchi_is,ierr) >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); >> >>>>>>>>>> call ISDestroy(gdofp_is,ierr) >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); >> >>>>>>>>>> call ISDestroy(gdofphi_is,ierr) >> >>>>>>>>>> >> >>>>>>>>>> call PCFieldSplitGetSubKSP(pc,nsplit,subksp,ierr); >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> These SetOperators() calls are wrong. I am not sure why you want >> >>>>>>>>> them >> >>>>>>>>> here. >> >>>>>>>>> Also, that means you do not need the call above. >> >>>>>>>>> >> >>>>>>>>> Thanks, >> >>>>>>>>> >> >>>>>>>>> Matt >> >>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> call KSPSetOperators(subksp(1),Amat,Amat,ierr); >> >>>>>>>>>> call KSPSetOperators(subksp(2),Amat,Amat,ierr); >> >>>>>>>>>> call KSPSetOperators(subksp(3),Amat,Amat,ierr); >> >>>>>>>>>> call KSPSetOperators(subksp(4),Amat,Amat,ierr); >> >>>>>>>>>> >> >>>>>>>>>> call KSPSolve(ksp,bvec,xvec,ierr) >> >>>>>>>>>> ====================================================== >> >>>>>>>>>> >> >>>>>>>>>> But I am getting the following error: >> >>>>>>>>>> [79]PETSC ERROR: Null argument, when expecting valid pointer >> >>>>>>>>>> [79]PETSC ERROR: Null Object: Parameter # 1 >> >>>>>>>>>> [79]PETSC ERROR: #1 KSPSetOperators() line 536 in >> >>>>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interf >> >>>>>>>>>> >> >>>>>>>>>> It looks like I am doing something wrong in "call >> >>>>>>>>>> KSPSetOperators" >> >>>>>>>>>> but >> >>>>>>>>>> I could not figure that out. >> >>>>>>>>>> >> >>>>>>>>>> Could anybody help me to fix this problem? I looked into almost >> >>>>>>>>>> all >> >>>>>>>>>> related examples but I could not really figure out the correct >> >>>>>>>>>> steps >> >>>>>>>>>> after "call PCFieldSplitSetIS". >> >>>>>>>>>> >> >>>>>>>>>> Any help will be greatly appreciated. >> >>>>>>>>>> >> >>>>>>>>>> Best, >> >>>>>>>>>> Hom nath >> >>>>>>>>>> >> >>>>>>>>>> On Sun, Jan 24, 2016 at 7:14 PM, Hom Nath Gharti >> >>>>>>>>>> >> >>>>>>>>>> wrote: >> >>>>>>>>>>> Thank you so much Matt! I will try. >> >>>>>>>>>>> >> >>>>>>>>>>> Hom Nath >> >>>>>>>>>>> >> >>>>>>>>>>> On Sun, Jan 24, 2016 at 6:26 AM, Matthew Knepley >> >>>>>>>>>>> >> >>>>>>>>>>> wrote: >> >>>>>>>>>>>> On Fri, Jan 22, 2016 at 2:19 PM, Hom Nath Gharti >> >>>>>>>>>>>> >> >>>>>>>>>>>> wrote: >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Dear all, >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> I am new to PcFieldSplit. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> I have a matrix formed using MATMPIAIJ. Is it possible to >> >>>>>>>>>>>>> use >> >>>>>>>>>>>>> PCFIELDSPLIT operations in this type of matrix? Or does it >> >>>>>>>>>>>>> have >> >>>>>>>>>>>>> to >> >>>>>>>>>>>>> be >> >>>>>>>>>>>>> MATMPIBIJ or MATNEST format? >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> Yes, you can split AIJ. >> >>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> If possible for MATMPIAIJ, could anybody provide me a simple >> >>>>>>>>>>>>> example >> >>>>>>>>>>>>> or few steps? Variables in the equations are displacement >> >>>>>>>>>>>>> vector, >> >>>>>>>>>>>>> scalar potential and pressure. >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> If you do not have a collocated discretization, then you have >> >>>>>>>>>>>> to >> >>>>>>>>>>>> use >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html >> >>>>>>>>>>>> >> >>>>>>>>>>>> Thanks, >> >>>>>>>>>>>> >> >>>>>>>>>>>> Matt >> >>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Thanks for help. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Hom Nath >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> -- >> >>>>>>>>>>>> What most experimenters take for granted before they begin >> >>>>>>>>>>>> their >> >>>>>>>>>>>> experiments >> >>>>>>>>>>>> is infinitely more interesting than any results to which >> >>>>>>>>>>>> their >> >>>>>>>>>>>> experiments >> >>>>>>>>>>>> lead. >> >>>>>>>>>>>> -- Norbert Wiener >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> -- >> >>>>>>>>> What most experimenters take for granted before they begin their >> >>>>>>>>> experiments >> >>>>>>>>> is infinitely more interesting than any results to which their >> >>>>>>>>> experiments >> >>>>>>>>> lead. >> >>>>>>>>> -- Norbert Wiener >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> -- >> >>>>>>> What most experimenters take for granted before they begin their >> >>>>>>> experiments >> >>>>>>> is infinitely more interesting than any results to which their >> >>>>>>> experiments >> >>>>>>> lead. >> >>>>>>> -- Norbert Wiener >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> -- >> >>>>> What most experimenters take for granted before they begin their >> >>>>> experiments >> >>>>> is infinitely more interesting than any results to which their >> >>>>> experiments >> >>>>> lead. >> >>>>> -- Norbert Wiener >> >>> >> >>> >> >>> >> >>> >> >>> -- >> >>> What most experimenters take for granted before they begin their >> >>> experiments >> >>> is infinitely more interesting than any results to which their >> >>> experiments >> >>> lead. >> >>> -- Norbert Wiener >> > > > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener From knepley at gmail.com Tue Feb 9 09:36:15 2016 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 9 Feb 2016 09:36:15 -0600 Subject: [petsc-users] PCFIELDSPLIT question In-Reply-To: References: Message-ID: On Tue, Feb 9, 2016 at 9:31 AM, Hom Nath Gharti wrote: > Thanks a lot Matt! > > Were you referring to > > http://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex62.c.html > ? > > I do not see any statements related to PCFieldSplit there. Am I > missing something here? > Yes. The whole point is not to put anything specific to FIELDSPLIT in the code. It is all in options. For example, ./ex62 -run_type full -refinement_limit 0.00625 -bc_type dirichlet -interpolate 1 -vel_petscspace_order 2 -pres_petscspace_order 1 -ksp_type fgmres -ksp_gmres_restart 100 -ksp_rtol 1.0e-9 -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_schur_factorization_type upper -fieldsplit_pressure_ksp_rtol 1e-10 -fieldsplit_velocity_ksp_type gmres -fieldsplit_velocity_pc_type lu -fieldsplit_pressure_pc_type jacobi -snes_monitor_short -ksp_monitor_short -snes_converged_reason -ksp_converged_reason -snes_view -show_solution 0 The reason that it works is that the DM calls PCFIeldSplitSetIS() for each field in the DM. Thanks, Matt > Thanks, > Hom Nath > > On Tue, Feb 9, 2016 at 10:19 AM, Matthew Knepley > wrote: > > On Tue, Feb 9, 2016 at 9:10 AM, Hom Nath Gharti > wrote: > >> > >> Thank you so much Barry! > >> > >> For my small test case, with -pc_fieldsplit_block_size 4, the program > >> runs, although the answer was not correct. At least now I get > >> something to look upon. I am using PCFieldSplitSetIS to set the > >> fields. Do I still need to use -pc_fieldsplit_block_size? > > > > > > No, this is only for co-located discretizations. > > > >> > >> In my case each grid point may have different variable sets. For > >> example, the grid point in the solid region has displacement and > >> gravity potential: ux, uy, uz, and \phi. Whereas the grid point in the > >> fluid region has scalar potential, pressure and gravity potential: > >> \chi, p, and \phi. And the solid-fluid interface has all of them. Do > >> you think I can still use PCFIELDSPLIT in this situation? > > > > > > We have examples, like SNES ex62, which do exactly this. > > > > Thanks, > > > > Matt > > > >> > >> Best, > >> Hom Nath > >> > >> > >> > >> On Mon, Feb 8, 2016 at 2:27 PM, Barry Smith wrote: > >> > > >> > > >> > In this case you need to provide two pieces of information to the > >> > PCFIELDSPLIT. What we call the "block size" or bs which is the number > of > >> > "basic fields" in the problem. For example if at each grid point you > have > >> > x-velocity, y-velocity, and pressure the block size is 3. The second > is you > >> > need to tell PCFIELDSPLIT what "basic fields" are in each split you > want to > >> > use. > >> > > >> > In your case you can do this with -pc_fieldsplit_block_size 4. > >> > (Note if you use a DM to define your problem the PCFIELDSPLIT will > >> > automatically get the bs from the DM so you do not need to provide it. > >> > Similarly if you set a block size on your Mat the PCFIELDSPLIT will > use that > >> > so you don't need to set it). > >> > > >> > Then use -pc_fieldsplit_0_fields 0,1 to indicate your first split > is > >> > the 0 and 1 basic fields and -pc_fieldsplit_1_fields 2,3 to indicate > the > >> > second split is the 2 and 3 basic fields. > >> > (By default if you do not provide this then PCFIELDSPLIT will use bs > >> > splits (4 in your case) one for each basic field. > >> > > >> > Barry > >> > > >> >> On Feb 8, 2016, at 11:20 AM, Hom Nath Gharti > >> >> wrote: > >> >> > >> >> Hi Matt, Hi all, > >> >> > >> >> I am trying to have some feel for PCFIELDSPLIT testing on a very > small > >> >> matrix (attached below). I have 4 degrees of freedoms. I use 4 > >> >> processors. I want to split 4 dofs into two fields each having two > >> >> dofs. I don't know whether this my be a problem for petsc. When I use > >> >> the command: > >> >> srun -n 4 ./xtestfs -ksp_monitor -ksp_view > >> >> > >> >> It runs fine. > >> >> > >> >> But when I want to use field split options using the command: > >> >> srun -n 4 ./xtestfs -ksp_monitor -ksp_type fgmres -pc_type fieldsplit > >> >> -fieldsplit_0_ksp_type gmres -fieldsplit_0_pc_type bjacobi > >> >> -fieldsplit_1_pc_type jacobi > >> >> > >> >> I get the following error message: > >> >> [0]PETSC ERROR: Petsc has generated inconsistent data > >> >> [0]PETSC ERROR: Unhandled case, must have at least two fields, not 1 > >> >> [0]PETSC ERROR: See > >> >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > >> >> shooting. > >> >> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > >> >> [0]PETSC ERROR: /tigress/hgharti/test/testfs/./xtestfs on a > >> >> arch-linux2-c-debug named tiger-r11n1 by hgharti Mon Feb 8 11:40:03 > >> >> 2016 > >> >> [0]PETSC ERROR: Configure options > >> >> > >> >> > --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64/ > >> >> --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > >> >> --with-mpiexec=mpiexec --with-debugging=1 --download-scalapack > >> >> --download-mumps --download-pastix --download-superlu > >> >> --download-superlu_dist --download-metis --download-parmetis > >> >> --download-ptscotch --download-hypre > >> >> [0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 469 in > >> >> > >> >> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > >> >> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 486 in > >> >> > >> >> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > >> >> [0]PETSC ERROR: #3 PCSetUp() line 983 in > >> >> > >> >> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/interface/precon.c > >> >> [0]PETSC ERROR: #4 KSPSetUp() line 332 in > >> >> > >> >> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c > >> >> [0]PETSC ERROR: #5 KSPSolve() line 546 in > >> >> > >> >> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c > >> >> WARNING! There are options you set that were not used! > >> >> WARNING! could be spelling mistake, etc! > >> >> Option left: name:-fieldsplit_0_ksp_type value: gmres > >> >> Option left: name:-fieldsplit_0_pc_type value: bjacobi > >> >> Option left: name:-fieldsplit_1_pc_type value: jacobi > >> >> forrtl: error (76): Abort trap signal > >> >> > >> >> I tried several trials but could not fix the problem. Is it the > >> >> FORTRAN problem or am I doing something wrong? Any suggestions would > >> >> be greatly appreciated. > >> >> For your information I use: > >> >> PETSc-3.6.3 > >> >> intel-16.0 compiler > >> >> intel-mpi-5.1.1 > >> >> > >> >> Program > >> >> > >> >> > >> >> Best, > >> >> Hom Nath > >> >> > >> >> ! simple test case for PCFIELDSPLIT > >> >> ! > >> >> > ----------------------------------------------------------------------- > >> >> program testfs > >> >> implicit none > >> >> #include "petsc/finclude/petscsys.h" > >> >> #include "petsc/finclude/petscvec.h" > >> >> #include "petsc/finclude/petscvec.h90" > >> >> #include "petsc/finclude/petscmat.h" > >> >> #include "petsc/finclude/petscksp.h" > >> >> #include "petsc/finclude/petscpc.h" > >> >> #include "petsc/finclude/petscviewer.h" > >> >> #include "petsc/finclude/petscviewer.h90" > >> >> > >> >> Vec x,b,u > >> >> Mat A > >> >> KSP ksp > >> >> PC pc > >> >> PetscErrorCode ierr > >> >> PetscBool flg > >> >> PetscInt errcode,its,maxitr,myrank,nproc > >> >> PetscInt nedof,nelmt,ndof,nenod,ngdof,fixdof,nsparse,neq > >> >> PetscInt,allocatable :: krow_sparse(:),ggdof_elmt(:,:), & > >> >> inode_elmt(:,:) > >> >> > >> >> PetscReal,allocatable :: storef(:,:),storekmat(:,:,:) > >> >> PetscInt gdof0(2),gdof1(2) > >> >> > >> >> ! initialize MPI > >> >> > >> >> call MPI_INIT(errcode) > >> >> if(errcode /= 0)stop 'ERROR: cannot initialize MPI!' > >> >> call MPI_COMM_RANK(MPI_COMM_WORLD,myrank,errcode) > >> >> if(errcode /= 0)stop 'ERROR: cannot find processor ID' > >> >> call MPI_COMM_SIZE(MPI_COMM_WORLD,nproc,errcode) > >> >> if(errcode /= 0)stop 'ERROR: cannot find number of processors!' > >> >> > >> >> ! define some parameters > >> >> ! 1D model with 4 elements. Each elements consits of 2 nodes. Node 0 > is > >> >> fixed. > >> >> ! 1-----------2-----------3-----------4-----------5 > >> >> nelmt=1 ! per processor > >> >> nenod=2 ! number of nodes per element > >> >> nedof=2 ! number of DOFs per element > >> >> ndof=2 ! number of DOFs > >> >> ngdof=4 ! total number of global DOFs > >> >> fixdof=1 > >> >> > >> >> if(myrank==0)then > >> >> neq=1 ! number of equations > >> >> nsparse=1 > >> >> else > >> >> neq=2 ! number of equations > >> >> nsparse=4 > >> >> endif > >> >> > >> >> allocate(storef(nedof,nelmt),storekmat(nedof,nedof,nelmt), > >> >> & > >> >> ggdof_elmt(nedof,nelmt),inode_elmt(nenod,nelmt),krow_sparse(nsparse)) > >> >> > >> >> storef=0.0_8 ! local RHS vector > >> >> storekmat=0.0_8 ! local stiffness matrix > >> >> if(myrank==0)then > >> >> krow_sparse(:)=1 > >> >> storef(:,1)=(/12.5000_8, 12.5000_8/) > >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & > >> >> 0.0166666667_8, > >> >> 1.2333333332_8/),(/nedof,nedof/)) > >> >> inode_elmt(:,1) = (/1,2/) > >> >> ggdof_elmt(:,1) = (/0,1/) ! 0 because left node is fixed > >> >> elseif(myrank==1)then > >> >> krow_sparse(:)=(/1,1,2,2/) > >> >> storef(:,1)=(/12.5000_8, 12.5000_8/) > >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & > >> >> 0.0166666667_8, > >> >> 1.2333333332_8/),(/nedof,nedof/)) > >> >> inode_elmt(:,1) = (/1,2/) > >> >> ggdof_elmt(:,1) = (/1,2/) > >> >> elseif(myrank==2)then > >> >> krow_sparse(:)=(/1,1,2,2/) > >> >> storef(:,1)=(/12.5000_8, 12.5000_8/) > >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & > >> >> 0.0166666667_8, > >> >> 1.2333333332_8/),(/nedof,nedof/)) > >> >> inode_elmt(:,1) = (/1,2/) > >> >> ggdof_elmt(:,1) = (/2,3/) > >> >> elseif(myrank==3)then > >> >> krow_sparse(:)=(/1,1,2,2/) > >> >> storef(:,1)=(/12.5000_8, 22.5000_8/) > >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & > >> >> 0.0166666667_8, > >> >> 1.2333333332_8/),(/nedof,nedof/)) > >> >> inode_elmt(:,1) = (/1,2/) > >> >> ggdof_elmt(:,1) = (/3,4/) > >> >> endif > >> >> > >> >> call petsc_initialize > >> >> call petsc_set_matrix > >> >> call petsc_set_vector > >> >> call petsc_solve > >> >> call petsc_finalize > >> >> > >> >> ! deallocate > >> >> deallocate(storef,storekmat,ggdof_elmt,inode_elmt) > >> >> deallocate(krow_sparse) > >> >> contains > >> >> > >> >> > !------------------------------------------------------------------------------- > >> >> subroutine petsc_initialize() > >> >> implicit none > >> >> PetscInt :: istart,iend > >> >> PetscInt :: nzeros_max,nzeros_min > >> >> PetscReal :: atol,dtol,rtol > >> >> PetscInt,allocatable :: nzeros(:) > >> >> IS gdof0_is,gdof1_is > >> >> > >> >> call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > >> >> call PetscOptionsGetInt(PETSC_NULL_CHARACTER,'-n',ngdof,flg,ierr) > >> >> > >> >> ! count number of nonzeros per row > >> >> allocate(nzeros(neq)) > >> >> nzeros=0 > >> >> nzeros(krow_sparse)=nzeros(krow_sparse)+1 > >> >> nzeros_max=maxval(nzeros) > >> >> nzeros_min=minval(nzeros) > >> >> > >> >> ! create matrix > >> >> call MatCreate(PETSC_COMM_WORLD,A,ierr) > >> >> call MatSetType(A,MATMPIAIJ,ierr) > >> >> CHKERRQ(ierr) > >> >> call MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,ngdof,ngdof,ierr) > >> >> CHKERRQ(ierr) > >> >> ! preallocation > >> >> call > >> >> > MatMPIAIJSetPreallocation(A,nzeros_max,PETSC_NULL_INTEGER,nzeros_max, & > >> >> PETSC_NULL_INTEGER,ierr) > >> >> CHKERRQ(ierr) > >> >> > >> >> call MatGetOwnershipRange(A,istart,iend,ierr) > >> >> CHKERRQ(ierr) > >> >> print*,'ownership:',myrank,istart,iend > >> >> ! create vector > >> >> call VecCreate(PETSC_COMM_WORLD,x,ierr) > >> >> call VecSetSizes(x,PETSC_DECIDE,ngdof,ierr) > >> >> call VecSetType(x,VECMPI,ierr) > >> >> call VecDuplicate(x,b,ierr) > >> >> call VecDuplicate(x,u,ierr) > >> >> > >> >> > >> >> > !******************************************************************************* > >> >> ! get IS for split fields > >> >> gdof0=(/0,1/) > >> >> gdof1=(/2,3/) > >> >> call > >> >> > ISCreateGeneral(PETSC_COMM_WORLD,2,gdof0,PETSC_COPY_VALUES,gdof0_is,ierr) > >> >> CHKERRQ(ierr) > >> >> call > >> >> > ISCreateGeneral(PETSC_COMM_WORLD,2,gdof1,PETSC_COPY_VALUES,gdof1_is,ierr) > >> >> CHKERRQ(ierr) > >> >> > >> >> > !******************************************************************************* > >> >> > >> >> ! Create linear solver context > >> >> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) > >> >> call KSPSetOperators(ksp,A,A,ierr) > >> >> > >> >> call KSPGetPC(ksp,pc,ierr) > >> >> CHKERRQ(ierr) > >> >> > >> >> > >> >> > !******************************************************************************* > >> >> ! split PC > >> >> call PCFieldSplitSetIS(pc,"0",gdof0_is,ierr); > >> >> CHKERRQ(ierr) > >> >> call ISDestroy(gdof0_is,ierr) > >> >> CHKERRQ(ierr) > >> >> call PCFieldSplitSetIS(pc,"1",gdof1_is,ierr); > >> >> CHKERRQ(ierr) > >> >> call ISDestroy(gdof1_is,ierr) > >> >> CHKERRQ(ierr) > >> >> > >> >> > !******************************************************************************* > >> >> > >> >> rtol = 1.0d-6 > >> >> atol = 1.0d-6 > >> >> dtol = 1.0d10 > >> >> maxitr = 1000 > >> >> call KSPSetTolerances(ksp,rtol,atol,dtol,maxitr,ierr) > >> >> CHKERRQ(ierr) > >> >> call KSPSetFromOptions(ksp,ierr) > >> >> end subroutine petsc_initialize > >> >> > >> >> > !------------------------------------------------------------------------------- > >> >> > >> >> subroutine petsc_set_matrix() > >> >> > >> >> implicit none > >> >> integer :: i,i_elmt,j,ncount > >> >> integer :: ggdof(NEDOF),idof(NEDOF),igdof(NEDOF) > >> >> > >> >> ! Set and assemble matrix. > >> >> call MatZeroEntries(A,ierr) > >> >> CHKERRQ(ierr) > >> >> do i_elmt=1,nelmt > >> >> ggdof(:)=ggdof_elmt(:,i_elmt) > >> >> ggdof(:)=ggdof(:)-1 ! petsc index starts from 0 > >> >> ncount=0; idof=-1; igdof=-1 > >> >> do i=1,NEDOF > >> >> do j=1,NEDOF > >> >> if(ggdof(i).ge.0.and.ggdof(j).ge.0)then > >> >> call MatSetValues(A,1,ggdof(i),1,ggdof(j),storekmat(i,j,i_elmt), > >> >> & > >> >> ADD_VALUES,ierr) > >> >> CHKERRQ(ierr) > >> >> endif > >> >> enddo > >> >> enddo > >> >> enddo > >> >> call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) > >> >> call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) > >> >> if(myrank==0)print*,'matrix setting & assembly complete!' > >> >> > >> >> end subroutine petsc_set_matrix > >> >> > >> >> > !------------------------------------------------------------------------------- > >> >> > >> >> subroutine petsc_set_vector() > >> >> implicit none > >> >> PetscScalar zero > >> >> integer :: i,i_elmt,ncount > >> >> integer :: ggdof(NEDOF),idof(NEDOF),igdof(NEDOF) > >> >> > >> >> ! set vector > >> >> zero=0.0_8 > >> >> call VecSet(b,zero,ierr) > >> >> do i_elmt=1,nelmt > >> >> ggdof(:)=ggdof_elmt(:,i_elmt) > >> >> ggdof(:)=ggdof(:)-1 ! petsc index starts from 0 > >> >> ncount=0; idof=-1; igdof=-1 > >> >> do i=1,NEDOF > >> >> if(ggdof(i).ge.0)then > >> >> call > VecSetValues(b,1,ggdof(i),storef(i,i_elmt),ADD_VALUES,ierr); > >> >> CHKERRQ(ierr) > >> >> endif > >> >> enddo > >> >> enddo > >> >> > >> >> ! assemble vector > >> >> call VecAssemblyBegin(b,ierr) > >> >> call VecAssemblyEnd(b,ierr) > >> >> if(myrank==0)print*,'vector setting & assembly complete!' > >> >> > >> >> end subroutine petsc_set_vector > >> >> > >> >> > !------------------------------------------------------------------------------- > >> >> > >> >> subroutine petsc_solve() > >> >> implicit none > >> >> PetscInt ireason > >> >> PetscScalar a_v(1) > >> >> PetscOffset a_i > >> >> PetscInt n > >> >> PetscReal,allocatable :: sdata(:) > >> >> > >> >> call VecGetSize(b,n,ierr) > >> >> CHKERRQ(ierr) > >> >> allocate(sdata(n)) > >> >> sdata=0.0_8 > >> >> call VecGetArray(b,a_v,a_i,ierr) > >> >> CHKERRQ(ierr) > >> >> sdata(1:n)=a_v(a_i+1:a_i+n) > >> >> call VecRestoreArray(b,a_v,a_i,ierr) > >> >> CHKERRQ(ierr) > >> >> print*,'rhs:',sdata > >> >> > >> >> call KSPSolve(ksp,b,x,ierr) > >> >> sdata=0.0_8 > >> >> call VecGetArray(x,a_v,a_i,ierr) > >> >> CHKERRQ(ierr) > >> >> sdata(1:n)=a_v(a_i+1:a_i+n) > >> >> call VecRestoreArray(b,a_v,a_i,ierr) > >> >> CHKERRQ(ierr) > >> >> print*,'solution:',sdata > >> >> > >> >> ! Check convergence > >> >> call KSPGetConvergedReason(ksp,ireason,ierr) > >> >> if(myrank==0)print*,'converges reason',myrank,ireason > >> >> call KSPGetIterationNumber(ksp,its,ierr) > >> >> if(myrank==0)print*,'Iterations:',its > >> >> deallocate(sdata) > >> >> end subroutine petsc_solve > >> >> > >> >> > !------------------------------------------------------------------------------- > >> >> > >> >> subroutine petsc_finalize() > >> >> implicit none > >> >> > >> >> ! Free work space. > >> >> call VecDestroy(x,ierr) > >> >> call VecDestroy(u,ierr) > >> >> call VecDestroy(b,ierr) > >> >> call MatDestroy(A,ierr) > >> >> call KSPDestroy(ksp,ierr) > >> >> call PetscFinalize(ierr) > >> >> CHKERRQ(ierr) > >> >> > >> >> end subroutine petsc_finalize > >> >> > >> >> > !------------------------------------------------------------------------------- > >> >> > >> >> end program testfs > >> >> > >> >> On Tue, Feb 2, 2016 at 4:54 PM, Matthew Knepley > >> >> wrote: > >> >>> On Tue, Feb 2, 2016 at 3:11 PM, Hom Nath Gharti < > hng.email at gmail.com> > >> >>> wrote: > >> >>>> > >> >>>> Thanks a lot. I will try. > >> >>> > >> >>> > >> >>> Also, if you send a small test case, we can run it and tell you the > >> >>> problem. > >> >>> > >> >>> Matt > >> >>> > >> >>>> > >> >>>> Hom Nath > >> >>>> > >> >>>> On Tue, Feb 2, 2016 at 4:02 PM, Matthew Knepley > > >> >>>> wrote: > >> >>>>> On Tue, Feb 2, 2016 at 3:01 PM, Hom Nath Gharti > >> >>>>> > >> >>>>> wrote: > >> >>>>>> > >> >>>>>> Thanks again Matt. Unfortunately got the same errors with '0'. I > >> >>>>>> think > >> >>>>>> both are valid in Fortran. > >> >>>>> > >> >>>>> > >> >>>>> Then you will have to go in the debugger and see why its not > >> >>>>> creating 4 > >> >>>>> splits, since this is exactly > >> >>>>> what it does in our tests. This is how all the SNES tests that I > use > >> >>>>> work. I > >> >>>>> am sure its a Fortran > >> >>>>> problem. > >> >>>>> > >> >>>>> Thanks, > >> >>>>> > >> >>>>> Matt > >> >>>>> > >> >>>>>> > >> >>>>>> Hom Nath > >> >>>>>> > >> >>>>>> On Tue, Feb 2, 2016 at 3:42 PM, Matthew Knepley < > knepley at gmail.com> > >> >>>>>> wrote: > >> >>>>>>> On Tue, Feb 2, 2016 at 2:35 PM, Hom Nath Gharti > >> >>>>>>> > >> >>>>>>> wrote: > >> >>>>>>>> > >> >>>>>>>> Thank you so much Matt. > >> >>>>>>>> > >> >>>>>>>> I now tried the following: > >> >>>>>>>> > >> >>>>>>>> ====================================================== > >> >>>>>>>> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) > >> >>>>>>>> call KSPGetPC(ksp,pc,ierr) > >> >>>>>>>> > >> >>>>>>>> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); > >> >>>>>>>> call ISDestroy(gdofu_is,ierr) > >> >>>>>>>> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); > >> >>>>>>>> call ISDestroy(gdofchi_is,ierr) > >> >>>>>>>> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); > >> >>>>>>>> call ISDestroy(gdofp_is,ierr) > >> >>>>>>>> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); > >> >>>>>>>> call ISDestroy(gdofphi_is,ierr) > >> >>>>>>>> > >> >>>>>>>> ! Amat changes in each time steps > >> >>>>>>>> call KSPSetOperators(ksp,Amat,Amat,ierr) !version >= 3.5.0 > >> >>>>>>>> > >> >>>>>>>> call KSPSolve(ksp,bvec,xvec,ierr) > >> >>>>>>>> ====================================================== > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> I am guessing that "0" is not a valid string for your Fortran > >> >>>>>>> compiler. > >> >>>>>>> Are > >> >>>>>>> you sure > >> >>>>>>> its not single quotes '0'? > >> >>>>>>> > >> >>>>>>> Matt > >> >>>>>>> > >> >>>>>>>> > >> >>>>>>>> But I get the following error: > >> >>>>>>>> > >> >>>>>>>> [0]PETSC ERROR: --------------------- Error Message > >> >>>>>>>> -------------------------------------------------------------- > >> >>>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data > >> >>>>>>>> [0]PETSC ERROR: Unhandled case, must have at least two fields, > >> >>>>>>>> not 1 > >> >>>>>>>> [0]PETSC ERROR: See > >> >>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for > trouble > >> >>>>>>>> shooting. > >> >>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > >> >>>>>>>> [0]PETSC ERROR: > >> >>>>>>>> /tigress/hgharti/gitwork/SPECFEM3D_GLOBEVSPP/./bin/xspecfem3D > on > >> >>>>>>>> a > >> >>>>>>>> arch-linux2-c-debug named tiger-r3c1n7 by hgharti Tue Feb 2 > 15: > >> >>>>>>>> 29:30 2016 > >> >>>>>>>> [0]PETSC ERROR: Configure options > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> > --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64/ > >> >>>>>>>> --with-cc=mpicc --with-cxx=mpicxx --wit > >> >>>>>>>> h-fc=mpif90 --with-mpiexec=mpiexec --with-debugging=1 > >> >>>>>>>> --download-scalapack --download-mumps --download-pastix > >> >>>>>>>> --download-superlu --download-superlu_dist --download-metis > >> >>>>>>>> --download-parmetis --download-ptscotch --download-hypre > >> >>>>>>>> [0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 469 in > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > >> >>>>>>>> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 486 in > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > >> >>>>>>>> [0]PETSC ERROR: #3 PCSetUp() line 983 in > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/interface/precon.c > >> >>>>>>>> [0]PETSC ERROR: #4 KSPSetUp() line 332 in > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c > >> >>>>>>>> [0]PETSC ERROR: #5 KSPSolve() line 546 in > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c > >> >>>>>>>> forrtl: error (76): Abort trap signal > >> >>>>>>>> > >> >>>>>>>> Am I missing something? > >> >>>>>>>> > >> >>>>>>>> Thanks, > >> >>>>>>>> Hom Nath > >> >>>>>>>> > >> >>>>>>>> On Tue, Feb 2, 2016 at 3:24 PM, Matthew Knepley > >> >>>>>>>> > >> >>>>>>>> wrote: > >> >>>>>>>>> On Tue, Feb 2, 2016 at 2:20 PM, Hom Nath Gharti > >> >>>>>>>>> > >> >>>>>>>>> wrote: > >> >>>>>>>>>> > >> >>>>>>>>>> Hi Matt, hi all, > >> >>>>>>>>>> > >> >>>>>>>>>> I am trying to use PCFIELDSPLIT for my solver which consists > of > >> >>>>>>>>>> 4 > >> >>>>>>>>>> different variables, namely, u (displacement vector), \chi > >> >>>>>>>>>> (displacement potential), p(pressure), and \phi (gravity > >> >>>>>>>>>> potential). > >> >>>>>>>>>> > >> >>>>>>>>>> My code segment looks like the following: > >> >>>>>>>>>> ====================================================== > >> >>>>>>>>>> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) > >> >>>>>>>>>> call KSPGetPC(ksp,pc,ierr) > >> >>>>>>>>>> nsplit=4 > >> >>>>>>>>>> call PCFieldSplitSetBlockSize(pc,nsplit,ierr); > >> >>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>>> You do not need the statement above. > >> >>>>>>>>> > >> >>>>>>>>>> > >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); > >> >>>>>>>>>> call ISDestroy(gdofu_is,ierr) > >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); > >> >>>>>>>>>> call ISDestroy(gdofchi_is,ierr) > >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); > >> >>>>>>>>>> call ISDestroy(gdofp_is,ierr) > >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); > >> >>>>>>>>>> call ISDestroy(gdofphi_is,ierr) > >> >>>>>>>>>> > >> >>>>>>>>>> call PCFieldSplitGetSubKSP(pc,nsplit,subksp,ierr); > >> >>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>>> These SetOperators() calls are wrong. I am not sure why you > want > >> >>>>>>>>> them > >> >>>>>>>>> here. > >> >>>>>>>>> Also, that means you do not need the call above. > >> >>>>>>>>> > >> >>>>>>>>> Thanks, > >> >>>>>>>>> > >> >>>>>>>>> Matt > >> >>>>>>>>> > >> >>>>>>>>>> > >> >>>>>>>>>> call KSPSetOperators(subksp(1),Amat,Amat,ierr); > >> >>>>>>>>>> call KSPSetOperators(subksp(2),Amat,Amat,ierr); > >> >>>>>>>>>> call KSPSetOperators(subksp(3),Amat,Amat,ierr); > >> >>>>>>>>>> call KSPSetOperators(subksp(4),Amat,Amat,ierr); > >> >>>>>>>>>> > >> >>>>>>>>>> call KSPSolve(ksp,bvec,xvec,ierr) > >> >>>>>>>>>> ====================================================== > >> >>>>>>>>>> > >> >>>>>>>>>> But I am getting the following error: > >> >>>>>>>>>> [79]PETSC ERROR: Null argument, when expecting valid pointer > >> >>>>>>>>>> [79]PETSC ERROR: Null Object: Parameter # 1 > >> >>>>>>>>>> [79]PETSC ERROR: #1 KSPSetOperators() line 536 in > >> >>>>>>>>>> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interf > >> >>>>>>>>>> > >> >>>>>>>>>> It looks like I am doing something wrong in "call > >> >>>>>>>>>> KSPSetOperators" > >> >>>>>>>>>> but > >> >>>>>>>>>> I could not figure that out. > >> >>>>>>>>>> > >> >>>>>>>>>> Could anybody help me to fix this problem? I looked into > almost > >> >>>>>>>>>> all > >> >>>>>>>>>> related examples but I could not really figure out the > correct > >> >>>>>>>>>> steps > >> >>>>>>>>>> after "call PCFieldSplitSetIS". > >> >>>>>>>>>> > >> >>>>>>>>>> Any help will be greatly appreciated. > >> >>>>>>>>>> > >> >>>>>>>>>> Best, > >> >>>>>>>>>> Hom nath > >> >>>>>>>>>> > >> >>>>>>>>>> On Sun, Jan 24, 2016 at 7:14 PM, Hom Nath Gharti > >> >>>>>>>>>> > >> >>>>>>>>>> wrote: > >> >>>>>>>>>>> Thank you so much Matt! I will try. > >> >>>>>>>>>>> > >> >>>>>>>>>>> Hom Nath > >> >>>>>>>>>>> > >> >>>>>>>>>>> On Sun, Jan 24, 2016 at 6:26 AM, Matthew Knepley > >> >>>>>>>>>>> > >> >>>>>>>>>>> wrote: > >> >>>>>>>>>>>> On Fri, Jan 22, 2016 at 2:19 PM, Hom Nath Gharti > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> wrote: > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> Dear all, > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> I am new to PcFieldSplit. > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> I have a matrix formed using MATMPIAIJ. Is it possible to > >> >>>>>>>>>>>>> use > >> >>>>>>>>>>>>> PCFIELDSPLIT operations in this type of matrix? Or does it > >> >>>>>>>>>>>>> have > >> >>>>>>>>>>>>> to > >> >>>>>>>>>>>>> be > >> >>>>>>>>>>>>> MATMPIBIJ or MATNEST format? > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> Yes, you can split AIJ. > >> >>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> If possible for MATMPIAIJ, could anybody provide me a > simple > >> >>>>>>>>>>>>> example > >> >>>>>>>>>>>>> or few steps? Variables in the equations are displacement > >> >>>>>>>>>>>>> vector, > >> >>>>>>>>>>>>> scalar potential and pressure. > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> If you do not have a collocated discretization, then you > have > >> >>>>>>>>>>>> to > >> >>>>>>>>>>>> use > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> Thanks, > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> Matt > >> >>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> Thanks for help. > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> Hom Nath > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> -- > >> >>>>>>>>>>>> What most experimenters take for granted before they begin > >> >>>>>>>>>>>> their > >> >>>>>>>>>>>> experiments > >> >>>>>>>>>>>> is infinitely more interesting than any results to which > >> >>>>>>>>>>>> their > >> >>>>>>>>>>>> experiments > >> >>>>>>>>>>>> lead. > >> >>>>>>>>>>>> -- Norbert Wiener > >> >>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>>> -- > >> >>>>>>>>> What most experimenters take for granted before they begin > their > >> >>>>>>>>> experiments > >> >>>>>>>>> is infinitely more interesting than any results to which their > >> >>>>>>>>> experiments > >> >>>>>>>>> lead. > >> >>>>>>>>> -- Norbert Wiener > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> -- > >> >>>>>>> What most experimenters take for granted before they begin their > >> >>>>>>> experiments > >> >>>>>>> is infinitely more interesting than any results to which their > >> >>>>>>> experiments > >> >>>>>>> lead. > >> >>>>>>> -- Norbert Wiener > >> >>>>> > >> >>>>> > >> >>>>> > >> >>>>> > >> >>>>> -- > >> >>>>> What most experimenters take for granted before they begin their > >> >>>>> experiments > >> >>>>> is infinitely more interesting than any results to which their > >> >>>>> experiments > >> >>>>> lead. > >> >>>>> -- Norbert Wiener > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> -- > >> >>> What most experimenters take for granted before they begin their > >> >>> experiments > >> >>> is infinitely more interesting than any results to which their > >> >>> experiments > >> >>> lead. > >> >>> -- Norbert Wiener > >> > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments > > is infinitely more interesting than any results to which their > experiments > > lead. > > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hng.email at gmail.com Tue Feb 9 09:57:51 2016 From: hng.email at gmail.com (Hom Nath Gharti) Date: Tue, 9 Feb 2016 10:57:51 -0500 Subject: [petsc-users] PCFIELDSPLIT question In-Reply-To: References: Message-ID: Sounds interesting! Thanks a lot Matt! I will have a look. Hom Nath On Tue, Feb 9, 2016 at 10:36 AM, Matthew Knepley wrote: > On Tue, Feb 9, 2016 at 9:31 AM, Hom Nath Gharti wrote: >> >> Thanks a lot Matt! >> >> Were you referring to >> >> http://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex62.c.html >> ? >> >> I do not see any statements related to PCFieldSplit there. Am I >> missing something here? > > > Yes. The whole point is not to put anything specific to FIELDSPLIT in the > code. It is all in options. For example, > > ./ex62 -run_type full -refinement_limit 0.00625 -bc_type dirichlet > -interpolate 1 -vel_petscspace_order 2 -pres_petscspace_order 1 -ksp_type > fgmres -ksp_gmres_restart 100 -ksp_rtol 1.0e-9 -pc_type fieldsplit > -pc_fieldsplit_type schur -pc_fieldsplit_schur_factorization_type upper > -fieldsplit_pressure_ksp_rtol 1e-10 -fieldsplit_velocity_ksp_type gmres > -fieldsplit_velocity_pc_type lu -fieldsplit_pressure_pc_type jacobi > -snes_monitor_short -ksp_monitor_short -snes_converged_reason > -ksp_converged_reason -snes_view -show_solution 0 > > The reason that it works is that the DM calls PCFIeldSplitSetIS() for each > field in the DM. > > Thanks, > > Matt > >> >> Thanks, >> Hom Nath >> >> On Tue, Feb 9, 2016 at 10:19 AM, Matthew Knepley >> wrote: >> > On Tue, Feb 9, 2016 at 9:10 AM, Hom Nath Gharti >> > wrote: >> >> >> >> Thank you so much Barry! >> >> >> >> For my small test case, with -pc_fieldsplit_block_size 4, the program >> >> runs, although the answer was not correct. At least now I get >> >> something to look upon. I am using PCFieldSplitSetIS to set the >> >> fields. Do I still need to use -pc_fieldsplit_block_size? >> > >> > >> > No, this is only for co-located discretizations. >> > >> >> >> >> In my case each grid point may have different variable sets. For >> >> example, the grid point in the solid region has displacement and >> >> gravity potential: ux, uy, uz, and \phi. Whereas the grid point in the >> >> fluid region has scalar potential, pressure and gravity potential: >> >> \chi, p, and \phi. And the solid-fluid interface has all of them. Do >> >> you think I can still use PCFIELDSPLIT in this situation? >> > >> > >> > We have examples, like SNES ex62, which do exactly this. >> > >> > Thanks, >> > >> > Matt >> > >> >> >> >> Best, >> >> Hom Nath >> >> >> >> >> >> >> >> On Mon, Feb 8, 2016 at 2:27 PM, Barry Smith wrote: >> >> > >> >> > >> >> > In this case you need to provide two pieces of information to the >> >> > PCFIELDSPLIT. What we call the "block size" or bs which is the number >> >> > of >> >> > "basic fields" in the problem. For example if at each grid point you >> >> > have >> >> > x-velocity, y-velocity, and pressure the block size is 3. The second >> >> > is you >> >> > need to tell PCFIELDSPLIT what "basic fields" are in each split you >> >> > want to >> >> > use. >> >> > >> >> > In your case you can do this with -pc_fieldsplit_block_size 4. >> >> > (Note if you use a DM to define your problem the PCFIELDSPLIT will >> >> > automatically get the bs from the DM so you do not need to provide >> >> > it. >> >> > Similarly if you set a block size on your Mat the PCFIELDSPLIT will >> >> > use that >> >> > so you don't need to set it). >> >> > >> >> > Then use -pc_fieldsplit_0_fields 0,1 to indicate your first split >> >> > is >> >> > the 0 and 1 basic fields and -pc_fieldsplit_1_fields 2,3 to indicate >> >> > the >> >> > second split is the 2 and 3 basic fields. >> >> > (By default if you do not provide this then PCFIELDSPLIT will use bs >> >> > splits (4 in your case) one for each basic field. >> >> > >> >> > Barry >> >> > >> >> >> On Feb 8, 2016, at 11:20 AM, Hom Nath Gharti >> >> >> wrote: >> >> >> >> >> >> Hi Matt, Hi all, >> >> >> >> >> >> I am trying to have some feel for PCFIELDSPLIT testing on a very >> >> >> small >> >> >> matrix (attached below). I have 4 degrees of freedoms. I use 4 >> >> >> processors. I want to split 4 dofs into two fields each having two >> >> >> dofs. I don't know whether this my be a problem for petsc. When I >> >> >> use >> >> >> the command: >> >> >> srun -n 4 ./xtestfs -ksp_monitor -ksp_view >> >> >> >> >> >> It runs fine. >> >> >> >> >> >> But when I want to use field split options using the command: >> >> >> srun -n 4 ./xtestfs -ksp_monitor -ksp_type fgmres -pc_type >> >> >> fieldsplit >> >> >> -fieldsplit_0_ksp_type gmres -fieldsplit_0_pc_type bjacobi >> >> >> -fieldsplit_1_pc_type jacobi >> >> >> >> >> >> I get the following error message: >> >> >> [0]PETSC ERROR: Petsc has generated inconsistent data >> >> >> [0]PETSC ERROR: Unhandled case, must have at least two fields, not 1 >> >> >> [0]PETSC ERROR: See >> >> >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >> >> >> shooting. >> >> >> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >> >> >> [0]PETSC ERROR: /tigress/hgharti/test/testfs/./xtestfs on a >> >> >> arch-linux2-c-debug named tiger-r11n1 by hgharti Mon Feb 8 11:40:03 >> >> >> 2016 >> >> >> [0]PETSC ERROR: Configure options >> >> >> >> >> >> >> >> >> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64/ >> >> >> --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 >> >> >> --with-mpiexec=mpiexec --with-debugging=1 --download-scalapack >> >> >> --download-mumps --download-pastix --download-superlu >> >> >> --download-superlu_dist --download-metis --download-parmetis >> >> >> --download-ptscotch --download-hypre >> >> >> [0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 469 in >> >> >> >> >> >> >> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> >> >> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 486 in >> >> >> >> >> >> >> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> >> >> [0]PETSC ERROR: #3 PCSetUp() line 983 in >> >> >> >> >> >> >> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/interface/precon.c >> >> >> [0]PETSC ERROR: #4 KSPSetUp() line 332 in >> >> >> >> >> >> >> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >> >> >> [0]PETSC ERROR: #5 KSPSolve() line 546 in >> >> >> >> >> >> >> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >> >> >> WARNING! There are options you set that were not used! >> >> >> WARNING! could be spelling mistake, etc! >> >> >> Option left: name:-fieldsplit_0_ksp_type value: gmres >> >> >> Option left: name:-fieldsplit_0_pc_type value: bjacobi >> >> >> Option left: name:-fieldsplit_1_pc_type value: jacobi >> >> >> forrtl: error (76): Abort trap signal >> >> >> >> >> >> I tried several trials but could not fix the problem. Is it the >> >> >> FORTRAN problem or am I doing something wrong? Any suggestions would >> >> >> be greatly appreciated. >> >> >> For your information I use: >> >> >> PETSc-3.6.3 >> >> >> intel-16.0 compiler >> >> >> intel-mpi-5.1.1 >> >> >> >> >> >> Program >> >> >> >> >> >> >> >> >> Best, >> >> >> Hom Nath >> >> >> >> >> >> ! simple test case for PCFIELDSPLIT >> >> >> ! >> >> >> >> >> >> ----------------------------------------------------------------------- >> >> >> program testfs >> >> >> implicit none >> >> >> #include "petsc/finclude/petscsys.h" >> >> >> #include "petsc/finclude/petscvec.h" >> >> >> #include "petsc/finclude/petscvec.h90" >> >> >> #include "petsc/finclude/petscmat.h" >> >> >> #include "petsc/finclude/petscksp.h" >> >> >> #include "petsc/finclude/petscpc.h" >> >> >> #include "petsc/finclude/petscviewer.h" >> >> >> #include "petsc/finclude/petscviewer.h90" >> >> >> >> >> >> Vec x,b,u >> >> >> Mat A >> >> >> KSP ksp >> >> >> PC pc >> >> >> PetscErrorCode ierr >> >> >> PetscBool flg >> >> >> PetscInt errcode,its,maxitr,myrank,nproc >> >> >> PetscInt nedof,nelmt,ndof,nenod,ngdof,fixdof,nsparse,neq >> >> >> PetscInt,allocatable :: krow_sparse(:),ggdof_elmt(:,:), & >> >> >> inode_elmt(:,:) >> >> >> >> >> >> PetscReal,allocatable :: storef(:,:),storekmat(:,:,:) >> >> >> PetscInt gdof0(2),gdof1(2) >> >> >> >> >> >> ! initialize MPI >> >> >> >> >> >> call MPI_INIT(errcode) >> >> >> if(errcode /= 0)stop 'ERROR: cannot initialize MPI!' >> >> >> call MPI_COMM_RANK(MPI_COMM_WORLD,myrank,errcode) >> >> >> if(errcode /= 0)stop 'ERROR: cannot find processor ID' >> >> >> call MPI_COMM_SIZE(MPI_COMM_WORLD,nproc,errcode) >> >> >> if(errcode /= 0)stop 'ERROR: cannot find number of processors!' >> >> >> >> >> >> ! define some parameters >> >> >> ! 1D model with 4 elements. Each elements consits of 2 nodes. Node 0 >> >> >> is >> >> >> fixed. >> >> >> ! 1-----------2-----------3-----------4-----------5 >> >> >> nelmt=1 ! per processor >> >> >> nenod=2 ! number of nodes per element >> >> >> nedof=2 ! number of DOFs per element >> >> >> ndof=2 ! number of DOFs >> >> >> ngdof=4 ! total number of global DOFs >> >> >> fixdof=1 >> >> >> >> >> >> if(myrank==0)then >> >> >> neq=1 ! number of equations >> >> >> nsparse=1 >> >> >> else >> >> >> neq=2 ! number of equations >> >> >> nsparse=4 >> >> >> endif >> >> >> >> >> >> allocate(storef(nedof,nelmt),storekmat(nedof,nedof,nelmt), >> >> >> & >> >> >> >> >> >> ggdof_elmt(nedof,nelmt),inode_elmt(nenod,nelmt),krow_sparse(nsparse)) >> >> >> >> >> >> storef=0.0_8 ! local RHS vector >> >> >> storekmat=0.0_8 ! local stiffness matrix >> >> >> if(myrank==0)then >> >> >> krow_sparse(:)=1 >> >> >> storef(:,1)=(/12.5000_8, 12.5000_8/) >> >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & >> >> >> 0.0166666667_8, >> >> >> 1.2333333332_8/),(/nedof,nedof/)) >> >> >> inode_elmt(:,1) = (/1,2/) >> >> >> ggdof_elmt(:,1) = (/0,1/) ! 0 because left node is fixed >> >> >> elseif(myrank==1)then >> >> >> krow_sparse(:)=(/1,1,2,2/) >> >> >> storef(:,1)=(/12.5000_8, 12.5000_8/) >> >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & >> >> >> 0.0166666667_8, >> >> >> 1.2333333332_8/),(/nedof,nedof/)) >> >> >> inode_elmt(:,1) = (/1,2/) >> >> >> ggdof_elmt(:,1) = (/1,2/) >> >> >> elseif(myrank==2)then >> >> >> krow_sparse(:)=(/1,1,2,2/) >> >> >> storef(:,1)=(/12.5000_8, 12.5000_8/) >> >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & >> >> >> 0.0166666667_8, >> >> >> 1.2333333332_8/),(/nedof,nedof/)) >> >> >> inode_elmt(:,1) = (/1,2/) >> >> >> ggdof_elmt(:,1) = (/2,3/) >> >> >> elseif(myrank==3)then >> >> >> krow_sparse(:)=(/1,1,2,2/) >> >> >> storef(:,1)=(/12.5000_8, 22.5000_8/) >> >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & >> >> >> 0.0166666667_8, >> >> >> 1.2333333332_8/),(/nedof,nedof/)) >> >> >> inode_elmt(:,1) = (/1,2/) >> >> >> ggdof_elmt(:,1) = (/3,4/) >> >> >> endif >> >> >> >> >> >> call petsc_initialize >> >> >> call petsc_set_matrix >> >> >> call petsc_set_vector >> >> >> call petsc_solve >> >> >> call petsc_finalize >> >> >> >> >> >> ! deallocate >> >> >> deallocate(storef,storekmat,ggdof_elmt,inode_elmt) >> >> >> deallocate(krow_sparse) >> >> >> contains >> >> >> >> >> >> >> >> >> !------------------------------------------------------------------------------- >> >> >> subroutine petsc_initialize() >> >> >> implicit none >> >> >> PetscInt :: istart,iend >> >> >> PetscInt :: nzeros_max,nzeros_min >> >> >> PetscReal :: atol,dtol,rtol >> >> >> PetscInt,allocatable :: nzeros(:) >> >> >> IS gdof0_is,gdof1_is >> >> >> >> >> >> call PetscInitialize(PETSC_NULL_CHARACTER,ierr) >> >> >> call PetscOptionsGetInt(PETSC_NULL_CHARACTER,'-n',ngdof,flg,ierr) >> >> >> >> >> >> ! count number of nonzeros per row >> >> >> allocate(nzeros(neq)) >> >> >> nzeros=0 >> >> >> nzeros(krow_sparse)=nzeros(krow_sparse)+1 >> >> >> nzeros_max=maxval(nzeros) >> >> >> nzeros_min=minval(nzeros) >> >> >> >> >> >> ! create matrix >> >> >> call MatCreate(PETSC_COMM_WORLD,A,ierr) >> >> >> call MatSetType(A,MATMPIAIJ,ierr) >> >> >> CHKERRQ(ierr) >> >> >> call MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,ngdof,ngdof,ierr) >> >> >> CHKERRQ(ierr) >> >> >> ! preallocation >> >> >> call >> >> >> >> >> >> MatMPIAIJSetPreallocation(A,nzeros_max,PETSC_NULL_INTEGER,nzeros_max, & >> >> >> PETSC_NULL_INTEGER,ierr) >> >> >> CHKERRQ(ierr) >> >> >> >> >> >> call MatGetOwnershipRange(A,istart,iend,ierr) >> >> >> CHKERRQ(ierr) >> >> >> print*,'ownership:',myrank,istart,iend >> >> >> ! create vector >> >> >> call VecCreate(PETSC_COMM_WORLD,x,ierr) >> >> >> call VecSetSizes(x,PETSC_DECIDE,ngdof,ierr) >> >> >> call VecSetType(x,VECMPI,ierr) >> >> >> call VecDuplicate(x,b,ierr) >> >> >> call VecDuplicate(x,u,ierr) >> >> >> >> >> >> >> >> >> >> >> >> !******************************************************************************* >> >> >> ! get IS for split fields >> >> >> gdof0=(/0,1/) >> >> >> gdof1=(/2,3/) >> >> >> call >> >> >> >> >> >> ISCreateGeneral(PETSC_COMM_WORLD,2,gdof0,PETSC_COPY_VALUES,gdof0_is,ierr) >> >> >> CHKERRQ(ierr) >> >> >> call >> >> >> >> >> >> ISCreateGeneral(PETSC_COMM_WORLD,2,gdof1,PETSC_COPY_VALUES,gdof1_is,ierr) >> >> >> CHKERRQ(ierr) >> >> >> >> >> >> >> >> >> !******************************************************************************* >> >> >> >> >> >> ! Create linear solver context >> >> >> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >> >> >> call KSPSetOperators(ksp,A,A,ierr) >> >> >> >> >> >> call KSPGetPC(ksp,pc,ierr) >> >> >> CHKERRQ(ierr) >> >> >> >> >> >> >> >> >> >> >> >> !******************************************************************************* >> >> >> ! split PC >> >> >> call PCFieldSplitSetIS(pc,"0",gdof0_is,ierr); >> >> >> CHKERRQ(ierr) >> >> >> call ISDestroy(gdof0_is,ierr) >> >> >> CHKERRQ(ierr) >> >> >> call PCFieldSplitSetIS(pc,"1",gdof1_is,ierr); >> >> >> CHKERRQ(ierr) >> >> >> call ISDestroy(gdof1_is,ierr) >> >> >> CHKERRQ(ierr) >> >> >> >> >> >> >> >> >> !******************************************************************************* >> >> >> >> >> >> rtol = 1.0d-6 >> >> >> atol = 1.0d-6 >> >> >> dtol = 1.0d10 >> >> >> maxitr = 1000 >> >> >> call KSPSetTolerances(ksp,rtol,atol,dtol,maxitr,ierr) >> >> >> CHKERRQ(ierr) >> >> >> call KSPSetFromOptions(ksp,ierr) >> >> >> end subroutine petsc_initialize >> >> >> >> >> >> >> >> >> !------------------------------------------------------------------------------- >> >> >> >> >> >> subroutine petsc_set_matrix() >> >> >> >> >> >> implicit none >> >> >> integer :: i,i_elmt,j,ncount >> >> >> integer :: ggdof(NEDOF),idof(NEDOF),igdof(NEDOF) >> >> >> >> >> >> ! Set and assemble matrix. >> >> >> call MatZeroEntries(A,ierr) >> >> >> CHKERRQ(ierr) >> >> >> do i_elmt=1,nelmt >> >> >> ggdof(:)=ggdof_elmt(:,i_elmt) >> >> >> ggdof(:)=ggdof(:)-1 ! petsc index starts from 0 >> >> >> ncount=0; idof=-1; igdof=-1 >> >> >> do i=1,NEDOF >> >> >> do j=1,NEDOF >> >> >> if(ggdof(i).ge.0.and.ggdof(j).ge.0)then >> >> >> call >> >> >> MatSetValues(A,1,ggdof(i),1,ggdof(j),storekmat(i,j,i_elmt), >> >> >> & >> >> >> ADD_VALUES,ierr) >> >> >> CHKERRQ(ierr) >> >> >> endif >> >> >> enddo >> >> >> enddo >> >> >> enddo >> >> >> call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) >> >> >> call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) >> >> >> if(myrank==0)print*,'matrix setting & assembly complete!' >> >> >> >> >> >> end subroutine petsc_set_matrix >> >> >> >> >> >> >> >> >> !------------------------------------------------------------------------------- >> >> >> >> >> >> subroutine petsc_set_vector() >> >> >> implicit none >> >> >> PetscScalar zero >> >> >> integer :: i,i_elmt,ncount >> >> >> integer :: ggdof(NEDOF),idof(NEDOF),igdof(NEDOF) >> >> >> >> >> >> ! set vector >> >> >> zero=0.0_8 >> >> >> call VecSet(b,zero,ierr) >> >> >> do i_elmt=1,nelmt >> >> >> ggdof(:)=ggdof_elmt(:,i_elmt) >> >> >> ggdof(:)=ggdof(:)-1 ! petsc index starts from 0 >> >> >> ncount=0; idof=-1; igdof=-1 >> >> >> do i=1,NEDOF >> >> >> if(ggdof(i).ge.0)then >> >> >> call >> >> >> VecSetValues(b,1,ggdof(i),storef(i,i_elmt),ADD_VALUES,ierr); >> >> >> CHKERRQ(ierr) >> >> >> endif >> >> >> enddo >> >> >> enddo >> >> >> >> >> >> ! assemble vector >> >> >> call VecAssemblyBegin(b,ierr) >> >> >> call VecAssemblyEnd(b,ierr) >> >> >> if(myrank==0)print*,'vector setting & assembly complete!' >> >> >> >> >> >> end subroutine petsc_set_vector >> >> >> >> >> >> >> >> >> !------------------------------------------------------------------------------- >> >> >> >> >> >> subroutine petsc_solve() >> >> >> implicit none >> >> >> PetscInt ireason >> >> >> PetscScalar a_v(1) >> >> >> PetscOffset a_i >> >> >> PetscInt n >> >> >> PetscReal,allocatable :: sdata(:) >> >> >> >> >> >> call VecGetSize(b,n,ierr) >> >> >> CHKERRQ(ierr) >> >> >> allocate(sdata(n)) >> >> >> sdata=0.0_8 >> >> >> call VecGetArray(b,a_v,a_i,ierr) >> >> >> CHKERRQ(ierr) >> >> >> sdata(1:n)=a_v(a_i+1:a_i+n) >> >> >> call VecRestoreArray(b,a_v,a_i,ierr) >> >> >> CHKERRQ(ierr) >> >> >> print*,'rhs:',sdata >> >> >> >> >> >> call KSPSolve(ksp,b,x,ierr) >> >> >> sdata=0.0_8 >> >> >> call VecGetArray(x,a_v,a_i,ierr) >> >> >> CHKERRQ(ierr) >> >> >> sdata(1:n)=a_v(a_i+1:a_i+n) >> >> >> call VecRestoreArray(b,a_v,a_i,ierr) >> >> >> CHKERRQ(ierr) >> >> >> print*,'solution:',sdata >> >> >> >> >> >> ! Check convergence >> >> >> call KSPGetConvergedReason(ksp,ireason,ierr) >> >> >> if(myrank==0)print*,'converges reason',myrank,ireason >> >> >> call KSPGetIterationNumber(ksp,its,ierr) >> >> >> if(myrank==0)print*,'Iterations:',its >> >> >> deallocate(sdata) >> >> >> end subroutine petsc_solve >> >> >> >> >> >> >> >> >> !------------------------------------------------------------------------------- >> >> >> >> >> >> subroutine petsc_finalize() >> >> >> implicit none >> >> >> >> >> >> ! Free work space. >> >> >> call VecDestroy(x,ierr) >> >> >> call VecDestroy(u,ierr) >> >> >> call VecDestroy(b,ierr) >> >> >> call MatDestroy(A,ierr) >> >> >> call KSPDestroy(ksp,ierr) >> >> >> call PetscFinalize(ierr) >> >> >> CHKERRQ(ierr) >> >> >> >> >> >> end subroutine petsc_finalize >> >> >> >> >> >> >> >> >> !------------------------------------------------------------------------------- >> >> >> >> >> >> end program testfs >> >> >> >> >> >> On Tue, Feb 2, 2016 at 4:54 PM, Matthew Knepley >> >> >> wrote: >> >> >>> On Tue, Feb 2, 2016 at 3:11 PM, Hom Nath Gharti >> >> >>> >> >> >>> wrote: >> >> >>>> >> >> >>>> Thanks a lot. I will try. >> >> >>> >> >> >>> >> >> >>> Also, if you send a small test case, we can run it and tell you the >> >> >>> problem. >> >> >>> >> >> >>> Matt >> >> >>> >> >> >>>> >> >> >>>> Hom Nath >> >> >>>> >> >> >>>> On Tue, Feb 2, 2016 at 4:02 PM, Matthew Knepley >> >> >>>> >> >> >>>> wrote: >> >> >>>>> On Tue, Feb 2, 2016 at 3:01 PM, Hom Nath Gharti >> >> >>>>> >> >> >>>>> wrote: >> >> >>>>>> >> >> >>>>>> Thanks again Matt. Unfortunately got the same errors with '0'. I >> >> >>>>>> think >> >> >>>>>> both are valid in Fortran. >> >> >>>>> >> >> >>>>> >> >> >>>>> Then you will have to go in the debugger and see why its not >> >> >>>>> creating 4 >> >> >>>>> splits, since this is exactly >> >> >>>>> what it does in our tests. This is how all the SNES tests that I >> >> >>>>> use >> >> >>>>> work. I >> >> >>>>> am sure its a Fortran >> >> >>>>> problem. >> >> >>>>> >> >> >>>>> Thanks, >> >> >>>>> >> >> >>>>> Matt >> >> >>>>> >> >> >>>>>> >> >> >>>>>> Hom Nath >> >> >>>>>> >> >> >>>>>> On Tue, Feb 2, 2016 at 3:42 PM, Matthew Knepley >> >> >>>>>> >> >> >>>>>> wrote: >> >> >>>>>>> On Tue, Feb 2, 2016 at 2:35 PM, Hom Nath Gharti >> >> >>>>>>> >> >> >>>>>>> wrote: >> >> >>>>>>>> >> >> >>>>>>>> Thank you so much Matt. >> >> >>>>>>>> >> >> >>>>>>>> I now tried the following: >> >> >>>>>>>> >> >> >>>>>>>> ====================================================== >> >> >>>>>>>> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >> >> >>>>>>>> call KSPGetPC(ksp,pc,ierr) >> >> >>>>>>>> >> >> >>>>>>>> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); >> >> >>>>>>>> call ISDestroy(gdofu_is,ierr) >> >> >>>>>>>> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); >> >> >>>>>>>> call ISDestroy(gdofchi_is,ierr) >> >> >>>>>>>> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); >> >> >>>>>>>> call ISDestroy(gdofp_is,ierr) >> >> >>>>>>>> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); >> >> >>>>>>>> call ISDestroy(gdofphi_is,ierr) >> >> >>>>>>>> >> >> >>>>>>>> ! Amat changes in each time steps >> >> >>>>>>>> call KSPSetOperators(ksp,Amat,Amat,ierr) !version >= 3.5.0 >> >> >>>>>>>> >> >> >>>>>>>> call KSPSolve(ksp,bvec,xvec,ierr) >> >> >>>>>>>> ====================================================== >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>> I am guessing that "0" is not a valid string for your Fortran >> >> >>>>>>> compiler. >> >> >>>>>>> Are >> >> >>>>>>> you sure >> >> >>>>>>> its not single quotes '0'? >> >> >>>>>>> >> >> >>>>>>> Matt >> >> >>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> But I get the following error: >> >> >>>>>>>> >> >> >>>>>>>> [0]PETSC ERROR: --------------------- Error Message >> >> >>>>>>>> -------------------------------------------------------------- >> >> >>>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >> >> >>>>>>>> [0]PETSC ERROR: Unhandled case, must have at least two fields, >> >> >>>>>>>> not 1 >> >> >>>>>>>> [0]PETSC ERROR: See >> >> >>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for >> >> >>>>>>>> trouble >> >> >>>>>>>> shooting. >> >> >>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >> >> >>>>>>>> [0]PETSC ERROR: >> >> >>>>>>>> /tigress/hgharti/gitwork/SPECFEM3D_GLOBEVSPP/./bin/xspecfem3D >> >> >>>>>>>> on >> >> >>>>>>>> a >> >> >>>>>>>> arch-linux2-c-debug named tiger-r3c1n7 by hgharti Tue Feb 2 >> >> >>>>>>>> 15: >> >> >>>>>>>> 29:30 2016 >> >> >>>>>>>> [0]PETSC ERROR: Configure options >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64/ >> >> >>>>>>>> --with-cc=mpicc --with-cxx=mpicxx --wit >> >> >>>>>>>> h-fc=mpif90 --with-mpiexec=mpiexec --with-debugging=1 >> >> >>>>>>>> --download-scalapack --download-mumps --download-pastix >> >> >>>>>>>> --download-superlu --download-superlu_dist --download-metis >> >> >>>>>>>> --download-parmetis --download-ptscotch --download-hypre >> >> >>>>>>>> [0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 469 in >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> >> >>>>>>>> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 486 in >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> >> >>>>>>>> [0]PETSC ERROR: #3 PCSetUp() line 983 in >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/interface/precon.c >> >> >>>>>>>> [0]PETSC ERROR: #4 KSPSetUp() line 332 in >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >> >> >>>>>>>> [0]PETSC ERROR: #5 KSPSolve() line 546 in >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >> >> >>>>>>>> forrtl: error (76): Abort trap signal >> >> >>>>>>>> >> >> >>>>>>>> Am I missing something? >> >> >>>>>>>> >> >> >>>>>>>> Thanks, >> >> >>>>>>>> Hom Nath >> >> >>>>>>>> >> >> >>>>>>>> On Tue, Feb 2, 2016 at 3:24 PM, Matthew Knepley >> >> >>>>>>>> >> >> >>>>>>>> wrote: >> >> >>>>>>>>> On Tue, Feb 2, 2016 at 2:20 PM, Hom Nath Gharti >> >> >>>>>>>>> >> >> >>>>>>>>> wrote: >> >> >>>>>>>>>> >> >> >>>>>>>>>> Hi Matt, hi all, >> >> >>>>>>>>>> >> >> >>>>>>>>>> I am trying to use PCFIELDSPLIT for my solver which consists >> >> >>>>>>>>>> of >> >> >>>>>>>>>> 4 >> >> >>>>>>>>>> different variables, namely, u (displacement vector), \chi >> >> >>>>>>>>>> (displacement potential), p(pressure), and \phi (gravity >> >> >>>>>>>>>> potential). >> >> >>>>>>>>>> >> >> >>>>>>>>>> My code segment looks like the following: >> >> >>>>>>>>>> ====================================================== >> >> >>>>>>>>>> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >> >> >>>>>>>>>> call KSPGetPC(ksp,pc,ierr) >> >> >>>>>>>>>> nsplit=4 >> >> >>>>>>>>>> call PCFieldSplitSetBlockSize(pc,nsplit,ierr); >> >> >>>>>>>>> >> >> >>>>>>>>> >> >> >>>>>>>>> You do not need the statement above. >> >> >>>>>>>>> >> >> >>>>>>>>>> >> >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); >> >> >>>>>>>>>> call ISDestroy(gdofu_is,ierr) >> >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); >> >> >>>>>>>>>> call ISDestroy(gdofchi_is,ierr) >> >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); >> >> >>>>>>>>>> call ISDestroy(gdofp_is,ierr) >> >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); >> >> >>>>>>>>>> call ISDestroy(gdofphi_is,ierr) >> >> >>>>>>>>>> >> >> >>>>>>>>>> call PCFieldSplitGetSubKSP(pc,nsplit,subksp,ierr); >> >> >>>>>>>>> >> >> >>>>>>>>> >> >> >>>>>>>>> These SetOperators() calls are wrong. I am not sure why you >> >> >>>>>>>>> want >> >> >>>>>>>>> them >> >> >>>>>>>>> here. >> >> >>>>>>>>> Also, that means you do not need the call above. >> >> >>>>>>>>> >> >> >>>>>>>>> Thanks, >> >> >>>>>>>>> >> >> >>>>>>>>> Matt >> >> >>>>>>>>> >> >> >>>>>>>>>> >> >> >>>>>>>>>> call KSPSetOperators(subksp(1),Amat,Amat,ierr); >> >> >>>>>>>>>> call KSPSetOperators(subksp(2),Amat,Amat,ierr); >> >> >>>>>>>>>> call KSPSetOperators(subksp(3),Amat,Amat,ierr); >> >> >>>>>>>>>> call KSPSetOperators(subksp(4),Amat,Amat,ierr); >> >> >>>>>>>>>> >> >> >>>>>>>>>> call KSPSolve(ksp,bvec,xvec,ierr) >> >> >>>>>>>>>> ====================================================== >> >> >>>>>>>>>> >> >> >>>>>>>>>> But I am getting the following error: >> >> >>>>>>>>>> [79]PETSC ERROR: Null argument, when expecting valid pointer >> >> >>>>>>>>>> [79]PETSC ERROR: Null Object: Parameter # 1 >> >> >>>>>>>>>> [79]PETSC ERROR: #1 KSPSetOperators() line 536 in >> >> >>>>>>>>>> >> >> >>>>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interf >> >> >>>>>>>>>> >> >> >>>>>>>>>> It looks like I am doing something wrong in "call >> >> >>>>>>>>>> KSPSetOperators" >> >> >>>>>>>>>> but >> >> >>>>>>>>>> I could not figure that out. >> >> >>>>>>>>>> >> >> >>>>>>>>>> Could anybody help me to fix this problem? I looked into >> >> >>>>>>>>>> almost >> >> >>>>>>>>>> all >> >> >>>>>>>>>> related examples but I could not really figure out the >> >> >>>>>>>>>> correct >> >> >>>>>>>>>> steps >> >> >>>>>>>>>> after "call PCFieldSplitSetIS". >> >> >>>>>>>>>> >> >> >>>>>>>>>> Any help will be greatly appreciated. >> >> >>>>>>>>>> >> >> >>>>>>>>>> Best, >> >> >>>>>>>>>> Hom nath >> >> >>>>>>>>>> >> >> >>>>>>>>>> On Sun, Jan 24, 2016 at 7:14 PM, Hom Nath Gharti >> >> >>>>>>>>>> >> >> >>>>>>>>>> wrote: >> >> >>>>>>>>>>> Thank you so much Matt! I will try. >> >> >>>>>>>>>>> >> >> >>>>>>>>>>> Hom Nath >> >> >>>>>>>>>>> >> >> >>>>>>>>>>> On Sun, Jan 24, 2016 at 6:26 AM, Matthew Knepley >> >> >>>>>>>>>>> >> >> >>>>>>>>>>> wrote: >> >> >>>>>>>>>>>> On Fri, Jan 22, 2016 at 2:19 PM, Hom Nath Gharti >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>> wrote: >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>> Dear all, >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>> I am new to PcFieldSplit. >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>> I have a matrix formed using MATMPIAIJ. Is it possible to >> >> >>>>>>>>>>>>> use >> >> >>>>>>>>>>>>> PCFIELDSPLIT operations in this type of matrix? Or does >> >> >>>>>>>>>>>>> it >> >> >>>>>>>>>>>>> have >> >> >>>>>>>>>>>>> to >> >> >>>>>>>>>>>>> be >> >> >>>>>>>>>>>>> MATMPIBIJ or MATNEST format? >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>> Yes, you can split AIJ. >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>> If possible for MATMPIAIJ, could anybody provide me a >> >> >>>>>>>>>>>>> simple >> >> >>>>>>>>>>>>> example >> >> >>>>>>>>>>>>> or few steps? Variables in the equations are displacement >> >> >>>>>>>>>>>>> vector, >> >> >>>>>>>>>>>>> scalar potential and pressure. >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>> If you do not have a collocated discretization, then you >> >> >>>>>>>>>>>> have >> >> >>>>>>>>>>>> to >> >> >>>>>>>>>>>> use >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>> Thanks, >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>> Matt >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>> Thanks for help. >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>> Hom Nath >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>> -- >> >> >>>>>>>>>>>> What most experimenters take for granted before they begin >> >> >>>>>>>>>>>> their >> >> >>>>>>>>>>>> experiments >> >> >>>>>>>>>>>> is infinitely more interesting than any results to which >> >> >>>>>>>>>>>> their >> >> >>>>>>>>>>>> experiments >> >> >>>>>>>>>>>> lead. >> >> >>>>>>>>>>>> -- Norbert Wiener >> >> >>>>>>>>> >> >> >>>>>>>>> >> >> >>>>>>>>> >> >> >>>>>>>>> >> >> >>>>>>>>> -- >> >> >>>>>>>>> What most experimenters take for granted before they begin >> >> >>>>>>>>> their >> >> >>>>>>>>> experiments >> >> >>>>>>>>> is infinitely more interesting than any results to which >> >> >>>>>>>>> their >> >> >>>>>>>>> experiments >> >> >>>>>>>>> lead. >> >> >>>>>>>>> -- Norbert Wiener >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>> -- >> >> >>>>>>> What most experimenters take for granted before they begin >> >> >>>>>>> their >> >> >>>>>>> experiments >> >> >>>>>>> is infinitely more interesting than any results to which their >> >> >>>>>>> experiments >> >> >>>>>>> lead. >> >> >>>>>>> -- Norbert Wiener >> >> >>>>> >> >> >>>>> >> >> >>>>> >> >> >>>>> >> >> >>>>> -- >> >> >>>>> What most experimenters take for granted before they begin their >> >> >>>>> experiments >> >> >>>>> is infinitely more interesting than any results to which their >> >> >>>>> experiments >> >> >>>>> lead. >> >> >>>>> -- Norbert Wiener >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> -- >> >> >>> What most experimenters take for granted before they begin their >> >> >>> experiments >> >> >>> is infinitely more interesting than any results to which their >> >> >>> experiments >> >> >>> lead. >> >> >>> -- Norbert Wiener >> >> > >> > >> > >> > >> > >> > -- >> > What most experimenters take for granted before they begin their >> > experiments >> > is infinitely more interesting than any results to which their >> > experiments >> > lead. >> > -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener From wen.zhao at outlook.fr Wed Feb 10 08:06:31 2016 From: wen.zhao at outlook.fr (wen zhao) Date: Wed, 10 Feb 2016 15:06:31 +0100 Subject: [petsc-users] multiplication of two vetors Message-ID: Hello I have another question about the multiplication of two vector with the form (x) * (x^T), which x is of length n, That means, the result is a matrix. Is there some functions in Petsc. Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From epscodes at gmail.com Wed Feb 10 08:12:14 2016 From: epscodes at gmail.com (Xiangdong) Date: Wed, 10 Feb 2016 09:12:14 -0500 Subject: [petsc-users] log_summary time ratio and flops ratio In-Reply-To: <87h9hiahra.fsf@jedbrown.org> References: <874mdmeqsu.fsf@jedbrown.org> <87h9hiahra.fsf@jedbrown.org> Message-ID: On Mon, Feb 8, 2016 at 6:45 PM, Jed Brown wrote: > Xiangdong writes: > > > iii) since the time ratios of VecDot (2.5) and MatMult (1.5) are still > > high, I rerun the program with ipm module. The IPM summary is here: > > > https://drive.google.com/file/d/0BxEfb1tasJxhYXI0VkV0cjlLWUU/view?usp=sharing > . > > From this IPM reuslts, MPI_Allreduce takes 74% of MPI time. The > > communication by task figure (1st figure in p4) in above link showed that > > it is not well-balanced. Is this related to the hardware and network > (which > > the users cannot control) or can I do something on my codes to improve? > > Here are a few functions that don't have any communication, but still > have significant load imbalance. > > VecAXPY 1021815 1.0 2.2148e+01 2.1 1.89e+10 1.1 0.0e+00 0.0e+00 > 0.0e+00 2 4 0 0 0 2 4 0 0 0 207057 > VecMAXPY 613089 1.0 1.3276e+01 2.2 2.27e+10 1.1 0.0e+00 0.0e+00 > 0.0e+00 1 4 0 0 0 1 4 0 0 0 414499 > MatSOR 818390 1.0 1.9608e+02 1.5 2.00e+11 1.1 0.0e+00 0.0e+00 > 0.0e+00 22 40 0 0 0 22 40 0 0 0 247472 > > The result above is from a run with 256 cores (16 nodes * 16 cores/node). I did another run with 64 nodes * 4 cores/node. Now these functions are much better balanced ( a factor of 1.2-1.3, instead of 1.5-2.1). VecAXPY 987215 1.0 6.8469e+00 1.3 1.82e+10 1.1 0.0e+00 0.0e+00 0.0e+00 1 4 0 0 0 1 4 0 0 0 647096 VecMAXPY 592329 1.0 6.0866e+00 1.3 2.19e+10 1.1 0.0e+00 0.0e+00 0.0e+00 1 4 0 0 0 1 4 0 0 0 873511 MatSOR 790717 1.0 1.2933e+02 1.2 1.93e+11 1.1 0.0e+00 0.0e+00 0.0e+00 24 40 0 0 0 24 40 0 0 0 362525 For the functions requires communication, the time ratio is about (1.4-1.6) VecDot 789772 1.0 8.4804e+01 1.4 1.46e+10 1.1 0.0e+00 0.0e+00 7.9e+05 14 3 0 0 40 14 3 0 0 40 41794 VecNorm 597914 1.0 7.6259e+01 1.6 1.10e+10 1.1 0.0e+00 0.0e+00 6.0e+05 12 2 0 0 30 12 2 0 0 30 34996 The full logsummary for this new run is here: https://googledrive.com/host/0BxEfb1tasJxhVkZ2NHJkSmF4LUU Can we say now the load imbalance is from the network communication, instead of memory bandwidth? Thanks. Xiangdong You can and should improve load balance before stressing about network > costs. This could be that the nodes aren't clean (running at different > speeds) or that the partition is not balancing data. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From epscodes at gmail.com Wed Feb 10 08:15:16 2016 From: epscodes at gmail.com (Xiangdong) Date: Wed, 10 Feb 2016 09:15:16 -0500 Subject: [petsc-users] multiplication of two vetors In-Reply-To: References: Message-ID: What are you going to do with this matrix? Are you going to multiply this matrix with another vector? Xiangdong On Wed, Feb 10, 2016 at 9:06 AM, wen zhao wrote: > Hello > > I have another question about the multiplication of two vector with the > form > > (x) * (x^T), which x is of length n, > > That means, the result is a matrix. Is there some functions in Petsc. > > Thanks > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Feb 10 08:31:23 2016 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 10 Feb 2016 08:31:23 -0600 Subject: [petsc-users] log_summary time ratio and flops ratio In-Reply-To: References: <874mdmeqsu.fsf@jedbrown.org> <87h9hiahra.fsf@jedbrown.org> Message-ID: On Wed, Feb 10, 2016 at 8:12 AM, Xiangdong wrote: > On Mon, Feb 8, 2016 at 6:45 PM, Jed Brown wrote: > >> Xiangdong writes: >> >> > iii) since the time ratios of VecDot (2.5) and MatMult (1.5) are still >> > high, I rerun the program with ipm module. The IPM summary is here: >> > >> https://drive.google.com/file/d/0BxEfb1tasJxhYXI0VkV0cjlLWUU/view?usp=sharing >> . >> > From this IPM reuslts, MPI_Allreduce takes 74% of MPI time. The >> > communication by task figure (1st figure in p4) in above link showed >> that >> > it is not well-balanced. Is this related to the hardware and network >> (which >> > the users cannot control) or can I do something on my codes to improve? >> >> Here are a few functions that don't have any communication, but still >> have significant load imbalance. >> >> VecAXPY 1021815 1.0 2.2148e+01 2.1 1.89e+10 1.1 0.0e+00 >> 0.0e+00 0.0e+00 2 4 0 0 0 2 4 0 0 0 207057 >> VecMAXPY 613089 1.0 1.3276e+01 2.2 2.27e+10 1.1 0.0e+00 >> 0.0e+00 0.0e+00 1 4 0 0 0 1 4 0 0 0 414499 >> MatSOR 818390 1.0 1.9608e+02 1.5 2.00e+11 1.1 0.0e+00 >> 0.0e+00 0.0e+00 22 40 0 0 0 22 40 0 0 0 247472 >> >> > The result above is from a run with 256 cores (16 nodes * 16 cores/node). > I did another run with 64 nodes * 4 cores/node. Now these functions are > much better balanced ( a factor of 1.2-1.3, instead of 1.5-2.1). > > VecAXPY 987215 1.0 6.8469e+00 1.3 1.82e+10 1.1 0.0e+00 0.0e+00 > 0.0e+00 1 4 0 0 0 1 4 0 0 0 647096 > VecMAXPY 592329 1.0 6.0866e+00 1.3 2.19e+10 1.1 0.0e+00 0.0e+00 > 0.0e+00 1 4 0 0 0 1 4 0 0 0 873511 > MatSOR 790717 1.0 1.2933e+02 1.2 1.93e+11 1.1 0.0e+00 0.0e+00 > 0.0e+00 24 40 0 0 0 24 40 0 0 0 362525 > > For the functions requires communication, the time ratio is about (1.4-1.6) > VecDot 789772 1.0 8.4804e+01 1.4 1.46e+10 1.1 0.0e+00 0.0e+00 > 7.9e+05 14 3 0 0 40 14 3 0 0 40 41794 > VecNorm 597914 1.0 7.6259e+01 1.6 1.10e+10 1.1 0.0e+00 0.0e+00 > 6.0e+05 12 2 0 0 30 12 2 0 0 30 34996 > > The full logsummary for this new run is here: > https://googledrive.com/host/0BxEfb1tasJxhVkZ2NHJkSmF4LUU > > Can we say now the load imbalance is from the network communication, > instead of memory bandwidth? > Actually now it looks even more like what Jed was saying. The 4 cores have much more available bandwidth. Matt > Thanks. > > Xiangdong > > You can and should improve load balance before stressing about network >> costs. This could be that the nodes aren't clean (running at different >> speeds) or that the partition is not balancing data. >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Feb 10 08:33:12 2016 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 10 Feb 2016 08:33:12 -0600 Subject: [petsc-users] multiplication of two vetors In-Reply-To: References: Message-ID: On Wed, Feb 10, 2016 at 8:06 AM, wen zhao wrote: > Hello > > I have another question about the multiplication of two vector with the > form > > (x) * (x^T), which x is of length n, > > That means, the result is a matrix. Is there some functions in Petsc. > You do not want to form this matrix since it is dense but very low rank. We do not have a separate abstraction for outer product matrices, but it would not be hard to write. However, its not clear it would save people much programming, and it increases the interface size. Thanks, Matt > Thanks > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Wed Feb 10 08:35:21 2016 From: jed at jedbrown.org (Jed Brown) Date: Wed, 10 Feb 2016 07:35:21 -0700 Subject: [petsc-users] log_summary time ratio and flops ratio In-Reply-To: References: <874mdmeqsu.fsf@jedbrown.org> <87h9hiahra.fsf@jedbrown.org> Message-ID: <87h9hg63bq.fsf@jedbrown.org> Xiangdong writes: >> VecAXPY 1021815 1.0 2.2148e+01 2.1 1.89e+10 1.1 0.0e+00 0.0e+00 >> 0.0e+00 2 4 0 0 0 2 4 0 0 0 207057 >> VecMAXPY 613089 1.0 1.3276e+01 2.2 2.27e+10 1.1 0.0e+00 0.0e+00 >> 0.0e+00 1 4 0 0 0 1 4 0 0 0 414499 >> MatSOR 818390 1.0 1.9608e+02 1.5 2.00e+11 1.1 0.0e+00 0.0e+00 >> 0.0e+00 22 40 0 0 0 22 40 0 0 0 247472 >> >> > The result above is from a run with 256 cores (16 nodes * 16 cores/node). I > did another run with 64 nodes * 4 cores/node. Now these functions are much > better balanced ( a factor of 1.2-1.3, instead of 1.5-2.1). > > VecAXPY 987215 1.0 6.8469e+00 1.3 1.82e+10 1.1 0.0e+00 0.0e+00 > 0.0e+00 1 4 0 0 0 1 4 0 0 0 647096 > VecMAXPY 592329 1.0 6.0866e+00 1.3 2.19e+10 1.1 0.0e+00 0.0e+00 > 0.0e+00 1 4 0 0 0 1 4 0 0 0 873511 > MatSOR 790717 1.0 1.2933e+02 1.2 1.93e+11 1.1 0.0e+00 0.0e+00 > 0.0e+00 24 40 0 0 0 24 40 0 0 0 362525 So it's significantly faster in addition to being more balanced. I would attribute that to memory bandwidth. > For the functions requires communication, the time ratio is about (1.4-1.6) > VecDot 789772 1.0 8.4804e+01 1.4 1.46e+10 1.1 0.0e+00 0.0e+00 > 7.9e+05 14 3 0 0 40 14 3 0 0 40 41794 > VecNorm 597914 1.0 7.6259e+01 1.6 1.10e+10 1.1 0.0e+00 0.0e+00 > 6.0e+05 12 2 0 0 30 12 2 0 0 30 34996 > > The full logsummary for this new run is here: > https://googledrive.com/host/0BxEfb1tasJxhVkZ2NHJkSmF4LUU > > Can we say now the load imbalance is from the network communication, > instead of memory bandwidth? It is expected that synchronizing functions like these have higher "load imbalance", but it doesn't necessarily mean the network is running at different speeds for different nodes or some such. Rather, you've accumulated load imbalance over previous operations and now you have to wait for the slowest process before anyone can continue. So now the process that was fastest before logs the longest time for the Norm or Dot. I see 100?s per VecDot above, which is reasonable. If you get more exact load balance in the local computation, you might be able to improve it a bit. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From francesco.magaletti at uniroma1.it Wed Feb 10 09:38:03 2016 From: francesco.magaletti at uniroma1.it (Francesco Magaletti) Date: Wed, 10 Feb 2016 16:38:03 +0100 Subject: [petsc-users] TS and snes_grid_sequence Message-ID: <42B3AE22-CEDD-4FA3-8BA3-E95BD5AF217E@uniroma1.it> Hi everyone, I?m trying to solve a system of time dependent PDE?s, strongly non linear and quite stiff. The system is 1D, but I need a large number of grid points (1-10 million points). I?m facing some convergence problem of the Newton solver: if I use a small timestep (10^-5) the SNES converges correctly, but if I increase it, the Newton method diverges. Moreover, when I increase the number of grid points, the timestep for Newton convergence decreases. Such a restriction on the timestep is a big problem for my simulations. I thought that the option -snes_grid_sequence could help the SNES to converge, but when I run the code the first grid converges and then the simulations abort with errors. Here is an example with 60000grid points: mpiexec -n 4 ./code_petsc -ts_monitor -snes_monitor -snes_converged_reason -ksp_converged_reason -ftime 8.0 -ksp_rtol 1.0e-8 -snes_atol 1.0e-12 -snes_rtol 1.0e-14 -snes_stol 1.0e-12 -snes_max_it 2000 -dt 2.0e-4 -nx 60000 -snes_grid_sequence 1 0 TS dt 0.0002 time 0 0 SNES Function norm 4.597340391780e+04 Linear solve converged due to CONVERGED_RTOL iterations 9 1 SNES Function norm 3.986169680241e+03 Linear solve converged due to CONVERGED_RTOL iterations 14 2 SNES Function norm 4.292840674611e+01 Linear solve converged due to CONVERGED_RTOL iterations 15 3 SNES Function norm 1.044935520416e-02 Linear solve converged due to CONVERGED_RTOL iterations 14 4 SNES Function norm 2.936744161136e-06 Linear solve converged due to CONVERGED_RTOL iterations 10 5 SNES Function norm 4.471992600451e-08 Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 5 [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: [0]PETSC ERROR: Must call TSSetRHSFunction() and / or TSSetIFunction() [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-mpich-device=ch3:nemesis [0]PETSC ERROR: #1 TSComputeIFunction() line 723 in /home/magaletto/Scaricati/petsc-3.6.3/src/ts/interface/ts.c [0]PETSC ERROR: #2 SNESTSFormFunction_Theta() line 483 in /home/magaletto/Scaricati/petsc-3.6.3/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: #3 SNESTSFormFunction() line 4138 in /home/magaletto/Scaricati/petsc-3.6.3/src/ts/interface/ts.c [0]PETSC ERROR: #4 SNESComputeFunction() line 2067 in /home/magaletto/Scaricati/petsc-3.6.3/src/snes/interface/snes.c [0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 184 in /home/magaletto/Scaricati/petsc-3.6.3/src/snes/impls/ls/ls.c [0]PETSC ERROR: #6 SNESSolve() line 3906 in /home/magaletto/Scaricati/petsc-3.6.3/src/snes/interface/snes.c [0]PETSC ERROR: #7 TSStep_Theta() line 198 in /home/magaletto/Scaricati/petsc-3.6.3/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: #8 TSStep() line 3101 in /home/magaletto/Scaricati/petsc-3.6.3/src/ts/interface/ts.c [0]PETSC ERROR: #9 TSSolve() line 3285 in /home/magaletto/Scaricati/petsc-3.6.3/src/ts/interface/ts.c =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = EXIT CODE: 6 Where is the mistake? Could you suggest a "more convergent? SNES? Maybe a FAS as a nonlinear preconditioner? Hope someone can help here. best regards Francesco From bsmith at mcs.anl.gov Wed Feb 10 14:06:49 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 10 Feb 2016 14:06:49 -0600 Subject: [petsc-users] TS and snes_grid_sequence In-Reply-To: <42B3AE22-CEDD-4FA3-8BA3-E95BD5AF217E@uniroma1.it> References: <42B3AE22-CEDD-4FA3-8BA3-E95BD5AF217E@uniroma1.it> Message-ID: <74BC3C1B-75E8-4083-B58A-25A96BFA1364@mcs.anl.gov> Francesco, Unfortunately the -snes_grid_sequence option only works with a single SNES nonlinear solve and not directly within an outer time-integration scheme. We do not currently have library code to do grid sequence within a time-step. I believe that providing this requires a great deal of "replumbing" of the TS solvers to make them "grid sequence aware". This is true I believe for both traditional grid sequencing and FAS (I would be happy if Emil or Jed could prove me wrong). There are seemingly multiple ways to do the grid sequencing: 1) Restrict the current solution to a coarse mesh and do a single stage of the ODE integrator (which means solve a single nonlinear solve) on that coarse mesh, interpolate the solution to the next mesh and use it as an initial guess for the single stage on that mesh and then continue to up the finest mesh. Note that since the specific ODE integrator determines the "recipe" for the SNES function evaluation this is not something that can be done in pure user code, it must use the "plumbing" of the specific ODE integrator. an alternative is 2) Restrict the current solution to a coarse mesh and do the entire timestep of the ODE integrator (which means multiple stage and multiple nonlinear solvers) saving all the intermediate solutions (for each stage). Interpolate all the intermediate solutions (as well as the final solution) to the finer mesh and repeat the entire timestep computation but using the interpolated intermediate solutions as the initial guesses for the nonlinear solves). Continue to the finest mesh. Again this cannot be done in user code and requires replumbing the ODE integrator. I think 2 will produce different "results" from 1 since for 1 the coarse grid initial values on (for example) the second stage will be obtained from restricting the finest grid solution from the first stage while with 2 the initial values on the second stage will be directly the coarse solution from the first stage. To do the FAS is even more involved, I can only conceive of doing it for a single stage. Again the FAS has to be plumbed directly into the specific ODE solver to define the correct functions for the various levels. With the above approaches the coarser nonlinear problems should be "easier" than the finest because the time-step remains small but the space discretization is larger. But the only way the approaches can work is if the interpolated coarse grid solution used on the finer mesh is within the basin of attraction on that mesh, otherwise you will still have the same problem you have now of lack of convergence. I don't know if anyone actually does this and how much it helps or could help. I welcome input from others. Based on your convergence experience it sounds like your solution is changing rather rapidly in time? Is the solution somewhat smooth in space? If not then I don't think grid sequencing will help much. Barry > On Feb 10, 2016, at 9:38 AM, Francesco Magaletti wrote: > > Hi everyone, > > I?m trying to solve a system of time dependent PDE?s, strongly non linear and quite stiff. The system is 1D, but I need a large number of grid points (1-10 million points). I?m facing some convergence problem of the Newton solver: if I use a small timestep (10^-5) the SNES converges correctly, but if I increase it, the Newton method diverges. Moreover, when I increase the number of grid points, the timestep for Newton convergence decreases. Such a restriction on the timestep is a big problem for my simulations. > > I thought that the option -snes_grid_sequence could help the SNES to converge, but when I run the code the first grid converges and then the simulations abort with errors. Here is an example with 60000grid points: > > mpiexec -n 4 ./code_petsc -ts_monitor -snes_monitor -snes_converged_reason -ksp_converged_reason -ftime 8.0 -ksp_rtol 1.0e-8 -snes_atol 1.0e-12 -snes_rtol 1.0e-14 -snes_stol 1.0e-12 -snes_max_it 2000 -dt 2.0e-4 -nx 60000 -snes_grid_sequence 1 > > 0 TS dt 0.0002 time 0 > 0 SNES Function norm 4.597340391780e+04 > Linear solve converged due to CONVERGED_RTOL iterations 9 > 1 SNES Function norm 3.986169680241e+03 > Linear solve converged due to CONVERGED_RTOL iterations 14 > 2 SNES Function norm 4.292840674611e+01 > Linear solve converged due to CONVERGED_RTOL iterations 15 > 3 SNES Function norm 1.044935520416e-02 > Linear solve converged due to CONVERGED_RTOL iterations 14 > 4 SNES Function norm 2.936744161136e-06 > Linear solve converged due to CONVERGED_RTOL iterations 10 > 5 SNES Function norm 4.471992600451e-08 > Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 5 > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: > [0]PETSC ERROR: Must call TSSetRHSFunction() and / or TSSetIFunction() > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-mpich-device=ch3:nemesis > [0]PETSC ERROR: #1 TSComputeIFunction() line 723 in /home/magaletto/Scaricati/petsc-3.6.3/src/ts/interface/ts.c > [0]PETSC ERROR: #2 SNESTSFormFunction_Theta() line 483 in /home/magaletto/Scaricati/petsc-3.6.3/src/ts/impls/implicit/theta/theta.c > [0]PETSC ERROR: #3 SNESTSFormFunction() line 4138 in /home/magaletto/Scaricati/petsc-3.6.3/src/ts/interface/ts.c > [0]PETSC ERROR: #4 SNESComputeFunction() line 2067 in /home/magaletto/Scaricati/petsc-3.6.3/src/snes/interface/snes.c > [0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 184 in /home/magaletto/Scaricati/petsc-3.6.3/src/snes/impls/ls/ls.c > [0]PETSC ERROR: #6 SNESSolve() line 3906 in /home/magaletto/Scaricati/petsc-3.6.3/src/snes/interface/snes.c > [0]PETSC ERROR: #7 TSStep_Theta() line 198 in /home/magaletto/Scaricati/petsc-3.6.3/src/ts/impls/implicit/theta/theta.c > [0]PETSC ERROR: #8 TSStep() line 3101 in /home/magaletto/Scaricati/petsc-3.6.3/src/ts/interface/ts.c > [0]PETSC ERROR: #9 TSSolve() line 3285 in /home/magaletto/Scaricati/petsc-3.6.3/src/ts/interface/ts.c > > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = EXIT CODE: 6 > > > Where is the mistake? > Could you suggest a "more convergent? SNES? Maybe a FAS as a nonlinear preconditioner? > > Hope someone can help here. > best regards > > Francesco From jed at jedbrown.org Wed Feb 10 16:19:45 2016 From: jed at jedbrown.org (Jed Brown) Date: Wed, 10 Feb 2016 15:19:45 -0700 Subject: [petsc-users] TS and snes_grid_sequence In-Reply-To: <74BC3C1B-75E8-4083-B58A-25A96BFA1364@mcs.anl.gov> References: <42B3AE22-CEDD-4FA3-8BA3-E95BD5AF217E@uniroma1.it> <74BC3C1B-75E8-4083-B58A-25A96BFA1364@mcs.anl.gov> Message-ID: <87pow4439a.fsf@jedbrown.org> Barry Smith writes: > We do not currently have library code to do grid sequence within a time-step. I believe that providing this requires a great deal of "replumbing" of the TS solvers to make them "grid sequence aware". This is true I believe for both traditional grid sequencing and FAS (I would be happy if Emil or Jed could prove me wrong). There are seemingly multiple ways to do the grid sequencing: I'm pretty sure you could use TS solved by FAS when Peter and I were working on it. But my test just failed and it looks like some work vectors aren't being obtained correctly. I'm afraid I can't dig into it right now. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From hongzhang at anl.gov Wed Feb 10 16:39:33 2016 From: hongzhang at anl.gov (Hong Zhang) Date: Wed, 10 Feb 2016 16:39:33 -0600 Subject: [petsc-users] TS and snes_grid_sequence In-Reply-To: <74BC3C1B-75E8-4083-B58A-25A96BFA1364@mcs.anl.gov> References: <42B3AE22-CEDD-4FA3-8BA3-E95BD5AF217E@uniroma1.it> <74BC3C1B-75E8-4083-B58A-25A96BFA1364@mcs.anl.gov> Message-ID: There is one particular integration method that may make the life easier ? backward Euler. It has only one stage at each time step. To Francesco: which TS method are you using? Is it backward Euler or the default theta method? The default one is not stiffly accurate, thus not good for stiff problems. Hong > On Feb 10, 2016, at 2:06 PM, Barry Smith wrote: > > > Francesco, > > Unfortunately the -snes_grid_sequence option only works with a single SNES nonlinear solve and not directly within an outer time-integration scheme. > > We do not currently have library code to do grid sequence within a time-step. I believe that providing this requires a great deal of "replumbing" of the TS solvers to make them "grid sequence aware". This is true I believe for both traditional grid sequencing and FAS (I would be happy if Emil or Jed could prove me wrong). There are seemingly multiple ways to do the grid sequencing: > > 1) Restrict the current solution to a coarse mesh and do a single stage of the ODE integrator (which means solve a single nonlinear solve) on that coarse mesh, interpolate the solution to the next mesh and use it as an initial guess for the single stage on that mesh and then continue to up the finest mesh. Note that since the specific ODE integrator determines the "recipe" for the SNES function evaluation this is not something that can be done in pure user code, it must use the "plumbing" of the specific ODE integrator. > > an alternative is > 2) Restrict the current solution to a coarse mesh and do the entire timestep of the ODE integrator (which means multiple stage and multiple nonlinear solvers) saving all the intermediate solutions (for each stage). Interpolate all the intermediate solutions (as well as the final solution) to the finer mesh and repeat the entire timestep computation but using the interpolated intermediate solutions as the initial guesses for the nonlinear solves). Continue to the finest mesh. Again this cannot be done in user code and requires replumbing the ODE integrator. > > I think 2 will produce different "results" from 1 since for 1 the coarse grid initial values on (for example) the second stage will be obtained from restricting the finest grid solution from the first stage while with 2 the initial values on the second stage will be directly the coarse solution from the first stage. > > To do the FAS is even more involved, I can only conceive of doing it for a single stage. Again the FAS has to be plumbed directly into the specific ODE solver to define the correct functions for the various levels. > > With the above approaches the coarser nonlinear problems should be "easier" than the finest because the time-step remains small but the space discretization is larger. But the only way the approaches can work is if the interpolated coarse grid solution used on the finer mesh is within the basin of attraction on that mesh, otherwise you will still have the same problem you have now of lack of convergence. > > I don't know if anyone actually does this and how much it helps or could help. I welcome input from others. > > Based on your convergence experience it sounds like your solution is changing rather rapidly in time? Is the solution somewhat smooth in space? If not then I don't think grid sequencing will help much. > > > Barry > > > > > >> On Feb 10, 2016, at 9:38 AM, Francesco Magaletti wrote: >> >> Hi everyone, >> >> I?m trying to solve a system of time dependent PDE?s, strongly non linear and quite stiff. The system is 1D, but I need a large number of grid points (1-10 million points). I?m facing some convergence problem of the Newton solver: if I use a small timestep (10^-5) the SNES converges correctly, but if I increase it, the Newton method diverges. Moreover, when I increase the number of grid points, the timestep for Newton convergence decreases. Such a restriction on the timestep is a big problem for my simulations. >> >> I thought that the option -snes_grid_sequence could help the SNES to converge, but when I run the code the first grid converges and then the simulations abort with errors. Here is an example with 60000grid points: >> >> mpiexec -n 4 ./code_petsc -ts_monitor -snes_monitor -snes_converged_reason -ksp_converged_reason -ftime 8.0 -ksp_rtol 1.0e-8 -snes_atol 1.0e-12 -snes_rtol 1.0e-14 -snes_stol 1.0e-12 -snes_max_it 2000 -dt 2.0e-4 -nx 60000 -snes_grid_sequence 1 >> >> 0 TS dt 0.0002 time 0 >> 0 SNES Function norm 4.597340391780e+04 >> Linear solve converged due to CONVERGED_RTOL iterations 9 >> 1 SNES Function norm 3.986169680241e+03 >> Linear solve converged due to CONVERGED_RTOL iterations 14 >> 2 SNES Function norm 4.292840674611e+01 >> Linear solve converged due to CONVERGED_RTOL iterations 15 >> 3 SNES Function norm 1.044935520416e-02 >> Linear solve converged due to CONVERGED_RTOL iterations 14 >> 4 SNES Function norm 2.936744161136e-06 >> Linear solve converged due to CONVERGED_RTOL iterations 10 >> 5 SNES Function norm 4.471992600451e-08 >> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 5 >> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [0]PETSC ERROR: >> [0]PETSC ERROR: Must call TSSetRHSFunction() and / or TSSetIFunction() >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-mpich-device=ch3:nemesis >> [0]PETSC ERROR: #1 TSComputeIFunction() line 723 in /home/magaletto/Scaricati/petsc-3.6.3/src/ts/interface/ts.c >> [0]PETSC ERROR: #2 SNESTSFormFunction_Theta() line 483 in /home/magaletto/Scaricati/petsc-3.6.3/src/ts/impls/implicit/theta/theta.c >> [0]PETSC ERROR: #3 SNESTSFormFunction() line 4138 in /home/magaletto/Scaricati/petsc-3.6.3/src/ts/interface/ts.c >> [0]PETSC ERROR: #4 SNESComputeFunction() line 2067 in /home/magaletto/Scaricati/petsc-3.6.3/src/snes/interface/snes.c >> [0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 184 in /home/magaletto/Scaricati/petsc-3.6.3/src/snes/impls/ls/ls.c >> [0]PETSC ERROR: #6 SNESSolve() line 3906 in /home/magaletto/Scaricati/petsc-3.6.3/src/snes/interface/snes.c >> [0]PETSC ERROR: #7 TSStep_Theta() line 198 in /home/magaletto/Scaricati/petsc-3.6.3/src/ts/impls/implicit/theta/theta.c >> [0]PETSC ERROR: #8 TSStep() line 3101 in /home/magaletto/Scaricati/petsc-3.6.3/src/ts/interface/ts.c >> [0]PETSC ERROR: #9 TSSolve() line 3285 in /home/magaletto/Scaricati/petsc-3.6.3/src/ts/interface/ts.c >> >> =================================================================================== >> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> = EXIT CODE: 6 >> >> >> Where is the mistake? >> Could you suggest a "more convergent? SNES? Maybe a FAS as a nonlinear preconditioner? >> >> Hope someone can help here. >> best regards >> >> Francesco > From jed at jedbrown.org Wed Feb 10 16:45:35 2016 From: jed at jedbrown.org (Jed Brown) Date: Wed, 10 Feb 2016 15:45:35 -0700 Subject: [petsc-users] TS and snes_grid_sequence In-Reply-To: References: <42B3AE22-CEDD-4FA3-8BA3-E95BD5AF217E@uniroma1.it> <74BC3C1B-75E8-4083-B58A-25A96BFA1364@mcs.anl.gov> Message-ID: <87lh6s4228.fsf@jedbrown.org> Hong Zhang writes: > There is one particular integration method that may make the life easier ? backward Euler. It has only one stage at each time step. > > To Francesco: which TS method are you using? Is it backward Euler or > the default theta method? Default is forward Euler, which should almost never be used. Stages per step shouldn't be a serious concern. But -snes_grid_sequence will never work for TS (it's a weird option that refines the SNES during the solve -- it doesn't make semantic sense for TS). -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From bsmith at mcs.anl.gov Wed Feb 10 16:46:51 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 10 Feb 2016 16:46:51 -0600 Subject: [petsc-users] TS and snes_grid_sequence In-Reply-To: <87pow4439a.fsf@jedbrown.org> References: <42B3AE22-CEDD-4FA3-8BA3-E95BD5AF217E@uniroma1.it> <74BC3C1B-75E8-4083-B58A-25A96BFA1364@mcs.anl.gov> <87pow4439a.fsf@jedbrown.org> Message-ID: <6AC843B3-E0E5-4CA7-BDE0-13E76FF32533@mcs.anl.gov> > On Feb 10, 2016, at 4:19 PM, Jed Brown wrote: > > Barry Smith writes: >> We do not currently have library code to do grid sequence within a time-step. I believe that providing this requires a great deal of "replumbing" of the TS solvers to make them "grid sequence aware". This is true I believe for both traditional grid sequencing and FAS (I would be happy if Emil or Jed could prove me wrong). There are seemingly multiple ways to do the grid sequencing: > > I'm pretty sure you could use TS solved by FAS when Peter and I were > working on it. But my test just failed and it looks like some work > vectors aren't being obtained correctly. I'm afraid I can't dig into it > right now. Is there an example in the repository? From jed at jedbrown.org Wed Feb 10 17:18:15 2016 From: jed at jedbrown.org (Jed Brown) Date: Wed, 10 Feb 2016 16:18:15 -0700 Subject: [petsc-users] TS and snes_grid_sequence In-Reply-To: <6AC843B3-E0E5-4CA7-BDE0-13E76FF32533@mcs.anl.gov> References: <42B3AE22-CEDD-4FA3-8BA3-E95BD5AF217E@uniroma1.it> <74BC3C1B-75E8-4083-B58A-25A96BFA1364@mcs.anl.gov> <87pow4439a.fsf@jedbrown.org> <6AC843B3-E0E5-4CA7-BDE0-13E76FF32533@mcs.anl.gov> Message-ID: <87a8n840js.fsf@jedbrown.org> Barry Smith writes: >> On Feb 10, 2016, at 4:19 PM, Jed Brown wrote: >> >> Barry Smith writes: >>> We do not currently have library code to do grid sequence within a time-step. I believe that providing this requires a great deal of "replumbing" of the TS solvers to make them "grid sequence aware". This is true I believe for both traditional grid sequencing and FAS (I would be happy if Emil or Jed could prove me wrong). There are seemingly multiple ways to do the grid sequencing: >> >> I'm pretty sure you could use TS solved by FAS when Peter and I were >> working on it. But my test just failed and it looks like some work >> vectors aren't being obtained correctly. I'm afraid I can't dig into it >> right now. > > Is there an example in the repository? $ ./ex25 -ts_type beuler -snes_type fas -da_refine 1 [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Arguments are incompatible [0]PETSC ERROR: Vector local size 42 is not compatible with DMDA local sizes 22 22 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.6.3-3237-gc58487d GIT Date: 2016-01-27 18:34:57 -0600 [0]PETSC ERROR: ./ex25 on a mpich-clang named batura by jed Wed Feb 10 16:13:34 2016 [0]PETSC ERROR: Configure options --download-chaco --download-exodusii --download-sundials --download-triangle --with-fc=0 --with-hdf5-dir=/opt/mpich --with-mpi-dir=/home/jed/usr/ccache/mpich-clang --with-netcdf --with-pcbddc --with-shared-libraries --with-visibility --with-x --with-yaml PETSC_ARCH=mpich-clang [0]PETSC ERROR: #1 DMDAVecGetArray() line 70 in /home/jed/petsc/src/dm/impls/da/dagetarray.c [0]PETSC ERROR: #2 FormRHSFunction() line 210 in /home/jed/petsc/src/ts/examples/tutorials/ex25.c [0]PETSC ERROR: #3 TSComputeRHSFunction() line 627 in /home/jed/petsc/src/ts/interface/ts.c [0]PETSC ERROR: #4 TSComputeIFunction() line 837 in /home/jed/petsc/src/ts/interface/ts.c [0]PETSC ERROR: #5 SNESTSFormFunction_Theta() line 508 in /home/jed/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: #6 SNESTSFormFunction() line 4529 in /home/jed/petsc/src/ts/interface/ts.c [0]PETSC ERROR: #7 SNESComputeFunction() line 2136 in /home/jed/petsc/src/snes/interface/snes.c [0]PETSC ERROR: #8 SNESFASCoarseCorrection() line 659 in /home/jed/petsc/src/snes/impls/fas/fas.c [0]PETSC ERROR: #9 SNESFASCycle_Multiplicative() line 811 in /home/jed/petsc/src/snes/impls/fas/fas.c [0]PETSC ERROR: #10 SNESSolve_FAS() line 980 in /home/jed/petsc/src/snes/impls/fas/fas.c [0]PETSC ERROR: #11 SNESSolve() line 3992 in /home/jed/petsc/src/snes/interface/snes.c [0]PETSC ERROR: #12 TSStep_Theta() line 248 in /home/jed/petsc/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: #13 TSStep() line 3338 in /home/jed/petsc/src/ts/interface/ts.c [0]PETSC ERROR: #14 TSSolve() line 3544 in /home/jed/petsc/src/ts/interface/ts.c [0]PETSC ERROR: #15 main() line 119 in /home/jed/petsc/src/ts/examples/tutorials/ex25.c [0]PETSC ERROR: PETSc Option Table entries: [0]PETSC ERROR: -da_refine 1 [0]PETSC ERROR: -malloc_test [0]PETSC ERROR: -snes_type fas [0]PETSC ERROR: -ts_type beuler [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- Problem here is that TSComputeIFunction is getting the Frhs work vec From the TS rather than something associated with the DM. I don't know if there are other issues later. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From hongzhang at anl.gov Wed Feb 10 17:23:39 2016 From: hongzhang at anl.gov (Hong Zhang) Date: Wed, 10 Feb 2016 17:23:39 -0600 Subject: [petsc-users] TS and snes_grid_sequence In-Reply-To: <87lh6s4228.fsf@jedbrown.org> References: <42B3AE22-CEDD-4FA3-8BA3-E95BD5AF217E@uniroma1.it> <74BC3C1B-75E8-4083-B58A-25A96BFA1364@mcs.anl.gov> <87lh6s4228.fsf@jedbrown.org> Message-ID: <087ED5DC-1450-443D-BF96-86C5ACFCE117@anl.gov> The error messages provided by Francesco indicate a theta method is used in his code. So presumably he may happen to be using the default option of the theta method which is not good for stiff problems. I mentioned the one-stage method just in response to the two ways Barry proposed. It might be easer to do the implementation based on beuler than methods with multi stages. Hong > On Feb 10, 2016, at 4:45 PM, Jed Brown wrote: > > Hong Zhang writes: > >> There is one particular integration method that may make the life easier ? backward Euler. It has only one stage at each time step. >> >> To Francesco: which TS method are you using? Is it backward Euler or >> the default theta method? > > Default is forward Euler, which should almost never be used. > > Stages per step shouldn't be a serious concern. But -snes_grid_sequence > will never work for TS (it's a weird option that refines the SNES during > the solve -- it doesn't make semantic sense for TS). From pvsang002 at gmail.com Wed Feb 10 18:14:06 2016 From: pvsang002 at gmail.com (Sang pham van) Date: Thu, 11 Feb 2016 07:14:06 +0700 Subject: [petsc-users] Create DM matrix Message-ID: Hi, I am trying to create a DM matrix with DMCreateMatrix_DA_3d_MPIAIJ() instead of using DMCreateMatrix(). Which header file should I include to use that routine? also, what is the source file containing the DMCreateMatrix() routine? Many thanks in advance. Pham -------------- next part -------------- An HTML attachment was scrubbed... URL: From francesco.magaletti at uniroma1.it Wed Feb 10 18:33:23 2016 From: francesco.magaletti at uniroma1.it (Francesco Magaletti) Date: Thu, 11 Feb 2016 01:33:23 +0100 Subject: [petsc-users] TS and snes_grid_sequence In-Reply-To: <74BC3C1B-75E8-4083-B58A-25A96BFA1364@mcs.anl.gov> References: <42B3AE22-CEDD-4FA3-8BA3-E95BD5AF217E@uniroma1.it> <74BC3C1B-75E8-4083-B58A-25A96BFA1364@mcs.anl.gov> Message-ID: <3D824BE9-3B94-4CCF-850F-D7C9D9F99145@uniroma1.it> First of all thank you everybody for the fast replies. Barry, I think the solution 1 is a good suggestion but I confess I?m a bit scared of going deep into the modification of the ODE integrator. I?m now using TSCN, the classical Crank-Nicolson 2nd order full implicit time integration. Maybe I can try to implement your solution together with BEULER, as suggested by Hong. By the way the solution changes rapidly, but not so fast to justify that small timestep value. The space variation instead are really fast and this is the reason why I need so many grid points. Jed, I hope you could solve soon the problem with the FAS scheme. Thank you again Francesco From jed at jedbrown.org Wed Feb 10 18:39:24 2016 From: jed at jedbrown.org (Jed Brown) Date: Wed, 10 Feb 2016 17:39:24 -0700 Subject: [petsc-users] TS and snes_grid_sequence In-Reply-To: <3D824BE9-3B94-4CCF-850F-D7C9D9F99145@uniroma1.it> References: <42B3AE22-CEDD-4FA3-8BA3-E95BD5AF217E@uniroma1.it> <74BC3C1B-75E8-4083-B58A-25A96BFA1364@mcs.anl.gov> <3D824BE9-3B94-4CCF-850F-D7C9D9F99145@uniroma1.it> Message-ID: <87y4as2i83.fsf@jedbrown.org> Francesco Magaletti writes: > First of all thank you everybody for the fast replies. > > Barry, > I think the solution 1 is a good suggestion but I confess I?m a bit scared of going deep into the modification of the ODE integrator. > I?m now using TSCN, the classical Crank-Nicolson 2nd order full > implicit time integration. Unless you have experience for this problem, you should consider whether you'd be better off with an L-stable method (like a BDF or DIRK). > Maybe I can try to implement your solution together with BEULER, as > suggested by Hong. By the way the solution changes rapidly, but not > so fast to justify that small timestep value. How did you measure that? It's changing enough that the nonlinear solve is difficult. If you use almost any other implicit method in PETSc (or -ts_type sundials), you'll get an error-based adaptive controller. I do recommend trying those; you might find that shorter time steps are needed for accuracy too, in which case FAS is moot (this is typical). -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From emconsta at mcs.anl.gov Wed Feb 10 19:13:01 2016 From: emconsta at mcs.anl.gov (Emil Constantinescu) Date: Wed, 10 Feb 2016 19:13:01 -0600 Subject: [petsc-users] TS and snes_grid_sequence In-Reply-To: <3D824BE9-3B94-4CCF-850F-D7C9D9F99145@uniroma1.it> References: <42B3AE22-CEDD-4FA3-8BA3-E95BD5AF217E@uniroma1.it> <74BC3C1B-75E8-4083-B58A-25A96BFA1364@mcs.anl.gov> <3D824BE9-3B94-4CCF-850F-D7C9D9F99145@uniroma1.it> Message-ID: <56BBE01D.5010603@mcs.anl.gov> Francesco, It could be a problem of ODE stability, error control, nonlinear solver (cold start), etc. In addition to what's been recommended, I would suggest trying a different integrator with better properties: try -ts_type arkimex -ts_arkimex_type 2e -ts_arkimex_fully_implicit In case it's a cold-start problem, arkimex methods can provide warmer guesses: to do that try adding -ts_arkimex_initial_guess_extrapolate To take off the error controller, use -ts_adapt_type none This should take care of the common things. Emil On 2/10/16 6:33 PM, Francesco Magaletti wrote: > First of all thank you everybody for the fast replies. > > Barry, > I think the solution 1 is a good suggestion but I confess I?m a bit scared of going deep into the modification of the ODE integrator. > I?m now using TSCN, the classical Crank-Nicolson 2nd order full implicit time integration. Maybe I can try to implement your solution > together with BEULER, as suggested by Hong. > By the way the solution changes rapidly, but not so fast to justify that small timestep value. The space variation instead are really fast and this is the reason why I need so many grid points. > > Jed, > I hope you could solve soon the problem with the FAS scheme. > > Thank you again > Francesco > > From francesco.magaletti at uniroma1.it Wed Feb 10 19:31:14 2016 From: francesco.magaletti at uniroma1.it (Francesco Magaletti) Date: Thu, 11 Feb 2016 02:31:14 +0100 Subject: [petsc-users] TS and snes_grid_sequence In-Reply-To: <56BBE01D.5010603@mcs.anl.gov> References: <42B3AE22-CEDD-4FA3-8BA3-E95BD5AF217E@uniroma1.it> <74BC3C1B-75E8-4083-B58A-25A96BFA1364@mcs.anl.gov> <3D824BE9-3B94-4CCF-850F-D7C9D9F99145@uniroma1.it> <56BBE01D.5010603@mcs.anl.gov> Message-ID: <76F512BD-AC5B-4B66-9F11-658E56A0B418@uniroma1.it> Dear Jed and Emil, your suggestion to use L-stable time integrators gave great results, since the SNES now converges with higher timestep values. The arkimex 2e performs very well, in particular with the extrapolation of the initial guess. Thank you for your time Francesco > Il giorno 11/feb/2016, alle ore 02:13, Emil Constantinescu ha scritto: > > -ts_arkimex_initial_guess_extrapolate -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrosso at uci.edu Wed Feb 10 19:33:18 2016 From: mrosso at uci.edu (Michele Rosso) Date: Wed, 10 Feb 2016 17:33:18 -0800 Subject: [petsc-users] DIVERGED_PCSETUP_FAILED Message-ID: <1455154398.5948.15.camel@enterprise-A> Hi, I encountered the following error while solving a symmetric positive defined system: Linear solve did not converge due to DIVERGED_PCSETUP_FAILED iterations 0 PCSETUP_FAILED due to SUBPC_ERROR This error appears only if I use the optimized version of both petsc and my code ( compiler: gfortran, flags: -O3 ). It is weird since I am solving a time-dependent problem and everything, i.e. results and convergence rate, are as expected until the above error shows up. If I run both petsc and my code in debug mode, everything goes smooth till the end of the simulation. However, if I reduce the ksp_rtol, even the debug run fails, after running as expected for a while, because of a KSP_DIVERGED_INDEFINITE_PC . The options I am using are: -ksp_type cg -ksp_norm_type unpreconditioned -ksp_rtol 1e-8 -ksp_lag_norm -ksp_initial_guess_nonzero yes -pc_type mg -pc_mg_galerkin -pc_mg_levels 4 -mg_levels_ksp_type richardson -mg_coarse_ksp_constant_null_space -mg_coarse_pc_type lu -mg_coarse_pc_factor_mat_solver_package superlu_dist -options_left I attached a copy of ksp_view. I am currently using petsc-master (last updated yesterday). I would appreciate any suggestion on this matter. Thanks, Michele -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- KSP Object: 1 MPI processes type: cg maximum iterations=10000 tolerances: relative=1e-08, absolute=1e-50, divergence=10000. left preconditioning using nonzero initial guess using UNPRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: mg MG: type is MULTIPLICATIVE, levels=4 cycles=v Cycles per PCApply=1 Using Galerkin computed coarse grid matrices Coarse grid solver -- level ------------------------------- KSP Object: (mg_coarse_) 1 MPI processes type: preonly maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (mg_coarse_) 1 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 using diagonal shift on blocks to prevent zero pivot [INBLOCKS] matrix ordering: nd factor fill ratio given 0., needed 0. Factored matrix follows: Mat Object: 1 MPI processes type: seqaij rows=16, cols=16 package used to perform factorization: superlu_dist total: nonzeros=0, allocated nonzeros=0 total number of mallocs used during MatSetValues calls =0 SuperLU_DIST run parameters: Process grid nprow 1 x npcol 1 Equilibrate matrix TRUE Matrix input mode 0 Replace tiny pivots FALSE Use iterative refinement FALSE Processors in row 1 col partition 1 Row permutation LargeDiag Column permutation METIS_AT_PLUS_A Parallel symbolic factorization FALSE Repeated factorization SamePattern linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=16, cols=16 total: nonzeros=72, allocated nonzeros=72 total number of mallocs used during MatSetValues calls =0 not using I-node routines Down solver (pre-smoother) on level 1 ------------------------------- KSP Object: (mg_levels_1_) 1 MPI processes type: richardson Richardson: damping factor=1. maximum iterations=2 tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (mg_levels_1_) 1 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=64, cols=64 total: nonzeros=304, allocated nonzeros=304 total number of mallocs used during MatSetValues calls =0 not using I-node routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 2 ------------------------------- KSP Object: (mg_levels_2_) 1 MPI processes type: richardson Richardson: damping factor=1. maximum iterations=2 tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (mg_levels_2_) 1 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=256, cols=256 total: nonzeros=1248, allocated nonzeros=1248 total number of mallocs used during MatSetValues calls =0 not using I-node routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 3 ------------------------------- KSP Object: (mg_levels_3_) 1 MPI processes type: richardson Richardson: damping factor=1. maximum iterations=2 tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (mg_levels_3_) 1 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=1024, cols=1024 total: nonzeros=5056, allocated nonzeros=5056 total number of mallocs used during MatSetValues calls =0 has attached null space not using I-node routines Up solver (post-smoother) same as down solver (pre-smoother) linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=1024, cols=1024 total: nonzeros=5056, allocated nonzeros=5056 total number of mallocs used during MatSetValues calls =0 has attached null space not using I-node routines From knepley at gmail.com Wed Feb 10 19:35:29 2016 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 10 Feb 2016 19:35:29 -0600 Subject: [petsc-users] Create DM matrix In-Reply-To: References: Message-ID: On Wed, Feb 10, 2016 at 6:14 PM, Sang pham van wrote: > Hi, > > I am trying to create a DM matrix with DMCreateMatrix_DA_3d_MPIAIJ() > instead of using DMCreateMatrix(). > Why, that should be called automatically by DMCreateMatrix()? Matt > Which header file should I include to use that routine? also, what is the > source file containing the DMCreateMatrix() routine? > > Many thanks in advance. > Pham > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Wed Feb 10 19:36:34 2016 From: jed at jedbrown.org (Jed Brown) Date: Wed, 10 Feb 2016 18:36:34 -0700 Subject: [petsc-users] TS and snes_grid_sequence In-Reply-To: <76F512BD-AC5B-4B66-9F11-658E56A0B418@uniroma1.it> References: <42B3AE22-CEDD-4FA3-8BA3-E95BD5AF217E@uniroma1.it> <74BC3C1B-75E8-4083-B58A-25A96BFA1364@mcs.anl.gov> <3D824BE9-3B94-4CCF-850F-D7C9D9F99145@uniroma1.it> <56BBE01D.5010603@mcs.anl.gov> <76F512BD-AC5B-4B66-9F11-658E56A0B418@uniroma1.it> Message-ID: <87k2mc2fkt.fsf@jedbrown.org> Francesco Magaletti writes: > Dear Jed and Emil, > your suggestion to use L-stable time integrators gave great results, since the SNES now converges with higher timestep values. > The arkimex 2e performs very well, in particular with the extrapolation of the initial guess. Excellent! -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From knepley at gmail.com Wed Feb 10 19:37:16 2016 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 10 Feb 2016 19:37:16 -0600 Subject: [petsc-users] DIVERGED_PCSETUP_FAILED In-Reply-To: <1455154398.5948.15.camel@enterprise-A> References: <1455154398.5948.15.camel@enterprise-A> Message-ID: On Wed, Feb 10, 2016 at 7:33 PM, Michele Rosso wrote: > Hi, > > I encountered the following error while solving a symmetric positive > defined system: > > Linear solve did not converge due to DIVERGED_PCSETUP_FAILED iterations 0 > PCSETUP_FAILED due to SUBPC_ERROR > > This error appears only if I use the optimized version of both petsc and > my code ( compiler: gfortran, flags: -O3 ). > It is weird since I am solving a time-dependent problem and everything, > i.e. results and convergence rate, are as expected until the above error > shows up. If I run both petsc and my code in debug mode, everything goes > smooth till the end of the simulation. > However, if I reduce the ksp_rtol, even the debug run fails, after running > as expected for a while, because of a KSP_DIVERGED_INDEFINITE_PC . > The options I am using are: > > -ksp_type cg > -ksp_norm_type unpreconditioned > -ksp_rtol 1e-8 > -ksp_lag_norm > -ksp_initial_guess_nonzero yes > -pc_type mg > -pc_mg_galerkin > -pc_mg_levels 4 > -mg_levels_ksp_type richardson > -mg_coarse_ksp_constant_null_space > -mg_coarse_pc_type lu > -mg_coarse_pc_factor_mat_solver_package superlu_dist > -options_left > > I attached a copy of ksp_view. I am currently using petsc-master (last > updated yesterday). > I would appreciate any suggestion on this matter. > I suspect you have a nonlinear PC. Can you send the output of -ksp_view? Matt > Thanks, > Michele > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Feb 10 19:38:57 2016 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 10 Feb 2016 19:38:57 -0600 Subject: [petsc-users] TS and snes_grid_sequence In-Reply-To: <87pow4439a.fsf@jedbrown.org> References: <42B3AE22-CEDD-4FA3-8BA3-E95BD5AF217E@uniroma1.it> <74BC3C1B-75E8-4083-B58A-25A96BFA1364@mcs.anl.gov> <87pow4439a.fsf@jedbrown.org> Message-ID: On Wed, Feb 10, 2016 at 4:19 PM, Jed Brown wrote: > Barry Smith writes: > > We do not currently have library code to do grid sequence within a > time-step. I believe that providing this requires a great deal of > "replumbing" of the TS solvers to make them "grid sequence aware". This is > true I believe for both traditional grid sequencing and FAS (I would be > happy if Emil or Jed could prove me wrong). There are seemingly multiple > ways to do the grid sequencing: > > I'm pretty sure you could use TS solved by FAS when Peter and I were > working on it. But my test just failed and it looks like some work > vectors aren't being obtained correctly. I'm afraid I can't dig into it > right now. > I use FAS with TS for the magma runs and it works. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Wed Feb 10 19:43:32 2016 From: jed at jedbrown.org (Jed Brown) Date: Wed, 10 Feb 2016 18:43:32 -0700 Subject: [petsc-users] TS and snes_grid_sequence In-Reply-To: References: <42B3AE22-CEDD-4FA3-8BA3-E95BD5AF217E@uniroma1.it> <74BC3C1B-75E8-4083-B58A-25A96BFA1364@mcs.anl.gov> <87pow4439a.fsf@jedbrown.org> Message-ID: <87h9hg2f97.fsf@jedbrown.org> Matthew Knepley writes: > On Wed, Feb 10, 2016 at 4:19 PM, Jed Brown wrote: > >> Barry Smith writes: >> > We do not currently have library code to do grid sequence within a >> time-step. I believe that providing this requires a great deal of >> "replumbing" of the TS solvers to make them "grid sequence aware". This is >> true I believe for both traditional grid sequencing and FAS (I would be >> happy if Emil or Jed could prove me wrong). There are seemingly multiple >> ways to do the grid sequencing: >> >> I'm pretty sure you could use TS solved by FAS when Peter and I were >> working on it. But my test just failed and it looks like some work >> vectors aren't being obtained correctly. I'm afraid I can't dig into it >> right now. >> > > I use FAS with TS for the magma runs and it works. The problem I encountered would only affect the case where you have both an IFunction and an RHSFunction with a fully implicit method. Thanks for the report that it works otherwise. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From mrosso at uci.edu Wed Feb 10 20:21:31 2016 From: mrosso at uci.edu (Michele Rosso) Date: Wed, 10 Feb 2016 18:21:31 -0800 Subject: [petsc-users] DIVERGED_PCSETUP_FAILED In-Reply-To: References: <1455154398.5948.15.camel@enterprise-A> Message-ID: <1455157291.5948.19.camel@enterprise-A> Hi Matt, the ksp_view output was an attachment to my previous email. Here it is: KSP Object: 1 MPI processes type: cg maximum iterations=10000 tolerances: relative=1e-08, absolute=1e-50, divergence=10000. left preconditioning using nonzero initial guess using UNPRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: mg MG: type is MULTIPLICATIVE, levels=4 cycles=v Cycles per PCApply=1 Using Galerkin computed coarse grid matrices Coarse grid solver -- level ------------------------------- KSP Object: (mg_coarse_) 1 MPI processes type: preonly maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (mg_coarse_) 1 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 using diagonal shift on blocks to prevent zero pivot [INBLOCKS] matrix ordering: nd factor fill ratio given 0., needed 0. Factored matrix follows: Mat Object: 1 MPI processes type: seqaij rows=16, cols=16 package used to perform factorization: superlu_dist total: nonzeros=0, allocated nonzeros=0 total number of mallocs used during MatSetValues calls =0 SuperLU_DIST run parameters: Process grid nprow 1 x npcol 1 Equilibrate matrix TRUE Matrix input mode 0 Replace tiny pivots FALSE Use iterative refinement FALSE Processors in row 1 col partition 1 Row permutation LargeDiag Column permutation METIS_AT_PLUS_A Parallel symbolic factorization FALSE Repeated factorization SamePattern linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=16, cols=16 total: nonzeros=72, allocated nonzeros=72 total number of mallocs used during MatSetValues calls =0 not using I-node routines Down solver (pre-smoother) on level 1 ------------------------------- KSP Object: (mg_levels_1_) 1 MPI processes type: richardson Richardson: damping factor=1. maximum iterations=2 tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (mg_levels_1_) 1 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=64, cols=64 total: nonzeros=304, allocated nonzeros=304 total number of mallocs used during MatSetValues calls =0 not using I-node routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 2 ------------------------------- KSP Object: (mg_levels_2_) 1 MPI processes type: richardson Richardson: damping factor=1. maximum iterations=2 tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (mg_levels_2_) 1 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=256, cols=256 total: nonzeros=1248, allocated nonzeros=1248 total number of mallocs used during MatSetValues calls =0 not using I-node routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 3 ------------------------------- KSP Object: (mg_levels_3_) 1 MPI processes type: richardson Richardson: damping factor=1. maximum iterations=2 tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (mg_levels_3_) 1 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=1024, cols=1024 total: nonzeros=5056, allocated nonzeros=5056 total number of mallocs used during MatSetValues calls =0 has attached null space not using I-node routines Up solver (post-smoother) same as down solver (pre-smoother) linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=1024, cols=1024 total: nonzeros=5056, allocated nonzeros=5056 total number of mallocs used during MatSetValues calls =0 has attached null space not using I-node routines Michele On Wed, 2016-02-10 at 19:37 -0600, Matthew Knepley wrote: > On Wed, Feb 10, 2016 at 7:33 PM, Michele Rosso wrote: > > Hi, > > I encountered the following error while solving a symmetric > positive defined system: > > Linear solve did not converge due to DIVERGED_PCSETUP_FAILED > iterations 0 > PCSETUP_FAILED due to SUBPC_ERROR > > This error appears only if I use the optimized version of > both petsc and my code ( compiler: gfortran, flags: -O3 ). > It is weird since I am solving a time-dependent problem and > everything, i.e. results and convergence rate, are as expected > until the above error shows up. If I run both petsc and my > code in debug mode, everything goes smooth till the end of the > simulation. > However, if I reduce the ksp_rtol, even the debug run fails, > after running as expected for a while, because of a > KSP_DIVERGED_INDEFINITE_PC . > The options I am using are: > > -ksp_type cg > -ksp_norm_type unpreconditioned > -ksp_rtol 1e-8 > -ksp_lag_norm > -ksp_initial_guess_nonzero yes > -pc_type mg > -pc_mg_galerkin > -pc_mg_levels 4 > -mg_levels_ksp_type richardson > -mg_coarse_ksp_constant_null_space > -mg_coarse_pc_type lu > -mg_coarse_pc_factor_mat_solver_package superlu_dist > -options_left > > I attached a copy of ksp_view. I am currently using > petsc-master (last updated yesterday). > I would appreciate any suggestion on this matter. > > > > I suspect you have a nonlinear PC. Can you send the output of > -ksp_view? > > > Matt > > Thanks, > Michele > > > > > > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From pvsang002 at gmail.com Wed Feb 10 20:44:10 2016 From: pvsang002 at gmail.com (Sang pham van) Date: Thu, 11 Feb 2016 09:44:10 +0700 Subject: [petsc-users] Create DM matrix In-Reply-To: References: Message-ID: That is because my matrix has some rows which need more entries than usual. Where can i find source code of DMCreateMatrix()? Pham. On Feb 11, 2016 8:35 AM, "Matthew Knepley" wrote: > On Wed, Feb 10, 2016 at 6:14 PM, Sang pham van > wrote: > >> Hi, >> >> I am trying to create a DM matrix with DMCreateMatrix_DA_3d_MPIAIJ() >> instead of using DMCreateMatrix(). >> > > Why, that should be called automatically by DMCreateMatrix()? > > Matt > > >> Which header file should I include to use that routine? also, what is the >> source file containing the DMCreateMatrix() routine? >> >> Many thanks in advance. >> Pham >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Feb 10 20:52:29 2016 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 10 Feb 2016 20:52:29 -0600 Subject: [petsc-users] Create DM matrix In-Reply-To: References: Message-ID: On Wed, Feb 10, 2016 at 8:44 PM, Sang pham van wrote: > That is because my matrix has some rows which need more entries than usual. > If its only a few, you could just turn off the allocation error. > Where can i find source code of DMCreateMatrix()? > > https://bitbucket.org/petsc/petsc/src/827b69d6bb12709ff9b9a0dede31640477fc2b74/src/dm/impls/da/fdda.c?at=master&fileviewer=file-view-default#fdda.c-1024 Matt > Pham. > On Feb 11, 2016 8:35 AM, "Matthew Knepley" wrote: > >> On Wed, Feb 10, 2016 at 6:14 PM, Sang pham van >> wrote: >> >>> Hi, >>> >>> I am trying to create a DM matrix with DMCreateMatrix_DA_3d_MPIAIJ() >>> instead of using DMCreateMatrix(). >>> >> >> Why, that should be called automatically by DMCreateMatrix()? >> >> Matt >> >> >>> Which header file should I include to use that routine? also, what is >>> the source file containing the DMCreateMatrix() routine? >>> >>> Many thanks in advance. >>> Pham >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From pvsang002 at gmail.com Wed Feb 10 20:59:41 2016 From: pvsang002 at gmail.com (Sang pham van) Date: Thu, 11 Feb 2016 09:59:41 +0700 Subject: [petsc-users] Create DM matrix In-Reply-To: References: Message-ID: The irregular rows is quite many. The matrix really needs to be preallocated. Could you show me how to use DMCreateMatrix_DA_3d_MPIAIJ() directly? Pham On Feb 11, 2016 9:52 AM, "Matthew Knepley" wrote: > On Wed, Feb 10, 2016 at 8:44 PM, Sang pham van > wrote: > >> That is because my matrix has some rows which need more entries than >> usual. >> > > If its only a few, you could just turn off the allocation error. > > >> Where can i find source code of DMCreateMatrix()? >> >> > > https://bitbucket.org/petsc/petsc/src/827b69d6bb12709ff9b9a0dede31640477fc2b74/src/dm/impls/da/fdda.c?at=master&fileviewer=file-view-default#fdda.c-1024 > > Matt > >> Pham. >> On Feb 11, 2016 8:35 AM, "Matthew Knepley" wrote: >> >>> On Wed, Feb 10, 2016 at 6:14 PM, Sang pham van >>> wrote: >>> >>>> Hi, >>>> >>>> I am trying to create a DM matrix with DMCreateMatrix_DA_3d_MPIAIJ() >>>> instead of using DMCreateMatrix(). >>>> >>> >>> Why, that should be called automatically by DMCreateMatrix()? >>> >>> Matt >>> >>> >>>> Which header file should I include to use that routine? also, what is >>>> the source file containing the DMCreateMatrix() routine? >>>> >>>> Many thanks in advance. >>>> Pham >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Wed Feb 10 21:00:20 2016 From: hzhang at mcs.anl.gov (Hong) Date: Wed, 10 Feb 2016 21:00:20 -0600 Subject: [petsc-users] DIVERGED_PCSETUP_FAILED In-Reply-To: <1455157291.5948.19.camel@enterprise-A> References: <1455154398.5948.15.camel@enterprise-A> <1455157291.5948.19.camel@enterprise-A> Message-ID: Michele : Superlu_dist LU is used for coarse grid PC, which likely produces a zero-pivot. Run your code with '-info |grep pivot' to verify. Hong Hi Matt, > > the ksp_view output was an attachment to my previous email. > Here it is: > > KSP Object: 1 MPI processes > type: cg > maximum iterations=10000 > tolerances: relative=1e-08, absolute=1e-50, divergence=10000. > left preconditioning > using nonzero initial guess > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: mg > MG: type is MULTIPLICATIVE, levels=4 cycles=v > Cycles per PCApply=1 > Using Galerkin computed coarse grid matrices > Coarse grid solver -- level ------------------------------- > KSP Object: (mg_coarse_) 1 MPI processes > type: preonly > maximum iterations=1, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (mg_coarse_) 1 MPI processes > type: lu > LU: out-of-place factorization > tolerance for zero pivot 2.22045e-14 > using diagonal shift on blocks to prevent zero pivot [INBLOCKS] > matrix ordering: nd > factor fill ratio given 0., needed 0. > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqaij > rows=16, cols=16 > package used to perform factorization: superlu_dist > total: nonzeros=0, allocated nonzeros=0 > total number of mallocs used during MatSetValues calls =0 > SuperLU_DIST run parameters: > Process grid nprow 1 x npcol 1 > Equilibrate matrix TRUE > Matrix input mode 0 > Replace tiny pivots FALSE > Use iterative refinement FALSE > Processors in row 1 col partition 1 > Row permutation LargeDiag > Column permutation METIS_AT_PLUS_A > Parallel symbolic factorization FALSE > Repeated factorization SamePattern > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=16, cols=16 > total: nonzeros=72, allocated nonzeros=72 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > Down solver (pre-smoother) on level 1 ------------------------------- > KSP Object: (mg_levels_1_) 1 MPI processes > type: richardson > Richardson: damping factor=1. > maximum iterations=2 > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using nonzero initial guess > using NONE norm type for convergence test > PC Object: (mg_levels_1_) 1 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, > omega = 1. > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=64, cols=64 > total: nonzeros=304, allocated nonzeros=304 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > Up solver (post-smoother) same as down solver (pre-smoother) > Down solver (pre-smoother) on level 2 ------------------------------- > KSP Object: (mg_levels_2_) 1 MPI processes > type: richardson > Richardson: damping factor=1. > maximum iterations=2 > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using nonzero initial guess > using NONE norm type for convergence test > PC Object: (mg_levels_2_) 1 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, > omega = 1. > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=256, cols=256 > total: nonzeros=1248, allocated nonzeros=1248 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > Up solver (post-smoother) same as down solver (pre-smoother) > Down solver (pre-smoother) on level 3 ------------------------------- > KSP Object: (mg_levels_3_) 1 MPI processes > type: richardson > Richardson: damping factor=1. > maximum iterations=2 > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using nonzero initial guess > using NONE norm type for convergence test > PC Object: (mg_levels_3_) 1 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, > omega = 1. > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=1024, cols=1024 > total: nonzeros=5056, allocated nonzeros=5056 > total number of mallocs used during MatSetValues calls =0 > has attached null space > not using I-node routines > Up solver (post-smoother) same as down solver (pre-smoother) > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=1024, cols=1024 > total: nonzeros=5056, allocated nonzeros=5056 > total number of mallocs used during MatSetValues calls =0 > has attached null space > not using I-node routines > > > Michele > > > > > On Wed, 2016-02-10 at 19:37 -0600, Matthew Knepley wrote: > > On Wed, Feb 10, 2016 at 7:33 PM, Michele Rosso wrote: > > Hi, > > I encountered the following error while solving a symmetric positive > defined system: > > Linear solve did not converge due to DIVERGED_PCSETUP_FAILED iterations 0 > PCSETUP_FAILED due to SUBPC_ERROR > > This error appears only if I use the optimized version of both petsc and > my code ( compiler: gfortran, flags: -O3 ). > It is weird since I am solving a time-dependent problem and everything, > i.e. results and convergence rate, are as expected until the above error > shows up. If I run both petsc and my code in debug mode, everything goes > smooth till the end of the simulation. > However, if I reduce the ksp_rtol, even the debug run fails, after running > as expected for a while, because of a KSP_DIVERGED_INDEFINITE_PC . > The options I am using are: > > -ksp_type cg > -ksp_norm_type unpreconditioned > -ksp_rtol 1e-8 > -ksp_lag_norm > -ksp_initial_guess_nonzero yes > -pc_type mg > -pc_mg_galerkin > -pc_mg_levels 4 > -mg_levels_ksp_type richardson > -mg_coarse_ksp_constant_null_space > -mg_coarse_pc_type lu > -mg_coarse_pc_factor_mat_solver_package superlu_dist > -options_left > > I attached a copy of ksp_view. I am currently using petsc-master (last > updated yesterday). > I would appreciate any suggestion on this matter. > > > > I suspect you have a nonlinear PC. Can you send the output of -ksp_view? > > > > Matt > > > > Thanks, > Michele > > > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Feb 10 21:07:00 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 10 Feb 2016 21:07:00 -0600 Subject: [petsc-users] TS and snes_grid_sequence In-Reply-To: References: <42B3AE22-CEDD-4FA3-8BA3-E95BD5AF217E@uniroma1.it> <74BC3C1B-75E8-4083-B58A-25A96BFA1364@mcs.anl.gov> <87pow4439a.fsf@jedbrown.org> Message-ID: > On Feb 10, 2016, at 7:38 PM, Matthew Knepley wrote: > > On Wed, Feb 10, 2016 at 4:19 PM, Jed Brown wrote: > Barry Smith writes: > > We do not currently have library code to do grid sequence within a time-step. I believe that providing this requires a great deal of "replumbing" of the TS solvers to make them "grid sequence aware". This is true I believe for both traditional grid sequencing and FAS (I would be happy if Emil or Jed could prove me wrong). There are seemingly multiple ways to do the grid sequencing: > > I'm pretty sure you could use TS solved by FAS when Peter and I were > working on it. But my test just failed and it looks like some work > vectors aren't being obtained correctly. I'm afraid I can't dig into it > right now. > > I use FAS with TS for the magma runs and it works. Ok, there must be some code hidden away in TS that I didn't know about to have this work. > > Matt > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From knepley at gmail.com Wed Feb 10 21:08:36 2016 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 10 Feb 2016 21:08:36 -0600 Subject: [petsc-users] Create DM matrix In-Reply-To: References: Message-ID: On Wed, Feb 10, 2016 at 8:59 PM, Sang pham van wrote: > The irregular rows is quite many. The matrix really needs to be > preallocated. > Could you show me how to use DMCreateMatrix_DA_3d_MPIAIJ() directly? > > Just put the declaration right into your source. Matt > Pham > On Feb 11, 2016 9:52 AM, "Matthew Knepley" wrote: > >> On Wed, Feb 10, 2016 at 8:44 PM, Sang pham van >> wrote: >> >>> That is because my matrix has some rows which need more entries than >>> usual. >>> >> >> If its only a few, you could just turn off the allocation error. >> >> >>> Where can i find source code of DMCreateMatrix()? >>> >>> >> >> https://bitbucket.org/petsc/petsc/src/827b69d6bb12709ff9b9a0dede31640477fc2b74/src/dm/impls/da/fdda.c?at=master&fileviewer=file-view-default#fdda.c-1024 >> >> Matt >> >>> Pham. >>> On Feb 11, 2016 8:35 AM, "Matthew Knepley" wrote: >>> >>>> On Wed, Feb 10, 2016 at 6:14 PM, Sang pham van >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> I am trying to create a DM matrix with DMCreateMatrix_DA_3d_MPIAIJ() >>>>> instead of using DMCreateMatrix(). >>>>> >>>> >>>> Why, that should be called automatically by DMCreateMatrix()? >>>> >>>> Matt >>>> >>>> >>>>> Which header file should I include to use that routine? also, what is >>>>> the source file containing the DMCreateMatrix() routine? >>>>> >>>>> Many thanks in advance. >>>>> Pham >>>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Feb 10 21:22:33 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 10 Feb 2016 21:22:33 -0600 Subject: [petsc-users] DIVERGED_PCSETUP_FAILED In-Reply-To: References: <1455154398.5948.15.camel@enterprise-A> <1455157291.5948.19.camel@enterprise-A> Message-ID: <0903A02C-4B7A-4C1F-AD27-44CB6B1C6F07@mcs.anl.gov> > On Feb 10, 2016, at 9:00 PM, Hong wrote: > > Michele : > Superlu_dist LU is used for coarse grid PC, which likely produces a zero-pivot. > Run your code with '-info |grep pivot' to verify. Michele You can also run with -ksp_error_if_not_converged in or not in the debugger and it will stop immediately when the problem is detected and hopefully provide additional useful information about what has happened. Barry > > Hong > > Hi Matt, > > the ksp_view output was an attachment to my previous email. > Here it is: > > KSP Object: 1 MPI processes > type: cg > maximum iterations=10000 > tolerances: relative=1e-08, absolute=1e-50, divergence=10000. > left preconditioning > using nonzero initial guess > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: mg > MG: type is MULTIPLICATIVE, levels=4 cycles=v > Cycles per PCApply=1 > Using Galerkin computed coarse grid matrices > Coarse grid solver -- level ------------------------------- > KSP Object: (mg_coarse_) 1 MPI processes > type: preonly > maximum iterations=1, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (mg_coarse_) 1 MPI processes > type: lu > LU: out-of-place factorization > tolerance for zero pivot 2.22045e-14 > using diagonal shift on blocks to prevent zero pivot [INBLOCKS] > matrix ordering: nd > factor fill ratio given 0., needed 0. > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqaij > rows=16, cols=16 > package used to perform factorization: superlu_dist > total: nonzeros=0, allocated nonzeros=0 > total number of mallocs used during MatSetValues calls =0 > SuperLU_DIST run parameters: > Process grid nprow 1 x npcol 1 > Equilibrate matrix TRUE > Matrix input mode 0 > Replace tiny pivots FALSE > Use iterative refinement FALSE > Processors in row 1 col partition 1 > Row permutation LargeDiag > Column permutation METIS_AT_PLUS_A > Parallel symbolic factorization FALSE > Repeated factorization SamePattern > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=16, cols=16 > total: nonzeros=72, allocated nonzeros=72 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > Down solver (pre-smoother) on level 1 ------------------------------- > KSP Object: (mg_levels_1_) 1 MPI processes > type: richardson > Richardson: damping factor=1. > maximum iterations=2 > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using nonzero initial guess > using NONE norm type for convergence test > PC Object: (mg_levels_1_) 1 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=64, cols=64 > total: nonzeros=304, allocated nonzeros=304 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > Up solver (post-smoother) same as down solver (pre-smoother) > Down solver (pre-smoother) on level 2 ------------------------------- > KSP Object: (mg_levels_2_) 1 MPI processes > type: richardson > Richardson: damping factor=1. > maximum iterations=2 > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using nonzero initial guess > using NONE norm type for convergence test > PC Object: (mg_levels_2_) 1 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=256, cols=256 > total: nonzeros=1248, allocated nonzeros=1248 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > Up solver (post-smoother) same as down solver (pre-smoother) > Down solver (pre-smoother) on level 3 ------------------------------- > KSP Object: (mg_levels_3_) 1 MPI processes > type: richardson > Richardson: damping factor=1. > maximum iterations=2 > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using nonzero initial guess > using NONE norm type for convergence test > PC Object: (mg_levels_3_) 1 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=1024, cols=1024 > total: nonzeros=5056, allocated nonzeros=5056 > total number of mallocs used during MatSetValues calls =0 > has attached null space > not using I-node routines > Up solver (post-smoother) same as down solver (pre-smoother) > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=1024, cols=1024 > total: nonzeros=5056, allocated nonzeros=5056 > total number of mallocs used during MatSetValues calls =0 > has attached null space > not using I-node routines > > > Michele > > > > > On Wed, 2016-02-10 at 19:37 -0600, Matthew Knepley wrote: >> On Wed, Feb 10, 2016 at 7:33 PM, Michele Rosso wrote: >> Hi, >> >> I encountered the following error while solving a symmetric positive defined system: >> >> Linear solve did not converge due to DIVERGED_PCSETUP_FAILED iterations 0 >> PCSETUP_FAILED due to SUBPC_ERROR >> >> This error appears only if I use the optimized version of both petsc and my code ( compiler: gfortran, flags: -O3 ). >> It is weird since I am solving a time-dependent problem and everything, i.e. results and convergence rate, are as expected until the above error shows up. If I run both petsc and my code in debug mode, everything goes smooth till the end of the simulation. >> However, if I reduce the ksp_rtol, even the debug run fails, after running as expected for a while, because of a KSP_DIVERGED_INDEFINITE_PC . >> The options I am using are: >> >> -ksp_type cg >> -ksp_norm_type unpreconditioned >> -ksp_rtol 1e-8 >> -ksp_lag_norm >> -ksp_initial_guess_nonzero yes >> -pc_type mg >> -pc_mg_galerkin >> -pc_mg_levels 4 >> -mg_levels_ksp_type richardson >> -mg_coarse_ksp_constant_null_space >> -mg_coarse_pc_type lu >> -mg_coarse_pc_factor_mat_solver_package superlu_dist >> -options_left >> >> I attached a copy of ksp_view. I am currently using petsc-master (last updated yesterday). >> I would appreciate any suggestion on this matter. >> >> >> >> I suspect you have a nonlinear PC. Can you send the output of -ksp_view? >> >> >> Matt >> >> Thanks, >> Michele >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener > > From mrosso at uci.edu Wed Feb 10 21:49:00 2016 From: mrosso at uci.edu (Michele Rosso) Date: Wed, 10 Feb 2016 19:49:00 -0800 Subject: [petsc-users] DIVERGED_PCSETUP_FAILED In-Reply-To: <0903A02C-4B7A-4C1F-AD27-44CB6B1C6F07@mcs.anl.gov> References: <1455154398.5948.15.camel@enterprise-A> <1455157291.5948.19.camel@enterprise-A> <0903A02C-4B7A-4C1F-AD27-44CB6B1C6F07@mcs.anl.gov> Message-ID: <1455162540.5948.21.camel@enterprise-A> Hong, here if the output of grep -info: using diagonal shift on blocks to prevent zero pivot [INBLOCKS] Replace tiny pivots FALSE tolerance for zero pivot 2.22045e-14 It seems it is not replacing small pivots: could this be the problem? I will also try Barry's suggestion to diagnose the problem. Thanks, Michele On Wed, 2016-02-10 at 21:22 -0600, Barry Smith wrote: > > On Feb 10, 2016, at 9:00 PM, Hong wrote: > > > > Michele : > > Superlu_dist LU is used for coarse grid PC, which likely produces a zero-pivot. > > Run your code with '-info |grep pivot' to verify. > > Michele > > You can also run with -ksp_error_if_not_converged in or not in the debugger and it will stop immediately when the problem is detected and hopefully provide additional useful information about what has happened. > > Barry > > > > > Hong > > > > Hi Matt, > > > > the ksp_view output was an attachment to my previous email. > > Here it is: > > > > KSP Object: 1 MPI processes > > type: cg > > maximum iterations=10000 > > tolerances: relative=1e-08, absolute=1e-50, divergence=10000. > > left preconditioning > > using nonzero initial guess > > using UNPRECONDITIONED norm type for convergence test > > PC Object: 1 MPI processes > > type: mg > > MG: type is MULTIPLICATIVE, levels=4 cycles=v > > Cycles per PCApply=1 > > Using Galerkin computed coarse grid matrices > > Coarse grid solver -- level ------------------------------- > > KSP Object: (mg_coarse_) 1 MPI processes > > type: preonly > > maximum iterations=1, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using NONE norm type for convergence test > > PC Object: (mg_coarse_) 1 MPI processes > > type: lu > > LU: out-of-place factorization > > tolerance for zero pivot 2.22045e-14 > > using diagonal shift on blocks to prevent zero pivot [INBLOCKS] > > matrix ordering: nd > > factor fill ratio given 0., needed 0. > > Factored matrix follows: > > Mat Object: 1 MPI processes > > type: seqaij > > rows=16, cols=16 > > package used to perform factorization: superlu_dist > > total: nonzeros=0, allocated nonzeros=0 > > total number of mallocs used during MatSetValues calls =0 > > SuperLU_DIST run parameters: > > Process grid nprow 1 x npcol 1 > > Equilibrate matrix TRUE > > Matrix input mode 0 > > Replace tiny pivots FALSE > > Use iterative refinement FALSE > > Processors in row 1 col partition 1 > > Row permutation LargeDiag > > Column permutation METIS_AT_PLUS_A > > Parallel symbolic factorization FALSE > > Repeated factorization SamePattern > > linear system matrix = precond matrix: > > Mat Object: 1 MPI processes > > type: seqaij > > rows=16, cols=16 > > total: nonzeros=72, allocated nonzeros=72 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node routines > > Down solver (pre-smoother) on level 1 ------------------------------- > > KSP Object: (mg_levels_1_) 1 MPI processes > > type: richardson > > Richardson: damping factor=1. > > maximum iterations=2 > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using nonzero initial guess > > using NONE norm type for convergence test > > PC Object: (mg_levels_1_) 1 MPI processes > > type: sor > > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. > > linear system matrix = precond matrix: > > Mat Object: 1 MPI processes > > type: seqaij > > rows=64, cols=64 > > total: nonzeros=304, allocated nonzeros=304 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node routines > > Up solver (post-smoother) same as down solver (pre-smoother) > > Down solver (pre-smoother) on level 2 ------------------------------- > > KSP Object: (mg_levels_2_) 1 MPI processes > > type: richardson > > Richardson: damping factor=1. > > maximum iterations=2 > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using nonzero initial guess > > using NONE norm type for convergence test > > PC Object: (mg_levels_2_) 1 MPI processes > > type: sor > > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. > > linear system matrix = precond matrix: > > Mat Object: 1 MPI processes > > type: seqaij > > rows=256, cols=256 > > total: nonzeros=1248, allocated nonzeros=1248 > > total number of mallocs used during MatSetValues calls =0 > > not using I-node routines > > Up solver (post-smoother) same as down solver (pre-smoother) > > Down solver (pre-smoother) on level 3 ------------------------------- > > KSP Object: (mg_levels_3_) 1 MPI processes > > type: richardson > > Richardson: damping factor=1. > > maximum iterations=2 > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using nonzero initial guess > > using NONE norm type for convergence test > > PC Object: (mg_levels_3_) 1 MPI processes > > type: sor > > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. > > linear system matrix = precond matrix: > > Mat Object: 1 MPI processes > > type: seqaij > > rows=1024, cols=1024 > > total: nonzeros=5056, allocated nonzeros=5056 > > total number of mallocs used during MatSetValues calls =0 > > has attached null space > > not using I-node routines > > Up solver (post-smoother) same as down solver (pre-smoother) > > linear system matrix = precond matrix: > > Mat Object: 1 MPI processes > > type: seqaij > > rows=1024, cols=1024 > > total: nonzeros=5056, allocated nonzeros=5056 > > total number of mallocs used during MatSetValues calls =0 > > has attached null space > > not using I-node routines > > > > > > Michele > > > > > > > > > > On Wed, 2016-02-10 at 19:37 -0600, Matthew Knepley wrote: > >> On Wed, Feb 10, 2016 at 7:33 PM, Michele Rosso wrote: > >> Hi, > >> > >> I encountered the following error while solving a symmetric positive defined system: > >> > >> Linear solve did not converge due to DIVERGED_PCSETUP_FAILED iterations 0 > >> PCSETUP_FAILED due to SUBPC_ERROR > >> > >> This error appears only if I use the optimized version of both petsc and my code ( compiler: gfortran, flags: -O3 ). > >> It is weird since I am solving a time-dependent problem and everything, i.e. results and convergence rate, are as expected until the above error shows up. If I run both petsc and my code in debug mode, everything goes smooth till the end of the simulation. > >> However, if I reduce the ksp_rtol, even the debug run fails, after running as expected for a while, because of a KSP_DIVERGED_INDEFINITE_PC . > >> The options I am using are: > >> > >> -ksp_type cg > >> -ksp_norm_type unpreconditioned > >> -ksp_rtol 1e-8 > >> -ksp_lag_norm > >> -ksp_initial_guess_nonzero yes > >> -pc_type mg > >> -pc_mg_galerkin > >> -pc_mg_levels 4 > >> -mg_levels_ksp_type richardson > >> -mg_coarse_ksp_constant_null_space > >> -mg_coarse_pc_type lu > >> -mg_coarse_pc_factor_mat_solver_package superlu_dist > >> -options_left > >> > >> I attached a copy of ksp_view. I am currently using petsc-master (last updated yesterday). > >> I would appreciate any suggestion on this matter. > >> > >> > >> > >> I suspect you have a nonlinear PC. Can you send the output of -ksp_view? > >> > >> > >> Matt > >> > >> Thanks, > >> Michele > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> -- > >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > >> -- Norbert Wiener > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Feb 10 22:15:25 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 10 Feb 2016 22:15:25 -0600 Subject: [petsc-users] DIVERGED_PCSETUP_FAILED In-Reply-To: <1455162540.5948.21.camel@enterprise-A> References: <1455154398.5948.15.camel@enterprise-A> <1455157291.5948.19.camel@enterprise-A> <0903A02C-4B7A-4C1F-AD27-44CB6B1C6F07@mcs.anl.gov> <1455162540.5948.21.camel@enterprise-A> Message-ID: You can try the option -mat_superlu_dist_replacetinypivot true if you are luck it get you past the zero pivot but still give an adequate preconditioner. Barry > On Feb 10, 2016, at 9:49 PM, Michele Rosso wrote: > > Hong, > > here if the output of grep -info: > > using diagonal shift on blocks to prevent zero pivot [INBLOCKS] > Replace tiny pivots FALSE > tolerance for zero pivot 2.22045e-14 > > It seems it is not replacing small pivots: could this be the problem? > I will also try Barry's suggestion to diagnose the problem. > > Thanks, > Michele > > > On Wed, 2016-02-10 at 21:22 -0600, Barry Smith wrote: >> > On Feb 10, 2016, at 9:00 PM, Hong wrote: >> > >> > Michele : >> > Superlu_dist LU is used for coarse grid PC, which likely produces a zero-pivot. >> > Run your code with '-info |grep pivot' to verify. >> >> >> Michele >> >> You can also run with -ksp_error_if_not_converged in or not in the debugger and it will stop immediately when the problem is detected and hopefully provide additional useful information about what has happened. >> >> Barry >> >> >> > >> > Hong >> > >> > Hi Matt, >> > >> > the ksp_view output was an attachment to my previous email. >> > Here it is: >> > >> > KSP Object: 1 MPI processes >> > type: cg >> > maximum iterations=10000 >> > tolerances: relative=1e-08, absolute=1e-50, divergence=10000. >> > left preconditioning >> > using nonzero initial guess >> > using UNPRECONDITIONED norm type for convergence test >> > PC Object: 1 MPI processes >> > type: mg >> > MG: type is MULTIPLICATIVE, levels=4 cycles=v >> > Cycles per PCApply=1 >> > Using Galerkin computed coarse grid matrices >> > Coarse grid solver -- level ------------------------------- >> > KSP Object: (mg_coarse_) 1 MPI processes >> > type: preonly >> > maximum iterations=1, initial guess is zero >> > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. >> > left preconditioning >> > using NONE norm type for convergence test >> > PC Object: (mg_coarse_) 1 MPI processes >> > type: lu >> > LU: out-of-place factorization >> > tolerance for zero pivot 2.22045e-14 >> > using diagonal shift on blocks to prevent zero pivot [INBLOCKS] >> > matrix ordering: nd >> > factor fill ratio given 0., needed 0. >> > Factored matrix follows: >> > Mat Object: 1 MPI processes >> > type: seqaij >> > rows=16, cols=16 >> > package used to perform factorization: superlu_dist >> > total: nonzeros=0, allocated nonzeros=0 >> > total number of mallocs used during MatSetValues calls =0 >> > SuperLU_DIST run parameters: >> > Process grid nprow 1 x npcol 1 >> > Equilibrate matrix TRUE >> > Matrix input mode 0 >> > Replace tiny pivots FALSE >> > Use iterative refinement FALSE >> > Processors in row 1 col partition 1 >> > Row permutation LargeDiag >> > Column permutation METIS_AT_PLUS_A >> > Parallel symbolic factorization FALSE >> > Repeated factorization SamePattern >> > linear system matrix = precond matrix: >> > Mat Object: 1 MPI processes >> > type: seqaij >> > rows=16, cols=16 >> > total: nonzeros=72, allocated nonzeros=72 >> > total number of mallocs used during MatSetValues calls =0 >> > not using I-node routines >> > Down solver (pre-smoother) on level 1 ------------------------------- >> > KSP Object: (mg_levels_1_) 1 MPI processes >> > type: richardson >> > Richardson: damping factor=1. >> > maximum iterations=2 >> > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. >> > left preconditioning >> > using nonzero initial guess >> > using NONE norm type for convergence test >> > PC Object: (mg_levels_1_) 1 MPI processes >> > type: sor >> > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. >> > linear system matrix = precond matrix: >> > Mat Object: 1 MPI processes >> > type: seqaij >> > rows=64, cols=64 >> > total: nonzeros=304, allocated nonzeros=304 >> > total number of mallocs used during MatSetValues calls =0 >> > not using I-node routines >> > Up solver (post-smoother) same as down solver (pre-smoother) >> > Down solver (pre-smoother) on level 2 ------------------------------- >> > KSP Object: (mg_levels_2_) 1 MPI processes >> > type: richardson >> > Richardson: damping factor=1. >> > maximum iterations=2 >> > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. >> > left preconditioning >> > using nonzero initial guess >> > using NONE norm type for convergence test >> > PC Object: (mg_levels_2_) 1 MPI processes >> > type: sor >> > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. >> > linear system matrix = precond matrix: >> > Mat Object: 1 MPI processes >> > type: seqaij >> > rows=256, cols=256 >> > total: nonzeros=1248, allocated nonzeros=1248 >> > total number of mallocs used during MatSetValues calls =0 >> > not using I-node routines >> > Up solver (post-smoother) same as down solver (pre-smoother) >> > Down solver (pre-smoother) on level 3 ------------------------------- >> > KSP Object: (mg_levels_3_) 1 MPI processes >> > type: richardson >> > Richardson: damping factor=1. >> > maximum iterations=2 >> > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. >> > left preconditioning >> > using nonzero initial guess >> > using NONE norm type for convergence test >> > PC Object: (mg_levels_3_) 1 MPI processes >> > type: sor >> > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. >> > linear system matrix = precond matrix: >> > Mat Object: 1 MPI processes >> > type: seqaij >> > rows=1024, cols=1024 >> > total: nonzeros=5056, allocated nonzeros=5056 >> > total number of mallocs used during MatSetValues calls =0 >> > has attached null space >> > not using I-node routines >> > Up solver (post-smoother) same as down solver (pre-smoother) >> > linear system matrix = precond matrix: >> > Mat Object: 1 MPI processes >> > type: seqaij >> > rows=1024, cols=1024 >> > total: nonzeros=5056, allocated nonzeros=5056 >> > total number of mallocs used during MatSetValues calls =0 >> > has attached null space >> > not using I-node routines >> > >> > >> > Michele >> > >> > >> > >> > >> > On Wed, 2016-02-10 at 19:37 -0600, Matthew Knepley wrote: >> >> On Wed, Feb 10, 2016 at 7:33 PM, Michele Rosso wrote: >> >> Hi, >> >> >> >> I encountered the following error while solving a symmetric positive defined system: >> >> >> >> Linear solve did not converge due to DIVERGED_PCSETUP_FAILED iterations 0 >> >> PCSETUP_FAILED due to SUBPC_ERROR >> >> >> >> This error appears only if I use the optimized version of both petsc and my code ( compiler: gfortran, flags: -O3 ). >> >> It is weird since I am solving a time-dependent problem and everything, i.e. results and convergence rate, are as expected until the above error shows up. If I run both petsc and my code in debug mode, everything goes smooth till the end of the simulation. >> >> However, if I reduce the ksp_rtol, even the debug run fails, after running as expected for a while, because of a KSP_DIVERGED_INDEFINITE_PC . >> >> The options I am using are: >> >> >> >> -ksp_type cg >> >> -ksp_norm_type unpreconditioned >> >> -ksp_rtol 1e-8 >> >> -ksp_lag_norm >> >> -ksp_initial_guess_nonzero yes >> >> -pc_type mg >> >> -pc_mg_galerkin >> >> -pc_mg_levels 4 >> >> -mg_levels_ksp_type richardson >> >> -mg_coarse_ksp_constant_null_space >> >> -mg_coarse_pc_type lu >> >> -mg_coarse_pc_factor_mat_solver_package superlu_dist >> >> -options_left >> >> >> >> I attached a copy of ksp_view. I am currently using petsc-master (last updated yesterday). >> >> I would appreciate any suggestion on this matter. >> >> >> >> >> >> >> >> I suspect you have a nonlinear PC. Can you send the output of -ksp_view? >> >> >> >> >> >> Matt >> >> >> >> Thanks, >> >> Michele >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> >> -- Norbert Wiener >> > >> > >> >> >> > From mrosso at uci.edu Thu Feb 11 00:05:18 2016 From: mrosso at uci.edu (Michele Rosso) Date: Wed, 10 Feb 2016 22:05:18 -0800 Subject: [petsc-users] DIVERGED_PCSETUP_FAILED In-Reply-To: References: <1455154398.5948.15.camel@enterprise-A> <1455157291.5948.19.camel@enterprise-A> <0903A02C-4B7A-4C1F-AD27-44CB6B1C6F07@mcs.anl.gov> <1455162540.5948.21.camel@enterprise-A> Message-ID: <1455170718.5948.28.camel@enterprise-A> I tried setting -mat_superlu_dist_replacetinypivot true: it does help to advance the run past the previous "critical" point but eventually it stops later with the same error. I forgot to mention my system is singular: I remove the constant null space but I am not sure if the coarse solver needs to be explicity informed of this. Michele On Wed, 2016-02-10 at 22:15 -0600, Barry Smith wrote: > You can try the option > > -mat_superlu_dist_replacetinypivot true > > if you are luck it get you past the zero pivot but still give an adequate preconditioner. > > Barry > > > On Feb 10, 2016, at 9:49 PM, Michele Rosso wrote: > > > > Hong, > > > > here if the output of grep -info: > > > > using diagonal shift on blocks to prevent zero pivot [INBLOCKS] > > Replace tiny pivots FALSE > > tolerance for zero pivot 2.22045e-14 > > > > It seems it is not replacing small pivots: could this be the problem? > > I will also try Barry's suggestion to diagnose the problem. > > > > Thanks, > > Michele > > > > > > On Wed, 2016-02-10 at 21:22 -0600, Barry Smith wrote: > >> > On Feb 10, 2016, at 9:00 PM, Hong wrote: > >> > > >> > Michele : > >> > Superlu_dist LU is used for coarse grid PC, which likely produces a zero-pivot. > >> > Run your code with '-info |grep pivot' to verify. > >> > >> > >> Michele > >> > >> You can also run with -ksp_error_if_not_converged in or not in the debugger and it will stop immediately when the problem is detected and hopefully provide additional useful information about what has happened. > >> > >> Barry > >> > >> > >> > > >> > Hong > >> > > >> > Hi Matt, > >> > > >> > the ksp_view output was an attachment to my previous email. > >> > Here it is: > >> > > >> > KSP Object: 1 MPI processes > >> > type: cg > >> > maximum iterations=10000 > >> > tolerances: relative=1e-08, absolute=1e-50, divergence=10000. > >> > left preconditioning > >> > using nonzero initial guess > >> > using UNPRECONDITIONED norm type for convergence test > >> > PC Object: 1 MPI processes > >> > type: mg > >> > MG: type is MULTIPLICATIVE, levels=4 cycles=v > >> > Cycles per PCApply=1 > >> > Using Galerkin computed coarse grid matrices > >> > Coarse grid solver -- level ------------------------------- > >> > KSP Object: (mg_coarse_) 1 MPI processes > >> > type: preonly > >> > maximum iterations=1, initial guess is zero > >> > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > >> > left preconditioning > >> > using NONE norm type for convergence test > >> > PC Object: (mg_coarse_) 1 MPI processes > >> > type: lu > >> > LU: out-of-place factorization > >> > tolerance for zero pivot 2.22045e-14 > >> > using diagonal shift on blocks to prevent zero pivot [INBLOCKS] > >> > matrix ordering: nd > >> > factor fill ratio given 0., needed 0. > >> > Factored matrix follows: > >> > Mat Object: 1 MPI processes > >> > type: seqaij > >> > rows=16, cols=16 > >> > package used to perform factorization: superlu_dist > >> > total: nonzeros=0, allocated nonzeros=0 > >> > total number of mallocs used during MatSetValues calls =0 > >> > SuperLU_DIST run parameters: > >> > Process grid nprow 1 x npcol 1 > >> > Equilibrate matrix TRUE > >> > Matrix input mode 0 > >> > Replace tiny pivots FALSE > >> > Use iterative refinement FALSE > >> > Processors in row 1 col partition 1 > >> > Row permutation LargeDiag > >> > Column permutation METIS_AT_PLUS_A > >> > Parallel symbolic factorization FALSE > >> > Repeated factorization SamePattern > >> > linear system matrix = precond matrix: > >> > Mat Object: 1 MPI processes > >> > type: seqaij > >> > rows=16, cols=16 > >> > total: nonzeros=72, allocated nonzeros=72 > >> > total number of mallocs used during MatSetValues calls =0 > >> > not using I-node routines > >> > Down solver (pre-smoother) on level 1 ------------------------------- > >> > KSP Object: (mg_levels_1_) 1 MPI processes > >> > type: richardson > >> > Richardson: damping factor=1. > >> > maximum iterations=2 > >> > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > >> > left preconditioning > >> > using nonzero initial guess > >> > using NONE norm type for convergence test > >> > PC Object: (mg_levels_1_) 1 MPI processes > >> > type: sor > >> > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. > >> > linear system matrix = precond matrix: > >> > Mat Object: 1 MPI processes > >> > type: seqaij > >> > rows=64, cols=64 > >> > total: nonzeros=304, allocated nonzeros=304 > >> > total number of mallocs used during MatSetValues calls =0 > >> > not using I-node routines > >> > Up solver (post-smoother) same as down solver (pre-smoother) > >> > Down solver (pre-smoother) on level 2 ------------------------------- > >> > KSP Object: (mg_levels_2_) 1 MPI processes > >> > type: richardson > >> > Richardson: damping factor=1. > >> > maximum iterations=2 > >> > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > >> > left preconditioning > >> > using nonzero initial guess > >> > using NONE norm type for convergence test > >> > PC Object: (mg_levels_2_) 1 MPI processes > >> > type: sor > >> > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. > >> > linear system matrix = precond matrix: > >> > Mat Object: 1 MPI processes > >> > type: seqaij > >> > rows=256, cols=256 > >> > total: nonzeros=1248, allocated nonzeros=1248 > >> > total number of mallocs used during MatSetValues calls =0 > >> > not using I-node routines > >> > Up solver (post-smoother) same as down solver (pre-smoother) > >> > Down solver (pre-smoother) on level 3 ------------------------------- > >> > KSP Object: (mg_levels_3_) 1 MPI processes > >> > type: richardson > >> > Richardson: damping factor=1. > >> > maximum iterations=2 > >> > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > >> > left preconditioning > >> > using nonzero initial guess > >> > using NONE norm type for convergence test > >> > PC Object: (mg_levels_3_) 1 MPI processes > >> > type: sor > >> > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. > >> > linear system matrix = precond matrix: > >> > Mat Object: 1 MPI processes > >> > type: seqaij > >> > rows=1024, cols=1024 > >> > total: nonzeros=5056, allocated nonzeros=5056 > >> > total number of mallocs used during MatSetValues calls =0 > >> > has attached null space > >> > not using I-node routines > >> > Up solver (post-smoother) same as down solver (pre-smoother) > >> > linear system matrix = precond matrix: > >> > Mat Object: 1 MPI processes > >> > type: seqaij > >> > rows=1024, cols=1024 > >> > total: nonzeros=5056, allocated nonzeros=5056 > >> > total number of mallocs used during MatSetValues calls =0 > >> > has attached null space > >> > not using I-node routines > >> > > >> > > >> > Michele > >> > > >> > > >> > > >> > > >> > On Wed, 2016-02-10 at 19:37 -0600, Matthew Knepley wrote: > >> >> On Wed, Feb 10, 2016 at 7:33 PM, Michele Rosso wrote: > >> >> Hi, > >> >> > >> >> I encountered the following error while solving a symmetric positive defined system: > >> >> > >> >> Linear solve did not converge due to DIVERGED_PCSETUP_FAILED iterations 0 > >> >> PCSETUP_FAILED due to SUBPC_ERROR > >> >> > >> >> This error appears only if I use the optimized version of both petsc and my code ( compiler: gfortran, flags: -O3 ). > >> >> It is weird since I am solving a time-dependent problem and everything, i.e. results and convergence rate, are as expected until the above error shows up. If I run both petsc and my code in debug mode, everything goes smooth till the end of the simulation. > >> >> However, if I reduce the ksp_rtol, even the debug run fails, after running as expected for a while, because of a KSP_DIVERGED_INDEFINITE_PC . > >> >> The options I am using are: > >> >> > >> >> -ksp_type cg > >> >> -ksp_norm_type unpreconditioned > >> >> -ksp_rtol 1e-8 > >> >> -ksp_lag_norm > >> >> -ksp_initial_guess_nonzero yes > >> >> -pc_type mg > >> >> -pc_mg_galerkin > >> >> -pc_mg_levels 4 > >> >> -mg_levels_ksp_type richardson > >> >> -mg_coarse_ksp_constant_null_space > >> >> -mg_coarse_pc_type lu > >> >> -mg_coarse_pc_factor_mat_solver_package superlu_dist > >> >> -options_left > >> >> > >> >> I attached a copy of ksp_view. I am currently using petsc-master (last updated yesterday). > >> >> I would appreciate any suggestion on this matter. > >> >> > >> >> > >> >> > >> >> I suspect you have a nonlinear PC. Can you send the output of -ksp_view? > >> >> > >> >> > >> >> Matt > >> >> > >> >> Thanks, > >> >> Michele > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> -- > >> >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > >> >> -- Norbert Wiener > >> > > >> > > >> > >> > >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Thu Feb 11 00:18:01 2016 From: dave.mayhem23 at gmail.com (Dave May) Date: Thu, 11 Feb 2016 07:18:01 +0100 Subject: [petsc-users] Create DM matrix In-Reply-To: References: Message-ID: I think he wants the source location so that he can copy and implementation and "tweak" it slightly The location is here ${PETSC_DIR}/src/dm/impls/da/fdda.c /Users/dmay/software/petsc-3.6.0/src dmay at nikkan:~/software/petsc-3.6.0/src $ grep -r DMCreateMatrix_DA_3d_MPIAIJ * dm/impls/da/fdda.c:extern PetscErrorCode DMCreateMatrix_DA_3d_MPIAIJ(DM,Mat); dm/impls/da/fdda.c:extern PetscErrorCode DMCreateMatrix_DA_3d_MPIAIJ_Fill(DM,Mat); dm/impls/da/fdda.c: ierr = DMCreateMatrix_DA_3d_MPIAIJ_Fill(da,A);CHKERRQ(ierr); dm/impls/da/fdda.c: ierr = DMCreateMatrix_DA_3d_MPIAIJ(da,A);CHKERRQ(ierr); *dm/impls/da/fdda.c:#define __FUNCT__ "DMCreateMatrix_DA_3d_MPIAIJ"dm/impls/da/fdda.c:PetscErrorCode DMCreateMatrix_DA_3d_MPIAIJ(DM da,Mat J)*dm/impls/da/fdda.c:#define __FUNCT__ "DMCreateMatrix_DA_3d_MPIAIJ_Fill" dm/impls/da/fdda.c:PetscErrorCode DMCreateMatrix_DA_3d_MPIAIJ_Fill(DM da,Mat J) On 11 February 2016 at 04:08, Matthew Knepley wrote: > On Wed, Feb 10, 2016 at 8:59 PM, Sang pham van > wrote: > >> The irregular rows is quite many. The matrix really needs to be >> preallocated. >> Could you show me how to use DMCreateMatrix_DA_3d_MPIAIJ() directly? >> >> Just put the declaration right into your source. > > Matt > >> Pham >> On Feb 11, 2016 9:52 AM, "Matthew Knepley" wrote: >> >>> On Wed, Feb 10, 2016 at 8:44 PM, Sang pham van >>> wrote: >>> >>>> That is because my matrix has some rows which need more entries than >>>> usual. >>>> >>> >>> If its only a few, you could just turn off the allocation error. >>> >>> >>>> Where can i find source code of DMCreateMatrix()? >>>> >>>> >>> >>> https://bitbucket.org/petsc/petsc/src/827b69d6bb12709ff9b9a0dede31640477fc2b74/src/dm/impls/da/fdda.c?at=master&fileviewer=file-view-default#fdda.c-1024 >>> >>> Matt >>> >>>> Pham. >>>> On Feb 11, 2016 8:35 AM, "Matthew Knepley" wrote: >>>> >>>>> On Wed, Feb 10, 2016 at 6:14 PM, Sang pham van >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I am trying to create a DM matrix with DMCreateMatrix_DA_3d_MPIAIJ() >>>>>> instead of using DMCreateMatrix(). >>>>>> >>>>> >>>>> Why, that should be called automatically by DMCreateMatrix()? >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Which header file should I include to use that routine? also, what is >>>>>> the source file containing the DMCreateMatrix() routine? >>>>>> >>>>>> Many thanks in advance. >>>>>> Pham >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Thu Feb 11 00:30:14 2016 From: dave.mayhem23 at gmail.com (Dave May) Date: Thu, 11 Feb 2016 07:30:14 +0100 Subject: [petsc-users] DIVERGED_PCSETUP_FAILED In-Reply-To: <1455170718.5948.28.camel@enterprise-A> References: <1455154398.5948.15.camel@enterprise-A> <1455157291.5948.19.camel@enterprise-A> <0903A02C-4B7A-4C1F-AD27-44CB6B1C6F07@mcs.anl.gov> <1455162540.5948.21.camel@enterprise-A> <1455170718.5948.28.camel@enterprise-A> Message-ID: On 11 February 2016 at 07:05, Michele Rosso wrote: > I tried setting -mat_superlu_dist_replacetinypivot true: it does help to > advance the run past the previous "critical" point but eventually it stops > later with the same error. > I forgot to mention my system is singular: I remove the constant null > space but I am not sure if the coarse solver needs to be explicity informed > of this. > Right - are you using pure Newmann boundary conditions? To make the solution unique, are you (a) imposing a single Dichletet boundary condition on your field by explicitly modifying the matrix (b) imposing a a condition like \int \phi dV = 0 via something like -ksp_constant_null_space If you removed removed the null space by modifying the matrix explicitly (a), the sparse direct solver should go through. If you use (b), then this method cannot be used to help the direct solver. If this is the intended target problem size (16x16), gather the matrix and using petsc Cholesky or Umfpack. Cholesky is more stable than LU and can usually deal with a single zero eigenvaue without resorting to tricks. Umfpack will solve the problem easily as it uses clever re-ordering. If you have access to MKL-Pardiso, that will also work great. Thanks, Dave > > > Michele > > > On Wed, 2016-02-10 at 22:15 -0600, Barry Smith wrote: > > You can try the option > > -mat_superlu_dist_replacetinypivot true > > if you are luck it get you past the zero pivot but still give an adequate preconditioner. > > Barry > > On Feb 10, 2016, at 9:49 PM, Michele Rosso wrote:> > Hong,> > here if the output of grep -info:> > using diagonal shift on blocks to prevent zero pivot [INBLOCKS]> Replace tiny pivots FALSE> tolerance for zero pivot 2.22045e-14> > It seems it is not replacing small pivots: could this be the problem?> I will also try Barry's suggestion to diagnose the problem.> > Thanks,> Michele> > > On Wed, 2016-02-10 at 21:22 -0600, Barry Smith wrote:>> > On Feb 10, 2016, at 9:00 PM, Hong wrote:>> > >> > Michele :>> > Superlu_dist LU is used for coarse grid PC, which likely produces a zero-pivot.>> > Run your code with '-info |grep pivot' to verify.>> >> >> Michele>> >> You can also run with -ksp_error_if_not_converged in or not in the debugger and it will stop immediately when the problem is detected and hopefully provide additional useful information about what has happened.>> >> Barry>> >> >> > >> > Hong>> > >> > Hi Matt,>> > >> > the ksp_view output was an attachment to my previous email.>> > Here it is:>> > >> > KSP Object: 1 MPI processes>> > type: cg>> > maximum iterations=10000>> > tolerances: relative=1e-08, absolute=1e-50, divergence=10000.>> > left preconditioning>> > using nonzero initial guess>> > using UNPRECONDITIONED norm type for convergence test>> > PC Object: 1 MPI processes>> > type: mg>> > MG: type is MULTIPLICATIVE, levels=4 cycles=v>> > Cycles per PCApply=1>> > Using Galerkin computed coarse grid matrices>> > Coarse grid solver -- level ------------------------------->> > KSP Object: (mg_coarse_) 1 MPI processes>> > type: preonly>> > maximum iterations=1, initial guess is zero>> > tolerances: relative=1e-05, absolute=1e-50, divergence=10000.>> > left preconditioning>> > using NONE norm type for convergence test>> > PC Object: (mg_coarse_) 1 MPI processes>> > type: lu>> > LU: out-of-place factorization>> > tolerance for zero pivot 2.22045e-14>> > using diagonal shift on blocks to prevent zero pivot [INBLOCKS]>> > matrix ordering: nd>> > factor fill ratio given 0., needed 0.>> > Factored matrix follows:>> > Mat Object: 1 MPI processes>> > type: seqaij>> > rows=16, cols=16>> > package used to perform factorization: superlu_dist>> > total: nonzeros=0, allocated nonzeros=0>> > total number of mallocs used during MatSetValues calls =0>> > SuperLU_DIST run parameters:>> > Process grid nprow 1 x npcol 1 >> > Equilibrate matrix TRUE >> > Matrix input mode 0 >> > Replace tiny pivots FALSE >> > Use iterative refinement FALSE >> > Processors in row 1 col partition 1 >> > Row permutation LargeDiag >> > Column permutation METIS_AT_PLUS_A>> > Parallel symbolic factorization FALSE >> > Repeated factorization SamePattern>> > linear system matrix = precond matrix:>> > Mat Object: 1 MPI processes>> > type: seqaij>> > rows=16, cols=16>> > total: nonzeros=72, allocated nonzeros=72>> > total number of mallocs used during MatSetValues calls =0>> > not using I-node routines>> > Down solver (pre-smoother) on level 1 ------------------------------->> > KSP Object: (mg_levels_1_) 1 MPI processes>> > type: richardson>> > Richardson: damping factor=1.>> > maximum iterations=2>> > tolerances: relative=1e-05, absolute=1e-50, divergence=10000.>> > left preconditioning>> > using nonzero initial guess>> > using NONE norm type for convergence test>> > PC Object: (mg_levels_1_) 1 MPI processes>> > type: sor>> > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.>> > linear system matrix = precond matrix:>> > Mat Object: 1 MPI processes>> > type: seqaij>> > rows=64, cols=64>> > total: nonzeros=304, allocated nonzeros=304>> > total number of mallocs used during MatSetValues calls =0>> > not using I-node routines>> > Up solver (post-smoother) same as down solver (pre-smoother)>> > Down solver (pre-smoother) on level 2 ------------------------------->> > KSP Object: (mg_levels_2_) 1 MPI processes>> > type: richardson>> > Richardson: damping factor=1.>> > maximum iterations=2>> > tolerances: relative=1e-05, absolute=1e-50, divergence=10000.>> > left preconditioning>> > using nonzero initial guess>> > using NONE norm type for convergence test>> > PC Object: (mg_levels_2_) 1 MPI processes>> > type: sor>> > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.>> > linear system matrix = precond matrix:>> > Mat Object: 1 MPI processes>> > type: seqaij>> > rows=256, cols=256>> > total: nonzeros=1248, allocated nonzeros=1248>> > total number of mallocs used during MatSetValues calls =0>> > not using I-node routines>> > Up solver (post-smoother) same as down solver (pre-smoother)>> > Down solver (pre-smoother) on level 3 ------------------------------->> > KSP Object: (mg_levels_3_) 1 MPI processes>> > type: richardson>> > Richardson: damping factor=1.>> > maximum iterations=2>> > tolerances: relative=1e-05, absolute=1e-50, divergence=10000.>> > left preconditioning>> > using nonzero initial guess>> > using NONE norm type for convergence test>> > PC Object: (mg_levels_3_) 1 MPI processes>> > type: sor>> > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.>> > linear system matrix = precond matrix:>> > Mat Object: 1 MPI processes>> > type: seqaij>> > rows=1024, cols=1024>> > total: nonzeros=5056, allocated nonzeros=5056>> > total number of mallocs used during MatSetValues calls =0>> > has attached null space>> > not using I-node routines>> > Up solver (post-smoother) same as down solver (pre-smoother)>> > linear system matrix = precond matrix:>> > Mat Object: 1 MPI processes>> > type: seqaij>> > rows=1024, cols=1024>> > total: nonzeros=5056, allocated nonzeros=5056>> > total number of mallocs used during MatSetValues calls =0>> > has attached null space>> > not using I-node routines>> > >> > >> > Michele>> > >> > >> > >> > >> > On Wed, 2016-02-10 at 19:37 -0600, Matthew Knepley wrote:>> >> On Wed, Feb 10, 2016 at 7:33 PM, Michele Rosso wrote:>> >> Hi,>> >> >> >> I encountered the following error while solving a symmetric positive defined system:>> >> >> >> Linear solve did not converge due to DIVERGED_PCSETUP_FAILED iterations 0>> >> PCSETUP_FAILED due to SUBPC_ERROR >> >> >> >> This error appears only if I use the optimized version of both petsc and my code ( compiler: gfortran, flags: -O3 ).>> >> It is weird since I am solving a time-dependent problem and everything, i.e. results and convergence rate, are as expected until the above error shows up. If I run both petsc and my code in debug mode, everything goes smooth till the end of the simulation.>> >> However, if I reduce the ksp_rtol, even the debug run fails, after running as expected for a while, because of a KSP_DIVERGED_INDEFINITE_PC . >> >> The options I am using are:>> >> >> >> -ksp_type cg>> >> -ksp_norm_type unpreconditioned>> >> -ksp_rtol 1e-8>> >> -ksp_lag_norm>> >> -ksp_initial_guess_nonzero yes>> >> -pc_type mg>> >> -pc_mg_galerkin>> >> -pc_mg_levels 4>> >> -mg_levels_ksp_type richardson>> >> -mg_coarse_ksp_constant_null_space>> >> -mg_coarse_pc_type lu>> >> -mg_coarse_pc_factor_mat_solver_package superlu_dist>> >> -options_left>> >> >> >> I attached a copy of ksp_view. I am currently using petsc-master (last updated yesterday).>> >> I would appreciate any suggestion on this matter.>> >> >> >> >> >> >> >> I suspect you have a nonlinear PC. Can you send the output of -ksp_view?>> >> >> >> >> >> Matt>> >> >> >> Thanks,>> >> Michele>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -->> >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.>> >> -- Norbert Wiener>> > >> > >> >> >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pvsang002 at gmail.com Thu Feb 11 02:16:32 2016 From: pvsang002 at gmail.com (Sang pham van) Date: Thu, 11 Feb 2016 15:16:32 +0700 Subject: [petsc-users] Create DM matrix In-Reply-To: References: Message-ID: I insert the lines below into my code, but it does not work: PetscErrorCode ierr; PetscInt xs,ys,nx,ny,i,j,slot,gxs,gys,gnx,gny; PetscInt m,n,dim,s,*cols = NULL,k,nc,*rows = NULL,col,cnt,l,p,*dnz = NULL,*onz = NULL; PetscInt istart,iend,jstart,jend,kstart,kend,zs,nz,gzs,gnz,ii,jj,kk; MPI_Comm comm; PetscScalar *values; DMDABoundaryType bx,by,bz; ISLocalToGlobalMapping ltog,ltogb; DMDAStencilType st; PetscFunctionBegin; ierr = DMDAGetInfo(da,&dim,&m,&n,&p,0,0,0,&nc,&s,&bx,&by,&bz,&st);CHKERRQ(ierr); col = 2*s + 1; ierr = DMDAGetCorners(da,&xs,&ys,&zs,&nx,&ny,&nz);CHKERRQ(ierr); ierr = DMDAGetGhostCorners(da,&gxs,&gys,&gzs,&gnx,&gny,&gnz);CHKERRQ(ierr); ierr = PetscObjectGetComm((PetscObject)da,&comm);CHKERRQ(ierr); ierr = PetscMalloc2(nc,PetscInt,&rows,col*col*col*nc*nc,PetscInt,&cols);CHKERRQ(ierr); ierr = DMGetLocalToGlobalMapping(da,<og);CHKERRQ(ierr); ierr = MatSetBlockSize(pMat,nc);CHKERRQ(ierr); /// error happens ! /* determine the matrix preallocation information */ ierr = MatPreallocateInitialize(comm,nc*nx*ny*nz,nc*nx*ny*nz,dnz,onz);CHKERRQ(ierr); for (i=xs; i&, sasVector&, sasVector&, sasVector&)() line 165 in "unknowndirectory/"src/mat_vec/sasMatVecPetsc.cpp where is the mistake? Many thanks. Pham On Thu, Feb 11, 2016 at 1:18 PM, Dave May wrote: > I think he wants the source location so that he can copy and > implementation and "tweak" it slightly > > The location is here > ${PETSC_DIR}/src/dm/impls/da/fdda.c > > /Users/dmay/software/petsc-3.6.0/src > dmay at nikkan:~/software/petsc-3.6.0/src $ grep -r > DMCreateMatrix_DA_3d_MPIAIJ * > dm/impls/da/fdda.c:extern PetscErrorCode > DMCreateMatrix_DA_3d_MPIAIJ(DM,Mat); > dm/impls/da/fdda.c:extern PetscErrorCode > DMCreateMatrix_DA_3d_MPIAIJ_Fill(DM,Mat); > dm/impls/da/fdda.c: ierr = > DMCreateMatrix_DA_3d_MPIAIJ_Fill(da,A);CHKERRQ(ierr); > dm/impls/da/fdda.c: ierr = > DMCreateMatrix_DA_3d_MPIAIJ(da,A);CHKERRQ(ierr); > > > *dm/impls/da/fdda.c:#define __FUNCT__ > "DMCreateMatrix_DA_3d_MPIAIJ"dm/impls/da/fdda.c:PetscErrorCode > DMCreateMatrix_DA_3d_MPIAIJ(DM da,Mat J)*dm/impls/da/fdda.c:#define > __FUNCT__ "DMCreateMatrix_DA_3d_MPIAIJ_Fill" > dm/impls/da/fdda.c:PetscErrorCode DMCreateMatrix_DA_3d_MPIAIJ_Fill(DM > da,Mat J) > > > On 11 February 2016 at 04:08, Matthew Knepley wrote: > >> On Wed, Feb 10, 2016 at 8:59 PM, Sang pham van >> wrote: >> >>> The irregular rows is quite many. The matrix really needs to be >>> preallocated. >>> Could you show me how to use DMCreateMatrix_DA_3d_MPIAIJ() directly? >>> >>> Just put the declaration right into your source. >> >> Matt >> >>> Pham >>> On Feb 11, 2016 9:52 AM, "Matthew Knepley" wrote: >>> >>>> On Wed, Feb 10, 2016 at 8:44 PM, Sang pham van >>>> wrote: >>>> >>>>> That is because my matrix has some rows which need more entries than >>>>> usual. >>>>> >>>> >>>> If its only a few, you could just turn off the allocation error. >>>> >>>> >>>>> Where can i find source code of DMCreateMatrix()? >>>>> >>>>> >>>> >>>> https://bitbucket.org/petsc/petsc/src/827b69d6bb12709ff9b9a0dede31640477fc2b74/src/dm/impls/da/fdda.c?at=master&fileviewer=file-view-default#fdda.c-1024 >>>> >>>> Matt >>>> >>>>> Pham. >>>>> On Feb 11, 2016 8:35 AM, "Matthew Knepley" wrote: >>>>> >>>>>> On Wed, Feb 10, 2016 at 6:14 PM, Sang pham van >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I am trying to create a DM matrix with DMCreateMatrix_DA_3d_MPIAIJ() >>>>>>> instead of using DMCreateMatrix(). >>>>>>> >>>>>> >>>>>> Why, that should be called automatically by DMCreateMatrix()? >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Which header file should I include to use that routine? also, what >>>>>>> is the source file containing the DMCreateMatrix() routine? >>>>>>> >>>>>>> Many thanks in advance. >>>>>>> Pham >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Thu Feb 11 02:29:34 2016 From: dave.mayhem23 at gmail.com (Dave May) Date: Thu, 11 Feb 2016 09:29:34 +0100 Subject: [petsc-users] Create DM matrix In-Reply-To: References: Message-ID: On 11 February 2016 at 09:16, Sang pham van wrote: > I insert the lines below into my code, but it does not work: > Look more carefully at the calling pattern of the standard implementation DMCreateMatrix_DA() calls DMCreateMatrix_DA_3d_MPIAIJ() DMCreateMatrix_DA() allocates the matrix. If in your user code you simply call DMCreateMatrix_DA_3d_MPIAIJ_pvs() from your user code without first calling performing a similar setup and allocation as that occurring DMCreateMatrix_DA(), it clearly won't work. The error tells you everything. It indicates the first argument is not allocated, e.g. pMat has not been allocated. > PetscErrorCode ierr; > PetscInt xs,ys,nx,ny,i,j,slot,gxs,gys,gnx,gny; > PetscInt m,n,dim,s,*cols = NULL,k,nc,*rows = > NULL,col,cnt,l,p,*dnz = NULL,*onz = NULL; > PetscInt > istart,iend,jstart,jend,kstart,kend,zs,nz,gzs,gnz,ii,jj,kk; > MPI_Comm comm; > PetscScalar *values; > DMDABoundaryType bx,by,bz; > ISLocalToGlobalMapping ltog,ltogb; > DMDAStencilType st; > > PetscFunctionBegin; > ierr = > DMDAGetInfo(da,&dim,&m,&n,&p,0,0,0,&nc,&s,&bx,&by,&bz,&st);CHKERRQ(ierr); > col = 2*s + 1; > > ierr = DMDAGetCorners(da,&xs,&ys,&zs,&nx,&ny,&nz);CHKERRQ(ierr); > ierr = > DMDAGetGhostCorners(da,&gxs,&gys,&gzs,&gnx,&gny,&gnz);CHKERRQ(ierr); > ierr = PetscObjectGetComm((PetscObject)da,&comm);CHKERRQ(ierr); > > ierr = > PetscMalloc2(nc,PetscInt,&rows,col*col*col*nc*nc,PetscInt,&cols);CHKERRQ(ierr); > ierr = DMGetLocalToGlobalMapping(da,<og);CHKERRQ(ierr); > > ierr = MatSetBlockSize(pMat,nc);CHKERRQ(ierr); /// error happens ! > /* determine the matrix preallocation information */ > ierr = > MatPreallocateInitialize(comm,nc*nx*ny*nz,nc*nx*ny*nz,dnz,onz);CHKERRQ(ierr); > for (i=xs; i istart = (bx == DMDA_BOUNDARY_PERIODIC) ? -s : (PetscMax(-s,-i)); > iend = (bx == DMDA_BOUNDARY_PERIODIC) ? s : (PetscMin(s,m-i-1)); > for (j=ys; j jstart = (by == DMDA_BOUNDARY_PERIODIC) ? -s : (PetscMax(-s,-j)); > jend = (by == DMDA_BOUNDARY_PERIODIC) ? s : (PetscMin(s,n-j-1)); > for (k=zs; k kstart = (bz == DMDA_BOUNDARY_PERIODIC) ? -s : (PetscMax(-s,-k)); > kend = (bz == DMDA_BOUNDARY_PERIODIC) ? s : (PetscMin(s,p-k-1)); > > slot = i - gxs + gnx*(j - gys) + gnx*gny*(k - gzs); > > cnt = 0; > for (l=0; l for (ii=istart; ii for (jj=jstart; jj for (kk=kstart; kk if ((st == DMDA_STENCIL_BOX) || ((!ii && !jj) || (!jj && > !kk) || (!ii && !kk))) {/* entries on star*/ > cols[cnt++] = l + nc*(slot + ii + gnx*jj + gnx*gny*kk); > } > } > } > } > rows[l] = l + nc*(slot); > } > ierr = > MatPreallocateSetLocal(ltog,nc,rows,ltog,cnt,cols,dnz,onz);CHKERRQ(ierr); > } > } > } > ierr = MatSetBlockSize(pMat,nc);CHKERRQ(ierr); > ierr = MatSeqAIJSetPreallocation(pMat,0,dnz);CHKERRQ(ierr); > ierr = MatMPIAIJSetPreallocation(pMat,0,dnz,0,onz);CHKERRQ(ierr); > ierr = MatPreallocateFinalize(dnz,onz);CHKERRQ(ierr); > ierr = MatSetLocalToGlobalMapping(pMat,ltog,ltog);CHKERRQ(ierr); > ierr = PetscFree2(rows,cols);CHKERRQ(ierr); > > An error reported when it runs into ierr = > MatSetBlockSize(pMat,nc);CHKERRQ(ierr); > > "unknowndirectory/"src/mat_vec/sasMatVecPetsc.cpp:154: > __FUNCT__="DMCreateMatrix_DA_3d_MPIAIJ_pvs" does not agree with > __PRETTY_FUNCTION__="PetscErrorCode > sasMatVecPetsc::DMCreateMatrix_DA_3d_MPIAIJ_pvs(DM, sasSmesh*, > sasVector&, sasVector&, sasVector&, sasVector&)" > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Null argument, when expecting valid pointer! > [0]PETSC ERROR: Null Object: Parameter # 1! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.4.5, Jun, 29, 2014 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./sasMAIN on a arch-cxx-linux-deb named sang by sang Thu > Feb 11 14:54:12 2016 > [0]PETSC ERROR: Libraries linked from > /home/sang/petsc/petsc-3.4.5/arch-cxx-linux-deb/lib > [0]PETSC ERROR: Configure run at Thu Feb 11 11:44:35 2016 > [0]PETSC ERROR: Configure options --download-mpich=1 > --with-scalar-type=real --with-clanguage=cxx --download-mumps=1 > --download-blacs=1 --download-parmetis=1 --download-scalapack=1 > --with-debugging=1 --download-hypre=1 --with-fc=gfortran --download-metis=1 > -download-cmake=1 --download-f-blas-lapack=1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: MatSetBlockSize() line 6686 in > /home/sang/petsc/petsc-3.4.5/src/mat/interface/matrix.c > [0]PETSC ERROR: PetscErrorCode > sasMatVecPetsc::DMCreateMatrix_DA_3d_MPIAIJ_pvs(DM, sasSmesh*, > sasVector&, sasVector&, sasVector&, sasVector&)() line > 165 in "unknowndirectory/"src/mat_vec/sasMatVecPetsc.cpp > > where is the mistake? > > Many thanks. > > Pham > > On Thu, Feb 11, 2016 at 1:18 PM, Dave May wrote: > >> I think he wants the source location so that he can copy and >> implementation and "tweak" it slightly >> >> The location is here >> ${PETSC_DIR}/src/dm/impls/da/fdda.c >> >> /Users/dmay/software/petsc-3.6.0/src >> dmay at nikkan:~/software/petsc-3.6.0/src $ grep -r >> DMCreateMatrix_DA_3d_MPIAIJ * >> dm/impls/da/fdda.c:extern PetscErrorCode >> DMCreateMatrix_DA_3d_MPIAIJ(DM,Mat); >> dm/impls/da/fdda.c:extern PetscErrorCode >> DMCreateMatrix_DA_3d_MPIAIJ_Fill(DM,Mat); >> dm/impls/da/fdda.c: ierr = >> DMCreateMatrix_DA_3d_MPIAIJ_Fill(da,A);CHKERRQ(ierr); >> dm/impls/da/fdda.c: ierr = >> DMCreateMatrix_DA_3d_MPIAIJ(da,A);CHKERRQ(ierr); >> >> >> *dm/impls/da/fdda.c:#define __FUNCT__ >> "DMCreateMatrix_DA_3d_MPIAIJ"dm/impls/da/fdda.c:PetscErrorCode >> DMCreateMatrix_DA_3d_MPIAIJ(DM da,Mat J)*dm/impls/da/fdda.c:#define >> __FUNCT__ "DMCreateMatrix_DA_3d_MPIAIJ_Fill" >> dm/impls/da/fdda.c:PetscErrorCode DMCreateMatrix_DA_3d_MPIAIJ_Fill(DM >> da,Mat J) >> >> >> On 11 February 2016 at 04:08, Matthew Knepley wrote: >> >>> On Wed, Feb 10, 2016 at 8:59 PM, Sang pham van >>> wrote: >>> >>>> The irregular rows is quite many. The matrix really needs to be >>>> preallocated. >>>> Could you show me how to use DMCreateMatrix_DA_3d_MPIAIJ() directly? >>>> >>>> Just put the declaration right into your source. >>> >>> Matt >>> >>>> Pham >>>> On Feb 11, 2016 9:52 AM, "Matthew Knepley" wrote: >>>> >>>>> On Wed, Feb 10, 2016 at 8:44 PM, Sang pham van >>>>> wrote: >>>>> >>>>>> That is because my matrix has some rows which need more entries than >>>>>> usual. >>>>>> >>>>> >>>>> If its only a few, you could just turn off the allocation error. >>>>> >>>>> >>>>>> Where can i find source code of DMCreateMatrix()? >>>>>> >>>>>> >>>>> >>>>> https://bitbucket.org/petsc/petsc/src/827b69d6bb12709ff9b9a0dede31640477fc2b74/src/dm/impls/da/fdda.c?at=master&fileviewer=file-view-default#fdda.c-1024 >>>>> >>>>> Matt >>>>> >>>>>> Pham. >>>>>> On Feb 11, 2016 8:35 AM, "Matthew Knepley" wrote: >>>>>> >>>>>>> On Wed, Feb 10, 2016 at 6:14 PM, Sang pham van >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I am trying to create a DM matrix with DMCreateMatrix_DA_3d_MPIAIJ() >>>>>>>> instead of using DMCreateMatrix(). >>>>>>>> >>>>>>> >>>>>>> Why, that should be called automatically by DMCreateMatrix()? >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> Which header file should I include to use that routine? also, what >>>>>>>> is the source file containing the DMCreateMatrix() routine? >>>>>>>> >>>>>>>> Many thanks in advance. >>>>>>>> Pham >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mossaiby at yahoo.com Thu Feb 11 03:18:25 2016 From: mossaiby at yahoo.com (Farshid Mossaiby) Date: Thu, 11 Feb 2016 09:18:25 +0000 (UTC) Subject: [petsc-users] Parallel FEM code using PETSc References: <1649257156.2608323.1455182305418.JavaMail.yahoo.ref@mail.yahoo.com> Message-ID: <1649257156.2608323.1455182305418.JavaMail.yahoo@mail.yahoo.com> Hi all, Can somebody point me to a simple, linear, parallel FEM code based on PETSc, implementing efficient application of non-homogeneous Dirichlet BC? Domain decomposition code will is also welcome. TIA -------------- next part -------------- An HTML attachment was scrubbed... URL: From vincent.huber at cemosis.fr Thu Feb 11 04:03:11 2016 From: vincent.huber at cemosis.fr (Vincent Huber) Date: Thu, 11 Feb 2016 11:03:11 +0100 Subject: [petsc-users] Parallel FEM code using PETSc In-Reply-To: <1649257156.2608323.1455182305418.JavaMail.yahoo@mail.yahoo.com> References: <1649257156.2608323.1455182305418.JavaMail.yahoo.ref@mail.yahoo.com> <1649257156.2608323.1455182305418.JavaMail.yahoo@mail.yahoo.com> Message-ID: Hello, I?m not convinced the following will be find simple enough but Feel++ match the other expectations. In the example on the main page of the web site, you only have to change the line a+=on(_range=boundaryfaces(mesh), _rhs=l, _element=u, _expr=cst(0.) ); to a+=on(_range=boundaryfaces(mesh), _rhs=l, _element=u, _expr=expr(soption("functions.g")) ); and run the code with mpirun -np xxx ./feelpp_theApp --functions.g="2*x*y=x=y" To change the bc you want to impose. I (highly) recommend Ubuntu 14.04 to test the library. The documentation is here You may also have a look to FreeFem++ All the best, VH ? 2016-02-11 10:18 GMT+01:00 Farshid Mossaiby : > Hi all, > > Can somebody point me to a simple, linear, parallel FEM code based on > PETSc, implementing efficient application of non-homogeneous Dirichlet BC? > Domain decomposition code will is also welcome. > > TIA > -- Docteur Ing?nieur de recherche CeMoSiS - vincent.huber at cemosis.fr Tel: +33 (0)3 68 8*5 02 06* IRMA - 7, rue Ren? Descartes 67 000 Strasbourg -------------- next part -------------- An HTML attachment was scrubbed... URL: From mossaiby at yahoo.com Thu Feb 11 05:22:37 2016 From: mossaiby at yahoo.com (Farshid Mossaiby) Date: Thu, 11 Feb 2016 11:22:37 +0000 (UTC) Subject: [petsc-users] Parallel FEM code using PETSc In-Reply-To: References: <1649257156.2608323.1455182305418.JavaMail.yahoo.ref@mail.yahoo.com> <1649257156.2608323.1455182305418.JavaMail.yahoo@mail.yahoo.com> Message-ID: <2110283153.2611159.1455189757184.JavaMail.yahoo@mail.yahoo.com> Thanks for your reply. Indeed I want to expand the code to non-linear cases and use it as a study tool for some other researches. This requires plain C++ code to be available so that I can change different parts of it easily. Using a complex library unfortunately does not fit my needs for this research. Bests,FM From: Vincent Huber To: Farshid Mossaiby Cc: PETSc Users List Sent: Thursday, February 11, 2016 1:33 PM Subject: Re: [petsc-users] Parallel FEM code using PETSc Hello,I?m not convinced the following will be find simple enough but Feel++ match the other expectations.In the example on the main page of the web site, you only have to change the line a+=on(_range=boundaryfaces(mesh), _rhs=l, _element=u, _expr=cst(0.) ); to a+=on(_range=boundaryfaces(mesh), _rhs=l, _element=u, _expr=expr(soption("functions.g")) ); and run the code withmpirun -np xxx ./feelpp_theApp --functions.g="2*x*y=x=y" To change the bc you want to impose.I (highly) recommend Ubuntu 14.04 to test the library. The documentation is hereYou may also have a look to FreeFem++All the best,VH? 2016-02-11 10:18 GMT+01:00 Farshid Mossaiby : Hi all, Can somebody point me to a simple, linear, parallel FEM code based on PETSc, implementing efficient application of non-homogeneous Dirichlet BC? Domain decomposition code will is also welcome. TIA -- Docteur Ing?nieur de rechercheCeMoSiS?- vincent.huber at cemosis.frTel: +33 (0)3 68 85 02 06IRMA - 7, rue Ren? Descartes67 000 Strasbourg -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Feb 11 08:50:10 2016 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 11 Feb 2016 08:50:10 -0600 Subject: [petsc-users] DIVERGED_PCSETUP_FAILED In-Reply-To: References: <1455154398.5948.15.camel@enterprise-A> <1455157291.5948.19.camel@enterprise-A> <0903A02C-4B7A-4C1F-AD27-44CB6B1C6F07@mcs.anl.gov> <1455162540.5948.21.camel@enterprise-A> <1455170718.5948.28.camel@enterprise-A> Message-ID: On Thu, Feb 11, 2016 at 12:30 AM, Dave May wrote: > > > On 11 February 2016 at 07:05, Michele Rosso wrote: > >> I tried setting -mat_superlu_dist_replacetinypivot true: it does help to >> advance the run past the previous "critical" point but eventually it stops >> later with the same error. >> I forgot to mention my system is singular: I remove the constant null >> space but I am not sure if the coarse solver needs to be explicity informed >> of this. >> > > Right - are you using pure Newmann boundary conditions? > > To make the solution unique, are you > (a) imposing a single Dichletet boundary condition on your field > by explicitly modifying the matrix > (b) imposing a a condition like > \int \phi dV = 0 > via something like -ksp_constant_null_space > > If you removed removed the null space by modifying the matrix explicitly > (a), the sparse direct solver > should go through. If you use (b), then this method cannot be used to help > the direct solver. > > If this is the intended target problem size (16x16), gather the matrix and > using petsc Cholesky or Umfpack. > Cholesky is more stable than LU and can usually deal with a single zero > eigenvaue without resorting to tricks. Umfpack will solve the problem > easily as it uses clever re-ordering. If you have access to MKL-Pardiso, > that will also work great. > An easy fix is just to use -pc_type svd on the coarse grid. Matt > Thanks, > Dave > > > >> >> >> Michele >> >> >> On Wed, 2016-02-10 at 22:15 -0600, Barry Smith wrote: >> >> You can try the option >> >> -mat_superlu_dist_replacetinypivot true >> >> if you are luck it get you past the zero pivot but still give an adequate preconditioner. >> >> Barry >> > On Feb 10, 2016, at 9:49 PM, Michele Rosso wrote:> > Hong,> > here if the output of grep -info:> > using diagonal shift on blocks to prevent zero pivot [INBLOCKS]> Replace tiny pivots FALSE> tolerance for zero pivot 2.22045e-14> > It seems it is not replacing small pivots: could this be the problem?> I will also try Barry's suggestion to diagnose the problem.> > Thanks,> Michele> > > On Wed, 2016-02-10 at 21:22 -0600, Barry Smith wrote:>> > On Feb 10, 2016, at 9:00 PM, Hong wrote:>> > >> > Michele :>> > Superlu_dist LU is used for coarse grid PC, which likely produces a zero-pivot.>> > Run your code with '-info |grep pivot' to verify.>> >> >> Michele>> >> You can also run with -ksp_error_if_not_converged in or not in the debugger and it will stop immediately when the problem is detected and hopefully provide additional useful information about what has happened.>> >> Barry>> >> >> > >> > Hong>> > >> > Hi Matt,>> > >> > the ksp_view output was an attachment to my previous email.>> > Here it is:>> > >> > KSP Object: 1 MPI processes>> > type: cg>> > maximum iterations=10000>> > tolerances: relative=1e-08, absolute=1e-50, divergence=10000.>> > left preconditioning>> > using nonzero initial guess>> > using UNPRECONDITIONED norm type for convergence test>> > PC Object: 1 MPI processes>> > type: mg>> > MG: type is MULTIPLICATIVE, levels=4 cycles=v>> > Cycles per PCApply=1>> > Using Galerkin computed coarse grid matrices>> > Coarse grid solver -- level ------------------------------->> > KSP Object: (mg_coarse_) 1 MPI processes>> > type: preonly>> > maximum iterations=1, initial guess is zero>> > tolerances: relative=1e-05, absolute=1e-50, divergence=10000.>> > left preconditioning>> > using NONE norm type for convergence test>> > PC Object: (mg_coarse_) 1 MPI processes>> > type: lu>> > LU: out-of-place factorization>> > tolerance for zero pivot 2.22045e-14>> > using diagonal shift on blocks to prevent zero pivot [INBLOCKS]>> > matrix ordering: nd>> > factor fill ratio given 0., needed 0.>> > Factored matrix follows:>> > Mat Object: 1 MPI processes>> > type: seqaij>> > rows=16, cols=16>> > package used to perform factorization: superlu_dist>> > total: nonzeros=0, allocated nonzeros=0>> > total number of mallocs used during MatSetValues calls =0>> > SuperLU_DIST run parameters:>> > Process grid nprow 1 x npcol 1 >> > Equilibrate matrix TRUE >> > Matrix input mode 0 >> > Replace tiny pivots FALSE >> > Use iterative refinement FALSE >> > Processors in row 1 col partition 1 >> > Row permutation LargeDiag >> > Column permutation METIS_AT_PLUS_A>> > Parallel symbolic factorization FALSE >> > Repeated factorization SamePattern>> > linear system matrix = precond matrix:>> > Mat Object: 1 MPI processes>> > type: seqaij>> > rows=16, cols=16>> > total: nonzeros=72, allocated nonzeros=72>> > total number of mallocs used during MatSetValues calls =0>> > not using I-node routines>> > Down solver (pre-smoother) on level 1 ------------------------------->> > KSP Object: (mg_levels_1_) 1 MPI processes>> > type: richardson>> > Richardson: damping factor=1.>> > maximum iterations=2>> > tolerances: relative=1e-05, absolute=1e-50, divergence=10000.>> > left preconditioning>> > using nonzero initial guess>> > using NONE norm type for convergence test>> > PC Object: (mg_levels_1_) 1 MPI processes>> > type: sor>> > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.>> > linear system matrix = precond matrix:>> > Mat Object: 1 MPI processes>> > type: seqaij>> > rows=64, cols=64>> > total: nonzeros=304, allocated nonzeros=304>> > total number of mallocs used during MatSetValues calls =0>> > not using I-node routines>> > Up solver (post-smoother) same as down solver (pre-smoother)>> > Down solver (pre-smoother) on level 2 ------------------------------->> > KSP Object: (mg_levels_2_) 1 MPI processes>> > type: richardson>> > Richardson: damping factor=1.>> > maximum iterations=2>> > tolerances: relative=1e-05, absolute=1e-50, divergence=10000.>> > left preconditioning>> > using nonzero initial guess>> > using NONE norm type for convergence test>> > PC Object: (mg_levels_2_) 1 MPI processes>> > type: sor>> > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.>> > linear system matrix = precond matrix:>> > Mat Object: 1 MPI processes>> > type: seqaij>> > rows=256, cols=256>> > total: nonzeros=1248, allocated nonzeros=1248>> > total number of mallocs used during MatSetValues calls =0>> > not using I-node routines>> > Up solver (post-smoother) same as down solver (pre-smoother)>> > Down solver (pre-smoother) on level 3 ------------------------------->> > KSP Object: (mg_levels_3_) 1 MPI processes>> > type: richardson>> > Richardson: damping factor=1.>> > maximum iterations=2>> > tolerances: relative=1e-05, absolute=1e-50, divergence=10000.>> > left preconditioning>> > using nonzero initial guess>> > using NONE norm type for convergence test>> > PC Object: (mg_levels_3_) 1 MPI processes>> > type: sor>> > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.>> > linear system matrix = precond matrix:>> > Mat Object: 1 MPI processes>> > type: seqaij>> > rows=1024, cols=1024>> > total: nonzeros=5056, allocated nonzeros=5056>> > total number of mallocs used during MatSetValues calls =0>> > has attached null space>> > not using I-node routines>> > Up solver (post-smoother) same as down solver (pre-smoother)>> > linear system matrix = precond matrix:>> > Mat Object: 1 MPI processes>> > type: seqaij>> > rows=1024, cols=1024>> > total: nonzeros=5056, allocated nonzeros=5056>> > total number of mallocs used during MatSetValues calls =0>> > has attached null space>> > not using I-node routines>> > >> > >> > Michele>> > >> > >> > >> > >> > On Wed, 2016-02-10 at 19:37 -0600, Matthew Knepley wrote:>> >> On Wed, Feb 10, 2016 at 7:33 PM, Michele Rosso wrote:>> >> Hi,>> >> >> >> I encountered the following error while solving a symmetric positive defined system:>> >> >> >> Linear solve did not converge due to DIVERGED_PCSETUP_FAILED iterations 0>> >> PCSETUP_FAILED due to SUBPC_ERROR >> >> >> >> This error appears only if I use the optimized version of both petsc and my code ( compiler: gfortran, flags: -O3 ).>> >> It is weird since I am solving a time-dependent problem and everything, i.e. results and convergence rate, are as expected until the above error shows up. If I run both petsc and my code in debug mode, everything goes smooth till the end of the simulation.>> >> However, if I reduce the ksp_rtol, even the debug run fails, after running as expected for a while, because of a KSP_DIVERGED_INDEFINITE_PC . >> >> The options I am using are:>> >> >> >> -ksp_type cg>> >> -ksp_norm_type unpreconditioned>> >> -ksp_rtol 1e-8>> >> -ksp_lag_norm>> >> -ksp_initial_guess_nonzero yes>> >> -pc_type mg>> >> -pc_mg_galerkin>> >> -pc_mg_levels 4>> >> -mg_levels_ksp_type richardson>> >> -mg_coarse_ksp_constant_null_space>> >> -mg_coarse_pc_type lu>> >> -mg_coarse_pc_factor_mat_solver_package superlu_dist>> >> -options_left>> >> >> >> I attached a copy of ksp_view. I am currently using petsc-master (last updated yesterday).>> >> I would appreciate any suggestion on this matter.>> >> >> >> >> >> >> >> I suspect you have a nonlinear PC. Can you send the output of -ksp_view?>> >> >> >> >> >> Matt>> >> >> >> Thanks,>> >> Michele>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -->> >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.>> >> -- Norbert Wiener>> > >> > >> >> >> > >> >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hng.email at gmail.com Thu Feb 11 11:47:14 2016 From: hng.email at gmail.com (Hom Nath Gharti) Date: Thu, 11 Feb 2016 12:47:14 -0500 Subject: [petsc-users] PCFIELDSPLIT question In-Reply-To: References: Message-ID: Thanks all. With further help from Stefano Zhampini I was able to solve my small test case using PCFieldSPlit option. I had to add call PCSetType(pc,PCFIELDSPLIT,ierr) Now I am trying to use same technique in my actual problem and got the error: PETSC ERROR: Fields must be sorted when creating split Does this mean I have to sort the global matrix rows according the split fields? Thanks Hom Nath On Tue, Feb 9, 2016 at 10:57 AM, Hom Nath Gharti wrote: > Sounds interesting! Thanks a lot Matt! I will have a look. > > Hom Nath > > On Tue, Feb 9, 2016 at 10:36 AM, Matthew Knepley wrote: >> On Tue, Feb 9, 2016 at 9:31 AM, Hom Nath Gharti wrote: >>> >>> Thanks a lot Matt! >>> >>> Were you referring to >>> >>> http://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex62.c.html >>> ? >>> >>> I do not see any statements related to PCFieldSplit there. Am I >>> missing something here? >> >> >> Yes. The whole point is not to put anything specific to FIELDSPLIT in the >> code. It is all in options. For example, >> >> ./ex62 -run_type full -refinement_limit 0.00625 -bc_type dirichlet >> -interpolate 1 -vel_petscspace_order 2 -pres_petscspace_order 1 -ksp_type >> fgmres -ksp_gmres_restart 100 -ksp_rtol 1.0e-9 -pc_type fieldsplit >> -pc_fieldsplit_type schur -pc_fieldsplit_schur_factorization_type upper >> -fieldsplit_pressure_ksp_rtol 1e-10 -fieldsplit_velocity_ksp_type gmres >> -fieldsplit_velocity_pc_type lu -fieldsplit_pressure_pc_type jacobi >> -snes_monitor_short -ksp_monitor_short -snes_converged_reason >> -ksp_converged_reason -snes_view -show_solution 0 >> >> The reason that it works is that the DM calls PCFIeldSplitSetIS() for each >> field in the DM. >> >> Thanks, >> >> Matt >> >>> >>> Thanks, >>> Hom Nath >>> >>> On Tue, Feb 9, 2016 at 10:19 AM, Matthew Knepley >>> wrote: >>> > On Tue, Feb 9, 2016 at 9:10 AM, Hom Nath Gharti >>> > wrote: >>> >> >>> >> Thank you so much Barry! >>> >> >>> >> For my small test case, with -pc_fieldsplit_block_size 4, the program >>> >> runs, although the answer was not correct. At least now I get >>> >> something to look upon. I am using PCFieldSplitSetIS to set the >>> >> fields. Do I still need to use -pc_fieldsplit_block_size? >>> > >>> > >>> > No, this is only for co-located discretizations. >>> > >>> >> >>> >> In my case each grid point may have different variable sets. For >>> >> example, the grid point in the solid region has displacement and >>> >> gravity potential: ux, uy, uz, and \phi. Whereas the grid point in the >>> >> fluid region has scalar potential, pressure and gravity potential: >>> >> \chi, p, and \phi. And the solid-fluid interface has all of them. Do >>> >> you think I can still use PCFIELDSPLIT in this situation? >>> > >>> > >>> > We have examples, like SNES ex62, which do exactly this. >>> > >>> > Thanks, >>> > >>> > Matt >>> > >>> >> >>> >> Best, >>> >> Hom Nath >>> >> >>> >> >>> >> >>> >> On Mon, Feb 8, 2016 at 2:27 PM, Barry Smith wrote: >>> >> > >>> >> > >>> >> > In this case you need to provide two pieces of information to the >>> >> > PCFIELDSPLIT. What we call the "block size" or bs which is the number >>> >> > of >>> >> > "basic fields" in the problem. For example if at each grid point you >>> >> > have >>> >> > x-velocity, y-velocity, and pressure the block size is 3. The second >>> >> > is you >>> >> > need to tell PCFIELDSPLIT what "basic fields" are in each split you >>> >> > want to >>> >> > use. >>> >> > >>> >> > In your case you can do this with -pc_fieldsplit_block_size 4. >>> >> > (Note if you use a DM to define your problem the PCFIELDSPLIT will >>> >> > automatically get the bs from the DM so you do not need to provide >>> >> > it. >>> >> > Similarly if you set a block size on your Mat the PCFIELDSPLIT will >>> >> > use that >>> >> > so you don't need to set it). >>> >> > >>> >> > Then use -pc_fieldsplit_0_fields 0,1 to indicate your first split >>> >> > is >>> >> > the 0 and 1 basic fields and -pc_fieldsplit_1_fields 2,3 to indicate >>> >> > the >>> >> > second split is the 2 and 3 basic fields. >>> >> > (By default if you do not provide this then PCFIELDSPLIT will use bs >>> >> > splits (4 in your case) one for each basic field. >>> >> > >>> >> > Barry >>> >> > >>> >> >> On Feb 8, 2016, at 11:20 AM, Hom Nath Gharti >>> >> >> wrote: >>> >> >> >>> >> >> Hi Matt, Hi all, >>> >> >> >>> >> >> I am trying to have some feel for PCFIELDSPLIT testing on a very >>> >> >> small >>> >> >> matrix (attached below). I have 4 degrees of freedoms. I use 4 >>> >> >> processors. I want to split 4 dofs into two fields each having two >>> >> >> dofs. I don't know whether this my be a problem for petsc. When I >>> >> >> use >>> >> >> the command: >>> >> >> srun -n 4 ./xtestfs -ksp_monitor -ksp_view >>> >> >> >>> >> >> It runs fine. >>> >> >> >>> >> >> But when I want to use field split options using the command: >>> >> >> srun -n 4 ./xtestfs -ksp_monitor -ksp_type fgmres -pc_type >>> >> >> fieldsplit >>> >> >> -fieldsplit_0_ksp_type gmres -fieldsplit_0_pc_type bjacobi >>> >> >> -fieldsplit_1_pc_type jacobi >>> >> >> >>> >> >> I get the following error message: >>> >> >> [0]PETSC ERROR: Petsc has generated inconsistent data >>> >> >> [0]PETSC ERROR: Unhandled case, must have at least two fields, not 1 >>> >> >> [0]PETSC ERROR: See >>> >> >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >>> >> >> shooting. >>> >> >> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>> >> >> [0]PETSC ERROR: /tigress/hgharti/test/testfs/./xtestfs on a >>> >> >> arch-linux2-c-debug named tiger-r11n1 by hgharti Mon Feb 8 11:40:03 >>> >> >> 2016 >>> >> >> [0]PETSC ERROR: Configure options >>> >> >> >>> >> >> >>> >> >> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64/ >>> >> >> --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 >>> >> >> --with-mpiexec=mpiexec --with-debugging=1 --download-scalapack >>> >> >> --download-mumps --download-pastix --download-superlu >>> >> >> --download-superlu_dist --download-metis --download-parmetis >>> >> >> --download-ptscotch --download-hypre >>> >> >> [0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 469 in >>> >> >> >>> >> >> >>> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >>> >> >> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 486 in >>> >> >> >>> >> >> >>> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >>> >> >> [0]PETSC ERROR: #3 PCSetUp() line 983 in >>> >> >> >>> >> >> >>> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/interface/precon.c >>> >> >> [0]PETSC ERROR: #4 KSPSetUp() line 332 in >>> >> >> >>> >> >> >>> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >>> >> >> [0]PETSC ERROR: #5 KSPSolve() line 546 in >>> >> >> >>> >> >> >>> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >>> >> >> WARNING! There are options you set that were not used! >>> >> >> WARNING! could be spelling mistake, etc! >>> >> >> Option left: name:-fieldsplit_0_ksp_type value: gmres >>> >> >> Option left: name:-fieldsplit_0_pc_type value: bjacobi >>> >> >> Option left: name:-fieldsplit_1_pc_type value: jacobi >>> >> >> forrtl: error (76): Abort trap signal >>> >> >> >>> >> >> I tried several trials but could not fix the problem. Is it the >>> >> >> FORTRAN problem or am I doing something wrong? Any suggestions would >>> >> >> be greatly appreciated. >>> >> >> For your information I use: >>> >> >> PETSc-3.6.3 >>> >> >> intel-16.0 compiler >>> >> >> intel-mpi-5.1.1 >>> >> >> >>> >> >> Program >>> >> >> >>> >> >> >>> >> >> Best, >>> >> >> Hom Nath >>> >> >> >>> >> >> ! simple test case for PCFIELDSPLIT >>> >> >> ! >>> >> >> >>> >> >> ----------------------------------------------------------------------- >>> >> >> program testfs >>> >> >> implicit none >>> >> >> #include "petsc/finclude/petscsys.h" >>> >> >> #include "petsc/finclude/petscvec.h" >>> >> >> #include "petsc/finclude/petscvec.h90" >>> >> >> #include "petsc/finclude/petscmat.h" >>> >> >> #include "petsc/finclude/petscksp.h" >>> >> >> #include "petsc/finclude/petscpc.h" >>> >> >> #include "petsc/finclude/petscviewer.h" >>> >> >> #include "petsc/finclude/petscviewer.h90" >>> >> >> >>> >> >> Vec x,b,u >>> >> >> Mat A >>> >> >> KSP ksp >>> >> >> PC pc >>> >> >> PetscErrorCode ierr >>> >> >> PetscBool flg >>> >> >> PetscInt errcode,its,maxitr,myrank,nproc >>> >> >> PetscInt nedof,nelmt,ndof,nenod,ngdof,fixdof,nsparse,neq >>> >> >> PetscInt,allocatable :: krow_sparse(:),ggdof_elmt(:,:), & >>> >> >> inode_elmt(:,:) >>> >> >> >>> >> >> PetscReal,allocatable :: storef(:,:),storekmat(:,:,:) >>> >> >> PetscInt gdof0(2),gdof1(2) >>> >> >> >>> >> >> ! initialize MPI >>> >> >> >>> >> >> call MPI_INIT(errcode) >>> >> >> if(errcode /= 0)stop 'ERROR: cannot initialize MPI!' >>> >> >> call MPI_COMM_RANK(MPI_COMM_WORLD,myrank,errcode) >>> >> >> if(errcode /= 0)stop 'ERROR: cannot find processor ID' >>> >> >> call MPI_COMM_SIZE(MPI_COMM_WORLD,nproc,errcode) >>> >> >> if(errcode /= 0)stop 'ERROR: cannot find number of processors!' >>> >> >> >>> >> >> ! define some parameters >>> >> >> ! 1D model with 4 elements. Each elements consits of 2 nodes. Node 0 >>> >> >> is >>> >> >> fixed. >>> >> >> ! 1-----------2-----------3-----------4-----------5 >>> >> >> nelmt=1 ! per processor >>> >> >> nenod=2 ! number of nodes per element >>> >> >> nedof=2 ! number of DOFs per element >>> >> >> ndof=2 ! number of DOFs >>> >> >> ngdof=4 ! total number of global DOFs >>> >> >> fixdof=1 >>> >> >> >>> >> >> if(myrank==0)then >>> >> >> neq=1 ! number of equations >>> >> >> nsparse=1 >>> >> >> else >>> >> >> neq=2 ! number of equations >>> >> >> nsparse=4 >>> >> >> endif >>> >> >> >>> >> >> allocate(storef(nedof,nelmt),storekmat(nedof,nedof,nelmt), >>> >> >> & >>> >> >> >>> >> >> ggdof_elmt(nedof,nelmt),inode_elmt(nenod,nelmt),krow_sparse(nsparse)) >>> >> >> >>> >> >> storef=0.0_8 ! local RHS vector >>> >> >> storekmat=0.0_8 ! local stiffness matrix >>> >> >> if(myrank==0)then >>> >> >> krow_sparse(:)=1 >>> >> >> storef(:,1)=(/12.5000_8, 12.5000_8/) >>> >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & >>> >> >> 0.0166666667_8, >>> >> >> 1.2333333332_8/),(/nedof,nedof/)) >>> >> >> inode_elmt(:,1) = (/1,2/) >>> >> >> ggdof_elmt(:,1) = (/0,1/) ! 0 because left node is fixed >>> >> >> elseif(myrank==1)then >>> >> >> krow_sparse(:)=(/1,1,2,2/) >>> >> >> storef(:,1)=(/12.5000_8, 12.5000_8/) >>> >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & >>> >> >> 0.0166666667_8, >>> >> >> 1.2333333332_8/),(/nedof,nedof/)) >>> >> >> inode_elmt(:,1) = (/1,2/) >>> >> >> ggdof_elmt(:,1) = (/1,2/) >>> >> >> elseif(myrank==2)then >>> >> >> krow_sparse(:)=(/1,1,2,2/) >>> >> >> storef(:,1)=(/12.5000_8, 12.5000_8/) >>> >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & >>> >> >> 0.0166666667_8, >>> >> >> 1.2333333332_8/),(/nedof,nedof/)) >>> >> >> inode_elmt(:,1) = (/1,2/) >>> >> >> ggdof_elmt(:,1) = (/2,3/) >>> >> >> elseif(myrank==3)then >>> >> >> krow_sparse(:)=(/1,1,2,2/) >>> >> >> storef(:,1)=(/12.5000_8, 22.5000_8/) >>> >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & >>> >> >> 0.0166666667_8, >>> >> >> 1.2333333332_8/),(/nedof,nedof/)) >>> >> >> inode_elmt(:,1) = (/1,2/) >>> >> >> ggdof_elmt(:,1) = (/3,4/) >>> >> >> endif >>> >> >> >>> >> >> call petsc_initialize >>> >> >> call petsc_set_matrix >>> >> >> call petsc_set_vector >>> >> >> call petsc_solve >>> >> >> call petsc_finalize >>> >> >> >>> >> >> ! deallocate >>> >> >> deallocate(storef,storekmat,ggdof_elmt,inode_elmt) >>> >> >> deallocate(krow_sparse) >>> >> >> contains >>> >> >> >>> >> >> >>> >> >> !------------------------------------------------------------------------------- >>> >> >> subroutine petsc_initialize() >>> >> >> implicit none >>> >> >> PetscInt :: istart,iend >>> >> >> PetscInt :: nzeros_max,nzeros_min >>> >> >> PetscReal :: atol,dtol,rtol >>> >> >> PetscInt,allocatable :: nzeros(:) >>> >> >> IS gdof0_is,gdof1_is >>> >> >> >>> >> >> call PetscInitialize(PETSC_NULL_CHARACTER,ierr) >>> >> >> call PetscOptionsGetInt(PETSC_NULL_CHARACTER,'-n',ngdof,flg,ierr) >>> >> >> >>> >> >> ! count number of nonzeros per row >>> >> >> allocate(nzeros(neq)) >>> >> >> nzeros=0 >>> >> >> nzeros(krow_sparse)=nzeros(krow_sparse)+1 >>> >> >> nzeros_max=maxval(nzeros) >>> >> >> nzeros_min=minval(nzeros) >>> >> >> >>> >> >> ! create matrix >>> >> >> call MatCreate(PETSC_COMM_WORLD,A,ierr) >>> >> >> call MatSetType(A,MATMPIAIJ,ierr) >>> >> >> CHKERRQ(ierr) >>> >> >> call MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,ngdof,ngdof,ierr) >>> >> >> CHKERRQ(ierr) >>> >> >> ! preallocation >>> >> >> call >>> >> >> >>> >> >> MatMPIAIJSetPreallocation(A,nzeros_max,PETSC_NULL_INTEGER,nzeros_max, & >>> >> >> PETSC_NULL_INTEGER,ierr) >>> >> >> CHKERRQ(ierr) >>> >> >> >>> >> >> call MatGetOwnershipRange(A,istart,iend,ierr) >>> >> >> CHKERRQ(ierr) >>> >> >> print*,'ownership:',myrank,istart,iend >>> >> >> ! create vector >>> >> >> call VecCreate(PETSC_COMM_WORLD,x,ierr) >>> >> >> call VecSetSizes(x,PETSC_DECIDE,ngdof,ierr) >>> >> >> call VecSetType(x,VECMPI,ierr) >>> >> >> call VecDuplicate(x,b,ierr) >>> >> >> call VecDuplicate(x,u,ierr) >>> >> >> >>> >> >> >>> >> >> >>> >> >> !******************************************************************************* >>> >> >> ! get IS for split fields >>> >> >> gdof0=(/0,1/) >>> >> >> gdof1=(/2,3/) >>> >> >> call >>> >> >> >>> >> >> ISCreateGeneral(PETSC_COMM_WORLD,2,gdof0,PETSC_COPY_VALUES,gdof0_is,ierr) >>> >> >> CHKERRQ(ierr) >>> >> >> call >>> >> >> >>> >> >> ISCreateGeneral(PETSC_COMM_WORLD,2,gdof1,PETSC_COPY_VALUES,gdof1_is,ierr) >>> >> >> CHKERRQ(ierr) >>> >> >> >>> >> >> >>> >> >> !******************************************************************************* >>> >> >> >>> >> >> ! Create linear solver context >>> >> >> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >>> >> >> call KSPSetOperators(ksp,A,A,ierr) >>> >> >> >>> >> >> call KSPGetPC(ksp,pc,ierr) >>> >> >> CHKERRQ(ierr) >>> >> >> >>> >> >> >>> >> >> >>> >> >> !******************************************************************************* >>> >> >> ! split PC >>> >> >> call PCFieldSplitSetIS(pc,"0",gdof0_is,ierr); >>> >> >> CHKERRQ(ierr) >>> >> >> call ISDestroy(gdof0_is,ierr) >>> >> >> CHKERRQ(ierr) >>> >> >> call PCFieldSplitSetIS(pc,"1",gdof1_is,ierr); >>> >> >> CHKERRQ(ierr) >>> >> >> call ISDestroy(gdof1_is,ierr) >>> >> >> CHKERRQ(ierr) >>> >> >> >>> >> >> >>> >> >> !******************************************************************************* >>> >> >> >>> >> >> rtol = 1.0d-6 >>> >> >> atol = 1.0d-6 >>> >> >> dtol = 1.0d10 >>> >> >> maxitr = 1000 >>> >> >> call KSPSetTolerances(ksp,rtol,atol,dtol,maxitr,ierr) >>> >> >> CHKERRQ(ierr) >>> >> >> call KSPSetFromOptions(ksp,ierr) >>> >> >> end subroutine petsc_initialize >>> >> >> >>> >> >> >>> >> >> !------------------------------------------------------------------------------- >>> >> >> >>> >> >> subroutine petsc_set_matrix() >>> >> >> >>> >> >> implicit none >>> >> >> integer :: i,i_elmt,j,ncount >>> >> >> integer :: ggdof(NEDOF),idof(NEDOF),igdof(NEDOF) >>> >> >> >>> >> >> ! Set and assemble matrix. >>> >> >> call MatZeroEntries(A,ierr) >>> >> >> CHKERRQ(ierr) >>> >> >> do i_elmt=1,nelmt >>> >> >> ggdof(:)=ggdof_elmt(:,i_elmt) >>> >> >> ggdof(:)=ggdof(:)-1 ! petsc index starts from 0 >>> >> >> ncount=0; idof=-1; igdof=-1 >>> >> >> do i=1,NEDOF >>> >> >> do j=1,NEDOF >>> >> >> if(ggdof(i).ge.0.and.ggdof(j).ge.0)then >>> >> >> call >>> >> >> MatSetValues(A,1,ggdof(i),1,ggdof(j),storekmat(i,j,i_elmt), >>> >> >> & >>> >> >> ADD_VALUES,ierr) >>> >> >> CHKERRQ(ierr) >>> >> >> endif >>> >> >> enddo >>> >> >> enddo >>> >> >> enddo >>> >> >> call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) >>> >> >> call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) >>> >> >> if(myrank==0)print*,'matrix setting & assembly complete!' >>> >> >> >>> >> >> end subroutine petsc_set_matrix >>> >> >> >>> >> >> >>> >> >> !------------------------------------------------------------------------------- >>> >> >> >>> >> >> subroutine petsc_set_vector() >>> >> >> implicit none >>> >> >> PetscScalar zero >>> >> >> integer :: i,i_elmt,ncount >>> >> >> integer :: ggdof(NEDOF),idof(NEDOF),igdof(NEDOF) >>> >> >> >>> >> >> ! set vector >>> >> >> zero=0.0_8 >>> >> >> call VecSet(b,zero,ierr) >>> >> >> do i_elmt=1,nelmt >>> >> >> ggdof(:)=ggdof_elmt(:,i_elmt) >>> >> >> ggdof(:)=ggdof(:)-1 ! petsc index starts from 0 >>> >> >> ncount=0; idof=-1; igdof=-1 >>> >> >> do i=1,NEDOF >>> >> >> if(ggdof(i).ge.0)then >>> >> >> call >>> >> >> VecSetValues(b,1,ggdof(i),storef(i,i_elmt),ADD_VALUES,ierr); >>> >> >> CHKERRQ(ierr) >>> >> >> endif >>> >> >> enddo >>> >> >> enddo >>> >> >> >>> >> >> ! assemble vector >>> >> >> call VecAssemblyBegin(b,ierr) >>> >> >> call VecAssemblyEnd(b,ierr) >>> >> >> if(myrank==0)print*,'vector setting & assembly complete!' >>> >> >> >>> >> >> end subroutine petsc_set_vector >>> >> >> >>> >> >> >>> >> >> !------------------------------------------------------------------------------- >>> >> >> >>> >> >> subroutine petsc_solve() >>> >> >> implicit none >>> >> >> PetscInt ireason >>> >> >> PetscScalar a_v(1) >>> >> >> PetscOffset a_i >>> >> >> PetscInt n >>> >> >> PetscReal,allocatable :: sdata(:) >>> >> >> >>> >> >> call VecGetSize(b,n,ierr) >>> >> >> CHKERRQ(ierr) >>> >> >> allocate(sdata(n)) >>> >> >> sdata=0.0_8 >>> >> >> call VecGetArray(b,a_v,a_i,ierr) >>> >> >> CHKERRQ(ierr) >>> >> >> sdata(1:n)=a_v(a_i+1:a_i+n) >>> >> >> call VecRestoreArray(b,a_v,a_i,ierr) >>> >> >> CHKERRQ(ierr) >>> >> >> print*,'rhs:',sdata >>> >> >> >>> >> >> call KSPSolve(ksp,b,x,ierr) >>> >> >> sdata=0.0_8 >>> >> >> call VecGetArray(x,a_v,a_i,ierr) >>> >> >> CHKERRQ(ierr) >>> >> >> sdata(1:n)=a_v(a_i+1:a_i+n) >>> >> >> call VecRestoreArray(b,a_v,a_i,ierr) >>> >> >> CHKERRQ(ierr) >>> >> >> print*,'solution:',sdata >>> >> >> >>> >> >> ! Check convergence >>> >> >> call KSPGetConvergedReason(ksp,ireason,ierr) >>> >> >> if(myrank==0)print*,'converges reason',myrank,ireason >>> >> >> call KSPGetIterationNumber(ksp,its,ierr) >>> >> >> if(myrank==0)print*,'Iterations:',its >>> >> >> deallocate(sdata) >>> >> >> end subroutine petsc_solve >>> >> >> >>> >> >> >>> >> >> !------------------------------------------------------------------------------- >>> >> >> >>> >> >> subroutine petsc_finalize() >>> >> >> implicit none >>> >> >> >>> >> >> ! Free work space. >>> >> >> call VecDestroy(x,ierr) >>> >> >> call VecDestroy(u,ierr) >>> >> >> call VecDestroy(b,ierr) >>> >> >> call MatDestroy(A,ierr) >>> >> >> call KSPDestroy(ksp,ierr) >>> >> >> call PetscFinalize(ierr) >>> >> >> CHKERRQ(ierr) >>> >> >> >>> >> >> end subroutine petsc_finalize >>> >> >> >>> >> >> >>> >> >> !------------------------------------------------------------------------------- >>> >> >> >>> >> >> end program testfs >>> >> >> >>> >> >> On Tue, Feb 2, 2016 at 4:54 PM, Matthew Knepley >>> >> >> wrote: >>> >> >>> On Tue, Feb 2, 2016 at 3:11 PM, Hom Nath Gharti >>> >> >>> >>> >> >>> wrote: >>> >> >>>> >>> >> >>>> Thanks a lot. I will try. >>> >> >>> >>> >> >>> >>> >> >>> Also, if you send a small test case, we can run it and tell you the >>> >> >>> problem. >>> >> >>> >>> >> >>> Matt >>> >> >>> >>> >> >>>> >>> >> >>>> Hom Nath >>> >> >>>> >>> >> >>>> On Tue, Feb 2, 2016 at 4:02 PM, Matthew Knepley >>> >> >>>> >>> >> >>>> wrote: >>> >> >>>>> On Tue, Feb 2, 2016 at 3:01 PM, Hom Nath Gharti >>> >> >>>>> >>> >> >>>>> wrote: >>> >> >>>>>> >>> >> >>>>>> Thanks again Matt. Unfortunately got the same errors with '0'. I >>> >> >>>>>> think >>> >> >>>>>> both are valid in Fortran. >>> >> >>>>> >>> >> >>>>> >>> >> >>>>> Then you will have to go in the debugger and see why its not >>> >> >>>>> creating 4 >>> >> >>>>> splits, since this is exactly >>> >> >>>>> what it does in our tests. This is how all the SNES tests that I >>> >> >>>>> use >>> >> >>>>> work. I >>> >> >>>>> am sure its a Fortran >>> >> >>>>> problem. >>> >> >>>>> >>> >> >>>>> Thanks, >>> >> >>>>> >>> >> >>>>> Matt >>> >> >>>>> >>> >> >>>>>> >>> >> >>>>>> Hom Nath >>> >> >>>>>> >>> >> >>>>>> On Tue, Feb 2, 2016 at 3:42 PM, Matthew Knepley >>> >> >>>>>> >>> >> >>>>>> wrote: >>> >> >>>>>>> On Tue, Feb 2, 2016 at 2:35 PM, Hom Nath Gharti >>> >> >>>>>>> >>> >> >>>>>>> wrote: >>> >> >>>>>>>> >>> >> >>>>>>>> Thank you so much Matt. >>> >> >>>>>>>> >>> >> >>>>>>>> I now tried the following: >>> >> >>>>>>>> >>> >> >>>>>>>> ====================================================== >>> >> >>>>>>>> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >>> >> >>>>>>>> call KSPGetPC(ksp,pc,ierr) >>> >> >>>>>>>> >>> >> >>>>>>>> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); >>> >> >>>>>>>> call ISDestroy(gdofu_is,ierr) >>> >> >>>>>>>> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); >>> >> >>>>>>>> call ISDestroy(gdofchi_is,ierr) >>> >> >>>>>>>> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); >>> >> >>>>>>>> call ISDestroy(gdofp_is,ierr) >>> >> >>>>>>>> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); >>> >> >>>>>>>> call ISDestroy(gdofphi_is,ierr) >>> >> >>>>>>>> >>> >> >>>>>>>> ! Amat changes in each time steps >>> >> >>>>>>>> call KSPSetOperators(ksp,Amat,Amat,ierr) !version >= 3.5.0 >>> >> >>>>>>>> >>> >> >>>>>>>> call KSPSolve(ksp,bvec,xvec,ierr) >>> >> >>>>>>>> ====================================================== >>> >> >>>>>>> >>> >> >>>>>>> >>> >> >>>>>>> I am guessing that "0" is not a valid string for your Fortran >>> >> >>>>>>> compiler. >>> >> >>>>>>> Are >>> >> >>>>>>> you sure >>> >> >>>>>>> its not single quotes '0'? >>> >> >>>>>>> >>> >> >>>>>>> Matt >>> >> >>>>>>> >>> >> >>>>>>>> >>> >> >>>>>>>> But I get the following error: >>> >> >>>>>>>> >>> >> >>>>>>>> [0]PETSC ERROR: --------------------- Error Message >>> >> >>>>>>>> -------------------------------------------------------------- >>> >> >>>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>> >> >>>>>>>> [0]PETSC ERROR: Unhandled case, must have at least two fields, >>> >> >>>>>>>> not 1 >>> >> >>>>>>>> [0]PETSC ERROR: See >>> >> >>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for >>> >> >>>>>>>> trouble >>> >> >>>>>>>> shooting. >>> >> >>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>> >> >>>>>>>> [0]PETSC ERROR: >>> >> >>>>>>>> /tigress/hgharti/gitwork/SPECFEM3D_GLOBEVSPP/./bin/xspecfem3D >>> >> >>>>>>>> on >>> >> >>>>>>>> a >>> >> >>>>>>>> arch-linux2-c-debug named tiger-r3c1n7 by hgharti Tue Feb 2 >>> >> >>>>>>>> 15: >>> >> >>>>>>>> 29:30 2016 >>> >> >>>>>>>> [0]PETSC ERROR: Configure options >>> >> >>>>>>>> >>> >> >>>>>>>> >>> >> >>>>>>>> >>> >> >>>>>>>> >>> >> >>>>>>>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64/ >>> >> >>>>>>>> --with-cc=mpicc --with-cxx=mpicxx --wit >>> >> >>>>>>>> h-fc=mpif90 --with-mpiexec=mpiexec --with-debugging=1 >>> >> >>>>>>>> --download-scalapack --download-mumps --download-pastix >>> >> >>>>>>>> --download-superlu --download-superlu_dist --download-metis >>> >> >>>>>>>> --download-parmetis --download-ptscotch --download-hypre >>> >> >>>>>>>> [0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 469 in >>> >> >>>>>>>> >>> >> >>>>>>>> >>> >> >>>>>>>> >>> >> >>>>>>>> >>> >> >>>>>>>> >>> >> >>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >>> >> >>>>>>>> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 486 in >>> >> >>>>>>>> >>> >> >>>>>>>> >>> >> >>>>>>>> >>> >> >>>>>>>> >>> >> >>>>>>>> >>> >> >>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >>> >> >>>>>>>> [0]PETSC ERROR: #3 PCSetUp() line 983 in >>> >> >>>>>>>> >>> >> >>>>>>>> >>> >> >>>>>>>> >>> >> >>>>>>>> >>> >> >>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/interface/precon.c >>> >> >>>>>>>> [0]PETSC ERROR: #4 KSPSetUp() line 332 in >>> >> >>>>>>>> >>> >> >>>>>>>> >>> >> >>>>>>>> >>> >> >>>>>>>> >>> >> >>>>>>>> >>> >> >>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >>> >> >>>>>>>> [0]PETSC ERROR: #5 KSPSolve() line 546 in >>> >> >>>>>>>> >>> >> >>>>>>>> >>> >> >>>>>>>> >>> >> >>>>>>>> >>> >> >>>>>>>> >>> >> >>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >>> >> >>>>>>>> forrtl: error (76): Abort trap signal >>> >> >>>>>>>> >>> >> >>>>>>>> Am I missing something? >>> >> >>>>>>>> >>> >> >>>>>>>> Thanks, >>> >> >>>>>>>> Hom Nath >>> >> >>>>>>>> >>> >> >>>>>>>> On Tue, Feb 2, 2016 at 3:24 PM, Matthew Knepley >>> >> >>>>>>>> >>> >> >>>>>>>> wrote: >>> >> >>>>>>>>> On Tue, Feb 2, 2016 at 2:20 PM, Hom Nath Gharti >>> >> >>>>>>>>> >>> >> >>>>>>>>> wrote: >>> >> >>>>>>>>>> >>> >> >>>>>>>>>> Hi Matt, hi all, >>> >> >>>>>>>>>> >>> >> >>>>>>>>>> I am trying to use PCFIELDSPLIT for my solver which consists >>> >> >>>>>>>>>> of >>> >> >>>>>>>>>> 4 >>> >> >>>>>>>>>> different variables, namely, u (displacement vector), \chi >>> >> >>>>>>>>>> (displacement potential), p(pressure), and \phi (gravity >>> >> >>>>>>>>>> potential). >>> >> >>>>>>>>>> >>> >> >>>>>>>>>> My code segment looks like the following: >>> >> >>>>>>>>>> ====================================================== >>> >> >>>>>>>>>> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >>> >> >>>>>>>>>> call KSPGetPC(ksp,pc,ierr) >>> >> >>>>>>>>>> nsplit=4 >>> >> >>>>>>>>>> call PCFieldSplitSetBlockSize(pc,nsplit,ierr); >>> >> >>>>>>>>> >>> >> >>>>>>>>> >>> >> >>>>>>>>> You do not need the statement above. >>> >> >>>>>>>>> >>> >> >>>>>>>>>> >>> >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); >>> >> >>>>>>>>>> call ISDestroy(gdofu_is,ierr) >>> >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); >>> >> >>>>>>>>>> call ISDestroy(gdofchi_is,ierr) >>> >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); >>> >> >>>>>>>>>> call ISDestroy(gdofp_is,ierr) >>> >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); >>> >> >>>>>>>>>> call ISDestroy(gdofphi_is,ierr) >>> >> >>>>>>>>>> >>> >> >>>>>>>>>> call PCFieldSplitGetSubKSP(pc,nsplit,subksp,ierr); >>> >> >>>>>>>>> >>> >> >>>>>>>>> >>> >> >>>>>>>>> These SetOperators() calls are wrong. I am not sure why you >>> >> >>>>>>>>> want >>> >> >>>>>>>>> them >>> >> >>>>>>>>> here. >>> >> >>>>>>>>> Also, that means you do not need the call above. >>> >> >>>>>>>>> >>> >> >>>>>>>>> Thanks, >>> >> >>>>>>>>> >>> >> >>>>>>>>> Matt >>> >> >>>>>>>>> >>> >> >>>>>>>>>> >>> >> >>>>>>>>>> call KSPSetOperators(subksp(1),Amat,Amat,ierr); >>> >> >>>>>>>>>> call KSPSetOperators(subksp(2),Amat,Amat,ierr); >>> >> >>>>>>>>>> call KSPSetOperators(subksp(3),Amat,Amat,ierr); >>> >> >>>>>>>>>> call KSPSetOperators(subksp(4),Amat,Amat,ierr); >>> >> >>>>>>>>>> >>> >> >>>>>>>>>> call KSPSolve(ksp,bvec,xvec,ierr) >>> >> >>>>>>>>>> ====================================================== >>> >> >>>>>>>>>> >>> >> >>>>>>>>>> But I am getting the following error: >>> >> >>>>>>>>>> [79]PETSC ERROR: Null argument, when expecting valid pointer >>> >> >>>>>>>>>> [79]PETSC ERROR: Null Object: Parameter # 1 >>> >> >>>>>>>>>> [79]PETSC ERROR: #1 KSPSetOperators() line 536 in >>> >> >>>>>>>>>> >>> >> >>>>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interf >>> >> >>>>>>>>>> >>> >> >>>>>>>>>> It looks like I am doing something wrong in "call >>> >> >>>>>>>>>> KSPSetOperators" >>> >> >>>>>>>>>> but >>> >> >>>>>>>>>> I could not figure that out. >>> >> >>>>>>>>>> >>> >> >>>>>>>>>> Could anybody help me to fix this problem? I looked into >>> >> >>>>>>>>>> almost >>> >> >>>>>>>>>> all >>> >> >>>>>>>>>> related examples but I could not really figure out the >>> >> >>>>>>>>>> correct >>> >> >>>>>>>>>> steps >>> >> >>>>>>>>>> after "call PCFieldSplitSetIS". >>> >> >>>>>>>>>> >>> >> >>>>>>>>>> Any help will be greatly appreciated. >>> >> >>>>>>>>>> >>> >> >>>>>>>>>> Best, >>> >> >>>>>>>>>> Hom nath >>> >> >>>>>>>>>> >>> >> >>>>>>>>>> On Sun, Jan 24, 2016 at 7:14 PM, Hom Nath Gharti >>> >> >>>>>>>>>> >>> >> >>>>>>>>>> wrote: >>> >> >>>>>>>>>>> Thank you so much Matt! I will try. >>> >> >>>>>>>>>>> >>> >> >>>>>>>>>>> Hom Nath >>> >> >>>>>>>>>>> >>> >> >>>>>>>>>>> On Sun, Jan 24, 2016 at 6:26 AM, Matthew Knepley >>> >> >>>>>>>>>>> >>> >> >>>>>>>>>>> wrote: >>> >> >>>>>>>>>>>> On Fri, Jan 22, 2016 at 2:19 PM, Hom Nath Gharti >>> >> >>>>>>>>>>>> >>> >> >>>>>>>>>>>> wrote: >>> >> >>>>>>>>>>>>> >>> >> >>>>>>>>>>>>> Dear all, >>> >> >>>>>>>>>>>>> >>> >> >>>>>>>>>>>>> I am new to PcFieldSplit. >>> >> >>>>>>>>>>>>> >>> >> >>>>>>>>>>>>> I have a matrix formed using MATMPIAIJ. Is it possible to >>> >> >>>>>>>>>>>>> use >>> >> >>>>>>>>>>>>> PCFIELDSPLIT operations in this type of matrix? Or does >>> >> >>>>>>>>>>>>> it >>> >> >>>>>>>>>>>>> have >>> >> >>>>>>>>>>>>> to >>> >> >>>>>>>>>>>>> be >>> >> >>>>>>>>>>>>> MATMPIBIJ or MATNEST format? >>> >> >>>>>>>>>>>> >>> >> >>>>>>>>>>>> >>> >> >>>>>>>>>>>> Yes, you can split AIJ. >>> >> >>>>>>>>>>>> >>> >> >>>>>>>>>>>>> >>> >> >>>>>>>>>>>>> If possible for MATMPIAIJ, could anybody provide me a >>> >> >>>>>>>>>>>>> simple >>> >> >>>>>>>>>>>>> example >>> >> >>>>>>>>>>>>> or few steps? Variables in the equations are displacement >>> >> >>>>>>>>>>>>> vector, >>> >> >>>>>>>>>>>>> scalar potential and pressure. >>> >> >>>>>>>>>>>> >>> >> >>>>>>>>>>>> >>> >> >>>>>>>>>>>> If you do not have a collocated discretization, then you >>> >> >>>>>>>>>>>> have >>> >> >>>>>>>>>>>> to >>> >> >>>>>>>>>>>> use >>> >> >>>>>>>>>>>> >>> >> >>>>>>>>>>>> >>> >> >>>>>>>>>>>> >>> >> >>>>>>>>>>>> >>> >> >>>>>>>>>>>> >>> >> >>>>>>>>>>>> >>> >> >>>>>>>>>>>> >>> >> >>>>>>>>>>>> >>> >> >>>>>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html >>> >> >>>>>>>>>>>> >>> >> >>>>>>>>>>>> Thanks, >>> >> >>>>>>>>>>>> >>> >> >>>>>>>>>>>> Matt >>> >> >>>>>>>>>>>> >>> >> >>>>>>>>>>>>> >>> >> >>>>>>>>>>>>> Thanks for help. >>> >> >>>>>>>>>>>>> >>> >> >>>>>>>>>>>>> Hom Nath >>> >> >>>>>>>>>>>> >>> >> >>>>>>>>>>>> >>> >> >>>>>>>>>>>> >>> >> >>>>>>>>>>>> >>> >> >>>>>>>>>>>> -- >>> >> >>>>>>>>>>>> What most experimenters take for granted before they begin >>> >> >>>>>>>>>>>> their >>> >> >>>>>>>>>>>> experiments >>> >> >>>>>>>>>>>> is infinitely more interesting than any results to which >>> >> >>>>>>>>>>>> their >>> >> >>>>>>>>>>>> experiments >>> >> >>>>>>>>>>>> lead. >>> >> >>>>>>>>>>>> -- Norbert Wiener >>> >> >>>>>>>>> >>> >> >>>>>>>>> >>> >> >>>>>>>>> >>> >> >>>>>>>>> >>> >> >>>>>>>>> -- >>> >> >>>>>>>>> What most experimenters take for granted before they begin >>> >> >>>>>>>>> their >>> >> >>>>>>>>> experiments >>> >> >>>>>>>>> is infinitely more interesting than any results to which >>> >> >>>>>>>>> their >>> >> >>>>>>>>> experiments >>> >> >>>>>>>>> lead. >>> >> >>>>>>>>> -- Norbert Wiener >>> >> >>>>>>> >>> >> >>>>>>> >>> >> >>>>>>> >>> >> >>>>>>> >>> >> >>>>>>> -- >>> >> >>>>>>> What most experimenters take for granted before they begin >>> >> >>>>>>> their >>> >> >>>>>>> experiments >>> >> >>>>>>> is infinitely more interesting than any results to which their >>> >> >>>>>>> experiments >>> >> >>>>>>> lead. >>> >> >>>>>>> -- Norbert Wiener >>> >> >>>>> >>> >> >>>>> >>> >> >>>>> >>> >> >>>>> >>> >> >>>>> -- >>> >> >>>>> What most experimenters take for granted before they begin their >>> >> >>>>> experiments >>> >> >>>>> is infinitely more interesting than any results to which their >>> >> >>>>> experiments >>> >> >>>>> lead. >>> >> >>>>> -- Norbert Wiener >>> >> >>> >>> >> >>> >>> >> >>> >>> >> >>> >>> >> >>> -- >>> >> >>> What most experimenters take for granted before they begin their >>> >> >>> experiments >>> >> >>> is infinitely more interesting than any results to which their >>> >> >>> experiments >>> >> >>> lead. >>> >> >>> -- Norbert Wiener >>> >> > >>> > >>> > >>> > >>> > >>> > -- >>> > What most experimenters take for granted before they begin their >>> > experiments >>> > is infinitely more interesting than any results to which their >>> > experiments >>> > lead. >>> > -- Norbert Wiener >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments >> is infinitely more interesting than any results to which their experiments >> lead. >> -- Norbert Wiener From knepley at gmail.com Thu Feb 11 12:34:20 2016 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 11 Feb 2016 12:34:20 -0600 Subject: [petsc-users] PCFIELDSPLIT question In-Reply-To: References: Message-ID: On Thu, Feb 11, 2016 at 11:47 AM, Hom Nath Gharti wrote: > Thanks all. With further help from Stefano Zhampini I was able to > solve my small test case using PCFieldSPlit option. I had to add > call PCSetType(pc,PCFIELDSPLIT,ierr) > > Now I am trying to use same technique in my actual problem and got the > error: > > PETSC ERROR: Fields must be sorted when creating split > > Does this mean I have to sort the global matrix rows according the split > fields? > You have to sort the IS you pass to PCFieldSplitSetIS() which can be done using ISSort() Thanks, Matt > Thanks > Hom Nath > > > > On Tue, Feb 9, 2016 at 10:57 AM, Hom Nath Gharti > wrote: > > Sounds interesting! Thanks a lot Matt! I will have a look. > > > > Hom Nath > > > > On Tue, Feb 9, 2016 at 10:36 AM, Matthew Knepley > wrote: > >> On Tue, Feb 9, 2016 at 9:31 AM, Hom Nath Gharti > wrote: > >>> > >>> Thanks a lot Matt! > >>> > >>> Were you referring to > >>> > >>> > http://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex62.c.html > >>> ? > >>> > >>> I do not see any statements related to PCFieldSplit there. Am I > >>> missing something here? > >> > >> > >> Yes. The whole point is not to put anything specific to FIELDSPLIT in > the > >> code. It is all in options. For example, > >> > >> ./ex62 -run_type full -refinement_limit 0.00625 -bc_type dirichlet > >> -interpolate 1 -vel_petscspace_order 2 -pres_petscspace_order 1 > -ksp_type > >> fgmres -ksp_gmres_restart 100 -ksp_rtol 1.0e-9 -pc_type fieldsplit > >> -pc_fieldsplit_type schur -pc_fieldsplit_schur_factorization_type upper > >> -fieldsplit_pressure_ksp_rtol 1e-10 -fieldsplit_velocity_ksp_type gmres > >> -fieldsplit_velocity_pc_type lu -fieldsplit_pressure_pc_type jacobi > >> -snes_monitor_short -ksp_monitor_short -snes_converged_reason > >> -ksp_converged_reason -snes_view -show_solution 0 > >> > >> The reason that it works is that the DM calls PCFIeldSplitSetIS() for > each > >> field in the DM. > >> > >> Thanks, > >> > >> Matt > >> > >>> > >>> Thanks, > >>> Hom Nath > >>> > >>> On Tue, Feb 9, 2016 at 10:19 AM, Matthew Knepley > >>> wrote: > >>> > On Tue, Feb 9, 2016 at 9:10 AM, Hom Nath Gharti > > >>> > wrote: > >>> >> > >>> >> Thank you so much Barry! > >>> >> > >>> >> For my small test case, with -pc_fieldsplit_block_size 4, the > program > >>> >> runs, although the answer was not correct. At least now I get > >>> >> something to look upon. I am using PCFieldSplitSetIS to set the > >>> >> fields. Do I still need to use -pc_fieldsplit_block_size? > >>> > > >>> > > >>> > No, this is only for co-located discretizations. > >>> > > >>> >> > >>> >> In my case each grid point may have different variable sets. For > >>> >> example, the grid point in the solid region has displacement and > >>> >> gravity potential: ux, uy, uz, and \phi. Whereas the grid point in > the > >>> >> fluid region has scalar potential, pressure and gravity potential: > >>> >> \chi, p, and \phi. And the solid-fluid interface has all of them. Do > >>> >> you think I can still use PCFIELDSPLIT in this situation? > >>> > > >>> > > >>> > We have examples, like SNES ex62, which do exactly this. > >>> > > >>> > Thanks, > >>> > > >>> > Matt > >>> > > >>> >> > >>> >> Best, > >>> >> Hom Nath > >>> >> > >>> >> > >>> >> > >>> >> On Mon, Feb 8, 2016 at 2:27 PM, Barry Smith > wrote: > >>> >> > > >>> >> > > >>> >> > In this case you need to provide two pieces of information to > the > >>> >> > PCFIELDSPLIT. What we call the "block size" or bs which is the > number > >>> >> > of > >>> >> > "basic fields" in the problem. For example if at each grid point > you > >>> >> > have > >>> >> > x-velocity, y-velocity, and pressure the block size is 3. The > second > >>> >> > is you > >>> >> > need to tell PCFIELDSPLIT what "basic fields" are in each split > you > >>> >> > want to > >>> >> > use. > >>> >> > > >>> >> > In your case you can do this with -pc_fieldsplit_block_size 4. > >>> >> > (Note if you use a DM to define your problem the PCFIELDSPLIT will > >>> >> > automatically get the bs from the DM so you do not need to provide > >>> >> > it. > >>> >> > Similarly if you set a block size on your Mat the PCFIELDSPLIT > will > >>> >> > use that > >>> >> > so you don't need to set it). > >>> >> > > >>> >> > Then use -pc_fieldsplit_0_fields 0,1 to indicate your first > split > >>> >> > is > >>> >> > the 0 and 1 basic fields and -pc_fieldsplit_1_fields 2,3 to > indicate > >>> >> > the > >>> >> > second split is the 2 and 3 basic fields. > >>> >> > (By default if you do not provide this then PCFIELDSPLIT will use > bs > >>> >> > splits (4 in your case) one for each basic field. > >>> >> > > >>> >> > Barry > >>> >> > > >>> >> >> On Feb 8, 2016, at 11:20 AM, Hom Nath Gharti < > hng.email at gmail.com> > >>> >> >> wrote: > >>> >> >> > >>> >> >> Hi Matt, Hi all, > >>> >> >> > >>> >> >> I am trying to have some feel for PCFIELDSPLIT testing on a very > >>> >> >> small > >>> >> >> matrix (attached below). I have 4 degrees of freedoms. I use 4 > >>> >> >> processors. I want to split 4 dofs into two fields each having > two > >>> >> >> dofs. I don't know whether this my be a problem for petsc. When I > >>> >> >> use > >>> >> >> the command: > >>> >> >> srun -n 4 ./xtestfs -ksp_monitor -ksp_view > >>> >> >> > >>> >> >> It runs fine. > >>> >> >> > >>> >> >> But when I want to use field split options using the command: > >>> >> >> srun -n 4 ./xtestfs -ksp_monitor -ksp_type fgmres -pc_type > >>> >> >> fieldsplit > >>> >> >> -fieldsplit_0_ksp_type gmres -fieldsplit_0_pc_type bjacobi > >>> >> >> -fieldsplit_1_pc_type jacobi > >>> >> >> > >>> >> >> I get the following error message: > >>> >> >> [0]PETSC ERROR: Petsc has generated inconsistent data > >>> >> >> [0]PETSC ERROR: Unhandled case, must have at least two fields, > not 1 > >>> >> >> [0]PETSC ERROR: See > >>> >> >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > >>> >> >> shooting. > >>> >> >> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > >>> >> >> [0]PETSC ERROR: /tigress/hgharti/test/testfs/./xtestfs on a > >>> >> >> arch-linux2-c-debug named tiger-r11n1 by hgharti Mon Feb 8 > 11:40:03 > >>> >> >> 2016 > >>> >> >> [0]PETSC ERROR: Configure options > >>> >> >> > >>> >> >> > >>> >> >> > --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64/ > >>> >> >> --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > >>> >> >> --with-mpiexec=mpiexec --with-debugging=1 --download-scalapack > >>> >> >> --download-mumps --download-pastix --download-superlu > >>> >> >> --download-superlu_dist --download-metis --download-parmetis > >>> >> >> --download-ptscotch --download-hypre > >>> >> >> [0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 469 in > >>> >> >> > >>> >> >> > >>> >> >> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > >>> >> >> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 486 in > >>> >> >> > >>> >> >> > >>> >> >> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > >>> >> >> [0]PETSC ERROR: #3 PCSetUp() line 983 in > >>> >> >> > >>> >> >> > >>> >> >> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/interface/precon.c > >>> >> >> [0]PETSC ERROR: #4 KSPSetUp() line 332 in > >>> >> >> > >>> >> >> > >>> >> >> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c > >>> >> >> [0]PETSC ERROR: #5 KSPSolve() line 546 in > >>> >> >> > >>> >> >> > >>> >> >> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c > >>> >> >> WARNING! There are options you set that were not used! > >>> >> >> WARNING! could be spelling mistake, etc! > >>> >> >> Option left: name:-fieldsplit_0_ksp_type value: gmres > >>> >> >> Option left: name:-fieldsplit_0_pc_type value: bjacobi > >>> >> >> Option left: name:-fieldsplit_1_pc_type value: jacobi > >>> >> >> forrtl: error (76): Abort trap signal > >>> >> >> > >>> >> >> I tried several trials but could not fix the problem. Is it the > >>> >> >> FORTRAN problem or am I doing something wrong? Any suggestions > would > >>> >> >> be greatly appreciated. > >>> >> >> For your information I use: > >>> >> >> PETSc-3.6.3 > >>> >> >> intel-16.0 compiler > >>> >> >> intel-mpi-5.1.1 > >>> >> >> > >>> >> >> Program > >>> >> >> > >>> >> >> > >>> >> >> Best, > >>> >> >> Hom Nath > >>> >> >> > >>> >> >> ! simple test case for PCFIELDSPLIT > >>> >> >> ! > >>> >> >> > >>> >> >> > ----------------------------------------------------------------------- > >>> >> >> program testfs > >>> >> >> implicit none > >>> >> >> #include "petsc/finclude/petscsys.h" > >>> >> >> #include "petsc/finclude/petscvec.h" > >>> >> >> #include "petsc/finclude/petscvec.h90" > >>> >> >> #include "petsc/finclude/petscmat.h" > >>> >> >> #include "petsc/finclude/petscksp.h" > >>> >> >> #include "petsc/finclude/petscpc.h" > >>> >> >> #include "petsc/finclude/petscviewer.h" > >>> >> >> #include "petsc/finclude/petscviewer.h90" > >>> >> >> > >>> >> >> Vec x,b,u > >>> >> >> Mat A > >>> >> >> KSP ksp > >>> >> >> PC pc > >>> >> >> PetscErrorCode ierr > >>> >> >> PetscBool flg > >>> >> >> PetscInt errcode,its,maxitr,myrank,nproc > >>> >> >> PetscInt nedof,nelmt,ndof,nenod,ngdof,fixdof,nsparse,neq > >>> >> >> PetscInt,allocatable :: krow_sparse(:),ggdof_elmt(:,:), > & > >>> >> >> inode_elmt(:,:) > >>> >> >> > >>> >> >> PetscReal,allocatable :: storef(:,:),storekmat(:,:,:) > >>> >> >> PetscInt gdof0(2),gdof1(2) > >>> >> >> > >>> >> >> ! initialize MPI > >>> >> >> > >>> >> >> call MPI_INIT(errcode) > >>> >> >> if(errcode /= 0)stop 'ERROR: cannot initialize MPI!' > >>> >> >> call MPI_COMM_RANK(MPI_COMM_WORLD,myrank,errcode) > >>> >> >> if(errcode /= 0)stop 'ERROR: cannot find processor ID' > >>> >> >> call MPI_COMM_SIZE(MPI_COMM_WORLD,nproc,errcode) > >>> >> >> if(errcode /= 0)stop 'ERROR: cannot find number of processors!' > >>> >> >> > >>> >> >> ! define some parameters > >>> >> >> ! 1D model with 4 elements. Each elements consits of 2 nodes. > Node 0 > >>> >> >> is > >>> >> >> fixed. > >>> >> >> ! 1-----------2-----------3-----------4-----------5 > >>> >> >> nelmt=1 ! per processor > >>> >> >> nenod=2 ! number of nodes per element > >>> >> >> nedof=2 ! number of DOFs per element > >>> >> >> ndof=2 ! number of DOFs > >>> >> >> ngdof=4 ! total number of global DOFs > >>> >> >> fixdof=1 > >>> >> >> > >>> >> >> if(myrank==0)then > >>> >> >> neq=1 ! number of equations > >>> >> >> nsparse=1 > >>> >> >> else > >>> >> >> neq=2 ! number of equations > >>> >> >> nsparse=4 > >>> >> >> endif > >>> >> >> > >>> >> >> allocate(storef(nedof,nelmt),storekmat(nedof,nedof,nelmt), > >>> >> >> & > >>> >> >> > >>> >> >> > ggdof_elmt(nedof,nelmt),inode_elmt(nenod,nelmt),krow_sparse(nsparse)) > >>> >> >> > >>> >> >> storef=0.0_8 ! local RHS vector > >>> >> >> storekmat=0.0_8 ! local stiffness matrix > >>> >> >> if(myrank==0)then > >>> >> >> krow_sparse(:)=1 > >>> >> >> storef(:,1)=(/12.5000_8, 12.5000_8/) > >>> >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & > >>> >> >> 0.0166666667_8, > >>> >> >> 1.2333333332_8/),(/nedof,nedof/)) > >>> >> >> inode_elmt(:,1) = (/1,2/) > >>> >> >> ggdof_elmt(:,1) = (/0,1/) ! 0 because left node is fixed > >>> >> >> elseif(myrank==1)then > >>> >> >> krow_sparse(:)=(/1,1,2,2/) > >>> >> >> storef(:,1)=(/12.5000_8, 12.5000_8/) > >>> >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & > >>> >> >> 0.0166666667_8, > >>> >> >> 1.2333333332_8/),(/nedof,nedof/)) > >>> >> >> inode_elmt(:,1) = (/1,2/) > >>> >> >> ggdof_elmt(:,1) = (/1,2/) > >>> >> >> elseif(myrank==2)then > >>> >> >> krow_sparse(:)=(/1,1,2,2/) > >>> >> >> storef(:,1)=(/12.5000_8, 12.5000_8/) > >>> >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & > >>> >> >> 0.0166666667_8, > >>> >> >> 1.2333333332_8/),(/nedof,nedof/)) > >>> >> >> inode_elmt(:,1) = (/1,2/) > >>> >> >> ggdof_elmt(:,1) = (/2,3/) > >>> >> >> elseif(myrank==3)then > >>> >> >> krow_sparse(:)=(/1,1,2,2/) > >>> >> >> storef(:,1)=(/12.5000_8, 22.5000_8/) > >>> >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & > >>> >> >> 0.0166666667_8, > >>> >> >> 1.2333333332_8/),(/nedof,nedof/)) > >>> >> >> inode_elmt(:,1) = (/1,2/) > >>> >> >> ggdof_elmt(:,1) = (/3,4/) > >>> >> >> endif > >>> >> >> > >>> >> >> call petsc_initialize > >>> >> >> call petsc_set_matrix > >>> >> >> call petsc_set_vector > >>> >> >> call petsc_solve > >>> >> >> call petsc_finalize > >>> >> >> > >>> >> >> ! deallocate > >>> >> >> deallocate(storef,storekmat,ggdof_elmt,inode_elmt) > >>> >> >> deallocate(krow_sparse) > >>> >> >> contains > >>> >> >> > >>> >> >> > >>> >> >> > !------------------------------------------------------------------------------- > >>> >> >> subroutine petsc_initialize() > >>> >> >> implicit none > >>> >> >> PetscInt :: istart,iend > >>> >> >> PetscInt :: nzeros_max,nzeros_min > >>> >> >> PetscReal :: atol,dtol,rtol > >>> >> >> PetscInt,allocatable :: nzeros(:) > >>> >> >> IS gdof0_is,gdof1_is > >>> >> >> > >>> >> >> call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > >>> >> >> call PetscOptionsGetInt(PETSC_NULL_CHARACTER,'-n',ngdof,flg,ierr) > >>> >> >> > >>> >> >> ! count number of nonzeros per row > >>> >> >> allocate(nzeros(neq)) > >>> >> >> nzeros=0 > >>> >> >> nzeros(krow_sparse)=nzeros(krow_sparse)+1 > >>> >> >> nzeros_max=maxval(nzeros) > >>> >> >> nzeros_min=minval(nzeros) > >>> >> >> > >>> >> >> ! create matrix > >>> >> >> call MatCreate(PETSC_COMM_WORLD,A,ierr) > >>> >> >> call MatSetType(A,MATMPIAIJ,ierr) > >>> >> >> CHKERRQ(ierr) > >>> >> >> call MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,ngdof,ngdof,ierr) > >>> >> >> CHKERRQ(ierr) > >>> >> >> ! preallocation > >>> >> >> call > >>> >> >> > >>> >> >> > MatMPIAIJSetPreallocation(A,nzeros_max,PETSC_NULL_INTEGER,nzeros_max, & > >>> >> >> PETSC_NULL_INTEGER,ierr) > >>> >> >> CHKERRQ(ierr) > >>> >> >> > >>> >> >> call MatGetOwnershipRange(A,istart,iend,ierr) > >>> >> >> CHKERRQ(ierr) > >>> >> >> print*,'ownership:',myrank,istart,iend > >>> >> >> ! create vector > >>> >> >> call VecCreate(PETSC_COMM_WORLD,x,ierr) > >>> >> >> call VecSetSizes(x,PETSC_DECIDE,ngdof,ierr) > >>> >> >> call VecSetType(x,VECMPI,ierr) > >>> >> >> call VecDuplicate(x,b,ierr) > >>> >> >> call VecDuplicate(x,u,ierr) > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> > !******************************************************************************* > >>> >> >> ! get IS for split fields > >>> >> >> gdof0=(/0,1/) > >>> >> >> gdof1=(/2,3/) > >>> >> >> call > >>> >> >> > >>> >> >> > ISCreateGeneral(PETSC_COMM_WORLD,2,gdof0,PETSC_COPY_VALUES,gdof0_is,ierr) > >>> >> >> CHKERRQ(ierr) > >>> >> >> call > >>> >> >> > >>> >> >> > ISCreateGeneral(PETSC_COMM_WORLD,2,gdof1,PETSC_COPY_VALUES,gdof1_is,ierr) > >>> >> >> CHKERRQ(ierr) > >>> >> >> > >>> >> >> > >>> >> >> > !******************************************************************************* > >>> >> >> > >>> >> >> ! Create linear solver context > >>> >> >> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) > >>> >> >> call KSPSetOperators(ksp,A,A,ierr) > >>> >> >> > >>> >> >> call KSPGetPC(ksp,pc,ierr) > >>> >> >> CHKERRQ(ierr) > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> > !******************************************************************************* > >>> >> >> ! split PC > >>> >> >> call PCFieldSplitSetIS(pc,"0",gdof0_is,ierr); > >>> >> >> CHKERRQ(ierr) > >>> >> >> call ISDestroy(gdof0_is,ierr) > >>> >> >> CHKERRQ(ierr) > >>> >> >> call PCFieldSplitSetIS(pc,"1",gdof1_is,ierr); > >>> >> >> CHKERRQ(ierr) > >>> >> >> call ISDestroy(gdof1_is,ierr) > >>> >> >> CHKERRQ(ierr) > >>> >> >> > >>> >> >> > >>> >> >> > !******************************************************************************* > >>> >> >> > >>> >> >> rtol = 1.0d-6 > >>> >> >> atol = 1.0d-6 > >>> >> >> dtol = 1.0d10 > >>> >> >> maxitr = 1000 > >>> >> >> call KSPSetTolerances(ksp,rtol,atol,dtol,maxitr,ierr) > >>> >> >> CHKERRQ(ierr) > >>> >> >> call KSPSetFromOptions(ksp,ierr) > >>> >> >> end subroutine petsc_initialize > >>> >> >> > >>> >> >> > >>> >> >> > !------------------------------------------------------------------------------- > >>> >> >> > >>> >> >> subroutine petsc_set_matrix() > >>> >> >> > >>> >> >> implicit none > >>> >> >> integer :: i,i_elmt,j,ncount > >>> >> >> integer :: ggdof(NEDOF),idof(NEDOF),igdof(NEDOF) > >>> >> >> > >>> >> >> ! Set and assemble matrix. > >>> >> >> call MatZeroEntries(A,ierr) > >>> >> >> CHKERRQ(ierr) > >>> >> >> do i_elmt=1,nelmt > >>> >> >> ggdof(:)=ggdof_elmt(:,i_elmt) > >>> >> >> ggdof(:)=ggdof(:)-1 ! petsc index starts from 0 > >>> >> >> ncount=0; idof=-1; igdof=-1 > >>> >> >> do i=1,NEDOF > >>> >> >> do j=1,NEDOF > >>> >> >> if(ggdof(i).ge.0.and.ggdof(j).ge.0)then > >>> >> >> call > >>> >> >> MatSetValues(A,1,ggdof(i),1,ggdof(j),storekmat(i,j,i_elmt), > >>> >> >> & > >>> >> >> ADD_VALUES,ierr) > >>> >> >> CHKERRQ(ierr) > >>> >> >> endif > >>> >> >> enddo > >>> >> >> enddo > >>> >> >> enddo > >>> >> >> call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) > >>> >> >> call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) > >>> >> >> if(myrank==0)print*,'matrix setting & assembly complete!' > >>> >> >> > >>> >> >> end subroutine petsc_set_matrix > >>> >> >> > >>> >> >> > >>> >> >> > !------------------------------------------------------------------------------- > >>> >> >> > >>> >> >> subroutine petsc_set_vector() > >>> >> >> implicit none > >>> >> >> PetscScalar zero > >>> >> >> integer :: i,i_elmt,ncount > >>> >> >> integer :: ggdof(NEDOF),idof(NEDOF),igdof(NEDOF) > >>> >> >> > >>> >> >> ! set vector > >>> >> >> zero=0.0_8 > >>> >> >> call VecSet(b,zero,ierr) > >>> >> >> do i_elmt=1,nelmt > >>> >> >> ggdof(:)=ggdof_elmt(:,i_elmt) > >>> >> >> ggdof(:)=ggdof(:)-1 ! petsc index starts from 0 > >>> >> >> ncount=0; idof=-1; igdof=-1 > >>> >> >> do i=1,NEDOF > >>> >> >> if(ggdof(i).ge.0)then > >>> >> >> call > >>> >> >> VecSetValues(b,1,ggdof(i),storef(i,i_elmt),ADD_VALUES,ierr); > >>> >> >> CHKERRQ(ierr) > >>> >> >> endif > >>> >> >> enddo > >>> >> >> enddo > >>> >> >> > >>> >> >> ! assemble vector > >>> >> >> call VecAssemblyBegin(b,ierr) > >>> >> >> call VecAssemblyEnd(b,ierr) > >>> >> >> if(myrank==0)print*,'vector setting & assembly complete!' > >>> >> >> > >>> >> >> end subroutine petsc_set_vector > >>> >> >> > >>> >> >> > >>> >> >> > !------------------------------------------------------------------------------- > >>> >> >> > >>> >> >> subroutine petsc_solve() > >>> >> >> implicit none > >>> >> >> PetscInt ireason > >>> >> >> PetscScalar a_v(1) > >>> >> >> PetscOffset a_i > >>> >> >> PetscInt n > >>> >> >> PetscReal,allocatable :: sdata(:) > >>> >> >> > >>> >> >> call VecGetSize(b,n,ierr) > >>> >> >> CHKERRQ(ierr) > >>> >> >> allocate(sdata(n)) > >>> >> >> sdata=0.0_8 > >>> >> >> call VecGetArray(b,a_v,a_i,ierr) > >>> >> >> CHKERRQ(ierr) > >>> >> >> sdata(1:n)=a_v(a_i+1:a_i+n) > >>> >> >> call VecRestoreArray(b,a_v,a_i,ierr) > >>> >> >> CHKERRQ(ierr) > >>> >> >> print*,'rhs:',sdata > >>> >> >> > >>> >> >> call KSPSolve(ksp,b,x,ierr) > >>> >> >> sdata=0.0_8 > >>> >> >> call VecGetArray(x,a_v,a_i,ierr) > >>> >> >> CHKERRQ(ierr) > >>> >> >> sdata(1:n)=a_v(a_i+1:a_i+n) > >>> >> >> call VecRestoreArray(b,a_v,a_i,ierr) > >>> >> >> CHKERRQ(ierr) > >>> >> >> print*,'solution:',sdata > >>> >> >> > >>> >> >> ! Check convergence > >>> >> >> call KSPGetConvergedReason(ksp,ireason,ierr) > >>> >> >> if(myrank==0)print*,'converges reason',myrank,ireason > >>> >> >> call KSPGetIterationNumber(ksp,its,ierr) > >>> >> >> if(myrank==0)print*,'Iterations:',its > >>> >> >> deallocate(sdata) > >>> >> >> end subroutine petsc_solve > >>> >> >> > >>> >> >> > >>> >> >> > !------------------------------------------------------------------------------- > >>> >> >> > >>> >> >> subroutine petsc_finalize() > >>> >> >> implicit none > >>> >> >> > >>> >> >> ! Free work space. > >>> >> >> call VecDestroy(x,ierr) > >>> >> >> call VecDestroy(u,ierr) > >>> >> >> call VecDestroy(b,ierr) > >>> >> >> call MatDestroy(A,ierr) > >>> >> >> call KSPDestroy(ksp,ierr) > >>> >> >> call PetscFinalize(ierr) > >>> >> >> CHKERRQ(ierr) > >>> >> >> > >>> >> >> end subroutine petsc_finalize > >>> >> >> > >>> >> >> > >>> >> >> > !------------------------------------------------------------------------------- > >>> >> >> > >>> >> >> end program testfs > >>> >> >> > >>> >> >> On Tue, Feb 2, 2016 at 4:54 PM, Matthew Knepley < > knepley at gmail.com> > >>> >> >> wrote: > >>> >> >>> On Tue, Feb 2, 2016 at 3:11 PM, Hom Nath Gharti > >>> >> >>> > >>> >> >>> wrote: > >>> >> >>>> > >>> >> >>>> Thanks a lot. I will try. > >>> >> >>> > >>> >> >>> > >>> >> >>> Also, if you send a small test case, we can run it and tell you > the > >>> >> >>> problem. > >>> >> >>> > >>> >> >>> Matt > >>> >> >>> > >>> >> >>>> > >>> >> >>>> Hom Nath > >>> >> >>>> > >>> >> >>>> On Tue, Feb 2, 2016 at 4:02 PM, Matthew Knepley > >>> >> >>>> > >>> >> >>>> wrote: > >>> >> >>>>> On Tue, Feb 2, 2016 at 3:01 PM, Hom Nath Gharti > >>> >> >>>>> > >>> >> >>>>> wrote: > >>> >> >>>>>> > >>> >> >>>>>> Thanks again Matt. Unfortunately got the same errors with > '0'. I > >>> >> >>>>>> think > >>> >> >>>>>> both are valid in Fortran. > >>> >> >>>>> > >>> >> >>>>> > >>> >> >>>>> Then you will have to go in the debugger and see why its not > >>> >> >>>>> creating 4 > >>> >> >>>>> splits, since this is exactly > >>> >> >>>>> what it does in our tests. This is how all the SNES tests > that I > >>> >> >>>>> use > >>> >> >>>>> work. I > >>> >> >>>>> am sure its a Fortran > >>> >> >>>>> problem. > >>> >> >>>>> > >>> >> >>>>> Thanks, > >>> >> >>>>> > >>> >> >>>>> Matt > >>> >> >>>>> > >>> >> >>>>>> > >>> >> >>>>>> Hom Nath > >>> >> >>>>>> > >>> >> >>>>>> On Tue, Feb 2, 2016 at 3:42 PM, Matthew Knepley > >>> >> >>>>>> > >>> >> >>>>>> wrote: > >>> >> >>>>>>> On Tue, Feb 2, 2016 at 2:35 PM, Hom Nath Gharti > >>> >> >>>>>>> > >>> >> >>>>>>> wrote: > >>> >> >>>>>>>> > >>> >> >>>>>>>> Thank you so much Matt. > >>> >> >>>>>>>> > >>> >> >>>>>>>> I now tried the following: > >>> >> >>>>>>>> > >>> >> >>>>>>>> ====================================================== > >>> >> >>>>>>>> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) > >>> >> >>>>>>>> call KSPGetPC(ksp,pc,ierr) > >>> >> >>>>>>>> > >>> >> >>>>>>>> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); > >>> >> >>>>>>>> call ISDestroy(gdofu_is,ierr) > >>> >> >>>>>>>> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); > >>> >> >>>>>>>> call ISDestroy(gdofchi_is,ierr) > >>> >> >>>>>>>> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); > >>> >> >>>>>>>> call ISDestroy(gdofp_is,ierr) > >>> >> >>>>>>>> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); > >>> >> >>>>>>>> call ISDestroy(gdofphi_is,ierr) > >>> >> >>>>>>>> > >>> >> >>>>>>>> ! Amat changes in each time steps > >>> >> >>>>>>>> call KSPSetOperators(ksp,Amat,Amat,ierr) !version >= 3.5.0 > >>> >> >>>>>>>> > >>> >> >>>>>>>> call KSPSolve(ksp,bvec,xvec,ierr) > >>> >> >>>>>>>> ====================================================== > >>> >> >>>>>>> > >>> >> >>>>>>> > >>> >> >>>>>>> I am guessing that "0" is not a valid string for your > Fortran > >>> >> >>>>>>> compiler. > >>> >> >>>>>>> Are > >>> >> >>>>>>> you sure > >>> >> >>>>>>> its not single quotes '0'? > >>> >> >>>>>>> > >>> >> >>>>>>> Matt > >>> >> >>>>>>> > >>> >> >>>>>>>> > >>> >> >>>>>>>> But I get the following error: > >>> >> >>>>>>>> > >>> >> >>>>>>>> [0]PETSC ERROR: --------------------- Error Message > >>> >> >>>>>>>> > -------------------------------------------------------------- > >>> >> >>>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data > >>> >> >>>>>>>> [0]PETSC ERROR: Unhandled case, must have at least two > fields, > >>> >> >>>>>>>> not 1 > >>> >> >>>>>>>> [0]PETSC ERROR: See > >>> >> >>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for > >>> >> >>>>>>>> trouble > >>> >> >>>>>>>> shooting. > >>> >> >>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > >>> >> >>>>>>>> [0]PETSC ERROR: > >>> >> >>>>>>>> > /tigress/hgharti/gitwork/SPECFEM3D_GLOBEVSPP/./bin/xspecfem3D > >>> >> >>>>>>>> on > >>> >> >>>>>>>> a > >>> >> >>>>>>>> arch-linux2-c-debug named tiger-r3c1n7 by hgharti Tue Feb > 2 > >>> >> >>>>>>>> 15: > >>> >> >>>>>>>> 29:30 2016 > >>> >> >>>>>>>> [0]PETSC ERROR: Configure options > >>> >> >>>>>>>> > >>> >> >>>>>>>> > >>> >> >>>>>>>> > >>> >> >>>>>>>> > >>> >> >>>>>>>> > --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64/ > >>> >> >>>>>>>> --with-cc=mpicc --with-cxx=mpicxx --wit > >>> >> >>>>>>>> h-fc=mpif90 --with-mpiexec=mpiexec --with-debugging=1 > >>> >> >>>>>>>> --download-scalapack --download-mumps --download-pastix > >>> >> >>>>>>>> --download-superlu --download-superlu_dist --download-metis > >>> >> >>>>>>>> --download-parmetis --download-ptscotch --download-hypre > >>> >> >>>>>>>> [0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 469 in > >>> >> >>>>>>>> > >>> >> >>>>>>>> > >>> >> >>>>>>>> > >>> >> >>>>>>>> > >>> >> >>>>>>>> > >>> >> >>>>>>>> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > >>> >> >>>>>>>> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 486 in > >>> >> >>>>>>>> > >>> >> >>>>>>>> > >>> >> >>>>>>>> > >>> >> >>>>>>>> > >>> >> >>>>>>>> > >>> >> >>>>>>>> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > >>> >> >>>>>>>> [0]PETSC ERROR: #3 PCSetUp() line 983 in > >>> >> >>>>>>>> > >>> >> >>>>>>>> > >>> >> >>>>>>>> > >>> >> >>>>>>>> > >>> >> >>>>>>>> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/interface/precon.c > >>> >> >>>>>>>> [0]PETSC ERROR: #4 KSPSetUp() line 332 in > >>> >> >>>>>>>> > >>> >> >>>>>>>> > >>> >> >>>>>>>> > >>> >> >>>>>>>> > >>> >> >>>>>>>> > >>> >> >>>>>>>> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c > >>> >> >>>>>>>> [0]PETSC ERROR: #5 KSPSolve() line 546 in > >>> >> >>>>>>>> > >>> >> >>>>>>>> > >>> >> >>>>>>>> > >>> >> >>>>>>>> > >>> >> >>>>>>>> > >>> >> >>>>>>>> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c > >>> >> >>>>>>>> forrtl: error (76): Abort trap signal > >>> >> >>>>>>>> > >>> >> >>>>>>>> Am I missing something? > >>> >> >>>>>>>> > >>> >> >>>>>>>> Thanks, > >>> >> >>>>>>>> Hom Nath > >>> >> >>>>>>>> > >>> >> >>>>>>>> On Tue, Feb 2, 2016 at 3:24 PM, Matthew Knepley > >>> >> >>>>>>>> > >>> >> >>>>>>>> wrote: > >>> >> >>>>>>>>> On Tue, Feb 2, 2016 at 2:20 PM, Hom Nath Gharti > >>> >> >>>>>>>>> > >>> >> >>>>>>>>> wrote: > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> Hi Matt, hi all, > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> I am trying to use PCFIELDSPLIT for my solver which > consists > >>> >> >>>>>>>>>> of > >>> >> >>>>>>>>>> 4 > >>> >> >>>>>>>>>> different variables, namely, u (displacement vector), > \chi > >>> >> >>>>>>>>>> (displacement potential), p(pressure), and \phi (gravity > >>> >> >>>>>>>>>> potential). > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> My code segment looks like the following: > >>> >> >>>>>>>>>> ====================================================== > >>> >> >>>>>>>>>> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) > >>> >> >>>>>>>>>> call KSPGetPC(ksp,pc,ierr) > >>> >> >>>>>>>>>> nsplit=4 > >>> >> >>>>>>>>>> call PCFieldSplitSetBlockSize(pc,nsplit,ierr); > >>> >> >>>>>>>>> > >>> >> >>>>>>>>> > >>> >> >>>>>>>>> You do not need the statement above. > >>> >> >>>>>>>>> > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); > >>> >> >>>>>>>>>> call ISDestroy(gdofu_is,ierr) > >>> >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); > >>> >> >>>>>>>>>> call ISDestroy(gdofchi_is,ierr) > >>> >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); > >>> >> >>>>>>>>>> call ISDestroy(gdofp_is,ierr) > >>> >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); > >>> >> >>>>>>>>>> call ISDestroy(gdofphi_is,ierr) > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> call PCFieldSplitGetSubKSP(pc,nsplit,subksp,ierr); > >>> >> >>>>>>>>> > >>> >> >>>>>>>>> > >>> >> >>>>>>>>> These SetOperators() calls are wrong. I am not sure why > you > >>> >> >>>>>>>>> want > >>> >> >>>>>>>>> them > >>> >> >>>>>>>>> here. > >>> >> >>>>>>>>> Also, that means you do not need the call above. > >>> >> >>>>>>>>> > >>> >> >>>>>>>>> Thanks, > >>> >> >>>>>>>>> > >>> >> >>>>>>>>> Matt > >>> >> >>>>>>>>> > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> call KSPSetOperators(subksp(1),Amat,Amat,ierr); > >>> >> >>>>>>>>>> call KSPSetOperators(subksp(2),Amat,Amat,ierr); > >>> >> >>>>>>>>>> call KSPSetOperators(subksp(3),Amat,Amat,ierr); > >>> >> >>>>>>>>>> call KSPSetOperators(subksp(4),Amat,Amat,ierr); > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> call KSPSolve(ksp,bvec,xvec,ierr) > >>> >> >>>>>>>>>> ====================================================== > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> But I am getting the following error: > >>> >> >>>>>>>>>> [79]PETSC ERROR: Null argument, when expecting valid > pointer > >>> >> >>>>>>>>>> [79]PETSC ERROR: Null Object: Parameter # 1 > >>> >> >>>>>>>>>> [79]PETSC ERROR: #1 KSPSetOperators() line 536 in > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interf > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> It looks like I am doing something wrong in "call > >>> >> >>>>>>>>>> KSPSetOperators" > >>> >> >>>>>>>>>> but > >>> >> >>>>>>>>>> I could not figure that out. > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> Could anybody help me to fix this problem? I looked into > >>> >> >>>>>>>>>> almost > >>> >> >>>>>>>>>> all > >>> >> >>>>>>>>>> related examples but I could not really figure out the > >>> >> >>>>>>>>>> correct > >>> >> >>>>>>>>>> steps > >>> >> >>>>>>>>>> after "call PCFieldSplitSetIS". > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> Any help will be greatly appreciated. > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> Best, > >>> >> >>>>>>>>>> Hom nath > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> On Sun, Jan 24, 2016 at 7:14 PM, Hom Nath Gharti > >>> >> >>>>>>>>>> > >>> >> >>>>>>>>>> wrote: > >>> >> >>>>>>>>>>> Thank you so much Matt! I will try. > >>> >> >>>>>>>>>>> > >>> >> >>>>>>>>>>> Hom Nath > >>> >> >>>>>>>>>>> > >>> >> >>>>>>>>>>> On Sun, Jan 24, 2016 at 6:26 AM, Matthew Knepley > >>> >> >>>>>>>>>>> > >>> >> >>>>>>>>>>> wrote: > >>> >> >>>>>>>>>>>> On Fri, Jan 22, 2016 at 2:19 PM, Hom Nath Gharti > >>> >> >>>>>>>>>>>> > >>> >> >>>>>>>>>>>> wrote: > >>> >> >>>>>>>>>>>>> > >>> >> >>>>>>>>>>>>> Dear all, > >>> >> >>>>>>>>>>>>> > >>> >> >>>>>>>>>>>>> I am new to PcFieldSplit. > >>> >> >>>>>>>>>>>>> > >>> >> >>>>>>>>>>>>> I have a matrix formed using MATMPIAIJ. Is it > possible to > >>> >> >>>>>>>>>>>>> use > >>> >> >>>>>>>>>>>>> PCFIELDSPLIT operations in this type of matrix? Or > does > >>> >> >>>>>>>>>>>>> it > >>> >> >>>>>>>>>>>>> have > >>> >> >>>>>>>>>>>>> to > >>> >> >>>>>>>>>>>>> be > >>> >> >>>>>>>>>>>>> MATMPIBIJ or MATNEST format? > >>> >> >>>>>>>>>>>> > >>> >> >>>>>>>>>>>> > >>> >> >>>>>>>>>>>> Yes, you can split AIJ. > >>> >> >>>>>>>>>>>> > >>> >> >>>>>>>>>>>>> > >>> >> >>>>>>>>>>>>> If possible for MATMPIAIJ, could anybody provide me a > >>> >> >>>>>>>>>>>>> simple > >>> >> >>>>>>>>>>>>> example > >>> >> >>>>>>>>>>>>> or few steps? Variables in the equations are > displacement > >>> >> >>>>>>>>>>>>> vector, > >>> >> >>>>>>>>>>>>> scalar potential and pressure. > >>> >> >>>>>>>>>>>> > >>> >> >>>>>>>>>>>> > >>> >> >>>>>>>>>>>> If you do not have a collocated discretization, then > you > >>> >> >>>>>>>>>>>> have > >>> >> >>>>>>>>>>>> to > >>> >> >>>>>>>>>>>> use > >>> >> >>>>>>>>>>>> > >>> >> >>>>>>>>>>>> > >>> >> >>>>>>>>>>>> > >>> >> >>>>>>>>>>>> > >>> >> >>>>>>>>>>>> > >>> >> >>>>>>>>>>>> > >>> >> >>>>>>>>>>>> > >>> >> >>>>>>>>>>>> > >>> >> >>>>>>>>>>>> > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html > >>> >> >>>>>>>>>>>> > >>> >> >>>>>>>>>>>> Thanks, > >>> >> >>>>>>>>>>>> > >>> >> >>>>>>>>>>>> Matt > >>> >> >>>>>>>>>>>> > >>> >> >>>>>>>>>>>>> > >>> >> >>>>>>>>>>>>> Thanks for help. > >>> >> >>>>>>>>>>>>> > >>> >> >>>>>>>>>>>>> Hom Nath > >>> >> >>>>>>>>>>>> > >>> >> >>>>>>>>>>>> > >>> >> >>>>>>>>>>>> > >>> >> >>>>>>>>>>>> > >>> >> >>>>>>>>>>>> -- > >>> >> >>>>>>>>>>>> What most experimenters take for granted before they > begin > >>> >> >>>>>>>>>>>> their > >>> >> >>>>>>>>>>>> experiments > >>> >> >>>>>>>>>>>> is infinitely more interesting than any results to > which > >>> >> >>>>>>>>>>>> their > >>> >> >>>>>>>>>>>> experiments > >>> >> >>>>>>>>>>>> lead. > >>> >> >>>>>>>>>>>> -- Norbert Wiener > >>> >> >>>>>>>>> > >>> >> >>>>>>>>> > >>> >> >>>>>>>>> > >>> >> >>>>>>>>> > >>> >> >>>>>>>>> -- > >>> >> >>>>>>>>> What most experimenters take for granted before they begin > >>> >> >>>>>>>>> their > >>> >> >>>>>>>>> experiments > >>> >> >>>>>>>>> is infinitely more interesting than any results to which > >>> >> >>>>>>>>> their > >>> >> >>>>>>>>> experiments > >>> >> >>>>>>>>> lead. > >>> >> >>>>>>>>> -- Norbert Wiener > >>> >> >>>>>>> > >>> >> >>>>>>> > >>> >> >>>>>>> > >>> >> >>>>>>> > >>> >> >>>>>>> -- > >>> >> >>>>>>> What most experimenters take for granted before they begin > >>> >> >>>>>>> their > >>> >> >>>>>>> experiments > >>> >> >>>>>>> is infinitely more interesting than any results to which > their > >>> >> >>>>>>> experiments > >>> >> >>>>>>> lead. > >>> >> >>>>>>> -- Norbert Wiener > >>> >> >>>>> > >>> >> >>>>> > >>> >> >>>>> > >>> >> >>>>> > >>> >> >>>>> -- > >>> >> >>>>> What most experimenters take for granted before they begin > their > >>> >> >>>>> experiments > >>> >> >>>>> is infinitely more interesting than any results to which their > >>> >> >>>>> experiments > >>> >> >>>>> lead. > >>> >> >>>>> -- Norbert Wiener > >>> >> >>> > >>> >> >>> > >>> >> >>> > >>> >> >>> > >>> >> >>> -- > >>> >> >>> What most experimenters take for granted before they begin their > >>> >> >>> experiments > >>> >> >>> is infinitely more interesting than any results to which their > >>> >> >>> experiments > >>> >> >>> lead. > >>> >> >>> -- Norbert Wiener > >>> >> > > >>> > > >>> > > >>> > > >>> > > >>> > -- > >>> > What most experimenters take for granted before they begin their > >>> > experiments > >>> > is infinitely more interesting than any results to which their > >>> > experiments > >>> > lead. > >>> > -- Norbert Wiener > >> > >> > >> > >> > >> -- > >> What most experimenters take for granted before they begin their > experiments > >> is infinitely more interesting than any results to which their > experiments > >> lead. > >> -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hng.email at gmail.com Thu Feb 11 12:41:22 2016 From: hng.email at gmail.com (Hom Nath Gharti) Date: Thu, 11 Feb 2016 13:41:22 -0500 Subject: [petsc-users] PCFIELDSPLIT question In-Reply-To: References: Message-ID: Perfect! Thanks a lot Matt! Hom Nath On Thu, Feb 11, 2016 at 1:34 PM, Matthew Knepley wrote: > On Thu, Feb 11, 2016 at 11:47 AM, Hom Nath Gharti > wrote: >> >> Thanks all. With further help from Stefano Zhampini I was able to >> solve my small test case using PCFieldSPlit option. I had to add >> call PCSetType(pc,PCFIELDSPLIT,ierr) >> >> Now I am trying to use same technique in my actual problem and got the >> error: >> >> PETSC ERROR: Fields must be sorted when creating split >> >> Does this mean I have to sort the global matrix rows according the split >> fields? > > > You have to sort the IS you pass to PCFieldSplitSetIS() which can be done > using ISSort() > > Thanks, > > Matt > >> >> Thanks >> Hom Nath >> >> >> >> On Tue, Feb 9, 2016 at 10:57 AM, Hom Nath Gharti >> wrote: >> > Sounds interesting! Thanks a lot Matt! I will have a look. >> > >> > Hom Nath >> > >> > On Tue, Feb 9, 2016 at 10:36 AM, Matthew Knepley >> > wrote: >> >> On Tue, Feb 9, 2016 at 9:31 AM, Hom Nath Gharti >> >> wrote: >> >>> >> >>> Thanks a lot Matt! >> >>> >> >>> Were you referring to >> >>> >> >>> >> >>> http://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex62.c.html >> >>> ? >> >>> >> >>> I do not see any statements related to PCFieldSplit there. Am I >> >>> missing something here? >> >> >> >> >> >> Yes. The whole point is not to put anything specific to FIELDSPLIT in >> >> the >> >> code. It is all in options. For example, >> >> >> >> ./ex62 -run_type full -refinement_limit 0.00625 -bc_type dirichlet >> >> -interpolate 1 -vel_petscspace_order 2 -pres_petscspace_order 1 >> >> -ksp_type >> >> fgmres -ksp_gmres_restart 100 -ksp_rtol 1.0e-9 -pc_type fieldsplit >> >> -pc_fieldsplit_type schur -pc_fieldsplit_schur_factorization_type upper >> >> -fieldsplit_pressure_ksp_rtol 1e-10 -fieldsplit_velocity_ksp_type gmres >> >> -fieldsplit_velocity_pc_type lu -fieldsplit_pressure_pc_type jacobi >> >> -snes_monitor_short -ksp_monitor_short -snes_converged_reason >> >> -ksp_converged_reason -snes_view -show_solution 0 >> >> >> >> The reason that it works is that the DM calls PCFIeldSplitSetIS() for >> >> each >> >> field in the DM. >> >> >> >> Thanks, >> >> >> >> Matt >> >> >> >>> >> >>> Thanks, >> >>> Hom Nath >> >>> >> >>> On Tue, Feb 9, 2016 at 10:19 AM, Matthew Knepley >> >>> wrote: >> >>> > On Tue, Feb 9, 2016 at 9:10 AM, Hom Nath Gharti >> >>> > >> >>> > wrote: >> >>> >> >> >>> >> Thank you so much Barry! >> >>> >> >> >>> >> For my small test case, with -pc_fieldsplit_block_size 4, the >> >>> >> program >> >>> >> runs, although the answer was not correct. At least now I get >> >>> >> something to look upon. I am using PCFieldSplitSetIS to set the >> >>> >> fields. Do I still need to use -pc_fieldsplit_block_size? >> >>> > >> >>> > >> >>> > No, this is only for co-located discretizations. >> >>> > >> >>> >> >> >>> >> In my case each grid point may have different variable sets. For >> >>> >> example, the grid point in the solid region has displacement and >> >>> >> gravity potential: ux, uy, uz, and \phi. Whereas the grid point in >> >>> >> the >> >>> >> fluid region has scalar potential, pressure and gravity potential: >> >>> >> \chi, p, and \phi. And the solid-fluid interface has all of them. >> >>> >> Do >> >>> >> you think I can still use PCFIELDSPLIT in this situation? >> >>> > >> >>> > >> >>> > We have examples, like SNES ex62, which do exactly this. >> >>> > >> >>> > Thanks, >> >>> > >> >>> > Matt >> >>> > >> >>> >> >> >>> >> Best, >> >>> >> Hom Nath >> >>> >> >> >>> >> >> >>> >> >> >>> >> On Mon, Feb 8, 2016 at 2:27 PM, Barry Smith >> >>> >> wrote: >> >>> >> > >> >>> >> > >> >>> >> > In this case you need to provide two pieces of information to >> >>> >> > the >> >>> >> > PCFIELDSPLIT. What we call the "block size" or bs which is the >> >>> >> > number >> >>> >> > of >> >>> >> > "basic fields" in the problem. For example if at each grid point >> >>> >> > you >> >>> >> > have >> >>> >> > x-velocity, y-velocity, and pressure the block size is 3. The >> >>> >> > second >> >>> >> > is you >> >>> >> > need to tell PCFIELDSPLIT what "basic fields" are in each split >> >>> >> > you >> >>> >> > want to >> >>> >> > use. >> >>> >> > >> >>> >> > In your case you can do this with -pc_fieldsplit_block_size 4. >> >>> >> > (Note if you use a DM to define your problem the PCFIELDSPLIT >> >>> >> > will >> >>> >> > automatically get the bs from the DM so you do not need to >> >>> >> > provide >> >>> >> > it. >> >>> >> > Similarly if you set a block size on your Mat the PCFIELDSPLIT >> >>> >> > will >> >>> >> > use that >> >>> >> > so you don't need to set it). >> >>> >> > >> >>> >> > Then use -pc_fieldsplit_0_fields 0,1 to indicate your first >> >>> >> > split >> >>> >> > is >> >>> >> > the 0 and 1 basic fields and -pc_fieldsplit_1_fields 2,3 to >> >>> >> > indicate >> >>> >> > the >> >>> >> > second split is the 2 and 3 basic fields. >> >>> >> > (By default if you do not provide this then PCFIELDSPLIT will use >> >>> >> > bs >> >>> >> > splits (4 in your case) one for each basic field. >> >>> >> > >> >>> >> > Barry >> >>> >> > >> >>> >> >> On Feb 8, 2016, at 11:20 AM, Hom Nath Gharti >> >>> >> >> >> >>> >> >> wrote: >> >>> >> >> >> >>> >> >> Hi Matt, Hi all, >> >>> >> >> >> >>> >> >> I am trying to have some feel for PCFIELDSPLIT testing on a very >> >>> >> >> small >> >>> >> >> matrix (attached below). I have 4 degrees of freedoms. I use 4 >> >>> >> >> processors. I want to split 4 dofs into two fields each having >> >>> >> >> two >> >>> >> >> dofs. I don't know whether this my be a problem for petsc. When >> >>> >> >> I >> >>> >> >> use >> >>> >> >> the command: >> >>> >> >> srun -n 4 ./xtestfs -ksp_monitor -ksp_view >> >>> >> >> >> >>> >> >> It runs fine. >> >>> >> >> >> >>> >> >> But when I want to use field split options using the command: >> >>> >> >> srun -n 4 ./xtestfs -ksp_monitor -ksp_type fgmres -pc_type >> >>> >> >> fieldsplit >> >>> >> >> -fieldsplit_0_ksp_type gmres -fieldsplit_0_pc_type bjacobi >> >>> >> >> -fieldsplit_1_pc_type jacobi >> >>> >> >> >> >>> >> >> I get the following error message: >> >>> >> >> [0]PETSC ERROR: Petsc has generated inconsistent data >> >>> >> >> [0]PETSC ERROR: Unhandled case, must have at least two fields, >> >>> >> >> not 1 >> >>> >> >> [0]PETSC ERROR: See >> >>> >> >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >> >>> >> >> shooting. >> >>> >> >> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >> >>> >> >> [0]PETSC ERROR: /tigress/hgharti/test/testfs/./xtestfs on a >> >>> >> >> arch-linux2-c-debug named tiger-r11n1 by hgharti Mon Feb 8 >> >>> >> >> 11:40:03 >> >>> >> >> 2016 >> >>> >> >> [0]PETSC ERROR: Configure options >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64/ >> >>> >> >> --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 >> >>> >> >> --with-mpiexec=mpiexec --with-debugging=1 --download-scalapack >> >>> >> >> --download-mumps --download-pastix --download-superlu >> >>> >> >> --download-superlu_dist --download-metis --download-parmetis >> >>> >> >> --download-ptscotch --download-hypre >> >>> >> >> [0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 469 in >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> >>> >> >> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 486 in >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> >>> >> >> [0]PETSC ERROR: #3 PCSetUp() line 983 in >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/interface/precon.c >> >>> >> >> [0]PETSC ERROR: #4 KSPSetUp() line 332 in >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >> >>> >> >> [0]PETSC ERROR: #5 KSPSolve() line 546 in >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >> >>> >> >> WARNING! There are options you set that were not used! >> >>> >> >> WARNING! could be spelling mistake, etc! >> >>> >> >> Option left: name:-fieldsplit_0_ksp_type value: gmres >> >>> >> >> Option left: name:-fieldsplit_0_pc_type value: bjacobi >> >>> >> >> Option left: name:-fieldsplit_1_pc_type value: jacobi >> >>> >> >> forrtl: error (76): Abort trap signal >> >>> >> >> >> >>> >> >> I tried several trials but could not fix the problem. Is it the >> >>> >> >> FORTRAN problem or am I doing something wrong? Any suggestions >> >>> >> >> would >> >>> >> >> be greatly appreciated. >> >>> >> >> For your information I use: >> >>> >> >> PETSc-3.6.3 >> >>> >> >> intel-16.0 compiler >> >>> >> >> intel-mpi-5.1.1 >> >>> >> >> >> >>> >> >> Program >> >>> >> >> >> >>> >> >> >> >>> >> >> Best, >> >>> >> >> Hom Nath >> >>> >> >> >> >>> >> >> ! simple test case for PCFIELDSPLIT >> >>> >> >> ! >> >>> >> >> >> >>> >> >> >> >>> >> >> ----------------------------------------------------------------------- >> >>> >> >> program testfs >> >>> >> >> implicit none >> >>> >> >> #include "petsc/finclude/petscsys.h" >> >>> >> >> #include "petsc/finclude/petscvec.h" >> >>> >> >> #include "petsc/finclude/petscvec.h90" >> >>> >> >> #include "petsc/finclude/petscmat.h" >> >>> >> >> #include "petsc/finclude/petscksp.h" >> >>> >> >> #include "petsc/finclude/petscpc.h" >> >>> >> >> #include "petsc/finclude/petscviewer.h" >> >>> >> >> #include "petsc/finclude/petscviewer.h90" >> >>> >> >> >> >>> >> >> Vec x,b,u >> >>> >> >> Mat A >> >>> >> >> KSP ksp >> >>> >> >> PC pc >> >>> >> >> PetscErrorCode ierr >> >>> >> >> PetscBool flg >> >>> >> >> PetscInt errcode,its,maxitr,myrank,nproc >> >>> >> >> PetscInt nedof,nelmt,ndof,nenod,ngdof,fixdof,nsparse,neq >> >>> >> >> PetscInt,allocatable :: krow_sparse(:),ggdof_elmt(:,:), >> >>> >> >> & >> >>> >> >> inode_elmt(:,:) >> >>> >> >> >> >>> >> >> PetscReal,allocatable :: storef(:,:),storekmat(:,:,:) >> >>> >> >> PetscInt gdof0(2),gdof1(2) >> >>> >> >> >> >>> >> >> ! initialize MPI >> >>> >> >> >> >>> >> >> call MPI_INIT(errcode) >> >>> >> >> if(errcode /= 0)stop 'ERROR: cannot initialize MPI!' >> >>> >> >> call MPI_COMM_RANK(MPI_COMM_WORLD,myrank,errcode) >> >>> >> >> if(errcode /= 0)stop 'ERROR: cannot find processor ID' >> >>> >> >> call MPI_COMM_SIZE(MPI_COMM_WORLD,nproc,errcode) >> >>> >> >> if(errcode /= 0)stop 'ERROR: cannot find number of processors!' >> >>> >> >> >> >>> >> >> ! define some parameters >> >>> >> >> ! 1D model with 4 elements. Each elements consits of 2 nodes. >> >>> >> >> Node 0 >> >>> >> >> is >> >>> >> >> fixed. >> >>> >> >> ! 1-----------2-----------3-----------4-----------5 >> >>> >> >> nelmt=1 ! per processor >> >>> >> >> nenod=2 ! number of nodes per element >> >>> >> >> nedof=2 ! number of DOFs per element >> >>> >> >> ndof=2 ! number of DOFs >> >>> >> >> ngdof=4 ! total number of global DOFs >> >>> >> >> fixdof=1 >> >>> >> >> >> >>> >> >> if(myrank==0)then >> >>> >> >> neq=1 ! number of equations >> >>> >> >> nsparse=1 >> >>> >> >> else >> >>> >> >> neq=2 ! number of equations >> >>> >> >> nsparse=4 >> >>> >> >> endif >> >>> >> >> >> >>> >> >> allocate(storef(nedof,nelmt),storekmat(nedof,nedof,nelmt), >> >>> >> >> & >> >>> >> >> >> >>> >> >> >> >>> >> >> ggdof_elmt(nedof,nelmt),inode_elmt(nenod,nelmt),krow_sparse(nsparse)) >> >>> >> >> >> >>> >> >> storef=0.0_8 ! local RHS vector >> >>> >> >> storekmat=0.0_8 ! local stiffness matrix >> >>> >> >> if(myrank==0)then >> >>> >> >> krow_sparse(:)=1 >> >>> >> >> storef(:,1)=(/12.5000_8, 12.5000_8/) >> >>> >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & >> >>> >> >> 0.0166666667_8, >> >>> >> >> 1.2333333332_8/),(/nedof,nedof/)) >> >>> >> >> inode_elmt(:,1) = (/1,2/) >> >>> >> >> ggdof_elmt(:,1) = (/0,1/) ! 0 because left node is fixed >> >>> >> >> elseif(myrank==1)then >> >>> >> >> krow_sparse(:)=(/1,1,2,2/) >> >>> >> >> storef(:,1)=(/12.5000_8, 12.5000_8/) >> >>> >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & >> >>> >> >> 0.0166666667_8, >> >>> >> >> 1.2333333332_8/),(/nedof,nedof/)) >> >>> >> >> inode_elmt(:,1) = (/1,2/) >> >>> >> >> ggdof_elmt(:,1) = (/1,2/) >> >>> >> >> elseif(myrank==2)then >> >>> >> >> krow_sparse(:)=(/1,1,2,2/) >> >>> >> >> storef(:,1)=(/12.5000_8, 12.5000_8/) >> >>> >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & >> >>> >> >> 0.0166666667_8, >> >>> >> >> 1.2333333332_8/),(/nedof,nedof/)) >> >>> >> >> inode_elmt(:,1) = (/1,2/) >> >>> >> >> ggdof_elmt(:,1) = (/2,3/) >> >>> >> >> elseif(myrank==3)then >> >>> >> >> krow_sparse(:)=(/1,1,2,2/) >> >>> >> >> storef(:,1)=(/12.5000_8, 22.5000_8/) >> >>> >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & >> >>> >> >> 0.0166666667_8, >> >>> >> >> 1.2333333332_8/),(/nedof,nedof/)) >> >>> >> >> inode_elmt(:,1) = (/1,2/) >> >>> >> >> ggdof_elmt(:,1) = (/3,4/) >> >>> >> >> endif >> >>> >> >> >> >>> >> >> call petsc_initialize >> >>> >> >> call petsc_set_matrix >> >>> >> >> call petsc_set_vector >> >>> >> >> call petsc_solve >> >>> >> >> call petsc_finalize >> >>> >> >> >> >>> >> >> ! deallocate >> >>> >> >> deallocate(storef,storekmat,ggdof_elmt,inode_elmt) >> >>> >> >> deallocate(krow_sparse) >> >>> >> >> contains >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> !------------------------------------------------------------------------------- >> >>> >> >> subroutine petsc_initialize() >> >>> >> >> implicit none >> >>> >> >> PetscInt :: istart,iend >> >>> >> >> PetscInt :: nzeros_max,nzeros_min >> >>> >> >> PetscReal :: atol,dtol,rtol >> >>> >> >> PetscInt,allocatable :: nzeros(:) >> >>> >> >> IS gdof0_is,gdof1_is >> >>> >> >> >> >>> >> >> call PetscInitialize(PETSC_NULL_CHARACTER,ierr) >> >>> >> >> call >> >>> >> >> PetscOptionsGetInt(PETSC_NULL_CHARACTER,'-n',ngdof,flg,ierr) >> >>> >> >> >> >>> >> >> ! count number of nonzeros per row >> >>> >> >> allocate(nzeros(neq)) >> >>> >> >> nzeros=0 >> >>> >> >> nzeros(krow_sparse)=nzeros(krow_sparse)+1 >> >>> >> >> nzeros_max=maxval(nzeros) >> >>> >> >> nzeros_min=minval(nzeros) >> >>> >> >> >> >>> >> >> ! create matrix >> >>> >> >> call MatCreate(PETSC_COMM_WORLD,A,ierr) >> >>> >> >> call MatSetType(A,MATMPIAIJ,ierr) >> >>> >> >> CHKERRQ(ierr) >> >>> >> >> call MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,ngdof,ngdof,ierr) >> >>> >> >> CHKERRQ(ierr) >> >>> >> >> ! preallocation >> >>> >> >> call >> >>> >> >> >> >>> >> >> >> >>> >> >> MatMPIAIJSetPreallocation(A,nzeros_max,PETSC_NULL_INTEGER,nzeros_max, & >> >>> >> >> PETSC_NULL_INTEGER,ierr) >> >>> >> >> CHKERRQ(ierr) >> >>> >> >> >> >>> >> >> call MatGetOwnershipRange(A,istart,iend,ierr) >> >>> >> >> CHKERRQ(ierr) >> >>> >> >> print*,'ownership:',myrank,istart,iend >> >>> >> >> ! create vector >> >>> >> >> call VecCreate(PETSC_COMM_WORLD,x,ierr) >> >>> >> >> call VecSetSizes(x,PETSC_DECIDE,ngdof,ierr) >> >>> >> >> call VecSetType(x,VECMPI,ierr) >> >>> >> >> call VecDuplicate(x,b,ierr) >> >>> >> >> call VecDuplicate(x,u,ierr) >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> !******************************************************************************* >> >>> >> >> ! get IS for split fields >> >>> >> >> gdof0=(/0,1/) >> >>> >> >> gdof1=(/2,3/) >> >>> >> >> call >> >>> >> >> >> >>> >> >> >> >>> >> >> ISCreateGeneral(PETSC_COMM_WORLD,2,gdof0,PETSC_COPY_VALUES,gdof0_is,ierr) >> >>> >> >> CHKERRQ(ierr) >> >>> >> >> call >> >>> >> >> >> >>> >> >> >> >>> >> >> ISCreateGeneral(PETSC_COMM_WORLD,2,gdof1,PETSC_COPY_VALUES,gdof1_is,ierr) >> >>> >> >> CHKERRQ(ierr) >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> !******************************************************************************* >> >>> >> >> >> >>> >> >> ! Create linear solver context >> >>> >> >> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >> >>> >> >> call KSPSetOperators(ksp,A,A,ierr) >> >>> >> >> >> >>> >> >> call KSPGetPC(ksp,pc,ierr) >> >>> >> >> CHKERRQ(ierr) >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> !******************************************************************************* >> >>> >> >> ! split PC >> >>> >> >> call PCFieldSplitSetIS(pc,"0",gdof0_is,ierr); >> >>> >> >> CHKERRQ(ierr) >> >>> >> >> call ISDestroy(gdof0_is,ierr) >> >>> >> >> CHKERRQ(ierr) >> >>> >> >> call PCFieldSplitSetIS(pc,"1",gdof1_is,ierr); >> >>> >> >> CHKERRQ(ierr) >> >>> >> >> call ISDestroy(gdof1_is,ierr) >> >>> >> >> CHKERRQ(ierr) >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> !******************************************************************************* >> >>> >> >> >> >>> >> >> rtol = 1.0d-6 >> >>> >> >> atol = 1.0d-6 >> >>> >> >> dtol = 1.0d10 >> >>> >> >> maxitr = 1000 >> >>> >> >> call KSPSetTolerances(ksp,rtol,atol,dtol,maxitr,ierr) >> >>> >> >> CHKERRQ(ierr) >> >>> >> >> call KSPSetFromOptions(ksp,ierr) >> >>> >> >> end subroutine petsc_initialize >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> !------------------------------------------------------------------------------- >> >>> >> >> >> >>> >> >> subroutine petsc_set_matrix() >> >>> >> >> >> >>> >> >> implicit none >> >>> >> >> integer :: i,i_elmt,j,ncount >> >>> >> >> integer :: ggdof(NEDOF),idof(NEDOF),igdof(NEDOF) >> >>> >> >> >> >>> >> >> ! Set and assemble matrix. >> >>> >> >> call MatZeroEntries(A,ierr) >> >>> >> >> CHKERRQ(ierr) >> >>> >> >> do i_elmt=1,nelmt >> >>> >> >> ggdof(:)=ggdof_elmt(:,i_elmt) >> >>> >> >> ggdof(:)=ggdof(:)-1 ! petsc index starts from 0 >> >>> >> >> ncount=0; idof=-1; igdof=-1 >> >>> >> >> do i=1,NEDOF >> >>> >> >> do j=1,NEDOF >> >>> >> >> if(ggdof(i).ge.0.and.ggdof(j).ge.0)then >> >>> >> >> call >> >>> >> >> MatSetValues(A,1,ggdof(i),1,ggdof(j),storekmat(i,j,i_elmt), >> >>> >> >> & >> >>> >> >> ADD_VALUES,ierr) >> >>> >> >> CHKERRQ(ierr) >> >>> >> >> endif >> >>> >> >> enddo >> >>> >> >> enddo >> >>> >> >> enddo >> >>> >> >> call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) >> >>> >> >> call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) >> >>> >> >> if(myrank==0)print*,'matrix setting & assembly complete!' >> >>> >> >> >> >>> >> >> end subroutine petsc_set_matrix >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> !------------------------------------------------------------------------------- >> >>> >> >> >> >>> >> >> subroutine petsc_set_vector() >> >>> >> >> implicit none >> >>> >> >> PetscScalar zero >> >>> >> >> integer :: i,i_elmt,ncount >> >>> >> >> integer :: ggdof(NEDOF),idof(NEDOF),igdof(NEDOF) >> >>> >> >> >> >>> >> >> ! set vector >> >>> >> >> zero=0.0_8 >> >>> >> >> call VecSet(b,zero,ierr) >> >>> >> >> do i_elmt=1,nelmt >> >>> >> >> ggdof(:)=ggdof_elmt(:,i_elmt) >> >>> >> >> ggdof(:)=ggdof(:)-1 ! petsc index starts from 0 >> >>> >> >> ncount=0; idof=-1; igdof=-1 >> >>> >> >> do i=1,NEDOF >> >>> >> >> if(ggdof(i).ge.0)then >> >>> >> >> call >> >>> >> >> VecSetValues(b,1,ggdof(i),storef(i,i_elmt),ADD_VALUES,ierr); >> >>> >> >> CHKERRQ(ierr) >> >>> >> >> endif >> >>> >> >> enddo >> >>> >> >> enddo >> >>> >> >> >> >>> >> >> ! assemble vector >> >>> >> >> call VecAssemblyBegin(b,ierr) >> >>> >> >> call VecAssemblyEnd(b,ierr) >> >>> >> >> if(myrank==0)print*,'vector setting & assembly complete!' >> >>> >> >> >> >>> >> >> end subroutine petsc_set_vector >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> !------------------------------------------------------------------------------- >> >>> >> >> >> >>> >> >> subroutine petsc_solve() >> >>> >> >> implicit none >> >>> >> >> PetscInt ireason >> >>> >> >> PetscScalar a_v(1) >> >>> >> >> PetscOffset a_i >> >>> >> >> PetscInt n >> >>> >> >> PetscReal,allocatable :: sdata(:) >> >>> >> >> >> >>> >> >> call VecGetSize(b,n,ierr) >> >>> >> >> CHKERRQ(ierr) >> >>> >> >> allocate(sdata(n)) >> >>> >> >> sdata=0.0_8 >> >>> >> >> call VecGetArray(b,a_v,a_i,ierr) >> >>> >> >> CHKERRQ(ierr) >> >>> >> >> sdata(1:n)=a_v(a_i+1:a_i+n) >> >>> >> >> call VecRestoreArray(b,a_v,a_i,ierr) >> >>> >> >> CHKERRQ(ierr) >> >>> >> >> print*,'rhs:',sdata >> >>> >> >> >> >>> >> >> call KSPSolve(ksp,b,x,ierr) >> >>> >> >> sdata=0.0_8 >> >>> >> >> call VecGetArray(x,a_v,a_i,ierr) >> >>> >> >> CHKERRQ(ierr) >> >>> >> >> sdata(1:n)=a_v(a_i+1:a_i+n) >> >>> >> >> call VecRestoreArray(b,a_v,a_i,ierr) >> >>> >> >> CHKERRQ(ierr) >> >>> >> >> print*,'solution:',sdata >> >>> >> >> >> >>> >> >> ! Check convergence >> >>> >> >> call KSPGetConvergedReason(ksp,ireason,ierr) >> >>> >> >> if(myrank==0)print*,'converges reason',myrank,ireason >> >>> >> >> call KSPGetIterationNumber(ksp,its,ierr) >> >>> >> >> if(myrank==0)print*,'Iterations:',its >> >>> >> >> deallocate(sdata) >> >>> >> >> end subroutine petsc_solve >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> !------------------------------------------------------------------------------- >> >>> >> >> >> >>> >> >> subroutine petsc_finalize() >> >>> >> >> implicit none >> >>> >> >> >> >>> >> >> ! Free work space. >> >>> >> >> call VecDestroy(x,ierr) >> >>> >> >> call VecDestroy(u,ierr) >> >>> >> >> call VecDestroy(b,ierr) >> >>> >> >> call MatDestroy(A,ierr) >> >>> >> >> call KSPDestroy(ksp,ierr) >> >>> >> >> call PetscFinalize(ierr) >> >>> >> >> CHKERRQ(ierr) >> >>> >> >> >> >>> >> >> end subroutine petsc_finalize >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> !------------------------------------------------------------------------------- >> >>> >> >> >> >>> >> >> end program testfs >> >>> >> >> >> >>> >> >> On Tue, Feb 2, 2016 at 4:54 PM, Matthew Knepley >> >>> >> >> >> >>> >> >> wrote: >> >>> >> >>> On Tue, Feb 2, 2016 at 3:11 PM, Hom Nath Gharti >> >>> >> >>> >> >>> >> >>> wrote: >> >>> >> >>>> >> >>> >> >>>> Thanks a lot. I will try. >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> Also, if you send a small test case, we can run it and tell you >> >>> >> >>> the >> >>> >> >>> problem. >> >>> >> >>> >> >>> >> >>> Matt >> >>> >> >>> >> >>> >> >>>> >> >>> >> >>>> Hom Nath >> >>> >> >>>> >> >>> >> >>>> On Tue, Feb 2, 2016 at 4:02 PM, Matthew Knepley >> >>> >> >>>> >> >>> >> >>>> wrote: >> >>> >> >>>>> On Tue, Feb 2, 2016 at 3:01 PM, Hom Nath Gharti >> >>> >> >>>>> >> >>> >> >>>>> wrote: >> >>> >> >>>>>> >> >>> >> >>>>>> Thanks again Matt. Unfortunately got the same errors with >> >>> >> >>>>>> '0'. I >> >>> >> >>>>>> think >> >>> >> >>>>>> both are valid in Fortran. >> >>> >> >>>>> >> >>> >> >>>>> >> >>> >> >>>>> Then you will have to go in the debugger and see why its not >> >>> >> >>>>> creating 4 >> >>> >> >>>>> splits, since this is exactly >> >>> >> >>>>> what it does in our tests. This is how all the SNES tests >> >>> >> >>>>> that I >> >>> >> >>>>> use >> >>> >> >>>>> work. I >> >>> >> >>>>> am sure its a Fortran >> >>> >> >>>>> problem. >> >>> >> >>>>> >> >>> >> >>>>> Thanks, >> >>> >> >>>>> >> >>> >> >>>>> Matt >> >>> >> >>>>> >> >>> >> >>>>>> >> >>> >> >>>>>> Hom Nath >> >>> >> >>>>>> >> >>> >> >>>>>> On Tue, Feb 2, 2016 at 3:42 PM, Matthew Knepley >> >>> >> >>>>>> >> >>> >> >>>>>> wrote: >> >>> >> >>>>>>> On Tue, Feb 2, 2016 at 2:35 PM, Hom Nath Gharti >> >>> >> >>>>>>> >> >>> >> >>>>>>> wrote: >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> Thank you so much Matt. >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> I now tried the following: >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> ====================================================== >> >>> >> >>>>>>>> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >> >>> >> >>>>>>>> call KSPGetPC(ksp,pc,ierr) >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); >> >>> >> >>>>>>>> call ISDestroy(gdofu_is,ierr) >> >>> >> >>>>>>>> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); >> >>> >> >>>>>>>> call ISDestroy(gdofchi_is,ierr) >> >>> >> >>>>>>>> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); >> >>> >> >>>>>>>> call ISDestroy(gdofp_is,ierr) >> >>> >> >>>>>>>> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); >> >>> >> >>>>>>>> call ISDestroy(gdofphi_is,ierr) >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> ! Amat changes in each time steps >> >>> >> >>>>>>>> call KSPSetOperators(ksp,Amat,Amat,ierr) !version >= 3.5.0 >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> call KSPSolve(ksp,bvec,xvec,ierr) >> >>> >> >>>>>>>> ====================================================== >> >>> >> >>>>>>> >> >>> >> >>>>>>> >> >>> >> >>>>>>> I am guessing that "0" is not a valid string for your >> >>> >> >>>>>>> Fortran >> >>> >> >>>>>>> compiler. >> >>> >> >>>>>>> Are >> >>> >> >>>>>>> you sure >> >>> >> >>>>>>> its not single quotes '0'? >> >>> >> >>>>>>> >> >>> >> >>>>>>> Matt >> >>> >> >>>>>>> >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> But I get the following error: >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> [0]PETSC ERROR: --------------------- Error Message >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> -------------------------------------------------------------- >> >>> >> >>>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >> >>> >> >>>>>>>> [0]PETSC ERROR: Unhandled case, must have at least two >> >>> >> >>>>>>>> fields, >> >>> >> >>>>>>>> not 1 >> >>> >> >>>>>>>> [0]PETSC ERROR: See >> >>> >> >>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for >> >>> >> >>>>>>>> trouble >> >>> >> >>>>>>>> shooting. >> >>> >> >>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >> >>> >> >>>>>>>> [0]PETSC ERROR: >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> /tigress/hgharti/gitwork/SPECFEM3D_GLOBEVSPP/./bin/xspecfem3D >> >>> >> >>>>>>>> on >> >>> >> >>>>>>>> a >> >>> >> >>>>>>>> arch-linux2-c-debug named tiger-r3c1n7 by hgharti Tue Feb >> >>> >> >>>>>>>> 2 >> >>> >> >>>>>>>> 15: >> >>> >> >>>>>>>> 29:30 2016 >> >>> >> >>>>>>>> [0]PETSC ERROR: Configure options >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64/ >> >>> >> >>>>>>>> --with-cc=mpicc --with-cxx=mpicxx --wit >> >>> >> >>>>>>>> h-fc=mpif90 --with-mpiexec=mpiexec --with-debugging=1 >> >>> >> >>>>>>>> --download-scalapack --download-mumps --download-pastix >> >>> >> >>>>>>>> --download-superlu --download-superlu_dist >> >>> >> >>>>>>>> --download-metis >> >>> >> >>>>>>>> --download-parmetis --download-ptscotch --download-hypre >> >>> >> >>>>>>>> [0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 469 in >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> >>> >> >>>>>>>> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 486 in >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> >>> >> >>>>>>>> [0]PETSC ERROR: #3 PCSetUp() line 983 in >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/interface/precon.c >> >>> >> >>>>>>>> [0]PETSC ERROR: #4 KSPSetUp() line 332 in >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >> >>> >> >>>>>>>> [0]PETSC ERROR: #5 KSPSolve() line 546 in >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >> >>> >> >>>>>>>> forrtl: error (76): Abort trap signal >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> Am I missing something? >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> Thanks, >> >>> >> >>>>>>>> Hom Nath >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> On Tue, Feb 2, 2016 at 3:24 PM, Matthew Knepley >> >>> >> >>>>>>>> >> >>> >> >>>>>>>> wrote: >> >>> >> >>>>>>>>> On Tue, Feb 2, 2016 at 2:20 PM, Hom Nath Gharti >> >>> >> >>>>>>>>> >> >>> >> >>>>>>>>> wrote: >> >>> >> >>>>>>>>>> >> >>> >> >>>>>>>>>> Hi Matt, hi all, >> >>> >> >>>>>>>>>> >> >>> >> >>>>>>>>>> I am trying to use PCFIELDSPLIT for my solver which >> >>> >> >>>>>>>>>> consists >> >>> >> >>>>>>>>>> of >> >>> >> >>>>>>>>>> 4 >> >>> >> >>>>>>>>>> different variables, namely, u (displacement vector), >> >>> >> >>>>>>>>>> \chi >> >>> >> >>>>>>>>>> (displacement potential), p(pressure), and \phi (gravity >> >>> >> >>>>>>>>>> potential). >> >>> >> >>>>>>>>>> >> >>> >> >>>>>>>>>> My code segment looks like the following: >> >>> >> >>>>>>>>>> ====================================================== >> >>> >> >>>>>>>>>> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >> >>> >> >>>>>>>>>> call KSPGetPC(ksp,pc,ierr) >> >>> >> >>>>>>>>>> nsplit=4 >> >>> >> >>>>>>>>>> call PCFieldSplitSetBlockSize(pc,nsplit,ierr); >> >>> >> >>>>>>>>> >> >>> >> >>>>>>>>> >> >>> >> >>>>>>>>> You do not need the statement above. >> >>> >> >>>>>>>>> >> >>> >> >>>>>>>>>> >> >>> >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); >> >>> >> >>>>>>>>>> call ISDestroy(gdofu_is,ierr) >> >>> >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); >> >>> >> >>>>>>>>>> call ISDestroy(gdofchi_is,ierr) >> >>> >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); >> >>> >> >>>>>>>>>> call ISDestroy(gdofp_is,ierr) >> >>> >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); >> >>> >> >>>>>>>>>> call ISDestroy(gdofphi_is,ierr) >> >>> >> >>>>>>>>>> >> >>> >> >>>>>>>>>> call PCFieldSplitGetSubKSP(pc,nsplit,subksp,ierr); >> >>> >> >>>>>>>>> >> >>> >> >>>>>>>>> >> >>> >> >>>>>>>>> These SetOperators() calls are wrong. I am not sure why >> >>> >> >>>>>>>>> you >> >>> >> >>>>>>>>> want >> >>> >> >>>>>>>>> them >> >>> >> >>>>>>>>> here. >> >>> >> >>>>>>>>> Also, that means you do not need the call above. >> >>> >> >>>>>>>>> >> >>> >> >>>>>>>>> Thanks, >> >>> >> >>>>>>>>> >> >>> >> >>>>>>>>> Matt >> >>> >> >>>>>>>>> >> >>> >> >>>>>>>>>> >> >>> >> >>>>>>>>>> call KSPSetOperators(subksp(1),Amat,Amat,ierr); >> >>> >> >>>>>>>>>> call KSPSetOperators(subksp(2),Amat,Amat,ierr); >> >>> >> >>>>>>>>>> call KSPSetOperators(subksp(3),Amat,Amat,ierr); >> >>> >> >>>>>>>>>> call KSPSetOperators(subksp(4),Amat,Amat,ierr); >> >>> >> >>>>>>>>>> >> >>> >> >>>>>>>>>> call KSPSolve(ksp,bvec,xvec,ierr) >> >>> >> >>>>>>>>>> ====================================================== >> >>> >> >>>>>>>>>> >> >>> >> >>>>>>>>>> But I am getting the following error: >> >>> >> >>>>>>>>>> [79]PETSC ERROR: Null argument, when expecting valid >> >>> >> >>>>>>>>>> pointer >> >>> >> >>>>>>>>>> [79]PETSC ERROR: Null Object: Parameter # 1 >> >>> >> >>>>>>>>>> [79]PETSC ERROR: #1 KSPSetOperators() line 536 in >> >>> >> >>>>>>>>>> >> >>> >> >>>>>>>>>> >> >>> >> >>>>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interf >> >>> >> >>>>>>>>>> >> >>> >> >>>>>>>>>> It looks like I am doing something wrong in "call >> >>> >> >>>>>>>>>> KSPSetOperators" >> >>> >> >>>>>>>>>> but >> >>> >> >>>>>>>>>> I could not figure that out. >> >>> >> >>>>>>>>>> >> >>> >> >>>>>>>>>> Could anybody help me to fix this problem? I looked into >> >>> >> >>>>>>>>>> almost >> >>> >> >>>>>>>>>> all >> >>> >> >>>>>>>>>> related examples but I could not really figure out the >> >>> >> >>>>>>>>>> correct >> >>> >> >>>>>>>>>> steps >> >>> >> >>>>>>>>>> after "call PCFieldSplitSetIS". >> >>> >> >>>>>>>>>> >> >>> >> >>>>>>>>>> Any help will be greatly appreciated. >> >>> >> >>>>>>>>>> >> >>> >> >>>>>>>>>> Best, >> >>> >> >>>>>>>>>> Hom nath >> >>> >> >>>>>>>>>> >> >>> >> >>>>>>>>>> On Sun, Jan 24, 2016 at 7:14 PM, Hom Nath Gharti >> >>> >> >>>>>>>>>> >> >>> >> >>>>>>>>>> wrote: >> >>> >> >>>>>>>>>>> Thank you so much Matt! I will try. >> >>> >> >>>>>>>>>>> >> >>> >> >>>>>>>>>>> Hom Nath >> >>> >> >>>>>>>>>>> >> >>> >> >>>>>>>>>>> On Sun, Jan 24, 2016 at 6:26 AM, Matthew Knepley >> >>> >> >>>>>>>>>>> >> >>> >> >>>>>>>>>>> wrote: >> >>> >> >>>>>>>>>>>> On Fri, Jan 22, 2016 at 2:19 PM, Hom Nath Gharti >> >>> >> >>>>>>>>>>>> >> >>> >> >>>>>>>>>>>> wrote: >> >>> >> >>>>>>>>>>>>> >> >>> >> >>>>>>>>>>>>> Dear all, >> >>> >> >>>>>>>>>>>>> >> >>> >> >>>>>>>>>>>>> I am new to PcFieldSplit. >> >>> >> >>>>>>>>>>>>> >> >>> >> >>>>>>>>>>>>> I have a matrix formed using MATMPIAIJ. Is it >> >>> >> >>>>>>>>>>>>> possible to >> >>> >> >>>>>>>>>>>>> use >> >>> >> >>>>>>>>>>>>> PCFIELDSPLIT operations in this type of matrix? Or >> >>> >> >>>>>>>>>>>>> does >> >>> >> >>>>>>>>>>>>> it >> >>> >> >>>>>>>>>>>>> have >> >>> >> >>>>>>>>>>>>> to >> >>> >> >>>>>>>>>>>>> be >> >>> >> >>>>>>>>>>>>> MATMPIBIJ or MATNEST format? >> >>> >> >>>>>>>>>>>> >> >>> >> >>>>>>>>>>>> >> >>> >> >>>>>>>>>>>> Yes, you can split AIJ. >> >>> >> >>>>>>>>>>>> >> >>> >> >>>>>>>>>>>>> >> >>> >> >>>>>>>>>>>>> If possible for MATMPIAIJ, could anybody provide me a >> >>> >> >>>>>>>>>>>>> simple >> >>> >> >>>>>>>>>>>>> example >> >>> >> >>>>>>>>>>>>> or few steps? Variables in the equations are >> >>> >> >>>>>>>>>>>>> displacement >> >>> >> >>>>>>>>>>>>> vector, >> >>> >> >>>>>>>>>>>>> scalar potential and pressure. >> >>> >> >>>>>>>>>>>> >> >>> >> >>>>>>>>>>>> >> >>> >> >>>>>>>>>>>> If you do not have a collocated discretization, then >> >>> >> >>>>>>>>>>>> you >> >>> >> >>>>>>>>>>>> have >> >>> >> >>>>>>>>>>>> to >> >>> >> >>>>>>>>>>>> use >> >>> >> >>>>>>>>>>>> >> >>> >> >>>>>>>>>>>> >> >>> >> >>>>>>>>>>>> >> >>> >> >>>>>>>>>>>> >> >>> >> >>>>>>>>>>>> >> >>> >> >>>>>>>>>>>> >> >>> >> >>>>>>>>>>>> >> >>> >> >>>>>>>>>>>> >> >>> >> >>>>>>>>>>>> >> >>> >> >>>>>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html >> >>> >> >>>>>>>>>>>> >> >>> >> >>>>>>>>>>>> Thanks, >> >>> >> >>>>>>>>>>>> >> >>> >> >>>>>>>>>>>> Matt >> >>> >> >>>>>>>>>>>> >> >>> >> >>>>>>>>>>>>> >> >>> >> >>>>>>>>>>>>> Thanks for help. >> >>> >> >>>>>>>>>>>>> >> >>> >> >>>>>>>>>>>>> Hom Nath >> >>> >> >>>>>>>>>>>> >> >>> >> >>>>>>>>>>>> >> >>> >> >>>>>>>>>>>> >> >>> >> >>>>>>>>>>>> >> >>> >> >>>>>>>>>>>> -- >> >>> >> >>>>>>>>>>>> What most experimenters take for granted before they >> >>> >> >>>>>>>>>>>> begin >> >>> >> >>>>>>>>>>>> their >> >>> >> >>>>>>>>>>>> experiments >> >>> >> >>>>>>>>>>>> is infinitely more interesting than any results to >> >>> >> >>>>>>>>>>>> which >> >>> >> >>>>>>>>>>>> their >> >>> >> >>>>>>>>>>>> experiments >> >>> >> >>>>>>>>>>>> lead. >> >>> >> >>>>>>>>>>>> -- Norbert Wiener >> >>> >> >>>>>>>>> >> >>> >> >>>>>>>>> >> >>> >> >>>>>>>>> >> >>> >> >>>>>>>>> >> >>> >> >>>>>>>>> -- >> >>> >> >>>>>>>>> What most experimenters take for granted before they >> >>> >> >>>>>>>>> begin >> >>> >> >>>>>>>>> their >> >>> >> >>>>>>>>> experiments >> >>> >> >>>>>>>>> is infinitely more interesting than any results to which >> >>> >> >>>>>>>>> their >> >>> >> >>>>>>>>> experiments >> >>> >> >>>>>>>>> lead. >> >>> >> >>>>>>>>> -- Norbert Wiener >> >>> >> >>>>>>> >> >>> >> >>>>>>> >> >>> >> >>>>>>> >> >>> >> >>>>>>> >> >>> >> >>>>>>> -- >> >>> >> >>>>>>> What most experimenters take for granted before they begin >> >>> >> >>>>>>> their >> >>> >> >>>>>>> experiments >> >>> >> >>>>>>> is infinitely more interesting than any results to which >> >>> >> >>>>>>> their >> >>> >> >>>>>>> experiments >> >>> >> >>>>>>> lead. >> >>> >> >>>>>>> -- Norbert Wiener >> >>> >> >>>>> >> >>> >> >>>>> >> >>> >> >>>>> >> >>> >> >>>>> >> >>> >> >>>>> -- >> >>> >> >>>>> What most experimenters take for granted before they begin >> >>> >> >>>>> their >> >>> >> >>>>> experiments >> >>> >> >>>>> is infinitely more interesting than any results to which >> >>> >> >>>>> their >> >>> >> >>>>> experiments >> >>> >> >>>>> lead. >> >>> >> >>>>> -- Norbert Wiener >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> -- >> >>> >> >>> What most experimenters take for granted before they begin >> >>> >> >>> their >> >>> >> >>> experiments >> >>> >> >>> is infinitely more interesting than any results to which their >> >>> >> >>> experiments >> >>> >> >>> lead. >> >>> >> >>> -- Norbert Wiener >> >>> >> > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> > -- >> >>> > What most experimenters take for granted before they begin their >> >>> > experiments >> >>> > is infinitely more interesting than any results to which their >> >>> > experiments >> >>> > lead. >> >>> > -- Norbert Wiener >> >> >> >> >> >> >> >> >> >> -- >> >> What most experimenters take for granted before they begin their >> >> experiments >> >> is infinitely more interesting than any results to which their >> >> experiments >> >> lead. >> >> -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener From wen.zhao at outlook.fr Thu Feb 11 12:56:03 2016 From: wen.zhao at outlook.fr (wen zhao) Date: Thu, 11 Feb 2016 19:56:03 +0100 Subject: [petsc-users] The matrix viewer Message-ID: Hi barry I have a question about the viewer of a matrix. I use a PCcholesky method to factor matrix. and i use this codes to get a factored matrix F. ierr = PCSetType(pc,PCCHOLESKY);CHKERRQ(ierr); ierr = PCFactorSetMatSolverPackage(pc,MATSOLVERMUMPS);CHKERRQ(ierr); ierr = PCFactorSetUpMatSolverPackage(pc);CHKERRQ(ierr); ierr = PCFactorGetMatrix(pc,&F);CHKERRQ(ierr); But it seems that the matrix F can not be viewed with MATVIEW() function. It tells me the f is in ACII format i guess. So what should i do to view the matrix and export it in using PetscViewerBinaryOpen() function. Thanks very much -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Feb 11 12:57:37 2016 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 11 Feb 2016 12:57:37 -0600 Subject: [petsc-users] The matrix viewer In-Reply-To: References: Message-ID: On Thu, Feb 11, 2016 at 12:56 PM, wen zhao wrote: > Hi barry > > I have a question about the viewer of a matrix. I use a PCcholesky method > to factor matrix. and i use this codes to get a factored matrix F. > > ierr = PCSetType(pc,PCCHOLESKY);CHKERRQ(ierr); > ierr = PCFactorSetMatSolverPackage(pc,MATSOLVERMUMPS);CHKERRQ(ierr); > ierr = PCFactorSetUpMatSolverPackage(pc);CHKERRQ(ierr); > ierr = PCFactorGetMatrix(pc,&F);CHKERRQ(ierr); > > But it seems that the matrix F can not be viewed with MATVIEW() function. > It tells me the f is in ACII format i guess. So what should i do to view > the matrix and export it in using PetscViewerBinaryOpen() function. > We do not have viewers for matrices stored in the formats from other packages, like MUMPS. We have no idea what that internal format is, and have no guarantee that they will not change the format. Matt > Thanks very much > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrosso at uci.edu Thu Feb 11 13:17:50 2016 From: mrosso at uci.edu (Michele Rosso) Date: Thu, 11 Feb 2016 11:17:50 -0800 Subject: [petsc-users] DIVERGED_PCSETUP_FAILED In-Reply-To: References: <1455154398.5948.15.camel@enterprise-A> <1455157291.5948.19.camel@enterprise-A> <0903A02C-4B7A-4C1F-AD27-44CB6B1C6F07@mcs.anl.gov> <1455162540.5948.21.camel@enterprise-A> <1455170718.5948.28.camel@enterprise-A> Message-ID: <56BCDE5E.1060202@uci.edu> I am using periodic along x and Neumann along y. I remove the nullspace via -ksp_constant_null_space. Matt's suggestion worked; I will also give a try to umfpack and cholesky. Thanks, Michele On 02/11/2016 06:50 AM, Matthew Knepley wrote: > On Thu, Feb 11, 2016 at 12:30 AM, Dave May > wrote: > > > > On 11 February 2016 at 07:05, Michele Rosso > wrote: > > I tried setting -mat_superlu_dist_replacetinypivot true: it > does help to advance the run past the previous "critical" > point but eventually it stops later with the same error. > I forgot to mention my system is singular: I remove the > constant null space but I am not sure if the coarse solver > needs to be explicity informed of this. > > > Right - are you using pure Newmann boundary conditions? > > To make the solution unique, are you > (a) imposing a single Dichletet boundary condition on your field > by explicitly modifying the matrix > (b) imposing a a condition like > \int \phi dV = 0 > via something like -ksp_constant_null_space > > If you removed removed the null space by modifying the matrix > explicitly (a), the sparse direct solver > should go through. If you use (b), then this method cannot be used > to help the direct solver. > > If this is the intended target problem size (16x16), gather the > matrix and using petsc Cholesky or Umfpack. > Cholesky is more stable than LU and can usually deal with a single > zero eigenvaue without resorting to tricks. Umfpack will solve the > problem easily as it uses clever re-ordering. If you have access > to MKL-Pardiso, that will also work great. > > > An easy fix is just to use -pc_type svd on the coarse grid. > > Matt > > Thanks, > Dave > > > > Michele > > > On Wed, 2016-02-10 at 22:15 -0600, Barry Smith wrote: >> You can try the option >> >> -mat_superlu_dist_replacetinypivot true >> >> if you are luck it get you past the zero pivot but still give an adequate preconditioner. >> >> Barry >> >> > On Feb 10, 2016, at 9:49 PM, Michele Rosso > > wrote: >> > >> > Hong, >> > >> > here if the output of grep -info: >> > >> > using diagonal shift on blocks to prevent zero pivot [INBLOCKS] >> > Replace tiny pivots FALSE >> > tolerance for zero pivot 2.22045e-14 >> > >> > It seems it is not replacing small pivots: could this be >> the problem? >> > I will also try Barry's suggestion to diagnose the problem. >> > >> > Thanks, >> > Michele >> > >> > >> > On Wed, 2016-02-10 at 21:22 -0600, Barry Smith wrote: >> >> > On Feb 10, 2016, at 9:00 PM, Hong > > wrote: >> >> > >> >> > Michele : >> >> > Superlu_dist LU is used for coarse grid PC, which likely >> produces a zero-pivot. >> >> > Run your code with '-info |grep pivot' to verify. >> >> >> >> >> >> Michele >> >> >> >> You can also run with -ksp_error_if_not_converged in or >> not in the debugger and it will stop immediately when the >> problem is detected and hopefully provide additional useful >> information about what has happened. >> >> >> >> Barry >> >> >> >> >> >> > >> >> > Hong >> >> > >> >> > Hi Matt, >> >> > >> >> > the ksp_view output was an attachment to my previous email. >> >> > Here it is: >> >> > >> >> > KSP Object: 1 MPI processes >> >> > type: cg >> >> > maximum iterations=10000 >> >> > tolerances: relative=1e-08, absolute=1e-50, >> divergence=10000. >> >> > left preconditioning >> >> > using nonzero initial guess >> >> > using UNPRECONDITIONED norm type for convergence test >> >> > PC Object: 1 MPI processes >> >> > type: mg >> >> > MG: type is MULTIPLICATIVE, levels=4 cycles=v >> >> > Cycles per PCApply=1 >> >> > Using Galerkin computed coarse grid matrices >> >> > Coarse grid solver -- level ------------------------------- >> >> > KSP Object: (mg_coarse_) 1 MPI processes >> >> > type: preonly >> >> > maximum iterations=1, initial guess is zero >> >> > tolerances: relative=1e-05, absolute=1e-50, >> divergence=10000. >> >> > left preconditioning >> >> > using NONE norm type for convergence test >> >> > PC Object: (mg_coarse_) 1 MPI processes >> >> > type: lu >> >> > LU: out-of-place factorization >> >> > tolerance for zero pivot 2.22045e-14 >> >> > using diagonal shift on blocks to prevent zero pivot >> [INBLOCKS] >> >> > matrix ordering: nd >> >> > factor fill ratio given 0., needed 0. >> >> > Factored matrix follows: >> >> > Mat Object: 1 MPI processes >> >> > type: seqaij >> >> > rows=16, cols=16 >> >> > package used to perform factorization: superlu_dist >> >> > total: nonzeros=0, allocated nonzeros=0 >> >> > total number of mallocs used during MatSetValues calls =0 >> >> > SuperLU_DIST run parameters: >> >> > Process grid nprow 1 x npcol 1 >> >> > Equilibrate matrix TRUE >> >> > Matrix input mode 0 >> >> > Replace tiny pivots FALSE >> >> > Use iterative refinement FALSE >> >> > Processors in row 1 col partition 1 >> >> > Row permutation LargeDiag >> >> > Column permutation METIS_AT_PLUS_A >> >> > Parallel symbolic factorization FALSE >> >> > Repeated factorization SamePattern >> >> > linear system matrix = precond matrix: >> >> > Mat Object: 1 MPI processes >> >> > type: seqaij >> >> > rows=16, cols=16 >> >> > total: nonzeros=72, allocated nonzeros=72 >> >> > total number of mallocs used during MatSetValues calls =0 >> >> > not using I-node routines >> >> > Down solver (pre-smoother) on level 1 >> ------------------------------- >> >> > KSP Object: (mg_levels_1_) 1 MPI processes >> >> > type: richardson >> >> > Richardson: damping factor=1. >> >> > maximum iterations=2 >> >> > tolerances: relative=1e-05, absolute=1e-50, >> divergence=10000. >> >> > left preconditioning >> >> > using nonzero initial guess >> >> > using NONE norm type for convergence test >> >> > PC Object: (mg_levels_1_) 1 MPI processes >> >> > type: sor >> >> > SOR: type = local_symmetric, iterations = 1, local >> iterations = 1, omega = 1. >> >> > linear system matrix = precond matrix: >> >> > Mat Object: 1 MPI processes >> >> > type: seqaij >> >> > rows=64, cols=64 >> >> > total: nonzeros=304, allocated nonzeros=304 >> >> > total number of mallocs used during MatSetValues calls =0 >> >> > not using I-node routines >> >> > Up solver (post-smoother) same as down solver (pre-smoother) >> >> > Down solver (pre-smoother) on level 2 >> ------------------------------- >> >> > KSP Object: (mg_levels_2_) 1 MPI processes >> >> > type: richardson >> >> > Richardson: damping factor=1. >> >> > maximum iterations=2 >> >> > tolerances: relative=1e-05, absolute=1e-50, >> divergence=10000. >> >> > left preconditioning >> >> > using nonzero initial guess >> >> > using NONE norm type for convergence test >> >> > PC Object: (mg_levels_2_) 1 MPI processes >> >> > type: sor >> >> > SOR: type = local_symmetric, iterations = 1, local >> iterations = 1, omega = 1. >> >> > linear system matrix = precond matrix: >> >> > Mat Object: 1 MPI processes >> >> > type: seqaij >> >> > rows=256, cols=256 >> >> > total: nonzeros=1248, allocated nonzeros=1248 >> >> > total number of mallocs used during MatSetValues calls =0 >> >> > not using I-node routines >> >> > Up solver (post-smoother) same as down solver (pre-smoother) >> >> > Down solver (pre-smoother) on level 3 >> ------------------------------- >> >> > KSP Object: (mg_levels_3_) 1 MPI processes >> >> > type: richardson >> >> > Richardson: damping factor=1. >> >> > maximum iterations=2 >> >> > tolerances: relative=1e-05, absolute=1e-50, >> divergence=10000. >> >> > left preconditioning >> >> > using nonzero initial guess >> >> > using NONE norm type for convergence test >> >> > PC Object: (mg_levels_3_) 1 MPI processes >> >> > type: sor >> >> > SOR: type = local_symmetric, iterations = 1, local >> iterations = 1, omega = 1. >> >> > linear system matrix = precond matrix: >> >> > Mat Object: 1 MPI processes >> >> > type: seqaij >> >> > rows=1024, cols=1024 >> >> > total: nonzeros=5056, allocated nonzeros=5056 >> >> > total number of mallocs used during MatSetValues calls =0 >> >> > has attached null space >> >> > not using I-node routines >> >> > Up solver (post-smoother) same as down solver (pre-smoother) >> >> > linear system matrix = precond matrix: >> >> > Mat Object: 1 MPI processes >> >> > type: seqaij >> >> > rows=1024, cols=1024 >> >> > total: nonzeros=5056, allocated nonzeros=5056 >> >> > total number of mallocs used during MatSetValues calls =0 >> >> > has attached null space >> >> > not using I-node routines >> >> > >> >> > >> >> > Michele >> >> > >> >> > >> >> > >> >> > >> >> > On Wed, 2016-02-10 at 19:37 -0600, Matthew Knepley wrote: >> >> >> On Wed, Feb 10, 2016 at 7:33 PM, Michele Rosso >> > wrote: >> >> >> Hi, >> >> >> >> >> >> I encountered the following error while solving a >> symmetric positive defined system: >> >> >> >> >> >> Linear solve did not converge due to >> DIVERGED_PCSETUP_FAILED iterations 0 >> >> >> PCSETUP_FAILED due to SUBPC_ERROR >> >> >> >> >> >> This error appears only if I use the optimized version >> of both petsc and my code ( compiler: gfortran, flags: -O3 ). >> >> >> It is weird since I am solving a time-dependent problem >> and everything, i.e. results and convergence rate, are as >> expected until the above error shows up. If I run both petsc >> and my code in debug mode, everything goes smooth till the >> end of the simulation. >> >> >> However, if I reduce the ksp_rtol, even the debug run >> fails, after running as expected for a while, because of a >> KSP_DIVERGED_INDEFINITE_PC . >> >> >> The options I am using are: >> >> >> >> >> >> -ksp_type cg >> >> >> -ksp_norm_type unpreconditioned >> >> >> -ksp_rtol 1e-8 >> >> >> -ksp_lag_norm >> >> >> -ksp_initial_guess_nonzero yes >> >> >> -pc_type mg >> >> >> -pc_mg_galerkin >> >> >> -pc_mg_levels 4 >> >> >> -mg_levels_ksp_type richardson >> >> >> -mg_coarse_ksp_constant_null_space >> >> >> -mg_coarse_pc_type lu >> >> >> -mg_coarse_pc_factor_mat_solver_package superlu_dist >> >> >> -options_left >> >> >> >> >> >> I attached a copy of ksp_view. I am currently using >> petsc-master (last updated yesterday). >> >> >> I would appreciate any suggestion on this matter. >> >> >> >> >> >> >> >> >> >> >> >> I suspect you have a nonlinear PC. Can you send the >> output of -ksp_view? >> >> >> >> >> >> >> >> >> Matt >> >> >> >> >> >> Thanks, >> >> >> Michele >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> What most experimenters take for granted before they >> begin their experiments is infinitely more interesting than >> any results to which their experiments lead. >> >> >> -- Norbert Wiener >> >> > >> >> > >> >> >> >> >> >> >> > >> > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From borisbou at buffalo.edu Thu Feb 11 13:36:07 2016 From: borisbou at buffalo.edu (Boris Boutkov) Date: Thu, 11 Feb 2016 14:36:07 -0500 Subject: [petsc-users] Providing Interpolation/Injections to Multigrid infrastructure Message-ID: <56BCE2A7.8020203@buffalo.edu> Hello All, I'm currently working on hooking into the PETSc multi-grid infrastructure by creating my own DMShell and providing my own interpolation and injection operators. The issue I am currently facing is that while PCSetUp_MG is running, PETSc begins by attempting to create an interpolation (through DMCreateInterpolation) for the finest grid that I am passing it. What would be a good way to try and let the PCSetUp_MG procedure know the first mesh it is receiving is the finest one, so there should be no interpolation provided to it? Thanks for your help, Boris From bsmith at mcs.anl.gov Thu Feb 11 13:45:55 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 11 Feb 2016 13:45:55 -0600 Subject: [petsc-users] Providing Interpolation/Injections to Multigrid infrastructure In-Reply-To: <56BCE2A7.8020203@buffalo.edu> References: <56BCE2A7.8020203@buffalo.edu> Message-ID: <557E8E3E-EC3F-4BAE-8DF1-E8025F411968@mcs.anl.gov> > On Feb 11, 2016, at 1:36 PM, Boris Boutkov wrote: > > Hello All, > > I'm currently working on hooking into the PETSc multi-grid infrastructure by creating my own DMShell and providing my own interpolation and injection operators. The issue I am currently facing is that while PCSetUp_MG is running, PETSc begins by attempting to create an interpolation (through DMCreateInterpolation) for the finest grid that I am passing it. > > What would be a good way to try and let the PCSetUp_MG procedure know the first mesh it is receiving is the finest one, so there should be no interpolation provided to it? Boris, I do not understand the question, perhaps it is a matter of language. Say you have two levels total: call them coarse and fine. the PCSetUp_MG will try to create a single interpolation "for/to the fine grid" FROM the coarse grid. This is necessary to do the geometric multigrid. You would then through your DMSHELL provide the function that creates the interpolation matrix. The PCSetUp_MG will not attempt to create an interpolation "for the coarse mesh" since there is no mesh coarser than the coarse mesh. So PCSetUp_MG() should be trying to create only the needed interpolations, so you should not need to tell them not to try to create certain ones. Thanks Barry > > Thanks for your help, > Boris > From waynezw0618 at 163.com Thu Feb 11 21:40:35 2016 From: waynezw0618 at 163.com (Wei Zhang) Date: Fri, 12 Feb 2016 11:40:35 +0800 Subject: [petsc-users] example of Parallel running DG code with dynamic mesh refinement? Message-ID: <20AD3C37-0AFD-4294-B70C-47B6912C3E95@163.com> Dear everyone: I am planning to write a parallel running DG N-S solver where I want to do mesh refinement. I want to know if there is any example I can start with? /Wei From kaushikggg at gmail.com Fri Feb 12 01:51:38 2016 From: kaushikggg at gmail.com (Kaushik Kulkarni) Date: Fri, 12 Feb 2016 13:21:38 +0530 Subject: [petsc-users] Doubts related MatCreateDense() Message-ID: Hi all, Could you help me with my doubts: *Doubt 1*: Initially I tried to create a matrix with MatCreateMPIDense(); I received a compilation error stating that no such function existed. *Doubt 2*: So I continued working with MatCreateDense(). And I set the global size to 10 cross 10. Now when I called the function *MatGetLocalSize(A,&localrow,&localcolumn), *and ran the code with 2 processes the values returned were: *The local matrix size for the process 1 is 5 cross 5The local matrix size for the process 2 is 5 cross 5* How can it be possible that process 1 is only dealing with 25 elements and process two is dealing with 25 elements, while the global matrix contains 100 elements. Thanks, *Kaushik* -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Feb 12 02:58:40 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 12 Feb 2016 02:58:40 -0600 Subject: [petsc-users] Doubts related MatCreateDense() In-Reply-To: References: Message-ID: > On Feb 12, 2016, at 1:51 AM, Kaushik Kulkarni wrote: > > Hi all, > > Could you help me with my doubts: > > Doubt 1: Initially I tried to create a matrix with MatCreateMPIDense(); I received a compilation error stating that no such function existed. The name was changed > > Doubt 2: So I continued working with MatCreateDense(). And I set the global size to 10 cross 10. Now when I called the function MatGetLocalSize(A,&localrow,&localcolumn), and ran the code with 2 processes the values returned were: > The local matrix size for the process 1 is 5 cross 5 > The local matrix size for the process 2 is 5 cross 5 > How can it be possible that process 1 is only dealing with 25 elements and process two is dealing with 25 elements, while the global matrix contains 100 elements. The local size for columns is slightly mis-leading. For standard PETSc matrices such as "Dense" each process stores all the entries for its rows of the matrix. The term "local columns" refers to the rows of the vector which one can use to do a matrix vector product with the matrix. See the users manual for more details on the layout of vectors and matrices in PETSc. Barry > > Thanks, > Kaushik From kaushikggg at gmail.com Fri Feb 12 03:10:00 2016 From: kaushikggg at gmail.com (Kaushik Kulkarni) Date: Fri, 12 Feb 2016 14:40:00 +0530 Subject: [petsc-users] Doubts related MatCreateDense() In-Reply-To: References: Message-ID: Thanks Barry, Just one more doubt, does it mean that PETSc divides the global matrix among various processes bases on the rows, and no "ACTUAL" division of ?columns? occur ??? *Kaushik* On Fri, Feb 12, 2016 at 2:28 PM, Barry Smith wrote: > > > On Feb 12, 2016, at 1:51 AM, Kaushik Kulkarni > wrote: > > > > Hi all, > > > > Could you help me with my doubts: > > > > Doubt 1: Initially I tried to create a matrix with MatCreateMPIDense(); > I received a compilation error stating that no such function existed. > > The name was changed > > > > Doubt 2: So I continued working with MatCreateDense(). And I set the > global size to 10 cross 10. Now when I called the function > MatGetLocalSize(A,&localrow,&localcolumn), and ran the code with 2 > processes the values returned were: > > The local matrix size for the process 1 is 5 cross 5 > > The local matrix size for the process 2 is 5 cross 5 > > How can it be possible that process 1 is only dealing with 25 elements > and process two is dealing with 25 elements, while the global matrix > contains 100 elements. > > The local size for columns is slightly mis-leading. For standard PETSc > matrices such as "Dense" each process stores all the entries for its rows > of the matrix. The term "local columns" refers to the rows of the vector > which one can use to do a matrix vector product with the matrix. See the > users manual for more details on the layout of vectors and matrices in > PETSc. > > Barry > > > > > Thanks, > > Kaushik > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Feb 12 03:16:03 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 12 Feb 2016 03:16:03 -0600 Subject: [petsc-users] Doubts related MatCreateDense() In-Reply-To: References: Message-ID: <0522F63F-228A-4A4B-AAA4-B43CFD2B44F5@mcs.anl.gov> > On Feb 12, 2016, at 3:10 AM, Kaushik Kulkarni wrote: > > Thanks Barry, > Just one more doubt, does it mean that PETSc divides the global matrix among various processes bases on the rows, and no "ACTUAL" division of ?columns? occur??? For the PETSc matrices yes. But when it uses external packages such as Elemental that is not the case. Depending on what you are doing with the dense matrices it may be better for you to use the MATELEMENTAL http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MATELEMENTAL.html format. Barry > > Kaushik > On Fri, Feb 12, 2016 at 2:28 PM, Barry Smith wrote: > > > On Feb 12, 2016, at 1:51 AM, Kaushik Kulkarni wrote: > > > > Hi all, > > > > Could you help me with my doubts: > > > > Doubt 1: Initially I tried to create a matrix with MatCreateMPIDense(); I received a compilation error stating that no such function existed. > > The name was changed > > > > Doubt 2: So I continued working with MatCreateDense(). And I set the global size to 10 cross 10. Now when I called the function MatGetLocalSize(A,&localrow,&localcolumn), and ran the code with 2 processes the values returned were: > > The local matrix size for the process 1 is 5 cross 5 > > The local matrix size for the process 2 is 5 cross 5 > > How can it be possible that process 1 is only dealing with 25 elements and process two is dealing with 25 elements, while the global matrix contains 100 elements. > > The local size for columns is slightly mis-leading. For standard PETSc matrices such as "Dense" each process stores all the entries for its rows of the matrix. The term "local columns" refers to the rows of the vector which one can use to do a matrix vector product with the matrix. See the users manual for more details on the layout of vectors and matrices in PETSc. > > Barry > > > > > Thanks, > > Kaushik > > From mailinglists at xgm.de Fri Feb 12 03:26:24 2016 From: mailinglists at xgm.de (Florian Lindner) Date: Fri, 12 Feb 2016 10:26:24 +0100 Subject: [petsc-users] Viewer Draw just flickers Message-ID: <3603477.9L8cqCz3Y2@asaru> Hello, I have this code to draw the structure of a matrix: PetscErrorCode ierr; PetscViewer viewer; ierr = PetscViewerCreate(communicator, &viewer); CHKERRV(ierr); ierr = PetscViewerSetType(viewer, PETSCVIEWERDRAW); CHKERRV(ierr); ierr = MatView(matrix, viewer); CHKERRV(ierr); ierr = PetscViewerDestroy(&viewer); CHKERRV(ierr); The window just opens for a fraction of a second and closes again. How can I keep it open until the user closes it? -draw_pause -1 seems to do the trick, but I don't want to use a command line option every time. PetscDrawSetPause(PetscDraw draw,PetscReal lpause) and PetscDrawPause both need a PetscDraw object. Where do I get this from? Best, Florian From knepley at gmail.com Fri Feb 12 06:31:28 2016 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 12 Feb 2016 06:31:28 -0600 Subject: [petsc-users] Viewer Draw just flickers In-Reply-To: <3603477.9L8cqCz3Y2@asaru> References: <3603477.9L8cqCz3Y2@asaru> Message-ID: On Fri, Feb 12, 2016 at 3:26 AM, Florian Lindner wrote: > Hello, > > I have this code to draw the structure of a matrix: > > PetscErrorCode ierr; > PetscViewer viewer; > ierr = PetscViewerCreate(communicator, &viewer); CHKERRV(ierr); > ierr = PetscViewerSetType(viewer, PETSCVIEWERDRAW); CHKERRV(ierr); > ierr = MatView(matrix, viewer); CHKERRV(ierr); > ierr = PetscViewerDestroy(&viewer); CHKERRV(ierr); > > The window just opens for a fraction of a second and closes again. How can > I keep it open until the user closes it? > > -draw_pause -1 seems to do the trick, but I don't want to use a command > line option every time. > > PetscDrawSetPause(PetscDraw draw,PetscReal lpause) and PetscDrawPause both > need a PetscDraw object. Where do I get this from? > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerDrawGetDraw.html#PetscViewerDrawGetDraw Matt > Best, > Florian > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From kaushikggg at gmail.com Fri Feb 12 08:56:46 2016 From: kaushikggg at gmail.com (Kaushik Kulkarni) Date: Fri, 12 Feb 2016 20:26:46 +0530 Subject: [petsc-users] Doubts related MatCreateDense() In-Reply-To: <0522F63F-228A-4A4B-AAA4-B43CFD2B44F5@mcs.anl.gov> References: <0522F63F-228A-4A4B-AAA4-B43CFD2B44F5@mcs.anl.gov> Message-ID: So Barry does it mean that currently PETSc does not support a parallel implementation of Dense Matrices. If it does could you please provide me a link where could I find a proper documentation for the same. Thanks, Kaushik On Fri, Feb 12, 2016 at 2:46 PM, Barry Smith wrote: > > > On Feb 12, 2016, at 3:10 AM, Kaushik Kulkarni > wrote: > > > > Thanks Barry, > > Just one more doubt, does it mean that PETSc divides the global matrix > among various processes bases on the rows, and no "ACTUAL" division of > ?columns? occur??? > > For the PETSc matrices yes. But when it uses external packages such as > Elemental that is not the case. Depending on what you are doing with the > dense matrices it may be better for you to use the MATELEMENTAL > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MATELEMENTAL.html > format. > > > Barry > > > > > Kaushik > > On Fri, Feb 12, 2016 at 2:28 PM, Barry Smith wrote: > > > > > On Feb 12, 2016, at 1:51 AM, Kaushik Kulkarni > wrote: > > > > > > Hi all, > > > > > > Could you help me with my doubts: > > > > > > Doubt 1: Initially I tried to create a matrix with > MatCreateMPIDense(); I received a compilation error stating that no such > function existed. > > > > The name was changed > > > > > > Doubt 2: So I continued working with MatCreateDense(). And I set the > global size to 10 cross 10. Now when I called the function > MatGetLocalSize(A,&localrow,&localcolumn), and ran the code with 2 > processes the values returned were: > > > The local matrix size for the process 1 is 5 cross 5 > > > The local matrix size for the process 2 is 5 cross 5 > > > How can it be possible that process 1 is only dealing with 25 elements > and process two is dealing with 25 elements, while the global matrix > contains 100 elements. > > > > The local size for columns is slightly mis-leading. For standard PETSc > matrices such as "Dense" each process stores all the entries for its rows > of the matrix. The term "local columns" refers to the rows of the vector > which one can use to do a matrix vector product with the matrix. See the > users manual for more details on the layout of vectors and matrices in > PETSc. > > > > Barry > > > > > > > > Thanks, > > > Kaushik > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Feb 12 09:09:16 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 12 Feb 2016 09:09:16 -0600 Subject: [petsc-users] Doubts related MatCreateDense() In-Reply-To: References: <0522F63F-228A-4A4B-AAA4-B43CFD2B44F5@mcs.anl.gov> Message-ID: <2F25AAB3-0FC0-4F49-9EDE-D39F455D2863@mcs.anl.gov> > On Feb 12, 2016, at 8:56 AM, Kaushik Kulkarni wrote: > > So Barry does it mean that currently PETSc does not support a parallel implementation of Dense Matrices. No I did not say that. I said that the PETSc Dense matrix format does not divide columns across processes. Each process has certain rows of the matrix. The MATELEMENTAL format does a so-called cyclic decomposition of the dense matrix which is good for parallel algorithms such as dense LU factorization. This format is unfortunately more cumbersome to work with. So depending on what you want you want to do with your matrix you would pick one of the two formats. If, in fact, you do not have a matrix, but merely a two (or more) dimensional array (that represents, for example, unknowns on a finite difference mesh, sometimes called field variables) then you should not be using Mat at all (since that is for linear operators, not arrays) and you should be using Vec's for your field variables. The DMDA routines (for example DMDACreate2d()) are useful utility routines for creating Vec's that you want to treat as multidimensional arrays. Barry > If it does could you please provide me a link where could I find a proper documentation for the same. > Thanks, > Kaushik > > On Fri, Feb 12, 2016 at 2:46 PM, Barry Smith wrote: > > > On Feb 12, 2016, at 3:10 AM, Kaushik Kulkarni wrote: > > > > Thanks Barry, > > Just one more doubt, does it mean that PETSc divides the global matrix among various processes bases on the rows, and no "ACTUAL" division of ?columns? occur??? > > For the PETSc matrices yes. But when it uses external packages such as Elemental that is not the case. Depending on what you are doing with the dense matrices it may be better for you to use the MATELEMENTAL http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MATELEMENTAL.html format. > > > Barry > > > > > Kaushik > > On Fri, Feb 12, 2016 at 2:28 PM, Barry Smith wrote: > > > > > On Feb 12, 2016, at 1:51 AM, Kaushik Kulkarni wrote: > > > > > > Hi all, > > > > > > Could you help me with my doubts: > > > > > > Doubt 1: Initially I tried to create a matrix with MatCreateMPIDense(); I received a compilation error stating that no such function existed. > > > > The name was changed > > > > > > Doubt 2: So I continued working with MatCreateDense(). And I set the global size to 10 cross 10. Now when I called the function MatGetLocalSize(A,&localrow,&localcolumn), and ran the code with 2 processes the values returned were: > > > The local matrix size for the process 1 is 5 cross 5 > > > The local matrix size for the process 2 is 5 cross 5 > > > How can it be possible that process 1 is only dealing with 25 elements and process two is dealing with 25 elements, while the global matrix contains 100 elements. > > > > The local size for columns is slightly mis-leading. For standard PETSc matrices such as "Dense" each process stores all the entries for its rows of the matrix. The term "local columns" refers to the rows of the vector which one can use to do a matrix vector product with the matrix. See the users manual for more details on the layout of vectors and matrices in PETSc. > > > > Barry > > > > > > > > Thanks, > > > Kaushik > > > > > > From ptbauman at gmail.com Fri Feb 12 09:20:17 2016 From: ptbauman at gmail.com (Paul T. Bauman) Date: Fri, 12 Feb 2016 10:20:17 -0500 Subject: [petsc-users] Providing Interpolation/Injections to Multigrid infrastructure In-Reply-To: <557E8E3E-EC3F-4BAE-8DF1-E8025F411968@mcs.anl.gov> References: <56BCE2A7.8020203@buffalo.edu> <557E8E3E-EC3F-4BAE-8DF1-E8025F411968@mcs.anl.gov> Message-ID: Hi Barry, On Thu, Feb 11, 2016 at 2:45 PM, Barry Smith wrote: > > > On Feb 11, 2016, at 1:36 PM, Boris Boutkov wrote: > > > > Hello All, > > > > I'm currently working on hooking into the PETSc multi-grid > infrastructure by creating my own DMShell and providing my own > interpolation and injection operators. The issue I am currently facing is > that while PCSetUp_MG is running, PETSc begins by attempting to create an > interpolation (through DMCreateInterpolation) for the finest grid that I am > passing it. > > > > What would be a good way to try and let the PCSetUp_MG procedure know > the first mesh it is receiving is the finest one, so there should be no > interpolation provided to it? > > Boris, > > I do not understand the question, perhaps it is a matter of language. Say you have two levels total: call them coarse and fine. the PCSetUp_MG > will try to create a single interpolation "for/to the fine grid" FROM the > coarse grid. This is necessary to do the geometric multigrid. You would > then through your DMSHELL provide the function that creates the > interpolation matrix. The PCSetUp_MG will not attempt to create an > interpolation "for the coarse mesh" since there is no mesh coarser than the > coarse mesh. So > PCSetUp_MG() should be trying to create only the needed interpolations, so > you should not need to tell them not to try to create certain ones. The information I think we're missing is, more or less, the order of operations during the MG setup and how PETSc is expecting us to behave with our DMShell implementation. I'll stick with your coarse and fine grid. On our side, we can construct the coarse and the fine grids, supplying the DMs and accompanying PetscSections, at simulation setup time since we know what our grid hierarchy is (we're focusing on hierarchical meshes to start with). Then, for DMRefine and DMCoarsen, we can just return the correct DM that we've built. (Maybe this the wrong way to do it?) We also have implementations of DMCreateInterpolation and DMCreateInjection for moving between each of the levels. Our main confusion is, once we've set up of all that, which of the DMs we've constructed do we go ahead and give to SNES? The fine or the coarse? Is it expected that we should be checking the DM supplied to, e.g., DMCreateInterpolation for which of the operators PETSc is wanting? We were guessing that we should supply the fine DM to SNES (and then not expecting DMCreateInterpolation to be called yet). Clearly, there's some setup steps we're missing or we're just going about this all the wrong way. Is there an existing example of setting up a problem using a DMShell? I poked a little but couldn't find one. Thanks much. Best, Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Feb 12 09:30:49 2016 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 12 Feb 2016 09:30:49 -0600 Subject: [petsc-users] Doubts related MatCreateDense() In-Reply-To: References: <0522F63F-228A-4A4B-AAA4-B43CFD2B44F5@mcs.anl.gov> Message-ID: On Fri, Feb 12, 2016 at 8:56 AM, Kaushik Kulkarni wrote: > So Barry does it mean that currently PETSc does not support a parallel > implementation of Dense Matrices. If it does could you please provide me a > link where could I find a proper documentation for the same. > No, http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MATDENSE.html http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateDense.html Matt > Thanks, > Kaushik > > On Fri, Feb 12, 2016 at 2:46 PM, Barry Smith wrote: > >> >> > On Feb 12, 2016, at 3:10 AM, Kaushik Kulkarni >> wrote: >> > >> > Thanks Barry, >> > Just one more doubt, does it mean that PETSc divides the global matrix >> among various processes bases on the rows, and no "ACTUAL" division of >> ?columns? occur??? >> >> For the PETSc matrices yes. But when it uses external packages such as >> Elemental that is not the case. Depending on what you are doing with the >> dense matrices it may be better for you to use the MATELEMENTAL >> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MATELEMENTAL.html >> format. >> >> >> Barry >> >> > >> > Kaushik >> > On Fri, Feb 12, 2016 at 2:28 PM, Barry Smith >> wrote: >> > >> > > On Feb 12, 2016, at 1:51 AM, Kaushik Kulkarni >> wrote: >> > > >> > > Hi all, >> > > >> > > Could you help me with my doubts: >> > > >> > > Doubt 1: Initially I tried to create a matrix with >> MatCreateMPIDense(); I received a compilation error stating that no such >> function existed. >> > >> > The name was changed >> > > >> > > Doubt 2: So I continued working with MatCreateDense(). And I set the >> global size to 10 cross 10. Now when I called the function >> MatGetLocalSize(A,&localrow,&localcolumn), and ran the code with 2 >> processes the values returned were: >> > > The local matrix size for the process 1 is 5 cross 5 >> > > The local matrix size for the process 2 is 5 cross 5 >> > > How can it be possible that process 1 is only dealing with 25 >> elements and process two is dealing with 25 elements, while the global >> matrix contains 100 elements. >> > >> > The local size for columns is slightly mis-leading. For standard >> PETSc matrices such as "Dense" each process stores all the entries for its >> rows of the matrix. The term "local columns" refers to the rows of the >> vector which one can use to do a matrix vector product with the matrix. See >> the users manual for more details on the layout of vectors and matrices in >> PETSc. >> > >> > Barry >> > >> > > >> > > Thanks, >> > > Kaushik >> > >> > >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From lawrence.mitchell at imperial.ac.uk Fri Feb 12 09:32:13 2016 From: lawrence.mitchell at imperial.ac.uk (Lawrence Mitchell) Date: Fri, 12 Feb 2016 15:32:13 +0000 Subject: [petsc-users] Providing Interpolation/Injections to Multigrid infrastructure In-Reply-To: References: <56BCE2A7.8020203@buffalo.edu> <557E8E3E-EC3F-4BAE-8DF1-E8025F411968@mcs.anl.gov> Message-ID: <05A26E59-C074-469D-8308-3BF75F634181@imperial.ac.uk> > On 12 Feb 2016, at 15:20, Paul T. Bauman wrote: > > Hi Barry, > > On Thu, Feb 11, 2016 at 2:45 PM, Barry Smith wrote: > > > On Feb 11, 2016, at 1:36 PM, Boris Boutkov wrote: > > > > Hello All, > > > > I'm currently working on hooking into the PETSc multi-grid infrastructure by creating my own DMShell and providing my own interpolation and injection operators. The issue I am currently facing is that while PCSetUp_MG is running, PETSc begins by attempting to create an interpolation (through DMCreateInterpolation) for the finest grid that I am passing it. > > > > What would be a good way to try and let the PCSetUp_MG procedure know the first mesh it is receiving is the finest one, so there should be no interpolation provided to it? > > Boris, > > I do not understand the question, perhaps it is a matter of language. > Say you have two levels total: call them coarse and fine. the PCSetUp_MG will try to create a single interpolation "for/to the fine grid" FROM the coarse grid. This is necessary to do the geometric multigrid. You would then through your DMSHELL provide the function that creates the interpolation matrix. The PCSetUp_MG will not attempt to create an interpolation "for the coarse mesh" since there is no mesh coarser than the coarse mesh. So > PCSetUp_MG() should be trying to create only the needed interpolations, so you should not need to tell them not to try to create certain ones. > > The information I think we're missing is, more or less, the order of operations during the MG setup and how PETSc is expecting us to behave with our DMShell implementation. I'll stick with your coarse and fine grid. On our side, we can construct the coarse and the fine grids, supplying the DMs and accompanying PetscSections, at simulation setup time since we know what our grid hierarchy is (we're focusing on hierarchical meshes to start with). Then, for DMRefine and DMCoarsen, we can just return the correct DM that we've built. (Maybe this the wrong way to do it?) We also have implementations of DMCreateInterpolation and DMCreateInjection for moving between each of the levels. > > Our main confusion is, once we've set up of all that, which of the DMs we've constructed do we go ahead and give to SNES? The fine or the coarse? Is it expected that we should be checking the DM supplied to, e.g., DMCreateInterpolation for which of the operators PETSc is wanting? We were guessing that we should supply the fine DM to SNES (and then not expecting DMCreateInterpolation to be called yet). > > Clearly, there's some setup steps we're missing or we're just going about this all the wrong way. Is there an existing example of setting up a problem using a DMShell? I poked a little but couldn't find one. AIUI, PETSc needs to know how many refinements generated the fine DM that is attached to the operator inside the PC. That means, I think, you need to call DMRefine on the coarse DM (and so on all the way down until you get to the fine DM). Cheers, Lawrence -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 455 bytes Desc: Message signed with OpenPGP using GPGMail URL: From knepley at gmail.com Fri Feb 12 09:32:59 2016 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 12 Feb 2016 09:32:59 -0600 Subject: [petsc-users] Providing Interpolation/Injections to Multigrid infrastructure In-Reply-To: References: <56BCE2A7.8020203@buffalo.edu> <557E8E3E-EC3F-4BAE-8DF1-E8025F411968@mcs.anl.gov> Message-ID: On Fri, Feb 12, 2016 at 9:20 AM, Paul T. Bauman wrote: > Hi Barry, > > On Thu, Feb 11, 2016 at 2:45 PM, Barry Smith wrote: > >> >> > On Feb 11, 2016, at 1:36 PM, Boris Boutkov >> wrote: >> > >> > Hello All, >> > >> > I'm currently working on hooking into the PETSc multi-grid >> infrastructure by creating my own DMShell and providing my own >> interpolation and injection operators. The issue I am currently facing is >> that while PCSetUp_MG is running, PETSc begins by attempting to create an >> interpolation (through DMCreateInterpolation) for the finest grid that I am >> passing it. >> > >> > What would be a good way to try and let the PCSetUp_MG procedure know >> the first mesh it is receiving is the finest one, so there should be no >> interpolation provided to it? >> >> Boris, >> >> I do not understand the question, perhaps it is a matter of language. > > Say you have two levels total: call them coarse and fine. the PCSetUp_MG >> will try to create a single interpolation "for/to the fine grid" FROM the >> coarse grid. This is necessary to do the geometric multigrid. You would >> then through your DMSHELL provide the function that creates the >> interpolation matrix. The PCSetUp_MG will not attempt to create an >> interpolation "for the coarse mesh" since there is no mesh coarser than the >> coarse mesh. So >> PCSetUp_MG() should be trying to create only the needed interpolations, >> so you should not need to tell them not to try to create certain ones. > > > The information I think we're missing is, more or less, the order of > operations during the MG setup and how PETSc is expecting us to behave with > our DMShell implementation. I'll stick with your coarse and fine grid. On > our side, we can construct the coarse and the fine grids, supplying the DMs > and accompanying PetscSections, at simulation setup time since we know what > our grid hierarchy is (we're focusing on hierarchical meshes to start > with). Then, for DMRefine and DMCoarsen, we can just return the correct DM > that we've built. (Maybe this the wrong way to do it?) We also have > implementations of DMCreateInterpolation and DMCreateInjection for moving > between each of the levels. > > Our main confusion is, once we've set up of all that, which of the DMs > we've constructed do we go ahead and give to SNES? The fine or the coarse? > Is it expected that we should be checking the DM supplied to, e.g., > DMCreateInterpolation for which of the operators PETSc is wanting? We were > guessing that we should supply the fine DM to SNES (and then not expecting > DMCreateInterpolation to be called yet). > Ah, good question. I think it should be the fine grid, and this is the default for FAS. Some stuff used to take a coarse grid because DMDA was easier to refine, but that should all be fixed now. Also, it looks like Lawrence responded. Matt > Clearly, there's some setup steps we're missing or we're just going about > this all the wrong way. Is there an existing example of setting up a > problem using a DMShell? I poked a little but couldn't find one. > > Thanks much. > > Best, > > Paul > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Feb 12 09:37:40 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 12 Feb 2016 09:37:40 -0600 Subject: [petsc-users] Providing Interpolation/Injections to Multigrid infrastructure In-Reply-To: References: <56BCE2A7.8020203@buffalo.edu> <557E8E3E-EC3F-4BAE-8DF1-E8025F411968@mcs.anl.gov> Message-ID: <7E2629FB-9F9E-49A7-B380-6C92BA788D27@mcs.anl.gov> > On Feb 12, 2016, at 9:20 AM, Paul T. Bauman wrote: > > Hi Barry, > > On Thu, Feb 11, 2016 at 2:45 PM, Barry Smith wrote: > > > On Feb 11, 2016, at 1:36 PM, Boris Boutkov wrote: > > > > Hello All, > > > > I'm currently working on hooking into the PETSc multi-grid infrastructure by creating my own DMShell and providing my own interpolation and injection operators. The issue I am currently facing is that while PCSetUp_MG is running, PETSc begins by attempting to create an interpolation (through DMCreateInterpolation) for the finest grid that I am passing it. > > > > What would be a good way to try and let the PCSetUp_MG procedure know the first mesh it is receiving is the finest one, so there should be no interpolation provided to it? > > Boris, > > I do not understand the question, perhaps it is a matter of language. > Say you have two levels total: call them coarse and fine. the PCSetUp_MG will try to create a single interpolation "for/to the fine grid" FROM the coarse grid. This is necessary to do the geometric multigrid. You would then through your DMSHELL provide the function that creates the interpolation matrix. The PCSetUp_MG will not attempt to create an interpolation "for the coarse mesh" since there is no mesh coarser than the coarse mesh. So > PCSetUp_MG() should be trying to create only the needed interpolations, so you should not need to tell them not to try to create certain ones. > > The information I think we're missing is, more or less, the order of operations during the MG setup and how PETSc is expecting us to behave with our DMShell implementation. I'll stick with your coarse and fine grid. On our side, we can construct the coarse and the fine grids, supplying the DMs and accompanying PetscSections, at simulation setup time since we know what our grid hierarchy is (we're focusing on hierarchical meshes to start with). Then, for DMRefine and DMCoarsen, we can just return the correct DM that we've built. (Maybe this the wrong way to do it?) We also have implementations of DMCreateInterpolation and DMCreateInjection for moving between each of the levels. This is an ok way to do it. > > Our main confusion is, once we've set up of all that, which of the DMs we've constructed do we go ahead and give to SNES? The fine or the coarse? You pass the finest to SNES because that is the one SNES uses for the nonlinear solve. The linear multigrid solver will then call your DMCoarsen() to get the coarser DMs, so each time it is called you just pass the next coarser one from the DM that is the input argument to your DMCoarsen. > Is it expected that we should be checking the DM supplied to, e.g., DMCreateInterpolation for which of the operators PETSc is wanting? Absolutely. It gets passed two DMS and based on those two DMS you need to pass back the correct interpolation. > We were guessing that we should supply the fine DM to SNES (and then not expecting DMCreateInterpolation to be called yet). > > Clearly, there's some setup steps we're missing or we're just going about this all the wrong way. Is there an existing example of setting up a problem using a DMShell? I poked a little but couldn't find one. Unfortunately not. You are over thinking this. It is actually not complicated or tricky. You might try running the two level one in the debugger (on one process) and put break points in all your DMSHELL call back functions, then you can check exactly what arguments they are being called with. Barry > > Thanks much. > > Best, > > Paul From amneetb at live.unc.edu Fri Feb 12 12:16:05 2016 From: amneetb at live.unc.edu (Bhalla, Amneet Pal S) Date: Fri, 12 Feb 2016 18:16:05 +0000 Subject: [petsc-users] Extracting CSR format from matrices for CUDA Message-ID: Hi Folks, I want to extract the CSR format from PETSc matrices and ship it to CUDA. Is there an easy way of doing this? If so is there an example/source code showing this extraction? Thanks, ? Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Fri Feb 12 12:21:34 2016 From: dave.mayhem23 at gmail.com (Dave May) Date: Fri, 12 Feb 2016 19:21:34 +0100 Subject: [petsc-users] Extracting CSR format from matrices for CUDA In-Reply-To: References: Message-ID: On 12 February 2016 at 19:16, Bhalla, Amneet Pal S wrote: > Hi Folks, > > I want to extract the CSR format from PETSc matrices and ship it to CUDA. > Is there an easy way of doing this? > Yep, see these web pages http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetRowIJ.html http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetColumnIJ.html http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSeqAIJGetArray.html > If so is there an example/source code showing this extraction? > Not that I know of. But there isn't much to it, the description on these webpages should be sufficient. Thanks, Dave > > Thanks, > > ? Amneet > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Feb 12 12:24:32 2016 From: jed at jedbrown.org (Jed Brown) Date: Fri, 12 Feb 2016 11:24:32 -0700 Subject: [petsc-users] Extracting CSR format from matrices for CUDA In-Reply-To: References: Message-ID: <874mddzt0f.fsf@jedbrown.org> Dave May writes: >> If so is there an example/source code showing this extraction? >> > > Not that I know of. > But there isn't much to it, the description on these webpages should be > sufficient. See also the CUSP and ViennaCL matrix classes, which can run on CUDA. But also note that it stands a good chance of not solving your problem any faster, unless you have a very special problem. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From knepley at gmail.com Fri Feb 12 15:01:08 2016 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 12 Feb 2016 15:01:08 -0600 Subject: [petsc-users] Extracting CSR format from matrices for CUDA In-Reply-To: <874mddzt0f.fsf@jedbrown.org> References: <874mddzt0f.fsf@jedbrown.org> Message-ID: On Fri, Feb 12, 2016 at 12:24 PM, Jed Brown wrote: > Dave May writes: > >> If so is there an example/source code showing this extraction? > >> > > > > Not that I know of. > > But there isn't much to it, the description on these webpages should be > > sufficient. > > See also the CUSP and ViennaCL matrix classes, which can run on CUDA. > But also note that it stands a good chance of not solving your problem > any faster, unless you have a very special problem. > GPU! Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From griffith at cims.nyu.edu Fri Feb 12 15:06:48 2016 From: griffith at cims.nyu.edu (Boyce Griffith) Date: Fri, 12 Feb 2016 16:06:48 -0500 Subject: [petsc-users] Extracting CSR format from matrices for CUDA In-Reply-To: References: <874mddzt0f.fsf@jedbrown.org> Message-ID: <164FFB1F-AB42-4639-AC1D-C8EF4ACD6081@cims.nyu.edu> > On Feb 12, 2016, at 4:01 PM, Matthew Knepley wrote: > > On Fri, Feb 12, 2016 at 12:24 PM, Jed Brown > wrote: > Dave May > writes: > >> If so is there an example/source code showing this extraction? > >> > > > > Not that I know of. > > But there isn't much to it, the description on these webpages should be > > sufficient. > > See also the CUSP and ViennaCL matrix classes, which can run on CUDA. > But also note that it stands a good chance of not solving your problem > any faster, unless you have a very special problem. > > GPU! I think I've convinced Amneet to try out some other things before/instead of heading down this route. -- Boyce -------------- next part -------------- An HTML attachment was scrubbed... URL: From kaushikggg at gmail.com Sat Feb 13 02:19:36 2016 From: kaushikggg at gmail.com (Kaushik Kulkarni) Date: Sat, 13 Feb 2016 13:49:36 +0530 Subject: [petsc-users] Doubt related to MatSetValues() Message-ID: After preallocating a matrix do I have to change the way in which I have to use the function MatSetValues(). The reason I am asking this is that when I am executing the code with the line " *MatMPIAIJSetPreallocation(A,dnz,PETSC_NULL,0,PETSC_NULL);*" in which dnz is computed for each row and all the off diagonal terms are zero I am getting an error: " *[0]PETSC ERROR:Argument out of range[0]PETSC ERROR: New nonzero at (0,4) caused a malloc*" But when I am commenting the line corresponding to the Preallocation, the code is running fine. Thank you, *Kaushik* -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Feb 13 09:44:02 2016 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 13 Feb 2016 09:44:02 -0600 Subject: [petsc-users] Doubt related to MatSetValues() In-Reply-To: References: Message-ID: On Sat, Feb 13, 2016 at 2:19 AM, Kaushik Kulkarni wrote: > After preallocating a matrix do I have to change the way in which I have > to use the function MatSetValues(). The reason I am asking this is that > when I am executing the code with the line " > *MatMPIAIJSetPreallocation(A,dnz,PETSC_NULL,0,PETSC_NULL);*" in which dnz > is computed for each row and all the off diagonal terms are zero I am > getting an error: > " > *[0]PETSC ERROR:Argument out of range[0]PETSC ERROR: New nonzero at (0,4) > caused a malloc*" > It sounds like (0,4) is an off-diagonal. Matt > But when I am commenting the line corresponding to the Preallocation, the > code is running fine. > > Thank you, > > *Kaushik* > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bikash at umich.edu Sat Feb 13 16:43:55 2016 From: bikash at umich.edu (Bikash Kanungo) Date: Sat, 13 Feb 2016 17:43:55 -0500 Subject: [petsc-users] Segmentation Fault in MatAssembly Message-ID: Hi, I have a small (~ 2500x2500) dense matrix, say A, parallelized over 480 procs and stored as MPIAIJ instead of MPIDENSE. This is because I have to perform MatMatMult of A with an MPIDENSE matrix, say B. Since MatMatMult is not supported if both A and B are dense, I'm storing A as MPIAIJ. The entries in A are inserted only by processor 0 through MatSetValues. Now while assembling A, I sometimes get segmentation fault. This error is random,it shows up in one run and doesn't in another. I understand that it doesn't make much sense to distribute such a small matrix over 480 procs, but what can be a possible reason for such random segmentation faults? Regards, Bikash -- Bikash S. Kanungo PhD Student Computational Materials Physics Group Mechanical Engineering University of Michigan -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Feb 13 17:03:33 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 13 Feb 2016 17:03:33 -0600 Subject: [petsc-users] Segmentation Fault in MatAssembly In-Reply-To: References: Message-ID: <15B59285-689A-4794-A943-EC02956146D5@mcs.anl.gov> Run with valgrind http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > On Feb 13, 2016, at 4:43 PM, Bikash Kanungo wrote: > > Hi, > > I have a small (~ 2500x2500) dense matrix, say A, parallelized over 480 procs and stored as MPIAIJ instead of MPIDENSE. This is because I have to perform MatMatMult of A with an MPIDENSE matrix, say B. Since MatMatMult is not supported if both A and B are dense, I'm storing A as MPIAIJ. The entries in A are inserted only by processor 0 through MatSetValues. Now while assembling A, I sometimes get segmentation fault. This error is random,it shows up in one run and doesn't in another. I understand that it doesn't make much sense to distribute such a small matrix over 480 procs, but what can be a possible reason for such random segmentation faults? > > Regards, > Bikash > > -- > Bikash S. Kanungo > PhD Student > Computational Materials Physics Group > Mechanical Engineering > University of Michigan > From jcrean01 at gmail.com Mon Feb 15 22:31:04 2016 From: jcrean01 at gmail.com (Jared Crean) Date: Mon, 15 Feb 2016 23:31:04 -0500 Subject: [petsc-users] MatGetLocalSubMatrix Trouble Message-ID: <56C2A608.3090506@gmail.com> Hello, When I try to use MatGetLocalSubMatrix (line 55 of the attached code), I get the following error: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Null argument, when expecting valid pointer [0]PETSC ERROR: Null Object: Parameter # 2 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.0, Jun, 09, 2015 [0]PETSC ERROR: ./jc3 on a arch-linux2-cxx-debug named k475 by jared Mon Feb 15 23:19:29 2016 [0]PETSC ERROR: Configure options --with-clanguage=C++ [0]PETSC ERROR: #1 ISL2GCompose() line 117 in /home/jared/build/petsc-3.6.0/src/mat/impls/localref/mlocalref.c [0]PETSC ERROR: #2 MatCreateLocalRef() line 259 in /home/jared/build/petsc-3.6.0/src/mat/impls/localref/mlocalref.c [0]PETSC ERROR: #3 MatGetLocalSubMatrix() line 9686 in /home/jared/build/petsc-3.6.0/src/mat/interface/matrix.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Null argument, when expecting valid pointer [0]PETSC ERROR: Null Object: Parameter # 2 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.0, Jun, 09, 2015 [0]PETSC ERROR: ./jc3 on a arch-linux2-cxx-debug named k475 by jared Mon Feb 15 23:19:29 2016 [0]PETSC ERROR: Configure options --with-clanguage=C++ [0]PETSC ERROR: #4 ISL2GCompose() line 117 in /home/jared/build/petsc-3.6.0/src/mat/impls/localref/mlocalref.c [0]PETSC ERROR: #5 MatCreateLocalRef() line 259 in /home/jared/build/petsc-3.6.0/src/mat/impls/localref/mlocalref.c [0]PETSC ERROR: #6 MatGetLocalSubMatrix() line 9686 in /home/jared/build/petsc-3.6.0/src/mat/interface/matrix.c [0]PETSC ERROR: #7 main() line 55 in /home/jared/build/petsc-3.6.0/src/ksp/ksp/examples/tutorials/jc3.c [0]PETSC ERROR: No PETSc Option Table entries [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- application called MPI_Abort(MPI_COMM_WORLD, 85) - process 0 A did a little digging and found: Line 117-118 of mlocalref.c are: PetscValidHeaderSpecific(ltog,IS_LTOGM_CLASSID,2); PetscValidPointer(cltog,3); where ltog is the ISLocalToGlobalMapping from (line 259): A->rmap->mapping where A is the original matrix, cltog is a reference to the local to global mapping to be created . The code prints out A, so it appears to be a valid matrix. Is there something wrong with the way I am using MatGetLocalSubMatrix? Jared Crean -------------- next part -------------- A non-text attachment was scrubbed... Name: jc3.c Type: text/x-csrc Size: 1705 bytes Desc: not available URL: From bsmith at mcs.anl.gov Mon Feb 15 22:43:05 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 15 Feb 2016 22:43:05 -0600 Subject: [petsc-users] MatGetLocalSubMatrix Trouble In-Reply-To: <56C2A608.3090506@gmail.com> References: <56C2A608.3090506@gmail.com> Message-ID: <67FCA4CF-E86B-4568-B7AA-A5088CAA348B@mcs.anl.gov> Jared, Though it is not properly documented the matrix must have had a call to MatSetLocalToGlobalMapping() before you can call MatCreateLocalRef(). I'll add a little more error checking and docs. Barry Note that matrices obtained with DMCreateMat() generally have had the local to global mapping already provided to the matrix so you don't need to call it yourself. > On Feb 15, 2016, at 10:31 PM, Jared Crean wrote: > > Hello, > When I try to use MatGetLocalSubMatrix (line 55 of the attached code), I get the following error: > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Null argument, when expecting valid pointer > [0]PETSC ERROR: Null Object: Parameter # 2 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.0, Jun, 09, 2015 > [0]PETSC ERROR: ./jc3 on a arch-linux2-cxx-debug named k475 by jared Mon Feb 15 23:19:29 2016 > [0]PETSC ERROR: Configure options --with-clanguage=C++ > [0]PETSC ERROR: #1 ISL2GCompose() line 117 in /home/jared/build/petsc-3.6.0/src/mat/impls/localref/mlocalref.c > [0]PETSC ERROR: #2 MatCreateLocalRef() line 259 in /home/jared/build/petsc-3.6.0/src/mat/impls/localref/mlocalref.c > [0]PETSC ERROR: #3 MatGetLocalSubMatrix() line 9686 in /home/jared/build/petsc-3.6.0/src/mat/interface/matrix.c > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Null argument, when expecting valid pointer > [0]PETSC ERROR: Null Object: Parameter # 2 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.0, Jun, 09, 2015 > [0]PETSC ERROR: ./jc3 on a arch-linux2-cxx-debug named k475 by jared Mon Feb 15 23:19:29 2016 > [0]PETSC ERROR: Configure options --with-clanguage=C++ > [0]PETSC ERROR: #4 ISL2GCompose() line 117 in /home/jared/build/petsc-3.6.0/src/mat/impls/localref/mlocalref.c > [0]PETSC ERROR: #5 MatCreateLocalRef() line 259 in /home/jared/build/petsc-3.6.0/src/mat/impls/localref/mlocalref.c > [0]PETSC ERROR: #6 MatGetLocalSubMatrix() line 9686 in /home/jared/build/petsc-3.6.0/src/mat/interface/matrix.c > [0]PETSC ERROR: #7 main() line 55 in /home/jared/build/petsc-3.6.0/src/ksp/ksp/examples/tutorials/jc3.c > [0]PETSC ERROR: No PETSc Option Table entries > [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- > application called MPI_Abort(MPI_COMM_WORLD, 85) - process 0 > > > A did a little digging and found: > > Line 117-118 of mlocalref.c are: > PetscValidHeaderSpecific(ltog,IS_LTOGM_CLASSID,2); > PetscValidPointer(cltog,3); > > where ltog is the ISLocalToGlobalMapping from (line 259): > A->rmap->mapping > > where A is the original matrix, cltog is a reference to the local to global mapping to be created . > The code prints out A, so it appears to be a valid matrix. Is there something wrong with the way I am using MatGetLocalSubMatrix? > > Jared Crean > > > From rongliang.chan at gmail.com Tue Feb 16 06:45:04 2016 From: rongliang.chan at gmail.com (Rongliang Chen) Date: Tue, 16 Feb 2016 20:45:04 +0800 Subject: [petsc-users] SNESComputeJacobianDefaultColor use too much memory Message-ID: <56C319D0.9060208@gmail.com> Dear all, I am using the DMPlex to solve a PDE on a unstructured mesh and I use the SNESComputeJacobianDefaultColor to compute the Jacobian matrix. My code works well for small problems (such as problem with 3.3x10^5 cells using 120 cores) but when I increase the number of cells (2.6x10^6 cells using 1920 cores), I go out of memory in the function MatColoringApply. It shows that one of the cores uses over 11G memory. I think this is unreasonable. Do you have any suggestions for debugging this problem? Best regards, Rongliang From mailinglists at xgm.de Tue Feb 16 08:55:25 2016 From: mailinglists at xgm.de (Florian Lindner) Date: Tue, 16 Feb 2016 15:55:25 +0100 Subject: [petsc-users] Inverse local to global mapping? Message-ID: <4329728.aZqEON1xM9@asaru> Hello, I want to build a global vector (and matrix) from local data. The local data has a global index, which can be non-contiguous i.e. global index (0, 5, 2) is on rank 0, (1, 4, 3) is on rank 1. To keep all local data on the local part of vector I need a mapping: Application -> PETSc (0, 5, 2) -> (0, 1, 2) (1, 4, 3) -> (3, 4, 5) It can happen that rank 1 also need to touch vector[5] (application ordering) i.e. vector[1] (petsc ordering) on rank 0. I can produce a mapping using index sets: ierr = ISCreateGeneral(PETSC_COMM_WORLD, indizes.size(), indizes.data(), PETSC_COPY_VALUES, &ISlocal); CHKERRV(ierr); ierr = ISAllGather(ISlocal, &ISglobal); CHKERRV(ierr); // Gather the IS from all processors ierr = ISLocalToGlobalMappingCreateIS(ISglobal, &ISmapping); CHKERRV(ierr); // Make it a mapping I construct std::vector indizes by walking though all local indizes. This way I have a mapping ISmapping that looks like that: 0 0 1 5 2 2 3 1 4 4 5 3 Now I can use that mapping: VecSetLocalToGlobalMapping(vec, mapping); but this does not work like it should (for me). If I use VecSetValueLocal to set element 5 it gets mapped to 3, element 1 gets mapped to 5. I want to have it reversed, accessing element 3 gets mapped to 5, element 1 to 3. This loop: for (size_t i = 0; i < indizes.size(); i++) { ierr = VecSetValueLocal(vec, indizes[i], i*10, INSERT_VALUES); CHKERRQ(ierr); } should produce a vector = [0, 10, 20, 30, 40, 50] and not [0, 50, 20, 10, 40, 30]. Are AO (application orderings) more suited for that? But afaik I can not produce a ISLocalToGlobalMapping from an AO that can be used with VecSetValueLocal, can I? Is there a way to inverse a mapping? To produce: 0 0 1 3 2 2 3 5 4 4 5 1 Or am I down the wrong path? Thanks, Florian Example code: #include #include #include #include "petscmat.h" #include "petscviewer.h" #include "petscksp.h" using namespace std; int main(int argc, char **argv) { PetscInitialize(&argc, &argv, "", NULL); PetscErrorCode ierr; Vec vec; VecCreate(PETSC_COMM_SELF, &vec); std::vector indizes = {0, 5, 2, 1, 4, 3}; IS is; ISLocalToGlobalMapping mapping; ierr = ISCreateGeneral(PETSC_COMM_WORLD, indizes.size(),indizes.data(), PETSC_COPY_VALUES, &is); CHKERRQ(ierr); ierr = ISLocalToGlobalMappingCreateIS(is, &mapping); CHKERRQ(ierr); ierr = VecCreate(PETSC_COMM_WORLD, &vec); CHKERRQ(ierr); ierr = VecSetLocalToGlobalMapping(vec, mapping); CHKERRQ(ierr); ierr = ISLocalToGlobalMappingView(mapping, PETSC_VIEWER_STDOUT_SELF); CHKERRQ(ierr); ierr = VecSetSizes(vec, PETSC_DECIDE, indizes.size()); CHKERRQ(ierr); ierr = VecSetFromOptions(vec); CHKERRQ(ierr); for (size_t i = 0; i < indizes.size(); i++) { ierr = VecSetValueLocal(vec, indizes[i], i*10, INSERT_VALUES); CHKERRQ(ierr); } ierr = VecView(vec, PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr); PetscFinalize(); return 0; } From bsmith at mcs.anl.gov Tue Feb 16 11:13:57 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 16 Feb 2016 11:13:57 -0600 Subject: [petsc-users] SNESComputeJacobianDefaultColor use too much memory In-Reply-To: <56C319D0.9060208@gmail.com> References: <56C319D0.9060208@gmail.com> Message-ID: <684F7147-4F58-42E3-9C9E-2C2297383276@mcs.anl.gov> How many colors are needed? You need to produce a breakdown of where all the memory is being used. For example valgrind with the http://valgrind.org/docs/manual/ms-manual.html Barry > On Feb 16, 2016, at 6:45 AM, Rongliang Chen wrote: > > Dear all, > > I am using the DMPlex to solve a PDE on a unstructured mesh and I use the SNESComputeJacobianDefaultColor to compute the Jacobian matrix. > > My code works well for small problems (such as problem with 3.3x10^5 cells using 120 cores) but when I increase the number of cells (2.6x10^6 cells using 1920 cores), I go out of memory in the function MatColoringApply. It shows that one of the cores uses over 11G memory. I think this is unreasonable. Do you have any suggestions for debugging this problem? > > Best regards, > Rongliang From hng.email at gmail.com Tue Feb 16 13:46:42 2016 From: hng.email at gmail.com (Hom Nath Gharti) Date: Tue, 16 Feb 2016 14:46:42 -0500 Subject: [petsc-users] PCFIELDSPLIT question In-Reply-To: References: Message-ID: Hi Matt, Finally I was able to run my program with field split option. But it is very slow. One KSP iteration takes about 5mins! For comparison, without field split it would take couple of seconds. I have set three fields corresponding to solid displacement (u), fluid pressure (p), and fluid potential (\xi). The matrix is symmetric and has following structure: K_{uu} 0 K_{u\chi} 0 K_{pp} K_{p\chi} (K_{u\chi})^T (K_{p\chi})^T K_{\chi\chi} For testing purpose I use the command: srun -n 96 ./bin/xspecfem3D -ksp_monitor -ksp_type fgmres -fieldsplit_0_pc_type jacobi -fieldsplit_1_pc_type jacobi -fieldsplit_2_pc_type jacobi It seems that I may be doing something wrong. Any help would be greatly appreciated. Thanks, Hom Nath On Thu, Feb 11, 2016 at 1:41 PM, Hom Nath Gharti wrote: > Perfect! Thanks a lot Matt! > > Hom Nath > > On Thu, Feb 11, 2016 at 1:34 PM, Matthew Knepley wrote: >> On Thu, Feb 11, 2016 at 11:47 AM, Hom Nath Gharti >> wrote: >>> >>> Thanks all. With further help from Stefano Zhampini I was able to >>> solve my small test case using PCFieldSPlit option. I had to add >>> call PCSetType(pc,PCFIELDSPLIT,ierr) >>> >>> Now I am trying to use same technique in my actual problem and got the >>> error: >>> >>> PETSC ERROR: Fields must be sorted when creating split >>> >>> Does this mean I have to sort the global matrix rows according the split >>> fields? >> >> >> You have to sort the IS you pass to PCFieldSplitSetIS() which can be done >> using ISSort() >> >> Thanks, >> >> Matt >> >>> >>> Thanks >>> Hom Nath >>> >>> >>> >>> On Tue, Feb 9, 2016 at 10:57 AM, Hom Nath Gharti >>> wrote: >>> > Sounds interesting! Thanks a lot Matt! I will have a look. >>> > >>> > Hom Nath >>> > >>> > On Tue, Feb 9, 2016 at 10:36 AM, Matthew Knepley >>> > wrote: >>> >> On Tue, Feb 9, 2016 at 9:31 AM, Hom Nath Gharti >>> >> wrote: >>> >>> >>> >>> Thanks a lot Matt! >>> >>> >>> >>> Were you referring to >>> >>> >>> >>> >>> >>> http://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex62.c.html >>> >>> ? >>> >>> >>> >>> I do not see any statements related to PCFieldSplit there. Am I >>> >>> missing something here? >>> >> >>> >> >>> >> Yes. The whole point is not to put anything specific to FIELDSPLIT in >>> >> the >>> >> code. It is all in options. For example, >>> >> >>> >> ./ex62 -run_type full -refinement_limit 0.00625 -bc_type dirichlet >>> >> -interpolate 1 -vel_petscspace_order 2 -pres_petscspace_order 1 >>> >> -ksp_type >>> >> fgmres -ksp_gmres_restart 100 -ksp_rtol 1.0e-9 -pc_type fieldsplit >>> >> -pc_fieldsplit_type schur -pc_fieldsplit_schur_factorization_type upper >>> >> -fieldsplit_pressure_ksp_rtol 1e-10 -fieldsplit_velocity_ksp_type gmres >>> >> -fieldsplit_velocity_pc_type lu -fieldsplit_pressure_pc_type jacobi >>> >> -snes_monitor_short -ksp_monitor_short -snes_converged_reason >>> >> -ksp_converged_reason -snes_view -show_solution 0 >>> >> >>> >> The reason that it works is that the DM calls PCFIeldSplitSetIS() for >>> >> each >>> >> field in the DM. >>> >> >>> >> Thanks, >>> >> >>> >> Matt >>> >> >>> >>> >>> >>> Thanks, >>> >>> Hom Nath >>> >>> >>> >>> On Tue, Feb 9, 2016 at 10:19 AM, Matthew Knepley >>> >>> wrote: >>> >>> > On Tue, Feb 9, 2016 at 9:10 AM, Hom Nath Gharti >>> >>> > >>> >>> > wrote: >>> >>> >> >>> >>> >> Thank you so much Barry! >>> >>> >> >>> >>> >> For my small test case, with -pc_fieldsplit_block_size 4, the >>> >>> >> program >>> >>> >> runs, although the answer was not correct. At least now I get >>> >>> >> something to look upon. I am using PCFieldSplitSetIS to set the >>> >>> >> fields. Do I still need to use -pc_fieldsplit_block_size? >>> >>> > >>> >>> > >>> >>> > No, this is only for co-located discretizations. >>> >>> > >>> >>> >> >>> >>> >> In my case each grid point may have different variable sets. For >>> >>> >> example, the grid point in the solid region has displacement and >>> >>> >> gravity potential: ux, uy, uz, and \phi. Whereas the grid point in >>> >>> >> the >>> >>> >> fluid region has scalar potential, pressure and gravity potential: >>> >>> >> \chi, p, and \phi. And the solid-fluid interface has all of them. >>> >>> >> Do >>> >>> >> you think I can still use PCFIELDSPLIT in this situation? >>> >>> > >>> >>> > >>> >>> > We have examples, like SNES ex62, which do exactly this. >>> >>> > >>> >>> > Thanks, >>> >>> > >>> >>> > Matt >>> >>> > >>> >>> >> >>> >>> >> Best, >>> >>> >> Hom Nath >>> >>> >> >>> >>> >> >>> >>> >> >>> >>> >> On Mon, Feb 8, 2016 at 2:27 PM, Barry Smith >>> >>> >> wrote: >>> >>> >> > >>> >>> >> > >>> >>> >> > In this case you need to provide two pieces of information to >>> >>> >> > the >>> >>> >> > PCFIELDSPLIT. What we call the "block size" or bs which is the >>> >>> >> > number >>> >>> >> > of >>> >>> >> > "basic fields" in the problem. For example if at each grid point >>> >>> >> > you >>> >>> >> > have >>> >>> >> > x-velocity, y-velocity, and pressure the block size is 3. The >>> >>> >> > second >>> >>> >> > is you >>> >>> >> > need to tell PCFIELDSPLIT what "basic fields" are in each split >>> >>> >> > you >>> >>> >> > want to >>> >>> >> > use. >>> >>> >> > >>> >>> >> > In your case you can do this with -pc_fieldsplit_block_size 4. >>> >>> >> > (Note if you use a DM to define your problem the PCFIELDSPLIT >>> >>> >> > will >>> >>> >> > automatically get the bs from the DM so you do not need to >>> >>> >> > provide >>> >>> >> > it. >>> >>> >> > Similarly if you set a block size on your Mat the PCFIELDSPLIT >>> >>> >> > will >>> >>> >> > use that >>> >>> >> > so you don't need to set it). >>> >>> >> > >>> >>> >> > Then use -pc_fieldsplit_0_fields 0,1 to indicate your first >>> >>> >> > split >>> >>> >> > is >>> >>> >> > the 0 and 1 basic fields and -pc_fieldsplit_1_fields 2,3 to >>> >>> >> > indicate >>> >>> >> > the >>> >>> >> > second split is the 2 and 3 basic fields. >>> >>> >> > (By default if you do not provide this then PCFIELDSPLIT will use >>> >>> >> > bs >>> >>> >> > splits (4 in your case) one for each basic field. >>> >>> >> > >>> >>> >> > Barry >>> >>> >> > >>> >>> >> >> On Feb 8, 2016, at 11:20 AM, Hom Nath Gharti >>> >>> >> >> >>> >>> >> >> wrote: >>> >>> >> >> >>> >>> >> >> Hi Matt, Hi all, >>> >>> >> >> >>> >>> >> >> I am trying to have some feel for PCFIELDSPLIT testing on a very >>> >>> >> >> small >>> >>> >> >> matrix (attached below). I have 4 degrees of freedoms. I use 4 >>> >>> >> >> processors. I want to split 4 dofs into two fields each having >>> >>> >> >> two >>> >>> >> >> dofs. I don't know whether this my be a problem for petsc. When >>> >>> >> >> I >>> >>> >> >> use >>> >>> >> >> the command: >>> >>> >> >> srun -n 4 ./xtestfs -ksp_monitor -ksp_view >>> >>> >> >> >>> >>> >> >> It runs fine. >>> >>> >> >> >>> >>> >> >> But when I want to use field split options using the command: >>> >>> >> >> srun -n 4 ./xtestfs -ksp_monitor -ksp_type fgmres -pc_type >>> >>> >> >> fieldsplit >>> >>> >> >> -fieldsplit_0_ksp_type gmres -fieldsplit_0_pc_type bjacobi >>> >>> >> >> -fieldsplit_1_pc_type jacobi >>> >>> >> >> >>> >>> >> >> I get the following error message: >>> >>> >> >> [0]PETSC ERROR: Petsc has generated inconsistent data >>> >>> >> >> [0]PETSC ERROR: Unhandled case, must have at least two fields, >>> >>> >> >> not 1 >>> >>> >> >> [0]PETSC ERROR: See >>> >>> >> >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >>> >>> >> >> shooting. >>> >>> >> >> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>> >>> >> >> [0]PETSC ERROR: /tigress/hgharti/test/testfs/./xtestfs on a >>> >>> >> >> arch-linux2-c-debug named tiger-r11n1 by hgharti Mon Feb 8 >>> >>> >> >> 11:40:03 >>> >>> >> >> 2016 >>> >>> >> >> [0]PETSC ERROR: Configure options >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64/ >>> >>> >> >> --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 >>> >>> >> >> --with-mpiexec=mpiexec --with-debugging=1 --download-scalapack >>> >>> >> >> --download-mumps --download-pastix --download-superlu >>> >>> >> >> --download-superlu_dist --download-metis --download-parmetis >>> >>> >> >> --download-ptscotch --download-hypre >>> >>> >> >> [0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 469 in >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >>> >>> >> >> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 486 in >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >>> >>> >> >> [0]PETSC ERROR: #3 PCSetUp() line 983 in >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/interface/precon.c >>> >>> >> >> [0]PETSC ERROR: #4 KSPSetUp() line 332 in >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >>> >>> >> >> [0]PETSC ERROR: #5 KSPSolve() line 546 in >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >>> >>> >> >> WARNING! There are options you set that were not used! >>> >>> >> >> WARNING! could be spelling mistake, etc! >>> >>> >> >> Option left: name:-fieldsplit_0_ksp_type value: gmres >>> >>> >> >> Option left: name:-fieldsplit_0_pc_type value: bjacobi >>> >>> >> >> Option left: name:-fieldsplit_1_pc_type value: jacobi >>> >>> >> >> forrtl: error (76): Abort trap signal >>> >>> >> >> >>> >>> >> >> I tried several trials but could not fix the problem. Is it the >>> >>> >> >> FORTRAN problem or am I doing something wrong? Any suggestions >>> >>> >> >> would >>> >>> >> >> be greatly appreciated. >>> >>> >> >> For your information I use: >>> >>> >> >> PETSc-3.6.3 >>> >>> >> >> intel-16.0 compiler >>> >>> >> >> intel-mpi-5.1.1 >>> >>> >> >> >>> >>> >> >> Program >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> Best, >>> >>> >> >> Hom Nath >>> >>> >> >> >>> >>> >> >> ! simple test case for PCFIELDSPLIT >>> >>> >> >> ! >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> ----------------------------------------------------------------------- >>> >>> >> >> program testfs >>> >>> >> >> implicit none >>> >>> >> >> #include "petsc/finclude/petscsys.h" >>> >>> >> >> #include "petsc/finclude/petscvec.h" >>> >>> >> >> #include "petsc/finclude/petscvec.h90" >>> >>> >> >> #include "petsc/finclude/petscmat.h" >>> >>> >> >> #include "petsc/finclude/petscksp.h" >>> >>> >> >> #include "petsc/finclude/petscpc.h" >>> >>> >> >> #include "petsc/finclude/petscviewer.h" >>> >>> >> >> #include "petsc/finclude/petscviewer.h90" >>> >>> >> >> >>> >>> >> >> Vec x,b,u >>> >>> >> >> Mat A >>> >>> >> >> KSP ksp >>> >>> >> >> PC pc >>> >>> >> >> PetscErrorCode ierr >>> >>> >> >> PetscBool flg >>> >>> >> >> PetscInt errcode,its,maxitr,myrank,nproc >>> >>> >> >> PetscInt nedof,nelmt,ndof,nenod,ngdof,fixdof,nsparse,neq >>> >>> >> >> PetscInt,allocatable :: krow_sparse(:),ggdof_elmt(:,:), >>> >>> >> >> & >>> >>> >> >> inode_elmt(:,:) >>> >>> >> >> >>> >>> >> >> PetscReal,allocatable :: storef(:,:),storekmat(:,:,:) >>> >>> >> >> PetscInt gdof0(2),gdof1(2) >>> >>> >> >> >>> >>> >> >> ! initialize MPI >>> >>> >> >> >>> >>> >> >> call MPI_INIT(errcode) >>> >>> >> >> if(errcode /= 0)stop 'ERROR: cannot initialize MPI!' >>> >>> >> >> call MPI_COMM_RANK(MPI_COMM_WORLD,myrank,errcode) >>> >>> >> >> if(errcode /= 0)stop 'ERROR: cannot find processor ID' >>> >>> >> >> call MPI_COMM_SIZE(MPI_COMM_WORLD,nproc,errcode) >>> >>> >> >> if(errcode /= 0)stop 'ERROR: cannot find number of processors!' >>> >>> >> >> >>> >>> >> >> ! define some parameters >>> >>> >> >> ! 1D model with 4 elements. Each elements consits of 2 nodes. >>> >>> >> >> Node 0 >>> >>> >> >> is >>> >>> >> >> fixed. >>> >>> >> >> ! 1-----------2-----------3-----------4-----------5 >>> >>> >> >> nelmt=1 ! per processor >>> >>> >> >> nenod=2 ! number of nodes per element >>> >>> >> >> nedof=2 ! number of DOFs per element >>> >>> >> >> ndof=2 ! number of DOFs >>> >>> >> >> ngdof=4 ! total number of global DOFs >>> >>> >> >> fixdof=1 >>> >>> >> >> >>> >>> >> >> if(myrank==0)then >>> >>> >> >> neq=1 ! number of equations >>> >>> >> >> nsparse=1 >>> >>> >> >> else >>> >>> >> >> neq=2 ! number of equations >>> >>> >> >> nsparse=4 >>> >>> >> >> endif >>> >>> >> >> >>> >>> >> >> allocate(storef(nedof,nelmt),storekmat(nedof,nedof,nelmt), >>> >>> >> >> & >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> ggdof_elmt(nedof,nelmt),inode_elmt(nenod,nelmt),krow_sparse(nsparse)) >>> >>> >> >> >>> >>> >> >> storef=0.0_8 ! local RHS vector >>> >>> >> >> storekmat=0.0_8 ! local stiffness matrix >>> >>> >> >> if(myrank==0)then >>> >>> >> >> krow_sparse(:)=1 >>> >>> >> >> storef(:,1)=(/12.5000_8, 12.5000_8/) >>> >>> >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & >>> >>> >> >> 0.0166666667_8, >>> >>> >> >> 1.2333333332_8/),(/nedof,nedof/)) >>> >>> >> >> inode_elmt(:,1) = (/1,2/) >>> >>> >> >> ggdof_elmt(:,1) = (/0,1/) ! 0 because left node is fixed >>> >>> >> >> elseif(myrank==1)then >>> >>> >> >> krow_sparse(:)=(/1,1,2,2/) >>> >>> >> >> storef(:,1)=(/12.5000_8, 12.5000_8/) >>> >>> >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & >>> >>> >> >> 0.0166666667_8, >>> >>> >> >> 1.2333333332_8/),(/nedof,nedof/)) >>> >>> >> >> inode_elmt(:,1) = (/1,2/) >>> >>> >> >> ggdof_elmt(:,1) = (/1,2/) >>> >>> >> >> elseif(myrank==2)then >>> >>> >> >> krow_sparse(:)=(/1,1,2,2/) >>> >>> >> >> storef(:,1)=(/12.5000_8, 12.5000_8/) >>> >>> >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & >>> >>> >> >> 0.0166666667_8, >>> >>> >> >> 1.2333333332_8/),(/nedof,nedof/)) >>> >>> >> >> inode_elmt(:,1) = (/1,2/) >>> >>> >> >> ggdof_elmt(:,1) = (/2,3/) >>> >>> >> >> elseif(myrank==3)then >>> >>> >> >> krow_sparse(:)=(/1,1,2,2/) >>> >>> >> >> storef(:,1)=(/12.5000_8, 22.5000_8/) >>> >>> >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, 0.0166666667_8, & >>> >>> >> >> 0.0166666667_8, >>> >>> >> >> 1.2333333332_8/),(/nedof,nedof/)) >>> >>> >> >> inode_elmt(:,1) = (/1,2/) >>> >>> >> >> ggdof_elmt(:,1) = (/3,4/) >>> >>> >> >> endif >>> >>> >> >> >>> >>> >> >> call petsc_initialize >>> >>> >> >> call petsc_set_matrix >>> >>> >> >> call petsc_set_vector >>> >>> >> >> call petsc_solve >>> >>> >> >> call petsc_finalize >>> >>> >> >> >>> >>> >> >> ! deallocate >>> >>> >> >> deallocate(storef,storekmat,ggdof_elmt,inode_elmt) >>> >>> >> >> deallocate(krow_sparse) >>> >>> >> >> contains >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> !------------------------------------------------------------------------------- >>> >>> >> >> subroutine petsc_initialize() >>> >>> >> >> implicit none >>> >>> >> >> PetscInt :: istart,iend >>> >>> >> >> PetscInt :: nzeros_max,nzeros_min >>> >>> >> >> PetscReal :: atol,dtol,rtol >>> >>> >> >> PetscInt,allocatable :: nzeros(:) >>> >>> >> >> IS gdof0_is,gdof1_is >>> >>> >> >> >>> >>> >> >> call PetscInitialize(PETSC_NULL_CHARACTER,ierr) >>> >>> >> >> call >>> >>> >> >> PetscOptionsGetInt(PETSC_NULL_CHARACTER,'-n',ngdof,flg,ierr) >>> >>> >> >> >>> >>> >> >> ! count number of nonzeros per row >>> >>> >> >> allocate(nzeros(neq)) >>> >>> >> >> nzeros=0 >>> >>> >> >> nzeros(krow_sparse)=nzeros(krow_sparse)+1 >>> >>> >> >> nzeros_max=maxval(nzeros) >>> >>> >> >> nzeros_min=minval(nzeros) >>> >>> >> >> >>> >>> >> >> ! create matrix >>> >>> >> >> call MatCreate(PETSC_COMM_WORLD,A,ierr) >>> >>> >> >> call MatSetType(A,MATMPIAIJ,ierr) >>> >>> >> >> CHKERRQ(ierr) >>> >>> >> >> call MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,ngdof,ngdof,ierr) >>> >>> >> >> CHKERRQ(ierr) >>> >>> >> >> ! preallocation >>> >>> >> >> call >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> MatMPIAIJSetPreallocation(A,nzeros_max,PETSC_NULL_INTEGER,nzeros_max, & >>> >>> >> >> PETSC_NULL_INTEGER,ierr) >>> >>> >> >> CHKERRQ(ierr) >>> >>> >> >> >>> >>> >> >> call MatGetOwnershipRange(A,istart,iend,ierr) >>> >>> >> >> CHKERRQ(ierr) >>> >>> >> >> print*,'ownership:',myrank,istart,iend >>> >>> >> >> ! create vector >>> >>> >> >> call VecCreate(PETSC_COMM_WORLD,x,ierr) >>> >>> >> >> call VecSetSizes(x,PETSC_DECIDE,ngdof,ierr) >>> >>> >> >> call VecSetType(x,VECMPI,ierr) >>> >>> >> >> call VecDuplicate(x,b,ierr) >>> >>> >> >> call VecDuplicate(x,u,ierr) >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> !******************************************************************************* >>> >>> >> >> ! get IS for split fields >>> >>> >> >> gdof0=(/0,1/) >>> >>> >> >> gdof1=(/2,3/) >>> >>> >> >> call >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> ISCreateGeneral(PETSC_COMM_WORLD,2,gdof0,PETSC_COPY_VALUES,gdof0_is,ierr) >>> >>> >> >> CHKERRQ(ierr) >>> >>> >> >> call >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> ISCreateGeneral(PETSC_COMM_WORLD,2,gdof1,PETSC_COPY_VALUES,gdof1_is,ierr) >>> >>> >> >> CHKERRQ(ierr) >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> !******************************************************************************* >>> >>> >> >> >>> >>> >> >> ! Create linear solver context >>> >>> >> >> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >>> >>> >> >> call KSPSetOperators(ksp,A,A,ierr) >>> >>> >> >> >>> >>> >> >> call KSPGetPC(ksp,pc,ierr) >>> >>> >> >> CHKERRQ(ierr) >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> !******************************************************************************* >>> >>> >> >> ! split PC >>> >>> >> >> call PCFieldSplitSetIS(pc,"0",gdof0_is,ierr); >>> >>> >> >> CHKERRQ(ierr) >>> >>> >> >> call ISDestroy(gdof0_is,ierr) >>> >>> >> >> CHKERRQ(ierr) >>> >>> >> >> call PCFieldSplitSetIS(pc,"1",gdof1_is,ierr); >>> >>> >> >> CHKERRQ(ierr) >>> >>> >> >> call ISDestroy(gdof1_is,ierr) >>> >>> >> >> CHKERRQ(ierr) >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> !******************************************************************************* >>> >>> >> >> >>> >>> >> >> rtol = 1.0d-6 >>> >>> >> >> atol = 1.0d-6 >>> >>> >> >> dtol = 1.0d10 >>> >>> >> >> maxitr = 1000 >>> >>> >> >> call KSPSetTolerances(ksp,rtol,atol,dtol,maxitr,ierr) >>> >>> >> >> CHKERRQ(ierr) >>> >>> >> >> call KSPSetFromOptions(ksp,ierr) >>> >>> >> >> end subroutine petsc_initialize >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> !------------------------------------------------------------------------------- >>> >>> >> >> >>> >>> >> >> subroutine petsc_set_matrix() >>> >>> >> >> >>> >>> >> >> implicit none >>> >>> >> >> integer :: i,i_elmt,j,ncount >>> >>> >> >> integer :: ggdof(NEDOF),idof(NEDOF),igdof(NEDOF) >>> >>> >> >> >>> >>> >> >> ! Set and assemble matrix. >>> >>> >> >> call MatZeroEntries(A,ierr) >>> >>> >> >> CHKERRQ(ierr) >>> >>> >> >> do i_elmt=1,nelmt >>> >>> >> >> ggdof(:)=ggdof_elmt(:,i_elmt) >>> >>> >> >> ggdof(:)=ggdof(:)-1 ! petsc index starts from 0 >>> >>> >> >> ncount=0; idof=-1; igdof=-1 >>> >>> >> >> do i=1,NEDOF >>> >>> >> >> do j=1,NEDOF >>> >>> >> >> if(ggdof(i).ge.0.and.ggdof(j).ge.0)then >>> >>> >> >> call >>> >>> >> >> MatSetValues(A,1,ggdof(i),1,ggdof(j),storekmat(i,j,i_elmt), >>> >>> >> >> & >>> >>> >> >> ADD_VALUES,ierr) >>> >>> >> >> CHKERRQ(ierr) >>> >>> >> >> endif >>> >>> >> >> enddo >>> >>> >> >> enddo >>> >>> >> >> enddo >>> >>> >> >> call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) >>> >>> >> >> call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) >>> >>> >> >> if(myrank==0)print*,'matrix setting & assembly complete!' >>> >>> >> >> >>> >>> >> >> end subroutine petsc_set_matrix >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> !------------------------------------------------------------------------------- >>> >>> >> >> >>> >>> >> >> subroutine petsc_set_vector() >>> >>> >> >> implicit none >>> >>> >> >> PetscScalar zero >>> >>> >> >> integer :: i,i_elmt,ncount >>> >>> >> >> integer :: ggdof(NEDOF),idof(NEDOF),igdof(NEDOF) >>> >>> >> >> >>> >>> >> >> ! set vector >>> >>> >> >> zero=0.0_8 >>> >>> >> >> call VecSet(b,zero,ierr) >>> >>> >> >> do i_elmt=1,nelmt >>> >>> >> >> ggdof(:)=ggdof_elmt(:,i_elmt) >>> >>> >> >> ggdof(:)=ggdof(:)-1 ! petsc index starts from 0 >>> >>> >> >> ncount=0; idof=-1; igdof=-1 >>> >>> >> >> do i=1,NEDOF >>> >>> >> >> if(ggdof(i).ge.0)then >>> >>> >> >> call >>> >>> >> >> VecSetValues(b,1,ggdof(i),storef(i,i_elmt),ADD_VALUES,ierr); >>> >>> >> >> CHKERRQ(ierr) >>> >>> >> >> endif >>> >>> >> >> enddo >>> >>> >> >> enddo >>> >>> >> >> >>> >>> >> >> ! assemble vector >>> >>> >> >> call VecAssemblyBegin(b,ierr) >>> >>> >> >> call VecAssemblyEnd(b,ierr) >>> >>> >> >> if(myrank==0)print*,'vector setting & assembly complete!' >>> >>> >> >> >>> >>> >> >> end subroutine petsc_set_vector >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> !------------------------------------------------------------------------------- >>> >>> >> >> >>> >>> >> >> subroutine petsc_solve() >>> >>> >> >> implicit none >>> >>> >> >> PetscInt ireason >>> >>> >> >> PetscScalar a_v(1) >>> >>> >> >> PetscOffset a_i >>> >>> >> >> PetscInt n >>> >>> >> >> PetscReal,allocatable :: sdata(:) >>> >>> >> >> >>> >>> >> >> call VecGetSize(b,n,ierr) >>> >>> >> >> CHKERRQ(ierr) >>> >>> >> >> allocate(sdata(n)) >>> >>> >> >> sdata=0.0_8 >>> >>> >> >> call VecGetArray(b,a_v,a_i,ierr) >>> >>> >> >> CHKERRQ(ierr) >>> >>> >> >> sdata(1:n)=a_v(a_i+1:a_i+n) >>> >>> >> >> call VecRestoreArray(b,a_v,a_i,ierr) >>> >>> >> >> CHKERRQ(ierr) >>> >>> >> >> print*,'rhs:',sdata >>> >>> >> >> >>> >>> >> >> call KSPSolve(ksp,b,x,ierr) >>> >>> >> >> sdata=0.0_8 >>> >>> >> >> call VecGetArray(x,a_v,a_i,ierr) >>> >>> >> >> CHKERRQ(ierr) >>> >>> >> >> sdata(1:n)=a_v(a_i+1:a_i+n) >>> >>> >> >> call VecRestoreArray(b,a_v,a_i,ierr) >>> >>> >> >> CHKERRQ(ierr) >>> >>> >> >> print*,'solution:',sdata >>> >>> >> >> >>> >>> >> >> ! Check convergence >>> >>> >> >> call KSPGetConvergedReason(ksp,ireason,ierr) >>> >>> >> >> if(myrank==0)print*,'converges reason',myrank,ireason >>> >>> >> >> call KSPGetIterationNumber(ksp,its,ierr) >>> >>> >> >> if(myrank==0)print*,'Iterations:',its >>> >>> >> >> deallocate(sdata) >>> >>> >> >> end subroutine petsc_solve >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> !------------------------------------------------------------------------------- >>> >>> >> >> >>> >>> >> >> subroutine petsc_finalize() >>> >>> >> >> implicit none >>> >>> >> >> >>> >>> >> >> ! Free work space. >>> >>> >> >> call VecDestroy(x,ierr) >>> >>> >> >> call VecDestroy(u,ierr) >>> >>> >> >> call VecDestroy(b,ierr) >>> >>> >> >> call MatDestroy(A,ierr) >>> >>> >> >> call KSPDestroy(ksp,ierr) >>> >>> >> >> call PetscFinalize(ierr) >>> >>> >> >> CHKERRQ(ierr) >>> >>> >> >> >>> >>> >> >> end subroutine petsc_finalize >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> !------------------------------------------------------------------------------- >>> >>> >> >> >>> >>> >> >> end program testfs >>> >>> >> >> >>> >>> >> >> On Tue, Feb 2, 2016 at 4:54 PM, Matthew Knepley >>> >>> >> >> >>> >>> >> >> wrote: >>> >>> >> >>> On Tue, Feb 2, 2016 at 3:11 PM, Hom Nath Gharti >>> >>> >> >>> >>> >>> >> >>> wrote: >>> >>> >> >>>> >>> >>> >> >>>> Thanks a lot. I will try. >>> >>> >> >>> >>> >>> >> >>> >>> >>> >> >>> Also, if you send a small test case, we can run it and tell you >>> >>> >> >>> the >>> >>> >> >>> problem. >>> >>> >> >>> >>> >>> >> >>> Matt >>> >>> >> >>> >>> >>> >> >>>> >>> >>> >> >>>> Hom Nath >>> >>> >> >>>> >>> >>> >> >>>> On Tue, Feb 2, 2016 at 4:02 PM, Matthew Knepley >>> >>> >> >>>> >>> >>> >> >>>> wrote: >>> >>> >> >>>>> On Tue, Feb 2, 2016 at 3:01 PM, Hom Nath Gharti >>> >>> >> >>>>> >>> >>> >> >>>>> wrote: >>> >>> >> >>>>>> >>> >>> >> >>>>>> Thanks again Matt. Unfortunately got the same errors with >>> >>> >> >>>>>> '0'. I >>> >>> >> >>>>>> think >>> >>> >> >>>>>> both are valid in Fortran. >>> >>> >> >>>>> >>> >>> >> >>>>> >>> >>> >> >>>>> Then you will have to go in the debugger and see why its not >>> >>> >> >>>>> creating 4 >>> >>> >> >>>>> splits, since this is exactly >>> >>> >> >>>>> what it does in our tests. This is how all the SNES tests >>> >>> >> >>>>> that I >>> >>> >> >>>>> use >>> >>> >> >>>>> work. I >>> >>> >> >>>>> am sure its a Fortran >>> >>> >> >>>>> problem. >>> >>> >> >>>>> >>> >>> >> >>>>> Thanks, >>> >>> >> >>>>> >>> >>> >> >>>>> Matt >>> >>> >> >>>>> >>> >>> >> >>>>>> >>> >>> >> >>>>>> Hom Nath >>> >>> >> >>>>>> >>> >>> >> >>>>>> On Tue, Feb 2, 2016 at 3:42 PM, Matthew Knepley >>> >>> >> >>>>>> >>> >>> >> >>>>>> wrote: >>> >>> >> >>>>>>> On Tue, Feb 2, 2016 at 2:35 PM, Hom Nath Gharti >>> >>> >> >>>>>>> >>> >>> >> >>>>>>> wrote: >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> Thank you so much Matt. >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> I now tried the following: >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> ====================================================== >>> >>> >> >>>>>>>> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >>> >>> >> >>>>>>>> call KSPGetPC(ksp,pc,ierr) >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); >>> >>> >> >>>>>>>> call ISDestroy(gdofu_is,ierr) >>> >>> >> >>>>>>>> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); >>> >>> >> >>>>>>>> call ISDestroy(gdofchi_is,ierr) >>> >>> >> >>>>>>>> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); >>> >>> >> >>>>>>>> call ISDestroy(gdofp_is,ierr) >>> >>> >> >>>>>>>> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); >>> >>> >> >>>>>>>> call ISDestroy(gdofphi_is,ierr) >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> ! Amat changes in each time steps >>> >>> >> >>>>>>>> call KSPSetOperators(ksp,Amat,Amat,ierr) !version >= 3.5.0 >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> call KSPSolve(ksp,bvec,xvec,ierr) >>> >>> >> >>>>>>>> ====================================================== >>> >>> >> >>>>>>> >>> >>> >> >>>>>>> >>> >>> >> >>>>>>> I am guessing that "0" is not a valid string for your >>> >>> >> >>>>>>> Fortran >>> >>> >> >>>>>>> compiler. >>> >>> >> >>>>>>> Are >>> >>> >> >>>>>>> you sure >>> >>> >> >>>>>>> its not single quotes '0'? >>> >>> >> >>>>>>> >>> >>> >> >>>>>>> Matt >>> >>> >> >>>>>>> >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> But I get the following error: >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> [0]PETSC ERROR: --------------------- Error Message >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> -------------------------------------------------------------- >>> >>> >> >>>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>> >>> >> >>>>>>>> [0]PETSC ERROR: Unhandled case, must have at least two >>> >>> >> >>>>>>>> fields, >>> >>> >> >>>>>>>> not 1 >>> >>> >> >>>>>>>> [0]PETSC ERROR: See >>> >>> >> >>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for >>> >>> >> >>>>>>>> trouble >>> >>> >> >>>>>>>> shooting. >>> >>> >> >>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>> >>> >> >>>>>>>> [0]PETSC ERROR: >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> /tigress/hgharti/gitwork/SPECFEM3D_GLOBEVSPP/./bin/xspecfem3D >>> >>> >> >>>>>>>> on >>> >>> >> >>>>>>>> a >>> >>> >> >>>>>>>> arch-linux2-c-debug named tiger-r3c1n7 by hgharti Tue Feb >>> >>> >> >>>>>>>> 2 >>> >>> >> >>>>>>>> 15: >>> >>> >> >>>>>>>> 29:30 2016 >>> >>> >> >>>>>>>> [0]PETSC ERROR: Configure options >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64/ >>> >>> >> >>>>>>>> --with-cc=mpicc --with-cxx=mpicxx --wit >>> >>> >> >>>>>>>> h-fc=mpif90 --with-mpiexec=mpiexec --with-debugging=1 >>> >>> >> >>>>>>>> --download-scalapack --download-mumps --download-pastix >>> >>> >> >>>>>>>> --download-superlu --download-superlu_dist >>> >>> >> >>>>>>>> --download-metis >>> >>> >> >>>>>>>> --download-parmetis --download-ptscotch --download-hypre >>> >>> >> >>>>>>>> [0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 469 in >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >>> >>> >> >>>>>>>> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 486 in >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >>> >>> >> >>>>>>>> [0]PETSC ERROR: #3 PCSetUp() line 983 in >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/interface/precon.c >>> >>> >> >>>>>>>> [0]PETSC ERROR: #4 KSPSetUp() line 332 in >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >>> >>> >> >>>>>>>> [0]PETSC ERROR: #5 KSPSolve() line 546 in >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c >>> >>> >> >>>>>>>> forrtl: error (76): Abort trap signal >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> Am I missing something? >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> Thanks, >>> >>> >> >>>>>>>> Hom Nath >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> On Tue, Feb 2, 2016 at 3:24 PM, Matthew Knepley >>> >>> >> >>>>>>>> >>> >>> >> >>>>>>>> wrote: >>> >>> >> >>>>>>>>> On Tue, Feb 2, 2016 at 2:20 PM, Hom Nath Gharti >>> >>> >> >>>>>>>>> >>> >>> >> >>>>>>>>> wrote: >>> >>> >> >>>>>>>>>> >>> >>> >> >>>>>>>>>> Hi Matt, hi all, >>> >>> >> >>>>>>>>>> >>> >>> >> >>>>>>>>>> I am trying to use PCFIELDSPLIT for my solver which >>> >>> >> >>>>>>>>>> consists >>> >>> >> >>>>>>>>>> of >>> >>> >> >>>>>>>>>> 4 >>> >>> >> >>>>>>>>>> different variables, namely, u (displacement vector), >>> >>> >> >>>>>>>>>> \chi >>> >>> >> >>>>>>>>>> (displacement potential), p(pressure), and \phi (gravity >>> >>> >> >>>>>>>>>> potential). >>> >>> >> >>>>>>>>>> >>> >>> >> >>>>>>>>>> My code segment looks like the following: >>> >>> >> >>>>>>>>>> ====================================================== >>> >>> >> >>>>>>>>>> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >>> >>> >> >>>>>>>>>> call KSPGetPC(ksp,pc,ierr) >>> >>> >> >>>>>>>>>> nsplit=4 >>> >>> >> >>>>>>>>>> call PCFieldSplitSetBlockSize(pc,nsplit,ierr); >>> >>> >> >>>>>>>>> >>> >>> >> >>>>>>>>> >>> >>> >> >>>>>>>>> You do not need the statement above. >>> >>> >> >>>>>>>>> >>> >>> >> >>>>>>>>>> >>> >>> >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); >>> >>> >> >>>>>>>>>> call ISDestroy(gdofu_is,ierr) >>> >>> >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); >>> >>> >> >>>>>>>>>> call ISDestroy(gdofchi_is,ierr) >>> >>> >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); >>> >>> >> >>>>>>>>>> call ISDestroy(gdofp_is,ierr) >>> >>> >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); >>> >>> >> >>>>>>>>>> call ISDestroy(gdofphi_is,ierr) >>> >>> >> >>>>>>>>>> >>> >>> >> >>>>>>>>>> call PCFieldSplitGetSubKSP(pc,nsplit,subksp,ierr); >>> >>> >> >>>>>>>>> >>> >>> >> >>>>>>>>> >>> >>> >> >>>>>>>>> These SetOperators() calls are wrong. I am not sure why >>> >>> >> >>>>>>>>> you >>> >>> >> >>>>>>>>> want >>> >>> >> >>>>>>>>> them >>> >>> >> >>>>>>>>> here. >>> >>> >> >>>>>>>>> Also, that means you do not need the call above. >>> >>> >> >>>>>>>>> >>> >>> >> >>>>>>>>> Thanks, >>> >>> >> >>>>>>>>> >>> >>> >> >>>>>>>>> Matt >>> >>> >> >>>>>>>>> >>> >>> >> >>>>>>>>>> >>> >>> >> >>>>>>>>>> call KSPSetOperators(subksp(1),Amat,Amat,ierr); >>> >>> >> >>>>>>>>>> call KSPSetOperators(subksp(2),Amat,Amat,ierr); >>> >>> >> >>>>>>>>>> call KSPSetOperators(subksp(3),Amat,Amat,ierr); >>> >>> >> >>>>>>>>>> call KSPSetOperators(subksp(4),Amat,Amat,ierr); >>> >>> >> >>>>>>>>>> >>> >>> >> >>>>>>>>>> call KSPSolve(ksp,bvec,xvec,ierr) >>> >>> >> >>>>>>>>>> ====================================================== >>> >>> >> >>>>>>>>>> >>> >>> >> >>>>>>>>>> But I am getting the following error: >>> >>> >> >>>>>>>>>> [79]PETSC ERROR: Null argument, when expecting valid >>> >>> >> >>>>>>>>>> pointer >>> >>> >> >>>>>>>>>> [79]PETSC ERROR: Null Object: Parameter # 1 >>> >>> >> >>>>>>>>>> [79]PETSC ERROR: #1 KSPSetOperators() line 536 in >>> >>> >> >>>>>>>>>> >>> >>> >> >>>>>>>>>> >>> >>> >> >>>>>>>>>> /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interf >>> >>> >> >>>>>>>>>> >>> >>> >> >>>>>>>>>> It looks like I am doing something wrong in "call >>> >>> >> >>>>>>>>>> KSPSetOperators" >>> >>> >> >>>>>>>>>> but >>> >>> >> >>>>>>>>>> I could not figure that out. >>> >>> >> >>>>>>>>>> >>> >>> >> >>>>>>>>>> Could anybody help me to fix this problem? I looked into >>> >>> >> >>>>>>>>>> almost >>> >>> >> >>>>>>>>>> all >>> >>> >> >>>>>>>>>> related examples but I could not really figure out the >>> >>> >> >>>>>>>>>> correct >>> >>> >> >>>>>>>>>> steps >>> >>> >> >>>>>>>>>> after "call PCFieldSplitSetIS". >>> >>> >> >>>>>>>>>> >>> >>> >> >>>>>>>>>> Any help will be greatly appreciated. >>> >>> >> >>>>>>>>>> >>> >>> >> >>>>>>>>>> Best, >>> >>> >> >>>>>>>>>> Hom nath >>> >>> >> >>>>>>>>>> >>> >>> >> >>>>>>>>>> On Sun, Jan 24, 2016 at 7:14 PM, Hom Nath Gharti >>> >>> >> >>>>>>>>>> >>> >>> >> >>>>>>>>>> wrote: >>> >>> >> >>>>>>>>>>> Thank you so much Matt! I will try. >>> >>> >> >>>>>>>>>>> >>> >>> >> >>>>>>>>>>> Hom Nath >>> >>> >> >>>>>>>>>>> >>> >>> >> >>>>>>>>>>> On Sun, Jan 24, 2016 at 6:26 AM, Matthew Knepley >>> >>> >> >>>>>>>>>>> >>> >>> >> >>>>>>>>>>> wrote: >>> >>> >> >>>>>>>>>>>> On Fri, Jan 22, 2016 at 2:19 PM, Hom Nath Gharti >>> >>> >> >>>>>>>>>>>> >>> >>> >> >>>>>>>>>>>> wrote: >>> >>> >> >>>>>>>>>>>>> >>> >>> >> >>>>>>>>>>>>> Dear all, >>> >>> >> >>>>>>>>>>>>> >>> >>> >> >>>>>>>>>>>>> I am new to PcFieldSplit. >>> >>> >> >>>>>>>>>>>>> >>> >>> >> >>>>>>>>>>>>> I have a matrix formed using MATMPIAIJ. Is it >>> >>> >> >>>>>>>>>>>>> possible to >>> >>> >> >>>>>>>>>>>>> use >>> >>> >> >>>>>>>>>>>>> PCFIELDSPLIT operations in this type of matrix? Or >>> >>> >> >>>>>>>>>>>>> does >>> >>> >> >>>>>>>>>>>>> it >>> >>> >> >>>>>>>>>>>>> have >>> >>> >> >>>>>>>>>>>>> to >>> >>> >> >>>>>>>>>>>>> be >>> >>> >> >>>>>>>>>>>>> MATMPIBIJ or MATNEST format? >>> >>> >> >>>>>>>>>>>> >>> >>> >> >>>>>>>>>>>> >>> >>> >> >>>>>>>>>>>> Yes, you can split AIJ. >>> >>> >> >>>>>>>>>>>> >>> >>> >> >>>>>>>>>>>>> >>> >>> >> >>>>>>>>>>>>> If possible for MATMPIAIJ, could anybody provide me a >>> >>> >> >>>>>>>>>>>>> simple >>> >>> >> >>>>>>>>>>>>> example >>> >>> >> >>>>>>>>>>>>> or few steps? Variables in the equations are >>> >>> >> >>>>>>>>>>>>> displacement >>> >>> >> >>>>>>>>>>>>> vector, >>> >>> >> >>>>>>>>>>>>> scalar potential and pressure. >>> >>> >> >>>>>>>>>>>> >>> >>> >> >>>>>>>>>>>> >>> >>> >> >>>>>>>>>>>> If you do not have a collocated discretization, then >>> >>> >> >>>>>>>>>>>> you >>> >>> >> >>>>>>>>>>>> have >>> >>> >> >>>>>>>>>>>> to >>> >>> >> >>>>>>>>>>>> use >>> >>> >> >>>>>>>>>>>> >>> >>> >> >>>>>>>>>>>> >>> >>> >> >>>>>>>>>>>> >>> >>> >> >>>>>>>>>>>> >>> >>> >> >>>>>>>>>>>> >>> >>> >> >>>>>>>>>>>> >>> >>> >> >>>>>>>>>>>> >>> >>> >> >>>>>>>>>>>> >>> >>> >> >>>>>>>>>>>> >>> >>> >> >>>>>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html >>> >>> >> >>>>>>>>>>>> >>> >>> >> >>>>>>>>>>>> Thanks, >>> >>> >> >>>>>>>>>>>> >>> >>> >> >>>>>>>>>>>> Matt >>> >>> >> >>>>>>>>>>>> >>> >>> >> >>>>>>>>>>>>> >>> >>> >> >>>>>>>>>>>>> Thanks for help. >>> >>> >> >>>>>>>>>>>>> >>> >>> >> >>>>>>>>>>>>> Hom Nath >>> >>> >> >>>>>>>>>>>> >>> >>> >> >>>>>>>>>>>> >>> >>> >> >>>>>>>>>>>> >>> >>> >> >>>>>>>>>>>> >>> >>> >> >>>>>>>>>>>> -- >>> >>> >> >>>>>>>>>>>> What most experimenters take for granted before they >>> >>> >> >>>>>>>>>>>> begin >>> >>> >> >>>>>>>>>>>> their >>> >>> >> >>>>>>>>>>>> experiments >>> >>> >> >>>>>>>>>>>> is infinitely more interesting than any results to >>> >>> >> >>>>>>>>>>>> which >>> >>> >> >>>>>>>>>>>> their >>> >>> >> >>>>>>>>>>>> experiments >>> >>> >> >>>>>>>>>>>> lead. >>> >>> >> >>>>>>>>>>>> -- Norbert Wiener >>> >>> >> >>>>>>>>> >>> >>> >> >>>>>>>>> >>> >>> >> >>>>>>>>> >>> >>> >> >>>>>>>>> >>> >>> >> >>>>>>>>> -- >>> >>> >> >>>>>>>>> What most experimenters take for granted before they >>> >>> >> >>>>>>>>> begin >>> >>> >> >>>>>>>>> their >>> >>> >> >>>>>>>>> experiments >>> >>> >> >>>>>>>>> is infinitely more interesting than any results to which >>> >>> >> >>>>>>>>> their >>> >>> >> >>>>>>>>> experiments >>> >>> >> >>>>>>>>> lead. >>> >>> >> >>>>>>>>> -- Norbert Wiener >>> >>> >> >>>>>>> >>> >>> >> >>>>>>> >>> >>> >> >>>>>>> >>> >>> >> >>>>>>> >>> >>> >> >>>>>>> -- >>> >>> >> >>>>>>> What most experimenters take for granted before they begin >>> >>> >> >>>>>>> their >>> >>> >> >>>>>>> experiments >>> >>> >> >>>>>>> is infinitely more interesting than any results to which >>> >>> >> >>>>>>> their >>> >>> >> >>>>>>> experiments >>> >>> >> >>>>>>> lead. >>> >>> >> >>>>>>> -- Norbert Wiener >>> >>> >> >>>>> >>> >>> >> >>>>> >>> >>> >> >>>>> >>> >>> >> >>>>> >>> >>> >> >>>>> -- >>> >>> >> >>>>> What most experimenters take for granted before they begin >>> >>> >> >>>>> their >>> >>> >> >>>>> experiments >>> >>> >> >>>>> is infinitely more interesting than any results to which >>> >>> >> >>>>> their >>> >>> >> >>>>> experiments >>> >>> >> >>>>> lead. >>> >>> >> >>>>> -- Norbert Wiener >>> >>> >> >>> >>> >>> >> >>> >>> >>> >> >>> >>> >>> >> >>> >>> >>> >> >>> -- >>> >>> >> >>> What most experimenters take for granted before they begin >>> >>> >> >>> their >>> >>> >> >>> experiments >>> >>> >> >>> is infinitely more interesting than any results to which their >>> >>> >> >>> experiments >>> >>> >> >>> lead. >>> >>> >> >>> -- Norbert Wiener >>> >>> >> > >>> >>> > >>> >>> > >>> >>> > >>> >>> > >>> >>> > -- >>> >>> > What most experimenters take for granted before they begin their >>> >>> > experiments >>> >>> > is infinitely more interesting than any results to which their >>> >>> > experiments >>> >>> > lead. >>> >>> > -- Norbert Wiener >>> >> >>> >> >>> >> >>> >> >>> >> -- >>> >> What most experimenters take for granted before they begin their >>> >> experiments >>> >> is infinitely more interesting than any results to which their >>> >> experiments >>> >> lead. >>> >> -- Norbert Wiener >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments >> is infinitely more interesting than any results to which their experiments >> lead. >> -- Norbert Wiener From jcrean01 at gmail.com Tue Feb 16 18:45:39 2016 From: jcrean01 at gmail.com (Jared Crean) Date: Tue, 16 Feb 2016 19:45:39 -0500 Subject: [petsc-users] MatGetLocalSubMatrix Trouble In-Reply-To: <67FCA4CF-E86B-4568-B7AA-A5088CAA348B@mcs.anl.gov> References: <56C2A608.3090506@gmail.com> <67FCA4CF-E86B-4568-B7AA-A5088CAA348B@mcs.anl.gov> Message-ID: <56C3C2B3.1030700@gmail.com> Hello Barry, Thanks makes sense now. Thanks, Jared Crean On 02/15/2016 11:43 PM, Barry Smith wrote: > Jared, > > Though it is not properly documented the matrix must have had a call to MatSetLocalToGlobalMapping() before you can call MatCreateLocalRef(). > > I'll add a little more error checking and docs. > > Barry > > Note that matrices obtained with DMCreateMat() generally have had the local to global mapping already provided to the matrix so you don't need to call it yourself. > > >> On Feb 15, 2016, at 10:31 PM, Jared Crean wrote: >> >> Hello, >> When I try to use MatGetLocalSubMatrix (line 55 of the attached code), I get the following error: >> >> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [0]PETSC ERROR: Null argument, when expecting valid pointer >> [0]PETSC ERROR: Null Object: Parameter # 2 >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.6.0, Jun, 09, 2015 >> [0]PETSC ERROR: ./jc3 on a arch-linux2-cxx-debug named k475 by jared Mon Feb 15 23:19:29 2016 >> [0]PETSC ERROR: Configure options --with-clanguage=C++ >> [0]PETSC ERROR: #1 ISL2GCompose() line 117 in /home/jared/build/petsc-3.6.0/src/mat/impls/localref/mlocalref.c >> [0]PETSC ERROR: #2 MatCreateLocalRef() line 259 in /home/jared/build/petsc-3.6.0/src/mat/impls/localref/mlocalref.c >> [0]PETSC ERROR: #3 MatGetLocalSubMatrix() line 9686 in /home/jared/build/petsc-3.6.0/src/mat/interface/matrix.c >> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [0]PETSC ERROR: Null argument, when expecting valid pointer >> [0]PETSC ERROR: Null Object: Parameter # 2 >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.6.0, Jun, 09, 2015 >> [0]PETSC ERROR: ./jc3 on a arch-linux2-cxx-debug named k475 by jared Mon Feb 15 23:19:29 2016 >> [0]PETSC ERROR: Configure options --with-clanguage=C++ >> [0]PETSC ERROR: #4 ISL2GCompose() line 117 in /home/jared/build/petsc-3.6.0/src/mat/impls/localref/mlocalref.c >> [0]PETSC ERROR: #5 MatCreateLocalRef() line 259 in /home/jared/build/petsc-3.6.0/src/mat/impls/localref/mlocalref.c >> [0]PETSC ERROR: #6 MatGetLocalSubMatrix() line 9686 in /home/jared/build/petsc-3.6.0/src/mat/interface/matrix.c >> [0]PETSC ERROR: #7 main() line 55 in /home/jared/build/petsc-3.6.0/src/ksp/ksp/examples/tutorials/jc3.c >> [0]PETSC ERROR: No PETSc Option Table entries >> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- >> application called MPI_Abort(MPI_COMM_WORLD, 85) - process 0 >> >> >> A did a little digging and found: >> >> Line 117-118 of mlocalref.c are: >> PetscValidHeaderSpecific(ltog,IS_LTOGM_CLASSID,2); >> PetscValidPointer(cltog,3); >> >> where ltog is the ISLocalToGlobalMapping from (line 259): >> A->rmap->mapping >> >> where A is the original matrix, cltog is a reference to the local to global mapping to be created . >> The code prints out A, so it appears to be a valid matrix. Is there something wrong with the way I am using MatGetLocalSubMatrix? >> >> Jared Crean >> >> >> From knepley at gmail.com Tue Feb 16 19:27:31 2016 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 16 Feb 2016 19:27:31 -0600 Subject: [petsc-users] PCFIELDSPLIT question In-Reply-To: References: Message-ID: On Tue, Feb 16, 2016 at 1:46 PM, Hom Nath Gharti wrote: > Hi Matt, > > Finally I was able to run my program with field split option. But it > is very slow. One KSP iteration takes about 5mins! For comparison, > without field split it would take couple of seconds. > > I have set three fields corresponding to solid displacement (u), fluid > pressure (p), and fluid potential (\xi). The matrix is symmetric and > has following structure: > > K_{uu} 0 K_{u\chi} > 0 K_{pp} K_{p\chi} > (K_{u\chi})^T (K_{p\chi})^T K_{\chi\chi} > > For testing purpose I use the command: > srun -n 96 ./bin/xspecfem3D -ksp_monitor -ksp_type fgmres > -fieldsplit_0_pc_type jacobi -fieldsplit_1_pc_type jacobi > -fieldsplit_2_pc_type jacobi > > It seems that I may be doing something wrong. Any help would be > greatly appreciated. > 1) When trying something new, I would try it on 1 process to make sure i was doing the right thing 2) I would send the output of -ksp_view, so that other people could see what I was doing 3) For performance questions, I would attach the output of -log_summary. Matt > Thanks, > Hom Nath > > On Thu, Feb 11, 2016 at 1:41 PM, Hom Nath Gharti > wrote: > > Perfect! Thanks a lot Matt! > > > > Hom Nath > > > > On Thu, Feb 11, 2016 at 1:34 PM, Matthew Knepley > wrote: > >> On Thu, Feb 11, 2016 at 11:47 AM, Hom Nath Gharti > >> wrote: > >>> > >>> Thanks all. With further help from Stefano Zhampini I was able to > >>> solve my small test case using PCFieldSPlit option. I had to add > >>> call PCSetType(pc,PCFIELDSPLIT,ierr) > >>> > >>> Now I am trying to use same technique in my actual problem and got the > >>> error: > >>> > >>> PETSC ERROR: Fields must be sorted when creating split > >>> > >>> Does this mean I have to sort the global matrix rows according the > split > >>> fields? > >> > >> > >> You have to sort the IS you pass to PCFieldSplitSetIS() which can be > done > >> using ISSort() > >> > >> Thanks, > >> > >> Matt > >> > >>> > >>> Thanks > >>> Hom Nath > >>> > >>> > >>> > >>> On Tue, Feb 9, 2016 at 10:57 AM, Hom Nath Gharti > >>> wrote: > >>> > Sounds interesting! Thanks a lot Matt! I will have a look. > >>> > > >>> > Hom Nath > >>> > > >>> > On Tue, Feb 9, 2016 at 10:36 AM, Matthew Knepley > >>> > wrote: > >>> >> On Tue, Feb 9, 2016 at 9:31 AM, Hom Nath Gharti < > hng.email at gmail.com> > >>> >> wrote: > >>> >>> > >>> >>> Thanks a lot Matt! > >>> >>> > >>> >>> Were you referring to > >>> >>> > >>> >>> > >>> >>> > http://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex62.c.html > >>> >>> ? > >>> >>> > >>> >>> I do not see any statements related to PCFieldSplit there. Am I > >>> >>> missing something here? > >>> >> > >>> >> > >>> >> Yes. The whole point is not to put anything specific to FIELDSPLIT > in > >>> >> the > >>> >> code. It is all in options. For example, > >>> >> > >>> >> ./ex62 -run_type full -refinement_limit 0.00625 -bc_type dirichlet > >>> >> -interpolate 1 -vel_petscspace_order 2 -pres_petscspace_order 1 > >>> >> -ksp_type > >>> >> fgmres -ksp_gmres_restart 100 -ksp_rtol 1.0e-9 -pc_type fieldsplit > >>> >> -pc_fieldsplit_type schur -pc_fieldsplit_schur_factorization_type > upper > >>> >> -fieldsplit_pressure_ksp_rtol 1e-10 -fieldsplit_velocity_ksp_type > gmres > >>> >> -fieldsplit_velocity_pc_type lu -fieldsplit_pressure_pc_type jacobi > >>> >> -snes_monitor_short -ksp_monitor_short -snes_converged_reason > >>> >> -ksp_converged_reason -snes_view -show_solution 0 > >>> >> > >>> >> The reason that it works is that the DM calls PCFIeldSplitSetIS() > for > >>> >> each > >>> >> field in the DM. > >>> >> > >>> >> Thanks, > >>> >> > >>> >> Matt > >>> >> > >>> >>> > >>> >>> Thanks, > >>> >>> Hom Nath > >>> >>> > >>> >>> On Tue, Feb 9, 2016 at 10:19 AM, Matthew Knepley < > knepley at gmail.com> > >>> >>> wrote: > >>> >>> > On Tue, Feb 9, 2016 at 9:10 AM, Hom Nath Gharti > >>> >>> > > >>> >>> > wrote: > >>> >>> >> > >>> >>> >> Thank you so much Barry! > >>> >>> >> > >>> >>> >> For my small test case, with -pc_fieldsplit_block_size 4, the > >>> >>> >> program > >>> >>> >> runs, although the answer was not correct. At least now I get > >>> >>> >> something to look upon. I am using PCFieldSplitSetIS to set the > >>> >>> >> fields. Do I still need to use -pc_fieldsplit_block_size? > >>> >>> > > >>> >>> > > >>> >>> > No, this is only for co-located discretizations. > >>> >>> > > >>> >>> >> > >>> >>> >> In my case each grid point may have different variable sets. For > >>> >>> >> example, the grid point in the solid region has displacement and > >>> >>> >> gravity potential: ux, uy, uz, and \phi. Whereas the grid point > in > >>> >>> >> the > >>> >>> >> fluid region has scalar potential, pressure and gravity > potential: > >>> >>> >> \chi, p, and \phi. And the solid-fluid interface has all of > them. > >>> >>> >> Do > >>> >>> >> you think I can still use PCFIELDSPLIT in this situation? > >>> >>> > > >>> >>> > > >>> >>> > We have examples, like SNES ex62, which do exactly this. > >>> >>> > > >>> >>> > Thanks, > >>> >>> > > >>> >>> > Matt > >>> >>> > > >>> >>> >> > >>> >>> >> Best, > >>> >>> >> Hom Nath > >>> >>> >> > >>> >>> >> > >>> >>> >> > >>> >>> >> On Mon, Feb 8, 2016 at 2:27 PM, Barry Smith > > >>> >>> >> wrote: > >>> >>> >> > > >>> >>> >> > > >>> >>> >> > In this case you need to provide two pieces of information > to > >>> >>> >> > the > >>> >>> >> > PCFIELDSPLIT. What we call the "block size" or bs which is the > >>> >>> >> > number > >>> >>> >> > of > >>> >>> >> > "basic fields" in the problem. For example if at each grid > point > >>> >>> >> > you > >>> >>> >> > have > >>> >>> >> > x-velocity, y-velocity, and pressure the block size is 3. The > >>> >>> >> > second > >>> >>> >> > is you > >>> >>> >> > need to tell PCFIELDSPLIT what "basic fields" are in each > split > >>> >>> >> > you > >>> >>> >> > want to > >>> >>> >> > use. > >>> >>> >> > > >>> >>> >> > In your case you can do this with > -pc_fieldsplit_block_size 4. > >>> >>> >> > (Note if you use a DM to define your problem the PCFIELDSPLIT > >>> >>> >> > will > >>> >>> >> > automatically get the bs from the DM so you do not need to > >>> >>> >> > provide > >>> >>> >> > it. > >>> >>> >> > Similarly if you set a block size on your Mat the PCFIELDSPLIT > >>> >>> >> > will > >>> >>> >> > use that > >>> >>> >> > so you don't need to set it). > >>> >>> >> > > >>> >>> >> > Then use -pc_fieldsplit_0_fields 0,1 to indicate your first > >>> >>> >> > split > >>> >>> >> > is > >>> >>> >> > the 0 and 1 basic fields and -pc_fieldsplit_1_fields 2,3 to > >>> >>> >> > indicate > >>> >>> >> > the > >>> >>> >> > second split is the 2 and 3 basic fields. > >>> >>> >> > (By default if you do not provide this then PCFIELDSPLIT will > use > >>> >>> >> > bs > >>> >>> >> > splits (4 in your case) one for each basic field. > >>> >>> >> > > >>> >>> >> > Barry > >>> >>> >> > > >>> >>> >> >> On Feb 8, 2016, at 11:20 AM, Hom Nath Gharti > >>> >>> >> >> > >>> >>> >> >> wrote: > >>> >>> >> >> > >>> >>> >> >> Hi Matt, Hi all, > >>> >>> >> >> > >>> >>> >> >> I am trying to have some feel for PCFIELDSPLIT testing on a > very > >>> >>> >> >> small > >>> >>> >> >> matrix (attached below). I have 4 degrees of freedoms. I use > 4 > >>> >>> >> >> processors. I want to split 4 dofs into two fields each > having > >>> >>> >> >> two > >>> >>> >> >> dofs. I don't know whether this my be a problem for petsc. > When > >>> >>> >> >> I > >>> >>> >> >> use > >>> >>> >> >> the command: > >>> >>> >> >> srun -n 4 ./xtestfs -ksp_monitor -ksp_view > >>> >>> >> >> > >>> >>> >> >> It runs fine. > >>> >>> >> >> > >>> >>> >> >> But when I want to use field split options using the command: > >>> >>> >> >> srun -n 4 ./xtestfs -ksp_monitor -ksp_type fgmres -pc_type > >>> >>> >> >> fieldsplit > >>> >>> >> >> -fieldsplit_0_ksp_type gmres -fieldsplit_0_pc_type bjacobi > >>> >>> >> >> -fieldsplit_1_pc_type jacobi > >>> >>> >> >> > >>> >>> >> >> I get the following error message: > >>> >>> >> >> [0]PETSC ERROR: Petsc has generated inconsistent data > >>> >>> >> >> [0]PETSC ERROR: Unhandled case, must have at least two > fields, > >>> >>> >> >> not 1 > >>> >>> >> >> [0]PETSC ERROR: See > >>> >>> >> >> http://www.mcs.anl.gov/petsc/documentation/faq.html for > trouble > >>> >>> >> >> shooting. > >>> >>> >> >> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > >>> >>> >> >> [0]PETSC ERROR: /tigress/hgharti/test/testfs/./xtestfs on a > >>> >>> >> >> arch-linux2-c-debug named tiger-r11n1 by hgharti Mon Feb 8 > >>> >>> >> >> 11:40:03 > >>> >>> >> >> 2016 > >>> >>> >> >> [0]PETSC ERROR: Configure options > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64/ > >>> >>> >> >> --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > >>> >>> >> >> --with-mpiexec=mpiexec --with-debugging=1 > --download-scalapack > >>> >>> >> >> --download-mumps --download-pastix --download-superlu > >>> >>> >> >> --download-superlu_dist --download-metis --download-parmetis > >>> >>> >> >> --download-ptscotch --download-hypre > >>> >>> >> >> [0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 469 in > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > >>> >>> >> >> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 486 in > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > >>> >>> >> >> [0]PETSC ERROR: #3 PCSetUp() line 983 in > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/interface/precon.c > >>> >>> >> >> [0]PETSC ERROR: #4 KSPSetUp() line 332 in > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c > >>> >>> >> >> [0]PETSC ERROR: #5 KSPSolve() line 546 in > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c > >>> >>> >> >> WARNING! There are options you set that were not used! > >>> >>> >> >> WARNING! could be spelling mistake, etc! > >>> >>> >> >> Option left: name:-fieldsplit_0_ksp_type value: gmres > >>> >>> >> >> Option left: name:-fieldsplit_0_pc_type value: bjacobi > >>> >>> >> >> Option left: name:-fieldsplit_1_pc_type value: jacobi > >>> >>> >> >> forrtl: error (76): Abort trap signal > >>> >>> >> >> > >>> >>> >> >> I tried several trials but could not fix the problem. Is it > the > >>> >>> >> >> FORTRAN problem or am I doing something wrong? Any > suggestions > >>> >>> >> >> would > >>> >>> >> >> be greatly appreciated. > >>> >>> >> >> For your information I use: > >>> >>> >> >> PETSc-3.6.3 > >>> >>> >> >> intel-16.0 compiler > >>> >>> >> >> intel-mpi-5.1.1 > >>> >>> >> >> > >>> >>> >> >> Program > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> Best, > >>> >>> >> >> Hom Nath > >>> >>> >> >> > >>> >>> >> >> ! simple test case for PCFIELDSPLIT > >>> >>> >> >> ! > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > ----------------------------------------------------------------------- > >>> >>> >> >> program testfs > >>> >>> >> >> implicit none > >>> >>> >> >> #include "petsc/finclude/petscsys.h" > >>> >>> >> >> #include "petsc/finclude/petscvec.h" > >>> >>> >> >> #include "petsc/finclude/petscvec.h90" > >>> >>> >> >> #include "petsc/finclude/petscmat.h" > >>> >>> >> >> #include "petsc/finclude/petscksp.h" > >>> >>> >> >> #include "petsc/finclude/petscpc.h" > >>> >>> >> >> #include "petsc/finclude/petscviewer.h" > >>> >>> >> >> #include "petsc/finclude/petscviewer.h90" > >>> >>> >> >> > >>> >>> >> >> Vec x,b,u > >>> >>> >> >> Mat A > >>> >>> >> >> KSP ksp > >>> >>> >> >> PC pc > >>> >>> >> >> PetscErrorCode ierr > >>> >>> >> >> PetscBool flg > >>> >>> >> >> PetscInt errcode,its,maxitr,myrank,nproc > >>> >>> >> >> PetscInt > nedof,nelmt,ndof,nenod,ngdof,fixdof,nsparse,neq > >>> >>> >> >> PetscInt,allocatable :: krow_sparse(:),ggdof_elmt(:,:), > >>> >>> >> >> & > >>> >>> >> >> inode_elmt(:,:) > >>> >>> >> >> > >>> >>> >> >> PetscReal,allocatable :: storef(:,:),storekmat(:,:,:) > >>> >>> >> >> PetscInt gdof0(2),gdof1(2) > >>> >>> >> >> > >>> >>> >> >> ! initialize MPI > >>> >>> >> >> > >>> >>> >> >> call MPI_INIT(errcode) > >>> >>> >> >> if(errcode /= 0)stop 'ERROR: cannot initialize MPI!' > >>> >>> >> >> call MPI_COMM_RANK(MPI_COMM_WORLD,myrank,errcode) > >>> >>> >> >> if(errcode /= 0)stop 'ERROR: cannot find processor ID' > >>> >>> >> >> call MPI_COMM_SIZE(MPI_COMM_WORLD,nproc,errcode) > >>> >>> >> >> if(errcode /= 0)stop 'ERROR: cannot find number of > processors!' > >>> >>> >> >> > >>> >>> >> >> ! define some parameters > >>> >>> >> >> ! 1D model with 4 elements. Each elements consits of 2 nodes. > >>> >>> >> >> Node 0 > >>> >>> >> >> is > >>> >>> >> >> fixed. > >>> >>> >> >> ! 1-----------2-----------3-----------4-----------5 > >>> >>> >> >> nelmt=1 ! per processor > >>> >>> >> >> nenod=2 ! number of nodes per element > >>> >>> >> >> nedof=2 ! number of DOFs per element > >>> >>> >> >> ndof=2 ! number of DOFs > >>> >>> >> >> ngdof=4 ! total number of global DOFs > >>> >>> >> >> fixdof=1 > >>> >>> >> >> > >>> >>> >> >> if(myrank==0)then > >>> >>> >> >> neq=1 ! number of equations > >>> >>> >> >> nsparse=1 > >>> >>> >> >> else > >>> >>> >> >> neq=2 ! number of equations > >>> >>> >> >> nsparse=4 > >>> >>> >> >> endif > >>> >>> >> >> > >>> >>> >> >> allocate(storef(nedof,nelmt),storekmat(nedof,nedof,nelmt), > >>> >>> >> >> & > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > ggdof_elmt(nedof,nelmt),inode_elmt(nenod,nelmt),krow_sparse(nsparse)) > >>> >>> >> >> > >>> >>> >> >> storef=0.0_8 ! local RHS vector > >>> >>> >> >> storekmat=0.0_8 ! local stiffness matrix > >>> >>> >> >> if(myrank==0)then > >>> >>> >> >> krow_sparse(:)=1 > >>> >>> >> >> storef(:,1)=(/12.5000_8, 12.5000_8/) > >>> >>> >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, > 0.0166666667_8, & > >>> >>> >> >> 0.0166666667_8, > >>> >>> >> >> 1.2333333332_8/),(/nedof,nedof/)) > >>> >>> >> >> inode_elmt(:,1) = (/1,2/) > >>> >>> >> >> ggdof_elmt(:,1) = (/0,1/) ! 0 because left node is fixed > >>> >>> >> >> elseif(myrank==1)then > >>> >>> >> >> krow_sparse(:)=(/1,1,2,2/) > >>> >>> >> >> storef(:,1)=(/12.5000_8, 12.5000_8/) > >>> >>> >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, > 0.0166666667_8, & > >>> >>> >> >> 0.0166666667_8, > >>> >>> >> >> 1.2333333332_8/),(/nedof,nedof/)) > >>> >>> >> >> inode_elmt(:,1) = (/1,2/) > >>> >>> >> >> ggdof_elmt(:,1) = (/1,2/) > >>> >>> >> >> elseif(myrank==2)then > >>> >>> >> >> krow_sparse(:)=(/1,1,2,2/) > >>> >>> >> >> storef(:,1)=(/12.5000_8, 12.5000_8/) > >>> >>> >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, > 0.0166666667_8, & > >>> >>> >> >> 0.0166666667_8, > >>> >>> >> >> 1.2333333332_8/),(/nedof,nedof/)) > >>> >>> >> >> inode_elmt(:,1) = (/1,2/) > >>> >>> >> >> ggdof_elmt(:,1) = (/2,3/) > >>> >>> >> >> elseif(myrank==3)then > >>> >>> >> >> krow_sparse(:)=(/1,1,2,2/) > >>> >>> >> >> storef(:,1)=(/12.5000_8, 22.5000_8/) > >>> >>> >> >> storekmat(:,:,1) = reshape((/1.2333333332_8, > 0.0166666667_8, & > >>> >>> >> >> 0.0166666667_8, > >>> >>> >> >> 1.2333333332_8/),(/nedof,nedof/)) > >>> >>> >> >> inode_elmt(:,1) = (/1,2/) > >>> >>> >> >> ggdof_elmt(:,1) = (/3,4/) > >>> >>> >> >> endif > >>> >>> >> >> > >>> >>> >> >> call petsc_initialize > >>> >>> >> >> call petsc_set_matrix > >>> >>> >> >> call petsc_set_vector > >>> >>> >> >> call petsc_solve > >>> >>> >> >> call petsc_finalize > >>> >>> >> >> > >>> >>> >> >> ! deallocate > >>> >>> >> >> deallocate(storef,storekmat,ggdof_elmt,inode_elmt) > >>> >>> >> >> deallocate(krow_sparse) > >>> >>> >> >> contains > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > !------------------------------------------------------------------------------- > >>> >>> >> >> subroutine petsc_initialize() > >>> >>> >> >> implicit none > >>> >>> >> >> PetscInt :: istart,iend > >>> >>> >> >> PetscInt :: nzeros_max,nzeros_min > >>> >>> >> >> PetscReal :: atol,dtol,rtol > >>> >>> >> >> PetscInt,allocatable :: nzeros(:) > >>> >>> >> >> IS gdof0_is,gdof1_is > >>> >>> >> >> > >>> >>> >> >> call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > >>> >>> >> >> call > >>> >>> >> >> PetscOptionsGetInt(PETSC_NULL_CHARACTER,'-n',ngdof,flg,ierr) > >>> >>> >> >> > >>> >>> >> >> ! count number of nonzeros per row > >>> >>> >> >> allocate(nzeros(neq)) > >>> >>> >> >> nzeros=0 > >>> >>> >> >> nzeros(krow_sparse)=nzeros(krow_sparse)+1 > >>> >>> >> >> nzeros_max=maxval(nzeros) > >>> >>> >> >> nzeros_min=minval(nzeros) > >>> >>> >> >> > >>> >>> >> >> ! create matrix > >>> >>> >> >> call MatCreate(PETSC_COMM_WORLD,A,ierr) > >>> >>> >> >> call MatSetType(A,MATMPIAIJ,ierr) > >>> >>> >> >> CHKERRQ(ierr) > >>> >>> >> >> call > MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,ngdof,ngdof,ierr) > >>> >>> >> >> CHKERRQ(ierr) > >>> >>> >> >> ! preallocation > >>> >>> >> >> call > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > MatMPIAIJSetPreallocation(A,nzeros_max,PETSC_NULL_INTEGER,nzeros_max, & > >>> >>> >> >> PETSC_NULL_INTEGER,ierr) > >>> >>> >> >> CHKERRQ(ierr) > >>> >>> >> >> > >>> >>> >> >> call MatGetOwnershipRange(A,istart,iend,ierr) > >>> >>> >> >> CHKERRQ(ierr) > >>> >>> >> >> print*,'ownership:',myrank,istart,iend > >>> >>> >> >> ! create vector > >>> >>> >> >> call VecCreate(PETSC_COMM_WORLD,x,ierr) > >>> >>> >> >> call VecSetSizes(x,PETSC_DECIDE,ngdof,ierr) > >>> >>> >> >> call VecSetType(x,VECMPI,ierr) > >>> >>> >> >> call VecDuplicate(x,b,ierr) > >>> >>> >> >> call VecDuplicate(x,u,ierr) > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > !******************************************************************************* > >>> >>> >> >> ! get IS for split fields > >>> >>> >> >> gdof0=(/0,1/) > >>> >>> >> >> gdof1=(/2,3/) > >>> >>> >> >> call > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > ISCreateGeneral(PETSC_COMM_WORLD,2,gdof0,PETSC_COPY_VALUES,gdof0_is,ierr) > >>> >>> >> >> CHKERRQ(ierr) > >>> >>> >> >> call > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > ISCreateGeneral(PETSC_COMM_WORLD,2,gdof1,PETSC_COPY_VALUES,gdof1_is,ierr) > >>> >>> >> >> CHKERRQ(ierr) > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > !******************************************************************************* > >>> >>> >> >> > >>> >>> >> >> ! Create linear solver context > >>> >>> >> >> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) > >>> >>> >> >> call KSPSetOperators(ksp,A,A,ierr) > >>> >>> >> >> > >>> >>> >> >> call KSPGetPC(ksp,pc,ierr) > >>> >>> >> >> CHKERRQ(ierr) > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > !******************************************************************************* > >>> >>> >> >> ! split PC > >>> >>> >> >> call PCFieldSplitSetIS(pc,"0",gdof0_is,ierr); > >>> >>> >> >> CHKERRQ(ierr) > >>> >>> >> >> call ISDestroy(gdof0_is,ierr) > >>> >>> >> >> CHKERRQ(ierr) > >>> >>> >> >> call PCFieldSplitSetIS(pc,"1",gdof1_is,ierr); > >>> >>> >> >> CHKERRQ(ierr) > >>> >>> >> >> call ISDestroy(gdof1_is,ierr) > >>> >>> >> >> CHKERRQ(ierr) > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > !******************************************************************************* > >>> >>> >> >> > >>> >>> >> >> rtol = 1.0d-6 > >>> >>> >> >> atol = 1.0d-6 > >>> >>> >> >> dtol = 1.0d10 > >>> >>> >> >> maxitr = 1000 > >>> >>> >> >> call KSPSetTolerances(ksp,rtol,atol,dtol,maxitr,ierr) > >>> >>> >> >> CHKERRQ(ierr) > >>> >>> >> >> call KSPSetFromOptions(ksp,ierr) > >>> >>> >> >> end subroutine petsc_initialize > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > !------------------------------------------------------------------------------- > >>> >>> >> >> > >>> >>> >> >> subroutine petsc_set_matrix() > >>> >>> >> >> > >>> >>> >> >> implicit none > >>> >>> >> >> integer :: i,i_elmt,j,ncount > >>> >>> >> >> integer :: ggdof(NEDOF),idof(NEDOF),igdof(NEDOF) > >>> >>> >> >> > >>> >>> >> >> ! Set and assemble matrix. > >>> >>> >> >> call MatZeroEntries(A,ierr) > >>> >>> >> >> CHKERRQ(ierr) > >>> >>> >> >> do i_elmt=1,nelmt > >>> >>> >> >> ggdof(:)=ggdof_elmt(:,i_elmt) > >>> >>> >> >> ggdof(:)=ggdof(:)-1 ! petsc index starts from 0 > >>> >>> >> >> ncount=0; idof=-1; igdof=-1 > >>> >>> >> >> do i=1,NEDOF > >>> >>> >> >> do j=1,NEDOF > >>> >>> >> >> if(ggdof(i).ge.0.and.ggdof(j).ge.0)then > >>> >>> >> >> call > >>> >>> >> >> MatSetValues(A,1,ggdof(i),1,ggdof(j),storekmat(i,j,i_elmt), > >>> >>> >> >> & > >>> >>> >> >> ADD_VALUES,ierr) > >>> >>> >> >> CHKERRQ(ierr) > >>> >>> >> >> endif > >>> >>> >> >> enddo > >>> >>> >> >> enddo > >>> >>> >> >> enddo > >>> >>> >> >> call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) > >>> >>> >> >> call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) > >>> >>> >> >> if(myrank==0)print*,'matrix setting & assembly complete!' > >>> >>> >> >> > >>> >>> >> >> end subroutine petsc_set_matrix > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > !------------------------------------------------------------------------------- > >>> >>> >> >> > >>> >>> >> >> subroutine petsc_set_vector() > >>> >>> >> >> implicit none > >>> >>> >> >> PetscScalar zero > >>> >>> >> >> integer :: i,i_elmt,ncount > >>> >>> >> >> integer :: ggdof(NEDOF),idof(NEDOF),igdof(NEDOF) > >>> >>> >> >> > >>> >>> >> >> ! set vector > >>> >>> >> >> zero=0.0_8 > >>> >>> >> >> call VecSet(b,zero,ierr) > >>> >>> >> >> do i_elmt=1,nelmt > >>> >>> >> >> ggdof(:)=ggdof_elmt(:,i_elmt) > >>> >>> >> >> ggdof(:)=ggdof(:)-1 ! petsc index starts from 0 > >>> >>> >> >> ncount=0; idof=-1; igdof=-1 > >>> >>> >> >> do i=1,NEDOF > >>> >>> >> >> if(ggdof(i).ge.0)then > >>> >>> >> >> call > >>> >>> >> >> VecSetValues(b,1,ggdof(i),storef(i,i_elmt),ADD_VALUES,ierr); > >>> >>> >> >> CHKERRQ(ierr) > >>> >>> >> >> endif > >>> >>> >> >> enddo > >>> >>> >> >> enddo > >>> >>> >> >> > >>> >>> >> >> ! assemble vector > >>> >>> >> >> call VecAssemblyBegin(b,ierr) > >>> >>> >> >> call VecAssemblyEnd(b,ierr) > >>> >>> >> >> if(myrank==0)print*,'vector setting & assembly complete!' > >>> >>> >> >> > >>> >>> >> >> end subroutine petsc_set_vector > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > !------------------------------------------------------------------------------- > >>> >>> >> >> > >>> >>> >> >> subroutine petsc_solve() > >>> >>> >> >> implicit none > >>> >>> >> >> PetscInt ireason > >>> >>> >> >> PetscScalar a_v(1) > >>> >>> >> >> PetscOffset a_i > >>> >>> >> >> PetscInt n > >>> >>> >> >> PetscReal,allocatable :: sdata(:) > >>> >>> >> >> > >>> >>> >> >> call VecGetSize(b,n,ierr) > >>> >>> >> >> CHKERRQ(ierr) > >>> >>> >> >> allocate(sdata(n)) > >>> >>> >> >> sdata=0.0_8 > >>> >>> >> >> call VecGetArray(b,a_v,a_i,ierr) > >>> >>> >> >> CHKERRQ(ierr) > >>> >>> >> >> sdata(1:n)=a_v(a_i+1:a_i+n) > >>> >>> >> >> call VecRestoreArray(b,a_v,a_i,ierr) > >>> >>> >> >> CHKERRQ(ierr) > >>> >>> >> >> print*,'rhs:',sdata > >>> >>> >> >> > >>> >>> >> >> call KSPSolve(ksp,b,x,ierr) > >>> >>> >> >> sdata=0.0_8 > >>> >>> >> >> call VecGetArray(x,a_v,a_i,ierr) > >>> >>> >> >> CHKERRQ(ierr) > >>> >>> >> >> sdata(1:n)=a_v(a_i+1:a_i+n) > >>> >>> >> >> call VecRestoreArray(b,a_v,a_i,ierr) > >>> >>> >> >> CHKERRQ(ierr) > >>> >>> >> >> print*,'solution:',sdata > >>> >>> >> >> > >>> >>> >> >> ! Check convergence > >>> >>> >> >> call KSPGetConvergedReason(ksp,ireason,ierr) > >>> >>> >> >> if(myrank==0)print*,'converges reason',myrank,ireason > >>> >>> >> >> call KSPGetIterationNumber(ksp,its,ierr) > >>> >>> >> >> if(myrank==0)print*,'Iterations:',its > >>> >>> >> >> deallocate(sdata) > >>> >>> >> >> end subroutine petsc_solve > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > !------------------------------------------------------------------------------- > >>> >>> >> >> > >>> >>> >> >> subroutine petsc_finalize() > >>> >>> >> >> implicit none > >>> >>> >> >> > >>> >>> >> >> ! Free work space. > >>> >>> >> >> call VecDestroy(x,ierr) > >>> >>> >> >> call VecDestroy(u,ierr) > >>> >>> >> >> call VecDestroy(b,ierr) > >>> >>> >> >> call MatDestroy(A,ierr) > >>> >>> >> >> call KSPDestroy(ksp,ierr) > >>> >>> >> >> call PetscFinalize(ierr) > >>> >>> >> >> CHKERRQ(ierr) > >>> >>> >> >> > >>> >>> >> >> end subroutine petsc_finalize > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > >>> >>> >> >> > !------------------------------------------------------------------------------- > >>> >>> >> >> > >>> >>> >> >> end program testfs > >>> >>> >> >> > >>> >>> >> >> On Tue, Feb 2, 2016 at 4:54 PM, Matthew Knepley > >>> >>> >> >> > >>> >>> >> >> wrote: > >>> >>> >> >>> On Tue, Feb 2, 2016 at 3:11 PM, Hom Nath Gharti > >>> >>> >> >>> > >>> >>> >> >>> wrote: > >>> >>> >> >>>> > >>> >>> >> >>>> Thanks a lot. I will try. > >>> >>> >> >>> > >>> >>> >> >>> > >>> >>> >> >>> Also, if you send a small test case, we can run it and tell > you > >>> >>> >> >>> the > >>> >>> >> >>> problem. > >>> >>> >> >>> > >>> >>> >> >>> Matt > >>> >>> >> >>> > >>> >>> >> >>>> > >>> >>> >> >>>> Hom Nath > >>> >>> >> >>>> > >>> >>> >> >>>> On Tue, Feb 2, 2016 at 4:02 PM, Matthew Knepley > >>> >>> >> >>>> > >>> >>> >> >>>> wrote: > >>> >>> >> >>>>> On Tue, Feb 2, 2016 at 3:01 PM, Hom Nath Gharti > >>> >>> >> >>>>> > >>> >>> >> >>>>> wrote: > >>> >>> >> >>>>>> > >>> >>> >> >>>>>> Thanks again Matt. Unfortunately got the same errors with > >>> >>> >> >>>>>> '0'. I > >>> >>> >> >>>>>> think > >>> >>> >> >>>>>> both are valid in Fortran. > >>> >>> >> >>>>> > >>> >>> >> >>>>> > >>> >>> >> >>>>> Then you will have to go in the debugger and see why its > not > >>> >>> >> >>>>> creating 4 > >>> >>> >> >>>>> splits, since this is exactly > >>> >>> >> >>>>> what it does in our tests. This is how all the SNES tests > >>> >>> >> >>>>> that I > >>> >>> >> >>>>> use > >>> >>> >> >>>>> work. I > >>> >>> >> >>>>> am sure its a Fortran > >>> >>> >> >>>>> problem. > >>> >>> >> >>>>> > >>> >>> >> >>>>> Thanks, > >>> >>> >> >>>>> > >>> >>> >> >>>>> Matt > >>> >>> >> >>>>> > >>> >>> >> >>>>>> > >>> >>> >> >>>>>> Hom Nath > >>> >>> >> >>>>>> > >>> >>> >> >>>>>> On Tue, Feb 2, 2016 at 3:42 PM, Matthew Knepley > >>> >>> >> >>>>>> > >>> >>> >> >>>>>> wrote: > >>> >>> >> >>>>>>> On Tue, Feb 2, 2016 at 2:35 PM, Hom Nath Gharti > >>> >>> >> >>>>>>> > >>> >>> >> >>>>>>> wrote: > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> Thank you so much Matt. > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> I now tried the following: > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> ====================================================== > >>> >>> >> >>>>>>>> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) > >>> >>> >> >>>>>>>> call KSPGetPC(ksp,pc,ierr) > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); > >>> >>> >> >>>>>>>> call ISDestroy(gdofu_is,ierr) > >>> >>> >> >>>>>>>> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); > >>> >>> >> >>>>>>>> call ISDestroy(gdofchi_is,ierr) > >>> >>> >> >>>>>>>> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); > >>> >>> >> >>>>>>>> call ISDestroy(gdofp_is,ierr) > >>> >>> >> >>>>>>>> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); > >>> >>> >> >>>>>>>> call ISDestroy(gdofphi_is,ierr) > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> ! Amat changes in each time steps > >>> >>> >> >>>>>>>> call KSPSetOperators(ksp,Amat,Amat,ierr) !version >= > 3.5.0 > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> call KSPSolve(ksp,bvec,xvec,ierr) > >>> >>> >> >>>>>>>> ====================================================== > >>> >>> >> >>>>>>> > >>> >>> >> >>>>>>> > >>> >>> >> >>>>>>> I am guessing that "0" is not a valid string for your > >>> >>> >> >>>>>>> Fortran > >>> >>> >> >>>>>>> compiler. > >>> >>> >> >>>>>>> Are > >>> >>> >> >>>>>>> you sure > >>> >>> >> >>>>>>> its not single quotes '0'? > >>> >>> >> >>>>>>> > >>> >>> >> >>>>>>> Matt > >>> >>> >> >>>>>>> > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> But I get the following error: > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> [0]PETSC ERROR: --------------------- Error Message > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > -------------------------------------------------------------- > >>> >>> >> >>>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data > >>> >>> >> >>>>>>>> [0]PETSC ERROR: Unhandled case, must have at least two > >>> >>> >> >>>>>>>> fields, > >>> >>> >> >>>>>>>> not 1 > >>> >>> >> >>>>>>>> [0]PETSC ERROR: See > >>> >>> >> >>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html > for > >>> >>> >> >>>>>>>> trouble > >>> >>> >> >>>>>>>> shooting. > >>> >>> >> >>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, > 2015 > >>> >>> >> >>>>>>>> [0]PETSC ERROR: > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > /tigress/hgharti/gitwork/SPECFEM3D_GLOBEVSPP/./bin/xspecfem3D > >>> >>> >> >>>>>>>> on > >>> >>> >> >>>>>>>> a > >>> >>> >> >>>>>>>> arch-linux2-c-debug named tiger-r3c1n7 by hgharti Tue > Feb > >>> >>> >> >>>>>>>> 2 > >>> >>> >> >>>>>>>> 15: > >>> >>> >> >>>>>>>> 29:30 2016 > >>> >>> >> >>>>>>>> [0]PETSC ERROR: Configure options > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > --with-blas-lapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64/ > >>> >>> >> >>>>>>>> --with-cc=mpicc --with-cxx=mpicxx --wit > >>> >>> >> >>>>>>>> h-fc=mpif90 --with-mpiexec=mpiexec --with-debugging=1 > >>> >>> >> >>>>>>>> --download-scalapack --download-mumps --download-pastix > >>> >>> >> >>>>>>>> --download-superlu --download-superlu_dist > >>> >>> >> >>>>>>>> --download-metis > >>> >>> >> >>>>>>>> --download-parmetis --download-ptscotch > --download-hypre > >>> >>> >> >>>>>>>> [0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 469 > in > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > >>> >>> >> >>>>>>>> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 486 in > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > >>> >>> >> >>>>>>>> [0]PETSC ERROR: #3 PCSetUp() line 983 in > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/pc/interface/precon.c > >>> >>> >> >>>>>>>> [0]PETSC ERROR: #4 KSPSetUp() line 332 in > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c > >>> >>> >> >>>>>>>> [0]PETSC ERROR: #5 KSPSolve() line 546 in > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interface/itfunc.c > >>> >>> >> >>>>>>>> forrtl: error (76): Abort trap signal > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> Am I missing something? > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> Thanks, > >>> >>> >> >>>>>>>> Hom Nath > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> On Tue, Feb 2, 2016 at 3:24 PM, Matthew Knepley > >>> >>> >> >>>>>>>> > >>> >>> >> >>>>>>>> wrote: > >>> >>> >> >>>>>>>>> On Tue, Feb 2, 2016 at 2:20 PM, Hom Nath Gharti > >>> >>> >> >>>>>>>>> > >>> >>> >> >>>>>>>>> wrote: > >>> >>> >> >>>>>>>>>> > >>> >>> >> >>>>>>>>>> Hi Matt, hi all, > >>> >>> >> >>>>>>>>>> > >>> >>> >> >>>>>>>>>> I am trying to use PCFIELDSPLIT for my solver which > >>> >>> >> >>>>>>>>>> consists > >>> >>> >> >>>>>>>>>> of > >>> >>> >> >>>>>>>>>> 4 > >>> >>> >> >>>>>>>>>> different variables, namely, u (displacement vector), > >>> >>> >> >>>>>>>>>> \chi > >>> >>> >> >>>>>>>>>> (displacement potential), p(pressure), and \phi > (gravity > >>> >>> >> >>>>>>>>>> potential). > >>> >>> >> >>>>>>>>>> > >>> >>> >> >>>>>>>>>> My code segment looks like the following: > >>> >>> >> >>>>>>>>>> > ====================================================== > >>> >>> >> >>>>>>>>>> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) > >>> >>> >> >>>>>>>>>> call KSPGetPC(ksp,pc,ierr) > >>> >>> >> >>>>>>>>>> nsplit=4 > >>> >>> >> >>>>>>>>>> call PCFieldSplitSetBlockSize(pc,nsplit,ierr); > >>> >>> >> >>>>>>>>> > >>> >>> >> >>>>>>>>> > >>> >>> >> >>>>>>>>> You do not need the statement above. > >>> >>> >> >>>>>>>>> > >>> >>> >> >>>>>>>>>> > >>> >>> >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"0",gdofu_is,ierr); > >>> >>> >> >>>>>>>>>> call ISDestroy(gdofu_is,ierr) > >>> >>> >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"1",gdofchi_is,ierr); > >>> >>> >> >>>>>>>>>> call ISDestroy(gdofchi_is,ierr) > >>> >>> >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"2",gdofp_is,ierr); > >>> >>> >> >>>>>>>>>> call ISDestroy(gdofp_is,ierr) > >>> >>> >> >>>>>>>>>> call PCFieldSplitSetIS(pc,"3",gdofphi_is,ierr); > >>> >>> >> >>>>>>>>>> call ISDestroy(gdofphi_is,ierr) > >>> >>> >> >>>>>>>>>> > >>> >>> >> >>>>>>>>>> call PCFieldSplitGetSubKSP(pc,nsplit,subksp,ierr); > >>> >>> >> >>>>>>>>> > >>> >>> >> >>>>>>>>> > >>> >>> >> >>>>>>>>> These SetOperators() calls are wrong. I am not sure > why > >>> >>> >> >>>>>>>>> you > >>> >>> >> >>>>>>>>> want > >>> >>> >> >>>>>>>>> them > >>> >>> >> >>>>>>>>> here. > >>> >>> >> >>>>>>>>> Also, that means you do not need the call above. > >>> >>> >> >>>>>>>>> > >>> >>> >> >>>>>>>>> Thanks, > >>> >>> >> >>>>>>>>> > >>> >>> >> >>>>>>>>> Matt > >>> >>> >> >>>>>>>>> > >>> >>> >> >>>>>>>>>> > >>> >>> >> >>>>>>>>>> call KSPSetOperators(subksp(1),Amat,Amat,ierr); > >>> >>> >> >>>>>>>>>> call KSPSetOperators(subksp(2),Amat,Amat,ierr); > >>> >>> >> >>>>>>>>>> call KSPSetOperators(subksp(3),Amat,Amat,ierr); > >>> >>> >> >>>>>>>>>> call KSPSetOperators(subksp(4),Amat,Amat,ierr); > >>> >>> >> >>>>>>>>>> > >>> >>> >> >>>>>>>>>> call KSPSolve(ksp,bvec,xvec,ierr) > >>> >>> >> >>>>>>>>>> > ====================================================== > >>> >>> >> >>>>>>>>>> > >>> >>> >> >>>>>>>>>> But I am getting the following error: > >>> >>> >> >>>>>>>>>> [79]PETSC ERROR: Null argument, when expecting valid > >>> >>> >> >>>>>>>>>> pointer > >>> >>> >> >>>>>>>>>> [79]PETSC ERROR: Null Object: Parameter # 1 > >>> >>> >> >>>>>>>>>> [79]PETSC ERROR: #1 KSPSetOperators() line 536 in > >>> >>> >> >>>>>>>>>> > >>> >>> >> >>>>>>>>>> > >>> >>> >> >>>>>>>>>> > /home/hgharti/lsoft/petsc-3.6.3-intel16-mpi5/src/ksp/ksp/interf > >>> >>> >> >>>>>>>>>> > >>> >>> >> >>>>>>>>>> It looks like I am doing something wrong in "call > >>> >>> >> >>>>>>>>>> KSPSetOperators" > >>> >>> >> >>>>>>>>>> but > >>> >>> >> >>>>>>>>>> I could not figure that out. > >>> >>> >> >>>>>>>>>> > >>> >>> >> >>>>>>>>>> Could anybody help me to fix this problem? I looked > into > >>> >>> >> >>>>>>>>>> almost > >>> >>> >> >>>>>>>>>> all > >>> >>> >> >>>>>>>>>> related examples but I could not really figure out > the > >>> >>> >> >>>>>>>>>> correct > >>> >>> >> >>>>>>>>>> steps > >>> >>> >> >>>>>>>>>> after "call PCFieldSplitSetIS". > >>> >>> >> >>>>>>>>>> > >>> >>> >> >>>>>>>>>> Any help will be greatly appreciated. > >>> >>> >> >>>>>>>>>> > >>> >>> >> >>>>>>>>>> Best, > >>> >>> >> >>>>>>>>>> Hom nath > >>> >>> >> >>>>>>>>>> > >>> >>> >> >>>>>>>>>> On Sun, Jan 24, 2016 at 7:14 PM, Hom Nath Gharti > >>> >>> >> >>>>>>>>>> > >>> >>> >> >>>>>>>>>> wrote: > >>> >>> >> >>>>>>>>>>> Thank you so much Matt! I will try. > >>> >>> >> >>>>>>>>>>> > >>> >>> >> >>>>>>>>>>> Hom Nath > >>> >>> >> >>>>>>>>>>> > >>> >>> >> >>>>>>>>>>> On Sun, Jan 24, 2016 at 6:26 AM, Matthew Knepley > >>> >>> >> >>>>>>>>>>> > >>> >>> >> >>>>>>>>>>> wrote: > >>> >>> >> >>>>>>>>>>>> On Fri, Jan 22, 2016 at 2:19 PM, Hom Nath Gharti > >>> >>> >> >>>>>>>>>>>> > >>> >>> >> >>>>>>>>>>>> wrote: > >>> >>> >> >>>>>>>>>>>>> > >>> >>> >> >>>>>>>>>>>>> Dear all, > >>> >>> >> >>>>>>>>>>>>> > >>> >>> >> >>>>>>>>>>>>> I am new to PcFieldSplit. > >>> >>> >> >>>>>>>>>>>>> > >>> >>> >> >>>>>>>>>>>>> I have a matrix formed using MATMPIAIJ. Is it > >>> >>> >> >>>>>>>>>>>>> possible to > >>> >>> >> >>>>>>>>>>>>> use > >>> >>> >> >>>>>>>>>>>>> PCFIELDSPLIT operations in this type of matrix? Or > >>> >>> >> >>>>>>>>>>>>> does > >>> >>> >> >>>>>>>>>>>>> it > >>> >>> >> >>>>>>>>>>>>> have > >>> >>> >> >>>>>>>>>>>>> to > >>> >>> >> >>>>>>>>>>>>> be > >>> >>> >> >>>>>>>>>>>>> MATMPIBIJ or MATNEST format? > >>> >>> >> >>>>>>>>>>>> > >>> >>> >> >>>>>>>>>>>> > >>> >>> >> >>>>>>>>>>>> Yes, you can split AIJ. > >>> >>> >> >>>>>>>>>>>> > >>> >>> >> >>>>>>>>>>>>> > >>> >>> >> >>>>>>>>>>>>> If possible for MATMPIAIJ, could anybody provide > me a > >>> >>> >> >>>>>>>>>>>>> simple > >>> >>> >> >>>>>>>>>>>>> example > >>> >>> >> >>>>>>>>>>>>> or few steps? Variables in the equations are > >>> >>> >> >>>>>>>>>>>>> displacement > >>> >>> >> >>>>>>>>>>>>> vector, > >>> >>> >> >>>>>>>>>>>>> scalar potential and pressure. > >>> >>> >> >>>>>>>>>>>> > >>> >>> >> >>>>>>>>>>>> > >>> >>> >> >>>>>>>>>>>> If you do not have a collocated discretization, > then > >>> >>> >> >>>>>>>>>>>> you > >>> >>> >> >>>>>>>>>>>> have > >>> >>> >> >>>>>>>>>>>> to > >>> >>> >> >>>>>>>>>>>> use > >>> >>> >> >>>>>>>>>>>> > >>> >>> >> >>>>>>>>>>>> > >>> >>> >> >>>>>>>>>>>> > >>> >>> >> >>>>>>>>>>>> > >>> >>> >> >>>>>>>>>>>> > >>> >>> >> >>>>>>>>>>>> > >>> >>> >> >>>>>>>>>>>> > >>> >>> >> >>>>>>>>>>>> > >>> >>> >> >>>>>>>>>>>> > >>> >>> >> >>>>>>>>>>>> > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html > >>> >>> >> >>>>>>>>>>>> > >>> >>> >> >>>>>>>>>>>> Thanks, > >>> >>> >> >>>>>>>>>>>> > >>> >>> >> >>>>>>>>>>>> Matt > >>> >>> >> >>>>>>>>>>>> > >>> >>> >> >>>>>>>>>>>>> > >>> >>> >> >>>>>>>>>>>>> Thanks for help. > >>> >>> >> >>>>>>>>>>>>> > >>> >>> >> >>>>>>>>>>>>> Hom Nath > >>> >>> >> >>>>>>>>>>>> > >>> >>> >> >>>>>>>>>>>> > >>> >>> >> >>>>>>>>>>>> > >>> >>> >> >>>>>>>>>>>> > >>> >>> >> >>>>>>>>>>>> -- > >>> >>> >> >>>>>>>>>>>> What most experimenters take for granted before > they > >>> >>> >> >>>>>>>>>>>> begin > >>> >>> >> >>>>>>>>>>>> their > >>> >>> >> >>>>>>>>>>>> experiments > >>> >>> >> >>>>>>>>>>>> is infinitely more interesting than any results to > >>> >>> >> >>>>>>>>>>>> which > >>> >>> >> >>>>>>>>>>>> their > >>> >>> >> >>>>>>>>>>>> experiments > >>> >>> >> >>>>>>>>>>>> lead. > >>> >>> >> >>>>>>>>>>>> -- Norbert Wiener > >>> >>> >> >>>>>>>>> > >>> >>> >> >>>>>>>>> > >>> >>> >> >>>>>>>>> > >>> >>> >> >>>>>>>>> > >>> >>> >> >>>>>>>>> -- > >>> >>> >> >>>>>>>>> What most experimenters take for granted before they > >>> >>> >> >>>>>>>>> begin > >>> >>> >> >>>>>>>>> their > >>> >>> >> >>>>>>>>> experiments > >>> >>> >> >>>>>>>>> is infinitely more interesting than any results to > which > >>> >>> >> >>>>>>>>> their > >>> >>> >> >>>>>>>>> experiments > >>> >>> >> >>>>>>>>> lead. > >>> >>> >> >>>>>>>>> -- Norbert Wiener > >>> >>> >> >>>>>>> > >>> >>> >> >>>>>>> > >>> >>> >> >>>>>>> > >>> >>> >> >>>>>>> > >>> >>> >> >>>>>>> -- > >>> >>> >> >>>>>>> What most experimenters take for granted before they > begin > >>> >>> >> >>>>>>> their > >>> >>> >> >>>>>>> experiments > >>> >>> >> >>>>>>> is infinitely more interesting than any results to which > >>> >>> >> >>>>>>> their > >>> >>> >> >>>>>>> experiments > >>> >>> >> >>>>>>> lead. > >>> >>> >> >>>>>>> -- Norbert Wiener > >>> >>> >> >>>>> > >>> >>> >> >>>>> > >>> >>> >> >>>>> > >>> >>> >> >>>>> > >>> >>> >> >>>>> -- > >>> >>> >> >>>>> What most experimenters take for granted before they begin > >>> >>> >> >>>>> their > >>> >>> >> >>>>> experiments > >>> >>> >> >>>>> is infinitely more interesting than any results to which > >>> >>> >> >>>>> their > >>> >>> >> >>>>> experiments > >>> >>> >> >>>>> lead. > >>> >>> >> >>>>> -- Norbert Wiener > >>> >>> >> >>> > >>> >>> >> >>> > >>> >>> >> >>> > >>> >>> >> >>> > >>> >>> >> >>> -- > >>> >>> >> >>> What most experimenters take for granted before they begin > >>> >>> >> >>> their > >>> >>> >> >>> experiments > >>> >>> >> >>> is infinitely more interesting than any results to which > their > >>> >>> >> >>> experiments > >>> >>> >> >>> lead. > >>> >>> >> >>> -- Norbert Wiener > >>> >>> >> > > >>> >>> > > >>> >>> > > >>> >>> > > >>> >>> > > >>> >>> > -- > >>> >>> > What most experimenters take for granted before they begin their > >>> >>> > experiments > >>> >>> > is infinitely more interesting than any results to which their > >>> >>> > experiments > >>> >>> > lead. > >>> >>> > -- Norbert Wiener > >>> >> > >>> >> > >>> >> > >>> >> > >>> >> -- > >>> >> What most experimenters take for granted before they begin their > >>> >> experiments > >>> >> is infinitely more interesting than any results to which their > >>> >> experiments > >>> >> lead. > >>> >> -- Norbert Wiener > >> > >> > >> > >> > >> -- > >> What most experimenters take for granted before they begin their > experiments > >> is infinitely more interesting than any results to which their > experiments > >> lead. > >> -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From rongliang.chan at gmail.com Tue Feb 16 21:44:27 2016 From: rongliang.chan at gmail.com (Rongliang Chen) Date: Wed, 17 Feb 2016 11:44:27 +0800 Subject: [petsc-users] SNESComputeJacobianDefaultColor use too much memory In-Reply-To: <684F7147-4F58-42E3-9C9E-2C2297383276@mcs.anl.gov> References: <56C319D0.9060208@gmail.com> <684F7147-4F58-42E3-9C9E-2C2297383276@mcs.anl.gov> Message-ID: <56C3EC9B.5080306@gmail.com> Dear Barry, Many thanks for your reply. I checked with the valgrind and did not obtain any outputs (|massif.out.|) because the job was killed before it reached the end. Then I switch to a smaller case, it works well and one of the output is attached (I did not find any useful information in it). The output with the option -mat_coloring_view is followed, which shows that the number of colors is 65. Any ideas for this? MatColoring Object: 480 MPI processes type: sl Weight type: RANDOM Distance 2, Max. Colors 65535 Number of colors 65 Number of total columns 1637350 Best regards, Rongliang On 02/17/2016 01:13 AM, Barry Smith wrote: > How many colors are needed? > > You need to produce a breakdown of where all the memory is being used. For example valgrind with the http://valgrind.org/docs/manual/ms-manual.html > > Barry > >> On Feb 16, 2016, at 6:45 AM, Rongliang Chen wrote: >> >> Dear all, >> >> I am using the DMPlex to solve a PDE on a unstructured mesh and I use the SNESComputeJacobianDefaultColor to compute the Jacobian matrix. >> >> My code works well for small problems (such as problem with 3.3x10^5 cells using 120 cores) but when I increase the number of cells (2.6x10^6 cells using 1920 cores), I go out of memory in the function MatColoringApply. It shows that one of the cores uses over 11G memory. I think this is unreasonable. Do you have any suggestions for debugging this problem? >> >> Best regards, >> Rongliang -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- desc: (none) cmd: ./Nwtun -f /HOME/siat_crl_1/WORKSPACE/rlchen/meshes/FVM/3DNACA0012AIRFOIL_Hex75x25x50x40.exo -overlap 1 -options_file options.txt -ksp_monitor -time_integral_method 1 -initial_time 0.0 -final_time 1.0 -dt 1.e-3 -timestep 2 -max_time_its 0 -includeenergy -reconstruct -PressureFlux -testcase 3DNACA -output_solution -solutionfile /HOME/siat_crl_1/WORKSPACE/rlchen/results/FVM/3DNACA0012AIRFOIL_Hex150x50x100x80_imp -steps_output 100 -inflow_u 349.02 -viscosity 0.01 -k 0.026 -monitor_step_gap -R 287.02 -adiabatic 1.4 -Euler -Attack 10 -Mach 0.8 -fd_jacobian_color -limiter zero -RiemannSolver Rusanov -CFL 0.9 -reconstruct -second_order -parallel_csr -parallel_output -DiffFuncForJac -mat_coloring_view time_unit: i #----------- snapshot=0 #----------- time=0 mem_heap_B=0 mem_heap_extra_B=0 mem_stacks_B=0 heap_tree=empty From amneetb at live.unc.edu Tue Feb 16 22:37:18 2016 From: amneetb at live.unc.edu (Bhalla, Amneet Pal S) Date: Wed, 17 Feb 2016 04:37:18 +0000 Subject: [petsc-users] Error with MatSchurComplement Message-ID: Hi Folks, I am getting error with odd number of processors with PCFIELDSPLIT. The error log is attached. Any hints where to start digging? +++++++++++++++++++++++++++++++++++++++++++++++++++ At beginning of timestep # 0 Simulation time is 0 Residual norms for ib_ solve. 0 KSP unpreconditioned resid norm 5.093934451651e+05 true resid norm 5.093934451651e+05 ||r(i)||/||b|| 1.000000000000e+00 [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [2]PETSC ERROR: Nonconforming object sizes [2]PETSC ERROR: Local columns of A10 1072 do not equal local rows of A00 1104 [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [2]PETSC ERROR: Petsc Release Version 3.6.1, unknown [2]PETSC ERROR: ./main2d on a darwin-dbg named Amneets-MBP.attlocal.net by Taylor Tue Feb 16 20:30:45 2016 [2]PETSC ERROR: Configure options --CC=mpicc --CXX=mpicxx --FC=mpif90 --PETSC_ARCH=darwin-dbg --with-debugging=1 --with-c++-support=1 --with-hypre=1 --download-hypre=1 --with-hdf5=yes --with-hdf5-dir=/Users/Taylor/Documents/SOFTWARES/HDF5/ [2]PETSC ERROR: #1 MatSchurComplementSetSubMatrices() line 261 in /Users/Taylor/Documents/SOFTWARES/PETSc-BitBucket/PETSc/src/ksp/ksp/utils/schurm.c [2]PETSC ERROR: #2 PCSetUp_FieldSplit() line 714 in /Users/Taylor/Documents/SOFTWARES/PETSc-BitBucket/PETSc/src/ksp/pc/impls/fieldsplit/fieldsplit.c [2]PETSC ERROR: #3 PCSetUp() line 983 in /Users/Taylor/Documents/SOFTWARES/PETSc-BitBucket/PETSc/src/ksp/pc/interface/precon.c [2]PETSC ERROR: #4 KSPSetUp() line 332 in /Users/Taylor/Documents/SOFTWARES/PETSc-BitBucket/PETSc/src/ksp/ksp/interface/itfunc.c [2]PETSC ERROR: #5 KSPSolve() line 546 in /Users/Taylor/Documents/SOFTWARES/PETSc-BitBucket/PETSc/src/ksp/ksp/interface/itfunc.c [2]PETSC ERROR: #6 solveSystem() line 229 in ../../../IBAMR/ibtk/lib/../src/solvers/impls/PETScLevelSolver.cpp P=00002:Program abort called in file ``../../../IBAMR/ibtk/lib/../src/solvers/impls/PETScLevelSolver.cpp'' at line 229 P=00002:ERROR MESSAGE: P=00002: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 2 *** Warning, AutoPtr is deprecated and will be removed in a future library version! ./include/libmesh/auto_ptr.h, line 271, compiled Feb 3 2016 at 10:29:17 *** ? Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Feb 16 23:30:38 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 16 Feb 2016 23:30:38 -0600 Subject: [petsc-users] SNESComputeJacobianDefaultColor use too much memory In-Reply-To: <56C3EC9B.5080306@gmail.com> References: <56C319D0.9060208@gmail.com> <684F7147-4F58-42E3-9C9E-2C2297383276@mcs.anl.gov> <56C3EC9B.5080306@gmail.com> Message-ID: <1199051F-2EC1-4462-A7C0-E2DDCA95A4C3@mcs.anl.gov> Hmm, something didn't work right with massif. I should give a bunch of ASCII information about how much memory is used at different times in the code. You may need to play around with the massif options and google documentation on how to get it to provide useful information. Once it produces the useful information it will be very helpful. time=0 mem_heap_B=0 mem_heap_extra_B=0 mem_stacks_B=0 heap_tree=empty > On Feb 16, 2016, at 9:44 PM, Rongliang Chen wrote: > > Dear Barry, > > Many thanks for your reply. > > I checked with the valgrind and did not obtain any outputs (massif.out.) because the job was killed before it reached the end. > > Then I switch to a smaller case, it works well and one of the output is attached (I did not find any useful information in it). The output with the option -mat_coloring_view is followed, which shows that the number of colors is 65. Any ideas for this? > > MatColoring Object: 480 MPI processes > type: sl > Weight type: RANDOM > Distance 2, Max. Colors 65535 > Number of colors 65 > Number of total columns 1637350 > > Best regards, > Rongliang > > On 02/17/2016 01:13 AM, Barry Smith wrote: >> How many colors are needed? >> >> You need to produce a breakdown of where all the memory is being used. For example valgrind with the >> http://valgrind.org/docs/manual/ms-manual.html >> >> >> Barry >> >> >>> On Feb 16, 2016, at 6:45 AM, Rongliang Chen >>> wrote: >>> >>> Dear all, >>> >>> I am using the DMPlex to solve a PDE on a unstructured mesh and I use the SNESComputeJacobianDefaultColor to compute the Jacobian matrix. >>> >>> My code works well for small problems (such as problem with 3.3x10^5 cells using 120 cores) but when I increase the number of cells (2.6x10^6 cells using 1920 cores), I go out of memory in the function MatColoringApply. It shows that one of the cores uses over 11G memory. I think this is unreasonable. Do you have any suggestions for debugging this problem? >>> >>> Best regards, >>> Rongliang >>> > > From jed at jedbrown.org Tue Feb 16 16:49:33 2016 From: jed at jedbrown.org (Jed Brown) Date: Tue, 16 Feb 2016 15:49:33 -0700 Subject: [petsc-users] Inverse local to global mapping? In-Reply-To: <4329728.aZqEON1xM9@asaru> References: <4329728.aZqEON1xM9@asaru> Message-ID: <871t8cuv7m.fsf@jedbrown.org> Florian Lindner writes: > Hello, > > I want to build a global vector (and matrix) from local data. The local data has a global index, which can be non-contiguous i.e. global index (0, 5, 2) is on rank 0, (1, 4, 3) is on rank 1. To keep all local data on the local part of vector I need a mapping: > > Application -> PETSc > (0, 5, 2) -> (0, 1, 2) > (1, 4, 3) -> (3, 4, 5) > > It can happen that rank 1 also need to touch vector[5] (application ordering) i.e. vector[1] (petsc ordering) on rank 0. > > I can produce a mapping using index sets: > > ierr = ISCreateGeneral(PETSC_COMM_WORLD, indizes.size(), indizes.data(), PETSC_COPY_VALUES, &ISlocal); CHKERRV(ierr); > ierr = ISAllGather(ISlocal, &ISglobal); CHKERRV(ierr); // Gather the IS from all processors > ierr = ISLocalToGlobalMappingCreateIS(ISglobal, &ISmapping); CHKERRV(ierr); // Make it a mapping > > I construct std::vector indizes by walking though all local indizes. This way I have a mapping ISmapping that looks like that: > > 0 0 > 1 5 > 2 2 > 3 1 > 4 4 > 5 3 > > Now I can use that mapping: VecSetLocalToGlobalMapping(vec, mapping); > > but this does not work like it should (for me). If I use VecSetValueLocal to set element 5 it gets mapped to 3, element 1 gets mapped to 5. > I want to have it reversed, accessing element 3 gets mapped to 5, element 1 to 3. ISInvertPermutation would do what you're asking for, but beware that this is not memory scalable. I recommend fixing your algorithm so you don't need this inverse mapping. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From rongliang.chan at gmail.com Tue Feb 16 23:49:13 2016 From: rongliang.chan at gmail.com (Rongliang Chen) Date: Wed, 17 Feb 2016 13:49:13 +0800 Subject: [petsc-users] SNESComputeJacobianDefaultColor use too much memory In-Reply-To: <1199051F-2EC1-4462-A7C0-E2DDCA95A4C3@mcs.anl.gov> References: <56C319D0.9060208@gmail.com> <684F7147-4F58-42E3-9C9E-2C2297383276@mcs.anl.gov> <56C3EC9B.5080306@gmail.com> <1199051F-2EC1-4462-A7C0-E2DDCA95A4C3@mcs.anl.gov> Message-ID: <56C409D9.70002@gmail.com> Hi Barry, I run the code with valgrind on a workstation for a smaller case and it produces some ASCII information (see attached). Is this helpful? Best, Rongliang On 02/17/2016 01:30 PM, Barry Smith wrote: > Hmm, something didn't work right with massif. I should give a bunch of ASCII information about how much memory is used at different times in the code. You may need to play around with the massif options and google documentation on how to get it to provide useful information. Once it produces the useful information it will be very helpful. > > time=0 > mem_heap_B=0 > mem_heap_extra_B=0 > mem_stacks_B=0 > heap_tree=empty > > >> On Feb 16, 2016, at 9:44 PM, Rongliang Chen wrote: >> >> Dear Barry, >> >> Many thanks for your reply. >> >> I checked with the valgrind and did not obtain any outputs (massif.out.) because the job was killed before it reached the end. >> >> Then I switch to a smaller case, it works well and one of the output is attached (I did not find any useful information in it). The output with the option -mat_coloring_view is followed, which shows that the number of colors is 65. Any ideas for this? >> >> MatColoring Object: 480 MPI processes >> type: sl >> Weight type: RANDOM >> Distance 2, Max. Colors 65535 >> Number of colors 65 >> Number of total columns 1637350 >> >> Best regards, >> Rongliang >> >> On 02/17/2016 01:13 AM, Barry Smith wrote: >>> How many colors are needed? >>> >>> You need to produce a breakdown of where all the memory is being used. For example valgrind with the >>> http://valgrind.org/docs/manual/ms-manual.html >>> >>> >>> Barry >>> >>> >>>> On Feb 16, 2016, at 6:45 AM, Rongliang Chen >>>> wrote: >>>> >>>> Dear all, >>>> >>>> I am using the DMPlex to solve a PDE on a unstructured mesh and I use the SNESComputeJacobianDefaultColor to compute the Jacobian matrix. >>>> >>>> My code works well for small problems (such as problem with 3.3x10^5 cells using 120 cores) but when I increase the number of cells (2.6x10^6 cells using 1920 cores), I go out of memory in the function MatColoringApply. It shows that one of the cores uses over 11G memory. I think this is unreasonable. Do you have any suggestions for debugging this problem? >>>> >>>> Best regards, >>>> Rongliang >>>> >> -------------- next part -------------- desc: (none) cmd: ./Nwtun -malloc off -f ../meshes/2DNACA0012AIRFOIL_Hex50x30x10x2.exo -overlap 1 -options_file options.txt -time_integral_method 1 -initial_time 0.0 -final_time 0.5 -dt 1.e-3 -timestep 1 -max_time_its 0 -includeenergy -reconstruct -PressureFlux -testcase 2DNACA -symmetricbc -output_solution -solutionfile results/2DNACAImp -steps_output 2 -inflow_u 349.02 -viscosity 0.01 -k 0.026 -R 287.02 -adiabatic 1.4 -Euler -Attack 10 -Mach 0.8 -fd_jacobian_color -monitor_step_gap -monitormass -parallel_csr -parallel_output -limiter zero -RiemannSolver Rusanov -CFL 0.9 -reconstruct -second_order -DiffFuncForJac -mat_coloring_view time_unit: i #----------- snapshot=0 #----------- time=0 mem_heap_B=0 mem_heap_extra_B=0 mem_stacks_B=0 heap_tree=empty #----------- snapshot=1 #----------- time=466351390 mem_heap_B=5106022 mem_heap_extra_B=13618 mem_stacks_B=0 heap_tree=empty #----------- snapshot=2 #----------- time=746348011 mem_heap_B=5897454 mem_heap_extra_B=14138 mem_stacks_B=0 heap_tree=empty #----------- snapshot=3 #----------- time=1208542856 mem_heap_B=9098558 mem_heap_extra_B=14114 mem_stacks_B=0 heap_tree=empty #----------- snapshot=4 #----------- time=1585087744 mem_heap_B=6260499 mem_heap_extra_B=14581 mem_stacks_B=0 heap_tree=empty #----------- snapshot=5 #----------- time=1923773721 mem_heap_B=6299991 mem_heap_extra_B=14641 mem_stacks_B=0 heap_tree=empty #----------- snapshot=6 #----------- time=2304638724 mem_heap_B=30905886 mem_heap_extra_B=12810 mem_stacks_B=0 heap_tree=peak n2: 30905886 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n8: 30364925 0x57877E: PetscMallocAlign (mal.c:27) n1: 24893576 0x9832FA: DMPlexGetAdjacency_Internal (plexdistribute.c:220) n1: 24893576 0x9839A6: DMPlexGetAdjacency (plexdistribute.c:263) n1: 24893576 0x554357: DMPlexEnlargePartitionNwtun (plexpartition.c:507) n1: 24893576 0x555C98: DMPlexCreatePartitionNwtun (plexpartition.c:669) n1: 24893576 0x556B5F: DMPlexDistributeNwtun (plexdistribute.c:68) n1: 24893576 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 24893576 0x4076DF: main (Nwtun.c:55) n0: 1033045 in 104 places, all below massif's threshold (01.00%) n1: 855200 0x462A1D: DMSetUp_Plex (plex.c:1582) n2: 855200 0x50B88B: DMSetUp (dm.c:560) n1: 855200 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229) n1: 855200 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 855200 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 855200 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 855200 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 855200 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 855200 0x462AB1: DMSetUp_Plex (plex.c:1583) n2: 855200 0x50B88B: DMSetUp (dm.c:560) n1: 855200 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229) n1: 855200 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 855200 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 855200 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 855200 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 855200 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n2: 855200 0x46371F: DMPlexSymmetrize (plex.c:1655) n1: 855200 0x9E4D06: DMPlexInterpolateFaces_Internal (plexinterpolate.c:303) n1: 855200 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 855200 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 855200 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 855200 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 855200 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 710752 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n0: 710752 in 6 places, all below massif's threshold (01.00%) n1: 710752 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n0: 710752 in 6 places, all below massif's threshold (01.00%) n1: 451200 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132) n1: 451200 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 451200 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 451200 0x4076DF: main (Nwtun.c:55) n0: 540961 in 134 places, all below massif's threshold (01.00%) #----------- snapshot=7 #----------- time=2801678661 mem_heap_B=6355101 mem_heap_extra_B=11195 mem_stacks_B=0 heap_tree=detailed n5: 6355101 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n11: 5823604 0x57877E: PetscMallocAlign (mal.c:27) n1: 855200 0x462A1D: DMSetUp_Plex (plex.c:1582) n2: 855200 0x50B88B: DMSetUp (dm.c:560) n1: 855200 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229) n1: 855200 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 855200 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 855200 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 855200 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 855200 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 855200 0x462AB1: DMSetUp_Plex (plex.c:1583) n2: 855200 0x50B88B: DMSetUp (dm.c:560) n1: 855200 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229) n1: 855200 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 855200 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 855200 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 855200 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 855200 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n2: 855200 0x46371F: DMPlexSymmetrize (plex.c:1655) n1: 855200 0x9E4D06: DMPlexInterpolateFaces_Internal (plexinterpolate.c:303) n1: 855200 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 855200 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 855200 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 855200 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 855200 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n3: 710848 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n2: 308000 0x45A810: DMPlexSetChart (plex.c:729) n1: 308000 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 308000 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 308000 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 308000 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 308000 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 308000 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n2: 308000 0x45A874: DMPlexSetChart (plex.c:730) n1: 308000 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 308000 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 308000 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 308000 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 308000 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 308000 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n0: 94848 in 5 places, all below massif's threshold (01.00%) n3: 710848 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n2: 308000 0x45A810: DMPlexSetChart (plex.c:729) n1: 308000 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 308000 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 308000 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 308000 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 308000 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 308000 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n2: 308000 0x45A874: DMPlexSetChart (plex.c:730) n1: 308000 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 308000 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 308000 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 308000 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 308000 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 308000 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n0: 94848 in 5 places, all below massif's threshold (01.00%) n1: 473944 0x56B8A6: PetscSegBufferAlloc_Private (segbuffer.c:33) n1: 473944 0x56BFCC: PetscSegBufferGet (segbuffer.c:102) n2: 473944 0x97A8D6: PetscSegBufferGetInts (petscsys.h:2626) n1: 409728 0x98071B: DMPlexCreatePartitionClosure (plexpartition.c:580) n1: 409728 0x557178: DMPlexDistributeNwtun (plexdistribute.c:92) n1: 409728 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 409728 0x4076DF: main (Nwtun.c:55) n1: 64216 0x9805AF: DMPlexCreatePartitionClosure (plexpartition.c:573) n1: 64216 0x557178: DMPlexDistributeNwtun (plexdistribute.c:92) n1: 64216 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 64216 0x4076DF: main (Nwtun.c:55) n1: 451200 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132) n1: 451200 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 451200 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 451200 0x4076DF: main (Nwtun.c:55) n2: 308000 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46) n2: 308000 0x4E2DE5: DMLabelGetStratumIS (plexlabel.c:571) n1: 308000 0x46481D: DMPlexStratify (plex.c:1768) n1: 308000 0x9E4D5E: DMPlexInterpolateFaces_Internal (plexinterpolate.c:304) n1: 308000 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 308000 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 308000 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 308000 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 308000 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n0: 0 in 1 place, below massif's threshold (01.00%) n3: 288120 0x50FE4C: DMGetWorkArray (dm.c:1005) n2: 192080 0x45FFD3: DMPlexGetTransitiveClosure (plex.c:1291) n1: 153664 0x466336: DMPlexGetFullJoin (plex.c:1946) n1: 153664 0x445DE9: DMPlexCreateExodusNwtun (plexexodusii.c:372) n1: 153664 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 153664 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 153664 0x4076DF: main (Nwtun.c:55) n0: 38416 in 1 place, below massif's threshold (01.00%) n1: 76832 0x45FF3D: DMPlexGetTransitiveClosure (plex.c:1287) n0: 76832 in 2 places, all below massif's threshold (01.00%) n0: 19208 in 4 places, all below massif's threshold (01.00%) n0: 173204 in 114 places, all below massif's threshold (01.00%) n1: 141840 0xC99701: VecCreate_MPI_Private (pbvec.c:201) n1: 141840 0xC99C3D: VecCreate_MPI (pbvec.c:244) n1: 141840 0x6BF560: VecSetType (vecreg.c:53) n1: 141840 0xC99FA5: VecCreate_Standard (pbvec.c:271) n1: 141840 0x6BF560: VecSetType (vecreg.c:53) n1: 141840 0x44533A: DMPlexCreateExodusNwtun (plexexodusii.c:314) n1: 141840 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 141840 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 141840 0x4076DF: main (Nwtun.c:55) n1: 180480 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202) n1: 180480 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 180480 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 180480 0x4076DF: main (Nwtun.c:55) n1: 161240 0x4DBEB6: kh_resize_HASHI (hash.h:510) n2: 161240 0x4DC2CD: kh_put_HASHI (hash.h:510) n2: 161240 0x4E14DD: DMLabelSetValue (plexlabel.c:450) n1: 161240 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909) n0: 161240 in 3 places, all below massif's threshold (01.00%) n0: 0 in 3 places, all below massif's threshold (01.00%) n0: 0 in 1 place, below massif's threshold (01.00%) n0: 109157 in 131 places, all below massif's threshold (01.00%) n1: 80620 0x4DBE90: kh_resize_HASHI (hash.h:510) n2: 80620 0x4DC2CD: kh_put_HASHI (hash.h:510) n2: 80620 0x4E14DD: DMLabelSetValue (plexlabel.c:450) n1: 80620 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909) n0: 80620 in 3 places, all below massif's threshold (01.00%) n0: 0 in 3 places, all below massif's threshold (01.00%) n0: 0 in 1 place, below massif's threshold (01.00%) #----------- snapshot=8 #----------- time=3336978286 mem_heap_B=8295339 mem_heap_extra_B=12405 mem_stacks_B=0 heap_tree=empty #----------- snapshot=9 #----------- time=3820713760 mem_heap_B=1630326 mem_heap_extra_B=10986 mem_stacks_B=0 heap_tree=empty #----------- snapshot=10 #----------- time=4287539798 mem_heap_B=1686266 mem_heap_extra_B=13262 mem_stacks_B=0 heap_tree=empty #----------- snapshot=11 #----------- time=4618200211 mem_heap_B=1884316 mem_heap_extra_B=31492 mem_stacks_B=0 heap_tree=empty #----------- snapshot=12 #----------- time=5199848578 mem_heap_B=1893829 mem_heap_extra_B=33355 mem_stacks_B=0 heap_tree=detailed n6: 1893829 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n12: 1292450 0x57877E: PetscMallocAlign (mal.c:27) n0: 232010 in 224 places, all below massif's threshold (01.00%) n1: 155712 0xC9786A: VecCreate_Seq (bvec3.c:38) n1: 155712 0x6BF560: VecSetType (vecreg.c:53) n1: 155712 0xC99F47: VecCreate_Standard (pbvec.c:269) n1: 155712 0x6BF560: VecSetType (vecreg.c:53) n1: 155712 0xA0773A: DMCreateLocalVector_Section_Private (dmi.c:65) n1: 155712 0x976C9B: DMCreateLocalVector_Plex (plexcreate.c:931) n2: 155712 0x50D220: DMCreateLocalVector (dm.c:709) n1: 130752 0x417EA6: ConstructGeometryFVM (SetupMeshes.c:306) n0: 130752 0x407898: main (Nwtun.c:60) n1: 24960 0x4178BF: ConstructGeometryFVM (SetupMeshes.c:273) n0: 24960 0x407898: main (Nwtun.c:60) n3: 153664 0x50FE4C: DMGetWorkArray (dm.c:1005) n2: 76832 0x45FF3D: DMPlexGetTransitiveClosure (plex.c:1287) n2: 76832 0x47B16A: DMPlexVecGetClosure (plex.c:4614) n1: 38416 0x44F2E0: DMPlexComputeGeometryFVM_3D_Internal (plexgeometry.c:872) n1: 38416 0x44FF85: DMPlexComputeCellGeometryFVM (plexgeometry.c:977) n1: 38416 0x417A90: ConstructGeometryFVM (SetupMeshes.c:281) n0: 38416 0x407898: main (Nwtun.c:60) n1: 38416 0x44E4E6: DMPlexComputeGeometryFVM_2D_Internal (plexgeometry.c:793) n1: 38416 0x44FF11: DMPlexComputeCellGeometryFVM (plexgeometry.c:974) n1: 38416 0x4180BC: ConstructGeometryFVM (SetupMeshes.c:324) n0: 38416 0x407898: main (Nwtun.c:60) n0: 0 in 2 places, all below massif's threshold (01.00%) n2: 76832 0x45FFD3: DMPlexGetTransitiveClosure (plex.c:1291) n2: 76832 0x47B16A: DMPlexVecGetClosure (plex.c:4614) n1: 38416 0x44F2E0: DMPlexComputeGeometryFVM_3D_Internal (plexgeometry.c:872) n1: 38416 0x44FF85: DMPlexComputeCellGeometryFVM (plexgeometry.c:977) n1: 38416 0x417A90: ConstructGeometryFVM (SetupMeshes.c:281) n0: 38416 0x407898: main (Nwtun.c:60) n1: 38416 0x44E4E6: DMPlexComputeGeometryFVM_2D_Internal (plexgeometry.c:793) n1: 38416 0x44FF11: DMPlexComputeCellGeometryFVM (plexgeometry.c:974) n1: 38416 0x4180BC: ConstructGeometryFVM (SetupMeshes.c:324) n0: 38416 0x407898: main (Nwtun.c:60) n0: 0 in 2 places, all below massif's threshold (01.00%) n0: 0 in 4 places, all below massif's threshold (01.00%) n4: 135920 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n0: 41064 in 12 places, all below massif's threshold (01.00%) n2: 35464 0x45A810: DMPlexSetChart (plex.c:729) n1: 35464 0x4B39F0: DMPlexShiftSizes_Internal (plexsubmesh.c:189) n1: 35464 0x4B8A41: DMPlexConstructGhostCells_Internal (plexsubmesh.c:507) n1: 35464 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 35464 0x415800: CreateMesh (SetupMeshes.c:53) n0: 35464 0x4076DF: main (Nwtun.c:55) n0: 0 in 2 places, all below massif's threshold (01.00%) n2: 35464 0x45A874: DMPlexSetChart (plex.c:730) n1: 35464 0x4B39F0: DMPlexShiftSizes_Internal (plexsubmesh.c:189) n1: 35464 0x4B8A41: DMPlexConstructGhostCells_Internal (plexsubmesh.c:507) n1: 35464 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 35464 0x415800: CreateMesh (SetupMeshes.c:53) n0: 35464 0x4076DF: main (Nwtun.c:55) n0: 0 in 2 places, all below massif's threshold (01.00%) n1: 23928 0x52ED49: PetscSectionSetChart (vsectionis.c:392) n0: 23928 in 4 places, all below massif's threshold (01.00%) n4: 135920 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n0: 41064 in 12 places, all below massif's threshold (01.00%) n2: 35464 0x45A810: DMPlexSetChart (plex.c:729) n1: 35464 0x4B39F0: DMPlexShiftSizes_Internal (plexsubmesh.c:189) n1: 35464 0x4B8A41: DMPlexConstructGhostCells_Internal (plexsubmesh.c:507) n1: 35464 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 35464 0x415800: CreateMesh (SetupMeshes.c:53) n0: 35464 0x4076DF: main (Nwtun.c:55) n0: 0 in 2 places, all below massif's threshold (01.00%) n2: 35464 0x45A874: DMPlexSetChart (plex.c:730) n1: 35464 0x4B39F0: DMPlexShiftSizes_Internal (plexsubmesh.c:189) n1: 35464 0x4B8A41: DMPlexConstructGhostCells_Internal (plexsubmesh.c:507) n1: 35464 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 35464 0x415800: CreateMesh (SetupMeshes.c:53) n0: 35464 0x4076DF: main (Nwtun.c:55) n0: 0 in 2 places, all below massif's threshold (01.00%) n1: 23928 0x52ED49: PetscSectionSetChart (vsectionis.c:392) n0: 23928 in 4 places, all below massif's threshold (01.00%) n2: 116064 0x659AFA: ISCreate (isreg.c:41) n2: 116064 0x64D670: ISCreateGeneral (general.c:632) n2: 116064 0x4E222E: DMLabelGetValueIS (plexlabel.c:518) n2: 116064 0x4E8069: DMPlexGetLabelIdIS (plexlabel.c:1013) n1: 116064 0x41D333: SetUpSymetricBoundaryMaching (SetupMeshes.c:1030) n0: 116064 0x40796B: main (Nwtun.c:65) n0: 0 in 1 place, below massif's threshold (01.00%) n0: 0 in 2 places, all below massif's threshold (01.00%) n0: 0 in 4 places, all below massif's threshold (01.00%) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 90944 0x462A1D: DMSetUp_Plex (plex.c:1582) n2: 90944 0x50B88B: DMSetUp (dm.c:560) n1: 90944 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534) n1: 90944 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 90944 0x415800: CreateMesh (SetupMeshes.c:53) n0: 90944 0x4076DF: main (Nwtun.c:55) n0: 0 in 3 places, all below massif's threshold (01.00%) n1: 90944 0x462C3D: DMSetUp_Plex (plex.c:1587) n1: 90944 0x50B88B: DMSetUp (dm.c:560) n1: 90944 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534) n1: 90944 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 90944 0x415800: CreateMesh (SetupMeshes.c:53) n0: 90944 0x4076DF: main (Nwtun.c:55) n1: 90944 0x462AB1: DMSetUp_Plex (plex.c:1583) n2: 90944 0x50B88B: DMSetUp (dm.c:560) n1: 90944 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534) n1: 90944 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 90944 0x415800: CreateMesh (SetupMeshes.c:53) n0: 90944 0x4076DF: main (Nwtun.c:55) n0: 0 in 3 places, all below massif's threshold (01.00%) n2: 38728 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46) n2: 35464 0x4E2DE5: DMLabelGetStratumIS (plexlabel.c:571) n2: 35464 0x46481D: DMPlexStratify (plex.c:1768) n1: 35464 0x4B955D: DMPlexConstructGhostCells_Internal (plexsubmesh.c:560) n1: 35464 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 35464 0x415800: CreateMesh (SetupMeshes.c:53) n0: 35464 0x4076DF: main (Nwtun.c:55) n0: 0 in 2 places, all below massif's threshold (01.00%) n0: 0 in 1 place, below massif's threshold (01.00%) n0: 3264 in 3 places, all below massif's threshold (01.00%) n2: 26784 0x659BA0: ISCreate (isreg.c:41) n2: 26784 0x64D670: ISCreateGeneral (general.c:632) n2: 26784 0x4E222E: DMLabelGetValueIS (plexlabel.c:518) n2: 26784 0x4E8069: DMPlexGetLabelIdIS (plexlabel.c:1013) n1: 26784 0x41D333: SetUpSymetricBoundaryMaching (SetupMeshes.c:1030) n0: 26784 0x40796B: main (Nwtun.c:65) n0: 0 in 1 place, below massif's threshold (01.00%) n0: 0 in 2 places, all below massif's threshold (01.00%) n0: 0 in 4 places, all below massif's threshold (01.00%) n0: 0 in 1 place, below massif's threshold (01.00%) n2: 24816 0x505DF4: DMCreate (dm.c:43) n1: 20680 0x5066C4: DMClone (dm.c:99) n0: 20680 in 4 places, all below massif's threshold (01.00%) n0: 4136 in 4 places, all below massif's threshold (01.00%) n2: 266240 0x5E920BC: MPIU_Handle_obj_alloc_unsafe (in /home/rlchen/mpich2/lib/libmpich.so.3.0) n1: 192512 0x5EA339B: MPID_Isend (in /home/rlchen/mpich2/lib/libmpich.so.3.0) n1: 192512 0x5E9CBA9: PMPI_Isend (in /home/rlchen/mpich2/lib/libmpich.so.3.0) n1: 192512 0x11532F6: PetscCommBuildTwoSided_Allreduce (mpits.c:174) n1: 192512 0x11538F7: PetscCommBuildTwoSided (mpits.c:237) n1: 192512 0xC8603E: PetscSFSetUp_Basic (sfbasic.c:552) n1: 192512 0x67F6D7: PetscSFSetUp (sf.c:192) n1: 192512 0x689A32: PetscSFFetchAndOpBegin (sf.c:1138) n1: 192512 0x53DD1C: PetscSFConvertPartition (vsectionis.c:1916) n1: 192512 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101) n1: 192512 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 192512 0x4076DF: main (Nwtun.c:55) n1: 73728 0x5EAFDD0: MPID_Type_dup (in /home/rlchen/mpich2/lib/libmpich.so.3.0) n1: 73728 0x5EFFF19: PMPI_Type_dup (in /home/rlchen/mpich2/lib/libmpich.so.3.0) n1: 73728 0xC8750A: PetscSFBasicPackTypeSetup (sfbasic.c:654) n1: 73728 0xC88D96: PetscSFBasicGetPack (sfbasic.c:778) n1: 73728 0xC8B6AF: PetscSFReduceBegin_Basic (sfbasic.c:973) n1: 73728 0x68814F: PetscSFReduceBegin (sf.c:993) n1: 73728 0x682011: PetscSFCreateInverseSF (sf.c:459) n1: 73728 0x53E180: PetscSFConvertPartition (vsectionis.c:1937) n1: 73728 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101) n1: 73728 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 73728 0x4076DF: main (Nwtun.c:55) n1: 180480 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202) n1: 180480 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 180480 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 180480 0x4076DF: main (Nwtun.c:55) n0: 102947 in 141 places, all below massif's threshold (01.00%) n1: 30088 0x443EEF: DMPlexCreateExodusNwtun (plexexodusii.c:201) n1: 30088 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 30088 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 30088 0x4076DF: main (Nwtun.c:55) n1: 21624 0x4DBEB6: kh_resize_HASHI (hash.h:510) n2: 21624 0x4DC2CD: kh_put_HASHI (hash.h:510) n1: 21624 0x4E14DD: DMLabelSetValue (plexlabel.c:450) n0: 21624 in 8 places, all below massif's threshold (01.00%) n0: 0 in 1 place, below massif's threshold (01.00%) #----------- snapshot=13 #----------- time=5707788994 mem_heap_B=3019566 mem_heap_extra_B=38722 mem_stacks_B=0 heap_tree=empty #----------- snapshot=14 #----------- time=6349483701 mem_heap_B=4209659 mem_heap_extra_B=40661 mem_stacks_B=0 heap_tree=empty #----------- snapshot=15 #----------- time=6766057848 mem_heap_B=5309276 mem_heap_extra_B=42020 mem_stacks_B=0 heap_tree=empty #----------- snapshot=16 #----------- time=7358988205 mem_heap_B=5309276 mem_heap_extra_B=42020 mem_stacks_B=0 heap_tree=empty #----------- snapshot=17 #----------- time=7800734843 mem_heap_B=5309252 mem_heap_extra_B=42028 mem_stacks_B=0 heap_tree=empty #----------- snapshot=18 #----------- time=8095696846 mem_heap_B=5309252 mem_heap_extra_B=42028 mem_stacks_B=0 heap_tree=empty #----------- snapshot=19 #----------- time=8536311900 mem_heap_B=5309276 mem_heap_extra_B=42020 mem_stacks_B=0 heap_tree=empty #----------- snapshot=20 #----------- time=9120156697 mem_heap_B=5309276 mem_heap_extra_B=42020 mem_stacks_B=0 heap_tree=empty #----------- snapshot=21 #----------- time=9685257975 mem_heap_B=5309276 mem_heap_extra_B=42020 mem_stacks_B=0 heap_tree=empty #----------- snapshot=22 #----------- time=10085942687 mem_heap_B=5309284 mem_heap_extra_B=42044 mem_stacks_B=0 heap_tree=empty #----------- snapshot=23 #----------- time=10490188517 mem_heap_B=5309276 mem_heap_extra_B=42020 mem_stacks_B=0 heap_tree=empty #----------- snapshot=24 #----------- time=10889161638 mem_heap_B=5309252 mem_heap_extra_B=42028 mem_stacks_B=0 heap_tree=empty #----------- snapshot=25 #----------- time=11158478616 mem_heap_B=5309252 mem_heap_extra_B=42028 mem_stacks_B=0 heap_tree=detailed n4: 5309252 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n21: 4707353 0x57877E: PetscMallocAlign (mal.c:27) n1: 712000 0x8518E1: MatFDColoringSetUpBlocked_AIJ_Private (fdaij.c:121) n1: 712000 0x919037: MatFDColoringSetUp_MPIXAIJ (fdmpiaij.c:570) n1: 712000 0x74CD34: MatFDColoringSetUp (fdmatrix.c:250) n1: 712000 0x43E33A: SNESComputeJacobianDefaultColorNwtun (SetupJacobian.c:907) n1: 712000 0x434942: FormJacobian (SetupJacobian.c:56) n1: 712000 0xA8F850: SNESComputeJacobian (snes.c:2193) n1: 712000 0xAF7C15: SNESSolve_NEWTONLS (ls.c:230) n1: 712000 0xA9E3CC: SNESSolve (snes.c:3743) n1: 712000 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 712000 0x40BC62: main (Nwtun.c:417) n0: 520457 in 389 places, all below massif's threshold (01.00%) n1: 408112 0xC9786A: VecCreate_Seq (bvec3.c:38) n2: 408112 0x6BF560: VecSetType (vecreg.c:53) n1: 405312 0xC99F47: VecCreate_Standard (pbvec.c:269) n1: 405312 0x6BF560: VecSetType (vecreg.c:53) n1: 405312 0xA0773A: DMCreateLocalVector_Section_Private (dmi.c:65) n1: 405312 0x976C9B: DMCreateLocalVector_Plex (plexcreate.c:931) n3: 405312 0x50D220: DMCreateLocalVector (dm.c:709) n3: 249600 0xA035E0: DMGetLocalVector (dmget.c:49) n1: 93600 0x425614: CaculateLocalFunction_LS (SetupFunctions.c:742) n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93600 0x40BC62: main (Nwtun.c:417) n1: 93600 0x4257D8: CaculateLocalFunction_LS (SetupFunctions.c:748) n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93600 0x40BC62: main (Nwtun.c:417) n0: 62400 in 2 places, all below massif's threshold (01.00%) n1: 130752 0x417EA6: ConstructGeometryFVM (SetupMeshes.c:306) n0: 130752 0x407898: main (Nwtun.c:60) n0: 24960 in 1 place, below massif's threshold (01.00%) n0: 2800 in 1 place, below massif's threshold (01.00%) n1: 362400 0x8512B1: MatFDColoringSetUpBlocked_AIJ_Private (fdaij.c:70) n1: 362400 0x919037: MatFDColoringSetUp_MPIXAIJ (fdmpiaij.c:570) n1: 362400 0x74CD34: MatFDColoringSetUp (fdmatrix.c:250) n1: 362400 0x43E33A: SNESComputeJacobianDefaultColorNwtun (SetupJacobian.c:907) n1: 362400 0x434942: FormJacobian (SetupJacobian.c:56) n1: 362400 0xA8F850: SNESComputeJacobian (snes.c:2193) n1: 362400 0xAF7C15: SNESSolve_NEWTONLS (ls.c:230) n1: 362400 0xA9E3CC: SNESSolve (snes.c:3743) n1: 362400 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 362400 0x40BC62: main (Nwtun.c:417) n1: 356000 0x81EE1C: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719) n2: 356000 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664) n1: 340000 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255) n1: 340000 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959) n1: 340000 0x955310: MatXAIJSetPreallocation (gcreate.c:252) n1: 340000 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435) n1: 340000 0x459619: DMCreateMatrix_Plex (plex.c:615) n1: 340000 0x50F42F: DMCreateMatrix (dm.c:941) n0: 340000 0x40B906: main (Nwtun.c:396) n0: 16000 in 2 places, all below massif's threshold (01.00%) n1: 356000 0x81EDCA: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719) n2: 356000 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664) n1: 340000 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255) n1: 340000 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959) n1: 340000 0x955310: MatXAIJSetPreallocation (gcreate.c:252) n1: 340000 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435) n1: 340000 0x459619: DMCreateMatrix_Plex (plex.c:615) n1: 340000 0x50F42F: DMCreateMatrix (dm.c:941) n0: 340000 0x40B906: main (Nwtun.c:396) n0: 16000 in 2 places, all below massif's threshold (01.00%) n2: 233704 0xC99701: VecCreate_MPI_Private (pbvec.c:201) n1: 137064 0xC99C3D: VecCreate_MPI (pbvec.c:244) n1: 137064 0x6BF560: VecSetType (vecreg.c:53) n1: 137064 0xC99FA5: VecCreate_Standard (pbvec.c:271) n2: 137064 0x6BF560: VecSetType (vecreg.c:53) n1: 109360 0xA07131: DMCreateGlobalVector_Section_Private (dmi.c:34) n1: 109360 0x976932: DMCreateGlobalVector_Plex (plexcreate.c:917) n2: 109360 0x50CE38: DMCreateGlobalVector (dm.c:681) n2: 97280 0xA04005: DMGetGlobalVector (dmget.c:154) n1: 85200 0x426EC6: ConstructCellCentriodGradient (SetupFunctions.c:855) n1: 85200 0x42054B: FormTimeStepFunction (SetupFunctions.c:233) n1: 85200 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 85200 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 85200 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 85200 0xA9E3CC: SNESSolve (snes.c:3743) n1: 85200 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 85200 0x40BC62: main (Nwtun.c:417) n0: 12080 in 1 place, below massif's threshold (01.00%) n0: 12080 in 1 place, below massif's threshold (01.00%) n0: 27704 in 4 places, all below massif's threshold (01.00%) n1: 96640 0xC98DCF: VecDuplicate_MPI (pbvec.c:68) n1: 96640 0x6B04CF: VecDuplicate (vector.c:399) n0: 96640 in 8 places, all below massif's threshold (01.00%) n1: 216960 0x521E45: DMCreateDefaultSF (dm.c:3075) n1: 216960 0x520D8A: DMGetDefaultSF (dm.c:2991) n2: 216960 0x5153DB: DMGlobalToLocalBegin (dm.c:1649) n1: 187200 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 187200 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 187200 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 187200 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 187200 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 187200 0xA9E3CC: SNESSolve (snes.c:3743) n1: 187200 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 187200 0x40BC62: main (Nwtun.c:417) n0: 29760 in 1 place, below massif's threshold (01.00%) n1: 154640 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n0: 154640 in 19 places, all below massif's threshold (01.00%) n1: 154640 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n0: 154640 in 19 places, all below massif's threshold (01.00%) n3: 153664 0x50FE4C: DMGetWorkArray (dm.c:1005) n2: 76832 0x45FF3D: DMPlexGetTransitiveClosure (plex.c:1287) n1: 76832 0x47B16A: DMPlexVecGetClosure (plex.c:4614) n0: 76832 in 2 places, all below massif's threshold (01.00%) n0: 0 in 2 places, all below massif's threshold (01.00%) n2: 76832 0x45FFD3: DMPlexGetTransitiveClosure (plex.c:1291) n1: 76832 0x47B16A: DMPlexVecGetClosure (plex.c:4614) n0: 76832 in 2 places, all below massif's threshold (01.00%) n0: 0 in 2 places, all below massif's threshold (01.00%) n0: 0 in 4 places, all below massif's threshold (01.00%) n2: 117480 0x6812ED: PetscSFSetGraph (sf.c:349) n1: 108480 0x522622: DMCreateDefaultSF (dm.c:3116) n1: 108480 0x520D8A: DMGetDefaultSF (dm.c:2991) n2: 108480 0x5153DB: DMGlobalToLocalBegin (dm.c:1649) n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93600 0x40BC62: main (Nwtun.c:417) n0: 14880 in 1 place, below massif's threshold (01.00%) n0: 9000 in 9 places, all below massif's threshold (01.00%) n2: 117480 0x681299: PetscSFSetGraph (sf.c:349) n1: 108480 0x522622: DMCreateDefaultSF (dm.c:3116) n1: 108480 0x520D8A: DMGetDefaultSF (dm.c:2991) n2: 108480 0x5153DB: DMGlobalToLocalBegin (dm.c:1649) n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93600 0x40BC62: main (Nwtun.c:417) n0: 14880 in 1 place, below massif's threshold (01.00%) n0: 9000 in 9 places, all below massif's threshold (01.00%) n2: 117480 0xC88EE6: PetscSFBasicGetPack (sfbasic.c:779) n1: 117480 0xC8AA7F: PetscSFBcastBegin_Basic (sfbasic.c:914) n2: 117480 0x687407: PetscSFBcastBegin (sf.c:926) n2: 108480 0x51554F: DMGlobalToLocalBegin (dm.c:1656) n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93600 0x40BC62: main (Nwtun.c:417) n0: 14880 in 1 place, below massif's threshold (01.00%) n0: 9000 in 8 places, all below massif's threshold (01.00%) n0: 0 in 1 place, below massif's threshold (01.00%) n2: 116064 0x659AFA: ISCreate (isreg.c:41) n2: 116064 0x64D670: ISCreateGeneral (general.c:632) n2: 116064 0x4E222E: DMLabelGetValueIS (plexlabel.c:518) n2: 116064 0x4E8069: DMPlexGetLabelIdIS (plexlabel.c:1013) n1: 116064 0x41D333: SetUpSymetricBoundaryMaching (SetupMeshes.c:1030) n0: 116064 0x40796B: main (Nwtun.c:65) n0: 0 in 2 places, all below massif's threshold (01.00%) n0: 0 in 2 places, all below massif's threshold (01.00%) n0: 0 in 6 places, all below massif's threshold (01.00%) n0: 0 in 1 place, below massif's threshold (01.00%) n2: 114480 0xC88E56: PetscSFBasicGetPack (sfbasic.c:779) n1: 114480 0xC8AA7F: PetscSFBcastBegin_Basic (sfbasic.c:914) n2: 114480 0x687407: PetscSFBcastBegin (sf.c:926) n2: 108800 0x51554F: DMGlobalToLocalBegin (dm.c:1656) n1: 93840 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 93840 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93840 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93840 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93840 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93840 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93840 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93840 0x40BC62: main (Nwtun.c:417) n0: 14960 in 1 place, below massif's threshold (01.00%) n0: 5680 in 8 places, all below massif's threshold (01.00%) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 114480 0xC861ED: PetscSFSetUp_Basic (sfbasic.c:557) n2: 114480 0x67F6D7: PetscSFSetUp (sf.c:192) n2: 114480 0x68739A: PetscSFBcastBegin (sf.c:925) n2: 108800 0x51554F: DMGlobalToLocalBegin (dm.c:1656) n1: 93840 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 93840 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93840 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93840 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93840 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93840 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93840 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93840 0x40BC62: main (Nwtun.c:417) n0: 14960 in 1 place, below massif's threshold (01.00%) n0: 5680 in 7 places, all below massif's threshold (01.00%) n0: 0 in 2 places, all below massif's threshold (01.00%) n1: 108480 0x521DAD: DMCreateDefaultSF (dm.c:3074) n1: 108480 0x520D8A: DMGetDefaultSF (dm.c:2991) n2: 108480 0x5153DB: DMGlobalToLocalBegin (dm.c:1649) n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93600 0x40BC62: main (Nwtun.c:417) n0: 14880 in 1 place, below massif's threshold (01.00%) n1: 90944 0x462C3D: DMSetUp_Plex (plex.c:1587) n1: 90944 0x50B88B: DMSetUp (dm.c:560) n1: 90944 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534) n1: 90944 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 90944 0x415800: CreateMesh (SetupMeshes.c:53) n0: 90944 0x4076DF: main (Nwtun.c:55) n1: 90944 0x462AB1: DMSetUp_Plex (plex.c:1583) n2: 90944 0x50B88B: DMSetUp (dm.c:560) n1: 90944 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534) n1: 90944 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 90944 0x415800: CreateMesh (SetupMeshes.c:53) n0: 90944 0x4076DF: main (Nwtun.c:55) n0: 0 in 3 places, all below massif's threshold (01.00%) n1: 90944 0x462A1D: DMSetUp_Plex (plex.c:1582) n2: 90944 0x50B88B: DMSetUp (dm.c:560) n1: 90944 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534) n1: 90944 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 90944 0x415800: CreateMesh (SetupMeshes.c:53) n0: 90944 0x4076DF: main (Nwtun.c:55) n0: 0 in 3 places, all below massif's threshold (01.00%) n2: 266240 0x5E920BC: MPIU_Handle_obj_alloc_unsafe (in /home/rlchen/mpich2/lib/libmpich.so.3.0) n1: 192512 0x5EA339B: MPID_Isend (in /home/rlchen/mpich2/lib/libmpich.so.3.0) n1: 192512 0x5E9CBA9: PMPI_Isend (in /home/rlchen/mpich2/lib/libmpich.so.3.0) n1: 192512 0x11532F6: PetscCommBuildTwoSided_Allreduce (mpits.c:174) n1: 192512 0x11538F7: PetscCommBuildTwoSided (mpits.c:237) n1: 192512 0xC8603E: PetscSFSetUp_Basic (sfbasic.c:552) n1: 192512 0x67F6D7: PetscSFSetUp (sf.c:192) n1: 192512 0x689A32: PetscSFFetchAndOpBegin (sf.c:1138) n1: 192512 0x53DD1C: PetscSFConvertPartition (vsectionis.c:1916) n1: 192512 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101) n1: 192512 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 192512 0x4076DF: main (Nwtun.c:55) n1: 73728 0x5EAFDD0: MPID_Type_dup (in /home/rlchen/mpich2/lib/libmpich.so.3.0) n1: 73728 0x5EFFF19: PMPI_Type_dup (in /home/rlchen/mpich2/lib/libmpich.so.3.0) n1: 73728 0xC8750A: PetscSFBasicPackTypeSetup (sfbasic.c:654) n1: 73728 0xC88D96: PetscSFBasicGetPack (sfbasic.c:778) n1: 73728 0xC8B6AF: PetscSFReduceBegin_Basic (sfbasic.c:973) n1: 73728 0x68814F: PetscSFReduceBegin (sf.c:993) n1: 73728 0x682011: PetscSFCreateInverseSF (sf.c:459) n1: 73728 0x53E180: PetscSFConvertPartition (vsectionis.c:1937) n1: 73728 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101) n1: 73728 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 73728 0x4076DF: main (Nwtun.c:55) n1: 180480 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202) n1: 180480 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 180480 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 180480 0x4076DF: main (Nwtun.c:55) n0: 155179 in 150 places, all below massif's threshold (01.00%) #----------- snapshot=26 #----------- time=11692400418 mem_heap_B=5309252 mem_heap_extra_B=42028 mem_stacks_B=0 heap_tree=empty #----------- snapshot=27 #----------- time=11960769989 mem_heap_B=5309252 mem_heap_extra_B=42028 mem_stacks_B=0 heap_tree=empty #----------- snapshot=28 #----------- time=12362682614 mem_heap_B=5309276 mem_heap_extra_B=42020 mem_stacks_B=0 heap_tree=empty #----------- snapshot=29 #----------- time=12896123403 mem_heap_B=5309276 mem_heap_extra_B=42020 mem_stacks_B=0 heap_tree=empty #----------- snapshot=30 #----------- time=13162816658 mem_heap_B=5309276 mem_heap_extra_B=42020 mem_stacks_B=0 heap_tree=empty #----------- snapshot=31 #----------- time=13572082155 mem_heap_B=5309252 mem_heap_extra_B=42028 mem_stacks_B=0 heap_tree=empty #----------- snapshot=32 #----------- time=13854485968 mem_heap_B=5310218 mem_heap_extra_B=42094 mem_stacks_B=0 heap_tree=detailed n4: 5310218 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n21: 4708335 0x57877E: PetscMallocAlign (mal.c:27) n1: 712000 0x8518E1: MatFDColoringSetUpBlocked_AIJ_Private (fdaij.c:121) n1: 712000 0x919037: MatFDColoringSetUp_MPIXAIJ (fdmpiaij.c:570) n1: 712000 0x74CD34: MatFDColoringSetUp (fdmatrix.c:250) n1: 712000 0x43E33A: SNESComputeJacobianDefaultColorNwtun (SetupJacobian.c:907) n1: 712000 0x434942: FormJacobian (SetupJacobian.c:56) n1: 712000 0xA8F850: SNESComputeJacobian (snes.c:2193) n1: 712000 0xAF7C15: SNESSolve_NEWTONLS (ls.c:230) n1: 712000 0xA9E3CC: SNESSolve (snes.c:3743) n1: 712000 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 712000 0x40BC62: main (Nwtun.c:417) n0: 520815 in 389 places, all below massif's threshold (01.00%) n1: 408112 0xC9786A: VecCreate_Seq (bvec3.c:38) n2: 408112 0x6BF560: VecSetType (vecreg.c:53) n1: 405312 0xC99F47: VecCreate_Standard (pbvec.c:269) n1: 405312 0x6BF560: VecSetType (vecreg.c:53) n1: 405312 0xA0773A: DMCreateLocalVector_Section_Private (dmi.c:65) n1: 405312 0x976C9B: DMCreateLocalVector_Plex (plexcreate.c:931) n3: 405312 0x50D220: DMCreateLocalVector (dm.c:709) n3: 249600 0xA035E0: DMGetLocalVector (dmget.c:49) n1: 93600 0x425614: CaculateLocalFunction_LS (SetupFunctions.c:742) n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93600 0x40BC62: main (Nwtun.c:417) n1: 93600 0x4257D8: CaculateLocalFunction_LS (SetupFunctions.c:748) n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93600 0x40BC62: main (Nwtun.c:417) n0: 62400 in 2 places, all below massif's threshold (01.00%) n1: 130752 0x417EA6: ConstructGeometryFVM (SetupMeshes.c:306) n0: 130752 0x407898: main (Nwtun.c:60) n0: 24960 in 1 place, below massif's threshold (01.00%) n0: 2800 in 1 place, below massif's threshold (01.00%) n1: 362400 0x8512B1: MatFDColoringSetUpBlocked_AIJ_Private (fdaij.c:70) n1: 362400 0x919037: MatFDColoringSetUp_MPIXAIJ (fdmpiaij.c:570) n1: 362400 0x74CD34: MatFDColoringSetUp (fdmatrix.c:250) n1: 362400 0x43E33A: SNESComputeJacobianDefaultColorNwtun (SetupJacobian.c:907) n1: 362400 0x434942: FormJacobian (SetupJacobian.c:56) n1: 362400 0xA8F850: SNESComputeJacobian (snes.c:2193) n1: 362400 0xAF7C15: SNESSolve_NEWTONLS (ls.c:230) n1: 362400 0xA9E3CC: SNESSolve (snes.c:3743) n1: 362400 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 362400 0x40BC62: main (Nwtun.c:417) n1: 356000 0x81EE1C: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719) n2: 356000 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664) n1: 340000 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255) n1: 340000 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959) n1: 340000 0x955310: MatXAIJSetPreallocation (gcreate.c:252) n1: 340000 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435) n1: 340000 0x459619: DMCreateMatrix_Plex (plex.c:615) n1: 340000 0x50F42F: DMCreateMatrix (dm.c:941) n0: 340000 0x40B906: main (Nwtun.c:396) n0: 16000 in 2 places, all below massif's threshold (01.00%) n1: 356000 0x81EDCA: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719) n2: 356000 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664) n1: 340000 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255) n1: 340000 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959) n1: 340000 0x955310: MatXAIJSetPreallocation (gcreate.c:252) n1: 340000 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435) n1: 340000 0x459619: DMCreateMatrix_Plex (plex.c:615) n1: 340000 0x50F42F: DMCreateMatrix (dm.c:941) n0: 340000 0x40B906: main (Nwtun.c:396) n0: 16000 in 2 places, all below massif's threshold (01.00%) n2: 233704 0xC99701: VecCreate_MPI_Private (pbvec.c:201) n1: 137064 0xC99C3D: VecCreate_MPI (pbvec.c:244) n1: 137064 0x6BF560: VecSetType (vecreg.c:53) n1: 137064 0xC99FA5: VecCreate_Standard (pbvec.c:271) n2: 137064 0x6BF560: VecSetType (vecreg.c:53) n1: 109360 0xA07131: DMCreateGlobalVector_Section_Private (dmi.c:34) n1: 109360 0x976932: DMCreateGlobalVector_Plex (plexcreate.c:917) n2: 109360 0x50CE38: DMCreateGlobalVector (dm.c:681) n2: 97280 0xA04005: DMGetGlobalVector (dmget.c:154) n1: 85200 0x426EC6: ConstructCellCentriodGradient (SetupFunctions.c:855) n1: 85200 0x42054B: FormTimeStepFunction (SetupFunctions.c:233) n1: 85200 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 85200 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 85200 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 85200 0xA9E3CC: SNESSolve (snes.c:3743) n1: 85200 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 85200 0x40BC62: main (Nwtun.c:417) n0: 12080 in 1 place, below massif's threshold (01.00%) n0: 12080 in 1 place, below massif's threshold (01.00%) n0: 27704 in 4 places, all below massif's threshold (01.00%) n1: 96640 0xC98DCF: VecDuplicate_MPI (pbvec.c:68) n1: 96640 0x6B04CF: VecDuplicate (vector.c:399) n0: 96640 in 8 places, all below massif's threshold (01.00%) n1: 216960 0x521E45: DMCreateDefaultSF (dm.c:3075) n1: 216960 0x520D8A: DMGetDefaultSF (dm.c:2991) n2: 216960 0x5153DB: DMGlobalToLocalBegin (dm.c:1649) n1: 187200 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 187200 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 187200 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 187200 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 187200 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 187200 0xA9E3CC: SNESSolve (snes.c:3743) n1: 187200 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 187200 0x40BC62: main (Nwtun.c:417) n0: 29760 in 1 place, below massif's threshold (01.00%) n1: 154640 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n0: 154640 in 19 places, all below massif's threshold (01.00%) n1: 154640 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n0: 154640 in 19 places, all below massif's threshold (01.00%) n3: 153664 0x50FE4C: DMGetWorkArray (dm.c:1005) n2: 76832 0x45FF3D: DMPlexGetTransitiveClosure (plex.c:1287) n1: 76832 0x47B16A: DMPlexVecGetClosure (plex.c:4614) n0: 76832 in 2 places, all below massif's threshold (01.00%) n0: 0 in 2 places, all below massif's threshold (01.00%) n2: 76832 0x45FFD3: DMPlexGetTransitiveClosure (plex.c:1291) n1: 76832 0x47B16A: DMPlexVecGetClosure (plex.c:4614) n0: 76832 in 2 places, all below massif's threshold (01.00%) n0: 0 in 2 places, all below massif's threshold (01.00%) n0: 0 in 4 places, all below massif's threshold (01.00%) n2: 117480 0x6812ED: PetscSFSetGraph (sf.c:349) n1: 108480 0x522622: DMCreateDefaultSF (dm.c:3116) n1: 108480 0x520D8A: DMGetDefaultSF (dm.c:2991) n2: 108480 0x5153DB: DMGlobalToLocalBegin (dm.c:1649) n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93600 0x40BC62: main (Nwtun.c:417) n0: 14880 in 1 place, below massif's threshold (01.00%) n0: 9000 in 9 places, all below massif's threshold (01.00%) n2: 117480 0x681299: PetscSFSetGraph (sf.c:349) n1: 108480 0x522622: DMCreateDefaultSF (dm.c:3116) n1: 108480 0x520D8A: DMGetDefaultSF (dm.c:2991) n2: 108480 0x5153DB: DMGlobalToLocalBegin (dm.c:1649) n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93600 0x40BC62: main (Nwtun.c:417) n0: 14880 in 1 place, below massif's threshold (01.00%) n0: 9000 in 9 places, all below massif's threshold (01.00%) n2: 117480 0xC88EE6: PetscSFBasicGetPack (sfbasic.c:779) n1: 117480 0xC8AA7F: PetscSFBcastBegin_Basic (sfbasic.c:914) n2: 117480 0x687407: PetscSFBcastBegin (sf.c:926) n2: 108480 0x51554F: DMGlobalToLocalBegin (dm.c:1656) n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93600 0x40BC62: main (Nwtun.c:417) n0: 14880 in 1 place, below massif's threshold (01.00%) n0: 9000 in 8 places, all below massif's threshold (01.00%) n0: 0 in 1 place, below massif's threshold (01.00%) n2: 116688 0x659AFA: ISCreate (isreg.c:41) n2: 116688 0x64D670: ISCreateGeneral (general.c:632) n2: 116688 0x4E222E: DMLabelGetValueIS (plexlabel.c:518) n2: 116688 0x4E8069: DMPlexGetLabelIdIS (plexlabel.c:1013) n1: 116064 0x41D333: SetUpSymetricBoundaryMaching (SetupMeshes.c:1030) n0: 116064 0x40796B: main (Nwtun.c:65) n0: 624 in 2 places, all below massif's threshold (01.00%) n0: 0 in 2 places, all below massif's threshold (01.00%) n0: 0 in 6 places, all below massif's threshold (01.00%) n0: 0 in 1 place, below massif's threshold (01.00%) n2: 114480 0xC88E56: PetscSFBasicGetPack (sfbasic.c:779) n1: 114480 0xC8AA7F: PetscSFBcastBegin_Basic (sfbasic.c:914) n2: 114480 0x687407: PetscSFBcastBegin (sf.c:926) n2: 108800 0x51554F: DMGlobalToLocalBegin (dm.c:1656) n1: 93840 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 93840 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93840 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93840 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93840 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93840 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93840 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93840 0x40BC62: main (Nwtun.c:417) n0: 14960 in 1 place, below massif's threshold (01.00%) n0: 5680 in 8 places, all below massif's threshold (01.00%) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 114480 0xC861ED: PetscSFSetUp_Basic (sfbasic.c:557) n2: 114480 0x67F6D7: PetscSFSetUp (sf.c:192) n2: 114480 0x68739A: PetscSFBcastBegin (sf.c:925) n2: 108800 0x51554F: DMGlobalToLocalBegin (dm.c:1656) n1: 93840 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 93840 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93840 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93840 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93840 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93840 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93840 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93840 0x40BC62: main (Nwtun.c:417) n0: 14960 in 1 place, below massif's threshold (01.00%) n0: 5680 in 7 places, all below massif's threshold (01.00%) n0: 0 in 2 places, all below massif's threshold (01.00%) n1: 108480 0x521DAD: DMCreateDefaultSF (dm.c:3074) n1: 108480 0x520D8A: DMGetDefaultSF (dm.c:2991) n2: 108480 0x5153DB: DMGlobalToLocalBegin (dm.c:1649) n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93600 0x40BC62: main (Nwtun.c:417) n0: 14880 in 1 place, below massif's threshold (01.00%) n1: 90944 0x462C3D: DMSetUp_Plex (plex.c:1587) n1: 90944 0x50B88B: DMSetUp (dm.c:560) n1: 90944 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534) n1: 90944 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 90944 0x415800: CreateMesh (SetupMeshes.c:53) n0: 90944 0x4076DF: main (Nwtun.c:55) n1: 90944 0x462AB1: DMSetUp_Plex (plex.c:1583) n2: 90944 0x50B88B: DMSetUp (dm.c:560) n1: 90944 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534) n1: 90944 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 90944 0x415800: CreateMesh (SetupMeshes.c:53) n0: 90944 0x4076DF: main (Nwtun.c:55) n0: 0 in 3 places, all below massif's threshold (01.00%) n1: 90944 0x462A1D: DMSetUp_Plex (plex.c:1582) n2: 90944 0x50B88B: DMSetUp (dm.c:560) n1: 90944 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534) n1: 90944 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 90944 0x415800: CreateMesh (SetupMeshes.c:53) n0: 90944 0x4076DF: main (Nwtun.c:55) n0: 0 in 3 places, all below massif's threshold (01.00%) n2: 266240 0x5E920BC: MPIU_Handle_obj_alloc_unsafe (in /home/rlchen/mpich2/lib/libmpich.so.3.0) n1: 192512 0x5EA339B: MPID_Isend (in /home/rlchen/mpich2/lib/libmpich.so.3.0) n1: 192512 0x5E9CBA9: PMPI_Isend (in /home/rlchen/mpich2/lib/libmpich.so.3.0) n1: 192512 0x11532F6: PetscCommBuildTwoSided_Allreduce (mpits.c:174) n1: 192512 0x11538F7: PetscCommBuildTwoSided (mpits.c:237) n1: 192512 0xC8603E: PetscSFSetUp_Basic (sfbasic.c:552) n1: 192512 0x67F6D7: PetscSFSetUp (sf.c:192) n1: 192512 0x689A32: PetscSFFetchAndOpBegin (sf.c:1138) n1: 192512 0x53DD1C: PetscSFConvertPartition (vsectionis.c:1916) n1: 192512 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101) n1: 192512 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 192512 0x4076DF: main (Nwtun.c:55) n1: 73728 0x5EAFDD0: MPID_Type_dup (in /home/rlchen/mpich2/lib/libmpich.so.3.0) n1: 73728 0x5EFFF19: PMPI_Type_dup (in /home/rlchen/mpich2/lib/libmpich.so.3.0) n1: 73728 0xC8750A: PetscSFBasicPackTypeSetup (sfbasic.c:654) n1: 73728 0xC88D96: PetscSFBasicGetPack (sfbasic.c:778) n1: 73728 0xC8B6AF: PetscSFReduceBegin_Basic (sfbasic.c:973) n1: 73728 0x68814F: PetscSFReduceBegin (sf.c:993) n1: 73728 0x682011: PetscSFCreateInverseSF (sf.c:459) n1: 73728 0x53E180: PetscSFConvertPartition (vsectionis.c:1937) n1: 73728 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101) n1: 73728 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 73728 0x4076DF: main (Nwtun.c:55) n1: 180480 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202) n1: 180480 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 180480 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 180480 0x4076DF: main (Nwtun.c:55) n0: 155163 in 150 places, all below massif's threshold (01.00%) #----------- snapshot=33 #----------- time=14373198930 mem_heap_B=5309284 mem_heap_extra_B=42044 mem_stacks_B=0 heap_tree=empty #----------- snapshot=34 #----------- time=14654959061 mem_heap_B=5309268 mem_heap_extra_B=42036 mem_stacks_B=0 heap_tree=empty #----------- snapshot=35 #----------- time=15172359569 mem_heap_B=5309252 mem_heap_extra_B=42028 mem_stacks_B=0 heap_tree=empty #----------- snapshot=36 #----------- time=15458754406 mem_heap_B=5309268 mem_heap_extra_B=42036 mem_stacks_B=0 heap_tree=detailed n4: 5309268 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n21: 4707353 0x57877E: PetscMallocAlign (mal.c:27) n1: 712000 0x8518E1: MatFDColoringSetUpBlocked_AIJ_Private (fdaij.c:121) n1: 712000 0x919037: MatFDColoringSetUp_MPIXAIJ (fdmpiaij.c:570) n1: 712000 0x74CD34: MatFDColoringSetUp (fdmatrix.c:250) n1: 712000 0x43E33A: SNESComputeJacobianDefaultColorNwtun (SetupJacobian.c:907) n1: 712000 0x434942: FormJacobian (SetupJacobian.c:56) n1: 712000 0xA8F850: SNESComputeJacobian (snes.c:2193) n1: 712000 0xAF7C15: SNESSolve_NEWTONLS (ls.c:230) n1: 712000 0xA9E3CC: SNESSolve (snes.c:3743) n1: 712000 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 712000 0x40BC62: main (Nwtun.c:417) n0: 520457 in 389 places, all below massif's threshold (01.00%) n1: 408112 0xC9786A: VecCreate_Seq (bvec3.c:38) n2: 408112 0x6BF560: VecSetType (vecreg.c:53) n1: 405312 0xC99F47: VecCreate_Standard (pbvec.c:269) n1: 405312 0x6BF560: VecSetType (vecreg.c:53) n1: 405312 0xA0773A: DMCreateLocalVector_Section_Private (dmi.c:65) n1: 405312 0x976C9B: DMCreateLocalVector_Plex (plexcreate.c:931) n3: 405312 0x50D220: DMCreateLocalVector (dm.c:709) n3: 249600 0xA035E0: DMGetLocalVector (dmget.c:49) n1: 93600 0x425614: CaculateLocalFunction_LS (SetupFunctions.c:742) n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93600 0x40BC62: main (Nwtun.c:417) n1: 93600 0x4257D8: CaculateLocalFunction_LS (SetupFunctions.c:748) n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93600 0x40BC62: main (Nwtun.c:417) n0: 62400 in 2 places, all below massif's threshold (01.00%) n1: 130752 0x417EA6: ConstructGeometryFVM (SetupMeshes.c:306) n0: 130752 0x407898: main (Nwtun.c:60) n0: 24960 in 1 place, below massif's threshold (01.00%) n0: 2800 in 1 place, below massif's threshold (01.00%) n1: 362400 0x8512B1: MatFDColoringSetUpBlocked_AIJ_Private (fdaij.c:70) n1: 362400 0x919037: MatFDColoringSetUp_MPIXAIJ (fdmpiaij.c:570) n1: 362400 0x74CD34: MatFDColoringSetUp (fdmatrix.c:250) n1: 362400 0x43E33A: SNESComputeJacobianDefaultColorNwtun (SetupJacobian.c:907) n1: 362400 0x434942: FormJacobian (SetupJacobian.c:56) n1: 362400 0xA8F850: SNESComputeJacobian (snes.c:2193) n1: 362400 0xAF7C15: SNESSolve_NEWTONLS (ls.c:230) n1: 362400 0xA9E3CC: SNESSolve (snes.c:3743) n1: 362400 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 362400 0x40BC62: main (Nwtun.c:417) n1: 356000 0x81EE1C: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719) n2: 356000 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664) n1: 340000 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255) n1: 340000 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959) n1: 340000 0x955310: MatXAIJSetPreallocation (gcreate.c:252) n1: 340000 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435) n1: 340000 0x459619: DMCreateMatrix_Plex (plex.c:615) n1: 340000 0x50F42F: DMCreateMatrix (dm.c:941) n0: 340000 0x40B906: main (Nwtun.c:396) n0: 16000 in 2 places, all below massif's threshold (01.00%) n1: 356000 0x81EDCA: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719) n2: 356000 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664) n1: 340000 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255) n1: 340000 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959) n1: 340000 0x955310: MatXAIJSetPreallocation (gcreate.c:252) n1: 340000 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435) n1: 340000 0x459619: DMCreateMatrix_Plex (plex.c:615) n1: 340000 0x50F42F: DMCreateMatrix (dm.c:941) n0: 340000 0x40B906: main (Nwtun.c:396) n0: 16000 in 2 places, all below massif's threshold (01.00%) n2: 233704 0xC99701: VecCreate_MPI_Private (pbvec.c:201) n1: 137064 0xC99C3D: VecCreate_MPI (pbvec.c:244) n1: 137064 0x6BF560: VecSetType (vecreg.c:53) n1: 137064 0xC99FA5: VecCreate_Standard (pbvec.c:271) n2: 137064 0x6BF560: VecSetType (vecreg.c:53) n1: 109360 0xA07131: DMCreateGlobalVector_Section_Private (dmi.c:34) n1: 109360 0x976932: DMCreateGlobalVector_Plex (plexcreate.c:917) n2: 109360 0x50CE38: DMCreateGlobalVector (dm.c:681) n2: 97280 0xA04005: DMGetGlobalVector (dmget.c:154) n1: 85200 0x426EC6: ConstructCellCentriodGradient (SetupFunctions.c:855) n1: 85200 0x42054B: FormTimeStepFunction (SetupFunctions.c:233) n1: 85200 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 85200 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 85200 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 85200 0xA9E3CC: SNESSolve (snes.c:3743) n1: 85200 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 85200 0x40BC62: main (Nwtun.c:417) n0: 12080 in 1 place, below massif's threshold (01.00%) n0: 12080 in 1 place, below massif's threshold (01.00%) n0: 27704 in 4 places, all below massif's threshold (01.00%) n1: 96640 0xC98DCF: VecDuplicate_MPI (pbvec.c:68) n1: 96640 0x6B04CF: VecDuplicate (vector.c:399) n0: 96640 in 8 places, all below massif's threshold (01.00%) n1: 216960 0x521E45: DMCreateDefaultSF (dm.c:3075) n1: 216960 0x520D8A: DMGetDefaultSF (dm.c:2991) n2: 216960 0x5153DB: DMGlobalToLocalBegin (dm.c:1649) n1: 187200 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 187200 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 187200 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 187200 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 187200 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 187200 0xA9E3CC: SNESSolve (snes.c:3743) n1: 187200 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 187200 0x40BC62: main (Nwtun.c:417) n0: 29760 in 1 place, below massif's threshold (01.00%) n1: 154640 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n0: 154640 in 19 places, all below massif's threshold (01.00%) n1: 154640 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n0: 154640 in 19 places, all below massif's threshold (01.00%) n3: 153664 0x50FE4C: DMGetWorkArray (dm.c:1005) n2: 76832 0x45FF3D: DMPlexGetTransitiveClosure (plex.c:1287) n1: 76832 0x47B16A: DMPlexVecGetClosure (plex.c:4614) n0: 76832 in 2 places, all below massif's threshold (01.00%) n0: 0 in 2 places, all below massif's threshold (01.00%) n2: 76832 0x45FFD3: DMPlexGetTransitiveClosure (plex.c:1291) n1: 76832 0x47B16A: DMPlexVecGetClosure (plex.c:4614) n0: 76832 in 2 places, all below massif's threshold (01.00%) n0: 0 in 2 places, all below massif's threshold (01.00%) n0: 0 in 4 places, all below massif's threshold (01.00%) n2: 117480 0x6812ED: PetscSFSetGraph (sf.c:349) n1: 108480 0x522622: DMCreateDefaultSF (dm.c:3116) n1: 108480 0x520D8A: DMGetDefaultSF (dm.c:2991) n2: 108480 0x5153DB: DMGlobalToLocalBegin (dm.c:1649) n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93600 0x40BC62: main (Nwtun.c:417) n0: 14880 in 1 place, below massif's threshold (01.00%) n0: 9000 in 9 places, all below massif's threshold (01.00%) n2: 117480 0x681299: PetscSFSetGraph (sf.c:349) n1: 108480 0x522622: DMCreateDefaultSF (dm.c:3116) n1: 108480 0x520D8A: DMGetDefaultSF (dm.c:2991) n2: 108480 0x5153DB: DMGlobalToLocalBegin (dm.c:1649) n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93600 0x40BC62: main (Nwtun.c:417) n0: 14880 in 1 place, below massif's threshold (01.00%) n0: 9000 in 9 places, all below massif's threshold (01.00%) n2: 117480 0xC88EE6: PetscSFBasicGetPack (sfbasic.c:779) n1: 117480 0xC8AA7F: PetscSFBcastBegin_Basic (sfbasic.c:914) n2: 117480 0x687407: PetscSFBcastBegin (sf.c:926) n2: 108480 0x51554F: DMGlobalToLocalBegin (dm.c:1656) n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93600 0x40BC62: main (Nwtun.c:417) n0: 14880 in 1 place, below massif's threshold (01.00%) n0: 9000 in 8 places, all below massif's threshold (01.00%) n0: 0 in 1 place, below massif's threshold (01.00%) n2: 116064 0x659AFA: ISCreate (isreg.c:41) n2: 116064 0x64D670: ISCreateGeneral (general.c:632) n2: 116064 0x4E222E: DMLabelGetValueIS (plexlabel.c:518) n2: 116064 0x4E8069: DMPlexGetLabelIdIS (plexlabel.c:1013) n1: 116064 0x41D333: SetUpSymetricBoundaryMaching (SetupMeshes.c:1030) n0: 116064 0x40796B: main (Nwtun.c:65) n0: 0 in 2 places, all below massif's threshold (01.00%) n0: 0 in 2 places, all below massif's threshold (01.00%) n0: 0 in 6 places, all below massif's threshold (01.00%) n0: 0 in 1 place, below massif's threshold (01.00%) n2: 114480 0xC88E56: PetscSFBasicGetPack (sfbasic.c:779) n1: 114480 0xC8AA7F: PetscSFBcastBegin_Basic (sfbasic.c:914) n2: 114480 0x687407: PetscSFBcastBegin (sf.c:926) n2: 108800 0x51554F: DMGlobalToLocalBegin (dm.c:1656) n1: 93840 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 93840 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93840 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93840 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93840 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93840 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93840 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93840 0x40BC62: main (Nwtun.c:417) n0: 14960 in 1 place, below massif's threshold (01.00%) n0: 5680 in 8 places, all below massif's threshold (01.00%) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 114480 0xC861ED: PetscSFSetUp_Basic (sfbasic.c:557) n2: 114480 0x67F6D7: PetscSFSetUp (sf.c:192) n2: 114480 0x68739A: PetscSFBcastBegin (sf.c:925) n2: 108800 0x51554F: DMGlobalToLocalBegin (dm.c:1656) n1: 93840 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 93840 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93840 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93840 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93840 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93840 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93840 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93840 0x40BC62: main (Nwtun.c:417) n0: 14960 in 1 place, below massif's threshold (01.00%) n0: 5680 in 7 places, all below massif's threshold (01.00%) n0: 0 in 2 places, all below massif's threshold (01.00%) n1: 108480 0x521DAD: DMCreateDefaultSF (dm.c:3074) n1: 108480 0x520D8A: DMGetDefaultSF (dm.c:2991) n2: 108480 0x5153DB: DMGlobalToLocalBegin (dm.c:1649) n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93600 0x40BC62: main (Nwtun.c:417) n0: 14880 in 1 place, below massif's threshold (01.00%) n1: 90944 0x462C3D: DMSetUp_Plex (plex.c:1587) n1: 90944 0x50B88B: DMSetUp (dm.c:560) n1: 90944 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534) n1: 90944 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 90944 0x415800: CreateMesh (SetupMeshes.c:53) n0: 90944 0x4076DF: main (Nwtun.c:55) n1: 90944 0x462AB1: DMSetUp_Plex (plex.c:1583) n2: 90944 0x50B88B: DMSetUp (dm.c:560) n1: 90944 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534) n1: 90944 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 90944 0x415800: CreateMesh (SetupMeshes.c:53) n0: 90944 0x4076DF: main (Nwtun.c:55) n0: 0 in 3 places, all below massif's threshold (01.00%) n1: 90944 0x462A1D: DMSetUp_Plex (plex.c:1582) n2: 90944 0x50B88B: DMSetUp (dm.c:560) n1: 90944 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534) n1: 90944 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 90944 0x415800: CreateMesh (SetupMeshes.c:53) n0: 90944 0x4076DF: main (Nwtun.c:55) n0: 0 in 3 places, all below massif's threshold (01.00%) n2: 266240 0x5E920BC: MPIU_Handle_obj_alloc_unsafe (in /home/rlchen/mpich2/lib/libmpich.so.3.0) n1: 192512 0x5EA339B: MPID_Isend (in /home/rlchen/mpich2/lib/libmpich.so.3.0) n1: 192512 0x5E9CBA9: PMPI_Isend (in /home/rlchen/mpich2/lib/libmpich.so.3.0) n1: 192512 0x11532F6: PetscCommBuildTwoSided_Allreduce (mpits.c:174) n1: 192512 0x11538F7: PetscCommBuildTwoSided (mpits.c:237) n1: 192512 0xC8603E: PetscSFSetUp_Basic (sfbasic.c:552) n1: 192512 0x67F6D7: PetscSFSetUp (sf.c:192) n1: 192512 0x689A32: PetscSFFetchAndOpBegin (sf.c:1138) n1: 192512 0x53DD1C: PetscSFConvertPartition (vsectionis.c:1916) n1: 192512 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101) n1: 192512 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 192512 0x4076DF: main (Nwtun.c:55) n1: 73728 0x5EAFDD0: MPID_Type_dup (in /home/rlchen/mpich2/lib/libmpich.so.3.0) n1: 73728 0x5EFFF19: PMPI_Type_dup (in /home/rlchen/mpich2/lib/libmpich.so.3.0) n1: 73728 0xC8750A: PetscSFBasicPackTypeSetup (sfbasic.c:654) n1: 73728 0xC88D96: PetscSFBasicGetPack (sfbasic.c:778) n1: 73728 0xC8B6AF: PetscSFReduceBegin_Basic (sfbasic.c:973) n1: 73728 0x68814F: PetscSFReduceBegin (sf.c:993) n1: 73728 0x682011: PetscSFCreateInverseSF (sf.c:459) n1: 73728 0x53E180: PetscSFConvertPartition (vsectionis.c:1937) n1: 73728 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101) n1: 73728 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 73728 0x4076DF: main (Nwtun.c:55) n1: 180480 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202) n1: 180480 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 180480 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 180480 0x4076DF: main (Nwtun.c:55) n0: 155195 in 150 places, all below massif's threshold (01.00%) #----------- snapshot=37 #----------- time=15977774523 mem_heap_B=5309284 mem_heap_extra_B=42044 mem_stacks_B=0 heap_tree=empty #----------- snapshot=38 #----------- time=16396490298 mem_heap_B=5309268 mem_heap_extra_B=42036 mem_stacks_B=0 heap_tree=empty #----------- snapshot=39 #----------- time=16768519490 mem_heap_B=5309284 mem_heap_extra_B=42044 mem_stacks_B=0 heap_tree=empty #----------- snapshot=40 #----------- time=17331041754 mem_heap_B=5309252 mem_heap_extra_B=42028 mem_stacks_B=0 heap_tree=empty #----------- snapshot=41 #----------- time=17841489784 mem_heap_B=5309252 mem_heap_extra_B=42028 mem_stacks_B=0 heap_tree=empty #----------- snapshot=42 #----------- time=18369461201 mem_heap_B=5309252 mem_heap_extra_B=42028 mem_stacks_B=0 heap_tree=empty #----------- snapshot=43 #----------- time=18660261684 mem_heap_B=5310218 mem_heap_extra_B=42094 mem_stacks_B=0 heap_tree=detailed n4: 5310218 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n21: 4708335 0x57877E: PetscMallocAlign (mal.c:27) n1: 712000 0x8518E1: MatFDColoringSetUpBlocked_AIJ_Private (fdaij.c:121) n1: 712000 0x919037: MatFDColoringSetUp_MPIXAIJ (fdmpiaij.c:570) n1: 712000 0x74CD34: MatFDColoringSetUp (fdmatrix.c:250) n1: 712000 0x43E33A: SNESComputeJacobianDefaultColorNwtun (SetupJacobian.c:907) n1: 712000 0x434942: FormJacobian (SetupJacobian.c:56) n1: 712000 0xA8F850: SNESComputeJacobian (snes.c:2193) n1: 712000 0xAF7C15: SNESSolve_NEWTONLS (ls.c:230) n1: 712000 0xA9E3CC: SNESSolve (snes.c:3743) n1: 712000 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 712000 0x40BC62: main (Nwtun.c:417) n0: 520815 in 389 places, all below massif's threshold (01.00%) n1: 408112 0xC9786A: VecCreate_Seq (bvec3.c:38) n2: 408112 0x6BF560: VecSetType (vecreg.c:53) n1: 405312 0xC99F47: VecCreate_Standard (pbvec.c:269) n1: 405312 0x6BF560: VecSetType (vecreg.c:53) n1: 405312 0xA0773A: DMCreateLocalVector_Section_Private (dmi.c:65) n1: 405312 0x976C9B: DMCreateLocalVector_Plex (plexcreate.c:931) n3: 405312 0x50D220: DMCreateLocalVector (dm.c:709) n3: 249600 0xA035E0: DMGetLocalVector (dmget.c:49) n1: 93600 0x425614: CaculateLocalFunction_LS (SetupFunctions.c:742) n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93600 0x40BC62: main (Nwtun.c:417) n1: 93600 0x4257D8: CaculateLocalFunction_LS (SetupFunctions.c:748) n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93600 0x40BC62: main (Nwtun.c:417) n0: 62400 in 2 places, all below massif's threshold (01.00%) n1: 130752 0x417EA6: ConstructGeometryFVM (SetupMeshes.c:306) n0: 130752 0x407898: main (Nwtun.c:60) n0: 24960 in 1 place, below massif's threshold (01.00%) n0: 2800 in 1 place, below massif's threshold (01.00%) n1: 362400 0x8512B1: MatFDColoringSetUpBlocked_AIJ_Private (fdaij.c:70) n1: 362400 0x919037: MatFDColoringSetUp_MPIXAIJ (fdmpiaij.c:570) n1: 362400 0x74CD34: MatFDColoringSetUp (fdmatrix.c:250) n1: 362400 0x43E33A: SNESComputeJacobianDefaultColorNwtun (SetupJacobian.c:907) n1: 362400 0x434942: FormJacobian (SetupJacobian.c:56) n1: 362400 0xA8F850: SNESComputeJacobian (snes.c:2193) n1: 362400 0xAF7C15: SNESSolve_NEWTONLS (ls.c:230) n1: 362400 0xA9E3CC: SNESSolve (snes.c:3743) n1: 362400 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 362400 0x40BC62: main (Nwtun.c:417) n1: 356000 0x81EE1C: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719) n2: 356000 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664) n1: 340000 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255) n1: 340000 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959) n1: 340000 0x955310: MatXAIJSetPreallocation (gcreate.c:252) n1: 340000 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435) n1: 340000 0x459619: DMCreateMatrix_Plex (plex.c:615) n1: 340000 0x50F42F: DMCreateMatrix (dm.c:941) n0: 340000 0x40B906: main (Nwtun.c:396) n0: 16000 in 2 places, all below massif's threshold (01.00%) n1: 356000 0x81EDCA: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719) n2: 356000 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664) n1: 340000 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255) n1: 340000 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959) n1: 340000 0x955310: MatXAIJSetPreallocation (gcreate.c:252) n1: 340000 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435) n1: 340000 0x459619: DMCreateMatrix_Plex (plex.c:615) n1: 340000 0x50F42F: DMCreateMatrix (dm.c:941) n0: 340000 0x40B906: main (Nwtun.c:396) n0: 16000 in 2 places, all below massif's threshold (01.00%) n2: 233704 0xC99701: VecCreate_MPI_Private (pbvec.c:201) n1: 137064 0xC99C3D: VecCreate_MPI (pbvec.c:244) n1: 137064 0x6BF560: VecSetType (vecreg.c:53) n1: 137064 0xC99FA5: VecCreate_Standard (pbvec.c:271) n2: 137064 0x6BF560: VecSetType (vecreg.c:53) n1: 109360 0xA07131: DMCreateGlobalVector_Section_Private (dmi.c:34) n1: 109360 0x976932: DMCreateGlobalVector_Plex (plexcreate.c:917) n2: 109360 0x50CE38: DMCreateGlobalVector (dm.c:681) n2: 97280 0xA04005: DMGetGlobalVector (dmget.c:154) n1: 85200 0x426EC6: ConstructCellCentriodGradient (SetupFunctions.c:855) n1: 85200 0x42054B: FormTimeStepFunction (SetupFunctions.c:233) n1: 85200 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 85200 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 85200 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 85200 0xA9E3CC: SNESSolve (snes.c:3743) n1: 85200 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 85200 0x40BC62: main (Nwtun.c:417) n0: 12080 in 1 place, below massif's threshold (01.00%) n0: 12080 in 1 place, below massif's threshold (01.00%) n0: 27704 in 4 places, all below massif's threshold (01.00%) n1: 96640 0xC98DCF: VecDuplicate_MPI (pbvec.c:68) n1: 96640 0x6B04CF: VecDuplicate (vector.c:399) n0: 96640 in 8 places, all below massif's threshold (01.00%) n1: 216960 0x521E45: DMCreateDefaultSF (dm.c:3075) n1: 216960 0x520D8A: DMGetDefaultSF (dm.c:2991) n2: 216960 0x5153DB: DMGlobalToLocalBegin (dm.c:1649) n1: 187200 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 187200 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 187200 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 187200 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 187200 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 187200 0xA9E3CC: SNESSolve (snes.c:3743) n1: 187200 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 187200 0x40BC62: main (Nwtun.c:417) n0: 29760 in 1 place, below massif's threshold (01.00%) n1: 154640 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n0: 154640 in 19 places, all below massif's threshold (01.00%) n1: 154640 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n0: 154640 in 19 places, all below massif's threshold (01.00%) n3: 153664 0x50FE4C: DMGetWorkArray (dm.c:1005) n2: 76832 0x45FF3D: DMPlexGetTransitiveClosure (plex.c:1287) n1: 76832 0x47B16A: DMPlexVecGetClosure (plex.c:4614) n0: 76832 in 2 places, all below massif's threshold (01.00%) n0: 0 in 2 places, all below massif's threshold (01.00%) n2: 76832 0x45FFD3: DMPlexGetTransitiveClosure (plex.c:1291) n1: 76832 0x47B16A: DMPlexVecGetClosure (plex.c:4614) n0: 76832 in 2 places, all below massif's threshold (01.00%) n0: 0 in 2 places, all below massif's threshold (01.00%) n0: 0 in 4 places, all below massif's threshold (01.00%) n2: 117480 0x6812ED: PetscSFSetGraph (sf.c:349) n1: 108480 0x522622: DMCreateDefaultSF (dm.c:3116) n1: 108480 0x520D8A: DMGetDefaultSF (dm.c:2991) n2: 108480 0x5153DB: DMGlobalToLocalBegin (dm.c:1649) n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93600 0x40BC62: main (Nwtun.c:417) n0: 14880 in 1 place, below massif's threshold (01.00%) n0: 9000 in 9 places, all below massif's threshold (01.00%) n2: 117480 0x681299: PetscSFSetGraph (sf.c:349) n1: 108480 0x522622: DMCreateDefaultSF (dm.c:3116) n1: 108480 0x520D8A: DMGetDefaultSF (dm.c:2991) n2: 108480 0x5153DB: DMGlobalToLocalBegin (dm.c:1649) n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93600 0x40BC62: main (Nwtun.c:417) n0: 14880 in 1 place, below massif's threshold (01.00%) n0: 9000 in 9 places, all below massif's threshold (01.00%) n2: 117480 0xC88EE6: PetscSFBasicGetPack (sfbasic.c:779) n1: 117480 0xC8AA7F: PetscSFBcastBegin_Basic (sfbasic.c:914) n2: 117480 0x687407: PetscSFBcastBegin (sf.c:926) n2: 108480 0x51554F: DMGlobalToLocalBegin (dm.c:1656) n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93600 0x40BC62: main (Nwtun.c:417) n0: 14880 in 1 place, below massif's threshold (01.00%) n0: 9000 in 8 places, all below massif's threshold (01.00%) n0: 0 in 1 place, below massif's threshold (01.00%) n2: 116688 0x659AFA: ISCreate (isreg.c:41) n2: 116688 0x64D670: ISCreateGeneral (general.c:632) n2: 116688 0x4E222E: DMLabelGetValueIS (plexlabel.c:518) n2: 116688 0x4E8069: DMPlexGetLabelIdIS (plexlabel.c:1013) n1: 116064 0x41D333: SetUpSymetricBoundaryMaching (SetupMeshes.c:1030) n0: 116064 0x40796B: main (Nwtun.c:65) n0: 624 in 2 places, all below massif's threshold (01.00%) n0: 0 in 2 places, all below massif's threshold (01.00%) n0: 0 in 6 places, all below massif's threshold (01.00%) n0: 0 in 1 place, below massif's threshold (01.00%) n2: 114480 0xC88E56: PetscSFBasicGetPack (sfbasic.c:779) n1: 114480 0xC8AA7F: PetscSFBcastBegin_Basic (sfbasic.c:914) n2: 114480 0x687407: PetscSFBcastBegin (sf.c:926) n2: 108800 0x51554F: DMGlobalToLocalBegin (dm.c:1656) n1: 93840 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 93840 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93840 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93840 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93840 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93840 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93840 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93840 0x40BC62: main (Nwtun.c:417) n0: 14960 in 1 place, below massif's threshold (01.00%) n0: 5680 in 8 places, all below massif's threshold (01.00%) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 114480 0xC861ED: PetscSFSetUp_Basic (sfbasic.c:557) n2: 114480 0x67F6D7: PetscSFSetUp (sf.c:192) n2: 114480 0x68739A: PetscSFBcastBegin (sf.c:925) n2: 108800 0x51554F: DMGlobalToLocalBegin (dm.c:1656) n1: 93840 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 93840 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93840 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93840 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93840 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93840 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93840 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93840 0x40BC62: main (Nwtun.c:417) n0: 14960 in 1 place, below massif's threshold (01.00%) n0: 5680 in 7 places, all below massif's threshold (01.00%) n0: 0 in 2 places, all below massif's threshold (01.00%) n1: 108480 0x521DAD: DMCreateDefaultSF (dm.c:3074) n1: 108480 0x520D8A: DMGetDefaultSF (dm.c:2991) n2: 108480 0x5153DB: DMGlobalToLocalBegin (dm.c:1649) n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743) n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806) n0: 93600 0x40BC62: main (Nwtun.c:417) n0: 14880 in 1 place, below massif's threshold (01.00%) n1: 90944 0x462C3D: DMSetUp_Plex (plex.c:1587) n1: 90944 0x50B88B: DMSetUp (dm.c:560) n1: 90944 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534) n1: 90944 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 90944 0x415800: CreateMesh (SetupMeshes.c:53) n0: 90944 0x4076DF: main (Nwtun.c:55) n1: 90944 0x462AB1: DMSetUp_Plex (plex.c:1583) n2: 90944 0x50B88B: DMSetUp (dm.c:560) n1: 90944 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534) n1: 90944 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 90944 0x415800: CreateMesh (SetupMeshes.c:53) n0: 90944 0x4076DF: main (Nwtun.c:55) n0: 0 in 3 places, all below massif's threshold (01.00%) n1: 90944 0x462A1D: DMSetUp_Plex (plex.c:1582) n2: 90944 0x50B88B: DMSetUp (dm.c:560) n1: 90944 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534) n1: 90944 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 90944 0x415800: CreateMesh (SetupMeshes.c:53) n0: 90944 0x4076DF: main (Nwtun.c:55) n0: 0 in 3 places, all below massif's threshold (01.00%) n2: 266240 0x5E920BC: MPIU_Handle_obj_alloc_unsafe (in /home/rlchen/mpich2/lib/libmpich.so.3.0) n1: 192512 0x5EA339B: MPID_Isend (in /home/rlchen/mpich2/lib/libmpich.so.3.0) n1: 192512 0x5E9CBA9: PMPI_Isend (in /home/rlchen/mpich2/lib/libmpich.so.3.0) n1: 192512 0x11532F6: PetscCommBuildTwoSided_Allreduce (mpits.c:174) n1: 192512 0x11538F7: PetscCommBuildTwoSided (mpits.c:237) n1: 192512 0xC8603E: PetscSFSetUp_Basic (sfbasic.c:552) n1: 192512 0x67F6D7: PetscSFSetUp (sf.c:192) n1: 192512 0x689A32: PetscSFFetchAndOpBegin (sf.c:1138) n1: 192512 0x53DD1C: PetscSFConvertPartition (vsectionis.c:1916) n1: 192512 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101) n1: 192512 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 192512 0x4076DF: main (Nwtun.c:55) n1: 73728 0x5EAFDD0: MPID_Type_dup (in /home/rlchen/mpich2/lib/libmpich.so.3.0) n1: 73728 0x5EFFF19: PMPI_Type_dup (in /home/rlchen/mpich2/lib/libmpich.so.3.0) n1: 73728 0xC8750A: PetscSFBasicPackTypeSetup (sfbasic.c:654) n1: 73728 0xC88D96: PetscSFBasicGetPack (sfbasic.c:778) n1: 73728 0xC8B6AF: PetscSFReduceBegin_Basic (sfbasic.c:973) n1: 73728 0x68814F: PetscSFReduceBegin (sf.c:993) n1: 73728 0x682011: PetscSFCreateInverseSF (sf.c:459) n1: 73728 0x53E180: PetscSFConvertPartition (vsectionis.c:1937) n1: 73728 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101) n1: 73728 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 73728 0x4076DF: main (Nwtun.c:55) n1: 180480 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202) n1: 180480 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 180480 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 180480 0x4076DF: main (Nwtun.c:55) n0: 155163 in 150 places, all below massif's threshold (01.00%) #----------- snapshot=44 #----------- time=19180269224 mem_heap_B=5309284 mem_heap_extra_B=42044 mem_stacks_B=0 heap_tree=empty #----------- snapshot=45 #----------- time=19461214172 mem_heap_B=5310218 mem_heap_extra_B=42094 mem_stacks_B=0 heap_tree=empty #----------- snapshot=46 #----------- time=19980095559 mem_heap_B=5309252 mem_heap_extra_B=42028 mem_stacks_B=0 heap_tree=empty #----------- snapshot=47 #----------- time=20484932365 mem_heap_B=11658972 mem_heap_extra_B=46180 mem_stacks_B=0 heap_tree=empty #----------- snapshot=48 #----------- time=20818609640 mem_heap_B=8938979 mem_heap_extra_B=46125 mem_stacks_B=0 heap_tree=empty #----------- snapshot=49 #----------- time=21431132383 mem_heap_B=8970627 mem_heap_extra_B=46525 mem_stacks_B=0 heap_tree=empty #----------- snapshot=50 #----------- time=21762203435 mem_heap_B=8970627 mem_heap_extra_B=46525 mem_stacks_B=0 heap_tree=empty #----------- snapshot=51 #----------- time=22092956732 mem_heap_B=8970627 mem_heap_extra_B=46525 mem_stacks_B=0 heap_tree=empty From bsmith at mcs.anl.gov Tue Feb 16 23:57:21 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 16 Feb 2016 23:57:21 -0600 Subject: [petsc-users] Error with MatSchurComplement In-Reply-To: References: Message-ID: <1FC93CCE-F389-4555-BDDF-FEF60ABCC154@mcs.anl.gov> Hmm, it seems this should never happen. It is like inconsistent (and in correct) IS are being generated. 1 is an odd number of processes, does it happen then? Are you providing the IS yourself to the PCFIELDSPLIT or using the default? If you are providing them is it possible that you have a bug in generating them for certain configurations? Of course it could also be a bug in PETSc. I recommend trying to produce the crash on a small a problem as possible then run in the debugger to check the IS at each stage to see why one of them becomes bonkers. It could be we should have more asserts in PETSc to insure that input IS are compatible. Barry > On Feb 16, 2016, at 10:37 PM, Bhalla, Amneet Pal S wrote: > > Hi Folks, > > I am getting error with odd number of processors with PCFIELDSPLIT. The error log is attached. Any hints where to start digging? > > > +++++++++++++++++++++++++++++++++++++++++++++++++++ > At beginning of timestep # 0 > Simulation time is 0 > Residual norms for ib_ solve. > 0 KSP unpreconditioned resid norm 5.093934451651e+05 true resid norm 5.093934451651e+05 ||r(i)||/||b|| 1.000000000000e+00 > [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [2]PETSC ERROR: Nonconforming object sizes > [2]PETSC ERROR: Local columns of A10 1072 do not equal local rows of A00 1104 > [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [2]PETSC ERROR: Petsc Release Version 3.6.1, unknown > [2]PETSC ERROR: ./main2d on a darwin-dbg named Amneets-MBP.attlocal.net by Taylor Tue Feb 16 20:30:45 2016 > [2]PETSC ERROR: Configure options --CC=mpicc --CXX=mpicxx --FC=mpif90 --PETSC_ARCH=darwin-dbg --with-debugging=1 --with-c++-support=1 --with-hypre=1 --download-hypre=1 --with-hdf5=yes --with-hdf5-dir=/Users/Taylor/Documents/SOFTWARES/HDF5/ > [2]PETSC ERROR: #1 MatSchurComplementSetSubMatrices() line 261 in /Users/Taylor/Documents/SOFTWARES/PETSc-BitBucket/PETSc/src/ksp/ksp/utils/schurm.c > [2]PETSC ERROR: #2 PCSetUp_FieldSplit() line 714 in /Users/Taylor/Documents/SOFTWARES/PETSc-BitBucket/PETSc/src/ksp/pc/impls/fieldsplit/fieldsplit.c > [2]PETSC ERROR: #3 PCSetUp() line 983 in /Users/Taylor/Documents/SOFTWARES/PETSc-BitBucket/PETSc/src/ksp/pc/interface/precon.c > [2]PETSC ERROR: #4 KSPSetUp() line 332 in /Users/Taylor/Documents/SOFTWARES/PETSc-BitBucket/PETSc/src/ksp/ksp/interface/itfunc.c > [2]PETSC ERROR: #5 KSPSolve() line 546 in /Users/Taylor/Documents/SOFTWARES/PETSc-BitBucket/PETSc/src/ksp/ksp/interface/itfunc.c > [2]PETSC ERROR: #6 solveSystem() line 229 in ../../../IBAMR/ibtk/lib/../src/solvers/impls/PETScLevelSolver.cpp > P=00002:Program abort called in file ``../../../IBAMR/ibtk/lib/../src/solvers/impls/PETScLevelSolver.cpp'' at line 229 > P=00002:ERROR MESSAGE: > P=00002: > application called MPI_Abort(MPI_COMM_WORLD, -1) - process 2 > *** Warning, AutoPtr is deprecated and will be removed in a future library version! ./include/libmesh/auto_ptr.h, line 271, compiled Feb 3 2016 at 10:29:17 *** > > > ? Amneet > > From bsmith at mcs.anl.gov Wed Feb 17 00:09:27 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 17 Feb 2016 00:09:27 -0600 Subject: [petsc-users] SNESComputeJacobianDefaultColor use too much memory In-Reply-To: <56C409D9.70002@gmail.com> References: <56C319D0.9060208@gmail.com> <684F7147-4F58-42E3-9C9E-2C2297383276@mcs.anl.gov> <56C3EC9B.5080306@gmail.com> <1199051F-2EC1-4462-A7C0-E2DDCA95A4C3@mcs.anl.gov> <56C409D9.70002@gmail.com> Message-ID: <2424BDF0-AD22-40EB-9920-AF697BFF4DAF@mcs.anl.gov> Yes, this is the type of output I was expecting. Now you need to produce it for a large case. Barry > On Feb 16, 2016, at 11:49 PM, Rongliang Chen wrote: > > Hi Barry, > > I run the code with valgrind on a workstation for a smaller case and it produces some ASCII information (see attached). Is this helpful? > > Best, > Rongliang > > On 02/17/2016 01:30 PM, Barry Smith wrote: >> Hmm, something didn't work right with massif. I should give a bunch of ASCII information about how much memory is used at different times in the code. You may need to play around with the massif options and google documentation on how to get it to provide useful information. Once it produces the useful information it will be very helpful. >> >> time=0 >> mem_heap_B=0 >> mem_heap_extra_B=0 >> mem_stacks_B=0 >> heap_tree=empty >> >> >>> On Feb 16, 2016, at 9:44 PM, Rongliang Chen wrote: >>> >>> Dear Barry, >>> >>> Many thanks for your reply. >>> >>> I checked with the valgrind and did not obtain any outputs (massif.out.) because the job was killed before it reached the end. >>> >>> Then I switch to a smaller case, it works well and one of the output is attached (I did not find any useful information in it). The output with the option -mat_coloring_view is followed, which shows that the number of colors is 65. Any ideas for this? >>> >>> MatColoring Object: 480 MPI processes >>> type: sl >>> Weight type: RANDOM >>> Distance 2, Max. Colors 65535 >>> Number of colors 65 >>> Number of total columns 1637350 >>> >>> Best regards, >>> Rongliang >>> >>> On 02/17/2016 01:13 AM, Barry Smith wrote: >>>> How many colors are needed? >>>> >>>> You need to produce a breakdown of where all the memory is being used. For example valgrind with the >>>> http://valgrind.org/docs/manual/ms-manual.html >>>> >>>> >>>> Barry >>>> >>>> >>>>> On Feb 16, 2016, at 6:45 AM, Rongliang Chen >>>>> wrote: >>>>> >>>>> Dear all, >>>>> >>>>> I am using the DMPlex to solve a PDE on a unstructured mesh and I use the SNESComputeJacobianDefaultColor to compute the Jacobian matrix. >>>>> >>>>> My code works well for small problems (such as problem with 3.3x10^5 cells using 120 cores) but when I increase the number of cells (2.6x10^6 cells using 1920 cores), I go out of memory in the function MatColoringApply. It shows that one of the cores uses over 11G memory. I think this is unreasonable. Do you have any suggestions for debugging this problem? >>>>> >>>>> Best regards, >>>>> Rongliang >>>>> >>> > > From rongliang.chan at gmail.com Wed Feb 17 01:49:12 2016 From: rongliang.chan at gmail.com (Rongliang Chen) Date: Wed, 17 Feb 2016 15:49:12 +0800 Subject: [petsc-users] SNESComputeJacobianDefaultColor use too much memory In-Reply-To: <2424BDF0-AD22-40EB-9920-AF697BFF4DAF@mcs.anl.gov> References: <56C319D0.9060208@gmail.com> <684F7147-4F58-42E3-9C9E-2C2297383276@mcs.anl.gov> <56C3EC9B.5080306@gmail.com> <1199051F-2EC1-4462-A7C0-E2DDCA95A4C3@mcs.anl.gov> <56C409D9.70002@gmail.com> <2424BDF0-AD22-40EB-9920-AF697BFF4DAF@mcs.anl.gov> Message-ID: <56C425F8.1060306@gmail.com> Dear Barry, The massif output for the large case is attached. It shows that the job was kill in the function MatGetSubMatrix_MPIAIJ_All(). Any suggestions? -------------------- [8]PETSC ERROR: #1 MatGetSubMatrix_MPIAIJ_All() line 622 in /home/rlchen/soft/petsc-3.5.2/src/mat/impls/aij/mpi/mpiov.c [8]PETSC ERROR: #2 MatGetSeqNonzeroStructure_MPIAIJ() line 3010 in /home/rlchen/soft/petsc-3.5.2/src/mat/impls/aij/mpi/mpiaij.c [8]PETSC ERROR: #3 MatGetSeqNonzeroStructure() line 6487 in /home/rlchen/soft/petsc-3.5.2/src/mat/interface/matrix.c [8]PETSC ERROR: #4 MatColoringApply_SL() line 78 in /home/rlchen/soft/petsc-3.5.2/src/mat/color/impls/minpack/color.c [8]PETSC ERROR: #5 MatColoringApply() line 379 in /home/rlchen/soft/petsc-3.5.2/src/mat/color/interface/matcoloring.c [8]PETSC ERROR: #6 SNESComputeJacobianDefaultColor() line 71 in /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snesj2.c [8]PETSC ERROR: #7 FormJacobian() line 58 in /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/SetupJacobian.c [8]PETSC ERROR: #8 SNESComputeJacobian() line 2193 in /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c [8]PETSC ERROR: #9 SNESSolve_NEWTONLS() line 230 in /home/rlchen/soft/petsc-3.5.2/src/snes/impls/ls/ls.c [8]PETSC ERROR: #10 SNESSolve() line 3743 in /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c [8]PETSC ERROR: #11 SolveTimeDependent() line 758 in /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/Nwtun.c [8]PETSC ERROR: #12 main() line 417 in /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/Nwtun.c ------------------- Best, Rongliang On 02/17/2016 02:09 PM, Barry Smith wrote: > Yes, this is the type of output I was expecting. Now you need to produce it for a large case. > > Barry > >> On Feb 16, 2016, at 11:49 PM, Rongliang Chen wrote: >> >> Hi Barry, >> >> I run the code with valgrind on a workstation for a smaller case and it produces some ASCII information (see attached). Is this helpful? >> >> Best, >> Rongliang >> >> On 02/17/2016 01:30 PM, Barry Smith wrote: >>> Hmm, something didn't work right with massif. I should give a bunch of ASCII information about how much memory is used at different times in the code. You may need to play around with the massif options and google documentation on how to get it to provide useful information. Once it produces the useful information it will be very helpful. >>> >>> time=0 >>> mem_heap_B=0 >>> mem_heap_extra_B=0 >>> mem_stacks_B=0 >>> heap_tree=empty >>> >>> >>>> On Feb 16, 2016, at 9:44 PM, Rongliang Chen wrote: >>>> >>>> Dear Barry, >>>> >>>> Many thanks for your reply. >>>> >>>> I checked with the valgrind and did not obtain any outputs (massif.out.) because the job was killed before it reached the end. >>>> >>>> Then I switch to a smaller case, it works well and one of the output is attached (I did not find any useful information in it). The output with the option -mat_coloring_view is followed, which shows that the number of colors is 65. Any ideas for this? >>>> >>>> MatColoring Object: 480 MPI processes >>>> type: sl >>>> Weight type: RANDOM >>>> Distance 2, Max. Colors 65535 >>>> Number of colors 65 >>>> Number of total columns 1637350 >>>> >>>> Best regards, >>>> Rongliang >>>> >>>> On 02/17/2016 01:13 AM, Barry Smith wrote: >>>>> How many colors are needed? >>>>> >>>>> You need to produce a breakdown of where all the memory is being used. For example valgrind with the >>>>> http://valgrind.org/docs/manual/ms-manual.html >>>>> >>>>> >>>>> Barry >>>>> >>>>> >>>>>> On Feb 16, 2016, at 6:45 AM, Rongliang Chen >>>>>> wrote: >>>>>> >>>>>> Dear all, >>>>>> >>>>>> I am using the DMPlex to solve a PDE on a unstructured mesh and I use the SNESComputeJacobianDefaultColor to compute the Jacobian matrix. >>>>>> >>>>>> My code works well for small problems (such as problem with 3.3x10^5 cells using 120 cores) but when I increase the number of cells (2.6x10^6 cells using 1920 cores), I go out of memory in the function MatColoringApply. It shows that one of the cores uses over 11G memory. I think this is unreasonable. Do you have any suggestions for debugging this problem? >>>>>> >>>>>> Best regards, >>>>>> Rongliang >>>>>> >>>> >> -------------- next part -------------- desc: (none) cmd: ./Nwtun -f ./../meshes/3DNACA0012AIRFOIL_Hex150x50x100x80.exo -overlap 1 -options_file options.txt -time_integral_method 1 -initial_time 0.0 -final_time 4.0 -dt 1.e-3 -timestep 2 -max_time_its 0 -includeenergy -reconstruct -PressureFlux -testcase 3DNACA -output_solution -solutionfile results/3DNACA0012AIRFOIL_Hex150x50x100x80_imp -steps_output 4000 -inflow_u 349.02 -viscosity 0.01 -k 0.026 -monitor_step_gap -R 287.02 -adiabatic 1.4 -Euler -Attack 10 -Mach 0.8 -fd_jacobian_color -limiter zero -RiemannSolver Rusanov -CFL 0.9 -reconstruct -second_order -parallel_csr -parallel_output time_unit: i #----------- snapshot=0 #----------- time=0 mem_heap_B=0 mem_heap_extra_B=0 mem_stacks_B=0 heap_tree=empty #----------- snapshot=1 #----------- time=224006276 mem_heap_B=134033 mem_heap_extra_B=8831 mem_stacks_B=0 heap_tree=empty #----------- snapshot=2 #----------- time=85544444037 mem_heap_B=139207 mem_heap_extra_B=8913 mem_stacks_B=0 heap_tree=empty #----------- snapshot=3 #----------- time=85546748915 mem_heap_B=147017 mem_heap_extra_B=9191 mem_stacks_B=0 heap_tree=empty #----------- snapshot=4 #----------- time=288378803176 mem_heap_B=148049 mem_heap_extra_B=9239 mem_stacks_B=0 heap_tree=empty #----------- snapshot=5 #----------- time=288380578373 mem_heap_B=137997 mem_heap_extra_B=8811 mem_stacks_B=0 heap_tree=empty #----------- snapshot=6 #----------- time=667538944418 mem_heap_B=139029 mem_heap_extra_B=8859 mem_stacks_B=0 heap_tree=detailed n9: 139029 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n12: 81895 0x57877E: PetscMallocAlign (mal.c:27) n1: 19200 0x5DF2BC: EventPerfLogEnsureSize (eventlog.c:211) n1: 19200 0x5C78F5: PetscLogEventRegister (plog.c:755) n1: 19200 0x7BA2C9: MatInitializePackage (dlregismat.c:174) n1: 19200 0x505D39: DMCreate (dm.c:40) n1: 19200 0x443017: DMPlexCreateExodusNwtun (plexexodusii.c:102) n1: 19200 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 19200 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 19200 0x4076DF: main (Nwtun.c:55) n0: 16880 0x4072F7: main (Nwtun.c:43) n1: 10032 0x56BC70: PetscSegBufferCreate (segbuffer.c:68) n1: 10032 0x5B076C: PetscCitationsInitialize (pinit.c:450) n1: 10032 0x5B228E: PetscInitialize (pinit.c:883) n0: 10032 0x406F74: main (Nwtun.c:24) n0: 9784 in 44 places, all below massif's threshold (01.00%) n3: 8272 0x505DF4: DMCreate (dm.c:43) n1: 4136 0x9E5266: DMPlexInterpolate (plexinterpolate.c:344) n1: 4136 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 4136 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 4136 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 4136 0x4076DF: main (Nwtun.c:55) n1: 4136 0x5066C4: DMClone (dm.c:99) n1: 4136 0x47819A: DMCreateCoordinateDM_Plex (plex.c:4355) n1: 4136 0x52788B: DMGetCoordinateDM (dm.c:3557) n1: 4136 0x528290: DMGetCoordinateSection (dm.c:3618) n1: 4136 0x444E4E: DMPlexCreateExodusNwtun (plexexodusii.c:301) n1: 4136 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 4136 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 4136 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n2: 3271 0x561F27: PetscStrallocpy (str.c:188) n1: 2318 0x5DFA3D: EventRegLogRegister (eventlog.c:317) n1: 2318 0x5C7873: PetscLogEventRegister (plog.c:753) n0: 2318 in 166 places, all below massif's threshold (01.00%) n0: 953 in 15 places, all below massif's threshold (01.00%) n1: 3200 0x5DF8BC: EventRegLogRegister (eventlog.c:310) n1: 3200 0x5C7873: PetscLogEventRegister (plog.c:753) n1: 3200 0x7BA2C9: MatInitializePackage (dlregismat.c:174) n1: 3200 0x505D39: DMCreate (dm.c:40) n1: 3200 0x443017: DMPlexCreateExodusNwtun (plexexodusii.c:102) n1: 3200 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 3200 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 3200 0x4076DF: main (Nwtun.c:55) n1: 3200 0x5D5CD5: ClassPerfLogCreate (classlog.c:121) n1: 3200 0x5DA811: PetscStageLogRegister (stagelog.c:230) n1: 3200 0x5C4C70: PetscLogBegin_Private (plog.c:214) n1: 3200 0x5B22E5: PetscInitialize (pinit.c:891) n0: 3200 0x406F74: main (Nwtun.c:24) n1: 2560 0xC169B0: PetscThreadCommWorldInitialize (threadcomm.c:1240) n1: 2560 0xC0DA8A: PetscGetThreadCommWorld (threadcomm.c:82) n1: 2560 0xC0DD9D: PetscCommGetThreadComm (threadcomm.c:117) n1: 2560 0x58238E: PetscCommDuplicate (tagm.c:195) n1: 2560 0x586EE2: PetscHeaderCreate_Private (inherit.c:59) n1: 2560 0x505EEE: DMCreate (dm.c:43) n1: 2560 0x443017: DMPlexCreateExodusNwtun (plexexodusii.c:102) n1: 2560 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 2560 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 2560 0x4076DF: main (Nwtun.c:55) n1: 2136 0x67D9C3: PetscSFCreate (sf.c:44) n0: 2136 in 2 places, all below massif's threshold (01.00%) n1: 1760 0x56EE38: PetscFunctionListAdd_Private (reg.c:210) n0: 1760 in 10 places, all below massif's threshold (01.00%) n1: 1600 0x5D52AA: PetscClassRegLogCreate (classlog.c:36) n1: 1600 0x5DD4FB: PetscStageLogCreate (stagelog.c:638) n1: 1600 0x5C4C0F: PetscLogBegin_Private (plog.c:213) n1: 1600 0x5B22E5: PetscInitialize (pinit.c:891) n0: 1600 0x406F74: main (Nwtun.c:24) n1: 17152 0x5A3ED5: PetscOptionsCreate (options.c:2322) n1: 17152 0x5B1876: PetscInitialize (pinit.c:763) n0: 17152 0x406F74: main (Nwtun.c:24) n0: 14910 in 68 places, all below massif's threshold (01.00%) n1: 13152 0x5EB3FDA: MPIDI_PG_Create (in /home/rlchen/mpich2/lib/libmpich.so.3.0) n1: 13152 0x5EA24B9: MPID_Init (in /home/rlchen/mpich2/lib/libmpich.so.3.0) n1: 13152 0x5E98088: MPIR_Init_thread (in /home/rlchen/mpich2/lib/libmpich.so.3.0) n1: 13152 0x5E981CF: PMPI_Init_thread (in /home/rlchen/mpich2/lib/libmpich.so.3.0) n1: 13152 0x5B1A7E: PetscInitialize (pinit.c:781) n0: 13152 0x406F74: main (Nwtun.c:24) n1: 4096 0x660F8EA: register_printf_specifier (reg-printf.c:58) n1: 4096 0x6BA4770: ??? (in /usr/lib/x86_64-linux-gnu/libquadmath.so.0.0.0) n1: 4096 0x4010138: call_init.part.0 (dl-init.c:78) n1: 4096 0x4010221: _dl_init (dl-init.c:36) n1: 4096 0x4001308: ??? (in /lib/x86_64-linux-gnu/ld-2.19.so) n1: 4096 0x36: ??? n1: 4096 0xFFF0002F5: ??? n1: 4096 0xFFF0002FD: ??? n1: 4096 0xFFF000300: ??? n1: 4096 0xFFF000333: ??? n1: 4096 0xFFF00033C: ??? n1: 4096 0xFFF00033E: ??? n1: 4096 0xFFF00034C: ??? n1: 4096 0xFFF000358: ??? n1: 4096 0xFFF00036E: ??? n1: 4096 0xFFF000370: ??? n1: 4096 0xFFF00037E: ??? n1: 4096 0xFFF000382: ??? n1: 4096 0xFFF00038E: ??? n1: 4096 0xFFF000392: ??? n1: 4096 0xFFF000396: ??? n1: 4096 0xFFF00039C: ??? n1: 4096 0xFFF0003A6: ??? n1: 4096 0xFFF0003A8: ??? n1: 4096 0xFFF0003B6: ??? n1: 4096 0xFFF0003B8: ??? n1: 4096 0xFFF0003C7: ??? n1: 4096 0xFFF0003D4: ??? n1: 4096 0xFFF0003E2: ??? n0: 4096 0xFFF0003EC: ??? n2: 2256 0x58BBC21: ??? (in /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0) n1: 2016 0x597F755: ??? (in /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0) n0: 2016 in 3 places, all below massif's threshold (01.00%) n0: 240 in 1 place, below massif's threshold (01.00%) n1: 2040 0x6611753: register_printf_modifier (reg-modifier.c:69) n1: 2040 0x6BA4749: ??? (in /usr/lib/x86_64-linux-gnu/libquadmath.so.0.0.0) n1: 2040 0x4010138: call_init.part.0 (dl-init.c:78) n1: 2040 0x4010221: _dl_init (dl-init.c:36) n1: 2040 0x4001308: ??? (in /lib/x86_64-linux-gnu/ld-2.19.so) n1: 2040 0x36: ??? n1: 2040 0xFFF0002F5: ??? n1: 2040 0xFFF0002FD: ??? n1: 2040 0xFFF000300: ??? n1: 2040 0xFFF000333: ??? n1: 2040 0xFFF00033C: ??? n1: 2040 0xFFF00033E: ??? n1: 2040 0xFFF00034C: ??? n1: 2040 0xFFF000358: ??? n1: 2040 0xFFF00036E: ??? n1: 2040 0xFFF000370: ??? n1: 2040 0xFFF00037E: ??? n1: 2040 0xFFF000382: ??? n1: 2040 0xFFF00038E: ??? n1: 2040 0xFFF000392: ??? n1: 2040 0xFFF000396: ??? n1: 2040 0xFFF00039C: ??? n1: 2040 0xFFF0003A6: ??? n1: 2040 0xFFF0003A8: ??? n1: 2040 0xFFF0003B6: ??? n1: 2040 0xFFF0003B8: ??? n1: 2040 0xFFF0003C7: ??? n1: 2040 0xFFF0003D4: ??? n1: 2040 0xFFF0003E2: ??? n0: 2040 0xFFF0003EC: ??? n1: 1984 0x6611A0D: register_printf_type (reg-type.c:43) n1: 1984 0x6BA472E: ??? (in /usr/lib/x86_64-linux-gnu/libquadmath.so.0.0.0) n1: 1984 0x4010138: call_init.part.0 (dl-init.c:78) n1: 1984 0x4010221: _dl_init (dl-init.c:36) n1: 1984 0x4001308: ??? (in /lib/x86_64-linux-gnu/ld-2.19.so) n1: 1984 0x36: ??? n1: 1984 0xFFF0002F5: ??? n1: 1984 0xFFF0002FD: ??? n1: 1984 0xFFF000300: ??? n1: 1984 0xFFF000333: ??? n1: 1984 0xFFF00033C: ??? n1: 1984 0xFFF00033E: ??? n1: 1984 0xFFF00034C: ??? n1: 1984 0xFFF000358: ??? n1: 1984 0xFFF00036E: ??? n1: 1984 0xFFF000370: ??? n1: 1984 0xFFF00037E: ??? n1: 1984 0xFFF000382: ??? n1: 1984 0xFFF00038E: ??? n1: 1984 0xFFF000392: ??? n1: 1984 0xFFF000396: ??? n1: 1984 0xFFF00039C: ??? n1: 1984 0xFFF0003A6: ??? n1: 1984 0xFFF0003A8: ??? n1: 1984 0xFFF0003B6: ??? n1: 1984 0xFFF0003B8: ??? n1: 1984 0xFFF0003C7: ??? n1: 1984 0xFFF0003D4: ??? n1: 1984 0xFFF0003E2: ??? n0: 1984 0xFFF0003EC: ??? n1: 1544 0x63F2B3: PetscStackCreate (pstack.c:105) n1: 1544 0x5B2739: PetscInitialize (pinit.c:946) n0: 1544 0x406F74: main (Nwtun.c:24) #----------- snapshot=7 #----------- time=669178970344 mem_heap_B=143791 mem_heap_extra_B=9305 mem_stacks_B=0 heap_tree=empty #----------- snapshot=8 #----------- time=686722556504 mem_heap_B=143807 mem_heap_extra_B=9305 mem_stacks_B=0 heap_tree=empty #----------- snapshot=9 #----------- time=694044915791 mem_heap_B=146405 mem_heap_extra_B=9475 mem_stacks_B=0 heap_tree=empty #----------- snapshot=10 #----------- time=733393978137 mem_heap_B=147309 mem_heap_extra_B=9515 mem_stacks_B=0 heap_tree=empty #----------- snapshot=11 #----------- time=733467252555 mem_heap_B=150111 mem_heap_extra_B=9729 mem_stacks_B=0 heap_tree=empty #----------- snapshot=12 #----------- time=1139843034827 mem_heap_B=150191 mem_heap_extra_B=9737 mem_stacks_B=0 heap_tree=empty #----------- snapshot=13 #----------- time=1141977712727 mem_heap_B=30255663 mem_heap_extra_B=10697 mem_stacks_B=0 heap_tree=empty #----------- snapshot=14 #----------- time=1180154243665 mem_heap_B=30255711 mem_heap_extra_B=10705 mem_stacks_B=0 heap_tree=empty #----------- snapshot=15 #----------- time=1188784700125 mem_heap_B=105582168 mem_heap_extra_B=11936 mem_stacks_B=0 heap_tree=empty #----------- snapshot=16 #----------- time=1200655909356 mem_heap_B=359461085 mem_heap_extra_B=14739 mem_stacks_B=0 heap_tree=detailed n2: 359461085 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n13: 359321645 0x57877E: PetscMallocAlign (mal.c:27) n1: 89512864 0x54037E: PetscSFCreateSectionSF (vsectionis.c:2098) n1: 89512864 0x557BF3: DMPlexDistributeNwtun (plexdistribute.c:128) n1: 89512864 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 89512864 0x4076DF: main (Nwtun.c:55) n1: 44756432 0x462AB1: DMSetUp_Plex (plex.c:1583) n1: 44756432 0x50B88B: DMSetUp (dm.c:560) n1: 44756432 0x557A48: DMPlexDistributeNwtun (plexdistribute.c:116) n1: 44756432 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 44756432 0x4076DF: main (Nwtun.c:55) n1: 44756432 0x5402EC: PetscSFCreateSectionSF (vsectionis.c:2097) n1: 44756432 0x557BF3: DMPlexDistributeNwtun (plexdistribute.c:128) n1: 44756432 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 44756432 0x4076DF: main (Nwtun.c:55) n1: 44756432 0x462A1D: DMSetUp_Plex (plex.c:1582) n1: 44756432 0x50B88B: DMSetUp (dm.c:560) n1: 44756432 0x557A48: DMPlexDistributeNwtun (plexdistribute.c:116) n1: 44756432 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 44756432 0x4076DF: main (Nwtun.c:55) n1: 30096944 0x680E6B: PetscSFSetGraph (sf.c:328) n1: 30096944 0x6822D2: PetscSFCreateInverseSF (sf.c:478) n1: 30096944 0x53E180: PetscSFConvertPartition (vsectionis.c:1937) n1: 30096944 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101) n1: 30096944 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 30096944 0x4076DF: main (Nwtun.c:55) n2: 15048664 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n1: 15048472 0x53EA67: PetscSFDistributeSection (vsectionis.c:1998) n1: 15048472 0x5579E4: DMPlexDistributeNwtun (plexdistribute.c:115) n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 15048472 0x4076DF: main (Nwtun.c:55) n0: 192 in 3 places, all below massif's threshold (01.00%) n2: 15048664 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n1: 15048472 0x53EA67: PetscSFDistributeSection (vsectionis.c:1998) n1: 15048472 0x5579E4: DMPlexDistributeNwtun (plexdistribute.c:115) n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 15048472 0x4076DF: main (Nwtun.c:55) n0: 192 in 3 places, all below massif's threshold (01.00%) n2: 15048472 0x681299: PetscSFSetGraph (sf.c:349) n1: 15048472 0x6822D2: PetscSFCreateInverseSF (sf.c:478) n1: 15048472 0x53E180: PetscSFConvertPartition (vsectionis.c:1937) n1: 15048472 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101) n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 15048472 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n2: 15048472 0x6812ED: PetscSFSetGraph (sf.c:349) n1: 15048472 0x6822D2: PetscSFCreateInverseSF (sf.c:478) n1: 15048472 0x53E180: PetscSFConvertPartition (vsectionis.c:1937) n1: 15048472 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101) n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 15048472 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 15048472 0x66B28C: ISLocalToGlobalMappingCreateSF (isltog.c:193) n1: 15048472 0x53E29B: PetscSFConvertPartition (vsectionis.c:1942) n1: 15048472 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101) n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 15048472 0x4076DF: main (Nwtun.c:55) n1: 15048472 0xC88EE6: PetscSFBasicGetPack (sfbasic.c:779) n1: 15048472 0xC8AA7F: PetscSFBcastBegin_Basic (sfbasic.c:914) n2: 15048472 0x687407: PetscSFBcastBegin (sf.c:926) n1: 15048472 0x66B362: ISLocalToGlobalMappingCreateSF (isltog.c:196) n1: 15048472 0x53E29B: PetscSFConvertPartition (vsectionis.c:1942) n1: 15048472 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101) n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 15048472 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 15048472 0x53EDF4: PetscSFDistributeSection (vsectionis.c:2007) n1: 15048472 0x5579E4: DMPlexDistributeNwtun (plexdistribute.c:115) n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 15048472 0x4076DF: main (Nwtun.c:55) n0: 102853 in 108 places, all below massif's threshold (01.00%) n0: 139440 in 88 places, all below massif's threshold (01.00%) #----------- snapshot=17 #----------- time=1218361277415 mem_heap_B=642593563 mem_heap_extra_B=19765 mem_stacks_B=0 heap_tree=detailed n2: 642593563 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n13: 642453035 0x57877E: PetscMallocAlign (mal.c:27) n1: 163910184 0x66DAC9: ISGlobalToLocalMappingSetUp_Private (isltog.c:489) n1: 163910184 0x66E7B8: ISGlobalToLocalMappingApplyBlock (isltog.c:639) n1: 163910184 0x557EA3: DMPlexDistributeNwtun (plexdistribute.c:134) n1: 163910184 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 163910184 0x4076DF: main (Nwtun.c:55) n1: 89512864 0x54037E: PetscSFCreateSectionSF (vsectionis.c:2098) n1: 89512864 0x557BF3: DMPlexDistributeNwtun (plexdistribute.c:128) n1: 89512864 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 89512864 0x4076DF: main (Nwtun.c:55) n1: 59804904 0xC88EE6: PetscSFBasicGetPack (sfbasic.c:779) n1: 59804904 0xC8AA7F: PetscSFBcastBegin_Basic (sfbasic.c:914) n3: 59804904 0x687407: PetscSFBcastBegin (sf.c:926) n1: 44756432 0x557D40: DMPlexDistributeNwtun (plexdistribute.c:131) n1: 44756432 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 44756432 0x4076DF: main (Nwtun.c:55) n1: 15048472 0x66B362: ISLocalToGlobalMappingCreateSF (isltog.c:196) n1: 15048472 0x53E29B: PetscSFConvertPartition (vsectionis.c:1942) n1: 15048472 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101) n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 15048472 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n3: 59804904 0x681299: PetscSFSetGraph (sf.c:349) n1: 44756432 0x5406F5: PetscSFCreateSectionSF (vsectionis.c:2119) n1: 44756432 0x557BF3: DMPlexDistributeNwtun (plexdistribute.c:128) n1: 44756432 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 44756432 0x4076DF: main (Nwtun.c:55) n1: 15048472 0x6822D2: PetscSFCreateInverseSF (sf.c:478) n1: 15048472 0x53E180: PetscSFConvertPartition (vsectionis.c:1937) n1: 15048472 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101) n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 15048472 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n3: 59804904 0x6812ED: PetscSFSetGraph (sf.c:349) n1: 44756432 0x5406F5: PetscSFCreateSectionSF (vsectionis.c:2119) n1: 44756432 0x557BF3: DMPlexDistributeNwtun (plexdistribute.c:128) n1: 44756432 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 44756432 0x4076DF: main (Nwtun.c:55) n1: 15048472 0x6822D2: PetscSFCreateInverseSF (sf.c:478) n1: 15048472 0x53E180: PetscSFConvertPartition (vsectionis.c:1937) n1: 15048472 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101) n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 15048472 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 44756432 0x5402EC: PetscSFCreateSectionSF (vsectionis.c:2097) n1: 44756432 0x557BF3: DMPlexDistributeNwtun (plexdistribute.c:128) n1: 44756432 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 44756432 0x4076DF: main (Nwtun.c:55) n1: 44756432 0x462AB1: DMSetUp_Plex (plex.c:1583) n1: 44756432 0x50B88B: DMSetUp (dm.c:560) n1: 44756432 0x557A48: DMPlexDistributeNwtun (plexdistribute.c:116) n1: 44756432 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 44756432 0x4076DF: main (Nwtun.c:55) n1: 44756432 0x462A1D: DMSetUp_Plex (plex.c:1582) n1: 44756432 0x50B88B: DMSetUp (dm.c:560) n1: 44756432 0x557A48: DMPlexDistributeNwtun (plexdistribute.c:116) n1: 44756432 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 44756432 0x4076DF: main (Nwtun.c:55) n1: 30096944 0x680E6B: PetscSFSetGraph (sf.c:328) n1: 30096944 0x6822D2: PetscSFCreateInverseSF (sf.c:478) n1: 30096944 0x53E180: PetscSFConvertPartition (vsectionis.c:1937) n1: 30096944 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101) n1: 30096944 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 30096944 0x4076DF: main (Nwtun.c:55) n2: 15048664 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n1: 15048472 0x53EA67: PetscSFDistributeSection (vsectionis.c:1998) n1: 15048472 0x5579E4: DMPlexDistributeNwtun (plexdistribute.c:115) n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 15048472 0x4076DF: main (Nwtun.c:55) n0: 192 in 3 places, all below massif's threshold (01.00%) n2: 15048664 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n1: 15048472 0x53EA67: PetscSFDistributeSection (vsectionis.c:1998) n1: 15048472 0x5579E4: DMPlexDistributeNwtun (plexdistribute.c:115) n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 15048472 0x4076DF: main (Nwtun.c:55) n0: 192 in 3 places, all below massif's threshold (01.00%) n1: 15048472 0x66B28C: ISLocalToGlobalMappingCreateSF (isltog.c:193) n1: 15048472 0x53E29B: PetscSFConvertPartition (vsectionis.c:1942) n1: 15048472 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101) n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 15048472 0x4076DF: main (Nwtun.c:55) n0: 103235 in 109 places, all below massif's threshold (01.00%) n0: 140528 in 88 places, all below massif's threshold (01.00%) #----------- snapshot=18 #----------- time=1236225419622 mem_heap_B=463955737 mem_heap_extra_B=17695 mem_stacks_B=0 heap_tree=empty #----------- snapshot=19 #----------- time=1252147611144 mem_heap_B=463959867 mem_heap_extra_B=17589 mem_stacks_B=0 heap_tree=empty #----------- snapshot=20 #----------- time=1263783858129 mem_heap_B=330597269 mem_heap_extra_B=16011 mem_stacks_B=0 heap_tree=detailed n2: 330597269 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n13: 330457997 0x57877E: PetscMallocAlign (mal.c:27) n1: 44756432 0x462AB1: DMSetUp_Plex (plex.c:1583) n1: 44756432 0x50B88B: DMSetUp (dm.c:560) n1: 44756432 0x557A48: DMPlexDistributeNwtun (plexdistribute.c:116) n1: 44756432 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 44756432 0x4076DF: main (Nwtun.c:55) n1: 44756432 0x462A1D: DMSetUp_Plex (plex.c:1582) n1: 44756432 0x50B88B: DMSetUp (dm.c:560) n1: 44756432 0x557A48: DMPlexDistributeNwtun (plexdistribute.c:116) n1: 44756432 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 44756432 0x4076DF: main (Nwtun.c:55) n1: 44756432 0x46371F: DMPlexSymmetrize (plex.c:1655) n1: 44756432 0x5585AB: DMPlexDistributeNwtun (plexdistribute.c:156) n1: 44756432 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 44756432 0x4076DF: main (Nwtun.c:55) n3: 33957328 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n2: 16978568 0x53EA67: PetscSFDistributeSection (vsectionis.c:1998) n1: 15048472 0x5579E4: DMPlexDistributeNwtun (plexdistribute.c:115) n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 15048472 0x4076DF: main (Nwtun.c:55) n0: 1930096 in 1 place, below massif's threshold (01.00%) n1: 15048472 0x558547: DMPlexDistributeNwtun (plexdistribute.c:154) n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 15048472 0x4076DF: main (Nwtun.c:55) n0: 1930288 in 4 places, all below massif's threshold (01.00%) n3: 33957328 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n2: 16978568 0x53EA67: PetscSFDistributeSection (vsectionis.c:1998) n1: 15048472 0x5579E4: DMPlexDistributeNwtun (plexdistribute.c:115) n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 15048472 0x4076DF: main (Nwtun.c:55) n0: 1930096 in 1 place, below massif's threshold (01.00%) n1: 15048472 0x558547: DMPlexDistributeNwtun (plexdistribute.c:154) n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 15048472 0x4076DF: main (Nwtun.c:55) n0: 1930288 in 4 places, all below massif's threshold (01.00%) n1: 30096944 0x680E6B: PetscSFSetGraph (sf.c:328) n1: 30096944 0x6822D2: PetscSFCreateInverseSF (sf.c:478) n1: 30096944 0x53E180: PetscSFConvertPartition (vsectionis.c:1937) n1: 30096944 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101) n1: 30096944 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 30096944 0x4076DF: main (Nwtun.c:55) n1: 30096944 0x559905: DMPlexDistributeNwtun (plexdistribute.c:247) n1: 30096944 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 30096944 0x4076DF: main (Nwtun.c:55) n2: 15048472 0x681299: PetscSFSetGraph (sf.c:349) n1: 15048472 0x6822D2: PetscSFCreateInverseSF (sf.c:478) n1: 15048472 0x53E180: PetscSFConvertPartition (vsectionis.c:1937) n1: 15048472 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101) n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 15048472 0x4076DF: main (Nwtun.c:55) n0: 0 in 2 places, all below massif's threshold (01.00%) n1: 15048472 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46) n2: 15048472 0x4E2DE5: DMLabelGetStratumIS (plexlabel.c:571) n1: 15048472 0x46481D: DMPlexStratify (plex.c:1768) n1: 15048472 0x55860F: DMPlexDistributeNwtun (plexdistribute.c:157) n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 15048472 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 15048472 0xC88EE6: PetscSFBasicGetPack (sfbasic.c:779) n1: 15048472 0xC8AA7F: PetscSFBcastBegin_Basic (sfbasic.c:914) n2: 15048472 0x687407: PetscSFBcastBegin (sf.c:926) n1: 15048472 0x66B362: ISLocalToGlobalMappingCreateSF (isltog.c:196) n1: 15048472 0x53E29B: PetscSFConvertPartition (vsectionis.c:1942) n1: 15048472 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101) n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 15048472 0x4076DF: main (Nwtun.c:55) n0: 0 in 3 places, all below massif's threshold (01.00%) n2: 15048472 0x6812ED: PetscSFSetGraph (sf.c:349) n1: 15048472 0x6822D2: PetscSFCreateInverseSF (sf.c:478) n1: 15048472 0x53E180: PetscSFConvertPartition (vsectionis.c:1937) n1: 15048472 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101) n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 15048472 0x4076DF: main (Nwtun.c:55) n0: 0 in 2 places, all below massif's threshold (01.00%) n1: 5790288 0xC99701: VecCreate_MPI_Private (pbvec.c:201) n1: 5790288 0xC99C3D: VecCreate_MPI (pbvec.c:244) n1: 5790288 0x6BF560: VecSetType (vecreg.c:53) n1: 5790288 0xC99FA5: VecCreate_Standard (pbvec.c:271) n1: 5790288 0x6BF560: VecSetType (vecreg.c:53) n1: 5790288 0x983E7C: DMPlexDistributeField (plexdistribute.c:301) n1: 5790288 0x558916: DMPlexDistributeNwtun (plexdistribute.c:172) n1: 5790288 0x4154E6: CreateMesh (SetupMeshes.c:34) n0: 5790288 0x4076DF: main (Nwtun.c:55) n0: 2095981 in 125 places, all below massif's threshold (01.00%) n0: 139272 in 92 places, all below massif's threshold (01.00%) #----------- snapshot=21 #----------- time=1688188453621 mem_heap_B=330598157 mem_heap_extra_B=16059 mem_stacks_B=0 heap_tree=empty #----------- snapshot=22 #----------- time=1696654881238 mem_heap_B=423787503 mem_heap_extra_B=28321 mem_stacks_B=0 heap_tree=empty #----------- snapshot=23 #----------- time=1714030650389 mem_heap_B=423787567 mem_heap_extra_B=28337 mem_stacks_B=0 heap_tree=empty #----------- snapshot=24 #----------- time=1728929437932 mem_heap_B=269391227 mem_heap_extra_B=32885 mem_stacks_B=0 heap_tree=empty #----------- snapshot=25 #----------- time=1788668420970 mem_heap_B=269472603 mem_heap_extra_B=32973 mem_stacks_B=0 heap_tree=empty #----------- snapshot=26 #----------- time=1791515716070 mem_heap_B=347795229 mem_heap_extra_B=38819 mem_stacks_B=0 heap_tree=empty #----------- snapshot=27 #----------- time=1832411906731 mem_heap_B=347872613 mem_heap_extra_B=38899 mem_stacks_B=0 heap_tree=detailed n4: 347872613 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n9: 336692827 0x57877E: PetscMallocAlign (mal.c:27) n1: 74765536 0xC9786A: VecCreate_Seq (bvec3.c:38) n1: 74765536 0x6BF560: VecSetType (vecreg.c:53) n1: 74765536 0xC99F47: VecCreate_Standard (pbvec.c:269) n1: 74765536 0x6BF560: VecSetType (vecreg.c:53) n1: 74765536 0xA0773A: DMCreateLocalVector_Section_Private (dmi.c:65) n1: 74765536 0x976C9B: DMCreateLocalVector_Plex (plexcreate.c:931) n2: 74765536 0x50D220: DMCreateLocalVector (dm.c:709) n1: 67129728 0x417EA6: ConstructGeometryFVM (SetupMeshes.c:306) n0: 67129728 0x407898: main (Nwtun.c:60) n1: 7635808 0x4178BF: ConstructGeometryFVM (SetupMeshes.c:273) n0: 7635808 0x407898: main (Nwtun.c:60) n5: 51157200 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n1: 15124576 0x45A810: DMPlexSetChart (plex.c:729) n1: 15124576 0x4B39F0: DMPlexShiftSizes_Internal (plexsubmesh.c:189) n1: 15124576 0x4B8A41: DMPlexConstructGhostCells_Internal (plexsubmesh.c:507) n1: 15124576 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 15124576 0x415800: CreateMesh (SetupMeshes.c:53) n0: 15124576 0x4076DF: main (Nwtun.c:55) n1: 15124576 0x45A874: DMPlexSetChart (plex.c:730) n1: 15124576 0x4B39F0: DMPlexShiftSizes_Internal (plexsubmesh.c:189) n1: 15124576 0x4B8A41: DMPlexConstructGhostCells_Internal (plexsubmesh.c:507) n1: 15124576 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 15124576 0x415800: CreateMesh (SetupMeshes.c:53) n0: 15124576 0x4076DF: main (Nwtun.c:55) n2: 7656952 0x52ED49: PetscSectionSetChart (vsectionis.c:392) n1: 5726856 0x4160EF: SetUpLocalSpace (SetupMeshes.c:112) n0: 5726856 0x40781E: main (Nwtun.c:59) n0: 1930096 in 2 places, all below massif's threshold (01.00%) n0: 7656952 in 9 places, all below massif's threshold (01.00%) n1: 5594144 0x417C82: ConstructGeometryFVM (SetupMeshes.c:292) n0: 5594144 0x407898: main (Nwtun.c:60) n5: 51157200 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n1: 15124576 0x45A810: DMPlexSetChart (plex.c:729) n1: 15124576 0x4B39F0: DMPlexShiftSizes_Internal (plexsubmesh.c:189) n1: 15124576 0x4B8A41: DMPlexConstructGhostCells_Internal (plexsubmesh.c:507) n1: 15124576 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 15124576 0x415800: CreateMesh (SetupMeshes.c:53) n0: 15124576 0x4076DF: main (Nwtun.c:55) n1: 15124576 0x45A874: DMPlexSetChart (plex.c:730) n1: 15124576 0x4B39F0: DMPlexShiftSizes_Internal (plexsubmesh.c:189) n1: 15124576 0x4B8A41: DMPlexConstructGhostCells_Internal (plexsubmesh.c:507) n1: 15124576 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 15124576 0x415800: CreateMesh (SetupMeshes.c:53) n0: 15124576 0x4076DF: main (Nwtun.c:55) n2: 7656952 0x52ED49: PetscSectionSetChart (vsectionis.c:392) n1: 5726856 0x4160EF: SetUpLocalSpace (SetupMeshes.c:112) n0: 5726856 0x40781E: main (Nwtun.c:59) n0: 1930096 in 2 places, all below massif's threshold (01.00%) n0: 7656952 in 9 places, all below massif's threshold (01.00%) n1: 5594144 0x417C82: ConstructGeometryFVM (SetupMeshes.c:292) n0: 5594144 0x407898: main (Nwtun.c:60) n1: 44832536 0x462C3D: DMSetUp_Plex (plex.c:1587) n1: 44832536 0x50B88B: DMSetUp (dm.c:560) n1: 44832536 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534) n1: 44832536 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 44832536 0x415800: CreateMesh (SetupMeshes.c:53) n0: 44832536 0x4076DF: main (Nwtun.c:55) n1: 44832536 0x462A1D: DMSetUp_Plex (plex.c:1582) n2: 44832536 0x50B88B: DMSetUp (dm.c:560) n1: 44832536 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534) n1: 44832536 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 44832536 0x415800: CreateMesh (SetupMeshes.c:53) n0: 44832536 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 44832536 0x462AB1: DMSetUp_Plex (plex.c:1583) n2: 44832536 0x50B88B: DMSetUp (dm.c:560) n1: 44832536 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534) n1: 44832536 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 44832536 0x415800: CreateMesh (SetupMeshes.c:53) n0: 44832536 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 15124576 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46) n2: 15124576 0x4E2DE5: DMLabelGetStratumIS (plexlabel.c:571) n2: 15124576 0x46481D: DMPlexStratify (plex.c:1768) n1: 15124576 0x4B955D: DMPlexConstructGhostCells_Internal (plexsubmesh.c:560) n1: 15124576 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 15124576 0x415800: CreateMesh (SetupMeshes.c:53) n0: 15124576 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 5790288 0xC99701: VecCreate_MPI_Private (pbvec.c:201) n1: 5790288 0xC99C3D: VecCreate_MPI (pbvec.c:244) n1: 5790288 0x6BF560: VecSetType (vecreg.c:53) n1: 5790288 0xC99FA5: VecCreate_Standard (pbvec.c:271) n2: 5790288 0x6BF560: VecSetType (vecreg.c:53) n1: 5790288 0x4B5708: DMPlexShiftCoordinates_Internal (plexsubmesh.c:289) n1: 5790288 0x4B95CF: DMPlexConstructGhostCells_Internal (plexsubmesh.c:561) n1: 5790288 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 5790288 0x415800: CreateMesh (SetupMeshes.c:53) n0: 5790288 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n0: 4200419 in 167 places, all below massif's threshold (01.00%) n1: 7078832 0x4DBEB6: kh_resize_HASHI (hash.h:510) n2: 7078832 0x4DC2CD: kh_put_HASHI (hash.h:510) n2: 7078832 0x4E14DD: DMLabelSetValue (plexlabel.c:450) n1: 3539416 0x4B7611: DMPlexShiftLabels_Internal (plexsubmesh.c:410) n1: 3539416 0x4B96B3: DMPlexConstructGhostCells_Internal (plexsubmesh.c:563) n1: 3539416 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 3539416 0x415800: CreateMesh (SetupMeshes.c:53) n0: 3539416 0x4076DF: main (Nwtun.c:55) n0: 3539416 in 5 places, all below massif's threshold (01.00%) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 3539416 0x4DBE90: kh_resize_HASHI (hash.h:510) n2: 3539416 0x4DC2CD: kh_put_HASHI (hash.h:510) n1: 3539416 0x4E14DD: DMLabelSetValue (plexlabel.c:450) n0: 3539416 in 6 places, all below massif's threshold (01.00%) n0: 0 in 1 place, below massif's threshold (01.00%) n0: 561538 in 90 places, all below massif's threshold (01.00%) #----------- snapshot=28 #----------- time=1842874956987 mem_heap_B=347873909 mem_heap_extra_B=38947 mem_stacks_B=0 heap_tree=empty #----------- snapshot=29 #----------- time=1854113575440 mem_heap_B=347873909 mem_heap_extra_B=38947 mem_stacks_B=0 heap_tree=empty #----------- snapshot=30 #----------- time=1863868911696 mem_heap_B=347873909 mem_heap_extra_B=38947 mem_stacks_B=0 heap_tree=empty #----------- snapshot=31 #----------- time=1937202400181 mem_heap_B=347877637 mem_heap_extra_B=38955 mem_stacks_B=0 heap_tree=empty #----------- snapshot=32 #----------- time=1946196214265 mem_heap_B=480322021 mem_heap_extra_B=81331 mem_stacks_B=0 heap_tree=empty #----------- snapshot=33 #----------- time=1957929625920 mem_heap_B=868356835 mem_heap_extra_B=151861 mem_stacks_B=0 heap_tree=empty #----------- snapshot=34 #----------- time=1973771437326 mem_heap_B=803234327 mem_heap_extra_B=118041 mem_stacks_B=0 heap_tree=empty #----------- snapshot=35 #----------- time=1984387969511 mem_heap_B=1305045055 mem_heap_extra_B=124545 mem_stacks_B=0 heap_tree=empty #----------- snapshot=36 #----------- time=2002831468916 mem_heap_B=1305045055 mem_heap_extra_B=124545 mem_stacks_B=0 heap_tree=detailed n2: 1305045055 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n13: 1294166241 0x57877E: PetscMallocAlign (mal.c:27) n1: 303810000 0x9EC1BF: DMPlexPreallocateOperator (plexpreallocate.c:337) n1: 303810000 0x459619: DMCreateMatrix_Plex (plex.c:615) n1: 303810000 0x50F42F: DMCreateMatrix (dm.c:941) n0: 303810000 0x40B906: main (Nwtun.c:396) n4: 273974304 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n1: 198000728 0x9EB86B: DMPlexPreallocateOperator (plexpreallocate.c:291) n1: 198000728 0x459619: DMCreateMatrix_Plex (plex.c:615) n1: 198000728 0x50F42F: DMCreateMatrix (dm.c:941) n0: 198000728 0x40B906: main (Nwtun.c:396) n0: 45724424 in 15 places, all below massif's threshold (01.00%) n1: 15124576 0x45A810: DMPlexSetChart (plex.c:729) n1: 15124576 0x4B39F0: DMPlexShiftSizes_Internal (plexsubmesh.c:189) n1: 15124576 0x4B8A41: DMPlexConstructGhostCells_Internal (plexsubmesh.c:507) n1: 15124576 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 15124576 0x415800: CreateMesh (SetupMeshes.c:53) n0: 15124576 0x4076DF: main (Nwtun.c:55) n1: 15124576 0x45A874: DMPlexSetChart (plex.c:730) n1: 15124576 0x4B39F0: DMPlexShiftSizes_Internal (plexsubmesh.c:189) n1: 15124576 0x4B8A41: DMPlexConstructGhostCells_Internal (plexsubmesh.c:507) n1: 15124576 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 15124576 0x415800: CreateMesh (SetupMeshes.c:53) n0: 15124576 0x4076DF: main (Nwtun.c:55) n4: 273974304 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n1: 198000728 0x9EB86B: DMPlexPreallocateOperator (plexpreallocate.c:291) n1: 198000728 0x459619: DMCreateMatrix_Plex (plex.c:615) n1: 198000728 0x50F42F: DMCreateMatrix (dm.c:941) n0: 198000728 0x40B906: main (Nwtun.c:396) n0: 45724424 in 15 places, all below massif's threshold (01.00%) n1: 15124576 0x45A810: DMPlexSetChart (plex.c:729) n1: 15124576 0x4B39F0: DMPlexShiftSizes_Internal (plexsubmesh.c:189) n1: 15124576 0x4B8A41: DMPlexConstructGhostCells_Internal (plexsubmesh.c:507) n1: 15124576 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 15124576 0x415800: CreateMesh (SetupMeshes.c:53) n0: 15124576 0x4076DF: main (Nwtun.c:55) n1: 15124576 0x45A874: DMPlexSetChart (plex.c:730) n1: 15124576 0x4B39F0: DMPlexShiftSizes_Internal (plexsubmesh.c:189) n1: 15124576 0x4B8A41: DMPlexConstructGhostCells_Internal (plexsubmesh.c:507) n1: 15124576 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 15124576 0x415800: CreateMesh (SetupMeshes.c:53) n0: 15124576 0x4076DF: main (Nwtun.c:55) n0: 97551865 in 216 places, all below massif's threshold (01.00%) n1: 84310296 0xC9786A: VecCreate_Seq (bvec3.c:38) n1: 84310296 0x6BF560: VecSetType (vecreg.c:53) n1: 84310296 0xC99F47: VecCreate_Standard (pbvec.c:269) n1: 84310296 0x6BF560: VecSetType (vecreg.c:53) n1: 84310296 0xA0773A: DMCreateLocalVector_Section_Private (dmi.c:65) n1: 84310296 0x976C9B: DMCreateLocalVector_Plex (plexcreate.c:931) n2: 84310296 0x50D220: DMCreateLocalVector (dm.c:709) n1: 67129728 0x417EA6: ConstructGeometryFVM (SetupMeshes.c:306) n0: 67129728 0x407898: main (Nwtun.c:60) n0: 17180568 in 2 places, all below massif's threshold (01.00%) n1: 44832536 0x462C3D: DMSetUp_Plex (plex.c:1587) n1: 44832536 0x50B88B: DMSetUp (dm.c:560) n1: 44832536 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534) n1: 44832536 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 44832536 0x415800: CreateMesh (SetupMeshes.c:53) n0: 44832536 0x4076DF: main (Nwtun.c:55) n1: 44832536 0x462AB1: DMSetUp_Plex (plex.c:1583) n2: 44832536 0x50B88B: DMSetUp (dm.c:560) n1: 44832536 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534) n1: 44832536 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 44832536 0x415800: CreateMesh (SetupMeshes.c:53) n0: 44832536 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 44832536 0x462A1D: DMSetUp_Plex (plex.c:1582) n2: 44832536 0x50B88B: DMSetUp (dm.c:560) n1: 44832536 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534) n1: 44832536 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 44832536 0x415800: CreateMesh (SetupMeshes.c:53) n0: 44832536 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n2: 40721328 0xC99701: VecCreate_MPI_Private (pbvec.c:201) n1: 26198280 0xC98DCF: VecDuplicate_MPI (pbvec.c:68) n1: 26198280 0x6B04CF: VecDuplicate (vector.c:399) n0: 26198280 in 3 places, all below massif's threshold (01.00%) n1: 14523048 0xC99C3D: VecCreate_MPI (pbvec.c:244) n1: 14523048 0x6BF560: VecSetType (vecreg.c:53) n1: 14523048 0xC99FA5: VecCreate_Standard (pbvec.c:271) n1: 14523048 0x6BF560: VecSetType (vecreg.c:53) n0: 14523048 in 3 places, all below massif's threshold (01.00%) n1: 26903800 0x9EA61B: DMPlexPreallocateOperator (plexpreallocate.c:204) n1: 26903800 0x459619: DMCreateMatrix_Plex (plex.c:615) n1: 26903800 0x50F42F: DMCreateMatrix (dm.c:941) n0: 26903800 0x40B906: main (Nwtun.c:396) n1: 24893576 0x9832FA: DMPlexGetAdjacency_Internal (plexdistribute.c:220) n1: 24893576 0x9839A6: DMPlexGetAdjacency (plexdistribute.c:263) n1: 24893576 0x9E917B: DMPlexPreallocateOperator (plexpreallocate.c:101) n1: 24893576 0x459619: DMCreateMatrix_Plex (plex.c:615) n1: 24893576 0x50F42F: DMCreateMatrix (dm.c:941) n0: 24893576 0x40B906: main (Nwtun.c:396) n1: 18328480 0x521E45: DMCreateDefaultSF (dm.c:3075) n1: 18328480 0x520D8A: DMGetDefaultSF (dm.c:2991) n1: 18328480 0x5153DB: DMGlobalToLocalBegin (dm.c:1649) n1: 18328480 0x411EF2: SetInitialCondition (Nwtun.c:938) n0: 18328480 0x40B823: main (Nwtun.c:385) n2: 15200680 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46) n2: 15124576 0x4E2DE5: DMLabelGetStratumIS (plexlabel.c:571) n2: 15124576 0x46481D: DMPlexStratify (plex.c:1768) n1: 15124576 0x4B955D: DMPlexConstructGhostCells_Internal (plexsubmesh.c:560) n1: 15124576 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 15124576 0x415800: CreateMesh (SetupMeshes.c:53) n0: 15124576 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n0: 0 in 1 place, below massif's threshold (01.00%) n0: 76104 in 1 place, below massif's threshold (01.00%) n0: 10878814 in 95 places, all below massif's threshold (01.00%) #----------- snapshot=37 #----------- time=2031323634051 mem_heap_B=1215067359 mem_heap_extra_B=108017 mem_stacks_B=0 heap_tree=empty #----------- snapshot=38 #----------- time=2035978518347 mem_heap_B=1875093065 mem_heap_extra_B=139567 mem_stacks_B=0 heap_tree=empty #----------- snapshot=39 #----------- time=2053498338602 mem_heap_B=1875093065 mem_heap_extra_B=139567 mem_stacks_B=0 heap_tree=detailed n2: 1875093065 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n11: 1864214251 0x57877E: PetscMallocAlign (mal.c:27) n1: 303810000 0x81EE1C: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719) n2: 303810000 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664) n1: 301101800 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255) n1: 301101800 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959) n1: 301101800 0x955310: MatXAIJSetPreallocation (gcreate.c:252) n1: 301101800 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435) n1: 301101800 0x459619: DMCreateMatrix_Plex (plex.c:615) n1: 301101800 0x50F42F: DMCreateMatrix (dm.c:941) n0: 301101800 0x40B906: main (Nwtun.c:396) n0: 2708200 in 1 place, below massif's threshold (01.00%) n1: 303810000 0x81EDCA: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719) n2: 303810000 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664) n1: 301101800 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255) n1: 301101800 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959) n1: 301101800 0x955310: MatXAIJSetPreallocation (gcreate.c:252) n1: 301101800 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435) n1: 301101800 0x459619: DMCreateMatrix_Plex (plex.c:615) n1: 301101800 0x50F42F: DMCreateMatrix (dm.c:941) n0: 301101800 0x40B906: main (Nwtun.c:396) n0: 2708200 in 1 place, below massif's threshold (01.00%) n1: 303810000 0x9EC1BF: DMPlexPreallocateOperator (plexpreallocate.c:337) n1: 303810000 0x459619: DMCreateMatrix_Plex (plex.c:615) n1: 303810000 0x50F42F: DMCreateMatrix (dm.c:941) n0: 303810000 0x40B906: main (Nwtun.c:396) n2: 254884784 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n1: 198000728 0x9EB86B: DMPlexPreallocateOperator (plexpreallocate.c:291) n1: 198000728 0x459619: DMCreateMatrix_Plex (plex.c:615) n1: 198000728 0x50F42F: DMCreateMatrix (dm.c:941) n0: 198000728 0x40B906: main (Nwtun.c:396) n0: 56884056 in 17 places, all below massif's threshold (01.00%) n2: 254884784 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n1: 198000728 0x9EB86B: DMPlexPreallocateOperator (plexpreallocate.c:291) n1: 198000728 0x459619: DMCreateMatrix_Plex (plex.c:615) n1: 198000728 0x50F42F: DMCreateMatrix (dm.c:941) n0: 198000728 0x40B906: main (Nwtun.c:396) n0: 56884056 in 17 places, all below massif's threshold (01.00%) n0: 183485451 in 225 places, all below massif's threshold (01.00%) n1: 84310296 0xC9786A: VecCreate_Seq (bvec3.c:38) n1: 84310296 0x6BF560: VecSetType (vecreg.c:53) n1: 84310296 0xC99F47: VecCreate_Standard (pbvec.c:269) n1: 84310296 0x6BF560: VecSetType (vecreg.c:53) n1: 84310296 0xA0773A: DMCreateLocalVector_Section_Private (dmi.c:65) n1: 84310296 0x976C9B: DMCreateLocalVector_Plex (plexcreate.c:931) n2: 84310296 0x50D220: DMCreateLocalVector (dm.c:709) n1: 67129728 0x417EA6: ConstructGeometryFVM (SetupMeshes.c:306) n0: 67129728 0x407898: main (Nwtun.c:60) n0: 17180568 in 2 places, all below massif's threshold (01.00%) n1: 44832536 0x462AB1: DMSetUp_Plex (plex.c:1583) n2: 44832536 0x50B88B: DMSetUp (dm.c:560) n1: 44832536 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534) n1: 44832536 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 44832536 0x415800: CreateMesh (SetupMeshes.c:53) n0: 44832536 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 44832536 0x462A1D: DMSetUp_Plex (plex.c:1582) n2: 44832536 0x50B88B: DMSetUp (dm.c:560) n1: 44832536 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534) n1: 44832536 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 44832536 0x415800: CreateMesh (SetupMeshes.c:53) n0: 44832536 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 44832536 0x462C3D: DMSetUp_Plex (plex.c:1587) n1: 44832536 0x50B88B: DMSetUp (dm.c:560) n1: 44832536 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534) n1: 44832536 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 44832536 0x415800: CreateMesh (SetupMeshes.c:53) n0: 44832536 0x4076DF: main (Nwtun.c:55) n2: 40721328 0xC99701: VecCreate_MPI_Private (pbvec.c:201) n1: 26198280 0xC98DCF: VecDuplicate_MPI (pbvec.c:68) n1: 26198280 0x6B04CF: VecDuplicate (vector.c:399) n0: 26198280 in 3 places, all below massif's threshold (01.00%) n0: 14523048 in 1 place, below massif's threshold (01.00%) n0: 10878814 in 95 places, all below massif's threshold (01.00%) #----------- snapshot=40 #----------- time=2068776186221 mem_heap_B=1919188393 mem_heap_extra_B=155015 mem_stacks_B=0 heap_tree=detailed n2: 1919188393 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n11: 1908309579 0x57877E: PetscMallocAlign (mal.c:27) n1: 303810000 0x81EE1C: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719) n2: 303810000 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664) n1: 301101800 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255) n1: 301101800 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959) n1: 301101800 0x955310: MatXAIJSetPreallocation (gcreate.c:252) n1: 301101800 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435) n1: 301101800 0x459619: DMCreateMatrix_Plex (plex.c:615) n1: 301101800 0x50F42F: DMCreateMatrix (dm.c:941) n0: 301101800 0x40B906: main (Nwtun.c:396) n0: 2708200 in 1 place, below massif's threshold (01.00%) n1: 303810000 0x81EDCA: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719) n2: 303810000 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664) n1: 301101800 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255) n1: 301101800 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959) n1: 301101800 0x955310: MatXAIJSetPreallocation (gcreate.c:252) n1: 301101800 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435) n1: 301101800 0x459619: DMCreateMatrix_Plex (plex.c:615) n1: 301101800 0x50F42F: DMCreateMatrix (dm.c:941) n0: 301101800 0x40B906: main (Nwtun.c:396) n0: 2708200 in 1 place, below massif's threshold (01.00%) n1: 303810000 0x9EC1BF: DMPlexPreallocateOperator (plexpreallocate.c:337) n1: 303810000 0x459619: DMCreateMatrix_Plex (plex.c:615) n1: 303810000 0x50F42F: DMCreateMatrix (dm.c:941) n0: 303810000 0x40B906: main (Nwtun.c:396) n2: 254884784 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n1: 198000728 0x9EB86B: DMPlexPreallocateOperator (plexpreallocate.c:291) n1: 198000728 0x459619: DMCreateMatrix_Plex (plex.c:615) n1: 198000728 0x50F42F: DMCreateMatrix (dm.c:941) n0: 198000728 0x40B906: main (Nwtun.c:396) n0: 56884056 in 17 places, all below massif's threshold (01.00%) n2: 254884784 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n1: 198000728 0x9EB86B: DMPlexPreallocateOperator (plexpreallocate.c:291) n1: 198000728 0x459619: DMCreateMatrix_Plex (plex.c:615) n1: 198000728 0x50F42F: DMCreateMatrix (dm.c:941) n0: 198000728 0x40B906: main (Nwtun.c:396) n0: 56884056 in 17 places, all below massif's threshold (01.00%) n0: 227580779 in 236 places, all below massif's threshold (01.00%) n1: 84310296 0xC9786A: VecCreate_Seq (bvec3.c:38) n1: 84310296 0x6BF560: VecSetType (vecreg.c:53) n1: 84310296 0xC99F47: VecCreate_Standard (pbvec.c:269) n1: 84310296 0x6BF560: VecSetType (vecreg.c:53) n1: 84310296 0xA0773A: DMCreateLocalVector_Section_Private (dmi.c:65) n1: 84310296 0x976C9B: DMCreateLocalVector_Plex (plexcreate.c:931) n2: 84310296 0x50D220: DMCreateLocalVector (dm.c:709) n1: 67129728 0x417EA6: ConstructGeometryFVM (SetupMeshes.c:306) n0: 67129728 0x407898: main (Nwtun.c:60) n0: 17180568 in 2 places, all below massif's threshold (01.00%) n1: 44832536 0x462AB1: DMSetUp_Plex (plex.c:1583) n2: 44832536 0x50B88B: DMSetUp (dm.c:560) n1: 44832536 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534) n1: 44832536 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 44832536 0x415800: CreateMesh (SetupMeshes.c:53) n0: 44832536 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 44832536 0x462A1D: DMSetUp_Plex (plex.c:1582) n2: 44832536 0x50B88B: DMSetUp (dm.c:560) n1: 44832536 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534) n1: 44832536 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 44832536 0x415800: CreateMesh (SetupMeshes.c:53) n0: 44832536 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 44832536 0x462C3D: DMSetUp_Plex (plex.c:1587) n1: 44832536 0x50B88B: DMSetUp (dm.c:560) n1: 44832536 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534) n1: 44832536 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 44832536 0x415800: CreateMesh (SetupMeshes.c:53) n0: 44832536 0x4076DF: main (Nwtun.c:55) n2: 40721328 0xC99701: VecCreate_MPI_Private (pbvec.c:201) n1: 26198280 0xC98DCF: VecDuplicate_MPI (pbvec.c:68) n1: 26198280 0x6B04CF: VecDuplicate (vector.c:399) n0: 26198280 in 3 places, all below massif's threshold (01.00%) n0: 14523048 in 1 place, below massif's threshold (01.00%) n0: 10878814 in 95 places, all below massif's threshold (01.00%) #----------- snapshot=41 #----------- time=2085612748660 mem_heap_B=1254369333 mem_heap_extra_B=150667 mem_stacks_B=0 heap_tree=empty #----------- snapshot=42 #----------- time=2099570407105 mem_heap_B=1281711289 mem_heap_extra_B=151679 mem_stacks_B=0 heap_tree=empty #----------- snapshot=43 #----------- time=2111798447020 mem_heap_B=1310347781 mem_heap_extra_B=152587 mem_stacks_B=0 heap_tree=empty #----------- snapshot=44 #----------- time=2130810038150 mem_heap_B=1310347885 mem_heap_extra_B=152603 mem_stacks_B=0 heap_tree=empty #----------- snapshot=45 #----------- time=2137035244977 mem_heap_B=1540712799 mem_heap_extra_B=165705 mem_stacks_B=0 heap_tree=empty #----------- snapshot=46 #----------- time=2189002748743 mem_heap_B=1540710923 mem_heap_extra_B=165669 mem_stacks_B=0 heap_tree=empty #----------- snapshot=47 #----------- time=2190640466102 mem_heap_B=1540711699 mem_heap_extra_B=165653 mem_stacks_B=0 heap_tree=empty #----------- snapshot=48 #----------- time=2255398115514 mem_heap_B=1540713443 mem_heap_extra_B=165685 mem_stacks_B=0 heap_tree=empty #----------- snapshot=49 #----------- time=2256803909597 mem_heap_B=1645503790 mem_heap_extra_B=167378 mem_stacks_B=0 heap_tree=empty #----------- snapshot=50 #----------- time=2256808280096 mem_heap_B=1645503766 mem_heap_extra_B=167362 mem_stacks_B=0 heap_tree=empty #----------- snapshot=51 #----------- time=2256811949338 mem_heap_B=1645502870 mem_heap_extra_B=167330 mem_stacks_B=0 heap_tree=empty #----------- snapshot=52 #----------- time=2256814355812 mem_heap_B=1645503766 mem_heap_extra_B=167362 mem_stacks_B=0 heap_tree=empty #----------- snapshot=53 #----------- time=2256824000668 mem_heap_B=1645502022 mem_heap_extra_B=167330 mem_stacks_B=0 heap_tree=empty #----------- snapshot=54 #----------- time=2256826285854 mem_heap_B=1645502894 mem_heap_extra_B=167346 mem_stacks_B=0 heap_tree=detailed n2: 1645502894 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n22: 1634623016 0x57877E: PetscMallocAlign (mal.c:27) n1: 303810000 0x81EE1C: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719) n2: 303810000 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664) n1: 301101800 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255) n1: 301101800 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959) n1: 301101800 0x955310: MatXAIJSetPreallocation (gcreate.c:252) n1: 301101800 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435) n1: 301101800 0x459619: DMCreateMatrix_Plex (plex.c:615) n1: 301101800 0x50F42F: DMCreateMatrix (dm.c:941) n0: 301101800 0x40B906: main (Nwtun.c:396) n0: 2708200 in 1 place, below massif's threshold (01.00%) n1: 303810000 0x81EDCA: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719) n2: 303810000 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664) n1: 301101800 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255) n1: 301101800 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959) n1: 301101800 0x955310: MatXAIJSetPreallocation (gcreate.c:252) n1: 301101800 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435) n1: 301101800 0x459619: DMCreateMatrix_Plex (plex.c:615) n1: 301101800 0x50F42F: DMCreateMatrix (dm.c:941) n0: 301101800 0x40B906: main (Nwtun.c:396) n0: 2708200 in 1 place, below massif's threshold (01.00%) n1: 151555096 0xC9786A: VecCreate_Seq (bvec3.c:38) n2: 151555096 0x6BF560: VecSetType (vecreg.c:53) n1: 151123616 0xC99F47: VecCreate_Standard (pbvec.c:269) n1: 151123616 0x6BF560: VecSetType (vecreg.c:53) n1: 151123616 0xA0773A: DMCreateLocalVector_Section_Private (dmi.c:65) n1: 151123616 0x976C9B: DMCreateLocalVector_Plex (plexcreate.c:931) n3: 151123616 0x50D220: DMCreateLocalVector (dm.c:709) n3: 76358080 0xA035E0: DMGetLocalVector (dmget.c:49) n1: 28634280 0x425614: CaculateLocalFunction_LS (SetupFunctions.c:742) n1: 28634280 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 28634280 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 28634280 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 28634280 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 28634280 0xA9E3CC: SNESSolve (snes.c:3743) n1: 28634280 0x40FB9B: SolveTimeDependent (Nwtun.c:758) n0: 28634280 0x40BC62: main (Nwtun.c:417) n1: 28634280 0x4257D8: CaculateLocalFunction_LS (SetupFunctions.c:748) n1: 28634280 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 28634280 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 28634280 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 28634280 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 28634280 0xA9E3CC: SNESSolve (snes.c:3743) n1: 28634280 0x40FB9B: SolveTimeDependent (Nwtun.c:758) n0: 28634280 0x40BC62: main (Nwtun.c:417) n0: 19089520 in 2 places, all below massif's threshold (01.00%) n1: 67129728 0x417EA6: ConstructGeometryFVM (SetupMeshes.c:306) n0: 67129728 0x407898: main (Nwtun.c:60) n0: 7635808 in 1 place, below massif's threshold (01.00%) n0: 431480 in 1 place, below massif's threshold (01.00%) n2: 111724968 0xC99701: VecCreate_MPI_Private (pbvec.c:201) n1: 61129320 0xC98DCF: VecDuplicate_MPI (pbvec.c:68) n2: 61129320 0x6B04CF: VecDuplicate (vector.c:399) n0: 43663800 in 6 places, all below massif's threshold (01.00%) n1: 17465520 0x6B434F: VecDuplicateVecs_Default (vector.c:827) n1: 17465520 0x6B11C4: VecDuplicateVecs (vector.c:473) n1: 17465520 0xAAFB50: SNESSetWorkVecs (snesut.c:562) n1: 17465520 0xAF8B57: SNESSetUp_NEWTONLS (ls.c:317) n1: 17465520 0xA9489F: SNESSetUp (snes.c:2616) n1: 17465520 0xA9E184: SNESSolve (snes.c:3731) n1: 17465520 0x40FB9B: SolveTimeDependent (Nwtun.c:758) n0: 17465520 0x40BC62: main (Nwtun.c:417) n1: 50595648 0xC99C3D: VecCreate_MPI (pbvec.c:244) n1: 50595648 0x6BF560: VecSetType (vecreg.c:53) n1: 50595648 0xC99FA5: VecCreate_Standard (pbvec.c:271) n2: 50595648 0x6BF560: VecSetType (vecreg.c:53) n1: 44805360 0xA07131: DMCreateGlobalVector_Section_Private (dmi.c:34) n1: 44805360 0x976932: DMCreateGlobalVector_Plex (plexcreate.c:917) n2: 44805360 0x50CE38: DMCreateGlobalVector (dm.c:681) n2: 36072600 0xA04005: DMGetGlobalVector (dmget.c:154) n1: 27339840 0x426EC6: ConstructCellCentriodGradient (SetupFunctions.c:855) n1: 27339840 0x42054B: FormTimeStepFunction (SetupFunctions.c:233) n1: 27339840 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 27339840 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 27339840 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 27339840 0xA9E3CC: SNESSolve (snes.c:3743) n1: 27339840 0x40FB9B: SolveTimeDependent (Nwtun.c:758) n0: 27339840 0x40BC62: main (Nwtun.c:417) n0: 8732760 in 1 place, below massif's threshold (01.00%) n0: 8732760 in 1 place, below massif's threshold (01.00%) n0: 5790288 in 2 places, all below massif's threshold (01.00%) n1: 104790400 0x905DD7: MatGetSubMatrix_MPIAIJ_All (mpiov.c:549) n1: 104790400 0x8DE981: MatGetSeqNonzeroStructure_MPIAIJ (mpiaij.c:3010) n1: 104790400 0x78FDBF: MatGetSeqNonzeroStructure (matrix.c:6487) n1: 104790400 0xCA31E2: MatColoringApply_SL (color.c:78) n1: 104790400 0x746C6C: MatColoringApply (matcoloring.c:379) n1: 104790400 0xAB0879: SNESComputeJacobianDefaultColor (snesj2.c:71) n1: 104790400 0x4349AD: FormJacobian (SetupJacobian.c:58) n1: 104790400 0xA8F850: SNESComputeJacobian (snes.c:2193) n1: 104790400 0xAF7C15: SNESSolve_NEWTONLS (ls.c:230) n1: 104790400 0xA9E3CC: SNESSolve (snes.c:3743) n1: 104790400 0x40FB9B: SolveTimeDependent (Nwtun.c:758) n0: 104790400 0x40BC62: main (Nwtun.c:417) n1: 75597040 0x521E45: DMCreateDefaultSF (dm.c:3075) n1: 75597040 0x520D8A: DMGetDefaultSF (dm.c:2991) n2: 75597040 0x5153DB: DMGlobalToLocalBegin (dm.c:1649) n1: 57268560 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 57268560 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 57268560 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 57268560 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 57268560 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 57268560 0xA9E3CC: SNESSolve (snes.c:3743) n1: 57268560 0x40FB9B: SolveTimeDependent (Nwtun.c:758) n0: 57268560 0x40BC62: main (Nwtun.c:417) n1: 18328480 0x411EF2: SetInitialCondition (Nwtun.c:938) n0: 18328480 0x40B823: main (Nwtun.c:385) n1: 58793008 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n0: 58793008 in 18 places, all below massif's threshold (01.00%) n1: 58793008 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n0: 58793008 in 18 places, all below massif's threshold (01.00%) n1: 44832536 0x462A1D: DMSetUp_Plex (plex.c:1582) n2: 44832536 0x50B88B: DMSetUp (dm.c:560) n1: 44832536 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534) n1: 44832536 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 44832536 0x415800: CreateMesh (SetupMeshes.c:53) n0: 44832536 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 44832536 0x462C3D: DMSetUp_Plex (plex.c:1587) n1: 44832536 0x50B88B: DMSetUp (dm.c:560) n1: 44832536 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534) n1: 44832536 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 44832536 0x415800: CreateMesh (SetupMeshes.c:53) n0: 44832536 0x4076DF: main (Nwtun.c:55) n1: 44832536 0x462AB1: DMSetUp_Plex (plex.c:1583) n2: 44832536 0x50B88B: DMSetUp (dm.c:560) n1: 44832536 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534) n1: 44832536 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618) n1: 44832536 0x415800: CreateMesh (SetupMeshes.c:53) n0: 44832536 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 38949904 0xC861ED: PetscSFSetUp_Basic (sfbasic.c:557) n2: 38949904 0x67F6D7: PetscSFSetUp (sf.c:192) n2: 38949904 0x68739A: PetscSFBcastBegin (sf.c:925) n2: 37794200 0x51554F: DMGlobalToLocalBegin (dm.c:1656) n1: 28631040 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 28631040 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 28631040 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 28631040 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 28631040 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 28631040 0xA9E3CC: SNESSolve (snes.c:3743) n1: 28631040 0x40FB9B: SolveTimeDependent (Nwtun.c:758) n0: 28631040 0x40BC62: main (Nwtun.c:417) n0: 9163160 in 1 place, below massif's threshold (01.00%) n0: 1155704 in 3 places, all below massif's threshold (01.00%) n0: 0 in 2 places, all below massif's threshold (01.00%) n2: 38949904 0xC88E56: PetscSFBasicGetPack (sfbasic.c:779) n1: 38949904 0xC8AA7F: PetscSFBcastBegin_Basic (sfbasic.c:914) n2: 38949904 0x687407: PetscSFBcastBegin (sf.c:926) n2: 37794200 0x51554F: DMGlobalToLocalBegin (dm.c:1656) n1: 28631040 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 28631040 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 28631040 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 28631040 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 28631040 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 28631040 0xA9E3CC: SNESSolve (snes.c:3743) n1: 28631040 0x40FB9B: SolveTimeDependent (Nwtun.c:758) n0: 28631040 0x40BC62: main (Nwtun.c:417) n0: 9163160 in 1 place, below massif's threshold (01.00%) n0: 1155704 in 3 places, all below massif's threshold (01.00%) n0: 0 in 1 place, below massif's threshold (01.00%) n2: 38507800 0xC88EE6: PetscSFBasicGetPack (sfbasic.c:779) n1: 38507800 0xC8AA7F: PetscSFBcastBegin_Basic (sfbasic.c:914) n2: 38507800 0x687407: PetscSFBcastBegin (sf.c:926) n2: 37798520 0x51554F: DMGlobalToLocalBegin (dm.c:1656) n1: 28634280 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 28634280 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 28634280 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 28634280 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 28634280 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 28634280 0xA9E3CC: SNESSolve (snes.c:3743) n1: 28634280 0x40FB9B: SolveTimeDependent (Nwtun.c:758) n0: 28634280 0x40BC62: main (Nwtun.c:417) n0: 9164240 in 1 place, below massif's threshold (01.00%) n0: 709280 in 8 places, all below massif's threshold (01.00%) n0: 0 in 1 place, below massif's threshold (01.00%) n2: 38507800 0x6812ED: PetscSFSetGraph (sf.c:349) n1: 37798520 0x522622: DMCreateDefaultSF (dm.c:3116) n1: 37798520 0x520D8A: DMGetDefaultSF (dm.c:2991) n2: 37798520 0x5153DB: DMGlobalToLocalBegin (dm.c:1649) n1: 28634280 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 28634280 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 28634280 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 28634280 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 28634280 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 28634280 0xA9E3CC: SNESSolve (snes.c:3743) n1: 28634280 0x40FB9B: SolveTimeDependent (Nwtun.c:758) n0: 28634280 0x40BC62: main (Nwtun.c:417) n0: 9164240 in 1 place, below massif's threshold (01.00%) n0: 709280 in 7 places, all below massif's threshold (01.00%) n2: 38507800 0x681299: PetscSFSetGraph (sf.c:349) n1: 37798520 0x522622: DMCreateDefaultSF (dm.c:3116) n1: 37798520 0x520D8A: DMGetDefaultSF (dm.c:2991) n2: 37798520 0x5153DB: DMGlobalToLocalBegin (dm.c:1649) n1: 28634280 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 28634280 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 28634280 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 28634280 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 28634280 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 28634280 0xA9E3CC: SNESSolve (snes.c:3743) n1: 28634280 0x40FB9B: SolveTimeDependent (Nwtun.c:758) n0: 28634280 0x40BC62: main (Nwtun.c:417) n0: 9164240 in 1 place, below massif's threshold (01.00%) n0: 709280 in 7 places, all below massif's threshold (01.00%) n1: 37798520 0x521DAD: DMCreateDefaultSF (dm.c:3074) n1: 37798520 0x520D8A: DMGetDefaultSF (dm.c:2991) n2: 37798520 0x5153DB: DMGlobalToLocalBegin (dm.c:1649) n1: 28634280 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743) n1: 28634280 0x420744: FormTimeStepFunction (SetupFunctions.c:247) n1: 28634280 0x41E3AB: FormFunction (SetupFunctions.c:52) n1: 28634280 0xA8D3FC: SNESComputeFunction (snes.c:2033) n1: 28634280 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174) n1: 28634280 0xA9E3CC: SNESSolve (snes.c:3743) n1: 28634280 0x40FB9B: SolveTimeDependent (Nwtun.c:758) n0: 28634280 0x40BC62: main (Nwtun.c:417) n0: 9164240 in 1 place, below massif's threshold (01.00%) n0: 30168064 in 290 places, all below massif's threshold (01.00%) n1: 17465536 0x81EE87: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719) n1: 17465536 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664) n0: 17465536 in 2 places, all below massif's threshold (01.00%) n1: 17465520 0x809EF6: MatMarkDiagonal_SeqAIJ (aij.c:1654) n1: 17465520 0x80490C: MatAssemblyEnd_SeqAIJ (aij.c:1018) n1: 17465520 0x7832CB: MatAssemblyEnd (matrix.c:4905) n0: 17465520 in 2 places, all below massif's threshold (01.00%) n1: 17465520 0x81EA9A: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3702) n1: 17465520 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664) n0: 17465520 in 2 places, all below massif's threshold (01.00%) n1: 17465520 0x81EAFA: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3702) n1: 17465520 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664) n0: 17465520 in 2 places, all below massif's threshold (01.00%) n0: 10879878 in 95 places, all below massif's threshold (01.00%) #----------- snapshot=55 #----------- time=2256829364156 mem_heap_B=1645503742 mem_heap_extra_B=167346 mem_stacks_B=0 heap_tree=empty #----------- snapshot=56 #----------- time=2256831333111 mem_heap_B=1645503254 mem_heap_extra_B=167330 mem_stacks_B=0 heap_tree=empty #----------- snapshot=57 #----------- time=2257106618788 mem_heap_B=1750296811 mem_heap_extra_B=169525 mem_stacks_B=0 heap_tree=empty #----------- snapshot=58 #----------- time=2257538879453 mem_heap_B=5501562411 mem_heap_extra_B=172165 mem_stacks_B=0 heap_tree=empty #----------- snapshot=59 #----------- time=2257905647280 mem_heap_B=9252828027 mem_heap_extra_B=174813 mem_stacks_B=0 heap_tree=empty #----------- snapshot=60 #----------- time=2259161343696 mem_heap_B=9252828097 mem_heap_extra_B=174831 mem_stacks_B=0 heap_tree=empty #----------- snapshot=61 #----------- time=2259161345536 mem_heap_B=9252828097 mem_heap_extra_B=174831 mem_stacks_B=0 heap_tree=peak n2: 9252828097 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n9: 9241949045 0x57877E: PetscMallocAlign (mal.c:27) n1: 3950285200 0x81EE1C: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719) n3: 3950285200 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664) n1: 3646475200 0x906416: MatGetSubMatrix_MPIAIJ_All (mpiov.c:571) n1: 3646475200 0x8DE981: MatGetSeqNonzeroStructure_MPIAIJ (mpiaij.c:3010) n1: 3646475200 0x78FDBF: MatGetSeqNonzeroStructure (matrix.c:6487) n1: 3646475200 0xCA31E2: MatColoringApply_SL (color.c:78) n1: 3646475200 0x746C6C: MatColoringApply (matcoloring.c:379) n1: 3646475200 0xAB0879: SNESComputeJacobianDefaultColor (snesj2.c:71) n1: 3646475200 0x4349AD: FormJacobian (SetupJacobian.c:58) n1: 3646475200 0xA8F850: SNESComputeJacobian (snes.c:2193) n1: 3646475200 0xAF7C15: SNESSolve_NEWTONLS (ls.c:230) n1: 3646475200 0xA9E3CC: SNESSolve (snes.c:3743) n1: 3646475200 0x40FB9B: SolveTimeDependent (Nwtun.c:758) n0: 3646475200 0x40BC62: main (Nwtun.c:417) n1: 301101800 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255) n1: 301101800 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959) n1: 301101800 0x955310: MatXAIJSetPreallocation (gcreate.c:252) n1: 301101800 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435) n1: 301101800 0x459619: DMCreateMatrix_Plex (plex.c:615) n1: 301101800 0x50F42F: DMCreateMatrix (dm.c:941) n0: 301101800 0x40B906: main (Nwtun.c:396) n0: 2708200 in 1 place, below massif's threshold (01.00%) n1: 3950285200 0x81EDCA: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719) n3: 3950285200 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664) n1: 3646475200 0x906416: MatGetSubMatrix_MPIAIJ_All (mpiov.c:571) n1: 3646475200 0x8DE981: MatGetSeqNonzeroStructure_MPIAIJ (mpiaij.c:3010) n1: 3646475200 0x78FDBF: MatGetSeqNonzeroStructure (matrix.c:6487) n1: 3646475200 0xCA31E2: MatColoringApply_SL (color.c:78) n1: 3646475200 0x746C6C: MatColoringApply (matcoloring.c:379) n1: 3646475200 0xAB0879: SNESComputeJacobianDefaultColor (snesj2.c:71) n1: 3646475200 0x4349AD: FormJacobian (SetupJacobian.c:58) n1: 3646475200 0xA8F850: SNESComputeJacobian (snes.c:2193) n1: 3646475200 0xAF7C15: SNESSolve_NEWTONLS (ls.c:230) n1: 3646475200 0xA9E3CC: SNESSolve (snes.c:3743) n1: 3646475200 0x40FB9B: SolveTimeDependent (Nwtun.c:758) n0: 3646475200 0x40BC62: main (Nwtun.c:417) n1: 301101800 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255) n1: 301101800 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959) n1: 301101800 0x955310: MatXAIJSetPreallocation (gcreate.c:252) n1: 301101800 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435) n1: 301101800 0x459619: DMCreateMatrix_Plex (plex.c:615) n1: 301101800 0x50F42F: DMCreateMatrix (dm.c:941) n0: 301101800 0x40B906: main (Nwtun.c:396) n0: 2708200 in 1 place, below massif's threshold (01.00%) n0: 606540397 in 304 places, all below massif's threshold (01.00%) n1: 151555096 0xC9786A: VecCreate_Seq (bvec3.c:38) n2: 151555096 0x6BF560: VecSetType (vecreg.c:53) n1: 151123616 0xC99F47: VecCreate_Standard (pbvec.c:269) n1: 151123616 0x6BF560: VecSetType (vecreg.c:53) n1: 151123616 0xA0773A: DMCreateLocalVector_Section_Private (dmi.c:65) n1: 151123616 0x976C9B: DMCreateLocalVector_Plex (plexcreate.c:931) n1: 151123616 0x50D220: DMCreateLocalVector (dm.c:709) n0: 151123616 in 3 places, all below massif's threshold (01.00%) n0: 431480 in 1 place, below massif's threshold (01.00%) n1: 122255944 0x81EE87: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719) n2: 122255944 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664) n1: 104790408 0x906416: MatGetSubMatrix_MPIAIJ_All (mpiov.c:571) n1: 104790408 0x8DE981: MatGetSeqNonzeroStructure_MPIAIJ (mpiaij.c:3010) n1: 104790408 0x78FDBF: MatGetSeqNonzeroStructure (matrix.c:6487) n1: 104790408 0xCA31E2: MatColoringApply_SL (color.c:78) n1: 104790408 0x746C6C: MatColoringApply (matcoloring.c:379) n1: 104790408 0xAB0879: SNESComputeJacobianDefaultColor (snesj2.c:71) n1: 104790408 0x4349AD: FormJacobian (SetupJacobian.c:58) n1: 104790408 0xA8F850: SNESComputeJacobian (snes.c:2193) n1: 104790408 0xAF7C15: SNESSolve_NEWTONLS (ls.c:230) n1: 104790408 0xA9E3CC: SNESSolve (snes.c:3743) n1: 104790408 0x40FB9B: SolveTimeDependent (Nwtun.c:758) n0: 104790408 0x40BC62: main (Nwtun.c:417) n0: 17465536 in 2 places, all below massif's threshold (01.00%) n1: 122255920 0x81EAFA: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3702) n2: 122255920 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664) n1: 104790400 0x906416: MatGetSubMatrix_MPIAIJ_All (mpiov.c:571) n1: 104790400 0x8DE981: MatGetSeqNonzeroStructure_MPIAIJ (mpiaij.c:3010) n1: 104790400 0x78FDBF: MatGetSeqNonzeroStructure (matrix.c:6487) n1: 104790400 0xCA31E2: MatColoringApply_SL (color.c:78) n1: 104790400 0x746C6C: MatColoringApply (matcoloring.c:379) n1: 104790400 0xAB0879: SNESComputeJacobianDefaultColor (snesj2.c:71) n1: 104790400 0x4349AD: FormJacobian (SetupJacobian.c:58) n1: 104790400 0xA8F850: SNESComputeJacobian (snes.c:2193) n1: 104790400 0xAF7C15: SNESSolve_NEWTONLS (ls.c:230) n1: 104790400 0xA9E3CC: SNESSolve (snes.c:3743) n1: 104790400 0x40FB9B: SolveTimeDependent (Nwtun.c:758) n0: 104790400 0x40BC62: main (Nwtun.c:417) n0: 17465520 in 2 places, all below massif's threshold (01.00%) n1: 122255920 0x81EA9A: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3702) n2: 122255920 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664) n1: 104790400 0x906416: MatGetSubMatrix_MPIAIJ_All (mpiov.c:571) n1: 104790400 0x8DE981: MatGetSeqNonzeroStructure_MPIAIJ (mpiaij.c:3010) n1: 104790400 0x78FDBF: MatGetSeqNonzeroStructure (matrix.c:6487) n1: 104790400 0xCA31E2: MatColoringApply_SL (color.c:78) n1: 104790400 0x746C6C: MatColoringApply (matcoloring.c:379) n1: 104790400 0xAB0879: SNESComputeJacobianDefaultColor (snesj2.c:71) n1: 104790400 0x4349AD: FormJacobian (SetupJacobian.c:58) n1: 104790400 0xA8F850: SNESComputeJacobian (snes.c:2193) n1: 104790400 0xAF7C15: SNESSolve_NEWTONLS (ls.c:230) n1: 104790400 0xA9E3CC: SNESSolve (snes.c:3743) n1: 104790400 0x40FB9B: SolveTimeDependent (Nwtun.c:758) n0: 104790400 0x40BC62: main (Nwtun.c:417) n0: 17465520 in 2 places, all below massif's threshold (01.00%) n1: 111724968 0xC99701: VecCreate_MPI_Private (pbvec.c:201) n0: 111724968 in 2 places, all below massif's threshold (01.00%) n1: 104790400 0x905DD7: MatGetSubMatrix_MPIAIJ_All (mpiov.c:549) n1: 104790400 0x8DE981: MatGetSeqNonzeroStructure_MPIAIJ (mpiaij.c:3010) n1: 104790400 0x78FDBF: MatGetSeqNonzeroStructure (matrix.c:6487) n1: 104790400 0xCA31E2: MatColoringApply_SL (color.c:78) n1: 104790400 0x746C6C: MatColoringApply (matcoloring.c:379) n1: 104790400 0xAB0879: SNESComputeJacobianDefaultColor (snesj2.c:71) n1: 104790400 0x4349AD: FormJacobian (SetupJacobian.c:58) n1: 104790400 0xA8F850: SNESComputeJacobian (snes.c:2193) n1: 104790400 0xAF7C15: SNESSolve_NEWTONLS (ls.c:230) n1: 104790400 0xA9E3CC: SNESSolve (snes.c:3743) n1: 104790400 0x40FB9B: SolveTimeDependent (Nwtun.c:758) n0: 104790400 0x40BC62: main (Nwtun.c:417) n0: 10879052 in 95 places, all below massif's threshold (01.00%) From Sander.Arens at UGent.be Wed Feb 17 03:16:57 2016 From: Sander.Arens at UGent.be (Sander Arens) Date: Wed, 17 Feb 2016 10:16:57 +0100 Subject: [petsc-users] DMPlexTSComputeIJacobianFEM Message-ID: Hi all, Is there a DMPlexSNESComputeJacobianFEM equivalent for the TS, i.e. DMPlexTSComputeIJacobianFEM? Or is there a better way to assemble the Jacobian for the TS in the case of FEM? -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Feb 17 03:18:43 2016 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 17 Feb 2016 03:18:43 -0600 Subject: [petsc-users] DMPlexTSComputeIJacobianFEM In-Reply-To: References: Message-ID: I just added this, but it is only in 'next'. It will go into 'master' next week I think. Thanks, Matt On Wed, Feb 17, 2016 at 3:16 AM, Sander Arens wrote: > Hi all, > > Is there a DMPlexSNESComputeJacobianFEM equivalent for the TS, i.e. > DMPlexTSComputeIJacobianFEM? Or is there a better way to assemble the > Jacobian for the TS in the case of FEM? > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Sander.Arens at UGent.be Wed Feb 17 03:19:53 2016 From: Sander.Arens at UGent.be (Sander Arens) Date: Wed, 17 Feb 2016 10:19:53 +0100 Subject: [petsc-users] DMPlexTSComputeIJacobianFEM In-Reply-To: References: Message-ID: Ok, great! Thanks Matt. On 17 February 2016 at 10:18, Matthew Knepley wrote: > I just added this, but it is only in 'next'. It will go into 'master' next > week I think. > > Thanks, > > Matt > > On Wed, Feb 17, 2016 at 3:16 AM, Sander Arens > wrote: > >> Hi all, >> >> Is there a DMPlexSNESComputeJacobianFEM equivalent for the TS, i.e. >> DMPlexTSComputeIJacobianFEM? Or is there a better way to assemble the >> Jacobian for the TS in the case of FEM? >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From boyceg at email.unc.edu Wed Feb 17 11:06:01 2016 From: boyceg at email.unc.edu (Griffith, Boyce Eugene) Date: Wed, 17 Feb 2016 17:06:01 +0000 Subject: [petsc-users] Error with MatSchurComplement In-Reply-To: <1FC93CCE-F389-4555-BDDF-FEF60ABCC154@mcs.anl.gov> References: <1FC93CCE-F389-4555-BDDF-FEF60ABCC154@mcs.anl.gov> Message-ID: <5EBB1897-32AC-430B-BECC-11742E7F0BA9@email.unc.edu> > On Feb 17, 2016, at 12:57 AM, Barry Smith wrote: > > > Hmm, it seems this should never happen. It is like inconsistent (and in correct) IS are being generated. > > 1 is an odd number of processes, does it happen then? > > Are you providing the IS yourself to the PCFIELDSPLIT or using the default? If you are providing them is it possible that you have a bug in generating them for certain configurations? Of course it could also be a bug in PETSc. We are doing this ourselves. Could this happen if we accidentally duplicated entries in one of the IS'es on different processors? -- Boyce > I recommend trying to produce the crash on a small a problem as possible then run in the debugger to check the IS at each stage to see why one of them becomes bonkers. > > It could be we should have more asserts in PETSc to insure that input IS are compatible. > > Barry > >> On Feb 16, 2016, at 10:37 PM, Bhalla, Amneet Pal S wrote: >> >> Hi Folks, >> >> I am getting error with odd number of processors with PCFIELDSPLIT. The error log is attached. Any hints where to start digging? >> >> >> +++++++++++++++++++++++++++++++++++++++++++++++++++ >> At beginning of timestep # 0 >> Simulation time is 0 >> Residual norms for ib_ solve. >> 0 KSP unpreconditioned resid norm 5.093934451651e+05 true resid norm 5.093934451651e+05 ||r(i)||/||b|| 1.000000000000e+00 >> [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [2]PETSC ERROR: Nonconforming object sizes >> [2]PETSC ERROR: Local columns of A10 1072 do not equal local rows of A00 1104 >> [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [2]PETSC ERROR: Petsc Release Version 3.6.1, unknown >> [2]PETSC ERROR: ./main2d on a darwin-dbg named Amneets-MBP.attlocal.net by Taylor Tue Feb 16 20:30:45 2016 >> [2]PETSC ERROR: Configure options --CC=mpicc --CXX=mpicxx --FC=mpif90 --PETSC_ARCH=darwin-dbg --with-debugging=1 --with-c++-support=1 --with-hypre=1 --download-hypre=1 --with-hdf5=yes --with-hdf5-dir=/Users/Taylor/Documents/SOFTWARES/HDF5/ >> [2]PETSC ERROR: #1 MatSchurComplementSetSubMatrices() line 261 in /Users/Taylor/Documents/SOFTWARES/PETSc-BitBucket/PETSc/src/ksp/ksp/utils/schurm.c >> [2]PETSC ERROR: #2 PCSetUp_FieldSplit() line 714 in /Users/Taylor/Documents/SOFTWARES/PETSc-BitBucket/PETSc/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> [2]PETSC ERROR: #3 PCSetUp() line 983 in /Users/Taylor/Documents/SOFTWARES/PETSc-BitBucket/PETSc/src/ksp/pc/interface/precon.c >> [2]PETSC ERROR: #4 KSPSetUp() line 332 in /Users/Taylor/Documents/SOFTWARES/PETSc-BitBucket/PETSc/src/ksp/ksp/interface/itfunc.c >> [2]PETSC ERROR: #5 KSPSolve() line 546 in /Users/Taylor/Documents/SOFTWARES/PETSc-BitBucket/PETSc/src/ksp/ksp/interface/itfunc.c >> [2]PETSC ERROR: #6 solveSystem() line 229 in ../../../IBAMR/ibtk/lib/../src/solvers/impls/PETScLevelSolver.cpp >> P=00002:Program abort called in file ``../../../IBAMR/ibtk/lib/../src/solvers/impls/PETScLevelSolver.cpp'' at line 229 >> P=00002:ERROR MESSAGE: >> P=00002: >> application called MPI_Abort(MPI_COMM_WORLD, -1) - process 2 >> *** Warning, AutoPtr is deprecated and will be removed in a future library version! ./include/libmesh/auto_ptr.h, line 271, compiled Feb 3 2016 at 10:29:17 *** >> >> >> ? Amneet >> >> From rlmackie862 at gmail.com Wed Feb 17 11:33:18 2016 From: rlmackie862 at gmail.com (Randall Mackie) Date: Wed, 17 Feb 2016 09:33:18 -0800 Subject: [petsc-users] memory scalable AO Message-ID: What is the correct way to set the AO for a DMDA to be the memory scalable version? I have tried this: call DMDASetAOType(da,AOMEMORYSCALABLE,ierr) call DMDAGetAO(da,ao,ierr) The code compiles fine, but I simply get a Segmentation Violation when I run it: [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [3]PETSC ERROR: to get more information on the crash. [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [3]PETSC ERROR: Signal received [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [3]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [3]PETSC ERROR: Configure options PETSC_ARCH=linux-gfortran-opt --with-clean=1 --with-scalar-type=complex --with-debugging=0 --with-fortran=1 --download-mpich=./mpich-3.1.3.tar.gz --FOPTFLAGS=-O3 --COPTFLAGS=-O3 [3]PETSC ERROR: #1 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 3 [cli_3]: aborting job: In fact, I get the same thing even if I set the AOType to AOBASIC, in other words call DMDASetAOTYPE(da,AOBASIC,ierr) Previously, I did not set an AOType, and only called DMDAGetAO, which works fine, and from what I can tell sets the type to basic. Thanks, Randy M. From bsmith at mcs.anl.gov Wed Feb 17 12:51:39 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 17 Feb 2016 12:51:39 -0600 Subject: [petsc-users] Error with MatSchurComplement In-Reply-To: <5EBB1897-32AC-430B-BECC-11742E7F0BA9@email.unc.edu> References: <1FC93CCE-F389-4555-BDDF-FEF60ABCC154@mcs.anl.gov> <5EBB1897-32AC-430B-BECC-11742E7F0BA9@email.unc.edu> Message-ID: <6F26E83B-0CAA-4D52-9C44-EFFCF432CCB1@mcs.anl.gov> > On Feb 17, 2016, at 11:06 AM, Griffith, Boyce Eugene wrote: > >> >> On Feb 17, 2016, at 12:57 AM, Barry Smith wrote: >> >> >> Hmm, it seems this should never happen. It is like inconsistent (and in correct) IS are being generated. >> >> 1 is an odd number of processes, does it happen then? >> >> Are you providing the IS yourself to the PCFIELDSPLIT or using the default? If you are providing them is it possible that you have a bug in generating them for certain configurations? Of course it could also be a bug in PETSc. > > We are doing this ourselves. Could this happen if we accidentally duplicated entries in one of the IS'es on different processors? I don't know could be. Likely there is some wrong with your IS. > > -- Boyce > >> I recommend trying to produce the crash on a small a problem as possible then run in the debugger to check the IS at each stage to see why one of them becomes bonkers. >> >> It could be we should have more asserts in PETSc to insure that input IS are compatible. >> >> Barry >> >>> On Feb 16, 2016, at 10:37 PM, Bhalla, Amneet Pal S wrote: >>> >>> Hi Folks, >>> >>> I am getting error with odd number of processors with PCFIELDSPLIT. The error log is attached. Any hints where to start digging? >>> >>> >>> +++++++++++++++++++++++++++++++++++++++++++++++++++ >>> At beginning of timestep # 0 >>> Simulation time is 0 >>> Residual norms for ib_ solve. >>> 0 KSP unpreconditioned resid norm 5.093934451651e+05 true resid norm 5.093934451651e+05 ||r(i)||/||b|| 1.000000000000e+00 >>> [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>> [2]PETSC ERROR: Nonconforming object sizes >>> [2]PETSC ERROR: Local columns of A10 1072 do not equal local rows of A00 1104 >>> [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>> [2]PETSC ERROR: Petsc Release Version 3.6.1, unknown >>> [2]PETSC ERROR: ./main2d on a darwin-dbg named Amneets-MBP.attlocal.net by Taylor Tue Feb 16 20:30:45 2016 >>> [2]PETSC ERROR: Configure options --CC=mpicc --CXX=mpicxx --FC=mpif90 --PETSC_ARCH=darwin-dbg --with-debugging=1 --with-c++-support=1 --with-hypre=1 --download-hypre=1 --with-hdf5=yes --with-hdf5-dir=/Users/Taylor/Documents/SOFTWARES/HDF5/ >>> [2]PETSC ERROR: #1 MatSchurComplementSetSubMatrices() line 261 in /Users/Taylor/Documents/SOFTWARES/PETSc-BitBucket/PETSc/src/ksp/ksp/utils/schurm.c >>> [2]PETSC ERROR: #2 PCSetUp_FieldSplit() line 714 in /Users/Taylor/Documents/SOFTWARES/PETSc-BitBucket/PETSc/src/ksp/pc/impls/fieldsplit/fieldsplit.c >>> [2]PETSC ERROR: #3 PCSetUp() line 983 in /Users/Taylor/Documents/SOFTWARES/PETSc-BitBucket/PETSc/src/ksp/pc/interface/precon.c >>> [2]PETSC ERROR: #4 KSPSetUp() line 332 in /Users/Taylor/Documents/SOFTWARES/PETSc-BitBucket/PETSc/src/ksp/ksp/interface/itfunc.c >>> [2]PETSC ERROR: #5 KSPSolve() line 546 in /Users/Taylor/Documents/SOFTWARES/PETSc-BitBucket/PETSc/src/ksp/ksp/interface/itfunc.c >>> [2]PETSC ERROR: #6 solveSystem() line 229 in ../../../IBAMR/ibtk/lib/../src/solvers/impls/PETScLevelSolver.cpp >>> P=00002:Program abort called in file ``../../../IBAMR/ibtk/lib/../src/solvers/impls/PETScLevelSolver.cpp'' at line 229 >>> P=00002:ERROR MESSAGE: >>> P=00002: >>> application called MPI_Abort(MPI_COMM_WORLD, -1) - process 2 >>> *** Warning, AutoPtr is deprecated and will be removed in a future library version! ./include/libmesh/auto_ptr.h, line 271, compiled Feb 3 2016 at 10:29:17 *** >>> >>> >>> ? Amneet From amneetb at live.unc.edu Wed Feb 17 12:58:08 2016 From: amneetb at live.unc.edu (Bhalla, Amneet Pal S) Date: Wed, 17 Feb 2016 18:58:08 +0000 Subject: [petsc-users] Error with MatSchurComplement In-Reply-To: <6F26E83B-0CAA-4D52-9C44-EFFCF432CCB1@mcs.anl.gov> References: <1FC93CCE-F389-4555-BDDF-FEF60ABCC154@mcs.anl.gov> <5EBB1897-32AC-430B-BECC-11742E7F0BA9@email.unc.edu> <6F26E83B-0CAA-4D52-9C44-EFFCF432CCB1@mcs.anl.gov> Message-ID: On Feb 17, 2016, at 10:51 AM, Barry Smith > wrote: I don't know could be. Likely there is some wrong with your IS. I can try to remove duplicated entries while creating a parallel IS. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Feb 17 13:35:11 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 17 Feb 2016 13:35:11 -0600 Subject: [petsc-users] memory scalable AO In-Reply-To: References: Message-ID: Should be ok. Do you have implicit none and the correct include files so AOMEMORYSCALABLE is defined? I think you need to run in the debugger next to track why this happens. Barry > On Feb 17, 2016, at 11:33 AM, Randall Mackie wrote: > > What is the correct way to set the AO for a DMDA to be the memory scalable version? > > I have tried this: > > call DMDASetAOType(da,AOMEMORYSCALABLE,ierr) > call DMDAGetAO(da,ao,ierr) > > The code compiles fine, but I simply get a Segmentation Violation when I run it: > > [3]PETSC ERROR: ------------------------------------------------------------------------ > [3]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [3]PETSC ERROR: to get more information on the crash. > [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [3]PETSC ERROR: Signal received > [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [3]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > [3]PETSC ERROR: Configure options PETSC_ARCH=linux-gfortran-opt --with-clean=1 --with-scalar-type=complex --with-debugging=0 --with-fortran=1 --download-mpich=./mpich-3.1.3.tar.gz --FOPTFLAGS=-O3 --COPTFLAGS=-O3 > [3]PETSC ERROR: #1 User provided function() line 0 in unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 3 > [cli_3]: aborting job: > > > In fact, I get the same thing even if I set the AOType to AOBASIC, in other words > > call DMDASetAOTYPE(da,AOBASIC,ierr) > > > Previously, I did not set an AOType, and only called DMDAGetAO, which works fine, and from what I can tell sets the type to basic. > > > Thanks, Randy M. From bsmith at mcs.anl.gov Wed Feb 17 14:11:04 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 17 Feb 2016 14:11:04 -0600 Subject: [petsc-users] SNESComputeJacobianDefaultColor use too much memory In-Reply-To: <56C425F8.1060306@gmail.com> References: <56C319D0.9060208@gmail.com> <684F7147-4F58-42E3-9C9E-2C2297383276@mcs.anl.gov> <56C3EC9B.5080306@gmail.com> <1199051F-2EC1-4462-A7C0-E2DDCA95A4C3@mcs.anl.gov> <56C409D9.70002@gmail.com> <2424BDF0-AD22-40EB-9920-AF697BFF4DAF@mcs.anl.gov> <56C425F8.1060306@gmail.com> Message-ID: <68210D25-00D1-4273-9C83-5E6A299D17E7@mcs.anl.gov> Hmm, Matt, this is nuts. This if (size > 1) { /* create a sequential iscoloring on all processors */ ierr = MatGetSeqNonzeroStructure(mat,&mat_seq);CHKERRQ(ierr); } It sequentializes the graph. It looks like the only parallel coloring for Jacobians is MATCOLORINGGREEDY? Try that. Maybe that should be the default? Barry > On Feb 17, 2016, at 1:49 AM, Rongliang Chen wrote: > > Dear Barry, > > The massif output for the large case is attached. It shows that the job was kill in the function MatGetSubMatrix_MPIAIJ_All(). Any suggestions? > > -------------------- > [8]PETSC ERROR: #1 MatGetSubMatrix_MPIAIJ_All() line 622 in /home/rlchen/soft/petsc-3.5.2/src/mat/impls/aij/mpi/mpiov.c > [8]PETSC ERROR: #2 MatGetSeqNonzeroStructure_MPIAIJ() line 3010 in /home/rlchen/soft/petsc-3.5.2/src/mat/impls/aij/mpi/mpiaij.c > [8]PETSC ERROR: #3 MatGetSeqNonzeroStructure() line 6487 in /home/rlchen/soft/petsc-3.5.2/src/mat/interface/matrix.c > [8]PETSC ERROR: #4 MatColoringApply_SL() line 78 in /home/rlchen/soft/petsc-3.5.2/src/mat/color/impls/minpack/color.c > [8]PETSC ERROR: #5 MatColoringApply() line 379 in /home/rlchen/soft/petsc-3.5.2/src/mat/color/interface/matcoloring.c > [8]PETSC ERROR: #6 SNESComputeJacobianDefaultColor() line 71 in /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snesj2.c > [8]PETSC ERROR: #7 FormJacobian() line 58 in /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/SetupJacobian.c > [8]PETSC ERROR: #8 SNESComputeJacobian() line 2193 in /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c > [8]PETSC ERROR: #9 SNESSolve_NEWTONLS() line 230 in /home/rlchen/soft/petsc-3.5.2/src/snes/impls/ls/ls.c > [8]PETSC ERROR: #10 SNESSolve() line 3743 in /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c > [8]PETSC ERROR: #11 SolveTimeDependent() line 758 in /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/Nwtun.c > [8]PETSC ERROR: #12 main() line 417 in /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/Nwtun.c > ------------------- > > Best, > Rongliang > > On 02/17/2016 02:09 PM, Barry Smith wrote: >> Yes, this is the type of output I was expecting. Now you need to produce it for a large case. >> >> Barry >> >>> On Feb 16, 2016, at 11:49 PM, Rongliang Chen wrote: >>> >>> Hi Barry, >>> >>> I run the code with valgrind on a workstation for a smaller case and it produces some ASCII information (see attached). Is this helpful? >>> >>> Best, >>> Rongliang >>> >>> On 02/17/2016 01:30 PM, Barry Smith wrote: >>>> Hmm, something didn't work right with massif. I should give a bunch of ASCII information about how much memory is used at different times in the code. You may need to play around with the massif options and google documentation on how to get it to provide useful information. Once it produces the useful information it will be very helpful. >>>> >>>> time=0 >>>> mem_heap_B=0 >>>> mem_heap_extra_B=0 >>>> mem_stacks_B=0 >>>> heap_tree=empty >>>> >>>> >>>>> On Feb 16, 2016, at 9:44 PM, Rongliang Chen wrote: >>>>> >>>>> Dear Barry, >>>>> >>>>> Many thanks for your reply. >>>>> >>>>> I checked with the valgrind and did not obtain any outputs (massif.out.) because the job was killed before it reached the end. >>>>> >>>>> Then I switch to a smaller case, it works well and one of the output is attached (I did not find any useful information in it). The output with the option -mat_coloring_view is followed, which shows that the number of colors is 65. Any ideas for this? >>>>> >>>>> MatColoring Object: 480 MPI processes >>>>> type: sl >>>>> Weight type: RANDOM >>>>> Distance 2, Max. Colors 65535 >>>>> Number of colors 65 >>>>> Number of total columns 1637350 >>>>> >>>>> Best regards, >>>>> Rongliang >>>>> >>>>> On 02/17/2016 01:13 AM, Barry Smith wrote: >>>>>> How many colors are needed? >>>>>> >>>>>> You need to produce a breakdown of where all the memory is being used. For example valgrind with the >>>>>> http://valgrind.org/docs/manual/ms-manual.html >>>>>> >>>>>> >>>>>> Barry >>>>>> >>>>>> >>>>>>> On Feb 16, 2016, at 6:45 AM, Rongliang Chen >>>>>>> wrote: >>>>>>> >>>>>>> Dear all, >>>>>>> >>>>>>> I am using the DMPlex to solve a PDE on a unstructured mesh and I use the SNESComputeJacobianDefaultColor to compute the Jacobian matrix. >>>>>>> >>>>>>> My code works well for small problems (such as problem with 3.3x10^5 cells using 120 cores) but when I increase the number of cells (2.6x10^6 cells using 1920 cores), I go out of memory in the function MatColoringApply. It shows that one of the cores uses over 11G memory. I think this is unreasonable. Do you have any suggestions for debugging this problem? >>>>>>> >>>>>>> Best regards, >>>>>>> Rongliang >>>>>>> >>>>> >>> > > From rlmackie862 at gmail.com Wed Feb 17 15:46:58 2016 From: rlmackie862 at gmail.com (Randall Mackie) Date: Wed, 17 Feb 2016 13:46:58 -0800 Subject: [petsc-users] memory scalable AO In-Reply-To: References: Message-ID: The attached test program demonstrates the problem. When I run it, I get the following output: [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] PetscStrlen line 150 /home/rmackie/PETSc/petsc-3.6.3/src/sys/utils/str.c [0]PETSC ERROR: [0] PetscStrallocpy line 185 /home/rmackie/PETSc/petsc-3.6.3/src/sys/utils/str.c [0]PETSC ERROR: [0] DMDASetAOType line 84 /home/rmackie/PETSc/petsc-3.6.3/src/dm/impls/da/daindex.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. Randy > On Feb 17, 2016, at 11:35 AM, Barry Smith wrote: > > > Should be ok. Do you have implicit none and the correct include files so AOMEMORYSCALABLE is defined? > > I think you need to run in the debugger next to track why this happens. > > Barry > >> On Feb 17, 2016, at 11:33 AM, Randall Mackie wrote: >> >> What is the correct way to set the AO for a DMDA to be the memory scalable version? >> >> I have tried this: >> >> call DMDASetAOType(da,AOMEMORYSCALABLE,ierr) >> call DMDAGetAO(da,ao,ierr) >> >> The code compiles fine, but I simply get a Segmentation Violation when I run it: >> >> [3]PETSC ERROR: ------------------------------------------------------------------------ >> [3]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >> [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> [3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >> [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run >> [3]PETSC ERROR: to get more information on the crash. >> [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [3]PETSC ERROR: Signal received >> [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [3]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >> [3]PETSC ERROR: Configure options PETSC_ARCH=linux-gfortran-opt --with-clean=1 --with-scalar-type=complex --with-debugging=0 --with-fortran=1 --download-mpich=./mpich-3.1.3.tar.gz --FOPTFLAGS=-O3 --COPTFLAGS=-O3 >> [3]PETSC ERROR: #1 User provided function() line 0 in unknown file >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 3 >> [cli_3]: aborting job: >> >> >> In fact, I get the same thing even if I set the AOType to AOBASIC, in other words >> >> call DMDASetAOTYPE(da,AOBASIC,ierr) >> >> >> Previously, I did not set an AOType, and only called DMDAGetAO, which works fine, and from what I can tell sets the type to basic. >> >> >> Thanks, Randy M. > -------------- next part -------------- A non-text attachment was scrubbed... Name: test.F90 Type: application/octet-stream Size: 640 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: makefile Type: application/octet-stream Size: 369 bytes Desc: not available URL: From bsmith at mcs.anl.gov Wed Feb 17 16:01:02 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 17 Feb 2016 16:01:02 -0600 Subject: [petsc-users] memory scalable AO In-Reply-To: References: Message-ID: Ahh, yes. This is our fault. Because this function takes a string it needs a custom Fortran stub which it does not have. We'll get you a patch soon. Barry > On Feb 17, 2016, at 3:46 PM, Randall Mackie wrote: > > The attached test program demonstrates the problem. When I run it, I get the following output: > > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] PetscStrlen line 150 /home/rmackie/PETSc/petsc-3.6.3/src/sys/utils/str.c > [0]PETSC ERROR: [0] PetscStrallocpy line 185 /home/rmackie/PETSc/petsc-3.6.3/src/sys/utils/str.c > [0]PETSC ERROR: [0] DMDASetAOType line 84 /home/rmackie/PETSc/petsc-3.6.3/src/dm/impls/da/daindex.c > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Signal received > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > > Randy > > >> On Feb 17, 2016, at 11:35 AM, Barry Smith wrote: >> >> >> Should be ok. Do you have implicit none and the correct include files so AOMEMORYSCALABLE is defined? >> >> I think you need to run in the debugger next to track why this happens. >> >> Barry >> >>> On Feb 17, 2016, at 11:33 AM, Randall Mackie wrote: >>> >>> What is the correct way to set the AO for a DMDA to be the memory scalable version? >>> >>> I have tried this: >>> >>> call DMDASetAOType(da,AOMEMORYSCALABLE,ierr) >>> call DMDAGetAO(da,ao,ierr) >>> >>> The code compiles fine, but I simply get a Segmentation Violation when I run it: >>> >>> [3]PETSC ERROR: ------------------------------------------------------------------------ >>> [3]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>> [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>> [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>> [3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>> [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run >>> [3]PETSC ERROR: to get more information on the crash. >>> [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>> [3]PETSC ERROR: Signal received >>> [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>> [3]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>> [3]PETSC ERROR: Configure options PETSC_ARCH=linux-gfortran-opt --with-clean=1 --with-scalar-type=complex --with-debugging=0 --with-fortran=1 --download-mpich=./mpich-3.1.3.tar.gz --FOPTFLAGS=-O3 --COPTFLAGS=-O3 >>> [3]PETSC ERROR: #1 User provided function() line 0 in unknown file >>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 3 >>> [cli_3]: aborting job: >>> >>> >>> In fact, I get the same thing even if I set the AOType to AOBASIC, in other words >>> >>> call DMDASetAOTYPE(da,AOBASIC,ierr) >>> >>> >>> Previously, I did not set an AOType, and only called DMDAGetAO, which works fine, and from what I can tell sets the type to basic. >>> >>> >>> Thanks, Randy M. >> > From bsmith at mcs.anl.gov Wed Feb 17 17:15:39 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 17 Feb 2016 17:15:39 -0600 Subject: [petsc-users] memory scalable AO In-Reply-To: References: Message-ID: <0DD3688F-1E57-4AC7-89A9-DD96B6A676FA@mcs.anl.gov> Here is patch. -------------- next part -------------- A non-text attachment was scrubbed... Name: fix-fortran-dmdasetaotype.patch Type: application/octet-stream Size: 1495 bytes Desc: not available URL: -------------- next part -------------- If it works for you I'll put it in maint and master tomorrow. Barry > On Feb 17, 2016, at 3:46 PM, Randall Mackie wrote: > > The attached test program demonstrates the problem. When I run it, I get the following output: > > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] PetscStrlen line 150 /home/rmackie/PETSc/petsc-3.6.3/src/sys/utils/str.c > [0]PETSC ERROR: [0] PetscStrallocpy line 185 /home/rmackie/PETSc/petsc-3.6.3/src/sys/utils/str.c > [0]PETSC ERROR: [0] DMDASetAOType line 84 /home/rmackie/PETSc/petsc-3.6.3/src/dm/impls/da/daindex.c > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Signal received > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > > Randy > > >> On Feb 17, 2016, at 11:35 AM, Barry Smith wrote: >> >> >> Should be ok. Do you have implicit none and the correct include files so AOMEMORYSCALABLE is defined? >> >> I think you need to run in the debugger next to track why this happens. >> >> Barry >> >>> On Feb 17, 2016, at 11:33 AM, Randall Mackie wrote: >>> >>> What is the correct way to set the AO for a DMDA to be the memory scalable version? >>> >>> I have tried this: >>> >>> call DMDASetAOType(da,AOMEMORYSCALABLE,ierr) >>> call DMDAGetAO(da,ao,ierr) >>> >>> The code compiles fine, but I simply get a Segmentation Violation when I run it: >>> >>> [3]PETSC ERROR: ------------------------------------------------------------------------ >>> [3]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>> [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>> [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>> [3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>> [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run >>> [3]PETSC ERROR: to get more information on the crash. >>> [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>> [3]PETSC ERROR: Signal received >>> [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>> [3]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>> [3]PETSC ERROR: Configure options PETSC_ARCH=linux-gfortran-opt --with-clean=1 --with-scalar-type=complex --with-debugging=0 --with-fortran=1 --download-mpich=./mpich-3.1.3.tar.gz --FOPTFLAGS=-O3 --COPTFLAGS=-O3 >>> [3]PETSC ERROR: #1 User provided function() line 0 in unknown file >>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 3 >>> [cli_3]: aborting job: >>> >>> >>> In fact, I get the same thing even if I set the AOType to AOBASIC, in other words >>> >>> call DMDASetAOTYPE(da,AOBASIC,ierr) >>> >>> >>> Previously, I did not set an AOType, and only called DMDAGetAO, which works fine, and from what I can tell sets the type to basic. >>> >>> >>> Thanks, Randy M. >> > From bikash at umich.edu Wed Feb 17 19:03:58 2016 From: bikash at umich.edu (Bikash Kanungo) Date: Wed, 17 Feb 2016 20:03:58 -0500 Subject: [petsc-users] Slow MatAssembly and MatMat Mult Message-ID: Hi, I have two small (2500x2500) matrices parallelized across 480 processors. One of them is an MPIAIJ matrix while the other is an MPIDENSE matrix. I perform a MatMatMult involving these two matrices. I tried these operations on two machines, one is the local cluster at University of Michigan and the other is the XSEDE Comet machine. The Comet machine takes 10-20 times more time in steps involving MatAssembly and MatMatMult of the aforementioned matrices. I have other Petsc MatMult operations in the same code involving larger matrices (4 million x 4 million) which show similar timing on both machines. It's just those small parallel matrices that are inconsistent in terms of their timings. I used same the compilers and MPI libraries in both the machines except that I have suppressed "avx2" flag in Comet. I believe avx2 affects floating point operations and not communication. I would like to know what might be causing these inconsistencies only in case of the small matrices. Are there any network settings that I can look into and compare? Regards, Bikash -- Bikash S. Kanungo PhD Student Computational Materials Physics Group Mechanical Engineering University of Michigan -------------- next part -------------- An HTML attachment was scrubbed... URL: From rlmackie862 at gmail.com Wed Feb 17 19:29:11 2016 From: rlmackie862 at gmail.com (Randall Mackie) Date: Wed, 17 Feb 2016 17:29:11 -0800 Subject: [petsc-users] memory scalable AO In-Reply-To: <0DD3688F-1E57-4AC7-89A9-DD96B6A676FA@mcs.anl.gov> References: <0DD3688F-1E57-4AC7-89A9-DD96B6A676FA@mcs.anl.gov> Message-ID: <4A7DAAD2-9FE3-482D-AACE-604F89BB1B80@gmail.com> Unfortunately I am getting exactly the same result. I applied the patch (patch -p1 < fix-fortran-dmdasetaotype.patch) Then I recompiled the library, and my code, but the error remains. Perhaps there is something else I?m missing? Randy > On Feb 17, 2016, at 3:15 PM, Barry Smith wrote: > > > Here is patch. > > If it works for you I'll put it in maint and master tomorrow. > > Barry > >> On Feb 17, 2016, at 3:46 PM, Randall Mackie wrote: >> >> The attached test program demonstrates the problem. When I run it, I get the following output: >> >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >> [0]PETSC ERROR: likely location of problem given in stack below >> [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >> [0]PETSC ERROR: INSTEAD the line number of the start of the function >> [0]PETSC ERROR: is given. >> [0]PETSC ERROR: [0] PetscStrlen line 150 /home/rmackie/PETSc/petsc-3.6.3/src/sys/utils/str.c >> [0]PETSC ERROR: [0] PetscStrallocpy line 185 /home/rmackie/PETSc/petsc-3.6.3/src/sys/utils/str.c >> [0]PETSC ERROR: [0] DMDASetAOType line 84 /home/rmackie/PETSc/petsc-3.6.3/src/dm/impls/da/daindex.c >> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [0]PETSC ERROR: Signal received >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> >> >> Randy >> >> >>> On Feb 17, 2016, at 11:35 AM, Barry Smith wrote: >>> >>> >>> Should be ok. Do you have implicit none and the correct include files so AOMEMORYSCALABLE is defined? >>> >>> I think you need to run in the debugger next to track why this happens. >>> >>> Barry >>> >>>> On Feb 17, 2016, at 11:33 AM, Randall Mackie wrote: >>>> >>>> What is the correct way to set the AO for a DMDA to be the memory scalable version? >>>> >>>> I have tried this: >>>> >>>> call DMDASetAOType(da,AOMEMORYSCALABLE,ierr) >>>> call DMDAGetAO(da,ao,ierr) >>>> >>>> The code compiles fine, but I simply get a Segmentation Violation when I run it: >>>> >>>> [3]PETSC ERROR: ------------------------------------------------------------------------ >>>> [3]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>>> [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>> [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>>> [3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>> [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run >>>> [3]PETSC ERROR: to get more information on the crash. >>>> [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>> [3]PETSC ERROR: Signal received >>>> [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>>> [3]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>>> [3]PETSC ERROR: Configure options PETSC_ARCH=linux-gfortran-opt --with-clean=1 --with-scalar-type=complex --with-debugging=0 --with-fortran=1 --download-mpich=./mpich-3.1.3.tar.gz --FOPTFLAGS=-O3 --COPTFLAGS=-O3 >>>> [3]PETSC ERROR: #1 User provided function() line 0 in unknown file >>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 3 >>>> [cli_3]: aborting job: >>>> >>>> >>>> In fact, I get the same thing even if I set the AOType to AOBASIC, in other words >>>> >>>> call DMDASetAOTYPE(da,AOBASIC,ierr) >>>> >>>> >>>> Previously, I did not set an AOType, and only called DMDAGetAO, which works fine, and from what I can tell sets the type to basic. >>>> >>>> >>>> Thanks, Randy M. >>> >> > From rongliang.chan at gmail.com Wed Feb 17 19:53:16 2016 From: rongliang.chan at gmail.com (Rongliang Chen) Date: Thu, 18 Feb 2016 09:53:16 +0800 Subject: [petsc-users] SNESComputeJacobianDefaultColor use too much memory In-Reply-To: <68210D25-00D1-4273-9C83-5E6A299D17E7@mcs.anl.gov> References: <56C319D0.9060208@gmail.com> <684F7147-4F58-42E3-9C9E-2C2297383276@mcs.anl.gov> <56C3EC9B.5080306@gmail.com> <1199051F-2EC1-4462-A7C0-E2DDCA95A4C3@mcs.anl.gov> <56C409D9.70002@gmail.com> <2424BDF0-AD22-40EB-9920-AF697BFF4DAF@mcs.anl.gov> <56C425F8.1060306@gmail.com> <68210D25-00D1-4273-9C83-5E6A299D17E7@mcs.anl.gov> Message-ID: <56C5240C.7080907@gmail.com> The greedy coloring works. Thanks Barry. Best, Rongliang On 02/18/2016 04:11 AM, Barry Smith wrote: > Hmm, Matt, this is nuts. This > > if (size > 1) { > /* create a sequential iscoloring on all processors */ > ierr = MatGetSeqNonzeroStructure(mat,&mat_seq);CHKERRQ(ierr); > } > > It sequentializes the graph. > > It looks like the only parallel coloring for Jacobians is MATCOLORINGGREEDY? Try that. > > Maybe that should be the default? > > Barry > > >> On Feb 17, 2016, at 1:49 AM, Rongliang Chen wrote: >> >> Dear Barry, >> >> The massif output for the large case is attached. It shows that the job was kill in the function MatGetSubMatrix_MPIAIJ_All(). Any suggestions? >> >> -------------------- >> [8]PETSC ERROR: #1 MatGetSubMatrix_MPIAIJ_All() line 622 in /home/rlchen/soft/petsc-3.5.2/src/mat/impls/aij/mpi/mpiov.c >> [8]PETSC ERROR: #2 MatGetSeqNonzeroStructure_MPIAIJ() line 3010 in /home/rlchen/soft/petsc-3.5.2/src/mat/impls/aij/mpi/mpiaij.c >> [8]PETSC ERROR: #3 MatGetSeqNonzeroStructure() line 6487 in /home/rlchen/soft/petsc-3.5.2/src/mat/interface/matrix.c >> [8]PETSC ERROR: #4 MatColoringApply_SL() line 78 in /home/rlchen/soft/petsc-3.5.2/src/mat/color/impls/minpack/color.c >> [8]PETSC ERROR: #5 MatColoringApply() line 379 in /home/rlchen/soft/petsc-3.5.2/src/mat/color/interface/matcoloring.c >> [8]PETSC ERROR: #6 SNESComputeJacobianDefaultColor() line 71 in /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snesj2.c >> [8]PETSC ERROR: #7 FormJacobian() line 58 in /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/SetupJacobian.c >> [8]PETSC ERROR: #8 SNESComputeJacobian() line 2193 in /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c >> [8]PETSC ERROR: #9 SNESSolve_NEWTONLS() line 230 in /home/rlchen/soft/petsc-3.5.2/src/snes/impls/ls/ls.c >> [8]PETSC ERROR: #10 SNESSolve() line 3743 in /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c >> [8]PETSC ERROR: #11 SolveTimeDependent() line 758 in /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/Nwtun.c >> [8]PETSC ERROR: #12 main() line 417 in /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/Nwtun.c >> ------------------- >> >> Best, >> Rongliang >> >> On 02/17/2016 02:09 PM, Barry Smith wrote: >>> Yes, this is the type of output I was expecting. Now you need to produce it for a large case. >>> >>> Barry >>> >>>> On Feb 16, 2016, at 11:49 PM, Rongliang Chen wrote: >>>> >>>> Hi Barry, >>>> >>>> I run the code with valgrind on a workstation for a smaller case and it produces some ASCII information (see attached). Is this helpful? >>>> >>>> Best, >>>> Rongliang >>>> >>>> On 02/17/2016 01:30 PM, Barry Smith wrote: >>>>> Hmm, something didn't work right with massif. I should give a bunch of ASCII information about how much memory is used at different times in the code. You may need to play around with the massif options and google documentation on how to get it to provide useful information. Once it produces the useful information it will be very helpful. >>>>> >>>>> time=0 >>>>> mem_heap_B=0 >>>>> mem_heap_extra_B=0 >>>>> mem_stacks_B=0 >>>>> heap_tree=empty >>>>> >>>>> >>>>>> On Feb 16, 2016, at 9:44 PM, Rongliang Chen wrote: >>>>>> >>>>>> Dear Barry, >>>>>> >>>>>> Many thanks for your reply. >>>>>> >>>>>> I checked with the valgrind and did not obtain any outputs (massif.out.) because the job was killed before it reached the end. >>>>>> >>>>>> Then I switch to a smaller case, it works well and one of the output is attached (I did not find any useful information in it). The output with the option -mat_coloring_view is followed, which shows that the number of colors is 65. Any ideas for this? >>>>>> >>>>>> MatColoring Object: 480 MPI processes >>>>>> type: sl >>>>>> Weight type: RANDOM >>>>>> Distance 2, Max. Colors 65535 >>>>>> Number of colors 65 >>>>>> Number of total columns 1637350 >>>>>> >>>>>> Best regards, >>>>>> Rongliang >>>>>> >>>>>> On 02/17/2016 01:13 AM, Barry Smith wrote: >>>>>>> How many colors are needed? >>>>>>> >>>>>>> You need to produce a breakdown of where all the memory is being used. For example valgrind with the >>>>>>> http://valgrind.org/docs/manual/ms-manual.html >>>>>>> >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>> >>>>>>>> On Feb 16, 2016, at 6:45 AM, Rongliang Chen >>>>>>>> wrote: >>>>>>>> >>>>>>>> Dear all, >>>>>>>> >>>>>>>> I am using the DMPlex to solve a PDE on a unstructured mesh and I use the SNESComputeJacobianDefaultColor to compute the Jacobian matrix. >>>>>>>> >>>>>>>> My code works well for small problems (such as problem with 3.3x10^5 cells using 120 cores) but when I increase the number of cells (2.6x10^6 cells using 1920 cores), I go out of memory in the function MatColoringApply. It shows that one of the cores uses over 11G memory. I think this is unreasonable. Do you have any suggestions for debugging this problem? >>>>>>>> >>>>>>>> Best regards, >>>>>>>> Rongliang >>>>>>>> >>>>>> >>>> >> From bsmith at mcs.anl.gov Wed Feb 17 20:31:41 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 17 Feb 2016 20:31:41 -0600 Subject: [petsc-users] memory scalable AO In-Reply-To: <4A7DAAD2-9FE3-482D-AACE-604F89BB1B80@gmail.com> References: <0DD3688F-1E57-4AC7-89A9-DD96B6A676FA@mcs.anl.gov> <4A7DAAD2-9FE3-482D-AACE-604F89BB1B80@gmail.com> Message-ID: <9AC44E2C-39E2-47A7-86BA-AF5723C5EB9E@mcs.anl.gov> It is probably my fault. Edit src/dm/impls/da/ftn-auto/daindexf.c and remove the function function from that file and then run make on the library again. If that does not work then run in the debugger and put a break point in dmdasetaotype_ once it is there in the debugger you can make sure it is the right function and step until it crashes to see what I have done wrong. Baryr > On Feb 17, 2016, at 7:29 PM, Randall Mackie wrote: > > Unfortunately I am getting exactly the same result. > > I applied the patch (patch -p1 < fix-fortran-dmdasetaotype.patch) > > Then I recompiled the library, and my code, but the error remains. > > Perhaps there is something else I?m missing? > > Randy > > >> On Feb 17, 2016, at 3:15 PM, Barry Smith wrote: >> >> >> Here is patch. >> >> If it works for you I'll put it in maint and master tomorrow. >> >> Barry >> >>> On Feb 17, 2016, at 3:46 PM, Randall Mackie wrote: >>> >>> The attached test program demonstrates the problem. When I run it, I get the following output: >>> >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>> [0]PETSC ERROR: likely location of problem given in stack below >>> [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>> [0]PETSC ERROR: is given. >>> [0]PETSC ERROR: [0] PetscStrlen line 150 /home/rmackie/PETSc/petsc-3.6.3/src/sys/utils/str.c >>> [0]PETSC ERROR: [0] PetscStrallocpy line 185 /home/rmackie/PETSc/petsc-3.6.3/src/sys/utils/str.c >>> [0]PETSC ERROR: [0] DMDASetAOType line 84 /home/rmackie/PETSc/petsc-3.6.3/src/dm/impls/da/daindex.c >>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>> [0]PETSC ERROR: Signal received >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>> >>> >>> Randy >>> >>> >>>> On Feb 17, 2016, at 11:35 AM, Barry Smith wrote: >>>> >>>> >>>> Should be ok. Do you have implicit none and the correct include files so AOMEMORYSCALABLE is defined? >>>> >>>> I think you need to run in the debugger next to track why this happens. >>>> >>>> Barry >>>> >>>>> On Feb 17, 2016, at 11:33 AM, Randall Mackie wrote: >>>>> >>>>> What is the correct way to set the AO for a DMDA to be the memory scalable version? >>>>> >>>>> I have tried this: >>>>> >>>>> call DMDASetAOType(da,AOMEMORYSCALABLE,ierr) >>>>> call DMDAGetAO(da,ao,ierr) >>>>> >>>>> The code compiles fine, but I simply get a Segmentation Violation when I run it: >>>>> >>>>> [3]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [3]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>>>> [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>> [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>>>> [3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>> [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run >>>>> [3]PETSC ERROR: to get more information on the crash. >>>>> [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>> [3]PETSC ERROR: Signal received >>>>> [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>>>> [3]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>>>> [3]PETSC ERROR: Configure options PETSC_ARCH=linux-gfortran-opt --with-clean=1 --with-scalar-type=complex --with-debugging=0 --with-fortran=1 --download-mpich=./mpich-3.1.3.tar.gz --FOPTFLAGS=-O3 --COPTFLAGS=-O3 >>>>> [3]PETSC ERROR: #1 User provided function() line 0 in unknown file >>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 3 >>>>> [cli_3]: aborting job: >>>>> >>>>> >>>>> In fact, I get the same thing even if I set the AOType to AOBASIC, in other words >>>>> >>>>> call DMDASetAOTYPE(da,AOBASIC,ierr) >>>>> >>>>> >>>>> Previously, I did not set an AOType, and only called DMDAGetAO, which works fine, and from what I can tell sets the type to basic. >>>>> >>>>> >>>>> Thanks, Randy M. >>>> >>> >> > From bsmith at mcs.anl.gov Wed Feb 17 20:33:24 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 17 Feb 2016 20:33:24 -0600 Subject: [petsc-users] Slow MatAssembly and MatMat Mult In-Reply-To: References: Message-ID: <4AB6ECA9-5C8D-4C85-BA59-BBE1023554E1@mcs.anl.gov> This is an absolutely tiny problem for 480 processors I am not surprised by the terrible performance. You should run this sub calculation on a small subset of the processes. Barry > On Feb 17, 2016, at 7:03 PM, Bikash Kanungo wrote: > > Hi, > > I have two small (2500x2500) matrices parallelized across 480 processors. One of them is an MPIAIJ matrix while the other is an MPIDENSE matrix. I perform a MatMatMult involving these two matrices. I tried these operations on two machines, one is the local cluster at University of Michigan and the other is the XSEDE Comet machine. The Comet machine takes 10-20 times more time in steps involving MatAssembly and MatMatMult of the aforementioned matrices. I have other Petsc MatMult operations in the same code involving larger matrices (4 million x 4 million) which show similar timing on both machines. It's just those small parallel matrices that are inconsistent in terms of their timings. I used same the compilers and MPI libraries in both the machines except that I have suppressed "avx2" flag in Comet. I believe avx2 affects floating point operations and not communication. I would like to know what might be causing these inconsistencies only in case of the small matrices. Are there any network settings that I can look into and compare? > > Regards, > Bikash > > -- > Bikash S. Kanungo > PhD Student > Computational Materials Physics Group > Mechanical Engineering > University of Michigan > From rlmackie862 at gmail.com Wed Feb 17 20:58:56 2016 From: rlmackie862 at gmail.com (Randall Mackie) Date: Wed, 17 Feb 2016 18:58:56 -0800 Subject: [petsc-users] memory scalable AO In-Reply-To: <9AC44E2C-39E2-47A7-86BA-AF5723C5EB9E@mcs.anl.gov> References: <0DD3688F-1E57-4AC7-89A9-DD96B6A676FA@mcs.anl.gov> <4A7DAAD2-9FE3-482D-AACE-604F89BB1B80@gmail.com> <9AC44E2C-39E2-47A7-86BA-AF5723C5EB9E@mcs.anl.gov> Message-ID: <60AAF166-8B9E-4365-B68A-81BB35192BD3@gmail.com> If I remove lines 47-50 in daindexf.c (the PETSC_EXTERN ?for dmdasetaotype_), then when I try to compile the test code, it complains of an undefined reference to dmdasetaotype_. So I put the lines back in, and with your patch applied, and then run in the debugger, with a breakpoint at dmdasetaotype_, it stops at /src/dm/impls/da/ftn-auto/daindexf.c. 3 steps later, at *__ierr = DMDASetAOType( is where it bombs out. So it seems to be ignoring the dmdasetaotype_ in /src/dm/impls/da/ftn-custom/zdaindexf.c in favor of ../ftn-auto/daindexf.c Randy > On Feb 17, 2016, at 6:31 PM, Barry Smith wrote: > > > It is probably my fault. Edit src/dm/impls/da/ftn-auto/daindexf.c and remove the function function from that file and then run make on the library again. If that does not work then run in the debugger and put a break point in dmdasetaotype_ once it is there in the debugger you can make sure it is the right function and step until it crashes to see what I have done wrong. > > Baryr > >> On Feb 17, 2016, at 7:29 PM, Randall Mackie wrote: >> >> Unfortunately I am getting exactly the same result. >> >> I applied the patch (patch -p1 < fix-fortran-dmdasetaotype.patch) >> >> Then I recompiled the library, and my code, but the error remains. >> >> Perhaps there is something else I?m missing? >> >> Randy >> >> >>> On Feb 17, 2016, at 3:15 PM, Barry Smith wrote: >>> >>> >>> Here is patch. >>> >>> If it works for you I'll put it in maint and master tomorrow. >>> >>> Barry >>> >>>> On Feb 17, 2016, at 3:46 PM, Randall Mackie wrote: >>>> >>>> The attached test program demonstrates the problem. When I run it, I get the following output: >>>> >>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>> [0]PETSC ERROR: likely location of problem given in stack below >>>> [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>> [0]PETSC ERROR: is given. >>>> [0]PETSC ERROR: [0] PetscStrlen line 150 /home/rmackie/PETSc/petsc-3.6.3/src/sys/utils/str.c >>>> [0]PETSC ERROR: [0] PetscStrallocpy line 185 /home/rmackie/PETSc/petsc-3.6.3/src/sys/utils/str.c >>>> [0]PETSC ERROR: [0] DMDASetAOType line 84 /home/rmackie/PETSc/petsc-3.6.3/src/dm/impls/da/daindex.c >>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>> [0]PETSC ERROR: Signal received >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>>> >>>> >>>> Randy >>>> >>>> >>>>> On Feb 17, 2016, at 11:35 AM, Barry Smith wrote: >>>>> >>>>> >>>>> Should be ok. Do you have implicit none and the correct include files so AOMEMORYSCALABLE is defined? >>>>> >>>>> I think you need to run in the debugger next to track why this happens. >>>>> >>>>> Barry >>>>> >>>>>> On Feb 17, 2016, at 11:33 AM, Randall Mackie wrote: >>>>>> >>>>>> What is the correct way to set the AO for a DMDA to be the memory scalable version? >>>>>> >>>>>> I have tried this: >>>>>> >>>>>> call DMDASetAOType(da,AOMEMORYSCALABLE,ierr) >>>>>> call DMDAGetAO(da,ao,ierr) >>>>>> >>>>>> The code compiles fine, but I simply get a Segmentation Violation when I run it: >>>>>> >>>>>> [3]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [3]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>>>>> [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>>> [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>>>>> [3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>>> [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run >>>>>> [3]PETSC ERROR: to get more information on the crash. >>>>>> [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>> [3]PETSC ERROR: Signal received >>>>>> [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>>>>> [3]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>>>>> [3]PETSC ERROR: Configure options PETSC_ARCH=linux-gfortran-opt --with-clean=1 --with-scalar-type=complex --with-debugging=0 --with-fortran=1 --download-mpich=./mpich-3.1.3.tar.gz --FOPTFLAGS=-O3 --COPTFLAGS=-O3 >>>>>> [3]PETSC ERROR: #1 User provided function() line 0 in unknown file >>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 3 >>>>>> [cli_3]: aborting job: >>>>>> >>>>>> >>>>>> In fact, I get the same thing even if I set the AOType to AOBASIC, in other words >>>>>> >>>>>> call DMDASetAOTYPE(da,AOBASIC,ierr) >>>>>> >>>>>> >>>>>> Previously, I did not set an AOType, and only called DMDAGetAO, which works fine, and from what I can tell sets the type to basic. >>>>>> >>>>>> >>>>>> Thanks, Randy M. >>>>> >>>> >>> >> > From bsmith at mcs.anl.gov Wed Feb 17 21:01:53 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 17 Feb 2016 21:01:53 -0600 Subject: [petsc-users] memory scalable AO In-Reply-To: <60AAF166-8B9E-4365-B68A-81BB35192BD3@gmail.com> References: <0DD3688F-1E57-4AC7-89A9-DD96B6A676FA@mcs.anl.gov> <4A7DAAD2-9FE3-482D-AACE-604F89BB1B80@gmail.com> <9AC44E2C-39E2-47A7-86BA-AF5723C5EB9E@mcs.anl.gov> <60AAF166-8B9E-4365-B68A-81BB35192BD3@gmail.com> Message-ID: <521A447E-6EB0-4FA8-B4EF-C89EE9D6137F@mcs.anl.gov> > On Feb 17, 2016, at 8:58 PM, Randall Mackie wrote: > > If I remove lines 47-50 in daindexf.c (the PETSC_EXTERN ?for dmdasetaotype_), then when I try to compile the test code, it complains of an undefined reference to dmdasetaotype_. > > So I put the lines back in, and with your patch applied, and then run in the debugger, with a breakpoint at dmdasetaotype_, it stops at /src/dm/impls/da/ftn-auto/daindexf.c. > > 3 steps later, at *__ierr = DMDASetAOType( > > is where it bombs out. > > > So it seems to be ignoring the dmdasetaotype_ in /src/dm/impls/da/ftn-custom/zdaindexf.c in favor of ../ftn-auto/daindexf.c This makes no sense. Ok just try pasting the new definition in daindexf.c. instead of the old one, compile and run. Barry > > Randy > > >> On Feb 17, 2016, at 6:31 PM, Barry Smith wrote: >> >> >> It is probably my fault. Edit src/dm/impls/da/ftn-auto/daindexf.c and remove the function function from that file and then run make on the library again. If that does not work then run in the debugger and put a break point in dmdasetaotype_ once it is there in the debugger you can make sure it is the right function and step until it crashes to see what I have done wrong. >> >> Baryr >> >>> On Feb 17, 2016, at 7:29 PM, Randall Mackie wrote: >>> >>> Unfortunately I am getting exactly the same result. >>> >>> I applied the patch (patch -p1 < fix-fortran-dmdasetaotype.patch) >>> >>> Then I recompiled the library, and my code, but the error remains. >>> >>> Perhaps there is something else I?m missing? >>> >>> Randy >>> >>> >>>> On Feb 17, 2016, at 3:15 PM, Barry Smith wrote: >>>> >>>> >>>> Here is patch. >>>> >>>> If it works for you I'll put it in maint and master tomorrow. >>>> >>>> Barry >>>> >>>>> On Feb 17, 2016, at 3:46 PM, Randall Mackie wrote: >>>>> >>>>> The attached test program demonstrates the problem. When I run it, I get the following output: >>>>> >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>> [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>>> [0]PETSC ERROR: is given. >>>>> [0]PETSC ERROR: [0] PetscStrlen line 150 /home/rmackie/PETSc/petsc-3.6.3/src/sys/utils/str.c >>>>> [0]PETSC ERROR: [0] PetscStrallocpy line 185 /home/rmackie/PETSc/petsc-3.6.3/src/sys/utils/str.c >>>>> [0]PETSC ERROR: [0] DMDASetAOType line 84 /home/rmackie/PETSc/petsc-3.6.3/src/dm/impls/da/daindex.c >>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>> [0]PETSC ERROR: Signal received >>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>>>> >>>>> >>>>> Randy >>>>> >>>>> >>>>>> On Feb 17, 2016, at 11:35 AM, Barry Smith wrote: >>>>>> >>>>>> >>>>>> Should be ok. Do you have implicit none and the correct include files so AOMEMORYSCALABLE is defined? >>>>>> >>>>>> I think you need to run in the debugger next to track why this happens. >>>>>> >>>>>> Barry >>>>>> >>>>>>> On Feb 17, 2016, at 11:33 AM, Randall Mackie wrote: >>>>>>> >>>>>>> What is the correct way to set the AO for a DMDA to be the memory scalable version? >>>>>>> >>>>>>> I have tried this: >>>>>>> >>>>>>> call DMDASetAOType(da,AOMEMORYSCALABLE,ierr) >>>>>>> call DMDAGetAO(da,ao,ierr) >>>>>>> >>>>>>> The code compiles fine, but I simply get a Segmentation Violation when I run it: >>>>>>> >>>>>>> [3]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>> [3]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>>>>>> [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>>>> [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>>>>>> [3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>>>> [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run >>>>>>> [3]PETSC ERROR: to get more information on the crash. >>>>>>> [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>>> [3]PETSC ERROR: Signal received >>>>>>> [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>>>>>> [3]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>>>>>> [3]PETSC ERROR: Configure options PETSC_ARCH=linux-gfortran-opt --with-clean=1 --with-scalar-type=complex --with-debugging=0 --with-fortran=1 --download-mpich=./mpich-3.1.3.tar.gz --FOPTFLAGS=-O3 --COPTFLAGS=-O3 >>>>>>> [3]PETSC ERROR: #1 User provided function() line 0 in unknown file >>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 3 >>>>>>> [cli_3]: aborting job: >>>>>>> >>>>>>> >>>>>>> In fact, I get the same thing even if I set the AOType to AOBASIC, in other words >>>>>>> >>>>>>> call DMDASetAOTYPE(da,AOBASIC,ierr) >>>>>>> >>>>>>> >>>>>>> Previously, I did not set an AOType, and only called DMDAGetAO, which works fine, and from what I can tell sets the type to basic. >>>>>>> >>>>>>> >>>>>>> Thanks, Randy M. >>>>>> >>>>> >>>> >>> >> > From jed at jedbrown.org Wed Feb 17 21:05:16 2016 From: jed at jedbrown.org (Jed Brown) Date: Wed, 17 Feb 2016 20:05:16 -0700 Subject: [petsc-users] memory scalable AO In-Reply-To: <60AAF166-8B9E-4365-B68A-81BB35192BD3@gmail.com> References: <0DD3688F-1E57-4AC7-89A9-DD96B6A676FA@mcs.anl.gov> <4A7DAAD2-9FE3-482D-AACE-604F89BB1B80@gmail.com> <9AC44E2C-39E2-47A7-86BA-AF5723C5EB9E@mcs.anl.gov> <60AAF166-8B9E-4365-B68A-81BB35192BD3@gmail.com> Message-ID: <87twl6ra4z.fsf@jedbrown.org> Randall Mackie writes: > So it seems to be ignoring the dmdasetaotype_ in /src/dm/impls/da/ftn-custom/zdaindexf.c in favor of ../ftn-auto/daindexf.c Sounds like you need to generate your Fortran stubs so that the version in ftn-auto/daindexf.c is fixed. make allfortranstubs -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From bikash at umich.edu Wed Feb 17 21:17:55 2016 From: bikash at umich.edu (Bikash Kanungo) Date: Wed, 17 Feb 2016 22:17:55 -0500 Subject: [petsc-users] Slow MatAssembly and MatMat Mult In-Reply-To: <4AB6ECA9-5C8D-4C85-BA59-BBE1023554E1@mcs.anl.gov> References: <4AB6ECA9-5C8D-4C85-BA59-BBE1023554E1@mcs.anl.gov> Message-ID: Hi Barry, I had thought of using sub-communicator for these operations. But the matrix entries have contributions from all the processors. Moreover, after the MatMatMult operation is done, I need to retrieve certain values from the resultant matrix through non-local calls (MatGetSubMatrices) on all processors. Defining these matrices to be residing on sub-communicator will prohibit me from adding contributions and calling MatGetSubMatrices from processors not within the sub-communicator. What would be a good workaround for it? Regards, Bikash On Wed, Feb 17, 2016 at 9:33 PM, Barry Smith wrote: > > This is an absolutely tiny problem for 480 processors I am not surprised > by the terrible performance. You should run this sub calculation on a small > subset of the processes. > > Barry > > > On Feb 17, 2016, at 7:03 PM, Bikash Kanungo wrote: > > > > Hi, > > > > I have two small (2500x2500) matrices parallelized across 480 > processors. One of them is an MPIAIJ matrix while the other is an MPIDENSE > matrix. I perform a MatMatMult involving these two matrices. I tried these > operations on two machines, one is the local cluster at University of > Michigan and the other is the XSEDE Comet machine. The Comet machine takes > 10-20 times more time in steps involving MatAssembly and MatMatMult of the > aforementioned matrices. I have other Petsc MatMult operations in the same > code involving larger matrices (4 million x 4 million) which show similar > timing on both machines. It's just those small parallel matrices that are > inconsistent in terms of their timings. I used same the compilers and MPI > libraries in both the machines except that I have suppressed "avx2" flag in > Comet. I believe avx2 affects floating point operations and not > communication. I would like to know what might be causing these > inconsistencies only in case of the small matrices. Are there any network > settings that I can look into and compare? > > > > Regards, > > Bikash > > > > -- > > Bikash S. Kanungo > > PhD Student > > Computational Materials Physics Group > > Mechanical Engineering > > University of Michigan > > > > -- Bikash S. Kanungo PhD Student Computational Materials Physics Group Mechanical Engineering University of Michigan -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Feb 17 21:30:34 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 17 Feb 2016 21:30:34 -0600 Subject: [petsc-users] Slow MatAssembly and MatMat Mult In-Reply-To: References: <4AB6ECA9-5C8D-4C85-BA59-BBE1023554E1@mcs.anl.gov> Message-ID: You need to bite the bullet and do the communication needed to get the info on the sub communicator and then get the result back out to the entire communicator. If the two matrices are generated on MPI_COMM_WORLD you can use MatCreateRedundantMatrix() to get entire copies of them on sub communicators. You can actually then have each sub communicator do a certain number of multiplies of the sparse matrix with different columns of the dense matrix so that instead of having one sub communicator do 2500 sparse matrix vector products you can have each sub communicator (say you have 5 of them) do 500 sparse matrix vector products (giving two levels of parallelism). The results are dense matrices so you would need to write some code to get the parts of the resulting dense matrices back to the processes where you want it. I would suggest using MPI calls directly for this, I don't PETSc has anything particularly useful to do that. Barry > On Feb 17, 2016, at 9:17 PM, Bikash Kanungo wrote: > > Hi Barry, > > I had thought of using sub-communicator for these operations. But the matrix entries have contributions from all the processors. Moreover, after the MatMatMult operation is done, I need to retrieve certain values from the resultant matrix through non-local calls (MatGetSubMatrices) on all processors. Defining these matrices to be residing on sub-communicator will prohibit me from adding contributions and calling MatGetSubMatrices from processors not within the sub-communicator. What would be a good workaround for it? > > Regards, > Bikash > > On Wed, Feb 17, 2016 at 9:33 PM, Barry Smith wrote: > > This is an absolutely tiny problem for 480 processors I am not surprised by the terrible performance. You should run this sub calculation on a small subset of the processes. > > Barry > > > On Feb 17, 2016, at 7:03 PM, Bikash Kanungo wrote: > > > > Hi, > > > > I have two small (2500x2500) matrices parallelized across 480 processors. One of them is an MPIAIJ matrix while the other is an MPIDENSE matrix. I perform a MatMatMult involving these two matrices. I tried these operations on two machines, one is the local cluster at University of Michigan and the other is the XSEDE Comet machine. The Comet machine takes 10-20 times more time in steps involving MatAssembly and MatMatMult of the aforementioned matrices. I have other Petsc MatMult operations in the same code involving larger matrices (4 million x 4 million) which show similar timing on both machines. It's just those small parallel matrices that are inconsistent in terms of their timings. I used same the compilers and MPI libraries in both the machines except that I have suppressed "avx2" flag in Comet. I believe avx2 affects floating point operations and not communication. I would like to know what might be causing these inconsistencies only in case of the small matrices. Are there any network settings that I can look into and compare? > > > > Regards, > > Bikash > > > > -- > > Bikash S. Kanungo > > PhD Student > > Computational Materials Physics Group > > Mechanical Engineering > > University of Michigan > > > > > > > -- > Bikash S. Kanungo > PhD Student > Computational Materials Physics Group > Mechanical Engineering > University of Michigan > From bikash at umich.edu Wed Feb 17 21:52:44 2016 From: bikash at umich.edu (Bikash Kanungo) Date: Wed, 17 Feb 2016 22:52:44 -0500 Subject: [petsc-users] Slow MatAssembly and MatMat Mult In-Reply-To: References: <4AB6ECA9-5C8D-4C85-BA59-BBE1023554E1@mcs.anl.gov> Message-ID: Thanks a lot Barry. I guess it's time to bite the bullet. Regards, Bikash On Wed, Feb 17, 2016 at 10:30 PM, Barry Smith wrote: > > You need to bite the bullet and do the communication needed to get the > info on the sub communicator and then get the result back out to the entire > communicator. > > If the two matrices are generated on MPI_COMM_WORLD you can use > MatCreateRedundantMatrix() to get entire copies of them on sub > communicators. You can actually then have each sub communicator do a > certain number of multiplies of the sparse matrix with different columns of > the dense matrix so that instead of having one sub communicator do 2500 > sparse matrix vector products you can have each sub communicator (say you > have 5 of them) do 500 sparse matrix vector products (giving two levels of > parallelism). The results are dense matrices so you would need to write > some code to get the parts of the resulting dense matrices back to the > processes where you want it. I would suggest using MPI calls directly for > this, I don't PETSc has anything particularly useful to do that. > > Barry > > > > On Feb 17, 2016, at 9:17 PM, Bikash Kanungo wrote: > > > > Hi Barry, > > > > I had thought of using sub-communicator for these operations. But the > matrix entries have contributions from all the processors. Moreover, after > the MatMatMult operation is done, I need to retrieve certain values from > the resultant matrix through non-local calls (MatGetSubMatrices) on all > processors. Defining these matrices to be residing on sub-communicator will > prohibit me from adding contributions and calling MatGetSubMatrices from > processors not within the sub-communicator. What would be a good workaround > for it? > > > > Regards, > > Bikash > > > > On Wed, Feb 17, 2016 at 9:33 PM, Barry Smith wrote: > > > > This is an absolutely tiny problem for 480 processors I am not > surprised by the terrible performance. You should run this sub calculation > on a small subset of the processes. > > > > Barry > > > > > On Feb 17, 2016, at 7:03 PM, Bikash Kanungo wrote: > > > > > > Hi, > > > > > > I have two small (2500x2500) matrices parallelized across 480 > processors. One of them is an MPIAIJ matrix while the other is an MPIDENSE > matrix. I perform a MatMatMult involving these two matrices. I tried these > operations on two machines, one is the local cluster at University of > Michigan and the other is the XSEDE Comet machine. The Comet machine takes > 10-20 times more time in steps involving MatAssembly and MatMatMult of the > aforementioned matrices. I have other Petsc MatMult operations in the same > code involving larger matrices (4 million x 4 million) which show similar > timing on both machines. It's just those small parallel matrices that are > inconsistent in terms of their timings. I used same the compilers and MPI > libraries in both the machines except that I have suppressed "avx2" flag in > Comet. I believe avx2 affects floating point operations and not > communication. I would like to know what might be causing these > inconsistencies only in case of the small matrices. Are there any network > settings that I can look into and compare? > > > > > > Regards, > > > Bikash > > > > > > -- > > > Bikash S. Kanungo > > > PhD Student > > > Computational Materials Physics Group > > > Mechanical Engineering > > > University of Michigan > > > > > > > > > > > > > -- > > Bikash S. Kanungo > > PhD Student > > Computational Materials Physics Group > > Mechanical Engineering > > University of Michigan > > > > -- Bikash S. Kanungo PhD Student Computational Materials Physics Group Mechanical Engineering University of Michigan -------------- next part -------------- An HTML attachment was scrubbed... URL: From rlmackie862 at gmail.com Wed Feb 17 22:29:31 2016 From: rlmackie862 at gmail.com (Randall Mackie) Date: Wed, 17 Feb 2016 20:29:31 -0800 Subject: [petsc-users] memory scalable AO In-Reply-To: <87twl6ra4z.fsf@jedbrown.org> References: <0DD3688F-1E57-4AC7-89A9-DD96B6A676FA@mcs.anl.gov> <4A7DAAD2-9FE3-482D-AACE-604F89BB1B80@gmail.com> <9AC44E2C-39E2-47A7-86BA-AF5723C5EB9E@mcs.anl.gov> <60AAF166-8B9E-4365-B68A-81BB35192BD3@gmail.com> <87twl6ra4z.fsf@jedbrown.org> Message-ID: <23E95BA9-BD97-4EAC-BCE1-5E8A92BE1E25@gmail.com> this leads to the error ?bin/maint/generatefortranstubs.py? ?No such file. there is no maint directory under bin. Randy > On Feb 17, 2016, at 7:05 PM, Jed Brown wrote: > > Randall Mackie writes: >> So it seems to be ignoring the dmdasetaotype_ in /src/dm/impls/da/ftn-custom/zdaindexf.c in favor of ../ftn-auto/daindexf.c > > Sounds like you need to generate your Fortran stubs so that the version > in ftn-auto/daindexf.c is fixed. > > make allfortranstubs From jed at jedbrown.org Wed Feb 17 22:45:07 2016 From: jed at jedbrown.org (Jed Brown) Date: Wed, 17 Feb 2016 21:45:07 -0700 Subject: [petsc-users] memory scalable AO In-Reply-To: <23E95BA9-BD97-4EAC-BCE1-5E8A92BE1E25@gmail.com> References: <0DD3688F-1E57-4AC7-89A9-DD96B6A676FA@mcs.anl.gov> <4A7DAAD2-9FE3-482D-AACE-604F89BB1B80@gmail.com> <9AC44E2C-39E2-47A7-86BA-AF5723C5EB9E@mcs.anl.gov> <60AAF166-8B9E-4365-B68A-81BB35192BD3@gmail.com> <87twl6ra4z.fsf@jedbrown.org> <23E95BA9-BD97-4EAC-BCE1-5E8A92BE1E25@gmail.com> Message-ID: <87lh6ir5ik.fsf@jedbrown.org> Randall Mackie writes: > this leads to the error ?bin/maint/generatefortranstubs.py? ?No such file. > > there is no maint directory under bin. Ah, I missed that you have a tarball, not a clone of the repository. Then you'll have to fix the file manually. I would start from a clean tarball because we have no idea what state your source tree is in after applying the match and manually editing files. Note that this would be easy and completely reliable if you used a clone instead of tarball. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From balay at mcs.anl.gov Wed Feb 17 22:54:20 2016 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 17 Feb 2016 22:54:20 -0600 Subject: [petsc-users] memory scalable AO In-Reply-To: <87lh6ir5ik.fsf@jedbrown.org> References: <0DD3688F-1E57-4AC7-89A9-DD96B6A676FA@mcs.anl.gov> <4A7DAAD2-9FE3-482D-AACE-604F89BB1B80@gmail.com> <9AC44E2C-39E2-47A7-86BA-AF5723C5EB9E@mcs.anl.gov> <60AAF166-8B9E-4365-B68A-81BB35192BD3@gmail.com> <87twl6ra4z.fsf@jedbrown.org> <23E95BA9-BD97-4EAC-BCE1-5E8A92BE1E25@gmail.com> <87lh6ir5ik.fsf@jedbrown.org> Message-ID: Attached are the modified src/dm/impls/da/ftn-auto/daindexf.c and src/dm/impls/da/ftn-custom/zdaindexf.c files. Satish On Wed, 17 Feb 2016, Jed Brown wrote: > Randall Mackie writes: > > > this leads to the error ?bin/maint/generatefortranstubs.py? ?No such file. > > > > there is no maint directory under bin. > > Ah, I missed that you have a tarball, not a clone of the repository. > Then you'll have to fix the file manually. I would start from a clean > tarball because we have no idea what state your source tree is in after > applying the match and manually editing files. Note that this would be > easy and completely reliable if you used a clone instead of tarball. > -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: daindexf.c URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: zdaindexf.c URL: From rlmackie862 at gmail.com Wed Feb 17 23:41:13 2016 From: rlmackie862 at gmail.com (Randall Mackie) Date: Wed, 17 Feb 2016 21:41:13 -0800 Subject: [petsc-users] memory scalable AO In-Reply-To: References: <0DD3688F-1E57-4AC7-89A9-DD96B6A676FA@mcs.anl.gov> <4A7DAAD2-9FE3-482D-AACE-604F89BB1B80@gmail.com> <9AC44E2C-39E2-47A7-86BA-AF5723C5EB9E@mcs.anl.gov> <60AAF166-8B9E-4365-B68A-81BB35192BD3@gmail.com> <87twl6ra4z.fsf@jedbrown.org> <23E95BA9-BD97-4EAC-BCE1-5E8A92BE1E25@gmail.com> <87lh6ir5ik.fsf@jedbrown.org> Message-ID: Satish, It is unclear from this email exchange if I am still suppose to do make allfortranstubs after updating these two files, or if these are the files that are suppose to result from that make operation. Although these files looked the same as what I tried before with Barry?s suggestions, I dropped these in, recompiled petsc, but again, when I try to compile the test code I sent, it still complains about an unresolved symbol to dmdasetaotype_. If I still need to do the make allfortranstubs, I?ll do that via a git clone tomorrow. Thanks, Randy > On Feb 17, 2016, at 8:54 PM, Satish Balay wrote: > > Attached are the modified src/dm/impls/da/ftn-auto/daindexf.c and > src/dm/impls/da/ftn-custom/zdaindexf.c files. > > Satish > > On Wed, 17 Feb 2016, Jed Brown wrote: > >> Randall Mackie writes: >> >>> this leads to the error ?bin/maint/generatefortranstubs.py? ?No such file. >>> >>> there is no maint directory under bin. >> >> Ah, I missed that you have a tarball, not a clone of the repository. >> Then you'll have to fix the file manually. I would start from a clean >> tarball because we have no idea what state your source tree is in after >> applying the match and manually editing files. Note that this would be >> easy and completely reliable if you used a clone instead of tarball. >> > From bsmith at mcs.anl.gov Wed Feb 17 23:50:31 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 17 Feb 2016 23:50:31 -0600 Subject: [petsc-users] memory scalable AO In-Reply-To: References: <0DD3688F-1E57-4AC7-89A9-DD96B6A676FA@mcs.anl.gov> <4A7DAAD2-9FE3-482D-AACE-604F89BB1B80@gmail.com> <9AC44E2C-39E2-47A7-86BA-AF5723C5EB9E@mcs.anl.gov> <60AAF166-8B9E-4365-B68A-81BB35192BD3@gmail.com> <87twl6ra4z.fsf@jedbrown.org> <23E95BA9-BD97-4EAC-BCE1-5E8A92BE1E25@gmail.com> <87lh6ir5ik.fsf@jedbrown.org> Message-ID: Randy, It is not suppose to be this hard! When you ran the make gnumake after you applied the patch did it recompile the file zdaindexf.c (in fact that file and daindexf.c should be the only files that changed and hence the only files that got recompiled). Barry > On Feb 17, 2016, at 11:41 PM, Randall Mackie wrote: > > Satish, > > It is unclear from this email exchange if I am still suppose to do make allfortranstubs after updating these two files, or if these are the files that are suppose to result from that make operation. > > Although these files looked the same as what I tried before with Barry?s suggestions, I dropped these in, recompiled petsc, but again, when I try to compile the test code I sent, it still complains about an unresolved symbol to dmdasetaotype_. > > If I still need to do the make allfortranstubs, I?ll do that via a git clone tomorrow. > > Thanks, Randy > > >> On Feb 17, 2016, at 8:54 PM, Satish Balay wrote: >> >> Attached are the modified src/dm/impls/da/ftn-auto/daindexf.c and >> src/dm/impls/da/ftn-custom/zdaindexf.c files. >> >> Satish >> >> On Wed, 17 Feb 2016, Jed Brown wrote: >> >>> Randall Mackie writes: >>> >>>> this leads to the error ?bin/maint/generatefortranstubs.py? ?No such file. >>>> >>>> there is no maint directory under bin. >>> >>> Ah, I missed that you have a tarball, not a clone of the repository. >>> Then you'll have to fix the file manually. I would start from a clean >>> tarball because we have no idea what state your source tree is in after >>> applying the match and manually editing files. Note that this would be >>> easy and completely reliable if you used a clone instead of tarball. >>> >> > From balay at mcs.anl.gov Wed Feb 17 23:58:35 2016 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 17 Feb 2016 23:58:35 -0600 Subject: [petsc-users] memory scalable AO In-Reply-To: References: <0DD3688F-1E57-4AC7-89A9-DD96B6A676FA@mcs.anl.gov> <4A7DAAD2-9FE3-482D-AACE-604F89BB1B80@gmail.com> <9AC44E2C-39E2-47A7-86BA-AF5723C5EB9E@mcs.anl.gov> <60AAF166-8B9E-4365-B68A-81BB35192BD3@gmail.com> <87twl6ra4z.fsf@jedbrown.org> <23E95BA9-BD97-4EAC-BCE1-5E8A92BE1E25@gmail.com> <87lh6ir5ik.fsf@jedbrown.org> Message-ID: Look like zdaindexf.c is not listed in makefile - so it never gets compiled. Also it has code for dmdagetglobalindices_() dmdarestoreglobalindices_(). But then src/docs/website/documentation/changes/35.html has: >>>>
  • DMDAGetGlobalIndices(DM,PetscInt*,const PetscInt*[]) and DMDARestoreGlobalIndices(DM,PetscInt*,const PetscInt*[]) are removed, use DMGetLocalToGlobalMapping() to get this information
    • <<<< So perhaps the attached modified patch should be used? [instead of the previous fix-fortran-dmdasetaotype.patch] patch -Np1 -R < fix-fortran-dmdasetaotype.patch patch -Np1 < dmdasetaotype-fixed.patch Might be easier to use 'maint' branch from git.. Satish On Wed, 17 Feb 2016, Barry Smith wrote: > > Randy, > > It is not suppose to be this hard! When you ran the make gnumake after you applied the patch did it recompile the file zdaindexf.c (in fact that file and daindexf.c should be the only files that changed and hence the only files that got recompiled). > > Barry > > > > > On Feb 17, 2016, at 11:41 PM, Randall Mackie wrote: > > > > Satish, > > > > It is unclear from this email exchange if I am still suppose to do make allfortranstubs after updating these two files, or if these are the files that are suppose to result from that make operation. > > > > Although these files looked the same as what I tried before with Barry?s suggestions, I dropped these in, recompiled petsc, but again, when I try to compile the test code I sent, it still complains about an unresolved symbol to dmdasetaotype_. > > > > If I still need to do the make allfortranstubs, I?ll do that via a git clone tomorrow. > > > > Thanks, Randy > > > > > >> On Feb 17, 2016, at 8:54 PM, Satish Balay wrote: > >> > >> Attached are the modified src/dm/impls/da/ftn-auto/daindexf.c and > >> src/dm/impls/da/ftn-custom/zdaindexf.c files. > >> > >> Satish > >> > >> On Wed, 17 Feb 2016, Jed Brown wrote: > >> > >>> Randall Mackie writes: > >>> > >>>> this leads to the error ?bin/maint/generatefortranstubs.py? ?No such file. > >>>> > >>>> there is no maint directory under bin. > >>> > >>> Ah, I missed that you have a tarball, not a clone of the repository. > >>> Then you'll have to fix the file manually. I would start from a clean > >>> tarball because we have no idea what state your source tree is in after > >>> applying the match and manually editing files. Note that this would be > >>> easy and completely reliable if you used a clone instead of tarball. > >>> > >> > > > > -------------- next part -------------- diff --git a/src/dm/impls/da/daindex.c b/src/dm/impls/da/daindex.c index 81d6121..24816e1 100644 --- a/src/dm/impls/da/daindex.c +++ b/src/dm/impls/da/daindex.c @@ -53,7 +53,7 @@ PetscErrorCode DMDAGetNatural_Private(DM da,PetscInt *outNlocal,IS *isnatural) #undef __FUNCT__ #define __FUNCT__ "DMDASetAOType" -/*@ +/*@C DMDASetAOType - Sets the type of application ordering for a distributed array. Collective on DMDA diff --git a/src/dm/impls/da/ftn-custom/makefile b/src/dm/impls/da/ftn-custom/makefile index deb3081..767b409 100644 --- a/src/dm/impls/da/ftn-custom/makefile +++ b/src/dm/impls/da/ftn-custom/makefile @@ -4,7 +4,7 @@ ALL: lib CFLAGS = FFLAGS = -SOURCEC = zdaf.c zda1f.c zda2f.c zda3f.c zdacornf.c zdagetscatterf.c zdaviewf.c +SOURCEC = zdaf.c zda1f.c zda2f.c zda3f.c zdacornf.c zdagetscatterf.c zdaviewf.c zdaindexf.c SOURCEF = SOURCEH = LIBBASE = libpetscdm diff --git a/src/dm/impls/da/ftn-custom/zdaindexf.c b/src/dm/impls/da/ftn-custom/zdaindexf.c index b5aa151..ab38d09 100644 --- a/src/dm/impls/da/ftn-custom/zdaindexf.c +++ b/src/dm/impls/da/ftn-custom/zdaindexf.c @@ -1,25 +1,16 @@ - #include #include #if defined(PETSC_HAVE_FORTRAN_CAPS) -#define dmdagetglobalindices_ DMDAGETGLOBALINDICES -#define dmdarestoreglobalindices_ DMDARESTOREGLOBALINDICES +#define dmdasetaotype_ DMDASETAOTYPE #elif !defined(PETSC_HAVE_FORTRAN_UNDERSCORE) #define dmdagetglobalindices_ dmdagetglobalindices -#define dmdarestoreglobalindices_ dmdarestoreglobalindices #endif -PETSC_EXTERN void PETSC_STDCALL dmdagetglobalindices_(DM *da,PetscInt *n,PetscInt *indices,size_t *ia,PetscErrorCode *ierr) +PETSC_EXTERN void PETSC_STDCALL dmdasetaotype_(DM *da,CHAR type PETSC_MIXED_LEN(len), PetscErrorCode *ierr PETSC_END_LEN(len) ) { - const PetscInt *idx; - *ierr = DMDAGetGlobalIndices(*da,n,&idx); - *ia = PetscIntAddressToFortran(indices,idx); + char *t; + FIXCHAR(type,len,t); + *ierr = DMDASetAOType(*da,t); + FREECHAR(type,t); } - -PETSC_EXTERN void PETSC_STDCALL dmdarestoreglobalindices_(DM *da,PetscInt *n,PetscInt *fa,size_t *ia,PetscErrorCode *ierr) -{ - const PetscInt *lx = PetscIntAddressFromFortran(fa,*ia); - *ierr = DMDARestoreGlobalIndices(*da,n,&lx); -} - From rlmackie862 at gmail.com Thu Feb 18 09:09:00 2016 From: rlmackie862 at gmail.com (Randall Mackie) Date: Thu, 18 Feb 2016 07:09:00 -0800 Subject: [petsc-users] memory scalable AO In-Reply-To: References: <0DD3688F-1E57-4AC7-89A9-DD96B6A676FA@mcs.anl.gov> <4A7DAAD2-9FE3-482D-AACE-604F89BB1B80@gmail.com> <9AC44E2C-39E2-47A7-86BA-AF5723C5EB9E@mcs.anl.gov> <60AAF166-8B9E-4365-B68A-81BB35192BD3@gmail.com> <87twl6ra4z.fsf@jedbrown.org> <23E95BA9-BD97-4EAC-BCE1-5E8A92BE1E25@gmail.com> <87lh6ir5ik.fsf@jedbrown.org> Message-ID: <0C1DBE27-7446-4D82-8176-525302EBDB23@gmail.com> Hi Satish, This patch works now. Thanks very much to all of you for your patience and help. Randy > On Feb 17, 2016, at 9:58 PM, Satish Balay wrote: > > Look like zdaindexf.c is not listed in makefile - so it never gets compiled. > > Also it has code for dmdagetglobalindices_() dmdarestoreglobalindices_(). But then > src/docs/website/documentation/changes/35.html has: > >>>>> >
  • DMDAGetGlobalIndices(DM,PetscInt*,const PetscInt*[]) and DMDARestoreGlobalIndices(DM,PetscInt*,const PetscInt*[]) are removed, use DMGetLocalToGlobalMapping() to get this information
    • > <<<< > > So perhaps the attached modified patch should be used? [instead of the previous fix-fortran-dmdasetaotype.patch] > > patch -Np1 -R < fix-fortran-dmdasetaotype.patch > patch -Np1 < dmdasetaotype-fixed.patch > > Might be easier to use 'maint' branch from git.. > > Satish > > On Wed, 17 Feb 2016, Barry Smith wrote: > >> >> Randy, >> >> It is not suppose to be this hard! When you ran the make gnumake after you applied the patch did it recompile the file zdaindexf.c (in fact that file and daindexf.c should be the only files that changed and hence the only files that got recompiled). >> >> Barry >> >> >> >>> On Feb 17, 2016, at 11:41 PM, Randall Mackie wrote: >>> >>> Satish, >>> >>> It is unclear from this email exchange if I am still suppose to do make allfortranstubs after updating these two files, or if these are the files that are suppose to result from that make operation. >>> >>> Although these files looked the same as what I tried before with Barry?s suggestions, I dropped these in, recompiled petsc, but again, when I try to compile the test code I sent, it still complains about an unresolved symbol to dmdasetaotype_. >>> >>> If I still need to do the make allfortranstubs, I?ll do that via a git clone tomorrow. >>> >>> Thanks, Randy >>> >>> >>>> On Feb 17, 2016, at 8:54 PM, Satish Balay wrote: >>>> >>>> Attached are the modified src/dm/impls/da/ftn-auto/daindexf.c and >>>> src/dm/impls/da/ftn-custom/zdaindexf.c files. >>>> >>>> Satish >>>> >>>> On Wed, 17 Feb 2016, Jed Brown wrote: >>>> >>>>> Randall Mackie writes: >>>>> >>>>>> this leads to the error ?bin/maint/generatefortranstubs.py? ?No such file. >>>>>> >>>>>> there is no maint directory under bin. >>>>> >>>>> Ah, I missed that you have a tarball, not a clone of the repository. >>>>> Then you'll have to fix the file manually. I would start from a clean >>>>> tarball because we have no idea what state your source tree is in after >>>>> applying the match and manually editing files. Note that this would be >>>>> easy and completely reliable if you used a clone instead of tarball. >>>>> >>>> >>> >> > From petertututk at gmail.com Thu Feb 18 09:13:23 2016 From: petertututk at gmail.com (peter tutuk) Date: Thu, 18 Feb 2016 16:13:23 +0100 Subject: [petsc-users] multiplication Message-ID: Hello, I need to perform following operation x = y^T * A, where y and x are R^n vectors and A is a matrix of dimension n x m. Is there any routine, which could help me, which I wasn't able to find? many thanks, peter -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Thu Feb 18 09:29:46 2016 From: hzhang at mcs.anl.gov (Hong) Date: Thu, 18 Feb 2016 09:29:46 -0600 Subject: [petsc-users] multiplication In-Reply-To: References: Message-ID: You can do x^T = (y^T * A)^T = A^T *y (MatMultTranspose) Hong On Thu, Feb 18, 2016 at 9:13 AM, peter tutuk wrote: > Hello, > > I need to perform following operation > > x = y^T * A, > > where y and x are R^n vectors and A is a matrix of dimension n x m. Is > there any routine, which could help me, which I wasn't able to find? > > > many thanks, > peter > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Feb 18 10:41:53 2016 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 18 Feb 2016 10:41:53 -0600 Subject: [petsc-users] memory scalable AO In-Reply-To: <0C1DBE27-7446-4D82-8176-525302EBDB23@gmail.com> References: <0DD3688F-1E57-4AC7-89A9-DD96B6A676FA@mcs.anl.gov> <4A7DAAD2-9FE3-482D-AACE-604F89BB1B80@gmail.com> <9AC44E2C-39E2-47A7-86BA-AF5723C5EB9E@mcs.anl.gov> <60AAF166-8B9E-4365-B68A-81BB35192BD3@gmail.com> <87twl6ra4z.fsf@jedbrown.org> <23E95BA9-BD97-4EAC-BCE1-5E8A92BE1E25@gmail.com> <87lh6ir5ik.fsf@jedbrown.org> <0C1DBE27-7446-4D82-8176-525302EBDB23@gmail.com> Message-ID: Thanks! pushed to maint - and merged to master. Satish On Thu, 18 Feb 2016, Randall Mackie wrote: > Hi Satish, > > This patch works now. > > Thanks very much to all of you for your patience and help. > > Randy > > > > On Feb 17, 2016, at 9:58 PM, Satish Balay wrote: > > > > Look like zdaindexf.c is not listed in makefile - so it never gets compiled. > > > > Also it has code for dmdagetglobalindices_() dmdarestoreglobalindices_(). But then > > src/docs/website/documentation/changes/35.html has: > > > >>>>> > >
  • DMDAGetGlobalIndices(DM,PetscInt*,const PetscInt*[]) and DMDARestoreGlobalIndices(DM,PetscInt*,const PetscInt*[]) are removed, use DMGetLocalToGlobalMapping() to get this information
    • > > <<<< > > > > So perhaps the attached modified patch should be used? [instead of the previous fix-fortran-dmdasetaotype.patch] > > > > patch -Np1 -R < fix-fortran-dmdasetaotype.patch > > patch -Np1 < dmdasetaotype-fixed.patch > > > > Might be easier to use 'maint' branch from git.. > > > > Satish > > > > On Wed, 17 Feb 2016, Barry Smith wrote: > > > >> > >> Randy, > >> > >> It is not suppose to be this hard! When you ran the make gnumake after you applied the patch did it recompile the file zdaindexf.c (in fact that file and daindexf.c should be the only files that changed and hence the only files that got recompiled). > >> > >> Barry > >> > >> > >> > >>> On Feb 17, 2016, at 11:41 PM, Randall Mackie wrote: > >>> > >>> Satish, > >>> > >>> It is unclear from this email exchange if I am still suppose to do make allfortranstubs after updating these two files, or if these are the files that are suppose to result from that make operation. > >>> > >>> Although these files looked the same as what I tried before with Barry?s suggestions, I dropped these in, recompiled petsc, but again, when I try to compile the test code I sent, it still complains about an unresolved symbol to dmdasetaotype_. > >>> > >>> If I still need to do the make allfortranstubs, I?ll do that via a git clone tomorrow. > >>> > >>> Thanks, Randy > >>> > >>> > >>>> On Feb 17, 2016, at 8:54 PM, Satish Balay wrote: > >>>> > >>>> Attached are the modified src/dm/impls/da/ftn-auto/daindexf.c and > >>>> src/dm/impls/da/ftn-custom/zdaindexf.c files. > >>>> > >>>> Satish > >>>> > >>>> On Wed, 17 Feb 2016, Jed Brown wrote: > >>>> > >>>>> Randall Mackie writes: > >>>>> > >>>>>> this leads to the error ?bin/maint/generatefortranstubs.py? ?No such file. > >>>>>> > >>>>>> there is no maint directory under bin. > >>>>> > >>>>> Ah, I missed that you have a tarball, not a clone of the repository. > >>>>> Then you'll have to fix the file manually. I would start from a clean > >>>>> tarball because we have no idea what state your source tree is in after > >>>>> applying the match and manually editing files. Note that this would be > >>>>> easy and completely reliable if you used a clone instead of tarball. > >>>>> > >>>> > >>> > >> > > > > From jychang48 at gmail.com Thu Feb 18 13:56:10 2016 From: jychang48 at gmail.com (Justin Chang) Date: Thu, 18 Feb 2016 13:56:10 -0600 Subject: [petsc-users] Do more solver iterations = more communication? Message-ID: Hi all, For a poisson problem with roughly 1 million dofs (using second-order elements), I solved the problem using two different solver/preconditioner combinations: CG/ILU and CG/GAMG. ILU takes roughly 82 solver iterations whereas with GAMG it takes 14 iterations (wall clock time is roughly 15 and 46 seconds respectively). I have seen from previous mailing threads that there is a strong correlation between solver iterations and communication (which could lead to less strong-scaling scalability). It makes sense to me if I strictly use one of these preconditioners to solve two different problems and compare the number of respective iterations, but what about solving the same problem with two different preconditioners? If GAMG takes 14 iterations whereas ILU takes 82 iterations, does this necessarily mean GAMG has less communication? I would think that the "bandwidth" that happens within a single GAMG iteration would be much greater than that within a single ILU iteration. Is there a way to officially determine this? I see from log_summary that we have this information: MPI Messages: 5.000e+00 1.00000 5.000e+00 5.000e+00 MPI Message Lengths: 5.816e+07 1.00000 1.163e+07 5.816e+07 MPI Reductions: 2.000e+01 1.00000 Can this information be used to determine the "bandwidth"? If so, does PETSc have the ability to document this for other preconditioner packages like HYPRE's BoomerAMG or Trilinos' ML? Thanks, Justin -------------- next part -------------- An HTML attachment was scrubbed... URL: From amneetb at live.unc.edu Thu Feb 18 14:28:01 2016 From: amneetb at live.unc.edu (Bhalla, Amneet Pal S) Date: Thu, 18 Feb 2016 20:28:01 +0000 Subject: [petsc-users] Error with MatSchurComplement In-Reply-To: References: <1FC93CCE-F389-4555-BDDF-FEF60ABCC154@mcs.anl.gov> <5EBB1897-32AC-430B-BECC-11742E7F0BA9@email.unc.edu> <6F26E83B-0CAA-4D52-9C44-EFFCF432CCB1@mcs.anl.gov> Message-ID: <5D9FC0A0-CDC1-4DC2-B799-AD86E3CC8C7E@ad.unc.edu> On Feb 17, 2016, at 10:58 AM, Bhalla, Amneet Pal Singh > wrote: I don't know could be. Likely there is some wrong with your IS. I can try to remove duplicated entries while creating a parallel IS. There were some duplicated local indices with Staggered velocity grid. Removing them made it work :-). Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Feb 18 16:05:29 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 18 Feb 2016 16:05:29 -0600 Subject: [petsc-users] Do more solver iterations = more communication? In-Reply-To: References: Message-ID: <0BEED677-9C98-465E-8339-CED8165F92A8@mcs.anl.gov> > On Feb 18, 2016, at 1:56 PM, Justin Chang wrote: > > Hi all, > > For a poisson problem with roughly 1 million dofs (using second-order elements), I solved the problem using two different solver/preconditioner combinations: CG/ILU and CG/GAMG. > > ILU takes roughly 82 solver iterations whereas with GAMG it takes 14 iterations (wall clock time is roughly 15 and 46 seconds respectively). I have seen from previous mailing threads that there is a strong correlation between solver iterations and communication (which could lead to less strong-scaling scalability). It makes sense to me if I strictly use one of these preconditioners to solve two different problems and compare the number of respective iterations, but what about solving the same problem with two different preconditioners? > > If GAMG takes 14 iterations whereas ILU takes 82 iterations, does this necessarily mean GAMG has less communication? No you can't say that at all. A single GAMG cycle will do more communication than a single block Jacobi cycle. > I would think that the "bandwidth" that happens within a single GAMG iteration would be much greater than that within a single ILU iteration. Is there a way to officially determine this? > > I see from log_summary that we have this information: > MPI Messages: 5.000e+00 1.00000 5.000e+00 5.000e+00 > MPI Message Lengths: 5.816e+07 1.00000 1.163e+07 5.816e+07 > MPI Reductions: 2.000e+01 1.00000 > > Can this information be used to determine the "bandwidth"? You can certainly use this data for each run to determine which algorithm is sending more messages, total length of messages is bigger etc. And if you divided by time it would tell the rate of communication for the different algorithms. Note that counts of messages and lengths are also given in the detailed table for each operation. There are also theoretical bounds on messages that can be derived for some iterations applied to some problems. > If so, does PETSc have the ability to document this for other preconditioner packages like HYPRE's BoomerAMG or Trilinos' ML? No, because they don't log this information. > > Thanks, > Justin From rongliang.chan at gmail.com Thu Feb 18 19:37:14 2016 From: rongliang.chan at gmail.com (Rongliang Chen) Date: Fri, 19 Feb 2016 09:37:14 +0800 Subject: [petsc-users] SNESComputeJacobianDefaultColor use too much memory In-Reply-To: <68210D25-00D1-4273-9C83-5E6A299D17E7@mcs.anl.gov> References: <56C319D0.9060208@gmail.com> <684F7147-4F58-42E3-9C9E-2C2297383276@mcs.anl.gov> <56C3EC9B.5080306@gmail.com> <1199051F-2EC1-4462-A7C0-E2DDCA95A4C3@mcs.anl.gov> <56C409D9.70002@gmail.com> <2424BDF0-AD22-40EB-9920-AF697BFF4DAF@mcs.anl.gov> <56C425F8.1060306@gmail.com> <68210D25-00D1-4273-9C83-5E6A299D17E7@mcs.anl.gov> Message-ID: <56C671CA.6030805@gmail.com> Hi Barry, When I increase the size of the mesh, another memory issue comes out. The DMPlexInterpolateFaces_Internal takes too much memory. The massif output is attached. Any suggestions for this? Thanks. Best, Rongliang On 02/18/2016 04:11 AM, Barry Smith wrote: > Hmm, Matt, this is nuts. This > > if (size > 1) { > /* create a sequential iscoloring on all processors */ > ierr = MatGetSeqNonzeroStructure(mat,&mat_seq);CHKERRQ(ierr); > } > > It sequentializes the graph. > > It looks like the only parallel coloring for Jacobians is MATCOLORINGGREEDY? Try that. > > Maybe that should be the default? > > Barry > > >> On Feb 17, 2016, at 1:49 AM, Rongliang Chen wrote: >> >> Dear Barry, >> >> The massif output for the large case is attached. It shows that the job was kill in the function MatGetSubMatrix_MPIAIJ_All(). Any suggestions? >> >> -------------------- >> [8]PETSC ERROR: #1 MatGetSubMatrix_MPIAIJ_All() line 622 in /home/rlchen/soft/petsc-3.5.2/src/mat/impls/aij/mpi/mpiov.c >> [8]PETSC ERROR: #2 MatGetSeqNonzeroStructure_MPIAIJ() line 3010 in /home/rlchen/soft/petsc-3.5.2/src/mat/impls/aij/mpi/mpiaij.c >> [8]PETSC ERROR: #3 MatGetSeqNonzeroStructure() line 6487 in /home/rlchen/soft/petsc-3.5.2/src/mat/interface/matrix.c >> [8]PETSC ERROR: #4 MatColoringApply_SL() line 78 in /home/rlchen/soft/petsc-3.5.2/src/mat/color/impls/minpack/color.c >> [8]PETSC ERROR: #5 MatColoringApply() line 379 in /home/rlchen/soft/petsc-3.5.2/src/mat/color/interface/matcoloring.c >> [8]PETSC ERROR: #6 SNESComputeJacobianDefaultColor() line 71 in /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snesj2.c >> [8]PETSC ERROR: #7 FormJacobian() line 58 in /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/SetupJacobian.c >> [8]PETSC ERROR: #8 SNESComputeJacobian() line 2193 in /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c >> [8]PETSC ERROR: #9 SNESSolve_NEWTONLS() line 230 in /home/rlchen/soft/petsc-3.5.2/src/snes/impls/ls/ls.c >> [8]PETSC ERROR: #10 SNESSolve() line 3743 in /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c >> [8]PETSC ERROR: #11 SolveTimeDependent() line 758 in /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/Nwtun.c >> [8]PETSC ERROR: #12 main() line 417 in /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/Nwtun.c >> ------------------- >> >> Best, >> Rongliang >> >> On 02/17/2016 02:09 PM, Barry Smith wrote: >>> Yes, this is the type of output I was expecting. Now you need to produce it for a large case. >>> >>> Barry >>> >>>> On Feb 16, 2016, at 11:49 PM, Rongliang Chen wrote: >>>> >>>> Hi Barry, >>>> >>>> I run the code with valgrind on a workstation for a smaller case and it produces some ASCII information (see attached). Is this helpful? >>>> >>>> Best, >>>> Rongliang >>>> >>>> On 02/17/2016 01:30 PM, Barry Smith wrote: >>>>> Hmm, something didn't work right with massif. I should give a bunch of ASCII information about how much memory is used at different times in the code. You may need to play around with the massif options and google documentation on how to get it to provide useful information. Once it produces the useful information it will be very helpful. >>>>> >>>>> time=0 >>>>> mem_heap_B=0 >>>>> mem_heap_extra_B=0 >>>>> mem_stacks_B=0 >>>>> heap_tree=empty >>>>> >>>>> >>>>>> On Feb 16, 2016, at 9:44 PM, Rongliang Chen wrote: >>>>>> >>>>>> Dear Barry, >>>>>> >>>>>> Many thanks for your reply. >>>>>> >>>>>> I checked with the valgrind and did not obtain any outputs (massif.out.) because the job was killed before it reached the end. >>>>>> >>>>>> Then I switch to a smaller case, it works well and one of the output is attached (I did not find any useful information in it). The output with the option -mat_coloring_view is followed, which shows that the number of colors is 65. Any ideas for this? >>>>>> >>>>>> MatColoring Object: 480 MPI processes >>>>>> type: sl >>>>>> Weight type: RANDOM >>>>>> Distance 2, Max. Colors 65535 >>>>>> Number of colors 65 >>>>>> Number of total columns 1637350 >>>>>> >>>>>> Best regards, >>>>>> Rongliang >>>>>> >>>>>> On 02/17/2016 01:13 AM, Barry Smith wrote: >>>>>>> How many colors are needed? >>>>>>> >>>>>>> You need to produce a breakdown of where all the memory is being used. For example valgrind with the >>>>>>> http://valgrind.org/docs/manual/ms-manual.html >>>>>>> >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>> >>>>>>>> On Feb 16, 2016, at 6:45 AM, Rongliang Chen >>>>>>>> wrote: >>>>>>>> >>>>>>>> Dear all, >>>>>>>> >>>>>>>> I am using the DMPlex to solve a PDE on a unstructured mesh and I use the SNESComputeJacobianDefaultColor to compute the Jacobian matrix. >>>>>>>> >>>>>>>> My code works well for small problems (such as problem with 3.3x10^5 cells using 120 cores) but when I increase the number of cells (2.6x10^6 cells using 1920 cores), I go out of memory in the function MatColoringApply. It shows that one of the cores uses over 11G memory. I think this is unreasonable. Do you have any suggestions for debugging this problem? >>>>>>>> >>>>>>>> Best regards, >>>>>>>> Rongliang >>>>>>>> >>>>>> >>>> >> -------------- next part -------------- desc: (none) cmd: ./Nwtun -f ./../meshes/3DNACA0012AIRFOIL_Hex300x100x200x160.exo -overlap 1 -options_file options.txt -time_integral_method 1 -initial_time 0.0 -final_time 4.0 -dt 1.e-3 -timestep 2 -max_time_its 0 -includeenergy -reconstruct -PressureFlux -testcase 3DNACA -output_solution -solutionfile results/3DNACA0012AIRFOIL_Hex150x50x100x80_imp -steps_output 4000 -inflow_u 349.02 -viscosity 0.01 -k 0.026 -monitor_step_gap -R 287.02 -adiabatic 1.4 -Euler -Attack 10 -Mach 0.8 -fd_jacobian_color -limiter zero -RiemannSolver Rusanov -CFL 0.9 -reconstruct -second_order -parallel_csr -parallel_output time_unit: i #----------- snapshot=0 #----------- time=0 mem_heap_B=0 mem_heap_extra_B=0 mem_stacks_B=0 heap_tree=empty #----------- snapshot=1 #----------- time=36881410931 mem_heap_B=7458834367 mem_heap_extra_B=33297 mem_stacks_B=0 heap_tree=detailed n3: 7458834367 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n7: 7302506748 0x57877E: PetscMallocAlign (mal.c:27) n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132) n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 2544974400 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x462A1D: DMSetUp_Plex (plex.c:1582) n1: 1357319680 0x50B88B: DMSetUp (dm.c:560) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x462AB1: DMSetUp_Plex (plex.c:1583) n1: 1357319680 0x50B88B: DMSetUp (dm.c:560) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n2: 682078080 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n1: 341039040 0x45A810: DMPlexSetChart (plex.c:729) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x45A874: DMPlexSetChart (plex.c:730) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 682078080 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n1: 341039040 0x45A810: DMPlexSetChart (plex.c:729) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x45A874: DMPlexSetChart (plex.c:730) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n1: 678659840 0x44390A: DMPlexCreateExodusNwtun (plexexodusii.c:149) n1: 678659840 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 678659840 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 678659840 0x4076DF: main (Nwtun.c:55) n0: 76988 in 59 places, all below massif's threshold (01.00%) n1: 100663336 0x4DBEB6: kh_resize_HASHI (hash.h:510) n1: 100663336 0x4DC2CD: kh_put_HASHI (hash.h:510) n1: 100663336 0x4E14DD: DMLabelSetValue (plexlabel.c:450) n1: 100663336 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909) n1: 100663336 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173) n1: 100663336 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 100663336 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 100663336 0x4076DF: main (Nwtun.c:55) n0: 55664283 in 105 places, all below massif's threshold (01.00%) #----------- snapshot=2 #----------- time=53537892721 mem_heap_B=7614548075 mem_heap_extra_B=37141 mem_stacks_B=0 heap_tree=detailed n4: 7614548075 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n7: 7302506748 0x57877E: PetscMallocAlign (mal.c:27) n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132) n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 2544974400 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x462A1D: DMSetUp_Plex (plex.c:1582) n1: 1357319680 0x50B88B: DMSetUp (dm.c:560) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x462AB1: DMSetUp_Plex (plex.c:1583) n1: 1357319680 0x50B88B: DMSetUp (dm.c:560) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n2: 682078080 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n1: 341039040 0x45A810: DMPlexSetChart (plex.c:729) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x45A874: DMPlexSetChart (plex.c:730) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 682078080 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n1: 341039040 0x45A810: DMPlexSetChart (plex.c:729) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x45A874: DMPlexSetChart (plex.c:730) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n1: 678659840 0x44390A: DMPlexCreateExodusNwtun (plexexodusii.c:149) n1: 678659840 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 678659840 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 678659840 0x4076DF: main (Nwtun.c:55) n0: 76988 in 59 places, all below massif's threshold (01.00%) n1: 201326744 0x4DBEB6: kh_resize_HASHI (hash.h:510) n1: 201326744 0x4DC2CD: kh_put_HASHI (hash.h:510) n1: 201326744 0x4E14DD: DMLabelSetValue (plexlabel.c:450) n1: 201326744 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909) n1: 201326744 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173) n1: 201326744 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 201326744 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 201326744 0x4076DF: main (Nwtun.c:55) n1: 100663372 0x4DBE90: kh_resize_HASHI (hash.h:510) n1: 100663372 0x4DC2CD: kh_put_HASHI (hash.h:510) n1: 100663372 0x4E14DD: DMLabelSetValue (plexlabel.c:450) n1: 100663372 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909) n1: 100663372 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173) n1: 100663372 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 100663372 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 100663372 0x4076DF: main (Nwtun.c:55) n0: 10051211 in 104 places, all below massif's threshold (01.00%) #----------- snapshot=3 #----------- time=83815558495 mem_heap_B=7623985259 mem_heap_extra_B=41157 mem_stacks_B=0 heap_tree=empty #----------- snapshot=4 #----------- time=115170250037 mem_heap_B=11993400907 mem_heap_extra_B=57749 mem_stacks_B=0 heap_tree=empty #----------- snapshot=5 #----------- time=204639329496 mem_heap_B=13007979483 mem_heap_extra_B=58317 mem_stacks_B=0 heap_tree=empty #----------- snapshot=6 #----------- time=296347515250 mem_heap_B=13007979483 mem_heap_extra_B=58317 mem_stacks_B=0 heap_tree=detailed n11: 13007979483 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n6: 6623846844 0x57877E: PetscMallocAlign (mal.c:27) n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132) n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 2544974400 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x462A1D: DMSetUp_Plex (plex.c:1582) n1: 1357319680 0x50B88B: DMSetUp (dm.c:560) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x462AB1: DMSetUp_Plex (plex.c:1583) n1: 1357319680 0x50B88B: DMSetUp (dm.c:560) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n2: 682078080 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n1: 341039040 0x45A810: DMPlexSetChart (plex.c:729) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x45A874: DMPlexSetChart (plex.c:730) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 682078080 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n1: 341039040 0x45A810: DMPlexSetChart (plex.c:729) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x45A874: DMPlexSetChart (plex.c:730) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n0: 76924 in 60 places, all below massif's threshold (01.00%) n2: 1868023688 0x13D6C59: gk_malloc (memory.c:147) n2: 1526984640 0x13DA642: libmetis__imalloc (gklib.c:24) n1: 1357319680 0x13F1019: libmetis__CreateGraphDual (mesh.c:177) n1: 1357319680 0x13F0CF7: METIS_MeshToDual (mesh.c:67) n1: 1357319680 0x4440CA: DMPlexCreateExodusNwtun (plexexodusii.c:218) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 169664960 0x13F12B6: libmetis__CreateGraphDual (mesh.c:202) n1: 169664960 0x13F0CF7: METIS_MeshToDual (mesh.c:67) n1: 169664960 0x4440CA: DMPlexCreateExodusNwtun (plexexodusii.c:218) n1: 169664960 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 169664960 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 169664960 0x4076DF: main (Nwtun.c:55) n2: 341039048 0x13DA6A8: libmetis__ismalloc (gklib.c:24) n1: 171374088 0x13F0FF0: libmetis__CreateGraphDual (mesh.c:176) n1: 171374088 0x13F0CF7: METIS_MeshToDual (mesh.c:67) n1: 171374088 0x4440CA: DMPlexCreateExodusNwtun (plexexodusii.c:218) n1: 171374088 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 171374088 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 171374088 0x4076DF: main (Nwtun.c:55) n1: 169664960 0x13F129F: libmetis__CreateGraphDual (mesh.c:201) n1: 169664960 0x13F0CF7: METIS_MeshToDual (mesh.c:67) n1: 169664960 0x4440CA: DMPlexCreateExodusNwtun (plexexodusii.c:218) n1: 169664960 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 169664960 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 169664960 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x443F5C: DMPlexCreateExodusNwtun (plexexodusii.c:205) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 1017989760 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202) n1: 1017989760 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1017989760 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1017989760 0x4076DF: main (Nwtun.c:55) n1: 1014578576 0x13F145D: libmetis__CreateGraphDual (mesh.c:213) n1: 1014578576 0x13F0CF7: METIS_MeshToDual (mesh.c:67) n1: 1014578576 0x4440CA: DMPlexCreateExodusNwtun (plexexodusii.c:218) n1: 1014578576 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1014578576 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1014578576 0x4076DF: main (Nwtun.c:55) n1: 402653224 0x4DBEB6: kh_resize_HASHI (hash.h:510) n1: 402653224 0x4DC2CD: kh_put_HASHI (hash.h:510) n1: 402653224 0x4E14DD: DMLabelSetValue (plexlabel.c:450) n1: 402653224 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909) n1: 402653224 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173) n1: 402653224 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 402653224 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 402653224 0x4076DF: main (Nwtun.c:55) n1: 201326612 0x4DBE90: kh_resize_HASHI (hash.h:510) n1: 201326612 0x4DC2CD: kh_put_HASHI (hash.h:510) n1: 201326612 0x4E14DD: DMLabelSetValue (plexlabel.c:450) n1: 201326612 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909) n1: 201326612 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173) n1: 201326612 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 201326612 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 201326612 0x4076DF: main (Nwtun.c:55) n1: 169664968 0x443EEF: DMPlexCreateExodusNwtun (plexexodusii.c:201) n1: 169664968 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 169664968 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 169664968 0x4076DF: main (Nwtun.c:55) n1: 169664968 0x443F34: DMPlexCreateExodusNwtun (plexexodusii.c:204) n1: 169664968 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 169664968 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 169664968 0x4076DF: main (Nwtun.c:55) n1: 169664968 0x13F123F: libmetis__CreateGraphDual (mesh.c:195) n1: 169664968 0x13F0CF7: METIS_MeshToDual (mesh.c:67) n1: 169664968 0x4440CA: DMPlexCreateExodusNwtun (plexexodusii.c:218) n1: 169664968 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 169664968 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 169664968 0x4076DF: main (Nwtun.c:55) n0: 13246195 in 106 places, all below massif's threshold (01.00%) #----------- snapshot=7 #----------- time=302711546355 mem_heap_B=10970241563 mem_heap_extra_B=44581 mem_stacks_B=0 heap_tree=empty #----------- snapshot=8 #----------- time=417185571136 mem_heap_B=9785998015 mem_heap_extra_B=40409 mem_stacks_B=0 heap_tree=empty #----------- snapshot=9 #----------- time=495477070490 mem_heap_B=9785998015 mem_heap_extra_B=40409 mem_stacks_B=0 heap_tree=empty #----------- snapshot=10 #----------- time=521966018039 mem_heap_B=9785998133 mem_heap_extra_B=40459 mem_stacks_B=0 heap_tree=empty #----------- snapshot=11 #----------- time=556872466967 mem_heap_B=10106862677 mem_heap_extra_B=56411 mem_stacks_B=0 heap_tree=empty #----------- snapshot=12 #----------- time=582193895580 mem_heap_B=10710842569 mem_heap_extra_B=64431 mem_stacks_B=0 heap_tree=empty #----------- snapshot=13 #----------- time=601361206419 mem_heap_B=11019123741 mem_heap_extra_B=64603 mem_stacks_B=0 heap_tree=empty #----------- snapshot=14 #----------- time=630993261443 mem_heap_B=11019123757 mem_heap_extra_B=64611 mem_stacks_B=0 heap_tree=empty #----------- snapshot=15 #----------- time=667400928559 mem_heap_B=10834836067 mem_heap_extra_B=50781 mem_stacks_B=0 heap_tree=empty #----------- snapshot=16 #----------- time=688635956717 mem_heap_B=11542625655 mem_heap_extra_B=54009 mem_stacks_B=0 heap_tree=detailed n8: 11542625655 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n8: 8322215448 0x57877E: PetscMallocAlign (mal.c:27) n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132) n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 2544974400 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x462AB1: DMSetUp_Plex (plex.c:1583) n1: 1357319680 0x50B88B: DMSetUp (dm.c:560) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x46371F: DMPlexSymmetrize (plex.c:1655) n1: 1357319680 0x444320: DMPlexCreateExodusNwtun (plexexodusii.c:247) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x462A1D: DMSetUp_Plex (plex.c:1582) n1: 1357319680 0x50B88B: DMSetUp (dm.c:560) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n2: 682078080 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n1: 341039040 0x45A810: DMPlexSetChart (plex.c:729) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x45A874: DMPlexSetChart (plex.c:730) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 682078080 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n1: 341039040 0x45A810: DMPlexSetChart (plex.c:729) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x45A874: DMPlexSetChart (plex.c:730) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46) n1: 341039040 0x4E1E75: DMLabelGetNumValues (plexlabel.c:504) n1: 341039040 0x473CE3: DMPlexGetDepth (plex.c:3971) n1: 341039040 0x9E510C: DMPlexInterpolate (plexinterpolate.c:335) n1: 341039040 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n0: 86808 in 68 places, all below massif's threshold (01.00%) n1: 1017989760 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202) n1: 1017989760 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1017989760 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1017989760 0x4076DF: main (Nwtun.c:55) n1: 805306976 0x9E01D5: kh_resize_HASHIJKL (hash.h:1199) n1: 805306976 0x9E07CD: kh_put_HASHIJKL (hash.h:1199) n1: 805306976 0x9E1116: PetscHashIJKLPut (hash.h:1262) n1: 805306976 0x9E3866: DMPlexInterpolateFaces_Internal (plexinterpolate.c:195) n1: 805306976 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 805306976 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 805306976 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 805306976 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 805306976 0x4076DF: main (Nwtun.c:55) n1: 603980232 0x9E020C: kh_resize_HASHIJKL (hash.h:1199) n1: 603980232 0x9E07CD: kh_put_HASHIJKL (hash.h:1199) n1: 603980232 0x9E1116: PetscHashIJKLPut (hash.h:1262) n1: 603980232 0x9E3866: DMPlexInterpolateFaces_Internal (plexinterpolate.c:195) n1: 603980232 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 603980232 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 603980232 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 603980232 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 603980232 0x4076DF: main (Nwtun.c:55) n1: 402653224 0x4DBEB6: kh_resize_HASHI (hash.h:510) n1: 402653224 0x4DC2CD: kh_put_HASHI (hash.h:510) n2: 402653224 0x4E14DD: DMLabelSetValue (plexlabel.c:450) n1: 402653224 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909) n1: 402653224 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173) n1: 402653224 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 402653224 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 402653224 0x4076DF: main (Nwtun.c:55) n0: 0 in 2 places, all below massif's threshold (01.00%) n1: 201326612 0x4DBE90: kh_resize_HASHI (hash.h:510) n1: 201326612 0x4DC2CD: kh_put_HASHI (hash.h:510) n2: 201326612 0x4E14DD: DMLabelSetValue (plexlabel.c:450) n1: 201326612 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909) n1: 201326612 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173) n1: 201326612 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 201326612 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 201326612 0x4076DF: main (Nwtun.c:55) n0: 0 in 2 places, all below massif's threshold (01.00%) n1: 169664968 0x443EEF: DMPlexCreateExodusNwtun (plexexodusii.c:201) n1: 169664968 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 169664968 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 169664968 0x4076DF: main (Nwtun.c:55) n0: 19488435 in 113 places, all below massif's threshold (01.00%) #----------- snapshot=17 #----------- time=727470680494 mem_heap_B=11555208571 mem_heap_extra_B=58045 mem_stacks_B=0 heap_tree=empty #----------- snapshot=18 #----------- time=731287750178 mem_heap_B=12958202467 mem_heap_extra_B=54797 mem_stacks_B=0 heap_tree=empty #----------- snapshot=19 #----------- time=808892033505 mem_heap_B=12983368299 mem_heap_extra_B=58829 mem_stacks_B=0 heap_tree=empty #----------- snapshot=20 #----------- time=816421281943 mem_heap_B=15801941595 mem_heap_extra_B=57821 mem_stacks_B=0 heap_tree=detailed n8: 15801941595 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n8: 8322215448 0x57877E: PetscMallocAlign (mal.c:27) n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132) n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 2544974400 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x462AB1: DMSetUp_Plex (plex.c:1583) n1: 1357319680 0x50B88B: DMSetUp (dm.c:560) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x46371F: DMPlexSymmetrize (plex.c:1655) n1: 1357319680 0x444320: DMPlexCreateExodusNwtun (plexexodusii.c:247) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x462A1D: DMSetUp_Plex (plex.c:1582) n1: 1357319680 0x50B88B: DMSetUp (dm.c:560) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n2: 682078080 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n1: 341039040 0x45A810: DMPlexSetChart (plex.c:729) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x45A874: DMPlexSetChart (plex.c:730) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 682078080 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n1: 341039040 0x45A810: DMPlexSetChart (plex.c:729) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x45A874: DMPlexSetChart (plex.c:730) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46) n1: 341039040 0x4E1E75: DMLabelGetNumValues (plexlabel.c:504) n1: 341039040 0x473CE3: DMPlexGetDepth (plex.c:3971) n1: 341039040 0x9E510C: DMPlexInterpolate (plexinterpolate.c:335) n1: 341039040 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n0: 86808 in 68 places, all below massif's threshold (01.00%) n1: 3221226208 0x9E01D5: kh_resize_HASHIJKL (hash.h:1199) n1: 3221226208 0x9E07CD: kh_put_HASHIJKL (hash.h:1199) n1: 3221226208 0x9E1116: PetscHashIJKLPut (hash.h:1262) n1: 3221226208 0x9E3866: DMPlexInterpolateFaces_Internal (plexinterpolate.c:195) n1: 3221226208 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 3221226208 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 3221226208 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 3221226208 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 3221226208 0x4076DF: main (Nwtun.c:55) n1: 2415919656 0x9E020C: kh_resize_HASHIJKL (hash.h:1199) n1: 2415919656 0x9E07CD: kh_put_HASHIJKL (hash.h:1199) n1: 2415919656 0x9E1116: PetscHashIJKLPut (hash.h:1262) n1: 2415919656 0x9E3866: DMPlexInterpolateFaces_Internal (plexinterpolate.c:195) n1: 2415919656 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 2415919656 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 2415919656 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 2415919656 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 2415919656 0x4076DF: main (Nwtun.c:55) n1: 1017989760 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202) n1: 1017989760 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1017989760 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1017989760 0x4076DF: main (Nwtun.c:55) n1: 402653224 0x4DBEB6: kh_resize_HASHI (hash.h:510) n1: 402653224 0x4DC2CD: kh_put_HASHI (hash.h:510) n2: 402653224 0x4E14DD: DMLabelSetValue (plexlabel.c:450) n1: 402653224 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909) n1: 402653224 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173) n1: 402653224 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 402653224 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 402653224 0x4076DF: main (Nwtun.c:55) n0: 0 in 2 places, all below massif's threshold (01.00%) n1: 201326612 0x4DBE90: kh_resize_HASHI (hash.h:510) n1: 201326612 0x4DC2CD: kh_put_HASHI (hash.h:510) n2: 201326612 0x4E14DD: DMLabelSetValue (plexlabel.c:450) n1: 201326612 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909) n1: 201326612 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173) n1: 201326612 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 201326612 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 201326612 0x4076DF: main (Nwtun.c:55) n0: 0 in 2 places, all below massif's threshold (01.00%) n1: 169664968 0x443EEF: DMPlexCreateExodusNwtun (plexexodusii.c:201) n1: 169664968 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 169664968 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 169664968 0x4076DF: main (Nwtun.c:55) n0: 50945719 in 113 places, all below massif's threshold (01.00%) #----------- snapshot=21 #----------- time=915052183074 mem_heap_B=12568132471 mem_heap_extra_B=50481 mem_stacks_B=0 heap_tree=empty #----------- snapshot=22 #----------- time=1053918439822 mem_heap_B=16594796631 mem_heap_extra_B=57177 mem_stacks_B=0 heap_tree=empty #----------- snapshot=23 #----------- time=1079607070946 mem_heap_B=20010269895 mem_heap_extra_B=66241 mem_stacks_B=0 heap_tree=detailed n6: 20010269895 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n8: 17850756792 0x57877E: PetscMallocAlign (mal.c:27) n1: 4418111328 0x462A1D: DMSetUp_Plex (plex.c:1582) n2: 4418111328 0x50B88B: DMSetUp (dm.c:560) n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229) n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 3060791648 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 4418111328 0x462AB1: DMSetUp_Plex (plex.c:1583) n2: 4418111328 0x50B88B: DMSetUp (dm.c:560) n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229) n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 3060791648 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132) n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 2544974400 0x4076DF: main (Nwtun.c:55) n2: 2385557104 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 2385557104 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x46371F: DMPlexSymmetrize (plex.c:1655) n1: 1357319680 0x444320: DMPlexCreateExodusNwtun (plexexodusii.c:247) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46) n1: 341039040 0x4E1E75: DMLabelGetNumValues (plexlabel.c:504) n1: 341039040 0x473CE3: DMPlexGetDepth (plex.c:3971) n1: 341039040 0x9E510C: DMPlexInterpolate (plexinterpolate.c:335) n1: 341039040 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n0: 86808 in 68 places, all below massif's threshold (01.00%) n1: 1017989760 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202) n1: 1017989760 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1017989760 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1017989760 0x4076DF: main (Nwtun.c:55) n1: 402653224 0x4DBEB6: kh_resize_HASHI (hash.h:510) n1: 402653224 0x4DC2CD: kh_put_HASHI (hash.h:510) n2: 402653224 0x4E14DD: DMLabelSetValue (plexlabel.c:450) n1: 402653224 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909) n1: 402653224 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173) n1: 402653224 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 402653224 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 402653224 0x4076DF: main (Nwtun.c:55) n0: 0 in 2 places, all below massif's threshold (01.00%) n0: 336216499 in 115 places, all below massif's threshold (01.00%) n1: 201327008 0x9E01D5: kh_resize_HASHIJKL (hash.h:1199) n1: 201327008 0x9E07CD: kh_put_HASHIJKL (hash.h:1199) n2: 201327008 0x9E1116: PetscHashIJKLPut (hash.h:1262) n1: 201327008 0x9E43F2: DMPlexInterpolateFaces_Internal (plexinterpolate.c:264) n1: 201327008 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 201327008 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 201327008 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 201327008 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 201327008 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 201326612 0x4DBE90: kh_resize_HASHI (hash.h:510) n1: 201326612 0x4DC2CD: kh_put_HASHI (hash.h:510) n2: 201326612 0x4E14DD: DMLabelSetValue (plexlabel.c:450) n1: 201326612 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909) n1: 201326612 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173) n1: 201326612 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 201326612 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 201326612 0x4076DF: main (Nwtun.c:55) n0: 0 in 2 places, all below massif's threshold (01.00%) #----------- snapshot=24 #----------- time=1106254629312 mem_heap_B=20364950279 mem_heap_extra_B=66689 mem_stacks_B=0 heap_tree=detailed n6: 20364950279 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n8: 17850756792 0x57877E: PetscMallocAlign (mal.c:27) n1: 4418111328 0x462A1D: DMSetUp_Plex (plex.c:1582) n2: 4418111328 0x50B88B: DMSetUp (dm.c:560) n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229) n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 3060791648 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 4418111328 0x462AB1: DMSetUp_Plex (plex.c:1583) n2: 4418111328 0x50B88B: DMSetUp (dm.c:560) n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229) n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 3060791648 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132) n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 2544974400 0x4076DF: main (Nwtun.c:55) n2: 2385557104 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 2385557104 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x46371F: DMPlexSymmetrize (plex.c:1655) n1: 1357319680 0x444320: DMPlexCreateExodusNwtun (plexexodusii.c:247) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46) n1: 341039040 0x4E1E75: DMLabelGetNumValues (plexlabel.c:504) n1: 341039040 0x473CE3: DMPlexGetDepth (plex.c:3971) n1: 341039040 0x9E510C: DMPlexInterpolate (plexinterpolate.c:335) n1: 341039040 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n0: 86808 in 68 places, all below massif's threshold (01.00%) n1: 1017989760 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202) n1: 1017989760 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1017989760 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1017989760 0x4076DF: main (Nwtun.c:55) n1: 402653344 0x9E01D5: kh_resize_HASHIJKL (hash.h:1199) n1: 402653344 0x9E07CD: kh_put_HASHIJKL (hash.h:1199) n2: 402653344 0x9E1116: PetscHashIJKLPut (hash.h:1262) n1: 402653344 0x9E43F2: DMPlexInterpolateFaces_Internal (plexinterpolate.c:264) n1: 402653344 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 402653344 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 402653344 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 402653344 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 402653344 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 402653224 0x4DBEB6: kh_resize_HASHI (hash.h:510) n1: 402653224 0x4DC2CD: kh_put_HASHI (hash.h:510) n2: 402653224 0x4E14DD: DMLabelSetValue (plexlabel.c:450) n1: 402653224 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909) n1: 402653224 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173) n1: 402653224 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 402653224 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 402653224 0x4076DF: main (Nwtun.c:55) n0: 0 in 2 places, all below massif's threshold (01.00%) n0: 388907151 in 115 places, all below massif's threshold (01.00%) n1: 301990008 0x9E020C: kh_resize_HASHIJKL (hash.h:1199) n1: 301990008 0x9E07CD: kh_put_HASHIJKL (hash.h:1199) n2: 301990008 0x9E1116: PetscHashIJKLPut (hash.h:1262) n1: 301990008 0x9E43F2: DMPlexInterpolateFaces_Internal (plexinterpolate.c:264) n1: 301990008 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 301990008 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 301990008 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 301990008 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 301990008 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) #----------- snapshot=25 #----------- time=1157665534790 mem_heap_B=20369668875 mem_heap_extra_B=70701 mem_stacks_B=0 heap_tree=empty #----------- snapshot=26 #----------- time=1159549737997 mem_heap_B=21074312731 mem_heap_extra_B=69917 mem_stacks_B=0 heap_tree=detailed n6: 21074312731 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n8: 17850756792 0x57877E: PetscMallocAlign (mal.c:27) n1: 4418111328 0x462A1D: DMSetUp_Plex (plex.c:1582) n2: 4418111328 0x50B88B: DMSetUp (dm.c:560) n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229) n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 3060791648 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 4418111328 0x462AB1: DMSetUp_Plex (plex.c:1583) n2: 4418111328 0x50B88B: DMSetUp (dm.c:560) n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229) n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 3060791648 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132) n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 2544974400 0x4076DF: main (Nwtun.c:55) n2: 2385557104 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 2385557104 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x46371F: DMPlexSymmetrize (plex.c:1655) n1: 1357319680 0x444320: DMPlexCreateExodusNwtun (plexexodusii.c:247) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46) n1: 341039040 0x4E1E75: DMLabelGetNumValues (plexlabel.c:504) n1: 341039040 0x473CE3: DMPlexGetDepth (plex.c:3971) n1: 341039040 0x9E510C: DMPlexInterpolate (plexinterpolate.c:335) n1: 341039040 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n0: 86808 in 68 places, all below massif's threshold (01.00%) n1: 1017989760 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202) n1: 1017989760 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1017989760 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1017989760 0x4076DF: main (Nwtun.c:55) n1: 805306976 0x9E01D5: kh_resize_HASHIJKL (hash.h:1199) n1: 805306976 0x9E07CD: kh_put_HASHIJKL (hash.h:1199) n2: 805306976 0x9E1116: PetscHashIJKLPut (hash.h:1262) n1: 805306976 0x9E43F2: DMPlexInterpolateFaces_Internal (plexinterpolate.c:264) n1: 805306976 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 805306976 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 805306976 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 805306976 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 805306976 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 603980232 0x9E020C: kh_resize_HASHIJKL (hash.h:1199) n1: 603980232 0x9E07CD: kh_put_HASHIJKL (hash.h:1199) n2: 603980232 0x9E1116: PetscHashIJKLPut (hash.h:1262) n1: 603980232 0x9E43F2: DMPlexInterpolateFaces_Internal (plexinterpolate.c:264) n1: 603980232 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 603980232 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 603980232 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 603980232 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 603980232 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 402653224 0x4DBEB6: kh_resize_HASHI (hash.h:510) n1: 402653224 0x4DC2CD: kh_put_HASHI (hash.h:510) n2: 402653224 0x4E14DD: DMLabelSetValue (plexlabel.c:450) n1: 402653224 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909) n1: 402653224 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173) n1: 402653224 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 402653224 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 402653224 0x4076DF: main (Nwtun.c:55) n0: 0 in 2 places, all below massif's threshold (01.00%) n0: 393625747 in 115 places, all below massif's threshold (01.00%) #----------- snapshot=27 #----------- time=1262479157617 mem_heap_B=21083749915 mem_heap_extra_B=73933 mem_stacks_B=0 heap_tree=empty #----------- snapshot=28 #----------- time=1266296227301 mem_heap_B=22493035275 mem_heap_extra_B=74717 mem_stacks_B=0 heap_tree=detailed n6: 22493035275 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n8: 17850756792 0x57877E: PetscMallocAlign (mal.c:27) n1: 4418111328 0x462A1D: DMSetUp_Plex (plex.c:1582) n2: 4418111328 0x50B88B: DMSetUp (dm.c:560) n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229) n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 3060791648 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 4418111328 0x462AB1: DMSetUp_Plex (plex.c:1583) n2: 4418111328 0x50B88B: DMSetUp (dm.c:560) n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229) n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 3060791648 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132) n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 2544974400 0x4076DF: main (Nwtun.c:55) n2: 2385557104 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 2385557104 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x46371F: DMPlexSymmetrize (plex.c:1655) n1: 1357319680 0x444320: DMPlexCreateExodusNwtun (plexexodusii.c:247) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46) n1: 341039040 0x4E1E75: DMLabelGetNumValues (plexlabel.c:504) n1: 341039040 0x473CE3: DMPlexGetDepth (plex.c:3971) n1: 341039040 0x9E510C: DMPlexInterpolate (plexinterpolate.c:335) n1: 341039040 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n0: 86808 in 68 places, all below massif's threshold (01.00%) n1: 1610612896 0x9E01D5: kh_resize_HASHIJKL (hash.h:1199) n1: 1610612896 0x9E07CD: kh_put_HASHIJKL (hash.h:1199) n2: 1610612896 0x9E1116: PetscHashIJKLPut (hash.h:1262) n1: 1610612896 0x9E43F2: DMPlexInterpolateFaces_Internal (plexinterpolate.c:264) n1: 1610612896 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 1610612896 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 1610612896 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1610612896 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1610612896 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 1207959672 0x9E020C: kh_resize_HASHIJKL (hash.h:1199) n1: 1207959672 0x9E07CD: kh_put_HASHIJKL (hash.h:1199) n2: 1207959672 0x9E1116: PetscHashIJKLPut (hash.h:1262) n1: 1207959672 0x9E43F2: DMPlexInterpolateFaces_Internal (plexinterpolate.c:264) n1: 1207959672 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 1207959672 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 1207959672 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1207959672 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1207959672 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 1017989760 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202) n1: 1017989760 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1017989760 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1017989760 0x4076DF: main (Nwtun.c:55) n0: 403062931 in 115 places, all below massif's threshold (01.00%) n1: 402653224 0x4DBEB6: kh_resize_HASHI (hash.h:510) n1: 402653224 0x4DC2CD: kh_put_HASHI (hash.h:510) n2: 402653224 0x4E14DD: DMLabelSetValue (plexlabel.c:450) n1: 402653224 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909) n1: 402653224 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173) n1: 402653224 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 402653224 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 402653224 0x4076DF: main (Nwtun.c:55) n0: 0 in 2 places, all below massif's threshold (01.00%) #----------- snapshot=29 #----------- time=1472102781759 mem_heap_B=22511909643 mem_heap_extra_B=74717 mem_stacks_B=0 heap_tree=empty #----------- snapshot=30 #----------- time=1479632030197 mem_heap_B=25330482939 mem_heap_extra_B=73709 mem_stacks_B=0 heap_tree=detailed n6: 25330482939 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n8: 17850756792 0x57877E: PetscMallocAlign (mal.c:27) n1: 4418111328 0x462A1D: DMSetUp_Plex (plex.c:1582) n2: 4418111328 0x50B88B: DMSetUp (dm.c:560) n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229) n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 3060791648 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 4418111328 0x462AB1: DMSetUp_Plex (plex.c:1583) n2: 4418111328 0x50B88B: DMSetUp (dm.c:560) n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229) n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 3060791648 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132) n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 2544974400 0x4076DF: main (Nwtun.c:55) n2: 2385557104 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 2385557104 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x46371F: DMPlexSymmetrize (plex.c:1655) n1: 1357319680 0x444320: DMPlexCreateExodusNwtun (plexexodusii.c:247) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46) n1: 341039040 0x4E1E75: DMLabelGetNumValues (plexlabel.c:504) n1: 341039040 0x473CE3: DMPlexGetDepth (plex.c:3971) n1: 341039040 0x9E510C: DMPlexInterpolate (plexinterpolate.c:335) n1: 341039040 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n0: 86808 in 68 places, all below massif's threshold (01.00%) n1: 3221226208 0x9E01D5: kh_resize_HASHIJKL (hash.h:1199) n1: 3221226208 0x9E07CD: kh_put_HASHIJKL (hash.h:1199) n2: 3221226208 0x9E1116: PetscHashIJKLPut (hash.h:1262) n1: 3221226208 0x9E43F2: DMPlexInterpolateFaces_Internal (plexinterpolate.c:264) n1: 3221226208 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 3221226208 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 3221226208 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 3221226208 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 3221226208 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 2415919656 0x9E020C: kh_resize_HASHIJKL (hash.h:1199) n1: 2415919656 0x9E07CD: kh_put_HASHIJKL (hash.h:1199) n2: 2415919656 0x9E1116: PetscHashIJKLPut (hash.h:1262) n1: 2415919656 0x9E43F2: DMPlexInterpolateFaces_Internal (plexinterpolate.c:264) n1: 2415919656 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 2415919656 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 2415919656 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 2415919656 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 2415919656 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 1017989760 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202) n1: 1017989760 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1017989760 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1017989760 0x4076DF: main (Nwtun.c:55) n0: 421937299 in 115 places, all below massif's threshold (01.00%) n1: 402653224 0x4DBEB6: kh_resize_HASHI (hash.h:510) n1: 402653224 0x4DC2CD: kh_put_HASHI (hash.h:510) n2: 402653224 0x4E14DD: DMLabelSetValue (plexlabel.c:450) n1: 402653224 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909) n1: 402653224 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173) n1: 402653224 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 402653224 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 402653224 0x4076DF: main (Nwtun.c:55) n0: 0 in 2 places, all below massif's threshold (01.00%) #----------- snapshot=31 #----------- time=1744264900288 mem_heap_B=25317899999 mem_heap_extra_B=69657 mem_stacks_B=0 heap_tree=empty #----------- snapshot=32 #----------- time=2012308656155 mem_heap_B=22716379879 mem_heap_extra_B=60425 mem_stacks_B=0 heap_tree=empty #----------- snapshot=33 #----------- time=2194841265212 mem_heap_B=22716379879 mem_heap_extra_B=60425 mem_stacks_B=0 heap_tree=empty #----------- snapshot=34 #----------- time=2221330212467 mem_heap_B=22716379997 mem_heap_extra_B=60491 mem_stacks_B=0 heap_tree=empty #----------- snapshot=35 #----------- time=2259009892248 mem_heap_B=23332942797 mem_heap_extra_B=72579 mem_stacks_B=0 heap_tree=empty #----------- snapshot=36 #----------- time=2341996204267 mem_heap_B=23332942805 mem_heap_extra_B=72595 mem_stacks_B=0 heap_tree=empty #----------- snapshot=37 #----------- time=2359064457778 mem_heap_B=23675692019 mem_heap_extra_B=77781 mem_stacks_B=0 heap_tree=empty #----------- snapshot=38 #----------- time=2384204482868 mem_heap_B=23812599551 mem_heap_extra_B=87121 mem_stacks_B=0 heap_tree=empty #----------- snapshot=39 #----------- time=2409252239559 mem_heap_B=24120880723 mem_heap_extra_B=87293 mem_stacks_B=0 heap_tree=empty #----------- snapshot=40 #----------- time=2453442168129 mem_heap_B=24146046555 mem_heap_extra_B=91325 mem_stacks_B=0 heap_tree=empty #----------- snapshot=41 #----------- time=2458984993722 mem_heap_B=24737443631 mem_heap_extra_B=87073 mem_stacks_B=0 heap_tree=empty #----------- snapshot=42 #----------- time=2516474050926 mem_heap_B=24566069551 mem_heap_extra_B=84569 mem_stacks_B=0 heap_tree=empty #----------- snapshot=43 #----------- time=2548399887187 mem_heap_B=23398454597 mem_heap_extra_B=64107 mem_stacks_B=0 heap_tree=empty #----------- snapshot=44 #----------- time=2576502409399 mem_heap_B=25771427531 mem_heap_extra_B=93285 mem_stacks_B=0 heap_tree=detailed n5: 25771427531 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n9: 22104324620 0x57877E: PetscMallocAlign (mal.c:27) n1: 4418111328 0x462AB1: DMSetUp_Plex (plex.c:1583) n2: 4418111328 0x50B88B: DMSetUp (dm.c:560) n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229) n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 3060791648 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n2: 4418111328 0x46371F: DMPlexSymmetrize (plex.c:1655) n1: 3060791648 0x9E4D06: DMPlexInterpolateFaces_Internal (plexinterpolate.c:303) n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 3060791648 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x444320: DMPlexCreateExodusNwtun (plexexodusii.c:247) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 4418111328 0x462A1D: DMSetUp_Plex (plex.c:1582) n2: 4418111328 0x50B88B: DMSetUp (dm.c:560) n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229) n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 3060791648 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132) n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 2544974400 0x4076DF: main (Nwtun.c:55) n2: 2385557104 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 2385557104 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 1023113592 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46) n2: 682074552 0x4E2DE5: DMLabelGetStratumIS (plexlabel.c:571) n1: 682074552 0x46481D: DMPlexStratify (plex.c:1768) n1: 682074552 0x9E4D5E: DMPlexInterpolateFaces_Internal (plexinterpolate.c:304) n1: 682074552 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 682074552 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 682074552 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 682074552 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 682074552 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 341039040 0x4E1E75: DMLabelGetNumValues (plexlabel.c:504) n1: 341039040 0x473CE3: DMPlexGetDepth (plex.c:3971) n1: 341039040 0x9E510C: DMPlexInterpolate (plexinterpolate.c:335) n1: 341039040 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n1: 510700472 0x64DFA9: ISGeneralSetIndices_General (general.c:681) n1: 510700472 0x64DA50: ISGeneralSetIndices (general.c:663) n1: 510700472 0x64D730: ISCreateGeneral (general.c:634) n2: 510700472 0x4E2EC9: DMLabelGetStratumIS (plexlabel.c:576) n1: 510700472 0x46481D: DMPlexStratify (plex.c:1768) n1: 510700472 0x9E4D5E: DMPlexInterpolateFaces_Internal (plexinterpolate.c:304) n1: 510700472 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 510700472 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 510700472 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 510700472 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 510700472 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n0: 87964 in 75 places, all below massif's threshold (01.00%) n1: 1610613000 0x4DBEB6: kh_resize_HASHI (hash.h:510) n2: 1610613000 0x4DC2CD: kh_put_HASHI (hash.h:510) n1: 1207959776 0x4DD75D: DMLabelMakeInvalid_Private (plexlabel.c:85) n1: 1207959776 0x4E1115: DMLabelSetValue (plexlabel.c:425) n1: 1207959776 0x4646BF: DMPlexStratify (plex.c:1762) n1: 1207959776 0x9E4D5E: DMPlexInterpolateFaces_Internal (plexinterpolate.c:304) n1: 1207959776 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 1207959776 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 1207959776 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1207959776 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1207959776 0x4076DF: main (Nwtun.c:55) n2: 402653224 0x4E14DD: DMLabelSetValue (plexlabel.c:450) n1: 402653224 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909) n1: 402653224 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173) n1: 402653224 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 402653224 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 402653224 0x4076DF: main (Nwtun.c:55) n0: 0 in 3 places, all below massif's threshold (01.00%) n1: 1017989760 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202) n1: 1017989760 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1017989760 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1017989760 0x4076DF: main (Nwtun.c:55) n1: 805306500 0x4DBE90: kh_resize_HASHI (hash.h:510) n2: 805306500 0x4DC2CD: kh_put_HASHI (hash.h:510) n1: 603979888 0x4DD75D: DMLabelMakeInvalid_Private (plexlabel.c:85) n1: 603979888 0x4E1115: DMLabelSetValue (plexlabel.c:425) n1: 603979888 0x4646BF: DMPlexStratify (plex.c:1762) n1: 603979888 0x9E4D5E: DMPlexInterpolateFaces_Internal (plexinterpolate.c:304) n1: 603979888 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 603979888 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 603979888 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 603979888 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 603979888 0x4076DF: main (Nwtun.c:55) n0: 201326612 in 1 place, below massif's threshold (01.00%) n0: 233193651 in 116 places, all below massif's threshold (01.00%) #----------- snapshot=45 #----------- time=2612741728105 mem_heap_B=25156986307 mem_heap_extra_B=93605 mem_stacks_B=0 heap_tree=empty #----------- snapshot=46 #----------- time=2647328222574 mem_heap_B=25385051659 mem_heap_extra_B=97525 mem_stacks_B=0 heap_tree=empty #----------- snapshot=47 #----------- time=2711933920814 mem_heap_B=25397634575 mem_heap_extra_B=101561 mem_stacks_B=0 heap_tree=empty #----------- snapshot=48 #----------- time=2729611625174 mem_heap_B=26034370889 mem_heap_extra_B=99663 mem_stacks_B=0 heap_tree=detailed n5: 26034370889 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n8: 21763288110 0x57877E: PetscMallocAlign (mal.c:27) n1: 4418111328 0x462AB1: DMSetUp_Plex (plex.c:1583) n2: 4418111328 0x50B88B: DMSetUp (dm.c:560) n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229) n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 3060791648 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n2: 4418111328 0x46371F: DMPlexSymmetrize (plex.c:1655) n1: 3060791648 0x9E4D06: DMPlexInterpolateFaces_Internal (plexinterpolate.c:303) n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 3060791648 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x444320: DMPlexCreateExodusNwtun (plexexodusii.c:247) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 4418111328 0x462A1D: DMSetUp_Plex (plex.c:1582) n2: 4418111328 0x50B88B: DMSetUp (dm.c:560) n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229) n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 3060791648 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132) n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 2544974400 0x4076DF: main (Nwtun.c:55) n2: 2385557104 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 2385557104 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 1192778552 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46) n2: 851739512 0x4E2DE5: DMLabelGetStratumIS (plexlabel.c:571) n1: 851739512 0x46481D: DMPlexStratify (plex.c:1768) n1: 851739512 0x9E4D5E: DMPlexInterpolateFaces_Internal (plexinterpolate.c:304) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 341039040 0x4E1E75: DMLabelGetNumValues (plexlabel.c:504) n1: 341039040 0x473CE3: DMPlexGetDepth (plex.c:3971) n1: 341039040 0x9E510C: DMPlexInterpolate (plexinterpolate.c:335) n1: 341039040 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n0: 86966 in 76 places, all below massif's threshold (01.00%) n1: 2013266224 0x4DBEB6: kh_resize_HASHI (hash.h:510) n2: 2013266224 0x4DC2CD: kh_put_HASHI (hash.h:510) n1: 1207959776 0x4DD75D: DMLabelMakeInvalid_Private (plexlabel.c:85) n1: 1207959776 0x4E1115: DMLabelSetValue (plexlabel.c:425) n1: 1207959776 0x4646BF: DMPlexStratify (plex.c:1762) n1: 1207959776 0x9E4D5E: DMPlexInterpolateFaces_Internal (plexinterpolate.c:304) n1: 1207959776 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 1207959776 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 1207959776 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1207959776 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1207959776 0x4076DF: main (Nwtun.c:55) n3: 805306448 0x4E14DD: DMLabelSetValue (plexlabel.c:450) n1: 402653224 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909) n1: 402653224 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173) n1: 402653224 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 402653224 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 402653224 0x4076DF: main (Nwtun.c:55) n1: 402653224 0x4646BF: DMPlexStratify (plex.c:1762) n1: 402653224 0x9E4D5E: DMPlexInterpolateFaces_Internal (plexinterpolate.c:304) n1: 402653224 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 402653224 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 402653224 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 402653224 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 402653224 0x4076DF: main (Nwtun.c:55) n0: 0 in 2 places, all below massif's threshold (01.00%) n1: 1017989760 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202) n1: 1017989760 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1017989760 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1017989760 0x4076DF: main (Nwtun.c:55) n1: 1006633112 0x4DBE90: kh_resize_HASHI (hash.h:510) n2: 1006633112 0x4DC2CD: kh_put_HASHI (hash.h:510) n1: 603979888 0x4DD75D: DMLabelMakeInvalid_Private (plexlabel.c:85) n1: 603979888 0x4E1115: DMLabelSetValue (plexlabel.c:425) n1: 603979888 0x4646BF: DMPlexStratify (plex.c:1762) n1: 603979888 0x9E4D5E: DMPlexInterpolateFaces_Internal (plexinterpolate.c:304) n1: 603979888 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 603979888 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 603979888 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 603979888 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 603979888 0x4076DF: main (Nwtun.c:55) n1: 402653224 0x4E14DD: DMLabelSetValue (plexlabel.c:450) n0: 402653224 in 4 places, all below massif's threshold (01.00%) n0: 233193683 in 116 places, all below massif's threshold (01.00%) #----------- snapshot=49 #----------- time=2761530816318 mem_heap_B=23983355785 mem_heap_extra_B=71719 mem_stacks_B=0 heap_tree=empty #----------- snapshot=50 #----------- time=2767533197595 mem_heap_B=23568119581 mem_heap_extra_B=63651 mem_stacks_B=0 heap_tree=empty #----------- snapshot=51 #----------- time=2794984499889 mem_heap_B=23568120619 mem_heap_extra_B=63821 mem_stacks_B=0 heap_tree=empty #----------- snapshot=52 #----------- time=2797183222241 mem_heap_B=23568126541 mem_heap_extra_B=63795 mem_stacks_B=0 heap_tree=empty #----------- snapshot=53 #----------- time=2799999920116 mem_heap_B=23590446779 mem_heap_extra_B=70773 mem_stacks_B=0 heap_tree=empty #----------- snapshot=54 #----------- time=2802219005929 mem_heap_B=23612760439 mem_heap_extra_B=72121 mem_stacks_B=0 heap_tree=empty #----------- snapshot=55 #----------- time=2806720304926 mem_heap_B=23657390679 mem_heap_extra_B=71897 mem_stacks_B=0 heap_tree=empty #----------- snapshot=56 #----------- time=2832414634760 mem_heap_B=23746651047 mem_heap_extra_B=71561 mem_stacks_B=0 heap_tree=empty #----------- snapshot=57 #----------- time=2850303228602 mem_heap_B=23925171223 mem_heap_extra_B=71449 mem_stacks_B=0 heap_tree=detailed n4: 23925171223 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n8: 21763298824 0x57877E: PetscMallocAlign (mal.c:27) n1: 4418111328 0x462AB1: DMSetUp_Plex (plex.c:1583) n2: 4418111328 0x50B88B: DMSetUp (dm.c:560) n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229) n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 3060791648 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n2: 4418111328 0x46371F: DMPlexSymmetrize (plex.c:1655) n1: 3060791648 0x9E4D06: DMPlexInterpolateFaces_Internal (plexinterpolate.c:303) n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 3060791648 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x444320: DMPlexCreateExodusNwtun (plexexodusii.c:247) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 4418111328 0x462A1D: DMSetUp_Plex (plex.c:1582) n2: 4418111328 0x50B88B: DMSetUp (dm.c:560) n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229) n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 3060791648 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132) n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 2544974400 0x4076DF: main (Nwtun.c:55) n2: 2385557104 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 2385557104 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 1192778552 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46) n2: 851739512 0x4E2DE5: DMLabelGetStratumIS (plexlabel.c:571) n1: 851739512 0x46481D: DMPlexStratify (plex.c:1768) n1: 851739512 0x9E4D5E: DMPlexInterpolateFaces_Internal (plexinterpolate.c:304) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 341039040 0x4E1E75: DMLabelGetNumValues (plexlabel.c:504) n1: 341039040 0x473CE3: DMPlexGetDepth (plex.c:3971) n1: 341039040 0x9E510C: DMPlexInterpolate (plexinterpolate.c:335) n1: 341039040 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n0: 97680 in 76 places, all below massif's threshold (01.00%) n1: 1017989760 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202) n1: 1017989760 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1017989760 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1017989760 0x4076DF: main (Nwtun.c:55) n0: 741229415 in 117 places, all below massif's threshold (01.00%) n1: 402653224 0x4DBEB6: kh_resize_HASHI (hash.h:510) n2: 402653224 0x4DC2CD: kh_put_HASHI (hash.h:510) n2: 402653224 0x4E14DD: DMLabelSetValue (plexlabel.c:450) n1: 402653224 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909) n1: 402653224 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173) n1: 402653224 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 402653224 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 402653224 0x4076DF: main (Nwtun.c:55) n0: 0 in 3 places, all below massif's threshold (01.00%) n0: 0 in 1 place, below massif's threshold (01.00%) #----------- snapshot=58 #----------- time=2891840800954 mem_heap_B=24282210907 mem_heap_extra_B=75909 mem_stacks_B=0 heap_tree=empty #----------- snapshot=59 #----------- time=2894223977411 mem_heap_B=24983709031 mem_heap_extra_B=75105 mem_stacks_B=0 heap_tree=empty #----------- snapshot=60 #----------- time=3056581214185 mem_heap_B=24996291947 mem_heap_extra_B=79141 mem_stacks_B=0 heap_tree=empty #----------- snapshot=61 #----------- time=3060855258657 mem_heap_B=26405577307 mem_heap_extra_B=79925 mem_stacks_B=0 heap_tree=detailed n6: 26405577307 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n8: 21763298824 0x57877E: PetscMallocAlign (mal.c:27) n1: 4418111328 0x462AB1: DMSetUp_Plex (plex.c:1583) n2: 4418111328 0x50B88B: DMSetUp (dm.c:560) n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229) n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 3060791648 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n2: 4418111328 0x46371F: DMPlexSymmetrize (plex.c:1655) n1: 3060791648 0x9E4D06: DMPlexInterpolateFaces_Internal (plexinterpolate.c:303) n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 3060791648 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x444320: DMPlexCreateExodusNwtun (plexexodusii.c:247) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 4418111328 0x462A1D: DMSetUp_Plex (plex.c:1582) n2: 4418111328 0x50B88B: DMSetUp (dm.c:560) n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229) n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 3060791648 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132) n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 2544974400 0x4076DF: main (Nwtun.c:55) n2: 2385557104 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 2385557104 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 1192778552 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46) n2: 851739512 0x4E2DE5: DMLabelGetStratumIS (plexlabel.c:571) n1: 851739512 0x46481D: DMPlexStratify (plex.c:1768) n1: 851739512 0x9E4D5E: DMPlexInterpolateFaces_Internal (plexinterpolate.c:304) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 341039040 0x4E1E75: DMLabelGetNumValues (plexlabel.c:504) n1: 341039040 0x473CE3: DMPlexGetDepth (plex.c:3971) n1: 341039040 0x9E510C: DMPlexInterpolate (plexinterpolate.c:335) n1: 341039040 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n0: 97680 in 76 places, all below massif's threshold (01.00%) n1: 1610612896 0x9E01D5: kh_resize_HASHIJKL (hash.h:1199) n1: 1610612896 0x9E07CD: kh_put_HASHIJKL (hash.h:1199) n2: 1610612896 0x9E1116: PetscHashIJKLPut (hash.h:1262) n1: 1610612896 0x9E3866: DMPlexInterpolateFaces_Internal (plexinterpolate.c:195) n1: 1610612896 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 1610612896 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 1610612896 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1610612896 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1610612896 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 1207959672 0x9E020C: kh_resize_HASHIJKL (hash.h:1199) n1: 1207959672 0x9E07CD: kh_put_HASHIJKL (hash.h:1199) n2: 1207959672 0x9E1116: PetscHashIJKLPut (hash.h:1262) n1: 1207959672 0x9E3866: DMPlexInterpolateFaces_Internal (plexinterpolate.c:195) n1: 1207959672 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 1207959672 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 1207959672 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1207959672 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1207959672 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 1017989760 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202) n1: 1017989760 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1017989760 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1017989760 0x4076DF: main (Nwtun.c:55) n0: 403062931 in 115 places, all below massif's threshold (01.00%) n1: 402653224 0x4DBEB6: kh_resize_HASHI (hash.h:510) n2: 402653224 0x4DC2CD: kh_put_HASHI (hash.h:510) n2: 402653224 0x4E14DD: DMLabelSetValue (plexlabel.c:450) n1: 402653224 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909) n1: 402653224 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173) n1: 402653224 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 402653224 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 402653224 0x4076DF: main (Nwtun.c:55) n0: 0 in 3 places, all below massif's threshold (01.00%) n0: 0 in 1 place, below massif's threshold (01.00%) #----------- snapshot=62 #----------- time=3200681585287 mem_heap_B=26424451675 mem_heap_extra_B=79925 mem_stacks_B=0 heap_tree=empty #----------- snapshot=63 #----------- time=3209016113153 mem_heap_B=29243024971 mem_heap_extra_B=78917 mem_stacks_B=0 heap_tree=detailed n6: 29243024971 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n8: 21763298824 0x57877E: PetscMallocAlign (mal.c:27) n1: 4418111328 0x462AB1: DMSetUp_Plex (plex.c:1583) n2: 4418111328 0x50B88B: DMSetUp (dm.c:560) n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229) n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 3060791648 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n2: 4418111328 0x46371F: DMPlexSymmetrize (plex.c:1655) n1: 3060791648 0x9E4D06: DMPlexInterpolateFaces_Internal (plexinterpolate.c:303) n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 3060791648 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x444320: DMPlexCreateExodusNwtun (plexexodusii.c:247) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 4418111328 0x462A1D: DMSetUp_Plex (plex.c:1582) n2: 4418111328 0x50B88B: DMSetUp (dm.c:560) n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229) n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 3060791648 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132) n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 2544974400 0x4076DF: main (Nwtun.c:55) n2: 2385557104 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 2385557104 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730) n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n2: 1192778552 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46) n2: 851739512 0x4E2DE5: DMLabelGetStratumIS (plexlabel.c:571) n1: 851739512 0x46481D: DMPlexStratify (plex.c:1768) n1: 851739512 0x9E4D5E: DMPlexInterpolateFaces_Internal (plexinterpolate.c:304) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 341039040 0x4E1E75: DMLabelGetNumValues (plexlabel.c:504) n1: 341039040 0x473CE3: DMPlexGetDepth (plex.c:3971) n1: 341039040 0x9E510C: DMPlexInterpolate (plexinterpolate.c:335) n1: 341039040 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 341039040 0x4076DF: main (Nwtun.c:55) n0: 97680 in 76 places, all below massif's threshold (01.00%) n1: 3221226208 0x9E01D5: kh_resize_HASHIJKL (hash.h:1199) n1: 3221226208 0x9E07CD: kh_put_HASHIJKL (hash.h:1199) n2: 3221226208 0x9E1116: PetscHashIJKLPut (hash.h:1262) n1: 3221226208 0x9E3866: DMPlexInterpolateFaces_Internal (plexinterpolate.c:195) n1: 3221226208 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 3221226208 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 3221226208 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 3221226208 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 3221226208 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 2415919656 0x9E020C: kh_resize_HASHIJKL (hash.h:1199) n1: 2415919656 0x9E07CD: kh_put_HASHIJKL (hash.h:1199) n2: 2415919656 0x9E1116: PetscHashIJKLPut (hash.h:1262) n1: 2415919656 0x9E3866: DMPlexInterpolateFaces_Internal (plexinterpolate.c:195) n1: 2415919656 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 2415919656 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 2415919656 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 2415919656 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 2415919656 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n1: 1017989760 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202) n1: 1017989760 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1017989760 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1017989760 0x4076DF: main (Nwtun.c:55) n0: 421937299 in 115 places, all below massif's threshold (01.00%) n1: 402653224 0x4DBEB6: kh_resize_HASHI (hash.h:510) n2: 402653224 0x4DC2CD: kh_put_HASHI (hash.h:510) n2: 402653224 0x4E14DD: DMLabelSetValue (plexlabel.c:450) n1: 402653224 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909) n1: 402653224 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173) n1: 402653224 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 402653224 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 402653224 0x4076DF: main (Nwtun.c:55) n0: 0 in 3 places, all below massif's threshold (01.00%) n0: 0 in 1 place, below massif's threshold (01.00%) #----------- snapshot=64 #----------- time=3385529782206 mem_heap_B=26009215847 mem_heap_extra_B=71577 mem_stacks_B=0 heap_tree=empty #----------- snapshot=65 #----------- time=3686934054551 mem_heap_B=33110337511 mem_heap_extra_B=78369 mem_stacks_B=0 heap_tree=empty #----------- snapshot=66 #----------- time=3815497683488 mem_heap_B=37195948379 mem_heap_extra_B=80093 mem_stacks_B=0 heap_tree=empty #----------- snapshot=67 #----------- time=3815497691074 mem_heap_B=37195948999 mem_heap_extra_B=80137 mem_stacks_B=0 heap_tree=peak n4: 37195948999 (heap allocation functions) malloc/new/new[], --alloc-fns, etc. n8: 35391116840 0x57877E: PetscMallocAlign (mal.c:27) n1: 8503722144 0x462AB1: DMSetUp_Plex (plex.c:1583) n2: 8503722144 0x50B88B: DMSetUp (dm.c:560) n1: 7146402464 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229) n1: 7146402464 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 7146402464 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 7146402464 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 7146402464 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 7146402464 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 8503722144 0x462A1D: DMSetUp_Plex (plex.c:1582) n2: 8503722144 0x50B88B: DMSetUp (dm.c:560) n1: 7146402464 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229) n1: 7146402464 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 7146402464 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 7146402464 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 7146402464 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 7146402464 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n2: 5113855296 0x52EBD7: PetscSectionSetChart (vsectionis.c:389) n2: 2556927648 0x45A810: DMPlexSetChart (plex.c:729) n1: 2215888608 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 2215888608 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 2215888608 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 2215888608 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 2215888608 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 2215888608 0x4076DF: main (Nwtun.c:55) n0: 341039040 in 1 place, below massif's threshold (01.00%) n2: 2556927648 0x45A874: DMPlexSetChart (plex.c:730) n1: 2215888608 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 2215888608 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 2215888608 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 2215888608 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 2215888608 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 2215888608 0x4076DF: main (Nwtun.c:55) n0: 341039040 in 1 place, below massif's threshold (01.00%) n2: 5113855296 0x52EC32: PetscSectionSetChart (vsectionis.c:389) n2: 2556927648 0x45A810: DMPlexSetChart (plex.c:729) n1: 2215888608 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 2215888608 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 2215888608 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 2215888608 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 2215888608 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 2215888608 0x4076DF: main (Nwtun.c:55) n0: 341039040 in 1 place, below massif's threshold (01.00%) n2: 2556927648 0x45A874: DMPlexSetChart (plex.c:730) n1: 2215888608 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206) n1: 2215888608 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 2215888608 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 2215888608 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 2215888608 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 2215888608 0x4076DF: main (Nwtun.c:55) n0: 341039040 in 1 place, below massif's threshold (01.00%) n2: 4418111328 0x46371F: DMPlexSymmetrize (plex.c:1655) n1: 3060791648 0x9E4D06: DMPlexInterpolateFaces_Internal (plexinterpolate.c:303) n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 3060791648 0x4076DF: main (Nwtun.c:55) n1: 1357319680 0x444320: DMPlexCreateExodusNwtun (plexexodusii.c:247) n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1357319680 0x4076DF: main (Nwtun.c:55) n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132) n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 2544974400 0x4076DF: main (Nwtun.c:55) n2: 1192778552 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46) n2: 851739512 0x4E2DE5: DMLabelGetStratumIS (plexlabel.c:571) n1: 851739512 0x46481D: DMPlexStratify (plex.c:1768) n1: 851739512 0x9E4D5E: DMPlexInterpolateFaces_Internal (plexinterpolate.c:304) n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347) n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252) n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 851739512 0x4076DF: main (Nwtun.c:55) n0: 0 in 1 place, below massif's threshold (01.00%) n0: 341039040 in 1 place, below massif's threshold (01.00%) n0: 97680 in 76 places, all below massif's threshold (01.00%) n1: 1017989760 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202) n1: 1017989760 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 1017989760 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 1017989760 0x4076DF: main (Nwtun.c:55) n1: 402653224 0x4DBEB6: kh_resize_HASHI (hash.h:510) n2: 402653224 0x4DC2CD: kh_put_HASHI (hash.h:510) n2: 402653224 0x4E14DD: DMLabelSetValue (plexlabel.c:450) n1: 402653224 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909) n1: 402653224 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173) n1: 402653224 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53) n1: 402653224 0x4152C9: CreateMesh (SetupMeshes.c:23) n0: 402653224 0x4076DF: main (Nwtun.c:55) n0: 0 in 3 places, all below massif's threshold (01.00%) n0: 0 in 1 place, below massif's threshold (01.00%) n0: 384189175 in 117 places, all below massif's threshold (01.00%) #----------- snapshot=68 #----------- time=3818343537621 mem_heap_B=37207106171 mem_heap_extra_B=87909 mem_stacks_B=0 heap_tree=empty #----------- snapshot=69 #----------- time=3821057220870 mem_heap_B=37218264795 mem_heap_extra_B=86789 mem_stacks_B=0 heap_tree=empty #----------- snapshot=70 #----------- time=3826940637881 mem_heap_B=37240578455 mem_heap_extra_B=88137 mem_stacks_B=0 heap_tree=empty #----------- snapshot=71 #----------- time=3840182989443 mem_heap_B=37285208695 mem_heap_extra_B=87913 mem_stacks_B=0 heap_tree=empty #----------- snapshot=72 #----------- time=3883346460203 mem_heap_B=37374469063 mem_heap_extra_B=87577 mem_stacks_B=0 heap_tree=empty #----------- snapshot=73 #----------- time=3936192268561 mem_heap_B=37552989239 mem_heap_extra_B=87465 mem_stacks_B=0 heap_tree=empty From knepley at gmail.com Thu Feb 18 19:58:14 2016 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 18 Feb 2016 19:58:14 -0600 Subject: [petsc-users] SNESComputeJacobianDefaultColor use too much memory In-Reply-To: <56C671CA.6030805@gmail.com> References: <56C319D0.9060208@gmail.com> <684F7147-4F58-42E3-9C9E-2C2297383276@mcs.anl.gov> <56C3EC9B.5080306@gmail.com> <1199051F-2EC1-4462-A7C0-E2DDCA95A4C3@mcs.anl.gov> <56C409D9.70002@gmail.com> <2424BDF0-AD22-40EB-9920-AF697BFF4DAF@mcs.anl.gov> <56C425F8.1060306@gmail.com> <68210D25-00D1-4273-9C83-5E6A299D17E7@mcs.anl.gov> <56C671CA.6030805@gmail.com> Message-ID: On Thu, Feb 18, 2016 at 7:37 PM, Rongliang Chen wrote: > Hi Barry, > > When I increase the size of the mesh, another memory issue comes out. The > DMPlexInterpolateFaces_Internal takes too much memory. The massif output is > attached. Any suggestions for this? Thanks. > It is creating parts of the mesh. I do not believe it allocates much temp storage. Is that what you see? Matt > Best, > Rongliang > > On 02/18/2016 04:11 AM, Barry Smith wrote: > >> Hmm, Matt, this is nuts. This >> >> if (size > 1) { >> /* create a sequential iscoloring on all processors */ >> ierr = MatGetSeqNonzeroStructure(mat,&mat_seq);CHKERRQ(ierr); >> } >> >> It sequentializes the graph. >> >> It looks like the only parallel coloring for Jacobians is >> MATCOLORINGGREEDY? Try that. >> >> Maybe that should be the default? >> >> Barry >> >> >> On Feb 17, 2016, at 1:49 AM, Rongliang Chen >>> wrote: >>> >>> Dear Barry, >>> >>> The massif output for the large case is attached. It shows that the job >>> was kill in the function MatGetSubMatrix_MPIAIJ_All(). Any suggestions? >>> >>> -------------------- >>> [8]PETSC ERROR: #1 MatGetSubMatrix_MPIAIJ_All() line 622 in >>> /home/rlchen/soft/petsc-3.5.2/src/mat/impls/aij/mpi/mpiov.c >>> [8]PETSC ERROR: #2 MatGetSeqNonzeroStructure_MPIAIJ() line 3010 in >>> /home/rlchen/soft/petsc-3.5.2/src/mat/impls/aij/mpi/mpiaij.c >>> [8]PETSC ERROR: #3 MatGetSeqNonzeroStructure() line 6487 in >>> /home/rlchen/soft/petsc-3.5.2/src/mat/interface/matrix.c >>> [8]PETSC ERROR: #4 MatColoringApply_SL() line 78 in >>> /home/rlchen/soft/petsc-3.5.2/src/mat/color/impls/minpack/color.c >>> [8]PETSC ERROR: #5 MatColoringApply() line 379 in >>> /home/rlchen/soft/petsc-3.5.2/src/mat/color/interface/matcoloring.c >>> [8]PETSC ERROR: #6 SNESComputeJacobianDefaultColor() line 71 in >>> /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snesj2.c >>> [8]PETSC ERROR: #7 FormJacobian() line 58 in >>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/SetupJacobian.c >>> [8]PETSC ERROR: #8 SNESComputeJacobian() line 2193 in >>> /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c >>> [8]PETSC ERROR: #9 SNESSolve_NEWTONLS() line 230 in >>> /home/rlchen/soft/petsc-3.5.2/src/snes/impls/ls/ls.c >>> [8]PETSC ERROR: #10 SNESSolve() line 3743 in >>> /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c >>> [8]PETSC ERROR: #11 SolveTimeDependent() line 758 in >>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/Nwtun.c >>> [8]PETSC ERROR: #12 main() line 417 in >>> /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/Nwtun.c >>> ------------------- >>> >>> Best, >>> Rongliang >>> >>> On 02/17/2016 02:09 PM, Barry Smith wrote: >>> >>>> Yes, this is the type of output I was expecting. Now you need to >>>> produce it for a large case. >>>> >>>> Barry >>>> >>>> On Feb 16, 2016, at 11:49 PM, Rongliang Chen >>>>> wrote: >>>>> >>>>> Hi Barry, >>>>> >>>>> I run the code with valgrind on a workstation for a smaller case and >>>>> it produces some ASCII information (see attached). Is this helpful? >>>>> >>>>> Best, >>>>> Rongliang >>>>> >>>>> On 02/17/2016 01:30 PM, Barry Smith wrote: >>>>> >>>>>> Hmm, something didn't work right with massif. I should give a >>>>>> bunch of ASCII information about how much memory is used at different times >>>>>> in the code. You may need to play around with the massif options and google >>>>>> documentation on how to get it to provide useful information. Once it >>>>>> produces the useful information it will be very helpful. >>>>>> >>>>>> time=0 >>>>>> mem_heap_B=0 >>>>>> mem_heap_extra_B=0 >>>>>> mem_stacks_B=0 >>>>>> heap_tree=empty >>>>>> >>>>>> >>>>>> On Feb 16, 2016, at 9:44 PM, Rongliang Chen >>>>>>> wrote: >>>>>>> >>>>>>> Dear Barry, >>>>>>> >>>>>>> Many thanks for your reply. >>>>>>> >>>>>>> I checked with the valgrind and did not obtain any outputs >>>>>>> (massif.out.) because the job was killed before it reached the end. >>>>>>> >>>>>>> Then I switch to a smaller case, it works well and one of the output >>>>>>> is attached (I did not find any useful information in it). The output with >>>>>>> the option -mat_coloring_view is followed, which shows that the number of >>>>>>> colors is 65. Any ideas for this? >>>>>>> >>>>>>> MatColoring Object: 480 MPI processes >>>>>>> type: sl >>>>>>> Weight type: RANDOM >>>>>>> Distance 2, Max. Colors 65535 >>>>>>> Number of colors 65 >>>>>>> Number of total columns 1637350 >>>>>>> >>>>>>> Best regards, >>>>>>> Rongliang >>>>>>> >>>>>>> On 02/17/2016 01:13 AM, Barry Smith wrote: >>>>>>> >>>>>>>> How many colors are needed? >>>>>>>> >>>>>>>> You need to produce a breakdown of where all the memory is being >>>>>>>> used. For example valgrind with the >>>>>>>> http://valgrind.org/docs/manual/ms-manual.html >>>>>>>> >>>>>>>> >>>>>>>> Barry >>>>>>>> >>>>>>>> >>>>>>>> On Feb 16, 2016, at 6:45 AM, Rongliang Chen < >>>>>>>>> rongliang.chan at gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Dear all, >>>>>>>>> >>>>>>>>> I am using the DMPlex to solve a PDE on a unstructured mesh and I >>>>>>>>> use the SNESComputeJacobianDefaultColor to compute the Jacobian matrix. >>>>>>>>> >>>>>>>>> My code works well for small problems (such as problem with >>>>>>>>> 3.3x10^5 cells using 120 cores) but when I increase the number of cells >>>>>>>>> (2.6x10^6 cells using 1920 cores), I go out of memory in the function >>>>>>>>> MatColoringApply. It shows that one of the cores uses over 11G memory. I >>>>>>>>> think this is unreasonable. Do you have any suggestions for debugging this >>>>>>>>> problem? >>>>>>>>> >>>>>>>>> Best regards, >>>>>>>>> Rongliang >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From rongliang.chan at gmail.com Thu Feb 18 20:07:45 2016 From: rongliang.chan at gmail.com (Rongliang Chen) Date: Fri, 19 Feb 2016 10:07:45 +0800 Subject: [petsc-users] SNESComputeJacobianDefaultColor use too much memory In-Reply-To: References: <56C319D0.9060208@gmail.com> <684F7147-4F58-42E3-9C9E-2C2297383276@mcs.anl.gov> <56C3EC9B.5080306@gmail.com> <1199051F-2EC1-4462-A7C0-E2DDCA95A4C3@mcs.anl.gov> <56C409D9.70002@gmail.com> <2424BDF0-AD22-40EB-9920-AF697BFF4DAF@mcs.anl.gov> <56C425F8.1060306@gmail.com> <68210D25-00D1-4273-9C83-5E6A299D17E7@mcs.anl.gov> <56C671CA.6030805@gmail.com> Message-ID: <56C678F1.4090902@gmail.com> Hi Matt, Thanks for your reply. The job was killed in the function DMPlexInterpolateFaces_Internal and the administrator told me that the reason is out of memory. The error messages are followed: --------------------------- [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] PetscHashIJKLPut line 1261 /home/rlchen/soft/petsc-3.5.2/include/../src/sys/utils/hash.h [0]PETSC ERROR: [0] DMPlexInterpolateFaces_Internal line 154 /home/rlchen/soft/petsc-3.5.2/src/dm/impls/plex/plexinterpolate.c [0]PETSC ERROR: [0] DMPlexInterpolate line 333 /home/rlchen/soft/petsc-3.5.2/src/dm/impls/plex/plexinterpolate.c [0]PETSC ERROR: [0] DMPlexCreateExodusNwtun line 98 /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/src/plexexodusii.c [0]PETSC ERROR: [0] DMPlexCreateExodusFromFileNwtun line 44 /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/src/plexexodusii.c [0]PETSC ERROR: [0] CreateMesh line 19 /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/SetupMeshes.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.5.2, unknown [0]PETSC ERROR: ./Nwtun on a 64bit-debug named leszek-ThinkStation-C20 by rlchen Thu Feb 18 23:07:32 2016 [0]PETSC ERROR: Configure options --download-fblaslapack --download-blacs --download-scalapack --download-metis --download-parmetis --download-exodusii --download-netcdf --download-hdf5 --with-64-bit-indices --with-c2html=0 --with-mpi=1 --with-debugging=1 --with-shared-libraries=0 [0]PETSC ERROR: #1 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 ===================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = EXIT CODE: 15104 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES ===================================================================================== ---------------------------------- Best, Rongliang On 02/19/2016 09:58 AM, Matthew Knepley wrote: > On Thu, Feb 18, 2016 at 7:37 PM, Rongliang Chen > > wrote: > > Hi Barry, > > When I increase the size of the mesh, another memory issue comes > out. The DMPlexInterpolateFaces_Internal takes too much memory. > The massif output is attached. Any suggestions for this? Thanks. > > > It is creating parts of the mesh. I do not believe it allocates much > temp storage. Is that what you see? > > Matt > > Best, > Rongliang > > On 02/18/2016 04:11 AM, Barry Smith wrote: > > Hmm, Matt, this is nuts. This > > if (size > 1) { > /* create a sequential iscoloring on all processors */ > ierr = MatGetSeqNonzeroStructure(mat,&mat_seq);CHKERRQ(ierr); > } > > It sequentializes the graph. > > It looks like the only parallel coloring for Jacobians is > MATCOLORINGGREEDY? Try that. > > Maybe that should be the default? > > Barry > > > On Feb 17, 2016, at 1:49 AM, Rongliang Chen > > wrote: > > Dear Barry, > > The massif output for the large case is attached. It shows > that the job was kill in the function > MatGetSubMatrix_MPIAIJ_All(). Any suggestions? > > -------------------- > [8]PETSC ERROR: #1 MatGetSubMatrix_MPIAIJ_All() line 622 > in /home/rlchen/soft/petsc-3.5.2/src/mat/impls/aij/mpi/mpiov.c > [8]PETSC ERROR: #2 MatGetSeqNonzeroStructure_MPIAIJ() line > 3010 in > /home/rlchen/soft/petsc-3.5.2/src/mat/impls/aij/mpi/mpiaij.c > [8]PETSC ERROR: #3 MatGetSeqNonzeroStructure() line 6487 > in /home/rlchen/soft/petsc-3.5.2/src/mat/interface/matrix.c > [8]PETSC ERROR: #4 MatColoringApply_SL() line 78 in > /home/rlchen/soft/petsc-3.5.2/src/mat/color/impls/minpack/color.c > [8]PETSC ERROR: #5 MatColoringApply() line 379 in > /home/rlchen/soft/petsc-3.5.2/src/mat/color/interface/matcoloring.c > [8]PETSC ERROR: #6 SNESComputeJacobianDefaultColor() line > 71 in > /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snesj2.c > [8]PETSC ERROR: #7 FormJacobian() line 58 in > /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/SetupJacobian.c > [8]PETSC ERROR: #8 SNESComputeJacobian() line 2193 in > /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c > [8]PETSC ERROR: #9 SNESSolve_NEWTONLS() line 230 in > /home/rlchen/soft/petsc-3.5.2/src/snes/impls/ls/ls.c > [8]PETSC ERROR: #10 SNESSolve() line 3743 in > /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c > [8]PETSC ERROR: #11 SolveTimeDependent() line 758 in > /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/Nwtun.c > [8]PETSC ERROR: #12 main() line 417 in > /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/Nwtun.c > ------------------- > > Best, > Rongliang > > On 02/17/2016 02:09 PM, Barry Smith wrote: > > Yes, this is the type of output I was expecting. > Now you need to produce it for a large case. > > Barry > > On Feb 16, 2016, at 11:49 PM, Rongliang Chen > > wrote: > > Hi Barry, > > I run the code with valgrind on a workstation for > a smaller case and it produces some ASCII > information (see attached). Is this helpful? > > Best, > Rongliang > > On 02/17/2016 01:30 PM, Barry Smith wrote: > > Hmm, something didn't work right with > massif. I should give a bunch of ASCII > information about how much memory is used at > different times in the code. You may need to > play around with the massif options and google > documentation on how to get it to provide > useful information. Once it produces the > useful information it will be very helpful. > > time=0 > mem_heap_B=0 > mem_heap_extra_B=0 > mem_stacks_B=0 > heap_tree=empty > > > On Feb 16, 2016, at 9:44 PM, Rongliang > Chen > wrote: > > Dear Barry, > > Many thanks for your reply. > > I checked with the valgrind and did not > obtain any outputs (massif.out.) > because the job was killed before it > reached the end. > > Then I switch to a smaller case, it works > well and one of the output is attached (I > did not find any useful information in > it). The output with the option > -mat_coloring_view is followed, which > shows that the number of colors is 65. Any > ideas for this? > > MatColoring Object: 480 MPI processes > type: sl > Weight type: RANDOM > Distance 2, Max. Colors 65535 > Number of colors 65 > Number of total columns 1637350 > > Best regards, > Rongliang > > On 02/17/2016 01:13 AM, Barry Smith wrote: > > How many colors are needed? > > You need to produce a breakdown of > where all the memory is being used. > For example valgrind with the > http://valgrind.org/docs/manual/ms-manual.html > > > Barry > > > On Feb 16, 2016, at 6:45 AM, > Rongliang Chen > > > wrote: > > Dear all, > > I am using the DMPlex to solve a > PDE on a unstructured mesh and I > use the > SNESComputeJacobianDefaultColor to > compute the Jacobian matrix. > > My code works well for small > problems (such as problem with > 3.3x10^5 cells using 120 cores) > but when I increase the number of > cells (2.6x10^6 cells using 1920 > cores), I go out of memory in the > function MatColoringApply. It > shows that one of the cores uses > over 11G memory. I think this is > unreasonable. Do you have any > suggestions for debugging this > problem? > > Best regards, > Rongliang > > > > > > > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From johncfdcfd at gmail.com Fri Feb 19 01:50:19 2016 From: johncfdcfd at gmail.com (John Albequerque) Date: Fri, 19 Feb 2016 13:20:19 +0530 Subject: [petsc-users] Doubt in MPIPreallocation Message-ID: Could someone help me setting up this matrix i.e from preallocation to Set Values for an MPIAIJ matrix which has its entries. *1 1 1 1 0 0 0 00 1 1 1 1 0 0 00 0 1 1 1 1 0 00 0 0 1 1 1 1 00 0 0 0 1 1 1 1* I am trying to do it from 3 days now but not able to figure it out. Could someone please help me do this. Thanks. ---- John Albequerque. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Feb 19 01:55:18 2016 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 19 Feb 2016 01:55:18 -0600 Subject: [petsc-users] Doubt in MPIPreallocation In-Reply-To: References: Message-ID: On Fri, Feb 19, 2016 at 1:50 AM, John Albequerque wrote: > Could someone help me setting up this matrix i.e from preallocation to Set > Values for an MPIAIJ matrix which has its entries. > > > > > > *1 1 1 1 0 0 0 00 1 1 1 1 0 0 00 0 1 1 1 1 0 00 0 > 0 1 1 1 1 00 0 0 0 1 1 1 1* > > I am trying to do it from 3 days now but not able to figure it out. Could > someone please help me do this. > It will be enough to say that each row has 3 off-diagonal values, so you can use the simplified preallocation rather than giving a value for each row individually. Matt > Thanks. > > ---- > John Albequerque. > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From johncfdcfd at gmail.com Fri Feb 19 03:27:09 2016 From: johncfdcfd at gmail.com (John Albequerque) Date: Fri, 19 Feb 2016 14:57:09 +0530 Subject: [petsc-users] Doubt in MPIPreallocation In-Reply-To: References: Message-ID: I came up with this could you please have a look at it as I am not getting it. I am getting an error *New nonzero at (0,4) caused a malloc*http://pastebin.com/ZxT7SWtc Could you please have a look at it. --------------- Thanks, John Albequerque On Fri, Feb 19, 2016 at 1:25 PM, Matthew Knepley wrote: > On Fri, Feb 19, 2016 at 1:50 AM, John Albequerque > wrote: > >> Could someone help me setting up this matrix i.e from preallocation to >> Set Values for an MPIAIJ matrix which has its entries. >> >> >> >> >> >> *1 1 1 1 0 0 0 00 1 1 1 1 0 0 00 0 1 1 1 1 0 00 0 >> 0 1 1 1 1 00 0 0 0 1 1 1 1* >> >> I am trying to do it from 3 days now but not able to figure it out. Could >> someone please help me do this. >> > > It will be enough to say that each row has 3 off-diagonal values, so you > can use the simplified preallocation rather > than giving a value for each row individually. > > Matt > > >> Thanks. >> >> ---- >> John Albequerque. >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Feb 19 03:31:49 2016 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 19 Feb 2016 03:31:49 -0600 Subject: [petsc-users] Doubt in MPIPreallocation In-Reply-To: References: Message-ID: On Fri, Feb 19, 2016 at 3:27 AM, John Albequerque wrote: > I came up with this could you please have a look at it as I am not getting > it. I am getting an error > > > > *New nonzero at (0,4) caused a malloc*http://pastebin.com/ZxT7SWtc > > Could you please have a look at it. > It looks like you are only allocating the upper triangle: if((j-i<=3) && (j>=i)) values[(i-low)*nc+j]=1; else values[(i-low)*nc+j]=0; Matt > --------------- > Thanks, > John Albequerque > > > On Fri, Feb 19, 2016 at 1:25 PM, Matthew Knepley > wrote: > >> On Fri, Feb 19, 2016 at 1:50 AM, John Albequerque >> wrote: >> >>> Could someone help me setting up this matrix i.e from preallocation to >>> Set Values for an MPIAIJ matrix which has its entries. >>> >>> >>> >>> >>> >>> *1 1 1 1 0 0 0 00 1 1 1 1 0 0 00 0 1 1 1 1 0 00 0 >>> 0 1 1 1 1 00 0 0 0 1 1 1 1* >>> >>> I am trying to do it from 3 days now but not able to figure it out. >>> Could someone please help me do this. >>> >> >> It will be enough to say that each row has 3 off-diagonal values, so you >> can use the simplified preallocation rather >> than giving a value for each row individually. >> >> Matt >> >> >>> Thanks. >>> >>> ---- >>> John Albequerque. >>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From johncfdcfd at gmail.com Fri Feb 19 04:34:46 2016 From: johncfdcfd at gmail.com (John Albequerque) Date: Fri, 19 Feb 2016 16:04:46 +0530 Subject: [petsc-users] Doubt in MPIPreallocation In-Reply-To: References: Message-ID: Currently I have used single processor. And without the Preallocation it works fine. And gives me the correct output as well. I have posted the full code for reference. *Here:* http://pastebin.com/kWUGHqSM On Fri, Feb 19, 2016 at 3:01 PM, Matthew Knepley wrote: > On Fri, Feb 19, 2016 at 3:27 AM, John Albequerque > wrote: > >> I came up with this could you please have a look at it as I am not >> getting it. I am getting an error >> >> >> >> *New nonzero at (0,4) caused a malloc*http://pastebin.com/ZxT7SWtc >> >> Could you please have a look at it. >> > > It looks like you are only allocating the upper triangle: > > if((j-i<=3) && (j>=i)) > values[(i-low)*nc+j]=1; > else > values[(i-low)*nc+j]=0; > > Matt > > >> --------------- >> Thanks, >> John Albequerque >> >> >> On Fri, Feb 19, 2016 at 1:25 PM, Matthew Knepley >> wrote: >> >>> On Fri, Feb 19, 2016 at 1:50 AM, John Albequerque >>> wrote: >>> >>>> Could someone help me setting up this matrix i.e from preallocation to >>>> Set Values for an MPIAIJ matrix which has its entries. >>>> >>>> >>>> >>>> >>>> >>>> *1 1 1 1 0 0 0 00 1 1 1 1 0 0 00 0 1 1 1 1 0 00 >>>> 0 0 1 1 1 1 00 0 0 0 1 1 1 1* >>>> >>>> I am trying to do it from 3 days now but not able to figure it out. >>>> Could someone please help me do this. >>>> >>> >>> It will be enough to say that each row has 3 off-diagonal values, so you >>> can use the simplified preallocation rather >>> than giving a value for each row individually. >>> >>> Matt >>> >>> >>>> Thanks. >>>> >>>> ---- >>>> John Albequerque. >>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Feb 19 04:47:07 2016 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 19 Feb 2016 04:47:07 -0600 Subject: [petsc-users] Doubt in MPIPreallocation In-Reply-To: References: Message-ID: On Fri, Feb 19, 2016 at 4:34 AM, John Albequerque wrote: > Currently I have used single processor. And without the Preallocation it > works fine. And gives me the correct output as well. I have posted the full > code for reference. > *Here:* http://pastebin.com/kWUGHqSM > Look at the last row. Your condition 'j >= i' will produce no off-diagonal rows here, but clearly you have them, and this is the error you are getting. Matt > On Fri, Feb 19, 2016 at 3:01 PM, Matthew Knepley > wrote: > >> On Fri, Feb 19, 2016 at 3:27 AM, John Albequerque >> wrote: >> >>> I came up with this could you please have a look at it as I am not >>> getting it. I am getting an error >>> >>> >>> >>> *New nonzero at (0,4) caused a malloc*http://pastebin.com/ZxT7SWtc >>> >>> Could you please have a look at it. >>> >> >> It looks like you are only allocating the upper triangle: >> >> if((j-i<=3) && (j>=i)) >> values[(i-low)*nc+j]=1; >> else >> values[(i-low)*nc+j]=0; >> >> Matt >> >> >>> --------------- >>> Thanks, >>> John Albequerque >>> >>> >>> On Fri, Feb 19, 2016 at 1:25 PM, Matthew Knepley >>> wrote: >>> >>>> On Fri, Feb 19, 2016 at 1:50 AM, John Albequerque >>> > wrote: >>>> >>>>> Could someone help me setting up this matrix i.e from preallocation to >>>>> Set Values for an MPIAIJ matrix which has its entries. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> *1 1 1 1 0 0 0 00 1 1 1 1 0 0 00 0 1 1 1 1 0 00 >>>>> 0 0 1 1 1 1 00 0 0 0 1 1 1 1* >>>>> >>>>> I am trying to do it from 3 days now but not able to figure it out. >>>>> Could someone please help me do this. >>>>> >>>> >>>> It will be enough to say that each row has 3 off-diagonal values, so >>>> you can use the simplified preallocation rather >>>> than giving a value for each row individually. >>>> >>>> Matt >>>> >>>> >>>>> Thanks. >>>>> >>>>> ---- >>>>> John Albequerque. >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From johncfdcfd at gmail.com Fri Feb 19 05:09:04 2016 From: johncfdcfd at gmail.com (John Albequerque) Date: Fri, 19 Feb 2016 16:39:04 +0530 Subject: [petsc-users] Doubt in MPIPreallocation In-Reply-To: References: Message-ID: Dear Matt, Sorry, I am not able to get you, as if I do not provide i>=j then the matrix mentioned in the earlier mail won't be generated. For the code I mentioned earlier, the output obtained is as follows - *Mat Object: 1 MPI processes type: mpiaijrow 0: (0, 1) (1, 1) (2, 1) (3, 1) (4, 0) (5, 0) (6, 0) (7, 0) row 1: (0, 0) (1, 1) (2, 1) (3, 1) (4, 1) (5, 0) (6, 0) (7, 0) row 2: (0, 0) (1, 0) (2, 1) (3, 1) (4, 1) (5, 1) (6, 0) (7, 0) row 3: (0, 0) (1, 0) (2, 0) (3, 1) (4, 1) (5, 1) (6, 1) (7, 0) row 4: (0, 0) (1, 0) (2, 0) (3, 0) (4, 1) (5, 1) (6, 1) (7, 1)* which is satisfactory, but once I use *MatMPIAIJSetPreallocation(A,0,d_nnz,0,o_nnz); *the code encounters a runtime error. And throws a malloc error as - *[0]PETSC ERROR: New nonzero at (0,4) caused a mallocUse MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check* Thanks ---- John Albequerque. On Fri, Feb 19, 2016 at 4:17 PM, Matthew Knepley wrote: > On Fri, Feb 19, 2016 at 4:34 AM, John Albequerque > wrote: > >> Currently I have used single processor. And without the Preallocation it >> works fine. And gives me the correct output as well. I have posted the full >> code for reference. >> *Here:* http://pastebin.com/kWUGHqSM >> > > Look at the last row. Your condition 'j >= i' will produce no off-diagonal > rows here, but > clearly you have them, and this is the error you are getting. > > Matt > > >> On Fri, Feb 19, 2016 at 3:01 PM, Matthew Knepley >> wrote: >> >>> On Fri, Feb 19, 2016 at 3:27 AM, John Albequerque >>> wrote: >>> >>>> I came up with this could you please have a look at it as I am not >>>> getting it. I am getting an error >>>> >>>> >>>> >>>> *New nonzero at (0,4) caused a malloc*http://pastebin.com/ZxT7SWtc >>>> >>>> Could you please have a look at it. >>>> >>> >>> It looks like you are only allocating the upper triangle: >>> >>> if((j-i<=3) && (j>=i)) >>> values[(i-low)*nc+j]=1; >>> else >>> values[(i-low)*nc+j]=0; >>> >>> Matt >>> >>> >>>> --------------- >>>> Thanks, >>>> John Albequerque >>>> >>>> >>>> On Fri, Feb 19, 2016 at 1:25 PM, Matthew Knepley >>>> wrote: >>>> >>>>> On Fri, Feb 19, 2016 at 1:50 AM, John Albequerque < >>>>> johncfdcfd at gmail.com> wrote: >>>>> >>>>>> Could someone help me setting up this matrix i.e from preallocation >>>>>> to Set Values for an MPIAIJ matrix which has its entries. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> *1 1 1 1 0 0 0 00 1 1 1 1 0 0 00 0 1 1 1 1 0 00 >>>>>> 0 0 1 1 1 1 00 0 0 0 1 1 1 1* >>>>>> >>>>>> I am trying to do it from 3 days now but not able to figure it out. >>>>>> Could someone please help me do this. >>>>>> >>>>> >>>>> It will be enough to say that each row has 3 off-diagonal values, so >>>>> you can use the simplified preallocation rather >>>>> than giving a value for each row individually. >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Thanks. >>>>>> >>>>>> ---- >>>>>> John Albequerque. >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Feb 19 05:45:50 2016 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 19 Feb 2016 05:45:50 -0600 Subject: [petsc-users] Doubt in MPIPreallocation In-Reply-To: References: Message-ID: On Fri, Feb 19, 2016 at 5:09 AM, John Albequerque wrote: > Dear Matt, > > Sorry, I am not able to get you, as if I do not provide i>=j then the > matrix mentioned in the earlier mail won't be generated. For the code I > mentioned earlier, the output obtained is as follows - > > > > > > > > > *Mat Object: 1 MPI processes type: mpiaijrow 0: (0, 1) (1, 1) (2, 1) > (3, 1) (4, 0) (5, 0) (6, 0) (7, 0) row 1: (0, 0) (1, 1) (2, 1) (3, > 1) (4, 1) (5, 0) (6, 0) (7, 0) row 2: (0, 0) (1, 0) (2, 1) (3, 1) > (4, 1) (5, 1) (6, 0) (7, 0) row 3: (0, 0) (1, 0) (2, 0) (3, 1) (4, > 1) (5, 1) (6, 1) (7, 0) row 4: (0, 0) (1, 0) (2, 0) (3, 0) (4, 1) > (5, 1) (6, 1) (7, 1)* > which is satisfactory, but once I use *MatMPIAIJSetPreallocation(A,0,d_nnz,0,o_nnz); > *the code encounters a runtime error. And throws a malloc error as - > > > *[0]PETSC ERROR: New nonzero at (0,4) caused a mallocUse MatSetOption(A, > MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check* > Print out your dnz and onz. You can clearly see they are wrong. The error is your condition which I pointed out above. Matt > Thanks > > ---- > John Albequerque. > > > On Fri, Feb 19, 2016 at 4:17 PM, Matthew Knepley > wrote: > >> On Fri, Feb 19, 2016 at 4:34 AM, John Albequerque >> wrote: >> >>> Currently I have used single processor. And without the Preallocation it >>> works fine. And gives me the correct output as well. I have posted the full >>> code for reference. >>> *Here:* http://pastebin.com/kWUGHqSM >>> >> >> Look at the last row. Your condition 'j >= i' will produce no >> off-diagonal rows here, but >> clearly you have them, and this is the error you are getting. >> >> Matt >> >> >>> On Fri, Feb 19, 2016 at 3:01 PM, Matthew Knepley >>> wrote: >>> >>>> On Fri, Feb 19, 2016 at 3:27 AM, John Albequerque >>> > wrote: >>>> >>>>> I came up with this could you please have a look at it as I am not >>>>> getting it. I am getting an error >>>>> >>>>> >>>>> >>>>> *New nonzero at (0,4) caused a malloc*http://pastebin.com/ZxT7SWtc >>>>> >>>>> Could you please have a look at it. >>>>> >>>> >>>> It looks like you are only allocating the upper triangle: >>>> >>>> if((j-i<=3) && (j>=i)) >>>> values[(i-low)*nc+j]=1; >>>> else >>>> values[(i-low)*nc+j]=0; >>>> >>>> Matt >>>> >>>> >>>>> --------------- >>>>> Thanks, >>>>> John Albequerque >>>>> >>>>> >>>>> On Fri, Feb 19, 2016 at 1:25 PM, Matthew Knepley >>>>> wrote: >>>>> >>>>>> On Fri, Feb 19, 2016 at 1:50 AM, John Albequerque < >>>>>> johncfdcfd at gmail.com> wrote: >>>>>> >>>>>>> Could someone help me setting up this matrix i.e from preallocation >>>>>>> to Set Values for an MPIAIJ matrix which has its entries. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> *1 1 1 1 0 0 0 00 1 1 1 1 0 0 00 0 1 1 1 1 0 >>>>>>> 00 0 0 1 1 1 1 00 0 0 0 1 1 1 1* >>>>>>> >>>>>>> I am trying to do it from 3 days now but not able to figure it out. >>>>>>> Could someone please help me do this. >>>>>>> >>>>>> >>>>>> It will be enough to say that each row has 3 off-diagonal values, so >>>>>> you can use the simplified preallocation rather >>>>>> than giving a value for each row individually. >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Thanks. >>>>>>> >>>>>>> ---- >>>>>>> John Albequerque. >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From johncfdcfd at gmail.com Fri Feb 19 08:01:26 2016 From: johncfdcfd at gmail.com (John Albequerque) Date: Fri, 19 Feb 2016 19:31:26 +0530 Subject: [petsc-users] Doubt in MPIPreallocation In-Reply-To: References: Message-ID: I am really sorry Matthew but I am not able to get your point. It would be very kind of you if you could show me the changes in the code. Thanks ---- John Albequerque. Materials Science Lab, UIUC On Fri, Feb 19, 2016 at 5:15 PM, Matthew Knepley wrote: > On Fri, Feb 19, 2016 at 5:09 AM, John Albequerque > wrote: > >> Dear Matt, >> >> Sorry, I am not able to get you, as if I do not provide i>=j then the >> matrix mentioned in the earlier mail won't be generated. For the code I >> mentioned earlier, the output obtained is as follows - >> >> >> >> >> >> >> >> >> *Mat Object: 1 MPI processes type: mpiaijrow 0: (0, 1) (1, 1) (2, 1) >> (3, 1) (4, 0) (5, 0) (6, 0) (7, 0) row 1: (0, 0) (1, 1) (2, 1) (3, >> 1) (4, 1) (5, 0) (6, 0) (7, 0) row 2: (0, 0) (1, 0) (2, 1) (3, 1) >> (4, 1) (5, 1) (6, 0) (7, 0) row 3: (0, 0) (1, 0) (2, 0) (3, 1) (4, >> 1) (5, 1) (6, 1) (7, 0) row 4: (0, 0) (1, 0) (2, 0) (3, 0) (4, 1) >> (5, 1) (6, 1) (7, 1)* >> which is satisfactory, but once I use *MatMPIAIJSetPreallocation(A,0,d_nnz,0,o_nnz); >> *the code encounters a runtime error. And throws a malloc error as - >> >> >> *[0]PETSC ERROR: New nonzero at (0,4) caused a mallocUse MatSetOption(A, >> MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check* >> > > Print out your dnz and onz. You can clearly see they are wrong. The error > is your condition which I pointed out above. > > Matt > > >> Thanks >> >> ---- >> John Albequerque. >> >> >> On Fri, Feb 19, 2016 at 4:17 PM, Matthew Knepley >> wrote: >> >>> On Fri, Feb 19, 2016 at 4:34 AM, John Albequerque >>> wrote: >>> >>>> Currently I have used single processor. And without the Preallocation >>>> it works fine. And gives me the correct output as well. I have posted the >>>> full code for reference. >>>> *Here:* http://pastebin.com/kWUGHqSM >>>> >>> >>> Look at the last row. Your condition 'j >= i' will produce no >>> off-diagonal rows here, but >>> clearly you have them, and this is the error you are getting. >>> >>> Matt >>> >>> >>>> On Fri, Feb 19, 2016 at 3:01 PM, Matthew Knepley >>>> wrote: >>>> >>>>> On Fri, Feb 19, 2016 at 3:27 AM, John Albequerque < >>>>> johncfdcfd at gmail.com> wrote: >>>>> >>>>>> I came up with this could you please have a look at it as I am not >>>>>> getting it. I am getting an error >>>>>> >>>>>> >>>>>> >>>>>> *New nonzero at (0,4) caused a malloc*http://pastebin.com/ZxT7SWtc >>>>>> >>>>>> Could you please have a look at it. >>>>>> >>>>> >>>>> It looks like you are only allocating the upper triangle: >>>>> >>>>> if((j-i<=3) && (j>=i)) >>>>> values[(i-low)*nc+j]=1; >>>>> else >>>>> values[(i-low)*nc+j]=0; >>>>> >>>>> Matt >>>>> >>>>> >>>>>> --------------- >>>>>> Thanks, >>>>>> John Albequerque >>>>>> >>>>>> >>>>>> On Fri, Feb 19, 2016 at 1:25 PM, Matthew Knepley >>>>>> wrote: >>>>>> >>>>>>> On Fri, Feb 19, 2016 at 1:50 AM, John Albequerque < >>>>>>> johncfdcfd at gmail.com> wrote: >>>>>>> >>>>>>>> Could someone help me setting up this matrix i.e from preallocation >>>>>>>> to Set Values for an MPIAIJ matrix which has its entries. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> *1 1 1 1 0 0 0 00 1 1 1 1 0 0 00 0 1 1 1 1 0 >>>>>>>> 00 0 0 1 1 1 1 00 0 0 0 1 1 1 1* >>>>>>>> >>>>>>>> I am trying to do it from 3 days now but not able to figure it out. >>>>>>>> Could someone please help me do this. >>>>>>>> >>>>>>> >>>>>>> It will be enough to say that each row has 3 off-diagonal values, so >>>>>>> you can use the simplified preallocation rather >>>>>>> than giving a value for each row individually. >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> Thanks. >>>>>>>> >>>>>>>> ---- >>>>>>>> John Albequerque. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Feb 19 08:13:17 2016 From: jed at jedbrown.org (Jed Brown) Date: Fri, 19 Feb 2016 07:13:17 -0700 Subject: [petsc-users] Doubt in MPIPreallocation In-Reply-To: References: Message-ID: <8760xkokjm.fsf@jedbrown.org> John Albequerque writes: > I am really sorry Matthew but I am not able to get your point. It would be > very kind of you if you could show me the changes in the code. MatSetValues sets a logically dense block. It's not a list of values in COO format. So this is setting a (high-low)*nc dense block (which includes explicit zeros): MatSetValues(A,(high-low),idxm,nc,idxn,values,INSERT_VALUES); You can see those stored zeros here: >>> *Mat Object: 1 MPI processes type: mpiaijrow 0: (0, 1) (1, 1) (2, 1) >>> (3, 1) (4, 0) (5, 0) (6, 0) (7, 0) row 1: (0, 0) (1, 1) (2, 1) (3, >>> 1) (4, 1) (5, 0) (6, 0) (7, 0) row 2: (0, 0) (1, 0) (2, 1) (3, 1) >>> (4, 1) (5, 1) (6, 0) (7, 0) row 3: (0, 0) (1, 0) (2, 0) (3, 1) (4, >>> 1) (5, 1) (6, 1) (7, 0) row 4: (0, 0) (1, 0) (2, 0) (3, 0) (4, 1) >>> (5, 1) (6, 1) (7, 1)* You need to allocate for all entries that you set explicitly. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From johncfdcfd at gmail.com Fri Feb 19 08:25:07 2016 From: johncfdcfd at gmail.com (John Albequerque) Date: Fri, 19 Feb 2016 19:55:07 +0530 Subject: [petsc-users] Doubt in MPIPreallocation In-Reply-To: <8760xkokjm.fsf@jedbrown.org> References: <8760xkokjm.fsf@jedbrown.org> Message-ID: So Jed, what you are suggesting is that I should set only the non-zero elements while using * MatSetValues(A,(high-low),**idxm,nc,idxn,values,INSERT_* *VALUES);* And not mention the zero elements and for that I should loop over all local rows and then set each of the value. Thanks ---- John Albequerque. On Fri, Feb 19, 2016 at 7:43 PM, Jed Brown wrote: > John Albequerque writes: > > > I am really sorry Matthew but I am not able to get your point. It would > be > > very kind of you if you could show me the changes in the code. > > MatSetValues sets a logically dense block. It's not a list of values in > COO format. So this is setting a (high-low)*nc dense block (which > includes explicit zeros): > > MatSetValues(A,(high-low),idxm,nc,idxn,values,INSERT_VALUES); > > You can see those stored zeros here: > > >>> *Mat Object: 1 MPI processes type: mpiaijrow 0: (0, 1) (1, 1) (2, 1) > >>> (3, 1) (4, 0) (5, 0) (6, 0) (7, 0) row 1: (0, 0) (1, 1) (2, 1) > (3, > >>> 1) (4, 1) (5, 0) (6, 0) (7, 0) row 2: (0, 0) (1, 0) (2, 1) (3, > 1) > >>> (4, 1) (5, 1) (6, 0) (7, 0) row 3: (0, 0) (1, 0) (2, 0) (3, 1) > (4, > >>> 1) (5, 1) (6, 1) (7, 0) row 4: (0, 0) (1, 0) (2, 0) (3, 0) (4, > 1) > >>> (5, 1) (6, 1) (7, 1)* > > You need to allocate for all entries that you set explicitly. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Feb 19 08:46:06 2016 From: jed at jedbrown.org (Jed Brown) Date: Fri, 19 Feb 2016 07:46:06 -0700 Subject: [petsc-users] Doubt in MPIPreallocation In-Reply-To: References: <8760xkokjm.fsf@jedbrown.org> Message-ID: <8737sooj0x.fsf@jedbrown.org> John Albequerque writes: > So Jed, what you are suggesting is that I should set only the non-zero > elements while using > * MatSetValues(A,(high-low),**idxm,nc,idxn,values,INSERT_* > > *VALUES);* > And not mention the zero elements and for that I should loop over all local > rows and then set each of the value. Yes. The whole point of a sparse matrix is to spend no time or storage on 0.0 entries. If you allocate a dense matrix and store all the zeros, you're just being inefficient relative to a dense format. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From johncfdcfd at gmail.com Fri Feb 19 08:47:31 2016 From: johncfdcfd at gmail.com (John Albequerque) Date: Fri, 19 Feb 2016 20:17:31 +0530 Subject: [petsc-users] Doubt in MPIPreallocation In-Reply-To: <8737sooj0x.fsf@jedbrown.org> References: <8760xkokjm.fsf@jedbrown.org> <8737sooj0x.fsf@jedbrown.org> Message-ID: Thank you very much, I will try it. Thanks ---- John Albequerque. On Fri, Feb 19, 2016 at 8:16 PM, Jed Brown wrote: > John Albequerque writes: > > > So Jed, what you are suggesting is that I should set only the non-zero > > elements while using > > * MatSetValues(A,(high-low),**idxm,nc,idxn,values,INSERT_* > > > > *VALUES);* > > And not mention the zero elements and for that I should loop over all > local > > rows and then set each of the value. > > Yes. The whole point of a sparse matrix is to spend no time or storage > on 0.0 entries. If you allocate a dense matrix and store all the zeros, > you're just being inefficient relative to a dense format. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eugenio.aulisa at ttu.edu Fri Feb 19 08:55:43 2016 From: eugenio.aulisa at ttu.edu (Aulisa, Eugenio) Date: Fri, 19 Feb 2016 14:55:43 +0000 Subject: [petsc-users] Some clarification on PCMG Message-ID: Hi, I would like a clarification how PCMG updates the residual on the level-l before restricting it to the level l-1 in the down-cycle. In particular I would expect that what is projected down is the true-residual, the one updated using Amat_l, however from what I see in my test it seams to be the residual updated using Pmat_l. My test is very simple: When setting up the ksp object on the l-level, in one case I use KSPSetOperators(_ksp_l, _Amat_l, _Pmat_l); in the other case I use KSPSetOperators(_ksp_l, _Pmat_l, _Pmat_l); when I run my application I get exactly the same results for the preconditioned residual: these are the very last 2 outputs: with Amat,Pmat 9 KSP preconditioned resid norm 2.939296481331e-16 true resid norm 1.460300485277e-02 ||r(i)||/||b|| 9.999999999999e-01 with Pmat, Pmat: 9 KSP preconditioned resid norm 2.939296481331e-16 true resid norm 5.165834461813e-15 ||r(i)||/||b|| 3.537514719672e-13 Since Amat and Pmat are 2 different matrices, and PCMG does not seams to see any difference, the only explanation I gave myself is that the residual projected down to the coarser level is the the one updated using Pmat. Now, if this is the case, is there any way I can use Amat_l to update the residual? If this is not the case, and Amat_l is already used, do you have any other explanation why the results are identical? Thanks, Eugenio Eugenio Aulisa Department of Mathematics and Statistics, Texas Tech University Lubbock TX, 79409-1042 room: 226 http://www.math.ttu.edu/~eaulisa/ phone: (806) 834-6684 fax: (806) 742-1112 From bsmith at mcs.anl.gov Fri Feb 19 10:42:57 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 19 Feb 2016 10:42:57 -0600 Subject: [petsc-users] Some clarification on PCMG In-Reply-To: References: Message-ID: > On Feb 19, 2016, at 8:55 AM, Aulisa, Eugenio wrote: > > Hi, > > I would like a clarification how PCMG updates > the residual on the level-l before restricting it to the level l-1 > in the down-cycle. > > In particular I would expect that what is projected down is the true-residual, > the one updated using Amat_l, however from what I see in my test it seams > to be the residual updated using Pmat_l. > > My test is very simple: > When setting up the ksp object on the l-level, in one case I use > > KSPSetOperators(_ksp_l, _Amat_l, _Pmat_l); > > in the other case I use > > KSPSetOperators(_ksp_l, _Pmat_l, _Pmat_l); > > when I run my application I get exactly the same results for the preconditioned residual: > > these are the very last 2 outputs: > > with Amat,Pmat > 9 KSP preconditioned resid norm 2.939296481331e-16 true resid norm 1.460300485277e-02 ||r(i)||/||b|| 9.999999999999e-01 It uses the Amat to update the residual. Note that this has not worked essentially at all the true residual norm 1.46e-2 is huge while using the Pmat below it is 5e-15. The fact that the "residual norm" is the same is a red herring. I suspect that the Amat, and the Pmat are "too far apart" hence using Amat to define the operator and Pmat to define the smoother produces a multigrid that really doesn't work. As a test try to make the Pmat be just a small perturbation of Amat (or if it is easier the other way around Amat a small perturbation of Pmat) then run and look at the true residual history. Barry > > with Pmat, Pmat: > 9 KSP preconditioned resid norm 2.939296481331e-16 true resid norm 5.165834461813e-15 ||r(i)||/||b|| 3.537514719672e-13 > > Since Amat and Pmat are 2 different matrices, and PCMG does not seams to see any difference, > the only explanation I gave myself is that the residual projected down to the coarser level is the > the one updated using Pmat. > > Now, if this is the case, is there any way I can use Amat_l to update the residual? > > If this is not the case, and Amat_l is already used, > do you have any other explanation why the results are identical? > > Thanks, > Eugenio > > > > > > > > > > Eugenio Aulisa > > Department of Mathematics and Statistics, > Texas Tech University > Lubbock TX, 79409-1042 > room: 226 > http://www.math.ttu.edu/~eaulisa/ > phone: (806) 834-6684 > fax: (806) 742-1112 > > > From jychang48 at gmail.com Fri Feb 19 11:18:45 2016 From: jychang48 at gmail.com (Justin Chang) Date: Fri, 19 Feb 2016 11:18:45 -0600 Subject: [petsc-users] Do more solver iterations = more communication? In-Reply-To: <0BEED677-9C98-465E-8339-CED8165F92A8@mcs.anl.gov> References: <0BEED677-9C98-465E-8339-CED8165F92A8@mcs.anl.gov> Message-ID: Thanks Barry, 1) Attached is the full log_summary of a serial run I did with ILU. I noticed that the MPI reductions/messages happen mostly in the SNESFunction/JacobianEval routines. Am I right to assume that these occur because of the required calls to Vec/MatAssemblyBegin/End at the end? 2) If I ran this program with at least 2 cores, will the other Vec and Mat functions have these MPI reductions/messages accumulated? 3) I don't know what all is happening inside BoomerAMG or ML, but do these packages perform their own Mat and Vec operations? Because if they still in part use PETSc' Vec and Mat operations, we could still somewhat quantify the corresponding MPI metrics no? 4) Suppose I stick with one of these preconditioner packages (e.g., ML) and solve the same problem with two different numerical methods.Is it more appropriate to infer that if both methods require the same amount of wall-clock time but one of them requires more iterations to achieve the solution, then it overall may have more communication and *might* have the tendency to not scale as well in the strong sense? Thanks, Justin On Thu, Feb 18, 2016 at 4:05 PM, Barry Smith wrote: > > > On Feb 18, 2016, at 1:56 PM, Justin Chang wrote: > > > > Hi all, > > > > For a poisson problem with roughly 1 million dofs (using second-order > elements), I solved the problem using two different solver/preconditioner > combinations: CG/ILU and CG/GAMG. > > > > ILU takes roughly 82 solver iterations whereas with GAMG it takes 14 > iterations (wall clock time is roughly 15 and 46 seconds respectively). I > have seen from previous mailing threads that there is a strong correlation > between solver iterations and communication (which could lead to less > strong-scaling scalability). It makes sense to me if I strictly use one of > these preconditioners to solve two different problems and compare the > number of respective iterations, but what about solving the same problem > with two different preconditioners? > > > > If GAMG takes 14 iterations whereas ILU takes 82 iterations, does this > necessarily mean GAMG has less communication? > > No you can't say that at all. A single GAMG cycle will do more > communication than a single block Jacobi cycle. > > > I would think that the "bandwidth" that happens within a single GAMG > iteration would be much greater than that within a single ILU iteration. Is > there a way to officially determine this? > > > > I see from log_summary that we have this information: > > MPI Messages: 5.000e+00 1.00000 5.000e+00 5.000e+00 > > MPI Message Lengths: 5.816e+07 1.00000 1.163e+07 5.816e+07 > > MPI Reductions: 2.000e+01 1.00000 > > > > Can this information be used to determine the "bandwidth"? > > You can certainly use this data for each run to determine which > algorithm is sending more messages, total length of messages is bigger etc. > And if you divided by time it would tell the rate of communication for the > different algorithms. > > Note that counts of messages and lengths are also given in the detailed > table for each operation. > > There are also theoretical bounds on messages that can be derived for some > iterations applied to some problems. > > > If so, does PETSc have the ability to document this for other > preconditioner packages like HYPRE's BoomerAMG or Trilinos' ML? > > No, because they don't log this information. > > > > Thanks, > > Justin > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./main on a arch-linux2-c-opt named pacotaco-xps with 1 processor, by justin Thu Feb 11 10:37:52 2016 Using Petsc Development GIT revision: v3.6.3-2084-g01047ab GIT Date: 2016-01-25 11:32:04 -0600 Max Max/Min Avg Total Time (sec): 1.548e+02 1.00000 1.548e+02 Objects: 2.930e+02 1.00000 2.930e+02 Flops: 7.338e+10 1.00000 7.338e+10 7.338e+10 Flops/sec: 4.740e+08 1.00000 4.740e+08 4.740e+08 MPI Messages: 5.000e+00 1.00000 5.000e+00 5.000e+00 MPI Message Lengths: 5.816e+07 1.00000 1.163e+07 5.816e+07 MPI Reductions: 2.000e+01 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 1.5481e+02 100.0% 7.3382e+10 100.0% 5.000e+00 100.0% 1.163e+07 100.0% 2.000e+01 100.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage CreateMesh 1 1.0 3.2756e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+01 21 0 0 0100 21 0 0 0100 0 BuildTwoSided 5 1.0 6.1989e-05 1.0 0.00e+00 0.0 1.0e+00 4.0e+00 0.0e+00 0 0 20 0 0 0 0 20 0 0 0 VecTDot 984 1.0 1.7839e+00 1.0 4.09e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 6 0 0 0 1 6 0 0 0 2291 VecNorm 493 1.0 8.9080e-01 1.0 2.05e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 3 0 0 0 1 3 0 0 0 2299 VecCopy 3 1.0 8.8620e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 25 1.0 1.7894e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 VecAXPY 984 1.0 2.5989e+00 1.0 4.09e+09 1.0 0.0e+00 0.0e+00 0.0e+00 2 6 0 0 0 2 6 0 0 0 1573 VecAYPX 491 1.0 1.2707e+00 1.0 2.04e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 3 0 0 0 1 3 0 0 0 1605 VecWAXPY 1 1.0 1.6074e-01 1.0 2.08e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 13 MatMult 492 1.0 2.1720e+01 1.0 2.92e+10 1.0 0.0e+00 0.0e+00 0.0e+00 14 40 0 0 0 14 40 0 0 0 1345 MatSolve 493 1.0 2.0103e+01 1.0 2.93e+10 1.0 0.0e+00 0.0e+00 0.0e+00 13 40 0 0 0 13 40 0 0 0 1457 MatLUFactorNum 1 1.0 4.6696e-01 1.0 2.85e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 610 MatILUFactorSym 1 1.0 6.9562e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 2 1.0 7.0739e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 2 1.0 2.2361e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 1 1.0 2.4049e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 6.6127e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatZeroEntries 1 1.0 2.5803e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 DMPlexInterp 3 1.0 5.2191e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 3 0 0 0 0 3 0 0 0 0 0 DMPlexStratify 11 1.0 7.3829e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 5 0 0 0 0 5 0 0 0 0 0 DMPlexPrealloc 1 1.0 1.3914e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 9 0 0 0 0 9 0 0 0 0 0 DMPlexResidualFE 1 1.0 1.0185e+01 1.0 6.41e+08 1.0 0.0e+00 0.0e+00 0.0e+00 7 1 0 0 0 7 1 0 0 0 63 DMPlexJacobianFE 1 1.0 3.5721e+01 1.0 1.30e+09 1.0 0.0e+00 0.0e+00 0.0e+00 23 2 0 0 0 23 2 0 0 0 36 SFSetGraph 6 1.0 2.2913e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SFBcastBegin 8 1.0 6.4482e-01 1.0 0.00e+00 0.0 4.0e+00 1.0e+07 0.0e+00 0 0 80 71 0 0 0 80 71 0 0 SFBcastEnd 8 1.0 8.5786e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SFReduceBegin 1 1.0 6.2540e-03 1.0 0.00e+00 0.0 1.0e+00 1.7e+07 0.0e+00 0 0 20 29 0 0 0 20 29 0 0 SFReduceEnd 1 1.0 4.0581e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SNESFunctionEval 1 1.0 1.2475e+01 1.0 6.41e+08 1.0 2.0e+00 1.7e+07 0.0e+00 8 1 40 57 0 8 1 40 57 0 51 SNESJacobianEval 1 1.0 3.5764e+01 1.0 1.30e+09 1.0 3.0e+00 8.3e+06 0.0e+00 23 2 60 43 0 23 2 60 43 0 36 KSPSetUp 1 1.0 2.5737e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 1 1.0 4.9169e+01 1.0 7.11e+10 1.0 0.0e+00 0.0e+00 0.0e+00 32 97 0 0 0 32 97 0 0 0 1445 PCSetUp 1 1.0 6.7844e-01 1.0 2.85e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 420 PCApply 493 1.0 2.0104e+01 1.0 2.93e+10 1.0 0.0e+00 0.0e+00 0.0e+00 13 40 0 0 0 13 40 0 0 0 1457 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Viewer 4 2 1520 0. Object 7 7 4088 0. Container 15 15 8640 0. Vector 20 20 3266330608 0. Matrix 2 2 804148444 0. Distributed Mesh 29 29 135576 0. GraphPartitioner 11 11 6732 0. Star Forest Bipartite Graph 62 62 50488 0. Discrete System 29 29 25056 0. Index Set 46 46 291423816 0. IS L to G Mapping 1 1 8701952 0. Section 56 54 36288 0. SNES 1 1 1340 0. SNESLineSearch 1 1 1000 0. DMSNES 1 1 672 0. Krylov Solver 1 1 1240 0. Preconditioner 1 1 1016 0. Linear Space 2 2 1296 0. Dual Space 2 2 1328 0. FE Space 2 2 1512 0. ======================================================================================================================== Average time to get PetscTime(): 0. #PETSc Option Table entries: -al 1 -am 0 -at 0.001 -bcloc 0,1,0,1,0,0,0,1,0,1,1,1,0,0,0,1,0,1,1,1,0,1,0,1,0,1,0,0,0,1,0,1,1,1,0,1,0.45,0.55,0.45,0.55,0.45,0.55 -bcnum 7 -bcval 0,0,0,0,0,0,1 -dim 3 -dm_refine 1 -dt 0.001 -edges 3,3 -floc 0.25,0.75,0.25,0.75,0.25,0.75 -fnum 0 -ftime 0,99 -fval 1 -ksp_atol 1.0e-8 -ksp_max_it 50000 -ksp_rtol 1.0e-8 -ksp_type cg -log_summary -lower 0,0 -mat_petscspace_order 0 -mesh datafiles/cube_with_hole5_mesh.dat -mu 1 -nonneg 0 -numsteps 0 -options_left 0 -pc_type ilu -petscpartitioner_type parmetis -progress 1 -simplex 1 -solution_petscspace_order 1 -tao_fatol 1e-8 -tao_frtol 1e-8 -tao_max_it 50000 -tao_type blmvm -trans datafiles/cube_with_hole5_trans.dat -upper 1,1 -vtuname figures/cube_with_hole_5 -vtuprint 0 #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: --download-chaco --download-ctetgen --download-fblaslapack --download-hdf5 --download-hypre --download-metis --download-ml --download-parmetis --download-triangle --with-debugging=0 --with-mpi-dir=/usr/lib/openmpi --with-papi=/usr/local --with-shared-libraries=1 --with-valgrind=1 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 PETSC_ARCH=arch-linux2-c-opt ----------------------------------------- Libraries compiled on Tue Jan 26 00:25:01 2016 on pacotaco-xps Machine characteristics: Linux-3.13.0-76-generic-x86_64-with-Ubuntu-14.04-trusty Using PETSc directory: /home/justin/Software/petsc-dev Using PETSc arch: arch-linux2-c-opt ----------------------------------------- Using C compiler: mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O2 ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: mpif90 -fPIC -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O2 ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/home/justin/Software/petsc-dev/arch-linux2-c-opt/include -I/home/justin/Software/petsc-dev/include -I/home/justin/Software/petsc-dev/include -I/home/justin/Software/petsc-dev/arch-linux2-c-opt/include -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi ----------------------------------------- Using C linker: mpicc Using Fortran linker: mpif90 Using libraries: -Wl,-rpath,/home/justin/Software/petsc-dev/arch-linux2-c-opt/lib -L/home/justin/Software/petsc-dev/arch-linux2-c-opt/lib -lpetsc -Wl,-rpath,/home/justin/Software/petsc-dev/arch-linux2-c-opt/lib -L/home/justin/Software/petsc-dev/arch-linux2-c-opt/lib -lml -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lmpi_cxx -lstdc++ -lHYPRE -lmpi_cxx -lstdc++ -lflapack -lfblas -lparmetis -lmetis -ltriangle -lX11 -lssl -lcrypto -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lctetgen -lchaco -lm -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -lmpi -lhwloc -lgcc_s -lpthread -ldl ----------------------------------------- From eugenio.aulisa at ttu.edu Fri Feb 19 11:38:13 2016 From: eugenio.aulisa at ttu.edu (Aulisa, Eugenio) Date: Fri, 19 Feb 2016 17:38:13 +0000 Subject: [petsc-users] Some clarification on PCMG In-Reply-To: References: , Message-ID: Thanks Berry, In Amat we have the weak form of a Jacobian with all Neumann Boundary conditions, while in Pmat we have the same Jacobian with the correct Dirichlet nodes imposed by penalty, So Pmat is the real problem and the fact that the preconditioned residual converges tell us that we are solving the problem correctly. We know that for many other reasons. The residual update with a Neumann Amat on the Dirichlet nodes is used only to generate the correct residual restriction on the coarse nodes whose support overlaps with the support of the fine boundary nodes. The fact that the true residual does not converge is then correct, because it involves the residual of those fake Neumann boundary nodes , that are never solved. Now, what puzzles us is that using Pmat or Amat does not change the convergence of the Multigrid not even of a single digit. We would expect a better convergence using Amat, but it is not the case. Thanks again, Eugenio Eugenio Aulisa Department of Mathematics and Statistics, Texas Tech University Lubbock TX, 79409-1042 room: 226 http://www.math.ttu.edu/~eaulisa/ phone: (806) 834-6684 fax: (806) 742-1112 ________________________________________ From: Barry Smith [bsmith at mcs.anl.gov] Sent: Friday, February 19, 2016 10:42 AM To: Aulisa, Eugenio Cc: PETSc; simone.bna at gmail.com; Bornia, Giorgio Subject: Re: [petsc-users] Some clarification on PCMG > On Feb 19, 2016, at 8:55 AM, Aulisa, Eugenio wrote: > > Hi, > > I would like a clarification how PCMG updates > the residual on the level-l before restricting it to the level l-1 > in the down-cycle. > > In particular I would expect that what is projected down is the true-residual, > the one updated using Amat_l, however from what I see in my test it seams > to be the residual updated using Pmat_l. > > My test is very simple: > When setting up the ksp object on the l-level, in one case I use > > KSPSetOperators(_ksp_l, _Amat_l, _Pmat_l); > > in the other case I use > > KSPSetOperators(_ksp_l, _Pmat_l, _Pmat_l); > > when I run my application I get exactly the same results for the preconditioned residual: > > these are the very last 2 outputs: > > with Amat,Pmat > 9 KSP preconditioned resid norm 2.939296481331e-16 true resid norm 1.460300485277e-02 ||r(i)||/||b|| 9.999999999999e-01 It uses the Amat to update the residual. Note that this has not worked essentially at all the true residual norm 1.46e-2 is huge while using the Pmat below it is 5e-15. The fact that the "residual norm" is the same is a red herring. I suspect that the Amat, and the Pmat are "too far apart" hence using Amat to define the operator and Pmat to define the smoother produces a multigrid that really doesn't work. As a test try to make the Pmat be just a small perturbation of Amat (or if it is easier the other way around Amat a small perturbation of Pmat) then run and look at the true residual history. Barry > > with Pmat, Pmat: > 9 KSP preconditioned resid norm 2.939296481331e-16 true resid norm 5.165834461813e-15 ||r(i)||/||b|| 3.537514719672e-13 > > Since Amat and Pmat are 2 different matrices, and PCMG does not seams to see any difference, > the only explanation I gave myself is that the residual projected down to the coarser level is the > the one updated using Pmat. > > Now, if this is the case, is there any way I can use Amat_l to update the residual? > > If this is not the case, and Amat_l is already used, > do you have any other explanation why the results are identical? > > Thanks, > Eugenio > > > > > > > > > > Eugenio Aulisa > > Department of Mathematics and Statistics, > Texas Tech University > Lubbock TX, 79409-1042 > room: 226 > http://www.math.ttu.edu/~eaulisa/ > phone: (806) 834-6684 > fax: (806) 742-1112 > > > From bsmith at mcs.anl.gov Fri Feb 19 11:42:19 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 19 Feb 2016 11:42:19 -0600 Subject: [petsc-users] Do more solver iterations = more communication? In-Reply-To: References: <0BEED677-9C98-465E-8339-CED8165F92A8@mcs.anl.gov> Message-ID: <03A9F764-2CEC-4730-85C3-9C5028EB0C2E@mcs.anl.gov> > On Feb 19, 2016, at 11:18 AM, Justin Chang wrote: > > Thanks Barry, > > 1) Attached is the full log_summary of a serial run I did with ILU. I noticed that the MPI reductions/messages happen mostly in the SNESFunction/JacobianEval routines. Am I right to assume that these occur because of the required calls to Vec/MatAssemblyBegin/End at the end? Likely it is two places 1) communicating the ghost points needed to compute the function and Jacobian (usually this will have no reductions) and then yes 2) in the AssemblyBegin/End > > 2) If I ran this program with at least 2 cores, will the other Vec and Mat functions have these MPI reductions/messages accumulated? Not sure what you mean by this. The logging does count MPI reductions and messages in any PETSc operations. > > 3) I don't know what all is happening inside BoomerAMG or ML, but do these packages perform their own Mat and Vec operations? Because if they still in part use PETSc' Vec and Mat operations, we could still somewhat quantify the corresponding MPI metrics no? BoomerAMG does everything internally so we have no information. For ML, it sets up the algebraic multigrid preconditioner completely internally so we have no information on that but the actually multigrid iterations are done with PETSc matrices and vectors so we do have that information. > > 4) Suppose I stick with one of these preconditioner packages (e.g., ML) and solve the same problem with two different numerical methods.Is it more appropriate to infer that if both methods require the same amount of wall-clock time but one of them requires more iterations to achieve the solution, then it overall may have more communication and *might* have the tendency to not scale as well in the strong sense? The is determined by seeing how the iterations scale with the number of processes, not with the absolute number of iterations. So you need to run at at least three grid resolutions and get the number of iterations and see how they grow. For example with bJacobi + ILU the number of iterations will grow but with GAMG they will be the same or grow very slowly. For a small number of processes bjacobi will be better than GAMG but for a large number of processes bJacobi will be far worse. > > Thanks, > Justin > > > On Thu, Feb 18, 2016 at 4:05 PM, Barry Smith wrote: > > > On Feb 18, 2016, at 1:56 PM, Justin Chang wrote: > > > > Hi all, > > > > For a poisson problem with roughly 1 million dofs (using second-order elements), I solved the problem using two different solver/preconditioner combinations: CG/ILU and CG/GAMG. > > > > ILU takes roughly 82 solver iterations whereas with GAMG it takes 14 iterations (wall clock time is roughly 15 and 46 seconds respectively). I have seen from previous mailing threads that there is a strong correlation between solver iterations and communication (which could lead to less strong-scaling scalability). It makes sense to me if I strictly use one of these preconditioners to solve two different problems and compare the number of respective iterations, but what about solving the same problem with two different preconditioners? > > > > If GAMG takes 14 iterations whereas ILU takes 82 iterations, does this necessarily mean GAMG has less communication? > > No you can't say that at all. A single GAMG cycle will do more communication than a single block Jacobi cycle. > > > I would think that the "bandwidth" that happens within a single GAMG iteration would be much greater than that within a single ILU iteration. Is there a way to officially determine this? > > > > I see from log_summary that we have this information: > > MPI Messages: 5.000e+00 1.00000 5.000e+00 5.000e+00 > > MPI Message Lengths: 5.816e+07 1.00000 1.163e+07 5.816e+07 > > MPI Reductions: 2.000e+01 1.00000 > > > > Can this information be used to determine the "bandwidth"? > > You can certainly use this data for each run to determine which algorithm is sending more messages, total length of messages is bigger etc. And if you divided by time it would tell the rate of communication for the different algorithms. > > Note that counts of messages and lengths are also given in the detailed table for each operation. > > There are also theoretical bounds on messages that can be derived for some iterations applied to some problems. > > > If so, does PETSc have the ability to document this for other preconditioner packages like HYPRE's BoomerAMG or Trilinos' ML? > > No, because they don't log this information. > > > > Thanks, > > Justin > > > From johncfdcfd at gmail.com Fri Feb 19 12:54:21 2016 From: johncfdcfd at gmail.com (John Albequerque) Date: Sat, 20 Feb 2016 00:24:21 +0530 Subject: [petsc-users] Doubt in MPIPreallocation In-Reply-To: References: <8760xkokjm.fsf@jedbrown.org> <8737sooj0x.fsf@jedbrown.org> Message-ID: Jed, one more *nnz cannot be greater than row length: local row 0 value 1 rowlength 0 * How do I deal with this error? Thanks. ---- John Albequerque. On Fri, Feb 19, 2016 at 8:17 PM, John Albequerque wrote: > Thank you very much, I will try it. > > > Thanks > > ---- > John Albequerque. > > > > On Fri, Feb 19, 2016 at 8:16 PM, Jed Brown wrote: > >> John Albequerque writes: >> >> > So Jed, what you are suggesting is that I should set only the non-zero >> > elements while using >> > * MatSetValues(A,(high-low),**idxm,nc,idxn,values,INSERT_* >> > >> > *VALUES);* >> > And not mention the zero elements and for that I should loop over all >> local >> > rows and then set each of the value. >> >> Yes. The whole point of a sparse matrix is to spend no time or storage >> on 0.0 entries. If you allocate a dense matrix and store all the zeros, >> you're just being inefficient relative to a dense format. >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Feb 19 12:59:19 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 19 Feb 2016 12:59:19 -0600 Subject: [petsc-users] Some clarification on PCMG In-Reply-To: References: Message-ID: <868455DC-795D-4BAE-A186-08F469ABDD25@mcs.anl.gov> > On Feb 19, 2016, at 11:38 AM, Aulisa, Eugenio wrote: > > Thanks Berry, > > In Amat we have the weak form of a Jacobian with all Neumann Boundary conditions, > while in Pmat we have the same Jacobian with the correct Dirichlet nodes imposed by penalty, Using Dirichlet problems to precondition a Neumann problem is a no-no. The spectrum are different and so it is not a spectrally equivalent preconditioner nor is there any good reason to do it IMHO. > So Pmat is the real problem and the fact that the preconditioned residual converges > tell us that we are solving the problem correctly. No it doesn't tell you that. You have not solved the problem correctly. > We know that for many other reasons. Some people do this but no self-respecting numerical analyst would accept the answers. > > The residual update with a Neumann Amat on the Dirichlet nodes > is used only to generate the correct residual restriction on the coarse nodes > whose support overlaps with the support of the fine boundary nodes. > > The fact that the true residual does not converge is then correct, > because it involves the residual of those fake Neumann boundary nodes , > that are never solved. I don't think there is any reason for you to build the multigrid preconditioner in this way. Geometric multigrid can work fine for Neumann problems with regular smoothers and no "special" Pmat. You just have to make sure you provide the null space information to the matrix on each level which can be done with MatSetNullSpace and use a coarse grid solver that can handle the null space. Usually what we do is have a very small coarse problem and then just use PCSVD on that problem. Feel free to ask for help in using this approach. Barry > > Now, what puzzles us is that using Pmat or Amat does > not change the convergence of the Multigrid not even of a single digit. > We would expect a better convergence using Amat, but it is not the case. Do you get the exact same convergence history? If so I agree it is odd. > > Thanks again, > Eugenio > > Eugenio Aulisa > > Department of Mathematics and Statistics, > Texas Tech University > Lubbock TX, 79409-1042 > room: 226 > http://www.math.ttu.edu/~eaulisa/ > phone: (806) 834-6684 > fax: (806) 742-1112 > > > > > ________________________________________ > From: Barry Smith [bsmith at mcs.anl.gov] > Sent: Friday, February 19, 2016 10:42 AM > To: Aulisa, Eugenio > Cc: PETSc; simone.bna at gmail.com; Bornia, Giorgio > Subject: Re: [petsc-users] Some clarification on PCMG > >> On Feb 19, 2016, at 8:55 AM, Aulisa, Eugenio wrote: >> >> Hi, >> >> I would like a clarification how PCMG updates >> the residual on the level-l before restricting it to the level l-1 >> in the down-cycle. >> >> In particular I would expect that what is projected down is the true-residual, >> the one updated using Amat_l, however from what I see in my test it seams >> to be the residual updated using Pmat_l. >> >> My test is very simple: >> When setting up the ksp object on the l-level, in one case I use >> >> KSPSetOperators(_ksp_l, _Amat_l, _Pmat_l); >> >> in the other case I use >> >> KSPSetOperators(_ksp_l, _Pmat_l, _Pmat_l); >> >> when I run my application I get exactly the same results for the preconditioned residual: >> >> these are the very last 2 outputs: >> >> with Amat,Pmat >> 9 KSP preconditioned resid norm 2.939296481331e-16 true resid norm 1.460300485277e-02 ||r(i)||/||b|| 9.999999999999e-01 > > It uses the Amat to update the residual. Note that this has not worked essentially at all the true residual norm 1.46e-2 is huge while using the Pmat below it is 5e-15. The fact that the "residual norm" is the same is a red herring. > > I suspect that the Amat, and the Pmat are "too far apart" hence using Amat to define the operator and Pmat to define the smoother produces a multigrid that really doesn't work. > > As a test try to make the Pmat be just a small perturbation of Amat (or if it is easier the other way around Amat a small perturbation of Pmat) then run and look at the true residual history. > > Barry > > >> >> with Pmat, Pmat: >> 9 KSP preconditioned resid norm 2.939296481331e-16 true resid norm 5.165834461813e-15 ||r(i)||/||b|| 3.537514719672e-13 >> >> Since Amat and Pmat are 2 different matrices, and PCMG does not seams to see any difference, >> the only explanation I gave myself is that the residual projected down to the coarser level is the >> the one updated using Pmat. >> >> Now, if this is the case, is there any way I can use Amat_l to update the residual? >> >> If this is not the case, and Amat_l is already used, >> do you have any other explanation why the results are identical? >> >> Thanks, >> Eugenio >> >> >> >> >> >> >> >> >> >> Eugenio Aulisa >> >> Department of Mathematics and Statistics, >> Texas Tech University >> Lubbock TX, 79409-1042 >> room: 226 >> http://www.math.ttu.edu/~eaulisa/ >> phone: (806) 834-6684 >> fax: (806) 742-1112 >> >> >> > From bsmith at mcs.anl.gov Fri Feb 19 13:05:02 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 19 Feb 2016 13:05:02 -0600 Subject: [petsc-users] Doubt in MPIPreallocation In-Reply-To: References: <8760xkokjm.fsf@jedbrown.org> <8737sooj0x.fsf@jedbrown.org> Message-ID: > On Feb 19, 2016, at 12:54 PM, John Albequerque wrote: > > Jed, one more > > nnz cannot be greater than row length: local row 0 value 1 rowlength 0 Always send the entire error message, this provides the context to know what the hey is going on. It looks like you set a matrix block (if you are using an MPI matrix on one process this could the the "off-diagonal" block) which has no columns (hence row length is zero) but you claim you need to preallocate an entry (the 1). Barry > > How do I deal with this error? > > Thanks. > > ---- > John Albequerque. > > > > On Fri, Feb 19, 2016 at 8:17 PM, John Albequerque wrote: > Thank you very much, I will try it. > > > Thanks > > ---- > John Albequerque. > > > > On Fri, Feb 19, 2016 at 8:16 PM, Jed Brown wrote: > John Albequerque writes: > > > So Jed, what you are suggesting is that I should set only the non-zero > > elements while using > > * MatSetValues(A,(high-low),**idxm,nc,idxn,values,INSERT_* > > > > *VALUES);* > > And not mention the zero elements and for that I should loop over all local > > rows and then set each of the value. > > Yes. The whole point of a sparse matrix is to spend no time or storage > on 0.0 entries. If you allocate a dense matrix and store all the zeros, > you're just being inefficient relative to a dense format. > > From johncfdcfd at gmail.com Fri Feb 19 14:29:05 2016 From: johncfdcfd at gmail.com (John Albequerque) Date: Sat, 20 Feb 2016 01:59:05 +0530 Subject: [petsc-users] Doubt in MPIPreallocation In-Reply-To: References: <8760xkokjm.fsf@jedbrown.org> <8737sooj0x.fsf@jedbrown.org> Message-ID: Dear Barry, I am sorry I could not get you. I have also posted the entire error message so that you could get a deeper insight into it. Many thanks. ---- John Albequerque. *========================================================================================Argument out of range[0]PETSC ERROR: nnz cannot be greater than row length: local row 0 value 1 rowlength 0[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.[0]PETSC ERROR: Petsc Release Version 3.6.3, unknown [0]PETSC ERROR: ./MatSparse on a linux-gnu-c-debug named John by johncfd Sat Feb 20 01:36:01 2016[0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich[0]PETSC ERROR: #1 MatSeqAIJSetPreallocation_SeqAIJ() line 3567 in /home//MyStuff/ClonedRepos/petsc/src/mat/impls/aij/seq/aij.c[0]PETSC ERROR: #2 MatSeqAIJSetPreallocation() line 3539 in /home/MyStuff/ClonedRepos/petsc/src/mat/impls/aij/seq/aij.c[0]PETSC ERROR: #3 MatMPIAIJSetPreallocation_MPIAIJ() line 2835 in /home/MyStuff/ClonedRepos/petsc/src/mat/impls/aij/mpi/mpiaij.c[0]PETSC ERROR: #4 MatMPIAIJSetPreallocation() line 3532 in /home/MyStuff/ClonedRepos/petsc/src/mat/impls/aij/mpi/mpiaij.c* ========================================================================================= On Sat, Feb 20, 2016 at 12:35 AM, Barry Smith wrote: > > > On Feb 19, 2016, at 12:54 PM, John Albequerque > wrote: > > > > Jed, one more > > > > nnz cannot be greater than row length: local row 0 value 1 rowlength 0 > > Always send the entire error message, this provides the context to know > what the hey is going on. > > It looks like you set a matrix block (if you are using an MPI matrix on > one process this could the the "off-diagonal" block) which has no columns > (hence row length is zero) but you claim you need to preallocate an entry > (the 1). > > Barry > > > > > How do I deal with this error? > > > > Thanks. > > > > ---- > > John Albequerque. > > > > > > > > On Fri, Feb 19, 2016 at 8:17 PM, John Albequerque > wrote: > > Thank you very much, I will try it. > > > > > > Thanks > > > > ---- > > John Albequerque. > > > > > > > > On Fri, Feb 19, 2016 at 8:16 PM, Jed Brown wrote: > > John Albequerque writes: > > > > > So Jed, what you are suggesting is that I should set only the non-zero > > > elements while using > > > * MatSetValues(A,(high-low),**idxm,nc,idxn,values,INSERT_* > > > > > > *VALUES);* > > > And not mention the zero elements and for that I should loop over all > local > > > rows and then set each of the value. > > > > Yes. The whole point of a sparse matrix is to spend no time or storage > > on 0.0 entries. If you allocate a dense matrix and store all the zeros, > > you're just being inefficient relative to a dense format. > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Feb 19 15:08:58 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 19 Feb 2016 15:08:58 -0600 Subject: [petsc-users] Doubt in MPIPreallocation In-Reply-To: References: <8760xkokjm.fsf@jedbrown.org> <8737sooj0x.fsf@jedbrown.org> Message-ID: <89FEE4EE-EE07-4109-94C8-541ACA02F87F@mcs.anl.gov> You are setting nonzero preallocations for the "off diagonal" portion of the matrix, but one one process there is no off diagonal portion so you should be setting those values to all zero. Barry > On Feb 19, 2016, at 2:29 PM, John Albequerque wrote: > > > Dear Barry, > I am sorry I could not get you. I have also posted the entire error message so that you could get a deeper insight into it. > > > Many thanks. > > ---- > John Albequerque. > ======================================================================================== > Argument out of range > [0]PETSC ERROR: nnz cannot be greater than row length: local row 0 value 1 rowlength 0 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.3, unknown > [0]PETSC ERROR: ./MatSparse on a linux-gnu-c-debug named John by johncfd Sat Feb 20 01:36:01 2016 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich > [0]PETSC ERROR: #1 MatSeqAIJSetPreallocation_SeqAIJ() line 3567 in /home//MyStuff/ClonedRepos/petsc/src/mat/impls/aij/seq/aij.c > [0]PETSC ERROR: #2 MatSeqAIJSetPreallocation() line 3539 in /home/MyStuff/ClonedRepos/petsc/src/mat/impls/aij/seq/aij.c > [0]PETSC ERROR: #3 MatMPIAIJSetPreallocation_MPIAIJ() line 2835 in /home/MyStuff/ClonedRepos/petsc/src/mat/impls/aij/mpi/mpiaij.c > [0]PETSC ERROR: #4 MatMPIAIJSetPreallocation() line 3532 in /home/MyStuff/ClonedRepos/petsc/src/mat/impls/aij/mpi/mpiaij.c > ========================================================================================= > > > On Sat, Feb 20, 2016 at 12:35 AM, Barry Smith wrote: > > > On Feb 19, 2016, at 12:54 PM, John Albequerque wrote: > > > > Jed, one more > > > > nnz cannot be greater than row length: local row 0 value 1 rowlength 0 > > Always send the entire error message, this provides the context to know what the hey is going on. > > It looks like you set a matrix block (if you are using an MPI matrix on one process this could the the "off-diagonal" block) which has no columns (hence row length is zero) but you claim you need to preallocate an entry (the 1). > > Barry > > > > > How do I deal with this error? > > > > Thanks. > > > > ---- > > John Albequerque. > > > > > > > > On Fri, Feb 19, 2016 at 8:17 PM, John Albequerque wrote: > > Thank you very much, I will try it. > > > > > > Thanks > > > > ---- > > John Albequerque. > > > > > > > > On Fri, Feb 19, 2016 at 8:16 PM, Jed Brown wrote: > > John Albequerque writes: > > > > > So Jed, what you are suggesting is that I should set only the non-zero > > > elements while using > > > * MatSetValues(A,(high-low),**idxm,nc,idxn,values,INSERT_* > > > > > > *VALUES);* > > > And not mention the zero elements and for that I should loop over all local > > > rows and then set each of the value. > > > > Yes. The whole point of a sparse matrix is to spend no time or storage > > on 0.0 entries. If you allocate a dense matrix and store all the zeros, > > you're just being inefficient relative to a dense format. > > > > > > From johncfdcfd at gmail.com Fri Feb 19 15:16:58 2016 From: johncfdcfd at gmail.com (John Albequerque) Date: Sat, 20 Feb 2016 02:46:58 +0530 Subject: [petsc-users] Doubt in MPIPreallocation In-Reply-To: <89FEE4EE-EE07-4109-94C8-541ACA02F87F@mcs.anl.gov> References: <8760xkokjm.fsf@jedbrown.org> <8737sooj0x.fsf@jedbrown.org> <89FEE4EE-EE07-4109-94C8-541ACA02F87F@mcs.anl.gov> Message-ID: Barry, I forgot to mention that the code is working fine with a single processor, but when I increase the number of processors, as in this case 3 processors have been used when the* error message was generated*. And yes, it has something to do with the *off-diagonal* terms as I am increasing the number of processes, the same error triggers for each row for which o_nnz[i] is not zero. Many thanks. ---- John Albequerque. On Sat, Feb 20, 2016 at 2:38 AM, Barry Smith wrote: > > You are setting nonzero preallocations for the "off diagonal" portion of > the matrix, but one one process there is no off diagonal portion so you > should be setting those values to all zero. > > Barry > > > On Feb 19, 2016, at 2:29 PM, John Albequerque > wrote: > > > > > > Dear Barry, > > I am sorry I could not get you. I have also posted the entire error > message so that you could get a deeper insight into it. > > > > > > Many thanks. > > > > ---- > > John Albequerque. > > > ======================================================================================== > > Argument out of range > > [0]PETSC ERROR: nnz cannot be greater than row length: local row 0 value > 1 rowlength 0 > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.6.3, unknown > > [0]PETSC ERROR: ./MatSparse on a linux-gnu-c-debug named John by johncfd > Sat Feb 20 01:36:01 2016 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-fblaslapack --download-mpich > > [0]PETSC ERROR: #1 MatSeqAIJSetPreallocation_SeqAIJ() line 3567 in > /home//MyStuff/ClonedRepos/petsc/src/mat/impls/aij/seq/aij.c > > [0]PETSC ERROR: #2 MatSeqAIJSetPreallocation() line 3539 in > /home/MyStuff/ClonedRepos/petsc/src/mat/impls/aij/seq/aij.c > > [0]PETSC ERROR: #3 MatMPIAIJSetPreallocation_MPIAIJ() line 2835 in > /home/MyStuff/ClonedRepos/petsc/src/mat/impls/aij/mpi/mpiaij.c > > [0]PETSC ERROR: #4 MatMPIAIJSetPreallocation() line 3532 in > /home/MyStuff/ClonedRepos/petsc/src/mat/impls/aij/mpi/mpiaij.c > > > ========================================================================================= > > > > > > On Sat, Feb 20, 2016 at 12:35 AM, Barry Smith > wrote: > > > > > On Feb 19, 2016, at 12:54 PM, John Albequerque > wrote: > > > > > > Jed, one more > > > > > > nnz cannot be greater than row length: local row 0 value 1 rowlength 0 > > > > Always send the entire error message, this provides the context to > know what the hey is going on. > > > > It looks like you set a matrix block (if you are using an MPI matrix > on one process this could the the "off-diagonal" block) which has no > columns (hence row length is zero) but you claim you need to preallocate an > entry (the 1). > > > > Barry > > > > > > > > How do I deal with this error? > > > > > > Thanks. > > > > > > ---- > > > John Albequerque. > > > > > > > > > > > > On Fri, Feb 19, 2016 at 8:17 PM, John Albequerque < > johncfdcfd at gmail.com> wrote: > > > Thank you very much, I will try it. > > > > > > > > > Thanks > > > > > > ---- > > > John Albequerque. > > > > > > > > > > > > On Fri, Feb 19, 2016 at 8:16 PM, Jed Brown wrote: > > > John Albequerque writes: > > > > > > > So Jed, what you are suggesting is that I should set only the > non-zero > > > > elements while using > > > > * MatSetValues(A,(high-low),**idxm,nc,idxn,values,INSERT_* > > > > > > > > *VALUES);* > > > > And not mention the zero elements and for that I should loop over > all local > > > > rows and then set each of the value. > > > > > > Yes. The whole point of a sparse matrix is to spend no time or storage > > > on 0.0 entries. If you allocate a dense matrix and store all the > zeros, > > > you're just being inefficient relative to a dense format. > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Feb 19 15:23:07 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 19 Feb 2016 15:23:07 -0600 Subject: [petsc-users] Doubt in MPIPreallocation In-Reply-To: References: <8760xkokjm.fsf@jedbrown.org> <8737sooj0x.fsf@jedbrown.org> <89FEE4EE-EE07-4109-94C8-541ACA02F87F@mcs.anl.gov> Message-ID: Are you sure you have divided up the matrix so each process gets some of the rows? You are just going to have to break down and run with -start_in_debugger to see exactly why it is crashing. Barry > On Feb 19, 2016, at 3:16 PM, John Albequerque wrote: > > Barry, > I forgot to mention that the code is working fine with a single processor, but when I increase the number of processors, as in this case 3 processors have been used when the error message was generated. And yes, it has something to do with the off-diagonal terms as I am increasing the number of processes, the same error triggers for each row for which o_nnz[i] is not zero. > > > Many thanks. > > ---- > John Albequerque. > > On Sat, Feb 20, 2016 at 2:38 AM, Barry Smith wrote: > > You are setting nonzero preallocations for the "off diagonal" portion of the matrix, but one one process there is no off diagonal portion so you should be setting those values to all zero. > > Barry > > > On Feb 19, 2016, at 2:29 PM, John Albequerque wrote: > > > > > > Dear Barry, > > I am sorry I could not get you. I have also posted the entire error message so that you could get a deeper insight into it. > > > > > > Many thanks. > > > > ---- > > John Albequerque. > > ======================================================================================== > > Argument out of range > > [0]PETSC ERROR: nnz cannot be greater than row length: local row 0 value 1 rowlength 0 > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.6.3, unknown > > [0]PETSC ERROR: ./MatSparse on a linux-gnu-c-debug named John by johncfd Sat Feb 20 01:36:01 2016 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich > > [0]PETSC ERROR: #1 MatSeqAIJSetPreallocation_SeqAIJ() line 3567 in /home//MyStuff/ClonedRepos/petsc/src/mat/impls/aij/seq/aij.c > > [0]PETSC ERROR: #2 MatSeqAIJSetPreallocation() line 3539 in /home/MyStuff/ClonedRepos/petsc/src/mat/impls/aij/seq/aij.c > > [0]PETSC ERROR: #3 MatMPIAIJSetPreallocation_MPIAIJ() line 2835 in /home/MyStuff/ClonedRepos/petsc/src/mat/impls/aij/mpi/mpiaij.c > > [0]PETSC ERROR: #4 MatMPIAIJSetPreallocation() line 3532 in /home/MyStuff/ClonedRepos/petsc/src/mat/impls/aij/mpi/mpiaij.c > > ========================================================================================= > > > > > > On Sat, Feb 20, 2016 at 12:35 AM, Barry Smith wrote: > > > > > On Feb 19, 2016, at 12:54 PM, John Albequerque wrote: > > > > > > Jed, one more > > > > > > nnz cannot be greater than row length: local row 0 value 1 rowlength 0 > > > > Always send the entire error message, this provides the context to know what the hey is going on. > > > > It looks like you set a matrix block (if you are using an MPI matrix on one process this could the the "off-diagonal" block) which has no columns (hence row length is zero) but you claim you need to preallocate an entry (the 1). > > > > Barry > > > > > > > > How do I deal with this error? > > > > > > Thanks. > > > > > > ---- > > > John Albequerque. > > > > > > > > > > > > On Fri, Feb 19, 2016 at 8:17 PM, John Albequerque wrote: > > > Thank you very much, I will try it. > > > > > > > > > Thanks > > > > > > ---- > > > John Albequerque. > > > > > > > > > > > > On Fri, Feb 19, 2016 at 8:16 PM, Jed Brown wrote: > > > John Albequerque writes: > > > > > > > So Jed, what you are suggesting is that I should set only the non-zero > > > > elements while using > > > > * MatSetValues(A,(high-low),**idxm,nc,idxn,values,INSERT_* > > > > > > > > *VALUES);* > > > > And not mention the zero elements and for that I should loop over all local > > > > rows and then set each of the value. > > > > > > Yes. The whole point of a sparse matrix is to spend no time or storage > > > on 0.0 entries. If you allocate a dense matrix and store all the zeros, > > > you're just being inefficient relative to a dense format. > > > > > > > > > > > > From johncfdcfd at gmail.com Fri Feb 19 15:31:42 2016 From: johncfdcfd at gmail.com (John Albequerque) Date: Sat, 20 Feb 2016 03:01:42 +0530 Subject: [petsc-users] Doubt in MPIPreallocation In-Reply-To: References: <8760xkokjm.fsf@jedbrown.org> <8737sooj0x.fsf@jedbrown.org> <89FEE4EE-EE07-4109-94C8-541ACA02F87F@mcs.anl.gov> Message-ID: Barry, I have let PETSc decide the local sizes I have only specified the global values, I will also check with the debugger and tell you the status. And just another query,is it a problem if I have already used *MatSetUp(A); *once and after that assembled the matrix and after that applying Preallocation. I guess that should not be a problem as the code ran fine with the one processor case. Just confirming though. Many thanks. ---- John Albequerque. On Sat, Feb 20, 2016 at 2:53 AM, Barry Smith wrote: > > Are you sure you have divided up the matrix so each process gets some of > the rows? > > You are just going to have to break down and run with -start_in_debugger > to see exactly why it is crashing. > > Barry > > > On Feb 19, 2016, at 3:16 PM, John Albequerque > wrote: > > > > Barry, > > I forgot to mention that the code is working fine with a single > processor, but when I increase the number of processors, as in this case 3 > processors have been used when the error message was generated. And yes, it > has something to do with the off-diagonal terms as I am increasing the > number of processes, the same error triggers for each row for which > o_nnz[i] is not zero. > > > > > > Many thanks. > > > > ---- > > John Albequerque. > > > > On Sat, Feb 20, 2016 at 2:38 AM, Barry Smith wrote: > > > > You are setting nonzero preallocations for the "off diagonal" portion > of the matrix, but one one process there is no off diagonal portion so you > should be setting those values to all zero. > > > > Barry > > > > > On Feb 19, 2016, at 2:29 PM, John Albequerque > wrote: > > > > > > > > > Dear Barry, > > > I am sorry I could not get you. I have also posted the entire error > message so that you could get a deeper insight into it. > > > > > > > > > Many thanks. > > > > > > ---- > > > John Albequerque. > > > > ======================================================================================== > > > Argument out of range > > > [0]PETSC ERROR: nnz cannot be greater than row length: local row 0 > value 1 rowlength 0 > > > [0]PETSC ERROR: See > http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > > [0]PETSC ERROR: Petsc Release Version 3.6.3, unknown > > > [0]PETSC ERROR: ./MatSparse on a linux-gnu-c-debug named John by > johncfd Sat Feb 20 01:36:01 2016 > > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-fblaslapack --download-mpich > > > [0]PETSC ERROR: #1 MatSeqAIJSetPreallocation_SeqAIJ() line 3567 in > /home//MyStuff/ClonedRepos/petsc/src/mat/impls/aij/seq/aij.c > > > [0]PETSC ERROR: #2 MatSeqAIJSetPreallocation() line 3539 in > /home/MyStuff/ClonedRepos/petsc/src/mat/impls/aij/seq/aij.c > > > [0]PETSC ERROR: #3 MatMPIAIJSetPreallocation_MPIAIJ() line 2835 in > /home/MyStuff/ClonedRepos/petsc/src/mat/impls/aij/mpi/mpiaij.c > > > [0]PETSC ERROR: #4 MatMPIAIJSetPreallocation() line 3532 in > /home/MyStuff/ClonedRepos/petsc/src/mat/impls/aij/mpi/mpiaij.c > > > > ========================================================================================= > > > > > > > > > On Sat, Feb 20, 2016 at 12:35 AM, Barry Smith > wrote: > > > > > > > On Feb 19, 2016, at 12:54 PM, John Albequerque > wrote: > > > > > > > > Jed, one more > > > > > > > > nnz cannot be greater than row length: local row 0 value 1 rowlength > 0 > > > > > > Always send the entire error message, this provides the context to > know what the hey is going on. > > > > > > It looks like you set a matrix block (if you are using an MPI matrix > on one process this could the the "off-diagonal" block) which has no > columns (hence row length is zero) but you claim you need to preallocate an > entry (the 1). > > > > > > Barry > > > > > > > > > > > How do I deal with this error? > > > > > > > > Thanks. > > > > > > > > ---- > > > > John Albequerque. > > > > > > > > > > > > > > > > On Fri, Feb 19, 2016 at 8:17 PM, John Albequerque < > johncfdcfd at gmail.com> wrote: > > > > Thank you very much, I will try it. > > > > > > > > > > > > Thanks > > > > > > > > ---- > > > > John Albequerque. > > > > > > > > > > > > > > > > On Fri, Feb 19, 2016 at 8:16 PM, Jed Brown wrote: > > > > John Albequerque writes: > > > > > > > > > So Jed, what you are suggesting is that I should set only the > non-zero > > > > > elements while using > > > > > * MatSetValues(A,(high-low),**idxm,nc,idxn,values,INSERT_* > > > > > > > > > > *VALUES);* > > > > > And not mention the zero elements and for that I should loop over > all local > > > > > rows and then set each of the value. > > > > > > > > Yes. The whole point of a sparse matrix is to spend no time or > storage > > > > on 0.0 entries. If you allocate a dense matrix and store all the > zeros, > > > > you're just being inefficient relative to a dense format. > > > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Feb 19 15:37:57 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 19 Feb 2016 15:37:57 -0600 Subject: [petsc-users] Doubt in MPIPreallocation In-Reply-To: References: <8760xkokjm.fsf@jedbrown.org> <8737sooj0x.fsf@jedbrown.org> <89FEE4EE-EE07-4109-94C8-541ACA02F87F@mcs.anl.gov> Message-ID: > On Feb 19, 2016, at 3:31 PM, John Albequerque wrote: > > Barry, > I have let PETSc decide the local sizes I have only specified the global values, This cannot work. The preallocation information is provided on a process by process information. Each process has to provide information for its rows. This means you need to know and set the number of local rows of the matrix you cannot use PETSC_DECIDE. > I will also check with the debugger and tell you the status. > > And just another query,is it a problem if I have already used MatSetUp(A); once and after that assembled the matrix and after that applying Preallocation. I guess that should not be a problem as the code ran fine with the one processor case. Just confirming though. No you only do one preallocation ever. If you need another matrix with a different preallocation you create a new matrix and set its preallocation (and delete the old one if you don't need it anymore). Barry > > Many thanks. > > ---- > John Albequerque. > > On Sat, Feb 20, 2016 at 2:53 AM, Barry Smith wrote: > > Are you sure you have divided up the matrix so each process gets some of the rows? > > You are just going to have to break down and run with -start_in_debugger to see exactly why it is crashing. > > Barry > > > On Feb 19, 2016, at 3:16 PM, John Albequerque wrote: > > > > Barry, > > I forgot to mention that the code is working fine with a single processor, but when I increase the number of processors, as in this case 3 processors have been used when the error message was generated. And yes, it has something to do with the off-diagonal terms as I am increasing the number of processes, the same error triggers for each row for which o_nnz[i] is not zero. > > > > > > Many thanks. > > > > ---- > > John Albequerque. > > > > On Sat, Feb 20, 2016 at 2:38 AM, Barry Smith wrote: > > > > You are setting nonzero preallocations for the "off diagonal" portion of the matrix, but one one process there is no off diagonal portion so you should be setting those values to all zero. > > > > Barry > > > > > On Feb 19, 2016, at 2:29 PM, John Albequerque wrote: > > > > > > > > > Dear Barry, > > > I am sorry I could not get you. I have also posted the entire error message so that you could get a deeper insight into it. > > > > > > > > > Many thanks. > > > > > > ---- > > > John Albequerque. > > > ======================================================================================== > > > Argument out of range > > > [0]PETSC ERROR: nnz cannot be greater than row length: local row 0 value 1 rowlength 0 > > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > > [0]PETSC ERROR: Petsc Release Version 3.6.3, unknown > > > [0]PETSC ERROR: ./MatSparse on a linux-gnu-c-debug named John by johncfd Sat Feb 20 01:36:01 2016 > > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich > > > [0]PETSC ERROR: #1 MatSeqAIJSetPreallocation_SeqAIJ() line 3567 in /home//MyStuff/ClonedRepos/petsc/src/mat/impls/aij/seq/aij.c > > > [0]PETSC ERROR: #2 MatSeqAIJSetPreallocation() line 3539 in /home/MyStuff/ClonedRepos/petsc/src/mat/impls/aij/seq/aij.c > > > [0]PETSC ERROR: #3 MatMPIAIJSetPreallocation_MPIAIJ() line 2835 in /home/MyStuff/ClonedRepos/petsc/src/mat/impls/aij/mpi/mpiaij.c > > > [0]PETSC ERROR: #4 MatMPIAIJSetPreallocation() line 3532 in /home/MyStuff/ClonedRepos/petsc/src/mat/impls/aij/mpi/mpiaij.c > > > ========================================================================================= > > > > > > > > > On Sat, Feb 20, 2016 at 12:35 AM, Barry Smith wrote: > > > > > > > On Feb 19, 2016, at 12:54 PM, John Albequerque wrote: > > > > > > > > Jed, one more > > > > > > > > nnz cannot be greater than row length: local row 0 value 1 rowlength 0 > > > > > > Always send the entire error message, this provides the context to know what the hey is going on. > > > > > > It looks like you set a matrix block (if you are using an MPI matrix on one process this could the the "off-diagonal" block) which has no columns (hence row length is zero) but you claim you need to preallocate an entry (the 1). > > > > > > Barry > > > > > > > > > > > How do I deal with this error? > > > > > > > > Thanks. > > > > > > > > ---- > > > > John Albequerque. > > > > > > > > > > > > > > > > On Fri, Feb 19, 2016 at 8:17 PM, John Albequerque wrote: > > > > Thank you very much, I will try it. > > > > > > > > > > > > Thanks > > > > > > > > ---- > > > > John Albequerque. > > > > > > > > > > > > > > > > On Fri, Feb 19, 2016 at 8:16 PM, Jed Brown wrote: > > > > John Albequerque writes: > > > > > > > > > So Jed, what you are suggesting is that I should set only the non-zero > > > > > elements while using > > > > > * MatSetValues(A,(high-low),**idxm,nc,idxn,values,INSERT_* > > > > > > > > > > *VALUES);* > > > > > And not mention the zero elements and for that I should loop over all local > > > > > rows and then set each of the value. > > > > > > > > Yes. The whole point of a sparse matrix is to spend no time or storage > > > > on 0.0 entries. If you allocate a dense matrix and store all the zeros, > > > > you're just being inefficient relative to a dense format. > > > > > > > > > > > > > > > > > > > > From rlmackie862 at gmail.com Fri Feb 19 18:33:50 2016 From: rlmackie862 at gmail.com (Randall Mackie) Date: Fri, 19 Feb 2016 16:33:50 -0800 Subject: [petsc-users] question about VecScatter from one global vector to another Message-ID: <9DC8B7C6-4159-44EA-928D-A6ABFE355D7E@gmail.com> I am trying to do a VecScatter of a subset of elements from a global vector on one DMDA to a global vector on a different DMDA (different sized DMDAs). I thought what made sense was to create a parallel IS using the local to global mapping obtained from the two DMDAs so that the local portion of each IS contained only the values on that processor. This works fine as long as the local size of the two IS?s are the same. But, say I have a situation like this: DMDA 1: proc 1: 1 value to scatter proc 2: 7 values to scatter DMDA 2: proc 1: 4 values to get proc 2: 4 values to get This doesn?t work because VecScatterCreate says the local size of the IS?s must be the same. Is the only way to set up this VecScatter to create a sequential IS on each processor of length 8 and duplicate the values? Thanks, Randy M. From knepley at gmail.com Fri Feb 19 18:39:24 2016 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 19 Feb 2016 18:39:24 -0600 Subject: [petsc-users] question about VecScatter from one global vector to another In-Reply-To: <9DC8B7C6-4159-44EA-928D-A6ABFE355D7E@gmail.com> References: <9DC8B7C6-4159-44EA-928D-A6ABFE355D7E@gmail.com> Message-ID: On Fri, Feb 19, 2016 at 6:33 PM, Randall Mackie wrote: > I am trying to do a VecScatter of a subset of elements from a global > vector on one DMDA to a global vector on a different DMDA (different sized > DMDAs). > > I thought what made sense was to create a parallel IS using the local to > global mapping obtained from the two DMDAs so that the local portion of > each IS contained only the values on that processor. > > This works fine as long as the local size of the two IS?s are the same. > > But, say I have a situation like this: > > DMDA 1: > proc 1: 1 value to scatter > proc 2: 7 values to scatter > > > DMDA 2: > proc 1: 4 values to get > proc 2: 4 values to get > > This doesn?t work because VecScatterCreate says the local size of the IS?s > must be the same. > > Is the only way to set up this VecScatter to create a sequential IS on > each processor of length 8 and duplicate the values? > The two inputs are index of input vector --> input of output vector and you can have as many entries of this map on any process as you want. Clearly, you need the same number of entires in each IS to make this map meaningful. Does that make sense? Thanks, Matt > Thanks, > > Randy M. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From rlmackie862 at gmail.com Fri Feb 19 19:00:20 2016 From: rlmackie862 at gmail.com (Randall Mackie) Date: Fri, 19 Feb 2016 17:00:20 -0800 Subject: [petsc-users] question about VecScatter from one global vector to another In-Reply-To: References: <9DC8B7C6-4159-44EA-928D-A6ABFE355D7E@gmail.com> Message-ID: <31FE8FF5-F53D-451E-9816-EBAA08DFCB15@gmail.com> Yes, it makes perfect sense. The issue is that the global size of my ISs are the same, but it?s the local size that is different (because of different DMDAs) Is there an easy way to put both IS?s on the same parallel layout. ISCopy doesn?t work in that case. Randy > On Feb 19, 2016, at 4:39 PM, Matthew Knepley wrote: > > On Fri, Feb 19, 2016 at 6:33 PM, Randall Mackie > wrote: > I am trying to do a VecScatter of a subset of elements from a global vector on one DMDA to a global vector on a different DMDA (different sized DMDAs). > > I thought what made sense was to create a parallel IS using the local to global mapping obtained from the two DMDAs so that the local portion of each IS contained only the values on that processor. > > This works fine as long as the local size of the two IS?s are the same. > > But, say I have a situation like this: > > DMDA 1: > proc 1: 1 value to scatter > proc 2: 7 values to scatter > > > DMDA 2: > proc 1: 4 values to get > proc 2: 4 values to get > > This doesn?t work because VecScatterCreate says the local size of the IS?s must be the same. > > Is the only way to set up this VecScatter to create a sequential IS on each processor of length 8 and duplicate the values? > > The two inputs are > > index of input vector --> input of output vector > > and you can have as many entries of this map on any process as you want. Clearly, you > need the same number of entires in each IS to make this map meaningful. Does that make > sense? > > Thanks, > > Matt > > Thanks, > > Randy M. > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Feb 19 19:11:50 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 19 Feb 2016 19:11:50 -0600 Subject: [petsc-users] question about VecScatter from one global vector to another In-Reply-To: <31FE8FF5-F53D-451E-9816-EBAA08DFCB15@gmail.com> References: <9DC8B7C6-4159-44EA-928D-A6ABFE355D7E@gmail.com> <31FE8FF5-F53D-451E-9816-EBAA08DFCB15@gmail.com> Message-ID: <91665C16-5FD6-4D1B-9188-528F653B8E34@mcs.anl.gov> How big are the scatters? If they are not large, say 100,000 or less then just gather the send and receive indices to one process and pass them all on that process while all other processes pass 0 indices (it doesn't matter from what process the indices are list in the VecScatterCreate()). If they are huge then you don't want to gather them all to one process, you need to figure out how to gather a bunch to each process (the from and two of course have to be one the same process) and then call the VecScatterCreate(). Barry > On Feb 19, 2016, at 7:00 PM, Randall Mackie wrote: > > Yes, it makes perfect sense. > > The issue is that the global size of my ISs are the same, but it?s the local size that is different (because of different DMDAs) > > Is there an easy way to put both IS?s on the same parallel layout. ISCopy doesn?t work in that case. > > Randy > >> On Feb 19, 2016, at 4:39 PM, Matthew Knepley wrote: >> >> On Fri, Feb 19, 2016 at 6:33 PM, Randall Mackie wrote: >> I am trying to do a VecScatter of a subset of elements from a global vector on one DMDA to a global vector on a different DMDA (different sized DMDAs). >> >> I thought what made sense was to create a parallel IS using the local to global mapping obtained from the two DMDAs so that the local portion of each IS contained only the values on that processor. >> >> This works fine as long as the local size of the two IS?s are the same. >> >> But, say I have a situation like this: >> >> DMDA 1: >> proc 1: 1 value to scatter >> proc 2: 7 values to scatter >> >> >> DMDA 2: >> proc 1: 4 values to get >> proc 2: 4 values to get >> >> This doesn?t work because VecScatterCreate says the local size of the IS?s must be the same. >> >> Is the only way to set up this VecScatter to create a sequential IS on each processor of length 8 and duplicate the values? >> >> The two inputs are >> >> index of input vector --> input of output vector >> >> and you can have as many entries of this map on any process as you want. Clearly, you >> need the same number of entires in each IS to make this map meaningful. Does that make >> sense? >> >> Thanks, >> >> Matt >> >> Thanks, >> >> Randy M. >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener > From johncfdcfd at gmail.com Sun Feb 21 08:11:55 2016 From: johncfdcfd at gmail.com (John Albequerque) Date: Sun, 21 Feb 2016 19:41:55 +0530 Subject: [petsc-users] Doubt in MPIPreallocation In-Reply-To: References: <8760xkokjm.fsf@jedbrown.org> <8737sooj0x.fsf@jedbrown.org> <89FEE4EE-EE07-4109-94C8-541ACA02F87F@mcs.anl.gov> Message-ID: Thank you very much *Barry* and *Matt*, for your help. Its working now. The way I did is I created a dummy matrix of the same layout with the name *'A'. *Then I set it up using MatSetUp(A); . Once this is done, I got the local sizes of the matrix, then created the o_nnz[] and d_nnz[]. After that I *destroyed* *A*, and made the actual working matrix *B, *with the same layout and set the preallocation with the already calculated o_nnz[] and d_nnz[]. And checked it via MatView. :) Many thanks. ---- John Albequerque. On Sat, Feb 20, 2016 at 3:07 AM, Barry Smith wrote: > > > On Feb 19, 2016, at 3:31 PM, John Albequerque > wrote: > > > > Barry, > > I have let PETSc decide the local sizes I have only specified the global > values, > > This cannot work. The preallocation information is provided on a > process by process information. Each process has to provide information for > its rows. This means you need to know and set the number of local rows of > the matrix you cannot use PETSC_DECIDE. > > > I will also check with the debugger and tell you the status. > > > > > > And just another query,is it a problem if I have already used > MatSetUp(A); once and after that assembled the matrix and after that > applying Preallocation. I guess that should not be a problem as the code > ran fine with the one processor case. Just confirming though. > > No you only do one preallocation ever. If you need another matrix with a > different preallocation you create a new matrix and set its preallocation > (and delete the old one if you don't need it anymore). > > Barry > > > > > > Many thanks. > > > > ---- > > John Albequerque. > > > > On Sat, Feb 20, 2016 at 2:53 AM, Barry Smith wrote: > > > > Are you sure you have divided up the matrix so each process gets some > of the rows? > > > > You are just going to have to break down and run with > -start_in_debugger to see exactly why it is crashing. > > > > Barry > > > > > On Feb 19, 2016, at 3:16 PM, John Albequerque > wrote: > > > > > > Barry, > > > I forgot to mention that the code is working fine with a single > processor, but when I increase the number of processors, as in this case 3 > processors have been used when the error message was generated. And yes, it > has something to do with the off-diagonal terms as I am increasing the > number of processes, the same error triggers for each row for which > o_nnz[i] is not zero. > > > > > > > > > Many thanks. > > > > > > ---- > > > John Albequerque. > > > > > > On Sat, Feb 20, 2016 at 2:38 AM, Barry Smith > wrote: > > > > > > You are setting nonzero preallocations for the "off diagonal" > portion of the matrix, but one one process there is no off diagonal portion > so you should be setting those values to all zero. > > > > > > Barry > > > > > > > On Feb 19, 2016, at 2:29 PM, John Albequerque > wrote: > > > > > > > > > > > > Dear Barry, > > > > I am sorry I could not get you. I have also posted the entire error > message so that you could get a deeper insight into it. > > > > > > > > > > > > Many thanks. > > > > > > > > ---- > > > > John Albequerque. > > > > > ======================================================================================== > > > > Argument out of range > > > > [0]PETSC ERROR: nnz cannot be greater than row length: local row 0 > value 1 rowlength 0 > > > > [0]PETSC ERROR: See > http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > > > [0]PETSC ERROR: Petsc Release Version 3.6.3, unknown > > > > [0]PETSC ERROR: ./MatSparse on a linux-gnu-c-debug named John by > johncfd Sat Feb 20 01:36:01 2016 > > > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-fblaslapack --download-mpich > > > > [0]PETSC ERROR: #1 MatSeqAIJSetPreallocation_SeqAIJ() line 3567 in > /home//MyStuff/ClonedRepos/petsc/src/mat/impls/aij/seq/aij.c > > > > [0]PETSC ERROR: #2 MatSeqAIJSetPreallocation() line 3539 in > /home/MyStuff/ClonedRepos/petsc/src/mat/impls/aij/seq/aij.c > > > > [0]PETSC ERROR: #3 MatMPIAIJSetPreallocation_MPIAIJ() line 2835 in > /home/MyStuff/ClonedRepos/petsc/src/mat/impls/aij/mpi/mpiaij.c > > > > [0]PETSC ERROR: #4 MatMPIAIJSetPreallocation() line 3532 in > /home/MyStuff/ClonedRepos/petsc/src/mat/impls/aij/mpi/mpiaij.c > > > > > ========================================================================================= > > > > > > > > > > > > On Sat, Feb 20, 2016 at 12:35 AM, Barry Smith > wrote: > > > > > > > > > On Feb 19, 2016, at 12:54 PM, John Albequerque < > johncfdcfd at gmail.com> wrote: > > > > > > > > > > Jed, one more > > > > > > > > > > nnz cannot be greater than row length: local row 0 value 1 > rowlength 0 > > > > > > > > Always send the entire error message, this provides the context to > know what the hey is going on. > > > > > > > > It looks like you set a matrix block (if you are using an MPI > matrix on one process this could the the "off-diagonal" block) which has > no columns (hence row length is zero) but you claim you need to preallocate > an entry (the 1). > > > > > > > > Barry > > > > > > > > > > > > > > How do I deal with this error? > > > > > > > > > > Thanks. > > > > > > > > > > ---- > > > > > John Albequerque. > > > > > > > > > > > > > > > > > > > > On Fri, Feb 19, 2016 at 8:17 PM, John Albequerque < > johncfdcfd at gmail.com> wrote: > > > > > Thank you very much, I will try it. > > > > > > > > > > > > > > > Thanks > > > > > > > > > > ---- > > > > > John Albequerque. > > > > > > > > > > > > > > > > > > > > On Fri, Feb 19, 2016 at 8:16 PM, Jed Brown > wrote: > > > > > John Albequerque writes: > > > > > > > > > > > So Jed, what you are suggesting is that I should set only the > non-zero > > > > > > elements while using > > > > > > * MatSetValues(A,(high-low),**idxm,nc,idxn,values,INSERT_* > > > > > > > > > > > > *VALUES);* > > > > > > And not mention the zero elements and for that I should loop > over all local > > > > > > rows and then set each of the value. > > > > > > > > > > Yes. The whole point of a sparse matrix is to spend no time or > storage > > > > > on 0.0 entries. If you allocate a dense matrix and store all the > zeros, > > > > > you're just being inefficient relative to a dense format. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ppratapa at gatech.edu Mon Feb 22 12:44:33 2016 From: ppratapa at gatech.edu (Phanisri Pradeep Pratapa) Date: Mon, 22 Feb 2016 13:44:33 -0500 Subject: [petsc-users] Variable wall times Message-ID: Hi, I am trying to compare run times for a function in petsc using ksp solve. When I run the code in parallel on a single node (64 processors), it seems to take different wall time for each run, although run on exactly same processors. For example one time it could take 10 sec where as the same run next time could take 20 sec. (This happens for other choices of processors too, not just 64) I am solving a linear system using gmres. I have attached four output files for the same inputs and run on same processors. You can see the differences. It seems like the variability comes from each processor taking different time to complete the MatMult and KSPGMRESOrthog (you can see profiling output from petsc in the attached files). Please let me know if you have any idea on how I can overcome this problem. Thank you, Regards, Pradeep -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: output_files.zip Type: application/zip Size: 13493 bytes Desc: not available URL: From jed at jedbrown.org Mon Feb 22 13:08:04 2016 From: jed at jedbrown.org (Jed Brown) Date: Mon, 22 Feb 2016 12:08:04 -0700 Subject: [petsc-users] Variable wall times In-Reply-To: References: Message-ID: <878u2ck1gr.fsf@jedbrown.org> Phanisri Pradeep Pratapa writes: > Hi, > > I am trying to compare run times for a function in petsc using ksp solve. > When I run the code in parallel on a single node (64 processors), it seems > to take different wall time for each run, although run on exactly same > processors. For example one time it could take 10 sec where as the same run > next time could take 20 sec. (This happens for other choices of processors > too, not just 64) You probably just have a noisy system. It could be due to the architecture and system software or it could be caused by contention with other jobs and/or misconfiguration. See this paper, especially Figure 1. http://unixer.de/publications/img/hoefler-scientific-benchmarking.pdf -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From bsmith at mcs.anl.gov Mon Feb 22 13:14:59 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 22 Feb 2016 13:14:59 -0600 Subject: [petsc-users] Variable wall times In-Reply-To: <878u2ck1gr.fsf@jedbrown.org> References: <878u2ck1gr.fsf@jedbrown.org> Message-ID: <16C6626B-D7F8-4713-AAD2-05F1FBE52954@mcs.anl.gov> Also make sure you are doing consistent binding of MPI processes to cores: see http://www.mcs.anl.gov/petsc/documentation/faq.html#computers and run the streams benchmarks several times to see if you get consistent results. With no other user or system processes consuming large amounts of cycles and suitable binding you should get pretty consistent results. Barry > On Feb 22, 2016, at 1:08 PM, Jed Brown wrote: > > Phanisri Pradeep Pratapa writes: > >> Hi, >> >> I am trying to compare run times for a function in petsc using ksp solve. >> When I run the code in parallel on a single node (64 processors), it seems >> to take different wall time for each run, although run on exactly same >> processors. For example one time it could take 10 sec where as the same run >> next time could take 20 sec. (This happens for other choices of processors >> too, not just 64) > > You probably just have a noisy system. It could be due to the > architecture and system software or it could be caused by contention > with other jobs and/or misconfiguration. > > See this paper, especially Figure 1. > > http://unixer.de/publications/img/hoefler-scientific-benchmarking.pdf From ppratapa at gatech.edu Mon Feb 22 15:03:13 2016 From: ppratapa at gatech.edu (Phanisri Pradeep Pratapa) Date: Mon, 22 Feb 2016 16:03:13 -0500 Subject: [petsc-users] Variable wall times In-Reply-To: <16C6626B-D7F8-4713-AAD2-05F1FBE52954@mcs.anl.gov> References: <878u2ck1gr.fsf@jedbrown.org> <16C6626B-D7F8-4713-AAD2-05F1FBE52954@mcs.anl.gov> Message-ID: @Barry: I am running the job on a single node with 64 processors and using all of them. So do I still have to do the binding? (From what I understand from the link you have provided, the switching of processes between cores could be the reason. But will that be a reason even when all processors are doing equal amount of computations?) @Jed: There are no other jobs running on that node (and also no other jobs on the queue). So could the noise be due to load balance issue or something else in computing dot products and MatMults? Is this to be expected? If so, is the only work around for this is to report a statistics of wall time for the runs? (When I compare two different algorithms, the speed-up of one over the other varies significantly because of this issue.) Thank you for the replies. Regards, Pradeep On Mon, Feb 22, 2016 at 2:14 PM, Barry Smith wrote: > > Also make sure you are doing consistent binding of MPI processes to > cores: see http://www.mcs.anl.gov/petsc/documentation/faq.html#computers > and run the streams benchmarks several times to see if you get consistent > results. > > With no other user or system processes consuming large amounts of > cycles and suitable binding you should get pretty consistent results. > > Barry > > > > > On Feb 22, 2016, at 1:08 PM, Jed Brown wrote: > > > > Phanisri Pradeep Pratapa writes: > > > >> Hi, > >> > >> I am trying to compare run times for a function in petsc using ksp > solve. > >> When I run the code in parallel on a single node (64 processors), it > seems > >> to take different wall time for each run, although run on exactly same > >> processors. For example one time it could take 10 sec where as the same > run > >> next time could take 20 sec. (This happens for other choices of > processors > >> too, not just 64) > > > > You probably just have a noisy system. It could be due to the > > architecture and system software or it could be caused by contention > > with other jobs and/or misconfiguration. > > > > See this paper, especially Figure 1. > > > > http://unixer.de/publications/img/hoefler-scientific-benchmarking.pdf > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Mon Feb 22 15:16:01 2016 From: jed at jedbrown.org (Jed Brown) Date: Mon, 22 Feb 2016 14:16:01 -0700 Subject: [petsc-users] Variable wall times In-Reply-To: References: <878u2ck1gr.fsf@jedbrown.org> <16C6626B-D7F8-4713-AAD2-05F1FBE52954@mcs.anl.gov> Message-ID: <87k2lwigz2.fsf@jedbrown.org> Phanisri Pradeep Pratapa writes: > @Jed: > There are no other jobs running on that node (and also no other jobs on the > queue). So could the noise be due to load balance issue or something else > in computing dot products and MatMults? Is this to be expected? If so, is > the only work around for this is to report a statistics of wall time for > the runs? (When I compare two different algorithms, the speed-up of one > over the other varies significantly because of this issue.) Try using only 32 processes. Hyperthreads aren't full threads and you could just be oversubscribed. Also note that there are system processes that can take time. If the performance varies between runs, you have to look at nondeterministic factors rather than deterministic factors like partition-induced load balance. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From bsmith at mcs.anl.gov Mon Feb 22 15:17:45 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 22 Feb 2016 15:17:45 -0600 Subject: [petsc-users] Variable wall times In-Reply-To: References: <878u2ck1gr.fsf@jedbrown.org> <16C6626B-D7F8-4713-AAD2-05F1FBE52954@mcs.anl.gov> Message-ID: <7DC946B8-F3E1-49EF-87BB-8C214E6E556E@mcs.anl.gov> > On Feb 22, 2016, at 3:03 PM, Phanisri Pradeep Pratapa wrote: > > @Barry: > I am running the job on a single node with 64 processors and using all of them. So do I still have to do the binding? (From what I understand from the link you have provided, the switching of processes between cores could be the reason. But will that be a reason even when all processors are doing equal amount of computations?) Absolutely! If processes are switched then all data in the cache is invalid so having the processes migrate around (even if you are the only one using the machine) can change the timing a lot. Barry > > @Jed: > There are no other jobs running on that node (and also no other jobs on the queue). So could the noise be due to load balance issue or something else in computing dot products and MatMults? Is this to be expected? If so, is the only work around for this is to report a statistics of wall time for the runs? (When I compare two different algorithms, the speed-up of one over the other varies significantly because of this issue.) > > Thank you for the replies. > > Regards, > > Pradeep > > On Mon, Feb 22, 2016 at 2:14 PM, Barry Smith wrote: > > Also make sure you are doing consistent binding of MPI processes to cores: see http://www.mcs.anl.gov/petsc/documentation/faq.html#computers and run the streams benchmarks several times to see if you get consistent results. > > With no other user or system processes consuming large amounts of cycles and suitable binding you should get pretty consistent results. > > Barry > > > > > On Feb 22, 2016, at 1:08 PM, Jed Brown wrote: > > > > Phanisri Pradeep Pratapa writes: > > > >> Hi, > >> > >> I am trying to compare run times for a function in petsc using ksp solve. > >> When I run the code in parallel on a single node (64 processors), it seems > >> to take different wall time for each run, although run on exactly same > >> processors. For example one time it could take 10 sec where as the same run > >> next time could take 20 sec. (This happens for other choices of processors > >> too, not just 64) > > > > You probably just have a noisy system. It could be due to the > > architecture and system software or it could be caused by contention > > with other jobs and/or misconfiguration. > > > > See this paper, especially Figure 1. > > > > http://unixer.de/publications/img/hoefler-scientific-benchmarking.pdf > > From timothee.nicolas at gmail.com Mon Feb 22 16:16:41 2016 From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=) Date: Mon, 22 Feb 2016 15:16:41 -0700 Subject: [petsc-users] Weighted Jacobi, and scaling 2nd matrix in KSPSetOperators Message-ID: Hi all, It sounds it should be obvious but I can't find it somehow. I would like to use weighted jacobi (more precisely point block jacobi) as a smoother for multigrid, but I don't find the options to set a weight omega, after I set -mg_levels_pc_type pbjacobi So I was thinking I could just artificially scale the second matrix in KSPSetOperators with a weight omega. Since it is the one used in retrieving the diagonal, it should be equivalent to have a weight and should not affect the residuals, right ? However, I notice no change of behaviour whatsoever when I do this. Is it normal ? Have I grossly misunderstood something ? Thanks Timothee -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Mon Feb 22 16:28:28 2016 From: dave.mayhem23 at gmail.com (Dave May) Date: Mon, 22 Feb 2016 23:28:28 +0100 Subject: [petsc-users] Weighted Jacobi, and scaling 2nd matrix in KSPSetOperators In-Reply-To: References: Message-ID: On 22 February 2016 at 23:16, Timoth?e Nicolas wrote: > Hi all, > > It sounds it should be obvious but I can't find it somehow. I would like > to use weighted jacobi (more precisely point block jacobi) as a smoother > for multigrid, but I don't find the options to set a weight omega, after I > set -mg_levels_pc_type pbjacobi > Why not use -ksp_type richardson -ksp_richardson_scale XXX ? Cheers, Dave So I was thinking I could just artificially scale the second matrix in > KSPSetOperators with a weight omega. Since it is the one used in retrieving > the diagonal, it should be equivalent to have a weight and should not > affect the residuals, right ? However, I notice no change of behaviour > whatsoever when I do this. > > Is it normal ? Have I grossly misunderstood something ? > > Thanks > > Timothee > -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothee.nicolas at gmail.com Mon Feb 22 16:38:55 2016 From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=) Date: Mon, 22 Feb 2016 15:38:55 -0700 Subject: [petsc-users] Weighted Jacobi, and scaling 2nd matrix in KSPSetOperators In-Reply-To: References: Message-ID: Yes, exactly. I'm really sorry, I just sent the e-mail without thinking enough... My bad. Thanks 2016-02-22 15:28 GMT-07:00 Dave May : > > > On 22 February 2016 at 23:16, Timoth?e Nicolas > wrote: > >> Hi all, >> >> It sounds it should be obvious but I can't find it somehow. I would like >> to use weighted jacobi (more precisely point block jacobi) as a smoother >> for multigrid, but I don't find the options to set a weight omega, after I >> set -mg_levels_pc_type pbjacobi >> > > Why not use -ksp_type richardson -ksp_richardson_scale XXX ? > > Cheers, > Dave > > So I was thinking I could just artificially scale the second matrix in >> KSPSetOperators with a weight omega. Since it is the one used in retrieving >> the diagonal, it should be equivalent to have a weight and should not >> affect the residuals, right ? However, I notice no change of behaviour >> whatsoever when I do this. >> >> Is it normal ? Have I grossly misunderstood something ? >> >> Thanks >> >> Timothee >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.knezevic at akselos.com Mon Feb 22 17:54:51 2016 From: david.knezevic at akselos.com (David Knezevic) Date: Mon, 22 Feb 2016 18:54:51 -0500 Subject: [petsc-users] Convergence trouble with SNES and CG Message-ID: I'm solving a "linear elasticity with contact" problem using SNES. The matrix is symmetric, and it corresponds to the standard linear elasticity matrix, plus some extra nonlinear "penalty spring" terms due to the contact conditions. The solve works fine if I use a direct solver (e.g. MUMPS). I've also found that CG usually works fine too, but in one specific case it fails to converge. (I thought that penalty springs + iterative solvers may not be a good combination, but it does seem to work fine in most cases.) Below I've pasted the (last part of the) output of "-ksp_type cg -ksp_norm_type natural -snes_view -ksp_monitor" for the case that fails. If anyone has some suggestions about other solver/preconditioner options that would be worth trying, that would be most appreciated. Thanks, David ------------------------------------------------------------------------------- 102 KSP Residual norm 8.910585826247e-05 103 KSP Residual norm 8.395178827803e-05 104 KSP Residual norm 7.758478472168e-05 105 KSP Residual norm 7.099534889984e-05 106 KSP Residual norm 6.615351528474e-05 107 KSP Residual norm 6.303646443688e-05 NL step 5, |residual|_2 = 1.996487e+01 0 KSP Residual norm 5.844456247091e+00 1 KSP Residual norm 1.308756943973e+00 2 KSP Residual norm 7.545133187463e-01 3 KSP Residual norm 1.346068942607e+00 SNES Object: 1 MPI processes type: newtonls maximum iterations=50, maximum function evaluations=10000 tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 total number of linear solver iterations=535 total number of function evaluations=11 norm schedule ALWAYS SNESLineSearch Object: 1 MPI processes type: bt interpolation: cubic alpha=1.000000e-04 maxstep=1.000000e+08, minlambda=1.000000e-12 tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 maximum iterations=40 KSP Object: 1 MPI processes type: cg maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NATURAL norm type for convergence test PC Object: 1 MPI processes type: ilu ILU: out-of-place factorization 0 levels of fill tolerance for zero pivot 2.22045e-14 matrix ordering: natural factor fill ratio given 1, needed 1 Factored matrix follows: Mat Object: 1 MPI processes type: seqaij rows=967608, cols=967608 package used to perform factorization: petsc total: nonzeros=7.55539e+07, allocated nonzeros=7.55539e+07 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 322536 nodes, limit used is 5 linear system matrix = precond matrix: Mat Object: () 1 MPI processes type: seqaij rows=967608, cols=967608 total: nonzeros=7.55539e+07, allocated nonzeros=7.55539e+07 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 322536 nodes, limit used is 5 Number of nonlinear iterations: 5 Nonlinear solver convergence/divergence reason: DIVERGED_LINEAR_SOLVE -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Feb 22 17:57:15 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 22 Feb 2016 17:57:15 -0600 Subject: [petsc-users] Convergence trouble with SNES and CG In-Reply-To: References: Message-ID: Add -ksp_converged_reason > On Feb 22, 2016, at 5:54 PM, David Knezevic wrote: > > I'm solving a "linear elasticity with contact" problem using SNES. The matrix is symmetric, and it corresponds to the standard linear elasticity matrix, plus some extra nonlinear "penalty spring" terms due to the contact conditions. > > The solve works fine if I use a direct solver (e.g. MUMPS). I've also found that CG usually works fine too, but in one specific case it fails to converge. (I thought that penalty springs + iterative solvers may not be a good combination, but it does seem to work fine in most cases.) > > Below I've pasted the (last part of the) output of "-ksp_type cg -ksp_norm_type natural -snes_view -ksp_monitor" for the case that fails. > > If anyone has some suggestions about other solver/preconditioner options that would be worth trying, that would be most appreciated. > > Thanks, > David > > ------------------------------------------------------------------------------- > > 102 KSP Residual norm 8.910585826247e-05 > 103 KSP Residual norm 8.395178827803e-05 > 104 KSP Residual norm 7.758478472168e-05 > 105 KSP Residual norm 7.099534889984e-05 > 106 KSP Residual norm 6.615351528474e-05 > 107 KSP Residual norm 6.303646443688e-05 > NL step 5, |residual|_2 = 1.996487e+01 > 0 KSP Residual norm 5.844456247091e+00 > 1 KSP Residual norm 1.308756943973e+00 > 2 KSP Residual norm 7.545133187463e-01 > 3 KSP Residual norm 1.346068942607e+00 > SNES Object: 1 MPI processes > type: newtonls > maximum iterations=50, maximum function evaluations=10000 > tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 > total number of linear solver iterations=535 > total number of function evaluations=11 > norm schedule ALWAYS > SNESLineSearch Object: 1 MPI processes > type: bt > interpolation: cubic > alpha=1.000000e-04 > maxstep=1.000000e+08, minlambda=1.000000e-12 > tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 > maximum iterations=40 > KSP Object: 1 MPI processes > type: cg > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using NATURAL norm type for convergence test > PC Object: 1 MPI processes > type: ilu > ILU: out-of-place factorization > 0 levels of fill > tolerance for zero pivot 2.22045e-14 > matrix ordering: natural > factor fill ratio given 1, needed 1 > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqaij > rows=967608, cols=967608 > package used to perform factorization: petsc > total: nonzeros=7.55539e+07, allocated nonzeros=7.55539e+07 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 322536 nodes, limit used is 5 > linear system matrix = precond matrix: > Mat Object: () 1 MPI processes > type: seqaij > rows=967608, cols=967608 > total: nonzeros=7.55539e+07, allocated nonzeros=7.55539e+07 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 322536 nodes, limit used is 5 > Number of nonlinear iterations: 5 > Nonlinear solver convergence/divergence reason: DIVERGED_LINEAR_SOLVE From david.knezevic at akselos.com Mon Feb 22 18:02:50 2016 From: david.knezevic at akselos.com (David Knezevic) Date: Mon, 22 Feb 2016 19:02:50 -0500 Subject: [petsc-users] Convergence trouble with SNES and CG In-Reply-To: References: Message-ID: On Mon, Feb 22, 2016 at 6:57 PM, Barry Smith wrote: > > Add -ksp_converged_reason > That gives "DIVERGED_INDEFINITE_PC". The full output is below: 105 KSP Residual norm 7.099534889984e-05 106 KSP Residual norm 6.615351528474e-05 107 KSP Residual norm 6.303646443688e-05 Linear solve converged due to CONVERGED_RTOL iterations 107 NL step 5, |residual|_2 = 1.996487e+01 0 KSP Residual norm 5.844456247091e+00 1 KSP Residual norm 1.308756943973e+00 2 KSP Residual norm 7.545133187463e-01 3 KSP Residual norm 1.346068942607e+00 Linear solve did not converge due to DIVERGED_INDEFINITE_PC iterations 4 SNES Object: 1 MPI processes type: newtonls maximum iterations=50, maximum function evaluations=10000 tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 total number of linear solver iterations=535 total number of function evaluations=11 norm schedule ALWAYS SNESLineSearch Object: 1 MPI processes type: bt interpolation: cubic alpha=1.000000e-04 maxstep=1.000000e+08, minlambda=1.000000e-12 tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 maximum iterations=40 KSP Object: 1 MPI processes type: cg maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NATURAL norm type for convergence test PC Object: 1 MPI processes type: ilu ILU: out-of-place factorization 0 levels of fill tolerance for zero pivot 2.22045e-14 matrix ordering: natural factor fill ratio given 1, needed 1 Factored matrix follows: Mat Object: 1 MPI processes type: seqaij rows=967608, cols=967608 package used to perform factorization: petsc total: nonzeros=7.55539e+07, allocated nonzeros=7.55539e+07 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 322536 nodes, limit used is 5 linear system matrix = precond matrix: Mat Object: () 1 MPI processes type: seqaij rows=967608, cols=967608 total: nonzeros=7.55539e+07, allocated nonzeros=7.55539e+07 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 322536 nodes, limit used is 5 Number of nonlinear iterations: 5 Nonlinear solver convergence/divergence reason: DIVERGED_LINEAR_SOLVE -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Feb 22 18:10:43 2016 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 22 Feb 2016 18:10:43 -0600 Subject: [petsc-users] Convergence trouble with SNES and CG In-Reply-To: References: Message-ID: On Mon, Feb 22, 2016 at 6:02 PM, David Knezevic wrote: > On Mon, Feb 22, 2016 at 6:57 PM, Barry Smith wrote: > >> >> Add -ksp_converged_reason >> > > > That gives "DIVERGED_INDEFINITE_PC". The full output is below: > Just for testing, replace CG with GMRES. Thanks, Matt > > 105 KSP Residual norm 7.099534889984e-05 > 106 KSP Residual norm 6.615351528474e-05 > 107 KSP Residual norm 6.303646443688e-05 > Linear solve converged due to CONVERGED_RTOL iterations 107 > NL step 5, |residual|_2 = 1.996487e+01 > 0 KSP Residual norm 5.844456247091e+00 > 1 KSP Residual norm 1.308756943973e+00 > 2 KSP Residual norm 7.545133187463e-01 > 3 KSP Residual norm 1.346068942607e+00 > Linear solve did not converge due to DIVERGED_INDEFINITE_PC iterations 4 > SNES Object: 1 MPI processes > type: newtonls > maximum iterations=50, maximum function evaluations=10000 > tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 > total number of linear solver iterations=535 > total number of function evaluations=11 > norm schedule ALWAYS > SNESLineSearch Object: 1 MPI processes > type: bt > interpolation: cubic > alpha=1.000000e-04 > maxstep=1.000000e+08, minlambda=1.000000e-12 > tolerances: relative=1.000000e-08, absolute=1.000000e-15, > lambda=1.000000e-08 > maximum iterations=40 > KSP Object: 1 MPI processes > type: cg > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > using NATURAL norm type for convergence test > PC Object: 1 MPI processes > type: ilu > ILU: out-of-place factorization > 0 levels of fill > tolerance for zero pivot 2.22045e-14 > matrix ordering: natural > factor fill ratio given 1, needed 1 > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqaij > rows=967608, cols=967608 > package used to perform factorization: petsc > total: nonzeros=7.55539e+07, allocated nonzeros=7.55539e+07 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 322536 nodes, limit used is 5 > linear system matrix = precond matrix: > Mat Object: () 1 MPI processes > type: seqaij > rows=967608, cols=967608 > total: nonzeros=7.55539e+07, allocated nonzeros=7.55539e+07 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 322536 nodes, limit used is 5 > Number of nonlinear iterations: 5 > Nonlinear solver convergence/divergence reason: DIVERGED_LINEAR_SOLVE > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.knezevic at akselos.com Mon Feb 22 18:27:16 2016 From: david.knezevic at akselos.com (David Knezevic) Date: Mon, 22 Feb 2016 19:27:16 -0500 Subject: [petsc-users] Convergence trouble with SNES and CG In-Reply-To: References: Message-ID: On Mon, Feb 22, 2016 at 7:10 PM, Matthew Knepley wrote: > On Mon, Feb 22, 2016 at 6:02 PM, David Knezevic < > david.knezevic at akselos.com> wrote: > >> On Mon, Feb 22, 2016 at 6:57 PM, Barry Smith wrote: >> >>> >>> Add -ksp_converged_reason >>> >> >> >> That gives "DIVERGED_INDEFINITE_PC". The full output is below: >> > > Just for testing, replace CG with GMRES. > With GMRES the first four nonlinear iterations behave much that same as before, converging at KSP iteration 150 or so. On the 5th nonlinear iteration, GMRES doesn't give a KSP convergence failure, but the convergence seems to stagnate. It's currently at KSP iteration 2000, and tolerance of 6e-3 (and still running). David -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Feb 22 19:06:55 2016 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 22 Feb 2016 19:06:55 -0600 Subject: [petsc-users] Convergence trouble with SNES and CG In-Reply-To: References: Message-ID: On Mon, Feb 22, 2016 at 6:27 PM, David Knezevic wrote: > On Mon, Feb 22, 2016 at 7:10 PM, Matthew Knepley > wrote: > >> On Mon, Feb 22, 2016 at 6:02 PM, David Knezevic < >> david.knezevic at akselos.com> wrote: >> >>> On Mon, Feb 22, 2016 at 6:57 PM, Barry Smith wrote: >>> >>>> >>>> Add -ksp_converged_reason >>>> >>> >>> >>> That gives "DIVERGED_INDEFINITE_PC". The full output is below: >>> >> >> Just for testing, replace CG with GMRES. >> > > With GMRES the first four nonlinear iterations behave much that same as > before, converging at KSP iteration 150 or so. > > On the 5th nonlinear iteration, GMRES doesn't give a KSP convergence > failure, but the convergence seems to stagnate. It's currently at KSP > iteration 2000, and tolerance of 6e-3 (and still running). > Can you use LU? The idea is to determine is the linear system becomes (almost) rank deficient or if the nonlinear problem is just hard. Matt > David > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From snakexf at gmail.com Tue Feb 23 02:46:02 2016 From: snakexf at gmail.com (Feng Xing) Date: Tue, 23 Feb 2016 09:46:02 +0100 Subject: [petsc-users] ILU(k) preconditioner version 3.6 Message-ID: <0A590646-90F6-481C-BB48-913801BC14B4@gmail.com> Hello everyone, I found in the release note of version 3.6 that "Removed -pc_hypre_type euclid due to bit-rot?. I would like to ask if there is no preconditioner ILU(k) for parallel aij matrix from 3.6? Thank you very much. Feng XING Postdoc INRIA From bsmith at mcs.anl.gov Tue Feb 23 11:39:37 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 23 Feb 2016 11:39:37 -0600 Subject: [petsc-users] ILU(k) preconditioner version 3.6 In-Reply-To: <0A590646-90F6-481C-BB48-913801BC14B4@gmail.com> References: <0A590646-90F6-481C-BB48-913801BC14B4@gmail.com> Message-ID: Yes, it is no longer available. If you have found that it is definitely the best preconditioner for your problem then you should likely stick with release 3.5 Otherwise you can experiment with other preconditioners to determine a good replacement; this, of course, if very problem specific. Barry > On Feb 23, 2016, at 2:46 AM, Feng Xing wrote: > > Hello everyone, > > I found in the release note of version 3.6 that "Removed -pc_hypre_type euclid due to bit-rot?. > > I would like to ask if there is no preconditioner ILU(k) for parallel aij matrix from 3.6? > > Thank you very much. > > Feng XING > Postdoc INRIA From snakexf at gmail.com Tue Feb 23 12:37:57 2016 From: snakexf at gmail.com (Feng Xing) Date: Tue, 23 Feb 2016 19:37:57 +0100 Subject: [petsc-users] ILU(k) preconditioner version 3.6 In-Reply-To: References: <0A590646-90F6-481C-BB48-913801BC14B4@gmail.com> Message-ID: <9DAD05BF-351B-41CD-BE7A-B37D92E24F87@gmail.com> OK, thank you. But, it?s for which reason that PETSc abandons ILU(k)? Feng XING Postdoc INRIA > On 23 Feb 2016, at 18:39, Barry Smith wrote: > > > Yes, it is no longer available. If you have found that it is definitely the best preconditioner for your problem then you should likely stick with release 3.5 Otherwise you can experiment with other preconditioners to determine a good replacement; this, of course, if very problem specific. > > Barry > >> On Feb 23, 2016, at 2:46 AM, Feng Xing wrote: >> >> Hello everyone, >> >> I found in the release note of version 3.6 that "Removed -pc_hypre_type euclid due to bit-rot?. >> >> I would like to ask if there is no preconditioner ILU(k) for parallel aij matrix from 3.6? >> >> Thank you very much. >> >> Feng XING >> Postdoc INRIA > From knepley at gmail.com Tue Feb 23 12:59:18 2016 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 23 Feb 2016 12:59:18 -0600 Subject: [petsc-users] ILU(k) preconditioner version 3.6 In-Reply-To: <9DAD05BF-351B-41CD-BE7A-B37D92E24F87@gmail.com> References: <0A590646-90F6-481C-BB48-913801BC14B4@gmail.com> <9DAD05BF-351B-41CD-BE7A-B37D92E24F87@gmail.com> Message-ID: On Tue, Feb 23, 2016 at 12:37 PM, Feng Xing wrote: > OK, thank you. But, it?s for which reason that PETSc abandons ILU(k)? > The code was not being mantained and it constantly broke in PETSc tests. It was not worth our time to keep in the package, since the maintainers did not seem to care. Matt > Feng XING > Postdoc INRIA > > > > On 23 Feb 2016, at 18:39, Barry Smith wrote: > > > > > > Yes, it is no longer available. If you have found that it is > definitely the best preconditioner for your problem then you should likely > stick with release 3.5 Otherwise you can experiment with other > preconditioners to determine a good replacement; this, of course, if very > problem specific. > > > > Barry > > > >> On Feb 23, 2016, at 2:46 AM, Feng Xing wrote: > >> > >> Hello everyone, > >> > >> I found in the release note of version 3.6 that "Removed -pc_hypre_type > euclid due to bit-rot?. > >> > >> I would like to ask if there is no preconditioner ILU(k) for parallel > aij matrix from 3.6? > >> > >> Thank you very much. > >> > >> Feng XING > >> Postdoc INRIA > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jychang48 at gmail.com Tue Feb 23 14:03:05 2016 From: jychang48 at gmail.com (Justin Chang) Date: Tue, 23 Feb 2016 14:03:05 -0600 Subject: [petsc-users] Difference between Block Jacobi and ILU? In-Reply-To: References: Message-ID: Two more questions, somewhat related maybe. Is there a practical case where one would use plain Jacobi preconditioning over ILU? Also, what exactly is happening when one uses -pc_bjacobi_blocks 2 ? Thanks, Justin On Wed, Jan 13, 2016 at 9:37 PM, Justin Chang wrote: > Thanks Satish, > > And yes I meant sequentially. > > On Wed, Jan 13, 2016 at 8:26 PM, Satish Balay wrote: > >> On Wed, 13 Jan 2016, Justin Chang wrote: >> >> > Hi all, >> > >> > What exactly is the difference between these two preconditioners? When I >> > use them to solve a Galerkin finite element poisson problem, I get the >> > exact same performance (iterations, wall-clock time, etc). >> >> you mean - when you run sequentially? >> >> With block jacobi - you decide the number of blocks. The default is >> 1-block/proc >> i.e - for sequnetial run you have only 1block i.e the whole matrix. >> >> So the following are essentially the same: >> -pc_type bjacobi -pc_bjacobi_blocks 1 [default] -sub_pc_type ilu [default] >> -pc_type ilu >> >> Satish >> >> > Only thing is I can't seem to use ILU in parallel though. >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Feb 23 14:13:05 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 23 Feb 2016 14:13:05 -0600 Subject: [petsc-users] Difference between Block Jacobi and ILU? In-Reply-To: References: Message-ID: > On Feb 23, 2016, at 2:03 PM, Justin Chang wrote: > > Two more questions, somewhat related maybe. > > Is there a practical case where one would use plain Jacobi preconditioning over ILU? For well conditioned problems an iteration of Jacobi is cheaper than an iteration of ILU (about 1/2 the work) so jacobi can beat ILU For problems where ILU produces 0 (or tiny) pivots and thus produces a bad preconditioner > > Also, what exactly is happening when one uses -pc_bjacobi_blocks 2 ? By default PETSc uses one block per MPI process. -pc_bjacobi_blocks 2 would produce exactly 2 blocks totally. See PCBJacobiSetTotalBlocks() and PCBJacobiSetLocalBlocks() > > Thanks, > Justin > > On Wed, Jan 13, 2016 at 9:37 PM, Justin Chang wrote: > Thanks Satish, > > And yes I meant sequentially. > > On Wed, Jan 13, 2016 at 8:26 PM, Satish Balay wrote: > On Wed, 13 Jan 2016, Justin Chang wrote: > > > Hi all, > > > > What exactly is the difference between these two preconditioners? When I > > use them to solve a Galerkin finite element poisson problem, I get the > > exact same performance (iterations, wall-clock time, etc). > > you mean - when you run sequentially? > > With block jacobi - you decide the number of blocks. The default is 1-block/proc > i.e - for sequnetial run you have only 1block i.e the whole matrix. > > So the following are essentially the same: > -pc_type bjacobi -pc_bjacobi_blocks 1 [default] -sub_pc_type ilu [default] > -pc_type ilu > > Satish > > > Only thing is I can't seem to use ILU in parallel though. > > > From zonexo at gmail.com Tue Feb 23 19:50:52 2016 From: zonexo at gmail.com (TAY wee-beng) Date: Wed, 24 Feb 2016 09:50:52 +0800 Subject: [petsc-users] Error - Out of memory. This could be due to allocating too large an object or bleeding by not properly ... Message-ID: <56CD0C7C.5060101@gmail.com> Hi, I got this error (also attached, full) when running my code. It happens after a few thousand time steps. The strange thing is that for 2 different clusters, it stops at 2 different time steps. I wonder if it's related to DM since this happens after I added DM into my code. In this case, how can I find out the error? I'm thinking valgrind may take very long and gives too many false errors. -- Thank you Yours sincerely, TAY wee-beng -------------- next part -------------- **************************************************************************** * hwloc has encountered what looks like an error from the operating system. * * L3 (cpuset 0x000003f0) intersects with NUMANode (P#0 cpuset 0x0000003f) without inclusion! * Error occurred in topology.c line 942 * * The following FAQ entry in a recent hwloc documentation may help: * What should I do when hwloc reports "operating system" warnings? * Otherwise please report this error message to the hwloc user's mailing list, * along with the output+tarball generated by the hwloc-gather-topology script. **************************************************************************** **************************************************************************** * hwloc has encountered what looks like an error from the operating system. * * L3 (cpuset 0x000003f0) intersects with NUMANode (P#0 cpuset 0x0000003f) without inclusion! * Error occurred in topology.c line 942 * * The following FAQ entry in a recent hwloc documentation may help: * What should I do when hwloc reports "operating system" warnings? * Otherwise please report this error message to the hwloc user's mailing list, * along with the output+tarball generated by the hwloc-gather-topology script. **************************************************************************** [35]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [35]PETSC ERROR: Out of memory. This could be due to allocating [35]PETSC ERROR: too large an object or bleeding by not properly [35]PETSC ERROR: destroying unneeded objects. [46]PETSC ERROR: [19]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [19]PETSC ERROR: Out of memory. This could be due to allocating [19]PETSC ERROR: too large an object or bleeding by not properly [19]PETSC ERROR: destroying unneeded objects. [19]PETSC ERROR: Memory allocated 0 Memory used by process 3551756288 [19]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. [19]PETSC ERROR: Memory requested 34905645 [19]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [19]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [32]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [32]PETSC ERROR: Out of memory. This could be due to allocating [32]PETSC ERROR: too large an object or bleeding by not properly [32]PETSC ERROR: destroying unneeded objects. [35]PETSC ERROR: Memory allocated 0 Memory used by process 3522326528 [35]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. [35]PETSC ERROR: Memory requested 34905645 [35]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [35]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [35]PETSC ERROR: ./a.out on a petsc-3.6.3_static_rel named n12-03 by wtay Sun Feb 21 08:24:37 2016 [35]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1 [35]PETSC ERROR: #1 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [35]PETSC ERROR: #2 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [35]PETSC ERROR: #3 MatLUFactorNumeric() line 2946 in /home/wtay/Codes/petsc-3.6.3/src/mat/interface/matrix.c [35]PETSC ERROR: #4 PCSetUp_ILU() line 233 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/factor/ilu/ilu.c [35]PETSC ERROR: #5 PCSetUp() line 983 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [35]PETSC ERROR: #6 KSPSetUp() line 332 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [35]PETSC ERROR: #7 KSPSolve() line 546 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [35]PETSC ERROR: #8 PCApply_BJacobi_Singleblock() line 670 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/bjacobi/bjacobi.c [35]PETSC ERROR: #9 PCApply() line 483 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [35]PETSC ERROR: #10 KSP_PCApply() line 242 in /home/wtay/Codes/petsc-3.6.3/include/petsc/private/kspimpl.h [35]PETSC ERROR: #11 KSPInitialResidual() line 63 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itres.c [35]PETSC ERROR: #12 KSPSolve_BCGS() line 50 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/impls/bcgs/bcgs.c [35]PETSC ERROR: #13 KSPSolve() line 604 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c --------------------- Error Message -------------------------------------------------------------- [46]PETSC ERROR: Out of memory. This could be due to allocating [46]PETSC ERROR: too large an object or bleeding by not properly [46]PETSC ERROR: destroying unneeded objects. [46]PETSC ERROR: Memory allocated 0 Memory used by process 3510972416 [46]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. [46]PETSC ERROR: Memory requested 34905645 [46]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [46]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [46]PETSC ERROR: ./a.out on a petsc-3.6.3_static_rel named n12-03 by wtay Sun Feb 21 08:24:37 2016 [46]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1 [46]PETSC ERROR: #1 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [46]PETSC ERROR: #2 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [46]PETSC ERROR: #3 MatLUFactorNumeric() line 2946 in /home/wtay/Codes/petsc-3.6.3/src/mat/interface/matrix.c [46]PETSC ERROR: #4 PCSetUp_ILU() line 233 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/factor/ilu/ilu.c [46]PETSC ERROR: #5 PCSetUp() line 983 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [19]PETSC ERROR: ./a.out on a petsc-3.6.3_static_rel named n12-03 by wtay Sun Feb 21 08:24:37 2016 [19]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1 [19]PETSC ERROR: #1 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [34]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [34]PETSC ERROR: Out of memory. This could be due to allocating [34]PETSC ERROR: too large an object or bleeding by not properly [34]PETSC ERROR: destroying unneeded objects. [19]PETSC ERROR: #2 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [19]PETSC ERROR: #3 MatLUFactorNumeric() line 2946 in /home/wtay/Codes/petsc-3.6.3/src/mat/interface/matrix.c [19]PETSC ERROR: #4 PCSetUp_ILU() line 233 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/factor/ilu/ilu.c [19]PETSC ERROR: #5 PCSetUp() line 983 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [19]PETSC ERROR: #6 KSPSetUp() line 332 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [19]PETSC ERROR: #7 KSPSolve() line 546 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [19]PETSC ERROR: #8 PCApply_BJacobi_Singleblock() line 670 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/bjacobi/bjacobi.c [19]PETSC ERROR: #9 PCApply() line 483 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [19]PETSC ERROR: #10 KSP_PCApply() line 242 in /home/wtay/Codes/petsc-3.6.3/include/petsc/private/kspimpl.h [19]PETSC ERROR: #11 KSPInitialResidual() line 63 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itres.c [19]PETSC ERROR: #12 KSPSolve_BCGS() line 50 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/impls/bcgs/bcgs.c [19]PETSC ERROR: #13 KSPSolve() line 604 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [32]PETSC ERROR: Memory allocated 0 Memory used by process 3514494976 [32]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. [32]PETSC ERROR: Memory requested 34905645 [32]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [32]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [32]PETSC ERROR: ./a.out on a petsc-3.6.3_static_rel named n12-03 by wtay Sun Feb 21 08:24:37 2016 [32]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1 [32]PETSC ERROR: #1 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [32]PETSC ERROR: #2 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [32]PETSC ERROR: #3 MatLUFactorNumeric() line 2946 in /home/wtay/Codes/petsc-3.6.3/src/mat/interface/matrix.c [32]PETSC ERROR: #4 PCSetUp_ILU() line 233 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/factor/ilu/ilu.c [32]PETSC ERROR: #5 PCSetUp() line 983 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [32]PETSC ERROR: #6 KSPSetUp() line 332 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [32]PETSC ERROR: #7 KSPSolve() line 546 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [32]PETSC ERROR: #8 PCApply_BJacobi_Singleblock() line 670 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/bjacobi/bjacobi.c [32]PETSC ERROR: #9 PCApply() line 483 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [32]PETSC ERROR: #10 KSP_PCApply() line 242 in /home/wtay/Codes/petsc-3.6.3/include/petsc/private/kspimpl.h [32]PETSC ERROR: #11 KSPInitialResidual() line 63 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itres.c [32]PETSC ERROR: #12 KSPSolve_BCGS() line 50 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/impls/bcgs/bcgs.c [32]PETSC ERROR: #13 KSPSolve() line 604 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [34]PETSC ERROR: Memory allocated 0 Memory used by process 3541282816 [34]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. [34]PETSC ERROR: Memory requested 34905645 [34]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [34]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [34]PETSC ERROR: ./a.out on a petsc-3.6.3_static_rel named n12-03 by wtay Sun Feb 21 08:24:37 2016 [34]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1 [34]PETSC ERROR: #1 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [34]PETSC ERROR: #2 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [34]PETSC ERROR: #3 MatLUFactorNumeric() line 2946 in /home/wtay/Codes/petsc-3.6.3/src/mat/interface/matrix.c [34]PETSC ERROR: #4 PCSetUp_ILU() line 233 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/factor/ilu/ilu.c [34]PETSC ERROR: #5 PCSetUp() line 983 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [34]PETSC ERROR: #6 KSPSetUp() line 332 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [34]PETSC ERROR: #7 KSPSolve() line 546 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [34]PETSC ERROR: #8 PCApply_BJacobi_Singleblock() line 670 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/bjacobi/bjacobi.c [34]PETSC ERROR: #9 PCApply() line 483 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [34]PETSC ERROR: #10 KSP_PCApply() line 242 in /home/wtay/Codes/petsc-3.6.3/include/petsc/private/kspimpl.h [34]PETSC ERROR: #11 KSPInitialResidual() line 63 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itres.c [34]PETSC ERROR: #12 KSPSolve_BCGS() line 50 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/impls/bcgs/bcgs.c [34]PETSC ERROR: #13 KSPSolve() line 604 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [46]PETSC ERROR: [33]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [33]PETSC ERROR: Out of memory. This could be due to allocating [33]PETSC ERROR: too large an object or bleeding by not properly [33]PETSC ERROR: destroying unneeded objects. #6 KSPSetUp() line 332 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [33]PETSC ERROR: Memory allocated 0 Memory used by process 3514351616 [33]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. [33]PETSC ERROR: Memory requested 34905645 [33]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [33]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [33]PETSC ERROR: ./a.out on a petsc-3.6.3_static_rel named n12-03 by wtay Sun Feb 21 08:24:37 2016 [33]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1 [33]PETSC ERROR: #1 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [33]PETSC ERROR: #2 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [33]PETSC ERROR: #3 MatLUFactorNumeric() line 2946 in /home/wtay/Codes/petsc-3.6.3/src/mat/interface/matrix.c [33]PETSC ERROR: #4 PCSetUp_ILU() line 233 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/factor/ilu/ilu.c [33]PETSC ERROR: #5 PCSetUp() line 983 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [33]PETSC ERROR: #6 KSPSetUp() line 332 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [33]PETSC ERROR: #7 KSPSolve() line 546 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [33]PETSC ERROR: #8 PCApply_BJacobi_Singleblock() line 670 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/bjacobi/bjacobi.c [33]PETSC ERROR: #9 PCApply() line 483 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [33]PETSC ERROR: #10 KSP_PCApply() line 242 in /home/wtay/Codes/petsc-3.6.3/include/petsc/private/kspimpl.h [33]PETSC ERROR: #11 KSPInitialResidual() line 63 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itres.c [33]PETSC ERROR: #12 KSPSolve_BCGS() line 50 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/impls/bcgs/bcgs.c [33]PETSC ERROR: #13 KSPSolve() line 604 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [46]PETSC ERROR: #7 KSPSolve() line 546 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [46]PETSC ERROR: #8 PCApply_BJacobi_Singleblock() line 670 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/bjacobi/bjacobi.c [46]PETSC ERROR: #9 PCApply() line 483 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [46]PETSC ERROR: #10 KSP_PCApply() line 242 in /home/wtay/Codes/petsc-3.6.3/include/petsc/private/kspimpl.h [46]PETSC ERROR: #11 KSPInitialResidual() line 63 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itres.c [46]PETSC ERROR: #12 KSPSolve_BCGS() line 50 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/impls/bcgs/bcgs.c [46]PETSC ERROR: #13 KSPSolve() line 604 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [16]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [16]PETSC ERROR: Out of memory. This could be due to allocating [12]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [12]PETSC ERROR: Out of memory. This could be due to allocating [12]PETSC ERROR: too large an object or bleeding by not properly [12]PETSC ERROR: destroying unneeded objects. [12]PETSC ERROR: Memory allocated 0 Memory used by process 3533832192 [12]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. [12]PETSC ERROR: Memory requested 4363204 [12]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [12]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [12]PETSC ERROR: ./a.out on a petsc-3.6.3_static_rel named n12-03 by wtay Sun Feb 21 08:24:37 2016 [12]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1 [12]PETSC ERROR: #1 MatLUFactorNumeric_SeqAIJ_Inode() line 1228 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [12]PETSC ERROR: #2 MatLUFactorNumeric_SeqAIJ_Inode() line 1228 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [12]PETSC ERROR: #3 MatLUFactorNumeric() line 2946 in /home/wtay/Codes/petsc-3.6.3/src/mat/interface/matrix.c [12]PETSC ERROR: #4 PCSetUp_ILU() line 233 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/factor/ilu/ilu.c [12]PETSC ERROR: #5 PCSetUp() line 983 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [12]PETSC ERROR: #6 KSPSetUp() line 332 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [12]PETSC ERROR: #7 KSPSolve() line 546 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [12]PETSC ERROR: #8 PCApply_BJacobi_Singleblock() line 670 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/bjacobi/bjacobi.c [12]PETSC ERROR: #9 PCApply() line 483 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [12]PETSC ERROR: #10 KSP_PCApply() line 242 in /home/wtay/Codes/petsc-3.6.3/include/petsc/private/kspimpl.h [12]PETSC ERROR: #11 KSPInitialResidual() line 63 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itres.c [12]PETSC ERROR: #12 KSPSolve_BCGS() line 50 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/impls/bcgs/bcgs.c [12]PETSC ERROR: #13 KSPSolve() line 604 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [16]PETSC ERROR: too large an object or bleeding by not properly [16]PETSC ERROR: destroying unneeded objects. [16]PETSC ERROR: Memory allocated 0 Memory used by process 3527131136 [16]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. [16]PETSC ERROR: Memory requested 4363204 [16]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [16]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [16]PETSC ERROR: ./a.out on a petsc-3.6.3_static_rel named n12-03 by wtay Sun Feb 21 08:24:37 2016 [16]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1 [16]PETSC ERROR: #1 MatLUFactorNumeric_SeqAIJ_Inode() line 1228 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [16]PETSC ERROR: #2 MatLUFactorNumeric_SeqAIJ_Inode() line 1228 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [16]PETSC ERROR: #3 MatLUFactorNumeric() line 2946 in /home/wtay/Codes/petsc-3.6.3/src/mat/interface/matrix.c [16]PETSC ERROR: #4 PCSetUp_ILU() line 233 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/factor/ilu/ilu.c [16]PETSC ERROR: #5 PCSetUp() line 983 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [16]PETSC ERROR: #6 KSPSetUp() line 332 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [16]PETSC ERROR: #7 KSPSolve() line 546 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [16]PETSC ERROR: #8 PCApply_BJacobi_Singleblock() line 670 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/bjacobi/bjacobi.c [16]PETSC ERROR: #9 PCApply() line 483 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [16]PETSC ERROR: #10 KSP_PCApply() line 242 in /home/wtay/Codes/petsc-3.6.3/include/petsc/private/kspimpl.h [16]PETSC ERROR: #11 KSPInitialResidual() line 63 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itres.c [16]PETSC ERROR: #12 KSPSolve_BCGS() line 50 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/impls/bcgs/bcgs.c [16]PETSC ERROR: #13 KSPSolve() line 604 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c mpiexec: abort is already in progress...hit ctrl-c again to forcibly terminate -------------- next part -------------- 0.000000000000000E+000 0.710000000000000 0.000000000000000E+000 90.0000000000000 0.000000000000000E+000 -0.350000000000000 1.00000000000000 0.200000000000000 0 -400000 AB,AA,BB,CC -3.13300005381461 3.47950007027248 3.69050008029444 3.79850008542417 size_x,size_y,size_z 202 300 577 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 0 1 50 1 37 1 373700 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 1 51 100 1 37 373701 747400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 2 101 150 1 37 747401 1121100 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 3 151 200 1 37 1121101 1494800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 4 201 250 1 37 1494801 1868500 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 5 251 300 1 37 1868501 2242200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 6 1 50 38 73 2242201 2605800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 7 51 100 38 73 2605801 2969400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 8 101 150 38 73 2969401 3333000 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 9 151 200 38 73 3333001 3696600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 10 201 250 38 73 3696601 4060200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 11 251 300 38 73 4060201 4423800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 12 1 50 74 109 4423801 4787400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 13 51 100 74 109 4787401 5151000 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 14 101 150 74 109 5151001 5514600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 15 151 200 74 109 5514601 5878200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 16 201 250 74 109 5878201 6241800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 17 251 300 74 109 6241801 6605400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 18 1 50 110 145 6605401 6969000 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 19 51 100 110 145 6969001 7332600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 20 101 150 110 145 7332601 7696200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 21 151 200 110 145 7696201 8059800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 22 201 250 110 145 8059801 8423400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 23 251 300 110 145 8423401 8787000 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 24 1 50 146 181 8787001 9150600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 25 51 100 146 181 9150601 9514200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 26 101 150 146 181 9514201 9877800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 27 151 200 146 181 9877801 10241400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 28 201 250 146 181 10241401 10605000 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 29 251 300 146 181 10605001 10968600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 30 1 50 182 217 10968601 11332200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 31 51 100 182 217 11332201 11695800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 32 101 150 182 217 11695801 12059400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 33 151 200 182 217 12059401 12423000 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 34 201 250 182 217 12423001 12786600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 35 251 300 182 217 12786601 13150200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 36 1 50 218 253 13150201 13513800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 37 51 100 218 253 13513801 13877400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 38 101 150 218 253 13877401 14241000 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 39 151 200 218 253 14241001 14604600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 40 201 250 218 253 14604601 14968200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 41 251 300 218 253 14968201 15331800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 42 1 50 254 289 15331801 15695400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 43 51 100 254 289 15695401 16059000 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 44 101 150 254 289 16059001 16422600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 45 151 200 254 289 16422601 16786200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 46 201 250 254 289 16786201 17149800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 47 251 300 254 289 17149801 17513400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 48 1 50 290 325 17513401 17877000 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 49 51 100 290 325 17877001 18240600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 50 101 150 290 325 18240601 18604200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 51 151 200 290 325 18604201 18967800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 52 201 250 290 325 18967801 19331400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 53 251 300 290 325 19331401 19695000 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 54 1 50 326 361 19695001 20058600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 55 51 100 326 361 20058601 20422200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 56 101 150 326 361 20422201 20785800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 57 151 200 326 361 20785801 21149400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 58 201 250 326 361 21149401 21513000 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 59 251 300 326 361 21513001 21876600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 60 1 50 362 397 21876601 22240200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 61 51 100 362 397 22240201 22603800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 62 101 150 362 397 22603801 22967400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 63 151 200 362 397 22967401 23331000 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 64 201 250 362 397 23331001 23694600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 65 251 300 362 397 23694601 24058200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 66 1 50 398 433 24058201 24421800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 67 51 100 398 433 24421801 24785400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 68 101 150 398 433 24785401 25149000 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 69 151 200 398 433 25149001 25512600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 70 201 250 398 433 25512601 25876200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 71 251 300 398 433 25876201 26239800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 72 1 50 434 469 26239801 26603400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 73 51 100 434 469 26603401 26967000 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 74 101 150 434 469 26967001 27330600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 75 151 200 434 469 27330601 27694200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 76 201 250 434 469 27694201 28057800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 77 251 300 434 469 28057801 28421400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 78 1 50 470 505 28421401 28785000 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 79 51 100 470 505 28785001 29148600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 80 101 150 470 505 29148601 29512200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 81 151 200 470 505 29512201 29875800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 82 201 250 470 505 29875801 30239400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 83 251 300 470 505 30239401 30603000 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 84 1 50 506 541 30603001 30966600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 85 51 100 506 541 30966601 31330200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 86 101 150 506 541 31330201 31693800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 87 151 200 506 541 31693801 32057400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 88 201 250 506 541 32057401 32421000 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 89 251 300 506 541 32421001 32784600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 90 1 50 542 577 32784601 33148200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 91 51 100 542 577 33148201 33511800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 92 101 150 542 577 33511801 33875400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 93 151 200 542 577 33875401 34239000 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 94 201 250 542 577 34239001 34602600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 95 251 300 542 577 34602601 34966200 body_cg_ini 9.999999999255602E-007 1.710991388089775E-003 10.2254852004451 672 surfaces with correct vertex ordering Warning - length difference between element and cell max_element_length,min_element_length,min_delta 0.297282883542864 5.747832995728071E-002 1.800000000000000E-002 maximum ngh_surfaces and ngh_vertics are 1 1 minimum ngh_surfaces and ngh_vertics are 1 1 body_cg_ini -9.199712417665717E-004 0.891121393810939 6.75773429502958 1776 surfaces with correct vertex ordering Warning - length difference between element and cell max_element_length,min_element_length,min_delta 0.294864814733624 1.648655999999882E-002 1.800000000000000E-002 maximum ngh_surfaces and ngh_vertics are 1 2 minimum ngh_surfaces and ngh_vertics are 1 1 min IIB_cell_no 0 max IIB_cell_no 1237 final initial IIB_cell_no 61850 min I_cell_no 0 max I_cell_no 4428 final initial I_cell_no 221400 size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u) 61850 221400 61850 221400 min IIB_cell_no 0 max IIB_cell_no 1350 final initial IIB_cell_no 67500 min I_cell_no 0 max I_cell_no 22199 final initial I_cell_no 1109950 size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u) 67500 1109950 67500 1109950 IIB_I_cell_no_uvw_total1 18956 18602 18748 57477 49290 49710 IIB_I_cell_no_uvw_total2 37851 37876 38095 500842 504527 506410 1500 0.00021825 9.98988642 10.43276011 8.24268002 0.64333010E+06 0.41880252E+04 0.34216012E+08 1800 0.00023513 10.68688767 9.49082300 8.14022395 0.47192748E+06 0.37796673E+04 0.34397558E+08 2100 0.00023612 11.14050288 9.53122678 8.10449896 0.24054404E+06 0.22516421E+04 0.34588837E+08 -------------------------------------------------------------------------- ORTE has lost communication with its daemon located on node: hostname: n12-09 This is usually due to either a failure of the TCP network connection to the node, or possibly an internal failure of the daemon itself. We cannot recover from this failure, and therefore will terminate the job. -------------------------------------------------------------------------- From knepley at gmail.com Tue Feb 23 20:28:09 2016 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 23 Feb 2016 20:28:09 -0600 Subject: [petsc-users] Error - Out of memory. This could be due to allocating too large an object or bleeding by not properly ... In-Reply-To: <56CD0C7C.5060101@gmail.com> References: <56CD0C7C.5060101@gmail.com> Message-ID: On Tue, Feb 23, 2016 at 7:50 PM, TAY wee-beng wrote: > Hi, > > I got this error (also attached, full) when running my code. It happens > after a few thousand time steps. > > The strange thing is that for 2 different clusters, it stops at 2 > different time steps. > > I wonder if it's related to DM since this happens after I added DM into my > code. > > In this case, how can I find out the error? I'm thinking valgrind may take > very long and gives too many false errors. It is very easy to find leaks. You just run a few steps with -malloc_dump and see what is left over. Matt > > -- > Thank you > > Yours sincerely, > > TAY wee-beng > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.knezevic at akselos.com Tue Feb 23 21:35:18 2016 From: david.knezevic at akselos.com (David Knezevic) Date: Tue, 23 Feb 2016 22:35:18 -0500 Subject: [petsc-users] MatPtAP Message-ID: I'm using MatPtAP, which works well for me, but in some examples I've tested the PtAP calculation dominates the overall solve time (e.g. see attached -log_summary output). In my case, A is a stiffness matrix, and P is the identity matrix except for a small number of columns (e.g. about 10 or so) which are dense. In this situation, I was wondering if there is a more efficient way to proceed than using MatPtAP? For example, would it be noticeably faster to calculate P^T A P directly using MatMults for the dense columns, rather than using MatPtAP? Thanks! David -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- /home/dknez/akselos-dev/scrbe/build/bin/fe_solver-opt_real on a arch-linux2-c-opt named david-Lenovo with 1 processor, by dknez Tue Feb 23 14:17:28 2016 Using Petsc Release Version 3.6.1, Jul, 22, 2015 Max Max/Min Avg Total Time (sec): 1.214e+02 1.00000 1.214e+02 Objects: 4.150e+02 1.00000 4.150e+02 Flops: 1.294e+09 1.00000 1.294e+09 1.294e+09 Flops/sec: 1.066e+07 1.00000 1.066e+07 1.066e+07 MPI Messages: 0.000e+00 0.00000 0.000e+00 0.000e+00 MPI Message Lengths: 0.000e+00 0.00000 0.000e+00 0.000e+00 MPI Reductions: 0.000e+00 0.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 1.2142e+02 100.0% 1.2945e+09 100.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage VecNorm 15 1.0 2.5439e-04 1.0 1.12e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 4384 VecCopy 102 1.0 2.6283e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 315 1.0 3.1855e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 104 1.0 3.3436e-03 1.0 1.25e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 374 VecWAXPY 7 1.0 4.2009e-04 1.0 2.60e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 620 VecAssemblyBegin 368 1.0 1.2109e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd 368 1.0 2.5558e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 106 1.0 8.7023e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecReduceArith 21 1.0 5.3144e-04 1.0 1.56e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 2940 VecReduceComm 7 1.0 1.1921e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatMultAdd 41 1.0 3.2815e-02 1.0 5.99e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 5 0 0 0 0 5 0 0 0 1824 MatMultTrAdd 9 1.0 6.2582e-03 1.0 8.70e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 1390 MatSolve 19 1.0 7.3646e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 MatCholFctrSym 2 1.0 8.0401e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 MatCholFctrNum 8 1.0 1.3437e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 11 0 0 0 0 11 0 0 0 0 0 MatAssemblyBegin 117 1.0 2.6464e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 117 1.0 1.7539e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetValues 32 1.0 3.2902e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 2 1.0 9.5367e-07 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 2 1.0 7.9417e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatZeroEntries 48 1.0 5.5495e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAXPY 7 1.0 2.9074e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatPtAP 8 1.0 8.5691e+01 1.0 1.22e+09 1.0 0.0e+00 0.0e+00 0.0e+00 71 94 0 0 0 71 94 0 0 0 14 MatPtAPSymbolic 1 1.0 2.2405e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 18 0 0 0 0 0 MatPtAPNumeric 8 1.0 6.3285e+01 1.0 1.22e+09 1.0 0.0e+00 0.0e+00 0.0e+00 52 94 0 0 0 52 94 0 0 0 19 MatGetSymTrans 1 1.0 1.6050e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSetUp 19 1.0 8.3447e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 19 1.0 1.4982e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 12 0 0 0 0 12 0 0 0 0 0 PCSetUp 8 1.0 1.4245e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 12 0 0 0 0 12 0 0 0 0 0 PCApply 19 1.0 7.3649e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 SNESSolve 2 1.0 6.9693e+01 1.0 1.14e+09 1.0 0.0e+00 0.0e+00 0.0e+00 57 88 0 0 0 57 88 0 0 0 16 SNESFunctionEval 9 1.0 1.1762e+00 1.0 6.20e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 5 0 0 0 1 5 0 0 0 53 SNESJacobianEval 7 1.0 5.5601e+01 1.0 1.08e+09 1.0 0.0e+00 0.0e+00 0.0e+00 46 83 0 0 0 46 83 0 0 0 19 SNESLineSearch 7 1.0 1.6607e-01 1.0 5.44e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 4 0 0 0 0 4 0 0 0 328 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Vector 254 254 29415312 0 Vector Scatter 41 41 26896 0 Index Set 65 65 744616 0 IS L to G Mapping 16 16 775904 0 Matrix 22 22 149945168 0 Krylov Solver 2 2 2448 0 DMKSP interface 1 1 648 0 Preconditioner 2 2 2136 0 SNES 1 1 1332 0 SNESLineSearch 1 1 856 0 DMSNES 1 1 664 0 Distributed Mesh 2 2 8992 0 Star Forest Bipartite Graph 4 4 3200 0 Discrete System 2 2 1696 0 Viewer 1 0 0 0 ======================================================================================================================== Average time to get PetscTime(): 9.53674e-08 #PETSc Option Table entries: -JSON_INIT /home/dknez/akselos-dev/data/instance/workers/fe_solver/810b60d7534448ce8ef67eb6a5e2267e/json_init.json -JSON_INPUT /home/dknez/akselos-dev/data/instance/workers/fe_solver/810b60d7534448ce8ef67eb6a5e2267e/json_input.json -log_summary #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: --with-shared-libraries=1 --with-debugging=0 --download-suitesparse --download-parmetis --download-blacs --with-blas-lapack-dir=/opt/intel/system_studio_2015.2.050/mkl --CXXFLAGS=-Wl,--no-as-needed --download-scalapack --download-mumps --download-metis --download-superlu_dist --prefix=/home/dknez/software/libmesh_install/opt_real/petsc --download-hypre --download-ml ----------------------------------------- Libraries compiled on Thu Aug 13 16:37:37 2015 on david-Lenovo Machine characteristics: Linux-3.13.0-61-generic-x86_64-with-Ubuntu-14.04-trusty Using PETSc directory: /home/dknez/software/petsc-3.6.1 Using PETSc arch: arch-linux2-c-opt ----------------------------------------- Using C compiler: mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: mpif90 -fPIC -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/home/dknez/software/petsc-3.6.1/arch-linux2-c-opt/include -I/home/dknez/software/petsc-3.6.1/include -I/home/dknez/software/petsc-3.6.1/include -I/home/dknez/software/petsc-3.6.1/arch-linux2-c-opt/include -I/home/dknez/software/libmesh_install/opt_real/petsc/include -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi ----------------------------------------- Using C linker: mpicc Using Fortran linker: mpif90 Using libraries: -Wl,-rpath,/home/dknez/software/petsc-3.6.1/arch-linux2-c-opt/lib -L/home/dknez/software/petsc-3.6.1/arch-linux2-c-opt/lib -lpetsc -Wl,-rpath,/home/dknez/software/libmesh_install/opt_real/petsc/lib -L/home/dknez/software/libmesh_install/opt_real/petsc/lib -lsuperlu_dist_4.0 -lHYPRE -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lmpi_cxx -lstdc++ -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lumfpack -lklu -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd -lsuitesparseconfig -lml -lmpi_cxx -lstdc++ -Wl,-rpath,/opt/intel/system_studio_2015.2.050/mkl/lib/intel64 -L/opt/intel/system_studio_2015.2.050/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -lparmetis -lmetis -lm -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -lmpi -lhwloc -lgcc_s -lpthread -ldl ----------------------------------------- From bsmith at mcs.anl.gov Tue Feb 23 21:40:07 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 23 Feb 2016 21:40:07 -0600 Subject: [petsc-users] MatPtAP In-Reply-To: References: Message-ID: <05BF9D61-23FF-4DC5-A6A6-A60B43B1E036@mcs.anl.gov> A custom MatPtAP would almost surely pay off, but it is only an optimization so you need to ask if this computation is your main "blocker" to getting more "science" done. Can you send a picture of the exact structure of P? Barry > On Feb 23, 2016, at 9:35 PM, David Knezevic wrote: > > I'm using MatPtAP, which works well for me, but in some examples I've tested the PtAP calculation dominates the overall solve time (e.g. see attached -log_summary output). > > In my case, A is a stiffness matrix, and P is the identity matrix except for a small number of columns (e.g. about 10 or so) which are dense. > > In this situation, I was wondering if there is a more efficient way to proceed than using MatPtAP? For example, would it be noticeably faster to calculate P^T A P directly using MatMults for the dense columns, rather than using MatPtAP? > > Thanks! > David > > > From david.knezevic at akselos.com Tue Feb 23 22:08:41 2016 From: david.knezevic at akselos.com (David Knezevic) Date: Tue, 23 Feb 2016 23:08:41 -0500 Subject: [petsc-users] MatPtAP In-Reply-To: <05BF9D61-23FF-4DC5-A6A6-A60B43B1E036@mcs.anl.gov> References: <05BF9D61-23FF-4DC5-A6A6-A60B43B1E036@mcs.anl.gov> Message-ID: On Tue, Feb 23, 2016 at 10:40 PM, Barry Smith wrote: > > A custom MatPtAP would almost surely pay off, but it is only an > optimization so you need to ask if this computation is your main "blocker" > to getting more "science" done. Can you send a picture of the exact > structure of P? > I've attached a picture of the sparsity pattern of P. There are 0s and 1s on the diagonal, and some dense columns in the first few columns. David > > > On Feb 23, 2016, at 9:35 PM, David Knezevic > wrote: > > > > I'm using MatPtAP, which works well for me, but in some examples I've > tested the PtAP calculation dominates the overall solve time (e.g. see > attached -log_summary output). > > > > In my case, A is a stiffness matrix, and P is the identity matrix except > for a small number of columns (e.g. about 10 or so) which are dense. > > > > In this situation, I was wondering if there is a more efficient way to > proceed than using MatPtAP? For example, would it be noticeably faster to > calculate P^T A P directly using MatMults for the dense columns, rather > than using MatPtAP? > > > > Thanks! > > David > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: P_sparsity.png Type: image/png Size: 22116 bytes Desc: not available URL: From mirzadeh at gmail.com Tue Feb 23 23:35:53 2016 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Wed, 24 Feb 2016 00:35:53 -0500 Subject: [petsc-users] Neumann BC with non-symmetric matrix Message-ID: Dear all, I am dealing with a situation I was hoping to get some suggestions here. Suppose after discretizing a poisson equation with purely neumann (or periodic) bc I end up with a matrix that is *almost* symmetric, i.e. it is symmetric for almost all grid points with the exception of a few points. The correct way of handling this problem is by specifying the nullspace to MatSetNullSpace. However, since the matrix is non-symmetric in general I would need to pass the nullspace of A^T. Now it turns out that if A is *sufficiently close to being symmetric*, I can get away with the constant vector, which is the nullspace of A and not A^T, but obviously this does not always work. Sometimes the KSP converges and in other situations the residual stagnates which is to be expected. Now, here are my questions (sorry if they are too many!): 1) Is there any efficient way of calculating nullspace of A^T in this case? Is SVD the only way? 2) I have tried fixing the solution at an arbitrary point, and while it generally works, for some problems I get numerical artifacts, e.g. slight asymmetry in the solution and/or increased error close to the point where I fix the solution. Is this, more or less, expected as a known artifact? 3) An alternative to 2 is to enforce some global constraint on the solution, e.g. to require that the average be zero. My question here is two-fold: 3-1) Is this generally any better than solution 2, in terms of not messing too much with the condition number of the matrix? 3-2) I don't quite know how to implement this using PETSc. Generally speaking I'd like to solve | A U | | X | | B | | | * | | = | | | U^T 0 | | s | | 0 | where U is a constant vector (of ones) and s is effectively a Lagrange multiplier. I suspect I need to use MatCreateSchurComplement and pass that to the KSP? Or do I need create my own matrix type from scratch through MatCreateShell? Any help is appreciated! Thanks, Mohammad -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Feb 23 23:52:16 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 23 Feb 2016 23:52:16 -0600 Subject: [petsc-users] Neumann BC with non-symmetric matrix In-Reply-To: References: Message-ID: > On Feb 23, 2016, at 11:35 PM, Mohammad Mirzadeh wrote: > > Dear all, > > I am dealing with a situation I was hoping to get some suggestions here. Suppose after discretizing a poisson equation with purely neumann (or periodic) bc I end up with a matrix that is *almost* symmetric, i.e. it is symmetric for almost all grid points with the exception of a few points. How come it is not purely symmetric? The usual finite elements with pure Neumann or periodic bc will give a completely symmetric matrix. Barry > > The correct way of handling this problem is by specifying the nullspace to MatSetNullSpace. However, since the matrix is non-symmetric in general I would need to pass the nullspace of A^T. Now it turns out that if A is *sufficiently close to being symmetric*, I can get away with the constant vector, which is the nullspace of A and not A^T, but obviously this does not always work. Sometimes the KSP converges and in other situations the residual stagnates which is to be expected. > > Now, here are my questions (sorry if they are too many!): > > 1) Is there any efficient way of calculating nullspace of A^T in this case? Is SVD the only way? > > 2) I have tried fixing the solution at an arbitrary point, and while it generally works, for some problems I get numerical artifacts, e.g. slight asymmetry in the solution and/or increased error close to the point where I fix the solution. Is this, more or less, expected as a known artifact? > > 3) An alternative to 2 is to enforce some global constraint on the solution, e.g. to require that the average be zero. My question here is two-fold: Requiring the average be zero is exactly the same as providing a null space of the constant function. Saying the average is zero is the same as saying the solution is orthogonal to the constant function. I don't see any reason to introduce the Lagrange multiplier and all its complications inside of just providing the constant null space. > > 3-1) Is this generally any better than solution 2, in terms of not messing too much with the condition number of the matrix? > > 3-2) I don't quite know how to implement this using PETSc. Generally speaking I'd like to solve > > | A U | | X | | B | > | | * | | = | | > | U^T 0 | | s | | 0 | > > > where U is a constant vector (of ones) and s is effectively a Lagrange multiplier. I suspect I need to use MatCreateSchurComplement and pass that to the KSP? Or do I need create my own matrix type from scratch through MatCreateShell? > > Any help is appreciated! > > Thanks, > Mohammad > > From bsmith at mcs.anl.gov Wed Feb 24 00:06:17 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 24 Feb 2016 00:06:17 -0600 Subject: [petsc-users] MatPtAP In-Reply-To: References: <05BF9D61-23FF-4DC5-A6A6-A60B43B1E036@mcs.anl.gov> Message-ID: <30D70E43-29E9-457D-93CA-183DAF7C4A41@mcs.anl.gov> Yes, could definitely be faster with a custom code but a bit of a pain to write. > On Feb 23, 2016, at 10:08 PM, David Knezevic wrote: > > On Tue, Feb 23, 2016 at 10:40 PM, Barry Smith wrote: > > A custom MatPtAP would almost surely pay off, but it is only an optimization so you need to ask if this computation is your main "blocker" to getting more "science" done. Can you send a picture of the exact structure of P? > > I've attached a picture of the sparsity pattern of P. There are 0s and 1s on the diagonal, and some dense columns in the first few columns. > > David > > > > > > > > > > On Feb 23, 2016, at 9:35 PM, David Knezevic wrote: > > > > I'm using MatPtAP, which works well for me, but in some examples I've tested the PtAP calculation dominates the overall solve time (e.g. see attached -log_summary output). > > > > In my case, A is a stiffness matrix, and P is the identity matrix except for a small number of columns (e.g. about 10 or so) which are dense. > > > > In this situation, I was wondering if there is a more efficient way to proceed than using MatPtAP? For example, would it be noticeably faster to calculate P^T A P directly using MatMults for the dense columns, rather than using MatPtAP? > > > > Thanks! > > David > > > > > > > > > From mirzadeh at gmail.com Wed Feb 24 00:07:07 2016 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Wed, 24 Feb 2016 01:07:07 -0500 Subject: [petsc-users] Neumann BC with non-symmetric matrix In-Reply-To: References: Message-ID: Barry, On Wednesday, February 24, 2016, Barry Smith wrote: > > > On Feb 23, 2016, at 11:35 PM, Mohammad Mirzadeh > wrote: > > > > Dear all, > > > > I am dealing with a situation I was hoping to get some suggestions here. > Suppose after discretizing a poisson equation with purely neumann (or > periodic) bc I end up with a matrix that is *almost* symmetric, i.e. it is > symmetric for almost all grid points with the exception of a few points. > > How come it is not purely symmetric? The usual finite elements with pure > Neumann or periodic bc will give a completely symmetric matrix. > > Barry > > So this is a finite difference discretization on adaptive Cartesian grids. It turns out that the discretization is non-symmetric at the corse-fine interface. It's actually not because of the BC itself. > > > > The correct way of handling this problem is by specifying the nullspace > to MatSetNullSpace. However, since the matrix is non-symmetric in general I > would need to pass the nullspace of A^T. Now it turns out that if A is > *sufficiently close to being symmetric*, I can get away with the constant > vector, which is the nullspace of A and not A^T, but obviously this does > not always work. Sometimes the KSP converges and in other situations the > residual stagnates which is to be expected. > > > > Now, here are my questions (sorry if they are too many!): > > > > 1) Is there any efficient way of calculating nullspace of A^T in this > case? Is SVD the only way? > > > > 2) I have tried fixing the solution at an arbitrary point, and while it > generally works, for some problems I get numerical artifacts, e.g. slight > asymmetry in the solution and/or increased error close to the point where I > fix the solution. Is this, more or less, expected as a known artifact? > > > > 3) An alternative to 2 is to enforce some global constraint on the > solution, e.g. to require that the average be zero. My question here is > two-fold: > > Requiring the average be zero is exactly the same as providing a null > space of the constant function. Saying the average is zero is the same as > saying the solution is orthogonal to the constant function. I don't see any > reason to introduce the Lagrange multiplier and all its complications > inside of just providing the constant null space. Is this also true at the discrete level when the matrix is non-symmetric? I have always viewed this as just a constraint that could really be anything. > > > > > 3-1) Is this generally any better than solution 2, in terms of not > messing too much with the condition number of the matrix? > > > > 3-2) I don't quite know how to implement this using PETSc. Generally > speaking I'd like to solve > > > > | A U | | X | | B | > > | | * | | = | | > > | U^T 0 | | s | | 0 | > > > > > > where U is a constant vector (of ones) and s is effectively a Lagrange > multiplier. I suspect I need to use MatCreateSchurComplement and pass that > to the KSP? Or do I need create my own matrix type from scratch through > MatCreateShell? > > > > Any help is appreciated! > > > > Thanks, > > Mohammad > > > > > > -- Sent from Gmail Mobile -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.knezevic at akselos.com Wed Feb 24 00:14:00 2016 From: david.knezevic at akselos.com (David Knezevic) Date: Wed, 24 Feb 2016 01:14:00 -0500 Subject: [petsc-users] MatPtAP In-Reply-To: <30D70E43-29E9-457D-93CA-183DAF7C4A41@mcs.anl.gov> References: <05BF9D61-23FF-4DC5-A6A6-A60B43B1E036@mcs.anl.gov> <30D70E43-29E9-457D-93CA-183DAF7C4A41@mcs.anl.gov> Message-ID: On Wed, Feb 24, 2016 at 1:06 AM, Barry Smith wrote: > > Yes, could definitely be faster with a custom code but a bit of a pain > to write. > OK, thanks for your input! David > > > On Feb 23, 2016, at 10:08 PM, David Knezevic > wrote: > > > > On Tue, Feb 23, 2016 at 10:40 PM, Barry Smith > wrote: > > > > A custom MatPtAP would almost surely pay off, but it is only an > optimization so you need to ask if this computation is your main "blocker" > to getting more "science" done. Can you send a picture of the exact > structure of P? > > > > I've attached a picture of the sparsity pattern of P. There are 0s and > 1s on the diagonal, and some dense columns in the first few columns. > > > > David > > > > > > > > > > > > > > > > > > > On Feb 23, 2016, at 9:35 PM, David Knezevic < > david.knezevic at akselos.com> wrote: > > > > > > I'm using MatPtAP, which works well for me, but in some examples I've > tested the PtAP calculation dominates the overall solve time (e.g. see > attached -log_summary output). > > > > > > In my case, A is a stiffness matrix, and P is the identity matrix > except for a small number of columns (e.g. about 10 or so) which are dense. > > > > > > In this situation, I was wondering if there is a more efficient way to > proceed than using MatPtAP? For example, would it be noticeably faster to > calculate P^T A P directly using MatMults for the dense columns, rather > than using MatPtAP? > > > > > > Thanks! > > > David > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Feb 24 00:20:49 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 24 Feb 2016 00:20:49 -0600 Subject: [petsc-users] Neumann BC with non-symmetric matrix In-Reply-To: References: Message-ID: <715E1F2D-F735-4419-811A-590E2218516F@mcs.anl.gov> > On Feb 24, 2016, at 12:07 AM, Mohammad Mirzadeh wrote: > > > Barry, > On Wednesday, February 24, 2016, Barry Smith wrote: > > > On Feb 23, 2016, at 11:35 PM, Mohammad Mirzadeh wrote: > > > > Dear all, > > > > I am dealing with a situation I was hoping to get some suggestions here. Suppose after discretizing a poisson equation with purely neumann (or periodic) bc I end up with a matrix that is *almost* symmetric, i.e. it is symmetric for almost all grid points with the exception of a few points. > > How come it is not purely symmetric? The usual finite elements with pure Neumann or periodic bc will give a completely symmetric matrix. > > Barry > > > So this is a finite difference discretization on adaptive Cartesian grids. It turns out that the discretization is non-symmetric at the corse-fine interface. It's actually not because of the BC itself. Oh yes, I remember that issue vaguely now. > > > > The correct way of handling this problem is by specifying the nullspace to MatSetNullSpace. However, since the matrix is non-symmetric in general I would need to pass the nullspace of A^T. Now it turns out that if A is *sufficiently close to being symmetric*, I can get away with the constant vector, which is the nullspace of A and not A^T, but obviously this does not always work. Sometimes the KSP converges and in other situations the residual stagnates which is to be expected. > > > > Now, here are my questions (sorry if they are too many!): > > > > 1) Is there any efficient way of calculating nullspace of A^T in this case? Not that I know of. > Is SVD the only way? I think if you make sure you only use right preconditioning such as with -ksp_type gmres -ksp_pc_side right AND you know that the right hand side is automatically in the range of A then the null space of A^T is never needed in the computation. Is your right hand side in the range of A? Barry > > > > 2) I have tried fixing the solution at an arbitrary point, and while it generally works, for some problems I get numerical artifacts, e.g. slight asymmetry in the solution and/or increased error close to the point where I fix the solution. Is this, more or less, expected as a known artifact? Yeah this approach is not good. We never recommend it. > > > > 3) An alternative to 2 is to enforce some global constraint on the solution, e.g. to require that the average be zero. My question here is two-fold: > > Requiring the average be zero is exactly the same as providing a null space of the constant function. Saying the average is zero is the same as saying the solution is orthogonal to the constant function. I don't see any reason to introduce the Lagrange multiplier and all its complications inside of just providing the constant null space. > > Is this also true at the discrete level when the matrix is non-symmetric? I have always viewed this as just a constraint that could really be anything. > > > > > 3-1) Is this generally any better than solution 2, in terms of not messing too much with the condition number of the matrix? > > > > 3-2) I don't quite know how to implement this using PETSc. Generally speaking I'd like to solve > > > > | A U | | X | | B | > > | | * | | = | | > > | U^T 0 | | s | | 0 | > > > > > > where U is a constant vector (of ones) and s is effectively a Lagrange multiplier. I suspect I need to use MatCreateSchurComplement and pass that to the KSP? Or do I need create my own matrix type from scratch through MatCreateShell? > > > > Any help is appreciated! > > > > Thanks, > > Mohammad > > > > > > > > -- > Sent from Gmail Mobile From zonexo at gmail.com Wed Feb 24 01:54:47 2016 From: zonexo at gmail.com (TAY wee-beng) Date: Wed, 24 Feb 2016 15:54:47 +0800 Subject: [petsc-users] Error - Out of memory. This could be due to allocating too large an object or bleeding by not properly ... In-Reply-To: References: <56CD0C7C.5060101@gmail.com> Message-ID: <56CD61C7.3040200@gmail.com> On 24/2/2016 10:28 AM, Matthew Knepley wrote: > On Tue, Feb 23, 2016 at 7:50 PM, TAY wee-beng > wrote: > > Hi, > > I got this error (also attached, full) when running my code. It > happens after a few thousand time steps. > > The strange thing is that for 2 different clusters, it stops at 2 > different time steps. > > I wonder if it's related to DM since this happens after I added DM > into my code. > > In this case, how can I find out the error? I'm thinking valgrind > may take very long and gives too many false errors. > > > It is very easy to find leaks. You just run a few steps with > -malloc_dump and see what is left over. > > Matt Hi Matt, Do you mean running my a.out with the -malloc_dump and stop after a few time steps? What and how should I "see" then? > > -- > Thank you > > Yours sincerely, > > TAY wee-beng > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hgbk2008 at gmail.com Wed Feb 24 03:16:54 2016 From: hgbk2008 at gmail.com (Hoang Giang Bui) Date: Wed, 24 Feb 2016 10:16:54 +0100 Subject: [petsc-users] Petsc reread options file and FIELDSPLIT null size Message-ID: Hello Is it possible to reread the options file since I like to experiment different solver configuration during time stepping? In addition, I got an error with FIELDSPLIT when a split has zero size (That happens in the first step when one field is disabled). Below is the error message on the proc 0: [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 8 FPE: Floating Point Exception,probably divide by zero [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [0]PETSC ERROR: python on a arch-linux2-cxx-opt named bermuda by hbui Wed Feb 24 10:00:34 2016 [0]PETSC ERROR: Configure options --with-shared-libraries --with-debugging=0 --with-pic --with-clanguage=cxx --download-fblas-lapack=yes --download-ptscotch=yes --download-metis=yes --download-parmetis=yes --download-scalapack=yes --download-mumps=yes --download-hypre=yes --download-ml=yes --download-klu=yes --download-pastix=yes --with-mpi-dir=/opt/openmpi-1.10.1 --prefix=/home/hbui/opt/petsc-3.6.3 [0]PETSC ERROR: #1 User provided function() line 0 in unknown file Do you have any comments? Giang -------------- next part -------------- An HTML attachment was scrubbed... URL: From juan at tf.uni-kiel.de Wed Feb 24 06:47:59 2016 From: juan at tf.uni-kiel.de (Julian Andrej) Date: Wed, 24 Feb 2016 13:47:59 +0100 Subject: [petsc-users] Mass matrix with PetscFE Message-ID: <56CDA67F.6000906@tf.uni-kiel.de> Hi, i'm trying to assemble a mass matrix with the PetscFE/DMPlex interface. I found something in the examples of TAO https://bitbucket.org/petsc/petsc/src/da8116b0e8d067e39fd79740a8a864b0fe207998/src/tao/examples/tutorials/ex3.c?at=master&fileviewer=file-view-default but using the lines DMClone(dm, &dm_mass); DMSetNumFields(dm_mass, 1); DMPlexCopyCoordinates(dm, dm_mass); DMGetDS(dm_mass, &prob_mass); PetscDSSetJacobian(prob_mass, 0, 0, mass_kernel, NULL, NULL, NULL); PetscDSSetDiscretization(prob_mass, 0, (PetscObject) fe); DMPlexSNESComputeJacobianFEM(dm_mass, u, M, M, NULL); DMCreateMatrix(dm_mass, &M); leads to errors in DMPlexSNESComputeJacobianFEM (u is a global vector). I don't can understand the necessary commands until DMPlexSNESComputeJacobianFEM. What does it do and why is it necessary? (especially why does the naming involve SNES?) Is there another/easier/better way to create a mass matrix (the inner product of the function space and the test space)? Regards Julian Andrej From knepley at gmail.com Wed Feb 24 06:53:24 2016 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 24 Feb 2016 06:53:24 -0600 Subject: [petsc-users] Petsc reread options file and FIELDSPLIT null size In-Reply-To: References: Message-ID: On Wed, Feb 24, 2016 at 3:16 AM, Hoang Giang Bui wrote: > Hello > > Is it possible to reread the options file since I like to experiment > different solver configuration during time stepping? > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscOptionsInsertFile.html#PetscOptionsInsertFile > In addition, I got an error with FIELDSPLIT when a split has zero size > (That happens in the first step when one field is disabled). Below is the > error message on the proc 0: > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 8 FPE: Floating Point > Exception,probably divide by zero > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS > X to find memory corruption errors > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and > run > [0]PETSC ERROR: to get more information on the crash. > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Signal received > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > [0]PETSC ERROR: python on a arch-linux2-cxx-opt named bermuda by hbui Wed > Feb 24 10:00:34 2016 > [0]PETSC ERROR: Configure options --with-shared-libraries > --with-debugging=0 --with-pic --with-clanguage=cxx > --download-fblas-lapack=yes --download-ptscotch=yes --download-metis=yes > --download-parmetis=yes --download-scalapack=yes --download-mumps=yes > --download-hypre=yes --download-ml=yes --download-klu=yes > --download-pastix=yes --with-mpi-dir=/opt/openmpi-1.10.1 > --prefix=/home/hbui/opt/petsc-3.6.3 > [0]PETSC ERROR: #1 User provided function() line 0 in unknown file > The SEGV does not occur in PETSc code, but rather your code, as this stack trace shows. Thanks, Matt > > Do you have any comments? > > Giang > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Feb 24 06:58:38 2016 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 24 Feb 2016 06:58:38 -0600 Subject: [petsc-users] Mass matrix with PetscFE In-Reply-To: <56CDA67F.6000906@tf.uni-kiel.de> References: <56CDA67F.6000906@tf.uni-kiel.de> Message-ID: On Wed, Feb 24, 2016 at 6:47 AM, Julian Andrej wrote: > Hi, > > i'm trying to assemble a mass matrix with the PetscFE/DMPlex interface. I > found something in the examples of TAO > > > https://bitbucket.org/petsc/petsc/src/da8116b0e8d067e39fd79740a8a864b0fe207998/src/tao/examples/tutorials/ex3.c?at=master&fileviewer=file-view-default > > but using the lines > > DMClone(dm, &dm_mass); > DMSetNumFields(dm_mass, 1); > DMPlexCopyCoordinates(dm, dm_mass); > DMGetDS(dm_mass, &prob_mass); > PetscDSSetJacobian(prob_mass, 0, 0, mass_kernel, NULL, NULL, NULL); > PetscDSSetDiscretization(prob_mass, 0, (PetscObject) fe); > DMPlexSNESComputeJacobianFEM(dm_mass, u, M, M, NULL); > DMCreateMatrix(dm_mass, &M); > > leads to errors in DMPlexSNESComputeJacobianFEM (u is a global vector). > > I don't can understand the necessary commands until > DMPlexSNESComputeJacobianFEM. What does it do and why is it necessary? > (especially why does the naming involve SNES?) > > Is there another/easier/better way to create a mass matrix (the inner > product of the function space and the test space)? > 1) That example needs updating. First, look at SNES ex12 which is up to date. 2) I assume you are using 3.6. If you use the development version, you can remove DMPlexCopyCoordinates(). 3) You need to create the matrix BEFORE calling the assembly 4) Always always always send the entire error messge Matt > Regards > Julian Andrej > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Feb 24 07:12:45 2016 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 24 Feb 2016 07:12:45 -0600 Subject: [petsc-users] Error - Out of memory. This could be due to allocating too large an object or bleeding by not properly ... In-Reply-To: <56CD61C7.3040200@gmail.com> References: <56CD0C7C.5060101@gmail.com> <56CD61C7.3040200@gmail.com> Message-ID: On Wed, Feb 24, 2016 at 1:54 AM, TAY wee-beng wrote: > > On 24/2/2016 10:28 AM, Matthew Knepley wrote: > > On Tue, Feb 23, 2016 at 7:50 PM, TAY wee-beng wrote: > >> Hi, >> >> I got this error (also attached, full) when running my code. It happens >> after a few thousand time steps. >> >> The strange thing is that for 2 different clusters, it stops at 2 >> different time steps. >> >> I wonder if it's related to DM since this happens after I added DM into >> my code. >> >> In this case, how can I find out the error? I'm thinking valgrind may >> take very long and gives too many false errors. > > > It is very easy to find leaks. You just run a few steps with -malloc_dump > and see what is left over. > > Matt > > Hi Matt, > > Do you mean running my a.out with the -malloc_dump and stop after a few > time steps? > > What and how should I "see" then? > -malloc_dump outputs all unfreed memory to the screen after PetscFinalize(), so you should see the leak. I guess it might be possible to keep creating things that you freed all at once at the end, but that is less likely. Matt > > > >> -- >> Thank you >> >> Yours sincerely, >> >> TAY wee-beng >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From juan at tf.uni-kiel.de Wed Feb 24 07:47:21 2016 From: juan at tf.uni-kiel.de (Julian Andrej) Date: Wed, 24 Feb 2016 14:47:21 +0100 Subject: [petsc-users] Mass matrix with PetscFE In-Reply-To: References: <56CDA67F.6000906@tf.uni-kiel.de> Message-ID: <56CDB469.806@tf.uni-kiel.de> I'm now using the petsc git master branch. I tried adding my code to the ex12 DM dm_mass; PetscDS prob_mass; PetscFE fe; Mat M; PetscFECreateDefault(dm, user.dim, 1, PETSC_TRUE, NULL, -1, &fe); DMClone(dm, &dm_mass); DMGetDS(dm_mass, &prob_mass); PetscDSSetDiscretization(prob_mass, 0, (PetscObject) fe); PetscDSSetJacobian(prob_mass, 0, 0, mass_kernel, NULL, NULL, NULL); DMCreateMatrix(dm_mass, &M); MatSetOptionsPrefix(M, "M_";) and receive the error on running ./exe -interpolate -refinement_limit 0.0125 -petscspace_order 2 -M_mat_view binary WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-M_mat_view value: binary I don't know if the matrix is actually there and assembled or if the option is ommitted because something is wrong. Using MatView(M, PETSC_VIEWER_STDOUT_WORLD); gives me a reasonable output to stdout. But saving the matrix and analysing it in matlab, results in an all zero matrix. PetscViewerBinaryOpen(PETSC_COMM_WORLD, "Mout",FILE_MODE_WRITE, &viewer); MatView(M, viewer); Any hints? On 24.02.2016 13:58, Matthew Knepley wrote: > On Wed, Feb 24, 2016 at 6:47 AM, Julian Andrej > wrote: > > Hi, > > i'm trying to assemble a mass matrix with the PetscFE/DMPlex > interface. I found something in the examples of TAO > > https://bitbucket.org/petsc/petsc/src/da8116b0e8d067e39fd79740a8a864b0fe207998/src/tao/examples/tutorials/ex3.c?at=master&fileviewer=file-view-default > > but using the lines > > DMClone(dm, &dm_mass); > DMSetNumFields(dm_mass, 1); > DMPlexCopyCoordinates(dm, dm_mass); > DMGetDS(dm_mass, &prob_mass); > PetscDSSetJacobian(prob_mass, 0, 0, mass_kernel, NULL, NULL, NULL); > PetscDSSetDiscretization(prob_mass, 0, (PetscObject) fe); > DMPlexSNESComputeJacobianFEM(dm_mass, u, M, M, NULL); > DMCreateMatrix(dm_mass, &M); > > leads to errors in DMPlexSNESComputeJacobianFEM (u is a global vector). > > I don't can understand the necessary commands until > DMPlexSNESComputeJacobianFEM. What does it do and why is it > necessary? (especially why does the naming involve SNES?) > > Is there another/easier/better way to create a mass matrix (the > inner product of the function space and the test space)? > > > 1) That example needs updating. First, look at SNES ex12 which is up to > date. > > 2) I assume you are using 3.6. If you use the development version, you > can remove DMPlexCopyCoordinates(). > > 3) You need to create the matrix BEFORE calling the assembly > > 4) Always always always send the entire error messge > > Matt > > Regards > Julian Andrej > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener From knepley at gmail.com Wed Feb 24 07:56:14 2016 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 24 Feb 2016 07:56:14 -0600 Subject: [petsc-users] Mass matrix with PetscFE In-Reply-To: <56CDB469.806@tf.uni-kiel.de> References: <56CDA67F.6000906@tf.uni-kiel.de> <56CDB469.806@tf.uni-kiel.de> Message-ID: On Wed, Feb 24, 2016 at 7:47 AM, Julian Andrej wrote: > I'm now using the petsc git master branch. > > I tried adding my code to the ex12 > > DM dm_mass; > PetscDS prob_mass; > PetscFE fe; > Mat M; > PetscFECreateDefault(dm, user.dim, 1, PETSC_TRUE, NULL, -1, &fe); > > DMClone(dm, &dm_mass); > DMGetDS(dm_mass, &prob_mass); > PetscDSSetDiscretization(prob_mass, 0, (PetscObject) fe); > PetscDSSetJacobian(prob_mass, 0, 0, mass_kernel, NULL, NULL, NULL); > DMCreateMatrix(dm_mass, &M); > > MatSetOptionsPrefix(M, "M_";) > > and receive the error on running > ./exe -interpolate -refinement_limit 0.0125 -petscspace_order 2 > -M_mat_view binary > > WARNING! There are options you set that were not used! > WARNING! could be spelling mistake, etc! > Option left: name:-M_mat_view value: binary > > I don't know if the matrix is actually there and assembled or if the > option is ommitted because something is wrong. > Its difficult to know when I cannot see the whole code. You can always insert MatViewFromOptions(M, NULL, "-mat_view"); > Using > MatView(M, PETSC_VIEWER_STDOUT_WORLD); > > gives me a reasonable output to stdout. > Good. > But saving the matrix and analysing it in matlab, results in an all zero > matrix. > > PetscViewerBinaryOpen(PETSC_COMM_WORLD, "Mout",FILE_MODE_WRITE, &viewer); > MatView(M, viewer); > I cannot explain this, but it has to be something like you are viewing the matrix before it is actually assembled. Feel free to send the code. It sounds like it is mostly working. Matt > Any hints? > > > On 24.02.2016 13:58, Matthew Knepley wrote: > >> On Wed, Feb 24, 2016 at 6:47 AM, Julian Andrej > > wrote: >> >> Hi, >> >> i'm trying to assemble a mass matrix with the PetscFE/DMPlex >> interface. I found something in the examples of TAO >> >> >> https://bitbucket.org/petsc/petsc/src/da8116b0e8d067e39fd79740a8a864b0fe207998/src/tao/examples/tutorials/ex3.c?at=master&fileviewer=file-view-default >> >> but using the lines >> >> DMClone(dm, &dm_mass); >> DMSetNumFields(dm_mass, 1); >> DMPlexCopyCoordinates(dm, dm_mass); >> DMGetDS(dm_mass, &prob_mass); >> PetscDSSetJacobian(prob_mass, 0, 0, mass_kernel, NULL, NULL, NULL); >> PetscDSSetDiscretization(prob_mass, 0, (PetscObject) fe); >> DMPlexSNESComputeJacobianFEM(dm_mass, u, M, M, NULL); >> DMCreateMatrix(dm_mass, &M); >> >> leads to errors in DMPlexSNESComputeJacobianFEM (u is a global >> vector). >> >> I don't can understand the necessary commands until >> DMPlexSNESComputeJacobianFEM. What does it do and why is it >> necessary? (especially why does the naming involve SNES?) >> >> Is there another/easier/better way to create a mass matrix (the >> inner product of the function space and the test space)? >> >> >> 1) That example needs updating. First, look at SNES ex12 which is up to >> date. >> >> 2) I assume you are using 3.6. If you use the development version, you >> can remove DMPlexCopyCoordinates(). >> >> 3) You need to create the matrix BEFORE calling the assembly >> >> 4) Always always always send the entire error messge >> >> Matt >> >> Regards >> Julian Andrej >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which >> their experiments lead. >> -- Norbert Wiener >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From juan at tf.uni-kiel.de Wed Feb 24 08:03:59 2016 From: juan at tf.uni-kiel.de (Julian Andrej) Date: Wed, 24 Feb 2016 15:03:59 +0100 Subject: [petsc-users] Mass matrix with PetscFE In-Reply-To: References: <56CDA67F.6000906@tf.uni-kiel.de> <56CDB469.806@tf.uni-kiel.de> Message-ID: <56CDB84F.8020309@tf.uni-kiel.de> Thanks Matt, I attached the modified example. the corresponding code (and only changes to ex12) is starting at line 832. It also seems that the mass matrix is of size 169x169 and the stiffness matrix is of dimension 225x225. I'd assume that if i multiply test and trial function i'd get a matrix of same size (if the space/quadrature is the same for the stiffness matrix) On 24.02.2016 14:56, Matthew Knepley wrote: > On Wed, Feb 24, 2016 at 7:47 AM, Julian Andrej > wrote: > > I'm now using the petsc git master branch. > > I tried adding my code to the ex12 > > DM dm_mass; > PetscDS prob_mass; > PetscFE fe; > Mat M; > PetscFECreateDefault(dm, user.dim, 1, PETSC_TRUE, NULL, -1, &fe); > > DMClone(dm, &dm_mass); > DMGetDS(dm_mass, &prob_mass); > PetscDSSetDiscretization(prob_mass, 0, (PetscObject) fe); > PetscDSSetJacobian(prob_mass, 0, 0, mass_kernel, NULL, NULL, NULL); > DMCreateMatrix(dm_mass, &M); > > MatSetOptionsPrefix(M, "M_";) > > and receive the error on running > ./exe -interpolate -refinement_limit 0.0125 -petscspace_order 2 > -M_mat_view binary > > WARNING! There are options you set that were not used! > WARNING! could be spelling mistake, etc! > Option left: name:-M_mat_view value: binary > > I don't know if the matrix is actually there and assembled or if the > option is ommitted because something is wrong. > > > Its difficult to know when I cannot see the whole code. You can always > insert > > MatViewFromOptions(M, NULL, "-mat_view"); > > Using > MatView(M, PETSC_VIEWER_STDOUT_WORLD); > > gives me a reasonable output to stdout. > > > Good. > > But saving the matrix and analysing it in matlab, results in an all > zero matrix. > > PetscViewerBinaryOpen(PETSC_COMM_WORLD, "Mout",FILE_MODE_WRITE, > &viewer); > MatView(M, viewer); > > > I cannot explain this, but it has to be something like you are viewing > the matrix before it is > actually assembled. Feel free to send the code. It sounds like it is > mostly working. > > Matt > > Any hints? > > > On 24.02.2016 13 :58, Matthew Knepley wrote: > > On Wed, Feb 24, 2016 at 6:47 AM, Julian Andrej > > >> wrote: > > Hi, > > i'm trying to assemble a mass matrix with the PetscFE/DMPlex > interface. I found something in the examples of TAO > > https://bitbucket.org/petsc/petsc/src/da8116b0e8d067e39fd79740a8a864b0fe207998/src/tao/examples/tutorials/ex3.c?at=master&fileviewer=file-view-default > > but using the lines > > DMClone(dm, &dm_mass); > DMSetNumFields(dm_mass, 1); > DMPlexCopyCoordinates(dm, dm_mass); > DMGetDS(dm_mass, &prob_mass); > PetscDSSetJacobian(prob_mass, 0, 0, mass_kernel, NULL, > NULL, NULL); > PetscDSSetDiscretization(prob_mass, 0, (PetscObject) fe); > DMPlexSNESComputeJacobianFEM(dm_mass, u, M, M, NULL); > DMCreateMatrix(dm_mass, &M); > > leads to errors in DMPlexSNESComputeJacobianFEM (u is a > global vector). > > I don't can understand the necessary commands until > DMPlexSNESComputeJacobianFEM. What does it do and why is it > necessary? (especially why does the naming involve SNES?) > > Is there another/easier/better way to create a mass matrix (the > inner product of the function space and the test space)? > > > 1) That example needs updating. First, look at SNES ex12 which > is up to > date. > > 2) I assume you are using 3.6. If you use the development > version, you > can remove DMPlexCopyCoordinates(). > > 3) You need to create the matrix BEFORE calling the assembly > > 4) Always always always send the entire error messge > > Matt > > Regards > Julian Andrej > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- A non-text attachment was scrubbed... Name: main.c Type: text/x-csrc Size: 39070 bytes Desc: not available URL: From mirzadeh at gmail.com Wed Feb 24 09:07:16 2016 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Wed, 24 Feb 2016 10:07:16 -0500 Subject: [petsc-users] Neumann BC with non-symmetric matrix In-Reply-To: <715E1F2D-F735-4419-811A-590E2218516F@mcs.anl.gov> References: <715E1F2D-F735-4419-811A-590E2218516F@mcs.anl.gov> Message-ID: Barry, On Wednesday, February 24, 2016, Barry Smith wrote: > > > On Feb 24, 2016, at 12:07 AM, Mohammad Mirzadeh > wrote: > > > > > > Barry, > > On Wednesday, February 24, 2016, Barry Smith > wrote: > > > > > On Feb 23, 2016, at 11:35 PM, Mohammad Mirzadeh > wrote: > > > > > > Dear all, > > > > > > I am dealing with a situation I was hoping to get some suggestions > here. Suppose after discretizing a poisson equation with purely neumann (or > periodic) bc I end up with a matrix that is *almost* symmetric, i.e. it is > symmetric for almost all grid points with the exception of a few points. > > > > How come it is not purely symmetric? The usual finite elements with > pure Neumann or periodic bc will give a completely symmetric matrix. > > > > Barry > > > > > > So this is a finite difference discretization on adaptive Cartesian > grids. It turns out that the discretization is non-symmetric at the > corse-fine interface. It's actually not because of the BC itself. > > Oh yes, I remember that issue vaguely now. > > > > > > > The correct way of handling this problem is by specifying the > nullspace to MatSetNullSpace. However, since the matrix is non-symmetric in > general I would need to pass the nullspace of A^T. Now it turns out that if > A is *sufficiently close to being symmetric*, I can get away with the > constant vector, which is the nullspace of A and not A^T, but obviously > this does not always work. Sometimes the KSP converges and in other > situations the residual stagnates which is to be expected. > > > > > > Now, here are my questions (sorry if they are too many!): > > > > > > 1) Is there any efficient way of calculating nullspace of A^T in this > case? > > Not that I know of. > > > Is SVD the only way? > > I think if you make sure you only use right preconditioning such as with > -ksp_type gmres -ksp_pc_side right AND you know that the right hand side is > automatically in the range of A then the null space of A^T is never needed > in the computation. Is your right hand side in the range of A? > > Barry At the PDE level the compatibility condition is satisfied. However I suspect that at the discrete level the rhs is not not exactly in the range. The reason for my suspicion is that I know for a fact that my discretization is not conservative at the hanging nodes. Thanks for the suggestion. I'll give it a try. Howerver, does GMRES fundamentally behave differently than BiCGSTAB for these systems? I have seen people saying that it can keep the solution in the range if rhs is already in the range but I thought any KSP would do the same? > > > > > > 2) I have tried fixing the solution at an arbitrary point, and while > it generally works, for some problems I get numerical artifacts, e.g. > slight asymmetry in the solution and/or increased error close to the point > where I fix the solution. Is this, more or less, expected as a known > artifact? > > Yeah this approach is not good. We never recommend it. > > > > > > 3) An alternative to 2 is to enforce some global constraint on the > solution, e.g. to require that the average be zero. My question here is > two-fold: > > > > Requiring the average be zero is exactly the same as providing a null > space of the constant function. Saying the average is zero is the same as > saying the solution is orthogonal to the constant function. I don't see any > reason to introduce the Lagrange multiplier and all its complications > inside of just providing the constant null space. > > > > Is this also true at the discrete level when the matrix is > non-symmetric? I have always viewed this as just a constraint that could > really be anything. > > > > > > > > 3-1) Is this generally any better than solution 2, in terms of not > messing too much with the condition number of the matrix? > > > > > > 3-2) I don't quite know how to implement this using PETSc. Generally > speaking I'd like to solve > > > > > > | A U | | X | | B | > > > | | * | | = | | > > > | U^T 0 | | s | | 0 | > > > > > > > > > where U is a constant vector (of ones) and s is effectively a Lagrange > multiplier. I suspect I need to use MatCreateSchurComplement and pass that > to the KSP? Or do I need create my own matrix type from scratch through > MatCreateShell? > > > > > > Any help is appreciated! > > > > > > Thanks, > > > Mohammad > > > > > > > > > > > > > > -- > > Sent from Gmail Mobile > > -- Sent from Gmail Mobile -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Wed Feb 24 09:16:49 2016 From: zonexo at gmail.com (TAY wee-beng) Date: Wed, 24 Feb 2016 23:16:49 +0800 Subject: [petsc-users] Error - Out of memory. This could be due to allocating too large an object or bleeding by not properly ... In-Reply-To: References: <56CD0C7C.5060101@gmail.com> <56CD61C7.3040200@gmail.com> Message-ID: <56CDC961.40004@gmail.com> On 24/2/2016 9:12 PM, Matthew Knepley wrote: > On Wed, Feb 24, 2016 at 1:54 AM, TAY wee-beng > wrote: > > > On 24/2/2016 10:28 AM, Matthew Knepley wrote: >> On Tue, Feb 23, 2016 at 7:50 PM, TAY wee-beng > > wrote: >> >> Hi, >> >> I got this error (also attached, full) when running my code. >> It happens after a few thousand time steps. >> >> The strange thing is that for 2 different clusters, it stops >> at 2 different time steps. >> >> I wonder if it's related to DM since this happens after I >> added DM into my code. >> >> In this case, how can I find out the error? I'm thinking >> valgrind may take very long and gives too many false errors. >> >> >> It is very easy to find leaks. You just run a few steps with >> -malloc_dump and see what is left over. >> >> Matt > Hi Matt, > > Do you mean running my a.out with the -malloc_dump and stop after > a few time steps? > > What and how should I "see" then? > > > -malloc_dump outputs all unfreed memory to the screen after > PetscFinalize(), so you should see the leak. > I guess it might be possible to keep creating things that you freed > all at once at the end, but that is less likely. > > Matt Hi, I got the output. I have zipped it since it's rather big. So it seems to be from DM routines but can you help me where the error is from? Thanks. > >> >> -- >> Thank you >> >> Yours sincerely, >> >> TAY wee-beng >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: log.7z Type: application/octet-stream Size: 166657 bytes Desc: not available URL: From knepley at gmail.com Wed Feb 24 09:18:29 2016 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 24 Feb 2016 09:18:29 -0600 Subject: [petsc-users] Error - Out of memory. This could be due to allocating too large an object or bleeding by not properly ... In-Reply-To: <56CDC961.40004@gmail.com> References: <56CD0C7C.5060101@gmail.com> <56CD61C7.3040200@gmail.com> <56CDC961.40004@gmail.com> Message-ID: On Wed, Feb 24, 2016 at 9:16 AM, TAY wee-beng wrote: > > On 24/2/2016 9:12 PM, Matthew Knepley wrote: > > On Wed, Feb 24, 2016 at 1:54 AM, TAY wee-beng wrote: > >> >> On 24/2/2016 10:28 AM, Matthew Knepley wrote: >> >> On Tue, Feb 23, 2016 at 7:50 PM, TAY wee-beng < >> zonexo at gmail.com> wrote: >> >>> Hi, >>> >>> I got this error (also attached, full) when running my code. It happens >>> after a few thousand time steps. >>> >>> The strange thing is that for 2 different clusters, it stops at 2 >>> different time steps. >>> >>> I wonder if it's related to DM since this happens after I added DM into >>> my code. >>> >>> In this case, how can I find out the error? I'm thinking valgrind may >>> take very long and gives too many false errors. >> >> >> It is very easy to find leaks. You just run a few steps with -malloc_dump >> and see what is left over. >> >> Matt >> >> Hi Matt, >> >> Do you mean running my a.out with the -malloc_dump and stop after a few >> time steps? >> >> What and how should I "see" then? >> > > -malloc_dump outputs all unfreed memory to the screen after > PetscFinalize(), so you should see the leak. > I guess it might be possible to keep creating things that you freed all at > once at the end, but that is less likely. > > Matt > > > Hi, > > I got the output. I have zipped it since it's rather big. So it seems to > be from DM routines but can you help me where the error is from? > Its really hard to tell by looking at it. What I do is remove things until there is no leak, then progressively put thing back in until I have the culprit. Then you can think about what is not destroyed. Matt > Thanks. > > >> >> >>> -- >>> Thank you >>> >>> Yours sincerely, >>> >>> TAY wee-beng >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Wed Feb 24 09:28:19 2016 From: zonexo at gmail.com (TAY wee-beng) Date: Wed, 24 Feb 2016 23:28:19 +0800 Subject: [petsc-users] Error - Out of memory. This could be due to allocating too large an object or bleeding by not properly ... In-Reply-To: References: <56CD0C7C.5060101@gmail.com> <56CD61C7.3040200@gmail.com> <56CDC961.40004@gmail.com> Message-ID: <56CDCC13.4070807@gmail.com> On 24/2/2016 11:18 PM, Matthew Knepley wrote: > On Wed, Feb 24, 2016 at 9:16 AM, TAY wee-beng > wrote: > > > On 24/2/2016 9:12 PM, Matthew Knepley wrote: >> On Wed, Feb 24, 2016 at 1:54 AM, TAY wee-beng > > wrote: >> >> >> On 24/2/2016 10:28 AM, Matthew Knepley wrote: >>> On Tue, Feb 23, 2016 at 7:50 PM, TAY wee-beng >>> > wrote: >>> >>> Hi, >>> >>> I got this error (also attached, full) when running my >>> code. It happens after a few thousand time steps. >>> >>> The strange thing is that for 2 different clusters, it >>> stops at 2 different time steps. >>> >>> I wonder if it's related to DM since this happens after >>> I added DM into my code. >>> >>> In this case, how can I find out the error? I'm thinking >>> valgrind may take very long and gives too many false errors. >>> >>> >>> It is very easy to find leaks. You just run a few steps with >>> -malloc_dump and see what is left over. >>> >>> Matt >> Hi Matt, >> >> Do you mean running my a.out with the -malloc_dump and stop >> after a few time steps? >> >> What and how should I "see" then? >> >> >> -malloc_dump outputs all unfreed memory to the screen after >> PetscFinalize(), so you should see the leak. >> I guess it might be possible to keep creating things that you >> freed all at once at the end, but that is less likely. >> >> Matt > Hi, > > I got the output. I have zipped it since it's rather big. So it > seems to be from DM routines but can you help me where the error > is from? > > > Its really hard to tell by looking at it. What I do is remove things > until there is no leak, then progressively > put thing back in until I have the culprit. Then you can think about > what is not destroyed. > > Matt Ok so let me get this clear. When it shows: [21]Total space allocated 1728961264 bytes [21]1861664 bytes MatCheckCompressedRow() line 60 in /home/wtay/Codes/petsc-3.6.3/src/mat/utils/compressedrow.c [21]16 bytes PetscStrallocpy() line 188 in /home/wtay/Codes/petsc-3.6.3/src/sys/utils/str.c [21]624 bytes ISLocalToGlobalMappingCreate() line 270 in /home/wtay/Codes .... Does it mean that it's simply allocating space ie normal? Or does it show that there's memory leak ie error? If it's error, should I zoom in and debug around this time at this region? Thanks > > Thanks. >> >>> >>> -- >>> Thank you >>> >>> Yours sincerely, >>> >>> TAY wee-beng >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin >>> their experiments is infinitely more interesting than any >>> results to which their experiments lead. >>> -- Norbert Wiener >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Feb 24 09:33:33 2016 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 24 Feb 2016 09:33:33 -0600 Subject: [petsc-users] Error - Out of memory. This could be due to allocating too large an object or bleeding by not properly ... In-Reply-To: <56CDCC13.4070807@gmail.com> References: <56CD0C7C.5060101@gmail.com> <56CD61C7.3040200@gmail.com> <56CDC961.40004@gmail.com> <56CDCC13.4070807@gmail.com> Message-ID: On Wed, Feb 24, 2016 at 9:28 AM, TAY wee-beng wrote: > > On 24/2/2016 11:18 PM, Matthew Knepley wrote: > > On Wed, Feb 24, 2016 at 9:16 AM, TAY wee-beng wrote: > >> >> On 24/2/2016 9:12 PM, Matthew Knepley wrote: >> >> On Wed, Feb 24, 2016 at 1:54 AM, TAY wee-beng < >> zonexo at gmail.com> wrote: >> >>> >>> On 24/2/2016 10:28 AM, Matthew Knepley wrote: >>> >>> On Tue, Feb 23, 2016 at 7:50 PM, TAY wee-beng < >>> zonexo at gmail.com> wrote: >>> >>>> Hi, >>>> >>>> I got this error (also attached, full) when running my code. It happens >>>> after a few thousand time steps. >>>> >>>> The strange thing is that for 2 different clusters, it stops at 2 >>>> different time steps. >>>> >>>> I wonder if it's related to DM since this happens after I added DM into >>>> my code. >>>> >>>> In this case, how can I find out the error? I'm thinking valgrind may >>>> take very long and gives too many false errors. >>> >>> >>> It is very easy to find leaks. You just run a few steps with >>> -malloc_dump and see what is left over. >>> >>> Matt >>> >>> Hi Matt, >>> >>> Do you mean running my a.out with the -malloc_dump and stop after a few >>> time steps? >>> >>> What and how should I "see" then? >>> >> >> -malloc_dump outputs all unfreed memory to the screen after >> PetscFinalize(), so you should see the leak. >> I guess it might be possible to keep creating things that you freed all >> at once at the end, but that is less likely. >> >> Matt >> >> >> Hi, >> >> I got the output. I have zipped it since it's rather big. So it seems to >> be from DM routines but can you help me where the error is from? >> > > Its really hard to tell by looking at it. What I do is remove things until > there is no leak, then progressively > put thing back in until I have the culprit. Then you can think about what > is not destroyed. > > Matt > > Ok so let me get this clear. When it shows: > > [21]Total space allocated 1728961264 bytes > [21]1861664 bytes MatCheckCompressedRow() line 60 in > /home/wtay/Codes/petsc-3.6.3/src/mat/utils/compressedrow.c > [21]16 bytes PetscStrallocpy() line 188 in > /home/wtay/Codes/petsc-3.6.3/src/sys/utils/str.c > [21]624 bytes ISLocalToGlobalMappingCreate() line 270 in /home/wtay/Codes > > .... > > Does it mean that it's simply allocating space ie normal? Or does it show > that there's memory leak ie error? > I gave the wrong option. That dumps everything. Lets just look at the leaks with -malloc_test. Sorry about that, Matt > If it's error, should I zoom in and debug around this time at this region? > > Thanks > > > >> Thanks. >> >> >>> >>> >>>> -- >>>> Thank you >>>> >>>> Yours sincerely, >>>> >>>> TAY wee-beng >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bhatiamanav at gmail.com Wed Feb 24 09:45:09 2016 From: bhatiamanav at gmail.com (Manav Bhatia) Date: Wed, 24 Feb 2016 09:45:09 -0600 Subject: [petsc-users] GAMG and near-null-space when applying Dirichlet Conditions Message-ID: <21F632BF-5E41-48B2-B273-763A7EA97BD4@gmail.com> Hi, I typically apply Dirichlet BCs by modifying the Jacobin and rhs: zero constrained rows of matrix with 1.0 at diagonal, and zero corresponding rows of rhs. While using GAMG, is it still recommended to provide the near-null space (given that the zero-eigenvalues have been removed by specification of DIrichlet BCs)? If that information is still needed, should the vectors be modified in any manner to be consistent with the Dirichlet BCs? Thanks, Manav From zonexo at gmail.com Wed Feb 24 09:57:06 2016 From: zonexo at gmail.com (TAY wee-beng) Date: Wed, 24 Feb 2016 23:57:06 +0800 Subject: [petsc-users] Error - Out of memory. This could be due to allocating too large an object or bleeding by not properly ... In-Reply-To: References: <56CD0C7C.5060101@gmail.com> <56CD61C7.3040200@gmail.com> <56CDC961.40004@gmail.com> <56CDCC13.4070807@gmail.com> Message-ID: <56CDD2D2.3030407@gmail.com> On 24/2/2016 11:33 PM, Matthew Knepley wrote: > On Wed, Feb 24, 2016 at 9:28 AM, TAY wee-beng > wrote: > > > On 24/2/2016 11:18 PM, Matthew Knepley wrote: >> On Wed, Feb 24, 2016 at 9:16 AM, TAY wee-beng > > wrote: >> >> >> On 24/2/2016 9:12 PM, Matthew Knepley wrote: >>> On Wed, Feb 24, 2016 at 1:54 AM, TAY wee-beng >>> > wrote: >>> >>> >>> On 24/2/2016 10:28 AM, Matthew Knepley wrote: >>>> On Tue, Feb 23, 2016 at 7:50 PM, TAY wee-beng >>>> > wrote: >>>> >>>> Hi, >>>> >>>> I got this error (also attached, full) when running >>>> my code. It happens after a few thousand time steps. >>>> >>>> The strange thing is that for 2 different clusters, >>>> it stops at 2 different time steps. >>>> >>>> I wonder if it's related to DM since this happens >>>> after I added DM into my code. >>>> >>>> In this case, how can I find out the error? I'm >>>> thinking valgrind may take very long and gives too >>>> many false errors. >>>> >>>> >>>> It is very easy to find leaks. You just run a few steps >>>> with -malloc_dump and see what is left over. >>>> >>>> Matt >>> Hi Matt, >>> >>> Do you mean running my a.out with the -malloc_dump and >>> stop after a few time steps? >>> >>> What and how should I "see" then? >>> >>> >>> -malloc_dump outputs all unfreed memory to the screen after >>> PetscFinalize(), so you should see the leak. >>> I guess it might be possible to keep creating things that >>> you freed all at once at the end, but that is less likely. >>> >>> Matt >> Hi, >> >> I got the output. I have zipped it since it's rather big. So >> it seems to be from DM routines but can you help me where the >> error is from? >> >> >> Its really hard to tell by looking at it. What I do is remove >> things until there is no leak, then progressively >> put thing back in until I have the culprit. Then you can think >> about what is not destroyed. >> >> Matt > Ok so let me get this clear. When it shows: > > [21]Total space allocated 1728961264 bytes > [21]1861664 bytes MatCheckCompressedRow() line 60 in > /home/wtay/Codes/petsc-3.6.3/src/mat/utils/compressedrow.c > [21]16 bytes PetscStrallocpy() line 188 in > /home/wtay/Codes/petsc-3.6.3/src/sys/utils/str.c > [21]624 bytes ISLocalToGlobalMappingCreate() line 270 in > /home/wtay/Codes > > .... > > Does it mean that it's simply allocating space ie normal? Or does > it show that there's memory leak ie error? > > > I gave the wrong option. That dumps everything. Lets just look at the > leaks with -malloc_test. > > Sorry about that, > > Matt Oh ic .. no prob. I'll try again then. > > If it's error, should I zoom in and debug around this time at this > region? > > Thanks >> >> Thanks. >>> >>>> >>>> -- >>>> Thank you >>>> >>>> Yours sincerely, >>>> >>>> TAY wee-beng >>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they >>>> begin their experiments is infinitely more interesting >>>> than any results to which their experiments lead. >>>> -- Norbert Wiener >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin >>> their experiments is infinitely more interesting than any >>> results to which their experiments lead. >>> -- Norbert Wiener >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Feb 24 09:57:41 2016 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 24 Feb 2016 09:57:41 -0600 Subject: [petsc-users] GAMG and near-null-space when applying Dirichlet Conditions In-Reply-To: <21F632BF-5E41-48B2-B273-763A7EA97BD4@gmail.com> References: <21F632BF-5E41-48B2-B273-763A7EA97BD4@gmail.com> Message-ID: On Wed, Feb 24, 2016 at 9:45 AM, Manav Bhatia wrote: > Hi, > > I typically apply Dirichlet BCs by modifying the Jacobin and rhs: zero > constrained rows of matrix with 1.0 at diagonal, and zero corresponding > rows of rhs. > > While using GAMG, is it still recommended to provide the near-null > space (given that the zero-eigenvalues have been removed by specification > of DIrichlet BCs)? > Yes. > If that information is still needed, should the vectors be modified in > any manner to be consistent with the Dirichlet BCs? > No. You can see that if you take a small piece of the domain, apart from the boundary, it will have this as a null space. Matt > Thanks, > Manav > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mailinglists at xgm.de Wed Feb 24 09:58:52 2016 From: mailinglists at xgm.de (Florian Lindner) Date: Wed, 24 Feb 2016 16:58:52 +0100 Subject: [petsc-users] When to destroy an IS? Message-ID: <728167329.xDZi72PuD8@asaru> Hello, I create a couple of index sets: ISLocalToGlobalMapping _ISmapping; IS ISlocal, ISlocalInv, ISglobal, ISidentity, ISidentityGlobal; ISLocalToGlobalMapping ISidentityMapping; // Create an index set which maps myIndizes to continous chunks of matrix rows. ierr = ISCreateGeneral(PETSC_COMM_WORLD, myIndizes.size(), myIndizes.data(), PETSC_COPY_VALUES, &ISlocal); CHKERRV(ierr); ierr = ISSetPermutation(ISlocal); CHKERRV(ierr); ierr = ISInvertPermutation(ISlocal, myIndizes.size(), &ISlocalInv); CHKERRV(ierr); ierr = ISAllGather(ISlocalInv, &ISglobal); CHKERRV(ierr); // Gather the IS from all processors ierr = ISLocalToGlobalMappingCreateIS(ISglobal, &_ISmapping); CHKERRV(ierr); // Make it a mapping // Create an identity mapping and use that for the rows of matrixA. ierr = ISCreateStride(PETSC_COMM_WORLD, _matrixA.ownerRange().second - _matrixA.ownerRange().first, _matrixA.ownerRange().first, 1, &ISidentity); CHKERRV(ierr); ierr = ISSetIdentity(ISidentity); CHKERRV(ierr); ierr = ISAllGather(ISidentity, &ISidentityGlobal); CHKERRV(ierr); ierr = ISLocalToGlobalMappingCreateIS(ISidentityGlobal, &ISidentityMapping); CHKERRV(ierr); ierr = MatSetLocalToGlobalMapping(_matrixC.matrix, _ISmapping, _ISmapping); CHKERRV(ierr); // Set mapping for rows and cols ierr = MatSetLocalToGlobalMapping(_matrixA.matrix, ISidentityMapping, _ISmapping); CHKERRV(ierr); // Set mapping only for cols, use identity for rows Two questions: 1) Since I only need the _ISmapping again, is there any way to optimize that, i.e. use less temporary variables? 2) Do all IS variables (_ISmapping, ISlocal, ISlocalInv, ISglobal, ISidentity, ISidentityGlobal, ISidentityMapping) need to be destroyed using ISDestroy at the end? Or only the ones that were created using a ISCreate* function (ISlocal, ISidentity)? Thanks, Florian From knepley at gmail.com Wed Feb 24 11:41:09 2016 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 24 Feb 2016 11:41:09 -0600 Subject: [petsc-users] When to destroy an IS? In-Reply-To: <728167329.xDZi72PuD8@asaru> References: <728167329.xDZi72PuD8@asaru> Message-ID: On Wed, Feb 24, 2016 at 9:58 AM, Florian Lindner wrote: > Hello, > > I create a couple of index sets: > > ISLocalToGlobalMapping _ISmapping; > IS ISlocal, ISlocalInv, ISglobal, ISidentity, ISidentityGlobal; > ISLocalToGlobalMapping ISidentityMapping; > > // Create an index set which maps myIndizes to continous chunks of > matrix rows. > ierr = ISCreateGeneral(PETSC_COMM_WORLD, myIndizes.size(), > myIndizes.data(), PETSC_COPY_VALUES, &ISlocal); CHKERRV(ierr); > ierr = ISSetPermutation(ISlocal); CHKERRV(ierr); > ierr = ISInvertPermutation(ISlocal, myIndizes.size(), &ISlocalInv); > CHKERRV(ierr); > ierr = ISAllGather(ISlocalInv, &ISglobal); CHKERRV(ierr); // Gather the > IS from all processors > ierr = ISLocalToGlobalMappingCreateIS(ISglobal, &_ISmapping); > CHKERRV(ierr); // Make it a mapping > > // Create an identity mapping and use that for the rows of matrixA. > ierr = ISCreateStride(PETSC_COMM_WORLD, _matrixA.ownerRange().second - > _matrixA.ownerRange().first, _matrixA.ownerRange().first, 1, &ISidentity); > CHKERRV(ierr); > ierr = ISSetIdentity(ISidentity); CHKERRV(ierr); > ierr = ISAllGather(ISidentity, &ISidentityGlobal); CHKERRV(ierr); > ierr = ISLocalToGlobalMappingCreateIS(ISidentityGlobal, > &ISidentityMapping); CHKERRV(ierr); > > ierr = MatSetLocalToGlobalMapping(_matrixC.matrix, _ISmapping, > _ISmapping); CHKERRV(ierr); // Set mapping for rows and cols > ierr = MatSetLocalToGlobalMapping(_matrixA.matrix, ISidentityMapping, > _ISmapping); CHKERRV(ierr); // Set mapping only for cols, use identity for > rows > > > Two questions: > > 1) Since I only need the _ISmapping again, is there any way to optimize > that, i.e. use less temporary variables? > It is not clear to me what is being done. > 2) Do all IS variables (_ISmapping, ISlocal, ISlocalInv, ISglobal, > ISidentity, ISidentityGlobal, ISidentityMapping) need to be destroyed using > ISDestroy at the end? Or only the ones that were created using a ISCreate* > function (ISlocal, ISidentity)? > All of them since you create new ISes with all those functions like ISInvertPermutation. Matt > Thanks, > > Florian > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From amneetb at live.unc.edu Wed Feb 24 12:48:14 2016 From: amneetb at live.unc.edu (Bhalla, Amneet Pal S) Date: Wed, 24 Feb 2016 18:48:14 +0000 Subject: [petsc-users] ISDestroy performance Message-ID: <2463B8DD-CBEB-4309-9620-4AE6C80DD3A6@ad.unc.edu> Hi Folks, In our MG algorithm for Stokes like system we are using PETSc solvers at each level for smoothers. In particular, we are using PCASM and PCFIELDSPLIT as smoothers (with 1/2 iterations of Chebyshev or Richardson solvers). We are also defining our own domains (IS'es) for these two PCs. The solvers at each level are initialized at the beginning of the timestep and destroyed at the end of the timestep (it's a dynamic problem). The HPCToolKit shows that about 74% of the time is used in doing ISDestroy, whereas we expect it to use in SNES solve. This seems very weird. I also tried Instruments (OSX) just to be sure that it was not an anomaly with HPCToolKit reporting, but the same performance sink shows up there too. Attached is the profiling from both profilers. Can anything be done to mitigate this time sink for ISDestroy? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Instruments_FieldsplitRun.trace.zip Type: application/zip Size: 4349185 bytes Desc: Instruments_FieldsplitRun.trace.zip URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: hpctoolkit-Fielsplit-database.zip Type: application/zip Size: 1674847 bytes Desc: hpctoolkit-Fielsplit-database.zip URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Feb 24 13:21:11 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 24 Feb 2016 13:21:11 -0600 Subject: [petsc-users] Neumann BC with non-symmetric matrix In-Reply-To: References: <715E1F2D-F735-4419-811A-590E2218516F@mcs.anl.gov> Message-ID: > On Feb 24, 2016, at 9:07 AM, Mohammad Mirzadeh wrote: > > Barry, > > On Wednesday, February 24, 2016, Barry Smith wrote: > > > On Feb 24, 2016, at 12:07 AM, Mohammad Mirzadeh wrote: > > > At the PDE level the compatibility condition is satisfied. However I suspect that at the discrete level the rhs is not not exactly in the range. The reason for my suspicion is that I know for a fact that my discretization is not conservative at the hanging nodes. Could be. > > Thanks for the suggestion. I'll give it a try. Howerver, does GMRES fundamentally behave differently than BiCGSTAB for these systems? I have seen people saying that it can keep the solution in the range if rhs is already in the range but I thought any KSP would do the same? Here is the deal. There are two issues here 1) Making sure that b is the in the range of A. If b is not in the range of A then it is obviously impossible to find an x such that A x = b. If we divide b into a part in the range of A called br and the rest call it bp then b = br + bp and one can solve Ax = br and the left over residual is bp. Normally if you run GMRES with an inconsistent right hand side (that is bp != 0) it will solve Ax = br automatically and thus give you a "least squares" answer which is likely what you want. These means in some sense you don't really need to worry about making sure that b is in the range of A. But the residuals of GMRES will stagnate, which makes sense because it cannot get rid of the bp part. In the least squares sense however GMRES has converged. If you provide MatSetTransposeNullSpace() then KSP automatically removes this space from b when it starts the Krylov method, this is nice because the GMRES residuals will go to zero. 2) Making sure that huge multiples of the null space of A do not get into the computed solution. With left preconditioning Krylov methods construct the solution in the space {Bb, BABb, BABABb, ..} if the range of B contains entries in the null space of A then the Krylov space will contain vectors in the null space of A. What can happen is that in each iteration of the Krylov space those entries grow and you end up with a solution x + xn where xn is in the null space of A and very large, thus it looks like GMRES is not converging since the solution "jump" each iteration. If you force the range of B to exclude the null space of A, that is project out the null space of A after applying B then nothing in the null space ever gets into the Krylov space and you get the "minimum norm" solution to the least squares problem which is what you almost always want. When you provide MatSetNullSpace() the KSP solvers automatically remove the null space after applying B. All this stuff applies for any Krylov method. So based on my understanding, you should just provide the null space that you can and forget out the transpose null space and use left preconditioning with GMRES (forget about what I said about trying with right preconditioning). Let GMRES iterate until the residual norm has stabilized and use the resulting solution which is the least squares solution. If you are using KSP inside SNES you may need to set SNESSetMaxLinearSolveFailures() to a large value so it doesn't stop when it thinks the GMRES has failed. Barry Matt and Jed, please check my logic I often flip my ranges/null spaces and may have incorrect presentation above. > > > > > > > > 2) I have tried fixing the solution at an arbitrary point, and while it generally works, for some problems I get numerical artifacts, e.g. slight asymmetry in the solution and/or increased error close to the point where I fix the solution. Is this, more or less, expected as a known artifact? > > Yeah this approach is not good. We never recommend it. > > > > > > 3) An alternative to 2 is to enforce some global constraint on the solution, e.g. to require that the average be zero. My question here is two-fold: > > > > Requiring the average be zero is exactly the same as providing a null space of the constant function. Saying the average is zero is the same as saying the solution is orthogonal to the constant function. I don't see any reason to introduce the Lagrange multiplier and all its complications inside of just providing the constant null space. > > > > Is this also true at the discrete level when the matrix is non-symmetric? I have always viewed this as just a constraint that could really be anything. > > > > > > > > 3-1) Is this generally any better than solution 2, in terms of not messing too much with the condition number of the matrix? > > > > > > 3-2) I don't quite know how to implement this using PETSc. Generally speaking I'd like to solve > > > > > > | A U | | X | | B | > > > | | * | | = | | > > > | U^T 0 | | s | | 0 | > > > > > > > > > where U is a constant vector (of ones) and s is effectively a Lagrange multiplier. I suspect I need to use MatCreateSchurComplement and pass that to the KSP? Or do I need create my own matrix type from scratch through MatCreateShell? > > > > > > Any help is appreciated! > > > > > > Thanks, > > > Mohammad > > > > > > > > > > > > > > -- > > Sent from Gmail Mobile > > > > -- > Sent from Gmail Mobile From bsmith at mcs.anl.gov Wed Feb 24 13:45:37 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 24 Feb 2016 13:45:37 -0600 Subject: [petsc-users] ISDestroy performance In-Reply-To: <2463B8DD-CBEB-4309-9620-4AE6C80DD3A6@ad.unc.edu> References: <2463B8DD-CBEB-4309-9620-4AE6C80DD3A6@ad.unc.edu> Message-ID: Yikes, you are good at finding the bad hidden backwaters of PETSc that kill performance. Edit src/sys/dll/reg.c and look for the lines #if defined(PETSC_USE_LOG) /* add this new list to list of all lists */ if (!dlallhead) { dlallhead = *fl; (*fl)->next_list = 0; } else { ne = dlallhead; dlallhead = *fl; (*fl)->next_list = ne; } #endif Remove them. Let us know if this works. This logging of functions is completely non-scalable for large numbers of PETSc objects. Thanks for sending the trace it made it easy for me to determine the problem. Barry > On Feb 24, 2016, at 12:48 PM, Bhalla, Amneet Pal S wrote: > > Hi Folks, > > In our MG algorithm for Stokes like system we are using PETSc solvers at each level for smoothers. In particular, we are using PCASM and PCFIELDSPLIT as smoothers (with 1/2 iterations of Chebyshev or Richardson solvers). > We are also defining our own domains (IS'es) for these two PCs. The solvers at each level are initialized at the beginning of the timestep and destroyed at the end of the > timestep (it's a dynamic problem). The HPCToolKit shows that about 74% of the time is used in doing ISDestroy, whereas we expect it to use in SNES solve. This seems very weird. I also tried Instruments (OSX) just to > be sure that it was not an anomaly with HPCToolKit reporting, but the same performance sink shows up there too. Attached is the profiling from both profilers. Can anything be done to mitigate > this time sink for ISDestroy? > > > > Thanks, > > > ? Amneet > ===================================================== > Amneet Bhalla > Postdoctoral Research Associate > Department of Mathematics and McAllister Heart Institute > University of North Carolina at Chapel Hill > Email: amneet at unc.edu > Web: https://abhalla.web.unc.edu > ===================================================== > From juan at tf.uni-kiel.de Wed Feb 24 13:54:27 2016 From: juan at tf.uni-kiel.de (juan) Date: Wed, 24 Feb 2016 20:54:27 +0100 Subject: [petsc-users] Mass matrix with PetscFE In-Reply-To: <56CDB84F.8020309@tf.uni-kiel.de> References: <56CDA67F.6000906@tf.uni-kiel.de> <56CDB469.806@tf.uni-kiel.de> <56CDB84F.8020309@tf.uni-kiel.de> Message-ID: <518cc2f74e6b2267660acaf3871d52f9@tf.uni-kiel.de> I attached another example which creates the correct mass matrix but also overwrites the DM for the SNES solve. Somehow i cannot manage to really copy the DM to dm_mass and use that. If i try to do that with DMClone(dm, &dm_mass) i get a smaller mass matrix (which is not of size A). Maybe this helps in the discussion. Relevant code starts at line 455. On 2016-02-24 15:03, Julian Andrej wrote: > Thanks Matt, > > I attached the modified example. > > the corresponding code (and only changes to ex12) is starting at line > 832. > > It also seems that the mass matrix is of size 169x169 and the > stiffness matrix is of dimension 225x225. I'd assume that if i > multiply test and trial function i'd get a matrix of same size (if the > space/quadrature is the same for the stiffness matrix) > > On 24.02.2016 14:56, Matthew Knepley wrote: >> On Wed, Feb 24, 2016 at 7:47 AM, Julian Andrej > > wrote: >> >> I'm now using the petsc git master branch. >> >> I tried adding my code to the ex12 >> >> DM dm_mass; >> PetscDS prob_mass; >> PetscFE fe; >> Mat M; >> PetscFECreateDefault(dm, user.dim, 1, PETSC_TRUE, NULL, -1, >> &fe); >> >> DMClone(dm, &dm_mass); >> DMGetDS(dm_mass, &prob_mass); >> PetscDSSetDiscretization(prob_mass, 0, (PetscObject) fe); >> PetscDSSetJacobian(prob_mass, 0, 0, mass_kernel, NULL, NULL, >> NULL); >> DMCreateMatrix(dm_mass, &M); >> >> MatSetOptionsPrefix(M, "M_";) >> >> and receive the error on running >> ./exe -interpolate -refinement_limit 0.0125 -petscspace_order 2 >> -M_mat_view binary >> >> WARNING! There are options you set that were not used! >> WARNING! could be spelling mistake, etc! >> Option left: name:-M_mat_view value: binary >> >> I don't know if the matrix is actually there and assembled or if >> the >> option is ommitted because something is wrong. >> >> >> Its difficult to know when I cannot see the whole code. You can always >> insert >> >> MatViewFromOptions(M, NULL, "-mat_view"); >> >> Using >> MatView(M, PETSC_VIEWER_STDOUT_WORLD); >> >> gives me a reasonable output to stdout. >> >> >> Good. >> >> But saving the matrix and analysing it in matlab, results in an >> all >> zero matrix. >> >> PetscViewerBinaryOpen(PETSC_COMM_WORLD, "Mout",FILE_MODE_WRITE, >> &viewer); >> MatView(M, viewer); >> >> >> I cannot explain this, but it has to be something like you are viewing >> the matrix before it is >> actually assembled. Feel free to send the code. It sounds like it is >> mostly working. >> >> Matt >> >> Any hints? >> >> >> On 24.02.2016 13 :58, Matthew Knepley wrote: >> >> On Wed, Feb 24, 2016 at 6:47 AM, Julian Andrej >> >> >> >> wrote: >> >> Hi, >> >> i'm trying to assemble a mass matrix with the >> PetscFE/DMPlex >> interface. I found something in the examples of TAO >> >> >> https://bitbucket.org/petsc/petsc/src/da8116b0e8d067e39fd79740a8a864b0fe207998/src/tao/examples/tutorials/ex3.c?at=master&fileviewer=file-view-default >> >> but using the lines >> >> DMClone(dm, &dm_mass); >> DMSetNumFields(dm_mass, 1); >> DMPlexCopyCoordinates(dm, dm_mass); >> DMGetDS(dm_mass, &prob_mass); >> PetscDSSetJacobian(prob_mass, 0, 0, mass_kernel, NULL, >> NULL, NULL); >> PetscDSSetDiscretization(prob_mass, 0, (PetscObject) fe); >> DMPlexSNESComputeJacobianFEM(dm_mass, u, M, M, NULL); >> DMCreateMatrix(dm_mass, &M); >> >> leads to errors in DMPlexSNESComputeJacobianFEM (u is a >> global vector). >> >> I don't can understand the necessary commands until >> DMPlexSNESComputeJacobianFEM. What does it do and why is >> it >> necessary? (especially why does the naming involve SNES?) >> >> Is there another/easier/better way to create a mass >> matrix (the >> inner product of the function space and the test space)? >> >> >> 1) That example needs updating. First, look at SNES ex12 which >> is up to >> date. >> >> 2) I assume you are using 3.6. If you use the development >> version, you >> can remove DMPlexCopyCoordinates(). >> >> 3) You need to create the matrix BEFORE calling the assembly >> >> 4) Always always always send the entire error messge >> >> Matt >> >> Regards >> Julian Andrej >> >> >> >> >> -- >> What most experimenters take for granted before they begin >> their >> experiments is infinitely more interesting than any results to >> which >> their experiments lead. >> -- Norbert Wiener >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which >> their experiments lead. >> -- Norbert Wiener -------------- next part -------------- A non-text attachment was scrubbed... Name: main.c Type: text/x-csrc Size: 21021 bytes Desc: not available URL: From amneetb at live.unc.edu Wed Feb 24 14:48:52 2016 From: amneetb at live.unc.edu (Bhalla, Amneet Pal S) Date: Wed, 24 Feb 2016 20:48:52 +0000 Subject: [petsc-users] ISDestroy performance In-Reply-To: References: <2463B8DD-CBEB-4309-9620-4AE6C80DD3A6@ad.unc.edu> Message-ID: Thanks Barry --- that changed the runtime substantially and SNES dominates. Sanity is restored ;-). Looking at the new trace is there something else that jumps out from PETSc calls which can be made more efficient for this case? -------------- next part -------------- A non-text attachment was scrubbed... Name: Instruments_FieldsplitRun_dll_changed.trace.zip Type: application/zip Size: 3372568 bytes Desc: Instruments_FieldsplitRun_dll_changed.trace.zip URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: ATT00001.txt URL: From bsmith at mcs.anl.gov Wed Feb 24 16:15:52 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 24 Feb 2016 16:15:52 -0600 Subject: [petsc-users] ISDestroy performance In-Reply-To: References: <2463B8DD-CBEB-4309-9620-4AE6C80DD3A6@ad.unc.edu> Message-ID: <9DD5F938-2786-4AFD-8310-14E74D4DE1EC@mcs.anl.gov> When you use ISCreateGeneral() do you use PETSC_COPY_VALUES, PETSC_OWN_POINTER, PETSC_USE_POINTER ? Using either PETSC_OWN_POINTER, PETSC_USE_POINTER will be better than PETSC_COPY_VALUES Otherwise I don't see any oddities. Barry > On Feb 24, 2016, at 2:48 PM, Bhalla, Amneet Pal S wrote: > > Thanks Barry --- that changed the runtime substantially and SNES dominates. Sanity is restored ;-). Looking at the new trace is > there something else that jumps out from PETSc calls which can be made more efficient for this case? > > >> On Feb 24, 2016, at 11:45 AM, Barry Smith wrote: >> >> >> Yikes, you are good at finding the bad hidden backwaters of PETSc that kill performance. >> >> Edit src/sys/dll/reg.c and look for the lines >> >> #if defined(PETSC_USE_LOG) >> /* add this new list to list of all lists */ >> if (!dlallhead) { >> dlallhead = *fl; >> (*fl)->next_list = 0; >> } else { >> ne = dlallhead; >> dlallhead = *fl; >> (*fl)->next_list = ne; >> } >> #endif >> >> Remove them. Let us know if this works. >> >> This logging of functions is completely non-scalable for large numbers of PETSc objects. >> >> Thanks for sending the trace it made it easy for me to determine the problem. >> >> Barry >> >>> On Feb 24, 2016, at 12:48 PM, Bhalla, Amneet Pal S wrote: >>> >>> Hi Folks, >>> >>> In our MG algorithm for Stokes like system we are using PETSc solvers at each level for smoothers. In particular, we are using PCASM and PCFIELDSPLIT as smoothers (with 1/2 iterations of Chebyshev or Richardson solvers). >>> We are also defining our own domains (IS'es) for these two PCs. The solvers at each level are initialized at the beginning of the timestep and destroyed at the end of the >>> timestep (it's a dynamic problem). The HPCToolKit shows that about 74% of the time is used in doing ISDestroy, whereas we expect it to use in SNES solve. This seems very weird. I also tried Instruments (OSX) just to >>> be sure that it was not an anomaly with HPCToolKit reporting, but the same performance sink shows up there too. Attached is the profiling from both profilers. Can anything be done to mitigate >>> this time sink for ISDestroy? >>> >>> >>> >>> Thanks, >>> >>> >>> ? Amneet >>> ===================================================== >>> Amneet Bhalla >>> Postdoctoral Research Associate >>> Department of Mathematics and McAllister Heart Institute >>> University of North Carolina at Chapel Hill >>> Email: amneet at unc.edu >>> Web: https://abhalla.web.unc.edu >>> ===================================================== >>> >> > From rongliang.chan at gmail.com Thu Feb 25 00:19:38 2016 From: rongliang.chan at gmail.com (Rongliang Chen) Date: Thu, 25 Feb 2016 14:19:38 +0800 Subject: [petsc-users] DMRefine fails Message-ID: <56CE9CFA.4040809@gmail.com> Dear All, In my application, I need an overlapping partition of the mesh, that is to set the overlapping size in DMPlexDistribute(*dm, partitioner, overlap, NULL, &distributedMesh) to be nonzero (overlap=1). At the same time, I want to use the DMRefine to refine the distributed mesh. I found that the refined mesh obtained by DMRefine is totally wrong. But if I set the overlapping size in DMPlexDistribute to be zero, then the refined mesh is correct. Please let me know if you have any ideas to use the DMRefine for a distributed mesh with nonzero overlapping size. Best regards, Rongliang From zonexo at gmail.com Thu Feb 25 00:33:30 2016 From: zonexo at gmail.com (TAY wee-beng) Date: Thu, 25 Feb 2016 14:33:30 +0800 Subject: [petsc-users] Error - Out of memory. This could be due to allocating too large an object or bleeding by not properly ... In-Reply-To: References: <56CD0C7C.5060101@gmail.com> <56CD61C7.3040200@gmail.com> <56CDC961.40004@gmail.com> <56CDCC13.4070807@gmail.com> Message-ID: <56CEA03A.2000407@gmail.com> Hi, I ran the code and it hangs again. However, adding -malloc_test doesn't seem to do any thing. The output (attached) is the same w/o it. Wonder if there's anything else I can do. Thank you Yours sincerely, TAY wee-beng On 24/2/2016 11:33 PM, Matthew Knepley wrote: > On Wed, Feb 24, 2016 at 9:28 AM, TAY wee-beng > wrote: > > > On 24/2/2016 11:18 PM, Matthew Knepley wrote: >> On Wed, Feb 24, 2016 at 9:16 AM, TAY wee-beng > > wrote: >> >> >> On 24/2/2016 9:12 PM, Matthew Knepley wrote: >>> On Wed, Feb 24, 2016 at 1:54 AM, TAY wee-beng >>> > wrote: >>> >>> >>> On 24/2/2016 10:28 AM, Matthew Knepley wrote: >>>> On Tue, Feb 23, 2016 at 7:50 PM, TAY wee-beng >>>> > wrote: >>>> >>>> Hi, >>>> >>>> I got this error (also attached, full) when running >>>> my code. It happens after a few thousand time steps. >>>> >>>> The strange thing is that for 2 different clusters, >>>> it stops at 2 different time steps. >>>> >>>> I wonder if it's related to DM since this happens >>>> after I added DM into my code. >>>> >>>> In this case, how can I find out the error? I'm >>>> thinking valgrind may take very long and gives too >>>> many false errors. >>>> >>>> >>>> It is very easy to find leaks. You just run a few steps >>>> with -malloc_dump and see what is left over. >>>> >>>> Matt >>> Hi Matt, >>> >>> Do you mean running my a.out with the -malloc_dump and >>> stop after a few time steps? >>> >>> What and how should I "see" then? >>> >>> >>> -malloc_dump outputs all unfreed memory to the screen after >>> PetscFinalize(), so you should see the leak. >>> I guess it might be possible to keep creating things that >>> you freed all at once at the end, but that is less likely. >>> >>> Matt >> Hi, >> >> I got the output. I have zipped it since it's rather big. So >> it seems to be from DM routines but can you help me where the >> error is from? >> >> >> Its really hard to tell by looking at it. What I do is remove >> things until there is no leak, then progressively >> put thing back in until I have the culprit. Then you can think >> about what is not destroyed. >> >> Matt > Ok so let me get this clear. When it shows: > > [21]Total space allocated 1728961264 bytes > [21]1861664 bytes MatCheckCompressedRow() line 60 in > /home/wtay/Codes/petsc-3.6.3/src/mat/utils/compressedrow.c > [21]16 bytes PetscStrallocpy() line 188 in > /home/wtay/Codes/petsc-3.6.3/src/sys/utils/str.c > [21]624 bytes ISLocalToGlobalMappingCreate() line 270 in > /home/wtay/Codes > > .... > > Does it mean that it's simply allocating space ie normal? Or does > it show that there's memory leak ie error? > > > I gave the wrong option. That dumps everything. Lets just look at the > leaks with -malloc_test. > > Sorry about that, > > Matt > > If it's error, should I zoom in and debug around this time at this > region? > > Thanks >> >> Thanks. >>> >>>> >>>> -- >>>> Thank you >>>> >>>> Yours sincerely, >>>> >>>> TAY wee-beng >>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they >>>> begin their experiments is infinitely more interesting >>>> than any results to which their experiments lead. >>>> -- Norbert Wiener >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin >>> their experiments is infinitely more interesting than any >>> results to which their experiments lead. >>> -- Norbert Wiener >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- **************************************************************************** * hwloc has encountered what looks like an error from the operating system. * * L3 (cpuset 0x000003f0) intersects with NUMANode (P#0 cpuset 0x0000003f) without inclusion! * Error occurred in topology.c line 942 * * The following FAQ entry in a recent hwloc documentation may help: * What should I do when hwloc reports "operating system" warnings? * Otherwise please report this error message to the hwloc user's mailing list, * along with the output+tarball generated by the hwloc-gather-topology script. **************************************************************************** **************************************************************************** * hwloc has encountered what looks like an error from the operating system. * * L3 (cpuset 0x000003f0) intersects with NUMANode (P#0 cpuset 0x0000003f) without inclusion! * Error occurred in topology.c line 942 * * The following FAQ entry in a recent hwloc documentation may help: * What should I do when hwloc reports "operating system" warnings? * Otherwise please report this error message to the hwloc user's mailing list, * along with the output+tarball generated by the hwloc-gather-topology script. **************************************************************************** [13]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [13]PETSC ERROR: Out of memory. This could be due to allocating [13]PETSC ERROR: too large an object or bleeding by not properly [13]PETSC ERROR: destroying unneeded objects. [6]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [6]PETSC ERROR: Out of memory. This could be due to allocating [6]PETSC ERROR: too large an object or bleeding by not properly [6]PETSC ERROR: destroying unneeded objects. [17]PETSC ERROR: [6]PETSC ERROR: Memory allocated 0 Memory used by process 3595563008 [6]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. [6]PETSC ERROR: Memory requested 37814445 [6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [6]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [6]PETSC ERROR: ./a.out on a petsc-3.6.3_static_rel named n12-09 by wtay Wed Feb 24 16:57:03 2016 [6]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1 [6]PETSC ERROR: #1 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [6]PETSC ERROR: #2 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [6]PETSC ERROR: #3 MatLUFactorNumeric() line 2946 in /home/wtay/Codes/petsc-3.6.3/src/mat/interface/matrix.c [6]PETSC ERROR: #4 PCSetUp_ILU() line 233 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/factor/ilu/ilu.c [6]PETSC ERROR: #5 PCSetUp() line 983 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [6]PETSC ERROR: #6 KSPSetUp() line 332 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [6]PETSC ERROR: #7 KSPSolve() line 546 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [6]PETSC ERROR: #8 PCApply_BJacobi_Singleblock() line 670 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/bjacobi/bjacobi.c [6]PETSC ERROR: #9 PCApply() line 483 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [6]PETSC ERROR: #10 KSP_PCApply() line 242 in /home/wtay/Codes/petsc-3.6.3/include/petsc/private/kspimpl.h [6]PETSC ERROR: #11 KSPInitialResidual() line 63 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itres.c [6]PETSC ERROR: #12 KSPSolve_BCGS() line 50 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/impls/bcgs/bcgs.c [6]PETSC ERROR: #13 KSPSolve() line 604 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [13]PETSC ERROR: Memory allocated 0 Memory used by process 3522146304 [13]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. [13]PETSC ERROR: Memory requested 36360045 [13]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [13]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [13]PETSC ERROR: ./a.out on a petsc-3.6.3_static_rel named n12-09 by wtay Wed Feb 24 16:57:03 2016 [13]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1 [13]PETSC ERROR: #1 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [13]PETSC ERROR: #2 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [13]PETSC ERROR: #3 MatLUFactorNumeric() line 2946 in /home/wtay/Codes/petsc-3.6.3/src/mat/interface/matrix.c [13]PETSC ERROR: #4 PCSetUp_ILU() line 233 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/factor/ilu/ilu.c [13]PETSC ERROR: #5 PCSetUp() line 983 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [13]PETSC ERROR: #6 KSPSetUp() line 332 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [13]PETSC ERROR: #7 KSPSolve() line 546 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [13]PETSC ERROR: #8 PCApply_BJacobi_Singleblock() line 670 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/bjacobi/bjacobi.c [13]PETSC ERROR: #9 PCApply() line 483 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [13]PETSC ERROR: #10 KSP_PCApply() line 242 in /home/wtay/Codes/petsc-3.6.3/include/petsc/private/kspimpl.h [13]PETSC ERROR: #11 KSPInitialResidual() line 63 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itres.c [13]PETSC ERROR: #12 KSPSolve_BCGS() line 50 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/impls/bcgs/bcgs.c [13]PETSC ERROR: #13 KSPSolve() line 604 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c --------------------- Error Message -------------------------------------------------------------- [33]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [33]PETSC ERROR: Out of memory. This could be due to allocating [33]PETSC ERROR: too large an object or bleeding by not properly [33]PETSC ERROR: destroying unneeded objects. [33]PETSC ERROR: Memory allocated 0 Memory used by process 3518820352 [33]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. [33]PETSC ERROR: Memory requested 36360045 [33]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [33]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [33]PETSC ERROR: ./a.out on a petsc-3.6.3_static_rel named n12-09 by wtay Wed Feb 24 16:57:03 2016 [33]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1 [33]PETSC ERROR: #1 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [33]PETSC ERROR: #2 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [33]PETSC ERROR: #3 MatLUFactorNumeric() line 2946 in /home/wtay/Codes/petsc-3.6.3/src/mat/interface/matrix.c [33]PETSC ERROR: #4 PCSetUp_ILU() line 233 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/factor/ilu/ilu.c [33]PETSC ERROR: #5 PCSetUp() line 983 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [33]PETSC ERROR: #6 KSPSetUp() line 332 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [33]PETSC ERROR: #7 KSPSolve() line 546 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [33]PETSC ERROR: #8 PCApply_BJacobi_Singleblock() line 670 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/bjacobi/bjacobi.c [33]PETSC ERROR: #9 PCApply() line 483 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [33]PETSC ERROR: #10 KSP_PCApply() line 242 in /home/wtay/Codes/petsc-3.6.3/include/petsc/private/kspimpl.h [33]PETSC ERROR: #11 KSPInitialResidual() line 63 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itres.c [33]PETSC ERROR: #12 KSPSolve_BCGS() line 50 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/impls/bcgs/bcgs.c [33]PETSC ERROR: #13 KSPSolve() line 604 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [17]PETSC ERROR: Out of memory. This could be due to allocating [9]PETSC ERROR: [17]PETSC ERROR: too large an object or bleeding by not properly [17]PETSC ERROR: destroying unneeded objects. [21]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [21]PETSC ERROR: Out of memory. This could be due to allocating [21]PETSC ERROR: too large an object or bleeding by not properly [21]PETSC ERROR: destroying unneeded objects. [17]PETSC ERROR: Memory allocated 0 Memory used by process 3505258496 [17]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. [17]PETSC ERROR: Memory requested 36360045 [17]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [17]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [17]PETSC ERROR: ./a.out on a petsc-3.6.3_static_rel named n12-09 by wtay Wed Feb 24 16:57:03 2016 [17]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1 [17]PETSC ERROR: #1 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [17]PETSC ERROR: #2 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [21]PETSC ERROR: Memory allocated 0 Memory used by process 3549990912 [21]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. [21]PETSC ERROR: Memory requested 36360045 [21]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [21]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [21]PETSC ERROR: ./a.out on a petsc-3.6.3_static_rel named n12-09 by wtay Wed Feb 24 16:57:03 2016 [21]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1 [21]PETSC ERROR: #1 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [21]PETSC ERROR: #2 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [21]PETSC ERROR: #3 MatLUFactorNumeric() line 2946 in /home/wtay/Codes/petsc-3.6.3/src/mat/interface/matrix.c [21]PETSC ERROR: #4 PCSetUp_ILU() line 233 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/factor/ilu/ilu.c [21]PETSC ERROR: #5 PCSetUp() line 983 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [21]PETSC ERROR: #6 KSPSetUp() line 332 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [21]PETSC ERROR: [25]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [25]PETSC ERROR: Out of memory. This could be due to allocating [25]PETSC ERROR: too large an object or bleeding by not properly [25]PETSC ERROR: destroying unneeded objects. [17]PETSC ERROR: #3 MatLUFactorNumeric() line 2946 in /home/wtay/Codes/petsc-3.6.3/src/mat/interface/matrix.c [17]PETSC ERROR: #4 PCSetUp_ILU() line 233 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/factor/ilu/ilu.c [17]PETSC ERROR: #7 KSPSolve() line 546 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [21]PETSC ERROR: #8 PCApply_BJacobi_Singleblock() line 670 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/bjacobi/bjacobi.c [21]PETSC ERROR: #9 PCApply() line 483 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [21]PETSC ERROR: #10 KSP_PCApply() line 242 in /home/wtay/Codes/petsc-3.6.3/include/petsc/private/kspimpl.h [21]PETSC ERROR: #11 KSPInitialResidual() line 63 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itres.c [21]PETSC ERROR: #12 KSPSolve_BCGS() line 50 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/impls/bcgs/bcgs.c [21]PETSC ERROR: #13 KSPSolve() line 604 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c #5 PCSetUp() line 983 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [17]PETSC ERROR: #6 KSPSetUp() line 332 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [17]PETSC ERROR: #7 KSPSolve() line 546 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [17]PETSC ERROR: #8 PCApply_BJacobi_Singleblock() line 670 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/bjacobi/bjacobi.c [17]PETSC ERROR: #9 PCApply() line 483 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [17]PETSC ERROR: #10 KSP_PCApply() line 242 in /home/wtay/Codes/petsc-3.6.3/include/petsc/private/kspimpl.h [17]PETSC ERROR: #11 KSPInitialResidual() line 63 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itres.c [17]PETSC ERROR: #12 KSPSolve_BCGS() line 50 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/impls/bcgs/bcgs.c [17]PETSC ERROR: #13 KSPSolve() line 604 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Out of memory. This could be due to allocating [1]PETSC ERROR: too large an object or bleeding by not properly [1]PETSC ERROR: destroying unneeded objects. [25]PETSC ERROR: Memory allocated 0 Memory used by process 3515305984 [25]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. [25]PETSC ERROR: Memory requested 36360045 [25]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [25]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [25]PETSC ERROR: ./a.out on a petsc-3.6.3_static_rel named n12-09 by wtay Wed Feb 24 16:57:03 2016 [25]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1 [25]PETSC ERROR: #1 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [25]PETSC ERROR: #2 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [25]PETSC ERROR: #3 MatLUFactorNumeric() line 2946 in /home/wtay/Codes/petsc-3.6.3/src/mat/interface/matrix.c [25]PETSC ERROR: #4 PCSetUp_ILU() line 233 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/factor/ilu/ilu.c [25]PETSC ERROR: #5 PCSetUp() line 983 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [25]PETSC ERROR: #6 KSPSetUp() line 332 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [25]PETSC ERROR: #7 KSPSolve() line 546 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [25]PETSC ERROR: #8 PCApply_BJacobi_Singleblock() line 670 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/bjacobi/bjacobi.c [1]PETSC ERROR: Memory allocated 0 Memory used by process 3497189376 [1]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. [1]PETSC ERROR: Memory requested 37814445 [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [1]PETSC ERROR: ./a.out on a petsc-3.6.3_static_rel named n12-09 by wtay Wed Feb 24 16:57:03 2016 [1]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1 [1]PETSC ERROR: #1 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [1]PETSC ERROR: #2 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [1]PETSC ERROR: #3 MatLUFactorNumeric() line 2946 in /home/wtay/Codes/petsc-3.6.3/src/mat/interface/matrix.c [1]PETSC ERROR: #4 PCSetUp_ILU() line 233 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/factor/ilu/ilu.c [1]PETSC ERROR: #5 PCSetUp() line 983 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [1]PETSC ERROR: #6 KSPSetUp() line 332 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [25]PETSC ERROR: #9 PCApply() line 483 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [25]PETSC ERROR: #10 KSP_PCApply() line 242 in /home/wtay/Codes/petsc-3.6.3/include/petsc/private/kspimpl.h [25]PETSC ERROR: #11 KSPInitialResidual() line 63 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itres.c [25]PETSC ERROR: #12 KSPSolve_BCGS() line 50 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/impls/bcgs/bcgs.c [25]PETSC ERROR: #13 KSPSolve() line 604 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [1]PETSC ERROR: #7 KSPSolve() line 546 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [1]PETSC ERROR: #8 PCApply_BJacobi_Singleblock() line 670 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/bjacobi/bjacobi.c [1]PETSC ERROR: #9 PCApply() line 483 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [1]PETSC ERROR: #10 KSP_PCApply() line 242 in /home/wtay/Codes/petsc-3.6.3/include/petsc/private/kspimpl.h [1]PETSC ERROR: #11 KSPInitialResidual() line 63 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itres.c [1]PETSC ERROR: #12 KSPSolve_BCGS() line 50 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/impls/bcgs/bcgs.c [1]PETSC ERROR: #13 KSPSolve() line 604 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [41]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [41]PETSC ERROR: Out of memory. This could be due to allocating [41]PETSC ERROR: too large an object or bleeding by not properly [41]PETSC ERROR: destroying unneeded objects. [41]PETSC ERROR: Memory allocated 0 Memory used by process 3544576000 [41]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. [41]PETSC ERROR: Memory requested 36360045 [41]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [41]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [41]PETSC ERROR: ./a.out on a petsc-3.6.3_static_rel named n12-09 by wtay Wed Feb 24 16:57:03 2016 [41]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1 [41]PETSC ERROR: #1 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [41]PETSC ERROR: #2 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [41]PETSC ERROR: #3 MatLUFactorNumeric() line 2946 in /home/wtay/Codes/petsc-3.6.3/src/mat/interface/matrix.c [41]PETSC ERROR: #4 PCSetUp_ILU() line 233 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/factor/ilu/ilu.c [41]PETSC ERROR: #5 PCSetUp() line 983 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [41]PETSC ERROR: #6 KSPSetUp() line 332 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [41]PETSC ERROR: #7 KSPSolve() line 546 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [41]PETSC ERROR: #8 PCApply_BJacobi_Singleblock() line 670 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/bjacobi/bjacobi.c [41]PETSC ERROR: #9 PCApply() line 483 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [41]PETSC ERROR: #10 KSP_PCApply() line 242 in /home/wtay/Codes/petsc-3.6.3/include/petsc/private/kspimpl.h [41]PETSC ERROR: #11 KSPInitialResidual() line 63 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itres.c [41]PETSC ERROR: #12 KSPSolve_BCGS() line 50 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/impls/bcgs/bcgs.c [41]PETSC ERROR: #13 KSPSolve() line 604 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c --------------------- Error Message -------------------------------------------------------------- [9]PETSC ERROR: Out of memory. This could be due to allocating [9]PETSC ERROR: too large an object or bleeding by not properly [9]PETSC ERROR: destroying unneeded objects. [9]PETSC ERROR: Memory allocated 0 Memory used by process 3521380352 [9]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. [9]PETSC ERROR: Memory requested 36360045 [9]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [9]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [9]PETSC ERROR: ./a.out on a petsc-3.6.3_static_rel named n12-09 by wtay Wed Feb 24 16:57:03 2016 [9]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1 [9]PETSC ERROR: #1 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [9]PETSC ERROR: #2 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [9]PETSC ERROR: #3 MatLUFactorNumeric() line 2946 in /home/wtay/Codes/petsc-3.6.3/src/mat/interface/matrix.c [9]PETSC ERROR: #4 PCSetUp_ILU() line 233 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/factor/ilu/ilu.c [9]PETSC ERROR: #5 PCSetUp() line 983 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [9]PETSC ERROR: #6 KSPSetUp() line 332 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [9]PETSC ERROR: #7 KSPSolve() line 546 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [9]PETSC ERROR: #8 PCApply_BJacobi_Singleblock() line 670 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/bjacobi/bjacobi.c [9]PETSC ERROR: #9 PCApply() line 483 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [9]PETSC ERROR: #10 KSP_PCApply() line 242 in /home/wtay/Codes/petsc-3.6.3/include/petsc/private/kspimpl.h [9]PETSC ERROR: #11 KSPInitialResidual() line 63 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itres.c [9]PETSC ERROR: #12 KSPSolve_BCGS() line 50 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/impls/bcgs/bcgs.c [9]PETSC ERROR: #13 KSPSolve() line 604 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [45]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [45]PETSC ERROR: Out of memory. This could be due to allocating [45]PETSC ERROR: too large an object or bleeding by not properly [45]PETSC ERROR: destroying unneeded objects. [45]PETSC ERROR: Memory allocated 0 Memory used by process 3541323776 [45]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info. [45]PETSC ERROR: Memory requested 4545004 [45]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [45]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [45]PETSC ERROR: ./a.out on a petsc-3.6.3_static_rel named n12-09 by wtay Wed Feb 24 16:57:03 2016 [45]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1 [45]PETSC ERROR: #1 MatLUFactorNumeric_SeqAIJ_Inode() line 1243 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [45]PETSC ERROR: #2 MatLUFactorNumeric_SeqAIJ_Inode() line 1243 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c [45]PETSC ERROR: #3 MatLUFactorNumeric() line 2946 in /home/wtay/Codes/petsc-3.6.3/src/mat/interface/matrix.c [45]PETSC ERROR: #4 PCSetUp_ILU() line 233 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/factor/ilu/ilu.c [45]PETSC ERROR: #5 PCSetUp() line 983 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [45]PETSC ERROR: #6 KSPSetUp() line 332 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [45]PETSC ERROR: #7 KSPSolve() line 546 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c [45]PETSC ERROR: #8 PCApply_BJacobi_Singleblock() line 670 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/bjacobi/bjacobi.c [45]PETSC ERROR: #9 PCApply() line 483 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c [45]PETSC ERROR: #10 KSP_PCApply() line 242 in /home/wtay/Codes/petsc-3.6.3/include/petsc/private/kspimpl.h [45]PETSC ERROR: #11 KSPInitialResidual() line 63 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itres.c [45]PETSC ERROR: #12 KSPSolve_BCGS() line 50 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/impls/bcgs/bcgs.c [45]PETSC ERROR: #13 KSPSolve() line 604 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c From juan at tf.uni-kiel.de Thu Feb 25 02:03:49 2016 From: juan at tf.uni-kiel.de (Julian Andrej) Date: Thu, 25 Feb 2016 09:03:49 +0100 Subject: [petsc-users] Mass matrix with PetscFE In-Reply-To: <518cc2f74e6b2267660acaf3871d52f9@tf.uni-kiel.de> References: <56CDA67F.6000906@tf.uni-kiel.de> <56CDB469.806@tf.uni-kiel.de> <56CDB84F.8020309@tf.uni-kiel.de> <518cc2f74e6b2267660acaf3871d52f9@tf.uni-kiel.de> Message-ID: <56CEB565.5010203@tf.uni-kiel.de> After a bit of rethinking the problem, the discrepancy between the size of matrix A and the mass matrix M arises because of the Dirichlet boundary conditions. So why aren't the BCs not imposed on the mass matrix? Do I need to handle Dirichlet BCs differently in this context (like zero rows and put one the diagonal?) On 24.02.2016 20:54, juan wrote: > I attached another example which creates the correct mass matrix > but also overwrites the DM for the SNES solve. Somehow i cannot manage > to really copy the DM to dm_mass and use that. If i try to do that with > DMClone(dm, &dm_mass) i get a smaller mass matrix (which is not of size A). > > Maybe this helps in the discussion. > > Relevant code starts at line 455. > > On 2016-02-24 15:03, Julian Andrej wrote: >> Thanks Matt, >> >> I attached the modified example. >> >> the corresponding code (and only changes to ex12) is starting at line >> 832. >> >> It also seems that the mass matrix is of size 169x169 and the >> stiffness matrix is of dimension 225x225. I'd assume that if i >> multiply test and trial function i'd get a matrix of same size (if the >> space/quadrature is the same for the stiffness matrix) >> >> On 24.02.2016 14:56, Matthew Knepley wrote: >>> On Wed, Feb 24, 2016 at 7:47 AM, Julian Andrej >> > wrote: >>> >>> I'm now using the petsc git master branch. >>> >>> I tried adding my code to the ex12 >>> >>> DM dm_mass; >>> PetscDS prob_mass; >>> PetscFE fe; >>> Mat M; >>> PetscFECreateDefault(dm, user.dim, 1, PETSC_TRUE, NULL, -1, &fe); >>> >>> DMClone(dm, &dm_mass); >>> DMGetDS(dm_mass, &prob_mass); >>> PetscDSSetDiscretization(prob_mass, 0, (PetscObject) fe); >>> PetscDSSetJacobian(prob_mass, 0, 0, mass_kernel, NULL, NULL, >>> NULL); >>> DMCreateMatrix(dm_mass, &M); >>> >>> MatSetOptionsPrefix(M, "M_";) >>> >>> and receive the error on running >>> ./exe -interpolate -refinement_limit 0.0125 -petscspace_order 2 >>> -M_mat_view binary >>> >>> WARNING! There are options you set that were not used! >>> WARNING! could be spelling mistake, etc! >>> Option left: name:-M_mat_view value: binary >>> >>> I don't know if the matrix is actually there and assembled or if the >>> option is ommitted because something is wrong. >>> >>> >>> Its difficult to know when I cannot see the whole code. You can always >>> insert >>> >>> MatViewFromOptions(M, NULL, "-mat_view"); >>> >>> Using >>> MatView(M, PETSC_VIEWER_STDOUT_WORLD); >>> >>> gives me a reasonable output to stdout. >>> >>> >>> Good. >>> >>> But saving the matrix and analysing it in matlab, results in an all >>> zero matrix. >>> >>> PetscViewerBinaryOpen(PETSC_COMM_WORLD, "Mout",FILE_MODE_WRITE, >>> &viewer); >>> MatView(M, viewer); >>> >>> >>> I cannot explain this, but it has to be something like you are viewing >>> the matrix before it is >>> actually assembled. Feel free to send the code. It sounds like it is >>> mostly working. >>> >>> Matt >>> >>> Any hints? >>> >>> >>> On 24.02.2016 13 :58, Matthew Knepley wrote: >>> >>> On Wed, Feb 24, 2016 at 6:47 AM, Julian Andrej >>> >>> >> >>> wrote: >>> >>> Hi, >>> >>> i'm trying to assemble a mass matrix with the >>> PetscFE/DMPlex >>> interface. I found something in the examples of TAO >>> >>> https://bitbucket.org/petsc/petsc/src/da8116b0e8d067e39fd79740a8a864b0fe207998/src/tao/examples/tutorials/ex3.c?at=master&fileviewer=file-view-default >>> >>> >>> but using the lines >>> >>> DMClone(dm, &dm_mass); >>> DMSetNumFields(dm_mass, 1); >>> DMPlexCopyCoordinates(dm, dm_mass); >>> DMGetDS(dm_mass, &prob_mass); >>> PetscDSSetJacobian(prob_mass, 0, 0, mass_kernel, NULL, >>> NULL, NULL); >>> PetscDSSetDiscretization(prob_mass, 0, (PetscObject) fe); >>> DMPlexSNESComputeJacobianFEM(dm_mass, u, M, M, NULL); >>> DMCreateMatrix(dm_mass, &M); >>> >>> leads to errors in DMPlexSNESComputeJacobianFEM (u is a >>> global vector). >>> >>> I don't can understand the necessary commands until >>> DMPlexSNESComputeJacobianFEM. What does it do and why is it >>> necessary? (especially why does the naming involve SNES?) >>> >>> Is there another/easier/better way to create a mass >>> matrix (the >>> inner product of the function space and the test space)? >>> >>> >>> 1) That example needs updating. First, look at SNES ex12 which >>> is up to >>> date. >>> >>> 2) I assume you are using 3.6. If you use the development >>> version, you >>> can remove DMPlexCopyCoordinates(). >>> >>> 3) You need to create the matrix BEFORE calling the assembly >>> >>> 4) Always always always send the entire error messge >>> >>> Matt >>> >>> Regards >>> Julian Andrej >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results >>> to which >>> their experiments lead. >>> -- Norbert Wiener >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which >>> their experiments lead. >>> -- Norbert Wiener From lawrence.mitchell at imperial.ac.uk Thu Feb 25 04:47:22 2016 From: lawrence.mitchell at imperial.ac.uk (Lawrence Mitchell) Date: Thu, 25 Feb 2016 10:47:22 +0000 Subject: [petsc-users] DMRefine fails In-Reply-To: <56CE9CFA.4040809@gmail.com> References: <56CE9CFA.4040809@gmail.com> Message-ID: <56CEDBBA.7040309@imperial.ac.uk> On 25/02/16 06:19, Rongliang Chen wrote: > Dear All, > > In my application, I need an overlapping partition of the mesh, that is > to set the overlapping size in DMPlexDistribute(*dm, partitioner, > overlap, NULL, &distributedMesh) to be nonzero (overlap=1). > > At the same time, I want to use the DMRefine to refine the distributed > mesh. I found that the refined mesh obtained by DMRefine is totally > wrong. What do you mean by "totally wrong"? Is it just that the overlapped region on the coarse mesh is also refined (such that rather than being one level deep, it is 2**nrefinements deep)? > But if I set the overlapping size in DMPlexDistribute to be zero, > then the refined mesh is correct. Please let me know if you have any > ideas to use the DMRefine for a distributed mesh with nonzero > overlapping size. I do this as follows: - Create the initial DM. - Distribute with zero overlap DMPlexDistribute(..., overlap=0, ...); - Refine the distributed DM as many times as you want. - Grow the overlap on the refined DM: DMPlexDistributeOverlap(refined_dm, 1, &pointsf, &overlapped_dm); Now the overlapped_dm is refined and has a 1-deep overlap. The only thing that, AFAIK, doesn't work out of the box with this scheme is the automatic computation of multigrid restriction and interpolation. The reason for this is that the code currently assumes that, given a coarse DM point, it can compute the corresponding fine DM cells by using: for r in range(num_sub_cells): fine_cell = coarse_cell*num_sub_cells + r There is a TODO comment in plex.c (in DMPlexMatSetClosureRefined): for (r = 0, q = 0; r < numSubcells; ++r) { /* TODO Map from coarse to fine cells */ ierr = DMPlexGetTransitiveClosure(dmf, point*numSubcells + r, PETSC_TRUE, &numFPoints, &fpoints);CHKERRQ(ierr); The two-step refine then grow the overlap method I outlined above destroys this mapping. Hence the approach currently coded in plex.c will not work. This is for regular refinement. For non-nested refinement, I think you're good because the computation of the interpolator spins over the fine cells and searches for the coarse cells. Hope that helps! Lawrence -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: OpenPGP digital signature URL: From mfadams at lbl.gov Thu Feb 25 06:21:21 2016 From: mfadams at lbl.gov (Mark Adams) Date: Thu, 25 Feb 2016 07:21:21 -0500 Subject: [petsc-users] GAMG and near-null-space when applying Dirichlet Conditions In-Reply-To: References: <21F632BF-5E41-48B2-B273-763A7EA97BD4@gmail.com> Message-ID: I added ", which is often the null space of the operator without boundary conditions" to the web page doc for MatSetNearNullSpace. On Wed, Feb 24, 2016 at 10:57 AM, Matthew Knepley wrote: > On Wed, Feb 24, 2016 at 9:45 AM, Manav Bhatia > wrote: > >> Hi, >> >> I typically apply Dirichlet BCs by modifying the Jacobin and rhs: >> zero constrained rows of matrix with 1.0 at diagonal, and zero >> corresponding rows of rhs. >> >> While using GAMG, is it still recommended to provide the near-null >> space (given that the zero-eigenvalues have been removed by specification >> of DIrichlet BCs)? >> > > Yes. > > >> If that information is still needed, should the vectors be modified >> in any manner to be consistent with the Dirichlet BCs? >> > > No. You can see that if you take a small piece of the domain, apart from > the boundary, it will have this as a null space. > > Matt > > >> Thanks, >> Manav >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Feb 25 11:56:35 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 25 Feb 2016 11:56:35 -0600 Subject: [petsc-users] Error - Out of memory. This could be due to allocating too large an object or bleeding by not properly ... In-Reply-To: <56CEA03A.2000407@gmail.com> References: <56CD0C7C.5060101@gmail.com> <56CD61C7.3040200@gmail.com> <56CDC961.40004@gmail.com> <56CDCC13.4070807@gmail.com> <56CEA03A.2000407@gmail.com> Message-ID: <43AC24CD-4727-4C4C-B016-6927CD206425@mcs.anl.gov> Run a much smaller problem for a few time steps, making sure you free all the objects at the end, with the option -malloc_dump this will print all the memory that was not freed and hopefully help you track down which objects you forgot to free. Barry > On Feb 25, 2016, at 12:33 AM, TAY wee-beng wrote: > > Hi, > > I ran the code and it hangs again. However, adding -malloc_test doesn't seem to do any thing. The output (attached) is the same w/o it. > > Wonder if there's anything else I can do. > Thank you > > Yours sincerely, > > TAY wee-beng > > On 24/2/2016 11:33 PM, Matthew Knepley wrote: >> On Wed, Feb 24, 2016 at 9:28 AM, TAY wee-beng wrote: >> >> On 24/2/2016 11:18 PM, Matthew Knepley wrote: >>> On Wed, Feb 24, 2016 at 9:16 AM, TAY wee-beng wrote: >>> >>> On 24/2/2016 9:12 PM, Matthew Knepley wrote: >>>> On Wed, Feb 24, 2016 at 1:54 AM, TAY wee-beng wrote: >>>> >>>> On 24/2/2016 10:28 AM, Matthew Knepley wrote: >>>>> On Tue, Feb 23, 2016 at 7:50 PM, TAY wee-beng wrote: >>>>> Hi, >>>>> >>>>> I got this error (also attached, full) when running my code. It happens after a few thousand time steps. >>>>> >>>>> The strange thing is that for 2 different clusters, it stops at 2 different time steps. >>>>> >>>>> I wonder if it's related to DM since this happens after I added DM into my code. >>>>> >>>>> In this case, how can I find out the error? I'm thinking valgrind may take very long and gives too many false errors. >>>>> >>>>> It is very easy to find leaks. You just run a few steps with -malloc_dump and see what is left over. >>>>> >>>>> Matt >>>> Hi Matt, >>>> >>>> Do you mean running my a.out with the -malloc_dump and stop after a few time steps? >>>> >>>> What and how should I "see" then? >>>> >>>> -malloc_dump outputs all unfreed memory to the screen after PetscFinalize(), so you should see the leak. >>>> I guess it might be possible to keep creating things that you freed all at once at the end, but that is less likely. >>>> >>>> Matt >>>> >>> Hi, >>> >>> I got the output. I have zipped it since it's rather big. So it seems to be from DM routines but can you help me where the error is from? >>> >>> Its really hard to tell by looking at it. What I do is remove things until there is no leak, then progressively >>> put thing back in until I have the culprit. Then you can think about what is not destroyed. >>> >>> Matt >> Ok so let me get this clear. When it shows: >> >> [21]Total space allocated 1728961264 bytes >> [21]1861664 bytes MatCheckCompressedRow() line 60 in /home/wtay/Codes/petsc-3.6.3/src/mat/utils/compressedrow.c >> [21]16 bytes PetscStrallocpy() line 188 in /home/wtay/Codes/petsc-3.6.3/src/sys/utils/str.c >> [21]624 bytes ISLocalToGlobalMappingCreate() line 270 in /home/wtay/Codes >> >> .... >> >> Does it mean that it's simply allocating space ie normal? Or does it show that there's memory leak ie error? >> >> I gave the wrong option. That dumps everything. Lets just look at the leaks with -malloc_test. >> >> Sorry about that, >> >> Matt >> >> If it's error, should I zoom in and debug around this time at this region? >> >> Thanks >>> >>> Thanks. >>>> >>>>> >>>>> -- >>>>> Thank you >>>>> >>>>> Yours sincerely, >>>>> >>>>> TAY wee-beng >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>> -- Norbert Wiener >>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>> -- Norbert Wiener >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener > > From mirzadeh at gmail.com Thu Feb 25 16:18:34 2016 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Thu, 25 Feb 2016 17:18:34 -0500 Subject: [petsc-users] Neumann BC with non-symmetric matrix In-Reply-To: References: <715E1F2D-F735-4419-811A-590E2218516F@mcs.anl.gov> Message-ID: Barry, Thanks a lot for the detailed discussion. Things make much more sense now, especially I was confused why the manual says to provide call both 'MatSetNullSpace' and 'MatSetTransposeNullSpace'. I have couple of more questions and some observations I have made since yesterday. 1) Is there a systematic way to tell KSP to stop when it stagnates? I am _not_ using SNES. 2) Once KSP converges to the least-square solution, the residual must be in the nullspace of A^T because otherwise it would have been reduced by the solver. So is it (at lest theoretically) possible to use the residual vector as an approximate basis for the n(A^T)? In general this wouldn't be enough but I'm thinking since the nullspace is one-dimensional, maybe I could use the residual itself to manually project solution onto range of A after calling KSPSolve? 3) Are preconditoners applied to the left by default? If not which one are and which one are not? 4) So then if I provide the nullspace of A, KSP residual should converge, correct? I have made a few observations: 4-1) When I provide the nullspace of A through MatSetNullSpace, I (generally) do see the residual (preconditioned) become very small (~1e-12 or so) but then it sometimes stagnates (say around 1e-10). Is this normal? 4-2) Another observation is that the true residual stagnates way earlier which I assume is an indication that the RHS is in fact not in the range of A. 4-3) Also, I've seen that the choice of KSP and PC have considerable effects on the solver. For instance, by default I use hypre+bcgs and I have noticed I need to change coarse-grid relaxation from Gaussian Elimination to symmetric-sor/Jacobi otherwise hypre has issues. I assume this is because the AMG preconditioner is also singular on the coarse level? 4-4) Along the same lines, I tried a couple of other PCs such as {jacobi, sor, gamg, ilu} and none of them were able to converge with bcgs as the KSP. However, with gmres, almost all of them converge with the exception of gamg. 4-5) If I use gmres instead of bcgs, and for any of {jacobi, sor, ilu}, the true residual seems to be generally 2-3 orders of magnitude smaller (1e-6 vs 1e-3). I suppose this is just because gmres is more robust? 4-6) With hypre, the true residual is always larger (~1e-3) than other PCs no matter if I use bcgs or gmres. Sorry for the long discussion but this has turned out to be quite educational for me! Thanks, Mohammad On Wed, Feb 24, 2016 at 2:21 PM, Barry Smith wrote: > > > On Feb 24, 2016, at 9:07 AM, Mohammad Mirzadeh > wrote: > > > > Barry, > > > > On Wednesday, February 24, 2016, Barry Smith wrote: > > > > > On Feb 24, 2016, at 12:07 AM, Mohammad Mirzadeh > wrote: > > > > > > At the PDE level the compatibility condition is satisfied. However I > suspect that at the discrete level the rhs is not not exactly in the range. > The reason for my suspicion is that I know for a fact that my > discretization is not conservative at the hanging nodes. > > Could be. > > > > > Thanks for the suggestion. I'll give it a try. Howerver, does GMRES > fundamentally behave differently than BiCGSTAB for these systems? I have > seen people saying that it can keep the solution in the range if rhs is > already in the range but I thought any KSP would do the same? > > > Here is the deal. There are two issues here > > 1) Making sure that b is the in the range of A. If b is not in the range > of A then it is obviously impossible to find an x such that A x = b. If we > divide b into a part in the range of A called br and the rest call it bp > then b = br + bp and one can solve Ax = br and the left over residual > is bp. Normally if you run GMRES with an inconsistent right hand side (that > is bp != 0) it will solve Ax = br automatically and thus give you a "least > squares" answer which is likely what you want. These means in some sense > you don't really need to worry about making sure that b is in the range of > A. But the residuals of GMRES will stagnate, which makes sense because it > cannot get rid of the bp part. In the least squares sense however GMRES has > converged. If you provide MatSetTransposeNullSpace() then KSP automatically > removes this space from b when it starts the Krylov method, this is nice > because the GMRES residuals will go to zero. > > 2) Making sure that huge multiples of the null space of A do not get into > the computed solution. > > With left preconditioning Krylov methods construct the solution in the > space {Bb, BABb, BABABb, ..} if the range of B contains entries in the null > space of A then the Krylov space will contain vectors in the null space of > A. What can happen is that in each iteration of the Krylov space those > entries grow and you end up with a solution x + xn where xn is in the null > space of A and very large, thus it looks like GMRES is not converging since > the solution "jump" each iteration. If you force the range of B to exclude > the null space of A, that is project out the null space of A after applying > B then nothing in the null space ever gets into the Krylov space and you > get the "minimum norm" solution to the least squares problem which is what > you almost always want. When you provide MatSetNullSpace() the KSP solvers > automatically remove the null space after applying B. > > All this stuff applies for any Krylov method. > > So based on my understanding, you should just provide the null space > that you can and forget out the transpose null space and use left > preconditioning with GMRES (forget about what I said about trying with > right preconditioning). Let GMRES iterate until the residual norm has > stabilized and use the resulting solution which is the least squares > solution. If you are using KSP inside SNES you may need to set > SNESSetMaxLinearSolveFailures() to a large value so it doesn't stop when it > thinks the GMRES has failed. > > Barry > > Matt and Jed, please check my logic I often flip my ranges/null spaces and > may have incorrect presentation above. > > > > > > > > > > > > 2) I have tried fixing the solution at an arbitrary point, and while > it generally works, for some problems I get numerical artifacts, e.g. > slight asymmetry in the solution and/or increased error close to the point > where I fix the solution. Is this, more or less, expected as a known > artifact? > > > > Yeah this approach is not good. We never recommend it. > > > > > > > > 3) An alternative to 2 is to enforce some global constraint on the > solution, e.g. to require that the average be zero. My question here is > two-fold: > > > > > > Requiring the average be zero is exactly the same as providing a > null space of the constant function. Saying the average is zero is the same > as saying the solution is orthogonal to the constant function. I don't see > any reason to introduce the Lagrange multiplier and all its complications > inside of just providing the constant null space. > > > > > > Is this also true at the discrete level when the matrix is > non-symmetric? I have always viewed this as just a constraint that could > really be anything. > > > > > > > > > > > 3-1) Is this generally any better than solution 2, in terms of not > messing too much with the condition number of the matrix? > > > > > > > > 3-2) I don't quite know how to implement this using PETSc. Generally > speaking I'd like to solve > > > > > > > > | A U | | X | | B | > > > > | | * | | = | | > > > > | U^T 0 | | s | | 0 | > > > > > > > > > > > > where U is a constant vector (of ones) and s is effectively a > Lagrange multiplier. I suspect I need to use MatCreateSchurComplement and > pass that to the KSP? Or do I need create my own matrix type from scratch > through MatCreateShell? > > > > > > > > Any help is appreciated! > > > > > > > > Thanks, > > > > Mohammad > > > > > > > > > > > > > > > > > > > > -- > > > Sent from Gmail Mobile > > > > > > > > -- > > Sent from Gmail Mobile > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Feb 25 17:05:44 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 25 Feb 2016 17:05:44 -0600 Subject: [petsc-users] Neumann BC with non-symmetric matrix In-Reply-To: References: <715E1F2D-F735-4419-811A-590E2218516F@mcs.anl.gov> Message-ID: <3798056C-7C20-459A-8AE3-A6768CC0301C@mcs.anl.gov> > On Feb 25, 2016, at 4:18 PM, Mohammad Mirzadeh wrote: > > Barry, > > Thanks a lot for the detailed discussion. Things make much more sense now, especially I was confused why the manual says to provide call both 'MatSetNullSpace' and 'MatSetTransposeNullSpace'. I have couple of more questions and some observations I have made since yesterday. > > 1) Is there a systematic way to tell KSP to stop when it stagnates? I am _not_ using SNES. No, I added the issue https://bitbucket.org/petsc/petsc/issues/122/ksp-convergence-test-for-inconsistent writing the code for a new test is pretty simple, but you need to decide mathematically how you are going to detect "stagnation". > > 2) Once KSP converges to the least-square solution, the residual must be in the nullspace of A^T because otherwise it would have been reduced by the solver. So is it (at lest theoretically) possible to use the residual vector as an approximate basis for the n(A^T)? In general this wouldn't be enough but I'm thinking since the nullspace is one-dimensional, maybe I could use the residual itself to manually project solution onto range of A after calling KSPSolve? I don't see why this wouldn't work. Just run one initial solve till stagnation and then use the resulting residual as the null space for A^T for future solves (with the same matrix, of course). It could be interesting to plot the residual to see what it looks like and it that makes sense physically > > 3) Are preconditoners applied to the left by default? If not which one are and which one are not? It actually depends on the KSP, some algorithms only support right preconditioning, some implementations only support one or the other (depending on who wrote it) and some support both. In PETSc CG, GMRES, and BiCGstab use left by default, both GMRES and BiCGstab also support right. > > 4) So then if I provide the nullspace of A, KSP residual should converge, correct? I have made a few observations: > > 4-1) When I provide the nullspace of A through MatSetNullSpace, I (generally) do see the residual (preconditioned) become very small (~1e-12 or so) but then it sometimes stagnates (say around 1e-10). Is this normal? There is only some much convergence one can expect for linear solvers; how far one can drive down the residual depends at least in part on the conditioning of the matrix. The higher the conditioning the less you can get in tolerance. > > 4-2) Another observation is that the true residual stagnates way earlier which I assume is an indication that the RHS is in fact not in the range of A. You can hope this is the case. Of course one cannot really know if the residual is stagnating due to an inconsistent right hand side or for some other reason. But if it stagnates at 10^-4 it is probably due to inconsistent right hand side if it stagnates at 10^-12 the right hand side is probably consistent. If it stagnates at 10^-8 then it could be due to inconsistent rhs and or some other reason. > > 4-3) Also, I've seen that the choice of KSP and PC have considerable effects on the solver. For instance, by default I use hypre+bcgs and I have noticed I need to change coarse-grid relaxation from Gaussian Elimination to symmetric-sor/Jacobi otherwise hypre has issues. I assume this is because the AMG preconditioner is also singular on the coarse level? Yes this is likely the reason. > > 4-4) Along the same lines, I tried a couple of other PCs such as {jacobi, sor, gamg, ilu} and none of them were able to converge with bcgs as the KSP. However, with gmres, almost all of them converge Sure this is expected > with the exception of gamg. > > 4-5) If I use gmres instead of bcgs, and for any of {jacobi, sor, ilu}, the true residual seems to be generally 2-3 orders of magnitude smaller (1e-6 vs 1e-3). I suppose this is just because gmres is more robust? Yes, with a single system I would recommend sticking with GMRES and avoiding Bcgs. > > 4-6) With hypre, the true residual is always larger (~1e-3) than other PCs no matter if I use bcgs or gmres. ok. > > Sorry for the long discussion but this has turned out to be quite educational for me! > > Thanks, > Mohammad > > On Wed, Feb 24, 2016 at 2:21 PM, Barry Smith wrote: > > > On Feb 24, 2016, at 9:07 AM, Mohammad Mirzadeh wrote: > > > > Barry, > > > > On Wednesday, February 24, 2016, Barry Smith wrote: > > > > > On Feb 24, 2016, at 12:07 AM, Mohammad Mirzadeh wrote: > > > > > > At the PDE level the compatibility condition is satisfied. However I suspect that at the discrete level the rhs is not not exactly in the range. The reason for my suspicion is that I know for a fact that my discretization is not conservative at the hanging nodes. > > Could be. > > > > > Thanks for the suggestion. I'll give it a try. Howerver, does GMRES fundamentally behave differently than BiCGSTAB for these systems? I have seen people saying that it can keep the solution in the range if rhs is already in the range but I thought any KSP would do the same? > > > Here is the deal. There are two issues here > > 1) Making sure that b is the in the range of A. If b is not in the range of A then it is obviously impossible to find an x such that A x = b. If we divide b into a part in the range of A called br and the rest call it bp then b = br + bp and one can solve Ax = br and the left over residual is bp. Normally if you run GMRES with an inconsistent right hand side (that is bp != 0) it will solve Ax = br automatically and thus give you a "least squares" answer which is likely what you want. These means in some sense you don't really need to worry about making sure that b is in the range of A. But the residuals of GMRES will stagnate, which makes sense because it cannot get rid of the bp part. In the least squares sense however GMRES has converged. If you provide MatSetTransposeNullSpace() then KSP automatically removes this space from b when it starts the Krylov method, this is nice because the GMRES residuals will go to zero. > > 2) Making sure that huge multiples of the null space of A do not get into the computed solution. > > With left preconditioning Krylov methods construct the solution in the space {Bb, BABb, BABABb, ..} if the range of B contains entries in the null space of A then the Krylov space will contain vectors in the null space of A. What can happen is that in each iteration of the Krylov space those entries grow and you end up with a solution x + xn where xn is in the null space of A and very large, thus it looks like GMRES is not converging since the solution "jump" each iteration. If you force the range of B to exclude the null space of A, that is project out the null space of A after applying B then nothing in the null space ever gets into the Krylov space and you get the "minimum norm" solution to the least squares problem which is what you almost always want. When you provide MatSetNullSpace() the KSP solvers automatically remove the null space after applying B. > > All this stuff applies for any Krylov method. > > So based on my understanding, you should just provide the null space that you can and forget out the transpose null space and use left preconditioning with GMRES (forget about what I said about trying with right preconditioning). Let GMRES iterate until the residual norm has stabilized and use the resulting solution which is the least squares solution. If you are using KSP inside SNES you may need to set SNESSetMaxLinearSolveFailures() to a large value so it doesn't stop when it thinks the GMRES has failed. > > Barry > > Matt and Jed, please check my logic I often flip my ranges/null spaces and may have incorrect presentation above. > > > > > > > > > > > > 2) I have tried fixing the solution at an arbitrary point, and while it generally works, for some problems I get numerical artifacts, e.g. slight asymmetry in the solution and/or increased error close to the point where I fix the solution. Is this, more or less, expected as a known artifact? > > > > Yeah this approach is not good. We never recommend it. > > > > > > > > 3) An alternative to 2 is to enforce some global constraint on the solution, e.g. to require that the average be zero. My question here is two-fold: > > > > > > Requiring the average be zero is exactly the same as providing a null space of the constant function. Saying the average is zero is the same as saying the solution is orthogonal to the constant function. I don't see any reason to introduce the Lagrange multiplier and all its complications inside of just providing the constant null space. > > > > > > Is this also true at the discrete level when the matrix is non-symmetric? I have always viewed this as just a constraint that could really be anything. > > > > > > > > > > > 3-1) Is this generally any better than solution 2, in terms of not messing too much with the condition number of the matrix? > > > > > > > > 3-2) I don't quite know how to implement this using PETSc. Generally speaking I'd like to solve > > > > > > > > | A U | | X | | B | > > > > | | * | | = | | > > > > | U^T 0 | | s | | 0 | > > > > > > > > > > > > where U is a constant vector (of ones) and s is effectively a Lagrange multiplier. I suspect I need to use MatCreateSchurComplement and pass that to the KSP? Or do I need create my own matrix type from scratch through MatCreateShell? > > > > > > > > Any help is appreciated! > > > > > > > > Thanks, > > > > Mohammad > > > > > > > > > > > > > > > > > > > > -- > > > Sent from Gmail Mobile > > > > > > > > -- > > Sent from Gmail Mobile > > From knepley at gmail.com Thu Feb 25 18:22:04 2016 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 25 Feb 2016 18:22:04 -0600 Subject: [petsc-users] Mass matrix with PetscFE In-Reply-To: <56CEB565.5010203@tf.uni-kiel.de> References: <56CDA67F.6000906@tf.uni-kiel.de> <56CDB469.806@tf.uni-kiel.de> <56CDB84F.8020309@tf.uni-kiel.de> <518cc2f74e6b2267660acaf3871d52f9@tf.uni-kiel.de> <56CEB565.5010203@tf.uni-kiel.de> Message-ID: I am sorry about the delay. I have your example working but it exposed a bug in Plex so I need to push the fix first. I should have everything for you early next week. Thanks Matt On Feb 25, 2016 2:04 AM, "Julian Andrej" wrote: > After a bit of rethinking the problem, the discrepancy between the size of > matrix A and the mass matrix M arises because of the Dirichlet boundary > conditions. So why aren't the BCs not imposed on the mass matrix? Do I need > to handle Dirichlet BCs differently in this context (like zero rows and put > one the diagonal?) > > On 24.02.2016 20:54, juan wrote: > >> I attached another example which creates the correct mass matrix >> but also overwrites the DM for the SNES solve. Somehow i cannot manage >> to really copy the DM to dm_mass and use that. If i try to do that with >> DMClone(dm, &dm_mass) i get a smaller mass matrix (which is not of size >> A). >> >> Maybe this helps in the discussion. >> >> Relevant code starts at line 455. >> >> On 2016-02-24 15:03, Julian Andrej wrote: >> >>> Thanks Matt, >>> >>> I attached the modified example. >>> >>> the corresponding code (and only changes to ex12) is starting at line >>> 832. >>> >>> It also seems that the mass matrix is of size 169x169 and the >>> stiffness matrix is of dimension 225x225. I'd assume that if i >>> multiply test and trial function i'd get a matrix of same size (if the >>> space/quadrature is the same for the stiffness matrix) >>> >>> On 24.02.2016 14:56, Matthew Knepley wrote: >>> >>>> On Wed, Feb 24, 2016 at 7:47 AM, Julian Andrej >>> > wrote: >>>> >>>> I'm now using the petsc git master branch. >>>> >>>> I tried adding my code to the ex12 >>>> >>>> DM dm_mass; >>>> PetscDS prob_mass; >>>> PetscFE fe; >>>> Mat M; >>>> PetscFECreateDefault(dm, user.dim, 1, PETSC_TRUE, NULL, -1, &fe); >>>> >>>> DMClone(dm, &dm_mass); >>>> DMGetDS(dm_mass, &prob_mass); >>>> PetscDSSetDiscretization(prob_mass, 0, (PetscObject) fe); >>>> PetscDSSetJacobian(prob_mass, 0, 0, mass_kernel, NULL, NULL, >>>> NULL); >>>> DMCreateMatrix(dm_mass, &M); >>>> >>>> MatSetOptionsPrefix(M, "M_";) >>>> >>>> and receive the error on running >>>> ./exe -interpolate -refinement_limit 0.0125 -petscspace_order 2 >>>> -M_mat_view binary >>>> >>>> WARNING! There are options you set that were not used! >>>> WARNING! could be spelling mistake, etc! >>>> Option left: name:-M_mat_view value: binary >>>> >>>> I don't know if the matrix is actually there and assembled or if the >>>> option is ommitted because something is wrong. >>>> >>>> >>>> Its difficult to know when I cannot see the whole code. You can always >>>> insert >>>> >>>> MatViewFromOptions(M, NULL, "-mat_view"); >>>> >>>> Using >>>> MatView(M, PETSC_VIEWER_STDOUT_WORLD); >>>> >>>> gives me a reasonable output to stdout. >>>> >>>> >>>> Good. >>>> >>>> But saving the matrix and analysing it in matlab, results in an all >>>> zero matrix. >>>> >>>> PetscViewerBinaryOpen(PETSC_COMM_WORLD, "Mout",FILE_MODE_WRITE, >>>> &viewer); >>>> MatView(M, viewer); >>>> >>>> >>>> I cannot explain this, but it has to be something like you are viewing >>>> the matrix before it is >>>> actually assembled. Feel free to send the code. It sounds like it is >>>> mostly working. >>>> >>>> Matt >>>> >>>> Any hints? >>>> >>>> >>>> On 24.02.2016 13 :58, Matthew Knepley wrote: >>>> >>>> On Wed, Feb 24, 2016 at 6:47 AM, Julian Andrej >>>> >>>> >> >>>> wrote: >>>> >>>> Hi, >>>> >>>> i'm trying to assemble a mass matrix with the >>>> PetscFE/DMPlex >>>> interface. I found something in the examples of TAO >>>> >>>> >>>> https://bitbucket.org/petsc/petsc/src/da8116b0e8d067e39fd79740a8a864b0fe207998/src/tao/examples/tutorials/ex3.c?at=master&fileviewer=file-view-default >>>> >>>> >>>> but using the lines >>>> >>>> DMClone(dm, &dm_mass); >>>> DMSetNumFields(dm_mass, 1); >>>> DMPlexCopyCoordinates(dm, dm_mass); >>>> DMGetDS(dm_mass, &prob_mass); >>>> PetscDSSetJacobian(prob_mass, 0, 0, mass_kernel, NULL, >>>> NULL, NULL); >>>> PetscDSSetDiscretization(prob_mass, 0, (PetscObject) fe); >>>> DMPlexSNESComputeJacobianFEM(dm_mass, u, M, M, NULL); >>>> DMCreateMatrix(dm_mass, &M); >>>> >>>> leads to errors in DMPlexSNESComputeJacobianFEM (u is a >>>> global vector). >>>> >>>> I don't can understand the necessary commands until >>>> DMPlexSNESComputeJacobianFEM. What does it do and why is it >>>> necessary? (especially why does the naming involve SNES?) >>>> >>>> Is there another/easier/better way to create a mass >>>> matrix (the >>>> inner product of the function space and the test space)? >>>> >>>> >>>> 1) That example needs updating. First, look at SNES ex12 which >>>> is up to >>>> date. >>>> >>>> 2) I assume you are using 3.6. If you use the development >>>> version, you >>>> can remove DMPlexCopyCoordinates(). >>>> >>>> 3) You need to create the matrix BEFORE calling the assembly >>>> >>>> 4) Always always always send the entire error messge >>>> >>>> Matt >>>> >>>> Regards >>>> Julian Andrej >>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results >>>> to which >>>> their experiments lead. >>>> -- Norbert Wiener >>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which >>>> their experiments lead. >>>> -- Norbert Wiener >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mirzadeh at gmail.com Thu Feb 25 20:32:19 2016 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Thu, 25 Feb 2016 21:32:19 -0500 Subject: [petsc-users] Neumann BC with non-symmetric matrix In-Reply-To: <3798056C-7C20-459A-8AE3-A6768CC0301C@mcs.anl.gov> References: <715E1F2D-F735-4419-811A-590E2218516F@mcs.anl.gov> <3798056C-7C20-459A-8AE3-A6768CC0301C@mcs.anl.gov> Message-ID: Thanks a lot Barry. This was very helpful :) On Thu, Feb 25, 2016 at 6:05 PM, Barry Smith wrote: > > > On Feb 25, 2016, at 4:18 PM, Mohammad Mirzadeh > wrote: > > > > Barry, > > > > Thanks a lot for the detailed discussion. Things make much more sense > now, especially I was confused why the manual says to provide call both > 'MatSetNullSpace' and 'MatSetTransposeNullSpace'. I have couple of more > questions and some observations I have made since yesterday. > > > > 1) Is there a systematic way to tell KSP to stop when it stagnates? I am > _not_ using SNES. > > No, I added the issue > https://bitbucket.org/petsc/petsc/issues/122/ksp-convergence-test-for-inconsistent > writing the code for a new test is pretty simple, but you need to decide > mathematically how you are going to detect "stagnation". > > > > > 2) Once KSP converges to the least-square solution, the residual must be > in the nullspace of A^T because otherwise it would have been reduced by the > solver. So is it (at lest theoretically) possible to use the residual > vector as an approximate basis for the n(A^T)? In general this wouldn't be > enough but I'm thinking since the nullspace is one-dimensional, maybe I > could use the residual itself to manually project solution onto range of A > after calling KSPSolve? > > I don't see why this wouldn't work. Just run one initial solve till > stagnation and then use the resulting residual as the null space for A^T > for future solves (with the same matrix, of course). It could be > interesting to plot the residual to see what it looks like and it that > makes sense physically > > > > > 3) Are preconditoners applied to the left by default? If not which one > are and which one are not? > > It actually depends on the KSP, some algorithms only support right > preconditioning, some implementations only support one or the other > (depending on who wrote it) and some support both. In PETSc CG, GMRES, and > BiCGstab use left by default, both GMRES and BiCGstab also support right. > > > > > 4) So then if I provide the nullspace of A, KSP residual should > converge, correct? I have made a few observations: > > > > 4-1) When I provide the nullspace of A through MatSetNullSpace, I > (generally) do see the residual (preconditioned) become very small (~1e-12 > or so) but then it sometimes stagnates (say around 1e-10). Is this normal? > > There is only some much convergence one can expect for linear solvers; > how far one can drive down the residual depends at least in part on the > conditioning of the matrix. The higher the conditioning the less you can > get in tolerance. > > > > > > 4-2) Another observation is that the true residual stagnates way > earlier which I assume is an indication that the RHS is in fact not in the > range of A. > > You can hope this is the case. > > Of course one cannot really know if the residual is stagnating due to > an inconsistent right hand side or for some other reason. But if it > stagnates at 10^-4 it is probably due to inconsistent right hand side if it > stagnates at 10^-12 the right hand side is probably consistent. If it > stagnates at 10^-8 then it could be due to inconsistent rhs and or some > other reason. > > > > 4-3) Also, I've seen that the choice of KSP and PC have considerable > effects on the solver. For instance, by default I use hypre+bcgs and I have > noticed I need to change coarse-grid relaxation from Gaussian Elimination > to symmetric-sor/Jacobi otherwise hypre has issues. I assume this is > because the AMG preconditioner is also singular on the coarse level? > > Yes this is likely the reason. > > > > 4-4) Along the same lines, I tried a couple of other PCs such as > {jacobi, sor, gamg, ilu} and none of them were able to converge with bcgs > as the KSP. However, with gmres, almost all of them converge > > Sure this is expected > > > with the exception of gamg. > > > > 4-5) If I use gmres instead of bcgs, and for any of {jacobi, sor, > ilu}, the true residual seems to be generally 2-3 orders of magnitude > smaller (1e-6 vs 1e-3). I suppose this is just because gmres is more robust? > > Yes, with a single system I would recommend sticking with GMRES and > avoiding Bcgs. > > > > 4-6) With hypre, the true residual is always larger (~1e-3) than > other PCs no matter if I use bcgs or gmres. > > ok. > > > > > Sorry for the long discussion but this has turned out to be quite > educational for me! > > > > Thanks, > > Mohammad > > > > On Wed, Feb 24, 2016 at 2:21 PM, Barry Smith wrote: > > > > > On Feb 24, 2016, at 9:07 AM, Mohammad Mirzadeh > wrote: > > > > > > Barry, > > > > > > On Wednesday, February 24, 2016, Barry Smith > wrote: > > > > > > > On Feb 24, 2016, at 12:07 AM, Mohammad Mirzadeh > wrote: > > > > > > > > > At the PDE level the compatibility condition is satisfied. However I > suspect that at the discrete level the rhs is not not exactly in the range. > The reason for my suspicion is that I know for a fact that my > discretization is not conservative at the hanging nodes. > > > > Could be. > > > > > > > > Thanks for the suggestion. I'll give it a try. Howerver, does GMRES > fundamentally behave differently than BiCGSTAB for these systems? I have > seen people saying that it can keep the solution in the range if rhs is > already in the range but I thought any KSP would do the same? > > > > > > Here is the deal. There are two issues here > > > > 1) Making sure that b is the in the range of A. If b is not in the > range of A then it is obviously impossible to find an x such that A x = b. > If we divide b into a part in the range of A called br and the rest call it > bp then b = br + bp and one can solve Ax = br and the left over > residual is bp. Normally if you run GMRES with an inconsistent right hand > side (that is bp != 0) it will solve Ax = br automatically and thus give > you a "least squares" answer which is likely what you want. These means in > some sense you don't really need to worry about making sure that b is in > the range of A. But the residuals of GMRES will stagnate, which makes sense > because it cannot get rid of the bp part. In the least squares sense > however GMRES has converged. If you provide MatSetTransposeNullSpace() then > KSP automatically removes this space from b when it starts the Krylov > method, this is nice because the GMRES residuals will go to zero. > > > > 2) Making sure that huge multiples of the null space of A do not get > into the computed solution. > > > > With left preconditioning Krylov methods construct the solution in the > space {Bb, BABb, BABABb, ..} if the range of B contains entries in the null > space of A then the Krylov space will contain vectors in the null space of > A. What can happen is that in each iteration of the Krylov space those > entries grow and you end up with a solution x + xn where xn is in the null > space of A and very large, thus it looks like GMRES is not converging since > the solution "jump" each iteration. If you force the range of B to exclude > the null space of A, that is project out the null space of A after applying > B then nothing in the null space ever gets into the Krylov space and you > get the "minimum norm" solution to the least squares problem which is what > you almost always want. When you provide MatSetNullSpace() the KSP solvers > automatically remove the null space after applying B. > > > > All this stuff applies for any Krylov method. > > > > So based on my understanding, you should just provide the null space > that you can and forget out the transpose null space and use left > preconditioning with GMRES (forget about what I said about trying with > right preconditioning). Let GMRES iterate until the residual norm has > stabilized and use the resulting solution which is the least squares > solution. If you are using KSP inside SNES you may need to set > SNESSetMaxLinearSolveFailures() to a large value so it doesn't stop when it > thinks the GMRES has failed. > > > > Barry > > > > Matt and Jed, please check my logic I often flip my ranges/null spaces > and may have incorrect presentation above. > > > > > > > > > > > > > > > > 2) I have tried fixing the solution at an arbitrary point, and > while it generally works, for some problems I get numerical artifacts, e.g. > slight asymmetry in the solution and/or increased error close to the point > where I fix the solution. Is this, more or less, expected as a known > artifact? > > > > > > Yeah this approach is not good. We never recommend it. > > > > > > > > > > 3) An alternative to 2 is to enforce some global constraint on the > solution, e.g. to require that the average be zero. My question here is > two-fold: > > > > > > > > Requiring the average be zero is exactly the same as providing a > null space of the constant function. Saying the average is zero is the same > as saying the solution is orthogonal to the constant function. I don't see > any reason to introduce the Lagrange multiplier and all its complications > inside of just providing the constant null space. > > > > > > > > Is this also true at the discrete level when the matrix is > non-symmetric? I have always viewed this as just a constraint that could > really be anything. > > > > > > > > > > > > > > 3-1) Is this generally any better than solution 2, in terms of not > messing too much with the condition number of the matrix? > > > > > > > > > > 3-2) I don't quite know how to implement this using PETSc. > Generally speaking I'd like to solve > > > > > > > > > > | A U | | X | | B | > > > > > | | * | | = | | > > > > > | U^T 0 | | s | | 0 | > > > > > > > > > > > > > > > where U is a constant vector (of ones) and s is effectively a > Lagrange multiplier. I suspect I need to use MatCreateSchurComplement and > pass that to the KSP? Or do I need create my own matrix type from scratch > through MatCreateShell? > > > > > > > > > > Any help is appreciated! > > > > > > > > > > Thanks, > > > > > Mohammad > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Sent from Gmail Mobile > > > > > > > > > > > > -- > > > Sent from Gmail Mobile > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bhatiamanav at gmail.com Thu Feb 25 22:31:09 2016 From: bhatiamanav at gmail.com (Manav Bhatia) Date: Thu, 25 Feb 2016 22:31:09 -0600 Subject: [petsc-users] compilation on mac Message-ID: Hi, I am attempting to compile petsc 3.6.3 on my mac with the following options: ./configure --prefix=${PWD}/../ --CC=mpicc --CXX=mpicxx --FC=mpif90 --with-clanguage=c++ --with-fortran=0 --with-mpi-include=/opt/local/include/openmpi-mp --with-mpiexec=/opt/local/bin/mpiexec -with-mpi-lib=[/opt/local/lib/openmpi-mp/libmpi_cxx.dylib,/opt/local/lib/openmpi-mp/libmpi.dylib] --with-shared-libraries=1 --with-x=1 --with-x-dir=/opt/X11 --with-debugging=0 --with-lapack-lib=/usr/lib/liblapack.dylib --with-blas-lib=/usr/lib/libblas.dylib --download-superlu=yes --download-superlu_dist=yes --download-suitesparse=yes --download-mumps=yes --download-scalapack=yes --download-parmetis=yes --download-parmetis-shared=1 --download-metis=yes --download-metis-shared=1 --download-hypre=yes --download-hypre-shared=1 --download-ml=yes --download-ml-shared=1 --download-sundials=yes --download-sundials-shared=1 The configuration script is putting oddly named libraries in hypre and sundials files in the directory arch-darwin-cxx-opt/lib/petsc/conf in hypre: --with-MPI-libs="mpi_cxx.dyl mpi.dyl" in sundials: --with-mpi-libs="-lmpi_cxx -lmpi_cxx.dyl -lmpi -lmpi.dyl" This is creating problems elsewhere in compilation. Are there ways to get around this? Thanks, Manav -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Feb 25 22:37:28 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 25 Feb 2016 22:37:28 -0600 Subject: [petsc-users] compilation on mac In-Reply-To: References: Message-ID: <89BEECEE-C137-46B7-A665-F0931D41CF52@mcs.anl.gov> Since you are passing mpicc, mpicxx, mpif90 arguments, do NOT pass --with-mpi-include and --with-mpi-lib If the problem persists after rebuilding with this change. Send configure.log make.log and $PETSC_ARCH/lib/petsc/conf/petscvariables. Barry > On Feb 25, 2016, at 10:31 PM, Manav Bhatia wrote: > > Hi, > > I am attempting to compile petsc 3.6.3 on my mac with the following options: > > ./configure --prefix=${PWD}/../ --CC=mpicc --CXX=mpicxx --FC=mpif90 --with-clanguage=c++ --with-fortran=0 --with-mpi-include=/opt/local/include/openmpi-mp --with-mpiexec=/opt/local/bin/mpiexec -with-mpi-lib=[/opt/local/lib/openmpi-mp/libmpi_cxx.dylib,/opt/local/lib/openmpi-mp/libmpi.dylib] --with-shared-libraries=1 --with-x=1 --with-x-dir=/opt/X11 --with-debugging=0 --with-lapack-lib=/usr/lib/liblapack.dylib --with-blas-lib=/usr/lib/libblas.dylib --download-superlu=yes --download-superlu_dist=yes --download-suitesparse=yes --download-mumps=yes --download-scalapack=yes --download-parmetis=yes --download-parmetis-shared=1 --download-metis=yes --download-metis-shared=1 --download-hypre=yes --download-hypre-shared=1 --download-ml=yes --download-ml-shared=1 --download-sundials=yes --download-sundials-shared=1 > > The configuration script is putting oddly named libraries in hypre and sundials files in the directory arch-darwin-cxx-opt/lib/petsc/conf > > in hypre: --with-MPI-libs="mpi_cxx.dyl mpi.dyl" > > in sundials: --with-mpi-libs="-lmpi_cxx -lmpi_cxx.dyl -lmpi -lmpi.dyl" > > > This is creating problems elsewhere in compilation. > > Are there ways to get around this? > > Thanks, > Manav > > From zonexo at gmail.com Fri Feb 26 01:14:40 2016 From: zonexo at gmail.com (TAY wee-beng) Date: Fri, 26 Feb 2016 15:14:40 +0800 Subject: [petsc-users] Error - Out of memory. This could be due to allocating too large an object or bleeding by not properly ... In-Reply-To: <43AC24CD-4727-4C4C-B016-6927CD206425@mcs.anl.gov> References: <56CD0C7C.5060101@gmail.com> <56CD61C7.3040200@gmail.com> <56CDC961.40004@gmail.com> <56CDCC13.4070807@gmail.com> <56CEA03A.2000407@gmail.com> <43AC24CD-4727-4C4C-B016-6927CD206425@mcs.anl.gov> Message-ID: <56CFFB60.7020702@gmail.com> On 26/2/2016 1:56 AM, Barry Smith wrote: > Run a much smaller problem for a few time steps, making sure you free all the objects at the end, with the option -malloc_dump this will print all the memory that was not freed and hopefully help you track down which objects you forgot to free. > > Barry Hi, I run a smaller problem and lots of things are shown in the log. How can I know which exactly are not freed from the memory? Is this info helpful? Or should I run in a single core? Thanks >> On Feb 25, 2016, at 12:33 AM, TAY wee-beng wrote: >> >> Hi, >> >> I ran the code and it hangs again. However, adding -malloc_test doesn't seem to do any thing. The output (attached) is the same w/o it. >> >> Wonder if there's anything else I can do. >> Thank you >> >> Yours sincerely, >> >> TAY wee-beng >> >> On 24/2/2016 11:33 PM, Matthew Knepley wrote: >>> On Wed, Feb 24, 2016 at 9:28 AM, TAY wee-beng wrote: >>> >>> On 24/2/2016 11:18 PM, Matthew Knepley wrote: >>>> On Wed, Feb 24, 2016 at 9:16 AM, TAY wee-beng wrote: >>>> >>>> On 24/2/2016 9:12 PM, Matthew Knepley wrote: >>>>> On Wed, Feb 24, 2016 at 1:54 AM, TAY wee-beng wrote: >>>>> >>>>> On 24/2/2016 10:28 AM, Matthew Knepley wrote: >>>>>> On Tue, Feb 23, 2016 at 7:50 PM, TAY wee-beng wrote: >>>>>> Hi, >>>>>> >>>>>> I got this error (also attached, full) when running my code. It happens after a few thousand time steps. >>>>>> >>>>>> The strange thing is that for 2 different clusters, it stops at 2 different time steps. >>>>>> >>>>>> I wonder if it's related to DM since this happens after I added DM into my code. >>>>>> >>>>>> In this case, how can I find out the error? I'm thinking valgrind may take very long and gives too many false errors. >>>>>> >>>>>> It is very easy to find leaks. You just run a few steps with -malloc_dump and see what is left over. >>>>>> >>>>>> Matt >>>>> Hi Matt, >>>>> >>>>> Do you mean running my a.out with the -malloc_dump and stop after a few time steps? >>>>> >>>>> What and how should I "see" then? >>>>> >>>>> -malloc_dump outputs all unfreed memory to the screen after PetscFinalize(), so you should see the leak. >>>>> I guess it might be possible to keep creating things that you freed all at once at the end, but that is less likely. >>>>> >>>>> Matt >>>>> >>>> Hi, >>>> >>>> I got the output. I have zipped it since it's rather big. So it seems to be from DM routines but can you help me where the error is from? >>>> >>>> Its really hard to tell by looking at it. What I do is remove things until there is no leak, then progressively >>>> put thing back in until I have the culprit. Then you can think about what is not destroyed. >>>> >>>> Matt >>> Ok so let me get this clear. When it shows: >>> >>> [21]Total space allocated 1728961264 bytes >>> [21]1861664 bytes MatCheckCompressedRow() line 60 in /home/wtay/Codes/petsc-3.6.3/src/mat/utils/compressedrow.c >>> [21]16 bytes PetscStrallocpy() line 188 in /home/wtay/Codes/petsc-3.6.3/src/sys/utils/str.c >>> [21]624 bytes ISLocalToGlobalMappingCreate() line 270 in /home/wtay/Codes >>> >>> .... >>> >>> Does it mean that it's simply allocating space ie normal? Or does it show that there's memory leak ie error? >>> >>> I gave the wrong option. That dumps everything. Lets just look at the leaks with -malloc_test. >>> >>> Sorry about that, >>> >>> Matt >>> >>> If it's error, should I zoom in and debug around this time at this region? >>> >>> Thanks >>>> >>>> Thanks. >>>>> >>>>>> -- >>>>>> Thank you >>>>>> >>>>>> Yours sincerely, >>>>>> >>>>>> TAY wee-beng >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>> -- Norbert Wiener >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>> -- Norbert Wiener >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>> -- Norbert Wiener >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >> -------------- next part -------------- A non-text attachment was scrubbed... Name: log.7z Type: application/octet-stream Size: 28945 bytes Desc: not available URL: From rongliang.chan at gmail.com Fri Feb 26 01:59:38 2016 From: rongliang.chan at gmail.com (Rongliang Chen) Date: Fri, 26 Feb 2016 15:59:38 +0800 Subject: [petsc-users] DMRefine fails In-Reply-To: References: Message-ID: <56D005EA.2070406@gmail.com> Dear Lawrence, Thank you very much for you suggestions. I said "totally wrong" because I found that the connectivity and number of points of the refined mesh is wrong. I viewed the refined mesh with paraview and found that there are some holes in it (see the attached figures: Fig1.png for the non-overlapping case and Fig2.png for the overlapping case). My package is based on petsc-3.5 which does not have DMPlexDistributeOverlap. Petsc-3.5 only has DMPlexEnlargePartition and it seems can not add overlap for a distributed mesh. So I need to update my package to petsc-3.6 before try your suggestions. It will takes me some time to do it since my package is a little too large. Best regards, Rongliang > ------------------------------ > > Message: 2 > Date: Thu, 25 Feb 2016 10:47:22 +0000 > From: Lawrence Mitchell > To:petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] DMRefine fails > Message-ID:<56CEDBBA.7040309 at imperial.ac.uk> > Content-Type: text/plain; charset="windows-1252" > > > > On 25/02/16 06:19, Rongliang Chen wrote: >> >Dear All, >> > >> >In my application, I need an overlapping partition of the mesh, that is >> >to set the overlapping size in DMPlexDistribute(*dm, partitioner, >> >overlap, NULL, &distributedMesh) to be nonzero (overlap=1). >> > >> >At the same time, I want to use the DMRefine to refine the distributed >> >mesh. I found that the refined mesh obtained by DMRefine is totally >> >wrong. > What do you mean by "totally wrong"? Is it just that the overlapped > region on the coarse mesh is also refined (such that rather than being > one level deep, it is 2**nrefinements deep)? > >> >But if I set the overlapping size in DMPlexDistribute to be zero, >> >then the refined mesh is correct. Please let me know if you have any >> >ideas to use the DMRefine for a distributed mesh with nonzero >> >overlapping size. > I do this as follows: > > - Create the initial DM. > > - Distribute with zero overlap DMPlexDistribute(..., overlap=0, ...); > > - Refine the distributed DM as many times as you want. > > - Grow the overlap on the refined DM: > > DMPlexDistributeOverlap(refined_dm, 1, &pointsf, &overlapped_dm); > > Now the overlapped_dm is refined and has a 1-deep overlap. > > The only thing that, AFAIK, doesn't work out of the box with this > scheme is the automatic computation of multigrid restriction and > interpolation. The reason for this is that the code currently assumes > that, given a coarse DM point, it can compute the corresponding fine > DM cells by using: > > for r in range(num_sub_cells): > fine_cell = coarse_cell*num_sub_cells + r > > There is a TODO comment in plex.c (in DMPlexMatSetClosureRefined): > > for (r = 0, q = 0; r < numSubcells; ++r) { > /* TODO Map from coarse to fine cells */ > ierr = DMPlexGetTransitiveClosure(dmf, point*numSubcells + r, > PETSC_TRUE, &numFPoints, &fpoints);CHKERRQ(ierr); > > > The two-step refine then grow the overlap method I outlined above > destroys this mapping. Hence the approach currently coded in plex.c > will not work. > > This is for regular refinement. For non-nested refinement, I think > you're good because the computation of the interpolator spins over the > fine cells and searches for the coarse cells. > > Hope that helps! > > Lawrence -------------- next part -------------- A non-text attachment was scrubbed... Name: Fig1.png Type: image/png Size: 86051 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Fig2.png Type: image/png Size: 115932 bytes Desc: not available URL: From mfadams at lbl.gov Fri Feb 26 07:12:52 2016 From: mfadams at lbl.gov (Mark Adams) Date: Fri, 26 Feb 2016 08:12:52 -0500 Subject: [petsc-users] Neumann BC with non-symmetric matrix In-Reply-To: References: <715E1F2D-F735-4419-811A-590E2218516F@mcs.anl.gov> Message-ID: > > > 4-4) Along the same lines, I tried a couple of other PCs such as > {jacobi, sor, gamg, ilu} and none of them were able to converge with bcgs > as the KSP. However, with gmres, almost all of them converge with the > exception of gamg. > Note, I'm not sure why you need the null space of A^T, you want the null space of A. And for singular systems like yours you need to use a pseudo inverse of the coarse grid because it is singular -- if you represent the null space exactly. GAMG is use for AMR problems like this a lot in BISICLES. You need to use an 'svd' coarse grid solver, or an appropriate iterative solver. LU is the default. Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Feb 26 07:21:51 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 26 Feb 2016 07:21:51 -0600 Subject: [petsc-users] Error - Out of memory. This could be due to allocating too large an object or bleeding by not properly ... In-Reply-To: <56CFFB60.7020702@gmail.com> References: <56CD0C7C.5060101@gmail.com> <56CD61C7.3040200@gmail.com> <56CDC961.40004@gmail.com> <56CDCC13.4070807@gmail.com> <56CEA03A.2000407@gmail.com> <43AC24CD-4727-4C4C-B016-6927CD206425@mcs.anl.gov> <56CFFB60.7020702@gmail.com> Message-ID: <256DA688-D052-4F80-AD4F-06B8295FED32@mcs.anl.gov> > On Feb 26, 2016, at 1:14 AM, TAY wee-beng wrote: > > > On 26/2/2016 1:56 AM, Barry Smith wrote: >> Run a much smaller problem for a few time steps, making sure you free all the objects at the end, with the option -malloc_dump this will print all the memory that was not freed and hopefully help you track down which objects you forgot to free. >> >> Barry > Hi, > > I run a smaller problem and lots of things are shown in the log. How can I know which exactly are not freed from the memory? Everything in in the log represents unfreed memory. You need to hunt through all the objects you create and make sure you destroy all of them. Barry > > Is this info helpful? Or should I run in a single core? > > Thanks >>> On Feb 25, 2016, at 12:33 AM, TAY wee-beng wrote: >>> >>> Hi, >>> >>> I ran the code and it hangs again. However, adding -malloc_test doesn't seem to do any thing. The output (attached) is the same w/o it. >>> >>> Wonder if there's anything else I can do. >>> Thank you >>> >>> Yours sincerely, >>> >>> TAY wee-beng >>> >>> On 24/2/2016 11:33 PM, Matthew Knepley wrote: >>>> On Wed, Feb 24, 2016 at 9:28 AM, TAY wee-beng wrote: >>>> >>>> On 24/2/2016 11:18 PM, Matthew Knepley wrote: >>>>> On Wed, Feb 24, 2016 at 9:16 AM, TAY wee-beng wrote: >>>>> >>>>> On 24/2/2016 9:12 PM, Matthew Knepley wrote: >>>>>> On Wed, Feb 24, 2016 at 1:54 AM, TAY wee-beng wrote: >>>>>> >>>>>> On 24/2/2016 10:28 AM, Matthew Knepley wrote: >>>>>>> On Tue, Feb 23, 2016 at 7:50 PM, TAY wee-beng wrote: >>>>>>> Hi, >>>>>>> >>>>>>> I got this error (also attached, full) when running my code. It happens after a few thousand time steps. >>>>>>> >>>>>>> The strange thing is that for 2 different clusters, it stops at 2 different time steps. >>>>>>> >>>>>>> I wonder if it's related to DM since this happens after I added DM into my code. >>>>>>> >>>>>>> In this case, how can I find out the error? I'm thinking valgrind may take very long and gives too many false errors. >>>>>>> >>>>>>> It is very easy to find leaks. You just run a few steps with -malloc_dump and see what is left over. >>>>>>> >>>>>>> Matt >>>>>> Hi Matt, >>>>>> >>>>>> Do you mean running my a.out with the -malloc_dump and stop after a few time steps? >>>>>> >>>>>> What and how should I "see" then? >>>>>> >>>>>> -malloc_dump outputs all unfreed memory to the screen after PetscFinalize(), so you should see the leak. >>>>>> I guess it might be possible to keep creating things that you freed all at once at the end, but that is less likely. >>>>>> >>>>>> Matt >>>>>> >>>>> Hi, >>>>> >>>>> I got the output. I have zipped it since it's rather big. So it seems to be from DM routines but can you help me where the error is from? >>>>> >>>>> Its really hard to tell by looking at it. What I do is remove things until there is no leak, then progressively >>>>> put thing back in until I have the culprit. Then you can think about what is not destroyed. >>>>> >>>>> Matt >>>> Ok so let me get this clear. When it shows: >>>> >>>> [21]Total space allocated 1728961264 bytes >>>> [21]1861664 bytes MatCheckCompressedRow() line 60 in /home/wtay/Codes/petsc-3.6.3/src/mat/utils/compressedrow.c >>>> [21]16 bytes PetscStrallocpy() line 188 in /home/wtay/Codes/petsc-3.6.3/src/sys/utils/str.c >>>> [21]624 bytes ISLocalToGlobalMappingCreate() line 270 in /home/wtay/Codes >>>> >>>> .... >>>> >>>> Does it mean that it's simply allocating space ie normal? Or does it show that there's memory leak ie error? >>>> >>>> I gave the wrong option. That dumps everything. Lets just look at the leaks with -malloc_test. >>>> >>>> Sorry about that, >>>> >>>> Matt >>>> If it's error, should I zoom in and debug around this time at this region? >>>> >>>> Thanks >>>>> Thanks. >>>>>> >>>>>>> -- >>>>>>> Thank you >>>>>>> >>>>>>> Yours sincerely, >>>>>>> >>>>>>> TAY wee-beng >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>> -- Norbert Wiener >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>> -- Norbert Wiener >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>> -- Norbert Wiener >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>> -- Norbert Wiener >>> > > From zonexo at gmail.com Fri Feb 26 09:28:22 2016 From: zonexo at gmail.com (TAY wee-beng) Date: Fri, 26 Feb 2016 23:28:22 +0800 Subject: [petsc-users] Investigate parallel code to improve parallelism Message-ID: <56D06F16.9000200@gmail.com> Hi, I have got a 3D code. When I ran with 48 procs and 11 million cells, it runs for 83 min. When I ran with 96 procs and 22 million cells, it ran for 99 min. So it's not that parallel. I want to find out which part of the code I need to improve. Also if PETsc and hypre is working well in parallel. What's the best way to do it? I thought of doing profiling but if the code is optimized, I wonder if it still works well. -- Thank you. Yours sincerely, TAY wee-beng From bsmith at mcs.anl.gov Fri Feb 26 09:32:25 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 26 Feb 2016 09:32:25 -0600 Subject: [petsc-users] Investigate parallel code to improve parallelism In-Reply-To: <56D06F16.9000200@gmail.com> References: <56D06F16.9000200@gmail.com> Message-ID: <141087E6-A7C1-41FF-B3E8-74FF587E3535@mcs.anl.gov> > On Feb 26, 2016, at 9:28 AM, TAY wee-beng wrote: > > Hi, > > I have got a 3D code. When I ran with 48 procs and 11 million cells, it runs for 83 min. When I ran with 96 procs and 22 million cells, it ran for 99 min. This is actually pretty good! > > So it's not that parallel. I want to find out which part of the code I need to improve. Also if PETsc and hypre is working well in parallel. What's the best way to do it? Run both with -log_summary and send the output for each case. This will show where the time is being spent and which parts are scaling less well. Barry > > I thought of doing profiling but if the code is optimized, I wonder if it still works well. > > -- > Thank you. > > Yours sincerely, > > TAY wee-beng > From zonexo at gmail.com Fri Feb 26 10:27:56 2016 From: zonexo at gmail.com (TAY wee-beng) Date: Sat, 27 Feb 2016 00:27:56 +0800 Subject: [petsc-users] Investigate parallel code to improve parallelism In-Reply-To: <141087E6-A7C1-41FF-B3E8-74FF587E3535@mcs.anl.gov> References: <56D06F16.9000200@gmail.com> <141087E6-A7C1-41FF-B3E8-74FF587E3535@mcs.anl.gov> Message-ID: <56D07D0C.7050109@gmail.com> On 26/2/2016 11:32 PM, Barry Smith wrote: >> On Feb 26, 2016, at 9:28 AM, TAY wee-beng wrote: >> >> Hi, >> >> I have got a 3D code. When I ran with 48 procs and 11 million cells, it runs for 83 min. When I ran with 96 procs and 22 million cells, it ran for 99 min. > This is actually pretty good! But if I'm not wrong, if I increase the no. of cells, the parallelism will keep on decreasing. I hope it scales up to maybe 300 - 400 procs. >> So it's not that parallel. I want to find out which part of the code I need to improve. Also if PETsc and hypre is working well in parallel. What's the best way to do it? > Run both with -log_summary and send the output for each case. This will show where the time is being spent and which parts are scaling less well. > > Barry That's only for the PETSc part, right? So for other parts of the code, including hypre part, I will not be able to find out. If so, what can I use to check these parts? >> I thought of doing profiling but if the code is optimized, I wonder if it still works well. >> >> -- >> Thank you. >> >> Yours sincerely, >> >> TAY wee-beng >> From knepley at gmail.com Fri Feb 26 10:41:31 2016 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 26 Feb 2016 10:41:31 -0600 Subject: [petsc-users] Investigate parallel code to improve parallelism In-Reply-To: <56D07D0C.7050109@gmail.com> References: <56D06F16.9000200@gmail.com> <141087E6-A7C1-41FF-B3E8-74FF587E3535@mcs.anl.gov> <56D07D0C.7050109@gmail.com> Message-ID: On Fri, Feb 26, 2016 at 10:27 AM, TAY wee-beng wrote: > > On 26/2/2016 11:32 PM, Barry Smith wrote: > >> On Feb 26, 2016, at 9:28 AM, TAY wee-beng wrote: >>> >>> Hi, >>> >>> I have got a 3D code. When I ran with 48 procs and 11 million cells, it >>> runs for 83 min. When I ran with 96 procs and 22 million cells, it ran for >>> 99 min. >>> >> This is actually pretty good! >> > But if I'm not wrong, if I increase the no. of cells, the parallelism will > keep on decreasing. I hope it scales up to maybe 300 - 400 procs. > 100% efficiency does not happen, and here you get 83%. You could probably get that into the 90s depending on your algorithm, but this is pretty good. Matt > So it's not that parallel. I want to find out which part of the code I >>> need to improve. Also if PETsc and hypre is working well in parallel. >>> What's the best way to do it? >>> >> Run both with -log_summary and send the output for each case. This >> will show where the time is being spent and which parts are scaling less >> well. >> >> Barry >> > That's only for the PETSc part, right? So for other parts of the code, > including hypre part, I will not be able to find out. If so, what can I use > to check these parts? > >> I thought of doing profiling but if the code is optimized, I wonder if it >>> still works well. >>> >>> -- >>> Thank you. >>> >>> Yours sincerely, >>> >>> TAY wee-beng >>> >>> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Feb 26 10:53:16 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 26 Feb 2016 10:53:16 -0600 Subject: [petsc-users] Investigate parallel code to improve parallelism In-Reply-To: <56D07D0C.7050109@gmail.com> References: <56D06F16.9000200@gmail.com> <141087E6-A7C1-41FF-B3E8-74FF587E3535@mcs.anl.gov> <56D07D0C.7050109@gmail.com> Message-ID: <5FBB830C-4A5E-40B2-9845-E0DC68B02BD3@mcs.anl.gov> > On Feb 26, 2016, at 10:27 AM, TAY wee-beng wrote: > > > On 26/2/2016 11:32 PM, Barry Smith wrote: >>> On Feb 26, 2016, at 9:28 AM, TAY wee-beng wrote: >>> >>> Hi, >>> >>> I have got a 3D code. When I ran with 48 procs and 11 million cells, it runs for 83 min. When I ran with 96 procs and 22 million cells, it ran for 99 min. >> This is actually pretty good! > But if I'm not wrong, if I increase the no. of cells, the parallelism will keep on decreasing. I hope it scales up to maybe 300 - 400 procs. >>> So it's not that parallel. I want to find out which part of the code I need to improve. Also if PETsc and hypre is working well in parallel. What's the best way to do it? >> Run both with -log_summary and send the output for each case. This will show where the time is being spent and which parts are scaling less well. >> >> Barry > That's only for the PETSc part, right? So for other parts of the code, including hypre part, I will not be able to find out. If so, what can I use to check these parts? You will still be able to see what percentage of the time is spent in hypre and if it increases with the problem size and how much. So the information will still be useful. Barry >>> I thought of doing profiling but if the code is optimized, I wonder if it still works well. >>> >>> -- >>> Thank you. >>> >>> Yours sincerely, >>> >>> TAY wee-beng >>> > From bhatiamanav at gmail.com Fri Feb 26 11:01:07 2016 From: bhatiamanav at gmail.com (Manav Bhatia) Date: Fri, 26 Feb 2016 11:01:07 -0600 Subject: [petsc-users] Blocked matrix storage Message-ID: <91C08E9D-8A81-42A7-91C3-401BF93166A2@gmail.com> Hi, I am working on a problem with system of equations in complex numbers, and am splitting it up to solve it with petsc compiled for real number support: (J_R + i J_I) (x_R + i x_I) + (r_R + i r_I) = 0 is then defined as [ J_R -J_I] {x_R} + {r_R} = {0} [ J_I J_R] {x_I} + {r_I} = {0} I have tried block-Schur solver with the Jacobian defined using the nested matrix format with a separate matrix for each quadrant of the Jacobian. This works well for diagonally dominant cases, but for high-frequency problems (J_I scales with frequency), the solver convergence is very slow. Now, I am attempting to create a single matrix so that I can use the other ksp+pc combinations for this system of equations. I am wondering if there is a natural way to create an mpiaij or an mpibaij matrix from the nested matrix defined above. I would appreciate any guidance on this. Regards, Manav -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Feb 26 11:13:10 2016 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 26 Feb 2016 11:13:10 -0600 Subject: [petsc-users] Blocked matrix storage In-Reply-To: <91C08E9D-8A81-42A7-91C3-401BF93166A2@gmail.com> References: <91C08E9D-8A81-42A7-91C3-401BF93166A2@gmail.com> Message-ID: On Fri, Feb 26, 2016 at 11:01 AM, Manav Bhatia wrote: > Hi, > > I am working on a problem with system of equations in complex numbers, > and am splitting it up to solve it with petsc compiled for real number > support: > > (J_R + i J_I) (x_R + i x_I) + (r_R + i r_I) = 0 > > is then defined as > > [ J_R -J_I] {x_R} + {r_R} = {0} > [ J_I J_R] {x_I} + {r_I} = {0} > > I have tried block-Schur solver with the Jacobian defined using the nested > matrix format with a separate matrix for each quadrant of the Jacobian. > This works well for diagonally dominant cases, but for high-frequency > problems (J_I scales with frequency), the solver convergence is very slow. > > Now, I am attempting to create a single matrix so that I can use the other > ksp+pc combinations for this system of equations. > > I am wondering if there is a natural way to create an mpiaij or an mpibaij > matrix from the nested matrix defined above. > You can use this http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetLocalSubMatrix.html to have the same assembly code insert into a global matrix. That way you can use Fieldsplit or other preconditioners. Also, a full Schur complement would converge in 1 iteration with exact solves. I always start there and then back off. Matt > I would appreciate any guidance on this. > > Regards, > Manav > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bhatiamanav at gmail.com Fri Feb 26 11:19:48 2016 From: bhatiamanav at gmail.com (Manav Bhatia) Date: Fri, 26 Feb 2016 11:19:48 -0600 Subject: [petsc-users] GAMG and near-null-space when applying Dirichlet Conditions In-Reply-To: References: <21F632BF-5E41-48B2-B273-763A7EA97BD4@gmail.com> Message-ID: <5EBAA0B1-E848-4782-A72B-3E67D0D99279@gmail.com> I have now experimented with different AMG solvers (gamg, ML, hypre ) through petsc, and have a mixed bag of results. I have used -pc_gamg_threshold 0.1 for all cases. The problem is that of plate-bending that is clamped on all ends, and has a uniformly distributed load. The problem has 6 dofs per node: {u, v, w, tx, ty, tz}. u, v are the in-plane deformations related to membrane action. w, tx, ty get the stiffness from the Mandlin first-order shear deformation theory. tz doesn?t really do anything in the problem, and the stiffness matrix has small diagonal values to avoid singularity problems. I have tested AMG solvers for number of unknowns from a few hundred to about 1.5e6. First off, I am absolutely thrilled to be able to solve that large a system of equations coming from a bending operator on my laptop! So a big thanks to the petsc team for giving us the tools! I have not done a very thorough convergence study, but following are some general observations: ? Without providing the near null space, all three solvers work. ? The convergence of the solvers is significantly better when the near null space is provided. There are 6 near-null space modes provided: 3 rigid-body translations and 3-rigid body rotations. ? With the near null space provided, both hypre and ML work without problems, but GAMG quits the error of zero-pivot in LU decomposition. I am guessing this happens for the coarsest level. I was able to get around this with -mg_levels_pc_type jacobi . (I saw some earlier discussion on the mailing list about this, and got the sense that this may be a non-deterministic issue (?) ). ? With -pc_gamg_threshold 0.1 and -pc_mg_type full, I get the fastest convergence from ML. ? GAMG seems to take about twice the amount of memory than ML. I am now keen to play around with various parameters to see how to influence the convergence. Any comments would be greatly appreciated. Regards, Manav > On Feb 25, 2016, at 6:21 AM, Mark Adams wrote: > > I added ", which is often the null space of the operator without boundary conditions" to the web page doc for MatSetNearNullSpace. > > On Wed, Feb 24, 2016 at 10:57 AM, Matthew Knepley > wrote: > On Wed, Feb 24, 2016 at 9:45 AM, Manav Bhatia > wrote: > Hi, > > I typically apply Dirichlet BCs by modifying the Jacobin and rhs: zero constrained rows of matrix with 1.0 at diagonal, and zero corresponding rows of rhs. > > While using GAMG, is it still recommended to provide the near-null space (given that the zero-eigenvalues have been removed by specification of DIrichlet BCs)? > > Yes. > > If that information is still needed, should the vectors be modified in any manner to be consistent with the Dirichlet BCs? > > No. You can see that if you take a small piece of the domain, apart from the boundary, it will have this as a null space. > > Matt > > Thanks, > Manav > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Feb 26 11:23:57 2016 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 26 Feb 2016 11:23:57 -0600 Subject: [petsc-users] GAMG and near-null-space when applying Dirichlet Conditions In-Reply-To: <5EBAA0B1-E848-4782-A72B-3E67D0D99279@gmail.com> References: <21F632BF-5E41-48B2-B273-763A7EA97BD4@gmail.com> <5EBAA0B1-E848-4782-A72B-3E67D0D99279@gmail.com> Message-ID: On Fri, Feb 26, 2016 at 11:19 AM, Manav Bhatia wrote: > I have now experimented with different AMG solvers (gamg, ML, hypre ) > through petsc, and have a mixed bag of results. I have used > -pc_gamg_threshold 0.1 for all cases. > > The problem is that of plate-bending that is clamped on all ends, and has > a uniformly distributed load. > > The problem has 6 dofs per node: {u, v, w, tx, ty, tz}. u, v are the > in-plane deformations related to membrane action. w, tx, ty get the > stiffness from the Mandlin first-order shear deformation theory. tz doesn?t > really do anything in the problem, and the stiffness matrix has small > diagonal values to avoid singularity problems. > > > I have tested AMG solvers for number of unknowns from a few hundred to > about 1.5e6. > > First off, I am absolutely thrilled to be able to solve that large a > system of equations coming from a bending operator on my laptop! So a big > thanks to the petsc team for giving us the tools! > > I have not done a very thorough convergence study, but following are some > general observations: > > ? Without providing the near null space, all three solvers work. > > ? The convergence of the solvers is significantly better when the near > null space is provided. There are 6 near-null space modes provided: 3 > rigid-body translations and 3-rigid body rotations. > > ? With the near null space provided, both hypre and ML work without > problems, but GAMG quits the error of zero-pivot in LU decomposition. I am > guessing this happens for the coarsest level. I was able to get around this > with -mg_levels_pc_type jacobi . (I saw some earlier discussion on the > mailing list about this, and got the sense that this may be a > non-deterministic issue (?) ). > No, you want to solve this using -mg_coarse_pc_type svd and also send the output of -ksp_view. > ? With -pc_gamg_threshold 0.1 and -pc_mg_type full, I get the fastest > convergence from ML. > > ? GAMG seems to take about twice the amount of memory than ML. > Then the agglomeration parameters must be different since the algorithms are almost identical. Thanks, Matt > > I am now keen to play around with various parameters to see how to > influence the convergence. > > Any comments would be greatly appreciated. > > Regards, > Manav > > > > On Feb 25, 2016, at 6:21 AM, Mark Adams wrote: > > I added ", which is often the null space of the operator without boundary > conditions" to the web page doc for MatSetNearNullSpace. > > On Wed, Feb 24, 2016 at 10:57 AM, Matthew Knepley > wrote: > >> On Wed, Feb 24, 2016 at 9:45 AM, Manav Bhatia >> wrote: >> >>> Hi, >>> >>> I typically apply Dirichlet BCs by modifying the Jacobin and rhs: >>> zero constrained rows of matrix with 1.0 at diagonal, and zero >>> corresponding rows of rhs. >>> >>> While using GAMG, is it still recommended to provide the near-null >>> space (given that the zero-eigenvalues have been removed by specification >>> of DIrichlet BCs)? >>> >> >> Yes. >> >> >>> If that information is still needed, should the vectors be modified >>> in any manner to be consistent with the Dirichlet BCs? >>> >> >> No. You can see that if you take a small piece of the domain, apart from >> the boundary, it will have this as a null space. >> >> Matt >> >> >>> Thanks, >>> Manav >>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mirzadeh at gmail.com Fri Feb 26 12:46:10 2016 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Fri, 26 Feb 2016 13:46:10 -0500 Subject: [petsc-users] Neumann BC with non-symmetric matrix In-Reply-To: References: <715E1F2D-F735-4419-811A-590E2218516F@mcs.anl.gov> Message-ID: Mark, On Fri, Feb 26, 2016 at 8:12 AM, Mark Adams wrote: > >> 4-4) Along the same lines, I tried a couple of other PCs such as >> {jacobi, sor, gamg, ilu} and none of them were able to converge with bcgs >> as the KSP. However, with gmres, almost all of them converge with the >> exception of gamg. >> > > Note, I'm not sure why you need the null space of A^T, you want the null > space of A. > > So the idea was to provide nullspace of A^T to make sure the true residual also converges to zero by projecting the RHS onto the range of A. It however looks like that GMRES (and sometimes BiCGSTAB) converge in the least-square sense for which you only need the nullspace of A and not A^T. And for singular systems like yours you need to use a pseudo inverse of the > coarse grid because it is singular -- if you represent the null space > exactly. > > GAMG is use for AMR problems like this a lot in BISICLES. > Thanks for the reference. However, a quick look at their paper suggests they are using a finite volume discretization which should be symmetric and avoid all the shenanigans I'm going through! I think it would actually be a good idea for me to swap my solver with a conservative one and see if it makes things better. > You need to use an 'svd' coarse grid solver, or an appropriate iterative > solver. LU is the default. > > I see. How can I change the GAMG coarse grid solver? Is there an analogue of "-pc_hypre_boomeramg_relax_type_coarse"? Mark > Thanks, Mohammad -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Feb 26 20:47:27 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 26 Feb 2016 20:47:27 -0600 Subject: [petsc-users] Neumann BC with non-symmetric matrix In-Reply-To: References: <715E1F2D-F735-4419-811A-590E2218516F@mcs.anl.gov> Message-ID: <855BC5E6-55B3-41DA-A113-5B2F8847E86F@mcs.anl.gov> > On Feb 26, 2016, at 12:46 PM, Mohammad Mirzadeh wrote: > > Mark, > > On Fri, Feb 26, 2016 at 8:12 AM, Mark Adams wrote: > > 4-4) Along the same lines, I tried a couple of other PCs such as {jacobi, sor, gamg, ilu} and none of them were able to converge with bcgs as the KSP. However, with gmres, almost all of them converge with the exception of gamg. > > Note, I'm not sure why you need the null space of A^T, you want the null space of A. > > > So the idea was to provide nullspace of A^T to make sure the true residual also converges to zero by projecting the RHS onto the range of A. It however looks like that GMRES (and sometimes BiCGSTAB) converge in the least-square sense for which you only need the nullspace of A and not A^T. > > And for singular systems like yours you need to use a pseudo inverse of the coarse grid because it is singular -- if you represent the null space exactly. > > GAMG is use for AMR problems like this a lot in BISICLES. > > Thanks for the reference. However, a quick look at their paper suggests they are using a finite volume discretization which should be symmetric and avoid all the shenanigans I'm going through! I think it would actually be a good idea for me to swap my solver with a conservative one and see if it makes things better. > > > You need to use an 'svd' coarse grid solver, or an appropriate iterative solver. LU is the default. > > > I see. How can I change the GAMG coarse grid solver? Is there an analogue of "-pc_hypre_boomeramg_relax_type_coarse"? -mg_coarse_pc_type svd or maybe -mg_coarse_redundant_pc_type svd or maybe -mg_coarse_pc_type redundant -mg_coarse_redundant_pc_type svd run with -help and grep for coarse for the exact syntax. > > Mark > > Thanks, > Mohammad From hguo at anl.gov Fri Feb 26 21:05:35 2016 From: hguo at anl.gov (Guo, Hanqi) Date: Fri, 26 Feb 2016 21:05:35 -0600 Subject: [petsc-users] Compile petsc with bgclang compilers on mira/vesta? Message-ID: <8477600C-3935-4187-8F7B-F1FA1EE23C6C@anl.gov> Hi, I am trying to compile petsc on mira with bgclang, because my application needs C++11 support. I have tried several ways to configure but didn?t work. Does anyone have ideas on this? Thanks in advance! The followings are several combinations I have tried: > ./configure --prefix=$HOME/local.bgclang/petsc-3.6.3 --with-cc=mpicc --with-cxx=mpic++11 --with-fc=0 --with-clanguage=cxx =============================================================================== Configuring PETSc to compile on your system =============================================================================== TESTING: checkCxxLibraries from config.compilers(config/BuildSystem/config/compilers.py:413) ******************************************************************************* UNABLE to EXECUTE BINARIES for ./configure ------------------------------------------------------------------------------- Cannot run executables created with C. If this machine uses a batch system to submit jobs you will need to configure using ./configure with the additional option --with-batch. Otherwise there is problem with the compilers. Can you compile and run code with your compiler 'mpicc'? ******************************************************************************* > ./configure --prefix=$HOME/local.bgclang/petsc-3.6.3 --with-cc=mpicc --with-cxx=mpic++11 --with-fc=0 --with-clanguage=cxx --with-batch =============================================================================== Configuring PETSc to compile on your system =============================================================================== TESTING: check from config.libraries(config/BuildSystem/config/libraries.py:146) ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Unable to find mpi in default locations! Perhaps you can specify with --with-mpi-dir= If you do not want MPI, then give --with-mpi=0 You might also consider using --download-mpich instead ******************************************************************************* > ./configure --prefix=$HOME/local.bgclang/petsc-3.6.3 --with-fc=0 --with-clanguage=cxx --with-batch --with-mpi-dir=/soft/compilers/bgclang/mpi/bgclang =============================================================================== Configuring PETSc to compile on your system =============================================================================== TESTING: configureFortranCommandLine from config.utilities.fortranCommandLine(config/BuildSystem/config/utilities/fortranCommandLine.py:27) TESTING: configureLibrary from config.packages.MPI(config/BuildSystem/config/packages/MPI.py:476) ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- --with-mpi-dir=/soft/compilers/bgclang/mpi/bgclang did not work ******************************************************************************* ... --------------- Hanqi Guo Postdoctoral Appointee Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Ave., Bldg 240 Argonne, IL 60439 e-mail: hguo at anl.gov web: http://www.mcs.anl.gov/~hguo phone: 630-252-7225 -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Fri Feb 26 22:23:14 2016 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 26 Feb 2016 22:23:14 -0600 Subject: [petsc-users] Compile petsc with bgclang compilers on mira/vesta? In-Reply-To: <8477600C-3935-4187-8F7B-F1FA1EE23C6C@anl.gov> References: <8477600C-3935-4187-8F7B-F1FA1EE23C6C@anl.gov> Message-ID: looks like '-fPIC' with clang is causing problems. The following works for me. ./configure --with-cc=mpicc --with-cxx=mpic++11 --with-fc=0 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-shared-libraries=0 --download-f2cblaslapack=1 Satish On Fri, 26 Feb 2016, Guo, Hanqi wrote: > Hi, > > I am trying to compile petsc on mira with bgclang, because my application needs C++11 support. I have tried several ways to configure but didn?t work. Does anyone have ideas on this? Thanks in advance! > > The followings are several combinations I have tried: > > > ./configure --prefix=$HOME/local.bgclang/petsc-3.6.3 --with-cc=mpicc --with-cxx=mpic++11 --with-fc=0 --with-clanguage=cxx > =============================================================================== > Configuring PETSc to compile on your system > =============================================================================== > TESTING: checkCxxLibraries from config.compilers(config/BuildSystem/config/compilers.py:413) ******************************************************************************* > UNABLE to EXECUTE BINARIES for ./configure > ------------------------------------------------------------------------------- > Cannot run executables created with C. If this machine uses a batch system > to submit jobs you will need to configure using ./configure with the additional option --with-batch. > Otherwise there is problem with the compilers. Can you compile and run code with your compiler 'mpicc'? > ******************************************************************************* > > > ./configure --prefix=$HOME/local.bgclang/petsc-3.6.3 --with-cc=mpicc --with-cxx=mpic++11 --with-fc=0 --with-clanguage=cxx --with-batch > =============================================================================== > Configuring PETSc to compile on your system > =============================================================================== > TESTING: check from config.libraries(config/BuildSystem/config/libraries.py:146) ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > ------------------------------------------------------------------------------- > Unable to find mpi in default locations! > Perhaps you can specify with --with-mpi-dir= > If you do not want MPI, then give --with-mpi=0 > You might also consider using --download-mpich instead > ******************************************************************************* > > > ./configure --prefix=$HOME/local.bgclang/petsc-3.6.3 --with-fc=0 --with-clanguage=cxx --with-batch --with-mpi-dir=/soft/compilers/bgclang/mpi/bgclang > =============================================================================== > Configuring PETSc to compile on your system > =============================================================================== > TESTING: configureFortranCommandLine from config.utilities.fortranCommandLine(config/BuildSystem/config/utilities/fortranCommandLine.py:27) > TESTING: configureLibrary from config.packages.MPI(config/BuildSystem/config/packages/MPI.py:476) ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > ------------------------------------------------------------------------------- > --with-mpi-dir=/soft/compilers/bgclang/mpi/bgclang did not work > ******************************************************************************* > > ... > > --------------- > Hanqi Guo > Postdoctoral Appointee > Mathematics and Computer Science Division > Argonne National Laboratory > 9700 S. Cass Ave., Bldg 240 > Argonne, IL 60439 > e-mail: hguo at anl.gov > web: http://www.mcs.anl.gov/~hguo > phone: 630-252-7225 > > From hguo at anl.gov Fri Feb 26 23:01:57 2016 From: hguo at anl.gov (Guo, Hanqi) Date: Fri, 26 Feb 2016 23:01:57 -0600 Subject: [petsc-users] Compile petsc with bgclang compilers on mira/vesta? In-Reply-To: References: <8477600C-3935-4187-8F7B-F1FA1EE23C6C@anl.gov> Message-ID: Great. Thanks! -Hanqi --------------- Hanqi Guo Postdoctoral Appointee Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Ave., Bldg 240 Argonne, IL 60439 e-mail: hguo at anl.gov web: http://www.mcs.anl.gov/~hguo phone: 630-252-7225 > On Feb 26, 2016, at 10:23 PM, Satish Balay wrote: > > looks like '-fPIC' with clang is causing problems. The following works for me. > > ./configure --with-cc=mpicc --with-cxx=mpic++11 --with-fc=0 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-shared-libraries=0 --download-f2cblaslapack=1 > > Satish > > On Fri, 26 Feb 2016, Guo, Hanqi wrote: > >> Hi, >> >> I am trying to compile petsc on mira with bgclang, because my application needs C++11 support. I have tried several ways to configure but didn?t work. Does anyone have ideas on this? Thanks in advance! >> >> The followings are several combinations I have tried: >> >>> ./configure --prefix=$HOME/local.bgclang/petsc-3.6.3 --with-cc=mpicc --with-cxx=mpic++11 --with-fc=0 --with-clanguage=cxx >> =============================================================================== >> Configuring PETSc to compile on your system >> =============================================================================== >> TESTING: checkCxxLibraries from config.compilers(config/BuildSystem/config/compilers.py:413) ******************************************************************************* >> UNABLE to EXECUTE BINARIES for ./configure >> ------------------------------------------------------------------------------- >> Cannot run executables created with C. If this machine uses a batch system >> to submit jobs you will need to configure using ./configure with the additional option --with-batch. >> Otherwise there is problem with the compilers. Can you compile and run code with your compiler 'mpicc'? >> ******************************************************************************* >> >>> ./configure --prefix=$HOME/local.bgclang/petsc-3.6.3 --with-cc=mpicc --with-cxx=mpic++11 --with-fc=0 --with-clanguage=cxx --with-batch >> =============================================================================== >> Configuring PETSc to compile on your system >> =============================================================================== >> TESTING: check from config.libraries(config/BuildSystem/config/libraries.py:146) ******************************************************************************* >> UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): >> ------------------------------------------------------------------------------- >> Unable to find mpi in default locations! >> Perhaps you can specify with --with-mpi-dir= >> If you do not want MPI, then give --with-mpi=0 >> You might also consider using --download-mpich instead >> ******************************************************************************* >> >>> ./configure --prefix=$HOME/local.bgclang/petsc-3.6.3 --with-fc=0 --with-clanguage=cxx --with-batch --with-mpi-dir=/soft/compilers/bgclang/mpi/bgclang >> =============================================================================== >> Configuring PETSc to compile on your system >> =============================================================================== >> TESTING: configureFortranCommandLine from config.utilities.fortranCommandLine(config/BuildSystem/config/utilities/fortranCommandLine.py:27) >> TESTING: configureLibrary from config.packages.MPI(config/BuildSystem/config/packages/MPI.py:476) ******************************************************************************* >> UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): >> ------------------------------------------------------------------------------- >> --with-mpi-dir=/soft/compilers/bgclang/mpi/bgclang did not work >> ******************************************************************************* >> >> ... >> >> --------------- >> Hanqi Guo >> Postdoctoral Appointee >> Mathematics and Computer Science Division >> Argonne National Laboratory >> 9700 S. Cass Ave., Bldg 240 >> Argonne, IL 60439 >> e-mail: hguo at anl.gov >> web: http://www.mcs.anl.gov/~hguo >> phone: 630-252-7225 >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Feb 27 14:36:35 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 27 Feb 2016 14:36:35 -0600 Subject: [petsc-users] PETSc release soon, request for input on needed fixes or enhancements Message-ID: PETSc Users, We are planning the PETSc release 3.7 shortly. If you know of any bugs that need to be fixed or enhancements added before the release please let us know. You can think of the master branch of the PETSc repository obtainable with git clone https://bitbucket.org/petsc/petsc petsc as a release candidate for 3.7. Changes for the release are listed at http://www.mcs.anl.gov/petsc/documentation/changes/dev.html Thanks Barry From hguo at anl.gov Sat Feb 27 21:57:54 2016 From: hguo at anl.gov (Guo, Hanqi) Date: Sat, 27 Feb 2016 21:57:54 -0600 Subject: [petsc-users] Writing matrix to file with MPIIO Message-ID: <5CF75596-F3C2-4809-A19E-BA7FEB2BB916@anl.gov> Hi, I am trying to write a sparse matrix to file with MPIIO but failed. I got the same error on vesta (gpfs), my Macbook (local disk), and a linux machine (nfs). The followings are my code snippet and error output. Could anyone help me check the code? Thanks! ... MatCreate(PETSC_COMM_WORLD, &mat); MatSetType(mat, MATMPIAIJ); MatSetSizes(mat, PETSC_DECIDE, PETSC_DECIDE, n, n); MatSetUp(mat); ... // write matrix ... MatAssemblyBegin(mat, MAT_FINAL_ASSEMBLY); MatAssemblyEnd(mat, MAT_FINAL_ASSEMBLY); PetscViewer viewer; PetscViewerCreate(PETSC_COMM_WORLD, &viewer); PetscViewerSetType(viewer, PETSCVIEWERBINARY); PetscViewerFileSetMode(viewer, FILE_MODE_WRITE); PetscViewerFileSetName(viewer, "sfm.mat"); PetscViewerBinarySetUseMPIIO(viewer, PETSC_TRUE); MatView(mat, viewer); PetscViewerDestroy(&viewer); ... [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Write to file failed [0]PETSC ERROR: Error writing to file total size 4 err -1 wsize 16 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 [0]PETSC ERROR: Unknown Name on a arch-linux2-c-debug named compute001 by hguo Sat Feb 27 21:40:04 2016 [0]PETSC ERROR: Configure options ?prefix=(hidden) --with-fc=0 --download-f2cblaslapack --with-mpi-dir=(hidden) --with-precision=single [0]PETSC ERROR: #1 PetscBinaryWrite() line 405 in (hidden)/petsc-3.6.3/src/sys/fileio/sysio.c [0]PETSC ERROR: #2 MatView_MPIAIJ_Binary() line 1219 in (hidden)/petsc-3.6.3/src/mat/impls/aij/mpi/mpiaij.c [0]PETSC ERROR: #3 MatView_MPIAIJ_ASCIIorDraworSocket() line 1385 in (hidden)/petsc-3.6.3/src/mat/impls/aij/mpi/mpiaij.c [0]PETSC ERROR: #4 MatView_MPIAIJ() line 1470 in (hidden)/petsc-3.6.3/src/mat/impls/aij/mpi/mpiaij.c [0]PETSC ERROR: #5 MatView() line 886 in (hidden)/petsc-3.6.3/src/mat/interface/matrix.c --------------- Hanqi Guo Postdoctoral Appointee Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Ave., Bldg 240 Argonne, IL 60439 e-mail: hguo at anl.gov web: http://www.mcs.anl.gov/~hguo phone: 630-252-7225 -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Sun Feb 28 10:29:48 2016 From: zonexo at gmail.com (TAY wee-beng) Date: Mon, 29 Feb 2016 00:29:48 +0800 Subject: [petsc-users] Investigate parallel code to improve parallelism In-Reply-To: <5FBB830C-4A5E-40B2-9845-E0DC68B02BD3@mcs.anl.gov> References: <56D06F16.9000200@gmail.com> <141087E6-A7C1-41FF-B3E8-74FF587E3535@mcs.anl.gov> <56D07D0C.7050109@gmail.com> <5FBB830C-4A5E-40B2-9845-E0DC68B02BD3@mcs.anl.gov> Message-ID: <56D3207C.6090808@gmail.com> On 27/2/2016 12:53 AM, Barry Smith wrote: >> On Feb 26, 2016, at 10:27 AM, TAY wee-beng wrote: >> >> >> On 26/2/2016 11:32 PM, Barry Smith wrote: >>>> On Feb 26, 2016, at 9:28 AM, TAY wee-beng wrote: >>>> >>>> Hi, >>>> >>>> I have got a 3D code. When I ran with 48 procs and 11 million cells, it runs for 83 min. When I ran with 96 procs and 22 million cells, it ran for 99 min. >>> This is actually pretty good! >> But if I'm not wrong, if I increase the no. of cells, the parallelism will keep on decreasing. I hope it scales up to maybe 300 - 400 procs. Hi, I think I may have mentioned this before, that is, I need to submit a proposal to request for computing nodes. In the proposal, I'm supposed to run some simulations to estimate the time it takes to run my code. Then an excel file will use my input to estimate the efficiency when I run my code with more cells. They use 2 mtds to estimate: 1. strong scaling, whereby I run 2 cases - 1st with n cells and x procs, then with n cells and 2x procs. From there, they can estimate my expected efficiency when I have y procs. The formula is attached in the pdf. 2. weak scaling, whereby I run 2 cases - 1st with n cells and x procs, then with 2n cells and 2x procs. From there, they can estimate my expected efficiency when I have y procs. The formula is attached in the pdf. So if I use 48 and 96 procs and get maybe 80% efficiency, by the time I hit 800 procs, I get 32% efficiency for strong scaling. They expect at least 50% efficiency for my code. To reach that, I need to achieve 89% efficiency when I use 48 and 96 procs. So now my qn is how accurate is this type of calculation, especially wrt to PETSc? Similarly, for weak scaling, is it accurate? Can I argue that this estimation is not suitable for PETSc or hypre? Thanks >>>> So it's not that parallel. I want to find out which part of the code I need to improve. Also if PETsc and hypre is working well in parallel. What's the best way to do it? >>> Run both with -log_summary and send the output for each case. This will show where the time is being spent and which parts are scaling less well. >>> >>> Barry >> That's only for the PETSc part, right? So for other parts of the code, including hypre part, I will not be able to find out. If so, what can I use to check these parts? > You will still be able to see what percentage of the time is spent in hypre and if it increases with the problem size and how much. So the information will still be useful. > > Barry > >>>> I thought of doing profiling but if the code is optimized, I wonder if it still works well. >>>> >>>> -- >>>> Thank you. >>>> >>>> Yours sincerely, >>>> >>>> TAY wee-beng >>>> -------------- next part -------------- A non-text attachment was scrubbed... Name: temp.pdf Type: application/pdf Size: 311059 bytes Desc: not available URL: From knepley at gmail.com Sun Feb 28 10:50:41 2016 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 28 Feb 2016 10:50:41 -0600 Subject: [petsc-users] Investigate parallel code to improve parallelism In-Reply-To: <56D3207C.6090808@gmail.com> References: <56D06F16.9000200@gmail.com> <141087E6-A7C1-41FF-B3E8-74FF587E3535@mcs.anl.gov> <56D07D0C.7050109@gmail.com> <5FBB830C-4A5E-40B2-9845-E0DC68B02BD3@mcs.anl.gov> <56D3207C.6090808@gmail.com> Message-ID: On Sun, Feb 28, 2016 at 10:29 AM, TAY wee-beng wrote: > > On 27/2/2016 12:53 AM, Barry Smith wrote: > >> On Feb 26, 2016, at 10:27 AM, TAY wee-beng wrote: >>> >>> >>> On 26/2/2016 11:32 PM, Barry Smith wrote: >>> >>>> On Feb 26, 2016, at 9:28 AM, TAY wee-beng wrote: >>>>> >>>>> Hi, >>>>> >>>>> I have got a 3D code. When I ran with 48 procs and 11 million cells, >>>>> it runs for 83 min. When I ran with 96 procs and 22 million cells, it ran >>>>> for 99 min. >>>>> >>>> This is actually pretty good! >>>> >>> But if I'm not wrong, if I increase the no. of cells, the parallelism >>> will keep on decreasing. I hope it scales up to maybe 300 - 400 procs. >>> >> Hi, > > I think I may have mentioned this before, that is, I need to submit a > proposal to request for computing nodes. In the proposal, I'm supposed to > run some simulations to estimate the time it takes to run my code. Then an > excel file will use my input to estimate the efficiency when I run my code > with more cells. They use 2 mtds to estimate: > > 1. strong scaling, whereby I run 2 cases - 1st with n cells and x procs, > then with n cells and 2x procs. From there, they can estimate my expected > efficiency when I have y procs. The formula is attached in the pdf. > > 2. weak scaling, whereby I run 2 cases - 1st with n cells and x procs, > then with 2n cells and 2x procs. From there, they can estimate my expected > efficiency when I have y procs. The formula is attached in the pdf. > > So if I use 48 and 96 procs and get maybe 80% efficiency, by the time I > hit 800 procs, I get 32% efficiency for strong scaling. They expect at > least 50% efficiency for my code. To reach that, I need to achieve 89% > efficiency when I use 48 and 96 procs. > Only if things obey the kind of scaling you expect. You could have your code degrade less quickly by changing the algorithm. > So now my qn is how accurate is this type of calculation, especially wrt > to PETSc? > You should also distinguish between "setup" and "run", where the later will happen at each step. Thanks, Matt > Similarly, for weak scaling, is it accurate? > > Can I argue that this estimation is not suitable for PETSc or hypre? > > Thanks > > > So it's not that parallel. I want to find out which part of the code I >>>>> need to improve. Also if PETsc and hypre is working well in parallel. >>>>> What's the best way to do it? >>>>> >>>> Run both with -log_summary and send the output for each case. This >>>> will show where the time is being spent and which parts are scaling less >>>> well. >>>> >>>> Barry >>>> >>> That's only for the PETSc part, right? So for other parts of the code, >>> including hypre part, I will not be able to find out. If so, what can I use >>> to check these parts? >>> >> You will still be able to see what percentage of the time is spent in >> hypre and if it increases with the problem size and how much. So the >> information will still be useful. >> >> Barry >> >> I thought of doing profiling but if the code is optimized, I wonder if it >>>>> still works well. >>>>> >>>>> -- >>>>> Thank you. >>>>> >>>>> Yours sincerely, >>>>> >>>>> TAY wee-beng >>>>> >>>>> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Feb 28 11:05:02 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 28 Feb 2016 11:05:02 -0600 Subject: [petsc-users] Writing matrix to file with MPIIO In-Reply-To: <5CF75596-F3C2-4809-A19E-BA7FEB2BB916@anl.gov> References: <5CF75596-F3C2-4809-A19E-BA7FEB2BB916@anl.gov> Message-ID: <19B90160-4B00-480B-AE94-3CDABF547953@mcs.anl.gov> Does it work without the PetscViewerBinarySetUseMPIIO(viewer, PETSC_TRUE); ? Barry > On Feb 27, 2016, at 9:57 PM, Guo, Hanqi wrote: > > Hi, > > I am trying to write a sparse matrix to file with MPIIO but failed. I got the same error on vesta (gpfs), my Macbook (local disk), and a linux machine (nfs). The followings are my code snippet and error output. Could anyone help me check the code? Thanks! > > ... > MatCreate(PETSC_COMM_WORLD, &mat); > MatSetType(mat, MATMPIAIJ); > MatSetSizes(mat, PETSC_DECIDE, PETSC_DECIDE, n, n); > MatSetUp(mat); > ... > // write matrix > ... > MatAssemblyBegin(mat, MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(mat, MAT_FINAL_ASSEMBLY); > > PetscViewer viewer; > PetscViewerCreate(PETSC_COMM_WORLD, &viewer); > PetscViewerSetType(viewer, PETSCVIEWERBINARY); > PetscViewerFileSetMode(viewer, FILE_MODE_WRITE); > PetscViewerFileSetName(viewer, "sfm.mat"); > PetscViewerBinarySetUseMPIIO(viewer, PETSC_TRUE); > MatView(mat, viewer); > PetscViewerDestroy(&viewer); > ... > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Write to file failed > [0]PETSC ERROR: Error writing to file total size 4 err -1 wsize 16 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 > [0]PETSC ERROR: Unknown Name on a arch-linux2-c-debug named compute001 by hguo Sat Feb 27 21:40:04 2016 > [0]PETSC ERROR: Configure options ?prefix=(hidden) --with-fc=0 --download-f2cblaslapack --with-mpi-dir=(hidden) --with-precision=single > [0]PETSC ERROR: #1 PetscBinaryWrite() line 405 in (hidden)/petsc-3.6.3/src/sys/fileio/sysio.c > [0]PETSC ERROR: #2 MatView_MPIAIJ_Binary() line 1219 in (hidden)/petsc-3.6.3/src/mat/impls/aij/mpi/mpiaij.c > [0]PETSC ERROR: #3 MatView_MPIAIJ_ASCIIorDraworSocket() line 1385 in (hidden)/petsc-3.6.3/src/mat/impls/aij/mpi/mpiaij.c > [0]PETSC ERROR: #4 MatView_MPIAIJ() line 1470 in (hidden)/petsc-3.6.3/src/mat/impls/aij/mpi/mpiaij.c > [0]PETSC ERROR: #5 MatView() line 886 in (hidden)/petsc-3.6.3/src/mat/interface/matrix.c > > > --------------- > Hanqi Guo > Postdoctoral Appointee > Mathematics and Computer Science Division > Argonne National Laboratory > 9700 S. Cass Ave., Bldg 240 > Argonne, IL 60439 > e-mail: hguo at anl.gov > web: http://www.mcs.anl.gov/~hguo > phone: 630-252-7225 > From bsmith at mcs.anl.gov Sun Feb 28 11:11:44 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 28 Feb 2016 11:11:44 -0600 Subject: [petsc-users] Investigate parallel code to improve parallelism In-Reply-To: <56D3207C.6090808@gmail.com> References: <56D06F16.9000200@gmail.com> <141087E6-A7C1-41FF-B3E8-74FF587E3535@mcs.anl.gov> <56D07D0C.7050109@gmail.com> <5FBB830C-4A5E-40B2-9845-E0DC68B02BD3@mcs.anl.gov> <56D3207C.6090808@gmail.com> Message-ID: <91716C74-7716-4C7D-98B5-1606A8A93766@mcs.anl.gov> As I said before, send the -log_summary output for the two processor sizes and we'll look at where it is spending its time and how it could possibly be improved. Barry > On Feb 28, 2016, at 10:29 AM, TAY wee-beng wrote: > > > On 27/2/2016 12:53 AM, Barry Smith wrote: >>> On Feb 26, 2016, at 10:27 AM, TAY wee-beng wrote: >>> >>> >>> On 26/2/2016 11:32 PM, Barry Smith wrote: >>>>> On Feb 26, 2016, at 9:28 AM, TAY wee-beng wrote: >>>>> >>>>> Hi, >>>>> >>>>> I have got a 3D code. When I ran with 48 procs and 11 million cells, it runs for 83 min. When I ran with 96 procs and 22 million cells, it ran for 99 min. >>>> This is actually pretty good! >>> But if I'm not wrong, if I increase the no. of cells, the parallelism will keep on decreasing. I hope it scales up to maybe 300 - 400 procs. > Hi, > > I think I may have mentioned this before, that is, I need to submit a proposal to request for computing nodes. In the proposal, I'm supposed to run some simulations to estimate the time it takes to run my code. Then an excel file will use my input to estimate the efficiency when I run my code with more cells. They use 2 mtds to estimate: > > 1. strong scaling, whereby I run 2 cases - 1st with n cells and x procs, then with n cells and 2x procs. From there, they can estimate my expected efficiency when I have y procs. The formula is attached in the pdf. > > 2. weak scaling, whereby I run 2 cases - 1st with n cells and x procs, then with 2n cells and 2x procs. From there, they can estimate my expected efficiency when I have y procs. The formula is attached in the pdf. > > So if I use 48 and 96 procs and get maybe 80% efficiency, by the time I hit 800 procs, I get 32% efficiency for strong scaling. They expect at least 50% efficiency for my code. To reach that, I need to achieve 89% efficiency when I use 48 and 96 procs. > > So now my qn is how accurate is this type of calculation, especially wrt to PETSc? > > Similarly, for weak scaling, is it accurate? > > Can I argue that this estimation is not suitable for PETSc or hypre? > > Thanks > > >>>>> So it's not that parallel. I want to find out which part of the code I need to improve. Also if PETsc and hypre is working well in parallel. What's the best way to do it? >>>> Run both with -log_summary and send the output for each case. This will show where the time is being spent and which parts are scaling less well. >>>> >>>> Barry >>> That's only for the PETSc part, right? So for other parts of the code, including hypre part, I will not be able to find out. If so, what can I use to check these parts? >> You will still be able to see what percentage of the time is spent in hypre and if it increases with the problem size and how much. So the information will still be useful. >> >> Barry >> >>>>> I thought of doing profiling but if the code is optimized, I wonder if it still works well. >>>>> >>>>> -- >>>>> Thank you. >>>>> >>>>> Yours sincerely, >>>>> >>>>> TAY wee-beng >>>>> > > From hguo at anl.gov Sun Feb 28 11:18:04 2016 From: hguo at anl.gov (Guo, Hanqi) Date: Sun, 28 Feb 2016 11:18:04 -0600 Subject: [petsc-users] Writing matrix to file with MPIIO In-Reply-To: <19B90160-4B00-480B-AE94-3CDABF547953@mcs.anl.gov> References: <5CF75596-F3C2-4809-A19E-BA7FEB2BB916@anl.gov> <19B90160-4B00-480B-AE94-3CDABF547953@mcs.anl.gov> Message-ID: <0C85B217-9D57-49F4-BD31-16E0512EBE2D@anl.gov> Yes. It works without MPIIO. -Hanqi --------------- Hanqi Guo Postdoctoral Appointee Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Ave., Bldg 240 Argonne, IL 60439 e-mail: hguo at anl.gov web: http://www.mcs.anl.gov/~hguo phone: 630-252-7225 > On Feb 28, 2016, at 11:05 AM, Barry Smith wrote: > > > Does it work without the PetscViewerBinarySetUseMPIIO(viewer, PETSC_TRUE); ? > > Barry > >> On Feb 27, 2016, at 9:57 PM, Guo, Hanqi wrote: >> >> Hi, >> >> I am trying to write a sparse matrix to file with MPIIO but failed. I got the same error on vesta (gpfs), my Macbook (local disk), and a linux machine (nfs). The followings are my code snippet and error output. Could anyone help me check the code? Thanks! >> >> ... >> MatCreate(PETSC_COMM_WORLD, &mat); >> MatSetType(mat, MATMPIAIJ); >> MatSetSizes(mat, PETSC_DECIDE, PETSC_DECIDE, n, n); >> MatSetUp(mat); >> ... >> // write matrix >> ... >> MatAssemblyBegin(mat, MAT_FINAL_ASSEMBLY); >> MatAssemblyEnd(mat, MAT_FINAL_ASSEMBLY); >> >> PetscViewer viewer; >> PetscViewerCreate(PETSC_COMM_WORLD, &viewer); >> PetscViewerSetType(viewer, PETSCVIEWERBINARY); >> PetscViewerFileSetMode(viewer, FILE_MODE_WRITE); >> PetscViewerFileSetName(viewer, "sfm.mat"); >> PetscViewerBinarySetUseMPIIO(viewer, PETSC_TRUE); >> MatView(mat, viewer); >> PetscViewerDestroy(&viewer); >> ... >> >> >> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [0]PETSC ERROR: Write to file failed >> [0]PETSC ERROR: Error writing to file total size 4 err -1 wsize 16 >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >> [0]PETSC ERROR: Unknown Name on a arch-linux2-c-debug named compute001 by hguo Sat Feb 27 21:40:04 2016 >> [0]PETSC ERROR: Configure options ?prefix=(hidden) --with-fc=0 --download-f2cblaslapack --with-mpi-dir=(hidden) --with-precision=single >> [0]PETSC ERROR: #1 PetscBinaryWrite() line 405 in (hidden)/petsc-3.6.3/src/sys/fileio/sysio.c >> [0]PETSC ERROR: #2 MatView_MPIAIJ_Binary() line 1219 in (hidden)/petsc-3.6.3/src/mat/impls/aij/mpi/mpiaij.c >> [0]PETSC ERROR: #3 MatView_MPIAIJ_ASCIIorDraworSocket() line 1385 in (hidden)/petsc-3.6.3/src/mat/impls/aij/mpi/mpiaij.c >> [0]PETSC ERROR: #4 MatView_MPIAIJ() line 1470 in (hidden)/petsc-3.6.3/src/mat/impls/aij/mpi/mpiaij.c >> [0]PETSC ERROR: #5 MatView() line 886 in (hidden)/petsc-3.6.3/src/mat/interface/matrix.c >> >> >> --------------- >> Hanqi Guo >> Postdoctoral Appointee >> Mathematics and Computer Science Division >> Argonne National Laboratory >> 9700 S. Cass Ave., Bldg 240 >> Argonne, IL 60439 >> e-mail: hguo at anl.gov >> web: http://www.mcs.anl.gov/~hguo >> phone: 630-252-7225 >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Feb 28 11:29:03 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 28 Feb 2016 11:29:03 -0600 Subject: [petsc-users] Writing matrix to file with MPIIO In-Reply-To: <0C85B217-9D57-49F4-BD31-16E0512EBE2D@anl.gov> References: <5CF75596-F3C2-4809-A19E-BA7FEB2BB916@anl.gov> <19B90160-4B00-480B-AE94-3CDABF547953@mcs.anl.gov> <0C85B217-9D57-49F4-BD31-16E0512EBE2D@anl.gov> Message-ID: > On Feb 28, 2016, at 11:18 AM, Guo, Hanqi wrote: > > Yes. It works without MPIIO. We've never added support for saving sparse matrices with MPIIO; likely it would not be much faster and it would be a lot of work to code. If you are just saving matrices for testing it doesn't matter anyways; I'd never save matrices in production runs. Barry > > -Hanqi > > --------------- > Hanqi Guo > Postdoctoral Appointee > Mathematics and Computer Science Division > Argonne National Laboratory > 9700 S. Cass Ave., Bldg 240 > Argonne, IL 60439 > e-mail: hguo at anl.gov > web: http://www.mcs.anl.gov/~hguo > phone: 630-252-7225 > >> On Feb 28, 2016, at 11:05 AM, Barry Smith wrote: >> >> >> Does it work without the PetscViewerBinarySetUseMPIIO(viewer, PETSC_TRUE); ? >> >> Barry >> >>> On Feb 27, 2016, at 9:57 PM, Guo, Hanqi wrote: >>> >>> Hi, >>> >>> I am trying to write a sparse matrix to file with MPIIO but failed. I got the same error on vesta (gpfs), my Macbook (local disk), and a linux machine (nfs). The followings are my code snippet and error output. Could anyone help me check the code? Thanks! >>> >>> ... >>> MatCreate(PETSC_COMM_WORLD, &mat); >>> MatSetType(mat, MATMPIAIJ); >>> MatSetSizes(mat, PETSC_DECIDE, PETSC_DECIDE, n, n); >>> MatSetUp(mat); >>> ... >>> // write matrix >>> ... >>> MatAssemblyBegin(mat, MAT_FINAL_ASSEMBLY); >>> MatAssemblyEnd(mat, MAT_FINAL_ASSEMBLY); >>> >>> PetscViewer viewer; >>> PetscViewerCreate(PETSC_COMM_WORLD, &viewer); >>> PetscViewerSetType(viewer, PETSCVIEWERBINARY); >>> PetscViewerFileSetMode(viewer, FILE_MODE_WRITE); >>> PetscViewerFileSetName(viewer, "sfm.mat"); >>> PetscViewerBinarySetUseMPIIO(viewer, PETSC_TRUE); >>> MatView(mat, viewer); >>> PetscViewerDestroy(&viewer); >>> ... >>> >>> >>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>> [0]PETSC ERROR: Write to file failed >>> [0]PETSC ERROR: Error writing to file total size 4 err -1 wsize 16 >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>> [0]PETSC ERROR: Unknown Name on a arch-linux2-c-debug named compute001 by hguo Sat Feb 27 21:40:04 2016 >>> [0]PETSC ERROR: Configure options ?prefix=(hidden) --with-fc=0 --download-f2cblaslapack --with-mpi-dir=(hidden) --with-precision=single >>> [0]PETSC ERROR: #1 PetscBinaryWrite() line 405 in (hidden)/petsc-3.6.3/src/sys/fileio/sysio.c >>> [0]PETSC ERROR: #2 MatView_MPIAIJ_Binary() line 1219 in (hidden)/petsc-3.6.3/src/mat/impls/aij/mpi/mpiaij.c >>> [0]PETSC ERROR: #3 MatView_MPIAIJ_ASCIIorDraworSocket() line 1385 in (hidden)/petsc-3.6.3/src/mat/impls/aij/mpi/mpiaij.c >>> [0]PETSC ERROR: #4 MatView_MPIAIJ() line 1470 in (hidden)/petsc-3.6.3/src/mat/impls/aij/mpi/mpiaij.c >>> [0]PETSC ERROR: #5 MatView() line 886 in (hidden)/petsc-3.6.3/src/mat/interface/matrix.c >>> >>> >>> --------------- >>> Hanqi Guo >>> Postdoctoral Appointee >>> Mathematics and Computer Science Division >>> Argonne National Laboratory >>> 9700 S. Cass Ave., Bldg 240 >>> Argonne, IL 60439 >>> e-mail: hguo at anl.gov >>> web: http://www.mcs.anl.gov/~hguo >>> phone: 630-252-7225 >>> >> > From hguo at anl.gov Sun Feb 28 11:39:51 2016 From: hguo at anl.gov (Guo, Hanqi) Date: Sun, 28 Feb 2016 11:39:51 -0600 Subject: [petsc-users] Writing matrix to file with MPIIO In-Reply-To: References: <5CF75596-F3C2-4809-A19E-BA7FEB2BB916@anl.gov> <19B90160-4B00-480B-AE94-3CDABF547953@mcs.anl.gov> <0C85B217-9D57-49F4-BD31-16E0512EBE2D@anl.gov> Message-ID: > On Feb 28, 2016, at 11:29 AM, Barry Smith wrote: > > >> On Feb 28, 2016, at 11:18 AM, Guo, Hanqi wrote: >> >> Yes. It works without MPIIO. > > We've never added support for saving sparse matrices with MPIIO; likely it would not be much faster and it would be a lot of work to code. If you are just saving matrices for testing it doesn't matter anyways; I'd never save matrices in production runs. Hi Barry, The matrices are the final products of my application. I?ll use serial IO instead. Thanks! -Hanqi > > Barry > >> >> -Hanqi >> >> --------------- >> Hanqi Guo >> Postdoctoral Appointee >> Mathematics and Computer Science Division >> Argonne National Laboratory >> 9700 S. Cass Ave., Bldg 240 >> Argonne, IL 60439 >> e-mail: hguo at anl.gov >> web: http://www.mcs.anl.gov/~hguo >> phone: 630-252-7225 >> >>> On Feb 28, 2016, at 11:05 AM, Barry Smith wrote: >>> >>> >>> Does it work without the PetscViewerBinarySetUseMPIIO(viewer, PETSC_TRUE); ? >>> >>> Barry >>> >>>> On Feb 27, 2016, at 9:57 PM, Guo, Hanqi wrote: >>>> >>>> Hi, >>>> >>>> I am trying to write a sparse matrix to file with MPIIO but failed. I got the same error on vesta (gpfs), my Macbook (local disk), and a linux machine (nfs). The followings are my code snippet and error output. Could anyone help me check the code? Thanks! >>>> >>>> ... >>>> MatCreate(PETSC_COMM_WORLD, &mat); >>>> MatSetType(mat, MATMPIAIJ); >>>> MatSetSizes(mat, PETSC_DECIDE, PETSC_DECIDE, n, n); >>>> MatSetUp(mat); >>>> ... >>>> // write matrix >>>> ... >>>> MatAssemblyBegin(mat, MAT_FINAL_ASSEMBLY); >>>> MatAssemblyEnd(mat, MAT_FINAL_ASSEMBLY); >>>> >>>> PetscViewer viewer; >>>> PetscViewerCreate(PETSC_COMM_WORLD, &viewer); >>>> PetscViewerSetType(viewer, PETSCVIEWERBINARY); >>>> PetscViewerFileSetMode(viewer, FILE_MODE_WRITE); >>>> PetscViewerFileSetName(viewer, "sfm.mat"); >>>> PetscViewerBinarySetUseMPIIO(viewer, PETSC_TRUE); >>>> MatView(mat, viewer); >>>> PetscViewerDestroy(&viewer); >>>> ... >>>> >>>> >>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>> [0]PETSC ERROR: Write to file failed >>>> [0]PETSC ERROR: Error writing to file total size 4 err -1 wsize 16 >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>>> [0]PETSC ERROR: Unknown Name on a arch-linux2-c-debug named compute001 by hguo Sat Feb 27 21:40:04 2016 >>>> [0]PETSC ERROR: Configure options ?prefix=(hidden) --with-fc=0 --download-f2cblaslapack --with-mpi-dir=(hidden) --with-precision=single >>>> [0]PETSC ERROR: #1 PetscBinaryWrite() line 405 in (hidden)/petsc-3.6.3/src/sys/fileio/sysio.c >>>> [0]PETSC ERROR: #2 MatView_MPIAIJ_Binary() line 1219 in (hidden)/petsc-3.6.3/src/mat/impls/aij/mpi/mpiaij.c >>>> [0]PETSC ERROR: #3 MatView_MPIAIJ_ASCIIorDraworSocket() line 1385 in (hidden)/petsc-3.6.3/src/mat/impls/aij/mpi/mpiaij.c >>>> [0]PETSC ERROR: #4 MatView_MPIAIJ() line 1470 in (hidden)/petsc-3.6.3/src/mat/impls/aij/mpi/mpiaij.c >>>> [0]PETSC ERROR: #5 MatView() line 886 in (hidden)/petsc-3.6.3/src/mat/interface/matrix.c >>>> >>>> >>>> --------------- >>>> Hanqi Guo >>>> Postdoctoral Appointee >>>> Mathematics and Computer Science Division >>>> Argonne National Laboratory >>>> 9700 S. Cass Ave., Bldg 240 >>>> Argonne, IL 60439 >>>> e-mail: hguo at anl.gov >>>> web: http://www.mcs.anl.gov/~hguo >>>> phone: 630-252-7225 >>>> >>> >> > From bsmith at mcs.anl.gov Sun Feb 28 13:11:39 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 28 Feb 2016 13:11:39 -0600 Subject: [petsc-users] Writing matrix to file with MPIIO In-Reply-To: References: <5CF75596-F3C2-4809-A19E-BA7FEB2BB916@anl.gov> <19B90160-4B00-480B-AE94-3CDABF547953@mcs.anl.gov> <0C85B217-9D57-49F4-BD31-16E0512EBE2D@anl.gov> Message-ID: > On Feb 28, 2016, at 11:39 AM, Guo, Hanqi wrote: > > >> On Feb 28, 2016, at 11:29 AM, Barry Smith wrote: >> >> >>> On Feb 28, 2016, at 11:18 AM, Guo, Hanqi wrote: >>> >>> Yes. It works without MPIIO. >> >> We've never added support for saving sparse matrices with MPIIO; likely it would not be much faster and it would be a lot of work to code. If you are just saving matrices for testing it doesn't matter anyways; I'd never save matrices in production runs. > > Hi Barry, > > The matrices are the final products of my application. I?ll use serial IO instead. Thanks! \ What do you use the matrices for? Just curious > > -Hanqi > >> >> Barry >> >>> >>> -Hanqi >>> >>> --------------- >>> Hanqi Guo >>> Postdoctoral Appointee >>> Mathematics and Computer Science Division >>> Argonne National Laboratory >>> 9700 S. Cass Ave., Bldg 240 >>> Argonne, IL 60439 >>> e-mail: hguo at anl.gov >>> web: http://www.mcs.anl.gov/~hguo >>> phone: 630-252-7225 >>> >>>> On Feb 28, 2016, at 11:05 AM, Barry Smith wrote: >>>> >>>> >>>> Does it work without the PetscViewerBinarySetUseMPIIO(viewer, PETSC_TRUE); ? >>>> >>>> Barry >>>> >>>>> On Feb 27, 2016, at 9:57 PM, Guo, Hanqi wrote: >>>>> >>>>> Hi, >>>>> >>>>> I am trying to write a sparse matrix to file with MPIIO but failed. I got the same error on vesta (gpfs), my Macbook (local disk), and a linux machine (nfs). The followings are my code snippet and error output. Could anyone help me check the code? Thanks! >>>>> >>>>> ... >>>>> MatCreate(PETSC_COMM_WORLD, &mat); >>>>> MatSetType(mat, MATMPIAIJ); >>>>> MatSetSizes(mat, PETSC_DECIDE, PETSC_DECIDE, n, n); >>>>> MatSetUp(mat); >>>>> ... >>>>> // write matrix >>>>> ... >>>>> MatAssemblyBegin(mat, MAT_FINAL_ASSEMBLY); >>>>> MatAssemblyEnd(mat, MAT_FINAL_ASSEMBLY); >>>>> >>>>> PetscViewer viewer; >>>>> PetscViewerCreate(PETSC_COMM_WORLD, &viewer); >>>>> PetscViewerSetType(viewer, PETSCVIEWERBINARY); >>>>> PetscViewerFileSetMode(viewer, FILE_MODE_WRITE); >>>>> PetscViewerFileSetName(viewer, "sfm.mat"); >>>>> PetscViewerBinarySetUseMPIIO(viewer, PETSC_TRUE); >>>>> MatView(mat, viewer); >>>>> PetscViewerDestroy(&viewer); >>>>> ... >>>>> >>>>> >>>>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>> [0]PETSC ERROR: Write to file failed >>>>> [0]PETSC ERROR: Error writing to file total size 4 err -1 wsize 16 >>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 >>>>> [0]PETSC ERROR: Unknown Name on a arch-linux2-c-debug named compute001 by hguo Sat Feb 27 21:40:04 2016 >>>>> [0]PETSC ERROR: Configure options ?prefix=(hidden) --with-fc=0 --download-f2cblaslapack --with-mpi-dir=(hidden) --with-precision=single >>>>> [0]PETSC ERROR: #1 PetscBinaryWrite() line 405 in (hidden)/petsc-3.6.3/src/sys/fileio/sysio.c >>>>> [0]PETSC ERROR: #2 MatView_MPIAIJ_Binary() line 1219 in (hidden)/petsc-3.6.3/src/mat/impls/aij/mpi/mpiaij.c >>>>> [0]PETSC ERROR: #3 MatView_MPIAIJ_ASCIIorDraworSocket() line 1385 in (hidden)/petsc-3.6.3/src/mat/impls/aij/mpi/mpiaij.c >>>>> [0]PETSC ERROR: #4 MatView_MPIAIJ() line 1470 in (hidden)/petsc-3.6.3/src/mat/impls/aij/mpi/mpiaij.c >>>>> [0]PETSC ERROR: #5 MatView() line 886 in (hidden)/petsc-3.6.3/src/mat/interface/matrix.c >>>>> >>>>> >>>>> --------------- >>>>> Hanqi Guo >>>>> Postdoctoral Appointee >>>>> Mathematics and Computer Science Division >>>>> Argonne National Laboratory >>>>> 9700 S. Cass Ave., Bldg 240 >>>>> Argonne, IL 60439 >>>>> e-mail: hguo at anl.gov >>>>> web: http://www.mcs.anl.gov/~hguo >>>>> phone: 630-252-7225 >>>>> >>>> >>> >> > From zonexo at gmail.com Sun Feb 28 19:08:05 2016 From: zonexo at gmail.com (TAY Wee Beng) Date: Mon, 29 Feb 2016 09:08:05 +0800 Subject: [petsc-users] Investigate parallel code to improve parallelism In-Reply-To: <91716C74-7716-4C7D-98B5-1606A8A93766@mcs.anl.gov> References: <56D06F16.9000200@gmail.com> <141087E6-A7C1-41FF-B3E8-74FF587E3535@mcs.anl.gov> <56D07D0C.7050109@gmail.com> <5FBB830C-4A5E-40B2-9845-E0DC68B02BD3@mcs.anl.gov> <56D3207C.6090808@gmail.com> <91716C74-7716-4C7D-98B5-1606A8A93766@mcs.anl.gov> Message-ID: <56D399F5.9030709@gmail.com> Hi, I've attached the files for x cells running y procs. hypre is called natively I'm not sure if PETSc catches it. Thanks On 29/2/2016 1:11 AM, Barry Smith wrote: > As I said before, send the -log_summary output for the two processor sizes and we'll look at where it is spending its time and how it could possibly be improved. > > Barry > >> On Feb 28, 2016, at 10:29 AM, TAY wee-beng wrote: >> >> >> On 27/2/2016 12:53 AM, Barry Smith wrote: >>>> On Feb 26, 2016, at 10:27 AM, TAY wee-beng wrote: >>>> >>>> >>>> On 26/2/2016 11:32 PM, Barry Smith wrote: >>>>>> On Feb 26, 2016, at 9:28 AM, TAY wee-beng wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> I have got a 3D code. When I ran with 48 procs and 11 million cells, it runs for 83 min. When I ran with 96 procs and 22 million cells, it ran for 99 min. >>>>> This is actually pretty good! >>>> But if I'm not wrong, if I increase the no. of cells, the parallelism will keep on decreasing. I hope it scales up to maybe 300 - 400 procs. >> Hi, >> >> I think I may have mentioned this before, that is, I need to submit a proposal to request for computing nodes. In the proposal, I'm supposed to run some simulations to estimate the time it takes to run my code. Then an excel file will use my input to estimate the efficiency when I run my code with more cells. They use 2 mtds to estimate: >> >> 1. strong scaling, whereby I run 2 cases - 1st with n cells and x procs, then with n cells and 2x procs. From there, they can estimate my expected efficiency when I have y procs. The formula is attached in the pdf. >> >> 2. weak scaling, whereby I run 2 cases - 1st with n cells and x procs, then with 2n cells and 2x procs. From there, they can estimate my expected efficiency when I have y procs. The formula is attached in the pdf. >> >> So if I use 48 and 96 procs and get maybe 80% efficiency, by the time I hit 800 procs, I get 32% efficiency for strong scaling. They expect at least 50% efficiency for my code. To reach that, I need to achieve 89% efficiency when I use 48 and 96 procs. >> >> So now my qn is how accurate is this type of calculation, especially wrt to PETSc? >> >> Similarly, for weak scaling, is it accurate? >> >> Can I argue that this estimation is not suitable for PETSc or hypre? >> >> Thanks >> >> >>>>>> So it's not that parallel. I want to find out which part of the code I need to improve. Also if PETsc and hypre is working well in parallel. What's the best way to do it? >>>>> Run both with -log_summary and send the output for each case. This will show where the time is being spent and which parts are scaling less well. >>>>> >>>>> Barry >>>> That's only for the PETSc part, right? So for other parts of the code, including hypre part, I will not be able to find out. If so, what can I use to check these parts? >>> You will still be able to see what percentage of the time is spent in hypre and if it increases with the problem size and how much. So the information will still be useful. >>> >>> Barry >>> >>>>>> I thought of doing profiling but if the code is optimized, I wonder if it still works well. >>>>>> >>>>>> -- >>>>>> Thank you. >>>>>> >>>>>> Yours sincerely, >>>>>> >>>>>> TAY wee-beng >>>>>> >> -- Thank you Yours sincerely, TAY wee-beng -------------- next part -------------- 0.000000000000000E+000 0.600000000000000 17.5000000000000 120.000000000000 0.000000000000000E+000 0.250000000000000 1.00000000000000 0.400000000000000 0 -400000 AB,AA,BB -2.78150003711926 2.76500003633555 2.78150003711926 2.70650003355695 size_x,size_y,size_z 100 172 171 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 0 1 22 1 29 1 63800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 1 23 44 1 29 63801 127600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 2 45 66 1 29 127601 191400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 3 67 88 1 29 191401 255200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 4 89 109 1 29 255201 316100 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 5 110 130 1 29 316101 377000 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 6 131 151 1 29 377001 437900 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 7 152 172 1 29 437901 498800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 8 1 22 30 58 498801 562600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 9 23 44 30 58 562601 626400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 10 45 66 30 58 626401 690200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 11 67 88 30 58 690201 754000 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 12 89 109 30 58 754001 814900 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 13 110 130 30 58 814901 875800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 14 131 151 30 58 875801 936700 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 15 152 172 30 58 936701 997600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 16 1 22 59 87 997601 1061400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 17 23 44 59 87 1061401 1125200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 18 45 66 59 87 1125201 1189000 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 19 67 88 59 87 1189001 1252800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 20 89 109 59 87 1252801 1313700 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 21 110 130 59 87 1313701 1374600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 22 131 151 59 87 1374601 1435500 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 23 152 172 59 87 1435501 1496400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 24 1 22 88 115 1496401 1558000 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 25 23 44 88 115 1558001 1619600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 26 45 66 88 115 1619601 1681200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 27 67 88 88 115 1681201 1742800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 28 89 109 88 115 1742801 1801600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 29 110 130 88 115 1801601 1860400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 30 131 151 88 115 1860401 1919200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 31 152 172 88 115 1919201 1978000 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 32 1 22 116 143 1978001 2039600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 33 23 44 116 143 2039601 2101200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 34 45 66 116 143 2101201 2162800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 35 67 88 116 143 2162801 2224400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 36 89 109 116 143 2224401 2283200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 37 110 130 116 143 2283201 2342000 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 38 131 151 116 143 2342001 2400800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 39 152 172 116 143 2400801 2459600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 40 1 22 144 171 2459601 2521200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 41 23 44 144 171 2521201 2582800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 42 45 66 144 171 2582801 2644400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 43 67 88 144 171 2644401 2706000 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 44 89 109 144 171 2706001 2764800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 45 110 130 144 171 2764801 2823600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 46 131 151 144 171 2823601 2882400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 47 152 172 144 171 2882401 2941200 body_cg_ini 0.850000999999998 9.999999998273846E-007 6.95771875020604 3104 surfaces with wrong vertex ordering Warning - length difference between element and cell max_element_length,min_element_length,min_delta 7.847540176996057E-002 3.349995610000001E-002 3.500000000000000E-002 maximum ngh_surfaces and ngh_vertics are 28 10 minimum ngh_surfaces and ngh_vertics are 14 5 min IIB_cell_no 0 max IIB_cell_no 160 final initial IIB_cell_no 8000 min I_cell_no 0 max I_cell_no 152 final initial I_cell_no 7600 size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u) 8000 7600 8000 7600 IIB_I_cell_no_uvw_total1 2230 2227 2166 1930 1926 1847 1 0.01411765 0.30104754 0.32529731 1.15440698 -0.30539502E+03 -0.29715696E+02 0.29394159E+07 2 0.00973086 0.41244275 0.45087918 1.22116705 -0.34883625E+03 -0.24811937E+02 0.29392112E+07 3 0.00918174 0.45383297 0.51181352 1.27753799 -0.35798168E+03 -0.24574571E+02 0.29391681E+07 4 0.00885756 0.47398509 0.55172076 1.31014211 -0.36230580E+03 -0.25263557E+02 0.29391476E+07 5 0.00872228 0.48832349 0.57971242 1.32671388 -0.36519380E+03 -0.26103818E+02 0.29391332E+07 300 0.00157913 4.12249858 6.52091323 3.82250060 -0.19591967E+04 0.43743742E+05 0.29433413E+07 600 0.00164417 3.72487671 5.92911552 5.37207663 -0.52948695E+03 0.20658440E+05 0.29461349E+07 900 0.00135887 4.43677663 5.89462632 4.98710090 -0.86683472E+04 -0.13481580E+05 0.29508693E+07 1200 0.00164451 4.13807017 6.05251254 4.63339167 -0.98993801E+04 0.17681878E+05 0.29569013E+07 1500 0.00150966 3.40576464 5.20747744 4.68945079 -0.16263794E+05 0.26012616E+05 0.29604137E+07 1800 0.00132562 3.67104160 10.44718173 4.20460908 -0.18839985E+05 -0.18296757E+05 0.29677661E+07 escape_time reached, so abort cd_cl_cs_mom_implicit1 -0.892081024888283 -1.32621439664773 -5.584506755655451E-003 0.350447346081129 -9.754897341839677E-002 -2.09284777911273 ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./a.out on a petsc-3.6.3_static_rel named n12-04 with 48 processors, by wtay Sat Feb 27 16:15:12 2016 Using Petsc Release Version 3.6.3, Dec, 03, 2015 Max Max/Min Avg Total Time (sec): 3.252e+03 1.00001 3.252e+03 Objects: 2.008e+04 1.00000 2.008e+04 Flops: 1.590e+11 1.09643 1.523e+11 7.310e+12 Flops/sec: 4.891e+07 1.09643 4.683e+07 2.248e+09 MPI Messages: 8.531e+04 1.93829 7.180e+04 3.446e+06 MPI Message Lengths: 4.573e+09 2.08162 5.352e+04 1.844e+11 MPI Reductions: 4.427e+04 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 3.2521e+03 100.0% 7.3101e+12 100.0% 3.446e+06 100.0% 5.352e+04 100.0% 4.427e+04 100.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage VecDot 3998 1.0 2.8182e+01 3.2 1.53e+09 1.1 0.0e+00 0.0e+00 4.0e+03 0 1 0 0 9 0 1 0 0 9 2503 VecDotNorm2 1999 1.0 2.3131e+01 5.3 1.53e+09 1.1 0.0e+00 0.0e+00 2.0e+03 0 1 0 0 5 0 1 0 0 5 3050 VecNorm 3998 1.0 1.6799e+01 3.4 1.53e+09 1.1 0.0e+00 0.0e+00 4.0e+03 0 1 0 0 9 0 1 0 0 9 4200 VecCopy 3998 1.0 4.9803e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 12002 1.0 1.1035e+01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPBYCZ 3998 1.0 9.5659e+00 1.1 3.06e+09 1.1 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 14751 VecWAXPY 3998 1.0 1.0318e+01 1.1 1.53e+09 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 6838 VecAssemblyBegin 3998 1.0 1.1847e+01 8.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+04 0 0 0 0 27 0 0 0 0 27 0 VecAssemblyEnd 3998 1.0 1.0632e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 16002 1.0 1.3053e+01 1.4 0.00e+00 0.0 2.6e+06 7.0e+04 0.0e+00 0 0 76100 0 0 0 76100 0 0 VecScatterEnd 16002 1.0 2.4842e+01 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 MatMult 3998 1.0 1.5690e+02 1.2 5.86e+10 1.1 6.6e+05 1.2e+05 0.0e+00 4 37 19 43 0 4 37 19 43 0 17137 MatSolve 5997 1.0 2.3313e+02 1.1 8.47e+10 1.1 0.0e+00 0.0e+00 0.0e+00 7 53 0 0 0 7 53 0 0 0 16729 MatLUFactorNum 104 1.0 2.2413e+01 1.2 6.54e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 4 0 0 0 1 4 0 0 0 13280 MatILUFactorSym 1 1.0 1.9445e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatScale 1 1.0 5.4134e-02 1.5 7.43e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 6293 MatAssemblyBegin 105 1.0 2.0050e+0010.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.1e+02 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 105 1.0 2.4443e+00 1.1 0.00e+00 0.0 3.3e+02 3.0e+04 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 1 1.0 8.1062e-06 4.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 8.0609e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSetUp 105 1.0 1.2339e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 1999 1.0 4.6014e+02 1.0 1.59e+11 1.1 6.6e+05 1.2e+05 1.0e+04 14100 19 43 23 14100 19 43 23 15886 PCSetUp 208 1.0 2.2611e+01 1.2 6.54e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 4 0 0 0 1 4 0 0 0 13163 PCSetUpOnBlocks 1999 1.0 4.2470e-01 1.2 6.29e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 6739 PCApply 5997 1.0 2.5994e+02 1.1 9.11e+10 1.1 0.0e+00 0.0e+00 0.0e+00 8 57 0 0 0 8 57 0 0 0 16138 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Vector 4032 4032 30946112 0 Vector Scatter 2010 15 3596160 0 Matrix 4 4 182686524 0 Distributed Mesh 2003 8 39680 0 Star Forest Bipartite Graph 4006 16 13696 0 Discrete System 2003 8 6784 0 Index Set 4013 4013 15434720 0 IS L to G Mapping 2003 8 2092944 0 Krylov Solver 2 2 2296 0 Preconditioner 2 2 1896 0 Viewer 1 0 0 0 ======================================================================================================================== Average time to get PetscTime(): 9.53674e-08 Average time for MPI_Barrier(): 1.01566e-05 Average time for zero size MPI_Send(): 7.45555e-06 #PETSc Option Table entries: -log_summary #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1 ----------------------------------------- Libraries compiled on Thu Jan 7 04:05:35 2016 on hpc12 Machine characteristics: Linux-3.10.0-123.20.1.el7.x86_64-x86_64-with-centos-7.1.1503-Core Using PETSc directory: /home/wtay/Codes/petsc-3.6.3 Using PETSc arch: petsc-3.6.3_static_rel ----------------------------------------- Using C compiler: /opt/ud/openmpi-1.8.8/bin/mpicc -wd1572 -O3 ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: /opt/ud/openmpi-1.8.8/bin/mpif90 -O3 ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/include -I/home/wtay/Codes/petsc-3.6.3/include -I/home/wtay/Codes/petsc-3.6.3/include -I/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/include -I/home/wtay/Lib/petsc-3.6.3_static_rel/include -I/opt/ud/openmpi-1.8.8/include ----------------------------------------- Using C linker: /opt/ud/openmpi-1.8.8/bin/mpicc Using Fortran linker: /opt/ud/openmpi-1.8.8/bin/mpif90 Using libraries: -Wl,-rpath,/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/lib -L/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/lib -lpetsc -Wl,-rpath,/home/wtay/Lib/petsc-3.6.3_static_rel/lib -L/home/wtay/Lib/petsc-3.6.3_static_rel/lib -lHYPRE -L/opt/ud/openmpi-1.8.8/lib -L/opt/ud/intel_xe_2013sp1/composer_xe_2013_sp1.2.144/compiler/lib/intel64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.3 -lmpi_cxx -Wl,-rpath,/opt/ud/openmpi-1.8.8/lib -Wl,-rpath,/opt/ud/intel_xe_2013sp1/mkl/lib/intel64 -L/opt/ud/intel_xe_2013sp1/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -lX11 -lhwloc -lssl -lcrypto -lmpi_usempi -lmpi_mpifh -lifport -lifcore -lm -lmpi_cxx -ldl -L/opt/ud/openmpi-1.8.8/lib -lmpi -L/opt/ud/openmpi-1.8.8/lib -L/opt/ud/intel_xe_2013sp1/composer_xe_2013_sp1.2.144/compiler/lib/intel64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.3 -Wl,-rpath,/opt/ud/openmpi-1.8.8/lib -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -L/opt/ud/openmpi-1.8.8/lib -L/opt/ud/intel_xe_2013sp1/composer_xe_2013_sp1.2.144/compiler/lib/intel64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.3 -ldl ----------------------------------------- -------------- next part -------------- 0.000000000000000E+000 0.600000000000000 17.5000000000000 120.000000000000 0.000000000000000E+000 0.250000000000000 1.00000000000000 0.400000000000000 0 -400000 AB,AA,BB -2.78150003711926 2.76500003633555 2.78150003711926 2.70650003355695 size_x,size_y,size_z 100 172 171 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 0 1 29 1 43 1 124700 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 1 30 58 1 43 124701 249400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 2 59 87 1 43 249401 374100 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 3 88 116 1 43 374101 498800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 4 117 144 1 43 498801 619200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 5 145 172 1 43 619201 739600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 6 1 29 44 86 739601 864300 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 7 30 58 44 86 864301 989000 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 8 59 87 44 86 989001 1113700 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 9 88 116 44 86 1113701 1238400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 10 117 144 44 86 1238401 1358800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 11 145 172 44 86 1358801 1479200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 12 1 29 87 129 1479201 1603900 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 13 30 58 87 129 1603901 1728600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 14 59 87 87 129 1728601 1853300 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 15 88 116 87 129 1853301 1978000 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 16 117 144 87 129 1978001 2098400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 17 145 172 87 129 2098401 2218800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 18 1 29 130 171 2218801 2340600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 19 30 58 130 171 2340601 2462400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 20 59 87 130 171 2462401 2584200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 21 88 116 130 171 2584201 2706000 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 22 117 144 130 171 2706001 2823600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 23 145 172 130 171 2823601 2941200 body_cg_ini 0.850000999999998 9.999999998273846E-007 6.95771875020604 3104 surfaces with wrong vertex ordering Warning - length difference between element and cell max_element_length,min_element_length,min_delta 7.847540176996057E-002 3.349995610000001E-002 3.500000000000000E-002 maximum ngh_surfaces and ngh_vertics are 28 12 minimum ngh_surfaces and ngh_vertics are 14 5 min IIB_cell_no 0 max IIB_cell_no 229 final initial IIB_cell_no 11450 min I_cell_no 0 max I_cell_no 200 final initial I_cell_no 10000 size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u) 11450 10000 11450 10000 IIB_I_cell_no_uvw_total1 2230 2227 2166 1930 1926 1847 1 0.01411765 0.30104754 0.32529731 1.15440698 -0.30539502E+03 -0.29715696E+02 0.29394159E+07 2 0.00973086 0.41244573 0.45086899 1.22116550 -0.34890134E+03 -0.25062690E+02 0.29392110E+07 3 0.00918177 0.45383616 0.51179402 1.27757073 -0.35811483E+03 -0.25027396E+02 0.29391677E+07 4 0.00885764 0.47398774 0.55169119 1.31019526 -0.36250500E+03 -0.25910050E+02 0.29391470E+07 5 0.00872241 0.48832538 0.57967282 1.32679047 -0.36545763E+03 -0.26947216E+02 0.29391325E+07 300 0.00163886 4.27898628 6.83028522 3.60837060 -0.19609891E+04 0.43984454E+05 0.29435194E+07 600 0.00160193 3.91014241 4.97460210 5.10461274 -0.61092521E+03 0.18910563E+05 0.29467790E+07 900 0.00150521 3.27352854 5.85427996 4.49166453 -0.89281765E+04 -0.12171584E+05 0.29507471E+07 1200 0.00165280 3.05922213 7.37243530 5.16434634 -0.10954640E+05 0.22049957E+05 0.29575213E+07 1500 0.00153718 3.54908044 5.42918256 4.84940953 -0.16430153E+05 0.24407130E+05 0.29608940E+07 1800 0.00155455 3.30956962 8.35799538 4.50638757 -0.20003619E+05 -0.20349497E+05 0.29676102E+07 escape_time reached, so abort cd_cl_cs_mom_implicit1 -1.29348921431473 -2.44525665200003 -0.238725356553914 0.644444280391413 -3.056662699041206E-002 -2.91791118488116 ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./a.out on a petsc-3.6.3_static_rel named n12-09 with 24 processors, by wtay Sat Feb 27 16:58:01 2016 Using Petsc Release Version 3.6.3, Dec, 03, 2015 Max Max/Min Avg Total Time (sec): 5.791e+03 1.00001 5.791e+03 Objects: 2.008e+04 1.00000 2.008e+04 Flops: 3.129e+11 1.06806 3.066e+11 7.360e+12 Flops/sec: 5.402e+07 1.06807 5.295e+07 1.271e+09 MPI Messages: 8.298e+04 1.89703 6.585e+04 1.580e+06 MPI Message Lengths: 6.456e+09 2.05684 7.780e+04 1.229e+11 MPI Reductions: 4.427e+04 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 5.7911e+03 100.0% 7.3595e+12 100.0% 1.580e+06 100.0% 7.780e+04 100.0% 4.427e+04 100.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage VecDot 3998 1.0 1.1437e+02 2.3 2.99e+09 1.1 0.0e+00 0.0e+00 4.0e+03 1 1 0 0 9 1 1 0 0 9 617 VecDotNorm2 1999 1.0 1.0442e+02 2.6 2.99e+09 1.1 0.0e+00 0.0e+00 2.0e+03 1 1 0 0 5 1 1 0 0 5 676 VecNorm 3998 1.0 8.5426e+01 2.2 2.99e+09 1.1 0.0e+00 0.0e+00 4.0e+03 1 1 0 0 9 1 1 0 0 9 826 VecCopy 3998 1.0 7.3321e+00 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 12002 1.0 1.2399e+01 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPBYCZ 3998 1.0 1.8118e+01 1.4 5.98e+09 1.1 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 7788 VecWAXPY 3998 1.0 1.6979e+01 1.3 2.99e+09 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 4155 VecAssemblyBegin 3998 1.0 4.1001e+01 5.6 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+04 0 0 0 0 27 0 0 0 0 27 0 VecAssemblyEnd 3998 1.0 1.4657e-02 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 16002 1.0 1.9519e+01 1.5 0.00e+00 0.0 1.2e+06 1.0e+05 0.0e+00 0 0 77100 0 0 0 77100 0 0 VecScatterEnd 16002 1.0 1.3223e+02 2.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 MatMult 3998 1.0 3.0904e+02 1.3 1.15e+11 1.1 3.0e+05 1.7e+05 0.0e+00 5 37 19 43 0 5 37 19 43 0 8700 MatSolve 5997 1.0 3.9285e+02 1.4 1.67e+11 1.1 0.0e+00 0.0e+00 0.0e+00 6 54 0 0 0 6 54 0 0 0 10040 MatLUFactorNum 104 1.0 4.2097e+01 1.2 1.30e+10 1.1 0.0e+00 0.0e+00 0.0e+00 1 4 0 0 0 1 4 0 0 0 7190 MatILUFactorSym 1 1.0 2.9875e-01 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatScale 1 1.0 1.3492e-01 3.3 1.45e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 2525 MatAssemblyBegin 105 1.0 5.9000e+00 4.7 0.00e+00 0.0 0.0e+00 0.0e+00 2.1e+02 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 105 1.0 4.7665e+00 1.1 0.00e+00 0.0 1.5e+02 4.3e+04 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 1 1.0 3.6001e-0518.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 1.6249e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSetUp 105 1.0 2.7945e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 1999 1.0 9.1973e+02 1.0 3.13e+11 1.1 3.0e+05 1.7e+05 1.0e+04 16100 19 43 23 16100 19 43 23 8001 PCSetUp 208 1.0 4.2401e+01 1.2 1.30e+10 1.1 0.0e+00 0.0e+00 0.0e+00 1 4 0 0 0 1 4 0 0 0 7138 PCSetUpOnBlocks 1999 1.0 7.2389e-01 1.2 1.25e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 4020 PCApply 5997 1.0 4.4054e+02 1.3 1.80e+11 1.1 0.0e+00 0.0e+00 0.0e+00 6 58 0 0 0 6 58 0 0 0 9634 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Vector 4032 4032 53827712 0 Vector Scatter 2010 15 7012720 0 Matrix 4 4 359683260 0 Distributed Mesh 2003 8 39680 0 Star Forest Bipartite Graph 4006 16 13696 0 Discrete System 2003 8 6784 0 Index Set 4013 4013 25819112 0 IS L to G Mapping 2003 8 3919440 0 Krylov Solver 2 2 2296 0 Preconditioner 2 2 1896 0 Viewer 1 0 0 0 ======================================================================================================================== Average time to get PetscTime(): 1.90735e-07 Average time for MPI_Barrier(): 7.20024e-06 Average time for zero size MPI_Send(): 1.83781e-06 #PETSc Option Table entries: -log_summary #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1 ----------------------------------------- Libraries compiled on Thu Jan 7 04:05:35 2016 on hpc12 Machine characteristics: Linux-3.10.0-123.20.1.el7.x86_64-x86_64-with-centos-7.1.1503-Core Using PETSc directory: /home/wtay/Codes/petsc-3.6.3 Using PETSc arch: petsc-3.6.3_static_rel ----------------------------------------- Using C compiler: /opt/ud/openmpi-1.8.8/bin/mpicc -wd1572 -O3 ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: /opt/ud/openmpi-1.8.8/bin/mpif90 -O3 ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/include -I/home/wtay/Codes/petsc-3.6.3/include -I/home/wtay/Codes/petsc-3.6.3/include -I/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/include -I/home/wtay/Lib/petsc-3.6.3_static_rel/include -I/opt/ud/openmpi-1.8.8/include ----------------------------------------- Using C linker: /opt/ud/openmpi-1.8.8/bin/mpicc Using Fortran linker: /opt/ud/openmpi-1.8.8/bin/mpif90 Using libraries: -Wl,-rpath,/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/lib -L/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/lib -lpetsc -Wl,-rpath,/home/wtay/Lib/petsc-3.6.3_static_rel/lib -L/home/wtay/Lib/petsc-3.6.3_static_rel/lib -lHYPRE -L/opt/ud/openmpi-1.8.8/lib -L/opt/ud/intel_xe_2013sp1/composer_xe_2013_sp1.2.144/compiler/lib/intel64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.3 -lmpi_cxx -Wl,-rpath,/opt/ud/openmpi-1.8.8/lib -Wl,-rpath,/opt/ud/intel_xe_2013sp1/mkl/lib/intel64 -L/opt/ud/intel_xe_2013sp1/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -lX11 -lhwloc -lssl -lcrypto -lmpi_usempi -lmpi_mpifh -lifport -lifcore -lm -lmpi_cxx -ldl -L/opt/ud/openmpi-1.8.8/lib -lmpi -L/opt/ud/openmpi-1.8.8/lib -L/opt/ud/intel_xe_2013sp1/composer_xe_2013_sp1.2.144/compiler/lib/intel64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.3 -Wl,-rpath,/opt/ud/openmpi-1.8.8/lib -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -L/opt/ud/openmpi-1.8.8/lib -L/opt/ud/intel_xe_2013sp1/composer_xe_2013_sp1.2.144/compiler/lib/intel64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.3 -ldl ----------------------------------------- -------------- next part -------------- 0.000000000000000E+000 0.600000000000000 17.5000000000000 120.000000000000 0.000000000000000E+000 0.250000000000000 1.00000000000000 0.400000000000000 0 -400000 AB,AA,BB -3.06400005053729 3.04150004946860 3.06400005053729 2.98650004685624 size_x,size_y,size_z 128 217 210 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 0 1 28 1 35 1 125440 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 1 29 55 1 35 125441 246400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 2 56 82 1 35 246401 367360 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 3 83 109 1 35 367361 488320 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 4 110 136 1 35 488321 609280 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 5 137 163 1 35 609281 730240 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 6 164 190 1 35 730241 851200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 7 191 217 1 35 851201 972160 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 8 1 28 36 70 972161 1097600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 9 29 55 36 70 1097601 1218560 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 10 56 82 36 70 1218561 1339520 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 11 83 109 36 70 1339521 1460480 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 12 110 136 36 70 1460481 1581440 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 13 137 163 36 70 1581441 1702400 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 14 164 190 36 70 1702401 1823360 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 15 191 217 36 70 1823361 1944320 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 16 1 28 71 105 1944321 2069760 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 17 29 55 71 105 2069761 2190720 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 18 56 82 71 105 2190721 2311680 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 19 83 109 71 105 2311681 2432640 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 20 110 136 71 105 2432641 2553600 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 21 137 163 71 105 2553601 2674560 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 22 164 190 71 105 2674561 2795520 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 23 191 217 71 105 2795521 2916480 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 24 1 28 106 140 2916481 3041920 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 25 29 55 106 140 3041921 3162880 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 26 56 82 106 140 3162881 3283840 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 27 83 109 106 140 3283841 3404800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 28 110 136 106 140 3404801 3525760 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 29 137 163 106 140 3525761 3646720 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 30 164 190 106 140 3646721 3767680 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 31 191 217 106 140 3767681 3888640 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 32 1 28 141 175 3888641 4014080 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 33 29 55 141 175 4014081 4135040 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 34 56 82 141 175 4135041 4256000 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 35 83 109 141 175 4256001 4376960 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 36 110 136 141 175 4376961 4497920 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 37 137 163 141 175 4497921 4618880 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 38 164 190 141 175 4618881 4739840 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 39 191 217 141 175 4739841 4860800 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 40 1 28 176 210 4860801 4986240 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 41 29 55 176 210 4986241 5107200 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 42 56 82 176 210 5107201 5228160 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 43 83 109 176 210 5228161 5349120 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 44 110 136 176 210 5349121 5470080 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 45 137 163 176 210 5470081 5591040 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 46 164 190 176 210 5591041 5712000 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 47 191 217 176 210 5712001 5832960 body_cg_ini 0.850000999999998 9.999999998273846E-007 6.95771875020604 3104 surfaces with wrong vertex ordering Warning - length difference between element and cell max_element_length,min_element_length,min_delta 7.847540176996057E-002 3.349995610000001E-002 2.600000000000000E-002 maximum ngh_surfaces and ngh_vertics are 15 5 minimum ngh_surfaces and ngh_vertics are 9 3 min IIB_cell_no 0 max IIB_cell_no 293 final initial IIB_cell_no 14650 min I_cell_no 0 max I_cell_no 330 final initial I_cell_no 16500 size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u) 14650 16500 14650 16500 IIB_I_cell_no_uvw_total1 4065 4064 3990 4692 4628 4632 1 0.01043478 0.34272324 0.35569220 1.16440147 -0.68429429E+03 -0.71309583E+02 0.58290411E+07 2 0.00697380 0.42763102 0.50182401 1.25425283 -0.77750642E+03 -0.58636989E+02 0.58286176E+07 3 0.00648381 0.45520992 0.57545722 1.32431608 -0.79650320E+03 -0.55533364E+02 0.58285305E+07 4 0.00611631 0.46806365 0.62117944 1.37204042 -0.80501258E+03 -0.54624483E+02 0.58284901E+07 5 0.00592372 0.47659245 0.65166569 1.40276768 -0.81047260E+03 -0.54497615E+02 0.58284626E+07 300 0.00115066 3.67033285 6.63969480 3.71347160 0.43279356E+04 0.80255072E+05 0.58398256E+07 600 0.00122543 3.75818573 5.23852131 5.46557154 -0.66201226E+04 0.11004125E+06 0.58427867E+07 900 0.00103133 3.87890472 6.22189985 4.38435056 -0.71743226E+03 0.42678349E+05 0.58534173E+07 1200 0.00112141 3.50676323 7.96027994 5.21426706 -0.16564613E+05 -0.28161927E+05 0.58651131E+07 1500 0.00116845 5.55355360 4.80276829 4.70049706 -0.22633631E+05 -0.12833717E+05 0.58692098E+07 1800 0.00106663 3.60921979 9.26223511 4.67749400 -0.30654568E+05 0.62643544E+05 0.58867013E+07 escape_time reached, so abort cd_cl_cs_mom_implicit1 0.515674932183890 -1.12997638014196 0.113834045912081 7.478720062203850E-002 -1.299985230479712E-002 -0.226394142710332 ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./a.out on a petsc-3.6.3_static_rel named n12-04 with 48 processors, by wtay Sat Feb 27 18:01:18 2016 Using Petsc Release Version 3.6.3, Dec, 03, 2015 Max Max/Min Avg Total Time (sec): 6.333e+03 1.00000 6.333e+03 Objects: 2.008e+04 1.00000 2.008e+04 Flops: 3.132e+11 1.04090 3.039e+11 1.459e+13 Flops/sec: 4.945e+07 1.04090 4.799e+07 2.303e+09 MPI Messages: 8.482e+04 1.92729 7.176e+04 3.444e+06 MPI Message Lengths: 7.117e+09 1.99989 8.514e+04 2.933e+11 MPI Reductions: 4.427e+04 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 6.3329e+03 100.0% 1.4588e+13 100.0% 3.444e+06 100.0% 8.514e+04 100.0% 4.427e+04 100.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage VecDot 3998 1.0 4.4692e+01 2.5 3.01e+09 1.0 0.0e+00 0.0e+00 4.0e+03 0 1 0 0 9 0 1 0 0 9 3131 VecDotNorm2 1999 1.0 3.7771e+01 3.2 3.01e+09 1.0 0.0e+00 0.0e+00 2.0e+03 0 1 0 0 5 0 1 0 0 5 3704 VecNorm 3998 1.0 6.2085e+01 5.2 3.01e+09 1.0 0.0e+00 0.0e+00 4.0e+03 1 1 0 0 9 1 1 0 0 9 2254 VecCopy 3998 1.0 8.7522e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 12002 1.0 1.5554e+01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPBYCZ 3998 1.0 1.9816e+01 1.1 6.02e+09 1.0 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 14122 VecWAXPY 3998 1.0 2.0635e+01 1.1 3.01e+09 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 6781 VecAssemblyBegin 3998 1.0 1.4403e+01 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+04 0 0 0 0 27 0 0 0 0 27 0 VecAssemblyEnd 3998 1.0 1.6043e-02 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 16002 1.0 2.8572e+01 1.3 0.00e+00 0.0 2.6e+06 1.1e+05 0.0e+00 0 0 76100 0 0 0 76100 0 0 VecScatterEnd 16002 1.0 5.2833e+01 2.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 MatMult 3998 1.0 3.1279e+02 1.1 1.14e+11 1.0 6.6e+05 1.9e+05 0.0e+00 5 37 19 43 0 5 37 19 43 0 17084 MatSolve 5997 1.0 4.8329e+02 1.1 1.68e+11 1.0 0.0e+00 0.0e+00 0.0e+00 7 53 0 0 0 7 53 0 0 0 16147 MatLUFactorNum 104 1.0 9.8655e+01 1.6 1.27e+10 1.1 0.0e+00 0.0e+00 0.0e+00 1 4 0 0 0 1 4 0 0 0 6084 MatILUFactorSym 1 1.0 3.9025e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatScale 1 1.0 1.0553e-01 1.4 1.45e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 6415 MatAssemblyBegin 105 1.0 2.1250e+00 5.5 0.00e+00 0.0 0.0e+00 0.0e+00 2.1e+02 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 105 1.0 4.9930e+00 1.1 0.00e+00 0.0 3.3e+02 4.8e+04 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 1 1.0 5.0068e-06 2.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 2.0131e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSetUp 105 1.0 4.2514e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 1999 1.0 9.8702e+02 1.0 3.13e+11 1.0 6.6e+05 1.9e+05 1.0e+04 16100 19 43 23 16100 19 43 23 14779 PCSetUp 208 1.0 9.9050e+01 1.6 1.27e+10 1.1 0.0e+00 0.0e+00 0.0e+00 1 4 0 0 0 1 4 0 0 0 6060 PCSetUpOnBlocks 1999 1.0 1.3700e+00 1.4 1.22e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 4213 PCApply 5997 1.0 5.8749e+02 1.2 1.81e+11 1.0 0.0e+00 0.0e+00 0.0e+00 8 58 0 0 0 8 58 0 0 0 14295 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Vector 4032 4032 54303424 0 Vector Scatter 2010 15 7054512 0 Matrix 4 4 361394700 0 Distributed Mesh 2003 8 39680 0 Star Forest Bipartite Graph 4006 16 13696 0 Discrete System 2003 8 6784 0 Index Set 4013 4013 27475284 0 IS L to G Mapping 2003 8 3991872 0 Krylov Solver 2 2 2296 0 Preconditioner 2 2 1896 0 Viewer 1 0 0 0 ======================================================================================================================== Average time to get PetscTime(): 9.53674e-08 Average time for MPI_Barrier(): 6.2561e-05 Average time for zero size MPI_Send(): 1.70867e-06 #PETSc Option Table entries: -log_summary #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1 ----------------------------------------- Libraries compiled on Thu Jan 7 04:05:35 2016 on hpc12 Machine characteristics: Linux-3.10.0-123.20.1.el7.x86_64-x86_64-with-centos-7.1.1503-Core Using PETSc directory: /home/wtay/Codes/petsc-3.6.3 Using PETSc arch: petsc-3.6.3_static_rel ----------------------------------------- Using C compiler: /opt/ud/openmpi-1.8.8/bin/mpicc -wd1572 -O3 ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: /opt/ud/openmpi-1.8.8/bin/mpif90 -O3 ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/include -I/home/wtay/Codes/petsc-3.6.3/include -I/home/wtay/Codes/petsc-3.6.3/include -I/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/include -I/home/wtay/Lib/petsc-3.6.3_static_rel/include -I/opt/ud/openmpi-1.8.8/include ----------------------------------------- Using C linker: /opt/ud/openmpi-1.8.8/bin/mpicc Using Fortran linker: /opt/ud/openmpi-1.8.8/bin/mpif90 Using libraries: -Wl,-rpath,/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/lib -L/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/lib -lpetsc -Wl,-rpath,/home/wtay/Lib/petsc-3.6.3_static_rel/lib -L/home/wtay/Lib/petsc-3.6.3_static_rel/lib -lHYPRE -L/opt/ud/openmpi-1.8.8/lib -L/opt/ud/intel_xe_2013sp1/composer_xe_2013_sp1.2.144/compiler/lib/intel64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.3 -lmpi_cxx -Wl,-rpath,/opt/ud/openmpi-1.8.8/lib -Wl,-rpath,/opt/ud/intel_xe_2013sp1/mkl/lib/intel64 -L/opt/ud/intel_xe_2013sp1/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -lX11 -lhwloc -lssl -lcrypto -lmpi_usempi -lmpi_mpifh -lifport -lifcore -lm -lmpi_cxx -ldl -L/opt/ud/openmpi-1.8.8/lib -lmpi -L/opt/ud/openmpi-1.8.8/lib -L/opt/ud/intel_xe_2013sp1/composer_xe_2013_sp1.2.144/compiler/lib/intel64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.3 -Wl,-rpath,/opt/ud/openmpi-1.8.8/lib -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -L/opt/ud/openmpi-1.8.8/lib -L/opt/ud/intel_xe_2013sp1/composer_xe_2013_sp1.2.144/compiler/lib/intel64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.3 -ldl ----------------------------------------- -------------- next part -------------- 0.000000000000000E+000 0.600000000000000 17.5000000000000 120.000000000000 0.000000000000000E+000 0.250000000000000 1.00000000000000 0.400000000000000 0 -400000 AB,AA,BB -2.51050002424745 2.47300002246629 2.51050002424745 2.43950002087513 size_x,size_y,size_z 79 137 141 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 0 1 35 1 24 1 66360 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 1 36 69 1 24 66361 130824 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 2 70 103 1 24 130825 195288 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 3 104 137 1 24 195289 259752 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 4 1 35 25 48 259753 326112 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 5 36 69 25 48 326113 390576 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 6 70 103 25 48 390577 455040 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 7 104 137 25 48 455041 519504 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 8 1 35 49 72 519505 585864 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 9 36 69 49 72 585865 650328 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 10 70 103 49 72 650329 714792 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 11 104 137 49 72 714793 779256 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 12 1 35 73 95 779257 842851 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 13 36 69 73 95 842852 904629 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 14 70 103 73 95 904630 966407 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 15 104 137 73 95 966408 1028185 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 16 1 35 96 118 1028186 1091780 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 17 36 69 96 118 1091781 1153558 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 18 70 103 96 118 1153559 1215336 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 19 104 137 96 118 1215337 1277114 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 20 1 35 119 141 1277115 1340709 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 21 36 69 119 141 1340710 1402487 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 22 70 103 119 141 1402488 1464265 myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 23 104 137 119 141 1464266 1526043 body_cg_ini 0.850000999999998 9.999999998273846E-007 6.95771875020604 3104 surfaces with wrong vertex ordering Warning - length difference between element and cell max_element_length,min_element_length,min_delta 7.847540176996057E-002 3.349995610000001E-002 4.700000000000000E-002 maximum ngh_surfaces and ngh_vertics are 47 22 minimum ngh_surfaces and ngh_vertics are 22 9 min IIB_cell_no 0 max IIB_cell_no 112 final initial IIB_cell_no 5600 min I_cell_no 0 max I_cell_no 96 final initial I_cell_no 4800 size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u) 5600 4800 5600 4800 IIB_I_cell_no_uvw_total1 1221 1206 1212 775 761 751 1 0.01904762 0.28410536 0.31610359 1.14440147 -0.14430869E+03 -0.13111542E+02 0.15251948E+07 2 0.01348578 0.34638018 0.42392119 1.23447223 -0.16528393E+03 -0.10238827E+02 0.15250907E+07 3 0.01252674 0.38305826 0.49569053 1.27891383 -0.16912542E+03 -0.95950253E+01 0.15250695E+07 4 0.01199639 0.41337279 0.54168038 1.29584768 -0.17048065E+03 -0.94814301E+01 0.15250602E+07 5 0.01165251 0.43544137 0.57347276 1.30255981 -0.17129184E+03 -0.95170304E+01 0.15250538E+07 300 0.00236362 3.56353622 5.06727508 4.03923148 -0.78697893E+03 0.15046453E+05 0.15263125E+07 600 0.00253142 2.94537779 5.74258126 4.71794271 -0.38271069E+04 -0.49150195E+04 0.15289768E+07 900 0.00220341 3.10439489 6.70144317 4.01105348 -0.71943943E+04 0.13728311E+05 0.15320532E+07 1200 0.00245748 3.53496741 7.33163591 4.01935315 -0.85017750E+04 -0.77550358E+04 0.15350351E+07 1500 0.00244299 3.71751725 5.93463559 4.12005108 -0.95364451E+04 0.81223334E+04 0.15373061E+07 1800 0.00237474 3.49908653 5.20866314 4.69712853 -0.10382365E+05 -0.18966840E+04 0.15385160E+07 escape_time reached, so abort cd_cl_cs_mom_implicit1 -1.03894256791350 -1.53179673343374 6.737940408853320E-002 0.357464909626058 -0.103698436387821 -2.42688484514611 ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./a.out on a petsc-3.6.3_static_rel named n12-09 with 24 processors, by wtay Sat Feb 27 16:09:41 2016 Using Petsc Release Version 3.6.3, Dec, 03, 2015 Max Max/Min Avg Total Time (sec): 2.922e+03 1.00001 2.922e+03 Objects: 2.008e+04 1.00000 2.008e+04 Flops: 1.651e+11 1.08049 1.582e+11 3.797e+12 Flops/sec: 5.652e+07 1.08049 5.414e+07 1.299e+09 MPI Messages: 8.293e+04 1.89333 6.588e+04 1.581e+06 MPI Message Lengths: 4.109e+09 2.03497 4.964e+04 7.849e+10 MPI Reductions: 4.427e+04 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 2.9219e+03 100.0% 3.7965e+12 100.0% 1.581e+06 100.0% 4.964e+04 100.0% 4.427e+04 100.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage VecDot 3998 1.0 4.4655e+01 5.1 1.59e+09 1.1 0.0e+00 0.0e+00 4.0e+03 1 1 0 0 9 1 1 0 0 9 820 VecDotNorm2 1999 1.0 4.0603e+01 7.6 1.59e+09 1.1 0.0e+00 0.0e+00 2.0e+03 1 1 0 0 5 1 1 0 0 5 902 VecNorm 3998 1.0 3.0557e+01 6.2 1.59e+09 1.1 0.0e+00 0.0e+00 4.0e+03 1 1 0 0 9 1 1 0 0 9 1198 VecCopy 3998 1.0 4.4206e+00 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 12002 1.0 9.3725e+00 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPBYCZ 3998 1.0 9.1178e+00 1.5 3.18e+09 1.1 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 8030 VecWAXPY 3998 1.0 9.3186e+00 1.5 1.59e+09 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 3928 VecAssemblyBegin 3998 1.0 1.5680e+01 4.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+04 0 0 0 0 27 0 0 0 0 27 0 VecAssemblyEnd 3998 1.0 1.1443e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 16002 1.0 9.0984e+00 1.4 0.00e+00 0.0 1.2e+06 6.4e+04 0.0e+00 0 0 77100 0 0 0 77100 0 0 VecScatterEnd 16002 1.0 4.4821e+01 4.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 MatMult 3998 1.0 1.4268e+02 1.3 6.05e+10 1.1 3.0e+05 1.1e+05 0.0e+00 4 37 19 43 0 4 37 19 43 0 9753 MatSolve 5997 1.0 2.0469e+02 1.4 8.84e+10 1.1 0.0e+00 0.0e+00 0.0e+00 6 53 0 0 0 6 53 0 0 0 9921 MatLUFactorNum 104 1.0 2.2332e+01 1.1 6.70e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 4 0 0 0 1 4 0 0 0 6922 MatILUFactorSym 1 1.0 1.0867e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatScale 1 1.0 3.8305e-02 1.9 7.67e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 4603 MatAssemblyBegin 105 1.0 2.0776e+00 3.6 0.00e+00 0.0 0.0e+00 0.0e+00 2.1e+02 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 105 1.0 2.4702e+00 1.1 0.00e+00 0.0 1.5e+02 2.8e+04 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 1 1.0 4.0531e-06 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 7.1249e-03 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSetUp 105 1.0 9.8758e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 1999 1.0 4.1857e+02 1.0 1.65e+11 1.1 3.0e+05 1.1e+05 1.0e+04 14100 19 43 23 14100 19 43 23 9070 PCSetUp 208 1.0 2.2440e+01 1.1 6.70e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 4 0 0 0 1 4 0 0 0 6888 PCSetUpOnBlocks 1999 1.0 2.7087e-01 1.1 6.44e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 5487 PCApply 5997 1.0 2.3123e+02 1.3 9.50e+10 1.1 0.0e+00 0.0e+00 0.0e+00 6 58 0 0 0 6 58 0 0 0 9444 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Vector 4032 4032 31782464 0 Vector Scatter 2010 15 3738624 0 Matrix 4 4 190398024 0 Distributed Mesh 2003 8 39680 0 Star Forest Bipartite Graph 4006 16 13696 0 Discrete System 2003 8 6784 0 Index Set 4013 4013 14715400 0 IS L to G Mapping 2003 8 2137148 0 Krylov Solver 2 2 2296 0 Preconditioner 2 2 1896 0 Viewer 1 0 0 0 ======================================================================================================================== Average time to get PetscTime(): 9.53674e-08 Average time for MPI_Barrier(): 8.15392e-06 Average time for zero size MPI_Send(): 1.12454e-05 #PETSc Option Table entries: -log_summary #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1 ----------------------------------------- Libraries compiled on Thu Jan 7 04:05:35 2016 on hpc12 Machine characteristics: Linux-3.10.0-123.20.1.el7.x86_64-x86_64-with-centos-7.1.1503-Core Using PETSc directory: /home/wtay/Codes/petsc-3.6.3 Using PETSc arch: petsc-3.6.3_static_rel ----------------------------------------- Using C compiler: /opt/ud/openmpi-1.8.8/bin/mpicc -wd1572 -O3 ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: /opt/ud/openmpi-1.8.8/bin/mpif90 -O3 ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/include -I/home/wtay/Codes/petsc-3.6.3/include -I/home/wtay/Codes/petsc-3.6.3/include -I/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/include -I/home/wtay/Lib/petsc-3.6.3_static_rel/include -I/opt/ud/openmpi-1.8.8/include ----------------------------------------- Using C linker: /opt/ud/openmpi-1.8.8/bin/mpicc Using Fortran linker: /opt/ud/openmpi-1.8.8/bin/mpif90 Using libraries: -Wl,-rpath,/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/lib -L/home/wtay/Codes/petsc-3.6.3/petsc-3.6.3_static_rel/lib -lpetsc -Wl,-rpath,/home/wtay/Lib/petsc-3.6.3_static_rel/lib -L/home/wtay/Lib/petsc-3.6.3_static_rel/lib -lHYPRE -L/opt/ud/openmpi-1.8.8/lib -L/opt/ud/intel_xe_2013sp1/composer_xe_2013_sp1.2.144/compiler/lib/intel64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.3 -lmpi_cxx -Wl,-rpath,/opt/ud/openmpi-1.8.8/lib -Wl,-rpath,/opt/ud/intel_xe_2013sp1/mkl/lib/intel64 -L/opt/ud/intel_xe_2013sp1/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -lX11 -lhwloc -lssl -lcrypto -lmpi_usempi -lmpi_mpifh -lifport -lifcore -lm -lmpi_cxx -ldl -L/opt/ud/openmpi-1.8.8/lib -lmpi -L/opt/ud/openmpi-1.8.8/lib -L/opt/ud/intel_xe_2013sp1/composer_xe_2013_sp1.2.144/compiler/lib/intel64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.3 -Wl,-rpath,/opt/ud/openmpi-1.8.8/lib -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -L/opt/ud/openmpi-1.8.8/lib -L/opt/ud/intel_xe_2013sp1/composer_xe_2013_sp1.2.144/compiler/lib/intel64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.3 -ldl ----------------------------------------- From bsmith at mcs.anl.gov Sun Feb 28 19:41:45 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 28 Feb 2016 19:41:45 -0600 Subject: [petsc-users] Investigate parallel code to improve parallelism In-Reply-To: <56D399F5.9030709@gmail.com> References: <56D06F16.9000200@gmail.com> <141087E6-A7C1-41FF-B3E8-74FF587E3535@mcs.anl.gov> <56D07D0C.7050109@gmail.com> <5FBB830C-4A5E-40B2-9845-E0DC68B02BD3@mcs.anl.gov> <56D3207C.6090808@gmail.com> <91716C74-7716-4C7D-98B5-1606A8A93766@mcs.anl.gov> <56D399F5.9030709@gmail.com> Message-ID: <65862BCD-10AB-4821-9816-981D1AFFE7DE@mcs.anl.gov> > On Feb 28, 2016, at 7:08 PM, TAY Wee Beng wrote: > > Hi, > > I've attached the files for x cells running y procs. hypre is called natively I'm not sure if PETSc catches it. So you are directly creating hypre matrices and calling the hypre solver in another piece of your code? In the PETSc part of the code if you compare the 2x_y to the x_y you see that doubling the problem size resulted in 2.2 as much time for the KSPSolve. Most of this large increase is due to the increased time in the scatter which went up to 150/54. = 2.7777777777777777 but the amount of data transferred only increased by 1e5/6.4e4 = 1.5625 Normally I would not expect to see this behavior and would not expect such a large increase in the communication time. Barry > > Thanks > > On 29/2/2016 1:11 AM, Barry Smith wrote: >> As I said before, send the -log_summary output for the two processor sizes and we'll look at where it is spending its time and how it could possibly be improved. >> >> Barry >> >>> On Feb 28, 2016, at 10:29 AM, TAY wee-beng wrote: >>> >>> >>> On 27/2/2016 12:53 AM, Barry Smith wrote: >>>>> On Feb 26, 2016, at 10:27 AM, TAY wee-beng wrote: >>>>> >>>>> >>>>> On 26/2/2016 11:32 PM, Barry Smith wrote: >>>>>>> On Feb 26, 2016, at 9:28 AM, TAY wee-beng wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I have got a 3D code. When I ran with 48 procs and 11 million cells, it runs for 83 min. When I ran with 96 procs and 22 million cells, it ran for 99 min. >>>>>> This is actually pretty good! >>>>> But if I'm not wrong, if I increase the no. of cells, the parallelism will keep on decreasing. I hope it scales up to maybe 300 - 400 procs. >>> Hi, >>> >>> I think I may have mentioned this before, that is, I need to submit a proposal to request for computing nodes. In the proposal, I'm supposed to run some simulations to estimate the time it takes to run my code. Then an excel file will use my input to estimate the efficiency when I run my code with more cells. They use 2 mtds to estimate: >>> >>> 1. strong scaling, whereby I run 2 cases - 1st with n cells and x procs, then with n cells and 2x procs. From there, they can estimate my expected efficiency when I have y procs. The formula is attached in the pdf. >>> >>> 2. weak scaling, whereby I run 2 cases - 1st with n cells and x procs, then with 2n cells and 2x procs. From there, they can estimate my expected efficiency when I have y procs. The formula is attached in the pdf. >>> >>> So if I use 48 and 96 procs and get maybe 80% efficiency, by the time I hit 800 procs, I get 32% efficiency for strong scaling. They expect at least 50% efficiency for my code. To reach that, I need to achieve 89% efficiency when I use 48 and 96 procs. >>> >>> So now my qn is how accurate is this type of calculation, especially wrt to PETSc? >>> >>> Similarly, for weak scaling, is it accurate? >>> >>> Can I argue that this estimation is not suitable for PETSc or hypre? >>> >>> Thanks >>> >>> >>>>>>> So it's not that parallel. I want to find out which part of the code I need to improve. Also if PETsc and hypre is working well in parallel. What's the best way to do it? >>>>>> Run both with -log_summary and send the output for each case. This will show where the time is being spent and which parts are scaling less well. >>>>>> >>>>>> Barry >>>>> That's only for the PETSc part, right? So for other parts of the code, including hypre part, I will not be able to find out. If so, what can I use to check these parts? >>>> You will still be able to see what percentage of the time is spent in hypre and if it increases with the problem size and how much. So the information will still be useful. >>>> >>>> Barry >>>> >>>>>>> I thought of doing profiling but if the code is optimized, I wonder if it still works well. >>>>>>> >>>>>>> -- >>>>>>> Thank you. >>>>>>> >>>>>>> Yours sincerely, >>>>>>> >>>>>>> TAY wee-beng >>>>>>> >>> > > -- > Thank you > > Yours sincerely, > > TAY wee-beng > > <2x_2y.txt><2x_y.txt><4x_2y.txt> From zonexo at gmail.com Sun Feb 28 20:26:36 2016 From: zonexo at gmail.com (TAY wee-beng) Date: Mon, 29 Feb 2016 10:26:36 +0800 Subject: [petsc-users] Investigate parallel code to improve parallelism In-Reply-To: <65862BCD-10AB-4821-9816-981D1AFFE7DE@mcs.anl.gov> References: <56D06F16.9000200@gmail.com> <141087E6-A7C1-41FF-B3E8-74FF587E3535@mcs.anl.gov> <56D07D0C.7050109@gmail.com> <5FBB830C-4A5E-40B2-9845-E0DC68B02BD3@mcs.anl.gov> <56D3207C.6090808@gmail.com> <91716C74-7716-4C7D-98B5-1606A8A93766@mcs.anl.gov> <56D399F5.9030709@gmail.com> <65862BCD-10AB-4821-9816-981D1AFFE7DE@mcs.anl.gov> Message-ID: <56D3AC5C.7040208@gmail.com> On 29/2/2016 9:41 AM, Barry Smith wrote: >> On Feb 28, 2016, at 7:08 PM, TAY Wee Beng wrote: >> >> Hi, >> >> I've attached the files for x cells running y procs. hypre is called natively I'm not sure if PETSc catches it. > So you are directly creating hypre matrices and calling the hypre solver in another piece of your code? Yes because I'm using the simple structure (struct) layout for Cartesian grids. It's about twice as fast compared to BoomerAMG. I can't create PETSc matrix and use the hypre struct layout, right? > > In the PETSc part of the code if you compare the 2x_y to the x_y you see that doubling the problem size resulted in 2.2 as much time for the KSPSolve. Most of this large increase is due to the increased time in the scatter which went up to 150/54. = 2.7777777777777777 but the amount of data transferred only increased by 1e5/6.4e4 = 1.5625 Normally I would not expect to see this behavior and would not expect such a large increase in the communication time. > > Barry > > > So ideally it should be 2 instead of 2.2, is that so? May I know where are you looking at? Because I can't find the nos. So where do you think the error comes from? Or how can I troubleshoot further? Thanks >> Thanks >> >> On 29/2/2016 1:11 AM, Barry Smith wrote: >>> As I said before, send the -log_summary output for the two processor sizes and we'll look at where it is spending its time and how it could possibly be improved. >>> >>> Barry >>> >>>> On Feb 28, 2016, at 10:29 AM, TAY wee-beng wrote: >>>> >>>> >>>> On 27/2/2016 12:53 AM, Barry Smith wrote: >>>>>> On Feb 26, 2016, at 10:27 AM, TAY wee-beng wrote: >>>>>> >>>>>> >>>>>> On 26/2/2016 11:32 PM, Barry Smith wrote: >>>>>>>> On Feb 26, 2016, at 9:28 AM, TAY wee-beng wrote: >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I have got a 3D code. When I ran with 48 procs and 11 million cells, it runs for 83 min. When I ran with 96 procs and 22 million cells, it ran for 99 min. >>>>>>> This is actually pretty good! >>>>>> But if I'm not wrong, if I increase the no. of cells, the parallelism will keep on decreasing. I hope it scales up to maybe 300 - 400 procs. >>>> Hi, >>>> >>>> I think I may have mentioned this before, that is, I need to submit a proposal to request for computing nodes. In the proposal, I'm supposed to run some simulations to estimate the time it takes to run my code. Then an excel file will use my input to estimate the efficiency when I run my code with more cells. They use 2 mtds to estimate: >>>> >>>> 1. strong scaling, whereby I run 2 cases - 1st with n cells and x procs, then with n cells and 2x procs. From there, they can estimate my expected efficiency when I have y procs. The formula is attached in the pdf. >>>> >>>> 2. weak scaling, whereby I run 2 cases - 1st with n cells and x procs, then with 2n cells and 2x procs. From there, they can estimate my expected efficiency when I have y procs. The formula is attached in the pdf. >>>> >>>> So if I use 48 and 96 procs and get maybe 80% efficiency, by the time I hit 800 procs, I get 32% efficiency for strong scaling. They expect at least 50% efficiency for my code. To reach that, I need to achieve 89% efficiency when I use 48 and 96 procs. >>>> >>>> So now my qn is how accurate is this type of calculation, especially wrt to PETSc? >>>> >>>> Similarly, for weak scaling, is it accurate? >>>> >>>> Can I argue that this estimation is not suitable for PETSc or hypre? >>>> >>>> Thanks >>>> >>>> >>>>>>>> So it's not that parallel. I want to find out which part of the code I need to improve. Also if PETsc and hypre is working well in parallel. What's the best way to do it? >>>>>>> Run both with -log_summary and send the output for each case. This will show where the time is being spent and which parts are scaling less well. >>>>>>> >>>>>>> Barry >>>>>> That's only for the PETSc part, right? So for other parts of the code, including hypre part, I will not be able to find out. If so, what can I use to check these parts? >>>>> You will still be able to see what percentage of the time is spent in hypre and if it increases with the problem size and how much. So the information will still be useful. >>>>> >>>>> Barry >>>>> >>>>>>>> I thought of doing profiling but if the code is optimized, I wonder if it still works well. >>>>>>>> >>>>>>>> -- >>>>>>>> Thank you. >>>>>>>> >>>>>>>> Yours sincerely, >>>>>>>> >>>>>>>> TAY wee-beng >>>>>>>> >>>> >> -- >> Thank you >> >> Yours sincerely, >> >> TAY wee-beng >> >> <2x_2y.txt><2x_y.txt><4x_2y.txt> From bsmith at mcs.anl.gov Sun Feb 28 21:21:41 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 28 Feb 2016 21:21:41 -0600 Subject: [petsc-users] Investigate parallel code to improve parallelism In-Reply-To: <56D3AC5C.7040208@gmail.com> References: <56D06F16.9000200@gmail.com> <141087E6-A7C1-41FF-B3E8-74FF587E3535@mcs.anl.gov> <56D07D0C.7050109@gmail.com> <5FBB830C-4A5E-40B2-9845-E0DC68B02BD3@mcs.anl.gov> <56D3207C.6090808@gmail.com> <91716C74-7716-4C7D-98B5-1606A8A93766@mcs.anl.gov> <56D399F5.9030709@gmail.com> <65862BCD-10AB-4821-9816-981D1AFFE7DE@mcs.anl.gov> <56D3AC5C.7040208@gmail.com> Message-ID: > On Feb 28, 2016, at 8:26 PM, TAY wee-beng wrote: > > > On 29/2/2016 9:41 AM, Barry Smith wrote: >>> On Feb 28, 2016, at 7:08 PM, TAY Wee Beng wrote: >>> >>> Hi, >>> >>> I've attached the files for x cells running y procs. hypre is called natively I'm not sure if PETSc catches it. >> So you are directly creating hypre matrices and calling the hypre solver in another piece of your code? > > Yes because I'm using the simple structure (struct) layout for Cartesian grids. It's about twice as fast compared to BoomerAMG Understood > . I can't create PETSc matrix and use the hypre struct layout, right? >> >> In the PETSc part of the code if you compare the 2x_y to the x_y you see that doubling the problem size resulted in 2.2 as much time for the KSPSolve. Most of this large increase is due to the increased time in the scatter which went up to 150/54. = 2.7777777777777777 but the amount of data transferred only increased by 1e5/6.4e4 = 1.5625 Normally I would not expect to see this behavior and would not expect such a large increase in the communication time. >> >> Barry >> >> >> > So ideally it should be 2 instead of 2.2, is that so? Ideally > > May I know where are you looking at? Because I can't find the nos. The column labeled Avg len tells the average length of messages which increases from 6.4e4 to 1e5 while the time max increase by 2.77 (I took the sum of the VecScatterBegin and VecScatter End rows. > > So where do you think the error comes from? It is not really an error it is just that it is taking more time then one would hope it would take. > Or how can I troubleshoot further? If you run the same problem several times how much different are the numerical timings for each run? > > Thanks >>> Thanks >>> >>> On 29/2/2016 1:11 AM, Barry Smith wrote: >>>> As I said before, send the -log_summary output for the two processor sizes and we'll look at where it is spending its time and how it could possibly be improved. >>>> >>>> Barry >>>> >>>>> On Feb 28, 2016, at 10:29 AM, TAY wee-beng wrote: >>>>> >>>>> >>>>> On 27/2/2016 12:53 AM, Barry Smith wrote: >>>>>>> On Feb 26, 2016, at 10:27 AM, TAY wee-beng wrote: >>>>>>> >>>>>>> >>>>>>> On 26/2/2016 11:32 PM, Barry Smith wrote: >>>>>>>>> On Feb 26, 2016, at 9:28 AM, TAY wee-beng wrote: >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I have got a 3D code. When I ran with 48 procs and 11 million cells, it runs for 83 min. When I ran with 96 procs and 22 million cells, it ran for 99 min. >>>>>>>> This is actually pretty good! >>>>>>> But if I'm not wrong, if I increase the no. of cells, the parallelism will keep on decreasing. I hope it scales up to maybe 300 - 400 procs. >>>>> Hi, >>>>> >>>>> I think I may have mentioned this before, that is, I need to submit a proposal to request for computing nodes. In the proposal, I'm supposed to run some simulations to estimate the time it takes to run my code. Then an excel file will use my input to estimate the efficiency when I run my code with more cells. They use 2 mtds to estimate: >>>>> >>>>> 1. strong scaling, whereby I run 2 cases - 1st with n cells and x procs, then with n cells and 2x procs. From there, they can estimate my expected efficiency when I have y procs. The formula is attached in the pdf. >>>>> >>>>> 2. weak scaling, whereby I run 2 cases - 1st with n cells and x procs, then with 2n cells and 2x procs. From there, they can estimate my expected efficiency when I have y procs. The formula is attached in the pdf. >>>>> >>>>> So if I use 48 and 96 procs and get maybe 80% efficiency, by the time I hit 800 procs, I get 32% efficiency for strong scaling. They expect at least 50% efficiency for my code. To reach that, I need to achieve 89% efficiency when I use 48 and 96 procs. >>>>> >>>>> So now my qn is how accurate is this type of calculation, especially wrt to PETSc? >>>>> >>>>> Similarly, for weak scaling, is it accurate? >>>>> >>>>> Can I argue that this estimation is not suitable for PETSc or hypre? >>>>> >>>>> Thanks >>>>> >>>>> >>>>>>>>> So it's not that parallel. I want to find out which part of the code I need to improve. Also if PETsc and hypre is working well in parallel. What's the best way to do it? >>>>>>>> Run both with -log_summary and send the output for each case. This will show where the time is being spent and which parts are scaling less well. >>>>>>>> >>>>>>>> Barry >>>>>>> That's only for the PETSc part, right? So for other parts of the code, including hypre part, I will not be able to find out. If so, what can I use to check these parts? >>>>>> You will still be able to see what percentage of the time is spent in hypre and if it increases with the problem size and how much. So the information will still be useful. >>>>>> >>>>>> Barry >>>>>> >>>>>>>>> I thought of doing profiling but if the code is optimized, I wonder if it still works well. >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Thank you. >>>>>>>>> >>>>>>>>> Yours sincerely, >>>>>>>>> >>>>>>>>> TAY wee-beng >>>>>>>>> >>>>> >>> -- >>> Thank you >>> >>> Yours sincerely, >>> >>> TAY wee-beng >>> >>> <2x_2y.txt><2x_y.txt><4x_2y.txt> > From davydden at gmail.com Mon Feb 29 01:45:41 2016 From: davydden at gmail.com (Denis Davydov) Date: Mon, 29 Feb 2016 08:45:41 +0100 Subject: [petsc-users] [SLEPc] record against which PETSc installation it was compiled Message-ID: <2C3C0E47-ED3A-4E5C-8F28-081D299E7BC1@gmail.com> Dear all, It would be good if SLEPc would store a location of PETSc used during the build in some config file, e.g. `slepcconf.h`, so that this information could be retrieved by external libraries (e.g. deal.ii) to prevent configuring with PETSc and SLEPc while SLEPc was linking to a different PETSc installation. See the discussion here https://github.com/dealii/dealii/issues/2167 Kind regards, Denis From jroman at dsic.upv.es Mon Feb 29 11:50:00 2016 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 29 Feb 2016 17:50:00 +0000 Subject: [petsc-users] [SLEPc] record against which PETSc installation it was compiled In-Reply-To: <2C3C0E47-ED3A-4E5C-8F28-081D299E7BC1@gmail.com> References: <2C3C0E47-ED3A-4E5C-8F28-081D299E7BC1@gmail.com> Message-ID: > El 29 feb 2016, a las 7:45, Denis Davydov escribi?: > > Dear all, > > It would be good if SLEPc would store a location of PETSc used during the build in some > config file, e.g. `slepcconf.h`, so that this information could be retrieved by external libraries (e.g. deal.ii) > to prevent configuring with PETSc and SLEPc while SLEPc was linking to a different PETSc installation. > See the discussion here https://github.com/dealii/dealii/issues/2167 > > Kind regards, > Denis > I have added this: https://bitbucket.org/slepc/slepc/branch/jose/configure However, I am not totally convinced of this change, because PETSC_DIR is then defined both in petscconf.h and slepcconf.h, so behaviour could change depending on whether user code includes one or the other in the first place. I will test this further before merging into master. Jose From davydden at gmail.com Mon Feb 29 11:58:16 2016 From: davydden at gmail.com (Denis Davydov) Date: Mon, 29 Feb 2016 18:58:16 +0100 Subject: [petsc-users] [SLEPc] record against which PETSc installation it was compiled In-Reply-To: References: <2C3C0E47-ED3A-4E5C-8F28-081D299E7BC1@gmail.com> Message-ID: <65F5A6CC-5E3A-43CD-B0C3-1315317692E2@gmail.com> > On 29 Feb 2016, at 18:50, Jose E. Roman wrote: > > >> El 29 feb 2016, a las 7:45, Denis Davydov escribi?: >> >> Dear all, >> >> It would be good if SLEPc would store a location of PETSc used during the build in some >> config file, e.g. `slepcconf.h`, so that this information could be retrieved by external libraries (e.g. deal.ii) >> to prevent configuring with PETSc and SLEPc while SLEPc was linking to a different PETSc installation. >> See the discussion here https://github.com/dealii/dealii/issues/2167 >> >> Kind regards, >> Denis >> > > I have added this: > https://bitbucket.org/slepc/slepc/branch/jose/configure > > However, I am not totally convinced of this change, because PETSC_DIR is then defined both in petscconf.h and slepcconf.h, so behaviour could change depending on whether user code includes one or the other in the first place. Thanks a lot, Jose. As an alternative one could call it PETSC_DIR_IN_SLEPC or alike, so that the two are different. Regards, Denis. > > I will test this further before merging into master. > > Jose > From bhatiamanav at gmail.com Mon Feb 29 14:04:03 2016 From: bhatiamanav at gmail.com (Manav Bhatia) Date: Mon, 29 Feb 2016 14:04:03 -0600 Subject: [petsc-users] VecGetValuesBlocked Message-ID: Hi, VecGetValuesBlocked is referenced by some documentation pages (http://www.mcs.anl.gov/petsc/petsc-3.6/docs/manualpages/Vec/VecGetValues.html), but does not seem to be in the code any more. Is VecGetValues the default method to get values from blocked storage of vectors as well? If so, then are the rows in a single block assumed to be continuously numbered? Thanks, Manav -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Feb 29 14:10:11 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 29 Feb 2016 14:10:11 -0600 Subject: [petsc-users] VecGetValuesBlocked In-Reply-To: References: Message-ID: <9EDA3C06-CF76-4E19-A7B6-1212DF6A101B@mcs.anl.gov> > On Feb 29, 2016, at 2:04 PM, Manav Bhatia wrote: > > Hi, > > VecGetValuesBlocked is referenced by some documentation pages (http://www.mcs.anl.gov/petsc/petsc-3.6/docs/manualpages/Vec/VecGetValues.html), but does not seem to be in the code any more. Yes looks like VecGetValuesBlocked doesn't exist > > Is VecGetValues the default method to get values from blocked storage of vectors as well? Yes > If so, then are the rows in a single block assumed to be continuously numbered? Yes An alternative to using VecGetValues() is to use VecGetArray() and then access the needed values from the array. To convert from global indices to the local indices needed in accessing in the array you can use VecGetOwnershipRange() and subtract off the "low" amount. Barry > > Thanks, > Manav > > From elbueler at alaska.edu Mon Feb 29 15:24:44 2016 From: elbueler at alaska.edu (Ed Bueler) Date: Mon, 29 Feb 2016 12:24:44 -0900 Subject: [petsc-users] how to get full trajectory from TS into petsc binary Message-ID: Dear PETSc -- I have a short C ode code that uses TS to solve y' = g(t,y) where y(t) is a 2-dim'l vector. My code defaults to -ts_type rk so it does adaptive time-stepping; thus using -ts_monitor shows times at stdout: $ ./ode -ts_monitor solving from t0 = 0.000 with initial time step dt = 0.10000 ... 0 TS dt 0.1 time 0. 1 TS dt 0.170141 time 0.1 2 TS dt 0.169917 time 0.270141 3 TS dt 0.171145 time 0.440058 4 TS dt 0.173931 time 0.611203 5 TS dt 0.178719 time 0.785134 6 TS dt 0.0361473 time 0.963853 7 TS dt 0.188252 time 1. error at tf = 1.000 : |y-y_exact|_inf = 0.000144484 I want to output the trajectory in PETSc binary and plot it in python using bin/PetscBinaryIO.py. Clearly I need the times shown above to do that. Note "-ts_monitor_solution binary:XX" gives me a binary file with only y values in it, but not the corresponding times. My question is, how to get those times in either the same binary file (preferred) or separate binary files? I have tried $ ./ode -ts_monitor binary:foo.dat # invalid $ ./ode -ts_monitor_solution binary:bar.dat # no t in file $ ./ode -ts_monitor_solution binary:baz.dat -ts_save_trajectory # no t in file without success. (I am not sure what the boolean option -ts_save_trajectory does, by the way.) Thanks! Ed PS Sorry if this is a "RTFM" question, but so far I can't find the documentation. -- Ed Bueler Dept of Math and Stat and Geophysical Institute University of Alaska Fairbanks Fairbanks, AK 99775-6660 301C Chapman and 410D Elvey 907 474-7693 and 907 474-7199 (fax 907 474-5394) -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Mon Feb 29 16:32:08 2016 From: mfadams at lbl.gov (Mark Adams) Date: Mon, 29 Feb 2016 17:32:08 -0500 Subject: [petsc-users] GAMG and near-null-space when applying Dirichlet Conditions In-Reply-To: <5EBAA0B1-E848-4782-A72B-3E67D0D99279@gmail.com> References: <21F632BF-5E41-48B2-B273-763A7EA97BD4@gmail.com> <5EBAA0B1-E848-4782-A72B-3E67D0D99279@gmail.com> Message-ID: On Fri, Feb 26, 2016 at 12:19 PM, Manav Bhatia wrote: > I have now experimented with different AMG solvers (gamg, ML, hypre ) > through petsc, and have a mixed bag of results. I have used > -pc_gamg_threshold 0.1 for all cases. > This should be -pc_gamg_agg_threshold X, and 0.1 is way too high. a negative number keeps all entries that you add in the matrix, zero drops only zero entries, > 0 drops stuff. 0.05 is at the high end of what is probably useful, but you can check. This is a very problem dependant parameter. > > The problem is that of plate-bending that is clamped on all ends, and has > a uniformly distributed load. > > The problem has 6 dofs per node: {u, v, w, tx, ty, tz}. u, v are the > in-plane deformations related to membrane action. w, tx, ty get the > stiffness from the Mandlin first-order shear deformation theory. tz doesn?t > really do anything in the problem, and the stiffness matrix has small > diagonal values to avoid singularity problems. > > > I have tested AMG solvers for number of unknowns from a few hundred to > about 1.5e6. > > First off, I am absolutely thrilled to be able to solve that large a > system of equations coming from a bending operator on my laptop! So a big > thanks to the petsc team for giving us the tools! > > I have not done a very thorough convergence study, but following are some > general observations: > > ? Without providing the near null space, all three solvers work. > > ? The convergence of the solvers is significantly better when the near > null space is provided. There are 6 near-null space modes provided: 3 > rigid-body translations and 3-rigid body rotations. > > ? With the near null space provided, both hypre and ML work without > problems, but GAMG quits the error of zero-pivot in LU decomposition. I am > guessing this happens for the coarsest level. I was able to get around this > with -mg_levels_pc_type jacobi . (I saw some earlier discussion on the > mailing list about this, and got the sense that this may be a > non-deterministic issue (?) ). > > ? With -pc_gamg_threshold 0.1 and -pc_mg_type full, I get the fastest > convergence from ML. > > ? GAMG seems to take about twice the amount of memory than ML. > > > I am now keen to play around with various parameters to see how to > influence the convergence. > > Any comments would be greatly appreciated. > > Regards, > Manav > > > > On Feb 25, 2016, at 6:21 AM, Mark Adams wrote: > > I added ", which is often the null space of the operator without boundary > conditions" to the web page doc for MatSetNearNullSpace. > > On Wed, Feb 24, 2016 at 10:57 AM, Matthew Knepley > wrote: > >> On Wed, Feb 24, 2016 at 9:45 AM, Manav Bhatia >> wrote: >> >>> Hi, >>> >>> I typically apply Dirichlet BCs by modifying the Jacobin and rhs: >>> zero constrained rows of matrix with 1.0 at diagonal, and zero >>> corresponding rows of rhs. >>> >>> While using GAMG, is it still recommended to provide the near-null >>> space (given that the zero-eigenvalues have been removed by specification >>> of DIrichlet BCs)? >>> >> >> Yes. >> >> >>> If that information is still needed, should the vectors be modified >>> in any manner to be consistent with the Dirichlet BCs? >>> >> >> No. You can see that if you take a small piece of the domain, apart from >> the boundary, it will have this as a null space. >> >> Matt >> >> >>> Thanks, >>> Manav >>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Mon Feb 29 16:36:07 2016 From: mfadams at lbl.gov (Mark Adams) Date: Mon, 29 Feb 2016 17:36:07 -0500 Subject: [petsc-users] Neumann BC with non-symmetric matrix In-Reply-To: References: <715E1F2D-F735-4419-811A-590E2218516F@mcs.anl.gov> Message-ID: > > >> GAMG is use for AMR problems like this a lot in BISICLES. >> > > Thanks for the reference. However, a quick look at their paper suggests > they are using a finite volume discretization which should be symmetric and > avoid all the shenanigans I'm going through! > No, they are not symmetric. FV is even worse than vertex centered methods. The BCs and the C-F interfaces add non-symmetry. -------------- next part -------------- An HTML attachment was scrubbed... URL: From griffith at cims.nyu.edu Mon Feb 29 16:42:19 2016 From: griffith at cims.nyu.edu (Boyce Griffith) Date: Mon, 29 Feb 2016 17:42:19 -0500 Subject: [petsc-users] Neumann BC with non-symmetric matrix In-Reply-To: References: <715E1F2D-F735-4419-811A-590E2218516F@mcs.anl.gov> Message-ID: <3C5B4DBF-17B5-434A-8BAD-C3D8DEE30279@cims.nyu.edu> > On Feb 29, 2016, at 5:36 PM, Mark Adams wrote: > > > GAMG is use for AMR problems like this a lot in BISICLES. > > Thanks for the reference. However, a quick look at their paper suggests they are using a finite volume discretization which should be symmetric and avoid all the shenanigans I'm going through! > > No, they are not symmetric. FV is even worse than vertex centered methods. The BCs and the C-F interfaces add non-symmetry. If you use a different discretization, it is possible to make the c-f interface discretization symmetric --- but symmetry appears to come at a cost of the reduction in the formal order of accuracy in the flux along the c-f interface. I can probably dig up some code that would make it easy to compare. -- Boyce -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Feb 29 23:53:24 2016 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 29 Feb 2016 23:53:24 -0600 Subject: [petsc-users] how to get full trajectory from TS into petsc binary In-Reply-To: References: Message-ID: Ed, I have added a branch barry/feature-ts-monitor-binary that supports -ts_monitor binary:timesteps that will store in simple binary format each of the time steps associated with each solution. This in conjugation with -ts_monitor_solution binary:solutions will give you two files you can read in. But note that timesteps is a simple binary file of double precision numbers you should read in directly in python, you cannot use PetscBinaryIO.py which is what you will use to read in the solutions file. Barry Currently PETSc has a binary file format where we can save Vec, Mat, IS, each is marked with a type id for PetscBinaryIO.py to detect, we do not have type ids for simple double precision numbers or arrays of numbers. This is why I have no way of saving the time steps in a way that PetscBinaryIO.py could read them in currently. I don't know how far we want to go in "spiffing up" the PETSc binary format to do more elaborate things since, presumably, other more power IO tools exist that would be used for "real" problems? > On Feb 29, 2016, at 3:24 PM, Ed Bueler wrote: > > Dear PETSc -- > > I have a short C ode code that uses TS to solve y' = g(t,y) where y(t) is a 2-dim'l vector. My code defaults to -ts_type rk so it does adaptive time-stepping; thus using -ts_monitor shows times at stdout: > > $ ./ode -ts_monitor > solving from t0 = 0.000 with initial time step dt = 0.10000 ... > 0 TS dt 0.1 time 0. > 1 TS dt 0.170141 time 0.1 > 2 TS dt 0.169917 time 0.270141 > 3 TS dt 0.171145 time 0.440058 > 4 TS dt 0.173931 time 0.611203 > 5 TS dt 0.178719 time 0.785134 > 6 TS dt 0.0361473 time 0.963853 > 7 TS dt 0.188252 time 1. > error at tf = 1.000 : |y-y_exact|_inf = 0.000144484 > > I want to output the trajectory in PETSc binary and plot it in python using bin/PetscBinaryIO.py. Clearly I need the times shown above to do that. > > Note "-ts_monitor_solution binary:XX" gives me a binary file with only y values in it, but not the corresponding times. > > My question is, how to get those times in either the same binary file (preferred) or separate binary files? I have tried > > $ ./ode -ts_monitor binary:foo.dat # invalid > $ ./ode -ts_monitor_solution binary:bar.dat # no t in file > $ ./ode -ts_monitor_solution binary:baz.dat -ts_save_trajectory # no t in file > > without success. (I am not sure what the boolean option -ts_save_trajectory does, by the way.) > > Thanks! > > Ed > > PS Sorry if this is a "RTFM" question, but so far I can't find the documentation. > > > -- > Ed Bueler > Dept of Math and Stat and Geophysical Institute > University of Alaska Fairbanks > Fairbanks, AK 99775-6660 > 301C Chapman and 410D Elvey > 907 474-7693 and 907 474-7199 (fax 907 474-5394)