From stali at geology.wisc.edu Sun Sep 1 07:37:07 2013 From: stali at geology.wisc.edu (Tabrez Ali) Date: Sun, 01 Sep 2013 07:37:07 -0500 Subject: [petsc-users] GAMG and linear elasticity In-Reply-To: <52224CD7.8080406@tudelft.nl> References: <521D01E6.7070003@geology.wisc.edu> <87bo4i99gx.fsf@mcs.anl.gov> <522218F9.7090805@geology.wisc.edu> <52224CD7.8080406@tudelft.nl> Message-ID: <522334F3.70505@geology.wisc.edu> I am using solid elements and here's what I get for the smallest problem with -pc_type svd -pc_svd_monitor SVD: condition number 4.184203727999e+13, 0 of 2187 singular values are (nearly) zero SVD: smallest singular values: 9.975131304660e-01 9.979349868120e-01 9.986761984005e-01 9.993267254639e-01 9.996135846381e-01 SVD: largest singular values : 3.933977100529e+13 3.933977100529e+13 4.162059658259e+13 4.162059658259e+13 4.173798159224e+13 Tabrez On 08/31/2013 03:06 PM, Umut Tabak wrote: > On 08/31/2013 06:25 PM, Tabrez Ali wrote: >> Hello >> >> So I used PCSetCoordinates and now GAMG seems to work really well in >> that the number of iterations are relatively constant. Here are the >> number of iterations on 4 cores >> >> DOF ASM GAMG >> 2187 15 22 >> 14739 26 22 >> 107811 51 29 > > Hi, > > Just curious, what is the approximate condition number of this matrix, > is that modelled with domain elements like solids or thin elements > like shells? > > Best, > U. From stali at geology.wisc.edu Sun Sep 1 07:57:42 2013 From: stali at geology.wisc.edu (Tabrez Ali) Date: Sun, 01 Sep 2013 07:57:42 -0500 Subject: [petsc-users] GAMG and linear elasticity In-Reply-To: <517B410A-A301-4E9C-A827-E5BFFBA17B1A@lbl.gov> References: <521D01E6.7070003@geology.wisc.edu> <87bo4i99gx.fsf@mcs.anl.gov> <522218F9.7090805@geology.wisc.edu> <517B410A-A301-4E9C-A827-E5BFFBA17B1A@lbl.gov> Message-ID: <522339C6.6050209@geology.wisc.edu> OK I will fix it to only pass local values. What should I do when I have additional linear constraint equations which are implemented using Lagrange multipliers (I use it even for things like non-zero displacement/pressure BCs)? Also with ex56.c I dont see a huge see any difference when PCSetCoordinates is used. Here's the number of iterations I get when I use it as it is (Note the increasing 'ne' value; Only the final iteration count is shown) stali at i5:~/petsc-3.3-p4/src/ksp/ksp/examples/tutorials$ mpirun -np 2 ./ex56 -ne 2 -alpha 1.e-3 -ksp_monitor 6 KSP Residual norm 7.616482944250e-05 stali at i5:~/petsc-3.3-p4/src/ksp/ksp/examples/tutorials$ mpirun -np 2 ./ex56 -ne 4 -alpha 1.e-3 -ksp_monitor 8 KSP Residual norm 2.561028520143e-03 stali at i5:~/petsc-3.3-p4/src/ksp/ksp/examples/tutorials$ mpirun -np 2 ./ex56 -ne 8 -alpha 1.e-3 -ksp_monitor 12 KSP Residual norm 6.460398845075e-03 stali at i5:~/petsc-3.3-p4/src/ksp/ksp/examples/tutorials$ mpirun -np 2 ./ex56 -ne 16 -alpha 1.e-3 -ksp_monitor 15 KSP Residual norm 1.888183406824e-02 And here's what I get if I comment out the line "ierr = PCSetCoordinates( pc, 3, m/3, coords );" stali at i5:~/petsc-3.3-p4/src/ksp/ksp/examples/tutorials$ mpirun -np 2 ./ex56 -ne 2 -alpha 1.e-3 -ksp_monitor [0]PCSetData_AGG bs=3 MM=81 4 KSP Residual norm 8.448311817203e-04 stali at i5:~/petsc-3.3-p4/src/ksp/ksp/examples/tutorials$ mpirun -np 2 ./ex56 -ne 4 -alpha 1.e-3 -ksp_monitor [0]PCSetData_AGG bs=3 MM=375 7 KSP Residual norm 3.281335307043e-03 stali at i5:~/petsc-3.3-p4/src/ksp/ksp/examples/tutorials$ mpirun -np 2 ./ex56 -ne 8 -alpha 1.e-3 -ksp_monitor [0]PCSetData_AGG bs=3 MM=2187 12 KSP Residual norm 4.324990561199e-03 stali at i5:~/petsc-3.3-p4/src/ksp/ksp/examples/tutorials$ mpirun -np 2 ./ex56 -ne 16 -alpha 1.e-3 -ksp_monitor [0]PCSetData_AGG bs=3 MM=14739 17 KSP Residual norm 7.038154621679e-03 Tabrez On 08/31/2013 04:06 PM, Mark F. Adams wrote: > On Aug 31, 2013, at 12:25 PM, Tabrez Ali wrote: > >> Hello >> >> So I used PCSetCoordinates and now GAMG seems to work really well in that the number of iterations are relatively constant. Here are the number of iterations on 4 cores >> >> DOF ASM GAMG >> 2187 15 22 >> 14739 26 22 >> 107811 51 29 >> >> So in PCSetCoordinates the 'coords' array should include values for the ghost nodes as well or only those values that correspond to the local owned sol'n vector? > Local only. > >> In the experiment above I included values of the ghost nodes as well (just had to add a line in my existing code) and it seems to have worked fine. >> > You tacked it onto the end of the array and so no harm done, we just did not read it. > > And you might want to use MatNullSpaceCreateRigidBody to create these vectors from the coordinates. This would add one extra step but it 1) is the preferred way and 2) it sounds like you want to something like Stokes and you could run with modify the vectors from MatNullSpaceCreateRigidBody to do an all MG solver (and dump this fieldsplit crap :) SOR smoothers inode matrices are actually vertex blocked smoothers and so they are stable even though they have a zero on the diagonal (just order pressure last). > > I think Jed mentioned this to you but specifically you can take the vectors that come out of MatNullSpaceCreateRigidBody and think if it as a tall skinny matrix: 3*n x 6. For the 3x6 matrix for each (n) vertex, call this Q, create a 4x7 matrix: > > Q 0 > 0 1.0 > > and give that to GAMG (i.e., 7 vectors of size 4*n). This would be very interesting to see how it works compared to fieldsplit. > > Oh, and pressure has to be a vertex variable. > >> Thanks in advance >> >> Tabrez >> >> On 08/27/2013 03:15 PM, Jed Brown wrote: >>> Tabrez Ali writes: >>> >>>> Hello >>>> >>>> What is the proper way to use GAMG on a vanilla 3D linear elasticity >>>> problem. Should I use >>>> >>>> -pc_type gamg -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 >>> Yeah, and only the first of these is needed because the others are >>> default with -pc_type gamg. >>> >>>> -pc_type fieldsplit -pc_fieldsplit_block_size 3 -fieldsplit_pc_type gamg >>>> -fieldsplit_pc_gamg_type agg -fieldsplit_pc_gamg_agg_nsmooths 1 >>>> >>>> Do these options even make sense? With the second set of options the % >>>> increase in number of iterations with increasing problem size is lower >>>> than the first but not optimal. >>> And it's probably more expensive because it has to do inner solves. >>> Also, if you have less compressible regions, it will get much worse. >>> >>>> Also, ksp/ksp/examples/ex56 performs much better in that the number of >>>> iterations remain more or less constant unlike what I see with my own >>>> problem. What am I doing wrong? >>> You probably forgot to set the near null space. You can use >>> MatSetNearNullSpace (and maybe MatNullSpaceCreateRigidBody) or the more >>> hacky (IMO) PCSetCoordinates. It's important to have translational >>> *and* rotational modes in the near null space that GAMG uses to build a >>> coarse space. >> >> -- >> No one trusts a model except the one who wrote it; Everyone trusts an observation except the one who made it- Harlow Shapley >> >> From u.tabak at tudelft.nl Sun Sep 1 09:33:53 2013 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sun, 01 Sep 2013 16:33:53 +0200 Subject: [petsc-users] GAMG and linear elasticity In-Reply-To: <522334F3.70505@geology.wisc.edu> References: <521D01E6.7070003@geology.wisc.edu> <87bo4i99gx.fsf@mcs.anl.gov> <522218F9.7090805@geology.wisc.edu> <52224CD7.8080406@tudelft.nl> <522334F3.70505@geology.wisc.edu> Message-ID: <52235051.5080903@tudelft.nl> On 09/01/2013 02:37 PM, Tabrez Ali wrote: > I am using solid elements and here's what I get for the smallest > problem with -pc_type svd -pc_svd_monitor > > SVD: condition number 4.184203727999e+13, 0 of 2187 singular > values are (nearly) zero I am quite surprised with this condition number by the use of solid elements. Do degrees of freedom include rotations also which might explain the difference in scaling? From mfadams at lbl.gov Sun Sep 1 10:40:53 2013 From: mfadams at lbl.gov (Mark F. Adams) Date: Sun, 1 Sep 2013 11:40:53 -0400 Subject: [petsc-users] GAMG and linear elasticity In-Reply-To: <522339C6.6050209@geology.wisc.edu> References: <521D01E6.7070003@geology.wisc.edu> <87bo4i99gx.fsf@mcs.anl.gov> <522218F9.7090805@geology.wisc.edu> <517B410A-A301-4E9C-A827-E5BFFBA17B1A@lbl.gov> <522339C6.6050209@geology.wisc.edu> Message-ID: <28CBAED3-DF1F-4909-B642-C4D5D05EEF14@lbl.gov> On Sep 1, 2013, at 8:57 AM, Tabrez Ali wrote: > OK I will fix it to only pass local values. > > What should I do when I have additional linear constraint equations which are implemented using Lagrange multipliers (I use it even for things like non-zero displacement/pressure BCs)? > > Also with ex56.c I dont see a huge see any difference when PCSetCoordinates is used. Here's the number of iterations I get when I use it as it is (Note the increasing 'ne' value; Only the final iteration count is shown) > The RHS/solution in ex56 does not have any rotational component so the rotation modes are not that important. > stali at i5:~/petsc-3.3-p4/src/ksp/ksp/examples/tutorials$ mpirun -np 2 ./ex56 -ne 2 -alpha 1.e-3 -ksp_monitor > 6 KSP Residual norm 7.616482944250e-05 > stali at i5:~/petsc-3.3-p4/src/ksp/ksp/examples/tutorials$ mpirun -np 2 ./ex56 -ne 4 -alpha 1.e-3 -ksp_monitor > 8 KSP Residual norm 2.561028520143e-03 > stali at i5:~/petsc-3.3-p4/src/ksp/ksp/examples/tutorials$ mpirun -np 2 ./ex56 -ne 8 -alpha 1.e-3 -ksp_monitor > 12 KSP Residual norm 6.460398845075e-03 > stali at i5:~/petsc-3.3-p4/src/ksp/ksp/examples/tutorials$ mpirun -np 2 ./ex56 -ne 16 -alpha 1.e-3 -ksp_monitor > 15 KSP Residual norm 1.888183406824e-02 > > And here's what I get if I comment out the line "ierr = PCSetCoordinates( pc, 3, m/3, coords );" > > stali at i5:~/petsc-3.3-p4/src/ksp/ksp/examples/tutorials$ mpirun -np 2 ./ex56 -ne 2 -alpha 1.e-3 -ksp_monitor > [0]PCSetData_AGG bs=3 MM=81 > 4 KSP Residual norm 8.448311817203e-04 > stali at i5:~/petsc-3.3-p4/src/ksp/ksp/examples/tutorials$ mpirun -np 2 ./ex56 -ne 4 -alpha 1.e-3 -ksp_monitor > [0]PCSetData_AGG bs=3 MM=375 > 7 KSP Residual norm 3.281335307043e-03 > stali at i5:~/petsc-3.3-p4/src/ksp/ksp/examples/tutorials$ mpirun -np 2 ./ex56 -ne 8 -alpha 1.e-3 -ksp_monitor > [0]PCSetData_AGG bs=3 MM=2187 > 12 KSP Residual norm 4.324990561199e-03 > stali at i5:~/petsc-3.3-p4/src/ksp/ksp/examples/tutorials$ mpirun -np 2 ./ex56 -ne 16 -alpha 1.e-3 -ksp_monitor > [0]PCSetData_AGG bs=3 MM=14739 > 17 KSP Residual norm 7.038154621679e-03 > > Tabrez > > > On 08/31/2013 04:06 PM, Mark F. Adams wrote: >> On Aug 31, 2013, at 12:25 PM, Tabrez Ali wrote: >> >>> Hello >>> >>> So I used PCSetCoordinates and now GAMG seems to work really well in that the number of iterations are relatively constant. Here are the number of iterations on 4 cores >>> >>> DOF ASM GAMG >>> 2187 15 22 >>> 14739 26 22 >>> 107811 51 29 >>> >>> So in PCSetCoordinates the 'coords' array should include values for the ghost nodes as well or only those values that correspond to the local owned sol'n vector? >> Local only. >> >>> In the experiment above I included values of the ghost nodes as well (just had to add a line in my existing code) and it seems to have worked fine. >>> >> You tacked it onto the end of the array and so no harm done, we just did not read it. >> >> And you might want to use MatNullSpaceCreateRigidBody to create these vectors from the coordinates. This would add one extra step but it 1) is the preferred way and 2) it sounds like you want to something like Stokes and you could run with modify the vectors from MatNullSpaceCreateRigidBody to do an all MG solver (and dump this fieldsplit crap :) SOR smoothers inode matrices are actually vertex blocked smoothers and so they are stable even though they have a zero on the diagonal (just order pressure last). >> >> I think Jed mentioned this to you but specifically you can take the vectors that come out of MatNullSpaceCreateRigidBody and think if it as a tall skinny matrix: 3*n x 6. For the 3x6 matrix for each (n) vertex, call this Q, create a 4x7 matrix: >> >> Q 0 >> 0 1.0 >> >> and give that to GAMG (i.e., 7 vectors of size 4*n). This would be very interesting to see how it works compared to fieldsplit. >> >> Oh, and pressure has to be a vertex variable. >> >>> Thanks in advance >>> >>> Tabrez >>> >>> On 08/27/2013 03:15 PM, Jed Brown wrote: >>>> Tabrez Ali writes: >>>> >>>>> Hello >>>>> >>>>> What is the proper way to use GAMG on a vanilla 3D linear elasticity >>>>> problem. Should I use >>>>> >>>>> -pc_type gamg -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 >>>> Yeah, and only the first of these is needed because the others are >>>> default with -pc_type gamg. >>>> >>>>> -pc_type fieldsplit -pc_fieldsplit_block_size 3 -fieldsplit_pc_type gamg >>>>> -fieldsplit_pc_gamg_type agg -fieldsplit_pc_gamg_agg_nsmooths 1 >>>>> >>>>> Do these options even make sense? With the second set of options the % >>>>> increase in number of iterations with increasing problem size is lower >>>>> than the first but not optimal. >>>> And it's probably more expensive because it has to do inner solves. >>>> Also, if you have less compressible regions, it will get much worse. >>>> >>>>> Also, ksp/ksp/examples/ex56 performs much better in that the number of >>>>> iterations remain more or less constant unlike what I see with my own >>>>> problem. What am I doing wrong? >>>> You probably forgot to set the near null space. You can use >>>> MatSetNearNullSpace (and maybe MatNullSpaceCreateRigidBody) or the more >>>> hacky (IMO) PCSetCoordinates. It's important to have translational >>>> *and* rotational modes in the near null space that GAMG uses to build a >>>> coarse space. >>> >>> -- >>> No one trusts a model except the one who wrote it; Everyone trusts an observation except the one who made it- Harlow Shapley >>> >>> > From mfadams at lbl.gov Sun Sep 1 10:44:10 2013 From: mfadams at lbl.gov (Mark F. Adams) Date: Sun, 1 Sep 2013 11:44:10 -0400 Subject: [petsc-users] GAMG and linear elasticity In-Reply-To: <522334F3.70505@geology.wisc.edu> References: <521D01E6.7070003@geology.wisc.edu> <87bo4i99gx.fsf@mcs.anl.gov> <522218F9.7090805@geology.wisc.edu> <52224CD7.8080406@tudelft.nl> <522334F3.70505@geology.wisc.edu> Message-ID: <90105643-2658-45AD-A6DA-778853B31FC5@lbl.gov> On Sep 1, 2013, at 8:37 AM, Tabrez Ali wrote: > I am using solid elements and here's what I get for the smallest problem with -pc_type svd -pc_svd_monitor > > SVD: condition number 4.184203727999e+13, 0 of 2187 singular values are (nearly) zero > SVD: smallest singular values: 9.975131304660e-01 9.979349868120e-01 9.986761984005e-01 9.993267254639e-01 9.996135846381e-01 > SVD: largest singular values : 3.933977100529e+13 3.933977100529e+13 4.162059658259e+13 4.162059658259e+13 4.173798159224e+13 > This is a very large condition number for a "vanilla 3D linear elasticity problem" [there are no rotations]. The 5 lowest values are very similar ? are you sure you have sufficient BCs? > Tabrez > > On 08/31/2013 03:06 PM, Umut Tabak wrote: >> On 08/31/2013 06:25 PM, Tabrez Ali wrote: >>> Hello >>> >>> So I used PCSetCoordinates and now GAMG seems to work really well in that the number of iterations are relatively constant. Here are the number of iterations on 4 cores >>> >>> DOF ASM GAMG >>> 2187 15 22 >>> 14739 26 22 >>> 107811 51 29 >> >> Hi, >> >> Just curious, what is the approximate condition number of this matrix, is that modelled with domain elements like solids or thin elements like shells? >> >> Best, >> U. > From u.tabak at tudelft.nl Sun Sep 1 10:56:16 2013 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sun, 01 Sep 2013 17:56:16 +0200 Subject: [petsc-users] GAMG and linear elasticity In-Reply-To: <90105643-2658-45AD-A6DA-778853B31FC5@lbl.gov> References: <521D01E6.7070003@geology.wisc.edu> <87bo4i99gx.fsf@mcs.anl.gov> <522218F9.7090805@geology.wisc.edu> <52224CD7.8080406@tudelft.nl> <522334F3.70505@geology.wisc.edu> <90105643-2658-45AD-A6DA-778853B31FC5@lbl.gov> Message-ID: <522363A0.9010507@tudelft.nl> On 09/01/2013 05:44 PM, Mark F. Adams wrote: > > This is a very large condition number for a "vanilla 3D linear elasticity problem" [there are no rotations]. > > The 5 lowest values are very similar ? are you sure you have sufficient BCs? I am also suspecting something fishy here ;-) From stali at geology.wisc.edu Sun Sep 1 11:01:55 2013 From: stali at geology.wisc.edu (Tabrez Ali) Date: Sun, 01 Sep 2013 11:01:55 -0500 Subject: [petsc-users] GAMG and linear elasticity In-Reply-To: <90105643-2658-45AD-A6DA-778853B31FC5@lbl.gov> References: <521D01E6.7070003@geology.wisc.edu> <87bo4i99gx.fsf@mcs.anl.gov> <522218F9.7090805@geology.wisc.edu> <52224CD7.8080406@tudelft.nl> <522334F3.70505@geology.wisc.edu> <90105643-2658-45AD-A6DA-778853B31FC5@lbl.gov> Message-ID: <522364F3.2020600@geology.wisc.edu> Mark/Umut Here's the code http://defmod.googlecode.com/ If you have PETSc installed with METIS then you can build the executable simply by running "make all PETSC_DIR=/home/user/petsc" The three meshes that I am testing GAMG on are available at http://www.geology.wisc.edu/~stali/defmod/gamg_test_input_files.tar.gz Here's how I run it: $ for i in small.inp medium.inp large.inp ; do mpirun -np 4 ./defmod -f $i -pc_type gamg -ksp_monitor; done and I get 16,28 & 52 iterations with ASM and 22,22 & 31 iterations with GAMG. A gif file showing the viz is attached. The problem simply involves application of a point force on a unit cube (nodes on the bottom and four side boundaries are fixed). I have little idea about AMG and am experimenting/investigating if I can easily use it for some of my problems. Tabrez On 09/01/2013 10:44 AM, Mark F. Adams wrote: > On Sep 1, 2013, at 8:37 AM, Tabrez Ali wrote: > >> I am using solid elements and here's what I get for the smallest problem with -pc_type svd -pc_svd_monitor >> >> SVD: condition number 4.184203727999e+13, 0 of 2187 singular values are (nearly) zero >> SVD: smallest singular values: 9.975131304660e-01 9.979349868120e-01 9.986761984005e-01 9.993267254639e-01 9.996135846381e-01 >> SVD: largest singular values : 3.933977100529e+13 3.933977100529e+13 4.162059658259e+13 4.162059658259e+13 4.173798159224e+13 >> > This is a very large condition number for a "vanilla 3D linear elasticity problem" [there are no rotations]. > > The 5 lowest values are very similar ? are you sure you have sufficient BCs? > >> Tabrez >> >> On 08/31/2013 03:06 PM, Umut Tabak wrote: >>> On 08/31/2013 06:25 PM, Tabrez Ali wrote: >>>> Hello >>>> >>>> So I used PCSetCoordinates and now GAMG seems to work really well in that the number of iterations are relatively constant. Here are the number of iterations on 4 cores >>>> >>>> DOF ASM GAMG >>>> 2187 15 22 >>>> 14739 26 22 >>>> 107811 51 29 >>> Hi, >>> >>> Just curious, what is the approximate condition number of this matrix, is that modelled with domain elements like solids or thin elements like shells? >>> >>> Best, >>> U. -------------- next part -------------- A non-text attachment was scrubbed... Name: viz.gif Type: image/gif Size: 146105 bytes Desc: not available URL: From u.tabak at tudelft.nl Sun Sep 1 11:12:57 2013 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sun, 01 Sep 2013 18:12:57 +0200 Subject: [petsc-users] GAMG and linear elasticity In-Reply-To: <522364F3.2020600@geology.wisc.edu> References: <521D01E6.7070003@geology.wisc.edu> <87bo4i99gx.fsf@mcs.anl.gov> <522218F9.7090805@geology.wisc.edu> <52224CD7.8080406@tudelft.nl> <522334F3.70505@geology.wisc.edu> <90105643-2658-45AD-A6DA-778853B31FC5@lbl.gov> <522364F3.2020600@geology.wisc.edu> Message-ID: <52236789.3030805@tudelft.nl> On 09/01/2013 06:01 PM, Tabrez Ali wrote: > Mark/Umut > > Here's the code http://defmod.googlecode.com/ > > If you have PETSc installed with METIS then you can build the > executable simply by running "make all PETSC_DIR=/home/user/petsc" > > The three meshes that I am testing GAMG on are available at > http://www.geology.wisc.edu/~stali/defmod/gamg_test_input_files.tar.gz Thanks for this information, BTW, is that software you are using for visualization is Mayavi2? From stali at geology.wisc.edu Sun Sep 1 11:38:47 2013 From: stali at geology.wisc.edu (Tabrez Ali) Date: Sun, 01 Sep 2013 11:38:47 -0500 Subject: [petsc-users] GAMG and linear elasticity In-Reply-To: <52236789.3030805@tudelft.nl> References: <521D01E6.7070003@geology.wisc.edu> <87bo4i99gx.fsf@mcs.anl.gov> <522218F9.7090805@geology.wisc.edu> <52224CD7.8080406@tudelft.nl> <522334F3.70505@geology.wisc.edu> <90105643-2658-45AD-A6DA-778853B31FC5@lbl.gov> <522364F3.2020600@geology.wisc.edu> <52236789.3030805@tudelft.nl> Message-ID: <52236D97.3060605@geology.wisc.edu> Its Paraview. On 09/01/2013 11:12 AM, Umut Tabak wrote: > On 09/01/2013 06:01 PM, Tabrez Ali wrote: >> Mark/Umut >> >> Here's the code http://defmod.googlecode.com/ >> >> If you have PETSc installed with METIS then you can build the >> executable simply by running "make all PETSC_DIR=/home/user/petsc" >> >> The three meshes that I am testing GAMG on are available at >> http://www.geology.wisc.edu/~stali/defmod/gamg_test_input_files.tar.gz > Thanks for this information, BTW, is that software you are using for > visualization is Mayavi2? From jnl at jennifernicolelee.com Tue Sep 3 10:47:04 2013 From: jnl at jennifernicolelee.com (Jennifer Nicole Lee) Date: Tue, 3 Sep 2013 11:47:04 -0400 (EDT) Subject: [petsc-users] Response Required: Please confirm your request for information. Message-ID: <20130903154704.452B11405B@appmail1.aweber.com> We received your request for information from the Fitness Model Program group. Before we begin sending you the information you requested, we want to be certain we have your permission. Once you do, you will immediately receive another email from me with a special link to download your Free Bikini Boot Camp Report! Smile, Jennifer Nicole Lee ----------------------------------------------------------- CONFIRM BY VISITING THE LINK BELOW: http://www.aweber.com/z/c/?zsbwc3cquzsa0rs6ms4tsbd2nm9m7ctytg4=7201 Click the link above to give us permission to send you information. It's fast and easy! If you cannot click the full URL above, please copy and paste it into your web browser. ----------------------------------------------------------- If you do not want to confirm, simply ignore this message. Thank You, Fitness Model Program 6619 S. Dixie Hwy., #178, Miami, FL 33143, USA Request generated by: IP: 75.147.208.109 Date: September 3, 2013 11:47 EDT From jnl at jennifernicolelee.com Tue Sep 3 10:47:17 2013 From: jnl at jennifernicolelee.com (Jennifer Nicole Lee) Date: Tue, 3 Sep 2013 11:47:17 -0400 (EDT) Subject: [petsc-users] Thank You For Confirming Your Subscription! Message-ID: <20130903154717.3B9EBDA05A@web-addlead4.aweber.com> Thank you for confirming your subscription to the Fitness Model Program Newsletter. Now to download your special Free Bikini Boot Camp Report simply click on the link below: http://www.fitnessmodelprogram.com/bikini_boot_camp.shtml Smile, Jennifer Nicole Lee 6619 S. Dixie Hwy., #178, Miami, FL 33143, USA To unsubscribe or change subscriber options visit: http://www.aweber.com/z/r/?TJzsbGwctKxsLGyMDGyMtGa0jByMrOxsDA== From suraj.prabhakaran at gmail.com Tue Sep 3 13:58:18 2013 From: suraj.prabhakaran at gmail.com (Suraj Prabhakaran) Date: Tue, 3 Sep 2013 20:58:18 +0200 Subject: [petsc-users] Re-initializing petsc again Message-ID: <315F25F7-7274-40E9-A244-5CD26F975CE5@gmail.com> Hello, Again, this relates to finalizing and initializing petsc in my application. I basically want to finalize petsc and initialize it again with a different MPI communicator. A rough sketch of my quite huge application is as follows. main() { PetscInitialize() ... ... AOCreate().. AO operations.. AODestroy().. ... ... PetscFinalize() PETSC_COMM_WORLD = A new communicator (which includes more processes) PetscInitialize() ... ... AOCreate().. AO operations.. AODestroy() ... ... PetscFinalize() } So after finalizing and initializing it back, I always get an error with one of the AO calls. Given below is the complete error I get when running with 8 procs. Can anyone give me a hint what the problem could be? I was told earlier, that finalizing and initializing petsc should not be a problem! Best, Suraj -------------------------ERROR starts here------------------------------- [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] PetscStrcmp line 414 /root/petsc/petsc-3.3-p7/src/sys/utils/str.c [0]PETSC ERROR: [0] PetscFListAdd line 202 /root/petsc/petsc-3.3-p7/src/sys/dll/reg.c [0]PETSC ERROR: [0] AORegister line 99 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/aoreg.c [0]PETSC ERROR: [0] AORegisterAll line 28 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/aoregall.c [0]PETSC ERROR: [0] AOInitializePackage line 49 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/dlregisdm.c [0]PETSC ERROR: [0] AOCreate line 530 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/ao.c [0]PETSC ERROR: [0] AOCreateBasicIS line 381 /root/petsc/petsc-3.3-p7/src/dm/ao/impls/basic/aobasic.c [0]PETSC ERROR: [0] AOCreateBasic line 336 /root/petsc/petsc-3.3-p7/src/dm/ao/impls/basic/aobasic.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 [0]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib [0]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 [0]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 59. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [1]PETSC ERROR: likely location of problem given in stack below [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [1]PETSC ERROR: INSTEAD the line number of the start of the function [1]PETSC ERROR: is given. [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Signal received! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 [1]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib [1]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 [1]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [2]PETSC ERROR: likely location of problem given in stack below [2]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [2]PETSC ERROR: INSTEAD the line number of the start of the function [2]PETSC ERROR: is given. [2]PETSC ERROR: --------------------- Error Message ------------------------------------ [2]PETSC ERROR: Signal received! [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 [2]PETSC ERROR: See docs/changes/index.html for recent updates. [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [2]PETSC ERROR: See docs/index.html for manual pages. [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 [2]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib [2]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 [2]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: User provided function() line 0 in unknown directory unknown file [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [3]PETSC ERROR: likely location of problem given in stack below [3]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [3]PETSC ERROR: INSTEAD the line number of the start of the function [3]PETSC ERROR: is given. [3]PETSC ERROR: --------------------- Error Message ------------------------------------ [3]PETSC ERROR: Signal received! [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 [3]PETSC ERROR: See docs/changes/index.html for recent updates. [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [3]PETSC ERROR: See docs/index.html for manual pages. [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 [3]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib [3]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 [3]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: User provided function() line 0 in unknown directory unknown file [4]PETSC ERROR: ------------------------------------------------------------------------ [4]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [4]PETSC ERROR: likely location of problem given in stack below [4]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [4]PETSC ERROR: INSTEAD the line number of the start of the function [4]PETSC ERROR: is given. [4]PETSC ERROR: --------------------- Error Message ------------------------------------ [4]PETSC ERROR: Signal received! [4]PETSC ERROR: ------------------------------------------------------------------------ [4]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 [4]PETSC ERROR: See docs/changes/index.html for recent updates. [4]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [4]PETSC ERROR: See docs/index.html for manual pages. [4]PETSC ERROR: ------------------------------------------------------------------------ [4]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 [4]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib [4]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 [4]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging [4]PETSC ERROR: ------------------------------------------------------------------------ [4]PETSC ERROR: User provided function() line 0 in unknown directory unknown file [5]PETSC ERROR: ------------------------------------------------------------------------ [5]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end [5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[5]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [5]PETSC ERROR: likely location of problem given in stack below [5]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [5]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [5]PETSC ERROR: INSTEAD the line number of the start of the function [5]PETSC ERROR: is given. [5]PETSC ERROR: --------------------- Error Message ------------------------------------ [5]PETSC ERROR: Signal received! [5]PETSC ERROR: ------------------------------------------------------------------------ [5]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 [5]PETSC ERROR: See docs/changes/index.html for recent updates. [5]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [5]PETSC ERROR: See docs/index.html for manual pages. [5]PETSC ERROR: ------------------------------------------------------------------------ [5]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 [5]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib [5]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 [5]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging [5]PETSC ERROR: ------------------------------------------------------------------------ [5]PETSC ERROR: User provided function() line 0 in unknown directory unknown file [6]PETSC ERROR: ------------------------------------------------------------------------ [6]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [6]PETSC ERROR: likely location of problem given in stack below [6]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [6]PETSC ERROR: INSTEAD the line number of the start of the function [6]PETSC ERROR: is given. [6]PETSC ERROR: --------------------- Error Message ------------------------------------ [6]PETSC ERROR: Signal received! [6]PETSC ERROR: ------------------------------------------------------------------------ [6]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 [6]PETSC ERROR: See docs/changes/index.html for recent updates. [6]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [6]PETSC ERROR: See docs/index.html for manual pages. [6]PETSC ERROR: ------------------------------------------------------------------------ [6]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 [6]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib [6]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 [6]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging [6]PETSC ERROR: ------------------------------------------------------------------------ [6]PETSC ERROR: User provided function() line 0 in unknown directory unknown file [7]PETSC ERROR: ------------------------------------------------------------------------ [7]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[7]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [7]PETSC ERROR: likely location of problem given in stack below [7]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [7]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [7]PETSC ERROR: INSTEAD the line number of the start of the function [7]PETSC ERROR: is given. [7]PETSC ERROR: --------------------- Error Message ------------------------------------ [7]PETSC ERROR: Signal received! [7]PETSC ERROR: ------------------------------------------------------------------------ [7]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 [7]PETSC ERROR: See docs/changes/index.html for recent updates. [7]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [7]PETSC ERROR: See docs/index.html for manual pages. [7]PETSC ERROR: ------------------------------------------------------------------------ [7]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 [7]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib [7]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 [7]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging [7]PETSC ERROR: ------------------------------------------------------------------------ [7]PETSC ERROR: User provided function() line 0 in unknown directory unknown file -------------------------------------------------------------------------- mpiexec has exited due to process rank 0 with PID 2670 on node grsacc17 exiting improperly. There are two reasons this could occur: 1. this process did not call "init" before exiting, but others in the job did. This can cause a job to hang indefinitely while it waits for all processes to call "init". By rule, if one process calls "init", then ALL processes must call "init" prior to termination. 2. this process called "init", but exited without calling "finalize". By rule, all processes that call "init" MUST call "finalize" prior to exiting or it will be considered an "abnormal termination" This may have caused other processes in the application to be terminated by signals sent by mpiexec (as reported here). From bsmith at mcs.anl.gov Tue Sep 3 14:01:41 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 3 Sep 2013 14:01:41 -0500 Subject: [petsc-users] Re-initializing petsc again In-Reply-To: <315F25F7-7274-40E9-A244-5CD26F975CE5@gmail.com> References: <315F25F7-7274-40E9-A244-5CD26F975CE5@gmail.com> Message-ID: <614D53AA-89C2-4C34-A192-5FA4BE0D1F27@mcs.anl.gov> There is a problem with the 3.3 series when reinitializing PETSc. Can you switch to PETSc 3.4 which properly supports calling PetscInitialize() several times. Barry On Sep 3, 2013, at 1:58 PM, Suraj Prabhakaran wrote: > Hello, > > Again, this relates to finalizing and initializing petsc in my application. I basically want to finalize petsc and initialize it again with a different MPI communicator. A rough sketch of my quite huge application is as follows. > > > main() > { > PetscInitialize() > ... > ... > AOCreate().. > AO operations.. > AODestroy().. > ... > ... > PetscFinalize() > PETSC_COMM_WORLD = A new communicator (which includes more processes) > PetscInitialize() > ... > ... > AOCreate().. > AO operations.. > AODestroy() > ... > ... > PetscFinalize() > } > > So after finalizing and initializing it back, I always get an error with one of the AO calls. Given below is the complete error I get when running with 8 procs. Can anyone give me a hint what the problem could be? I was told earlier, that finalizing and initializing petsc should not be a problem! > > Best, > Suraj > > -------------------------ERROR starts here------------------------------- > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] PetscStrcmp line 414 /root/petsc/petsc-3.3-p7/src/sys/utils/str.c > [0]PETSC ERROR: [0] PetscFListAdd line 202 /root/petsc/petsc-3.3-p7/src/sys/dll/reg.c > [0]PETSC ERROR: [0] AORegister line 99 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/aoreg.c > [0]PETSC ERROR: [0] AORegisterAll line 28 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/aoregall.c > [0]PETSC ERROR: [0] AOInitializePackage line 49 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/dlregisdm.c > [0]PETSC ERROR: [0] AOCreate line 530 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/ao.c > [0]PETSC ERROR: [0] AOCreateBasicIS line 381 /root/petsc/petsc-3.3-p7/src/dm/ao/impls/basic/aobasic.c > [0]PETSC ERROR: [0] AOCreateBasic line 336 /root/petsc/petsc-3.3-p7/src/dm/ao/impls/basic/aobasic.c > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > [0]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > [0]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > [0]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > with errorcode 59. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > [1]PETSC ERROR: ------------------------------------------------------------------------ > [1]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > [1]PETSC ERROR: likely location of problem given in stack below > [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [1]PETSC ERROR: INSTEAD the line number of the start of the function > [1]PETSC ERROR: is given. > [1]PETSC ERROR: --------------------- Error Message ------------------------------------ > [1]PETSC ERROR: Signal received! > [1]PETSC ERROR: ------------------------------------------------------------------------ > [1]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > [1]PETSC ERROR: See docs/changes/index.html for recent updates. > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [1]PETSC ERROR: See docs/index.html for manual pages. > [1]PETSC ERROR: ------------------------------------------------------------------------ > [1]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > [1]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > [1]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > [1]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > [1]PETSC ERROR: ------------------------------------------------------------------------ > [1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > [2]PETSC ERROR: ------------------------------------------------------------------------ > [2]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > [2]PETSC ERROR: likely location of problem given in stack below > [2]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [2]PETSC ERROR: INSTEAD the line number of the start of the function > [2]PETSC ERROR: is given. > [2]PETSC ERROR: --------------------- Error Message ------------------------------------ > [2]PETSC ERROR: Signal received! > [2]PETSC ERROR: ------------------------------------------------------------------------ > [2]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > [2]PETSC ERROR: See docs/changes/index.html for recent updates. > [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [2]PETSC ERROR: See docs/index.html for manual pages. > [2]PETSC ERROR: ------------------------------------------------------------------------ > [2]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > [2]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > [2]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > [2]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > [2]PETSC ERROR: ------------------------------------------------------------------------ > [2]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > [3]PETSC ERROR: ------------------------------------------------------------------------ > [3]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > [3]PETSC ERROR: likely location of problem given in stack below > [3]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [3]PETSC ERROR: INSTEAD the line number of the start of the function > [3]PETSC ERROR: is given. > [3]PETSC ERROR: --------------------- Error Message ------------------------------------ > [3]PETSC ERROR: Signal received! > [3]PETSC ERROR: ------------------------------------------------------------------------ > [3]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > [3]PETSC ERROR: See docs/changes/index.html for recent updates. > [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [3]PETSC ERROR: See docs/index.html for manual pages. > [3]PETSC ERROR: ------------------------------------------------------------------------ > [3]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > [3]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > [3]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > [3]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > [3]PETSC ERROR: ------------------------------------------------------------------------ > [3]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > [4]PETSC ERROR: ------------------------------------------------------------------------ > [4]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > [4]PETSC ERROR: likely location of problem given in stack below > [4]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [4]PETSC ERROR: INSTEAD the line number of the start of the function > [4]PETSC ERROR: is given. > [4]PETSC ERROR: --------------------- Error Message ------------------------------------ > [4]PETSC ERROR: Signal received! > [4]PETSC ERROR: ------------------------------------------------------------------------ > [4]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > [4]PETSC ERROR: See docs/changes/index.html for recent updates. > [4]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [4]PETSC ERROR: See docs/index.html for manual pages. > [4]PETSC ERROR: ------------------------------------------------------------------------ > [4]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > [4]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > [4]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > [4]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > [4]PETSC ERROR: ------------------------------------------------------------------------ > [4]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > [5]PETSC ERROR: ------------------------------------------------------------------------ > [5]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > [5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[5]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > [5]PETSC ERROR: likely location of problem given in stack below > [5]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [5]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [5]PETSC ERROR: INSTEAD the line number of the start of the function > [5]PETSC ERROR: is given. > [5]PETSC ERROR: --------------------- Error Message ------------------------------------ > [5]PETSC ERROR: Signal received! > [5]PETSC ERROR: ------------------------------------------------------------------------ > [5]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > [5]PETSC ERROR: See docs/changes/index.html for recent updates. > [5]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [5]PETSC ERROR: See docs/index.html for manual pages. > [5]PETSC ERROR: ------------------------------------------------------------------------ > [5]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > [5]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > [5]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > [5]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > [5]PETSC ERROR: ------------------------------------------------------------------------ > [5]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > [6]PETSC ERROR: ------------------------------------------------------------------------ > [6]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > [6]PETSC ERROR: likely location of problem given in stack below > [6]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [6]PETSC ERROR: INSTEAD the line number of the start of the function > [6]PETSC ERROR: is given. > [6]PETSC ERROR: --------------------- Error Message ------------------------------------ > [6]PETSC ERROR: Signal received! > [6]PETSC ERROR: ------------------------------------------------------------------------ > [6]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > [6]PETSC ERROR: See docs/changes/index.html for recent updates. > [6]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [6]PETSC ERROR: See docs/index.html for manual pages. > [6]PETSC ERROR: ------------------------------------------------------------------------ > [6]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > [6]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > [6]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > [6]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > [6]PETSC ERROR: ------------------------------------------------------------------------ > [6]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > [7]PETSC ERROR: ------------------------------------------------------------------------ > [7]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[7]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > [7]PETSC ERROR: likely location of problem given in stack below > [7]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [7]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [7]PETSC ERROR: INSTEAD the line number of the start of the function > [7]PETSC ERROR: is given. > [7]PETSC ERROR: --------------------- Error Message ------------------------------------ > [7]PETSC ERROR: Signal received! > [7]PETSC ERROR: ------------------------------------------------------------------------ > [7]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > [7]PETSC ERROR: See docs/changes/index.html for recent updates. > [7]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [7]PETSC ERROR: See docs/index.html for manual pages. > [7]PETSC ERROR: ------------------------------------------------------------------------ > [7]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > [7]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > [7]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > [7]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > [7]PETSC ERROR: ------------------------------------------------------------------------ > [7]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > -------------------------------------------------------------------------- > mpiexec has exited due to process rank 0 with PID 2670 on > node grsacc17 exiting improperly. There are two reasons this could occur: > > 1. this process did not call "init" before exiting, but others in > the job did. This can cause a job to hang indefinitely while it waits > for all processes to call "init". By rule, if one process calls "init", > then ALL processes must call "init" prior to termination. > > 2. this process called "init", but exited without calling "finalize". > By rule, all processes that call "init" MUST call "finalize" prior to > exiting or it will be considered an "abnormal termination" > > This may have caused other processes in the application to be > terminated by signals sent by mpiexec (as reported here). From balay at mcs.anl.gov Tue Sep 3 14:03:39 2013 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 3 Sep 2013 14:03:39 -0500 (CDT) Subject: [petsc-users] Re-initializing petsc again In-Reply-To: <614D53AA-89C2-4C34-A192-5FA4BE0D1F27@mcs.anl.gov> References: <315F25F7-7274-40E9-A244-5CD26F975CE5@gmail.com> <614D53AA-89C2-4C34-A192-5FA4BE0D1F27@mcs.anl.gov> Message-ID: Also you would have to call MPI_Init() before the first PetscInitialize() [and similarly MPI_Finalize() after the last call to PetscFinalize()] Satish On Tue, 3 Sep 2013, Barry Smith wrote: > > There is a problem with the 3.3 series when reinitializing PETSc. Can you switch to PETSc 3.4 which properly supports calling PetscInitialize() several times. > > Barry > > On Sep 3, 2013, at 1:58 PM, Suraj Prabhakaran wrote: > > > Hello, > > > > Again, this relates to finalizing and initializing petsc in my application. I basically want to finalize petsc and initialize it again with a different MPI communicator. A rough sketch of my quite huge application is as follows. > > > > > > main() > > { > > PetscInitialize() > > ... > > ... > > AOCreate().. > > AO operations.. > > AODestroy().. > > ... > > ... > > PetscFinalize() > > PETSC_COMM_WORLD = A new communicator (which includes more processes) > > PetscInitialize() > > ... > > ... > > AOCreate().. > > AO operations.. > > AODestroy() > > ... > > ... > > PetscFinalize() > > } > > > > So after finalizing and initializing it back, I always get an error with one of the AO calls. Given below is the complete error I get when running with 8 procs. Can anyone give me a hint what the problem could be? I was told earlier, that finalizing and initializing petsc should not be a problem! > > > > Best, > > Suraj > > > > -------------------------ERROR starts here------------------------------- > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > > [0]PETSC ERROR: likely location of problem given in stack below > > [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > > [0]PETSC ERROR: INSTEAD the line number of the start of the function > > [0]PETSC ERROR: is given. > > [0]PETSC ERROR: [0] PetscStrcmp line 414 /root/petsc/petsc-3.3-p7/src/sys/utils/str.c > > [0]PETSC ERROR: [0] PetscFListAdd line 202 /root/petsc/petsc-3.3-p7/src/sys/dll/reg.c > > [0]PETSC ERROR: [0] AORegister line 99 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/aoreg.c > > [0]PETSC ERROR: [0] AORegisterAll line 28 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/aoregall.c > > [0]PETSC ERROR: [0] AOInitializePackage line 49 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/dlregisdm.c > > [0]PETSC ERROR: [0] AOCreate line 530 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/ao.c > > [0]PETSC ERROR: [0] AOCreateBasicIS line 381 /root/petsc/petsc-3.3-p7/src/dm/ao/impls/basic/aobasic.c > > [0]PETSC ERROR: [0] AOCreateBasic line 336 /root/petsc/petsc-3.3-p7/src/dm/ao/impls/basic/aobasic.c > > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > > [0]PETSC ERROR: Signal received! > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > [0]PETSC ERROR: See docs/index.html for manual pages. > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > [0]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > > [0]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > > [0]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > > [0]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > > -------------------------------------------------------------------------- > > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > > with errorcode 59. > > > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > > You may or may not see output from other processes, depending on > > exactly when Open MPI kills them. > > -------------------------------------------------------------------------- > > [1]PETSC ERROR: ------------------------------------------------------------------------ > > [1]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > > [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > > [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > > [1]PETSC ERROR: likely location of problem given in stack below > > [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > > [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > > [1]PETSC ERROR: INSTEAD the line number of the start of the function > > [1]PETSC ERROR: is given. > > [1]PETSC ERROR: --------------------- Error Message ------------------------------------ > > [1]PETSC ERROR: Signal received! > > [1]PETSC ERROR: ------------------------------------------------------------------------ > > [1]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > > [1]PETSC ERROR: See docs/changes/index.html for recent updates. > > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > [1]PETSC ERROR: See docs/index.html for manual pages. > > [1]PETSC ERROR: ------------------------------------------------------------------------ > > [1]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > > [1]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > > [1]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > > [1]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > > [1]PETSC ERROR: ------------------------------------------------------------------------ > > [1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > > [2]PETSC ERROR: ------------------------------------------------------------------------ > > [2]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > > [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > > [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > > [2]PETSC ERROR: likely location of problem given in stack below > > [2]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > > [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > > [2]PETSC ERROR: INSTEAD the line number of the start of the function > > [2]PETSC ERROR: is given. > > [2]PETSC ERROR: --------------------- Error Message ------------------------------------ > > [2]PETSC ERROR: Signal received! > > [2]PETSC ERROR: ------------------------------------------------------------------------ > > [2]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > > [2]PETSC ERROR: See docs/changes/index.html for recent updates. > > [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > [2]PETSC ERROR: See docs/index.html for manual pages. > > [2]PETSC ERROR: ------------------------------------------------------------------------ > > [2]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > > [2]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > > [2]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > > [2]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > > [2]PETSC ERROR: ------------------------------------------------------------------------ > > [2]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > > [3]PETSC ERROR: ------------------------------------------------------------------------ > > [3]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > > [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > > [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > > [3]PETSC ERROR: likely location of problem given in stack below > > [3]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > > [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > > [3]PETSC ERROR: INSTEAD the line number of the start of the function > > [3]PETSC ERROR: is given. > > [3]PETSC ERROR: --------------------- Error Message ------------------------------------ > > [3]PETSC ERROR: Signal received! > > [3]PETSC ERROR: ------------------------------------------------------------------------ > > [3]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > > [3]PETSC ERROR: See docs/changes/index.html for recent updates. > > [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > [3]PETSC ERROR: See docs/index.html for manual pages. > > [3]PETSC ERROR: ------------------------------------------------------------------------ > > [3]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > > [3]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > > [3]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > > [3]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > > [3]PETSC ERROR: ------------------------------------------------------------------------ > > [3]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > > [4]PETSC ERROR: ------------------------------------------------------------------------ > > [4]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > > [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > > [4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > > [4]PETSC ERROR: likely location of problem given in stack below > > [4]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > > [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > > [4]PETSC ERROR: INSTEAD the line number of the start of the function > > [4]PETSC ERROR: is given. > > [4]PETSC ERROR: --------------------- Error Message ------------------------------------ > > [4]PETSC ERROR: Signal received! > > [4]PETSC ERROR: ------------------------------------------------------------------------ > > [4]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > > [4]PETSC ERROR: See docs/changes/index.html for recent updates. > > [4]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > [4]PETSC ERROR: See docs/index.html for manual pages. > > [4]PETSC ERROR: ------------------------------------------------------------------------ > > [4]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > > [4]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > > [4]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > > [4]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > > [4]PETSC ERROR: ------------------------------------------------------------------------ > > [4]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > > [5]PETSC ERROR: ------------------------------------------------------------------------ > > [5]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > > [5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > > [5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[5]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > > [5]PETSC ERROR: likely location of problem given in stack below > > [5]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > > [5]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > > [5]PETSC ERROR: INSTEAD the line number of the start of the function > > [5]PETSC ERROR: is given. > > [5]PETSC ERROR: --------------------- Error Message ------------------------------------ > > [5]PETSC ERROR: Signal received! > > [5]PETSC ERROR: ------------------------------------------------------------------------ > > [5]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > > [5]PETSC ERROR: See docs/changes/index.html for recent updates. > > [5]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > [5]PETSC ERROR: See docs/index.html for manual pages. > > [5]PETSC ERROR: ------------------------------------------------------------------------ > > [5]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > > [5]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > > [5]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > > [5]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > > [5]PETSC ERROR: ------------------------------------------------------------------------ > > [5]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > > [6]PETSC ERROR: ------------------------------------------------------------------------ > > [6]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > > [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > > [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > > [6]PETSC ERROR: likely location of problem given in stack below > > [6]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > > [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > > [6]PETSC ERROR: INSTEAD the line number of the start of the function > > [6]PETSC ERROR: is given. > > [6]PETSC ERROR: --------------------- Error Message ------------------------------------ > > [6]PETSC ERROR: Signal received! > > [6]PETSC ERROR: ------------------------------------------------------------------------ > > [6]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > > [6]PETSC ERROR: See docs/changes/index.html for recent updates. > > [6]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > [6]PETSC ERROR: See docs/index.html for manual pages. > > [6]PETSC ERROR: ------------------------------------------------------------------------ > > [6]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > > [6]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > > [6]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > > [6]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > > [6]PETSC ERROR: ------------------------------------------------------------------------ > > [6]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > > [7]PETSC ERROR: ------------------------------------------------------------------------ > > [7]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > > [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > > [7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[7]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > > [7]PETSC ERROR: likely location of problem given in stack below > > [7]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > > [7]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > > [7]PETSC ERROR: INSTEAD the line number of the start of the function > > [7]PETSC ERROR: is given. > > [7]PETSC ERROR: --------------------- Error Message ------------------------------------ > > [7]PETSC ERROR: Signal received! > > [7]PETSC ERROR: ------------------------------------------------------------------------ > > [7]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > > [7]PETSC ERROR: See docs/changes/index.html for recent updates. > > [7]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > [7]PETSC ERROR: See docs/index.html for manual pages. > > [7]PETSC ERROR: ------------------------------------------------------------------------ > > [7]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > > [7]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > > [7]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > > [7]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > > [7]PETSC ERROR: ------------------------------------------------------------------------ > > [7]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > > -------------------------------------------------------------------------- > > mpiexec has exited due to process rank 0 with PID 2670 on > > node grsacc17 exiting improperly. There are two reasons this could occur: > > > > 1. this process did not call "init" before exiting, but others in > > the job did. This can cause a job to hang indefinitely while it waits > > for all processes to call "init". By rule, if one process calls "init", > > then ALL processes must call "init" prior to termination. > > > > 2. this process called "init", but exited without calling "finalize". > > By rule, all processes that call "init" MUST call "finalize" prior to > > exiting or it will be considered an "abnormal termination" > > > > This may have caused other processes in the application to be > > terminated by signals sent by mpiexec (as reported here). > > From suraj.prabhakaran at gmail.com Tue Sep 3 22:01:07 2013 From: suraj.prabhakaran at gmail.com (Suraj Prabhakaran) Date: Wed, 4 Sep 2013 05:01:07 +0200 Subject: [petsc-users] Re-initializing petsc again In-Reply-To: References: <315F25F7-7274-40E9-A244-5CD26F975CE5@gmail.com> <614D53AA-89C2-4C34-A192-5FA4BE0D1F27@mcs.anl.gov> Message-ID: Thanks a lot for the confirmation!! I am now compiling it with 3.4.2, but I get a strange error and could be great if someone can just point out whats wrong. (Might be something stupid, but just in case...) Thanks in advance! Suraj -------------------------------ERROR----------------------------- In file included from /opt/petsc-3.4.2/include/petsc.h:8, from Main/time_integration_dir/compute_tau_w_mean.c:91: /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: /opt/petsc-3.4.2/include/petscbt.h:33: error: invalid storage class for function ?PetscBTLength? /opt/petsc-3.4.2/include/petscbt.h:38: error: invalid storage class for function ?PetscBTMemzero? /opt/petsc-3.4.2/include/petscbt.h:43: error: invalid storage class for function ?PetscBTDestroy? /opt/petsc-3.4.2/include/petscbt.h:48: error: invalid storage class for function ?PetscBTLookup? /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTLookup?: /opt/petsc-3.4.2/include/petscbt.h:47: warning: declaration of ?index? shadows a global declaration /usr/include/string.h:489: warning: shadowed declaration is here /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: /opt/petsc-3.4.2/include/petscbt.h:59: error: invalid storage class for function ?PetscBTView? /opt/petsc-3.4.2/include/petscbt.h:74: error: invalid storage class for function ?PetscBTCreate? /opt/petsc-3.4.2/include/petscbt.h:79: error: invalid storage class for function ?PetscBTLookupSet? /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTLookupSet?: /opt/petsc-3.4.2/include/petscbt.h:78: warning: declaration of ?index? shadows a global declaration /usr/include/string.h:489: warning: shadowed declaration is here /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: /opt/petsc-3.4.2/include/petscbt.h:91: error: invalid storage class for function ?PetscBTSet? /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTSet?: /opt/petsc-3.4.2/include/petscbt.h:90: warning: declaration of ?index? shadows a global declaration /usr/include/string.h:489: warning: shadowed declaration is here /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: /opt/petsc-3.4.2/include/petscbt.h:103: error: invalid storage class for function ?PetscBTLookupClear? /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTLookupClear?: /opt/petsc-3.4.2/include/petscbt.h:102: warning: declaration of ?index? shadows a global declaration /usr/include/string.h:489: warning: shadowed declaration is here /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: /opt/petsc-3.4.2/include/petscbt.h:115: error: invalid storage class for function ?PetscBTClear? /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTClear?: /opt/petsc-3.4.2/include/petscbt.h:114: warning: declaration of ?index? shadows a global declaration /usr/include/string.h:489: warning: shadowed declaration is here In file included from /opt/petsc-3.4.2/include/petsc.h:9, from Main/time_integration_dir/compute_tau_w_mean.c:91: /opt/petsc-3.4.2/include/petscctable.h: In function ?compute_tau_w_mean?: /opt/petsc-3.4.2/include/petscctable.h:20: error: invalid storage class for function ?PetscHash? /opt/petsc-3.4.2/include/petscctable.h:40: error: invalid storage class for function ?PetscTableAdd? In file included from /opt/petsc-3.4.2/include/petsc.h:9, from Main/time_integration_dir/compute_tau_w_mean.c:91: /opt/petsc-3.4.2/include/petscctable.h:83: error: invalid storage class for function ?PetscTableAddCount? /opt/petsc-3.4.2/include/petscctable.h:120: error: invalid storage class for function ?PetscTableFind? In file included from /opt/petsc-3.4.2/include/petscsnesfas.h:3, from /opt/petsc-3.4.2/include/petsc.h:27, from Main/time_integration_dir/compute_tau_w_mean.c:91: /opt/petsc-3.4.2/include/petscsnes.h:585: error: invalid storage class for function ?SNESGetSNESLineSearch? /opt/petsc-3.4.2/include/petscsnes.h:586: error: invalid storage class for function ?SNESSetSNESLineSearch? Main/time_integration_dir/compute_tau_w_mean.c:223: warning: implicit declaration of function ?wall_thermoquants? Main/time_integration_dir/compute_tau_w_mean.c:155: warning: unused variable ?iface_plus_one? Main/time_integration_dir/compute_tau_w_mean.c:330: warning: value computed is not used Main/time_integration_dir/compute_tau_w_mean.c:116: warning: unused variable ?ierr? Main/time_integration_dir/compute_tau_w_mean.c:111: warning: unused variable ?icell? Main/time_integration_dir/compute_tau_w_mean.c:105: warning: unused variable ?mach? /opt/petsc-3.4.2/include/petscts.h:328: warning: unused variable ?TSRegisterAllCalled? /opt/petsc-3.4.2/include/petscts.h:327: warning: unused variable ?TSList? /opt/petsc-3.4.2/include/petscts.h:168: warning: unused variable ?DMTS_CLASSID? /opt/petsc-3.4.2/include/petscts.h:167: warning: unused variable ?TS_CLASSID? /opt/petsc-3.4.2/include/petscts.h:163: warning: unused variable ?TSExactFinalTimeOptions? /opt/petsc-3.4.2/include/petscts.h:101: warning: unused variable ?TSConvergedReasons? /opt/petsc-3.4.2/include/petscts.h:80: warning: unused variable ?TSEquationTypes? /opt/petsc-3.4.2/include/petscsnesfas.h:19: warning: unused variable ?SNESFASTypes? /opt/petsc-3.4.2/include/petscsnes.h:679: warning: unused variable ?SNESQNRestartTypes? /opt/petsc-3.4.2/include/petscsnes.h:675: warning: unused variable ?SNESQNScaleTypes? /opt/petsc-3.4.2/include/petscsnes.h:667: warning: unused variable ?SNESNCGTypes? /opt/petsc-3.4.2/include/petscsnes.h:654: warning: unused variable ?SNESNGMRESSelectTypes? /opt/petsc-3.4.2/include/petscsnes.h:648: warning: unused variable ?SNESNGMRESRestartTypes? /opt/petsc-3.4.2/include/petscsnes.h:474: warning: unused variable ?SNESLineSearch_Apply? /opt/petsc-3.4.2/include/petscsnes.h:473: warning: unused variable ?SNESLineSearchList? /opt/petsc-3.4.2/include/petscsnes.h:472: warning: unused variable ?SNESLineSearchRegisterAllCalled? /opt/petsc-3.4.2/include/petscsnes.h:471: warning: unused variable ?SNESLINESEARCH_CLASSID? /opt/petsc-3.4.2/include/petscsnes.h:470: warning: unused variable ?SNESList? /opt/petsc-3.4.2/include/petscsnes.h:469: warning: unused variable ?SNESRegisterAllCalled? /opt/petsc-3.4.2/include/petscsnes.h:356: warning: unused variable ?SNESNormTypes? /opt/petsc-3.4.2/include/petscsnes.h:217: warning: unused variable ?SNESConvergedReasons? /opt/petsc-3.4.2/include/petscsnes.h:50: warning: unused variable ?DMSNES_CLASSID? /opt/petsc-3.4.2/include/petscsnes.h:49: warning: unused variable ?SNES_CLASSID? /opt/petsc-3.4.2/include/petsccharacteristic.h:44: warning: unused variable ?CharacteristicList? /opt/petsc-3.4.2/include/petsccharacteristic.h:43: warning: unused variable ?CharacteristicRegisterAllCalled? /opt/petsc-3.4.2/include/petscdmda.h:27: warning: unused variable ?DMDABoundaryTypes? /opt/petsc-3.4.2/include/petscpf.h:50: warning: unused variable ?PFRegisterAllCalled? /opt/petsc-3.4.2/include/petscpf.h:40: warning: unused variable ?PF_CLASSID? /opt/petsc-3.4.2/include/petscpf.h:12: warning: unused variable ?PFList? /opt/petsc-3.4.2/include/petscdm.h:34: warning: unused variable ?DMRegisterAllCalled? /opt/petsc-3.4.2/include/petscdm.h:33: warning: unused variable ?DMList? /opt/petsc-3.4.2/include/petscdm.h:11: warning: unused variable ?DM_CLASSID? /opt/petsc-3.4.2/include/petscdraw.h:239: warning: unused variable ?PETSC_DRAWHG_CLASSID? /opt/petsc-3.4.2/include/petscdraw.h:225: warning: unused variable ?PETSC_DRAWSP_CLASSID? /opt/petsc-3.4.2/include/petscdraw.h:206: warning: unused variable ?PETSC_DRAWLG_CLASSID? /opt/petsc-3.4.2/include/petscdraw.h:195: warning: unused variable ?PETSC_DRAWAXIS_CLASSID? /opt/petsc-3.4.2/include/petscdraw.h:26: warning: unused variable ?PetscDrawList? /opt/petsc-3.4.2/include/petscdraw.h:9: warning: unused variable ?PETSC_DRAW_CLASSID? /opt/petsc-3.4.2/include/petscmatlab.h:8: warning: unused variable ?MATLABENGINE_CLASSID? /opt/petsc-3.4.2/include/petscthreadcomm.h:35: warning: unused variable ?PetscThreadCommReductionOps? /opt/petsc-3.4.2/include/petscthreadcomm.h:32: warning: unused variable ?PetscThreadCommList? On Sep 3, 2013, at 9:03 PM, Satish Balay wrote: > Also you would have to call MPI_Init() before the first PetscInitialize() [and > similarly MPI_Finalize() after the last call to PetscFinalize()] > > Satish > > On Tue, 3 Sep 2013, Barry Smith wrote: > >> >> There is a problem with the 3.3 series when reinitializing PETSc. Can you switch to PETSc 3.4 which properly supports calling PetscInitialize() several times. >> >> Barry >> >> On Sep 3, 2013, at 1:58 PM, Suraj Prabhakaran wrote: >> >>> Hello, >>> >>> Again, this relates to finalizing and initializing petsc in my application. I basically want to finalize petsc and initialize it again with a different MPI communicator. A rough sketch of my quite huge application is as follows. >>> >>> >>> main() >>> { >>> PetscInitialize() >>> ... >>> ... >>> AOCreate().. >>> AO operations.. >>> AODestroy().. >>> ... >>> ... >>> PetscFinalize() >>> PETSC_COMM_WORLD = A new communicator (which includes more processes) >>> PetscInitialize() >>> ... >>> ... >>> AOCreate().. >>> AO operations.. >>> AODestroy() >>> ... >>> ... >>> PetscFinalize() >>> } >>> >>> So after finalizing and initializing it back, I always get an error with one of the AO calls. Given below is the complete error I get when running with 8 procs. Can anyone give me a hint what the problem could be? I was told earlier, that finalizing and initializing petsc should not be a problem! >>> >>> Best, >>> Suraj >>> >>> -------------------------ERROR starts here------------------------------- >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>> [0]PETSC ERROR: likely location of problem given in stack below >>> [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>> [0]PETSC ERROR: is given. >>> [0]PETSC ERROR: [0] PetscStrcmp line 414 /root/petsc/petsc-3.3-p7/src/sys/utils/str.c >>> [0]PETSC ERROR: [0] PetscFListAdd line 202 /root/petsc/petsc-3.3-p7/src/sys/dll/reg.c >>> [0]PETSC ERROR: [0] AORegister line 99 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/aoreg.c >>> [0]PETSC ERROR: [0] AORegisterAll line 28 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/aoregall.c >>> [0]PETSC ERROR: [0] AOInitializePackage line 49 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/dlregisdm.c >>> [0]PETSC ERROR: [0] AOCreate line 530 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/ao.c >>> [0]PETSC ERROR: [0] AOCreateBasicIS line 381 /root/petsc/petsc-3.3-p7/src/dm/ao/impls/basic/aobasic.c >>> [0]PETSC ERROR: [0] AOCreateBasic line 336 /root/petsc/petsc-3.3-p7/src/dm/ao/impls/basic/aobasic.c >>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>> [0]PETSC ERROR: Signal received! >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [0]PETSC ERROR: See docs/index.html for manual pages. >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>> [0]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>> [0]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>> [0]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>> -------------------------------------------------------------------------- >>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>> with errorcode 59. >>> >>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>> You may or may not see output from other processes, depending on >>> exactly when Open MPI kills them. >>> -------------------------------------------------------------------------- >>> [1]PETSC ERROR: ------------------------------------------------------------------------ >>> [1]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>> [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>> [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>> [1]PETSC ERROR: likely location of problem given in stack below >>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>> [1]PETSC ERROR: INSTEAD the line number of the start of the function >>> [1]PETSC ERROR: is given. >>> [1]PETSC ERROR: --------------------- Error Message ------------------------------------ >>> [1]PETSC ERROR: Signal received! >>> [1]PETSC ERROR: ------------------------------------------------------------------------ >>> [1]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>> [1]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [1]PETSC ERROR: See docs/index.html for manual pages. >>> [1]PETSC ERROR: ------------------------------------------------------------------------ >>> [1]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>> [1]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>> [1]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>> [1]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>> [1]PETSC ERROR: ------------------------------------------------------------------------ >>> [1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>> [2]PETSC ERROR: ------------------------------------------------------------------------ >>> [2]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>> [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>> [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>> [2]PETSC ERROR: likely location of problem given in stack below >>> [2]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>> [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>> [2]PETSC ERROR: INSTEAD the line number of the start of the function >>> [2]PETSC ERROR: is given. >>> [2]PETSC ERROR: --------------------- Error Message ------------------------------------ >>> [2]PETSC ERROR: Signal received! >>> [2]PETSC ERROR: ------------------------------------------------------------------------ >>> [2]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>> [2]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [2]PETSC ERROR: See docs/index.html for manual pages. >>> [2]PETSC ERROR: ------------------------------------------------------------------------ >>> [2]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>> [2]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>> [2]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>> [2]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>> [2]PETSC ERROR: ------------------------------------------------------------------------ >>> [2]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>> [3]PETSC ERROR: ------------------------------------------------------------------------ >>> [3]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>> [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>> [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>> [3]PETSC ERROR: likely location of problem given in stack below >>> [3]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>> [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>> [3]PETSC ERROR: INSTEAD the line number of the start of the function >>> [3]PETSC ERROR: is given. >>> [3]PETSC ERROR: --------------------- Error Message ------------------------------------ >>> [3]PETSC ERROR: Signal received! >>> [3]PETSC ERROR: ------------------------------------------------------------------------ >>> [3]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>> [3]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [3]PETSC ERROR: See docs/index.html for manual pages. >>> [3]PETSC ERROR: ------------------------------------------------------------------------ >>> [3]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>> [3]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>> [3]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>> [3]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>> [3]PETSC ERROR: ------------------------------------------------------------------------ >>> [3]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>> [4]PETSC ERROR: ------------------------------------------------------------------------ >>> [4]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>> [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>> [4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>> [4]PETSC ERROR: likely location of problem given in stack below >>> [4]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>> [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>> [4]PETSC ERROR: INSTEAD the line number of the start of the function >>> [4]PETSC ERROR: is given. >>> [4]PETSC ERROR: --------------------- Error Message ------------------------------------ >>> [4]PETSC ERROR: Signal received! >>> [4]PETSC ERROR: ------------------------------------------------------------------------ >>> [4]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>> [4]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [4]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [4]PETSC ERROR: See docs/index.html for manual pages. >>> [4]PETSC ERROR: ------------------------------------------------------------------------ >>> [4]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>> [4]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>> [4]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>> [4]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>> [4]PETSC ERROR: ------------------------------------------------------------------------ >>> [4]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>> [5]PETSC ERROR: ------------------------------------------------------------------------ >>> [5]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>> [5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>> [5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[5]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>> [5]PETSC ERROR: likely location of problem given in stack below >>> [5]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>> [5]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>> [5]PETSC ERROR: INSTEAD the line number of the start of the function >>> [5]PETSC ERROR: is given. >>> [5]PETSC ERROR: --------------------- Error Message ------------------------------------ >>> [5]PETSC ERROR: Signal received! >>> [5]PETSC ERROR: ------------------------------------------------------------------------ >>> [5]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>> [5]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [5]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [5]PETSC ERROR: See docs/index.html for manual pages. >>> [5]PETSC ERROR: ------------------------------------------------------------------------ >>> [5]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>> [5]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>> [5]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>> [5]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>> [5]PETSC ERROR: ------------------------------------------------------------------------ >>> [5]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>> [6]PETSC ERROR: ------------------------------------------------------------------------ >>> [6]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>> [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>> [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>> [6]PETSC ERROR: likely location of problem given in stack below >>> [6]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>> [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>> [6]PETSC ERROR: INSTEAD the line number of the start of the function >>> [6]PETSC ERROR: is given. >>> [6]PETSC ERROR: --------------------- Error Message ------------------------------------ >>> [6]PETSC ERROR: Signal received! >>> [6]PETSC ERROR: ------------------------------------------------------------------------ >>> [6]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>> [6]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [6]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [6]PETSC ERROR: See docs/index.html for manual pages. >>> [6]PETSC ERROR: ------------------------------------------------------------------------ >>> [6]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>> [6]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>> [6]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>> [6]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>> [6]PETSC ERROR: ------------------------------------------------------------------------ >>> [6]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>> [7]PETSC ERROR: ------------------------------------------------------------------------ >>> [7]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>> [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>> [7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[7]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>> [7]PETSC ERROR: likely location of problem given in stack below >>> [7]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>> [7]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>> [7]PETSC ERROR: INSTEAD the line number of the start of the function >>> [7]PETSC ERROR: is given. >>> [7]PETSC ERROR: --------------------- Error Message ------------------------------------ >>> [7]PETSC ERROR: Signal received! >>> [7]PETSC ERROR: ------------------------------------------------------------------------ >>> [7]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>> [7]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [7]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [7]PETSC ERROR: See docs/index.html for manual pages. >>> [7]PETSC ERROR: ------------------------------------------------------------------------ >>> [7]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>> [7]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>> [7]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>> [7]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>> [7]PETSC ERROR: ------------------------------------------------------------------------ >>> [7]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>> -------------------------------------------------------------------------- >>> mpiexec has exited due to process rank 0 with PID 2670 on >>> node grsacc17 exiting improperly. There are two reasons this could occur: >>> >>> 1. this process did not call "init" before exiting, but others in >>> the job did. This can cause a job to hang indefinitely while it waits >>> for all processes to call "init". By rule, if one process calls "init", >>> then ALL processes must call "init" prior to termination. >>> >>> 2. this process called "init", but exited without calling "finalize". >>> By rule, all processes that call "init" MUST call "finalize" prior to >>> exiting or it will be considered an "abnormal termination" >>> >>> This may have caused other processes in the application to be >>> terminated by signals sent by mpiexec (as reported here). >> >> > From bsmith at mcs.anl.gov Tue Sep 3 22:13:30 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 3 Sep 2013 22:13:30 -0500 Subject: [petsc-users] Re-initializing petsc again In-Reply-To: References: <315F25F7-7274-40E9-A244-5CD26F975CE5@gmail.com> <614D53AA-89C2-4C34-A192-5FA4BE0D1F27@mcs.anl.gov> Message-ID: <135162E5-14F4-49AF-831F-A5864CAC3A8C@mcs.anl.gov> Something is going wrong with the PETSC_STATIC_INLINE definition. Please send configure.log and make.log Barry On Sep 3, 2013, at 10:01 PM, Suraj Prabhakaran wrote: > Thanks a lot for the confirmation!! > I am now compiling it with 3.4.2, but I get a strange error and could be great if someone can just point out whats wrong. (Might be something stupid, but just in case...) > > Thanks in advance! > > Suraj > > -------------------------------ERROR----------------------------- > > In file included from /opt/petsc-3.4.2/include/petsc.h:8, > from Main/time_integration_dir/compute_tau_w_mean.c:91: > /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: > /opt/petsc-3.4.2/include/petscbt.h:33: error: invalid storage class for function ?PetscBTLength? > /opt/petsc-3.4.2/include/petscbt.h:38: error: invalid storage class for function ?PetscBTMemzero? > /opt/petsc-3.4.2/include/petscbt.h:43: error: invalid storage class for function ?PetscBTDestroy? > /opt/petsc-3.4.2/include/petscbt.h:48: error: invalid storage class for function ?PetscBTLookup? > /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTLookup?: > /opt/petsc-3.4.2/include/petscbt.h:47: warning: declaration of ?index? shadows a global declaration > /usr/include/string.h:489: warning: shadowed declaration is here > /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: > /opt/petsc-3.4.2/include/petscbt.h:59: error: invalid storage class for function ?PetscBTView? > /opt/petsc-3.4.2/include/petscbt.h:74: error: invalid storage class for function ?PetscBTCreate? > /opt/petsc-3.4.2/include/petscbt.h:79: error: invalid storage class for function ?PetscBTLookupSet? > /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTLookupSet?: > /opt/petsc-3.4.2/include/petscbt.h:78: warning: declaration of ?index? shadows a global declaration > /usr/include/string.h:489: warning: shadowed declaration is here > /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: > /opt/petsc-3.4.2/include/petscbt.h:91: error: invalid storage class for function ?PetscBTSet? > /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTSet?: > /opt/petsc-3.4.2/include/petscbt.h:90: warning: declaration of ?index? shadows a global declaration > /usr/include/string.h:489: warning: shadowed declaration is here > /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: > /opt/petsc-3.4.2/include/petscbt.h:103: error: invalid storage class for function ?PetscBTLookupClear? > /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTLookupClear?: > /opt/petsc-3.4.2/include/petscbt.h:102: warning: declaration of ?index? shadows a global declaration > /usr/include/string.h:489: warning: shadowed declaration is here > /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: > /opt/petsc-3.4.2/include/petscbt.h:115: error: invalid storage class for function ?PetscBTClear? > /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTClear?: > /opt/petsc-3.4.2/include/petscbt.h:114: warning: declaration of ?index? shadows a global declaration > /usr/include/string.h:489: warning: shadowed declaration is here > In file included from /opt/petsc-3.4.2/include/petsc.h:9, > from Main/time_integration_dir/compute_tau_w_mean.c:91: > /opt/petsc-3.4.2/include/petscctable.h: In function ?compute_tau_w_mean?: > /opt/petsc-3.4.2/include/petscctable.h:20: error: invalid storage class for function ?PetscHash? > /opt/petsc-3.4.2/include/petscctable.h:40: error: invalid storage class for function ?PetscTableAdd? > In file included from /opt/petsc-3.4.2/include/petsc.h:9, > from Main/time_integration_dir/compute_tau_w_mean.c:91: > /opt/petsc-3.4.2/include/petscctable.h:83: error: invalid storage class for function ?PetscTableAddCount? > /opt/petsc-3.4.2/include/petscctable.h:120: error: invalid storage class for function ?PetscTableFind? > In file included from /opt/petsc-3.4.2/include/petscsnesfas.h:3, > from /opt/petsc-3.4.2/include/petsc.h:27, > from Main/time_integration_dir/compute_tau_w_mean.c:91: > /opt/petsc-3.4.2/include/petscsnes.h:585: error: invalid storage class for function ?SNESGetSNESLineSearch? > /opt/petsc-3.4.2/include/petscsnes.h:586: error: invalid storage class for function ?SNESSetSNESLineSearch? > Main/time_integration_dir/compute_tau_w_mean.c:223: warning: implicit declaration of function ?wall_thermoquants? > Main/time_integration_dir/compute_tau_w_mean.c:155: warning: unused variable ?iface_plus_one? > Main/time_integration_dir/compute_tau_w_mean.c:330: warning: value computed is not used > Main/time_integration_dir/compute_tau_w_mean.c:116: warning: unused variable ?ierr? > Main/time_integration_dir/compute_tau_w_mean.c:111: warning: unused variable ?icell? > Main/time_integration_dir/compute_tau_w_mean.c:105: warning: unused variable ?mach? > /opt/petsc-3.4.2/include/petscts.h:328: warning: unused variable ?TSRegisterAllCalled? > /opt/petsc-3.4.2/include/petscts.h:327: warning: unused variable ?TSList? > /opt/petsc-3.4.2/include/petscts.h:168: warning: unused variable ?DMTS_CLASSID? > /opt/petsc-3.4.2/include/petscts.h:167: warning: unused variable ?TS_CLASSID? > /opt/petsc-3.4.2/include/petscts.h:163: warning: unused variable ?TSExactFinalTimeOptions? > /opt/petsc-3.4.2/include/petscts.h:101: warning: unused variable ?TSConvergedReasons? > /opt/petsc-3.4.2/include/petscts.h:80: warning: unused variable ?TSEquationTypes? > /opt/petsc-3.4.2/include/petscsnesfas.h:19: warning: unused variable ?SNESFASTypes? > /opt/petsc-3.4.2/include/petscsnes.h:679: warning: unused variable ?SNESQNRestartTypes? > /opt/petsc-3.4.2/include/petscsnes.h:675: warning: unused variable ?SNESQNScaleTypes? > /opt/petsc-3.4.2/include/petscsnes.h:667: warning: unused variable ?SNESNCGTypes? > /opt/petsc-3.4.2/include/petscsnes.h:654: warning: unused variable ?SNESNGMRESSelectTypes? > /opt/petsc-3.4.2/include/petscsnes.h:648: warning: unused variable ?SNESNGMRESRestartTypes? > /opt/petsc-3.4.2/include/petscsnes.h:474: warning: unused variable ?SNESLineSearch_Apply? > /opt/petsc-3.4.2/include/petscsnes.h:473: warning: unused variable ?SNESLineSearchList? > /opt/petsc-3.4.2/include/petscsnes.h:472: warning: unused variable ?SNESLineSearchRegisterAllCalled? > /opt/petsc-3.4.2/include/petscsnes.h:471: warning: unused variable ?SNESLINESEARCH_CLASSID? > /opt/petsc-3.4.2/include/petscsnes.h:470: warning: unused variable ?SNESList? > /opt/petsc-3.4.2/include/petscsnes.h:469: warning: unused variable ?SNESRegisterAllCalled? > /opt/petsc-3.4.2/include/petscsnes.h:356: warning: unused variable ?SNESNormTypes? > /opt/petsc-3.4.2/include/petscsnes.h:217: warning: unused variable ?SNESConvergedReasons? > /opt/petsc-3.4.2/include/petscsnes.h:50: warning: unused variable ?DMSNES_CLASSID? > /opt/petsc-3.4.2/include/petscsnes.h:49: warning: unused variable ?SNES_CLASSID? > /opt/petsc-3.4.2/include/petsccharacteristic.h:44: warning: unused variable ?CharacteristicList? > /opt/petsc-3.4.2/include/petsccharacteristic.h:43: warning: unused variable ?CharacteristicRegisterAllCalled? > /opt/petsc-3.4.2/include/petscdmda.h:27: warning: unused variable ?DMDABoundaryTypes? > /opt/petsc-3.4.2/include/petscpf.h:50: warning: unused variable ?PFRegisterAllCalled? > /opt/petsc-3.4.2/include/petscpf.h:40: warning: unused variable ?PF_CLASSID? > /opt/petsc-3.4.2/include/petscpf.h:12: warning: unused variable ?PFList? > /opt/petsc-3.4.2/include/petscdm.h:34: warning: unused variable ?DMRegisterAllCalled? > /opt/petsc-3.4.2/include/petscdm.h:33: warning: unused variable ?DMList? > /opt/petsc-3.4.2/include/petscdm.h:11: warning: unused variable ?DM_CLASSID? > /opt/petsc-3.4.2/include/petscdraw.h:239: warning: unused variable ?PETSC_DRAWHG_CLASSID? > /opt/petsc-3.4.2/include/petscdraw.h:225: warning: unused variable ?PETSC_DRAWSP_CLASSID? > /opt/petsc-3.4.2/include/petscdraw.h:206: warning: unused variable ?PETSC_DRAWLG_CLASSID? > /opt/petsc-3.4.2/include/petscdraw.h:195: warning: unused variable ?PETSC_DRAWAXIS_CLASSID? > /opt/petsc-3.4.2/include/petscdraw.h:26: warning: unused variable ?PetscDrawList? > /opt/petsc-3.4.2/include/petscdraw.h:9: warning: unused variable ?PETSC_DRAW_CLASSID? > /opt/petsc-3.4.2/include/petscmatlab.h:8: warning: unused variable ?MATLABENGINE_CLASSID? > /opt/petsc-3.4.2/include/petscthreadcomm.h:35: warning: unused variable ?PetscThreadCommReductionOps? > /opt/petsc-3.4.2/include/petscthreadcomm.h:32: warning: unused variable ?PetscThreadCommList? > > > > On Sep 3, 2013, at 9:03 PM, Satish Balay wrote: > >> Also you would have to call MPI_Init() before the first PetscInitialize() [and >> similarly MPI_Finalize() after the last call to PetscFinalize()] >> >> Satish >> >> On Tue, 3 Sep 2013, Barry Smith wrote: >> >>> >>> There is a problem with the 3.3 series when reinitializing PETSc. Can you switch to PETSc 3.4 which properly supports calling PetscInitialize() several times. >>> >>> Barry >>> >>> On Sep 3, 2013, at 1:58 PM, Suraj Prabhakaran wrote: >>> >>>> Hello, >>>> >>>> Again, this relates to finalizing and initializing petsc in my application. I basically want to finalize petsc and initialize it again with a different MPI communicator. A rough sketch of my quite huge application is as follows. >>>> >>>> >>>> main() >>>> { >>>> PetscInitialize() >>>> ... >>>> ... >>>> AOCreate().. >>>> AO operations.. >>>> AODestroy().. >>>> ... >>>> ... >>>> PetscFinalize() >>>> PETSC_COMM_WORLD = A new communicator (which includes more processes) >>>> PetscInitialize() >>>> ... >>>> ... >>>> AOCreate().. >>>> AO operations.. >>>> AODestroy() >>>> ... >>>> ... >>>> PetscFinalize() >>>> } >>>> >>>> So after finalizing and initializing it back, I always get an error with one of the AO calls. Given below is the complete error I get when running with 8 procs. Can anyone give me a hint what the problem could be? I was told earlier, that finalizing and initializing petsc should not be a problem! >>>> >>>> Best, >>>> Suraj >>>> >>>> -------------------------ERROR starts here------------------------------- >>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>> [0]PETSC ERROR: likely location of problem given in stack below >>>> [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>> [0]PETSC ERROR: is given. >>>> [0]PETSC ERROR: [0] PetscStrcmp line 414 /root/petsc/petsc-3.3-p7/src/sys/utils/str.c >>>> [0]PETSC ERROR: [0] PetscFListAdd line 202 /root/petsc/petsc-3.3-p7/src/sys/dll/reg.c >>>> [0]PETSC ERROR: [0] AORegister line 99 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/aoreg.c >>>> [0]PETSC ERROR: [0] AORegisterAll line 28 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/aoregall.c >>>> [0]PETSC ERROR: [0] AOInitializePackage line 49 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/dlregisdm.c >>>> [0]PETSC ERROR: [0] AOCreate line 530 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/ao.c >>>> [0]PETSC ERROR: [0] AOCreateBasicIS line 381 /root/petsc/petsc-3.3-p7/src/dm/ao/impls/basic/aobasic.c >>>> [0]PETSC ERROR: [0] AOCreateBasic line 336 /root/petsc/petsc-3.3-p7/src/dm/ao/impls/basic/aobasic.c >>>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>> [0]PETSC ERROR: Signal received! >>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>> [0]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>> [0]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>> [0]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>> -------------------------------------------------------------------------- >>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>>> with errorcode 59. >>>> >>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>> You may or may not see output from other processes, depending on >>>> exactly when Open MPI kills them. >>>> -------------------------------------------------------------------------- >>>> [1]PETSC ERROR: ------------------------------------------------------------------------ >>>> [1]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>> [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>> [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>> [1]PETSC ERROR: likely location of problem given in stack below >>>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>> [1]PETSC ERROR: INSTEAD the line number of the start of the function >>>> [1]PETSC ERROR: is given. >>>> [1]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>> [1]PETSC ERROR: Signal received! >>>> [1]PETSC ERROR: ------------------------------------------------------------------------ >>>> [1]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>> [1]PETSC ERROR: See docs/changes/index.html for recent updates. >>>> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>> [1]PETSC ERROR: See docs/index.html for manual pages. >>>> [1]PETSC ERROR: ------------------------------------------------------------------------ >>>> [1]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>> [1]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>> [1]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>> [1]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>> [1]PETSC ERROR: ------------------------------------------------------------------------ >>>> [1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>> [2]PETSC ERROR: ------------------------------------------------------------------------ >>>> [2]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>> [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>> [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>> [2]PETSC ERROR: likely location of problem given in stack below >>>> [2]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>> [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>> [2]PETSC ERROR: INSTEAD the line number of the start of the function >>>> [2]PETSC ERROR: is given. >>>> [2]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>> [2]PETSC ERROR: Signal received! >>>> [2]PETSC ERROR: ------------------------------------------------------------------------ >>>> [2]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>> [2]PETSC ERROR: See docs/changes/index.html for recent updates. >>>> [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>> [2]PETSC ERROR: See docs/index.html for manual pages. >>>> [2]PETSC ERROR: ------------------------------------------------------------------------ >>>> [2]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>> [2]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>> [2]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>> [2]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>> [2]PETSC ERROR: ------------------------------------------------------------------------ >>>> [2]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>> [3]PETSC ERROR: ------------------------------------------------------------------------ >>>> [3]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>> [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>> [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>> [3]PETSC ERROR: likely location of problem given in stack below >>>> [3]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>> [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>> [3]PETSC ERROR: INSTEAD the line number of the start of the function >>>> [3]PETSC ERROR: is given. >>>> [3]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>> [3]PETSC ERROR: Signal received! >>>> [3]PETSC ERROR: ------------------------------------------------------------------------ >>>> [3]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>> [3]PETSC ERROR: See docs/changes/index.html for recent updates. >>>> [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>> [3]PETSC ERROR: See docs/index.html for manual pages. >>>> [3]PETSC ERROR: ------------------------------------------------------------------------ >>>> [3]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>> [3]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>> [3]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>> [3]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>> [3]PETSC ERROR: ------------------------------------------------------------------------ >>>> [3]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>> [4]PETSC ERROR: ------------------------------------------------------------------------ >>>> [4]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>> [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>> [4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>> [4]PETSC ERROR: likely location of problem given in stack below >>>> [4]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>> [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>> [4]PETSC ERROR: INSTEAD the line number of the start of the function >>>> [4]PETSC ERROR: is given. >>>> [4]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>> [4]PETSC ERROR: Signal received! >>>> [4]PETSC ERROR: ------------------------------------------------------------------------ >>>> [4]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>> [4]PETSC ERROR: See docs/changes/index.html for recent updates. >>>> [4]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>> [4]PETSC ERROR: See docs/index.html for manual pages. >>>> [4]PETSC ERROR: ------------------------------------------------------------------------ >>>> [4]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>> [4]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>> [4]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>> [4]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>> [4]PETSC ERROR: ------------------------------------------------------------------------ >>>> [4]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>> [5]PETSC ERROR: ------------------------------------------------------------------------ >>>> [5]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>> [5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>> [5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[5]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>> [5]PETSC ERROR: likely location of problem given in stack below >>>> [5]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>> [5]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>> [5]PETSC ERROR: INSTEAD the line number of the start of the function >>>> [5]PETSC ERROR: is given. >>>> [5]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>> [5]PETSC ERROR: Signal received! >>>> [5]PETSC ERROR: ------------------------------------------------------------------------ >>>> [5]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>> [5]PETSC ERROR: See docs/changes/index.html for recent updates. >>>> [5]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>> [5]PETSC ERROR: See docs/index.html for manual pages. >>>> [5]PETSC ERROR: ------------------------------------------------------------------------ >>>> [5]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>> [5]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>> [5]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>> [5]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>> [5]PETSC ERROR: ------------------------------------------------------------------------ >>>> [5]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>> [6]PETSC ERROR: ------------------------------------------------------------------------ >>>> [6]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>> [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>> [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>> [6]PETSC ERROR: likely location of problem given in stack below >>>> [6]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>> [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>> [6]PETSC ERROR: INSTEAD the line number of the start of the function >>>> [6]PETSC ERROR: is given. >>>> [6]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>> [6]PETSC ERROR: Signal received! >>>> [6]PETSC ERROR: ------------------------------------------------------------------------ >>>> [6]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>> [6]PETSC ERROR: See docs/changes/index.html for recent updates. >>>> [6]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>> [6]PETSC ERROR: See docs/index.html for manual pages. >>>> [6]PETSC ERROR: ------------------------------------------------------------------------ >>>> [6]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>> [6]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>> [6]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>> [6]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>> [6]PETSC ERROR: ------------------------------------------------------------------------ >>>> [6]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>> [7]PETSC ERROR: ------------------------------------------------------------------------ >>>> [7]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>> [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>> [7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[7]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>> [7]PETSC ERROR: likely location of problem given in stack below >>>> [7]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>> [7]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>> [7]PETSC ERROR: INSTEAD the line number of the start of the function >>>> [7]PETSC ERROR: is given. >>>> [7]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>> [7]PETSC ERROR: Signal received! >>>> [7]PETSC ERROR: ------------------------------------------------------------------------ >>>> [7]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>> [7]PETSC ERROR: See docs/changes/index.html for recent updates. >>>> [7]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>> [7]PETSC ERROR: See docs/index.html for manual pages. >>>> [7]PETSC ERROR: ------------------------------------------------------------------------ >>>> [7]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>> [7]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>> [7]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>> [7]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>> [7]PETSC ERROR: ------------------------------------------------------------------------ >>>> [7]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>> -------------------------------------------------------------------------- >>>> mpiexec has exited due to process rank 0 with PID 2670 on >>>> node grsacc17 exiting improperly. There are two reasons this could occur: >>>> >>>> 1. this process did not call "init" before exiting, but others in >>>> the job did. This can cause a job to hang indefinitely while it waits >>>> for all processes to call "init". By rule, if one process calls "init", >>>> then ALL processes must call "init" prior to termination. >>>> >>>> 2. this process called "init", but exited without calling "finalize". >>>> By rule, all processes that call "init" MUST call "finalize" prior to >>>> exiting or it will be considered an "abnormal termination" >>>> >>>> This may have caused other processes in the application to be >>>> terminated by signals sent by mpiexec (as reported here). >>> >>> >> > From balay at mcs.anl.gov Tue Sep 3 22:14:47 2013 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 3 Sep 2013 22:14:47 -0500 (CDT) Subject: [petsc-users] Re-initializing petsc again In-Reply-To: References: <315F25F7-7274-40E9-A244-5CD26F975CE5@gmail.com> <614D53AA-89C2-4C34-A192-5FA4BE0D1F27@mcs.anl.gov> Message-ID: This is incomplete info. what compile command produced this error? And what do you get when you compile a petsc example with a petsc makefile? [complete output] Satish On Wed, 4 Sep 2013, Suraj Prabhakaran wrote: > Thanks a lot for the confirmation!! > I am now compiling it with 3.4.2, but I get a strange error and could be great if someone can just point out whats wrong. (Might be something stupid, but just in case...) > > Thanks in advance! > > Suraj > > -------------------------------ERROR----------------------------- > > In file included from /opt/petsc-3.4.2/include/petsc.h:8, > from Main/time_integration_dir/compute_tau_w_mean.c:91: > /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: > /opt/petsc-3.4.2/include/petscbt.h:33: error: invalid storage class for function ?PetscBTLength? > /opt/petsc-3.4.2/include/petscbt.h:38: error: invalid storage class for function ?PetscBTMemzero? > /opt/petsc-3.4.2/include/petscbt.h:43: error: invalid storage class for function ?PetscBTDestroy? > /opt/petsc-3.4.2/include/petscbt.h:48: error: invalid storage class for function ?PetscBTLookup? > /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTLookup?: > /opt/petsc-3.4.2/include/petscbt.h:47: warning: declaration of ?index? shadows a global declaration > /usr/include/string.h:489: warning: shadowed declaration is here > /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: > /opt/petsc-3.4.2/include/petscbt.h:59: error: invalid storage class for function ?PetscBTView? > /opt/petsc-3.4.2/include/petscbt.h:74: error: invalid storage class for function ?PetscBTCreate? > /opt/petsc-3.4.2/include/petscbt.h:79: error: invalid storage class for function ?PetscBTLookupSet? > /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTLookupSet?: > /opt/petsc-3.4.2/include/petscbt.h:78: warning: declaration of ?index? shadows a global declaration > /usr/include/string.h:489: warning: shadowed declaration is here > /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: > /opt/petsc-3.4.2/include/petscbt.h:91: error: invalid storage class for function ?PetscBTSet? > /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTSet?: > /opt/petsc-3.4.2/include/petscbt.h:90: warning: declaration of ?index? shadows a global declaration > /usr/include/string.h:489: warning: shadowed declaration is here > /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: > /opt/petsc-3.4.2/include/petscbt.h:103: error: invalid storage class for function ?PetscBTLookupClear? > /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTLookupClear?: > /opt/petsc-3.4.2/include/petscbt.h:102: warning: declaration of ?index? shadows a global declaration > /usr/include/string.h:489: warning: shadowed declaration is here > /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: > /opt/petsc-3.4.2/include/petscbt.h:115: error: invalid storage class for function ?PetscBTClear? > /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTClear?: > /opt/petsc-3.4.2/include/petscbt.h:114: warning: declaration of ?index? shadows a global declaration > /usr/include/string.h:489: warning: shadowed declaration is here > In file included from /opt/petsc-3.4.2/include/petsc.h:9, > from Main/time_integration_dir/compute_tau_w_mean.c:91: > /opt/petsc-3.4.2/include/petscctable.h: In function ?compute_tau_w_mean?: > /opt/petsc-3.4.2/include/petscctable.h:20: error: invalid storage class for function ?PetscHash? > /opt/petsc-3.4.2/include/petscctable.h:40: error: invalid storage class for function ?PetscTableAdd? > In file included from /opt/petsc-3.4.2/include/petsc.h:9, > from Main/time_integration_dir/compute_tau_w_mean.c:91: > /opt/petsc-3.4.2/include/petscctable.h:83: error: invalid storage class for function ?PetscTableAddCount? > /opt/petsc-3.4.2/include/petscctable.h:120: error: invalid storage class for function ?PetscTableFind? > In file included from /opt/petsc-3.4.2/include/petscsnesfas.h:3, > from /opt/petsc-3.4.2/include/petsc.h:27, > from Main/time_integration_dir/compute_tau_w_mean.c:91: > /opt/petsc-3.4.2/include/petscsnes.h:585: error: invalid storage class for function ?SNESGetSNESLineSearch? > /opt/petsc-3.4.2/include/petscsnes.h:586: error: invalid storage class for function ?SNESSetSNESLineSearch? > Main/time_integration_dir/compute_tau_w_mean.c:223: warning: implicit declaration of function ?wall_thermoquants? > Main/time_integration_dir/compute_tau_w_mean.c:155: warning: unused variable ?iface_plus_one? > Main/time_integration_dir/compute_tau_w_mean.c:330: warning: value computed is not used > Main/time_integration_dir/compute_tau_w_mean.c:116: warning: unused variable ?ierr? > Main/time_integration_dir/compute_tau_w_mean.c:111: warning: unused variable ?icell? > Main/time_integration_dir/compute_tau_w_mean.c:105: warning: unused variable ?mach? > /opt/petsc-3.4.2/include/petscts.h:328: warning: unused variable ?TSRegisterAllCalled? > /opt/petsc-3.4.2/include/petscts.h:327: warning: unused variable ?TSList? > /opt/petsc-3.4.2/include/petscts.h:168: warning: unused variable ?DMTS_CLASSID? > /opt/petsc-3.4.2/include/petscts.h:167: warning: unused variable ?TS_CLASSID? > /opt/petsc-3.4.2/include/petscts.h:163: warning: unused variable ?TSExactFinalTimeOptions? > /opt/petsc-3.4.2/include/petscts.h:101: warning: unused variable ?TSConvergedReasons? > /opt/petsc-3.4.2/include/petscts.h:80: warning: unused variable ?TSEquationTypes? > /opt/petsc-3.4.2/include/petscsnesfas.h:19: warning: unused variable ?SNESFASTypes? > /opt/petsc-3.4.2/include/petscsnes.h:679: warning: unused variable ?SNESQNRestartTypes? > /opt/petsc-3.4.2/include/petscsnes.h:675: warning: unused variable ?SNESQNScaleTypes? > /opt/petsc-3.4.2/include/petscsnes.h:667: warning: unused variable ?SNESNCGTypes? > /opt/petsc-3.4.2/include/petscsnes.h:654: warning: unused variable ?SNESNGMRESSelectTypes? > /opt/petsc-3.4.2/include/petscsnes.h:648: warning: unused variable ?SNESNGMRESRestartTypes? > /opt/petsc-3.4.2/include/petscsnes.h:474: warning: unused variable ?SNESLineSearch_Apply? > /opt/petsc-3.4.2/include/petscsnes.h:473: warning: unused variable ?SNESLineSearchList? > /opt/petsc-3.4.2/include/petscsnes.h:472: warning: unused variable ?SNESLineSearchRegisterAllCalled? > /opt/petsc-3.4.2/include/petscsnes.h:471: warning: unused variable ?SNESLINESEARCH_CLASSID? > /opt/petsc-3.4.2/include/petscsnes.h:470: warning: unused variable ?SNESList? > /opt/petsc-3.4.2/include/petscsnes.h:469: warning: unused variable ?SNESRegisterAllCalled? > /opt/petsc-3.4.2/include/petscsnes.h:356: warning: unused variable ?SNESNormTypes? > /opt/petsc-3.4.2/include/petscsnes.h:217: warning: unused variable ?SNESConvergedReasons? > /opt/petsc-3.4.2/include/petscsnes.h:50: warning: unused variable ?DMSNES_CLASSID? > /opt/petsc-3.4.2/include/petscsnes.h:49: warning: unused variable ?SNES_CLASSID? > /opt/petsc-3.4.2/include/petsccharacteristic.h:44: warning: unused variable ?CharacteristicList? > /opt/petsc-3.4.2/include/petsccharacteristic.h:43: warning: unused variable ?CharacteristicRegisterAllCalled? > /opt/petsc-3.4.2/include/petscdmda.h:27: warning: unused variable ?DMDABoundaryTypes? > /opt/petsc-3.4.2/include/petscpf.h:50: warning: unused variable ?PFRegisterAllCalled? > /opt/petsc-3.4.2/include/petscpf.h:40: warning: unused variable ?PF_CLASSID? > /opt/petsc-3.4.2/include/petscpf.h:12: warning: unused variable ?PFList? > /opt/petsc-3.4.2/include/petscdm.h:34: warning: unused variable ?DMRegisterAllCalled? > /opt/petsc-3.4.2/include/petscdm.h:33: warning: unused variable ?DMList? > /opt/petsc-3.4.2/include/petscdm.h:11: warning: unused variable ?DM_CLASSID? > /opt/petsc-3.4.2/include/petscdraw.h:239: warning: unused variable ?PETSC_DRAWHG_CLASSID? > /opt/petsc-3.4.2/include/petscdraw.h:225: warning: unused variable ?PETSC_DRAWSP_CLASSID? > /opt/petsc-3.4.2/include/petscdraw.h:206: warning: unused variable ?PETSC_DRAWLG_CLASSID? > /opt/petsc-3.4.2/include/petscdraw.h:195: warning: unused variable ?PETSC_DRAWAXIS_CLASSID? > /opt/petsc-3.4.2/include/petscdraw.h:26: warning: unused variable ?PetscDrawList? > /opt/petsc-3.4.2/include/petscdraw.h:9: warning: unused variable ?PETSC_DRAW_CLASSID? > /opt/petsc-3.4.2/include/petscmatlab.h:8: warning: unused variable ?MATLABENGINE_CLASSID? > /opt/petsc-3.4.2/include/petscthreadcomm.h:35: warning: unused variable ?PetscThreadCommReductionOps? > /opt/petsc-3.4.2/include/petscthreadcomm.h:32: warning: unused variable ?PetscThreadCommList? > > > > On Sep 3, 2013, at 9:03 PM, Satish Balay wrote: > > > Also you would have to call MPI_Init() before the first PetscInitialize() [and > > similarly MPI_Finalize() after the last call to PetscFinalize()] > > > > Satish > > > > On Tue, 3 Sep 2013, Barry Smith wrote: > > > >> > >> There is a problem with the 3.3 series when reinitializing PETSc. Can you switch to PETSc 3.4 which properly supports calling PetscInitialize() several times. > >> > >> Barry > >> > >> On Sep 3, 2013, at 1:58 PM, Suraj Prabhakaran wrote: > >> > >>> Hello, > >>> > >>> Again, this relates to finalizing and initializing petsc in my application. I basically want to finalize petsc and initialize it again with a different MPI communicator. A rough sketch of my quite huge application is as follows. > >>> > >>> > >>> main() > >>> { > >>> PetscInitialize() > >>> ... > >>> ... > >>> AOCreate().. > >>> AO operations.. > >>> AODestroy().. > >>> ... > >>> ... > >>> PetscFinalize() > >>> PETSC_COMM_WORLD = A new communicator (which includes more processes) > >>> PetscInitialize() > >>> ... > >>> ... > >>> AOCreate().. > >>> AO operations.. > >>> AODestroy() > >>> ... > >>> ... > >>> PetscFinalize() > >>> } > >>> > >>> So after finalizing and initializing it back, I always get an error with one of the AO calls. Given below is the complete error I get when running with 8 procs. Can anyone give me a hint what the problem could be? I was told earlier, that finalizing and initializing petsc should not be a problem! > >>> > >>> Best, > >>> Suraj > >>> > >>> -------------------------ERROR starts here------------------------------- > >>> [0]PETSC ERROR: ------------------------------------------------------------------------ > >>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > >>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > >>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > >>> [0]PETSC ERROR: likely location of problem given in stack below > >>> [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > >>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > >>> [0]PETSC ERROR: INSTEAD the line number of the start of the function > >>> [0]PETSC ERROR: is given. > >>> [0]PETSC ERROR: [0] PetscStrcmp line 414 /root/petsc/petsc-3.3-p7/src/sys/utils/str.c > >>> [0]PETSC ERROR: [0] PetscFListAdd line 202 /root/petsc/petsc-3.3-p7/src/sys/dll/reg.c > >>> [0]PETSC ERROR: [0] AORegister line 99 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/aoreg.c > >>> [0]PETSC ERROR: [0] AORegisterAll line 28 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/aoregall.c > >>> [0]PETSC ERROR: [0] AOInitializePackage line 49 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/dlregisdm.c > >>> [0]PETSC ERROR: [0] AOCreate line 530 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/ao.c > >>> [0]PETSC ERROR: [0] AOCreateBasicIS line 381 /root/petsc/petsc-3.3-p7/src/dm/ao/impls/basic/aobasic.c > >>> [0]PETSC ERROR: [0] AOCreateBasic line 336 /root/petsc/petsc-3.3-p7/src/dm/ao/impls/basic/aobasic.c > >>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > >>> [0]PETSC ERROR: Signal received! > >>> [0]PETSC ERROR: ------------------------------------------------------------------------ > >>> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > >>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. > >>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > >>> [0]PETSC ERROR: See docs/index.html for manual pages. > >>> [0]PETSC ERROR: ------------------------------------------------------------------------ > >>> [0]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > >>> [0]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > >>> [0]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > >>> [0]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > >>> [0]PETSC ERROR: ------------------------------------------------------------------------ > >>> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > >>> -------------------------------------------------------------------------- > >>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > >>> with errorcode 59. > >>> > >>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > >>> You may or may not see output from other processes, depending on > >>> exactly when Open MPI kills them. > >>> -------------------------------------------------------------------------- > >>> [1]PETSC ERROR: ------------------------------------------------------------------------ > >>> [1]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > >>> [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > >>> [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > >>> [1]PETSC ERROR: likely location of problem given in stack below > >>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > >>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > >>> [1]PETSC ERROR: INSTEAD the line number of the start of the function > >>> [1]PETSC ERROR: is given. > >>> [1]PETSC ERROR: --------------------- Error Message ------------------------------------ > >>> [1]PETSC ERROR: Signal received! > >>> [1]PETSC ERROR: ------------------------------------------------------------------------ > >>> [1]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > >>> [1]PETSC ERROR: See docs/changes/index.html for recent updates. > >>> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > >>> [1]PETSC ERROR: See docs/index.html for manual pages. > >>> [1]PETSC ERROR: ------------------------------------------------------------------------ > >>> [1]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > >>> [1]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > >>> [1]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > >>> [1]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > >>> [1]PETSC ERROR: ------------------------------------------------------------------------ > >>> [1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > >>> [2]PETSC ERROR: ------------------------------------------------------------------------ > >>> [2]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > >>> [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > >>> [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > >>> [2]PETSC ERROR: likely location of problem given in stack below > >>> [2]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > >>> [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > >>> [2]PETSC ERROR: INSTEAD the line number of the start of the function > >>> [2]PETSC ERROR: is given. > >>> [2]PETSC ERROR: --------------------- Error Message ------------------------------------ > >>> [2]PETSC ERROR: Signal received! > >>> [2]PETSC ERROR: ------------------------------------------------------------------------ > >>> [2]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > >>> [2]PETSC ERROR: See docs/changes/index.html for recent updates. > >>> [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > >>> [2]PETSC ERROR: See docs/index.html for manual pages. > >>> [2]PETSC ERROR: ------------------------------------------------------------------------ > >>> [2]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > >>> [2]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > >>> [2]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > >>> [2]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > >>> [2]PETSC ERROR: ------------------------------------------------------------------------ > >>> [2]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > >>> [3]PETSC ERROR: ------------------------------------------------------------------------ > >>> [3]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > >>> [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > >>> [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > >>> [3]PETSC ERROR: likely location of problem given in stack below > >>> [3]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > >>> [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > >>> [3]PETSC ERROR: INSTEAD the line number of the start of the function > >>> [3]PETSC ERROR: is given. > >>> [3]PETSC ERROR: --------------------- Error Message ------------------------------------ > >>> [3]PETSC ERROR: Signal received! > >>> [3]PETSC ERROR: ------------------------------------------------------------------------ > >>> [3]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > >>> [3]PETSC ERROR: See docs/changes/index.html for recent updates. > >>> [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > >>> [3]PETSC ERROR: See docs/index.html for manual pages. > >>> [3]PETSC ERROR: ------------------------------------------------------------------------ > >>> [3]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > >>> [3]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > >>> [3]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > >>> [3]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > >>> [3]PETSC ERROR: ------------------------------------------------------------------------ > >>> [3]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > >>> [4]PETSC ERROR: ------------------------------------------------------------------------ > >>> [4]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > >>> [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > >>> [4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > >>> [4]PETSC ERROR: likely location of problem given in stack below > >>> [4]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > >>> [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > >>> [4]PETSC ERROR: INSTEAD the line number of the start of the function > >>> [4]PETSC ERROR: is given. > >>> [4]PETSC ERROR: --------------------- Error Message ------------------------------------ > >>> [4]PETSC ERROR: Signal received! > >>> [4]PETSC ERROR: ------------------------------------------------------------------------ > >>> [4]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > >>> [4]PETSC ERROR: See docs/changes/index.html for recent updates. > >>> [4]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > >>> [4]PETSC ERROR: See docs/index.html for manual pages. > >>> [4]PETSC ERROR: ------------------------------------------------------------------------ > >>> [4]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > >>> [4]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > >>> [4]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > >>> [4]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > >>> [4]PETSC ERROR: ------------------------------------------------------------------------ > >>> [4]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > >>> [5]PETSC ERROR: ------------------------------------------------------------------------ > >>> [5]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > >>> [5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > >>> [5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[5]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > >>> [5]PETSC ERROR: likely location of problem given in stack below > >>> [5]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > >>> [5]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > >>> [5]PETSC ERROR: INSTEAD the line number of the start of the function > >>> [5]PETSC ERROR: is given. > >>> [5]PETSC ERROR: --------------------- Error Message ------------------------------------ > >>> [5]PETSC ERROR: Signal received! > >>> [5]PETSC ERROR: ------------------------------------------------------------------------ > >>> [5]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > >>> [5]PETSC ERROR: See docs/changes/index.html for recent updates. > >>> [5]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > >>> [5]PETSC ERROR: See docs/index.html for manual pages. > >>> [5]PETSC ERROR: ------------------------------------------------------------------------ > >>> [5]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > >>> [5]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > >>> [5]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > >>> [5]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > >>> [5]PETSC ERROR: ------------------------------------------------------------------------ > >>> [5]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > >>> [6]PETSC ERROR: ------------------------------------------------------------------------ > >>> [6]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > >>> [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > >>> [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > >>> [6]PETSC ERROR: likely location of problem given in stack below > >>> [6]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > >>> [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > >>> [6]PETSC ERROR: INSTEAD the line number of the start of the function > >>> [6]PETSC ERROR: is given. > >>> [6]PETSC ERROR: --------------------- Error Message ------------------------------------ > >>> [6]PETSC ERROR: Signal received! > >>> [6]PETSC ERROR: ------------------------------------------------------------------------ > >>> [6]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > >>> [6]PETSC ERROR: See docs/changes/index.html for recent updates. > >>> [6]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > >>> [6]PETSC ERROR: See docs/index.html for manual pages. > >>> [6]PETSC ERROR: ------------------------------------------------------------------------ > >>> [6]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > >>> [6]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > >>> [6]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > >>> [6]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > >>> [6]PETSC ERROR: ------------------------------------------------------------------------ > >>> [6]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > >>> [7]PETSC ERROR: ------------------------------------------------------------------------ > >>> [7]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > >>> [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > >>> [7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[7]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > >>> [7]PETSC ERROR: likely location of problem given in stack below > >>> [7]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > >>> [7]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > >>> [7]PETSC ERROR: INSTEAD the line number of the start of the function > >>> [7]PETSC ERROR: is given. > >>> [7]PETSC ERROR: --------------------- Error Message ------------------------------------ > >>> [7]PETSC ERROR: Signal received! > >>> [7]PETSC ERROR: ------------------------------------------------------------------------ > >>> [7]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > >>> [7]PETSC ERROR: See docs/changes/index.html for recent updates. > >>> [7]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > >>> [7]PETSC ERROR: See docs/index.html for manual pages. > >>> [7]PETSC ERROR: ------------------------------------------------------------------------ > >>> [7]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > >>> [7]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > >>> [7]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > >>> [7]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > >>> [7]PETSC ERROR: ------------------------------------------------------------------------ > >>> [7]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > >>> -------------------------------------------------------------------------- > >>> mpiexec has exited due to process rank 0 with PID 2670 on > >>> node grsacc17 exiting improperly. There are two reasons this could occur: > >>> > >>> 1. this process did not call "init" before exiting, but others in > >>> the job did. This can cause a job to hang indefinitely while it waits > >>> for all processes to call "init". By rule, if one process calls "init", > >>> then ALL processes must call "init" prior to termination. > >>> > >>> 2. this process called "init", but exited without calling "finalize". > >>> By rule, all processes that call "init" MUST call "finalize" prior to > >>> exiting or it will be considered an "abnormal termination" > >>> > >>> This may have caused other processes in the application to be > >>> terminated by signals sent by mpiexec (as reported here). > >> > >> > > > > From suraj.prabhakaran at gmail.com Tue Sep 3 22:21:10 2013 From: suraj.prabhakaran at gmail.com (Suraj Prabhakaran) Date: Wed, 4 Sep 2013 05:21:10 +0200 Subject: [petsc-users] Re-initializing petsc again In-Reply-To: <135162E5-14F4-49AF-831F-A5864CAC3A8C@mcs.anl.gov> References: <315F25F7-7274-40E9-A244-5CD26F975CE5@gmail.com> <614D53AA-89C2-4C34-A192-5FA4BE0D1F27@mcs.anl.gov> <135162E5-14F4-49AF-831F-A5864CAC3A8C@mcs.anl.gov> Message-ID: <7C516550-089B-4ACF-AE39-3DD4BC38EA41@gmail.com> Hi Barry, Thanks a lot! Here are the files attached. Suraj -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 2016815 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: make.log Type: application/octet-stream Size: 646994 bytes Desc: not available URL: -------------- next part -------------- On Sep 4, 2013, at 5:13 AM, Barry Smith wrote: > > Something is going wrong with the PETSC_STATIC_INLINE definition. Please send configure.log and make.log > > Barry > > On Sep 3, 2013, at 10:01 PM, Suraj Prabhakaran wrote: > >> Thanks a lot for the confirmation!! >> I am now compiling it with 3.4.2, but I get a strange error and could be great if someone can just point out whats wrong. (Might be something stupid, but just in case...) >> >> Thanks in advance! >> >> Suraj >> >> -------------------------------ERROR----------------------------- >> >> In file included from /opt/petsc-3.4.2/include/petsc.h:8, >> from Main/time_integration_dir/compute_tau_w_mean.c:91: >> /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: >> /opt/petsc-3.4.2/include/petscbt.h:33: error: invalid storage class for function ?PetscBTLength? >> /opt/petsc-3.4.2/include/petscbt.h:38: error: invalid storage class for function ?PetscBTMemzero? >> /opt/petsc-3.4.2/include/petscbt.h:43: error: invalid storage class for function ?PetscBTDestroy? >> /opt/petsc-3.4.2/include/petscbt.h:48: error: invalid storage class for function ?PetscBTLookup? >> /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTLookup?: >> /opt/petsc-3.4.2/include/petscbt.h:47: warning: declaration of ?index? shadows a global declaration >> /usr/include/string.h:489: warning: shadowed declaration is here >> /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: >> /opt/petsc-3.4.2/include/petscbt.h:59: error: invalid storage class for function ?PetscBTView? >> /opt/petsc-3.4.2/include/petscbt.h:74: error: invalid storage class for function ?PetscBTCreate? >> /opt/petsc-3.4.2/include/petscbt.h:79: error: invalid storage class for function ?PetscBTLookupSet? >> /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTLookupSet?: >> /opt/petsc-3.4.2/include/petscbt.h:78: warning: declaration of ?index? shadows a global declaration >> /usr/include/string.h:489: warning: shadowed declaration is here >> /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: >> /opt/petsc-3.4.2/include/petscbt.h:91: error: invalid storage class for function ?PetscBTSet? >> /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTSet?: >> /opt/petsc-3.4.2/include/petscbt.h:90: warning: declaration of ?index? shadows a global declaration >> /usr/include/string.h:489: warning: shadowed declaration is here >> /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: >> /opt/petsc-3.4.2/include/petscbt.h:103: error: invalid storage class for function ?PetscBTLookupClear? >> /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTLookupClear?: >> /opt/petsc-3.4.2/include/petscbt.h:102: warning: declaration of ?index? shadows a global declaration >> /usr/include/string.h:489: warning: shadowed declaration is here >> /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: >> /opt/petsc-3.4.2/include/petscbt.h:115: error: invalid storage class for function ?PetscBTClear? >> /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTClear?: >> /opt/petsc-3.4.2/include/petscbt.h:114: warning: declaration of ?index? shadows a global declaration >> /usr/include/string.h:489: warning: shadowed declaration is here >> In file included from /opt/petsc-3.4.2/include/petsc.h:9, >> from Main/time_integration_dir/compute_tau_w_mean.c:91: >> /opt/petsc-3.4.2/include/petscctable.h: In function ?compute_tau_w_mean?: >> /opt/petsc-3.4.2/include/petscctable.h:20: error: invalid storage class for function ?PetscHash? >> /opt/petsc-3.4.2/include/petscctable.h:40: error: invalid storage class for function ?PetscTableAdd? >> In file included from /opt/petsc-3.4.2/include/petsc.h:9, >> from Main/time_integration_dir/compute_tau_w_mean.c:91: >> /opt/petsc-3.4.2/include/petscctable.h:83: error: invalid storage class for function ?PetscTableAddCount? >> /opt/petsc-3.4.2/include/petscctable.h:120: error: invalid storage class for function ?PetscTableFind? >> In file included from /opt/petsc-3.4.2/include/petscsnesfas.h:3, >> from /opt/petsc-3.4.2/include/petsc.h:27, >> from Main/time_integration_dir/compute_tau_w_mean.c:91: >> /opt/petsc-3.4.2/include/petscsnes.h:585: error: invalid storage class for function ?SNESGetSNESLineSearch? >> /opt/petsc-3.4.2/include/petscsnes.h:586: error: invalid storage class for function ?SNESSetSNESLineSearch? >> Main/time_integration_dir/compute_tau_w_mean.c:223: warning: implicit declaration of function ?wall_thermoquants? >> Main/time_integration_dir/compute_tau_w_mean.c:155: warning: unused variable ?iface_plus_one? >> Main/time_integration_dir/compute_tau_w_mean.c:330: warning: value computed is not used >> Main/time_integration_dir/compute_tau_w_mean.c:116: warning: unused variable ?ierr? >> Main/time_integration_dir/compute_tau_w_mean.c:111: warning: unused variable ?icell? >> Main/time_integration_dir/compute_tau_w_mean.c:105: warning: unused variable ?mach? >> /opt/petsc-3.4.2/include/petscts.h:328: warning: unused variable ?TSRegisterAllCalled? >> /opt/petsc-3.4.2/include/petscts.h:327: warning: unused variable ?TSList? >> /opt/petsc-3.4.2/include/petscts.h:168: warning: unused variable ?DMTS_CLASSID? >> /opt/petsc-3.4.2/include/petscts.h:167: warning: unused variable ?TS_CLASSID? >> /opt/petsc-3.4.2/include/petscts.h:163: warning: unused variable ?TSExactFinalTimeOptions? >> /opt/petsc-3.4.2/include/petscts.h:101: warning: unused variable ?TSConvergedReasons? >> /opt/petsc-3.4.2/include/petscts.h:80: warning: unused variable ?TSEquationTypes? >> /opt/petsc-3.4.2/include/petscsnesfas.h:19: warning: unused variable ?SNESFASTypes? >> /opt/petsc-3.4.2/include/petscsnes.h:679: warning: unused variable ?SNESQNRestartTypes? >> /opt/petsc-3.4.2/include/petscsnes.h:675: warning: unused variable ?SNESQNScaleTypes? >> /opt/petsc-3.4.2/include/petscsnes.h:667: warning: unused variable ?SNESNCGTypes? >> /opt/petsc-3.4.2/include/petscsnes.h:654: warning: unused variable ?SNESNGMRESSelectTypes? >> /opt/petsc-3.4.2/include/petscsnes.h:648: warning: unused variable ?SNESNGMRESRestartTypes? >> /opt/petsc-3.4.2/include/petscsnes.h:474: warning: unused variable ?SNESLineSearch_Apply? >> /opt/petsc-3.4.2/include/petscsnes.h:473: warning: unused variable ?SNESLineSearchList? >> /opt/petsc-3.4.2/include/petscsnes.h:472: warning: unused variable ?SNESLineSearchRegisterAllCalled? >> /opt/petsc-3.4.2/include/petscsnes.h:471: warning: unused variable ?SNESLINESEARCH_CLASSID? >> /opt/petsc-3.4.2/include/petscsnes.h:470: warning: unused variable ?SNESList? >> /opt/petsc-3.4.2/include/petscsnes.h:469: warning: unused variable ?SNESRegisterAllCalled? >> /opt/petsc-3.4.2/include/petscsnes.h:356: warning: unused variable ?SNESNormTypes? >> /opt/petsc-3.4.2/include/petscsnes.h:217: warning: unused variable ?SNESConvergedReasons? >> /opt/petsc-3.4.2/include/petscsnes.h:50: warning: unused variable ?DMSNES_CLASSID? >> /opt/petsc-3.4.2/include/petscsnes.h:49: warning: unused variable ?SNES_CLASSID? >> /opt/petsc-3.4.2/include/petsccharacteristic.h:44: warning: unused variable ?CharacteristicList? >> /opt/petsc-3.4.2/include/petsccharacteristic.h:43: warning: unused variable ?CharacteristicRegisterAllCalled? >> /opt/petsc-3.4.2/include/petscdmda.h:27: warning: unused variable ?DMDABoundaryTypes? >> /opt/petsc-3.4.2/include/petscpf.h:50: warning: unused variable ?PFRegisterAllCalled? >> /opt/petsc-3.4.2/include/petscpf.h:40: warning: unused variable ?PF_CLASSID? >> /opt/petsc-3.4.2/include/petscpf.h:12: warning: unused variable ?PFList? >> /opt/petsc-3.4.2/include/petscdm.h:34: warning: unused variable ?DMRegisterAllCalled? >> /opt/petsc-3.4.2/include/petscdm.h:33: warning: unused variable ?DMList? >> /opt/petsc-3.4.2/include/petscdm.h:11: warning: unused variable ?DM_CLASSID? >> /opt/petsc-3.4.2/include/petscdraw.h:239: warning: unused variable ?PETSC_DRAWHG_CLASSID? >> /opt/petsc-3.4.2/include/petscdraw.h:225: warning: unused variable ?PETSC_DRAWSP_CLASSID? >> /opt/petsc-3.4.2/include/petscdraw.h:206: warning: unused variable ?PETSC_DRAWLG_CLASSID? >> /opt/petsc-3.4.2/include/petscdraw.h:195: warning: unused variable ?PETSC_DRAWAXIS_CLASSID? >> /opt/petsc-3.4.2/include/petscdraw.h:26: warning: unused variable ?PetscDrawList? >> /opt/petsc-3.4.2/include/petscdraw.h:9: warning: unused variable ?PETSC_DRAW_CLASSID? >> /opt/petsc-3.4.2/include/petscmatlab.h:8: warning: unused variable ?MATLABENGINE_CLASSID? >> /opt/petsc-3.4.2/include/petscthreadcomm.h:35: warning: unused variable ?PetscThreadCommReductionOps? >> /opt/petsc-3.4.2/include/petscthreadcomm.h:32: warning: unused variable ?PetscThreadCommList? >> >> >> >> On Sep 3, 2013, at 9:03 PM, Satish Balay wrote: >> >>> Also you would have to call MPI_Init() before the first PetscInitialize() [and >>> similarly MPI_Finalize() after the last call to PetscFinalize()] >>> >>> Satish >>> >>> On Tue, 3 Sep 2013, Barry Smith wrote: >>> >>>> >>>> There is a problem with the 3.3 series when reinitializing PETSc. Can you switch to PETSc 3.4 which properly supports calling PetscInitialize() several times. >>>> >>>> Barry >>>> >>>> On Sep 3, 2013, at 1:58 PM, Suraj Prabhakaran wrote: >>>> >>>>> Hello, >>>>> >>>>> Again, this relates to finalizing and initializing petsc in my application. I basically want to finalize petsc and initialize it again with a different MPI communicator. A rough sketch of my quite huge application is as follows. >>>>> >>>>> >>>>> main() >>>>> { >>>>> PetscInitialize() >>>>> ... >>>>> ... >>>>> AOCreate().. >>>>> AO operations.. >>>>> AODestroy().. >>>>> ... >>>>> ... >>>>> PetscFinalize() >>>>> PETSC_COMM_WORLD = A new communicator (which includes more processes) >>>>> PetscInitialize() >>>>> ... >>>>> ... >>>>> AOCreate().. >>>>> AO operations.. >>>>> AODestroy() >>>>> ... >>>>> ... >>>>> PetscFinalize() >>>>> } >>>>> >>>>> So after finalizing and initializing it back, I always get an error with one of the AO calls. Given below is the complete error I get when running with 8 procs. Can anyone give me a hint what the problem could be? I was told earlier, that finalizing and initializing petsc should not be a problem! >>>>> >>>>> Best, >>>>> Suraj >>>>> >>>>> -------------------------ERROR starts here------------------------------- >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>> [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>>> [0]PETSC ERROR: is given. >>>>> [0]PETSC ERROR: [0] PetscStrcmp line 414 /root/petsc/petsc-3.3-p7/src/sys/utils/str.c >>>>> [0]PETSC ERROR: [0] PetscFListAdd line 202 /root/petsc/petsc-3.3-p7/src/sys/dll/reg.c >>>>> [0]PETSC ERROR: [0] AORegister line 99 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/aoreg.c >>>>> [0]PETSC ERROR: [0] AORegisterAll line 28 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/aoregall.c >>>>> [0]PETSC ERROR: [0] AOInitializePackage line 49 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/dlregisdm.c >>>>> [0]PETSC ERROR: [0] AOCreate line 530 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/ao.c >>>>> [0]PETSC ERROR: [0] AOCreateBasicIS line 381 /root/petsc/petsc-3.3-p7/src/dm/ao/impls/basic/aobasic.c >>>>> [0]PETSC ERROR: [0] AOCreateBasic line 336 /root/petsc/petsc-3.3-p7/src/dm/ao/impls/basic/aobasic.c >>>>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>> [0]PETSC ERROR: Signal received! >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>> [0]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>> [0]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>> [0]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>> -------------------------------------------------------------------------- >>>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>>>> with errorcode 59. >>>>> >>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>>> You may or may not see output from other processes, depending on >>>>> exactly when Open MPI kills them. >>>>> -------------------------------------------------------------------------- >>>>> [1]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [1]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>>> [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>> [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>> [1]PETSC ERROR: likely location of problem given in stack below >>>>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the function >>>>> [1]PETSC ERROR: is given. >>>>> [1]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>> [1]PETSC ERROR: Signal received! >>>>> [1]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [1]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>> [1]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>> [1]PETSC ERROR: See docs/index.html for manual pages. >>>>> [1]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [1]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>> [1]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>> [1]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>> [1]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>> [1]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>> [2]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [2]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>>> [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>> [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>> [2]PETSC ERROR: likely location of problem given in stack below >>>>> [2]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>> [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>> [2]PETSC ERROR: INSTEAD the line number of the start of the function >>>>> [2]PETSC ERROR: is given. >>>>> [2]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>> [2]PETSC ERROR: Signal received! >>>>> [2]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [2]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>> [2]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>> [2]PETSC ERROR: See docs/index.html for manual pages. >>>>> [2]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [2]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>> [2]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>> [2]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>> [2]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>> [2]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [2]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>> [3]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [3]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>>> [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>> [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>> [3]PETSC ERROR: likely location of problem given in stack below >>>>> [3]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>> [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>> [3]PETSC ERROR: INSTEAD the line number of the start of the function >>>>> [3]PETSC ERROR: is given. >>>>> [3]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>> [3]PETSC ERROR: Signal received! >>>>> [3]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [3]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>> [3]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>> [3]PETSC ERROR: See docs/index.html for manual pages. >>>>> [3]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [3]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>> [3]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>> [3]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>> [3]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>> [3]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [3]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>> [4]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [4]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>>> [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>> [4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>> [4]PETSC ERROR: likely location of problem given in stack below >>>>> [4]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>> [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>> [4]PETSC ERROR: INSTEAD the line number of the start of the function >>>>> [4]PETSC ERROR: is given. >>>>> [4]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>> [4]PETSC ERROR: Signal received! >>>>> [4]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [4]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>> [4]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> [4]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>> [4]PETSC ERROR: See docs/index.html for manual pages. >>>>> [4]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [4]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>> [4]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>> [4]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>> [4]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>> [4]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [4]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>> [5]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [5]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>>> [5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>> [5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[5]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>> [5]PETSC ERROR: likely location of problem given in stack below >>>>> [5]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>> [5]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>> [5]PETSC ERROR: INSTEAD the line number of the start of the function >>>>> [5]PETSC ERROR: is given. >>>>> [5]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>> [5]PETSC ERROR: Signal received! >>>>> [5]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [5]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>> [5]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> [5]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>> [5]PETSC ERROR: See docs/index.html for manual pages. >>>>> [5]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [5]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>> [5]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>> [5]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>> [5]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>> [5]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [5]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>> [6]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [6]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>>> [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>> [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>> [6]PETSC ERROR: likely location of problem given in stack below >>>>> [6]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>> [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>> [6]PETSC ERROR: INSTEAD the line number of the start of the function >>>>> [6]PETSC ERROR: is given. >>>>> [6]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>> [6]PETSC ERROR: Signal received! >>>>> [6]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [6]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>> [6]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> [6]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>> [6]PETSC ERROR: See docs/index.html for manual pages. >>>>> [6]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [6]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>> [6]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>> [6]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>> [6]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>> [6]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [6]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>> [7]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [7]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>>> [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>> [7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[7]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>> [7]PETSC ERROR: likely location of problem given in stack below >>>>> [7]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>> [7]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>> [7]PETSC ERROR: INSTEAD the line number of the start of the function >>>>> [7]PETSC ERROR: is given. >>>>> [7]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>> [7]PETSC ERROR: Signal received! >>>>> [7]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [7]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>> [7]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> [7]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>> [7]PETSC ERROR: See docs/index.html for manual pages. >>>>> [7]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [7]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>> [7]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>> [7]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>> [7]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>> [7]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [7]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>> -------------------------------------------------------------------------- >>>>> mpiexec has exited due to process rank 0 with PID 2670 on >>>>> node grsacc17 exiting improperly. There are two reasons this could occur: >>>>> >>>>> 1. this process did not call "init" before exiting, but others in >>>>> the job did. This can cause a job to hang indefinitely while it waits >>>>> for all processes to call "init". By rule, if one process calls "init", >>>>> then ALL processes must call "init" prior to termination. >>>>> >>>>> 2. this process called "init", but exited without calling "finalize". >>>>> By rule, all processes that call "init" MUST call "finalize" prior to >>>>> exiting or it will be considered an "abnormal termination" >>>>> >>>>> This may have caused other processes in the application to be >>>>> terminated by signals sent by mpiexec (as reported here). >>>> >>>> >>> >> > From waynezw0618 at 163.com Tue Sep 3 23:42:43 2013 From: waynezw0618 at 163.com (=?GBK?B?1cXOsA==?=) Date: Wed, 4 Sep 2013 12:42:43 +0800 (CST) Subject: [petsc-users] qep results different from polyeig in matlab Message-ID: <6f9ea372.16f73.140e747992c.Coremail.waynezw0618@163.com> Hello every one. I am Wei. I want to use slepc to solver my quadratic eigenvalue problem. and now I start with the tutorial ex16 and compare with polyeig function matlab. but the two give different results. here is the results from slepc Number of requested eigenvalues: 6 Stopping condition: tol= 1e-08 , maxit= 100 Number of converged approximate eigenpairs: 8 k ||(k^2M+Ck+K)x||/||kx|| -------------------- ------------------------- -7.85046229415e-16 + 2.81636155327 J, 3.31272023702e-09 -7.85046229415e-16 + -2.81636155327 J, 3.31272023702e-09 6.28036983474e-16 + 2.79837285903 J, 2.31471998545e-09 6.28036983474e-16 + -2.79837285903 J, 2.31471998545e-09 -5.4953236053e-16 + 2.78026777804 J, 6.1734628482e-10 -5.4953236053e-16 + -2.78026777804 J, 6.1734628482e-10 7.85046229433e-17 + 2.76882293959 J, 8.38075971231e-10 7.85046229433e-17 + -2.76882293959 J, 8.38075971231e-10 and here is the results from polyeig in Matlab. tol = 1.0000e-008 ee = 0 + 2.4425i 0 - 2.4425i 0 + 2.4218i 0 - 2.4218i 0 + 2.3876i 0 - 2.3876i 0 + 2.3405i 0 - 2.3405i 0 + 2.2815i 0 - 2.2815i 0 + 2.2117i 0 - 2.2117i 0 + 2.1324i 0 - 2.1324i 0 + 2.0456i 0 - 2.0456i 0 + 1.4262i 0 - 1.4262i 0 + 1.4612i 0 - 1.4612i 0 + 1.5164i 0 - 1.5164i 0 + 1.5881i 0 - 1.5881i 0 + 1.9533i 0 - 1.9533i 0 + 1.6717i 0 - 1.6717i 0 + 1.7631i 0 - 1.7631i 0 + 1.8581i 0 - 1.8581i it is different a lot especially for imagine part even if both converged to 1e-8. I tried both linear solver and qarnoldi method. could any one tell me what is that? is it error? --------------------------------------- Wei zhang PhD -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Wed Sep 4 02:04:22 2013 From: jroman at dsic.upv.es (Jose E. Roman) Date: Wed, 4 Sep 2013 09:04:22 +0200 Subject: [petsc-users] qep results different from polyeig in matlab In-Reply-To: <6f9ea372.16f73.140e747992c.Coremail.waynezw0618@163.com> References: <6f9ea372.16f73.140e747992c.Coremail.waynezw0618@163.com> Message-ID: <1C145592-AF00-4FC6-A987-F1A60E8AE21F@dsic.upv.es> El 04/09/2013, a las 06:42, ?? escribi?: > Hello > every one. I am Wei. I want to use slepc to solver my quadratic eigenvalue problem. and now I start with the tutorial ex16 and compare with polyeig function matlab. but the two give different results. > here is the results from slepc > > Number of requested eigenvalues: 6 > Stopping condition: tol= 1e-08 , maxit= 100 > Number of converged approximate eigenpairs: 8 > > k ||(k^2M+Ck+K)x||/||kx|| > -------------------- ------------------------- > -7.85046229415e-16 + 2.81636155327 J, 3.31272023702e-09 > -7.85046229415e-16 + -2.81636155327 J, 3.31272023702e-09 > 6.28036983474e-16 + 2.79837285903 J, 2.31471998545e-09 > 6.28036983474e-16 + -2.79837285903 J, 2.31471998545e-09 > -5.4953236053e-16 + 2.78026777804 J, 6.1734628482e-10 > -5.4953236053e-16 + -2.78026777804 J, 6.1734628482e-10 > 7.85046229433e-17 + 2.76882293959 J, 8.38075971231e-10 > 7.85046229433e-17 + -2.76882293959 J, 8.38075971231e-10 > > and here is the results from polyeig in Matlab. > tol = > > &nb sp;1.0000e-008 > > ee = > > 0 + 2.4425i > 0 - 2.4425i > 0 + 2.4218i > 0 - 2.4218i > 0 + 2.3876i > 0 - 2.3876i > 0 + 2.3405i > 0 - 2.3405i > 0 + 2.2815i > 0 - 2.2815i > 0 + 2.2117i > 0 - 2.2117i > 0&nb sp;+ 2.1324i > 0 - 2.1324i > 0 + 2.0456i > 0 - 2.0456i > 0 + 1.4262i > 0 - 1.4262i > 0 + 1.4612i > 0 - 1.4612i > 0 + 1.5164i > 0 - 1.5164i > 0 + 1.5881i > 0 - 1.5881i > 0 + 1.9533i > 0 - 1.9533i > & nbsp; 0 + 1.6717i > 0 - 1.6717i > 0 + 1.7631i > 0 - 1.7631i > 0 + 1.8581i > 0 - 1.8581i > > it is different a lot especially for imagine part even if both converged to 1e-8. I tried both linear solver and qarnoldi method. could any one tell me what is that? is it error? > > --------------------------------------- > > Wei zhang > > PhD > Please do not send a query to both petsc-users and slepc-maint, only to one of them. I have checked the output of ex16 and quadeig and they match. Probably you are not solving exactly the same problem with polyeig as with SLEPc. Use the option -mat_view ::ascii_matlab in ex16 and paste the matrices in Matlab. [Note this option is for petsc-3.4, it is different in previous versions.] Jose From suraj.prabhakaran at gmail.com Wed Sep 4 04:49:23 2013 From: suraj.prabhakaran at gmail.com (Suraj Prabhakaran) Date: Wed, 4 Sep 2013 11:49:23 +0200 Subject: [petsc-users] Re-initializing petsc again In-Reply-To: References: <315F25F7-7274-40E9-A244-5CD26F975CE5@gmail.com> <614D53AA-89C2-4C34-A192-5FA4BE0D1F27@mcs.anl.gov> Message-ID: <93A68DDC-8B46-49F3-A3CF-041FFA1DF2A9@gmail.com> Dear satish, The make test after installing petsc works fine. The compile command that produced the error is as follows. mpicc -fPIC -c -g -Wshadow -fomit-frame-pointer -Wall -DSPACE_DIM=2 -DNUMBER_EQN=4 -DNUMBER_SPECIES=1 -DNUMBER_TEMPERATURES=1 -DNUMBER_ENERGYMODES=1 -D_idealgas_ -DEOS_PERFECT_GAS -DHAVE_FORTRAN_UNDERSCORE -DPARALLEL -include /Network/Servers/grsacx01.grs.rwth-aachen.de/Volumes/home/grsprabh/resmgmt/applications/Malleability/Quadflow/quadflow/Library/flow_solver/Source/Include/grspetsc_c.h -DPARFB -DMPI_Get_address=MPI_Address -DMPI_Type_create_struct=MPI_Type_struct -I/opt/petsc-3.4.2/include -I/opt/open-mpi/1.6/include -I/opt/metis/include -I/opt/parametis/include -I/Network/Servers/grsacx01.grs.rwth-aachen.de/Volumes/home/grsprabh/resmgmt/applications/Malleability/Quadflow/quadflow/Library/blitz++ -I/opt/cgns/include -I/Network/Servers/grsacx01.grs.rwth-aachen.de/Volumes/home/grsprabh/resmgmt/applications/Malleability/Quadflow/quadflow/Program/Main -I/Network/Servers/grsacx01.grs.rwth-aachen.de/Volumes/home/grsprabh/resmgmt/applications/Malleability/Quadflow/quadflow/Library/mstlib -I/Network/Servers/grsacx01.grs.rwth-aachen.de/Volumes/home/grsprabh/resmgmt/applications/Malleability/Quadflow/quadflow/Library/mstlib/EssIGPMlib -I/Network/Servers/grsacx01.grs.rwth-aachen.de/Volumes/home/grsprabh/resmgmt/applications/Malleability/Quadflow/quadflow/Library/igpm_t_lib -I/Network/Servers/grsacx01.grs.rwth-aachen.de/Volumes/home/grsprabh/resmgmt/applications/Malleability/Quadflow/quadflow/Library/fluid_t_lib/xrms/include -I/Network/Servers/grsacx01.grs.rwth-aachen.de/Volumes/home/grsprabh/resmgmt/applications/Malleability/Quadflow/quadflow/Library/qigpmesh -I/Network/Servers/grsacx01.grs.rwth-aachen.de/Volumes/home/grsprabh/resmgmt/applications/Malleability/Quadflow/quadflow/Library/parallel -I/Network/Servers/grsacx01.grs.rwth-aachen.de/Volumes/home/grsprabh/resmgmt/applications/Malleability/Quadflow/quadflow/Library/flow_solver/Source/Include -I/Network/Servers/grsacx01.grs.rwth-aachen.de/Volumes/home/grsprabh/resmgmt/applications/Malleability/Quadflow/quadflow/Library/flow_solver/Source/Main/grid_generation_dir/Flower_Converter -I/Network/Servers/grsacx01.grs.rwth-aachen.de/Volumes/home/grsprabh/resmgmt/applications/Malleability/Quadflow/quadflow/Library/flow_solver/Source/Main/grid_generation_dir/Tau_Converter -I/Network/Servers/grsacx01.grs.rwth-aachen.de/Volumes/home/grsprabh/resmgmt/applications/Malleability/Quadflow/quadflow/Library/flow_solver/Source/Main/reconstruction_dir -I/Network/Servers/grsacx01.grs.rwth-aachen.de/Volumes/home/grsprabh/resmgmt/applications/Malleability/Quadflow/quadflow/Library/flow_solver/Source/Main/implicit_dir -I/Network/Servers/grsacx01.grs.rwth-aachen.de/Volumes/home/grsprabh/resmgmt/applications/Malleability/Quadflow/quadflow/Library/flow_solver/Source/Main/implicit_dir/matvec_2ndorder -I/Network/Servers/grsacx01.grs.rwth-aachen.de/Volumes/home/grsprabh/resmgmt/applications/Malleability/Quadflow/quadflow/Library/visual -I/Network/Servers/grsacx01.grs.rwth-aachen.de/Volumes/home/grsprabh/resmgmt/applications/Malleability/Quadflow/quadflow/Library/eos -I/Network/Servers/grsacx01.grs.rwth-aachen.de/Volumes/home/grsprabh/resmgmt/applications/Malleability/Quadflow/quadflow/Library/flow_solver/Source/Main/ANN_kd_tree Main/time_integration_dir/compute_tau_w_mean.c -o /Network/Servers/grsacx01.grs.rwth-aachen.de/Volumes/home/grsprabh/resmgmt/applications/Malleability/Quadflow/quadflow/install/tmp/Library/flow_solver/compute_tau_w_mean.o Best, Suraj On Sep 4, 2013, at 5:14 AM, Satish Balay wrote: > This is incomplete info. > > what compile command produced this error? And what do you get when you > compile a petsc example with a petsc makefile? [complete output] > > Satish > > On Wed, 4 Sep 2013, Suraj Prabhakaran wrote: > >> Thanks a lot for the confirmation!! >> I am now compiling it with 3.4.2, but I get a strange error and could be great if someone can just point out whats wrong. (Might be something stupid, but just in case...) >> >> Thanks in advance! >> >> Suraj >> >> -------------------------------ERROR----------------------------- >> >> In file included from /opt/petsc-3.4.2/include/petsc.h:8, >> from Main/time_integration_dir/compute_tau_w_mean.c:91: >> /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: >> /opt/petsc-3.4.2/include/petscbt.h:33: error: invalid storage class for function ?PetscBTLength? >> /opt/petsc-3.4.2/include/petscbt.h:38: error: invalid storage class for function ?PetscBTMemzero? >> /opt/petsc-3.4.2/include/petscbt.h:43: error: invalid storage class for function ?PetscBTDestroy? >> /opt/petsc-3.4.2/include/petscbt.h:48: error: invalid storage class for function ?PetscBTLookup? >> /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTLookup?: >> /opt/petsc-3.4.2/include/petscbt.h:47: warning: declaration of ?index? shadows a global declaration >> /usr/include/string.h:489: warning: shadowed declaration is here >> /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: >> /opt/petsc-3.4.2/include/petscbt.h:59: error: invalid storage class for function ?PetscBTView? >> /opt/petsc-3.4.2/include/petscbt.h:74: error: invalid storage class for function ?PetscBTCreate? >> /opt/petsc-3.4.2/include/petscbt.h:79: error: invalid storage class for function ?PetscBTLookupSet? >> /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTLookupSet?: >> /opt/petsc-3.4.2/include/petscbt.h:78: warning: declaration of ?index? shadows a global declaration >> /usr/include/string.h:489: warning: shadowed declaration is here >> /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: >> /opt/petsc-3.4.2/include/petscbt.h:91: error: invalid storage class for function ?PetscBTSet? >> /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTSet?: >> /opt/petsc-3.4.2/include/petscbt.h:90: warning: declaration of ?index? shadows a global declaration >> /usr/include/string.h:489: warning: shadowed declaration is here >> /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: >> /opt/petsc-3.4.2/include/petscbt.h:103: error: invalid storage class for function ?PetscBTLookupClear? >> /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTLookupClear?: >> /opt/petsc-3.4.2/include/petscbt.h:102: warning: declaration of ?index? shadows a global declaration >> /usr/include/string.h:489: warning: shadowed declaration is here >> /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: >> /opt/petsc-3.4.2/include/petscbt.h:115: error: invalid storage class for function ?PetscBTClear? >> /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTClear?: >> /opt/petsc-3.4.2/include/petscbt.h:114: warning: declaration of ?index? shadows a global declaration >> /usr/include/string.h:489: warning: shadowed declaration is here >> In file included from /opt/petsc-3.4.2/include/petsc.h:9, >> from Main/time_integration_dir/compute_tau_w_mean.c:91: >> /opt/petsc-3.4.2/include/petscctable.h: In function ?compute_tau_w_mean?: >> /opt/petsc-3.4.2/include/petscctable.h:20: error: invalid storage class for function ?PetscHash? >> /opt/petsc-3.4.2/include/petscctable.h:40: error: invalid storage class for function ?PetscTableAdd? >> In file included from /opt/petsc-3.4.2/include/petsc.h:9, >> from Main/time_integration_dir/compute_tau_w_mean.c:91: >> /opt/petsc-3.4.2/include/petscctable.h:83: error: invalid storage class for function ?PetscTableAddCount? >> /opt/petsc-3.4.2/include/petscctable.h:120: error: invalid storage class for function ?PetscTableFind? >> In file included from /opt/petsc-3.4.2/include/petscsnesfas.h:3, >> from /opt/petsc-3.4.2/include/petsc.h:27, >> from Main/time_integration_dir/compute_tau_w_mean.c:91: >> /opt/petsc-3.4.2/include/petscsnes.h:585: error: invalid storage class for function ?SNESGetSNESLineSearch? >> /opt/petsc-3.4.2/include/petscsnes.h:586: error: invalid storage class for function ?SNESSetSNESLineSearch? >> Main/time_integration_dir/compute_tau_w_mean.c:223: warning: implicit declaration of function ?wall_thermoquants? >> Main/time_integration_dir/compute_tau_w_mean.c:155: warning: unused variable ?iface_plus_one? >> Main/time_integration_dir/compute_tau_w_mean.c:330: warning: value computed is not used >> Main/time_integration_dir/compute_tau_w_mean.c:116: warning: unused variable ?ierr? >> Main/time_integration_dir/compute_tau_w_mean.c:111: warning: unused variable ?icell? >> Main/time_integration_dir/compute_tau_w_mean.c:105: warning: unused variable ?mach? >> /opt/petsc-3.4.2/include/petscts.h:328: warning: unused variable ?TSRegisterAllCalled? >> /opt/petsc-3.4.2/include/petscts.h:327: warning: unused variable ?TSList? >> /opt/petsc-3.4.2/include/petscts.h:168: warning: unused variable ?DMTS_CLASSID? >> /opt/petsc-3.4.2/include/petscts.h:167: warning: unused variable ?TS_CLASSID? >> /opt/petsc-3.4.2/include/petscts.h:163: warning: unused variable ?TSExactFinalTimeOptions? >> /opt/petsc-3.4.2/include/petscts.h:101: warning: unused variable ?TSConvergedReasons? >> /opt/petsc-3.4.2/include/petscts.h:80: warning: unused variable ?TSEquationTypes? >> /opt/petsc-3.4.2/include/petscsnesfas.h:19: warning: unused variable ?SNESFASTypes? >> /opt/petsc-3.4.2/include/petscsnes.h:679: warning: unused variable ?SNESQNRestartTypes? >> /opt/petsc-3.4.2/include/petscsnes.h:675: warning: unused variable ?SNESQNScaleTypes? >> /opt/petsc-3.4.2/include/petscsnes.h:667: warning: unused variable ?SNESNCGTypes? >> /opt/petsc-3.4.2/include/petscsnes.h:654: warning: unused variable ?SNESNGMRESSelectTypes? >> /opt/petsc-3.4.2/include/petscsnes.h:648: warning: unused variable ?SNESNGMRESRestartTypes? >> /opt/petsc-3.4.2/include/petscsnes.h:474: warning: unused variable ?SNESLineSearch_Apply? >> /opt/petsc-3.4.2/include/petscsnes.h:473: warning: unused variable ?SNESLineSearchList? >> /opt/petsc-3.4.2/include/petscsnes.h:472: warning: unused variable ?SNESLineSearchRegisterAllCalled? >> /opt/petsc-3.4.2/include/petscsnes.h:471: warning: unused variable ?SNESLINESEARCH_CLASSID? >> /opt/petsc-3.4.2/include/petscsnes.h:470: warning: unused variable ?SNESList? >> /opt/petsc-3.4.2/include/petscsnes.h:469: warning: unused variable ?SNESRegisterAllCalled? >> /opt/petsc-3.4.2/include/petscsnes.h:356: warning: unused variable ?SNESNormTypes? >> /opt/petsc-3.4.2/include/petscsnes.h:217: warning: unused variable ?SNESConvergedReasons? >> /opt/petsc-3.4.2/include/petscsnes.h:50: warning: unused variable ?DMSNES_CLASSID? >> /opt/petsc-3.4.2/include/petscsnes.h:49: warning: unused variable ?SNES_CLASSID? >> /opt/petsc-3.4.2/include/petsccharacteristic.h:44: warning: unused variable ?CharacteristicList? >> /opt/petsc-3.4.2/include/petsccharacteristic.h:43: warning: unused variable ?CharacteristicRegisterAllCalled? >> /opt/petsc-3.4.2/include/petscdmda.h:27: warning: unused variable ?DMDABoundaryTypes? >> /opt/petsc-3.4.2/include/petscpf.h:50: warning: unused variable ?PFRegisterAllCalled? >> /opt/petsc-3.4.2/include/petscpf.h:40: warning: unused variable ?PF_CLASSID? >> /opt/petsc-3.4.2/include/petscpf.h:12: warning: unused variable ?PFList? >> /opt/petsc-3.4.2/include/petscdm.h:34: warning: unused variable ?DMRegisterAllCalled? >> /opt/petsc-3.4.2/include/petscdm.h:33: warning: unused variable ?DMList? >> /opt/petsc-3.4.2/include/petscdm.h:11: warning: unused variable ?DM_CLASSID? >> /opt/petsc-3.4.2/include/petscdraw.h:239: warning: unused variable ?PETSC_DRAWHG_CLASSID? >> /opt/petsc-3.4.2/include/petscdraw.h:225: warning: unused variable ?PETSC_DRAWSP_CLASSID? >> /opt/petsc-3.4.2/include/petscdraw.h:206: warning: unused variable ?PETSC_DRAWLG_CLASSID? >> /opt/petsc-3.4.2/include/petscdraw.h:195: warning: unused variable ?PETSC_DRAWAXIS_CLASSID? >> /opt/petsc-3.4.2/include/petscdraw.h:26: warning: unused variable ?PetscDrawList? >> /opt/petsc-3.4.2/include/petscdraw.h:9: warning: unused variable ?PETSC_DRAW_CLASSID? >> /opt/petsc-3.4.2/include/petscmatlab.h:8: warning: unused variable ?MATLABENGINE_CLASSID? >> /opt/petsc-3.4.2/include/petscthreadcomm.h:35: warning: unused variable ?PetscThreadCommReductionOps? >> /opt/petsc-3.4.2/include/petscthreadcomm.h:32: warning: unused variable ?PetscThreadCommList? >> >> >> >> On Sep 3, 2013, at 9:03 PM, Satish Balay wrote: >> >>> Also you would have to call MPI_Init() before the first PetscInitialize() [and >>> similarly MPI_Finalize() after the last call to PetscFinalize()] >>> >>> Satish >>> >>> On Tue, 3 Sep 2013, Barry Smith wrote: >>> >>>> >>>> There is a problem with the 3.3 series when reinitializing PETSc. Can you switch to PETSc 3.4 which properly supports calling PetscInitialize() several times. >>>> >>>> Barry >>>> >>>> On Sep 3, 2013, at 1:58 PM, Suraj Prabhakaran wrote: >>>> >>>>> Hello, >>>>> >>>>> Again, this relates to finalizing and initializing petsc in my application. I basically want to finalize petsc and initialize it again with a different MPI communicator. A rough sketch of my quite huge application is as follows. >>>>> >>>>> >>>>> main() >>>>> { >>>>> PetscInitialize() >>>>> ... >>>>> ... >>>>> AOCreate().. >>>>> AO operations.. >>>>> AODestroy().. >>>>> ... >>>>> ... >>>>> PetscFinalize() >>>>> PETSC_COMM_WORLD = A new communicator (which includes more processes) >>>>> PetscInitialize() >>>>> ... >>>>> ... >>>>> AOCreate().. >>>>> AO operations.. >>>>> AODestroy() >>>>> ... >>>>> ... >>>>> PetscFinalize() >>>>> } >>>>> >>>>> So after finalizing and initializing it back, I always get an error with one of the AO calls. Given below is the complete error I get when running with 8 procs. Can anyone give me a hint what the problem could be? I was told earlier, that finalizing and initializing petsc should not be a problem! >>>>> >>>>> Best, >>>>> Suraj >>>>> >>>>> -------------------------ERROR starts here------------------------------- >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>> [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>>> [0]PETSC ERROR: is given. >>>>> [0]PETSC ERROR: [0] PetscStrcmp line 414 /root/petsc/petsc-3.3-p7/src/sys/utils/str.c >>>>> [0]PETSC ERROR: [0] PetscFListAdd line 202 /root/petsc/petsc-3.3-p7/src/sys/dll/reg.c >>>>> [0]PETSC ERROR: [0] AORegister line 99 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/aoreg.c >>>>> [0]PETSC ERROR: [0] AORegisterAll line 28 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/aoregall.c >>>>> [0]PETSC ERROR: [0] AOInitializePackage line 49 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/dlregisdm.c >>>>> [0]PETSC ERROR: [0] AOCreate line 530 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/ao.c >>>>> [0]PETSC ERROR: [0] AOCreateBasicIS line 381 /root/petsc/petsc-3.3-p7/src/dm/ao/impls/basic/aobasic.c >>>>> [0]PETSC ERROR: [0] AOCreateBasic line 336 /root/petsc/petsc-3.3-p7/src/dm/ao/impls/basic/aobasic.c >>>>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>> [0]PETSC ERROR: Signal received! >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>> [0]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>> [0]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>> [0]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>> -------------------------------------------------------------------------- >>>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>>>> with errorcode 59. >>>>> >>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>>> You may or may not see output from other processes, depending on >>>>> exactly when Open MPI kills them. >>>>> -------------------------------------------------------------------------- >>>>> [1]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [1]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>>> [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>> [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>> [1]PETSC ERROR: likely location of problem given in stack below >>>>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the function >>>>> [1]PETSC ERROR: is given. >>>>> [1]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>> [1]PETSC ERROR: Signal received! >>>>> [1]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [1]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>> [1]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>> [1]PETSC ERROR: See docs/index.html for manual pages. >>>>> [1]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [1]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>> [1]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>> [1]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>> [1]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>> [1]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>> [2]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [2]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>>> [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>> [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>> [2]PETSC ERROR: likely location of problem given in stack below >>>>> [2]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>> [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>> [2]PETSC ERROR: INSTEAD the line number of the start of the function >>>>> [2]PETSC ERROR: is given. >>>>> [2]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>> [2]PETSC ERROR: Signal received! >>>>> [2]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [2]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>> [2]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>> [2]PETSC ERROR: See docs/index.html for manual pages. >>>>> [2]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [2]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>> [2]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>> [2]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>> [2]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>> [2]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [2]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>> [3]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [3]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>>> [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>> [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>> [3]PETSC ERROR: likely location of problem given in stack below >>>>> [3]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>> [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>> [3]PETSC ERROR: INSTEAD the line number of the start of the function >>>>> [3]PETSC ERROR: is given. >>>>> [3]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>> [3]PETSC ERROR: Signal received! >>>>> [3]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [3]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>> [3]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>> [3]PETSC ERROR: See docs/index.html for manual pages. >>>>> [3]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [3]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>> [3]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>> [3]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>> [3]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>> [3]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [3]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>> [4]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [4]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>>> [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>> [4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>> [4]PETSC ERROR: likely location of problem given in stack below >>>>> [4]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>> [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>> [4]PETSC ERROR: INSTEAD the line number of the start of the function >>>>> [4]PETSC ERROR: is given. >>>>> [4]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>> [4]PETSC ERROR: Signal received! >>>>> [4]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [4]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>> [4]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> [4]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>> [4]PETSC ERROR: See docs/index.html for manual pages. >>>>> [4]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [4]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>> [4]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>> [4]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>> [4]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>> [4]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [4]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>> [5]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [5]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>>> [5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>> [5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[5]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>> [5]PETSC ERROR: likely location of problem given in stack below >>>>> [5]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>> [5]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>> [5]PETSC ERROR: INSTEAD the line number of the start of the function >>>>> [5]PETSC ERROR: is given. >>>>> [5]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>> [5]PETSC ERROR: Signal received! >>>>> [5]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [5]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>> [5]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> [5]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>> [5]PETSC ERROR: See docs/index.html for manual pages. >>>>> [5]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [5]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>> [5]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>> [5]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>> [5]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>> [5]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [5]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>> [6]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [6]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>>> [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>> [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>> [6]PETSC ERROR: likely location of problem given in stack below >>>>> [6]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>> [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>> [6]PETSC ERROR: INSTEAD the line number of the start of the function >>>>> [6]PETSC ERROR: is given. >>>>> [6]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>> [6]PETSC ERROR: Signal received! >>>>> [6]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [6]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>> [6]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> [6]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>> [6]PETSC ERROR: See docs/index.html for manual pages. >>>>> [6]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [6]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>> [6]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>> [6]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>> [6]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>> [6]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [6]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>> [7]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [7]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>>> [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>> [7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[7]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>> [7]PETSC ERROR: likely location of problem given in stack below >>>>> [7]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>> [7]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>> [7]PETSC ERROR: INSTEAD the line number of the start of the function >>>>> [7]PETSC ERROR: is given. >>>>> [7]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>> [7]PETSC ERROR: Signal received! >>>>> [7]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [7]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>> [7]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> [7]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>> [7]PETSC ERROR: See docs/index.html for manual pages. >>>>> [7]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [7]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>> [7]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>> [7]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>> [7]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>> [7]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [7]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>> -------------------------------------------------------------------------- >>>>> mpiexec has exited due to process rank 0 with PID 2670 on >>>>> node grsacc17 exiting improperly. There are two reasons this could occur: >>>>> >>>>> 1. this process did not call "init" before exiting, but others in >>>>> the job did. This can cause a job to hang indefinitely while it waits >>>>> for all processes to call "init". By rule, if one process calls "init", >>>>> then ALL processes must call "init" prior to termination. >>>>> >>>>> 2. this process called "init", but exited without calling "finalize". >>>>> By rule, all processes that call "init" MUST call "finalize" prior to >>>>> exiting or it will be considered an "abnormal termination" >>>>> >>>>> This may have caused other processes in the application to be >>>>> terminated by signals sent by mpiexec (as reported here). >>>> >>>> >>> >> >> From bsmith at mcs.anl.gov Wed Sep 4 09:24:48 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 4 Sep 2013 09:24:48 -0500 Subject: [petsc-users] Re-initializing petsc again In-Reply-To: <7C516550-089B-4ACF-AE39-3DD4BC38EA41@gmail.com> References: <315F25F7-7274-40E9-A244-5CD26F975CE5@gmail.com> <614D53AA-89C2-4C34-A192-5FA4BE0D1F27@mcs.anl.gov> <135162E5-14F4-49AF-831F-A5864CAC3A8C@mcs.anl.gov> <7C516550-089B-4ACF-AE39-3DD4BC38EA41@gmail.com> Message-ID: <00164CC3-C3F6-4361-8055-4FB9C3F76855@mcs.anl.gov> >> In file included from /opt/petsc-3.4.2/include/petsc.h:8, >> from Main/time_integration_dir/compute_tau_w_mean.c:91: >> /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: >> /opt/petsc-3.4.2/include/petscbt.h:33: error: invalid storage class for function ?PetscBTLength? >> /opt/petsc-3.4.2/include/petscbt.h:38: error: invalid storage class for function ?PetscBTMemzero? >> /opt/petsc-3.4.2/include/petscbt.h:43: error: invalid storage class for function ?PetscBTDestroy? Ok, nothing wrong with the static inline or build; There must be something wonky in compute_tau_w_mean.c or a file it includes that is causing the error. I would start debugging by removing stuff from compute_tau_w_mean.c before the #include "petsc.h" until the error goes away and that will show the culprit. Barry On Sep 3, 2013, at 10:21 PM, Suraj Prabhakaran wrote: > Hi Barry, > > Thanks a lot! Here are the files attached. > > Suraj > > > > On Sep 4, 2013, at 5:13 AM, Barry Smith wrote: > >> >> Something is going wrong with the PETSC_STATIC_INLINE definition. Please send configure.log and make.log >> >> Barry >> >> On Sep 3, 2013, at 10:01 PM, Suraj Prabhakaran wrote: >> >>> Thanks a lot for the confirmation!! >>> I am now compiling it with 3.4.2, but I get a strange error and could be great if someone can just point out whats wrong. (Might be something stupid, but just in case...) >>> >>> Thanks in advance! >>> >>> Suraj >>> >>> -------------------------------ERROR----------------------------- >>> >>> In file included from /opt/petsc-3.4.2/include/petsc.h:8, >>> from Main/time_integration_dir/compute_tau_w_mean.c:91: >>> /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: >>> /opt/petsc-3.4.2/include/petscbt.h:33: error: invalid storage class for function ?PetscBTLength? >>> /opt/petsc-3.4.2/include/petscbt.h:38: error: invalid storage class for function ?PetscBTMemzero? >>> /opt/petsc-3.4.2/include/petscbt.h:43: error: invalid storage class for function ?PetscBTDestroy? >>> /opt/petsc-3.4.2/include/petscbt.h:48: error: invalid storage class for function ?PetscBTLookup? >>> /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTLookup?: >>> /opt/petsc-3.4.2/include/petscbt.h:47: warning: declaration of ?index? shadows a global declaration >>> /usr/include/string.h:489: warning: shadowed declaration is here >>> /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: >>> /opt/petsc-3.4.2/include/petscbt.h:59: error: invalid storage class for function ?PetscBTView? >>> /opt/petsc-3.4.2/include/petscbt.h:74: error: invalid storage class for function ?PetscBTCreate? >>> /opt/petsc-3.4.2/include/petscbt.h:79: error: invalid storage class for function ?PetscBTLookupSet? >>> /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTLookupSet?: >>> /opt/petsc-3.4.2/include/petscbt.h:78: warning: declaration of ?index? shadows a global declaration >>> /usr/include/string.h:489: warning: shadowed declaration is here >>> /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: >>> /opt/petsc-3.4.2/include/petscbt.h:91: error: invalid storage class for function ?PetscBTSet? >>> /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTSet?: >>> /opt/petsc-3.4.2/include/petscbt.h:90: warning: declaration of ?index? shadows a global declaration >>> /usr/include/string.h:489: warning: shadowed declaration is here >>> /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: >>> /opt/petsc-3.4.2/include/petscbt.h:103: error: invalid storage class for function ?PetscBTLookupClear? >>> /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTLookupClear?: >>> /opt/petsc-3.4.2/include/petscbt.h:102: warning: declaration of ?index? shadows a global declaration >>> /usr/include/string.h:489: warning: shadowed declaration is here >>> /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: >>> /opt/petsc-3.4.2/include/petscbt.h:115: error: invalid storage class for function ?PetscBTClear? >>> /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTClear?: >>> /opt/petsc-3.4.2/include/petscbt.h:114: warning: declaration of ?index? shadows a global declaration >>> /usr/include/string.h:489: warning: shadowed declaration is here >>> In file included from /opt/petsc-3.4.2/include/petsc.h:9, >>> from Main/time_integration_dir/compute_tau_w_mean.c:91: >>> /opt/petsc-3.4.2/include/petscctable.h: In function ?compute_tau_w_mean?: >>> /opt/petsc-3.4.2/include/petscctable.h:20: error: invalid storage class for function ?PetscHash? >>> /opt/petsc-3.4.2/include/petscctable.h:40: error: invalid storage class for function ?PetscTableAdd? >>> In file included from /opt/petsc-3.4.2/include/petsc.h:9, >>> from Main/time_integration_dir/compute_tau_w_mean.c:91: >>> /opt/petsc-3.4.2/include/petscctable.h:83: error: invalid storage class for function ?PetscTableAddCount? >>> /opt/petsc-3.4.2/include/petscctable.h:120: error: invalid storage class for function ?PetscTableFind? >>> In file included from /opt/petsc-3.4.2/include/petscsnesfas.h:3, >>> from /opt/petsc-3.4.2/include/petsc.h:27, >>> from Main/time_integration_dir/compute_tau_w_mean.c:91: >>> /opt/petsc-3.4.2/include/petscsnes.h:585: error: invalid storage class for function ?SNESGetSNESLineSearch? >>> /opt/petsc-3.4.2/include/petscsnes.h:586: error: invalid storage class for function ?SNESSetSNESLineSearch? >>> Main/time_integration_dir/compute_tau_w_mean.c:223: warning: implicit declaration of function ?wall_thermoquants? >>> Main/time_integration_dir/compute_tau_w_mean.c:155: warning: unused variable ?iface_plus_one? >>> Main/time_integration_dir/compute_tau_w_mean.c:330: warning: value computed is not used >>> Main/time_integration_dir/compute_tau_w_mean.c:116: warning: unused variable ?ierr? >>> Main/time_integration_dir/compute_tau_w_mean.c:111: warning: unused variable ?icell? >>> Main/time_integration_dir/compute_tau_w_mean.c:105: warning: unused variable ?mach? >>> /opt/petsc-3.4.2/include/petscts.h:328: warning: unused variable ?TSRegisterAllCalled? >>> /opt/petsc-3.4.2/include/petscts.h:327: warning: unused variable ?TSList? >>> /opt/petsc-3.4.2/include/petscts.h:168: warning: unused variable ?DMTS_CLASSID? >>> /opt/petsc-3.4.2/include/petscts.h:167: warning: unused variable ?TS_CLASSID? >>> /opt/petsc-3.4.2/include/petscts.h:163: warning: unused variable ?TSExactFinalTimeOptions? >>> /opt/petsc-3.4.2/include/petscts.h:101: warning: unused variable ?TSConvergedReasons? >>> /opt/petsc-3.4.2/include/petscts.h:80: warning: unused variable ?TSEquationTypes? >>> /opt/petsc-3.4.2/include/petscsnesfas.h:19: warning: unused variable ?SNESFASTypes? >>> /opt/petsc-3.4.2/include/petscsnes.h:679: warning: unused variable ?SNESQNRestartTypes? >>> /opt/petsc-3.4.2/include/petscsnes.h:675: warning: unused variable ?SNESQNScaleTypes? >>> /opt/petsc-3.4.2/include/petscsnes.h:667: warning: unused variable ?SNESNCGTypes? >>> /opt/petsc-3.4.2/include/petscsnes.h:654: warning: unused variable ?SNESNGMRESSelectTypes? >>> /opt/petsc-3.4.2/include/petscsnes.h:648: warning: unused variable ?SNESNGMRESRestartTypes? >>> /opt/petsc-3.4.2/include/petscsnes.h:474: warning: unused variable ?SNESLineSearch_Apply? >>> /opt/petsc-3.4.2/include/petscsnes.h:473: warning: unused variable ?SNESLineSearchList? >>> /opt/petsc-3.4.2/include/petscsnes.h:472: warning: unused variable ?SNESLineSearchRegisterAllCalled? >>> /opt/petsc-3.4.2/include/petscsnes.h:471: warning: unused variable ?SNESLINESEARCH_CLASSID? >>> /opt/petsc-3.4.2/include/petscsnes.h:470: warning: unused variable ?SNESList? >>> /opt/petsc-3.4.2/include/petscsnes.h:469: warning: unused variable ?SNESRegisterAllCalled? >>> /opt/petsc-3.4.2/include/petscsnes.h:356: warning: unused variable ?SNESNormTypes? >>> /opt/petsc-3.4.2/include/petscsnes.h:217: warning: unused variable ?SNESConvergedReasons? >>> /opt/petsc-3.4.2/include/petscsnes.h:50: warning: unused variable ?DMSNES_CLASSID? >>> /opt/petsc-3.4.2/include/petscsnes.h:49: warning: unused variable ?SNES_CLASSID? >>> /opt/petsc-3.4.2/include/petsccharacteristic.h:44: warning: unused variable ?CharacteristicList? >>> /opt/petsc-3.4.2/include/petsccharacteristic.h:43: warning: unused variable ?CharacteristicRegisterAllCalled? >>> /opt/petsc-3.4.2/include/petscdmda.h:27: warning: unused variable ?DMDABoundaryTypes? >>> /opt/petsc-3.4.2/include/petscpf.h:50: warning: unused variable ?PFRegisterAllCalled? >>> /opt/petsc-3.4.2/include/petscpf.h:40: warning: unused variable ?PF_CLASSID? >>> /opt/petsc-3.4.2/include/petscpf.h:12: warning: unused variable ?PFList? >>> /opt/petsc-3.4.2/include/petscdm.h:34: warning: unused variable ?DMRegisterAllCalled? >>> /opt/petsc-3.4.2/include/petscdm.h:33: warning: unused variable ?DMList? >>> /opt/petsc-3.4.2/include/petscdm.h:11: warning: unused variable ?DM_CLASSID? >>> /opt/petsc-3.4.2/include/petscdraw.h:239: warning: unused variable ?PETSC_DRAWHG_CLASSID? >>> /opt/petsc-3.4.2/include/petscdraw.h:225: warning: unused variable ?PETSC_DRAWSP_CLASSID? >>> /opt/petsc-3.4.2/include/petscdraw.h:206: warning: unused variable ?PETSC_DRAWLG_CLASSID? >>> /opt/petsc-3.4.2/include/petscdraw.h:195: warning: unused variable ?PETSC_DRAWAXIS_CLASSID? >>> /opt/petsc-3.4.2/include/petscdraw.h:26: warning: unused variable ?PetscDrawList? >>> /opt/petsc-3.4.2/include/petscdraw.h:9: warning: unused variable ?PETSC_DRAW_CLASSID? >>> /opt/petsc-3.4.2/include/petscmatlab.h:8: warning: unused variable ?MATLABENGINE_CLASSID? >>> /opt/petsc-3.4.2/include/petscthreadcomm.h:35: warning: unused variable ?PetscThreadCommReductionOps? >>> /opt/petsc-3.4.2/include/petscthreadcomm.h:32: warning: unused variable ?PetscThreadCommList? >>> >>> >>> >>> On Sep 3, 2013, at 9:03 PM, Satish Balay wrote: >>> >>>> Also you would have to call MPI_Init() before the first PetscInitialize() [and >>>> similarly MPI_Finalize() after the last call to PetscFinalize()] >>>> >>>> Satish >>>> >>>> On Tue, 3 Sep 2013, Barry Smith wrote: >>>> >>>>> >>>>> There is a problem with the 3.3 series when reinitializing PETSc. Can you switch to PETSc 3.4 which properly supports calling PetscInitialize() several times. >>>>> >>>>> Barry >>>>> >>>>> On Sep 3, 2013, at 1:58 PM, Suraj Prabhakaran wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> Again, this relates to finalizing and initializing petsc in my application. I basically want to finalize petsc and initialize it again with a different MPI communicator. A rough sketch of my quite huge application is as follows. >>>>>> >>>>>> >>>>>> main() >>>>>> { >>>>>> PetscInitialize() >>>>>> ... >>>>>> ... >>>>>> AOCreate().. >>>>>> AO operations.. >>>>>> AODestroy().. >>>>>> ... >>>>>> ... >>>>>> PetscFinalize() >>>>>> PETSC_COMM_WORLD = A new communicator (which includes more processes) >>>>>> PetscInitialize() >>>>>> ... >>>>>> ... >>>>>> AOCreate().. >>>>>> AO operations.. >>>>>> AODestroy() >>>>>> ... >>>>>> ... >>>>>> PetscFinalize() >>>>>> } >>>>>> >>>>>> So after finalizing and initializing it back, I always get an error with one of the AO calls. Given below is the complete error I get when running with 8 procs. Can anyone give me a hint what the problem could be? I was told earlier, that finalizing and initializing petsc should not be a problem! >>>>>> >>>>>> Best, >>>>>> Suraj >>>>>> >>>>>> -------------------------ERROR starts here------------------------------- >>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>> [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>> [0]PETSC ERROR: is given. >>>>>> [0]PETSC ERROR: [0] PetscStrcmp line 414 /root/petsc/petsc-3.3-p7/src/sys/utils/str.c >>>>>> [0]PETSC ERROR: [0] PetscFListAdd line 202 /root/petsc/petsc-3.3-p7/src/sys/dll/reg.c >>>>>> [0]PETSC ERROR: [0] AORegister line 99 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/aoreg.c >>>>>> [0]PETSC ERROR: [0] AORegisterAll line 28 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/aoregall.c >>>>>> [0]PETSC ERROR: [0] AOInitializePackage line 49 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/dlregisdm.c >>>>>> [0]PETSC ERROR: [0] AOCreate line 530 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/ao.c >>>>>> [0]PETSC ERROR: [0] AOCreateBasicIS line 381 /root/petsc/petsc-3.3-p7/src/dm/ao/impls/basic/aobasic.c >>>>>> [0]PETSC ERROR: [0] AOCreateBasic line 336 /root/petsc/petsc-3.3-p7/src/dm/ao/impls/basic/aobasic.c >>>>>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>>> [0]PETSC ERROR: Signal received! >>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [0]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>>> [0]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>>> [0]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>>> [0]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>>> -------------------------------------------------------------------------- >>>>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>>>>> with errorcode 59. >>>>>> >>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>>>> You may or may not see output from other processes, depending on >>>>>> exactly when Open MPI kills them. >>>>>> -------------------------------------------------------------------------- >>>>>> [1]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [1]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>>>> [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>>> [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>>> [1]PETSC ERROR: likely location of problem given in stack below >>>>>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>> [1]PETSC ERROR: is given. >>>>>> [1]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>>> [1]PETSC ERROR: Signal received! >>>>>> [1]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [1]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>>> [1]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>> [1]PETSC ERROR: See docs/index.html for manual pages. >>>>>> [1]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [1]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>>> [1]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>>> [1]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>>> [1]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>>> [1]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>>> [2]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [2]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>>>> [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>>> [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>>> [2]PETSC ERROR: likely location of problem given in stack below >>>>>> [2]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>> [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>> [2]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>> [2]PETSC ERROR: is given. >>>>>> [2]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>>> [2]PETSC ERROR: Signal received! >>>>>> [2]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [2]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>>> [2]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>> [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>> [2]PETSC ERROR: See docs/index.html for manual pages. >>>>>> [2]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [2]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>>> [2]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>>> [2]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>>> [2]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>>> [2]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [2]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>>> [3]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [3]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>>>> [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>>> [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>>> [3]PETSC ERROR: likely location of problem given in stack below >>>>>> [3]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>> [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>> [3]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>> [3]PETSC ERROR: is given. >>>>>> [3]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>>> [3]PETSC ERROR: Signal received! >>>>>> [3]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [3]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>>> [3]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>> [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>> [3]PETSC ERROR: See docs/index.html for manual pages. >>>>>> [3]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [3]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>>> [3]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>>> [3]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>>> [3]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>>> [3]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [3]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>>> [4]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [4]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>>>> [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>>> [4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>>> [4]PETSC ERROR: likely location of problem given in stack below >>>>>> [4]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>> [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>> [4]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>> [4]PETSC ERROR: is given. >>>>>> [4]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>>> [4]PETSC ERROR: Signal received! >>>>>> [4]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [4]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>>> [4]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>> [4]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>> [4]PETSC ERROR: See docs/index.html for manual pages. >>>>>> [4]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [4]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>>> [4]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>>> [4]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>>> [4]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>>> [4]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [4]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>>> [5]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [5]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>>>> [5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>>> [5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[5]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>>> [5]PETSC ERROR: likely location of problem given in stack below >>>>>> [5]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>> [5]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>> [5]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>> [5]PETSC ERROR: is given. >>>>>> [5]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>>> [5]PETSC ERROR: Signal received! >>>>>> [5]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [5]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>>> [5]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>> [5]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>> [5]PETSC ERROR: See docs/index.html for manual pages. >>>>>> [5]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [5]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>>> [5]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>>> [5]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>>> [5]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>>> [5]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [5]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>>> [6]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [6]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>>>> [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>>> [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>>> [6]PETSC ERROR: likely location of problem given in stack below >>>>>> [6]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>> [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>> [6]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>> [6]PETSC ERROR: is given. >>>>>> [6]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>>> [6]PETSC ERROR: Signal received! >>>>>> [6]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [6]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>>> [6]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>> [6]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>> [6]PETSC ERROR: See docs/index.html for manual pages. >>>>>> [6]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [6]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>>> [6]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>>> [6]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>>> [6]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>>> [6]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [6]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>>> [7]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [7]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>>>> [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>>> [7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[7]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>>> [7]PETSC ERROR: likely location of problem given in stack below >>>>>> [7]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>> [7]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>> [7]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>> [7]PETSC ERROR: is given. >>>>>> [7]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>>> [7]PETSC ERROR: Signal received! >>>>>> [7]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [7]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>>> [7]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>> [7]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>> [7]PETSC ERROR: See docs/index.html for manual pages. >>>>>> [7]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [7]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>>> [7]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>>> [7]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>>> [7]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>>> [7]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [7]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>>> -------------------------------------------------------------------------- >>>>>> mpiexec has exited due to process rank 0 with PID 2670 on >>>>>> node grsacc17 exiting improperly. There are two reasons this could occur: >>>>>> >>>>>> 1. this process did not call "init" before exiting, but others in >>>>>> the job did. This can cause a job to hang indefinitely while it waits >>>>>> for all processes to call "init". By rule, if one process calls "init", >>>>>> then ALL processes must call "init" prior to termination. >>>>>> >>>>>> 2. this process called "init", but exited without calling "finalize". >>>>>> By rule, all processes that call "init" MUST call "finalize" prior to >>>>>> exiting or it will be considered an "abnormal termination" >>>>>> >>>>>> This may have caused other processes in the application to be >>>>>> terminated by signals sent by mpiexec (as reported here). >>>>> >>>>> >>>> >>> >> > From ztdepyahoo at 163.com Wed Sep 4 10:16:30 2013 From: ztdepyahoo at 163.com (=?GBK?B?tqHAz8qm?=) Date: Wed, 4 Sep 2013 23:16:30 +0800 (CST) Subject: [petsc-users] Could you please give me some advice for the following error Message-ID: <32eead67.2729d.140e98bd661.Coremail.ztdepyahoo@163.com> 1] ***ASSERTION failed on line 243 of file /home/ztdep/Downloads/petsc-3.3-p6/externalpackages/parmetis-4.0.2-p3/libparmetis/match.c:k >= firstvtx && k < lastvtx [ 1] 2562 5124 10 0 0 main: /home/ztdep/Downloads/petsc-3.3-p6/externalpackages/parmetis-4.0.2-p3/libparmetis/match.c:243: libparmetis__Match_Global: Assertion `k >= firstvtx && k < lastvtx' failed. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Wed Sep 4 10:20:47 2013 From: mfadams at lbl.gov (Mark F. Adams) Date: Wed, 4 Sep 2013 11:20:47 -0400 Subject: [petsc-users] Could you please give me some advice for the following error In-Reply-To: <32eead67.2729d.140e98bd661.Coremail.ztdepyahoo@163.com> References: <32eead67.2729d.140e98bd661.Coremail.ztdepyahoo@163.com> Message-ID: <97AE0B7A-29FE-4C44-AA20-D527CA1C0A6D@lbl.gov> My guess is that your matrix is structurally unsymmetric. On Sep 4, 2013, at 11:16 AM, ??? wrote: > 1] ***ASSERTION failed on line 243 of file /home/ztdep/Downloads/petsc-3.3-p6/externalpackages/parmetis-4.0.2-p3/libparmetis/match.c:k >= firstvtx && k < lastvtx > [ 1] 2562 5124 10 0 0 > main: /home/ztdep/Downloads/petsc-3.3-p6/externalpackages/parmetis-4.0.2-p3/libparmetis/match.c:243: libparmetis__Match_Global: Assertion `k >= firstvtx && k < lastvtx' failed. > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ztdepyahoo at 163.com Wed Sep 4 10:23:26 2013 From: ztdepyahoo at 163.com (=?GBK?B?tqHAz8qm?=) Date: Wed, 4 Sep 2013 23:23:26 +0800 (CST) Subject: [petsc-users] Could you please give me some advice for the following error In-Reply-To: <97AE0B7A-29FE-4C44-AA20-D527CA1C0A6D@lbl.gov> References: <32eead67.2729d.140e98bd661.Coremail.ztdepyahoo@163.com> <97AE0B7A-29FE-4C44-AA20-D527CA1C0A6D@lbl.gov> Message-ID: <56589478.27365.140e9922e7d.Coremail.ztdepyahoo@163.com> yes, i am computing convection diffusion problem with a triangle unstructured grid. the matrix coefficients is unsymemetric. but is it a necessary for the parmetis to do the work? ? 2013-09-04 23:20:47?"Mark F. Adams" ??? My guess is that your matrix is structurally unsymmetric. On Sep 4, 2013, at 11:16 AM, ??? wrote: 1] ***ASSERTION failed on line 243 of file /home/ztdep/Downloads/petsc-3.3-p6/externalpackages/parmetis-4.0.2-p3/libparmetis/match.c:k >= firstvtx && k < lastvtx [ 1] 2562 5124 10 0 0 main: /home/ztdep/Downloads/petsc-3.3-p6/externalpackages/parmetis-4.0.2-p3/libparmetis/match.c:243: libparmetis__Match_Global: Assertion `k >= firstvtx && k < lastvtx' failed. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Sep 4 10:25:38 2013 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 4 Sep 2013 10:25:38 -0500 Subject: [petsc-users] Could you please give me some advice for the following error In-Reply-To: <56589478.27365.140e9922e7d.Coremail.ztdepyahoo@163.com> References: <32eead67.2729d.140e98bd661.Coremail.ztdepyahoo@163.com> <97AE0B7A-29FE-4C44-AA20-D527CA1C0A6D@lbl.gov> <56589478.27365.140e9922e7d.Coremail.ztdepyahoo@163.com> Message-ID: On Wed, Sep 4, 2013 at 10:23 AM, ??? wrote: > yes, i am computing convection diffusion problem with a triangle > unstructured grid. the matrix coefficients is unsymemetric. > but is it a necessary for the parmetis to do the work? > The matrix can be nonsymmetric, but it has to be structurally symmetric. Matt > ? 2013-09-04 23:20:47?"Mark F. Adams" ??? > > My guess is that your matrix is structurally unsymmetric. > > On Sep 4, 2013, at 11:16 AM, ??? wrote: > > 1] ***ASSERTION failed on line 243 of file > /home/ztdep/Downloads/petsc-3.3-p6/externalpackages/parmetis-4.0.2-p3/libparmetis/match.c:k > >= firstvtx && k < lastvtx > [ 1] 2562 5124 10 0 0 > main: > /home/ztdep/Downloads/petsc-3.3-p6/externalpackages/parmetis-4.0.2-p3/libparmetis/match.c:243: > libparmetis__Match_Global: Assertion `k >= firstvtx && k < lastvtx' failed. > > > > > > > > > > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From ztdepyahoo at 163.com Wed Sep 4 10:34:13 2013 From: ztdepyahoo at 163.com (=?GBK?B?tqHAz8qm?=) Date: Wed, 4 Sep 2013 23:34:13 +0800 (CST) Subject: [petsc-users] Could you please give me some advice for the following error In-Reply-To: References: <32eead67.2729d.140e98bd661.Coremail.ztdepyahoo@163.com> <97AE0B7A-29FE-4C44-AA20-D527CA1C0A6D@lbl.gov> <56589478.27365.140e9922e7d.Coremail.ztdepyahoo@163.com> Message-ID: <442e6a00.27478.140e99c0fd2.Coremail.ztdepyahoo@163.com> the input parameter of this partation is the drow and dcol. could you please tell me how to resolve this problem. MatCreateMPIAdj(PETSC_COMM_WORLD,drow.size()-1,CellsNumber,&drow[0],&dcol[0],NULL,&adj); ? 2013-09-04 23:25:38?"Matthew Knepley" ??? On Wed, Sep 4, 2013 at 10:23 AM, ??? wrote: yes, i am computing convection diffusion problem with a triangle unstructured grid. the matrix coefficients is unsymemetric. but is it a necessary for the parmetis to do the work? The matrix can be nonsymmetric, but it has to be structurally symmetric. Matt ? 2013-09-04 23:20:47?"Mark F. Adams" ??? My guess is that your matrix is structurally unsymmetric. On Sep 4, 2013, at 11:16 AM, ??? wrote: 1] ***ASSERTION failed on line 243 of file /home/ztdep/Downloads/petsc-3.3-p6/externalpackages/parmetis-4.0.2-p3/libparmetis/match.c:k >= firstvtx && k < lastvtx [ 1] 2562 5124 10 0 0 main: /home/ztdep/Downloads/petsc-3.3-p6/externalpackages/parmetis-4.0.2-p3/libparmetis/match.c:243: libparmetis__Match_Global: Assertion `k >= firstvtx && k < lastvtx' failed. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Sep 4 10:39:46 2013 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 4 Sep 2013 10:39:46 -0500 Subject: [petsc-users] Could you please give me some advice for the following error In-Reply-To: <442e6a00.27478.140e99c0fd2.Coremail.ztdepyahoo@163.com> References: <32eead67.2729d.140e98bd661.Coremail.ztdepyahoo@163.com> <97AE0B7A-29FE-4C44-AA20-D527CA1C0A6D@lbl.gov> <56589478.27365.140e9922e7d.Coremail.ztdepyahoo@163.com> <442e6a00.27478.140e99c0fd2.Coremail.ztdepyahoo@163.com> Message-ID: On Wed, Sep 4, 2013 at 10:34 AM, ??? wrote: > the input parameter of this partation is the drow and dcol. > could you please tell me how to resolve this problem. > > > MatCreateMPIAdj(PETSC_COMM_WORLD,drow.size()-1,CellsNumber,&drow[0],&dcol[0],NULL,&adj); > Graph partitioners like ParMetis partition undirected graphs. They are not suitable for directed graphs, which is what a nonsymmetric matrix corresponds to. I do not know what you should do here. Matt > ? 2013-09-04 23:25:38?"Matthew Knepley" ??? > > On Wed, Sep 4, 2013 at 10:23 AM, ??? wrote: > >> yes, i am computing convection diffusion problem with a triangle >> unstructured grid. the matrix coefficients is unsymemetric. >> but is it a necessary for the parmetis to do the work? >> > > The matrix can be nonsymmetric, but it has to be structurally symmetric. > > Matt > > >> ? 2013-09-04 23:20:47?"Mark F. Adams" ??? >> >> My guess is that your matrix is structurally unsymmetric. >> >> On Sep 4, 2013, at 11:16 AM, ??? wrote: >> >> 1] ***ASSERTION failed on line 243 of file >> /home/ztdep/Downloads/petsc-3.3-p6/externalpackages/parmetis-4.0.2-p3/libparmetis/match.c:k >> >= firstvtx && k < lastvtx >> [ 1] 2562 5124 10 0 0 >> main: >> /home/ztdep/Downloads/petsc-3.3-p6/externalpackages/parmetis-4.0.2-p3/libparmetis/match.c:243: >> libparmetis__Match_Global: Assertion `k >= firstvtx && k < lastvtx' failed. >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Sep 4 11:18:23 2013 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 4 Sep 2013 11:18:23 -0500 (CDT) Subject: [petsc-users] Re-initializing petsc again In-Reply-To: <00164CC3-C3F6-4361-8055-4FB9C3F76855@mcs.anl.gov> References: <315F25F7-7274-40E9-A244-5CD26F975CE5@gmail.com> <614D53AA-89C2-4C34-A192-5FA4BE0D1F27@mcs.anl.gov> <135162E5-14F4-49AF-831F-A5864CAC3A8C@mcs.anl.gov> <7C516550-089B-4ACF-AE39-3DD4BC38EA41@gmail.com> <00164CC3-C3F6-4361-8055-4FB9C3F76855@mcs.anl.gov> Message-ID: On Wed, 4 Sep 2013, Barry Smith wrote: > > >> In file included from /opt/petsc-3.4.2/include/petsc.h:8, > >> from Main/time_integration_dir/compute_tau_w_mean.c:91: > >> /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: > >> /opt/petsc-3.4.2/include/petscbt.h:33: error: invalid storage class for function ?PetscBTLength? > >> /opt/petsc-3.4.2/include/petscbt.h:38: error: invalid storage class for function ?PetscBTMemzero? > >> /opt/petsc-3.4.2/include/petscbt.h:43: error: invalid storage class for function ?PetscBTDestroy? > > > Ok, nothing wrong with the static inline or build; There must be something wonky in compute_tau_w_mean.c or a file it includes that is causing the error. > > I would start debugging by removing stuff from compute_tau_w_mean.c before the #include "petsc.h" until the error goes away and that will show the culprit. Also you have a huge list of -I dirs. Make sure a wrong petscconf.h isn't getting included from one of these locations. I would do a 'make CFLAGS=-E' compute_tau_w_mean.o and look at the preprocessed 'PetscBTLength' reference in there. Also 'grep petscconf.h compute_tau_w_mean.o' Satish > > Barry > > > > On Sep 3, 2013, at 10:21 PM, Suraj Prabhakaran wrote: > > > Hi Barry, > > > > Thanks a lot! Here are the files attached. > > > > Suraj > > > > > > > > On Sep 4, 2013, at 5:13 AM, Barry Smith wrote: > > > >> > >> Something is going wrong with the PETSC_STATIC_INLINE definition. Please send configure.log and make.log > >> > >> Barry > >> > >> On Sep 3, 2013, at 10:01 PM, Suraj Prabhakaran wrote: > >> > >>> Thanks a lot for the confirmation!! > >>> I am now compiling it with 3.4.2, but I get a strange error and could be great if someone can just point out whats wrong. (Might be something stupid, but just in case...) > >>> > >>> Thanks in advance! > >>> > >>> Suraj > >>> > >>> -------------------------------ERROR----------------------------- > >>> > >>> In file included from /opt/petsc-3.4.2/include/petsc.h:8, > >>> from Main/time_integration_dir/compute_tau_w_mean.c:91: > >>> /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: > >>> /opt/petsc-3.4.2/include/petscbt.h:33: error: invalid storage class for function ?PetscBTLength? > >>> /opt/petsc-3.4.2/include/petscbt.h:38: error: invalid storage class for function ?PetscBTMemzero? > >>> /opt/petsc-3.4.2/include/petscbt.h:43: error: invalid storage class for function ?PetscBTDestroy? > >>> /opt/petsc-3.4.2/include/petscbt.h:48: error: invalid storage class for function ?PetscBTLookup? > >>> /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTLookup?: > >>> /opt/petsc-3.4.2/include/petscbt.h:47: warning: declaration of ?index? shadows a global declaration > >>> /usr/include/string.h:489: warning: shadowed declaration is here > >>> /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: > >>> /opt/petsc-3.4.2/include/petscbt.h:59: error: invalid storage class for function ?PetscBTView? > >>> /opt/petsc-3.4.2/include/petscbt.h:74: error: invalid storage class for function ?PetscBTCreate? > >>> /opt/petsc-3.4.2/include/petscbt.h:79: error: invalid storage class for function ?PetscBTLookupSet? > >>> /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTLookupSet?: > >>> /opt/petsc-3.4.2/include/petscbt.h:78: warning: declaration of ?index? shadows a global declaration > >>> /usr/include/string.h:489: warning: shadowed declaration is here > >>> /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: > >>> /opt/petsc-3.4.2/include/petscbt.h:91: error: invalid storage class for function ?PetscBTSet? > >>> /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTSet?: > >>> /opt/petsc-3.4.2/include/petscbt.h:90: warning: declaration of ?index? shadows a global declaration > >>> /usr/include/string.h:489: warning: shadowed declaration is here > >>> /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: > >>> /opt/petsc-3.4.2/include/petscbt.h:103: error: invalid storage class for function ?PetscBTLookupClear? > >>> /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTLookupClear?: > >>> /opt/petsc-3.4.2/include/petscbt.h:102: warning: declaration of ?index? shadows a global declaration > >>> /usr/include/string.h:489: warning: shadowed declaration is here > >>> /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: > >>> /opt/petsc-3.4.2/include/petscbt.h:115: error: invalid storage class for function ?PetscBTClear? > >>> /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTClear?: > >>> /opt/petsc-3.4.2/include/petscbt.h:114: warning: declaration of ?index? shadows a global declaration > >>> /usr/include/string.h:489: warning: shadowed declaration is here > >>> In file included from /opt/petsc-3.4.2/include/petsc.h:9, > >>> from Main/time_integration_dir/compute_tau_w_mean.c:91: > >>> /opt/petsc-3.4.2/include/petscctable.h: In function ?compute_tau_w_mean?: > >>> /opt/petsc-3.4.2/include/petscctable.h:20: error: invalid storage class for function ?PetscHash? > >>> /opt/petsc-3.4.2/include/petscctable.h:40: error: invalid storage class for function ?PetscTableAdd? > >>> In file included from /opt/petsc-3.4.2/include/petsc.h:9, > >>> from Main/time_integration_dir/compute_tau_w_mean.c:91: > >>> /opt/petsc-3.4.2/include/petscctable.h:83: error: invalid storage class for function ?PetscTableAddCount? > >>> /opt/petsc-3.4.2/include/petscctable.h:120: error: invalid storage class for function ?PetscTableFind? > >>> In file included from /opt/petsc-3.4.2/include/petscsnesfas.h:3, > >>> from /opt/petsc-3.4.2/include/petsc.h:27, > >>> from Main/time_integration_dir/compute_tau_w_mean.c:91: > >>> /opt/petsc-3.4.2/include/petscsnes.h:585: error: invalid storage class for function ?SNESGetSNESLineSearch? > >>> /opt/petsc-3.4.2/include/petscsnes.h:586: error: invalid storage class for function ?SNESSetSNESLineSearch? > >>> Main/time_integration_dir/compute_tau_w_mean.c:223: warning: implicit declaration of function ?wall_thermoquants? > >>> Main/time_integration_dir/compute_tau_w_mean.c:155: warning: unused variable ?iface_plus_one? > >>> Main/time_integration_dir/compute_tau_w_mean.c:330: warning: value computed is not used > >>> Main/time_integration_dir/compute_tau_w_mean.c:116: warning: unused variable ?ierr? > >>> Main/time_integration_dir/compute_tau_w_mean.c:111: warning: unused variable ?icell? > >>> Main/time_integration_dir/compute_tau_w_mean.c:105: warning: unused variable ?mach? > >>> /opt/petsc-3.4.2/include/petscts.h:328: warning: unused variable ?TSRegisterAllCalled? > >>> /opt/petsc-3.4.2/include/petscts.h:327: warning: unused variable ?TSList? > >>> /opt/petsc-3.4.2/include/petscts.h:168: warning: unused variable ?DMTS_CLASSID? > >>> /opt/petsc-3.4.2/include/petscts.h:167: warning: unused variable ?TS_CLASSID? > >>> /opt/petsc-3.4.2/include/petscts.h:163: warning: unused variable ?TSExactFinalTimeOptions? > >>> /opt/petsc-3.4.2/include/petscts.h:101: warning: unused variable ?TSConvergedReasons? > >>> /opt/petsc-3.4.2/include/petscts.h:80: warning: unused variable ?TSEquationTypes? > >>> /opt/petsc-3.4.2/include/petscsnesfas.h:19: warning: unused variable ?SNESFASTypes? > >>> /opt/petsc-3.4.2/include/petscsnes.h:679: warning: unused variable ?SNESQNRestartTypes? > >>> /opt/petsc-3.4.2/include/petscsnes.h:675: warning: unused variable ?SNESQNScaleTypes? > >>> /opt/petsc-3.4.2/include/petscsnes.h:667: warning: unused variable ?SNESNCGTypes? > >>> /opt/petsc-3.4.2/include/petscsnes.h:654: warning: unused variable ?SNESNGMRESSelectTypes? > >>> /opt/petsc-3.4.2/include/petscsnes.h:648: warning: unused variable ?SNESNGMRESRestartTypes? > >>> /opt/petsc-3.4.2/include/petscsnes.h:474: warning: unused variable ?SNESLineSearch_Apply? > >>> /opt/petsc-3.4.2/include/petscsnes.h:473: warning: unused variable ?SNESLineSearchList? > >>> /opt/petsc-3.4.2/include/petscsnes.h:472: warning: unused variable ?SNESLineSearchRegisterAllCalled? > >>> /opt/petsc-3.4.2/include/petscsnes.h:471: warning: unused variable ?SNESLINESEARCH_CLASSID? > >>> /opt/petsc-3.4.2/include/petscsnes.h:470: warning: unused variable ?SNESList? > >>> /opt/petsc-3.4.2/include/petscsnes.h:469: warning: unused variable ?SNESRegisterAllCalled? > >>> /opt/petsc-3.4.2/include/petscsnes.h:356: warning: unused variable ?SNESNormTypes? > >>> /opt/petsc-3.4.2/include/petscsnes.h:217: warning: unused variable ?SNESConvergedReasons? > >>> /opt/petsc-3.4.2/include/petscsnes.h:50: warning: unused variable ?DMSNES_CLASSID? > >>> /opt/petsc-3.4.2/include/petscsnes.h:49: warning: unused variable ?SNES_CLASSID? > >>> /opt/petsc-3.4.2/include/petsccharacteristic.h:44: warning: unused variable ?CharacteristicList? > >>> /opt/petsc-3.4.2/include/petsccharacteristic.h:43: warning: unused variable ?CharacteristicRegisterAllCalled? > >>> /opt/petsc-3.4.2/include/petscdmda.h:27: warning: unused variable ?DMDABoundaryTypes? > >>> /opt/petsc-3.4.2/include/petscpf.h:50: warning: unused variable ?PFRegisterAllCalled? > >>> /opt/petsc-3.4.2/include/petscpf.h:40: warning: unused variable ?PF_CLASSID? > >>> /opt/petsc-3.4.2/include/petscpf.h:12: warning: unused variable ?PFList? > >>> /opt/petsc-3.4.2/include/petscdm.h:34: warning: unused variable ?DMRegisterAllCalled? > >>> /opt/petsc-3.4.2/include/petscdm.h:33: warning: unused variable ?DMList? > >>> /opt/petsc-3.4.2/include/petscdm.h:11: warning: unused variable ?DM_CLASSID? > >>> /opt/petsc-3.4.2/include/petscdraw.h:239: warning: unused variable ?PETSC_DRAWHG_CLASSID? > >>> /opt/petsc-3.4.2/include/petscdraw.h:225: warning: unused variable ?PETSC_DRAWSP_CLASSID? > >>> /opt/petsc-3.4.2/include/petscdraw.h:206: warning: unused variable ?PETSC_DRAWLG_CLASSID? > >>> /opt/petsc-3.4.2/include/petscdraw.h:195: warning: unused variable ?PETSC_DRAWAXIS_CLASSID? > >>> /opt/petsc-3.4.2/include/petscdraw.h:26: warning: unused variable ?PetscDrawList? > >>> /opt/petsc-3.4.2/include/petscdraw.h:9: warning: unused variable ?PETSC_DRAW_CLASSID? > >>> /opt/petsc-3.4.2/include/petscmatlab.h:8: warning: unused variable ?MATLABENGINE_CLASSID? > >>> /opt/petsc-3.4.2/include/petscthreadcomm.h:35: warning: unused variable ?PetscThreadCommReductionOps? > >>> /opt/petsc-3.4.2/include/petscthreadcomm.h:32: warning: unused variable ?PetscThreadCommList? > >>> > >>> > >>> > >>> On Sep 3, 2013, at 9:03 PM, Satish Balay wrote: > >>> > >>>> Also you would have to call MPI_Init() before the first PetscInitialize() [and > >>>> similarly MPI_Finalize() after the last call to PetscFinalize()] > >>>> > >>>> Satish > >>>> > >>>> On Tue, 3 Sep 2013, Barry Smith wrote: > >>>> > >>>>> > >>>>> There is a problem with the 3.3 series when reinitializing PETSc. Can you switch to PETSc 3.4 which properly supports calling PetscInitialize() several times. > >>>>> > >>>>> Barry > >>>>> > >>>>> On Sep 3, 2013, at 1:58 PM, Suraj Prabhakaran wrote: > >>>>> > >>>>>> Hello, > >>>>>> > >>>>>> Again, this relates to finalizing and initializing petsc in my application. I basically want to finalize petsc and initialize it again with a different MPI communicator. A rough sketch of my quite huge application is as follows. > >>>>>> > >>>>>> > >>>>>> main() > >>>>>> { > >>>>>> PetscInitialize() > >>>>>> ... > >>>>>> ... > >>>>>> AOCreate().. > >>>>>> AO operations.. > >>>>>> AODestroy().. > >>>>>> ... > >>>>>> ... > >>>>>> PetscFinalize() > >>>>>> PETSC_COMM_WORLD = A new communicator (which includes more processes) > >>>>>> PetscInitialize() > >>>>>> ... > >>>>>> ... > >>>>>> AOCreate().. > >>>>>> AO operations.. > >>>>>> AODestroy() > >>>>>> ... > >>>>>> ... > >>>>>> PetscFinalize() > >>>>>> } > >>>>>> > >>>>>> So after finalizing and initializing it back, I always get an error with one of the AO calls. Given below is the complete error I get when running with 8 procs. Can anyone give me a hint what the problem could be? I was told earlier, that finalizing and initializing petsc should not be a problem! > >>>>>> > >>>>>> Best, > >>>>>> Suraj > >>>>>> > >>>>>> -------------------------ERROR starts here------------------------------- > >>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > >>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > >>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > >>>>>> [0]PETSC ERROR: likely location of problem given in stack below > >>>>>> [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > >>>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > >>>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function > >>>>>> [0]PETSC ERROR: is given. > >>>>>> [0]PETSC ERROR: [0] PetscStrcmp line 414 /root/petsc/petsc-3.3-p7/src/sys/utils/str.c > >>>>>> [0]PETSC ERROR: [0] PetscFListAdd line 202 /root/petsc/petsc-3.3-p7/src/sys/dll/reg.c > >>>>>> [0]PETSC ERROR: [0] AORegister line 99 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/aoreg.c > >>>>>> [0]PETSC ERROR: [0] AORegisterAll line 28 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/aoregall.c > >>>>>> [0]PETSC ERROR: [0] AOInitializePackage line 49 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/dlregisdm.c > >>>>>> [0]PETSC ERROR: [0] AOCreate line 530 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/ao.c > >>>>>> [0]PETSC ERROR: [0] AOCreateBasicIS line 381 /root/petsc/petsc-3.3-p7/src/dm/ao/impls/basic/aobasic.c > >>>>>> [0]PETSC ERROR: [0] AOCreateBasic line 336 /root/petsc/petsc-3.3-p7/src/dm/ao/impls/basic/aobasic.c > >>>>>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > >>>>>> [0]PETSC ERROR: Signal received! > >>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > >>>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. > >>>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > >>>>>> [0]PETSC ERROR: See docs/index.html for manual pages. > >>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [0]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > >>>>>> [0]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > >>>>>> [0]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > >>>>>> [0]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > >>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > >>>>>> -------------------------------------------------------------------------- > >>>>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > >>>>>> with errorcode 59. > >>>>>> > >>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > >>>>>> You may or may not see output from other processes, depending on > >>>>>> exactly when Open MPI kills them. > >>>>>> -------------------------------------------------------------------------- > >>>>>> [1]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [1]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > >>>>>> [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > >>>>>> [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > >>>>>> [1]PETSC ERROR: likely location of problem given in stack below > >>>>>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > >>>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > >>>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the function > >>>>>> [1]PETSC ERROR: is given. > >>>>>> [1]PETSC ERROR: --------------------- Error Message ------------------------------------ > >>>>>> [1]PETSC ERROR: Signal received! > >>>>>> [1]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [1]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > >>>>>> [1]PETSC ERROR: See docs/changes/index.html for recent updates. > >>>>>> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > >>>>>> [1]PETSC ERROR: See docs/index.html for manual pages. > >>>>>> [1]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [1]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > >>>>>> [1]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > >>>>>> [1]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > >>>>>> [1]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > >>>>>> [1]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > >>>>>> [2]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [2]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > >>>>>> [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > >>>>>> [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > >>>>>> [2]PETSC ERROR: likely location of problem given in stack below > >>>>>> [2]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > >>>>>> [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > >>>>>> [2]PETSC ERROR: INSTEAD the line number of the start of the function > >>>>>> [2]PETSC ERROR: is given. > >>>>>> [2]PETSC ERROR: --------------------- Error Message ------------------------------------ > >>>>>> [2]PETSC ERROR: Signal received! > >>>>>> [2]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [2]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > >>>>>> [2]PETSC ERROR: See docs/changes/index.html for recent updates. > >>>>>> [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > >>>>>> [2]PETSC ERROR: See docs/index.html for manual pages. > >>>>>> [2]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [2]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > >>>>>> [2]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > >>>>>> [2]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > >>>>>> [2]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > >>>>>> [2]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [2]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > >>>>>> [3]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [3]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > >>>>>> [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > >>>>>> [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > >>>>>> [3]PETSC ERROR: likely location of problem given in stack below > >>>>>> [3]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > >>>>>> [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > >>>>>> [3]PETSC ERROR: INSTEAD the line number of the start of the function > >>>>>> [3]PETSC ERROR: is given. > >>>>>> [3]PETSC ERROR: --------------------- Error Message ------------------------------------ > >>>>>> [3]PETSC ERROR: Signal received! > >>>>>> [3]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [3]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > >>>>>> [3]PETSC ERROR: See docs/changes/index.html for recent updates. > >>>>>> [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > >>>>>> [3]PETSC ERROR: See docs/index.html for manual pages. > >>>>>> [3]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [3]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > >>>>>> [3]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > >>>>>> [3]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > >>>>>> [3]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > >>>>>> [3]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [3]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > >>>>>> [4]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [4]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > >>>>>> [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > >>>>>> [4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > >>>>>> [4]PETSC ERROR: likely location of problem given in stack below > >>>>>> [4]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > >>>>>> [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > >>>>>> [4]PETSC ERROR: INSTEAD the line number of the start of the function > >>>>>> [4]PETSC ERROR: is given. > >>>>>> [4]PETSC ERROR: --------------------- Error Message ------------------------------------ > >>>>>> [4]PETSC ERROR: Signal received! > >>>>>> [4]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [4]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > >>>>>> [4]PETSC ERROR: See docs/changes/index.html for recent updates. > >>>>>> [4]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > >>>>>> [4]PETSC ERROR: See docs/index.html for manual pages. > >>>>>> [4]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [4]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > >>>>>> [4]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > >>>>>> [4]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > >>>>>> [4]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > >>>>>> [4]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [4]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > >>>>>> [5]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [5]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > >>>>>> [5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > >>>>>> [5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[5]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > >>>>>> [5]PETSC ERROR: likely location of problem given in stack below > >>>>>> [5]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > >>>>>> [5]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > >>>>>> [5]PETSC ERROR: INSTEAD the line number of the start of the function > >>>>>> [5]PETSC ERROR: is given. > >>>>>> [5]PETSC ERROR: --------------------- Error Message ------------------------------------ > >>>>>> [5]PETSC ERROR: Signal received! > >>>>>> [5]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [5]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > >>>>>> [5]PETSC ERROR: See docs/changes/index.html for recent updates. > >>>>>> [5]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > >>>>>> [5]PETSC ERROR: See docs/index.html for manual pages. > >>>>>> [5]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [5]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > >>>>>> [5]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > >>>>>> [5]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > >>>>>> [5]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > >>>>>> [5]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [5]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > >>>>>> [6]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [6]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > >>>>>> [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > >>>>>> [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > >>>>>> [6]PETSC ERROR: likely location of problem given in stack below > >>>>>> [6]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > >>>>>> [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > >>>>>> [6]PETSC ERROR: INSTEAD the line number of the start of the function > >>>>>> [6]PETSC ERROR: is given. > >>>>>> [6]PETSC ERROR: --------------------- Error Message ------------------------------------ > >>>>>> [6]PETSC ERROR: Signal received! > >>>>>> [6]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [6]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > >>>>>> [6]PETSC ERROR: See docs/changes/index.html for recent updates. > >>>>>> [6]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > >>>>>> [6]PETSC ERROR: See docs/index.html for manual pages. > >>>>>> [6]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [6]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > >>>>>> [6]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > >>>>>> [6]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > >>>>>> [6]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > >>>>>> [6]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [6]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > >>>>>> [7]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [7]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end > >>>>>> [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > >>>>>> [7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[7]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > >>>>>> [7]PETSC ERROR: likely location of problem given in stack below > >>>>>> [7]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > >>>>>> [7]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > >>>>>> [7]PETSC ERROR: INSTEAD the line number of the start of the function > >>>>>> [7]PETSC ERROR: is given. > >>>>>> [7]PETSC ERROR: --------------------- Error Message ------------------------------------ > >>>>>> [7]PETSC ERROR: Signal received! > >>>>>> [7]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [7]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 > >>>>>> [7]PETSC ERROR: See docs/changes/index.html for recent updates. > >>>>>> [7]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > >>>>>> [7]PETSC ERROR: See docs/index.html for manual pages. > >>>>>> [7]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [7]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 > >>>>>> [7]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib > >>>>>> [7]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 > >>>>>> [7]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging > >>>>>> [7]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> [7]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > >>>>>> -------------------------------------------------------------------------- > >>>>>> mpiexec has exited due to process rank 0 with PID 2670 on > >>>>>> node grsacc17 exiting improperly. There are two reasons this could occur: > >>>>>> > >>>>>> 1. this process did not call "init" before exiting, but others in > >>>>>> the job did. This can cause a job to hang indefinitely while it waits > >>>>>> for all processes to call "init". By rule, if one process calls "init", > >>>>>> then ALL processes must call "init" prior to termination. > >>>>>> > >>>>>> 2. this process called "init", but exited without calling "finalize". > >>>>>> By rule, all processes that call "init" MUST call "finalize" prior to > >>>>>> exiting or it will be considered an "abnormal termination" > >>>>>> > >>>>>> This may have caused other processes in the application to be > >>>>>> terminated by signals sent by mpiexec (as reported here). > >>>>> > >>>>> > >>>> > >>> > >> > > > > From suraj.prabhakaran at gmail.com Wed Sep 4 11:22:36 2013 From: suraj.prabhakaran at gmail.com (Suraj Prabhakaran) Date: Wed, 4 Sep 2013 18:22:36 +0200 Subject: [petsc-users] Re-initializing petsc again In-Reply-To: References: <315F25F7-7274-40E9-A244-5CD26F975CE5@gmail.com> <614D53AA-89C2-4C34-A192-5FA4BE0D1F27@mcs.anl.gov> <135162E5-14F4-49AF-831F-A5864CAC3A8C@mcs.anl.gov> <7C516550-089B-4ACF-AE39-3DD4BC38EA41@gmail.com> <00164CC3-C3F6-4361-8055-4FB9C3F76855@mcs.anl.gov> Message-ID: <92356A3E-9EF0-408F-A69F-C3148F14C382@gmail.com> Dear Barry and Satish, Thank you for your responses and help. I got it all cleared now. If it will be interesting for you, what I am trying to do is spawn (using MPI_COMM_SPAWN) new processes and re-initialize petsc along with the parents and children in one communicator. At the moment, after a couple of tests, it seems to work fine with Petsc 3.4.0. Best, Suraj On Sep 4, 2013, at 6:18 PM, Satish Balay wrote: > On Wed, 4 Sep 2013, Barry Smith wrote: > >> >>>> In file included from /opt/petsc-3.4.2/include/petsc.h:8, >>>> from Main/time_integration_dir/compute_tau_w_mean.c:91: >>>> /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: >>>> /opt/petsc-3.4.2/include/petscbt.h:33: error: invalid storage class for function ?PetscBTLength? >>>> /opt/petsc-3.4.2/include/petscbt.h:38: error: invalid storage class for function ?PetscBTMemzero? >>>> /opt/petsc-3.4.2/include/petscbt.h:43: error: invalid storage class for function ?PetscBTDestroy? >> >> >> Ok, nothing wrong with the static inline or build; There must be something wonky in compute_tau_w_mean.c or a file it includes that is causing the error. >> >> I would start debugging by removing stuff from compute_tau_w_mean.c before the #include "petsc.h" until the error goes away and that will show the culprit. > > Also you have a huge list of -I dirs. Make sure a wrong petscconf.h isn't getting > included from one of these locations. > > I would do a 'make CFLAGS=-E' compute_tau_w_mean.o > > and look at the preprocessed 'PetscBTLength' reference in there. > > Also 'grep petscconf.h compute_tau_w_mean.o' > > Satish > > >> >> Barry >> >> >> >> On Sep 3, 2013, at 10:21 PM, Suraj Prabhakaran wrote: >> >>> Hi Barry, >>> >>> Thanks a lot! Here are the files attached. >>> >>> Suraj >>> >>> >>> >>> On Sep 4, 2013, at 5:13 AM, Barry Smith wrote: >>> >>>> >>>> Something is going wrong with the PETSC_STATIC_INLINE definition. Please send configure.log and make.log >>>> >>>> Barry >>>> >>>> On Sep 3, 2013, at 10:01 PM, Suraj Prabhakaran wrote: >>>> >>>>> Thanks a lot for the confirmation!! >>>>> I am now compiling it with 3.4.2, but I get a strange error and could be great if someone can just point out whats wrong. (Might be something stupid, but just in case...) >>>>> >>>>> Thanks in advance! >>>>> >>>>> Suraj >>>>> >>>>> -------------------------------ERROR----------------------------- >>>>> >>>>> In file included from /opt/petsc-3.4.2/include/petsc.h:8, >>>>> from Main/time_integration_dir/compute_tau_w_mean.c:91: >>>>> /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: >>>>> /opt/petsc-3.4.2/include/petscbt.h:33: error: invalid storage class for function ?PetscBTLength? >>>>> /opt/petsc-3.4.2/include/petscbt.h:38: error: invalid storage class for function ?PetscBTMemzero? >>>>> /opt/petsc-3.4.2/include/petscbt.h:43: error: invalid storage class for function ?PetscBTDestroy? >>>>> /opt/petsc-3.4.2/include/petscbt.h:48: error: invalid storage class for function ?PetscBTLookup? >>>>> /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTLookup?: >>>>> /opt/petsc-3.4.2/include/petscbt.h:47: warning: declaration of ?index? shadows a global declaration >>>>> /usr/include/string.h:489: warning: shadowed declaration is here >>>>> /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: >>>>> /opt/petsc-3.4.2/include/petscbt.h:59: error: invalid storage class for function ?PetscBTView? >>>>> /opt/petsc-3.4.2/include/petscbt.h:74: error: invalid storage class for function ?PetscBTCreate? >>>>> /opt/petsc-3.4.2/include/petscbt.h:79: error: invalid storage class for function ?PetscBTLookupSet? >>>>> /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTLookupSet?: >>>>> /opt/petsc-3.4.2/include/petscbt.h:78: warning: declaration of ?index? shadows a global declaration >>>>> /usr/include/string.h:489: warning: shadowed declaration is here >>>>> /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: >>>>> /opt/petsc-3.4.2/include/petscbt.h:91: error: invalid storage class for function ?PetscBTSet? >>>>> /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTSet?: >>>>> /opt/petsc-3.4.2/include/petscbt.h:90: warning: declaration of ?index? shadows a global declaration >>>>> /usr/include/string.h:489: warning: shadowed declaration is here >>>>> /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: >>>>> /opt/petsc-3.4.2/include/petscbt.h:103: error: invalid storage class for function ?PetscBTLookupClear? >>>>> /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTLookupClear?: >>>>> /opt/petsc-3.4.2/include/petscbt.h:102: warning: declaration of ?index? shadows a global declaration >>>>> /usr/include/string.h:489: warning: shadowed declaration is here >>>>> /opt/petsc-3.4.2/include/petscbt.h: In function ?compute_tau_w_mean?: >>>>> /opt/petsc-3.4.2/include/petscbt.h:115: error: invalid storage class for function ?PetscBTClear? >>>>> /opt/petsc-3.4.2/include/petscbt.h: In function ?PetscBTClear?: >>>>> /opt/petsc-3.4.2/include/petscbt.h:114: warning: declaration of ?index? shadows a global declaration >>>>> /usr/include/string.h:489: warning: shadowed declaration is here >>>>> In file included from /opt/petsc-3.4.2/include/petsc.h:9, >>>>> from Main/time_integration_dir/compute_tau_w_mean.c:91: >>>>> /opt/petsc-3.4.2/include/petscctable.h: In function ?compute_tau_w_mean?: >>>>> /opt/petsc-3.4.2/include/petscctable.h:20: error: invalid storage class for function ?PetscHash? >>>>> /opt/petsc-3.4.2/include/petscctable.h:40: error: invalid storage class for function ?PetscTableAdd? >>>>> In file included from /opt/petsc-3.4.2/include/petsc.h:9, >>>>> from Main/time_integration_dir/compute_tau_w_mean.c:91: >>>>> /opt/petsc-3.4.2/include/petscctable.h:83: error: invalid storage class for function ?PetscTableAddCount? >>>>> /opt/petsc-3.4.2/include/petscctable.h:120: error: invalid storage class for function ?PetscTableFind? >>>>> In file included from /opt/petsc-3.4.2/include/petscsnesfas.h:3, >>>>> from /opt/petsc-3.4.2/include/petsc.h:27, >>>>> from Main/time_integration_dir/compute_tau_w_mean.c:91: >>>>> /opt/petsc-3.4.2/include/petscsnes.h:585: error: invalid storage class for function ?SNESGetSNESLineSearch? >>>>> /opt/petsc-3.4.2/include/petscsnes.h:586: error: invalid storage class for function ?SNESSetSNESLineSearch? >>>>> Main/time_integration_dir/compute_tau_w_mean.c:223: warning: implicit declaration of function ?wall_thermoquants? >>>>> Main/time_integration_dir/compute_tau_w_mean.c:155: warning: unused variable ?iface_plus_one? >>>>> Main/time_integration_dir/compute_tau_w_mean.c:330: warning: value computed is not used >>>>> Main/time_integration_dir/compute_tau_w_mean.c:116: warning: unused variable ?ierr? >>>>> Main/time_integration_dir/compute_tau_w_mean.c:111: warning: unused variable ?icell? >>>>> Main/time_integration_dir/compute_tau_w_mean.c:105: warning: unused variable ?mach? >>>>> /opt/petsc-3.4.2/include/petscts.h:328: warning: unused variable ?TSRegisterAllCalled? >>>>> /opt/petsc-3.4.2/include/petscts.h:327: warning: unused variable ?TSList? >>>>> /opt/petsc-3.4.2/include/petscts.h:168: warning: unused variable ?DMTS_CLASSID? >>>>> /opt/petsc-3.4.2/include/petscts.h:167: warning: unused variable ?TS_CLASSID? >>>>> /opt/petsc-3.4.2/include/petscts.h:163: warning: unused variable ?TSExactFinalTimeOptions? >>>>> /opt/petsc-3.4.2/include/petscts.h:101: warning: unused variable ?TSConvergedReasons? >>>>> /opt/petsc-3.4.2/include/petscts.h:80: warning: unused variable ?TSEquationTypes? >>>>> /opt/petsc-3.4.2/include/petscsnesfas.h:19: warning: unused variable ?SNESFASTypes? >>>>> /opt/petsc-3.4.2/include/petscsnes.h:679: warning: unused variable ?SNESQNRestartTypes? >>>>> /opt/petsc-3.4.2/include/petscsnes.h:675: warning: unused variable ?SNESQNScaleTypes? >>>>> /opt/petsc-3.4.2/include/petscsnes.h:667: warning: unused variable ?SNESNCGTypes? >>>>> /opt/petsc-3.4.2/include/petscsnes.h:654: warning: unused variable ?SNESNGMRESSelectTypes? >>>>> /opt/petsc-3.4.2/include/petscsnes.h:648: warning: unused variable ?SNESNGMRESRestartTypes? >>>>> /opt/petsc-3.4.2/include/petscsnes.h:474: warning: unused variable ?SNESLineSearch_Apply? >>>>> /opt/petsc-3.4.2/include/petscsnes.h:473: warning: unused variable ?SNESLineSearchList? >>>>> /opt/petsc-3.4.2/include/petscsnes.h:472: warning: unused variable ?SNESLineSearchRegisterAllCalled? >>>>> /opt/petsc-3.4.2/include/petscsnes.h:471: warning: unused variable ?SNESLINESEARCH_CLASSID? >>>>> /opt/petsc-3.4.2/include/petscsnes.h:470: warning: unused variable ?SNESList? >>>>> /opt/petsc-3.4.2/include/petscsnes.h:469: warning: unused variable ?SNESRegisterAllCalled? >>>>> /opt/petsc-3.4.2/include/petscsnes.h:356: warning: unused variable ?SNESNormTypes? >>>>> /opt/petsc-3.4.2/include/petscsnes.h:217: warning: unused variable ?SNESConvergedReasons? >>>>> /opt/petsc-3.4.2/include/petscsnes.h:50: warning: unused variable ?DMSNES_CLASSID? >>>>> /opt/petsc-3.4.2/include/petscsnes.h:49: warning: unused variable ?SNES_CLASSID? >>>>> /opt/petsc-3.4.2/include/petsccharacteristic.h:44: warning: unused variable ?CharacteristicList? >>>>> /opt/petsc-3.4.2/include/petsccharacteristic.h:43: warning: unused variable ?CharacteristicRegisterAllCalled? >>>>> /opt/petsc-3.4.2/include/petscdmda.h:27: warning: unused variable ?DMDABoundaryTypes? >>>>> /opt/petsc-3.4.2/include/petscpf.h:50: warning: unused variable ?PFRegisterAllCalled? >>>>> /opt/petsc-3.4.2/include/petscpf.h:40: warning: unused variable ?PF_CLASSID? >>>>> /opt/petsc-3.4.2/include/petscpf.h:12: warning: unused variable ?PFList? >>>>> /opt/petsc-3.4.2/include/petscdm.h:34: warning: unused variable ?DMRegisterAllCalled? >>>>> /opt/petsc-3.4.2/include/petscdm.h:33: warning: unused variable ?DMList? >>>>> /opt/petsc-3.4.2/include/petscdm.h:11: warning: unused variable ?DM_CLASSID? >>>>> /opt/petsc-3.4.2/include/petscdraw.h:239: warning: unused variable ?PETSC_DRAWHG_CLASSID? >>>>> /opt/petsc-3.4.2/include/petscdraw.h:225: warning: unused variable ?PETSC_DRAWSP_CLASSID? >>>>> /opt/petsc-3.4.2/include/petscdraw.h:206: warning: unused variable ?PETSC_DRAWLG_CLASSID? >>>>> /opt/petsc-3.4.2/include/petscdraw.h:195: warning: unused variable ?PETSC_DRAWAXIS_CLASSID? >>>>> /opt/petsc-3.4.2/include/petscdraw.h:26: warning: unused variable ?PetscDrawList? >>>>> /opt/petsc-3.4.2/include/petscdraw.h:9: warning: unused variable ?PETSC_DRAW_CLASSID? >>>>> /opt/petsc-3.4.2/include/petscmatlab.h:8: warning: unused variable ?MATLABENGINE_CLASSID? >>>>> /opt/petsc-3.4.2/include/petscthreadcomm.h:35: warning: unused variable ?PetscThreadCommReductionOps? >>>>> /opt/petsc-3.4.2/include/petscthreadcomm.h:32: warning: unused variable ?PetscThreadCommList? >>>>> >>>>> >>>>> >>>>> On Sep 3, 2013, at 9:03 PM, Satish Balay wrote: >>>>> >>>>>> Also you would have to call MPI_Init() before the first PetscInitialize() [and >>>>>> similarly MPI_Finalize() after the last call to PetscFinalize()] >>>>>> >>>>>> Satish >>>>>> >>>>>> On Tue, 3 Sep 2013, Barry Smith wrote: >>>>>> >>>>>>> >>>>>>> There is a problem with the 3.3 series when reinitializing PETSc. Can you switch to PETSc 3.4 which properly supports calling PetscInitialize() several times. >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>> On Sep 3, 2013, at 1:58 PM, Suraj Prabhakaran wrote: >>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> Again, this relates to finalizing and initializing petsc in my application. I basically want to finalize petsc and initialize it again with a different MPI communicator. A rough sketch of my quite huge application is as follows. >>>>>>>> >>>>>>>> >>>>>>>> main() >>>>>>>> { >>>>>>>> PetscInitialize() >>>>>>>> ... >>>>>>>> ... >>>>>>>> AOCreate().. >>>>>>>> AO operations.. >>>>>>>> AODestroy().. >>>>>>>> ... >>>>>>>> ... >>>>>>>> PetscFinalize() >>>>>>>> PETSC_COMM_WORLD = A new communicator (which includes more processes) >>>>>>>> PetscInitialize() >>>>>>>> ... >>>>>>>> ... >>>>>>>> AOCreate().. >>>>>>>> AO operations.. >>>>>>>> AODestroy() >>>>>>>> ... >>>>>>>> ... >>>>>>>> PetscFinalize() >>>>>>>> } >>>>>>>> >>>>>>>> So after finalizing and initializing it back, I always get an error with one of the AO calls. Given below is the complete error I get when running with 8 procs. Can anyone give me a hint what the problem could be? I was told earlier, that finalizing and initializing petsc should not be a problem! >>>>>>>> >>>>>>>> Best, >>>>>>>> Suraj >>>>>>>> >>>>>>>> -------------------------ERROR starts here------------------------------- >>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>>>> [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>> [0]PETSC ERROR: is given. >>>>>>>> [0]PETSC ERROR: [0] PetscStrcmp line 414 /root/petsc/petsc-3.3-p7/src/sys/utils/str.c >>>>>>>> [0]PETSC ERROR: [0] PetscFListAdd line 202 /root/petsc/petsc-3.3-p7/src/sys/dll/reg.c >>>>>>>> [0]PETSC ERROR: [0] AORegister line 99 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/aoreg.c >>>>>>>> [0]PETSC ERROR: [0] AORegisterAll line 28 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/aoregall.c >>>>>>>> [0]PETSC ERROR: [0] AOInitializePackage line 49 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/dlregisdm.c >>>>>>>> [0]PETSC ERROR: [0] AOCreate line 530 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/ao.c >>>>>>>> [0]PETSC ERROR: [0] AOCreateBasicIS line 381 /root/petsc/petsc-3.3-p7/src/dm/ao/impls/basic/aobasic.c >>>>>>>> [0]PETSC ERROR: [0] AOCreateBasic line 336 /root/petsc/petsc-3.3-p7/src/dm/ao/impls/basic/aobasic.c >>>>>>>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>>>>> [0]PETSC ERROR: Signal received! >>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [0]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>>>>> [0]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>>>>> [0]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>>>>> [0]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>>>>> -------------------------------------------------------------------------- >>>>>>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>>>>>>> with errorcode 59. >>>>>>>> >>>>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>>>>>> You may or may not see output from other processes, depending on >>>>>>>> exactly when Open MPI kills them. >>>>>>>> -------------------------------------------------------------------------- >>>>>>>> [1]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [1]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>>>>>> [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>>>>> [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>>>>> [1]PETSC ERROR: likely location of problem given in stack below >>>>>>>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>> [1]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>> [1]PETSC ERROR: is given. >>>>>>>> [1]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>>>>> [1]PETSC ERROR: Signal received! >>>>>>>> [1]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [1]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>>>>> [1]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>>>> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>>>> [1]PETSC ERROR: See docs/index.html for manual pages. >>>>>>>> [1]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [1]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>>>>> [1]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>>>>> [1]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>>>>> [1]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>>>>> [1]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>>>>> [2]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [2]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>>>>>> [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>>>>> [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>>>>> [2]PETSC ERROR: likely location of problem given in stack below >>>>>>>> [2]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>>>> [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>> [2]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>> [2]PETSC ERROR: is given. >>>>>>>> [2]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>>>>> [2]PETSC ERROR: Signal received! >>>>>>>> [2]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [2]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>>>>> [2]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>>>> [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>>>> [2]PETSC ERROR: See docs/index.html for manual pages. >>>>>>>> [2]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [2]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>>>>> [2]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>>>>> [2]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>>>>> [2]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>>>>> [2]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [2]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>>>>> [3]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [3]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>>>>>> [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>>>>> [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>>>>> [3]PETSC ERROR: likely location of problem given in stack below >>>>>>>> [3]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>>>> [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>> [3]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>> [3]PETSC ERROR: is given. >>>>>>>> [3]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>>>>> [3]PETSC ERROR: Signal received! >>>>>>>> [3]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [3]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>>>>> [3]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>>>> [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>>>> [3]PETSC ERROR: See docs/index.html for manual pages. >>>>>>>> [3]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [3]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>>>>> [3]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>>>>> [3]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>>>>> [3]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>>>>> [3]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [3]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>>>>> [4]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [4]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>>>>>> [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>>>>> [4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>>>>> [4]PETSC ERROR: likely location of problem given in stack below >>>>>>>> [4]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>>>> [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>> [4]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>> [4]PETSC ERROR: is given. >>>>>>>> [4]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>>>>> [4]PETSC ERROR: Signal received! >>>>>>>> [4]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [4]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>>>>> [4]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>>>> [4]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>>>> [4]PETSC ERROR: See docs/index.html for manual pages. >>>>>>>> [4]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [4]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>>>>> [4]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>>>>> [4]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>>>>> [4]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>>>>> [4]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [4]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>>>>> [5]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [5]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>>>>>> [5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>>>>> [5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[5]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>>>>> [5]PETSC ERROR: likely location of problem given in stack below >>>>>>>> [5]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>>>> [5]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>> [5]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>> [5]PETSC ERROR: is given. >>>>>>>> [5]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>>>>> [5]PETSC ERROR: Signal received! >>>>>>>> [5]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [5]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>>>>> [5]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>>>> [5]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>>>> [5]PETSC ERROR: See docs/index.html for manual pages. >>>>>>>> [5]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [5]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>>>>> [5]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>>>>> [5]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>>>>> [5]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>>>>> [5]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [5]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>>>>> [6]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [6]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>>>>>> [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>>>>> [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>>>>> [6]PETSC ERROR: likely location of problem given in stack below >>>>>>>> [6]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>>>> [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>> [6]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>> [6]PETSC ERROR: is given. >>>>>>>> [6]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>>>>> [6]PETSC ERROR: Signal received! >>>>>>>> [6]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [6]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>>>>> [6]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>>>> [6]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>>>> [6]PETSC ERROR: See docs/index.html for manual pages. >>>>>>>> [6]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [6]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>>>>> [6]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>>>>> [6]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>>>>> [6]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>>>>> [6]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [6]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>>>>> [7]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [7]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end >>>>>>>> [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>>>>> [7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[7]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>>>>> [7]PETSC ERROR: likely location of problem given in stack below >>>>>>>> [7]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>>>> [7]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>> [7]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>> [7]PETSC ERROR: is given. >>>>>>>> [7]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>>>>> [7]PETSC ERROR: Signal received! >>>>>>>> [7]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [7]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 >>>>>>>> [7]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>>>> [7]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>>>> [7]PETSC ERROR: See docs/index.html for manual pages. >>>>>>>> [7]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [7]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013 >>>>>>>> [7]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib >>>>>>>> [7]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013 >>>>>>>> [7]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging >>>>>>>> [7]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [7]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>>>>> -------------------------------------------------------------------------- >>>>>>>> mpiexec has exited due to process rank 0 with PID 2670 on >>>>>>>> node grsacc17 exiting improperly. There are two reasons this could occur: >>>>>>>> >>>>>>>> 1. this process did not call "init" before exiting, but others in >>>>>>>> the job did. This can cause a job to hang indefinitely while it waits >>>>>>>> for all processes to call "init". By rule, if one process calls "init", >>>>>>>> then ALL processes must call "init" prior to termination. >>>>>>>> >>>>>>>> 2. this process called "init", but exited without calling "finalize". >>>>>>>> By rule, all processes that call "init" MUST call "finalize" prior to >>>>>>>> exiting or it will be considered an "abnormal termination" >>>>>>>> >>>>>>>> This may have caused other processes in the application to be >>>>>>>> terminated by signals sent by mpiexec (as reported here). >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> From hsahasra at purdue.edu Thu Sep 5 09:57:31 2013 From: hsahasra at purdue.edu (Harshad Sahasrabudhe) Date: Thu, 5 Sep 2013 10:57:31 -0400 (EDT) Subject: [petsc-users] cusp complex and std complex In-Reply-To: <947669505.98440.1378392845574.JavaMail.root@mailhub027.itcs.purdue.edu> Message-ID: <1013432031.98458.1378393051980.JavaMail.root@mailhub027.itcs.purdue.edu> Hi, I am trying to compile my code, which uses std:complex with PETSc (Complex) compiled with CUSP. At certain places, we use references to send vectors to PETSc for VecDot. In such a case, I get following errors: no known conversion for argument 3 from 'cusp::complex' to 'std::complex&' Is there any way to get around these errors other than using PetscScalar instead of std::complex? Thanks, Harshad From knepley at gmail.com Thu Sep 5 10:31:40 2013 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 5 Sep 2013 10:31:40 -0500 Subject: [petsc-users] cusp complex and std complex In-Reply-To: <1013432031.98458.1378393051980.JavaMail.root@mailhub027.itcs.purdue.edu> References: <947669505.98440.1378392845574.JavaMail.root@mailhub027.itcs.purdue.edu> <1013432031.98458.1378393051980.JavaMail.root@mailhub027.itcs.purdue.edu> Message-ID: On Thu, Sep 5, 2013 at 9:57 AM, Harshad Sahasrabudhe wrote: > Hi, > > I am trying to compile my code, which uses std:complex with PETSc > (Complex) compiled with CUSP. At certain places, we use references to send > vectors to PETSc for VecDot. In such a case, I get following errors: > > no known conversion for argument 3 from 'cusp::complex' to > 'std::complex&' > > Is there any way to get around these errors other than using PetscScalar > instead of std::complex? > Are we supposed to guess where this happens? Please send the entire compile error, and if its in your code, send a simple code that demonstrates the problem. Matt > Thanks, > Harshad > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From frtr at fysik.dtu.dk Thu Sep 5 10:47:14 2013 From: frtr at fysik.dtu.dk (Frederik Treue) Date: Thu, 5 Sep 2013 17:47:14 +0200 Subject: [petsc-users] zero pattern of result of matmatmult Message-ID: <1378396034.2282.15.camel@frtr-laptop> Is it somehow possible to restrict the nonzero pattern of the result of matmatmult, ie. can I get petsc to only consider a subset of the elements of the result matrix, if I know the rest are going to be zero? Example: I have a backward and forward first order finite difference differentiation matrix, ie. B_(i,i-1)=-1/dx B_(i,i)=1/dx and F_(i,i)=-1/dx F_(i,i+1)=1/dx If I make the product B*F I should get the centered 2nd order differentiation matrix, ie. P_(i,i-1)=1/dx^2 P_(i,i)=-2/dx^2 P_(i,i+1)1/dx^2 and I do, but petsc allocates 5 entries points in each row, if I had 3 in both B and F, since P_(i,i-2) and P_(i,i+2) _might_ have been nonzero, but ofcourse they never are. Can I get petsc to stop allocating more than 3 entries per row in P? /Frederik Treue From jedbrown at mcs.anl.gov Thu Sep 5 11:03:51 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 05 Sep 2013 09:03:51 -0700 Subject: [petsc-users] zero pattern of result of matmatmult In-Reply-To: <1378396034.2282.15.camel@frtr-laptop> References: <1378396034.2282.15.camel@frtr-laptop> Message-ID: <87ioyf6ys8.fsf@mcs.anl.gov> Frederik Treue writes: > Is it somehow possible to restrict the nonzero pattern of the result of > matmatmult, ie. can I get petsc to only consider a subset of the > elements of the result matrix, if I know the rest are going to be zero? If I understand correctly, you don't want to allocate for entries that are *computed* to be 0.0 in the product matrix? Is this a real scenario? Multiplying difference operators together is really brittle (boundary conditions, geometry, etc) and uses more memory than necessary. > Example: I have a backward and forward first order finite difference > differentiation matrix, ie. > B_(i,i-1)=-1/dx B_(i,i)=1/dx > and > F_(i,i)=-1/dx F_(i,i+1)=1/dx > If I make the product B*F I should get the centered 2nd order > differentiation matrix, ie. > P_(i,i-1)=1/dx^2 P_(i,i)=-2/dx^2 P_(i,i+1)1/dx^2 > and I do, but petsc allocates 5 entries points in each row, if I had 3 > in both B and F, since P_(i,i-2) and P_(i,i+2) _might_ have been > nonzero, but ofcourse they never are. Can I get petsc to stop allocating > more than 3 entries per row in P? > > /Frederik Treue -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From hsahasra at purdue.edu Thu Sep 5 12:29:38 2013 From: hsahasra at purdue.edu (Harshad Sahasrabudhe) Date: Thu, 5 Sep 2013 13:29:38 -0400 (EDT) Subject: [petsc-users] cusp complex and std complex In-Reply-To: Message-ID: <2079042502.99468.1378402178739.JavaMail.root@mailhub027.itcs.purdue.edu> Hi Matt, I have attached a small test case which reproduces the problem. Here is the compilation command with the error: gcc -c test.cpp -I/u/sciteam/sahasrab/NEMO5/libs/petsc/include -I/u/sciteam/sahasrab/NEMO5/libs/petsc/arch-linux2-cxx-opt-cplx/include/ -I/opt/cray/mpt/6.0.1/gni/mpich2- cray/81/include/ -I/u/sciteam/sahasrab/NEMO5/libs/petsc/cusplibrary-master/ -I/opt/nvidia/cudatoolkit/5.0.35/include/ test.cpp: In function 'int main()': test.cpp:24:27: error: no matching function for call to 'VectorP >::dot_product(VectorP >&, VectorP >&, cusp::complex&)' a.dot_product(a, b, p[1]); ^ test.cpp:24:27: note: candidate is: test.cpp:9:8: note: void VectorP::dot_product(VectorP&, VectorP&, T&) [with T = std::complex] void dot_product(VectorP& vecA, VectorP& vecB, T& prod) ^ test.cpp:9:8: note: no known conversion for argument 3 from 'cusp::complex' to 'std::complex? NOTE: PETSc is compiled with CUSP. Thanks, Harshad >Are we supposed to guess where this happens? Please send the entire compile error, and if its in your code, send >a simple code that demonstrates the problem. >Matt >-- >What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >-- Norbert Wiener -------------- next part -------------- A non-text attachment was scrubbed... Name: test.cpp Type: text/x-c++src Size: 374 bytes Desc: not available URL: From hsahasra at purdue.edu Thu Sep 5 12:31:37 2013 From: hsahasra at purdue.edu (Harshad Sahasrabudhe) Date: Thu, 5 Sep 2013 13:31:37 -0400 (EDT) Subject: [petsc-users] cusp complex and std complex In-Reply-To: <2079042502.99468.1378402178739.JavaMail.root@mailhub027.itcs.purdue.edu> Message-ID: <1140734881.99501.1378402297661.JavaMail.root@mailhub027.itcs.purdue.edu> Sorry the last line of the error should read like this: test.cpp:9:8: note: no known conversion for argument 3 from 'cusp::complex' to 'std::complex&' Didn't copy it properly the last time. Harshad ----- Original Message ----- From: "Harshad Sahasrabudhe" To: "Matthew Knepley" Cc: petsc-users at mcs.anl.gov Sent: Thursday, September 5, 2013 1:29:38 PM Subject: Re: [petsc-users] cusp complex and std complex Hi Matt, I have attached a small test case which reproduces the problem. Here is the compilation command with the error: gcc -c test.cpp -I/u/sciteam/sahasrab/NEMO5/libs/petsc/include -I/u/sciteam/sahasrab/NEMO5/libs/petsc/arch-linux2-cxx-opt-cplx/include/ -I/opt/cray/mpt/6.0.1/gni/mpich2- cray/81/include/ -I/u/sciteam/sahasrab/NEMO5/libs/petsc/cusplibrary-master/ -I/opt/nvidia/cudatoolkit/5.0.35/include/ test.cpp: In function 'int main()': test.cpp:24:27: error: no matching function for call to 'VectorP >::dot_product(VectorP >&, VectorP >&, cusp::complex&)' a.dot_product(a, b, p[1]); ^ test.cpp:24:27: note: candidate is: test.cpp:9:8: note: void VectorP::dot_product(VectorP&, VectorP&, T&) [with T = std::complex] void dot_product(VectorP& vecA, VectorP& vecB, T& prod) ^ test.cpp:9:8: note: no known conversion for argument 3 from 'cusp::complex' to 'std::complex? NOTE: PETSc is compiled with CUSP. Thanks, Harshad >Are we supposed to guess where this happens? Please send the entire compile error, and if its in your code, send >a simple code that demonstrates the problem. >Matt >-- >What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >-- Norbert Wiener From knepley at gmail.com Thu Sep 5 12:38:04 2013 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 5 Sep 2013 12:38:04 -0500 Subject: [petsc-users] cusp complex and std complex In-Reply-To: <1140734881.99501.1378402297661.JavaMail.root@mailhub027.itcs.purdue.edu> References: <2079042502.99468.1378402178739.JavaMail.root@mailhub027.itcs.purdue.edu> <1140734881.99501.1378402297661.JavaMail.root@mailhub027.itcs.purdue.edu> Message-ID: On Thu, Sep 5, 2013 at 12:31 PM, Harshad Sahasrabudhe wrote: > Sorry the last line of the error should read like this: > > test.cpp:9:8: note: no known conversion for argument 3 from > 'cusp::complex' to 'std::complex&' > > Didn't copy it properly the last time. > > Harshad > > ----- Original Message ----- > From: "Harshad Sahasrabudhe" > To: "Matthew Knepley" > Cc: petsc-users at mcs.anl.gov > Sent: Thursday, September 5, 2013 1:29:38 PM > Subject: Re: [petsc-users] cusp complex and std complex > > Hi Matt, > > I have attached a small test case which reproduces the problem. Here is > the compilation command with the error: > > > gcc -c test.cpp -I/u/sciteam/sahasrab/NEMO5/libs/petsc/include > -I/u/sciteam/sahasrab/NEMO5/libs/petsc/arch-linux2-cxx-opt-cplx/include/ > -I/opt/cray/mpt/6.0.1/gni/mpich2- > cray/81/include/ > -I/u/sciteam/sahasrab/NEMO5/libs/petsc/cusplibrary-master/ > -I/opt/nvidia/cudatoolkit/5.0.35/include/ > > test.cpp: In function 'int main()': > test.cpp:24:27: error: no matching function for call to > 'VectorP >::dot_product(VectorP > >&, VectorP >&, > cusp::complex&)' > a.dot_product(a, b, p[1]); > ^ > test.cpp:24:27: note: candidate is: > test.cpp:9:8: note: void VectorP::dot_product(VectorP&, > VectorP&, T&) [with T = std::complex] > void dot_product(VectorP& vecA, VectorP& vecB, T& prod) > ^ > test.cpp:9:8: note: no known conversion for argument 3 from > 'cusp::complex' to 'std::complex > My question is, is there no alternative to using VectorP? > I am not sure what you want. This is a problem with your own type definitions. If you want a dot_product that returns PetscScalar, define the overloaded function. If that is a pain, consider using C++. Personally, I would counsel anyone writing a new code to abandon C++. Matt > NOTE: PETSc is compiled with CUSP. > > Thanks, > Harshad > > > > > >Are we supposed to guess where this happens? Please send the entire > compile error, and if its in your code, send > >a simple code that demonstrates the problem. > > > >Matt > > >-- > >What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > >-- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jnl at jennifernicolelee.com Thu Sep 5 13:44:51 2013 From: jnl at jennifernicolelee.com (Jennifer Nicole Lee) Date: Thu, 5 Sep 2013 14:44:51 -0400 Subject: [petsc-users] Buy 1 Get 1 Free! JNL Clothing Flash Sale! Buy 1 Get 1 When You Order Over $75 Worth of JNL Clothing! Sale Ends at 8 EST Message-ID: <6C.EF.22484.321D8225@ec2> BUY 1 GET 1 FREE! Never Done Before & Will Never Be Repeated! Buy 1 & Get 1 Free of the exact same item you ordered at www.JNLClothing.com [ http://clicks.aweber.com/y/ct/?l=MR6Fo&m=JORR_t5sAIUBKv&b=3MZEjiKoXGnyk_nMR9Nf.Q ] when you purchase over $75! If you don't order over $75, then you will only receive one of the items you ordered! HURRY! Flash Sales Ends at 8:00 PM EST! CLICK HERE FOR FAVORITE V-NECKS: http://clicks.aweber.com/y/ct/?l=MR6Fo&m=JORR_t5sAIUBKv&b=VQ4BTjENqX4izxNW33ZXDg CLICK HERE FOR FAVORITE "Strong is the New Skinny" SHIRTS: http://clicks.aweber.com/y/ct/?l=MR6Fo&m=JORR_t5sAIUBKv&b=VQ4BTjENqX4izxNW33ZXDg Click here for Your Favorite Super Comfy T-shirt Dresses: http://clicks.aweber.com/y/ct/?l=MR6Fo&m=JORR_t5sAIUBKv&b=.7gpcd2wKqaoj18RkTAvPA Click here for your favorite Hoodies: http://clicks.aweber.com/y/ct/?l=MR6Fo&m=JORR_t5sAIUBKv&b=svUakacK9P9yVZWlwUCjJA REMEMBER TO STOCK UP! WHATEVER YOU ORDER, YOU WILL GET 2 DELIVERED TO YOU IF YOUR ORDER IS OVER $75! Sale Ends at 8:00 PM EST TONIGHT! ORDER NOW AT www.JNLClothing.com [ http://clicks.aweber.com/y/ct/?l=MR6Fo&m=JORR_t5sAIUBKv&b=3MZEjiKoXGnyk_nMR9Nf.Q ] NEW JNL CLOTHING FALL COLLECTION COMING SOON! 6619 S. Dixie Hwy., #178, Miami, FL 33143, USA To unsubscribe or change subscriber options visit: http://www.aweber.com/z/r/?TJzsbGwctKxsLGyMDGyMtEa0TAwMbBwMHCw= -------------- next part -------------- An HTML attachment was scrubbed... URL: From frtr at fysik.dtu.dk Fri Sep 6 04:51:27 2013 From: frtr at fysik.dtu.dk (Frederik Treue) Date: Fri, 6 Sep 2013 11:51:27 +0200 Subject: [petsc-users] zero pattern of result of matmatmult In-Reply-To: <87ioyf6ys8.fsf@mcs.anl.gov> References: <1378396034.2282.15.camel@frtr-laptop> <87ioyf6ys8.fsf@mcs.anl.gov> Message-ID: <1378461087.13721.10.camel@frtr-laptop> Resending to mail-list, sorry about the double post, Jed. PS. is there a reason why the listserver doesn't set the Reply-To header field? /Frederik Treue On Thu, 2013-09-05 at 09:03 -0700, Jed Brown wrote: > Frederik Treue writes: > > > Is it somehow possible to restrict the nonzero pattern of the result of > > matmatmult, ie. can I get petsc to only consider a subset of the > > elements of the result matrix, if I know the rest are going to be zero? > > If I understand correctly, you don't want to allocate for entries that > are *computed* to be 0.0 in the product matrix? Is this a real > scenario? Yes, exactly - It's a mathematical identity that most of the entries in the product matrix is zero. I can't specify my operators beforehand, as they depend on the fields that I'm solving for. > Multiplying difference operators together is really brittle > (boundary conditions, geometry, etc) and uses more memory than > necessary. Why? Ofcourse, it could go wrong if you get the non-zero pattern of the product matrix wrong, but that's the same as with all other instances of SAME_NONZERO_PATTERN throughout petsc. Why is this one special? Anyway, is there, as an alternative, a mat copy that doesn't reallocate the matrix? If I use the standard MatCopy, the stencil size of the resulting matrix seems to be the same as the original, regardless of how I've allocated the target matrix. /Frederik Treue From Wadud.Miah at awe.co.uk Fri Sep 6 05:00:20 2013 From: Wadud.Miah at awe.co.uk (Wadud.Miah at awe.co.uk) Date: Fri, 6 Sep 2013 10:00:20 +0000 Subject: [petsc-users] -no_signal_handler environment variable Message-ID: <201309061000.r86A0PfO013101@msw2.awe.co.uk> Hello, Is there an environment variable that has the same effect as the -no_signal_handler switch? Regards, -------------------------- Wadud Miah HPC, Design and Theoretical Physics Direct: 0118 98 56220 AWE, Aldermaston, Reading, RG7 4PR ___________________________________________________ ____________________________ The information in this email and in any attachment(s) is commercial in confidence. If you are not the named addressee(s) or if you receive this email in error then any distribution, copying or use of this communication or the information in it is strictly prohibited. Please notify us immediately by email at admin.internet(at)awe.co.uk, and then delete this message from your computer. While attachments are virus checked, AWE plc does not accept any liability in respect of any virus which is not detected. AWE Plc Registered in England and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Sep 6 05:16:53 2013 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 6 Sep 2013 05:16:53 -0500 Subject: [petsc-users] -no_signal_handler environment variable In-Reply-To: <201309061000.r86A0PfO013101@msw2.awe.co.uk> References: <201309061000.r86A0PfO013101@msw2.awe.co.uk> Message-ID: On Fri, Sep 6, 2013 at 5:00 AM, wrote: > ****** > > Hello,**** > > ** ** > > Is there an environment variable that has the same effect as the > -no_signal_handler switch? > PETSC_OPTIONS=-no_signal_handler Matt > > > Regards,**** > > ** ** > > *--------------------------***** > > *Wadud Miah* > *HPC, Design and Theoretical Physics** > *Direct: 0118 98 56220 > AWE, Aldermaston, ****Reading**, ** RG7 4PR******** > > **** > > ** ** > > ___________________________________________________ > ____________________________ The information in this email and in any > attachment(s) is commercial in confidence. If you are not the named > addressee(s) or if you receive this email in error then any distribution, > copying or use of this communication or the information in it is strictly > prohibited. Please notify us immediately by email at admin.internet(at) > awe.co.uk, and then delete this message from your computer. While > attachments are virus checked, AWE plc does not accept any liability in > respect of any virus which is not detected. AWE Plc Registered in England > and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Sep 6 08:36:26 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 06 Sep 2013 08:36:26 -0500 Subject: [petsc-users] zero pattern of result of matmatmult In-Reply-To: <1378461087.13721.10.camel@frtr-laptop> References: <1378396034.2282.15.camel@frtr-laptop> <87ioyf6ys8.fsf@mcs.anl.gov> <1378461087.13721.10.camel@frtr-laptop> Message-ID: <87ppsm9in9.fsf@mcs.anl.gov> Frederik Treue writes: > Resending to mail-list, sorry about the double post, Jed. > > PS. is there a reason why the listserver doesn't set the Reply-To header field? http://mail-archive.com/search?l=mid&q=8738uog8i3.fsf at mcs.anl.gov Just configure your mailer to properly do group replies. >> Multiplying difference operators together is really brittle >> (boundary conditions, geometry, etc) and uses more memory than >> necessary. > > Why? Ofcourse, it could go wrong if you get the non-zero pattern of the > product matrix wrong, but that's the same as with all other instances of > SAME_NONZERO_PATTERN throughout petsc. Why is this one special? The symbolic part of the matrix multiply does not use entries, and ensures that the numeric part has no memory allocation and no conditionals. Implementing boundary conditions and variable mesh spacing by splitting into first order operators is quite clumsy and a lot less flexible when you want to change the discretization. I think your approach is a false economy and you should just write the local routines that evaluate the actual Jacobian. You can factor those functions however you like (e.g., finite element and finite volume would look a bit different). > Anyway, is there, as an alternative, a mat copy that doesn't reallocate > the matrix? If I use the standard MatCopy, the stencil size of the > resulting matrix seems to be the same as the original, regardless of how > I've allocated the target matrix. See MatChop() -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From u.tabak at tudelft.nl Fri Sep 6 10:48:45 2013 From: u.tabak at tudelft.nl (Umut Tabak) Date: Fri, 06 Sep 2013 17:48:45 +0200 Subject: [petsc-users] confusion on the use of fieldsplit Message-ID: <5229F95D.1010604@tudelft.nl> Dear all, I am trying to use PETSc to solve linear systems where the operator matrices are composed of blocks. But I am having hard time to set up the PCFieldSplit, and a bit lost in the documentation. I sucessfully created the matrix represented as A = A11 A12 A21 A22 with MatCreateNest(PETSC_COMM_WORLD,nr,NULL,nc,NULL,sub_matrices,&blockA); but I could not understand really the following steps on how to associate a KSP object with Field split properties to use different conditioners on different blocks? In the mean time, continue on reading. If you can outline the basic steps on how to construct this scheme and submit analysis with basic command line options, that would of great help. By the way, is not possible to output blockA that is assembled above on the standard output? Best regards, Umut From knepley at gmail.com Fri Sep 6 11:16:13 2013 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 6 Sep 2013 11:16:13 -0500 Subject: [petsc-users] confusion on the use of fieldsplit In-Reply-To: <5229F95D.1010604@tudelft.nl> References: <5229F95D.1010604@tudelft.nl> Message-ID: On Fri, Sep 6, 2013 at 10:48 AM, Umut Tabak wrote: > Dear all, > > I am trying to use PETSc to solve linear systems where the operator > matrices are composed of blocks. But I am having hard time to set up the > PCFieldSplit, and a bit lost in the documentation. I sucessfully created > the matrix represented as > > A > = > A11 A12 > A21 A22 > > with > > MatCreateNest(PETSC_COMM_**WORLD,nr,NULL,nc,NULL,sub_**matrices,&blockA); > 1) Forget about MatNest. It is completely orthogonal to FieldSplit. Its just an optimization. > but I could not understand really the following steps on how to associate > a KSP object with Field split properties to use different conditioners on > different blocks? In the mean time, continue on reading. If you can outline > the basic steps on how to construct this scheme and submit analysis with > basic command line options, that would of great help. > 2) If you have the ISes to make a MatNest, just use them to make the FS: http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html so you have PCFieldSplitSetIS(pc, "1", is1); PCFieldSplitSetIS(pc, "2", is2); Now you can use all the fieldsplit options. There are many examples in PETSc, see src/snes/examples/tutorials/makefile and ex19 for instance. Matt > By the way, is not possible to output blockA that is assembled above on > the standard output? > > Best regards, > Umut > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From Lukasz.Kaczmarczyk at glasgow.ac.uk Fri Sep 6 16:25:28 2013 From: Lukasz.Kaczmarczyk at glasgow.ac.uk (Lukasz Kaczmarczyk) Date: Fri, 6 Sep 2013 22:25:28 +0100 Subject: [petsc-users] converting parallel matrix MATMPIJ to MATAIJ Message-ID: Hello, I solve system of eq. generated by finite element method. I apply some projection matrix to stiffness matrix K, P=I-CT[(CTC)^-1]C where C is some not square matrix. Resulting stiffness matrix K' has form K' = PT K P, with that at hand I solve problem K' *x = f' I manage to build shell matrix where I use sub ksp solver to get solution for (CTC)*b = C*x, where [ b = (CTC^-1*C*x)] . Using penalised matrix for preconditioner, where K_prec = alpha*CCT + K, where alpha is penalty I can get solution in efficient way. Now I like to avoid penalty parameter, in order to do that I will need to apply penalty matrix for each individual finite element matrix before it is assembled into K. No problem with that, using scattering it can be done. Problem is with solution (CTC)*b = C*x, C and CTC matrices are parallel, since I have parallelised assembly functions, problem (CTC)*b = C*x need to be solved on each processor independently without communication. It is not problem, but to do that I need to transform C and CTC matrix form MATMPIAIJ to MATAIJ. I know that MatConvert will not do it. I wonder it is any other way that form very beginning to assemble matrix C as a serial matrix. Regards, Lukasz From knepley at gmail.com Fri Sep 6 16:52:13 2013 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 6 Sep 2013 16:52:13 -0500 Subject: [petsc-users] converting parallel matrix MATMPIJ to MATAIJ In-Reply-To: References: Message-ID: On Fri, Sep 6, 2013 at 4:25 PM, Lukasz Kaczmarczyk < Lukasz.Kaczmarczyk at glasgow.ac.uk> wrote: > Hello, > > I solve system of eq. generated by finite element method. > > I apply some projection matrix to stiffness matrix K, > P=I-CT[(CTC)^-1]C > where C is some not square matrix. > > Resulting stiffness matrix K' has form > K' = PT K P, > with that at hand I solve problem K' *x = f' > > I manage to build shell matrix where I use sub ksp solver to get solution > for (CTC)*b = C*x, where [ b = (CTC^-1*C*x)] . Using penalised matrix for > preconditioner, where K_prec = alpha*CCT + K, where alpha is penalty I can > get solution in efficient way. > > Now I like to avoid penalty parameter, in order to do that I will need to > apply penalty matrix for each individual finite element matrix before it > is assembled into K. No problem with that, using scattering it can be done. > > Problem is with solution (CTC)*b = C*x, C and CTC matrices are parallel, > since I have parallelised assembly functions, problem (CTC)*b = C*x need to > be solved on each processor independently without communication. It is not > problem, but to do that I need to transform C and CTC matrix form MATMPIAIJ > to MATAIJ. > I think this might be what you want: http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetSubMatrices.html Thanks, Matt > I know that MatConvert will not do it. I wonder it is any other way that > form very beginning to assemble matrix C as a serial matrix. > > Regards, > Lukasz > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Sep 6 17:00:09 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 6 Sep 2013 17:00:09 -0500 Subject: [petsc-users] converting parallel matrix MATMPIJ to MATAIJ In-Reply-To: References: Message-ID: <9C361631-C61D-4B66-8E4E-01FEEAF67401@mcs.anl.gov> I would use MatGetRedundantMatrix() On Sep 6, 2013, at 4:52 PM, Matthew Knepley wrote: > On Fri, Sep 6, 2013 at 4:25 PM, Lukasz Kaczmarczyk wrote: > Hello, > > I solve system of eq. generated by finite element method. > > I apply some projection matrix to stiffness matrix K, > P=I-CT[(CTC)^-1]C > where C is some not square matrix. > > Resulting stiffness matrix K' has form > K' = PT K P, > with that at hand I solve problem K' *x = f' > > I manage to build shell matrix where I use sub ksp solver to get solution for (CTC)*b = C*x, where [ b = (CTC^-1*C*x)] . Using penalised matrix for preconditioner, where K_prec = alpha*CCT + K, where alpha is penalty I can get solution in efficient way. > > Now I like to avoid penalty parameter, in order to do that I will need to apply penalty matrix for each individual finite element matrix before it is assembled into K. No problem with that, using scattering it can be done. > > Problem is with solution (CTC)*b = C*x, C and CTC matrices are parallel, since I have parallelised assembly functions, problem (CTC)*b = C*x need to be solved on each processor independently without communication. It is not problem, but to do that I need to transform C and CTC matrix form MATMPIAIJ to MATAIJ. > > I think this might be what you want: > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetSubMatrices.html > > Thanks, > > Matt > > I know that MatConvert will not do it. I wonder it is any other way that form very beginning to assemble matrix C as a serial matrix. > > Regards, > Lukasz > > > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From Lukasz.Kaczmarczyk at glasgow.ac.uk Fri Sep 6 17:32:44 2013 From: Lukasz.Kaczmarczyk at glasgow.ac.uk (Lukasz Kaczmarczyk) Date: Fri, 6 Sep 2013 23:32:44 +0100 Subject: [petsc-users] converting parallel matrix MATMPIJ to MATAIJ In-Reply-To: <9C361631-C61D-4B66-8E4E-01FEEAF67401@mcs.anl.gov> References: <9C361631-C61D-4B66-8E4E-01FEEAF67401@mcs.anl.gov> Message-ID: <4F264DCC-E6BF-4F68-821E-D369B2AF2FD8@glasgow.ac.uk> On 6 Sep 2013, at 23:00, Barry Smith wrote: > > I would use MatGetRedundantMatrix() > > > On Sep 6, 2013, at 4:52 PM, Matthew Knepley wrote: > >> On Fri, Sep 6, 2013 at 4:25 PM, Lukasz Kaczmarczyk wrote: >> Hello, >> >> I solve system of eq. generated by finite element method. >> >> I apply some projection matrix to stiffness matrix K, >> P=I-CT[(CTC)^-1]C >> where C is some not square matrix. >> >> Resulting stiffness matrix K' has form >> K' = PT K P, >> with that at hand I solve problem K' *x = f' >> >> I manage to build shell matrix where I use sub ksp solver to get solution for (CTC)*b = C*x, where [ b = (CTC^-1*C*x)] . Using penalised matrix for preconditioner, where K_prec = alpha*CCT + K, where alpha is penalty I can get solution in efficient way. >> >> Now I like to avoid penalty parameter, in order to do that I will need to apply penalty matrix for each individual finite element matrix before it is assembled into K. No problem with that, using scattering it can be done. >> >> Problem is with solution (CTC)*b = C*x, C and CTC matrices are parallel, since I have parallelised assembly functions, problem (CTC)*b = C*x need to be solved on each processor independently without communication. It is not problem, but to do that I need to transform C and CTC matrix form MATMPIAIJ to MATAIJ. >> >> I think this might be what you want: >> >> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetSubMatrices.html >> >> Thanks, >> >> Matt >> >> I know that MatConvert will not do it. I wonder it is any other way that form very beginning to assemble matrix C as a serial matrix. >> >> Regards, >> Lukasz >> >> >> >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener > Thanks Matt and Barry, Sorry for taking your time, I should find that by myself. How to use MatGetSubMatrices is clear. I have feeling that MatGetRedundantMatrix could be more efficient. However, apologise my ignorance, I have problem with MatGetRedundantMatrix, *) is this should be equal to number of process in the communicator group, in my case 1 since I like to use PETSC_COMM_SELF? nsubcomm - the number of subcommunicators (= number of redundant parallel or sequential matrices) subcomm - MPI communicator split from the communicator where mat resides in *) it could be equal to total number of rows in MPIAIJ? What if this number is smaller, the first mlocal_red are stored in redundant matrix? mlocal_red - number of local rows of the redundant matrix Regards, Lukasz From bsmith at mcs.anl.gov Fri Sep 6 17:37:36 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 6 Sep 2013 17:37:36 -0500 Subject: [petsc-users] converting parallel matrix MATMPIJ to MATAIJ In-Reply-To: <4F264DCC-E6BF-4F68-821E-D369B2AF2FD8@glasgow.ac.uk> References: <9C361631-C61D-4B66-8E4E-01FEEAF67401@mcs.anl.gov> <4F264DCC-E6BF-4F68-821E-D369B2AF2FD8@glasgow.ac.uk> Message-ID: <57BA51FE-8371-4086-BB23-3B4567E3C9C9@mcs.anl.gov> MPI_Comm_size(PETSC_COMM_WORLD,&Np); Use Np and MPI_COMM_NULL. Note that you should be using PETSc from https://bitbucket.org/petsc/petsc Barry Yes, the phrasing is a little confusing. On Sep 6, 2013, at 5:32 PM, Lukasz Kaczmarczyk wrote: > On 6 Sep 2013, at 23:00, Barry Smith wrote: > >> >> I would use MatGetRedundantMatrix() >> >> >> On Sep 6, 2013, at 4:52 PM, Matthew Knepley wrote: >> >>> On Fri, Sep 6, 2013 at 4:25 PM, Lukasz Kaczmarczyk wrote: >>> Hello, >>> >>> I solve system of eq. generated by finite element method. >>> >>> I apply some projection matrix to stiffness matrix K, >>> P=I-CT[(CTC)^-1]C >>> where C is some not square matrix. >>> >>> Resulting stiffness matrix K' has form >>> K' = PT K P, >>> with that at hand I solve problem K' *x = f' >>> >>> I manage to build shell matrix where I use sub ksp solver to get solution for (CTC)*b = C*x, where [ b = (CTC^-1*C*x)] . Using penalised matrix for preconditioner, where K_prec = alpha*CCT + K, where alpha is penalty I can get solution in efficient way. >>> >>> Now I like to avoid penalty parameter, in order to do that I will need to apply penalty matrix for each individual finite element matrix before it is assembled into K. No problem with that, using scattering it can be done. >>> >>> Problem is with solution (CTC)*b = C*x, C and CTC matrices are parallel, since I have parallelised assembly functions, problem (CTC)*b = C*x need to be solved on each processor independently without communication. It is not problem, but to do that I need to transform C and CTC matrix form MATMPIAIJ to MATAIJ. >>> >>> I think this might be what you want: >>> >>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetSubMatrices.html >>> >>> Thanks, >>> >>> Matt >>> >>> I know that MatConvert will not do it. I wonder it is any other way that form very beginning to assemble matrix C as a serial matrix. >>> >>> Regards, >>> Lukasz >>> >>> >>> >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >> > > Thanks Matt and Barry, > > Sorry for taking your time, I should find that by myself. > How to use MatGetSubMatrices is clear. > > I have feeling that MatGetRedundantMatrix could be more efficient. However, apologise my ignorance, I have problem with MatGetRedundantMatrix, > > *) is this should be equal to number of process in the communicator group, in my case 1 since I like to use PETSC_COMM_SELF? > nsubcomm - the number of subcommunicators (= number of redundant parallel or sequential matrices) > subcomm - MPI communicator split from the communicator where mat resides in > > *) it could be equal to total number of rows in MPIAIJ? What if this number is smaller, the first mlocal_red are stored in redundant matrix? > mlocal_red - number of local rows of the redundant matrix > > > Regards, > Lukasz > > From Lukasz.Kaczmarczyk at glasgow.ac.uk Fri Sep 6 17:44:48 2013 From: Lukasz.Kaczmarczyk at glasgow.ac.uk (Lukasz Kaczmarczyk) Date: Fri, 6 Sep 2013 23:44:48 +0100 Subject: [petsc-users] converting parallel matrix MATMPIJ to MATAIJ In-Reply-To: <57BA51FE-8371-4086-BB23-3B4567E3C9C9@mcs.anl.gov> References: <9C361631-C61D-4B66-8E4E-01FEEAF67401@mcs.anl.gov> <4F264DCC-E6BF-4F68-821E-D369B2AF2FD8@glasgow.ac.uk> <57BA51FE-8371-4086-BB23-3B4567E3C9C9@mcs.anl.gov> Message-ID: Thanks, for very fast help, Last qestion, how I should interpret, PetscInt mlocal_red, i.e. number of local rows of the redundant matrix, what if mlocal_red is smaller than total number or rows in mat. Lukasz On 6 Sep 2013, at 23:37, Barry Smith wrote: > > > MPI_Comm_size(PETSC_COMM_WORLD,&Np); > > Use Np and MPI_COMM_NULL. Note that you should be using PETSc from https://bitbucket.org/petsc/petsc > > Barry > > Yes, the phrasing is a little confusing. > > On Sep 6, 2013, at 5:32 PM, Lukasz Kaczmarczyk wrote: > >> On 6 Sep 2013, at 23:00, Barry Smith wrote: >> >>> >>> I would use MatGetRedundantMatrix() >>> >>> >>> On Sep 6, 2013, at 4:52 PM, Matthew Knepley wrote: >>> >>>> On Fri, Sep 6, 2013 at 4:25 PM, Lukasz Kaczmarczyk wrote: >>>> Hello, >>>> >>>> I solve system of eq. generated by finite element method. >>>> >>>> I apply some projection matrix to stiffness matrix K, >>>> P=I-CT[(CTC)^-1]C >>>> where C is some not square matrix. >>>> >>>> Resulting stiffness matrix K' has form >>>> K' = PT K P, >>>> with that at hand I solve problem K' *x = f' >>>> >>>> I manage to build shell matrix where I use sub ksp solver to get solution for (CTC)*b = C*x, where [ b = (CTC^-1*C*x)] . Using penalised matrix for preconditioner, where K_prec = alpha*CCT + K, where alpha is penalty I can get solution in efficient way. >>>> >>>> Now I like to avoid penalty parameter, in order to do that I will need to apply penalty matrix for each individual finite element matrix before it is assembled into K. No problem with that, using scattering it can be done. >>>> >>>> Problem is with solution (CTC)*b = C*x, C and CTC matrices are parallel, since I have parallelised assembly functions, problem (CTC)*b = C*x need to be solved on each processor independently without communication. It is not problem, but to do that I need to transform C and CTC matrix form MATMPIAIJ to MATAIJ. >>>> >>>> I think this might be what you want: >>>> >>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetSubMatrices.html >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> I know that MatConvert will not do it. I wonder it is any other way that form very beginning to assemble matrix C as a serial matrix. >>>> >>>> Regards, >>>> Lukasz >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>> -- Norbert Wiener >>> >> >> Thanks Matt and Barry, >> >> Sorry for taking your time, I should find that by myself. >> How to use MatGetSubMatrices is clear. >> >> I have feeling that MatGetRedundantMatrix could be more efficient. However, apologise my ignorance, I have problem with MatGetRedundantMatrix, >> >> *) is this should be equal to number of process in the communicator group, in my case 1 since I like to use PETSC_COMM_SELF? >> nsubcomm - the number of subcommunicators (= number of redundant parallel or sequential matrices) >> subcomm - MPI communicator split from the communicator where mat resides in >> >> *) it could be equal to total number of rows in MPIAIJ? What if this number is smaller, the first mlocal_red are stored in redundant matrix? >> mlocal_red - number of local rows of the redundant matrix >> >> >> Regards, >> Lukasz >> >> > From bsmith at mcs.anl.gov Fri Sep 6 19:59:51 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 6 Sep 2013 19:59:51 -0500 Subject: [petsc-users] converting parallel matrix MATMPIJ to MATAIJ In-Reply-To: References: <9C361631-C61D-4B66-8E4E-01FEEAF67401@mcs.anl.gov> <4F264DCC-E6BF-4F68-821E-D369B2AF2FD8@glasgow.ac.uk> <57BA51FE-8371-4086-BB23-3B4567E3C9C9@mcs.anl.gov> Message-ID: <8933C957-DF09-4063-9C14-B4C730EBA42A@mcs.anl.gov> It is gone. Use the development version of petsc On Sep 6, 2013, at 5:44 PM, Lukasz Kaczmarczyk wrote: > Thanks, for very fast help, > > Last qestion, how I should interpret, > PetscInt mlocal_red, i.e. number of local rows of the redundant matrix, > what if mlocal_red is smaller than total number or rows in mat. > > Lukasz > > On 6 Sep 2013, at 23:37, Barry Smith wrote: > >> >> >> MPI_Comm_size(PETSC_COMM_WORLD,&Np); >> >> Use Np and MPI_COMM_NULL. Note that you should be using PETSc from https://bitbucket.org/petsc/petsc >> >> Barry >> >> Yes, the phrasing is a little confusing. >> >> On Sep 6, 2013, at 5:32 PM, Lukasz Kaczmarczyk wrote: >> >>> On 6 Sep 2013, at 23:00, Barry Smith wrote: >>> >>>> >>>> I would use MatGetRedundantMatrix() >>>> >>>> >>>> On Sep 6, 2013, at 4:52 PM, Matthew Knepley wrote: >>>> >>>>> On Fri, Sep 6, 2013 at 4:25 PM, Lukasz Kaczmarczyk wrote: >>>>> Hello, >>>>> >>>>> I solve system of eq. generated by finite element method. >>>>> >>>>> I apply some projection matrix to stiffness matrix K, >>>>> P=I-CT[(CTC)^-1]C >>>>> where C is some not square matrix. >>>>> >>>>> Resulting stiffness matrix K' has form >>>>> K' = PT K P, >>>>> with that at hand I solve problem K' *x = f' >>>>> >>>>> I manage to build shell matrix where I use sub ksp solver to get solution for (CTC)*b = C*x, where [ b = (CTC^-1*C*x)] . Using penalised matrix for preconditioner, where K_prec = alpha*CCT + K, where alpha is penalty I can get solution in efficient way. >>>>> >>>>> Now I like to avoid penalty parameter, in order to do that I will need to apply penalty matrix for each individual finite element matrix before it is assembled into K. No problem with that, using scattering it can be done. >>>>> >>>>> Problem is with solution (CTC)*b = C*x, C and CTC matrices are parallel, since I have parallelised assembly functions, problem (CTC)*b = C*x need to be solved on each processor independently without communication. It is not problem, but to do that I need to transform C and CTC matrix form MATMPIAIJ to MATAIJ. >>>>> >>>>> I think this might be what you want: >>>>> >>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetSubMatrices.html >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> I know that MatConvert will not do it. I wonder it is any other way that form very beginning to assemble matrix C as a serial matrix. >>>>> >>>>> Regards, >>>>> Lukasz >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>> -- Norbert Wiener >>>> >>> >>> Thanks Matt and Barry, >>> >>> Sorry for taking your time, I should find that by myself. >>> How to use MatGetSubMatrices is clear. >>> >>> I have feeling that MatGetRedundantMatrix could be more efficient. However, apologise my ignorance, I have problem with MatGetRedundantMatrix, >>> >>> *) is this should be equal to number of process in the communicator group, in my case 1 since I like to use PETSC_COMM_SELF? >>> nsubcomm - the number of subcommunicators (= number of redundant parallel or sequential matrices) >>> subcomm - MPI communicator split from the communicator where mat resides in >>> >>> *) it could be equal to total number of rows in MPIAIJ? What if this number is smaller, the first mlocal_red are stored in redundant matrix? >>> mlocal_red - number of local rows of the redundant matrix >>> >>> >>> Regards, >>> Lukasz >>> >>> >> > From u.tabak at tudelft.nl Sat Sep 7 04:59:20 2013 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sat, 07 Sep 2013 11:59:20 +0200 Subject: [petsc-users] confusion on the use of fieldsplit In-Reply-To: References: <5229F95D.1010604@tudelft.nl> Message-ID: <522AF8F8.5000308@tudelft.nl> On 09/06/2013 06:16 PM, Matthew Knepley wrote: > > > 1) Forget about MatNest. It is completely orthogonal to FieldSplit. > Its just an optimization. Hi Matt, Thanks, slowly getting there. Ok I built up a simple matrix in MATLAB and interfaced it to PETSc what I would like to try to try with PETSc at the end is a similar system with a similar operator like A = A C1 C2 B as mentioned in my previous mail. I also found out your tutorial www.mcs.anl.gov/*petsc*/.../*ACTS*Tutorial2012.pdf? and there are some examples on the usage of Stokes problems. That is exactly what I want to try for my problems, setting different preconditioners and solvers for different blocks(I guess these are denoted as splits, correct me if I am wrong). I investigated that presentation but I still have a problem somewhere. The simple code is also attached with a simple matrix and the right hand side vector. I have a problem on the field numbers. What I defined was the row index sets to set the fields but apparently the error I get is telling me that I still have a single field when I try to run ./ex2 -pc_type fieldsplit > simple_log And the output is [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Petsc has generated inconsistent data! [0]PETSC ERROR: Unhandled case, must have at least two fields, not 1! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./ex2 on a linux-gnu-c-debug named hp8570w by utabak Sat Sep 7 11:58:25 2013 [0]PETSC ERROR: Libraries linked from /home/utabak/external_libraries/petsc-3.4.2/linux-gnu-c-debug/lib [0]PETSC ERROR: Configure run at Fri Sep 6 14:18:08 2013 [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --download-f-blas-lapack=1 --with-mpi=1 --with-clanguage=cxx --download-mumps=1 --download-scalapack=1 --download-blacs=1 --download-superlu=1 --download-spooles=1 --download-umfpack=1 --download-mpich=1 --download-metis=1 --download-parmetis=1 -download-hypre=1 COPTFLAGS=-O3 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: PCFieldSplitSetDefaults() line 440 in /home/utabak/external_libraries/petsc-3.4.2/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: PCSetUp_FieldSplit() line 458 in /home/utabak/external_libraries/petsc-3.4.2/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: PCSetUp() line 890 in /home/utabak/external_libraries/petsc-3.4.2/src/ksp/pc/interface/precon.c [0]PETSC ERROR: KSPSetUp() line 278 in /home/utabak/external_libraries/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: KSPSolve() line 399 in /home/utabak/external_libraries/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c How can I correct this error? Could you please help me at this point? Best, Umut > but I could not understand really the following steps on how to > associate a KSP object with Field split properties to use > different conditioners on different blocks? In the mean time, > continue on reading. If you can outline the basic steps on how to > construct this scheme and submit analysis with basic command line > options, that would of great help. > > > 2) If you have the ISes to make a MatNest, just use them to make the FS: > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html > > so you have > > PCFieldSplitSetIS(pc, "1", is1); > PCFieldSplitSetIS(pc, "2", is2); > > Now you can use all the fieldsplit options. There are many examples in > PETSc, see > > src/snes/examples/tutorials/makefile > > and ex19 for instance. > > Matt > > By the way, is not possible to output blockA that is assembled > above on the standard output? > > Best regards, > Umut > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Ablock.bin Type: application/octet-stream Size: 128 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: b.bin Type: application/octet-stream Size: 40 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex2.c Type: text/x-csrc Size: 2656 bytes Desc: not available URL: From jedbrown at mcs.anl.gov Sat Sep 7 13:54:04 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 07 Sep 2013 13:54:04 -0500 Subject: [petsc-users] confusion on the use of fieldsplit In-Reply-To: <522AF8F8.5000308@tudelft.nl> References: <5229F95D.1010604@tudelft.nl> <522AF8F8.5000308@tudelft.nl> Message-ID: <878uz8a2er.fsf@mcs.anl.gov> Umut Tabak writes: > On 09/06/2013 06:16 PM, Matthew Knepley wrote: >> >> >> 1) Forget about MatNest. It is completely orthogonal to FieldSplit. >> Its just an optimization. > Hi Matt, > > Thanks, slowly getting there. > > Ok I built up a simple matrix in MATLAB and interfaced it to PETSc what > I would like to try to try with PETSc at the end is a similar system > with a similar operator like > > A = > A C1 > C2 B > > as mentioned in my previous mail. I also found out your tutorial > > www.mcs.anl.gov/*petsc*/.../*ACTS*Tutorial2012.pdf? > > and there are some examples on the usage of Stokes problems. That is > exactly what I want to try for my problems, setting different > preconditioners and solvers for different blocks(I guess these are > denoted as splits, correct me if I am wrong). > > I investigated that presentation but I still have a problem somewhere. > The simple code is also attached with a simple matrix and the right hand > side vector. I have a problem on the field numbers. > > What I defined was the row index sets to set the fields but apparently > the error I get is telling me that I still have a single field when I > try to run > > ./ex2 -pc_type fieldsplit > simple_log > > And the output is > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Petsc has generated inconsistent data! > [0]PETSC ERROR: Unhandled case, must have at least two fields, not 1! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./ex2 on a linux-gnu-c-debug named hp8570w by utabak Sat > Sep 7 11:58:25 2013 > [0]PETSC ERROR: Libraries linked from > /home/utabak/external_libraries/petsc-3.4.2/linux-gnu-c-debug/lib > [0]PETSC ERROR: Configure run at Fri Sep 6 14:18:08 2013 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran > --with-cxx=g++ --download-f-blas-lapack=1 --with-mpi=1 > --with-clanguage=cxx --download-mumps=1 --download-scalapack=1 > --download-blacs=1 --download-superlu=1 --download-spooles=1 > --download-umfpack=1 --download-mpich=1 --download-metis=1 > --download-parmetis=1 -download-hypre=1 COPTFLAGS=-O3 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: PCFieldSplitSetDefaults() line 440 in > /home/utabak/external_libraries/petsc-3.4.2/src/ksp/pc/impls/fieldsplit/fieldsplit.c > [0]PETSC ERROR: PCSetUp_FieldSplit() line 458 in > /home/utabak/external_libraries/petsc-3.4.2/src/ksp/pc/impls/fieldsplit/fieldsplit.c > [0]PETSC ERROR: PCSetUp() line 890 in > /home/utabak/external_libraries/petsc-3.4.2/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPSetUp() line 278 in > /home/utabak/external_libraries/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: KSPSolve() line 399 in > /home/utabak/external_libraries/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c > > How can I correct this error? You probably needed PCFieldSplitSetIS(). -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From u.tabak at tudelft.nl Sat Sep 7 15:51:51 2013 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sat, 07 Sep 2013 22:51:51 +0200 Subject: [petsc-users] confusion on the use of fieldsplit In-Reply-To: <878uz8a2er.fsf@mcs.anl.gov> References: <5229F95D.1010604@tudelft.nl> <522AF8F8.5000308@tudelft.nl> <878uz8a2er.fsf@mcs.anl.gov> Message-ID: <522B91E7.503@tudelft.nl> On 09/07/2013 08:54 PM, Jed Brown wrote: > Umut Tabak writes: > >> > You probably needed PCFieldSplitSetIS(). Dear Jed, I am using PCFieldSplitSetIS() in the code however, I am not sure whether I am using it at the right location or not. You can see the attached code, ex2.c. BR, Umut From jedbrown at mcs.anl.gov Sat Sep 7 16:00:18 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 07 Sep 2013 16:00:18 -0500 Subject: [petsc-users] confusion on the use of fieldsplit In-Reply-To: <522B91E7.503@tudelft.nl> References: <5229F95D.1010604@tudelft.nl> <522AF8F8.5000308@tudelft.nl> <878uz8a2er.fsf@mcs.anl.gov> <522B91E7.503@tudelft.nl> Message-ID: <8761uc9wkd.fsf@mcs.anl.gov> Umut Tabak writes: > I am using > > PCFieldSplitSetIS() > > in the code however, I am not sure whether I am using it at the right > location or not. > > You can see the attached code, ex2.c. There is no attached code. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From u.tabak at tudelft.nl Sat Sep 7 16:19:37 2013 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sat, 07 Sep 2013 23:19:37 +0200 Subject: [petsc-users] confusion on the use of fieldsplit In-Reply-To: <8761uc9wkd.fsf@mcs.anl.gov> References: <5229F95D.1010604@tudelft.nl> <522AF8F8.5000308@tudelft.nl> <878uz8a2er.fsf@mcs.anl.gov> <522B91E7.503@tudelft.nl> <8761uc9wkd.fsf@mcs.anl.gov> Message-ID: <522B9869.5060803@tudelft.nl> On 09/07/2013 11:00 PM, Jed Brown wrote: > Umut Tabak writes: >> I am using >> >> PCFieldSplitSetIS() >> >> in the code however, I am not sure whether I am using it at the right >> location or not. >> >> You can see the attached code, ex2.c. > There is no attached code. Ok, I meant the original mail, but here it is ;-) Thx. -------------- next part -------------- A non-text attachment was scrubbed... Name: ex2.c Type: text/x-csrc Size: 2656 bytes Desc: not available URL: From jnl at jennifernicolelee.com Sat Sep 7 18:55:23 2013 From: jnl at jennifernicolelee.com (Jennifer Nicole Lee) Date: Sat, 7 Sep 2013 19:55:23 -0400 Subject: [petsc-users] New JNL Podcast! JNL Goes Global! 3 Women from France, Sweden, & Germany Go to London & Miami for JNL World Conf! Message-ID: <85.60.07016.BECBB225@ec1> NEW JNL PODCAST! CLICK HERE TO LISTEN NOW: http://clicks.aweber.com/y/ct/?l=MR6Fo&m=JwQLAf0FwIUBKv&b=o.hdkjPqwzP5tyNAYCd6ZQ Three JNL Fusion Master Trainers from Germany, France & Sweden are all flying to London, England for the JNL UK Fitness Model Factory 1 Day Mega Event! These very same women are also flying to Miami for the JNL World Conference in January! These women speak French Swedish, German, and yes even Spanish! They are all highly sought after Fitness Experts, professionals, and true JNL Fusion Weight loss success stories! These 3 women, have lost a total combined weight of over 125 pounds collectively on the JNL Fusion workout method! Listen now, and you must share this powerful podcast by forwarding this link to your entire email database! It would be a sin not to share this joy and true success break thru's with others! Listen to this podcast and and meet these amazing women! Maria Bystrom of Sweden-A 50 year old who is a Body FX Success Advisor! Maricarmen aka "La Chica" from France-A Cert. Nutrition Coach with a PhD in Physical Chemistry and Carolin of Germany! She lost over 77 pounds with the JNL Fusion workout method, is an IFBB Pro, and a Nutrition Coach! BRACE YOURSELF FOR JNL'S MOST RIVETING AND POWERFUL PODCAST TO DATE! LISTEN NOW! And if you want to also make your dreams come true, apply at www.FitnessModelFactory.com [ http://clicks.aweber.com/y/ct/?l=MR6Fo&m=JwQLAf0FwIUBKv&b=u888pxWam7O99jUnpUX9fQ ] and for the JNL World Conference Jan 17-20th apply at www.JNLWorldwide.net [ http://clicks.aweber.com/y/ct/?l=MR6Fo&m=JwQLAf0FwIUBKv&b=_gN96ez1q029OigjGbZ1LQ ] I believe in you! JNL DONT MISS THE JNL WORLD CONFERENCE! APPLY AT www.JNLWorldwide.net [ http://clicks.aweber.com/y/ct/?l=MR6Fo&m=JwQLAf0FwIUBKv&b=OPwIVZIyFYqkdcPQKQYA0w ] Follow us on Facebook and Twitter 6619 S. Dixie Hwy., #178, Miami, FL 33143, USA To unsubscribe or change subscriber options visit: http://www.aweber.com/z/r/?TJzsbGwctKxsLGyMDGyMtEa0TAwMnCxMTIw= -------------- next part -------------- An HTML attachment was scrubbed... URL: From jnl at jennifernicolelee.com Sat Sep 7 18:57:16 2013 From: jnl at jennifernicolelee.com (Jennifer Nicole Lee) Date: Sat, 7 Sep 2013 19:57:16 -0400 Subject: [petsc-users] MIAMI, ITS ON! JNL's Gonna Make You Sweat at www.SweatUSA. com Fitness Concert! Message-ID: JENNIFER NICOLE LEE & OTHER TOP FITNESS CELEBRITIES AT SWEAT MIAMI OCT 11-13TH! LARGEST FITNESS CELEBRITY FESTIVAL! CLASSES ARE "FITNESS ROCK CONCERTS"! THIS IS NOT EXERCISE, BUT AN EXPERIENCE! NEW JNL PODCAST WITH ALL THE DETAILS ON HOW YOU CAN MINGLE WITH HER, TONY HORTON, SHAUN T INSANITY, JILLIAN MICHAELS & OTHER BIG FITNESS CELEBRITIES ALL UNDER ONE ROOF! Click here to listen now: ?http://clicks.aweber.com/y/ct/?l=MR6Fo&m=J9ivmp1FwIUBKv&b=mbI49upImW61csoccRM3NA? VIP Fitness Celebrity Cocktail Reception Meet & Greet at Private Location Oct 11th-All Fitness Celebrities will be there for your photo opportunities, from 6-8PM! Purchase your ticket at www.SweatUSA.com [ http://clicks.aweber.com/y/ct/?l=MR6Fo&m=J9ivmp1FwIUBKv&b=b9uxWykKxHXpl_mWYEK5ug ] under VIP Cocktail Reception! Oct 12th at the Miami Convention Center at 3:00 and 5:00 JNL will be teaching and instructing on the main stage. Get your tix today at www.SweatUSA.com [ http://clicks.aweber.com/y/ct/?l=MR6Fo&m=J9ivmp1FwIUBKv&b=b9uxWykKxHXpl_mWYEK5ug ] Click on the Miami Link, and then select the $59, $199, or $349 package. Then sign up for JNL's Class! Believe, Team JNL Follow us on Facebook and Twitter 6619 S. Dixie Hwy., #178, Miami, FL 33143, USA To unsubscribe or change subscriber options visit: http://www.aweber.com/z/r/?TJzsbGwctKxsLGyMDGyMtEa0TAwMnCxMTGw= -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sun Sep 8 00:54:49 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 8 Sep 2013 00:54:49 -0500 Subject: [petsc-users] [PATCH 1/2] Close memory leaks In-Reply-To: <522B9869.5060803@tudelft.nl> References: <522B9869.5060803@tudelft.nl> Message-ID: <1378619690-13533-1-git-send-email-jedbrown@mcs.anl.gov> --- ex2.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/ex2.c b/ex2.c index 8762937..82f1d97 100644 --- a/ex2.c +++ b/ex2.c @@ -28,11 +28,13 @@ int main(int argc,char **args) MatCreate(PETSC_COMM_WORLD,&A); MatSetType(A,MATSEQAIJ); MatLoad(A, viewer1); + PetscViewerDestroy(&viewer1); /* ----- */ ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD,"b.bin", FILE_MODE_READ,&viewer2); VecCreate(PETSC_COMM_WORLD,&b); VecLoad(b,viewer2); + PetscViewerDestroy(&viewer2); /* ---- */ MatView(A,PETSC_VIEWER_STDOUT_WORLD); VecView(b,PETSC_VIEWER_STDOUT_WORLD); @@ -54,6 +56,8 @@ int main(int argc,char **args) PCFieldSplitSetIS(pc, "1", is_row1); PCFieldSplitSetIS(pc, "2", is_row2); + ISDestroy(&is_row1); + ISDestroy(&is_row2); KSPSetFromOptions(ksp); -- 1.8.4 From jedbrown at mcs.anl.gov Sun Sep 8 00:54:50 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 8 Sep 2013 00:54:50 -0500 Subject: [petsc-users] [PATCH 2/2] Call KSPSetFromOptions before setting fieldsplit-specific options In-Reply-To: <1378619690-13533-1-git-send-email-jedbrown@mcs.anl.gov> References: <522B9869.5060803@tudelft.nl> <1378619690-13533-1-git-send-email-jedbrown@mcs.anl.gov> Message-ID: <1378619690-13533-2-git-send-email-jedbrown@mcs.anl.gov> PCFieldSplitSetIS is ignored if the PC type is not set to "fieldsplit". The command line option -pc_type is processed in KSPSetFromOptions and any manual fieldsplit-specific configuration must be done afterward. --- ex2.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/ex2.c b/ex2.c index 82f1d97..94cea2c 100644 --- a/ex2.c +++ b/ex2.c @@ -54,13 +54,13 @@ int main(int argc,char **args) KSPGetPC(ksp,&pc); KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN); + KSPSetFromOptions(ksp); + PCFieldSplitSetIS(pc, "1", is_row1); PCFieldSplitSetIS(pc, "2", is_row2); ISDestroy(&is_row1); ISDestroy(&is_row2); - - KSPSetFromOptions(ksp); - + /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Solve the linear system - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */ -- 1.8.4 From u.tabak at tudelft.nl Sun Sep 8 01:22:49 2013 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sun, 08 Sep 2013 08:22:49 +0200 Subject: [petsc-users] [PATCH 2/2] Call KSPSetFromOptions before setting fieldsplit-specific options In-Reply-To: <1378619690-13533-2-git-send-email-jedbrown@mcs.anl.gov> References: <522B9869.5060803@tudelft.nl> <1378619690-13533-1-git-send-email-jedbrown@mcs.anl.gov> <1378619690-13533-2-git-send-email-jedbrown@mcs.anl.gov> Message-ID: <522C17B9.9000307@tudelft.nl> On 09/08/2013 07:54 AM, Jed Brown wrote: > PCFieldSplitSetIS is ignored if the PC type is not set to "fieldsplit". > The command line option -pc_type is processed in KSPSetFromOptions and > any manual fieldsplit-specific configuration must be done afterward. Ok, Thanks. that was the problem, I could run the simple code now. Maybe a naive question but on this simple system the solution is wrong at the moment which is surprising to me, could you see immediately the reason of this? I run with ./ex2 -pc_type fieldsplit And get 0.0909091 0.0909091 0 0 as the solution vector. > --- > ex2.c | 6 +++--- > 1 file changed, 3 insertions(+), 3 deletions(-) > > diff --git a/ex2.c b/ex2.c > index 82f1d97..94cea2c 100644 > --- a/ex2.c > +++ b/ex2.c > @@ -54,13 +54,13 @@ int main(int argc,char **args) > KSPGetPC(ksp,&pc); > KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN); > > + KSPSetFromOptions(ksp); > + > PCFieldSplitSetIS(pc, "1", is_row1); > PCFieldSplitSetIS(pc, "2", is_row2); > ISDestroy(&is_row1); > ISDestroy(&is_row2); > - > - KSPSetFromOptions(ksp); > - > + > /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > Solve the linear system > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */ From jedbrown at mcs.anl.gov Sun Sep 8 01:43:32 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 08 Sep 2013 01:43:32 -0500 Subject: [petsc-users] [PATCH 2/2] Call KSPSetFromOptions before setting fieldsplit-specific options In-Reply-To: <522C17B9.9000307@tudelft.nl> References: <522B9869.5060803@tudelft.nl> <1378619690-13533-1-git-send-email-jedbrown@mcs.anl.gov> <1378619690-13533-2-git-send-email-jedbrown@mcs.anl.gov> <522C17B9.9000307@tudelft.nl> Message-ID: <87mwnn95kb.fsf@mcs.anl.gov> Umut Tabak writes: > Maybe a naive question but on this simple system the solution is wrong > at the moment which is surprising to me, could you see immediately the > reason of this? > > I run with > > ./ex2 -pc_type fieldsplit > > And get > > 0.0909091 > 0.0909091 > 0 > 0 > > as the solution vector. Your splits were both the same, so the preconditioner was applied twice to the first block and zero to the second block. That made the preconditioner singular. ISCreateBlock(PETSC_COMM_SELF,bs,n1,inputindices,PETSC_COPY_VALUES,&is_row1); ISCreateBlock(PETSC_COMM_SELF,bs,n2,inputindices,PETSC_COPY_VALUES,&is_row2); We could check for this case, but it would require a little code and isn't currently done. This is probably what you wanted: diff --git i/ex2.c w/ex2.c index 94cea2c..0cce3e6 100644 --- i/ex2.c +++ w/ex2.c @@ -45,7 +45,7 @@ int main(int argc,char **args) PetscInt bs = 2, n1=1, n2=1, inputindices[]={0,1}; /* block size is 2 */ ISCreateBlock(PETSC_COMM_SELF,bs,n1,inputindices,PETSC_COPY_VALUES,&is_row1); - ISCreateBlock(PETSC_COMM_SELF,bs,n2,inputindices,PETSC_COPY_VALUES,&is_row2); + ISCreateBlock(PETSC_COMM_SELF,bs,n2,inputindices+n1,PETSC_COPY_VALUES,&is_row2); ISView(is_row1,PETSC_VIEWER_STDOUT_SELF); ISView(is_row2,PETSC_VIEWER_STDOUT_SELF); -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From u.tabak at tudelft.nl Sun Sep 8 01:54:55 2013 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sun, 08 Sep 2013 08:54:55 +0200 Subject: [petsc-users] [PATCH 2/2] Call KSPSetFromOptions before setting fieldsplit-specific options In-Reply-To: <87mwnn95kb.fsf@mcs.anl.gov> References: <522B9869.5060803@tudelft.nl> <1378619690-13533-1-git-send-email-jedbrown@mcs.anl.gov> <1378619690-13533-2-git-send-email-jedbrown@mcs.anl.gov> <522C17B9.9000307@tudelft.nl> <87mwnn95kb.fsf@mcs.anl.gov> Message-ID: <522C1F3F.5020009@tudelft.nl> On 09/08/2013 08:43 AM, Jed Brown wrote: > Umut Tabak writes: > > Your splits were both the same, Some questions: + Splits represent different row and column blocks, right? + And as far as I understand from you(maybe this is in the manual, I am not sure) and they should differ in size, right? Thanks for the help. Umut > so the preconditioner was applied twice > to the first block and zero to the second block. That made the > preconditioner singular. > > ISCreateBlock(PETSC_COMM_SELF,bs,n1,inputindices,PETSC_COPY_VALUES,&is_row1); > ISCreateBlock(PETSC_COMM_SELF,bs,n2,inputindices,PETSC_COPY_VALUES,&is_row2); > > We could check for this case, but it would require a little code and > isn't currently done. This is probably what you wanted: > > diff --git i/ex2.c w/ex2.c > index 94cea2c..0cce3e6 100644 > --- i/ex2.c > +++ w/ex2.c > @@ -45,7 +45,7 @@ int main(int argc,char **args) > PetscInt bs = 2, n1=1, n2=1, inputindices[]={0,1}; /* block size is 2 */ > > ISCreateBlock(PETSC_COMM_SELF,bs,n1,inputindices,PETSC_COPY_VALUES,&is_row1); > - ISCreateBlock(PETSC_COMM_SELF,bs,n2,inputindices,PETSC_COPY_VALUES,&is_row2); > + ISCreateBlock(PETSC_COMM_SELF,bs,n2,inputindices+n1,PETSC_COPY_VALUES,&is_row2); > > ISView(is_row1,PETSC_VIEWER_STDOUT_SELF); > ISView(is_row2,PETSC_VIEWER_STDOUT_SELF); From u.tabak at tudelft.nl Sun Sep 8 04:23:51 2013 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sun, 08 Sep 2013 11:23:51 +0200 Subject: [petsc-users] [PATCH 2/2] Call KSPSetFromOptions before setting fieldsplit-specific options In-Reply-To: <87mwnn95kb.fsf@mcs.anl.gov> References: <522B9869.5060803@tudelft.nl> <1378619690-13533-1-git-send-email-jedbrown@mcs.anl.gov> <1378619690-13533-2-git-send-email-jedbrown@mcs.anl.gov> <522C17B9.9000307@tudelft.nl> <87mwnn95kb.fsf@mcs.anl.gov> Message-ID: <522C4227.5050903@tudelft.nl> > Your splits were both the same, so the preconditioner was applied twice > to the first block and zero to the second block. That made the > preconditioner singular. Dear Jed, Thanks a lot for your time and help, attached you can find a very challenging ill-conditioned problem: The first block comes from a structural problem and its size is 1490. The second block results from a fluid discretization and its size is 2432. So in total the coupled system is of size 3922. I created the blocks as given in the simple problem. However, with ./ex3 -pc_type fieldsplit -pc_fieldsplit_type schur -fieldsplit_1_pc_type icc -fieldsplit_1_ksp_type cg -fieldsplit_1_pc_type icc -fieldsplit_1_ksp_type minres -pc_fieldsplit_schur_factorization_type upper -ksp_monitor_true_residual The solver just hangs. The system is a symmetric system in this case given as A = A11 C C^T A22 C is of size 1490X2432 Could you please comment on this point for me, do I still have an error on the problem setup or on the usage of the options? BR, Umut > > ISCreateBlock(PETSC_COMM_SELF,bs,n1,inputindices,PETSC_COPY_VALUES,&is_row1); > ISCreateBlock(PETSC_COMM_SELF,bs,n2,inputindices,PETSC_COPY_VALUES,&is_row2); > > We could check for this case, but it would require a little code and > isn't currently done. This is probably what you wanted: > > diff --git i/ex2.c w/ex2.c > index 94cea2c..0cce3e6 100644 > --- i/ex2.c > +++ w/ex2.c > @@ -45,7 +45,7 @@ int main(int argc,char **args) > PetscInt bs = 2, n1=1, n2=1, inputindices[]={0,1}; /* block size is 2 */ > > ISCreateBlock(PETSC_COMM_SELF,bs,n1,inputindices,PETSC_COPY_VALUES,&is_row1); > - ISCreateBlock(PETSC_COMM_SELF,bs,n2,inputindices,PETSC_COPY_VALUES,&is_row2); > + ISCreateBlock(PETSC_COMM_SELF,bs,n2,inputindices+n1,PETSC_COPY_VALUES,&is_row2); > > ISView(is_row1,PETSC_VIEWER_STDOUT_SELF); > ISView(is_row2,PETSC_VIEWER_STDOUT_SELF); -------------- next part -------------- A non-text attachment was scrubbed... Name: borg.bin Type: application/octet-stream Size: 31384 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex3.c Type: text/x-csrc Size: 2861 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Korg.bin Type: application/octet-stream Size: 1546016 bytes Desc: not available URL: From ztdepyahoo at 163.com Sun Sep 8 07:50:17 2013 From: ztdepyahoo at 163.com (=?GBK?B?tqHAz8qm?=) Date: Sun, 8 Sep 2013 20:50:17 +0800 (CST) Subject: [petsc-users] does kspsolve perform the reordering to reduce the bandwith of sparse matrix Message-ID: <71425030.e38e.140fd9f69d5.Coremail.ztdepyahoo@163.com> -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Sep 8 08:00:38 2013 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 8 Sep 2013 08:00:38 -0500 Subject: [petsc-users] [PATCH 2/2] Call KSPSetFromOptions before setting fieldsplit-specific options In-Reply-To: <522C4227.5050903@tudelft.nl> References: <522B9869.5060803@tudelft.nl> <1378619690-13533-1-git-send-email-jedbrown@mcs.anl.gov> <1378619690-13533-2-git-send-email-jedbrown@mcs.anl.gov> <522C17B9.9000307@tudelft.nl> <87mwnn95kb.fsf@mcs.anl.gov> <522C4227.5050903@tudelft.nl> Message-ID: On Sun, Sep 8, 2013 at 4:23 AM, Umut Tabak wrote: > > Your splits were both the same, so the preconditioner was applied twice >> to the first block and zero to the second block. That made the >> preconditioner singular. >> > > Dear Jed, > > Thanks a lot for your time and help, attached you can find a very > challenging ill-conditioned problem: > > The first block comes from a structural problem and its size is 1490. The > second block results from a fluid discretization and its size is 2432. So > in total the coupled system is of size 3922. I created the blocks as given > in the simple problem. However, with > > ./ex3 -pc_type fieldsplit -pc_fieldsplit_type schur -fieldsplit_1_pc_type > icc -fieldsplit_1_ksp_type cg -fieldsplit_1_pc_type icc > -fieldsplit_1_ksp_type minres -pc_fieldsplit_schur_**factorization_type > upper -ksp_monitor_true_residual > > The solver just hangs. > If you want to solve hard systems, you must learn to debug a solver yourself. What does "just hangs" mean? Start with a full Schur complement factorization with LU on block 1 and a tiny tolerance on block 2. This will converge in 1 iterate. Then start slowly making the preconditioner weaker. Matt > The system is a symmetric system in this case given as > > A = > A11 C > C^T A22 > > C is of size 1490X2432 > > Could you please comment on this point for me, do I still have an error on > the problem setup or on the usage of the options? > > BR, > Umut > > >> ISCreateBlock(PETSC_COMM_SELF,**bs,n1,inputindices,PETSC_COPY_** >> VALUES,&is_row1); >> ISCreateBlock(PETSC_COMM_SELF,**bs,n2,inputindices,PETSC_COPY_** >> VALUES,&is_row2); >> >> We could check for this case, but it would require a little code and >> isn't currently done. This is probably what you wanted: >> >> diff --git i/ex2.c w/ex2.c >> index 94cea2c..0cce3e6 100644 >> --- i/ex2.c >> +++ w/ex2.c >> @@ -45,7 +45,7 @@ int main(int argc,char **args) >> PetscInt bs = 2, n1=1, n2=1, inputindices[]={0,1}; /* block size is 2 >> */ >> ISCreateBlock(PETSC_COMM_SELF,**bs,n1,inputindices,PETSC_COPY_* >> *VALUES,&is_row1); >> - ISCreateBlock(PETSC_COMM_SELF,**bs,n2,inputindices,PETSC_COPY_** >> VALUES,&is_row2); >> + ISCreateBlock(PETSC_COMM_SELF,**bs,n2,inputindices+n1,PETSC_** >> COPY_VALUES,&is_row2); >> ISView(is_row1,PETSC_VIEWER_**STDOUT_SELF); >> ISView(is_row2,PETSC_VIEWER_**STDOUT_SELF); >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Sep 8 08:22:23 2013 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 8 Sep 2013 08:22:23 -0500 Subject: [petsc-users] does kspsolve perform the reordering to reduce the bandwith of sparse matrix In-Reply-To: <71425030.e38e.140fd9f69d5.Coremail.ztdepyahoo@163.com> References: <71425030.e38e.140fd9f69d5.Coremail.ztdepyahoo@163.com> Message-ID: http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/MatOrderings/MatGetOrdering.html Matt On Sun, Sep 8, 2013 at 7:50 AM, ??? wrote: > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From u.tabak at tudelft.nl Sun Sep 8 08:32:49 2013 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sun, 08 Sep 2013 15:32:49 +0200 Subject: [petsc-users] [PATCH 2/2] Call KSPSetFromOptions before setting fieldsplit-specific options In-Reply-To: References: <522B9869.5060803@tudelft.nl> <1378619690-13533-1-git-send-email-jedbrown@mcs.anl.gov> <1378619690-13533-2-git-send-email-jedbrown@mcs.anl.gov> <522C17B9.9000307@tudelft.nl> <87mwnn95kb.fsf@mcs.anl.gov> <522C4227.5050903@tudelft.nl> Message-ID: <522C7C81.5050000@tudelft.nl> On 09/08/2013 03:00 PM, Matthew Knepley wrote: > > > If you want to solve hard systems, you must learn to debug a solver > yourself. What does "just hangs" mean? Start with a full > Schur complement factorization with LU on block 1 and a tiny tolerance > on block 2. This will converge in 1 iterate. Then > start slowly making the preconditioner weaker. Hi Matt, Thanks for the suggestion but I did not completely understand the options to use for this, I am still learning the fieldsplit interface. It was my mistake to not to give more in formation on what 'Just hangs' means, sorry. For the previous message: that meant extremely slow iterations, the iteration process was not advancing at all. After your reply and looking at the fieldsplit options a bit more, I came up with these set of options, now iterations are going fast but the decrease in relative residual is so slow. But apparently this does not converge in 1 iteration, I am still mistaken at some point. ./ex3 -pc_type fieldsplit -pc_fieldsplit_type schur -fieldsplit_1_pc_type lu -fieldsplit_1_ksp_type preonly -fieldsplit_1_pc_type icc -fieldsplit_2_ksp_type minres -fieldsplit_2_ksp_rtol 1e-8 -pc_fieldsplit_schur_factorization_type full -ksp_monitor_true_residual Maybe you can help me at this point again. Thanks. Umut > > Matt > > The system is a symmetric system in this case given as > > A = > A11 C > C^T A22 > > C is of size 1490X2432 > > Could you please comment on this point for me, do I still have an > error on the problem setup or on the usage of the options? > > BR, > Umut > > > > ISCreateBlock(PETSC_COMM_SELF,bs,n1,inputindices,PETSC_COPY_VALUES,&is_row1); > > ISCreateBlock(PETSC_COMM_SELF,bs,n2,inputindices,PETSC_COPY_VALUES,&is_row2); > > We could check for this case, but it would require a little > code and > isn't currently done. This is probably what you wanted: > > diff --git i/ex2.c w/ex2.c > index 94cea2c..0cce3e6 100644 > --- i/ex2.c > +++ w/ex2.c > @@ -45,7 +45,7 @@ int main(int argc,char **args) > PetscInt bs = 2, n1=1, n2=1, inputindices[]={0,1}; /* > block size is 2 */ > > ISCreateBlock(PETSC_COMM_SELF,bs,n1,inputindices,PETSC_COPY_VALUES,&is_row1); > - > ISCreateBlock(PETSC_COMM_SELF,bs,n2,inputindices,PETSC_COPY_VALUES,&is_row2); > + > ISCreateBlock(PETSC_COMM_SELF,bs,n2,inputindices+n1,PETSC_COPY_VALUES,&is_row2); > ISView(is_row1,PETSC_VIEWER_STDOUT_SELF); > ISView(is_row2,PETSC_VIEWER_STDOUT_SELF); > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Sun Sep 8 08:38:38 2013 From: dave.mayhem23 at gmail.com (Dave May) Date: Sun, 8 Sep 2013 15:38:38 +0200 Subject: [petsc-users] [PATCH 2/2] Call KSPSetFromOptions before setting fieldsplit-specific options In-Reply-To: <522C7C81.5050000@tudelft.nl> References: <522B9869.5060803@tudelft.nl> <1378619690-13533-1-git-send-email-jedbrown@mcs.anl.gov> <1378619690-13533-2-git-send-email-jedbrown@mcs.anl.gov> <522C17B9.9000307@tudelft.nl> <87mwnn95kb.fsf@mcs.anl.gov> <522C4227.5050903@tudelft.nl> <522C7C81.5050000@tudelft.nl> Message-ID: Remove the option -fieldsplit_1_pc_type icc It is clobbering the LU option Matt suggested you try first ( -fieldsplit_1_pc_type lu) When debigging such solvers, you should always run with -ksp_view to check your solver is configured correctly. On 8 September 2013 15:32, Umut Tabak wrote: > On 09/08/2013 03:00 PM, Matthew Knepley wrote: > > > > If you want to solve hard systems, you must learn to debug a solver > yourself. What does "just hangs" mean? Start with a full > Schur complement factorization with LU on block 1 and a tiny tolerance on > block 2. This will converge in 1 iterate. Then > start slowly making the preconditioner weaker. > > Hi Matt, > > Thanks for the suggestion but I did not completely understand the options > to use for this, I am still learning the fieldsplit interface. It was my > mistake to not to give more in formation on what 'Just hangs' means, sorry. > > For the previous message: that meant extremely slow iterations, the > iteration process was not advancing at all. > > After your reply and looking at the fieldsplit options a bit more, I came > up with these set of options, now iterations are going fast but the > decrease in relative residual is so slow. But apparently this does not > converge in 1 iteration, I am still mistaken at some point. > > ./ex3 -pc_type fieldsplit -pc_fieldsplit_type schur -fieldsplit_1_pc_type > lu -fieldsplit_1_ksp_type preonly -fieldsplit_1_pc_type icc > -fieldsplit_2_ksp_type minres -fieldsplit_2_ksp_rtol 1e-8 > -pc_fieldsplit_schur_factorization_type full -ksp_monitor_true_residual > > Maybe you can help me at this point again. > > Thanks. > Umut > > > > > Matt > > >> The system is a symmetric system in this case given as >> >> A = >> A11 C >> C^T A22 >> >> C is of size 1490X2432 >> >> Could you please comment on this point for me, do I still have an error >> on the problem setup or on the usage of the options? >> >> BR, >> Umut >> >> >>> >>> ISCreateBlock(PETSC_COMM_SELF,bs,n1,inputindices,PETSC_COPY_VALUES,&is_row1); >>> >>> ISCreateBlock(PETSC_COMM_SELF,bs,n2,inputindices,PETSC_COPY_VALUES,&is_row2); >>> >>> We could check for this case, but it would require a little code and >>> isn't currently done. This is probably what you wanted: >>> >>> diff --git i/ex2.c w/ex2.c >>> index 94cea2c..0cce3e6 100644 >>> --- i/ex2.c >>> +++ w/ex2.c >>> @@ -45,7 +45,7 @@ int main(int argc,char **args) >>> PetscInt bs = 2, n1=1, n2=1, inputindices[]={0,1}; /* block size is >>> 2 */ >>> >>> ISCreateBlock(PETSC_COMM_SELF,bs,n1,inputindices,PETSC_COPY_VALUES,&is_row1); >>> - >>> ISCreateBlock(PETSC_COMM_SELF,bs,n2,inputindices,PETSC_COPY_VALUES,&is_row2); >>> + >>> ISCreateBlock(PETSC_COMM_SELF,bs,n2,inputindices+n1,PETSC_COPY_VALUES,&is_row2); >>> ISView(is_row1,PETSC_VIEWER_STDOUT_SELF); >>> ISView(is_row2,PETSC_VIEWER_STDOUT_SELF); >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sun Sep 8 08:38:42 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 08 Sep 2013 08:38:42 -0500 Subject: [petsc-users] [PATCH 2/2] Call KSPSetFromOptions before setting fieldsplit-specific options In-Reply-To: <522C1F3F.5020009@tudelft.nl> References: <522B9869.5060803@tudelft.nl> <1378619690-13533-1-git-send-email-jedbrown@mcs.anl.gov> <1378619690-13533-2-git-send-email-jedbrown@mcs.anl.gov> <522C17B9.9000307@tudelft.nl> <87mwnn95kb.fsf@mcs.anl.gov> <522C1F3F.5020009@tudelft.nl> Message-ID: <87hadv8mcd.fsf@mcs.anl.gov> Umut Tabak writes: > On 09/08/2013 08:43 AM, Jed Brown wrote: >> Umut Tabak writes: >> >> Your splits were both the same, > Some questions: > > + Splits represent different row and column blocks, right? They should be different, though they can overlap (rarely what you want). > + And as far as I understand from you(maybe this is in the manual, I am > not sure) and they should differ in size, right? They may or may not be the same size, but if you look at what your code actually did, it was creating two identical index sets (both only used the first entry of your 'inputindices'). -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From u.tabak at tudelft.nl Sun Sep 8 08:42:58 2013 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sun, 08 Sep 2013 15:42:58 +0200 Subject: [petsc-users] [PATCH 2/2] Call KSPSetFromOptions before setting fieldsplit-specific options In-Reply-To: References: <522B9869.5060803@tudelft.nl> <1378619690-13533-1-git-send-email-jedbrown@mcs.anl.gov> <1378619690-13533-2-git-send-email-jedbrown@mcs.anl.gov> <522C17B9.9000307@tudelft.nl> <87mwnn95kb.fsf@mcs.anl.gov> <522C4227.5050903@tudelft.nl> <522C7C81.5050000@tudelft.nl> Message-ID: <522C7EE2.7090104@tudelft.nl> On 09/08/2013 03:38 PM, Dave May wrote: > Remove the option > -fieldsplit_1_pc_type icc > It is clobbering the LU option Matt suggested you try first ( > -fieldsplit_1_pc_type lu) Dear Dave, I also realized this but and changed that with -fieldsplit_2_pc_type icc but thanks for looking at that, need a break I guess ;-) > > When debigging such solvers, you should always run with -ksp_view to > check your solver is configured correctly. Thanks for this > > > > On 8 September 2013 15:32, Umut Tabak > wrote: > > On 09/08/2013 03:00 PM, Matthew Knepley wrote: >> >> >> If you want to solve hard systems, you must learn to debug a >> solver yourself. What does "just hangs" mean? Start with a full >> Schur complement factorization with LU on block 1 and a tiny >> tolerance on block 2. This will converge in 1 iterate. Then >> start slowly making the preconditioner weaker. > Hi Matt, > > Thanks for the suggestion but I did not completely understand the > options to use for this, I am still learning the fieldsplit > interface. It was my mistake to not to give more in formation on > what 'Just hangs' means, sorry. > > For the previous message: that meant extremely slow iterations, > the iteration process was not advancing at all. > > After your reply and looking at the fieldsplit options a bit more, > I came up with these set of options, now iterations are going fast > but the decrease in relative residual is so slow. But apparently > this does not converge in 1 iteration, I am still mistaken at some > point. > > ./ex3 -pc_type fieldsplit -pc_fieldsplit_type schur > -fieldsplit_1_pc_type lu -fieldsplit_1_ksp_type preonly > -fieldsplit_1_pc_type icc -fieldsplit_2_ksp_type minres > -fieldsplit_2_ksp_rtol 1e-8 > -pc_fieldsplit_schur_factorization_type full > -ksp_monitor_true_residual > > Maybe you can help me at this point again. > > Thanks. > Umut > > >> >> Matt >> >> The system is a symmetric system in this case given as >> >> A = >> A11 C >> C^T A22 >> >> C is of size 1490X2432 >> >> Could you please comment on this point for me, do I still >> have an error on the problem setup or on the usage of the >> options? >> >> BR, >> Umut >> >> >> ISCreateBlock(PETSC_COMM_SELF,bs,n1,inputindices,PETSC_COPY_VALUES,&is_row1); >> ISCreateBlock(PETSC_COMM_SELF,bs,n2,inputindices,PETSC_COPY_VALUES,&is_row2); >> >> We could check for this case, but it would require a >> little code and >> isn't currently done. This is probably what you wanted: >> >> diff --git i/ex2.c w/ex2.c >> index 94cea2c..0cce3e6 100644 >> --- i/ex2.c >> +++ w/ex2.c >> @@ -45,7 +45,7 @@ int main(int argc,char **args) >> PetscInt bs = 2, n1=1, n2=1, inputindices[]={0,1}; /* >> block size is 2 */ >> ISCreateBlock(PETSC_COMM_SELF,bs,n1,inputindices,PETSC_COPY_VALUES,&is_row1); >> - >> ISCreateBlock(PETSC_COMM_SELF,bs,n2,inputindices,PETSC_COPY_VALUES,&is_row2); >> + >> ISCreateBlock(PETSC_COMM_SELF,bs,n2,inputindices+n1,PETSC_COPY_VALUES,&is_row2); >> ISView(is_row1,PETSC_VIEWER_STDOUT_SELF); >> ISView(is_row2,PETSC_VIEWER_STDOUT_SELF); >> >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sun Sep 8 08:45:11 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 08 Sep 2013 08:45:11 -0500 Subject: [petsc-users] [PATCH 2/2] Call KSPSetFromOptions before setting fieldsplit-specific options In-Reply-To: <522C4227.5050903@tudelft.nl> References: <522B9869.5060803@tudelft.nl> <1378619690-13533-1-git-send-email-jedbrown@mcs.anl.gov> <1378619690-13533-2-git-send-email-jedbrown@mcs.anl.gov> <522C17B9.9000307@tudelft.nl> <87mwnn95kb.fsf@mcs.anl.gov> <522C4227.5050903@tudelft.nl> Message-ID: <87eh8z8m1k.fsf@mcs.anl.gov> Umut Tabak writes: >> Your splits were both the same, so the preconditioner was applied twice >> to the first block and zero to the second block. That made the >> preconditioner singular. > > Dear Jed, > > Thanks a lot for your time and help, attached you can find a very > challenging ill-conditioned problem: > > The first block comes from a structural problem and its size is 1490. > The second block results from a fluid discretization and its size is > 2432. So in total the coupled system is of size 3922. I created the > blocks as given in the simple problem. However, with > > ./ex3 -pc_type fieldsplit -pc_fieldsplit_type schur > -fieldsplit_1_pc_type icc -fieldsplit_1_ksp_type cg > -fieldsplit_1_pc_type icc -fieldsplit_1_ksp_type minres You have given nearly the same options twice. Did you mean to control the second split? > -pc_fieldsplit_schur_factorization_type upper -ksp_monitor_true_residual > > The solver just hangs. > > The system is a symmetric system in this case given as > > A = > A11 C > C^T A22 > > C is of size 1490X2432 > > Could you please comment on this point for me, do I still have an error > on the problem setup or on the usage of the options? This is an invalid memory access and does not use ISCreateBlock correctly. PetscInt sz1 = 1490, sz2 = 2432; PetscInt bs = 2, n1=1, n2=1, inputindices[]={0}; /* block size is 2 */ ISCreateBlock(PETSC_COMM_SELF,sz1,n1,inputindices, PETSC_COPY_VALUES,&is_row1); ISCreateBlock(PETSC_COMM_SELF,sz2,n2,inputindices+n1, PETSC_COPY_VALUES,&is_row2); -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From u.tabak at tudelft.nl Sun Sep 8 08:45:59 2013 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sun, 08 Sep 2013 15:45:59 +0200 Subject: [petsc-users] [PATCH 2/2] Call KSPSetFromOptions before setting fieldsplit-specific options In-Reply-To: References: <522B9869.5060803@tudelft.nl> <1378619690-13533-1-git-send-email-jedbrown@mcs.anl.gov> <1378619690-13533-2-git-send-email-jedbrown@mcs.anl.gov> <522C17B9.9000307@tudelft.nl> <87mwnn95kb.fsf@mcs.anl.gov> <522C4227.5050903@tudelft.nl> Message-ID: <522C7F97.2050905@tudelft.nl> On 09/08/2013 03:00 PM, Matthew Knepley wrote: > > > If you want to solve hard systems, you must learn to debug a solver > yourself. What does "just hangs" mean? Start with a full > Schur complement factorization with LU on block 1 and a tiny tolerance > on block 2. This will converge in 1 iterate. Then > start slowly making the preconditioner weaker. Ok just for reference for future search on the topic for the others, here is the correct set of options ./ex3 -pc_type fieldsplit -fieldsplit_1_pc_type lu -fieldsplit_1_ksp_type preonly -fieldsplit_2_pc_type icc -fieldsplit_2_ksp_type minres -fieldsplit_2_ksp_rtol 1e-4 -ksp_view > > Matt > > The system is a symmetric system in this case given as > > A = > A11 C > C^T A22 > > C is of size 1490X2432 > > Could you please comment on this point for me, do I still have an > error on the problem setup or on the usage of the options? > > BR, > Umut > > > > ISCreateBlock(PETSC_COMM_SELF,bs,n1,inputindices,PETSC_COPY_VALUES,&is_row1); > > ISCreateBlock(PETSC_COMM_SELF,bs,n2,inputindices,PETSC_COPY_VALUES,&is_row2); > > We could check for this case, but it would require a little > code and > isn't currently done. This is probably what you wanted: > > diff --git i/ex2.c w/ex2.c > index 94cea2c..0cce3e6 100644 > --- i/ex2.c > +++ w/ex2.c > @@ -45,7 +45,7 @@ int main(int argc,char **args) > PetscInt bs = 2, n1=1, n2=1, inputindices[]={0,1}; /* > block size is 2 */ > > ISCreateBlock(PETSC_COMM_SELF,bs,n1,inputindices,PETSC_COPY_VALUES,&is_row1); > - > ISCreateBlock(PETSC_COMM_SELF,bs,n2,inputindices,PETSC_COPY_VALUES,&is_row2); > + > ISCreateBlock(PETSC_COMM_SELF,bs,n2,inputindices+n1,PETSC_COPY_VALUES,&is_row2); > ISView(is_row1,PETSC_VIEWER_STDOUT_SELF); > ISView(is_row2,PETSC_VIEWER_STDOUT_SELF); > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From u.tabak at tudelft.nl Sun Sep 8 08:48:10 2013 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sun, 08 Sep 2013 15:48:10 +0200 Subject: [petsc-users] [PATCH 2/2] Call KSPSetFromOptions before setting fieldsplit-specific options In-Reply-To: <87eh8z8m1k.fsf@mcs.anl.gov> References: <522B9869.5060803@tudelft.nl> <1378619690-13533-1-git-send-email-jedbrown@mcs.anl.gov> <1378619690-13533-2-git-send-email-jedbrown@mcs.anl.gov> <522C17B9.9000307@tudelft.nl> <87mwnn95kb.fsf@mcs.anl.gov> <522C4227.5050903@tudelft.nl> <87eh8z8m1k.fsf@mcs.anl.gov> Message-ID: <522C801A.6080907@tudelft.nl> On 09/08/2013 03:45 PM, Jed Brown wrote: > > This is an invalid memory access and does not use ISCreateBlock correctly. > > PetscInt sz1 = 1490, sz2 = 2432; > > PetscInt bs = 2, n1=1, n2=1, inputindices[]={0}; /* block size is 2 */ > > ISCreateBlock(PETSC_COMM_SELF,sz1,n1,inputindices, > PETSC_COPY_VALUES,&is_row1); > ISCreateBlock(PETSC_COMM_SELF,sz2,n2,inputindices+n1, > PETSC_COPY_VALUES,&is_row2); Ok, I am trying to understand what my mistake is. From jedbrown at mcs.anl.gov Sun Sep 8 08:57:55 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 08 Sep 2013 08:57:55 -0500 Subject: [petsc-users] [PATCH 2/2] Call KSPSetFromOptions before setting fieldsplit-specific options In-Reply-To: <522C801A.6080907@tudelft.nl> References: <522B9869.5060803@tudelft.nl> <1378619690-13533-1-git-send-email-jedbrown@mcs.anl.gov> <1378619690-13533-2-git-send-email-jedbrown@mcs.anl.gov> <522C17B9.9000307@tudelft.nl> <87mwnn95kb.fsf@mcs.anl.gov> <522C4227.5050903@tudelft.nl> <87eh8z8m1k.fsf@mcs.anl.gov> <522C801A.6080907@tudelft.nl> Message-ID: <87a9jn8lgc.fsf@mcs.anl.gov> Umut Tabak writes: > On 09/08/2013 03:45 PM, Jed Brown wrote: >> >> This is an invalid memory access and does not use ISCreateBlock correctly. >> >> PetscInt sz1 = 1490, sz2 = 2432; >> >> PetscInt bs = 2, n1=1, n2=1, inputindices[]={0}; /* block size is 2 */ >> >> ISCreateBlock(PETSC_COMM_SELF,sz1,n1,inputindices, >> PETSC_COPY_VALUES,&is_row1); >> ISCreateBlock(PETSC_COMM_SELF,sz2,n2,inputindices+n1, >> PETSC_COPY_VALUES,&is_row2); > Ok, I am trying to understand what my mistake is. In the second call, ISCreateBlock looks at the array of block indices of length n2 starting from inputindices+n1. That references inputindices[n1], which is off the end of the array you created above. Simply extending that array won't work either because you can't describe the starting point correctly for one "block" of different length. I.e., if inputindices[1] == 1, then the second call above would be creating the index set {sz2, sz2+1,..., 2*sz2-1} This should be obvious if you look at ISView(). If your vectors are ordered so that the blocks are separate and contiguous (as opposed to interlaced in some way), I suggest using ISCreateStride(PETSC_COMM_SELF,sz1,0,1,&is_row1); ISCreateStride(PETSC_COMM_SELF,sz2,sz1,1,is_row2); -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From u.tabak at tudelft.nl Sun Sep 8 09:09:28 2013 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sun, 08 Sep 2013 16:09:28 +0200 Subject: [petsc-users] [PATCH 2/2] Call KSPSetFromOptions before setting fieldsplit-specific options In-Reply-To: <87a9jn8lgc.fsf@mcs.anl.gov> References: <522B9869.5060803@tudelft.nl> <1378619690-13533-1-git-send-email-jedbrown@mcs.anl.gov> <1378619690-13533-2-git-send-email-jedbrown@mcs.anl.gov> <522C17B9.9000307@tudelft.nl> <87mwnn95kb.fsf@mcs.anl.gov> <522C4227.5050903@tudelft.nl> <87eh8z8m1k.fsf@mcs.anl.gov> <522C801A.6080907@tudelft.nl> <87a9jn8lgc.fsf@mcs.anl.gov> Message-ID: <522C8518.6070609@tudelft.nl> On 09/08/2013 03:57 PM, Jed Brown wrote: > > In the second call, ISCreateBlock looks at the array of block indices of > length n2 starting from inputindices+n1. That references > inputindices[n1], which is off the end of the array you created above. > Simply extending that array won't work either because you can't describe > the starting point correctly for one "block" of different length. I.e., > if inputindices[1] == 1, then the second call above would be creating > the index set > > {sz2, sz2+1,..., 2*sz2-1} > > This should be obvious if you look at ISView(). Yes, but then there is a problem with my understanding of the block structure. What I would like to do at the end is to create a first block of size 1490 and then create a block of size 2432 and stack these. This is how the original system is built up. Of what if I use the below code, PetscInt sz1 = 1490, sz2 = 2432; /* ----- */ PetscInt n1=1, n2=1, inputindices1[]={0}, inputindices2[]={1}; /* block size is 2 */ ISCreateBlock(PETSC_COMM_SELF,sz1,n1,inputindices1, PETSC_COPY_VALUES,&is_row1); ISCreateBlock(PETSC_COMM_SELF,sz2,n2,inputindices2, PETSC_COPY_VALUES,&is_row2); Because if I look at http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/IS/ISCreateBlock.html I am getting the impression that bs and n are the same and on top idx is the global block number of the large matrix blocks. I will look at the below option in a minute. > > If your vectors are ordered so that the blocks are separate and > contiguous (as opposed to interlaced in some way), I suggest using > > ISCreateStride(PETSC_COMM_SELF,sz1,0,1,&is_row1); > ISCreateStride(PETSC_COMM_SELF,sz2,sz1,1,is_row2); From u.tabak at tudelft.nl Sun Sep 8 09:15:12 2013 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sun, 08 Sep 2013 16:15:12 +0200 Subject: [petsc-users] [PATCH 2/2] Call KSPSetFromOptions before setting fieldsplit-specific options In-Reply-To: <522C8518.6070609@tudelft.nl> References: <522B9869.5060803@tudelft.nl> <1378619690-13533-1-git-send-email-jedbrown@mcs.anl.gov> <1378619690-13533-2-git-send-email-jedbrown@mcs.anl.gov> <522C17B9.9000307@tudelft.nl> <87mwnn95kb.fsf@mcs.anl.gov> <522C4227.5050903@tudelft.nl> <87eh8z8m1k.fsf@mcs.anl.gov> <522C801A.6080907@tudelft.nl> <87a9jn8lgc.fsf@mcs.anl.gov> <522C8518.6070609@tudelft.nl> Message-ID: <522C8670.1080207@tudelft.nl> On 09/08/2013 04:09 PM, Umut Tabak wrote: > >> >> If your vectors are ordered so that the blocks are separate and >> contiguous (as opposed to interlaced in some way), I suggest using >> >> ISCreateStride(PETSC_COMM_SELF,sz1,0,1,&is_row1); >> ISCreateStride(PETSC_COMM_SELF,sz2,sz1,1,is_row2); > ok this was the thing that was missing and what I was looking for, thanks From jedbrown at mcs.anl.gov Sun Sep 8 09:19:07 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 08 Sep 2013 09:19:07 -0500 Subject: [petsc-users] [PATCH 2/2] Call KSPSetFromOptions before setting fieldsplit-specific options In-Reply-To: <522C8518.6070609@tudelft.nl> References: <522B9869.5060803@tudelft.nl> <1378619690-13533-1-git-send-email-jedbrown@mcs.anl.gov> <1378619690-13533-2-git-send-email-jedbrown@mcs.anl.gov> <522C17B9.9000307@tudelft.nl> <87mwnn95kb.fsf@mcs.anl.gov> <522C4227.5050903@tudelft.nl> <87eh8z8m1k.fsf@mcs.anl.gov> <522C801A.6080907@tudelft.nl> <87a9jn8lgc.fsf@mcs.anl.gov> <522C8518.6070609@tudelft.nl> Message-ID: <877ger8kh0.fsf@mcs.anl.gov> Umut Tabak writes: > On 09/08/2013 03:57 PM, Jed Brown wrote: >> >> In the second call, ISCreateBlock looks at the array of block indices of >> length n2 starting from inputindices+n1. That references >> inputindices[n1], which is off the end of the array you created above. >> Simply extending that array won't work either because you can't describe >> the starting point correctly for one "block" of different length. I.e., >> if inputindices[1] == 1, then the second call above would be creating >> the index set >> >> {sz2, sz2+1,..., 2*sz2-1} >> >> This should be obvious if you look at ISView(). > Yes, but then there is a problem with my understanding of the block > structure. > > What I would like to do at the end is to create a first block of size > 1490 and then create a block of size 2432 and stack these. This is how > the original system is built up. Then use the code I sent. > Of what if I use the below code, > > PetscInt sz1 = 1490, sz2 = 2432; > > /* ----- */ > > PetscInt n1=1, n2=1, > inputindices1[]={0}, inputindices2[]={1}; /* block size is 2 */ > > ISCreateBlock(PETSC_COMM_SELF,sz1,n1,inputindices1, > PETSC_COPY_VALUES,&is_row1); > ISCreateBlock(PETSC_COMM_SELF,sz2,n2,inputindices2, > PETSC_COPY_VALUES,&is_row2); > > Because if I look at > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/IS/ISCreateBlock.html > > I am getting the impression that bs and n are the same and on top idx is > the global block number of the large matrix blocks. What gives you this impression? ISCreateBlock is meant to describe index sets like the following (bs=2): {0,1,8,9,10,11,16,17,20,21} via ISCreateBlock(comm,2,5,[0,4,5,8,10],&is); -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From bsmith at mcs.anl.gov Sun Sep 8 09:57:36 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 8 Sep 2013 09:57:36 -0500 Subject: [petsc-users] does kspsolve perform the reordering to reduce the bandwith of sparse matrix In-Reply-To: <71425030.e38e.140fd9f69d5.Coremail.ztdepyahoo@163.com> References: <71425030.e38e.140fd9f69d5.Coremail.ztdepyahoo@163.com> Message-ID: <174DF7FD-A139-41F0-A333-CDAD6DB6CBF5@mcs.anl.gov> No. The reordering should be part of the process when setting up your mesh. That is you order the unknowns on the mesh properly and you do not need to reorder the matrix. Note that this improves not only the linear solver (a bit) but also improves the time of the mesh operations and function and Jacobian evaluations on the mesh. On Sep 8, 2013, at 7:50 AM, ??? wrote: > > > From u.tabak at tudelft.nl Sun Sep 8 10:01:05 2013 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sun, 08 Sep 2013 17:01:05 +0200 Subject: [petsc-users] [PATCH 2/2] Call KSPSetFromOptions before setting fieldsplit-specific options In-Reply-To: <877ger8kh0.fsf@mcs.anl.gov> References: <522B9869.5060803@tudelft.nl> <1378619690-13533-1-git-send-email-jedbrown@mcs.anl.gov> <1378619690-13533-2-git-send-email-jedbrown@mcs.anl.gov> <522C17B9.9000307@tudelft.nl> <87mwnn95kb.fsf@mcs.anl.gov> <522C4227.5050903@tudelft.nl> <87eh8z8m1k.fsf@mcs.anl.gov> <522C801A.6080907@tudelft.nl> <87a9jn8lgc.fsf@mcs.anl.gov> <522C8518.6070609@tudelft.nl> <877ger8kh0.fsf@mcs.anl.gov> Message-ID: <522C9131.708@tudelft.nl> On 09/08/2013 04:19 PM, Jed Brown wrote: > Umut Tabak writes: Dear Jed and others, Thanks a lot for your help. One last question, trying to run with ./ex3 -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_schur_factorization_type diag -fieldsplit_1_pc_type icc -fieldsplit_1_ksp_type minres -fieldsplit_2_pc_type icc -fieldsplit_2_ksp_type minres -ksp_converged_reason -ksp_monitor_short With this input, I would expect to see a flow of residual information even if increasing, but no information is being printed on the screen, what could be the reason for this? >> On 09/08/2013 03:57 PM, Jed Brown wrote: >>> In the second call, ISCreateBlock looks at the array of block indices of >>> length n2 starting from inputindices+n1. That references >>> inputindices[n1], which is off the end of the array you created above. >>> Simply extending that array won't work either because you can't describe >>> the starting point correctly for one "block" of different length. I.e., >>> if inputindices[1] == 1, then the second call above would be creating >>> the index set >>> >>> {sz2, sz2+1,..., 2*sz2-1} >>> >>> This should be obvious if you look at ISView(). >> Yes, but then there is a problem with my understanding of the block >> structure. >> >> What I would like to do at the end is to create a first block of size >> 1490 and then create a block of size 2432 and stack these. This is how >> the original system is built up. > Then use the code I sent. > >> Of what if I use the below code, >> >> PetscInt sz1 = 1490, sz2 = 2432; >> >> /* ----- */ >> >> PetscInt n1=1, n2=1, >> inputindices1[]={0}, inputindices2[]={1}; /* block size is 2 */ >> >> ISCreateBlock(PETSC_COMM_SELF,sz1,n1,inputindices1, >> PETSC_COPY_VALUES,&is_row1); >> ISCreateBlock(PETSC_COMM_SELF,sz2,n2,inputindices2, >> PETSC_COPY_VALUES,&is_row2); >> >> Because if I look at >> >> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/IS/ISCreateBlock.html >> >> I am getting the impression that bs and n are the same and on top idx is >> the global block number of the large matrix blocks. > What gives you this impression? > > ISCreateBlock is meant to describe index sets like the following (bs=2): > > {0,1,8,9,10,11,16,17,20,21} > > via > > ISCreateBlock(comm,2,5,[0,4,5,8,10],&is); From hariseldon99 at gmail.com Sun Sep 8 10:00:47 2013 From: hariseldon99 at gmail.com (Analabha Roy) Date: Sun, 8 Sep 2013 20:30:47 +0530 Subject: [petsc-users] Petsc HDF5. Storing matrices and adding attributes Message-ID: Hi all, Firstly: I need to store a matrix in HDF5. I was reading through petsc manual pages and examples and only saw information for storing vectors in HDF5. Using MatView() on a HDF5 viewer doesn't work. Is it even possible? For now, I'm working around by extracting the columns from my Mat and storing them as vecs by looping VecView(). Secondly: Is there a way to add attributes to HDF5 data using the PetSc API? I didn't see any docs on this. Thanks in advance, AR -- --- *Analabha Roy* C.S.I.R Senior Research Associate Saha Institute of Nuclear Physics Section 1, Block AF Bidhannagar, Calcutta 700064 India *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com *Webpage*: http://www.ph.utexas.edu/~daneel/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Sun Sep 8 10:03:25 2013 From: dave.mayhem23 at gmail.com (Dave May) Date: Sun, 8 Sep 2013 17:03:25 +0200 Subject: [petsc-users] [PATCH 2/2] Call KSPSetFromOptions before setting fieldsplit-specific options In-Reply-To: <522C9131.708@tudelft.nl> References: <522B9869.5060803@tudelft.nl> <1378619690-13533-1-git-send-email-jedbrown@mcs.anl.gov> <1378619690-13533-2-git-send-email-jedbrown@mcs.anl.gov> <522C17B9.9000307@tudelft.nl> <87mwnn95kb.fsf@mcs.anl.gov> <522C4227.5050903@tudelft.nl> <87eh8z8m1k.fsf@mcs.anl.gov> <522C801A.6080907@tudelft.nl> <87a9jn8lgc.fsf@mcs.anl.gov> <522C8518.6070609@tudelft.nl> <877ger8kh0.fsf@mcs.anl.gov> <522C9131.708@tudelft.nl> Message-ID: It's stuck in one of the inner solves. Turn on the montitors for fieldsplit_2 On Sunday, 8 September 2013, Umut Tabak wrote: > On 09/08/2013 04:19 PM, Jed Brown wrote: > >> Umut Tabak writes: >> > Dear Jed and others, > > Thanks a lot for your help. > > One last question, trying to run with > > ./ex3 -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_schur_* > *factorization_type diag -fieldsplit_1_pc_type icc -fieldsplit_1_ksp_type > minres -fieldsplit_2_pc_type icc -fieldsplit_2_ksp_type minres > -ksp_converged_reason -ksp_monitor_short > > With this input, I would expect to see a flow of residual information even > if increasing, but no information is being printed on the screen, what > could be the reason for this? > > On 09/08/2013 03:57 PM, Jed Brown wrote: >>> >>>> In the second call, ISCreateBlock looks at the array of block indices of >>>> length n2 starting from inputindices+n1. That references >>>> inputindices[n1], which is off the end of the array you created above. >>>> Simply extending that array won't work either because you can't describe >>>> the starting point correctly for one "block" of different length. I.e., >>>> if inputindices[1] == 1, then the second call above would be creating >>>> the index set >>>> >>>> {sz2, sz2+1,..., 2*sz2-1} >>>> >>>> This should be obvious if you look at ISView(). >>>> >>> Yes, but then there is a problem with my understanding of the block >>> structure. >>> >>> What I would like to do at the end is to create a first block of size >>> 1490 and then create a block of size 2432 and stack these. This is how >>> the original system is built up. >>> >> Then use the code I sent. >> >> Of what if I use the below code, >>> >>> PetscInt sz1 = 1490, sz2 = 2432; >>> >>> /* ----- */ >>> >>> PetscInt n1=1, n2=1, >>> inputindices1[]={0}, inputindices2[]={1}; /* block size is 2 */ >>> >>> ISCreateBlock(PETSC_COMM_SELF,**sz1,n1,inputindices1, >>> PETSC_COPY_VALUES,&is_row1); >>> ISCreateBlock(PETSC_COMM_SELF,**sz2,n2,inputindices2, >>> PETSC_COPY_VALUES,&is_row2); >>> >>> Because if I look at >>> >>> http://www.mcs.anl.gov/petsc/**petsc-current/docs/** >>> manualpages/IS/ISCreateBlock.**html >>> >>> I am getting the impression that bs and n are the same and on top idx is >>> the global block number of the large matrix blocks. >>> >> What gives you this impression? >> >> ISCreateBlock is meant to describe index sets like the following (bs=2): >> >> {0,1,8,9,10,11,16,17,20,21} >> >> via >> >> ISCreateBlock(comm,2,5,[0,4,5,**8,10],&is); >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Sep 8 10:09:54 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 8 Sep 2013 10:09:54 -0500 Subject: [petsc-users] Petsc HDF5. Storing matrices and adding attributes In-Reply-To: References: Message-ID: <216E7BC1-9B64-4D02-98AC-C178FB007BA5@mcs.anl.gov> On Sep 8, 2013, at 10:00 AM, Analabha Roy wrote: > Hi all, > > Firstly: > I need to store a matrix in HDF5. I was reading through petsc manual pages and examples and only saw information for storing vectors in HDF5. Using MatView() on a HDF5 viewer doesn't work. Is it even possible? > Is your matrix dense? If your matrix is sparse why do you even want to store it? and if you do want to store it why do you want to use HDF5 (it won't be any faster than just using the PETSc binary format). Generally people store vectors in HDF5 because they have some other tool (like a visualization package) that can read HDF5 easily. Don't think that exists for sparse matrices. Barry > For now, I'm working around by extracting the columns from my Mat and storing them as vecs by looping VecView(). > > > Secondly: > Is there a way to add attributes to HDF5 data using the PetSc API? I didn't see any docs on this. > > > > Thanks in advance, > AR > > -- > --- > Analabha Roy > C.S.I.R Senior Research Associate > Saha Institute of Nuclear Physics > Section 1, Block AF > Bidhannagar, Calcutta 700064 > India > Emails: daneel at physics.utexas.edu, hariseldon99 at gmail.com > Webpage: http://www.ph.utexas.edu/~daneel/ From u.tabak at tudelft.nl Sun Sep 8 10:11:25 2013 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sun, 08 Sep 2013 17:11:25 +0200 Subject: [petsc-users] [PATCH 2/2] Call KSPSetFromOptions before setting fieldsplit-specific options In-Reply-To: References: <522B9869.5060803@tudelft.nl> <1378619690-13533-1-git-send-email-jedbrown@mcs.anl.gov> <1378619690-13533-2-git-send-email-jedbrown@mcs.anl.gov> <522C17B9.9000307@tudelft.nl> <87mwnn95kb.fsf@mcs.anl.gov> <522C4227.5050903@tudelft.nl> <87eh8z8m1k.fsf@mcs.anl.gov> <522C801A.6080907@tudelft.nl> <87a9jn8lgc.fsf@mcs.anl.gov> <522C8518.6070609@tudelft.nl> <877ger8kh0.fsf@mcs.anl.gov> <522C9131.708@tudelft.nl> Message-ID: <522C939D.1070704@tudelft.nl> On 09/08/2013 05:03 PM, Dave May wrote: > It's stuck in one of the inner solves. Turn on the montitors for > fieldsplit_2 ./ex3 -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_schur_factorization_type diag -fieldsplit_1_pc_type icc -fieldsplit_1_ksp_type minres -fieldsplit_2_pc_type icc -fieldsplit_2_ksp_type minres -fieldsplit_2_ksp_monitor_short -ksp_converged_reason -ksp_monitor_short Residual norms for fieldsplit_2_ solve. 0 KSP Residual norm 1.16308e+16 1 KSP Residual norm 1.05027e+16 2 KSP Residual norm 9.23221e+15 3 KSP Residual norm 7.11411e+15 4 KSP Residual norm 5.08588e+15 5 KSP Residual norm 2.55846e+15 6 KSP Residual norm 1.80045e+15 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sun Sep 8 10:51:49 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 08 Sep 2013 10:51:49 -0500 Subject: [petsc-users] [PATCH 2/2] Call KSPSetFromOptions before setting fieldsplit-specific options In-Reply-To: <522C939D.1070704@tudelft.nl> References: <522B9869.5060803@tudelft.nl> <1378619690-13533-1-git-send-email-jedbrown@mcs.anl.gov> <1378619690-13533-2-git-send-email-jedbrown@mcs.anl.gov> <522C17B9.9000307@tudelft.nl> <87mwnn95kb.fsf@mcs.anl.gov> <522C4227.5050903@tudelft.nl> <87eh8z8m1k.fsf@mcs.anl.gov> <522C801A.6080907@tudelft.nl> <87a9jn8lgc.fsf@mcs.anl.gov> <522C8518.6070609@tudelft.nl> <877ger8kh0.fsf@mcs.anl.gov> <522C9131.708@tudelft.nl> <522C939D.1070704@tudelft.nl> Message-ID: <871u4z8g6i.fsf@mcs.anl.gov> Umut Tabak writes: > On 09/08/2013 05:03 PM, Dave May wrote: >> It's stuck in one of the inner solves. Turn on the montitors for >> fieldsplit_2 > ./ex3 -pc_type fieldsplit -pc_fieldsplit_type schur > -pc_fieldsplit_schur_factorization_type diag -fieldsplit_1_pc_type icc > -fieldsplit_1_ksp_type minres -fieldsplit_2_pc_type icc > -fieldsplit_2_ksp_type minres -fieldsplit_2_ksp_monitor_short > -ksp_converged_reason -ksp_monitor_short > Residual norms for fieldsplit_2_ solve. > 0 KSP Residual norm 1.16308e+16 > 1 KSP Residual norm 1.05027e+16 > 2 KSP Residual norm 9.23221e+15 > 3 KSP Residual norm 7.11411e+15 > 4 KSP Residual norm 5.08588e+15 > 5 KSP Residual norm 2.55846e+15 > 6 KSP Residual norm 1.80045e+15 Looks like it's converging, albeit somewhat slowly, and your operator scaling is terrible (not necessarily a problem, but full-space fieldsplit methods work much better if you non-dimensionalize so that the fields have similar scales). -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From u.tabak at tudelft.nl Sun Sep 8 11:01:34 2013 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sun, 08 Sep 2013 18:01:34 +0200 Subject: [petsc-users] [PATCH 2/2] Call KSPSetFromOptions before setting fieldsplit-specific options In-Reply-To: <871u4z8g6i.fsf@mcs.anl.gov> References: <522B9869.5060803@tudelft.nl> <1378619690-13533-1-git-send-email-jedbrown@mcs.anl.gov> <1378619690-13533-2-git-send-email-jedbrown@mcs.anl.gov> <522C17B9.9000307@tudelft.nl> <87mwnn95kb.fsf@mcs.anl.gov> <522C4227.5050903@tudelft.nl> <87eh8z8m1k.fsf@mcs.anl.gov> <522C801A.6080907@tudelft.nl> <87a9jn8lgc.fsf@mcs.anl.gov> <522C8518.6070609@tudelft.nl> <877ger8kh0.fsf@mcs.anl.gov> <522C9131.708@tudelft.nl> <522C939D.1070704@tudelft.nl> <871u4z8g6i.fsf@mcs.anl.gov> Message-ID: <522C9F5E.5000301@tudelft.nl> On 09/08/2013 05:51 PM, Jed Brown wrote: > Umut Tabak writes: > > > Looks like it's converging, albeit somewhat slowly, and your operator > scaling is terrible (not necessarily a problem, but full-space > fieldsplit methods work much better if you non-dimensionalize so that > the fields have similar scales). Indeed, something I am aware of, A00 is ill-conditioned A11 block is indefinite, even worse. I have been looking for some scaling options that might work but until now no success so far... From jedbrown at mcs.anl.gov Sun Sep 8 11:17:23 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 08 Sep 2013 11:17:23 -0500 Subject: [petsc-users] [PATCH 2/2] Call KSPSetFromOptions before setting fieldsplit-specific options In-Reply-To: <522C9F5E.5000301@tudelft.nl> References: <522B9869.5060803@tudelft.nl> <1378619690-13533-1-git-send-email-jedbrown@mcs.anl.gov> <1378619690-13533-2-git-send-email-jedbrown@mcs.anl.gov> <522C17B9.9000307@tudelft.nl> <87mwnn95kb.fsf@mcs.anl.gov> <522C4227.5050903@tudelft.nl> <87eh8z8m1k.fsf@mcs.anl.gov> <522C801A.6080907@tudelft.nl> <87a9jn8lgc.fsf@mcs.anl.gov> <522C8518.6070609@tudelft.nl> <877ger8kh0.fsf@mcs.anl.gov> <522C9131.708@tudelft.nl> <522C939D.1070704@tudelft.nl> <871u4z8g6i.fsf@mcs.anl.gov> <522C9F5E.5000301@tudelft.nl> Message-ID: <87sixf70fg.fsf@mcs.anl.gov> Umut Tabak writes: > On 09/08/2013 05:51 PM, Jed Brown wrote: >> Umut Tabak writes: >> >> >> Looks like it's converging, albeit somewhat slowly, and your operator >> scaling is terrible (not necessarily a problem, but full-space >> fieldsplit methods work much better if you non-dimensionalize so that >> the fields have similar scales). > > Indeed, something I am aware of, > > A00 > > is ill-conditioned > > A11 > > block is indefinite, even worse. A11 indefinite could lead to the Schur complement being singular. The approach breaks down if that's the case. A11 singular or negative semi-definite is the standard case. > I have been looking for some scaling options that might work but until > now no success so far... You need to read the literature for problems of your type. There are a few possible approaches to approximating the Schur complement. The splitting into blocks also might not be very good. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From u.tabak at tudelft.nl Sun Sep 8 11:34:12 2013 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sun, 08 Sep 2013 18:34:12 +0200 Subject: [petsc-users] [PATCH 2/2] Call KSPSetFromOptions before setting fieldsplit-specific options In-Reply-To: <87sixf70fg.fsf@mcs.anl.gov> References: <522B9869.5060803@tudelft.nl> <1378619690-13533-1-git-send-email-jedbrown@mcs.anl.gov> <1378619690-13533-2-git-send-email-jedbrown@mcs.anl.gov> <522C17B9.9000307@tudelft.nl> <87mwnn95kb.fsf@mcs.anl.gov> <522C4227.5050903@tudelft.nl> <87eh8z8m1k.fsf@mcs.anl.gov> <522C801A.6080907@tudelft.nl> <87a9jn8lgc.fsf@mcs.anl.gov> <522C8518.6070609@tudelft.nl> <877ger8kh0.fsf@mcs.anl.gov> <522C9131.708@tudelft.nl> <522C939D.1070704@tudelft.nl> <871u4z8g6i.fsf@mcs.anl.gov> <522C9F5E.5000301@tudelft.nl> <87sixf70fg.fsf@mcs.anl.gov> Message-ID: <522CA704.3040204@tudelft.nl> On 09/08/2013 06:17 PM, Jed Brown wrote: > block is indefinite, even worse. > A11 indefinite could lead to the Schur complement being singular. The > approach breaks down if that's the case. A11 singular or negative > semi-definite is the standard case. Even not singular, an already ill-conditioned problem becomes worse with a Schur complement approach. And it is almost impossible to define a SPD preconditioner that could work with S to result in decent iteration counts. This was a question I asked a couple of weeks ago but did not have the time to go into details. I guess these kinds of separation ideas and use of independent factorizations or combined preconditioners for different blocks will not work for this problem... > > You need to read the literature for problems of your type. There are a > few possible approaches to approximating the Schur complement. The > splitting into blocks also might not be very good. well I took a look at the literature but could not really find something useful up until now. The main difference was one of the blocks was either SPD or well-conditioned(for A00 or A11). More specifically, my problem is a kind of shifted problem for eigenvalue solutions without going into too much detail. The symmetric operator matrix is written as AA = (A-\sigmaB) C C^T (D-\sigmaE) and C is a rather sparse coupling matrix A00 = A-\sigmaB block is ill conditioned due to the shift and A11 = (D-\sigmaE) block is indefinite where D is a singular matrix with one zero eigenvalue with 1 vector in the null space. E is rather well conditioned but in combination it is not attackable by iterative methods. Moreover, Schur complements are defined on this problematic system, AA. There is some literature on these shifted problems but I am not sure if I should dive into that field or not at this point. Trying the 'fieldsplit' approach was some kind of a 'what-if' for me... But thanks for the help and comments anyway. From hariseldon99 at gmail.com Sun Sep 8 11:34:19 2013 From: hariseldon99 at gmail.com (Analabha Roy) Date: Sun, 8 Sep 2013 22:04:19 +0530 Subject: [petsc-users] Petsc HDF5. Storing matrices and adding attributes In-Reply-To: <216E7BC1-9B64-4D02-98AC-C178FB007BA5@mcs.anl.gov> References: <216E7BC1-9B64-4D02-98AC-C178FB007BA5@mcs.anl.gov> Message-ID: On Sep 8, 2013 8:39 PM, "Barry Smith" wrote: > > > On Sep 8, 2013, at 10:00 AM, Analabha Roy wrote: > > > Hi all, > > > > Firstly: > > I need to store a matrix in HDF5. I was reading through petsc manual pages and examples and only saw information for storing vectors in HDF5. Using MatView() on a HDF5 viewer doesn't work. Is it even possible? > > > Is your matrix dense? If your matrix is sparse why do you even want to store it? and if you do want to store it why do you want to use HDF5 (it won't be any faster than just using the PETSc binary format). It's sparse, but will binary output be cross platform portable? > Generally people store vectors in HDF5 because they have some other tool (like a visualization package) that can read HDF5 easily. Don't think that exists for sparse matrices. > > Barry > > > > > For now, I'm working around by extracting the columns from my Mat and storing them as vecs by looping VecView(). > > > > > > Secondly: > > Is there a way to add attributes to HDF5 data using the PetSc API? I didn't see any docs on this. > > > > > > > > Thanks in advance, > > AR > > > > -- > > --- > > Analabha Roy > > C.S.I.R Senior Research Associate > > Saha Institute of Nuclear Physics > > Section 1, Block AF > > Bidhannagar, Calcutta 700064 > > India > > Emails: daneel at physics.utexas.edu, hariseldon99 at gmail.com > > Webpage: http://www.ph.utexas.edu/~daneel/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Sep 8 11:35:01 2013 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 8 Sep 2013 11:35:01 -0500 Subject: [petsc-users] Petsc HDF5. Storing matrices and adding attributes In-Reply-To: References: <216E7BC1-9B64-4D02-98AC-C178FB007BA5@mcs.anl.gov> Message-ID: On Sun, Sep 8, 2013 at 11:34 AM, Analabha Roy wrote: > > On Sep 8, 2013 8:39 PM, "Barry Smith" wrote: > > > > > > On Sep 8, 2013, at 10:00 AM, Analabha Roy > wrote: > > > > > Hi all, > > > > > > Firstly: > > > I need to store a matrix in HDF5. I was reading through petsc manual > pages and examples and only saw information for storing vectors in HDF5. > Using MatView() on a HDF5 viewer doesn't work. Is it even possible? > > > > > Is your matrix dense? If your matrix is sparse why do you even want > to store it? and if you do want to store it why do you want to use HDF5 (it > won't be any faster than just using the PETSc binary format). > > It's sparse, but will binary output be cross platform portable? > Yes. Matt > > Generally people store vectors in HDF5 because they have some other > tool (like a visualization package) that can read HDF5 easily. Don't think > that exists for sparse matrices. > > > > > Barry > > > > > > > > > For now, I'm working around by extracting the columns from my Mat and > storing them as vecs by looping VecView(). > > > > > > > > > Secondly: > > > Is there a way to add attributes to HDF5 data using the PetSc API? I > didn't see any docs on this. > > > > > > > > > > > > Thanks in advance, > > > AR > > > > > > -- > > > --- > > > Analabha Roy > > > C.S.I.R Senior Research Associate > > > Saha Institute of Nuclear Physics > > > Section 1, Block AF > > > Bidhannagar, Calcutta 700064 > > > India > > > Emails: daneel at physics.utexas.edu, hariseldon99 at gmail.com > > > Webpage: http://www.ph.utexas.edu/~daneel/ > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sun Sep 8 11:45:44 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 08 Sep 2013 11:45:44 -0500 Subject: [petsc-users] [PATCH 2/2] Call KSPSetFromOptions before setting fieldsplit-specific options In-Reply-To: <522CA704.3040204@tudelft.nl> References: <522B9869.5060803@tudelft.nl> <1378619690-13533-1-git-send-email-jedbrown@mcs.anl.gov> <1378619690-13533-2-git-send-email-jedbrown@mcs.anl.gov> <522C17B9.9000307@tudelft.nl> <87mwnn95kb.fsf@mcs.anl.gov> <522C4227.5050903@tudelft.nl> <87eh8z8m1k.fsf@mcs.anl.gov> <522C801A.6080907@tudelft.nl> <87a9jn8lgc.fsf@mcs.anl.gov> <522C8518.6070609@tudelft.nl> <877ger8kh0.fsf@mcs.anl.gov> <522C9131.708@tudelft.nl> <522C939D.1070704@tudelft.nl> <871u4z8g6i.fsf@mcs.anl.gov> <522C9F5E.5000301@tudelft.nl> <87sixf70fg.fsf@mcs.anl.gov> <522CA704.3040204@tudelft.nl> Message-ID: <87ppsj6z47.fsf@mcs.anl.gov> Umut Tabak writes: > On 09/08/2013 06:17 PM, Jed Brown wrote: >> block is indefinite, even worse. >> A11 indefinite could lead to the Schur complement being singular. The >> approach breaks down if that's the case. A11 singular or negative >> semi-definite is the standard case. > Even not singular, an already ill-conditioned problem becomes worse with > a Schur complement approach. I don't know what you're trying to say, but using factorization makes sense when you set the problem up so that the Schur complement is better behaved than the original problem (either because it's better conditioned or because you have a practical preconditioner for it). Of course you can choose bad splittings so that the Schur complement is worse (or singular). Note that the Schur complement can be completely different from what was in that block of the matrix before factorization. > And it is almost impossible to define a SPD preconditioner that could > work with S to result in decent iteration counts. > > This was a question I asked a couple of weeks ago but did not have the > time to go into details. > > I guess these kinds of separation ideas and use of independent > factorizations or combined preconditioners for different blocks will not > work for this problem... Possible, or you're doing it wrong. >> You need to read the literature for problems of your type. There are a >> few possible approaches to approximating the Schur complement. The >> splitting into blocks also might not be very good. > well I took a look at the literature but could not really find something > useful up until now. The main difference was one of the blocks was > either SPD or well-conditioned(for A00 or A11). > > More specifically, my problem is a kind of shifted problem for > eigenvalue solutions without going into too much detail. The symmetric > operator matrix is written as > > AA > = > (A-\sigmaB) C > C^T (D-\sigmaE) > > and C is a rather sparse coupling matrix > > A00 = A-\sigmaB block is ill conditioned due to the shift > > and > > A11 = (D-\sigmaE) > > block is indefinite where D is a singular matrix with one zero > eigenvalue with 1 vector in the null space. E is rather well conditioned > but in combination it is not attackable by iterative methods. Moreover, > Schur complements are defined on this problematic system, AA. Take a small problem size and use SVD to compute the extreme singular values of S = A11 - C^T A00^{-1} C. Also get an estimate of the spectrum so you can tell whether it's indefinite. > There is some literature on these shifted problems but I am not sure if > I should dive into that field or not at this point. > > Trying the 'fieldsplit' approach was some kind of a 'what-if' for me... > > But thanks for the help and comments anyway. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From info at jenniferharwellart.com Sun Sep 8 13:38:03 2013 From: info at jenniferharwellart.com (Jennifer Harwell) Date: Sun, 08 Sep 2013 18:38:03 +0000 (GMT) Subject: [petsc-users] Start Your Morning With A Jennifer Harwell Angel Message-ID: <1378665483.42656336822240342@mf67.sendgrid.net> Email not displaying correctly? View it ( http://sendgrid.org/wf/webmail?rp=ZTI1bGQzTnNaWFIwWlhKZmFXUTZNVEl6TkN4MWMyVnlYMmxrT2pJMU5qVTBmUWV5SnVaWGR6YkdWMGRHVnlYMmxrSWpvaU1qRXlOamcwTWlJc0ltNWxkM05zWlhSMFpYSmZkWE5sY2w5cFpDSTZNelUwTkRNeE16a3hORE45 ) in your browser -------------------------------------------------------------------------------- Jennifer Harwell ================ ? ? ? ? ? ? ? ? ? -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- To unsubscribe please click here ( http://u352642.sendgrid.org/wf/unsubscribe?upn=hso2fBppV7DoRsl5Oo6VoiKnhTkwdPRRSDibcyZycu-2FMsSh38MPB-2FcjS4An7NuRGUjyeneaYHZ94pb-2BGStQ6uyHU8e-2FHmVidbBNLZkCF9fJ8XiyM9yRvZggUzDVZxIkoBqueuaQ8TunA-2FHNzBe4jFFu5Swg8ddb9gtBS4Z-2F1isi5K3chL2UOnEPqjy7EXcvv9celloUN-2F5JMQ94wuBhmXVFYeTLt1tj8tPk0wgsb4VVrToBXSg8Jojabwo2uBuxzSdyWQWjm21KA-2BMYP13vBKlMC-2BkUuoSJO-2Bui9IiRW4TgyiZmC-2F61GDUg1xQS7dY2Cd2ZRwFGiGkBYEV4lnKVGLVUAjbTTZr3FQm-2B-2BcZ0jDBVMhPTgE75X0KehxUgxGKDlh5MhUW4UJmshXcksXi6cnQ-3D-3D ) jenniferharwell PO Box 59258 Birmingham, AL 35259 -------------- next part -------------- An HTML attachment was scrubbed... URL: From fangxingjun0319 at gmail.com Mon Sep 9 15:10:33 2013 From: fangxingjun0319 at gmail.com (Frank) Date: Mon, 09 Sep 2013 15:10:33 -0500 Subject: [petsc-users] How to reuse matrix A Message-ID: <522E2B39.7040401@gmail.com> Hi, I am solving Ax=b repeatedly, and A does not change all the time. I did things like this: %1. Set up entries for matrix A%%% CALL MATASSEMBLYBEGIN(A,MAT_FINAL_ASSEMBLY,IERR) CALL MATASSEMBLYEND(A,MAT_FINAL_ASSEMBLY,IERR) CALL MATSETOPTION(A,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE,ierr) CALL KSPSetOperators(ksp,A,A,SAME_NONZERO_PATTERN,ierr) Call KSPSetOperators(ksp,A,A,SAME_NONZERO_PATTERN,ierr) Do i=1,100 % Set up entries for vector b%%%%% CALL VECASSEMBLYBEGIN(b,IERR) CALL VECASSEMBLYEND(b,IERR) CALL KSPSolve(ksp,b,x,ierr) ENDDO However, it does not work. I don't know why. Could anybody help me with this? Thank you very much. Sincerely Xingjun Fang From knepley at gmail.com Mon Sep 9 15:14:37 2013 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 9 Sep 2013 15:14:37 -0500 Subject: [petsc-users] How to reuse matrix A In-Reply-To: <522E2B39.7040401@gmail.com> References: <522E2B39.7040401@gmail.com> Message-ID: On Mon, Sep 9, 2013 at 3:10 PM, Frank wrote: > Hi, > > I am solving Ax=b repeatedly, and A does not change all the time. I did > things like this: > > %1. Set up entries for matrix A%%% > CALL MATASSEMBLYBEGIN(A,MAT_FINAL_**ASSEMBLY,IERR) > CALL MATASSEMBLYEND(A,MAT_FINAL_**ASSEMBLY,IERR) > CALL MATSETOPTION(A,MAT_NEW_**NONZERO_LOCATIONS,PETSC_FALSE,**ierr) > CALL KSPSetOperators(ksp,A,A,SAME_**NONZERO_PATTERN,ierr) > > Call KSPSetOperators(ksp,A,A,SAME_**NONZERO_PATTERN,ierr) > > Do i=1,100 > % Set up entries for vector b%%%%% > CALL VECASSEMBLYBEGIN(b,IERR) > CALL VECASSEMBLYEND(b,IERR) > CALL KSPSolve(ksp,b,x,ierr) > ENDDO > > However, it does not work. I don't know why. Could anybody help me with > this? Thank you very much. > "Does not work" is an unacceptable bug report. With that you are likely to get an answer like "yes it does". Matt > Sincerely > Xingjun Fang > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From u.tabak at tudelft.nl Mon Sep 9 16:47:25 2013 From: u.tabak at tudelft.nl (Umut Tabak) Date: Mon, 09 Sep 2013 23:47:25 +0200 Subject: [petsc-users] [PATCH 2/2] Call KSPSetFromOptions before setting fieldsplit-specific options In-Reply-To: <87ppsj6z47.fsf@mcs.anl.gov> References: <522B9869.5060803@tudelft.nl> <1378619690-13533-1-git-send-email-jedbrown@mcs.anl.gov> <1378619690-13533-2-git-send-email-jedbrown@mcs.anl.gov> <522C17B9.9000307@tudelft.nl> <87mwnn95kb.fsf@mcs.anl.gov> <522C4227.5050903@tudelft.nl> <87eh8z8m1k.fsf@mcs.anl.gov> <522C801A.6080907@tudelft.nl> <87a9jn8lgc.fsf@mcs.anl.gov> <522C8518.6070609@tudelft.nl> <877ger8kh0.fsf@mcs.anl.gov> <522C9131.708@tudelft.nl> <522C939D.1070704@tudelft.nl> <871u4z8g6i.fsf@mcs.anl.gov> <522C9F5E.5000301@tudelft.nl> <87sixf70fg.fsf@mcs.anl.gov> <522CA704.3040204@tudelft.nl> <87ppsj6z47.fsf@mcs.anl.gov> Message-ID: <522E41ED.9050100@tudelft.nl> On 09/08/2013 06:45 PM, Jed Brown wrote: > Umut Tabak writes: > >> On 09/08/2013 06:17 PM, Jed Brown wrote: >>> block is indefinite, even worse. >>> A11 indefinite could lead to the Schur complement being singular. The >>> approach breaks down if that's the case. A11 singular or negative >>> semi-definite is the standard case. >> Even not singular, an already ill-conditioned problem becomes worse with >> a Schur complement approach. > I don't know what you're trying to say, Hi Jed, I thought a bit more on this problem, if there are some matrices (A-\sigmaB) C C^T (D-\sigmaE) in blocks. The Schur complement is S = (D-\sigmaE) - C^T(A-\sigmaB)^{-1}C Is there a way to approximate the condition number of S, cheaply, not that I know of but just asking? > but using factorization makes > sense when you set the problem up so that the Schur complement is better > behaved than the original problem (either because it's better > conditioned or because you have a practical preconditioner for it). Of > course you can choose bad splittings so that the Schur complement is > worse (or singular). Can you rephrase the above, I could not understand which factorization is mentioned above for instance? > > Note that the Schur complement can be completely different from what was > in that block of the matrix before factorization. ok which factorization again? > >> And it is almost impossible to define a SPD preconditioner that could >> work with S to result in decent iteration counts. >> >> This was a question I asked a couple of weeks ago but did not have the >> time to go into details. >> >> I guess these kinds of separation ideas and use of independent >> factorizations or combined preconditioners for different blocks will not >> work for this problem... > Possible, or you're doing it wrong. > >>> You need to read the literature for problems of your type. There are a >>> few possible approaches to approximating the Schur complement. The >>> splitting into blocks also might not be very good. >> well I took a look at the literature but could not really find something >> useful up until now. The main difference was one of the blocks was >> either SPD or well-conditioned(for A00 or A11). >> >> More specifically, my problem is a kind of shifted problem for >> eigenvalue solutions without going into too much detail. The symmetric >> operator matrix is written as >> >> AA >> = >> (A-\sigmaB) C >> C^T (D-\sigmaE) >> >> and C is a rather sparse coupling matrix >> >> A00 = A-\sigmaB block is ill conditioned due to the shift >> >> and >> >> A11 = (D-\sigmaE) >> >> block is indefinite where D is a singular matrix with one zero >> eigenvalue with 1 vector in the null space. E is rather well conditioned >> but in combination it is not attackable by iterative methods. Moreover, >> Schur complements are defined on this problematic system, AA. > Take a small problem size and use SVD to compute the extreme singular > values of S = A11 - C^T A00^{-1} C. Also get an estimate of the > spectrum so you can tell whether it's indefinite. Ok took a small problem and explicitly computed S MATLAB eigs returns s = -9.573038036040949e-04 -1.030047395371149e-02 1.217726918930923e-02 1.532790465307093e-02 -2.037544234417839e-02 2.038856908922073e-02 -3.080232110141994e-02 3.217578262967955e-02 -3.621899833039557e-02 -4.812298722347947e-02 as the smallest eigenvalues smallest singular values with svds in MATLAB again are 4.812298722126469e-02 3.621899832617895e-02 3.217578263194974e-02 3.080232110105591e-02 2.038856907069070e-02 2.037544233721312e-02 1.532790465411290e-02 1.217726916867209e-02 1.030047395359284e-02 9.573038068612779e-04 > >> There is some literature on these shifted problems but I am not sure if >> I should dive into that field or not at this point. >> >> Trying the 'fieldsplit' approach was some kind of a 'what-if' for me... >> >> But thanks for the help and comments anyway. From knepley at gmail.com Mon Sep 9 17:23:10 2013 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 9 Sep 2013 17:23:10 -0500 Subject: [petsc-users] [PATCH 2/2] Call KSPSetFromOptions before setting fieldsplit-specific options In-Reply-To: <522E41ED.9050100@tudelft.nl> References: <522B9869.5060803@tudelft.nl> <1378619690-13533-1-git-send-email-jedbrown@mcs.anl.gov> <1378619690-13533-2-git-send-email-jedbrown@mcs.anl.gov> <522C17B9.9000307@tudelft.nl> <87mwnn95kb.fsf@mcs.anl.gov> <522C4227.5050903@tudelft.nl> <87eh8z8m1k.fsf@mcs.anl.gov> <522C801A.6080907@tudelft.nl> <87a9jn8lgc.fsf@mcs.anl.gov> <522C8518.6070609@tudelft.nl> <877ger8kh0.fsf@mcs.anl.gov> <522C9131.708@tudelft.nl> <522C939D.1070704@tudelft.nl> <871u4z8g6i.fsf@mcs.anl.gov> <522C9F5E.5000301@tudelft.nl> <87sixf70fg.fsf@mcs.anl.gov> <522CA704.3040204@tudelft.nl> <87ppsj6z47.fsf@mcs.anl.gov> <522E41ED.9050100@tudelft.nl> Message-ID: On Mon, Sep 9, 2013 at 4:47 PM, Umut Tabak wrote: > On 09/08/2013 06:45 PM, Jed Brown wrote: > >> Umut Tabak writes: >> >> On 09/08/2013 06:17 PM, Jed Brown wrote: >>> >>>> block is indefinite, even worse. >>>> A11 indefinite could lead to the Schur complement being singular. The >>>> approach breaks down if that's the case. A11 singular or negative >>>> semi-definite is the standard case. >>>> >>> Even not singular, an already ill-conditioned problem becomes worse with >>> a Schur complement approach. >>> >> I don't know what you're trying to say, >> > Hi Jed, > > I thought a bit more on this problem, if there are some matrices > > (A-\sigmaB) C > C^T (D-\sigmaE) > > in blocks. > > The Schur complement is > > S = (D-\sigmaE) - C^T(A-\sigmaB)^{-1}C > > Is there a way to approximate the condition number of S, cheaply, not that > I know of but just asking? > Not without more information. Have you looked at any papers for your problem? > but using factorization makes >> sense when you set the problem up so that the Schur complement is better >> behaved than the original problem (either because it's better >> conditioned or because you have a practical preconditioner for it). Of >> course you can choose bad splittings so that the Schur complement is >> worse (or singular). >> > Can you rephrase the above, I could not understand which factorization is > mentioned above for instance? > This is only one factorization here, the Schur factorization. > >> Note that the Schur complement can be completely different from what was >> in that block of the matrix before factorization. >> > ok which factorization again? > Above. > >> And it is almost impossible to define a SPD preconditioner that could >>> work with S to result in decent iteration counts. >>> >>> This was a question I asked a couple of weeks ago but did not have the >>> time to go into details. >>> >>> I guess these kinds of separation ideas and use of independent >>> factorizations or combined preconditioners for different blocks will not >>> work for this problem... >>> >> Possible, or you're doing it wrong. >> >> You need to read the literature for problems of your type. There are a >>>> few possible approaches to approximating the Schur complement. The >>>> splitting into blocks also might not be very good. >>>> >>> well I took a look at the literature but could not really find something >>> useful up until now. The main difference was one of the blocks was >>> either SPD or well-conditioned(for A00 or A11). >>> >>> More specifically, my problem is a kind of shifted problem for >>> eigenvalue solutions without going into too much detail. The symmetric >>> operator matrix is written as >>> >>> AA >>> = >>> (A-\sigmaB) C >>> C^T (D-\sigmaE) >>> >>> and C is a rather sparse coupling matrix >>> >>> A00 = A-\sigmaB block is ill conditioned due to the shift >>> >>> and >>> >>> A11 = (D-\sigmaE) >>> >>> block is indefinite where D is a singular matrix with one zero >>> eigenvalue with 1 vector in the null space. E is rather well conditioned >>> but in combination it is not attackable by iterative methods. Moreover, >>> Schur complements are defined on this problematic system, AA. >>> >> Take a small problem size and use SVD to compute the extreme singular >> values of S = A11 - C^T A00^{-1} C. Also get an estimate of the >> spectrum so you can tell whether it's indefinite. >> > Ok took a small problem and explicitly computed S > > MATLAB eigs returns > > s = > > -9.573038036040949e-04 > -1.030047395371149e-02 > 1.217726918930923e-02 > 1.532790465307093e-02 > -2.037544234417839e-02 > 2.038856908922073e-02 > -3.080232110141994e-02 > 3.217578262967955e-02 > -3.621899833039557e-02 > -4.812298722347947e-02 > > as the smallest eigenvalues > > smallest singular values with svds in MATLAB again are > > 4.812298722126469e-02 > 3.621899832617895e-02 > 3.217578263194974e-02 > 3.080232110105591e-02 > 2.038856907069070e-02 > 2.037544233721312e-02 > 1.532790465411290e-02 > 1.217726916867209e-02 > 1.030047395359284e-02 > 9.573038068612779e-04 > Its not that badly conditioned, so you need to understand your operators better to discover why your solver is not converging. Matt > >> There is some literature on these shifted problems but I am not sure if >>> I should dive into that field or not at this point. >>> >>> Trying the 'fieldsplit' approach was some kind of a 'what-if' for me... >>> >>> But thanks for the help and comments anyway. >>> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From pvsang002 at gmail.com Tue Sep 10 14:48:43 2013 From: pvsang002 at gmail.com (sang pham van) Date: Tue, 10 Sep 2013 15:48:43 -0400 Subject: [petsc-users] Basic question on making SNES continue its iterations In-Reply-To: References: Message-ID: Hi Matt, Is it true that the option -snes_linesearch_type basic will make SNES run with the pure Newton method (without line search nor true region)? Thank you. Sang On Wed, Jun 5, 2013 at 1:04 PM, Matthew Knepley wrote: > On Wed, Jun 5, 2013 at 12:33 PM, Gaurish Telang wrote: > >> I am trying to solve a non-linear system with PETSs's SNES solvers. >> >> To run my code, I am using the following options for snes set through the >> command-line, >> seen below. >> >> mpirun -np 1 ./Testvector \ >> -fBus $1/Bus.mat \ >> -fYm $1/Ym.mat \ >> -ft $1/t.mat \ >> -flinedata $1/linedata.mat \ >> -nox \ >> -snes_atol 1.e-5\ >> -snes_stol 1.e-6 \ >> -snes_rtol 1.e-12 \ >> -snes_max_it 100 \ >> -snes_max_linear_solve_fail 100 \ >> -snes_converged_reason \ >> -ksp_type preonly \ >> -pc_type lu \ >> -pc_factor_mat_solver_package superlu_dist \ >> -snes_monitor \ >> > result_superLU_dist.tmp 2>&1; >> >> >> After feeding my data, I get the following output >> 0 SNES Function norm 2.438306296478e+02 >> 1 SNES Function norm 1.679437793666e+02 >> 2 SNES Function norm 1.493618467597e+02 >> 3 SNES Function norm 4.622194635741e+01 >> 4 SNES Function norm 4.195017632984e+01 >> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 4 >> >> As you can see the the SNES function norm is *decreasing*, which >> indicates it is probably converging slowly to the actual solution. However >> after just 4 iterations, it stops and indicates non-convergence, even >> though my -snes_max_it = 100 as shown above. >> >> Is there a way to tell PETSc to continue its SNES iterations? Maybe some >> there are some >> default parameters which have to be changed? >> > > You can try -snes_max_fail or maybe -snes_linesearch_type basic. > > Matt > > >> Thank you >> >> Gaurish >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Sep 10 14:55:46 2013 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 10 Sep 2013 14:55:46 -0500 Subject: [petsc-users] Basic question on making SNES continue its iterations In-Reply-To: References: Message-ID: On Tue, Sep 10, 2013 at 2:48 PM, sang pham van wrote: > Hi Matt, > > Is it true that the option -snes_linesearch_type basic will make SNES run > with the pure Newton method (without line search nor true region)? > Yes. Matt > Thank you. > Sang > > > On Wed, Jun 5, 2013 at 1:04 PM, Matthew Knepley wrote: > >> On Wed, Jun 5, 2013 at 12:33 PM, Gaurish Telang wrote: >> >>> I am trying to solve a non-linear system with PETSs's SNES solvers. >>> >>> To run my code, I am using the following options for snes set through >>> the command-line, >>> seen below. >>> >>> mpirun -np 1 ./Testvector \ >>> -fBus $1/Bus.mat \ >>> -fYm $1/Ym.mat \ >>> -ft $1/t.mat \ >>> -flinedata $1/linedata.mat \ >>> -nox \ >>> -snes_atol 1.e-5\ >>> -snes_stol 1.e-6 \ >>> -snes_rtol 1.e-12 \ >>> -snes_max_it 100 \ >>> -snes_max_linear_solve_fail 100 \ >>> -snes_converged_reason \ >>> -ksp_type preonly \ >>> -pc_type lu \ >>> -pc_factor_mat_solver_package superlu_dist \ >>> -snes_monitor \ >>> > result_superLU_dist.tmp 2>&1; >>> >>> >>> After feeding my data, I get the following output >>> 0 SNES Function norm 2.438306296478e+02 >>> 1 SNES Function norm 1.679437793666e+02 >>> 2 SNES Function norm 1.493618467597e+02 >>> 3 SNES Function norm 4.622194635741e+01 >>> 4 SNES Function norm 4.195017632984e+01 >>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 4 >>> >>> As you can see the the SNES function norm is *decreasing*, which >>> indicates it is probably converging slowly to the actual solution. However >>> after just 4 iterations, it stops and indicates non-convergence, even >>> though my -snes_max_it = 100 as shown above. >>> >>> Is there a way to tell PETSc to continue its SNES iterations? Maybe some >>> there are some >>> default parameters which have to be changed? >>> >> >> You can try -snes_max_fail or maybe -snes_linesearch_type basic. >> >> Matt >> >> >>> Thank you >>> >>> Gaurish >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Sep 10 16:16:24 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 10 Sep 2013 16:16:24 -0500 Subject: [petsc-users] Basic question on making SNES continue its iterations In-Reply-To: References: Message-ID: <87ppsgwf6f.fsf@mcs.anl.gov> sang pham van writes: > Hi Matt, > > Is it true that the option -snes_linesearch_type basic will make SNES run > with the pure Newton method (without line search nor true region)? Yes. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From Shuangshuang.Jin at pnnl.gov Tue Sep 10 18:58:48 2013 From: Shuangshuang.Jin at pnnl.gov (Jin, Shuangshuang) Date: Tue, 10 Sep 2013 16:58:48 -0700 Subject: [petsc-users] MatGetRow and MatSetValues Message-ID: <6778DE83AB681D49BFC2CD850610FEB1018FDC98F36A@EMAIL04.pnl.gov> Hello, PETSc developers, I have a question regarding the usage of MatGetRow and MatSetValues. As shown in the piece of code below. I have a matrix A with n rows and N_Steps columns, for each time step I_Steps, I want to set A[x, I_Steps+1] = A[x, I_Steps] + someComputedValue; As I understand, in order to read A[x, I_Steps], I have to get the x's rows values in A by MatGetRow; In order to set the A[x, I_Steps+1], I have to use the MatSetValues. And these operations require the MatAssemblyBegin and MatAssemblyEnd as well. The problem is when I run the code, it only works when the number of processors equal to 1 or n. Otherwise, it's hanging there forever. I feel like the MatAssemblyBegin and MatAssemblyEnd may be the source of the problem. How to fix the error? Or is there a better way to implement this? Thanks, Shuangshuang Mat A; ierr = MatGetOwnershipRange(A, &lo, &hi); CHKERRQ(ierr); const PetscScalar *AVals; for (I_Steps = 0; I_Steps < N_Steps; I_Steps++) { for (k = lo; k < hi; k++) { ierr = MatGetRow(A, k, &ncols, &cols, &AVals); CHKERRQ(ierr); val[0] = AVals[cols[I_Steps]] + someComputedValue; ierr = MatRestoreRow(A, k, &ncols, &cols, &AVals); CHKERRQ(ierr); j = I_Steps+1; ierr = MatSetValues(A, 1, &k, 1, &j, val, INSERT_VALUES); CHKERRQ(ierr); ierr = MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); ierr = MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); MatView(A, PETSC_VIEWER_STDOUT_WORLD); } } -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Sep 10 20:02:56 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 10 Sep 2013 20:02:56 -0500 Subject: [petsc-users] MatGetRow and MatSetValues In-Reply-To: <6778DE83AB681D49BFC2CD850610FEB1018FDC98F36A@EMAIL04.pnl.gov> References: <6778DE83AB681D49BFC2CD850610FEB1018FDC98F36A@EMAIL04.pnl.gov> Message-ID: <87k3iouq4f.fsf@mcs.anl.gov> "Jin, Shuangshuang" writes: > Hello, PETSc developers, I have a question regarding the usage of > MatGetRow and MatSetValues. > > As shown in the piece of code below. I have a matrix A with n rows and > N_Steps columns, for each time step I_Steps, I want to set A[x, > I_Steps+1] = A[x, I_Steps] + someComputedValue; > > As I understand, in order to read A[x, I_Steps], I have to get the x's > rows values in A by MatGetRow; In order to set the A[x, I_Steps+1], I > have to use the MatSetValues. And these operations require the > MatAssemblyBegin and MatAssemblyEnd as well. Yes. > The problem is when I run the code, it only works when the number of > processors equal to 1 or n. Otherwise, it's hanging there forever. > > I feel like the MatAssemblyBegin and MatAssemblyEnd may be the source > of the problem. MatAssemblyBegin and MatAssemblyEnd are collective, but you are not calling them collectively when the process count is not 1 or evenly dividing the problem size. You can get all the values you need in one shot, then set all the values with MatSetValues, then call assembly once. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From Shuangshuang.Jin at pnnl.gov Tue Sep 10 23:32:19 2013 From: Shuangshuang.Jin at pnnl.gov (Jin, Shuangshuang) Date: Tue, 10 Sep 2013 21:32:19 -0700 Subject: [petsc-users] MatGetRow and MatSetValues In-Reply-To: <87k3iouq4f.fsf@mcs.anl.gov> References: <6778DE83AB681D49BFC2CD850610FEB1018FDC98F36A@EMAIL04.pnl.gov> <87k3iouq4f.fsf@mcs.anl.gov> Message-ID: <31C1B683-FB2A-46CA-A0AB-62DE82E4F1A5@pnnl.gov> Jed, thank you very much. The way you suggested works! Thanks, Shuangshuang Sent from my iPhone On Sep 10, 2013, at 18:03, "Jed Brown" wrote: > "Jin, Shuangshuang" writes: > >> Hello, PETSc developers, I have a question regarding the usage of >> MatGetRow and MatSetValues. >> >> As shown in the piece of code below. I have a matrix A with n rows and >> N_Steps columns, for each time step I_Steps, I want to set A[x, >> I_Steps+1] = A[x, I_Steps] + someComputedValue; >> >> As I understand, in order to read A[x, I_Steps], I have to get the x's >> rows values in A by MatGetRow; In order to set the A[x, I_Steps+1], I >> have to use the MatSetValues. And these operations require the >> MatAssemblyBegin and MatAssemblyEnd as well. > > Yes. > >> The problem is when I run the code, it only works when the number of >> processors equal to 1 or n. Otherwise, it's hanging there forever. >> >> I feel like the MatAssemblyBegin and MatAssemblyEnd may be the source >> of the problem. > > MatAssemblyBegin and MatAssemblyEnd are collective, but you are not > calling them collectively when the process count is not 1 or evenly > dividing the problem size. > > You can get all the values you need in one shot, then set all the values > with MatSetValues, then call assembly once. From choi240 at purdue.edu Thu Sep 12 02:31:42 2013 From: choi240 at purdue.edu (Joon Hee Choi) Date: Thu, 12 Sep 2013 03:31:42 -0400 (EDT) Subject: [petsc-users] About MatGetOwnershipRange() In-Reply-To: <1029719976.99392.1378969615445.JavaMail.root@mailhub028.itcs.purdue.edu> Message-ID: <1826029717.99414.1378971102526.JavaMail.root@mailhub028.itcs.purdue.edu> Hello, I am trying to set up a large MATMPIAIJ matrix from tuples. The tuples is pairs of (row, column, value). I will use 12~20 machines for this implementation. However, I think I need to know rows on each process before preallocation, because for large matrices the exact preallocation is important for good performance. Thus, I tried to use MatGetOwnershipRange(), but I found out that I cannot use it before preallocation. Could you let me know how to change my code? I am attaching the code. Thank you, Joon PetscErrorCode SetUpMatrix(Mat *X, vector< std::tr1::tuple< PetscInt, PetscInt, PetscInt > >, PetscInt I, PetscInt J) { PetscInt i, j; PetscScalar val; MatCreate(PETSC_COMM_WORLD, X); MatSetType(*X, MATMPIAIJ); MatSetSizes(*X, PETSC_DECIDE, PETSC_DECIDE, I, J); PetscInt begin, end; MatGetOwnershipRange(*X, &begin, &end); PetscInt localsize = end - begin; PetscInt d_nnz[localsize]; PetscInt o_nnz[localsize]; for (int x=0; x(tups[x]); if (i>=end) break; if (i>=begin) { j = std::tr1::get<1>(tups[x]); if (j>=begin && j(tups[x]); if (i>=end) break; if (i>=begin) { j = std::tr1::get<1>(tups[x]); val = std::tr1::get<2>(tups[x]); MatSetValues(*X, 1, &i, 1, &j, &val, INSERT_VALUES); } } MatAssemblyBegin(*X, MAT_FINAL_ASSEMBLY); MatAssemblyEnd(*X, MAT_FINAL_ASSEMBLY); } From jedbrown at mcs.anl.gov Thu Sep 12 03:04:17 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 12 Sep 2013 01:04:17 -0700 Subject: [petsc-users] About MatGetOwnershipRange() In-Reply-To: <1826029717.99414.1378971102526.JavaMail.root@mailhub028.itcs.purdue.edu> References: <1826029717.99414.1378971102526.JavaMail.root@mailhub028.itcs.purdue.edu> Message-ID: <874n9qv532.fsf@mcs.anl.gov> Joon Hee Choi writes: > Hello, > > I am trying to set up a large MATMPIAIJ matrix from tuples. The tuples > is pairs of (row, column, value). I will use 12~20 machines for this > implementation. However, I think I need to know rows on each process > before preallocation, because for large matrices the exact > preallocation is important for good performance. Thus, I tried to use > MatGetOwnershipRange(), but I found out that I cannot use it before > preallocation. Could you let me know how to change my code? I am > attaching the code. > > Thank you, > Joon > > > PetscErrorCode SetUpMatrix(Mat *X, vector< std::tr1::tuple< PetscInt, PetscInt, PetscInt > >, PetscInt I, PetscInt J) > { > PetscInt i, j; > PetscScalar val; > > MatCreate(PETSC_COMM_WORLD, X); > MatSetType(*X, MATMPIAIJ); > MatSetSizes(*X, PETSC_DECIDE, PETSC_DECIDE, I, J); > > PetscInt begin, end; > MatGetOwnershipRange(*X, &begin, &end); Use PetscSplitOwnership to determine sizes rows/columns, then MPI_Scan (or MPI_Exscan) to find your starting offset. Alternatively, use the following sequence (from PetscLayoutCreate). Notes: Typical calling sequence PetscLayoutCreate(MPI_Comm,PetscLayout *); PetscLayoutSetBlockSize(PetscLayout,1); PetscLayoutSetSize(PetscLayout,n) or PetscLayoutSetLocalSize(PetscLayout,N); PetscLayoutSetUp(PetscLayout); Optionally use any of the following: PetscLayoutGetSize(PetscLayout,PetscInt *); or PetscLayoutGetLocalSize(PetscLayout,PetscInt *;) PetscLayoutGetRange(PetscLayout,PetscInt *rstart,PetscInt *rend); or PetscLayoutGetRanges(PetscLayout,const PetscInt *range[]) PetscLayoutDestroy(PetscLayout*); > PetscInt localsize = end - begin; > > PetscInt d_nnz[localsize]; > PetscInt o_nnz[localsize]; This allocation will overflow your stack. You probably want to put it on the heap. > for (int x=0; x i = std::tr1::get<0>(tups[x]); > if (i>=end) break; Are your tuples sorted to begin with? Are they specified redundantly or distributed? Are any of the tuples nonlocal? If it is distributed without matching the row distribution, then you need to communicate d_nnz and o_nnz. > if (i>=begin) { > j = std::tr1::get<1>(tups[x]); > if (j>=begin && j else o_nnz[i-begin]++; > } > } > MatMPIAIJSetPreallocation(*X, 0, d_nnz, 0, o_nnz); > > for (int x=0; x i = std::tr1::get<0>(tups[x]); > if (i>=end) break; > if (i>=begin) { > j = std::tr1::get<1>(tups[x]); > val = std::tr1::get<2>(tups[x]); > MatSetValues(*X, 1, &i, 1, &j, &val, INSERT_VALUES); > } > } > MatAssemblyBegin(*X, MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(*X, MAT_FINAL_ASSEMBLY); > } -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From ztdepyahoo at 163.com Thu Sep 12 09:43:10 2013 From: ztdepyahoo at 163.com (=?GBK?B?tqHAz8qm?=) Date: Thu, 12 Sep 2013 22:43:10 +0800 (CST) Subject: [petsc-users] how to simultaneously perform the partition of the unstructured mesh and reduction bandwidth of the resulting sparsematrix Message-ID: <7d9de2eb.26557.14112a0334f.Coremail.ztdepyahoo@163.com> -------------- next part -------------- An HTML attachment was scrubbed... URL: From u.tabak at tudelft.nl Thu Sep 12 12:04:45 2013 From: u.tabak at tudelft.nl (Umut Tabak) Date: Thu, 12 Sep 2013 19:04:45 +0200 Subject: [petsc-users] any good references on matrix scaling Message-ID: <5231F42D.6030600@tudelft.nl> Dear all, I am looking for some references for matrix scaling where I would like to improve the conditioning of a block representation such as A C C^T B where the terms in A and B blocks are of very different orders, such as A is of 1e+8 and B for instance is of order 1e-4 . The ratios can get even worse but not better on the problems I tried. I am trying have an overview for some cheap conditioning approaches for this problem. Any ideas/references are appreciated. BR, Umut From jedbrown at mcs.anl.gov Thu Sep 12 12:59:59 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 12 Sep 2013 10:59:59 -0700 Subject: [petsc-users] how to simultaneously perform the partition of the unstructured mesh and reduction bandwidth of the resulting sparsematrix In-Reply-To: <7d9de2eb.26557.14112a0334f.Coremail.ztdepyahoo@163.com> References: <7d9de2eb.26557.14112a0334f.Coremail.ztdepyahoo@163.com> Message-ID: <87six9udi8.fsf@mcs.anl.gov> Matrix bandwidth minimization is an (imprecise) optimization to improve cache reuse. So just partition globally and then reorder locally. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From bsmith at mcs.anl.gov Thu Sep 12 13:48:08 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 12 Sep 2013 13:48:08 -0500 Subject: [petsc-users] any good references on matrix scaling In-Reply-To: <5231F42D.6030600@tudelft.nl> References: <5231F42D.6030600@tudelft.nl> Message-ID: <72EDE615-2689-434B-B1EC-E8F035B56D03@mcs.anl.gov> If the matrices already have that very different scaling then it is too late. You want to scale the variables in your physics initially before it ever gets to the discretization so that the scaling is as good as it can be. http://en.wikipedia.org/wiki/Nondimensionalization Barry On Sep 12, 2013, at 12:04 PM, Umut Tabak wrote: > Dear all, > > I am looking for some references for matrix scaling where I would like to improve the conditioning of a block representation such as > > A C > C^T B > > where the terms in A and B blocks are of very different orders, such as A is of 1e+8 and B for instance is of order 1e-4 . The ratios can get even worse but not better on the problems I tried. > > I am trying have an overview for some cheap conditioning approaches for this problem. > > Any ideas/references are appreciated. > > BR, > Umut From bsmith at mcs.anl.gov Thu Sep 12 13:54:38 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 12 Sep 2013 13:54:38 -0500 Subject: [petsc-users] how to simultaneously perform the partition of the unstructured mesh and reduction bandwidth of the resulting sparsematrix In-Reply-To: <87six9udi8.fsf@mcs.anl.gov> References: <7d9de2eb.26557.14112a0334f.Coremail.ztdepyahoo@163.com> <87six9udi8.fsf@mcs.anl.gov> Message-ID: <6F855A74-EED0-48F7-831A-1AD9C716BDB9@mcs.anl.gov> On Sep 12, 2013, at 12:59 PM, Jed Brown wrote: > Matrix bandwidth minimization is an (imprecise) optimization to improve > cache reuse. So just partition globally and then reorder locally. The mesh, before you start the linear algebra. From jedbrown at mcs.anl.gov Thu Sep 12 14:00:10 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 12 Sep 2013 12:00:10 -0700 Subject: [petsc-users] any good references on matrix scaling In-Reply-To: <5231F42D.6030600@tudelft.nl> References: <5231F42D.6030600@tudelft.nl> Message-ID: <87fvt9uapx.fsf@mcs.anl.gov> Umut Tabak writes: > Dear all, > > I am looking for some references for matrix scaling where I would like > to improve the conditioning of a block representation such as > > A C > C^T B > > where the terms in A and B blocks are of very different orders, such as > A is of 1e+8 and B for instance is of order 1e-4 . The ratios can get > even worse but not better on the problems I tried. I recommend non-dimensionalizing (i.e., choosing units) so the system is better scaled. > I am trying have an overview for some cheap conditioning approaches for > this problem. Most preconditioners use the diagonal enough to fix the disparate scales. If you are using full-space iteration with PCFIELDSPLIT, you'll likely have to scale the variables first. You can always use Schur complement reduction. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From u.tabak at tudelft.nl Thu Sep 12 15:42:30 2013 From: u.tabak at tudelft.nl (Umut Tabak) Date: Thu, 12 Sep 2013 22:42:30 +0200 Subject: [petsc-users] any good references on matrix scaling In-Reply-To: <72EDE615-2689-434B-B1EC-E8F035B56D03@mcs.anl.gov> References: <5231F42D.6030600@tudelft.nl> <72EDE615-2689-434B-B1EC-E8F035B56D03@mcs.anl.gov> Message-ID: <52322736.6080002@tudelft.nl> On 09/12/2013 08:48 PM, Barry Smith wrote: > > http://en.wikipedia.org/wiki/Nondimensionalization Hi Barry, For the kind of problems, I am working on, this is a general problem and unfortunately, it is not in general possible to get away from this property with the scaling on the initial differential equations. Moreover, the problem is two sided, a structure coupled to a fluid domain, where the physics is already inducing different scaling and math is obeying that ;-) This was a problem I discussed here extensively last week. Solving directly is not a problem, the challenge is, if possible, to use Schur complements and iterative solution techniques, where the source of the problem emerged from. Thanks for the comments anyway. BR, Umut > > Barry > > On Sep 12, 2013, at 12:04 PM, Umut Tabak wrote: > >> Dear all, >> >> I am looking for some references for matrix scaling where I would like to improve the conditioning of a block representation such as >> >> A C >> C^T B >> >> where the terms in A and B blocks are of very different orders, such as A is of 1e+8 and B for instance is of order 1e-4 . The ratios can get even worse but not better on the problems I tried. >> >> I am trying have an overview for some cheap conditioning approaches for this problem. >> >> Any ideas/references are appreciated. >> >> BR, >> Umut From bsmith at mcs.anl.gov Thu Sep 12 16:01:52 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 12 Sep 2013 16:01:52 -0500 Subject: [petsc-users] any good references on matrix scaling In-Reply-To: <52322736.6080002@tudelft.nl> References: <5231F42D.6030600@tudelft.nl> <72EDE615-2689-434B-B1EC-E8F035B56D03@mcs.anl.gov> <52322736.6080002@tudelft.nl> Message-ID: <83539F97-5063-4808-A6E0-F4D6C9E79697@mcs.anl.gov> If when properly pre scaled and then discretized there is a scaling difference in terms that actually matter of 10^12 then "I" would simply not believe results of any computation done in double precision. With the gnu compilers you can build a PETSc version that computes in quad precision, which gives more than 30 decimal digits to compute with, using the ./configure option --with-precision=__float128. You have to declare all variables as PetscScalar or PetscReal (not double) and be careful how you print them (for example cast to double) but otherwise it is pretty much just like double precision. The problems will still be difficult to solve in quad precision in terms of iterative methods because of the ill-conditioning but at least when you get answers they may have some meaning and convergence will actually tell you something. Good luck, Barry On Sep 12, 2013, at 3:42 PM, Umut Tabak wrote: > On 09/12/2013 08:48 PM, Barry Smith wrote: >> >> http://en.wikipedia.org/wiki/Nondimensionalization > Hi Barry, > > For the kind of problems, I am working on, this is a general problem and unfortunately, it is not in general possible to get away from this property with the scaling on the initial differential equations. > > Moreover, the problem is two sided, a structure coupled to a fluid domain, where the physics is already inducing different scaling and math is obeying that ;-) > > This was a problem I discussed here extensively last week. Solving directly is not a problem, the challenge is, if possible, to use Schur complements and iterative solution techniques, where the source of the problem emerged from. > > Thanks for the comments anyway. > > BR, > Umut >> >> Barry >> >> On Sep 12, 2013, at 12:04 PM, Umut Tabak wrote: >> >>> Dear all, >>> >>> I am looking for some references for matrix scaling where I would like to improve the conditioning of a block representation such as >>> >>> A C >>> C^T B >>> >>> where the terms in A and B blocks are of very different orders, such as A is of 1e+8 and B for instance is of order 1e-4 . The ratios can get even worse but not better on the problems I tried. >>> >>> I am trying have an overview for some cheap conditioning approaches for this problem. >>> >>> Any ideas/references are appreciated. >>> >>> BR, >>> Umut > From choi240 at purdue.edu Thu Sep 12 18:03:53 2013 From: choi240 at purdue.edu (Joon Hee Choi) Date: Thu, 12 Sep 2013 19:03:53 -0400 (EDT) Subject: [petsc-users] About MatGetOwnershipRange() In-Reply-To: <874n9qv532.fsf@mcs.anl.gov> Message-ID: <402438665.102280.1379027033472.JavaMail.root@mailhub028.itcs.purdue.edu> Hi Jed, Thank you for your reply. I tried to follow your method, but I think my code still has something wrong because "localsize" is the same as global row size("I"). Could you let me know what I am missing? My new code is as follows: PetscErrorCode SetUpMatrix(Mat *X, vector< std::tr1::tuple< PetscInt, PetscInt, PetscInt > >, PetscInt I, PetscInt J) { PetscInt i, j; PetscScalar val; MatCreate(PETSC_COMM_WORLD, X); MatSetType(*X, MATMPIAIJ); MatSetSizes(*X, PETSC_DECIDE, PETSC_DECIDE, I, J); PetscInt begin, end, localsize=PETSC_DECIDE; PetscSplitOwnership(PETSC_COMM_WORLD, &localsize, &I); MPI_Scan(&localsize, &end, 1, MPI_INT, MPI_SUM, PETSC_COMM_WORLD); begin = end - localsize; PetscPrintf(PETSC_COMM_WORLD, "Local Size: %d, begin: %d, end: %d \n", localsize, begin, end); ... } Also, tuples are the total pairs of (row dimension, column dimension, element), values of sparse matrix, which are read from a file. The tuples are sorted and distributed. Thus, I tried to set up d_nnz and o_nnz. I implemented d_nnz[i-begin]++ only if begin<="i"&&"j"(tups[x]); if (i>=end) break; if (i>=begin) { j = std::tr1::get<1>(tups[x]); if (j>=begin && j(tups[x]); if (i>=end) break; if (i>=begin) { j = std::tr1::get<1>(tups[x]); val = std::tr1::get<2>(tups[x]); MatSetValues(*X, 1, &i, 1, &j, &val, INSERT_VALUES); } } MatAssemblyBegin(*X, MAT_FINAL_ASSEMBLY); MatAssemblyEnd(*X, MAT_FINAL_ASSEMBLY); } ----- Original Message ----- From: "Jed Brown" To: "Joon Hee Choi" , petsc-users at mcs.anl.gov Sent: Thursday, September 12, 2013 4:04:17 AM Subject: Re: [petsc-users] About MatGetOwnershipRange() Joon Hee Choi writes: >> Hello, >> >> I am trying to set up a large MATMPIAIJ matrix from tuples. The tuples >> is pairs of (row, column, value). I will use 12~20 machines for this >> implementation. However, I think I need to know rows on each process >> before preallocation, because for large matrices the exact >> preallocation is important for good performance. Thus, I tried to use >> MatGetOwnershipRange(), but I found out that I cannot use it before >> preallocation. Could you let me know how to change my code? I am >> attaching the code. >> >> Thank you, >> Joon >> >> >> PetscErrorCode SetUpMatrix(Mat *X, vector< std::tr1::tuple< PetscInt, PetscInt, PetscInt > >, PetscInt I, PetscInt J) >> { >> PetscInt i, j; >> PetscScalar val; >> >> MatCreate(PETSC_COMM_WORLD, X); >> MatSetType(*X, MATMPIAIJ); >> MatSetSizes(*X, PETSC_DECIDE, PETSC_DECIDE, I, J); >> >> PetscInt begin, end; >> MatGetOwnershipRange(*X, &begin, &end); >Use PetscSplitOwnership to determine sizes rows/columns, then MPI_Scan >(or MPI_Exscan) to find your starting offset. Alternatively, use the >following sequence (from PetscLayoutCreate). > > Notes: Typical calling sequence > PetscLayoutCreate(MPI_Comm,PetscLayout *); > PetscLayoutSetBlockSize(PetscLayout,1); > PetscLayoutSetSize(PetscLayout,n) or PetscLayoutSetLocalSize(PetscLayout,N); > PetscLayoutSetUp(PetscLayout); > Optionally use any of the following: > PetscLayoutGetSize(PetscLayout,PetscInt *); or PetscLayoutGetLocalSize(PetscLayout,PetscInt *;) > PetscLayoutGetRange(PetscLayout,PetscInt *rstart,PetscInt *rend); or PetscLayoutGetRanges(PetscLayout,const PetscInt *range[]) > PetscLayoutDestroy(PetscLayout*); >> PetscInt localsize = end - begin; >> >> PetscInt d_nnz[localsize]; >> PetscInt o_nnz[localsize]; >This allocation will overflow your stack. You probably want to put it >on the heap. >> for (int x=0; x> i = std::tr1::get<0>(tups[x]); >> if (i>=end) break; >Are your tuples sorted to begin with? Are they specified redundantly or >distributed? Are any of the tuples nonlocal? If it is distributed >without matching the row distribution, then you need to communicate >d_nnz and o_nnz. >> if (i>=begin) { >> j = std::tr1::get<1>(tups[x]); >> if (j>=begin && j> else o_nnz[i-begin]++; >> } >> } >> MatMPIAIJSetPreallocation(*X, 0, d_nnz, 0, o_nnz); >> >> for (int x=0; x> i = std::tr1::get<0>(tups[x]); >> if (i>=end) break; >> if (i>=begin) { >> j = std::tr1::get<1>(tups[x]); >> val = std::tr1::get<2>(tups[x]); >> MatSetValues(*X, 1, &i, 1, &j, &val, INSERT_VALUES); >> } >> } >> MatAssemblyBegin(*X, MAT_FINAL_ASSEMBLY); >> MatAssemblyEnd(*X, MAT_FINAL_ASSEMBLY); >> } From ztdepyahoo at 163.com Thu Sep 12 18:44:31 2013 From: ztdepyahoo at 163.com (=?GBK?B?tqHAz8qm?=) Date: Fri, 13 Sep 2013 07:44:31 +0800 (CST) Subject: [petsc-users] how to simultaneously perform the partition of the unstructured mesh and reduction bandwidth of the resulting sparsematrix In-Reply-To: <87six9udi8.fsf@mcs.anl.gov> References: <7d9de2eb.26557.14112a0334f.Coremail.ztdepyahoo@163.com> <87six9udi8.fsf@mcs.anl.gov> Message-ID: <7a99d0af.5d6.141148fd0ec.Coremail.ztdepyahoo@163.com> i think bandwidth minimization will improve the convergence of the kspsolver, am i right? At 2013-09-13 01:59:59,"Jed Brown" wrote: >Matrix bandwidth minimization is an (imprecise) optimization to improve >cache reuse. So just partition globally and then reorder locally. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ztdepyahoo at 163.com Thu Sep 12 18:47:55 2013 From: ztdepyahoo at 163.com (=?GBK?B?tqHAz8qm?=) Date: Fri, 13 Sep 2013 07:47:55 +0800 (CST) Subject: [petsc-users] how to simultaneously perform the partition of the unstructured mesh and reduction bandwidth of the resulting sparsematrix In-Reply-To: <6F855A74-EED0-48F7-831A-1AD9C716BDB9@mcs.anl.gov> References: <7d9de2eb.26557.14112a0334f.Coremail.ztdepyahoo@163.com> <87six9udi8.fsf@mcs.anl.gov> <6F855A74-EED0-48F7-831A-1AD9C716BDB9@mcs.anl.gov> Message-ID: <7b0436a9.610.1411492ef46.Coremail.ztdepyahoo@163.com> which petsc function perform the reordering of the matrix At 2013-09-13 02:54:38,"Barry Smith" wrote: > >On Sep 12, 2013, at 12:59 PM, Jed Brown wrote: > >> Matrix bandwidth minimization is an (imprecise) optimization to improve >> cache reuse. So just partition globally and then reorder locally. > > The mesh, before you start the linear algebra. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Sep 12 18:48:12 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 12 Sep 2013 16:48:12 -0700 Subject: [petsc-users] how to simultaneously perform the partition of the unstructured mesh and reduction bandwidth of the resulting sparsematrix In-Reply-To: <7a99d0af.5d6.141148fd0ec.Coremail.ztdepyahoo@163.com> References: <7d9de2eb.26557.14112a0334f.Coremail.ztdepyahoo@163.com> <87six9udi8.fsf@mcs.anl.gov> <7a99d0af.5d6.141148fd0ec.Coremail.ztdepyahoo@163.com> Message-ID: <87mwnhsitf.fsf@mcs.anl.gov> ??? writes: > i think bandwidth minimization will improve the convergence of the kspsolver, am i right? Bandwidth is *irrelevant* for Krylov methods. It can affect preconditioning, but it's really not the right metric. Choose a partitioning that respects the coupling strength in your problem, then choose a local ordering that has the properties you like. (Low bandwidth is just one possibility.) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From jedbrown at mcs.anl.gov Thu Sep 12 18:51:21 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 12 Sep 2013 16:51:21 -0700 Subject: [petsc-users] About MatGetOwnershipRange() In-Reply-To: <402438665.102280.1379027033472.JavaMail.root@mailhub028.itcs.purdue.edu> References: <402438665.102280.1379027033472.JavaMail.root@mailhub028.itcs.purdue.edu> Message-ID: <87k3ilsio6.fsf@mcs.anl.gov> Joon Hee Choi writes: > Hi Jed, > > Thank you for your reply. I tried to follow your method, but I think > my code still has something wrong because "localsize" is the same as > global row size("I"). Did you run in parallel? > Could you let me know what I am missing? My new code is as follows: > > PetscErrorCode SetUpMatrix(Mat *X, vector< std::tr1::tuple< PetscInt, PetscInt, PetscInt > >, PetscInt I, PetscInt J) > { > PetscInt i, j; > PetscScalar val; > > MatCreate(PETSC_COMM_WORLD, X); > MatSetType(*X, MATMPIAIJ); > MatSetSizes(*X, PETSC_DECIDE, PETSC_DECIDE, I, J); > > PetscInt begin, end, localsize=PETSC_DECIDE; > PetscSplitOwnership(PETSC_COMM_WORLD, &localsize, &I); > MPI_Scan(&localsize, &end, 1, MPI_INT, MPI_SUM, PETSC_COMM_WORLD); > begin = end - localsize; > PetscPrintf(PETSC_COMM_WORLD, "Local Size: %d, begin: %d, end: %d \n", localsize, begin, end); > > ... > } > > Also, tuples are the total pairs of (row dimension, column dimension, > element), values of sparse matrix, which are read from a file. Using files this way is a massive bottleneck that you'll have to eliminate if you want your code to be scalable. > The tuples are sorted and distributed. When you distribute, are you sure that each process really gets the entire row, or would it be possible to cut in the middle of a row? -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From jedbrown at mcs.anl.gov Thu Sep 12 18:53:00 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 12 Sep 2013 16:53:00 -0700 Subject: [petsc-users] how to simultaneously perform the partition of the unstructured mesh and reduction bandwidth of the resulting sparsematrix In-Reply-To: <7b0436a9.610.1411492ef46.Coremail.ztdepyahoo@163.com> References: <7d9de2eb.26557.14112a0334f.Coremail.ztdepyahoo@163.com> <87six9udi8.fsf@mcs.anl.gov> <6F855A74-EED0-48F7-831A-1AD9C716BDB9@mcs.anl.gov> <7b0436a9.610.1411492ef46.Coremail.ztdepyahoo@163.com> Message-ID: <87hadpsilf.fsf@mcs.anl.gov> ??? writes: > which petsc function perform the reordering of the matrix MatGetOrdering() gives you the permutation vectors. You should be doing this reordering with the *mesh*. You can represent mesh connectivity in a matrix and use MatPartitioning and MatGetOrdering, but you should use the result to redistribute the mesh before assembling the matrices containing your physics. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From choi240 at purdue.edu Thu Sep 12 21:20:12 2013 From: choi240 at purdue.edu (Joon Hee Choi) Date: Thu, 12 Sep 2013 22:20:12 -0400 (EDT) Subject: [petsc-users] About MatGetOwnershipRange() In-Reply-To: <87k3ilsio6.fsf@mcs.anl.gov> Message-ID: <313513452.102807.1379038807888.JavaMail.root@mailhub028.itcs.purdue.edu> ----- Original Message ----- From: "Jed Brown" To: "Joon Hee Choi" Cc: petsc-users at mcs.anl.gov Sent: Thursday, September 12, 2013 7:51:21 PM Subject: Re: [petsc-users] About MatGetOwnershipRange() Joon Hee Choi writes: >> Hi Jed, >> >> Thank you for your reply. I tried to follow your method, but I think >> my code still has something wrong because "localsize" is the same as >> global row size("I"). >Did you run in parallel? Yes. I tested my code with 2 nodes and 4 ppn(processors per node). >> Could you let me know what I am missing? My new code is as follows: >> >> PetscErrorCode SetUpMatrix(Mat *X, vector< std::tr1::tuple< PetscInt, PetscInt, PetscInt > >, PetscInt I, PetscInt J) >> { >> PetscInt i, j; >> PetscScalar val; >> >> MatCreate(PETSC_COMM_WORLD, X); >> MatSetType(*X, MATMPIAIJ); >> MatSetSizes(*X, PETSC_DECIDE, PETSC_DECIDE, I, J); >> >> PetscInt begin, end, localsize=PETSC_DECIDE; >> PetscSplitOwnership(PETSC_COMM_WORLD, &localsize, &I); >> MPI_Scan(&localsize, &end, 1, MPI_INT, MPI_SUM, PETSC_COMM_WORLD); >> begin = end - localsize; >> PetscPrintf(PETSC_COMM_WORLD, "Local Size: %d, begin: %d, end: %d \n", localsize, begin, end); >> >> ... >> } >> >> Also, tuples are the total pairs of (row dimension, column dimension, >> element), values of sparse matrix, which are read from a file. >Using files this way is a massive bottleneck that you'll have to >eliminate if you want your code to be scalable. What do you mean a massive bottleneck? Actually, I succeeded in setting up the large matrix using seqaij type and calculating products between huge matrices. However, I need to use multiple machines because I have to run loops of the product many times. And I am a novice of mpi. >> The tuples are sorted and distributed. >When you distribute, are you sure that each process really gets the >entire row, or would it be possible to cut in the middle of a row? I think each process gets the entire row. The following is the code for getting tuples from text file: ierr = PetscFOpen(PETSC_COMM_WORLD, inputFile, "r", &file); CHKERRQ(ierr); while (fscanf(file, "%d %d %d", &ii, &jj, &vv) == 3) { tups.push_back(std::tr1::make_tuple (ii-1, jj-1, vv)); if (ii > I) I = ii; if (jj > J) J = jj; } ierr = PetscFClose(PETSC_COMM_WORLD, file); CHKERRQ(ierr); From fangfang at iscas.ac.cn Fri Sep 13 04:32:53 2013 From: fangfang at iscas.ac.cn (fangfang) Date: Fri, 13 Sep 2013 17:32:53 +0800 Subject: [petsc-users] Examples of petsc with superlu Message-ID: <201309131732529821572@iscas.ac.cn> Hi, I am trying to run the example ex52 in src/ksp/ksp/examples/tutorials with following command: make runex52_superlu but this generated the following error messages: 1c1,38 < Norm of error < 1.e-12 iterations 1 --- > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] SuperLU:StatInit line 616 /home/lff/petsc-3.4.2/src/mat/impls/aij/seq/superlu/superlu.c > [0]PETSC ERROR: [0] MatGetFactor_seqaij_superlu line 580 /home/lff/petsc-3.4.2/src/mat/impls/aij/seq/superlu/superlu.c > [0]PETSC ERROR: [0] MatGetFactor line 3949 /home/lff/petsc-3.4.2/src/mat/interface/matrix.c > [0]PETSC ERROR: [0] PCFactorSetUpMatSolverPackage_Factor line 14 /home/lff/petsc-3.4.2/src/ksp/pc/impls/factor/factimpl.c > [0]PETSC ERROR: [0] PCFactorSetUpMatSolverPackage line 24 /home/lff/petsc-3.4.2/src/ksp/pc/impls/factor/factor.c > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ./ex52 on a arch-linux2-c-debug named RDCPS by lff Fri Sep 13 17:12:46 2013 > [0]PETSC ERROR: Libraries linked from /home/lff/petsc-3.4.2/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Thu Sep 12 11:08:01 2013 > [0]PETSC ERROR: Configure options --with-cc=gcc --CFLAGS=-O3 --download-mpich --with-blas-lapack-dir=/home/lff/OpenBLAS-0.2.7/ --download-superlu=yes > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > [cli_0]: aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = EXIT CODE: 59 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > =================================================================================== /home/lff/petsc-3.4.2/src/ksp/ksp/examples/tutorials Possible problem with with ex52, diffs above I have rebuilt petsc with superlu, does this require changing the code in ex52.c or superlu.c ,or is there something else that I am doing wrong? Thanks a lot. 2013-09-13 fangfang research assistant Institute of Software China Acedemy of Sciences fax:010-61661632 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bisheshkh at gmail.com Fri Sep 13 06:21:28 2013 From: bisheshkh at gmail.com (Bishesh Khanal) Date: Fri, 13 Sep 2013 13:21:28 +0200 Subject: [petsc-users] problem in calling KSPSetComputeRHS and KSPSolve multiple times to solve different RHS. Message-ID: Dear all, I'm trying to reuse an operator for the linear system multiple times with different RHS. But it seems that the RHS is not changed in the KSP object. Here is the code to illustrate what I'm doing: Vec mX[3], mB[3]; //Solving Ax=b for 3 different RHS. ... ierr = KSPSetComputeOperators(mKsp,computeMatrix,this);CHKERRQ(ierr); ierr = KSPSetDM(mKsp,mDa);CHKERRQ(ierr); ierr = KSPSetFromOptions(mKsp);CHKERRQ(ierr); for(int i = 0; i<3; ++i) { mBComponent = i; //sets different component for RHS. ierr = KSPSetComputeRHS(mKsp,computeRhs,this);CHKERRQ(ierr); ierr = KSPSetDM(mKsp,mDa);CHKERRQ(ierr); ierr = KSPSolve(mKsp,NULL,NULL);CHKERRQ(ierr); ierr = KSPGetSolution(mKsp,&mX[i]);CHKERRQ(ierr); ierr = KSPGetRhs(mKsp,&mB[i]);CHKERRQ(ierr); } What I expect is to have three different solutions in mX array for the different Rhs computed by computeRhs function. However, the result when I see using VecView is that the Rhs mB[0], mB[1] and mB[2] are all identical (which should not have been the case). And mX also has three identical vectors as solution. What am I doing wrong here ? Thanks, Bishesh -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Sep 13 07:47:15 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 13 Sep 2013 07:47:15 -0500 Subject: [petsc-users] problem in calling KSPSetComputeRHS and KSPSolve multiple times to solve different RHS. In-Reply-To: References: Message-ID: <3746EAC3-3049-4873-BA53-E7CE2697FD43@mcs.anl.gov> On Sep 13, 2013, at 6:21 AM, Bishesh Khanal wrote: > Dear all, > I'm trying to reuse an operator for the linear system multiple times with different RHS. > But it seems that the RHS is not changed in the KSP object. Here is the code to illustrate what I'm doing: > > Vec mX[3], mB[3]; //Solving Ax=b for 3 different RHS. > ... > ierr = KSPSetComputeOperators(mKsp,computeMatrix,this);CHKERRQ(ierr); > ierr = KSPSetDM(mKsp,mDa);CHKERRQ(ierr); Move the line up above the previous line > ierr = KSPSetFromOptions(mKsp);CHKERRQ(ierr); > > for(int i = 0; i<3; ++i) { > mBComponent = i; //sets different component for RHS. > ierr = KSPSetComputeRHS(mKsp,computeRhs,this);CHKERRQ(ierr); Put the above line in the setup code above, not in the loop > ierr = KSPSetDM(mKsp,mDa);CHKERRQ(ierr); Remove the line above > ierr = KSPSolve(mKsp,NULL,NULL);CHKERRQ(ierr); > ierr = KSPGetSolution(mKsp,&mX[i]);CHKERRQ(ierr); > ierr = KSPGetRhs(mKsp,&mB[i]);CHKERRQ(ierr); > } After you make this changes it should work. If it does not then run with -start_in_debugger noxterm and put a break point in computeRHS, make sure it is being called. > > What I expect is to have three different solutions in mX array for the different Rhs computed by computeRhs function. However, the result when I see using VecView is that the Rhs mB[0], mB[1] and mB[2] are all identical (which should not have been the case). And mX also has three identical vectors as solution. > What am I doing wrong here ? > > > Thanks, > Bishesh > From bisheshkh at gmail.com Fri Sep 13 08:35:34 2013 From: bisheshkh at gmail.com (Bishesh Khanal) Date: Fri, 13 Sep 2013 15:35:34 +0200 Subject: [petsc-users] problem in calling KSPSetComputeRHS and KSPSolve multiple times to solve different RHS. In-Reply-To: <3746EAC3-3049-4873-BA53-E7CE2697FD43@mcs.anl.gov> References: <3746EAC3-3049-4873-BA53-E7CE2697FD43@mcs.anl.gov> Message-ID: On Fri, Sep 13, 2013 at 2:47 PM, Barry Smith wrote: > > On Sep 13, 2013, at 6:21 AM, Bishesh Khanal wrote: > > > Dear all, > > I'm trying to reuse an operator for the linear system multiple times > with different RHS. > > But it seems that the RHS is not changed in the KSP object. Here is the > code to illustrate what I'm doing: > > > > Vec mX[3], mB[3]; //Solving Ax=b for 3 different RHS. > > ... > > ierr = KSPSetComputeOperators(mKsp,computeMatrix,this);CHKERRQ(ierr); > > ierr = KSPSetDM(mKsp,mDa);CHKERRQ(ierr); > > Move the line up above the previous line > > > ierr = KSPSetFromOptions(mKsp);CHKERRQ(ierr); > > > > for(int i = 0; i<3; ++i) { > > mBComponent = i; //sets different component for RHS. > > ierr = KSPSetComputeRHS(mKsp,computeRhs,this);CHKERRQ(ierr); > > Put the above line in the setup code above, not in the loop > > > ierr = KSPSetDM(mKsp,mDa);CHKERRQ(ierr); > > Remove the line above > > > ierr = KSPSolve(mKsp,NULL,NULL);CHKERRQ(ierr); > > ierr = KSPGetSolution(mKsp,&mX[i]);CHKERRQ(ierr); > > ierr = KSPGetRhs(mKsp,&mB[i]);CHKERRQ(ierr); > > } > > After you make this changes it should work. If it does not then run > with -start_in_debugger noxterm and put a break point in computeRHS, make > sure it is being called. > Thanks, I did as you suggested but there is still a problem. I had kept KSPComputeRHS within the loop so that it gets called to update the RHS based on the value of mBComponent, using computeRhs function. Now I kept it outside the loop in the setup code as you suggested and while doing debugging, I see that the program does go inside the computeRhs function three times even in this case. However when I see the results with the following code, it still has the same rhs values for all three cases. ierr = PetscObjectSetName((PetscObject)mB[0],"rhsX");CHKERRQ(ierr); ierr = VecView(mB[0],viewer);CHKERRQ(ierr); ierr = PetscObjectSetName((PetscObject)mB[1],"rhsY");CHKERRQ(ierr); ierr = VecView(mB[1],viewer);CHKERRQ(ierr); ierr = PetscObjectSetName((PetscObject)mB[2],"rhsZ");CHKERRQ(ierr); ierr = VecView(mB[2],viewer);CHKERRQ(ierr); Now I am getting confused with two things: 1) It seems that computeRhs function will be called every time we call KspSolve. If this is the case, is it the same with KSPSetComputeOperators too ? Because I do not want the computeMatrix function to be called repeatedly to assemble the same matrix operator three times. Or there must be sth I'm missing here. 2) The computeRhs is called three times with different values of mBComponent, but I can't figure out why the rhs vectors in mB[] array are same. Here is my computeRhs function: PetscErrorCode HarmCurvRegSolver::computeRhs(KSP ksp, Vec b, void *ctx) { PetscFunctionBeginUser; HarmCurvRegSolver *context = (HarmCurvRegSolver*)ctx; PetscErrorCode ierr; DM da; ierr = KSPGetDM(ksp,&da);CHKERRQ(ierr); DMDALocalInfo info; ierr = DMDAGetLocalInfo(da,&info);CHKERRQ(ierr); PetscScalar ***rhs; ierr = DMDAVecGetArray(da,b,&rhs);CHKERRQ(ierr); //The RHS should be provided by the model! for (PetscInt k = info.zs; kgetProblem()->getRhsAtPosition(i,j,k,context->getCurrentRhsComponent()); //I've checked well the getRhsAtPosition(...) function! It does give different values of the 4th argument. //and getCurrentRhsComponent() just returns the value of mBComponent. } } } ierr = DMDAVecRestoreArray(da,b,&rhs);CHKERRQ(ierr); ierr = VecAssemblyBegin(b);CHKERRQ(ierr); ierr = VecAssemblyEnd(b);CHKERRQ(ierr); PetscFunctionReturn(0); } > > > > > What I expect is to have three different solutions in mX array for the > different Rhs computed by computeRhs function. However, the result when I > see using VecView is that the Rhs mB[0], mB[1] and mB[2] are all identical > (which should not have been the case). And mX also has three identical > vectors as solution. > > What am I doing wrong here ? > > > > > > Thanks, > > Bishesh > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Fri Sep 13 09:45:47 2013 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Fri, 13 Sep 2013 09:45:47 -0500 Subject: [petsc-users] Examples of petsc with superlu In-Reply-To: <201309131732529821572@iscas.ac.cn> References: <201309131732529821572@iscas.ac.cn> Message-ID: Fangfang, I tested 'runex52_superlu' on petsc-dev master branch which is virtually same as petsc-3.4 for superlu wrapper, and got correct result petsc/src/ksp/ksp/examples/tutorials (master) $ ./ex52 -use_superlu_lu Norm of error < 1.e-12 iterations 1 Your report puzzles me, because make runex52_superlu > but this generated the following error messages: > > > > 1c1,38 > > < Norm of error < 1.e-12 iterations 1 > this indicates that ex52 works fine to the line 265, i.e. superlu solve is done and gives correct answer. Then the error message below shows crash on > > [0]PETSC ERROR: [0] SuperLU:StatInit line 616 > /home/lff/petsc-3.4.2/src/mat/impls/aij/seq/superlu/superlu.c > > > [0]PETSC ERROR: [0] MatGetFactor_seqaij_superlu line 580 > /home/lff/petsc-3.4.2/src/mat/impls/aij/seq/superlu/superlu.c > > > [0]PETSC ERROR: [0] MatGetFactor line 3949 > /home/lff/petsc-3.4.2/src/mat/interface/matrix.c > > [0]PETSC ERROR: [0] PCFactorSetUpMatSolverPackage_Factor line 14 > which occurs way ahead of line 265. > /home/lff/petsc-3.4.2/src/ksp/pc/impls/factor/factimpl.c > > > [0]PETSC ERROR: [0] PCFactorSetUpMatSolverPackage line 24 > /home/lff/petsc-3.4.2/src/ksp/pc/impls/factor/factor.c > > > [0]PETSC ERROR: --------------------- Error Message > ----------------------------------- > > > /home/lff/petsc-3.4.2/src/ksp/ksp/examples/tutorials > > Possible problem with with ex52, diffs above > > > > I have rebuilt petsc with superlu, does this require changing the code in > ex52.c or superlu.c ,or is there something else that I am doing wrong? > No, there is no need to change ex52.c. Hong -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Sep 13 10:32:09 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 13 Sep 2013 10:32:09 -0500 Subject: [petsc-users] problem in calling KSPSetComputeRHS and KSPSolve multiple times to solve different RHS. In-Reply-To: References: <3746EAC3-3049-4873-BA53-E7CE2697FD43@mcs.anl.gov> Message-ID: On Sep 13, 2013, at 8:35 AM, Bishesh Khanal wrote: > > > > On Fri, Sep 13, 2013 at 2:47 PM, Barry Smith wrote: > > On Sep 13, 2013, at 6:21 AM, Bishesh Khanal wrote: > > > Dear all, > > I'm trying to reuse an operator for the linear system multiple times with different RHS. > > But it seems that the RHS is not changed in the KSP object. Here is the code to illustrate what I'm doing: > > > > Vec mX[3], mB[3]; //Solving Ax=b for 3 different RHS. > > ... > > ierr = KSPSetComputeOperators(mKsp,computeMatrix,this);CHKERRQ(ierr); > > ierr = KSPSetDM(mKsp,mDa);CHKERRQ(ierr); > > Move the line up above the previous line > > > ierr = KSPSetFromOptions(mKsp);CHKERRQ(ierr); > > > > for(int i = 0; i<3; ++i) { > > mBComponent = i; //sets different component for RHS. > > ierr = KSPSetComputeRHS(mKsp,computeRhs,this);CHKERRQ(ierr); > > Put the above line in the setup code above, not in the loop > > > ierr = KSPSetDM(mKsp,mDa);CHKERRQ(ierr); > > Remove the line above > > > ierr = KSPSolve(mKsp,NULL,NULL);CHKERRQ(ierr); > > ierr = KSPGetSolution(mKsp,&mX[i]);CHKERRQ(ierr); > > ierr = KSPGetRhs(mKsp,&mB[i]);CHKERRQ(ierr); > > } > > After you make this changes it should work. If it does not then run with -start_in_debugger noxterm and put a break point in computeRHS, make sure it is being called. > Thanks, I did as you suggested but there is still a problem. > I had kept KSPComputeRHS within the loop so that it gets called to update the RHS based on the value of mBComponent, using computeRhs function. > Now I kept it outside the loop in the setup code as you suggested and while doing debugging, I see that the program does go inside the computeRhs function three times even in this case. > However when I see the results with the following code, it still has the same rhs values for all three cases. > ierr = PetscObjectSetName((PetscObject)mB[0],"rhsX");CHKERRQ(ierr); > ierr = VecView(mB[0],viewer);CHKERRQ(ierr); > ierr = PetscObjectSetName((PetscObject)mB[1],"rhsY");CHKERRQ(ierr); > ierr = VecView(mB[1],viewer);CHKERRQ(ierr); > ierr = PetscObjectSetName((PetscObject)mB[2],"rhsZ");CHKERRQ(ierr); > ierr = VecView(mB[2],viewer);CHKERRQ(ierr); > > Now I am getting confused with two things: > 1) It seems that computeRhs function will be called every time we call KspSolve. Yes > If this is the case, is it the same with KSPSetComputeOperators too ? No, you need to call KSPSetComputeOperators() to trigger a new matrix generation on the next solve. > Because I do not want the computeMatrix function to be called repeatedly to assemble the same matrix operator three times. Or there must be sth I'm missing here. > > 2) The computeRhs is called three times with different values of mBComponent, but I can't figure out why the rhs vectors in mB[] array are same. Here is my computeRhs function: > > PetscErrorCode HarmCurvRegSolver::computeRhs(KSP ksp, Vec b, void *ctx) > { > PetscFunctionBeginUser; > HarmCurvRegSolver *context = (HarmCurvRegSolver*)ctx; > > PetscErrorCode ierr; > DM da; > ierr = KSPGetDM(ksp,&da);CHKERRQ(ierr); > DMDALocalInfo info; > ierr = DMDAGetLocalInfo(da,&info);CHKERRQ(ierr); > > PetscScalar ***rhs; > ierr = DMDAVecGetArray(da,b,&rhs);CHKERRQ(ierr); > > //The RHS should be provided by the model! > for (PetscInt k = info.zs; k for (PetscInt j = info.ys; j < info.ys+info.ym; ++j) { > for (PetscInt i = info.xs; i < info.xs+info.xm; ++i) { > rhs[k][j][i] = context->getProblem()->getRhsAtPosition(i,j,k,context->getCurrentRhsComponent()); put break point here if needed to check that values being set are different each call. Use a tiny da to help debugging. > //I've checked well the getRhsAtPosition(...) function! It does give different values of the 4th argument. > //and getCurrentRhsComponent() just returns the value of mBComponent. > } > } > } > > ierr = DMDAVecRestoreArray(da,b,&rhs);CHKERRQ(ierr); Call VecView() here on b; does it have different values each time or the same? If it does then you need to run in the debugger and break at the point indicated above and look at the values as they are copied over to make sure they are different each call. > ierr = VecAssemblyBegin(b);CHKERRQ(ierr); > ierr = VecAssemblyEnd(b);CHKERRQ(ierr); You don't need the VecAssemblyBegin/End here. > > PetscFunctionReturn(0); > > } > > > > > > > > What I expect is to have three different solutions in mX array for the different Rhs computed by computeRhs function. However, the result when I see using VecView is that the Rhs mB[0], mB[1] and mB[2] are all identical (which should not have been the case). And mX also has three identical vectors as solution. > > What am I doing wrong here ? > > > > > > Thanks, > > Bishesh > > > > From bsmith at mcs.anl.gov Fri Sep 13 10:46:19 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 13 Sep 2013 10:46:19 -0500 Subject: [petsc-users] Examples of petsc with superlu In-Reply-To: <201309131732529821572@iscas.ac.cn> References: <201309131732529821572@iscas.ac.cn> Message-ID: <030A91FE-D2A3-498C-83AE-F64070229D2A@mcs.anl.gov> You need to manually run the example with -start_in_debugger noxterm and see why it crashes It could be if you compile another PETSC_ARCH without those optimization flags it may not crash. You can also run with valgrind http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind Barry On Sep 13, 2013, at 4:32 AM, fangfang wrote: > Hi, I am trying to run the example ex52 in src/ksp/ksp/examples/tutorials with following command: > make runex52_superlu > but this generated the following error messages: > > 1c1,38 > < Norm of error < 1.e-12 iterations 1 > --- > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > > [0]PETSC ERROR: likely location of problem given in stack below > > [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > > [0]PETSC ERROR: INSTEAD the line number of the start of the function > > [0]PETSC ERROR: is given. > > [0]PETSC ERROR: [0] SuperLU:StatInit line 616 /home/lff/petsc-3.4.2/src/mat/impls/aij/seq/superlu/superlu.c > > [0]PETSC ERROR: [0] MatGetFactor_seqaij_superlu line 580 /home/lff/petsc-3.4.2/src/mat/impls/aij/seq/superlu/superlu.c > > [0]PETSC ERROR: [0] MatGetFactor line 3949 /home/lff/petsc-3.4.2/src/mat/interface/matrix.c > > [0]PETSC ERROR: [0] PCFactorSetUpMatSolverPackage_Factor line 14 /home/lff/petsc-3.4.2/src/ksp/pc/impls/factor/factimpl.c > > [0]PETSC ERROR: [0] PCFactorSetUpMatSolverPackage line 24 /home/lff/petsc-3.4.2/src/ksp/pc/impls/factor/factor.c > > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > > [0]PETSC ERROR: Signal received! > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > [0]PETSC ERROR: See docs/index.html for manual pages. > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > [0]PETSC ERROR: ./ex52 on a arch-linux2-c-debug named RDCPS by lff Fri Sep 13 17:12:46 2013 > > [0]PETSC ERROR: Libraries linked from /home/lff/petsc-3.4.2/arch-linux2-c-debug/lib > > [0]PETSC ERROR: Configure run at Thu Sep 12 11:08:01 2013 > > [0]PETSC ERROR: Configure options --with-cc=gcc --CFLAGS=-O3 --download-mpich --with-blas-lapack-dir=/home/lff/OpenBLAS-0.2.7/ --download-superlu=yes > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > > [cli_0]: aborting job: > > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > > > > =================================================================================== > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > = EXIT CODE: 59 > > = CLEANING UP REMAINING PROCESSES > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > =================================================================================== > /home/lff/petsc-3.4.2/src/ksp/ksp/examples/tutorials > Possible problem with with ex52, diffs above > > I have rebuilt petsc with superlu, does this require changing the code in ex52.c or superlu.c ,or is there something else that I am doing wrong? > Thanks a lot. > > > 2013-09-13 > fangfang > research assistant > Institute of Software China Acedemy of Sciences > fax:010-61661632 From parsani.matteo at gmail.com Fri Sep 13 15:41:30 2013 From: parsani.matteo at gmail.com (Matteo Parsani) Date: Fri, 13 Sep 2013 16:41:30 -0400 Subject: [petsc-users] Problem writing headers to log: 'Framework' object has no attribute 'compilerDefines Message-ID: Dear PETSc developers, I have recently updated PETSc from its git repository. I am trying to compile it as I have done 3 weeks ago with the following options: ./configure --prefix=/scratch/home0/pmatteo/research/workspace/codes/ssdc/deps/petsc --with-mpi-dir=/scratch/home0/pmatteo/research/lib_install --with-hdf5-dir=/scratch/home0/pmatteo/research/workspace/codes/ssdc/deps --with-metis-dir=/scratch/home0/pmatteo/research/workspace/codes/ssdc/deps --with-parmetis-dir=/scratch/home0/pmatteo/research/workspace/codes/ssdc/deps --with-valgrind-dir=/scratch/home0/pmatteo/research/lib_install --download-f-blas-lapack --download-hypre=yes --download-scalapack=yes --download-blacs=yes --download-superlu_dist=yes --download-umfpack=yes But I get an error which I cannot explain: =============================================================================== Configuring PETSc to compile on your system =============================================================================== TESTING: checkCCompiler from config.setCompilers(config/BuildSystem/config/setCompilers.py:525) ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- MPI compiler wrappers in /scratch/home0/pmatteo/research/lib_install/bin do not work. See http://www.mcs.anl.gov/petsc/documentation/faq.html#mpi-compilers Please, find in attachment the configure.log file. I have gcc 4.7.3 and openmpi 1.6.4 installed and they are in my path. Thank you. -- Matteo -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 71044 bytes Desc: not available URL: From bsmith at mcs.anl.gov Fri Sep 13 15:44:32 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 13 Sep 2013 15:44:32 -0500 Subject: [petsc-users] Problem writing headers to log: 'Framework' object has no attribute 'compilerDefines In-Reply-To: References: Message-ID: <21FCF58A-E97C-4C2A-83B4-6CBEEE3F0F5F@mcs.anl.gov> This directory does not seem to have any MPI compilers in that directory? --with-mpi-dir=/scratch/home0/pmatteo/research/lib_install TESTING: checkCCompiler from config.setCompilers(config/BuildSystem/config/setCompilers.py:525) Locate a functional C compiler Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpicc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpicc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpicc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpicc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpicc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpicc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpicc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpicc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpicc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpicc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpicc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpicc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpicc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpicc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpicc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpicc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpicc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpicc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/hcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/hcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/hcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/hcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/hcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/hcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/hcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/hcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/hcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/hcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/hcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/hcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/hcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/hcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/hcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/hcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/hcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/hcc...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc_r...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc_r...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc_r...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc_r...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc_r...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc_r...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc_r...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc_r...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc_r...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc_r...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc_r...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc_r...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc_r...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc_r...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc_r...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc_r...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc_r...not found Checking for program /scratch/home0/pmatteo/research/lib_install/bin/mpcc_r...not found On Sep 13, 2013, at 3:41 PM, Matteo Parsani wrote: > Dear PETSc developers, > I have recently updated PETSc from its git repository. > I am trying to compile it as I have done 3 weeks ago with the following options: > > ./configure --prefix=/scratch/home0/pmatteo/research/workspace/codes/ssdc/deps/petsc --with-mpi-dir=/scratch/home0/pmatteo/research/lib_install --with-hdf5-dir=/scratch/home0/pmatteo/research/workspace/codes/ssdc/deps --with-metis-dir=/scratch/home0/pmatteo/research/workspace/codes/ssdc/deps --with-parmetis-dir=/scratch/home0/pmatteo/research/workspace/codes/ssdc/deps --with-valgrind-dir=/scratch/home0/pmatteo/research/lib_install --download-f-blas-lapack --download-hypre=yes --download-scalapack=yes --download-blacs=yes --download-superlu_dist=yes --download-umfpack=yes > > But I get an error which I cannot explain: > > =============================================================================== > Configuring PETSc to compile on your system > =============================================================================== > TESTING: checkCCompiler from config.setCompilers(config/BuildSystem/config/setCompilers.py:525) ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > ------------------------------------------------------------------------------- > MPI compiler wrappers in /scratch/home0/pmatteo/research/lib_install/bin do not work. See http://www.mcs.anl.gov/petsc/documentation/faq.html#mpi-compilers > > > Please, find in attachment the configure.log file. > > I have gcc 4.7.3 and openmpi 1.6.4 installed and they are in my path. > > Thank you. > -- > Matteo > From jedbrown at mcs.anl.gov Fri Sep 13 16:34:33 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 13 Sep 2013 14:34:33 -0700 Subject: [petsc-users] About MatGetOwnershipRange() In-Reply-To: <313513452.102807.1379038807888.JavaMail.root@mailhub028.itcs.purdue.edu> References: <313513452.102807.1379038807888.JavaMail.root@mailhub028.itcs.purdue.edu> Message-ID: <87ppscquc6.fsf@mcs.anl.gov> Joon Hee Choi writes: >>Did you run in parallel? > > Yes. I tested my code with 2 nodes and 4 ppn(processors per node). Then print what you are passing in and what you get back on each process. The PetscSplitOwnership code is short. >>> Also, tuples are the total pairs of (row dimension, column dimension, >>> element), values of sparse matrix, which are read from a file. > >>Using files this way is a massive bottleneck that you'll have to >>eliminate if you want your code to be scalable. > > What do you mean a massive bottleneck? Writing to a file is at least 1000 times slower than accessing memory and it does not parallelize as well. > Actually, I succeeded in setting up the large matrix using seqaij type > and calculating products between huge matrices. However, I need to use > multiple machines because I have to run loops of the product many > times. And I am a novice of mpi. I'm not sure what you're asking, but you should create the matrices in memory, preferably in parallel and mostly where the entries will need to be stored. >>> The tuples are sorted and distributed. > >>When you distribute, are you sure that each process really gets the >>entire row, or would it be possible to cut in the middle of a row? > > I think each process gets the entire row. The following is the code for getting tuples from text file: > > ierr = PetscFOpen(PETSC_COMM_WORLD, inputFile, "r", &file); CHKERRQ(ierr); > while (fscanf(file, "%d %d %d", &ii, &jj, &vv) == 3) > { > tups.push_back(std::tr1::make_tuple (ii-1, jj-1, vv)); > if (ii > I) I = ii; > if (jj > J) J = jj; > } > ierr = PetscFClose(PETSC_COMM_WORLD, file); CHKERRQ(ierr); Nothing in this code snippet indicates that the row partition is non-overlapping. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From fangxingjun0319 at 163.com Fri Sep 13 17:07:56 2013 From: fangxingjun0319 at 163.com (Frank) Date: Fri, 13 Sep 2013 17:07:56 -0500 Subject: [petsc-users] Will matrix-free faster than solving linearized equation? Message-ID: <52338CBC.9070808@163.com> Hi, Currently, I am solving a nonlinear equation with some linearization method. I am thinking to modify it with a nonlinear solver. With PETSc library, I am confident to do it. I just want to ask those who have experiences with nonlinear solver if matrix-free method will be faster. Thank you very much. From jedbrown at mcs.anl.gov Fri Sep 13 17:37:07 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 13 Sep 2013 15:37:07 -0700 Subject: [petsc-users] Will matrix-free faster than solving linearized equation? In-Reply-To: <52338CBC.9070808@163.com> References: <52338CBC.9070808@163.com> Message-ID: <87bo3wqrfw.fsf@mcs.anl.gov> Frank writes: > Hi, > > Currently, I am solving a nonlinear equation with some linearization > method. I am thinking to modify it with a nonlinear solver. With PETSc > library, I am confident to do it. I just want to ask those who have > experiences with nonlinear solver if matrix-free method will be faster. Your question isn't very clear, but it depends on many things. Matrices are good for testing algorithms. Send -log_summary and convergence monitors for a solver configuration that works well for your problem. From that information and a description of your discretization, we can estimate the potential effect of switching to a matrix-free implementation. Don't start out with matrix-free unless you have a lot of experience with the problem class and relevant methods. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From choi240 at purdue.edu Fri Sep 13 21:59:57 2013 From: choi240 at purdue.edu (Joon Hee Choi) Date: Fri, 13 Sep 2013 22:59:57 -0400 (EDT) Subject: [petsc-users] About MatGetOwnershipRange() In-Reply-To: <87ppscquc6.fsf@mcs.anl.gov> Message-ID: <1576051771.106228.1379127597483.JavaMail.root@mailhub028.itcs.purdue.edu> Thank you for your reply. Actually, I succeeded in running with the PetscSpliOwnership() code that you gave to me. I think I didn't understand how to write a code in parallel. Thank you again. ----- Original Message ----- From: "Jed Brown" To: "Joon Hee Choi" Cc: petsc-users at mcs.anl.gov Sent: Friday, September 13, 2013 5:34:33 PM Subject: Re: [petsc-users] About MatGetOwnershipRange() Joon Hee Choi writes: >>>Did you run in parallel? >> >> Yes. I tested my code with 2 nodes and 4 ppn(processors per node). >Then print what you are passing in and what you get back on each >process. The PetscSplitOwnership code is short. >>>> Also, tuples are the total pairs of (row dimension, column dimension, >>>> element), values of sparse matrix, which are read from a file. >> >>>Using files this way is a massive bottleneck that you'll have to >>>eliminate if you want your code to be scalable. >> >> What do you mean a massive bottleneck? >Writing to a file is at least 1000 times slower than accessing memory >and it does not parallelize as well. >> Actually, I succeeded in setting up the large matrix using seqaij type >> and calculating products between huge matrices. However, I need to use >> multiple machines because I have to run loops of the product many >> times. And I am a novice of mpi. >I'm not sure what you're asking, but you should create the matrices in >memory, preferably in parallel and mostly where the entries will need to >be stored. >>>> The tuples are sorted and distributed. >> >>>When you distribute, are you sure that each process really gets the >>>entire row, or would it be possible to cut in the middle of a row? >> >> I think each process gets the entire row. The following is the code for getting >>tuples from text file: >> >> ierr = PetscFOpen(PETSC_COMM_WORLD, inputFile, "r", &file); CHKERRQ(ierr); >> while (fscanf(file, "%d %d %d", &ii, &jj, &vv) == 3) >> { >> tups.push_back(std::tr1::make_tuple (ii-1, jj-1, vv)); >> if (ii > I) I = ii; >> if (jj > J) J = jj; >> } >> ierr = PetscFClose(PETSC_COMM_WORLD, file); CHKERRQ(ierr); >Nothing in this code snippet indicates that the row partition is non-overlapping. From choi240 at purdue.edu Sat Sep 14 00:47:50 2013 From: choi240 at purdue.edu (Joon Hee Choi) Date: Sat, 14 Sep 2013 01:47:50 -0400 (EDT) Subject: [petsc-users] The product of two MPIDENSE matrices Message-ID: <1216881876.106328.1379137670026.JavaMail.root@mailhub028.itcs.purdue.edu> Hello, I want to calculate M = C x C^T. C is a MPIDENSE matrix. I tried the calculation with MatTransposeMatMult (C, C, MAT_INITIAL_MATRIX, PETSC_DEFAULT, &M), but it did not support MPIDENSE. Is there any method to calculate the product of two MPIDENSE matrices? Thanks, Joon From choi240 at purdue.edu Sat Sep 14 01:38:08 2013 From: choi240 at purdue.edu (Joon Hee Choi) Date: Sat, 14 Sep 2013 02:38:08 -0400 (EDT) Subject: [petsc-users] The product of two MPIDENSE matrices In-Reply-To: <1216881876.106328.1379137670026.JavaMail.root@mailhub028.itcs.purdue.edu> Message-ID: <91379376.106372.1379140688580.JavaMail.root@mailhub028.itcs.purdue.edu> Also, Is there any way to calculate C = A^T x B (A: MPIAIJ, B: MPIDENSE, c:MPIAIJ)? I just use the official version of PETSC. Thanks, Joon ----- Original Message ----- From: "Joon Hee Choi" To: petsc-users at mcs.anl.gov Sent: Saturday, September 14, 2013 1:47:50 AM Subject: [petsc-users] The product of two MPIDENSE matrices Hello, I want to calculate M = C x C^T. C is a MPIDENSE matrix. I tried the calculation with MatTransposeMatMult (C, C, MAT_INITIAL_MATRIX, PETSC_DEFAULT, &M), but it did not support MPIDENSE. Is there any method to calculate the product of two MPIDENSE matrices? Thanks, Joon From ztdepyahoo at 163.com Sat Sep 14 07:18:47 2013 From: ztdepyahoo at 163.com (=?GBK?B?tqHAz8qm?=) Date: Sat, 14 Sep 2013 20:18:47 +0800 (CST) Subject: [petsc-users] i use -pc_hypre_type euclid, but it does not work, the default preconditioner is still used. how to call the euclid preconditioner In-Reply-To: <7d9de2eb.26557.14112a0334f.Coremail.ztdepyahoo@163.com> References: <7d9de2eb.26557.14112a0334f.Coremail.ztdepyahoo@163.com> Message-ID: At 2013-09-12 22:43:10,"???" wrote: -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Sep 14 09:56:58 2013 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 14 Sep 2013 09:56:58 -0500 Subject: [petsc-users] i use -pc_hypre_type euclid, but it does not work, the default preconditioner is still used. how to call the euclid preconditioner In-Reply-To: References: <7d9de2eb.26557.14112a0334f.Coremail.ztdepyahoo@163.com> Message-ID: On Sat, Sep 14, 2013 at 7:18 AM, ??? wrote: For any solver question, always send the output from -ksp_view or -snes_view. For options questions, also give -options_left. Thanks, Matt At 2013-09-12 22:43:10,"???" wrote: > > > > > > > > > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Sep 14 09:43:14 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 14 Sep 2013 09:43:14 -0500 Subject: [petsc-users] i use -pc_hypre_type euclid, but it does not work, the default preconditioner is still used. how to call the euclid preconditioner In-Reply-To: References: <7d9de2eb.26557.14112a0334f.Coremail.ztdepyahoo@163.com> Message-ID: <11057074-C716-4A2F-AC05-1C99C08171E0@mcs.anl.gov> Did you also use -pc_type hypre ? That is also needed. On Sep 14, 2013, at 7:18 AM, ??? wrote: > > > > > > > At 2013-09-12 22:43:10,"???" wrote: > > > > > > > > > > > > > > From hzhang at mcs.anl.gov Sat Sep 14 20:18:44 2013 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Sat, 14 Sep 2013 20:18:44 -0500 Subject: [petsc-users] The product of two MPIDENSE matrices In-Reply-To: <1216881876.106328.1379137670026.JavaMail.root@mailhub028.itcs.purdue.edu> References: <1216881876.106328.1379137670026.JavaMail.root@mailhub028.itcs.purdue.edu> Message-ID: Petsc does not support C=A x B^T for MPIDENSE_MPIDENSE matrices, but supports it via elemental interface, see http://code.google.com/p/elemental/ and ~petsc//src/mat/impls/elemental/matelem.cxx. Hong On Sat, Sep 14, 2013 at 12:47 AM, Joon Hee Choi wrote: > Hello, > > I want to calculate M = C x C^T. C is a MPIDENSE matrix. > I tried the calculation with MatTransposeMatMult (C, C, > MAT_INITIAL_MATRIX, PETSC_DEFAULT, &M), but it did not support MPIDENSE. > Is there any method to calculate the product of two MPIDENSE matrices? > > Thanks, > Joon > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Sat Sep 14 20:22:39 2013 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Sat, 14 Sep 2013 20:22:39 -0500 Subject: [petsc-users] The product of two MPIDENSE matrices In-Reply-To: <91379376.106372.1379140688580.JavaMail.root@mailhub028.itcs.purdue.edu> References: <1216881876.106328.1379137670026.JavaMail.root@mailhub028.itcs.purdue.edu> <91379376.106372.1379140688580.JavaMail.root@mailhub028.itcs.purdue.edu> Message-ID: Joon : > Also, Is there any way to calculate C = A^T x B (A: MPIAIJ, B: MPIDENSE, > c:MPIAIJ)? > I just use the official version of PETSC. > Yes, see MatTransposeMatMult_MPIAIJ_MPIDense() in file petsc/src/mat/impls/aij/mpi/mpimattransposematmult.c Hong > > > ----- Original Message ----- > From: "Joon Hee Choi" > To: petsc-users at mcs.anl.gov > Sent: Saturday, September 14, 2013 1:47:50 AM > Subject: [petsc-users] The product of two MPIDENSE matrices > > Hello, > > I want to calculate M = C x C^T. C is a MPIDENSE matrix. > I tried the calculation with MatTransposeMatMult (C, C, > MAT_INITIAL_MATRIX, PETSC_DEFAULT, &M), but it did not support MPIDENSE. > Is there any method to calculate the product of two MPIDENSE matrices? > > Thanks, > Joon > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ztdepyahoo at 163.com Sat Sep 14 23:08:44 2013 From: ztdepyahoo at 163.com (=?GBK?B?tqHAz8qm?=) Date: Sun, 15 Sep 2013 12:08:44 +0800 (CST) Subject: [petsc-users] i use -pc_hypre_type euclid, but it does not work, the default preconditioner is still used. how to call the euclid preconditioner In-Reply-To: <11057074-C716-4A2F-AC05-1C99C08171E0@mcs.anl.gov> References: <7d9de2eb.26557.14112a0334f.Coremail.ztdepyahoo@163.com> <11057074-C716-4A2F-AC05-1C99C08171E0@mcs.anl.gov> Message-ID: <442a79eb.223d.1411fce6d4f.Coremail.ztdepyahoo@163.com> thank you. i forgotten to use it. ? 2013-09-15 06:13:35?"Barry Smith" ??? > > Did you also use -pc_type hypre ? That is also needed. > > >On Sep 14, 2013, at 7:18 AM, ??? wrote: > >> >> >> >> >> >> >> At 2013-09-12 22:43:10,"???" wrote: >> >> >> >> >> >> >> >> >> >> >> >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From choi240 at purdue.edu Sun Sep 15 04:25:10 2013 From: choi240 at purdue.edu (Joon Hee Choi) Date: Sun, 15 Sep 2013 05:25:10 -0400 (EDT) Subject: [petsc-users] The product of two MPIDENSE matrices In-Reply-To: Message-ID: <32352553.107452.1379237110334.JavaMail.root@mailhub028.itcs.purdue.edu> Thanks a lot. Joon ----- Original Message ----- From: "Hong Zhang" To: "Joon Hee Choi" Cc: "PETSc users list" Sent: Saturday, September 14, 2013 9:22:39 PM Subject: Re: [petsc-users] The product of two MPIDENSE matrices Joon : Also, Is there any way to calculate C = A^T x B (A: MPIAIJ, B: MPIDENSE, c:MPIAIJ)? I just use the official version of PETSC. Yes, see MatTransposeMatMult_MPIAIJ_MPIDense() in file petsc/src/mat/impls/aij/mpi/mpimattransposematmult.c Hong ----- Original Message ----- From: "Joon Hee Choi" < choi240 at purdue.edu > To: petsc-users at mcs.anl.gov Sent: Saturday, September 14, 2013 1:47:50 AM Subject: [petsc-users] The product of two MPIDENSE matrices Hello, I want to calculate M = C x C^T. C is a MPIDENSE matrix. I tried the calculation with MatTransposeMatMult (C, C, MAT_INITIAL_MATRIX, PETSC_DEFAULT, &M), but it did not support MPIDENSE. Is there any method to calculate the product of two MPIDENSE matrices? Thanks, Joon From choi240 at purdue.edu Sun Sep 15 04:25:19 2013 From: choi240 at purdue.edu (Joon Hee Choi) Date: Sun, 15 Sep 2013 05:25:19 -0400 (EDT) Subject: [petsc-users] The product of two MPIDENSE matrices In-Reply-To: Message-ID: <427841692.107454.1379237119983.JavaMail.root@mailhub028.itcs.purdue.edu> Thanks a lot. Joon ----- Original Message ----- From: "Hong Zhang" To: "Joon Hee Choi" Cc: "PETSc users list" Sent: Saturday, September 14, 2013 9:18:44 PM Subject: Re: [petsc-users] The product of two MPIDENSE matrices Petsc does not support C=A x B^T for MPIDENSE_ MPIDENSE matrices, but supports it via elemental interface, see http://code.google.com/p/elemental/ and ~petsc//src/mat/impls/elemental/matelem.cxx. Hong On Sat, Sep 14, 2013 at 12:47 AM, Joon Hee Choi < choi240 at purdue.edu > wrote: Hello, I want to calculate M = C x C^T. C is a MPIDENSE matrix. I tried the calculation with MatTransposeMatMult (C, C, MAT_INITIAL_MATRIX, PETSC_DEFAULT, &M), but it did not support MPIDENSE. Is there any method to calculate the product of two MPIDENSE matrices? Thanks, Joon From ztdepyahoo at 163.com Sun Sep 15 08:49:40 2013 From: ztdepyahoo at 163.com (=?GBK?B?tqHAz8qm?=) Date: Sun, 15 Sep 2013 21:49:40 +0800 (CST) Subject: [petsc-users] Does the kspsolver used in the DMDA take the special structrue of the cartesian coordinate into consideration. Message-ID: <476b3120.5fb7.14121e24c8c.Coremail.ztdepyahoo@163.com> since in cartesian coordinate system, the matrix vector operation can be explicitly known. Does the kspsolver used in the DMDA take the special structrue of the cartesian coordinate into consideration. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sun Sep 15 10:07:21 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 15 Sep 2013 08:07:21 -0700 Subject: [petsc-users] Does the kspsolver used in the DMDA take the special structrue of the cartesian coordinate into consideration. In-Reply-To: <476b3120.5fb7.14121e24c8c.Coremail.ztdepyahoo@163.com> References: <476b3120.5fb7.14121e24c8c.Coremail.ztdepyahoo@163.com> Message-ID: <877geirumu.fsf@mcs.anl.gov> ??? writes: > since in cartesian coordinate system, the matrix vector operation can > be explicitly known. What can be "explicitly known"? > Does the kspsolver used in the DMDA take the special structrue of the > cartesian coordinate into consideration. DMDA sets nonzero pattern, but you assemble the matrix. You can also define matrix-free products, but don't do that until you are happy with a preconditioner and profiling tells you that optimizing the matrix representation will give a speedup that is more valuable than either (a) further optimizing preconditioning algorithms or (b) adding new physics/analysis to your code. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From hariseldon99 at gmail.com Mon Sep 16 03:18:09 2013 From: hariseldon99 at gmail.com (Analabha Roy) Date: Mon, 16 Sep 2013 13:48:09 +0530 Subject: [petsc-users] make all freezes Message-ID: Hi, I just got a new laptop and installed linux mint KDE. Details below: samsung NP300E5X-U01IN: Kernel 3.8.0-26-generic #38-Ubuntu SMP Four Intel Unknown 800MHz processors, 16760.20 total bogomips, 3733M RAM System library 2.17.0 I tried to compile petsc-3.4.2 with the following configuration ./configure --with-scalar-type=complex --with-c2html=0 --with-debugging=no --with-fortran-kernels=1 --with-hdf5=yes --with-hdf5-dir=/usr --enable-parallel The configure log is posted here(dropbox) The configure script seems to run okay, but running "make PETSC_DIR=/usr/local/src/petsc-3.4.2 PETSC_ARCH=arch-linux2-c-opt all" or "make all" causes the compile to hang with no output. Downgrading to a previous version of petsc does not seem to help. The problem persists. Using default options "./configure --with-scalar-type=complex" doesn;t seem to change anything either. Running ./config/builder.py works, but then "make test" hangs similarly, as does running make on any example or any of my petsc sources. Thanks in advance, -- --- *Analabha Roy* C.S.I.R Senior Research Associate Saha Institute of Nuclear Physics Section 1, Block AF Bidhannagar, Calcutta 700064 India *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com *Webpage*: http://www.ph.utexas.edu/~daneel/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ztdepyahoo at 163.com Mon Sep 16 07:14:17 2013 From: ztdepyahoo at 163.com (=?GBK?B?tqHAz8qm?=) Date: Mon, 16 Sep 2013 20:14:17 +0800 (CST) Subject: [petsc-users] [2]PETSC ERROR: MatPermute not available for Mat type mpiaij! Message-ID: <242674af.18f0d.14126b153ca.Coremail.ztdepyahoo@163.com> i set up my matrix A with MatCreate(PETSC_COMM_WORLD,&A); MatSetSizes(A,PMesh->LocalSize,PMesh->LocalSize,PETSC_DECIDE,PETSC_DECIDE); MatSetFromOptions(A); MatMPIAIJSetPreallocation(A,5,PETSC_NULL,5,PETSC_NULL); then which type should i use to support the matpermute. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Sep 16 08:33:58 2013 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 16 Sep 2013 06:33:58 -0700 Subject: [petsc-users] make all freezes In-Reply-To: References: Message-ID: On Mon, Sep 16, 2013 at 1:18 AM, Analabha Roy wrote: > Hi, > > I just got a new laptop and installed linux mint KDE. Details below: > > samsung NP300E5X-U01IN: > Kernel 3.8.0-26-generic #38-Ubuntu SMP > Four Intel Unknown 800MHz processors, 16760.20 total bogomips, 3733M RAM > System library 2.17.0 > > > I tried to compile petsc-3.4.2 with the following configuration > > ./configure --with-scalar-type=complex --with-c2html=0 --with-debugging=no > --with-fortran-kernels=1 --with-hdf5=yes --with-hdf5-dir=/usr > --enable-parallel > > > The configure log is posted here(dropbox) > > The configure script seems to run okay, but running "make > PETSC_DIR=/usr/local/src/petsc-3.4.2 PETSC_ARCH=arch-linux2-c-opt all" or > "make all" causes the compile to hang with no output. > This sounds like a CMake problem on your machine. Please try running make all-legacy although it seems like your Builder run succeeded. Downgrading to a previous version of petsc does not seem to help. The > problem persists. Using default options "./configure > --with-scalar-type=complex" doesn;t seem to change anything either. > > Running ./config/builder.py works, but then "make test" hangs similarly, > as does running make on any example or any of my petsc sources. > This is different. It is likely that your MPI is not setup correctly to launch on this machine. Can you run any MPI job? Thanks, Matt > Thanks in advance, > > > > > > -- > --- > *Analabha Roy* > C.S.I.R Senior Research Associate > Saha Institute of Nuclear Physics > Section 1, Block AF > Bidhannagar, Calcutta 700064 > India > *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com > *Webpage*: http://www.ph.utexas.edu/~daneel/ > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Sep 16 11:19:39 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 16 Sep 2013 11:19:39 -0500 Subject: [petsc-users] [2]PETSC ERROR: MatPermute not available for Mat type mpiaij! In-Reply-To: <242674af.18f0d.14126b153ca.Coremail.ztdepyahoo@163.com> References: <242674af.18f0d.14126b153ca.Coremail.ztdepyahoo@163.com> Message-ID: <59E92E0D-2ABB-4A0E-B34E-6E85AD27F28E@mcs.anl.gov> I assume that his matrix A is suppose to represent the connections in the mesh. 1) You only want to reorder the mesh locally (assuming you have already partitioned the mesh in parallel for good load balancing and low number of cut eges) so each process will call MatCreate() with PETSC_COMM_SELF and each set a matrix associated only with the local part of the mesh. 2) You don't want to use MatPermute() rather you would use MatGetOrdering() which tells you how to reorder the unknowns on the mesh on each process. For the ordering I think you want to use MATORDERINGRCM. Now you just renumber the mesh nodes on each process and update the the ghost node information to reflect the new ordering. Barry On Sep 16, 2013, at 7:14 AM, ??? wrote: > i set up my matrix A with > MatCreate(PETSC_COMM_WORLD,&A); > MatSetSizes(A,PMesh->LocalSize,PMesh->LocalSize,PETSC_DECIDE,PETSC_DECIDE); > MatSetFromOptions(A); > MatMPIAIJSetPreallocation(A,5,PETSC_NULL,5,PETSC_NULL); > > > then which type should i use to support the matpermute. > > > > > > > > > > < span title="neteasefooter"> From jefonseca at gmail.com Mon Sep 16 13:08:34 2013 From: jefonseca at gmail.com (Jim Fonseca) Date: Mon, 16 Sep 2013 14:08:34 -0400 Subject: [petsc-users] PETSc 3.4 and TAO Message-ID: Hi, I found a couple postings online that say that TAO 2.1 is not compatible with PETSc 3.4. Can anyone please comment on that? Thanks, Jim -- Jim Fonseca, PhD Research Scientist Network for Computational Nanotechnology Purdue University 765-496-6495 www.jimfonseca.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From jefonseca at gmail.com Mon Sep 16 13:18:45 2013 From: jefonseca at gmail.com (Jim Fonseca) Date: Mon, 16 Sep 2013 14:18:45 -0400 Subject: [petsc-users] PETSc 3.4 and TAO In-Reply-To: References: Message-ID: Looks like TAO 2.2 with PETSc 3.4 compatibility will be released later this week. Thanks, Jim On Mon, Sep 16, 2013 at 2:08 PM, Jim Fonseca wrote: > Hi, > I found a couple postings online that say that TAO 2.1 is not compatible > with PETSc 3.4. Can anyone please comment on that? > > Thanks, > Jim > > -- > Jim Fonseca, PhD > Research Scientist > Network for Computational Nanotechnology > Purdue University > 765-496-6495 > www.jimfonseca.com > > > -- Jim Fonseca, PhD Research Scientist Network for Computational Nanotechnology Purdue University 765-496-6495 www.jimfonseca.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From boillod.france at gmail.com Mon Sep 16 16:10:32 2013 From: boillod.france at gmail.com (France Boillod-Cerneux) Date: Mon, 16 Sep 2013 23:10:32 +0200 Subject: [petsc-users] how to get info on matmpiaij distribution Message-ID: Dear PETSc-users, I need some help regarding the matmpiaij matrix format. In my application, I use currently the matmpiaij format, and i load the matrix using petsc binary file format. i understood that for mpiaij format, wether you know the distribution of your matrix and use the matcreateaij function therefore you can know with precision the data distribution, wether you create it without this knowledge and petsc will take care about efficient data distribution for you. i am in the second case, and i want to know the memory at runtime allowed by petsc to store the matrix on each process, for the format mpiaij. i saw the function matgetlocalsize but there's a warning on it (function is implementation dependant use with care). so i don't know if in my case i can use it or not. (if you have any documentation about it, i would like to have it) if not, can you tell me if there is any petsc function to get the memory size of submatrix on each process? i saw the function petscmemorygetcurrentusage but i am not sure if it would correspond to my issue. if still not, how can i estimate it (i mean by this, i can have approximately the parallel matrix distribution with mpiaij format)? thank you very much, France -- DEN/DANS/DM2S CEA Saclay 91191 Gif-sur-Yvette FRANCE -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Sep 16 16:33:41 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 16 Sep 2013 16:33:41 -0500 Subject: [petsc-users] how to get info on matmpiaij distribution In-Reply-To: References: Message-ID: MatGetInfo() On Sep 16, 2013, at 4:10 PM, France Boillod-Cerneux wrote: > Dear PETSc-users, > I need some help regarding the matmpiaij matrix format. > > In my application, I use currently the matmpiaij format, and i load the matrix using petsc binary file format. > > i understood that for mpiaij format, wether you know the distribution of your matrix and use the matcreateaij function therefore you can know with precision the data distribution, > wether you create it without this knowledge and petsc will take care about efficient data distribution for you. > > i am in the second case, and i want to know the memory at runtime allowed by petsc to store the matrix on each process, for the format mpiaij. > i saw the function matgetlocalsize but there's a warning on it (function is implementation dependant use with care). so i don't know if in my case i can use it or not. (if you have any documentation about it, i would like to have it) > > if not, can you tell me if there is any petsc function to get the memory size of submatrix on each process? i saw the function petscmemorygetcurrentusage but i am not sure if it would correspond to my issue. if still not, how can i estimate it (i mean by this, i can have approximately the parallel matrix distribution with mpiaij format)? > > thank you very much, > > France > > -- > DEN/DANS/DM2S > CEA Saclay > 91191 Gif-sur-Yvette > FRANCE > > From ztdepyahoo at 163.com Mon Sep 16 19:17:26 2013 From: ztdepyahoo at 163.com (=?GBK?B?tqHAz8qm?=) Date: Tue, 17 Sep 2013 08:17:26 +0800 (CST) Subject: [petsc-users] [2]PETSC ERROR: MatPermute not available for Mat type mpiaij! In-Reply-To: <59E92E0D-2ABB-4A0E-B34E-6E85AD27F28E@mcs.anl.gov> References: <242674af.18f0d.14126b153ca.Coremail.ztdepyahoo@163.com> <59E92E0D-2ABB-4A0E-B34E-6E85AD27F28E@mcs.anl.gov> Message-ID: Daer Barry: Matrix A is the coeffcients not the mesh. I use the MatPermute following the ksp examples in dev version ex18. After the line 77, we declared a parallel matrix , at line 78, we declare a seq matrix. so what is this seq matrix for ? 77: MatMPIAIJSetPreallocation(A,5,NULL,5,NULL); 78: MatSeqAIJSetPreallocation(A,5,NULL); At 2013-09-17 00:19:39,"Barry Smith" wrote: > > I assume that his matrix A is suppose to represent the connections in the mesh. > > 1) You only want to reorder the mesh locally (assuming you have already partitioned the mesh in parallel for good load balancing and low number of cut eges) so each process will call MatCreate() with PETSC_COMM_SELF and each set a matrix associated only with the local part of the mesh. > > 2) You don't want to use MatPermute() rather you would use MatGetOrdering() which tells you how to reorder the unknowns on the mesh on each process. For the ordering I think you want to use MATORDERINGRCM. Now you just renumber the mesh nodes on each process and update the the ghost node information to reflect the new ordering. > > Barry > > >On Sep 16, 2013, at 7:14 AM, ??? wrote: > >> i set up my matrix A with >> MatCreate(PETSC_COMM_WORLD,&A); >> MatSetSizes(A,PMesh->LocalSize,PMesh->LocalSize,PETSC_DECIDE,PETSC_DECIDE); >> MatSetFromOptions(A); >> MatMPIAIJSetPreallocation(A,5,PETSC_NULL,5,PETSC_NULL); >> >> >> then which type should i use to support the matpermute. >> >> >> >> >> >> >> >> >> >> < span title="neteasefooter"> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Sep 16 20:04:11 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 16 Sep 2013 20:04:11 -0500 Subject: [petsc-users] [2]PETSC ERROR: MatPermute not available for Mat type mpiaij! In-Reply-To: References: <242674af.18f0d.14126b153ca.Coremail.ztdepyahoo@163.com> <59E92E0D-2ABB-4A0E-B34E-6E85AD27F28E@mcs.anl.gov> Message-ID: I thought we discussed this before and stated that reordering the matrix, without reordering the mesh first for Krylov methods is not the correct approach. Barry On Sep 16, 2013, at 7:17 PM, ??? wrote: > Daer Barry: > Matrix A is the coeffcients not the mesh. I use the MatPermute following the ksp examples in dev version ex18. > After the line 77, we declared a parallel matrix , at line 78, we declare a seq matrix. > so what is this seq matrix for ? > 77: MatMPIAIJSetPreallocation > (A,5,NULL,5,NULL); > > 78: MatSeqAIJSetPreallocation(A,5,NULL); > > > > > > > > At 2013-09-17 00:19:39,"Barry Smith" wrote: > > > > I assume that his matrix A is suppose to represent the connections in the mesh. > > > > 1) You only want to reorder the mesh locally (assuming you have already partitioned the mesh in parallel for good load balancing and low number of cut eges) so each process will call MatCreate() with PETSC_COMM_SELF and each set a matrix associated only with the local part of the mesh. > > > > 2) You don't want to use MatPermute() rather you would use MatGetOrdering() which tells you how to reorder the unknowns on the mesh on each process. For the ordering I think you want to use MATORDERINGRCM. Now you just renumber the mesh nodes on each process and update the the ghost node information to reflect the new ordering. > > > > Barry > > > > > >On Sep 16, 2013, at 7:14 AM, ??? wrote: > > > >> i set up my matrix A with > >> MatCreate(PETSC_COMM_WORLD,&A); > >> MatSetSizes(A,PMesh->LocalSize,PMesh->LocalSize,PETSC_DECIDE,PETSC_DECIDE); > >> MatSetFromOptions(A); > >> MatMPIAIJSetPreallocation(A,5,PETSC_NULL,5,PETSC_NULL); > >> > >> > >> then which type should i use to support the matpermute. > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> < span title="neteasefooter"> > > > > > From hariseldon99 at gmail.com Tue Sep 17 00:47:19 2013 From: hariseldon99 at gmail.com (Analabha Roy) Date: Tue, 17 Sep 2013 11:17:19 +0530 Subject: [petsc-users] make all freezes In-Reply-To: References: Message-ID: Hi, Sorry for the late reply On Mon, Sep 16, 2013 at 7:03 PM, Matthew Knepley wrote: > On Mon, Sep 16, 2013 at 1:18 AM, Analabha Roy wrote: > >> Hi, >> >> I just got a new laptop and installed linux mint KDE. Details below: >> >> samsung NP300E5X-U01IN: >> Kernel 3.8.0-26-generic #38-Ubuntu SMP >> Four Intel Unknown 800MHz processors, 16760.20 total bogomips, 3733M RAM >> System library 2.17.0 >> >> >> I tried to compile petsc-3.4.2 with the following configuration >> >> ./configure --with-scalar-type=complex --with-c2html=0 >> --with-debugging=no --with-fortran-kernels=1 --with-hdf5=yes >> --with-hdf5-dir=/usr --enable-parallel >> >> >> The configure log is posted here(dropbox) >> >> The configure script seems to run okay, but running "make >> PETSC_DIR=/usr/local/src/petsc-3.4.2 PETSC_ARCH=arch-linux2-c-opt all" or >> "make all" causes the compile to hang with no output. >> > > This sounds like a CMake problem on your machine. Please try running > > make all-legacy > > although it seems like your Builder run succeeded. > I purged cmake using apt-get and reinstalled it from scratch. The problem persists I also ran "make all-legacy". Same problem. No output. Just hangs. > > Downgrading to a previous version of petsc does not seem to help. The >> problem persists. Using default options "./configure >> --with-scalar-type=complex" doesn;t seem to change anything either. >> >> Running ./config/builder.py works, but then "make test" hangs similarly, >> as does running make on any example or any of my petsc sources. >> > > This is different. It is likely that your MPI is not setup correctly to > launch on this machine. Can you > run any MPI job? > > Yes. Both this hello world exampleand this calculation of picompiles without errors and runs as expected. Should I try purging openmpi and reinstalling? > Thanks, > > Matt > > >> Thanks in advance, >> >> >> >> >> >> -- >> --- >> *Analabha Roy* >> C.S.I.R Senior Research Associate >> Saha Institute of Nuclear Physics >> Section 1, Block AF >> Bidhannagar, Calcutta 700064 >> India >> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >> *Webpage*: http://www.ph.utexas.edu/~daneel/ >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- --- *Analabha Roy* C.S.I.R Senior Research Associate Saha Institute of Nuclear Physics Section 1, Block AF Bidhannagar, Calcutta 700064 India *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com *Webpage*: http://www.ph.utexas.edu/~daneel/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From choi240 at purdue.edu Tue Sep 17 01:07:30 2013 From: choi240 at purdue.edu (Joon Hee Choi) Date: Tue, 17 Sep 2013 02:07:30 -0400 (EDT) Subject: [petsc-users] Inverse matrix of MATMPIDENSE matrix In-Reply-To: <103574335.114115.1379397692532.JavaMail.root@mailhub028.itcs.purdue.edu> Message-ID: <909698812.114127.1379398050337.JavaMail.root@mailhub028.itcs.purdue.edu> Hello, I want to get the inverse matrix of 10 x 10 MATMPIDENSE matrix. I know I cannot use MatLUFactor() for MATMPIDENSE. Is there any other method? Could you let me know the fastest way to get the inverse? Also, I don't want to rebuild or configure something for the inverse because I use petsc in my school server and I don't have permission to configure. Thanks, Joon From hariseldon99 at gmail.com Tue Sep 17 02:54:05 2013 From: hariseldon99 at gmail.com (Analabha Roy) Date: Tue, 17 Sep 2013 13:24:05 +0530 Subject: [petsc-users] make all freezes In-Reply-To: References: Message-ID: Update: Purging openmpi and reinstalling did not work. "make all" in $PETSC_DIR still freezes. On Tue, Sep 17, 2013 at 11:17 AM, Analabha Roy wrote: > Hi, > > Sorry for the late reply > > > On Mon, Sep 16, 2013 at 7:03 PM, Matthew Knepley wrote: > >> On Mon, Sep 16, 2013 at 1:18 AM, Analabha Roy wrote: >> >>> Hi, >>> >>> I just got a new laptop and installed linux mint KDE. Details below: >>> >>> samsung NP300E5X-U01IN: >>> Kernel 3.8.0-26-generic #38-Ubuntu SMP >>> Four Intel Unknown 800MHz processors, 16760.20 total bogomips, 3733M RAM >>> System library 2.17.0 >>> >>> >>> I tried to compile petsc-3.4.2 with the following configuration >>> >>> ./configure --with-scalar-type=complex --with-c2html=0 >>> --with-debugging=no --with-fortran-kernels=1 --with-hdf5=yes >>> --with-hdf5-dir=/usr --enable-parallel >>> >>> >>> The configure log is posted here(dropbox) >>> >>> The configure script seems to run okay, but running "make >>> PETSC_DIR=/usr/local/src/petsc-3.4.2 PETSC_ARCH=arch-linux2-c-opt all" or >>> "make all" causes the compile to hang with no output. >>> >> >> This sounds like a CMake problem on your machine. Please try running >> >> make all-legacy >> >> although it seems like your Builder run succeeded. >> > > > I purged cmake using apt-get and reinstalled it from scratch. The problem > persists > > I also ran "make all-legacy". Same problem. No output. Just hangs. > > > >> >> Downgrading to a previous version of petsc does not seem to help. The >>> problem persists. Using default options "./configure >>> --with-scalar-type=complex" doesn;t seem to change anything either. >>> >>> Running ./config/builder.py works, but then "make test" hangs similarly, >>> as does running make on any example or any of my petsc sources. >>> >> >> This is different. It is likely that your MPI is not setup correctly to >> launch on this machine. Can you >> run any MPI job? >> >> > Yes. Both this hello world exampleand this calculation > of picompiles without errors and runs as expected. > > > Should I try purging openmpi and reinstalling? > > >> Thanks, >> >> Matt >> >> >>> Thanks in advance, >>> >>> >>> >>> >>> >>> -- >>> --- >>> *Analabha Roy* >>> C.S.I.R Senior Research Associate >>> Saha Institute of Nuclear Physics >>> Section 1, Block AF >>> Bidhannagar, Calcutta 700064 >>> India >>> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >>> *Webpage*: http://www.ph.utexas.edu/~daneel/ >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > --- > *Analabha Roy* > C.S.I.R Senior Research Associate > Saha Institute of Nuclear Physics > Section 1, Block AF > Bidhannagar, Calcutta 700064 > India > *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com > *Webpage*: http://www.ph.utexas.edu/~daneel/ > -- --- *Analabha Roy* C.S.I.R Senior Research Associate Saha Institute of Nuclear Physics Section 1, Block AF Bidhannagar, Calcutta 700064 India *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com *Webpage*: http://www.ph.utexas.edu/~daneel/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From hariseldon99 at gmail.com Tue Sep 17 04:57:28 2013 From: hariseldon99 at gmail.com (Analabha Roy) Date: Tue, 17 Sep 2013 15:27:28 +0530 Subject: [petsc-users] make all freezes In-Reply-To: References: Message-ID: Update again: On a hunch, I went to a non-graphical tty where compilation worked! So I installed all xorg-devel packages from repository and now compilation seems to be working. Guess it just needed those... Thanks. On Tue, Sep 17, 2013 at 1:24 PM, Analabha Roy wrote: > Update: Purging openmpi and reinstalling did not work. "make all" in > $PETSC_DIR still freezes. > > > On Tue, Sep 17, 2013 at 11:17 AM, Analabha Roy wrote: > >> Hi, >> >> Sorry for the late reply >> >> >> On Mon, Sep 16, 2013 at 7:03 PM, Matthew Knepley wrote: >> >>> On Mon, Sep 16, 2013 at 1:18 AM, Analabha Roy wrote: >>> >>>> Hi, >>>> >>>> I just got a new laptop and installed linux mint KDE. Details below: >>>> >>>> samsung NP300E5X-U01IN: >>>> Kernel 3.8.0-26-generic #38-Ubuntu SMP >>>> Four Intel Unknown 800MHz processors, 16760.20 total bogomips, 3733M RAM >>>> System library 2.17.0 >>>> >>>> >>>> I tried to compile petsc-3.4.2 with the following configuration >>>> >>>> ./configure --with-scalar-type=complex --with-c2html=0 >>>> --with-debugging=no --with-fortran-kernels=1 --with-hdf5=yes >>>> --with-hdf5-dir=/usr --enable-parallel >>>> >>>> >>>> The configure log is posted here(dropbox) >>>> >>>> The configure script seems to run okay, but running "make >>>> PETSC_DIR=/usr/local/src/petsc-3.4.2 PETSC_ARCH=arch-linux2-c-opt all" or >>>> "make all" causes the compile to hang with no output. >>>> >>> >>> This sounds like a CMake problem on your machine. Please try running >>> >>> make all-legacy >>> >>> although it seems like your Builder run succeeded. >>> >> >> >> I purged cmake using apt-get and reinstalled it from scratch. The problem >> persists >> >> I also ran "make all-legacy". Same problem. No output. Just hangs. >> >> >> >>> >>> Downgrading to a previous version of petsc does not seem to help. The >>>> problem persists. Using default options "./configure >>>> --with-scalar-type=complex" doesn;t seem to change anything either. >>>> >>>> Running ./config/builder.py works, but then "make test" hangs >>>> similarly, as does running make on any example or any of my petsc sources. >>>> >>> >>> This is different. It is likely that your MPI is not setup correctly to >>> launch on this machine. Can you >>> run any MPI job? >>> >>> >> Yes. Both this hello world exampleand this calculation >> of picompiles without errors and runs as expected. >> >> >> Should I try purging openmpi and reinstalling? >> >> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thanks in advance, >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> --- >>>> *Analabha Roy* >>>> C.S.I.R Senior Research Associate >>>> Saha Institute of Nuclear Physics >>>> Section 1, Block AF >>>> Bidhannagar, Calcutta 700064 >>>> India >>>> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >>>> *Webpage*: http://www.ph.utexas.edu/~daneel/ >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> >> >> -- >> --- >> *Analabha Roy* >> C.S.I.R Senior Research Associate >> Saha Institute of Nuclear Physics >> Section 1, Block AF >> Bidhannagar, Calcutta 700064 >> India >> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >> *Webpage*: http://www.ph.utexas.edu/~daneel/ >> > > > > -- > --- > *Analabha Roy* > C.S.I.R Senior Research Associate > Saha Institute of Nuclear Physics > Section 1, Block AF > Bidhannagar, Calcutta 700064 > India > *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com > *Webpage*: http://www.ph.utexas.edu/~daneel/ > -- --- *Analabha Roy* C.S.I.R Senior Research Associate Saha Institute of Nuclear Physics Section 1, Block AF Bidhannagar, Calcutta 700064 India *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com *Webpage*: http://www.ph.utexas.edu/~daneel/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Sep 17 06:59:42 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 17 Sep 2013 06:59:42 -0500 Subject: [petsc-users] Inverse matrix of MATMPIDENSE matrix In-Reply-To: <909698812.114127.1379398050337.JavaMail.root@mailhub028.itcs.purdue.edu> References: <909698812.114127.1379398050337.JavaMail.root@mailhub028.itcs.purdue.edu> Message-ID: <0CF388B0-348C-440F-ABBD-A53397A45517@mcs.anl.gov> http://www.mcs.anl.gov/petsc/documentation/faq.html#invertmatrix On Sep 17, 2013, at 1:07 AM, Joon Hee Choi wrote: > Hello, > > I want to get the inverse matrix of 10 x 10 MATMPIDENSE matrix. I know I cannot use MatLUFactor() for MATMPIDENSE. Is there any other method? Could you let me know the fastest way to get the inverse? Also, I don't want to rebuild or configure something for the inverse because I use petsc in my school server and I don't have permission to configure. > > Thanks, > Joon From frtr at fysik.dtu.dk Tue Sep 17 07:59:46 2013 From: frtr at fysik.dtu.dk (Frederik Treue) Date: Tue, 17 Sep 2013 14:59:46 +0200 Subject: [petsc-users] zero pattern of result of matmatmult In-Reply-To: <87ppsm9in9.fsf@mcs.anl.gov> References: <1378396034.2282.15.camel@frtr-laptop> <87ioyf6ys8.fsf@mcs.anl.gov> <1378461087.13721.10.camel@frtr-laptop> <87ppsm9in9.fsf@mcs.anl.gov> Message-ID: <1379422786.6379.5.camel@frtr-laptop> On Fri, 2013-09-06 at 08:36 -0500, Jed Brown wrote: > Frederik Treue writes: > >> Multiplying difference operators together is really brittle > >> (boundary conditions, geometry, etc) and uses more memory than > >> necessary. > > > > Why? Ofcourse, it could go wrong if you get the non-zero pattern of the > > product matrix wrong, but that's the same as with all other instances of > > SAME_NONZERO_PATTERN throughout petsc. Why is this one special? > > The symbolic part of the matrix multiply does not use entries, and > ensures that the numeric part has no memory allocation and no > conditionals. > > Implementing boundary conditions and variable mesh spacing by splitting > into first order operators is quite clumsy and a lot less flexible when > you want to change the discretization. I think your approach is a false > economy and you should just write the local routines that evaluate the > actual Jacobian. You can factor those functions however you like > (e.g., finite element and finite volume would look a bit different). I have done this, but the problem still is that it takes too much time to recreate the matrix in each timestep (the time it takes to construct the matrix is roughly equivalent to the time it takes to solve it using CG with ~50 iterations), so: What is the fastest way to change the entries in a matrix? Right now I'm reduced to recreating the whole thing with MatSetValuesStencil in a for loop with MatAssembly afterwards. How would one ideally do this? /Frederik From bsmith at mcs.anl.gov Tue Sep 17 08:15:30 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 17 Sep 2013 08:15:30 -0500 Subject: [petsc-users] zero pattern of result of matmatmult In-Reply-To: <1379422786.6379.5.camel@frtr-laptop> References: <1378396034.2282.15.camel@frtr-laptop> <87ioyf6ys8.fsf@mcs.anl.gov> <1378461087.13721.10.camel@frtr-laptop> <87ppsm9in9.fsf@mcs.anl.gov> <1379422786.6379.5.camel@frtr-laptop> Message-ID: When the linear system is fairly easy it is possible for the construction of the matrix to take a significant portion of the total time. Run with -log_summary and send the output, from this we may be able to make suggestions as to what further improvements can be made. Is any part of the matrix constant? If so you can use MatStoreValues() and MatRetrieveValues() so that part does not have to be recomputed each time. Are you calling MatSetValuesStencil() for each single entry or once per row or block row? Row or block row will be faster than for each entry. Since you are using MatSetValuesStencil() I assume you are using a DA? How many degrees of freedom do you have per grid point? If it is more than one then MatSetValuesBlockedStencil() might be faster, also you might you use MatType BAIJ instead of AIJ. Or if the blocks are not dense then use AIJ but use DMDASetBlockFills() to eliminate all the entries that actually zero. Barry On Sep 17, 2013, at 7:59 AM, Frederik Treue wrote: > On Fri, 2013-09-06 at 08:36 -0500, Jed Brown wrote: >> Frederik Treue writes: >>>> Multiplying difference operators together is really brittle > > I have done this, but the problem still is that it takes too much time > to recreate the matrix in each timestep (the time it takes to construct > the matrix is roughly equivalent to the time it takes to solve it using > CG with ~50 iterations), so: What is the fastest way to change the > entries in a matrix? Right now I'm reduced to recreating the whole thing > with MatSetValuesStencil in a for loop with MatAssembly afterwards. How > would one ideally do this? > > /Frederik > From frtr at fysik.dtu.dk Tue Sep 17 08:57:22 2013 From: frtr at fysik.dtu.dk (Frederik Treue) Date: Tue, 17 Sep 2013 15:57:22 +0200 Subject: [petsc-users] zero pattern of result of matmatmult In-Reply-To: References: <1378396034.2282.15.camel@frtr-laptop> <87ioyf6ys8.fsf@mcs.anl.gov> <1378461087.13721.10.camel@frtr-laptop> <87ppsm9in9.fsf@mcs.anl.gov> <1379422786.6379.5.camel@frtr-laptop> Message-ID: <1379426242.2358.8.camel@frtr-laptop> On Tue, 2013-09-17 at 08:15 -0500, Barry Smith wrote: > When the linear system is fairly easy it is possible for the construction of the matrix to take a significant portion of the total time. Run with -log_summary and send the output, from this we may be able to make suggestions as to what further improvements can be made. Attached. > > Is any part of the matrix constant? If so you can use MatStoreValues() and MatRetrieveValues() so that part does not have to be recomputed each time. Not in terms of entries in the matrix. Mathematically, part of the operator is constant, but all entries has changing parts added to them. > > Are you calling MatSetValuesStencil() for each single entry or once per row or block row? Row or block row will be faster than for each entry. once per entry - this I should definitely improve. What's the logic behind row or block rows being faster? The fewer calls to MatSetValuesStencil, the better? > > Since you are using MatSetValuesStencil() I assume you are using a DA? I am, with one DOF per point. Are there better alternatives? PS. FYI: My stencil is essentially a standard 2D, 9-point box stencil, but the values in all rows are different (due, among other things, to the geometric tensor). Thanks for the help... /Frederik -------------- next part -------------- ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./petsclapl on a arch-linux2-c-debug named frtr-laptop with 2 processors, by frtr Tue Sep 17 14:24:35 2013 Using Petsc Release Version 3.4.1, Jun, 10, 2013 Max Max/Min Avg Total Time (sec): 4.045e+02 1.00001 4.045e+02 Objects: 1.336e+03 1.00000 1.336e+03 Flops: 8.196e+08 1.00000 8.196e+08 1.639e+09 Flops/sec: 2.026e+06 1.00001 2.026e+06 4.052e+06 Memory: 2.755e+07 1.00000 5.510e+07 MPI Messages: 3.902e+03 1.00000 3.902e+03 7.804e+03 MPI Message Lengths: 6.317e+06 1.00000 1.619e+03 1.263e+07 MPI Reductions: 3.193e+04 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 4.0449e+02 100.0% 1.6392e+09 100.0% 7.804e+03 100.0% 1.619e+03 100.0% 3.193e+04 100.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %f - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ ########################################################## # # # WARNING!!! # # # # This code was compiled with a debugging option, # # To get timing results run ./configure # # using --with-debugging=no, the performance will # # be generally two or three times faster. # # # ########################################################## Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage ThreadCommRunKer 1 1.0 9.8039e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 ThreadCommBarrie 1 1.0 6.7013e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecMDot 250 1.0 9.3867e-01 1.1 4.59e+06 1.0 0.0e+00 0.0e+00 2.5e+02 0 1 0 0 1 0 1 0 0 1 10 VecTDot 5782 1.0 5.8286e+01 1.0 5.90e+07 1.0 0.0e+00 0.0e+00 5.8e+03 14 7 0 0 18 14 7 0 0 18 2 VecNorm 3341 1.0 1.1062e+01 1.1 3.41e+07 1.0 0.0e+00 0.0e+00 3.3e+03 3 4 0 0 10 3 4 0 0 10 6 VecScale 452 1.0 3.6935e+00 1.0 2.31e+06 1.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 1 VecCopy 308 1.0 1.0530e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 3756 1.0 7.7467e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 6885 1.0 4.3978e+01 1.0 6.97e+07 1.0 0.0e+00 0.0e+00 0.0e+00 11 9 0 0 0 11 9 0 0 0 3 VecAYPX 2891 1.0 1.2989e+00 1.0 2.92e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 4 0 0 0 0 4 0 0 0 45 VecMAXPY 350 1.0 1.7310e-01 1.2 7.14e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 82 VecAssemblyBegin 36 1.0 2.2704e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 9.9e+01 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd 36 1.0 5.6790e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 491 1.0 1.9725e-01 1.1 2.50e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 25 VecScatterBegin 3900 1.0 1.3175e+00 1.0 0.00e+00 0.0 7.6e+03 1.6e+03 0.0e+00 0 0 97 99 0 0 0 97 99 0 0 VecScatterEnd 3900 1.0 3.3521e+00 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 VecNormalize 350 1.0 3.4656e+00 1.0 5.36e+06 1.0 0.0e+00 0.0e+00 3.5e+02 1 1 0 0 1 1 1 0 0 1 3 MatMult 3600 1.0 2.0704e+01 1.0 3.10e+08 1.0 7.2e+03 1.6e+03 0.0e+00 5 38 92 93 0 5 38 92 93 0 30 MatSolve 3341 1.0 1.2289e+01 1.0 2.84e+08 1.0 0.0e+00 0.0e+00 0.0e+00 3 35 0 0 0 3 35 0 0 0 46 MatLUFactorNum 56 1.0 3.4288e+00 1.0 7.72e+06 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 5 MatILUFactorSym 56 1.0 9.8914e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 5.6e+01 0 0 0 0 0 0 0 0 0 0 0 MatCopy 212 1.0 1.9453e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 7.2e+01 5 0 0 0 0 5 0 0 0 0 0 MatScale 176 1.0 5.4831e-01 1.0 8.07e+06 1.0 0.0e+00 0.0e+00 2.2e+01 0 1 0 0 0 0 1 0 0 0 29 MatAssemblyBegin 85 1.0 1.5119e+00 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.7e+02 0 0 0 0 1 0 0 0 0 1 0 MatAssemblyEnd 85 1.0 1.5816e+00 1.0 0.00e+00 0.0 1.9e+02 4.9e+02 1.1e+03 0 0 2 1 4 0 0 2 1 4 0 MatGetRow 285600 1.0 2.5095e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 6 0 0 0 0 6 0 0 0 0 0 MatGetRowIJ 56 1.0 4.7718e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 56 1.0 6.4522e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.1e+02 0 0 0 0 0 0 0 0 0 0 0 MatZeroEntries 18 1.0 1.8103e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAXPY 174 1.0 7.3996e+01 1.0 0.00e+00 0.0 8.8e+01 4.1e+02 1.0e+03 18 0 1 0 3 18 0 1 0 3 0 MatMatMult 1 1.0 9.7952e-01 1.0 8.17e+05 1.0 1.2e+01 6.5e+03 6.2e+01 0 0 0 1 0 0 0 0 1 0 2 MatMatMultSym 1 1.0 8.2768e-01 1.0 0.00e+00 0.0 1.0e+01 4.9e+03 5.6e+01 0 0 0 0 0 0 0 0 0 0 0 MatMatMultNum 1 1.0 1.5126e-01 1.0 8.17e+05 1.0 2.0e+00 1.5e+04 6.0e+00 0 0 0 0 0 0 0 0 0 0 11 MatGetLocalMat 2 1.0 3.4862e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetBrAoCol 2 1.0 1.4985e-02 1.0 0.00e+00 0.0 8.0e+00 9.3e+03 6.0e+00 0 0 0 1 0 0 0 0 1 0 0 KSPGMRESOrthog 250 1.0 1.2155e+00 1.1 9.18e+06 1.0 0.0e+00 0.0e+00 7.0e+02 0 1 0 0 2 0 1 0 0 2 15 KSPSetUp 112 1.0 6.0548e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 5.4e+02 0 0 0 0 2 0 0 0 0 2 0 KSPSolve 150 1.0 2.3976e+02 1.0 7.62e+08 1.0 6.4e+03 1.6e+03 2.7e+04 59 93 82 82 85 59 93 82 82 85 6 PCSetUp 112 1.0 5.5250e+00 1.0 7.72e+06 1.0 0.0e+00 0.0e+00 7.8e+02 1 1 0 0 2 1 1 0 0 2 3 PCSetUpOnBlocks 150 1.0 5.5561e+00 1.0 7.72e+06 1.0 0.0e+00 0.0e+00 3.4e+02 1 1 0 0 1 1 1 0 0 1 3 PCApply 3341 1.0 4.7031e+01 1.0 2.84e+08 1.0 0.0e+00 0.0e+00 6.7e+03 12 35 0 0 21 12 35 0 0 21 12 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Vector 561 547 14123724 0 Vector Scatter 55 50 31272 0 Matrix 199 216 68839696 0 Distributed Mesh 4 2 5572 0 Bipartite Graph 8 4 1808 0 Index Set 280 280 1354392 0 IS L to G Mapping 4 2 1076 0 Krylov Solver 112 112 181256 0 Preconditioner 112 112 60704 0 Viewer 1 0 0 0 ======================================================================================================================== Average time to get PetscTime(): 6.2055e-06 Average time for MPI_Barrier(): 0.000123 Average time for zero size MPI_Send(): 5e-05 #PETSc Option Table entries: -DIFFUSION 0.1 -DT 0.01 -ET 0.5 -G 2.0 -LOGGAUSSIAN 1,5,15,10.0,2.0 -NODRAW -OUTSTEP 1 -SQ 30.0,30.0 -SX 100,100,10 -SY 100,100,10 -T 19 -TIMEDEP /home/frtr/tstruns/param.ini -VISCOSITY 0.1 -log_summary #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 4 sizeof(void*) 4 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure run at: Mon Jul 8 14:50:33 2013 Configure options: --prefix=/home/frtr --download-fftw --download-blacs --download-blas --download-lapack --with-mpi-dir=/home/frtr --download-scalapack --with-debugging=1 COPTFLAGS="-O3 -march=native" ----------------------------------------- Libraries compiled on Mon Jul 8 14:50:33 2013 on frtr-laptop Machine characteristics: Linux-3.2.0-49-generic-pae-i686-with-Ubuntu-12.04-precise Using PETSc directory: /home/frtr/work_lib/petsc-3.4.1 Using PETSc arch: arch-linux2-c-debug ----------------------------------------- Using C compiler: /home/frtr/bin/mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O3 -march=native ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: /home/frtr/bin/mpif90 -fPIC -Wall -Wno-unused-variable -Wno-unused-dummy-argument -g ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/home/frtr/work_lib/petsc-3.4.1/arch-linux2-c-debug/include -I/home/frtr/work_lib/petsc-3.4.1/include -I/home/frtr/work_lib/petsc-3.4.1/include -I/home/frtr/work_lib/petsc-3.4.1/arch-linux2-c-debug/include -I/home/frtr/include ----------------------------------------- Using C linker: /home/frtr/bin/mpicc Using Fortran linker: /home/frtr/bin/mpif90 Using libraries: -Wl,-rpath,/home/frtr/work_lib/petsc-3.4.1/arch-linux2-c-debug/lib -L/home/frtr/work_lib/petsc-3.4.1/arch-linux2-c-debug/lib -lpetsc -Wl,-rpath,/home/frtr/work_lib/petsc-3.4.1/arch-linux2-c-debug/lib -L/home/frtr/work_lib/petsc-3.4.1/arch-linux2-c-debug/lib -lscalapack -llapack -lblas -lX11 -lpthread -lfftw3_mpi -lfftw3 -lm -Wl,-rpath,/home/frtr/lib -L/home/frtr/lib -Wl,-rpath,/usr/lib/gcc/i686-linux-gnu/4.6 -L/usr/lib/gcc/i686-linux-gnu/4.6 -Wl,-rpath,/usr/lib/i386-linux-gnu -L/usr/lib/i386-linux-gnu -Wl,-rpath,/lib/i386-linux-gnu -L/lib/i386-linux-gnu -lmpichf90 -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpichcxx -lstdc++ -ldl -lmpich -lopa -lmpl -lrt -lpthread -lgcc_s -ldl ----------------------------------------- From ztdepyahoo at 163.com Tue Sep 17 09:19:27 2013 From: ztdepyahoo at 163.com (=?GBK?B?tqHAz8qm?=) Date: Tue, 17 Sep 2013 22:19:27 +0800 (CST) Subject: [petsc-users] can i define a user defined residual function to determin the convergence of the kspsolve Message-ID: <5c93a96e.12468.1412c4a4713.Coremail.ztdepyahoo@163.com> i want to define a function for the computation of the equation residual in a special way. does petsc provide this capbility -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Sep 17 09:21:49 2013 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 17 Sep 2013 07:21:49 -0700 Subject: [petsc-users] can i define a user defined residual function to determin the convergence of the kspsolve In-Reply-To: <5c93a96e.12468.1412c4a4713.Coremail.ztdepyahoo@163.com> References: <5c93a96e.12468.1412c4a4713.Coremail.ztdepyahoo@163.com> Message-ID: On Tue, Sep 17, 2013 at 7:19 AM, ??? wrote: > i want to define a function for the computation of the equation residual > in a special way. > does petsc provide this capbility > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPSetConvergenceTest.html#KSPSetConvergenceTest Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Sep 17 09:24:11 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 17 Sep 2013 07:24:11 -0700 Subject: [petsc-users] zero pattern of result of matmatmult In-Reply-To: <1379426242.2358.8.camel@frtr-laptop> References: <1378396034.2282.15.camel@frtr-laptop> <87ioyf6ys8.fsf@mcs.anl.gov> <1378461087.13721.10.camel@frtr-laptop> <87ppsm9in9.fsf@mcs.anl.gov> <1379422786.6379.5.camel@frtr-laptop> <1379426242.2358.8.camel@frtr-laptop> Message-ID: <87ob7rsf04.fsf@mcs.anl.gov> Frederik Treue writes: > On Tue, 2013-09-17 at 08:15 -0500, Barry Smith wrote: >> Are you calling MatSetValuesStencil() for each single entry or once per row or block row? Row or block row will be faster than for each entry. > once per entry - this I should definitely improve. What's the logic > behind row or block rows being faster? The fewer calls to > MatSetValuesStencil, the better? It shares index translation and the call stack. >> >> Since you are using MatSetValuesStencil() I assume you are using a DA? > > I am, with one DOF per point. Are there better alternatives? No, what you're doing is fine. > PS. FYI: My stencil is essentially a standard 2D, 9-point box stencil, > but the values in all rows are different (due, among other things, to > the geometric tensor). That's normal. Is the geometry changing? > ########################################################## > # # > # WARNING!!! # > # # > # This code was compiled with a debugging option, # > # To get timing results run ./configure # > # using --with-debugging=no, the performance will # > # be generally two or three times faster. # > # # > ########################################################## We weren't joking. You should configure an optimized build using a different value of PETSC_ARCH so that you can easily switch back and forth between the debug and optimized versions. Everything will get faster, but especially insertion. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From bsmith at mcs.anl.gov Tue Sep 17 10:04:40 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 17 Sep 2013 10:04:40 -0500 Subject: [petsc-users] zero pattern of result of matmatmult In-Reply-To: <1379426242.2358.8.camel@frtr-laptop> References: <1378396034.2282.15.camel@frtr-laptop> <87ioyf6ys8.fsf@mcs.anl.gov> <1378461087.13721.10.camel@frtr-laptop> <87ppsm9in9.fsf@mcs.anl.gov> <1379422786.6379.5.camel@frtr-laptop> <1379426242.2358.8.camel@frtr-laptop> Message-ID: On Sep 17, 2013, at 8:57 AM, Frederik Treue wrote: > On Tue, 2013-09-17 at 08:15 -0500, Barry Smith wrote: >> When the linear system is fairly easy it is possible for the construction of the matrix to take a significant portion of the total time. Run with -log_summary and send the output, from this we may be able to make suggestions as to what further improvements can be made. > Attached. >> >> Is any part of the matrix constant? If so you can use MatStoreValues() and MatRetrieveValues() so that part does not have to be recomputed each time. > Not in terms of entries in the matrix. Mathematically, part of the > operator is constant, but all entries has changing parts added to them. If you matrix is A + B(x) where A is always the same but requires some non-trivial computation that currently gets repeated each time, you can do the following ONCE call MatSetValuesStencil() to put in the parts of A. Call MatAssemblyBegin/End(). Call MatStoreValues(). Then every time you need the matrix computed don't call MatZeroEntries() instead call MatRetreiveValues(), MatSetValuesStencil( ADD_VALUES) to put in the parts of B, MatAssemblyBegin/End(). If the "constant" part A requires almost no computation then you will not gain by this approach. Barry > >> >> Are you calling MatSetValuesStencil() for each single entry or once per row or block row? Row or block row will be faster than for each entry. > once per entry - this I should definitely improve. What's the logic > behind row or block rows being faster? The fewer calls to > MatSetValuesStencil, the better? > >> >> Since you are using MatSetValuesStencil() I assume you are using a DA? > > I am, with one DOF per point. Are there better alternatives? > > PS. FYI: My stencil is essentially a standard 2D, 9-point box stencil, > but the values in all rows are different (due, among other things, to > the geometric tensor). > > Thanks for the help... > > /Frederik > > > From s.prabhakaran at grs-sim.de Tue Sep 17 10:57:44 2013 From: s.prabhakaran at grs-sim.de (Suraj Prabhakaran) Date: Tue, 17 Sep 2013 17:57:44 +0200 Subject: [petsc-users] Petsc with open mpi 1.7 Message-ID: <0032BA89-0AC6-4430-B308-81E14DC35DA9@grs-sim.de> Hello, I am trying to compile petsc with openmpi 1.7 version. I can compile and install petsc successfully. but my application doesn't run since petsc is not able to find the libmpi_f77.so.1 and libmpi_f90.so.1. The problem is, in openmpi 1.7 version, these fotran libraries are completely replaced with libmpi_mpifh.so When I configure petsc with this mpi (I specifiy --with-mpi-dir option), I can see from the output that petsc only uses the libmpi_mpifh library rather than the other two. But still, when petsc is linked with my application, the binary ends up looking for libmpi_f90 and libmpi_f77, along with the correct libmpi_mpifh. To really be sure, I also used a hello world program and linked it with all libraries to see which one really asks for libmpi_f90 and libmpi_f77 and it was for sure libpetsc.so. Has anyone tried to compile and use petsc with openmpi 1.7 series? Is there anything special I have to do for this case? Best, Suraj -------------------------- Suraj Prabhakaran German Research School for Simulation Sciences GmbH Laboratory for Parallel Progreamming 52062 Aachen | Germany Tel +49 241 80 99743 Fax +49 241 80 92742 EMail s.prabhakaran at grs-sim.de Web www.grs-sim.de Members: Forschungszentrum J?lich GmbH | RWTH Aachen University Registered in the commercial register of the local court of D?ren (Amtsgericht D?ren) under registration number HRB 5268 Registered office: J?lich Executive board: Prof. Marek Behr Ph.D. | Prof. Dr. Sebastian M. Schmidt -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Sep 17 11:01:38 2013 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 17 Sep 2013 11:01:38 -0500 (CDT) Subject: [petsc-users] Petsc with open mpi 1.7 In-Reply-To: <0032BA89-0AC6-4430-B308-81E14DC35DA9@grs-sim.de> References: <0032BA89-0AC6-4430-B308-81E14DC35DA9@grs-sim.de> Message-ID: On Tue, 17 Sep 2013, Suraj Prabhakaran wrote: > Hello, > > I am trying to compile petsc with openmpi 1.7 version. I can compile and install petsc successfully. but my application doesn't run Do PETSc examples run? What do you get for 'make test'? If it doesn't work - send us the corresponding logs. Satish > since petsc is not able to find the libmpi_f77.so.1 and libmpi_f90.so.1. > The problem is, in openmpi 1.7 version, these fotran libraries are completely replaced with libmpi_mpifh.so > > When I configure petsc with this mpi (I specifiy --with-mpi-dir option), I can see from the output that petsc only uses the libmpi_mpifh library rather than the other two. But still, when petsc is linked with my application, the binary ends up looking for libmpi_f90 and libmpi_f77, along with the correct libmpi_mpifh. > > To really be sure, I also used a hello world program and linked it with all libraries to see which one really asks for libmpi_f90 and libmpi_f77 and it was for sure libpetsc.so. > > Has anyone tried to compile and use petsc with openmpi 1.7 series? > Is there anything special I have to do for this case? > > Best, > Suraj > > > -------------------------- > Suraj Prabhakaran > > German Research School for > Simulation Sciences GmbH > Laboratory for Parallel Progreamming > 52062 Aachen | Germany > > Tel +49 241 80 99743 > Fax +49 241 80 92742 > EMail s.prabhakaran at grs-sim.de > Web www.grs-sim.de > > Members: Forschungszentrum J?lich GmbH | RWTH Aachen University > Registered in the commercial register of the local court of D?ren (Amtsgericht D?ren) under registration number HRB 5268 > Registered office: J?lich > Executive board: Prof. Marek Behr Ph.D. | Prof. Dr. Sebastian M. Schmidt > > From bisheshkh at gmail.com Tue Sep 17 11:06:38 2013 From: bisheshkh at gmail.com (Bishesh Khanal) Date: Tue, 17 Sep 2013 18:06:38 +0200 Subject: [petsc-users] accessing individual elements of a global vector created from dmda Message-ID: Dear all, I have a vector say sol which is a vector created from a dmda with dof = 3. Now I need to provide an interface to the part of my code which does not use petsc but needs access to the values of sol. I was thinking of providing an interface such as: double solver::getSolutionAtPosition(int x, int y, int z, int field); which should return a value of the vector sol at the grid position (x,y,z) and dof = field. I know how to get the array and loop through all the values such as: DMDALocalInfo info; ierr = DMDAGetLocalInfo(da,&info);CHKERRQ(ierr); Field ***sol; ierr = DMDAVecGetArray(da,b,&rhs);CHKERRQ(ierr); for (PetscInt k = info.zs; k From balay at mcs.anl.gov Tue Sep 17 11:09:38 2013 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 17 Sep 2013 11:09:38 -0500 (CDT) Subject: [petsc-users] make all freezes In-Reply-To: References: Message-ID: This doesn't make any sense.. If the build is hanging - we'd like to know exactly where its hanging. [perhaps copy/paste] If the run is hanging - we'd like to see the stack trace [via gdb] of the location of the hang. xorg-devel should'nt make a difference [from what we know].. Satish On Tue, 17 Sep 2013, Analabha Roy wrote: > Update again: > > On a hunch, I went to a non-graphical tty where compilation worked! So I > installed all xorg-devel packages from repository and now compilation seems > to be working. > > Guess it just needed those... > > Thanks. > > > > On Tue, Sep 17, 2013 at 1:24 PM, Analabha Roy wrote: > > > Update: Purging openmpi and reinstalling did not work. "make all" in > > $PETSC_DIR still freezes. > > > > > > On Tue, Sep 17, 2013 at 11:17 AM, Analabha Roy wrote: > > > >> Hi, > >> > >> Sorry for the late reply > >> > >> > >> On Mon, Sep 16, 2013 at 7:03 PM, Matthew Knepley wrote: > >> > >>> On Mon, Sep 16, 2013 at 1:18 AM, Analabha Roy wrote: > >>> > >>>> Hi, > >>>> > >>>> I just got a new laptop and installed linux mint KDE. Details below: > >>>> > >>>> samsung NP300E5X-U01IN: > >>>> Kernel 3.8.0-26-generic #38-Ubuntu SMP > >>>> Four Intel Unknown 800MHz processors, 16760.20 total bogomips, 3733M RAM > >>>> System library 2.17.0 > >>>> > >>>> > >>>> I tried to compile petsc-3.4.2 with the following configuration > >>>> > >>>> ./configure --with-scalar-type=complex --with-c2html=0 > >>>> --with-debugging=no --with-fortran-kernels=1 --with-hdf5=yes > >>>> --with-hdf5-dir=/usr --enable-parallel > >>>> > >>>> > >>>> The configure log is posted here(dropbox) > >>>> > >>>> The configure script seems to run okay, but running "make > >>>> PETSC_DIR=/usr/local/src/petsc-3.4.2 PETSC_ARCH=arch-linux2-c-opt all" or > >>>> "make all" causes the compile to hang with no output. > >>>> > >>> > >>> This sounds like a CMake problem on your machine. Please try running > >>> > >>> make all-legacy > >>> > >>> although it seems like your Builder run succeeded. > >>> > >> > >> > >> I purged cmake using apt-get and reinstalled it from scratch. The problem > >> persists > >> > >> I also ran "make all-legacy". Same problem. No output. Just hangs. > >> > >> > >> > >>> > >>> Downgrading to a previous version of petsc does not seem to help. The > >>>> problem persists. Using default options "./configure > >>>> --with-scalar-type=complex" doesn;t seem to change anything either. > >>>> > >>>> Running ./config/builder.py works, but then "make test" hangs > >>>> similarly, as does running make on any example or any of my petsc sources. > >>>> > >>> > >>> This is different. It is likely that your MPI is not setup correctly to > >>> launch on this machine. Can you > >>> run any MPI job? > >>> > >>> > >> Yes. Both this hello world exampleand this calculation > >> of picompiles without errors and runs as expected. > >> > >> > >> Should I try purging openmpi and reinstalling? > >> > >> > >>> Thanks, > >>> > >>> Matt > >>> > >>> > >>>> Thanks in advance, > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> -- > >>>> --- > >>>> *Analabha Roy* > >>>> C.S.I.R Senior Research Associate > >>>> Saha Institute of Nuclear Physics > >>>> Section 1, Block AF > >>>> Bidhannagar, Calcutta 700064 > >>>> India > >>>> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com > >>>> *Webpage*: http://www.ph.utexas.edu/~daneel/ > >>>> > >>> > >>> > >>> > >>> -- > >>> What most experimenters take for granted before they begin their > >>> experiments is infinitely more interesting than any results to which their > >>> experiments lead. > >>> -- Norbert Wiener > >>> > >> > >> > >> > >> -- > >> --- > >> *Analabha Roy* > >> C.S.I.R Senior Research Associate > >> Saha Institute of Nuclear Physics > >> Section 1, Block AF > >> Bidhannagar, Calcutta 700064 > >> India > >> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com > >> *Webpage*: http://www.ph.utexas.edu/~daneel/ > >> > > > > > > > > -- > > --- > > *Analabha Roy* > > C.S.I.R Senior Research Associate > > Saha Institute of Nuclear Physics > > Section 1, Block AF > > Bidhannagar, Calcutta 700064 > > India > > *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com > > *Webpage*: http://www.ph.utexas.edu/~daneel/ > > > > > > From bsmith at mcs.anl.gov Tue Sep 17 11:15:02 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 17 Sep 2013 11:15:02 -0500 Subject: [petsc-users] accessing individual elements of a global vector created from dmda In-Reply-To: References: Message-ID: Bishesh, Do you need off process values? That is, might one process ask for a value that is not in the local part of the vector and belongs on a different process? If you know that the values accessed are always local then you can just use sol[k][j][i].vx the k,j,i are the global indices over the entire mesh. If any process might ask for any value on the entire mesh then you need to use DMDAGlobalToNaturalBegin/End() followed by VecScatterCreateToAll() and scatter all the values to all processes then you access them locally. Note this is not scalable. Barry Note that though with sol[k][j][i].vx it looks like you can access any value for k,j,i if you use a k, j, i that do not correspond to a value that belongs to this process you will get a memory error. On Sep 17, 2013, at 11:06 AM, Bishesh Khanal wrote: > Dear all, > I have a vector say sol which is a vector created from a dmda with dof = 3. > Now I need to provide an interface to the part of my code which does not use petsc but needs access to the values of sol. > > I was thinking of providing an interface such as: > double solver::getSolutionAtPosition(int x, int y, int z, int field); > which should return a value of the vector sol at the grid position (x,y,z) and dof = field. > > I know how to get the array and loop through all the values such as: > DMDALocalInfo info; > ierr = DMDAGetLocalInfo(da,&info);CHKERRQ(ierr); > > Field ***sol; > ierr = DMDAVecGetArray(da,b,&rhs);CHKERRQ(ierr); > > for (PetscInt k = info.zs; k for (PetscInt j = info.ys; j < info.ys+info.ym; ++j) { > for (PetscInt i = info.xs; i < info.xs+info.xm; ++i) { > //access sol[k][j][i].vx or sol[k][j][i].vy or sol[k][j][i].vz depending on field > > } > } > } > > But if I have a particular (x,y,z) based on global co-ordinate system, how do I get the value for the given field ? > > Thanks, > Bishesh From balay at mcs.anl.gov Tue Sep 17 11:25:07 2013 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 17 Sep 2013 11:25:07 -0500 (CDT) Subject: [petsc-users] Petsc with open mpi 1.7 In-Reply-To: References: <0032BA89-0AC6-4430-B308-81E14DC35DA9@grs-sim.de> Message-ID: I just attempted a build with openmpi 1.7 and 'make test' ran fine with it. Satish ------- ./configure --download-openmpi=http://www.open-mpi.org/software/ompi/v1.7/downloads/openmpi-1.7.2.tar.gz make all make test On Tue, 17 Sep 2013, Satish Balay wrote: > On Tue, 17 Sep 2013, Suraj Prabhakaran wrote: > > > Hello, > > > > I am trying to compile petsc with openmpi 1.7 version. I can compile and install petsc successfully. but my application doesn't run > > Do PETSc examples run? > > What do you get for 'make test'? > > If it doesn't work - send us the corresponding logs. > > Satish > > > since petsc is not able to find the libmpi_f77.so.1 and libmpi_f90.so.1. > > The problem is, in openmpi 1.7 version, these fotran libraries are completely replaced with libmpi_mpifh.so > > > > When I configure petsc with this mpi (I specifiy --with-mpi-dir option), I can see from the output that petsc only uses the libmpi_mpifh library rather than the other two. But still, when petsc is linked with my application, the binary ends up looking for libmpi_f90 and libmpi_f77, along with the correct libmpi_mpifh. > > > > To really be sure, I also used a hello world program and linked it with all libraries to see which one really asks for libmpi_f90 and libmpi_f77 and it was for sure libpetsc.so. > > > > Has anyone tried to compile and use petsc with openmpi 1.7 series? > > Is there anything special I have to do for this case? > > > > Best, > > Suraj > > > > > > -------------------------- > > Suraj Prabhakaran > > > > German Research School for > > Simulation Sciences GmbH > > Laboratory for Parallel Progreamming > > 52062 Aachen | Germany > > > > Tel +49 241 80 99743 > > Fax +49 241 80 92742 > > EMail s.prabhakaran at grs-sim.de > > Web www.grs-sim.de > > > > Members: Forschungszentrum J?lich GmbH | RWTH Aachen University > > Registered in the commercial register of the local court of D?ren (Amtsgericht D?ren) under registration number HRB 5268 > > Registered office: J?lich > > Executive board: Prof. Marek Behr Ph.D. | Prof. Dr. Sebastian M. Schmidt > > > > > From s.prabhakaran at grs-sim.de Tue Sep 17 11:36:00 2013 From: s.prabhakaran at grs-sim.de (Suraj Prabhakaran) Date: Tue, 17 Sep 2013 18:36:00 +0200 Subject: [petsc-users] Petsc with open mpi 1.7 In-Reply-To: References: <0032BA89-0AC6-4430-B308-81E14DC35DA9@grs-sim.de> Message-ID: <3BDE65F0-EA9D-4434-BE76-FF0B57142464@grs-sim.de> yes I also observed that. But still when I link petsc to a simple hello world example, I can see that the binary requires libmpi_f77 and libmpi_f90. Can you please compile the following snippet with mpicc and link it with petsc library, do a ldd on the binary and see if it asks for these 2 libraries or not? #include "stdio.h" int main() { printf("hello world\n"); return 0 } best, Suraj On Sep 17, 2013, at 6:25 PM, Satish Balay wrote: > I just attempted a build with openmpi 1.7 and 'make test' ran fine with it. > > Satish > > ------- > > ./configure --download-openmpi=http://www.open-mpi.org/software/ompi/v1.7/downloads/openmpi-1.7.2.tar.gz > make all > make test > > > On Tue, 17 Sep 2013, Satish Balay wrote: > >> On Tue, 17 Sep 2013, Suraj Prabhakaran wrote: >> >>> Hello, >>> >>> I am trying to compile petsc with openmpi 1.7 version. I can compile and install petsc successfully. but my application doesn't run >> >> Do PETSc examples run? >> >> What do you get for 'make test'? >> >> If it doesn't work - send us the corresponding logs. >> >> Satish >> >>> since petsc is not able to find the libmpi_f77.so.1 and libmpi_f90.so.1. >>> The problem is, in openmpi 1.7 version, these fotran libraries are completely replaced with libmpi_mpifh.so >>> >>> When I configure petsc with this mpi (I specifiy --with-mpi-dir option), I can see from the output that petsc only uses the libmpi_mpifh library rather than the other two. But still, when petsc is linked with my application, the binary ends up looking for libmpi_f90 and libmpi_f77, along with the correct libmpi_mpifh. >>> >>> To really be sure, I also used a hello world program and linked it with all libraries to see which one really asks for libmpi_f90 and libmpi_f77 and it was for sure libpetsc.so. >>> >>> Has anyone tried to compile and use petsc with openmpi 1.7 series? >>> Is there anything special I have to do for this case? >>> >>> Best, >>> Suraj >>> >>> >>> -------------------------- >>> Suraj Prabhakaran >>> >>> German Research School for >>> Simulation Sciences GmbH >>> Laboratory for Parallel Progreamming >>> 52062 Aachen | Germany >>> >>> Tel +49 241 80 99743 >>> Fax +49 241 80 92742 >>> EMail s.prabhakaran at grs-sim.de >>> Web www.grs-sim.de >>> >>> Members: Forschungszentrum J?lich GmbH | RWTH Aachen University >>> Registered in the commercial register of the local court of D?ren (Amtsgericht D?ren) under registration number HRB 5268 >>> Registered office: J?lich >>> Executive board: Prof. Marek Behr Ph.D. | Prof. Dr. Sebastian M. Schmidt >>> >>> >> -------------------------- Suraj Prabhakaran German Research School for Simulation Sciences GmbH Laboratory for Parallel Progreamming 52062 Aachen | Germany Tel +49 241 80 99743 Fax +49 241 80 92742 EMail s.prabhakaran at grs-sim.de Web www.grs-sim.de Members: Forschungszentrum J?lich GmbH | RWTH Aachen University Registered in the commercial register of the local court of D?ren (Amtsgericht D?ren) under registration number HRB 5268 Registered office: J?lich Executive board: Prof. Marek Behr Ph.D. | Prof. Dr. Sebastian M. Schmidt -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Sep 17 11:42:55 2013 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 17 Sep 2013 11:42:55 -0500 (CDT) Subject: [petsc-users] Petsc with open mpi 1.7 In-Reply-To: <3BDE65F0-EA9D-4434-BE76-FF0B57142464@grs-sim.de> References: <0032BA89-0AC6-4430-B308-81E14DC35DA9@grs-sim.de> <3BDE65F0-EA9D-4434-BE76-FF0B57142464@grs-sim.de> Message-ID: Are you using PETSc makefile format? Please compare your link command with the the link command used for a petsc example [say from "cd src/ksp/ksp/examples/tutorials && make ex2"] - and look for differences with your link command. We recommend using petsc makefile format to avoid such issues. Satish On Tue, 17 Sep 2013, Suraj Prabhakaran wrote: > yes I also observed that. But still when I link petsc to a simple hello world example, I can see that the binary requires libmpi_f77 and libmpi_f90. > Can you please compile the following snippet with mpicc and link it with petsc library, do a ldd on the binary and see if it asks for these 2 libraries or not? > > #include "stdio.h" > int main() > { > printf("hello world\n"); > return 0 > } > > best, > Suraj > > > > > On Sep 17, 2013, at 6:25 PM, Satish Balay wrote: > > > I just attempted a build with openmpi 1.7 and 'make test' ran fine with it. > > > > Satish > > > > ------- > > > > ./configure --download-openmpi=http://www.open-mpi.org/software/ompi/v1.7/downloads/openmpi-1.7.2.tar.gz > > make all > > make test > > > > > > On Tue, 17 Sep 2013, Satish Balay wrote: > > > >> On Tue, 17 Sep 2013, Suraj Prabhakaran wrote: > >> > >>> Hello, > >>> > >>> I am trying to compile petsc with openmpi 1.7 version. I can compile and install petsc successfully. but my application doesn't run > >> > >> Do PETSc examples run? > >> > >> What do you get for 'make test'? > >> > >> If it doesn't work - send us the corresponding logs. > >> > >> Satish > >> > >>> since petsc is not able to find the libmpi_f77.so.1 and libmpi_f90.so.1. > >>> The problem is, in openmpi 1.7 version, these fotran libraries are completely replaced with libmpi_mpifh.so > >>> > >>> When I configure petsc with this mpi (I specifiy --with-mpi-dir option), I can see from the output that petsc only uses the libmpi_mpifh library rather than the other two. But still, when petsc is linked with my application, the binary ends up looking for libmpi_f90 and libmpi_f77, along with the correct libmpi_mpifh. > >>> > >>> To really be sure, I also used a hello world program and linked it with all libraries to see which one really asks for libmpi_f90 and libmpi_f77 and it was for sure libpetsc.so. > >>> > >>> Has anyone tried to compile and use petsc with openmpi 1.7 series? > >>> Is there anything special I have to do for this case? > >>> > >>> Best, > >>> Suraj > >>> > >>> > >>> -------------------------- > >>> Suraj Prabhakaran > >>> > >>> German Research School for > >>> Simulation Sciences GmbH > >>> Laboratory for Parallel Progreamming > >>> 52062 Aachen | Germany > >>> > >>> Tel +49 241 80 99743 > >>> Fax +49 241 80 92742 > >>> EMail s.prabhakaran at grs-sim.de > >>> Web www.grs-sim.de > >>> > >>> Members: Forschungszentrum J?lich GmbH | RWTH Aachen University > >>> Registered in the commercial register of the local court of D?ren (Amtsgericht D?ren) under registration number HRB 5268 > >>> Registered office: J?lich > >>> Executive board: Prof. Marek Behr Ph.D. | Prof. Dr. Sebastian M. Schmidt > >>> > >>> > >> > > -------------------------- > Suraj Prabhakaran > > German Research School for > Simulation Sciences GmbH > Laboratory for Parallel Progreamming > 52062 Aachen | Germany > > Tel +49 241 80 99743 > Fax +49 241 80 92742 > EMail s.prabhakaran at grs-sim.de > Web www.grs-sim.de > > Members: Forschungszentrum J?lich GmbH | RWTH Aachen University > Registered in the commercial register of the local court of D?ren (Amtsgericht D?ren) under registration number HRB 5268 > Registered office: J?lich > Executive board: Prof. Marek Behr Ph.D. | Prof. Dr. Sebastian M. Schmidt > > From danyang.su at gmail.com Tue Sep 17 12:29:21 2013 From: danyang.su at gmail.com (Danyang Su) Date: Tue, 17 Sep 2013 10:29:21 -0700 Subject: [petsc-users] PETSc-dev and OpenMP Message-ID: <52389171.1090600@gmail.com> Hi All, I have a parallel Fortran project utilizing OpenMP parallelism. Before porting to PETSc utilizing MPICH, I want to know if it is possible to use PETSc solver with OpenMP. There is some information for petsc-dev openmp benchmarking as can be found here, http://lists.mcs.anl.gov/pipermail/petsc-dev/2012-March/007817.html. If this is possible, I would like to have a OpenMP version of PETSc solver first, otherwise, the project will be ported to PETSc utilizing MPI. Could anybody give me some advice on this? Thanks and regards, Danyang From choi240 at purdue.edu Tue Sep 17 13:04:13 2013 From: choi240 at purdue.edu (Joon Hee Choi) Date: Tue, 17 Sep 2013 14:04:13 -0400 (EDT) Subject: [petsc-users] Inverse matrix of MATMPIDENSE matrix In-Reply-To: <0CF388B0-348C-440F-ABBD-A53397A45517@mcs.anl.gov> Message-ID: <1977967275.115859.1379441053100.JavaMail.root@mailhub028.itcs.purdue.edu> Thank you for your reply. However, I followed the manual using MatLUFactor() and MatMatSolve(). However, MatLUFactor() did not support MATMPIDENSE. If my code is wrong, then please let me know. My code is following: MatGetOrdering(invM2T, MATORDERINGNATURAL, &isr, &isc); MatFactorInfoInitialize(&info); MatLUFactor(invM2T, isr, isc, &info); MatMatSolve(invM2T, M1T, *M3T); If not so, then please let me know which function I have to use. Thank you. Sincerely, Joon ----- Original Message ----- From: "Barry Smith" To: "Joon Hee Choi" Cc: petsc-users at mcs.anl.gov Sent: Tuesday, September 17, 2013 7:59:42 AM Subject: Re: [petsc-users] Inverse matrix of MATMPIDENSE matrix http://www.mcs.anl.gov/petsc/documentation/faq.html#invertmatrix On Sep 17, 2013, at 1:07 AM, Joon Hee Choi wrote: > Hello, > > I want to get the inverse matrix of 10 x 10 MATMPIDENSE matrix. I know I cannot use MatLUFactor() for MATMPIDENSE. Is there any other method? Could you let me know the fastest way to get the inverse? Also, I don't want to rebuild or configure something for the inverse because I use petsc in my school server and I don't have permission to configure. > > Thanks, > Joon From bsmith at mcs.anl.gov Tue Sep 17 13:10:15 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 17 Sep 2013 13:10:15 -0500 Subject: [petsc-users] Inverse matrix of MATMPIDENSE matrix In-Reply-To: <1977967275.115859.1379441053100.JavaMail.root@mailhub028.itcs.purdue.edu> References: <1977967275.115859.1379441053100.JavaMail.root@mailhub028.itcs.purdue.edu> Message-ID: <1566877B-4194-4DB5-9A06-1418E9724688@mcs.anl.gov> For parallel dense you need to use MatType MATELEMENTAL and configure PETSc with ../download-elemental Note that if N is 10 then it is insane to do the inverse in parallel. Just use SeqDense. If you want it on two processes just do it on one and send there result to the other or do it on both. Unless the matrix is at least 100s by 100s maybe more I don't see doing it in parallel as saving any time, rather it will be slower Barry On Sep 17, 2013, at 1:04 PM, Joon Hee Choi wrote: > Thank you for your reply. > However, I followed the manual using MatLUFactor() and MatMatSolve(). However, MatLUFactor() did not support MATMPIDENSE. If my code is wrong, then please let me know. My code is following: > > MatGetOrdering(invM2T, MATORDERINGNATURAL, &isr, &isc); > MatFactorInfoInitialize(&info); > MatLUFactor(invM2T, isr, isc, &info); > MatMatSolve(invM2T, M1T, *M3T); > > If not so, then please let me know which function I have to use. Thank you. > > Sincerely, > Joon > > ----- Original Message ----- > From: "Barry Smith" > To: "Joon Hee Choi" > Cc: petsc-users at mcs.anl.gov > Sent: Tuesday, September 17, 2013 7:59:42 AM > Subject: Re: [petsc-users] Inverse matrix of MATMPIDENSE matrix > > http://www.mcs.anl.gov/petsc/documentation/faq.html#invertmatrix > > > On Sep 17, 2013, at 1:07 AM, Joon Hee Choi wrote: > >> Hello, >> >> I want to get the inverse matrix of 10 x 10 MATMPIDENSE matrix. I know I cannot use MatLUFactor() for MATMPIDENSE. Is there any other method? Could you let me know the fastest way to get the inverse? Also, I don't want to rebuild or configure something for the inverse because I use petsc in my school server and I don't have permission to configure. >> >> Thanks, >> Joon > From bsmith at mcs.anl.gov Tue Sep 17 13:21:17 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 17 Sep 2013 13:21:17 -0500 Subject: [petsc-users] PETSc-dev and OpenMP In-Reply-To: <52389171.1090600@gmail.com> References: <52389171.1090600@gmail.com> Message-ID: <7AB30D0B-6ADA-427A-9CDE-A8D6B84E04E8@mcs.anl.gov> You can utilize the "OpenMP version of PETSc" with your OpenMP program but not all the solvers are OpenMP parallel currently. Use petsc-dev and configure PETSc with ./configure --with-mpi=0 --with-openmp see http://www.mcs.anl.gov/petsc/developers/index.html Barry On Sep 17, 2013, at 12:29 PM, Danyang Su wrote: > Hi All, > > I have a parallel Fortran project utilizing OpenMP parallelism. Before porting to PETSc utilizing MPICH, I want to know if it is possible to use PETSc solver with OpenMP. There is some information for petsc-dev openmp benchmarking as can be found here, http://lists.mcs.anl.gov/pipermail/petsc-dev/2012-March/007817.html. If this is possible, I would like to have a OpenMP version of PETSc solver first, otherwise, the project will be ported to PETSc utilizing MPI. Could anybody give me some advice on this? > > Thanks and regards, > > Danyang From choi240 at purdue.edu Tue Sep 17 13:24:38 2013 From: choi240 at purdue.edu (Joon Hee Choi) Date: Tue, 17 Sep 2013 14:24:38 -0400 (EDT) Subject: [petsc-users] Inverse matrix of MATMPIDENSE matrix In-Reply-To: <1566877B-4194-4DB5-9A06-1418E9724688@mcs.anl.gov> Message-ID: <813638769.115946.1379442278176.JavaMail.root@mailhub028.itcs.purdue.edu> Thank you for your reply. The 10x10 matrix is the result of product between large matrices with 10 different processors. So I think I need to gather entries into a processor and create SeqDense matrix. Could you let me know what to do for gathering entries saved on several processors? Sincerely, Joon ----- Original Message ----- From: "Barry Smith" To: "Joon Hee Choi" Cc: petsc-users at mcs.anl.gov Sent: Tuesday, September 17, 2013 2:10:15 PM Subject: Re: [petsc-users] Inverse matrix of MATMPIDENSE matrix For parallel dense you need to use MatType MATELEMENTAL and configure PETSc with ../download-elemental Note that if N is 10 then it is insane to do the inverse in parallel. Just use SeqDense. If you want it on two processes just do it on one and send there result to the other or do it on both. Unless the matrix is at least 100s by 100s maybe more I don't see doing it in parallel as saving any time, rather it will be slower Barry On Sep 17, 2013, at 1:04 PM, Joon Hee Choi wrote: > Thank you for your reply. > However, I followed the manual using MatLUFactor() and MatMatSolve(). However, MatLUFactor() did not support MATMPIDENSE. If my code is wrong, then please let me know. My code is following: > > MatGetOrdering(invM2T, MATORDERINGNATURAL, &isr, &isc); > MatFactorInfoInitialize(&info); > MatLUFactor(invM2T, isr, isc, &info); > MatMatSolve(invM2T, M1T, *M3T); > > If not so, then please let me know which function I have to use. Thank you. > > Sincerely, > Joon > > ----- Original Message ----- > From: "Barry Smith" > To: "Joon Hee Choi" > Cc: petsc-users at mcs.anl.gov > Sent: Tuesday, September 17, 2013 7:59:42 AM > Subject: Re: [petsc-users] Inverse matrix of MATMPIDENSE matrix > > http://www.mcs.anl.gov/petsc/documentation/faq.html#invertmatrix > > > On Sep 17, 2013, at 1:07 AM, Joon Hee Choi wrote: > >> Hello, >> >> I want to get the inverse matrix of 10 x 10 MATMPIDENSE matrix. I know I cannot use MatLUFactor() for MATMPIDENSE. Is there any other method? Could you let me know the fastest way to get the inverse? Also, I don't want to rebuild or configure something for the inverse because I use petsc in my school server and I don't have permission to configure. >> >> Thanks, >> Joon > From bsmith at mcs.anl.gov Tue Sep 17 13:28:53 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 17 Sep 2013 13:28:53 -0500 Subject: [petsc-users] Inverse matrix of MATMPIDENSE matrix In-Reply-To: <813638769.115946.1379442278176.JavaMail.root@mailhub028.itcs.purdue.edu> References: <813638769.115946.1379442278176.JavaMail.root@mailhub028.itcs.purdue.edu> Message-ID: On Sep 17, 2013, at 1:24 PM, Joon Hee Choi wrote: > Thank you for your reply. > The 10x10 matrix is the result of product between large matrices with 10 different processors. Yupe > So I think I need to gather entries into a processor and create SeqDense matrix. Since these are just dense arrays use MatDenseGetArray() /MatDenseRestoreArray() (previously called MatGetArray()/MatRestoreArray() in older versions of PETSc) and then MPI_Gatherv() to get them into a single array on process 0 then MatCreateSeqDense() to create the sequential matrix on process 0. Barry > Could you let me know what to do for gathering entries saved on several processors? > > Sincerely, > Joon > > ----- Original Message ----- > From: "Barry Smith" > To: "Joon Hee Choi" > Cc: petsc-users at mcs.anl.gov > Sent: Tuesday, September 17, 2013 2:10:15 PM > Subject: Re: [petsc-users] Inverse matrix of MATMPIDENSE matrix > > > For parallel dense you need to use MatType MATELEMENTAL and configure PETSc with ../download-elemental > > Note that if N is 10 then it is insane to do the inverse in parallel. Just use SeqDense. If you want it on two processes just do it on one and send there result to the other or do it on both. > > Unless the matrix is at least 100s by 100s maybe more I don't see doing it in parallel as saving any time, rather it will be slower > > > Barry > > > > On Sep 17, 2013, at 1:04 PM, Joon Hee Choi wrote: > >> Thank you for your reply. >> However, I followed the manual using MatLUFactor() and MatMatSolve(). However, MatLUFactor() did not support MATMPIDENSE. If my code is wrong, then please let me know. My code is following: >> >> MatGetOrdering(invM2T, MATORDERINGNATURAL, &isr, &isc); >> MatFactorInfoInitialize(&info); >> MatLUFactor(invM2T, isr, isc, &info); >> MatMatSolve(invM2T, M1T, *M3T); >> >> If not so, then please let me know which function I have to use. Thank you. >> >> Sincerely, >> Joon >> >> ----- Original Message ----- >> From: "Barry Smith" >> To: "Joon Hee Choi" >> Cc: petsc-users at mcs.anl.gov >> Sent: Tuesday, September 17, 2013 7:59:42 AM >> Subject: Re: [petsc-users] Inverse matrix of MATMPIDENSE matrix >> >> http://www.mcs.anl.gov/petsc/documentation/faq.html#invertmatrix >> >> >> On Sep 17, 2013, at 1:07 AM, Joon Hee Choi wrote: >> >>> Hello, >>> >>> I want to get the inverse matrix of 10 x 10 MATMPIDENSE matrix. I know I cannot use MatLUFactor() for MATMPIDENSE. Is there any other method? Could you let me know the fastest way to get the inverse? Also, I don't want to rebuild or configure something for the inverse because I use petsc in my school server and I don't have permission to configure. >>> >>> Thanks, >>> Joon >> > From danyang.su at gmail.com Tue Sep 17 14:31:29 2013 From: danyang.su at gmail.com (Danyang Su) Date: Tue, 17 Sep 2013 12:31:29 -0700 Subject: [petsc-users] PETSc-dev and OpenMP In-Reply-To: <7AB30D0B-6ADA-427A-9CDE-A8D6B84E04E8@mcs.anl.gov> References: <52389171.1090600@gmail.com> <7AB30D0B-6ADA-427A-9CDE-A8D6B84E04E8@mcs.anl.gov> Message-ID: <5238AE11.6050400@gmail.com> Hi Barry, Thanks so much. Which solvers are current OpenMP parallel currently? Can I use the external package (e.g., SuperLU OpenMP version) with OpenMP through PETSc? Is there any document or examples available for the usage of PETSc with OpenMP? Thanks again, Danyang On 17/09/2013 11:21 AM, Barry Smith wrote: > You can utilize the "OpenMP version of PETSc" with your OpenMP program but not all the solvers are OpenMP parallel currently. > > Use petsc-dev and configure PETSc with ./configure --with-mpi=0 --with-openmp see http://www.mcs.anl.gov/petsc/developers/index.html > > Barry > > > On Sep 17, 2013, at 12:29 PM, Danyang Su wrote: > >> Hi All, >> >> I have a parallel Fortran project utilizing OpenMP parallelism. Before porting to PETSc utilizing MPICH, I want to know if it is possible to use PETSc solver with OpenMP. There is some information for petsc-dev openmp benchmarking as can be found here, http://lists.mcs.anl.gov/pipermail/petsc-dev/2012-March/007817.html. If this is possible, I would like to have a OpenMP version of PETSc solver first, otherwise, the project will be ported to PETSc utilizing MPI. Could anybody give me some advice on this? >> >> Thanks and regards, >> >> Danyang From pengxwang at hotmail.com Tue Sep 17 20:35:41 2013 From: pengxwang at hotmail.com (Roc Wang) Date: Tue, 17 Sep 2013 20:35:41 -0500 Subject: [petsc-users] reuse an assembled matrix when a linear problem is solved with KSP and DM Message-ID: Hello, I am trying to solve a linear problem by using KSPSolve and DM. In the example /src/ksp/ksp/examples/tutorials/ex45.c, the Matrix and the RHS vector are computed by calling KSPSetComputeRHS and KSPSetComputeOperators in which functions ComputeRHS() and ComputeMatrix() are called and the matrix is assembled in ComputeMatrix(). Since the linear system is solved in an iteration, it will save time to reuse the matrix assembled in ComputeMatrix(). That is the ComputeMatrix() is called only once, but ComputeRHS() is called in each iteration. Can anyone tell me what should I do to keep the assembled matrix? I am thinking not to use KSPSetComputeOperators for assembling the matrix, but use DMCreateMatrix() alternatively. Am I right? Any hits would be appreciated very much. Thanks alot. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Sep 17 20:45:03 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 17 Sep 2013 20:45:03 -0500 Subject: [petsc-users] reuse an assembled matrix when a linear problem is solved with KSP and DM In-Reply-To: References: Message-ID: On Sep 17, 2013, at 8:35 PM, Roc Wang wrote: > Hello, > > I am trying to solve a linear problem by using KSPSolve and DM. In the example /src/ksp/ksp/examples/tutorials/ex45.c, the Matrix and the RHS vector are computed by calling KSPSetComputeRHS and KSPSetComputeOperators in which functions ComputeRHS() and ComputeMatrix() are called and the matrix is assembled in ComputeMatrix(). > > Since the linear system is solved in an iteration, it will save time to reuse the matrix assembled in ComputeMatrix(). That is the ComputeMatrix() is called only once, but ComputeRHS() is called in each iteration. What do you need mean by an "iteration" here? Do you mean one iteration of a KSP solvers, of which several are needed to produce the approximate solution or something else? The ComputeRHS() function is only called ONCE for each KSPSolve (even though KSP may take several iterations to solve the linear system). The ComputeMatrix() function is actually only called ONCE period, unless you call KSPSetComputeOperators() again, even if you call KSPSolve() several times it uses the same matrix until you call KSPSetComputerOperators() again. > Can anyone tell me what should I do to keep the assembled matrix? > > I am thinking not to use KSPSetComputeOperators for assembling the matrix, but use DMCreateMatrix() alternatively. Am I right? That is a completely orthogonal issue; DMCreateMatrix() only gives you an empty sparse matrix with the right nonzero structure for the mesh associated with DM. You still need to compute its entries yourself, possibly in a ComputeMatrix function you write. Barry > > Any hits would be appreciated very much. Thanks alot. > > From jbermeol at purdue.edu Tue Sep 17 21:11:20 2013 From: jbermeol at purdue.edu (Jose David Bermeol) Date: Tue, 17 Sep 2013 22:11:20 -0400 (EDT) Subject: [petsc-users] Join matrices Message-ID: <2082945003.155152.1379470280123.JavaMail.root@mailhub059.itcs.purdue.edu> Hi, right now I'm coding a solver for MATMPIAIJ matrices, as you know this kind of matrix has a diagonal matrix A of size n x n and a matrix B with the off-diagonal matrices of size n x (N - n). The problem is that my solver for each MPI process receive 3 arrays that represents the rows in the MPI process(aij format), that means the complete n x N matrix. What I did was a join of the A and B matrix, however this implies almost to duplicate everything in memory. I don't know if you know a better strategy for this(less memory overhead). Thanks Here is my code if you see something that can be improve: /* * MATMPIAIJ matrices are divided in two matrices, * diagonal matriA) and off-diagonal matriB), * both in aij format, as cpardiso only recives one matrix, * this method will join A and B matrix in one, return aij * representation and number of non-zero elements. * * This method will copy(duplicate) A and B elements in a new matrix. * * Input: * - Mat A: MATMPIAIJ matrix * - int shift: matrix index.(cpardiso requires fortan indexing) * - 0 for c representation * - 1 for fortran representation * - MatReuse reuse: * - MAT_INITIAL_MATRIX: Create a new aij representation * - MAT_REUSE_MATRIX: Reuse all aij representation and just change values * Output: * - int *nnz: Number of nonzero-elements. * - int **r pointer to i index * - int **c pointer to j elements * - PetscScalar **v: Non-zero elements */ #undef __FUNCT__ #define __FUNCT__ "MatJoin_CPARDISO" PetscErrorCode MatJoin_CPARDISO(Mat A, int shift, MatReuse reuse, int *nnz, int **r, int **c, PetscScalar **v){ const PetscInt *ai, *aj, *bi, *bj, *garray, m=A->rmap->n, *ajj, *bjj; PetscErrorCode ierr; PetscInt rstart, nz, i, jj, countA, countB, idxA, idxB; PetscInt *row,*col; const PetscScalar *av, *bv,*v1,*v2; PetscScalar *val; Mat_MPIAIJ *mat = (Mat_MPIAIJ*)A->data; Mat_SeqAIJ *aa = (Mat_SeqAIJ*)(mat->A)->data; Mat_SeqAIJ *bb = (Mat_SeqAIJ*)(mat->B)->data; PetscFunctionBegin; ai=aa->i; aj=aa->j; bi=bb->i; bj=bb->j; rstart= A->rmap->rstart; av=aa->a; bv=bb->a; garray = mat->garray; if (reuse == MAT_INITIAL_MATRIX) { nz = aa->nz + bb->nz; *nnz = nz; ierr = PetscMalloc(((nz + m + 1)*sizeof(PetscInt) + nz*sizeof(PetscScalar)), &row);CHKERRQ(ierr); col = row + m + 1; val = (PetscScalar*)(col + nz); *r = row; *c = col; *v = val; } else { row = *r; col = *c; val = *v; } jj = 0; for (i=0; i garray[bjj[idxB]]){ if(reuse == MAT_INITIAL_MATRIX){ col[jj] = garray[bjj[idxB]] + shift; } val[jj] = v2[idxB]; idxB++; jj++; } for (idxA = 0; idxA < countA; idxA++) { if(reuse == MAT_INITIAL_MATRIX){ col[jj] = rstart + ajj[idxA] + shift; } val[jj] = v1[idxA]; jj++; } for (; idxB < countB; idxB++) { if(reuse == MAT_INITIAL_MATRIX){ col[jj] = garray[bjj[idxB]] + shift; } val[jj] = v2[idxB]; jj++; } } } if(reuse == MAT_INITIAL_MATRIX){ row[m] = jj + shift; } PetscFunctionReturn(0); } From pengxwang at hotmail.com Tue Sep 17 22:01:58 2013 From: pengxwang at hotmail.com (Roc Wang) Date: Tue, 17 Sep 2013 22:01:58 -0500 Subject: [petsc-users] reuse an assembled matrix when a linear problem is solved with KSP and DM In-Reply-To: References: , Message-ID: > Subject: Re: [petsc-users] reuse an assembled matrix when a linear problem is solved with KSP and DM > From: bsmith at mcs.anl.gov > Date: Tue, 17 Sep 2013 20:45:03 -0500 > CC: petsc-users at mcs.anl.gov > To: pengxwang at hotmail.com > > > On Sep 17, 2013, at 8:35 PM, Roc Wang wrote: > > > Hello, > > > > I am trying to solve a linear problem by using KSPSolve and DM. In the example /src/ksp/ksp/examples/tutorials/ex45.c, the Matrix and the RHS vector are computed by calling KSPSetComputeRHS and KSPSetComputeOperators in which functions ComputeRHS() and ComputeMatrix() are called and the matrix is assembled in ComputeMatrix(). > > > > Since the linear system is solved in an iteration, it will save time to reuse the matrix assembled in ComputeMatrix(). That is the ComputeMatrix() is called only once, but ComputeRHS() is called in each iteration. > > What do you need mean by an "iteration" here? Do you mean one iteration of a KSP solvers, of which several are needed to produce the approximate solution or something else? > sorry for the misunderstanding. By iteration, I mean the linear system needs to be solved many times with different RHS, the RHS is changed in each time the linear sytem is solved. It's not the iteration that the KSP takes. > The ComputeRHS() function is only called ONCE for each KSPSolve (even though KSP may take several iterations to solve the linear system). > So, I need to call KSPSetComputeRHS() each time I want to change the RHS, right? > The ComputeMatrix() function is actually only called ONCE period, unless you call KSPSetComputeOperators() again, even if you call KSPSolve() several times it uses the same matrix until you call KSPSetComputerOperators() again. > Thus, I just need to call KSPSetComputerOperators() once when the matrix is assembled the first time to solve the linear system, after that only KSPSetComputeRHS(), when I want to change RHS vector. > > Can anyone tell me what should I do to keep the assembled matrix? > > > > I am thinking not to use KSPSetComputeOperators for assembling the matrix, but use DMCreateMatrix() alternatively. Am I right? > > That is a completely orthogonal issue; DMCreateMatrix() only gives you an empty sparse matrix with the right nonzero structure for the mesh associated with DM. You still need to compute its entries yourself, possibly in a ComputeMatrix function you write. Yes, I plan to compute the entries after calling DMCreateMatrix(), just like calling MatSetValuesStencil() in for loop and then MatAssemblyBegin(), MatAssemblyBegin(). > > Barry > > > > > Any hits would be appreciated very much. Thanks alot. > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Sep 17 22:06:41 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 17 Sep 2013 22:06:41 -0500 Subject: [petsc-users] reuse an assembled matrix when a linear problem is solved with KSP and DM In-Reply-To: References: , Message-ID: On Sep 17, 2013, at 10:01 PM, Roc Wang wrote: > > > > Subject: Re: [petsc-users] reuse an assembled matrix when a linear problem is solved with KSP and DM > > From: bsmith at mcs.anl.gov > > Date: Tue, 17 Sep 2013 20:45:03 -0500 > > CC: petsc-users at mcs.anl.gov > > To: pengxwang at hotmail.com > > > > > > On Sep 17, 2013, at 8:35 PM, Roc Wang wrote: > > > > > Hello, > > > > > > I am trying to solve a linear problem by using KSPSolve and DM. In the example /src/ksp/ksp/examples/tutorials/ex45.c, the Matrix and the RHS vector are computed by calling KSPSetComputeRHS and KSPSetComputeOperators in which functions ComputeRHS() and ComputeMatrix() are called and the matrix is assembled in ComputeMatrix(). > > > > > > Since the linear system is solved in an iteration, it will save time to reuse the matrix assembled in ComputeMatrix(). That is the ComputeMatrix() is called only once, but ComputeRHS() is called in each iteration. > > > > What do you need mean by an "iteration" here? Do you mean one iteration of a KSP solvers, of which several are needed to produce the approximate solution or something else? > > > sorry for the misunderstanding. By iteration, I mean the linear system needs to be solved many times with different RHS, the RHS is changed in each time the linear sytem is solved. It's not the iteration that the KSP takes. > > The ComputeRHS() function is only called ONCE for each KSPSolve (even though KSP may take several iterations to solve the linear system). > > > So, I need to call KSPSetComputeRHS() each time I want to change the RHS, right? > > The ComputeMatrix() function is actually only called ONCE period, unless you call KSPSetComputeOperators() again, even if you call KSPSolve() several times it uses the same matrix until you call KSPSetComputerOperators() again. > > > Thus, I just need to call KSPSetComputerOperators() once when the matrix is assembled the first time to solve the linear system, after that only KSPSetComputeRHS(), when I want to change RHS vector. No. Each new KSPSolve() will automatically call the ComputeRHS function again. You do NOT need to call the KSPSetComputeRHS() function more then once. Barry Yes the KSPSetComputeRHS() and KSPSetComputeOperators() functions behave slightly differently. > > > Can anyone tell me what should I do to keep the assembled matrix? > > > > > > I am thinking not to use KSPSetComputeOperators for assembling the matrix, but use DMCreateMatrix() alternatively. Am I right? > > > > That is a completely orthogonal issue; DMCreateMatrix() only gives you an empty sparse matrix with the right nonzero structure for the mesh associated with DM. You still need to compute its entries yourself, possibly in a ComputeMatrix function you write. > Yes, I plan to compute the entries after calling DMCreateMatrix(), just like calling MatSetValuesStencil() in for loop and then MatAssemblyBegin(), MatAssemblyBegin(). > > > > > > Barry > > > > > > > > Any hits would be appreciated very much. Thanks alot. > > > > > > > > From hariseldon99 at gmail.com Tue Sep 17 22:58:19 2013 From: hariseldon99 at gmail.com (Analabha Roy) Date: Wed, 18 Sep 2013 09:28:19 +0530 Subject: [petsc-users] make all freezes In-Reply-To: References: Message-ID: On Sep 17, 2013 9:39 PM, "Satish Balay" wrote: > > This doesn't make any sense.. > > If the build is hanging - we'd like to know exactly where its hanging. > [perhaps copy/paste] > It didn't dump any output, either to stdout or Stderr. There was nothing to copy/paste. > If the run is hanging - we'd like to see the stack trace [via gdb] of the > location of the hang. > > xorg-devel should'nt make a difference [from what we know].. > Doesn't it need xorg headers for rendering graphic output via petscdraw? Maybe it was one of the dependencies of xorg devel that did the trick. I'll try to reproduce the problem and see. > Satish > > On Tue, 17 Sep 2013, Analabha Roy wrote: > > > Update again: > > > > On a hunch, I went to a non-graphical tty where compilation worked! So I > > installed all xorg-devel packages from repository and now compilation seems > > to be working. > > > > Guess it just needed those... > > > > Thanks. > > > > > > > > On Tue, Sep 17, 2013 at 1:24 PM, Analabha Roy wrote: > > > > > Update: Purging openmpi and reinstalling did not work. "make all" in > > > $PETSC_DIR still freezes. > > > > > > > > > On Tue, Sep 17, 2013 at 11:17 AM, Analabha Roy wrote: > > > > > >> Hi, > > >> > > >> Sorry for the late reply > > >> > > >> > > >> On Mon, Sep 16, 2013 at 7:03 PM, Matthew Knepley wrote: > > >> > > >>> On Mon, Sep 16, 2013 at 1:18 AM, Analabha Roy < hariseldon99 at gmail.com>wrote: > > >>> > > >>>> Hi, > > >>>> > > >>>> I just got a new laptop and installed linux mint KDE. Details below: > > >>>> > > >>>> samsung NP300E5X-U01IN: > > >>>> Kernel 3.8.0-26-generic #38-Ubuntu SMP > > >>>> Four Intel Unknown 800MHz processors, 16760.20 total bogomips, 3733M RAM > > >>>> System library 2.17.0 > > >>>> > > >>>> > > >>>> I tried to compile petsc-3.4.2 with the following configuration > > >>>> > > >>>> ./configure --with-scalar-type=complex --with-c2html=0 > > >>>> --with-debugging=no --with-fortran-kernels=1 --with-hdf5=yes > > >>>> --with-hdf5-dir=/usr --enable-parallel > > >>>> > > >>>> > > >>>> The configure log is posted here< https://dl.dropboxusercontent.com/u/2769707/configure.log>(dropbox) > > >>>> > > >>>> The configure script seems to run okay, but running "make > > >>>> PETSC_DIR=/usr/local/src/petsc-3.4.2 PETSC_ARCH=arch-linux2-c-opt all" or > > >>>> "make all" causes the compile to hang with no output. > > >>>> > > >>> > > >>> This sounds like a CMake problem on your machine. Please try running > > >>> > > >>> make all-legacy > > >>> > > >>> although it seems like your Builder run succeeded. > > >>> > > >> > > >> > > >> I purged cmake using apt-get and reinstalled it from scratch. The problem > > >> persists > > >> > > >> I also ran "make all-legacy". Same problem. No output. Just hangs. > > >> > > >> > > >> > > >>> > > >>> Downgrading to a previous version of petsc does not seem to help. The > > >>>> problem persists. Using default options "./configure > > >>>> --with-scalar-type=complex" doesn;t seem to change anything either. > > >>>> > > >>>> Running ./config/builder.py works, but then "make test" hangs > > >>>> similarly, as does running make on any example or any of my petsc sources. > > >>>> > > >>> > > >>> This is different. It is likely that your MPI is not setup correctly to > > >>> launch on this machine. Can you > > >>> run any MPI job? > > >>> > > >>> > > >> Yes. Both this hello world example< http://www.dartmouth.edu/~rc/classes/intro_mpi/hello_world_ex.html>and this calculation > > >> of pi< http://www.mcs.anl.gov/research/projects/mpi/usingmpi/examples/simplempi/cpi_c.htm>compiles without errors and runs as expected. > > >> > > >> > > >> Should I try purging openmpi and reinstalling? > > >> > > >> > > >>> Thanks, > > >>> > > >>> Matt > > >>> > > >>> > > >>>> Thanks in advance, > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> -- > > >>>> --- > > >>>> *Analabha Roy* > > >>>> C.S.I.R Senior Research Associate< http://csirhrdg.res.in/poolsra.htm> > > >>>> Saha Institute of Nuclear Physics > > >>>> Section 1, Block AF > > >>>> Bidhannagar, Calcutta 700064 > > >>>> India > > >>>> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com > > >>>> *Webpage*: http://www.ph.utexas.edu/~daneel/ > > >>>> > > >>> > > >>> > > >>> > > >>> -- > > >>> What most experimenters take for granted before they begin their > > >>> experiments is infinitely more interesting than any results to which their > > >>> experiments lead. > > >>> -- Norbert Wiener > > >>> > > >> > > >> > > >> > > >> -- > > >> --- > > >> *Analabha Roy* > > >> C.S.I.R Senior Research Associate< http://csirhrdg.res.in/poolsra.htm> > > >> Saha Institute of Nuclear Physics > > >> Section 1, Block AF > > >> Bidhannagar, Calcutta 700064 > > >> India > > >> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com > > >> *Webpage*: http://www.ph.utexas.edu/~daneel/ > > >> > > > > > > > > > > > > -- > > > --- > > > *Analabha Roy* > > > C.S.I.R Senior Research Associate< http://csirhrdg.res.in/poolsra.htm> > > > Saha Institute of Nuclear Physics > > > Section 1, Block AF > > > Bidhannagar, Calcutta 700064 > > > India > > > *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com > > > *Webpage*: http://www.ph.utexas.edu/~daneel/ > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Sep 17 23:26:28 2013 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 17 Sep 2013 23:26:28 -0500 (CDT) Subject: [petsc-users] make all freezes In-Reply-To: References: Message-ID: On Wed, 18 Sep 2013, Analabha Roy wrote: > On Sep 17, 2013 9:39 PM, "Satish Balay" wrote: > > > > This doesn't make any sense.. > > > > If the build is hanging - we'd like to know exactly where its hanging. > > [perhaps copy/paste] > > > > It didn't dump any output, either to stdout or Stderr. There was nothing to > copy/paste. so you type in 'make all' and then see nothing on the screen? >>>>>>> all: chk_makej @${OMAKE} PETSC_ARCH=${PETSC_ARCH} PETSC_DIR=${PETSC_DIR} chk_petscdir chk_upgrade | tee ${PETSC_ARCH}/conf/make.log <<<<<< The only thing I can think off is chk_upgrade perhaps misconfigured network is affecting it. [if the behavior is reproduceable - perhaps we can narrow it down by invoking these commands one at a time]. > > If the run is hanging - we'd like to see the stack trace [via gdb] of the > > location of the hang. > > > > xorg-devel should'nt make a difference [from what we know].. > > > > Doesn't it need xorg headers for rendering graphic output via petscdraw? > > Maybe it was one of the dependencies of xorg devel that did the trick. If x11 includes/libs are not found - then petsc is configured without X11. > https://dl.dropboxusercontent.com/u/2769707/configure.log>(dropbox) The configure.log above confirms that it did not detect X11. [so none of the X11 stuff was enabled]. Even if X11 was enabled - 'make test' does not run any X11 related tests. If some make target caused hang for both 'make all' and 'make test' - then running the examples manually might confirm this. [and then invoking the steps in 'make test' one at a time might narrow down to the location of the hang]. Satish From security19 at sunivisiontech.com Wed Sep 18 01:18:20 2013 From: security19 at sunivisiontech.com (security) Date: Wed, 18 Sep 2013 14:18:20 +0800 Subject: [petsc-users] =?gb2312?b?QmlnIHNhbGVzIG9mIERWUiA0Q0ggRFZSIGp1c3Qg?= =?gb2312?b?bmVlZCAyNXVzZA==?= Message-ID: <381-22013931861820560@20130717-1059> Dear sir: Good news!We have Big sales of DVR 4CH DVR just need 25usd. The promotion will be end when all of them sold out! All of them have stock we can send to you right now, Contact us soon! Have a nice day! BR Tracy PS: China Sourcing Fair(HK) booth No: 2L32 Date: Oct 12th-15th, 2013 Add: Hong Kong AsiaWorld-Expo Welcome you to visit our booth and let's show you our latest products. China factory: Sunivision Technology Development Co.,Ltd Add:Floor 3, Building B, Taoyuan Industrial Park, Huangpu East Development Zone, Guangzhou, China Mobile: +86 15602392859 Alibaba Trade Manager:cn1001498382 SKYPE: sunivision19 Website:www.sunivision.com America office: SUNIVISION USA INC. 11721 183rd street #1175 Artesia, Ca 90702,USA Ph# 001818.687.8878 EMAIL: INFO at SUNIVISION.COM ----- Original Message ----- From: majid saeidiTo: sunivision19 at sunivision.comSent: 2012-12-26 17:14Subject: Re: Merry Christmas!---Tracy Hello dear Tracy Thank you very much for your attention to me !for start our relation , could you send me price list of your factory productions as camera , DVRs and accessories ?with specifications of them and as you know at Iran we have a terrible economic conditions and rate of currency has a great tolerance this days so I expect you give me your best price with maximum discount .I am looking forward to you with good news for me! Best regardsMajid Saeidi From: CCTV To: fgn.nasr Sent: Tuesday, 25 December 2012, 10:36 Subject: RE:Merry Christmas!---Tracy Dear customer:How are you?This is Tracy from Sunivision a manufacture of cctv camera and dvr.Merry Christmas!We are sincerely wish you and your family happy and business prosperous! I wonder whether you have any pending orders or inquiry recently ? If so, please contact me with no hesitate, we are sure any of your inquiry will get our prompt attention & reply. We are wondering if there is anything we can do for you?Pls don't hesitate to tell us! Have a nice day! Thanks&Regards Tracy Sunivision Technology Development Co.,Ltd Add:Floor 3, Building B, Taoyuan Industrial Park, Huangpu East Development Zone, Guangzhou, China Tel: +86 20 28217659-820 Mobile: +86 15602392859 MSN/Email: sunivision19 at sunivision.com SKYPE: sunivision19 Website:www.gzhuashi.en.alibaba.com 2013-09-17 15:25 ----- Original Message ----- From: fgn.nasr at yahoo.com To: sunivision19 at sunivision.com Sent: 2012-12-26 17:14 Subject: Re: Merry Christmas!---Tracy Hello dear Tracy Thank you very much for your attention to me ! for start our relation , could you send me price list of your factory productions as camera , DVRs and accessories ? with specifications of them and as you know at Iran we have a terrible economic conditions and rate of currency has a great tolerance this days so I expect you give me your best price with maximum discount . I am looking forward to you with good news for me! Best regards Majid Saeidi From: CCTV To: fgn.nasr Sent: Tuesday, 25 December 2012, 10:36 Subject: RE:Merry Christmas!---Tracy Dear customer: How are you?This is Tracy from Sunivision a manufacture of cctv camera and dvr.Merry Christmas! We are sincerely wish you and your family happy and business prosperous! I wonder whether you have any pending orders or inquiry recently ? If so, please contact me with no hesitate, we are sure any of your inquiry will get our prompt attention & reply. We are wondering if there is anything we can do for you?Pls don't hesitate to tell us! Have a nice day! Thanks&Regards Tracy Sunivision Technology Development Co.,Ltd Add:Floor 3, Building B, Taoyuan Industrial Park, Huangpu East Development Zone, Guangzhou, China Tel: +86 20 28217659-820 Mobile: +86 15602392859 MSN/Email: sunivision19 at sunivision.com SKYPE: sunivision19 Website:www.gzhuashi.en.alibaba.com gb(0,0,0); font-style: normal; font-family: arial, helvetica, sans-serif; background-color: transparent"How are you?This is Tracy from Sunivision a manufacture of cctv camera and dvr.Merry Christmas!We are sincerely wish you and your family happy and business prosperous! I wonder whether you have any pending orders or inquiry recently ? If so, please contact me with no hesitate, we are sure any of your inquiry will get our prompt attention & reply. We are wondering if there is anything we can do for you?Pls don't hesitate to tell us! Have a nice day! Thanks&Regards Tracy Sunivision Technology Development Co.,Ltd Add:Floor 3, Building B, Taoyuan Industrial Park, Huangpu East Development Zone, Guangzhou, China Tel: +86 20 28217659-820 Mobile: +86 15602392859 MSN/Email: sunivision19 at sunivision.com SKYPE: sunivision19 Website:www.gzhuashi.en.alibaba.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From frtr at fysik.dtu.dk Wed Sep 18 03:00:40 2013 From: frtr at fysik.dtu.dk (Frederik Treue) Date: Wed, 18 Sep 2013 10:00:40 +0200 Subject: [petsc-users] zero pattern of result of matmatmult In-Reply-To: <87ob7rsf04.fsf@mcs.anl.gov> References: <1378396034.2282.15.camel@frtr-laptop> <87ioyf6ys8.fsf@mcs.anl.gov> <1378461087.13721.10.camel@frtr-laptop> <87ppsm9in9.fsf@mcs.anl.gov> <1379422786.6379.5.camel@frtr-laptop> <1379426242.2358.8.camel@frtr-laptop> <87ob7rsf04.fsf@mcs.anl.gov> Message-ID: <1379491240.3677.8.camel@frtr-laptop> On Tue, 2013-09-17 at 07:24 -0700, Jed Brown wrote: > Frederik Treue writes: > > > On Tue, 2013-09-17 at 08:15 -0500, Barry Smith wrote: > >> Are you calling MatSetValuesStencil() for each single entry or once per row or block row? Row or block row will be faster than for each entry. > > once per entry - this I should definitely improve. What's the logic > > behind row or block rows being faster? The fewer calls to > > MatSetValuesStencil, the better? > > It shares index translation and the call stack. Just to be sure: If I could make the building of the matrix in one go (pr. mpi process), ie. give a list of stencils from all the points handled by the process, and then make one MatSetValuesStencil, would this always be faster? Assuming there is no memory concerns. > > PS. FYI: My stencil is essentially a standard 2D, 9-point box stencil, > > but the values in all rows are different (due, among other things, to > > the geometric tensor). > > That's normal. Is the geometry changing? Both that, and my operator depends explicitly on the fields, which obviously do change. > > > ########################################################## > > # # > > # WARNING!!! # > > # # > > # This code was compiled with a debugging option, # > > # To get timing results run ./configure # > > # using --with-debugging=no, the performance will # > > # be generally two or three times faster. # > > # # > > ########################################################## > > We weren't joking. > > You should configure an optimized build using a different value of > PETSC_ARCH so that you can easily switch back and forth between the > debug and optimized versions. Everything will get faster, but > especially insertion. Ah, so I can't assume that because version 1 of my code is X times faster than version 2, measured using a debug version of petsc, this will also hold for the production version of petsc... Thanks for the heads up :) /Frederik PS. Barry, the constant parts of my operator requires no calculation at all, but thanks for the info anyway. From bsmith at mcs.anl.gov Wed Sep 18 07:06:40 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 18 Sep 2013 07:06:40 -0500 Subject: [petsc-users] zero pattern of result of matmatmult In-Reply-To: <1379491240.3677.8.camel@frtr-laptop> References: <1378396034.2282.15.camel@frtr-laptop> <87ioyf6ys8.fsf@mcs.anl.gov> <1378461087.13721.10.camel@frtr-laptop> <87ppsm9in9.fsf@mcs.anl.gov> <1379422786.6379.5.camel@frtr-laptop> <1379426242.2358.8.camel@frtr-laptop> <87ob7rsf04.fsf@mcs.anl.gov> <1379491240.3677.8.camel@frtr-laptop> Message-ID: On Sep 18, 2013, at 3:00 AM, Frederik Treue wrote: > On Tue, 2013-09-17 at 07:24 -0700, Jed Brown wrote: >> Frederik Treue writes: >> >>> On Tue, 2013-09-17 at 08:15 -0500, Barry Smith wrote: >>>> Are you calling MatSetValuesStencil() for each single entry or once per row or block row? Row or block row will be faster than for each entry. >>> once per entry - this I should definitely improve. What's the logic >>> behind row or block rows being faster? The fewer calls to >>> MatSetValuesStencil, the better? >> >> It shares index translation and the call stack. > > Just to be sure: If I could make the building of the matrix in one go > (pr. mpi process), ie. give a list of stencils from all the points > handled by the process, and then make one MatSetValuesStencil, would > this always be faster? Assuming there is no memory concerns. No, no, no, do not do that! Just put in all the values at the same time that are naturally calculated together. Do not try to gather a big bunch of values and put them in together. For finite differences normally it is natural to have code that computes all values in a row together so put them in the matrix with one call to MatSetValuesStencil. Barry > >>> PS. FYI: My stencil is essentially a standard 2D, 9-point box stencil, >>> but the values in all rows are different (due, among other things, to >>> the geometric tensor). >> >> That's normal. Is the geometry changing? > > Both that, and my operator depends explicitly on the fields, which > obviously do change. >> >>> ########################################################## >>> # # >>> # WARNING!!! # >>> # # >>> # This code was compiled with a debugging option, # >>> # To get timing results run ./configure # >>> # using --with-debugging=no, the performance will # >>> # be generally two or three times faster. # >>> # # >>> ########################################################## >> >> We weren't joking. >> >> You should configure an optimized build using a different value of >> PETSC_ARCH so that you can easily switch back and forth between the >> debug and optimized versions. Everything will get faster, but >> especially insertion. > > Ah, so I can't assume that because version 1 of my code is X times > faster than version 2, measured using a debug version of petsc, this > will also hold for the production version of petsc... Thanks for the > heads up :) > > /Frederik > > PS. Barry, the constant parts of my operator requires no calculation at > all, but thanks for the info anyway. > From vijay.m at gmail.com Wed Sep 18 10:32:55 2013 From: vijay.m at gmail.com (Vijay S. Mahadevan) Date: Wed, 18 Sep 2013 10:32:55 -0500 Subject: [petsc-users] Error when using TSSetRHSJacobian+DMSetMatrixPreallocateOnly Message-ID: All, When I call DMSetMatrixPreallocateOnly and then TSSetRHSJacobian without a matrix input argument (NULL instead), then the code crashes during TSSolve and more specifically during MatDuplicate. This is replicable with src/ts/examples/tutorials/ex2.c. Here's the error with stack trace. [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Object is in wrong state! [0]PETSC ERROR: Not for unassembled matrix! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Development GIT revision: 0799ffdeffb47ef0458c1305293fae28c4d2cd92 GIT Date: 2013-08-01 06:36:16 +0800 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./ex2 on a arch-darwin-cxx-debug named Tachyon.local by mahadevan Tue Sep 17 21:26:54 2013 [0]PETSC ERROR: Libraries linked from /opt/petsc-dbg/lib [0]PETSC ERROR: Configure run at Wed Jul 31 18:03:46 2013 [0]PETSC ERROR: Configure options --prefix=/opt/petsc-dbg PETSC_ARCH=arch-darwin-cxx-debug --download-scalapack=1 --download-blacs=1 --with-mumps-dir=/Users/mahadevan/source/MUMPS_4.10.0-p3 --download-hypre=1 --with-metis=1 --with-parmetis=1 --known-mpi-shared-libraries=1 --with-blas-lapack-lib=/System/Library/Frameworks/vecLib.framework/vecLib --with-c++-support=1 --with-c-support=1 --with-cc=mpicc --with-clanguage=C++ --with-moab-dir=/opt/moab --with-dynamic-loading=1 --with-fc=mpif90 --with-fortran=1 --with-mpi=1 --with-shared-libraries=1 --with-valgrind=1 --with-valgrind-dir=/opt/local --with-cc=mpicc --with-cxx=mpicxx COPTFLAGS="-g -fPIC" CXXOPTFLAGS="-g -fPIC" FOPTFLAGS="-g -fPIC" --with-metis-dir=/usr/local --with-parmetis-dir=/usr/local --with-netcdf-dir=/usr/local --with-zoltan=1 --with-zoltan-lib="-L/usr/local/lib -lptscotch -lscotch -lscotchmetis -lscotcherr -lscotcherrexit -lzoltan" --with-zoltan-include=/usr/local/include --with-hdf5 --with-hdf5-include=/usr/local/include --with-hdf5-lib="-L/usr/local/lib -lhdf5_fortran -lhdf5" --with-netcdf-dir=/usr/local --with-cmake=/opt/local/bin/cmake --with-x-dir=/opt/local [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatDuplicate() line 4109 in /Users/mahadevan/source/petsc-dev/src/mat/interface/matrix.c [0]PETSC ERROR: TSGetRHSMats_Private() line 568 in /Users/mahadevan/source/petsc-dev/src/ts/interface/ts.c [0]PETSC ERROR: TSComputeIJacobian() line 740 in /Users/mahadevan/source/petsc-dev/src/ts/interface/ts.c [0]PETSC ERROR: SNESTSFormJacobian_Theta() line 307 in /Users/mahadevan/source/petsc-dev/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: SNESTSFormJacobian() line 3446 in /Users/mahadevan/source/petsc-dev/src/ts/interface/ts.c [0]PETSC ERROR: SNESComputeJacobian() line 2227 in /Users/mahadevan/source/petsc-dev/src/snes/interface/snes.c [0]PETSC ERROR: SNESSolve_NEWTONLS() line 233 in /Users/mahadevan/source/petsc-dev/src/snes/impls/ls/ls.c [0]PETSC ERROR: SNESSolve() line 3789 in /Users/mahadevan/source/petsc-dev/src/snes/interface/snes.c [0]PETSC ERROR: TSStep_Theta() line 183 in /Users/mahadevan/source/petsc-dev/src/ts/impls/implicit/theta/theta.c [0]PETSC ERROR: TSStep() line 2507 in /Users/mahadevan/source/petsc-dev/src/ts/interface/ts.c [0]PETSC ERROR: TSSolve() line 2632 in /Users/mahadevan/source/petsc-dev/src/ts/interface/ts.c [0]PETSC ERROR: main() line 196 in /Users/mahadevan/source/petsc-dev/src/ts/examples/tutorials/ex2.c I hope this is not supposed to happen. The fix seems to be to call TSSetRHSJacobian with a user created matrix instead of letting DM create one, which rectifies this issue. This behavior feels inconsistent and is there a way to fix this. Vijay PS: Just replace the following lines in ex2.c to replicate the error. /* ierr = TSSetRHSJacobian(ts,A,A,RHSJacobian,&appctx);CHKERRQ(ierr); */ ierr = TSSetRHSJacobian(ts,NULL,NULL,RHSJacobian,&appctx);CHKERRQ(ierr); ierr = DMSetMatrixPreallocateOnly(appctx.da,PETSC_TRUE);CHKERRQ(ierr); From bsmith at mcs.anl.gov Wed Sep 18 10:46:05 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 18 Sep 2013 10:46:05 -0500 Subject: [petsc-users] Error when using TSSetRHSJacobian+DMSetMatrixPreallocateOnly In-Reply-To: References: Message-ID: <541AD26E-C983-4807-A648-EFF9A5E7D03B@mcs.anl.gov> The code did not "crash", it ended with an error message On Sep 18, 2013, at 10:32 AM, "Vijay S. Mahadevan" wrote: > All, > > When I call DMSetMatrixPreallocateOnly Why are you setting this flag? If you don't set it then the code will go through. > and then TSSetRHSJacobian > without a matrix input argument (NULL instead), then the code crashes > during TSSolve and more specifically during MatDuplicate. This is > replicable with src/ts/examples/tutorials/ex2.c. Here's the error with Jed is reworking the logic here so the code will change. > stack trace. > > [0]PETSC ERROR: --------------------- Error Message > > > I hope this is not supposed to happen. The fix seems to be to call > TSSetRHSJacobian with a user created matrix instead of letting DM > create one, There is no reason to use a user created matrix since the DM can provide the correct one. One of the "main jobs" of DM is to create that matrix so we really don't intend people to use a DM but then create the matrix themselves. Barry > which rectifies this issue. This behavior feels > inconsistent and is there a way to fix this. > > Vijay > > PS: Just replace the following lines in ex2.c to replicate the error. > > /* ierr = TSSetRHSJacobian(ts,A,A,RHSJacobian,&appctx);CHKERRQ(ierr); */ > ierr = TSSetRHSJacobian(ts,NULL,NULL,RHSJacobian,&appctx);CHKERRQ(ierr); > ierr = DMSetMatrixPreallocateOnly(appctx.da,PETSC_TRUE);CHKERRQ(ierr); From vijay.m at gmail.com Wed Sep 18 10:52:30 2013 From: vijay.m at gmail.com (Vijay S. Mahadevan) Date: Wed, 18 Sep 2013 10:52:30 -0500 Subject: [petsc-users] Error when using TSSetRHSJacobian+DMSetMatrixPreallocateOnly In-Reply-To: <541AD26E-C983-4807-A648-EFF9A5E7D03B@mcs.anl.gov> References: <541AD26E-C983-4807-A648-EFF9A5E7D03B@mcs.anl.gov> Message-ID: > Why are you setting this flag? If you don't set it then the code will go through. Yes it passes through with DMDA. But with my custom DM implementation (MOAB based), this fails unless I explicitly set the NNZ pattern in DMCreateMatrix. I currently don't do that and the only way to replicate this behavior with DMDA was setting the flag explicitly. > There is no reason to use a user created matrix since the DM can provide the correct one. One of the "main jobs" of DM is to create that matrix so we really don't intend people to use a DM but then create the matrix themselves. Agreed, and that's what I expected when I passed in NULL args instead of user matrix. But the DM created matrix errors out during Duplicate while user provided matrix passes through, even though both of them are unassembled. And this is what is confusing. Vijay On Wed, Sep 18, 2013 at 10:46 AM, Barry Smith wrote: > > The code did not "crash", it ended with an error message > > > On Sep 18, 2013, at 10:32 AM, "Vijay S. Mahadevan" wrote: > >> All, >> >> When I call DMSetMatrixPreallocateOnly > > Why are you setting this flag? If you don't set it then the code will go through. > >> and then TSSetRHSJacobian >> without a matrix input argument (NULL instead), then the code crashes >> during TSSolve and more specifically during MatDuplicate. This is >> replicable with src/ts/examples/tutorials/ex2.c. Here's the error with > > Jed is reworking the logic here so the code will change. > > >> stack trace. >> >> [0]PETSC ERROR: --------------------- Error Message >> >> >> I hope this is not supposed to happen. The fix seems to be to call >> TSSetRHSJacobian with a user created matrix instead of letting DM >> create one, > > There is no reason to use a user created matrix since the DM can provide the correct one. One of the "main jobs" of DM is to create that matrix so we really don't intend people to use a DM but then create the matrix themselves. > > Barry > > >> which rectifies this issue. This behavior feels >> inconsistent and is there a way to fix this. >> >> Vijay >> >> PS: Just replace the following lines in ex2.c to replicate the error. >> >> /* ierr = TSSetRHSJacobian(ts,A,A,RHSJacobian,&appctx);CHKERRQ(ierr); */ >> ierr = TSSetRHSJacobian(ts,NULL,NULL,RHSJacobian,&appctx);CHKERRQ(ierr); >> ierr = DMSetMatrixPreallocateOnly(appctx.da,PETSC_TRUE);CHKERRQ(ierr); > From bsmith at mcs.anl.gov Wed Sep 18 11:17:52 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 18 Sep 2013 11:17:52 -0500 Subject: [petsc-users] Error when using TSSetRHSJacobian+DMSetMatrixPreallocateOnly In-Reply-To: References: <541AD26E-C983-4807-A648-EFF9A5E7D03B@mcs.anl.gov> Message-ID: On Sep 18, 2013, at 10:52 AM, "Vijay S. Mahadevan" wrote: >> Why are you setting this flag? If you don't set it then the code will go through. > > Yes it passes through with DMDA. But with my custom DM implementation > (MOAB based), this fails unless I explicitly set the NNZ pattern in > DMCreateMatrix. I currently don't do that and the only way to > replicate this behavior with DMDA was setting the flag explicitly. Until you can provide the preallocation you need to simply call MatSetUp() and then MatAssemblyBegin/End() on your matrix inside your DMCreateMatrix_Moab(). The expectation is that the DM matrix provided is "ready to go", we didn't expect to handle matrices that were "incomplete" in other parts of the code. > >> There is no reason to use a user created matrix since the DM can provide the correct one. One of the "main jobs" of DM is to create that matrix so we really don't intend people to use a DM but then create the matrix themselves. > > Agreed, and that's what I expected when I passed in NULL args instead > of user matrix. But the DM created matrix errors out during Duplicate > while user provided matrix passes through, even though both of them > are unassembled. PetscErrorCode MatDuplicate(Mat mat,MatDuplicateOption op,Mat *M) { PetscErrorCode ierr; Mat B; PetscInt i; PetscFunctionBegin; PetscValidHeaderSpecific(mat,MAT_CLASSID,1); PetscValidType(mat,1); PetscValidPointer(M,3); if (!mat->assembled) SETERRQ(PetscObjectComm((PetscObject)mat),PETSC_ERR_ARG_WRONGSTATE,"Not for unassembled matrix"); The "user matrix" must have been assembled otherwise it would have stopped above. > And this is what is confusing. > > Vijay > > On Wed, Sep 18, 2013 at 10:46 AM, Barry Smith wrote: >> >> The code did not "crash", it ended with an error message >> >> >> On Sep 18, 2013, at 10:32 AM, "Vijay S. Mahadevan" wrote: >> >>> All, >>> >>> When I call DMSetMatrixPreallocateOnly >> >> Why are you setting this flag? If you don't set it then the code will go through. >> >>> and then TSSetRHSJacobian >>> without a matrix input argument (NULL instead), then the code crashes >>> during TSSolve and more specifically during MatDuplicate. This is >>> replicable with src/ts/examples/tutorials/ex2.c. Here's the error with >> >> Jed is reworking the logic here so the code will change. >> >> >>> stack trace. >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> >>> >>> I hope this is not supposed to happen. The fix seems to be to call >>> TSSetRHSJacobian with a user created matrix instead of letting DM >>> create one, >> >> There is no reason to use a user created matrix since the DM can provide the correct one. One of the "main jobs" of DM is to create that matrix so we really don't intend people to use a DM but then create the matrix themselves. >> >> Barry >> >> >>> which rectifies this issue. This behavior feels >>> inconsistent and is there a way to fix this. >>> >>> Vijay >>> >>> PS: Just replace the following lines in ex2.c to replicate the error. >>> >>> /* ierr = TSSetRHSJacobian(ts,A,A,RHSJacobian,&appctx);CHKERRQ(ierr); */ >>> ierr = TSSetRHSJacobian(ts,NULL,NULL,RHSJacobian,&appctx);CHKERRQ(ierr); >>> ierr = DMSetMatrixPreallocateOnly(appctx.da,PETSC_TRUE);CHKERRQ(ierr); >> From vijay.m at gmail.com Wed Sep 18 12:06:41 2013 From: vijay.m at gmail.com (Vijay S. Mahadevan) Date: Wed, 18 Sep 2013 12:06:41 -0500 Subject: [petsc-users] Error when using TSSetRHSJacobian+DMSetMatrixPreallocateOnly In-Reply-To: References: <541AD26E-C983-4807-A648-EFF9A5E7D03B@mcs.anl.gov> Message-ID: > Until you can provide the preallocation you need to simply call MatSetUp() and then MatAssemblyBegin/End() on your matrix inside your DMCreateMatrix_Moab(). The expectation is that the DM matrix provided is "ready to go", we didn't expect to handle matrices that were "incomplete" in other parts of the code. Interesting. Calling MatAssemblyBegin/End() without any MatSetValues* wipes out the preallocation information. With -info turned on, I get the following for some sample run. [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 5192 X 5192; storage space: 6906304 unneeded,0 used [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 0 I do understand the expectation that matrix from DMCreateMatrix needs to be complete but just that traversing through the mesh and setting the right NNZ is difficult (I dont think this should exist in the library code since this is physics/problem dependent). Somewhat manageable for DMDA since it relies on a predefined stencil but I couldn't wrap my head around doing this for a generic enough implementation. I will think about this some more. > The "user matrix" must have been assembled otherwise it would have stopped above. It did. Vijay On Wed, Sep 18, 2013 at 11:17 AM, Barry Smith wrote: > > On Sep 18, 2013, at 10:52 AM, "Vijay S. Mahadevan" wrote: > >>> Why are you setting this flag? If you don't set it then the code will go through. >> >> Yes it passes through with DMDA. But with my custom DM implementation >> (MOAB based), this fails unless I explicitly set the NNZ pattern in >> DMCreateMatrix. I currently don't do that and the only way to >> replicate this behavior with DMDA was setting the flag explicitly. > > Until you can provide the preallocation you need to simply call MatSetUp() and then MatAssemblyBegin/End() on your matrix inside your DMCreateMatrix_Moab(). The expectation is that the DM matrix provided is "ready to go", we didn't expect to handle matrices that were "incomplete" in other parts of the code. > >> >>> There is no reason to use a user created matrix since the DM can provide the correct one. One of the "main jobs" of DM is to create that matrix so we really don't intend people to use a DM but then create the matrix themselves. >> >> Agreed, and that's what I expected when I passed in NULL args instead >> of user matrix. But the DM created matrix errors out during Duplicate >> while user provided matrix passes through, even though both of them >> are unassembled. > > PetscErrorCode MatDuplicate(Mat mat,MatDuplicateOption op,Mat *M) > { > PetscErrorCode ierr; > Mat B; > PetscInt i; > > PetscFunctionBegin; > PetscValidHeaderSpecific(mat,MAT_CLASSID,1); > PetscValidType(mat,1); > PetscValidPointer(M,3); > if (!mat->assembled) SETERRQ(PetscObjectComm((PetscObject)mat),PETSC_ERR_ARG_WRONGSTATE,"Not for unassembled matrix"); > > The "user matrix" must have been assembled otherwise it would have stopped above. > > >> And this is what is confusing. >> >> Vijay >> >> On Wed, Sep 18, 2013 at 10:46 AM, Barry Smith wrote: >>> >>> The code did not "crash", it ended with an error message >>> >>> >>> On Sep 18, 2013, at 10:32 AM, "Vijay S. Mahadevan" wrote: >>> >>>> All, >>>> >>>> When I call DMSetMatrixPreallocateOnly >>> >>> Why are you setting this flag? If you don't set it then the code will go through. >>> >>>> and then TSSetRHSJacobian >>>> without a matrix input argument (NULL instead), then the code crashes >>>> during TSSolve and more specifically during MatDuplicate. This is >>>> replicable with src/ts/examples/tutorials/ex2.c. Here's the error with >>> >>> Jed is reworking the logic here so the code will change. >>> >>> >>>> stack trace. >>>> >>>> [0]PETSC ERROR: --------------------- Error Message >>>> >>>> >>>> I hope this is not supposed to happen. The fix seems to be to call >>>> TSSetRHSJacobian with a user created matrix instead of letting DM >>>> create one, >>> >>> There is no reason to use a user created matrix since the DM can provide the correct one. One of the "main jobs" of DM is to create that matrix so we really don't intend people to use a DM but then create the matrix themselves. >>> >>> Barry >>> >>> >>>> which rectifies this issue. This behavior feels >>>> inconsistent and is there a way to fix this. >>>> >>>> Vijay >>>> >>>> PS: Just replace the following lines in ex2.c to replicate the error. >>>> >>>> /* ierr = TSSetRHSJacobian(ts,A,A,RHSJacobian,&appctx);CHKERRQ(ierr); */ >>>> ierr = TSSetRHSJacobian(ts,NULL,NULL,RHSJacobian,&appctx);CHKERRQ(ierr); >>>> ierr = DMSetMatrixPreallocateOnly(appctx.da,PETSC_TRUE);CHKERRQ(ierr); >>> > From knepley at gmail.com Wed Sep 18 12:10:52 2013 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 18 Sep 2013 10:10:52 -0700 Subject: [petsc-users] Error when using TSSetRHSJacobian+DMSetMatrixPreallocateOnly In-Reply-To: References: <541AD26E-C983-4807-A648-EFF9A5E7D03B@mcs.anl.gov> Message-ID: On Wed, Sep 18, 2013 at 10:06 AM, Vijay S. Mahadevan wrote: > > Until you can provide the preallocation you need to simply call > MatSetUp() and then MatAssemblyBegin/End() on your matrix inside your > DMCreateMatrix_Moab(). The expectation is that the DM matrix provided is > "ready to go", we didn't expect to handle matrices that were "incomplete" > in other parts of the code. > > Interesting. Calling MatAssemblyBegin/End() without any MatSetValues* > wipes out the preallocation information. With -info turned on, I get > the following for some sample run. > > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 5192 X 5192; storage space: > 6906304 unneeded,0 used > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 0 > > I do understand the expectation that matrix from DMCreateMatrix needs > to be complete but just that traversing through the mesh and setting > the right NNZ is difficult (I dont think this should exist in the > library code since this is physics/problem dependent). Somewhat > manageable for DMDA since it relies on a predefined stencil but I > couldn't wrap my head around doing this for a generic enough > implementation. I will think about this some more. > DMPlex does it. Matt > > The "user matrix" must have been assembled otherwise it would have > stopped above. > > It did. > > Vijay > > On Wed, Sep 18, 2013 at 11:17 AM, Barry Smith wrote: > > > > On Sep 18, 2013, at 10:52 AM, "Vijay S. Mahadevan" > wrote: > > > >>> Why are you setting this flag? If you don't set it then the code > will go through. > >> > >> Yes it passes through with DMDA. But with my custom DM implementation > >> (MOAB based), this fails unless I explicitly set the NNZ pattern in > >> DMCreateMatrix. I currently don't do that and the only way to > >> replicate this behavior with DMDA was setting the flag explicitly. > > > > Until you can provide the preallocation you need to simply call > MatSetUp() and then MatAssemblyBegin/End() on your matrix inside your > DMCreateMatrix_Moab(). The expectation is that the DM matrix provided is > "ready to go", we didn't expect to handle matrices that were "incomplete" > in other parts of the code. > > > >> > >>> There is no reason to use a user created matrix since the DM can > provide the correct one. One of the "main jobs" of DM is to create that > matrix so we really don't intend people to use a DM but then create the > matrix themselves. > >> > >> Agreed, and that's what I expected when I passed in NULL args instead > >> of user matrix. But the DM created matrix errors out during Duplicate > >> while user provided matrix passes through, even though both of them > >> are unassembled. > > > > PetscErrorCode MatDuplicate(Mat mat,MatDuplicateOption op,Mat *M) > > { > > PetscErrorCode ierr; > > Mat B; > > PetscInt i; > > > > PetscFunctionBegin; > > PetscValidHeaderSpecific(mat,MAT_CLASSID,1); > > PetscValidType(mat,1); > > PetscValidPointer(M,3); > > if (!mat->assembled) > SETERRQ(PetscObjectComm((PetscObject)mat),PETSC_ERR_ARG_WRONGSTATE,"Not for > unassembled matrix"); > > > > The "user matrix" must have been assembled otherwise it would have > stopped above. > > > > > >> And this is what is confusing. > >> > >> Vijay > >> > >> On Wed, Sep 18, 2013 at 10:46 AM, Barry Smith > wrote: > >>> > >>> The code did not "crash", it ended with an error message > >>> > >>> > >>> On Sep 18, 2013, at 10:32 AM, "Vijay S. Mahadevan" > wrote: > >>> > >>>> All, > >>>> > >>>> When I call DMSetMatrixPreallocateOnly > >>> > >>> Why are you setting this flag? If you don't set it then the code > will go through. > >>> > >>>> and then TSSetRHSJacobian > >>>> without a matrix input argument (NULL instead), then the code crashes > >>>> during TSSolve and more specifically during MatDuplicate. This is > >>>> replicable with src/ts/examples/tutorials/ex2.c. Here's the error with > >>> > >>> Jed is reworking the logic here so the code will change. > >>> > >>> > >>>> stack trace. > >>>> > >>>> [0]PETSC ERROR: --------------------- Error Message > >>>> > >>>> > >>>> I hope this is not supposed to happen. The fix seems to be to call > >>>> TSSetRHSJacobian with a user created matrix instead of letting DM > >>>> create one, > >>> > >>> There is no reason to use a user created matrix since the DM can > provide the correct one. One of the "main jobs" of DM is to create that > matrix so we really don't intend people to use a DM but then create the > matrix themselves. > >>> > >>> Barry > >>> > >>> > >>>> which rectifies this issue. This behavior feels > >>>> inconsistent and is there a way to fix this. > >>>> > >>>> Vijay > >>>> > >>>> PS: Just replace the following lines in ex2.c to replicate the error. > >>>> > >>>> /* ierr = > TSSetRHSJacobian(ts,A,A,RHSJacobian,&appctx);CHKERRQ(ierr); */ > >>>> ierr = > TSSetRHSJacobian(ts,NULL,NULL,RHSJacobian,&appctx);CHKERRQ(ierr); > >>>> ierr = DMSetMatrixPreallocateOnly(appctx.da,PETSC_TRUE);CHKERRQ(ierr); > >>> > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From vijay.m at gmail.com Wed Sep 18 12:13:30 2013 From: vijay.m at gmail.com (Vijay S. Mahadevan) Date: Wed, 18 Sep 2013 12:13:30 -0500 Subject: [petsc-users] Error when using TSSetRHSJacobian+DMSetMatrixPreallocateOnly In-Reply-To: References: <541AD26E-C983-4807-A648-EFF9A5E7D03B@mcs.anl.gov> Message-ID: > DMPlex does it. I was just thinking about how you managed this in DMPlex. I will take a look at this implementation. Vijay On Wed, Sep 18, 2013 at 12:10 PM, Matthew Knepley wrote: > On Wed, Sep 18, 2013 at 10:06 AM, Vijay S. Mahadevan > wrote: >> >> > Until you can provide the preallocation you need to simply call >> > MatSetUp() and then MatAssemblyBegin/End() on your matrix inside your >> > DMCreateMatrix_Moab(). The expectation is that the DM matrix provided is >> > "ready to go", we didn't expect to handle matrices that were "incomplete" in >> > other parts of the code. >> >> Interesting. Calling MatAssemblyBegin/End() without any MatSetValues* >> wipes out the preallocation information. With -info turned on, I get >> the following for some sample run. >> >> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 5192 X 5192; storage space: >> 6906304 unneeded,0 used >> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 0 >> >> I do understand the expectation that matrix from DMCreateMatrix needs >> to be complete but just that traversing through the mesh and setting >> the right NNZ is difficult (I dont think this should exist in the >> library code since this is physics/problem dependent). Somewhat >> manageable for DMDA since it relies on a predefined stencil but I >> couldn't wrap my head around doing this for a generic enough >> implementation. I will think about this some more. > > > DMPlex does it. > > Matt > >> >> > The "user matrix" must have been assembled otherwise it would have >> > stopped above. >> >> It did. >> >> Vijay >> >> On Wed, Sep 18, 2013 at 11:17 AM, Barry Smith wrote: >> > >> > On Sep 18, 2013, at 10:52 AM, "Vijay S. Mahadevan" >> > wrote: >> > >> >>> Why are you setting this flag? If you don't set it then the code >> >>> will go through. >> >> >> >> Yes it passes through with DMDA. But with my custom DM implementation >> >> (MOAB based), this fails unless I explicitly set the NNZ pattern in >> >> DMCreateMatrix. I currently don't do that and the only way to >> >> replicate this behavior with DMDA was setting the flag explicitly. >> > >> > Until you can provide the preallocation you need to simply call >> > MatSetUp() and then MatAssemblyBegin/End() on your matrix inside your >> > DMCreateMatrix_Moab(). The expectation is that the DM matrix provided is >> > "ready to go", we didn't expect to handle matrices that were "incomplete" in >> > other parts of the code. >> > >> >> >> >>> There is no reason to use a user created matrix since the DM can >> >>> provide the correct one. One of the "main jobs" of DM is to create that >> >>> matrix so we really don't intend people to use a DM but then create the >> >>> matrix themselves. >> >> >> >> Agreed, and that's what I expected when I passed in NULL args instead >> >> of user matrix. But the DM created matrix errors out during Duplicate >> >> while user provided matrix passes through, even though both of them >> >> are unassembled. >> > >> > PetscErrorCode MatDuplicate(Mat mat,MatDuplicateOption op,Mat *M) >> > { >> > PetscErrorCode ierr; >> > Mat B; >> > PetscInt i; >> > >> > PetscFunctionBegin; >> > PetscValidHeaderSpecific(mat,MAT_CLASSID,1); >> > PetscValidType(mat,1); >> > PetscValidPointer(M,3); >> > if (!mat->assembled) >> > SETERRQ(PetscObjectComm((PetscObject)mat),PETSC_ERR_ARG_WRONGSTATE,"Not for >> > unassembled matrix"); >> > >> > The "user matrix" must have been assembled otherwise it would have >> > stopped above. >> > >> > >> >> And this is what is confusing. >> >> >> >> Vijay >> >> >> >> On Wed, Sep 18, 2013 at 10:46 AM, Barry Smith >> >> wrote: >> >>> >> >>> The code did not "crash", it ended with an error message >> >>> >> >>> >> >>> On Sep 18, 2013, at 10:32 AM, "Vijay S. Mahadevan" >> >>> wrote: >> >>> >> >>>> All, >> >>>> >> >>>> When I call DMSetMatrixPreallocateOnly >> >>> >> >>> Why are you setting this flag? If you don't set it then the code >> >>> will go through. >> >>> >> >>>> and then TSSetRHSJacobian >> >>>> without a matrix input argument (NULL instead), then the code crashes >> >>>> during TSSolve and more specifically during MatDuplicate. This is >> >>>> replicable with src/ts/examples/tutorials/ex2.c. Here's the error >> >>>> with >> >>> >> >>> Jed is reworking the logic here so the code will change. >> >>> >> >>> >> >>>> stack trace. >> >>>> >> >>>> [0]PETSC ERROR: --------------------- Error Message >> >>>> >> >>>> >> >>>> I hope this is not supposed to happen. The fix seems to be to call >> >>>> TSSetRHSJacobian with a user created matrix instead of letting DM >> >>>> create one, >> >>> >> >>> There is no reason to use a user created matrix since the DM can >> >>> provide the correct one. One of the "main jobs" of DM is to create that >> >>> matrix so we really don't intend people to use a DM but then create the >> >>> matrix themselves. >> >>> >> >>> Barry >> >>> >> >>> >> >>>> which rectifies this issue. This behavior feels >> >>>> inconsistent and is there a way to fix this. >> >>>> >> >>>> Vijay >> >>>> >> >>>> PS: Just replace the following lines in ex2.c to replicate the error. >> >>>> >> >>>> /* ierr = >> >>>> TSSetRHSJacobian(ts,A,A,RHSJacobian,&appctx);CHKERRQ(ierr); */ >> >>>> ierr = >> >>>> TSSetRHSJacobian(ts,NULL,NULL,RHSJacobian,&appctx);CHKERRQ(ierr); >> >>>> ierr = >> >>>> DMSetMatrixPreallocateOnly(appctx.da,PETSC_TRUE);CHKERRQ(ierr); >> >>> >> > > > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener From ckhuangf at gmail.com Thu Sep 19 00:05:13 2013 From: ckhuangf at gmail.com (Chung-Kan Huang) Date: Thu, 19 Sep 2013 00:05:13 -0500 Subject: [petsc-users] Problem with MatCreateSeqAIJWithArrays Message-ID: Hi, I am writing a testing code that read CRS matrix M and rhs vector B from two files and solve M*B=X. I got error message indicating that I have segmentation fault but problem disappeared after I removed MatDestroy(& M); Another problem is that I am not sure if I should have MatAssemblyBegin(M, MAT_FINAL_ASSEMBLY); MatAssemblyEnd(M, MAT_FINAL_ASSEMBLY); after MatCreateSeqAIJWithArrays(PETSC_COMM_WORLD, nrow, ncol, rowptr, colptr, mat, &M); I attached my test code and hope someone can give me some pointers. Thanks, Kan #include "petscksp.h" #include "petscao.h" #include "petsc.h" #include "HYPRE.h" #include "boost/timer/timer.hpp" #include #include typedef boost::timer::cpu_timer Timer; void P_Solve(const char * matrixFileName, const char * rhsFileName, const char * solFileName) { Timer timer; PetscInitialize(PETSC_NULL, PETSC_NULL, PETSC_NULL, PETSC_NULL); Mat M; // initial matrix downloaded from disk Vec B; // initial RHS vector downloaded from disk Vec X; // initial guess=0/solution vector PC prec; KSP solver; FILE * fp; char temp[3]; PetscInt nrow, ncol, nnz; PetscFOpen(PETSC_COMM_WORLD, matrixFileName, "r", &fp); fscanf(fp, "%s", temp); //# fscanf(fp, "%i %i %i", &nrow, &ncol, &nnz); PetscInt * rowptr = new PetscInt [nrow + 1]; PetscInt * colptr = new PetscInt [nnz]; PetscReal * mat = new PetscReal [nnz]; fscanf(fp, "%s", temp); //# for (int i = 0; i < nrow + 1; i++) { fscanf(fp, "%i", &rowptr[i]); } fscanf(fp, "%s", temp); //# for (int i = 0; i < nnz; i++) { fscanf(fp, "%i", &colptr[i]); } fscanf(fp, "%s", temp); //# for (int i = 0; i < nnz; i++) { fscanf(fp, "%lf", &mat[i]); } fclose(fp); MatCreateSeqAIJWithArrays(PETSC_COMM_WORLD, nrow, ncol, rowptr, colptr, mat, &M); MatAssemblyBegin(M, MAT_FINAL_ASSEMBLY); MatAssemblyEnd(M, MAT_FINAL_ASSEMBLY); // MatView(M, PETSC_VIEWER_STDOUT_WORLD); PetscFOpen(PETSC_COMM_WORLD, rhsFileName, "r", &fp); PetscReal * rhs = new PetscReal [nrow]; PetscReal * x = new PetscReal [nrow]; for (int i = 0; i < nrow; i++) { fscanf(fp, "%lf", &rhs[i]); } fclose(fp); VecCreateSeqWithArray(PETSC_COMM_WORLD, 1, nrow, rhs, &B); // VecView(B, PETSC_VIEWER_STDOUT_WORLD); MatGetVecs(M, &X, NULL); KSPCreate(PETSC_COMM_WORLD, &solver); KSPSetFromOptions(solver); KSPConvergedReason reason; PetscReal rtol = 0; PetscReal atol = 0; PetscReal dtol = 0; PetscInt it = 0; PetscInt its = 0; KSPGetTolerances( solver, &rtol, &atol, &dtol, &it); rtol = 1e-6; it = 500; KSPSetTolerances( solver, rtol, atol, dtol, it); KSPSetOperators(solver, M, M, SAME_NONZERO_PATTERN); KSPSetType(solver, KSPBCGS); KSPSetPCSide(solver, PC_RIGHT); // right preconditioning KSPSetInitialGuessNonzero(solver,PETSC_FALSE); KSPGetPC(solver, &prec); PCSetType(prec, PCILU); KSPSetFromOptions(solver); KSPSetUp(solver); PCSetUp(prec); PCFactorGetMatrix(prec, &M); timer.resume(); KSPSolve(solver,B,X); timer.stop(); KSPGetConvergedReason(solver,&reason); if (reason==KSP_DIVERGED_INDEFINITE_PC) { PetscPrintf(PETSC_COMM_WORLD,"\nDivergence because of indefinite preconditioner;\n"); PetscPrintf(PETSC_COMM_WORLD,"Run the executable again but with '-pc_factor_shift_type POSITIVE_DEFINITE' option.\n"); } else if (reason<0) { PetscPrintf(PETSC_COMM_WORLD,"\nOther kind of divergence: this should not happen. %d\n",reason); } KSPGetIterationNumber(solver, &its); std::cout << "PETSc ITS = " << its << " " << "TIMES(SEC) = " << timer.elapsed().wall / 1.e9 << "\n"; // VecView(X, PETSC_VIEWER_STDOUT_WORLD); VecGetArray(X, &x); PetscFOpen(PETSC_COMM_WORLD, solFileName, "w", &fp); for (int i = 0; i < nrow; i++) { fprintf(fp, "%.20e\n", x[i]); } VecRestoreArray(X, &x); fclose(fp); KSPDestroy(& solver); VecDestroy(& B); VecDestroy(& X); MatDestroy(& M); if (rowptr == NULL) delete [] rowptr; if (colptr == NULL) delete [] colptr; if (mat == NULL) delete [] mat; if (rhs == NULL) delete [] rhs; PetscFinalize(); } -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Sep 19 01:30:13 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 19 Sep 2013 01:30:13 -0500 Subject: [petsc-users] Problem with MatCreateSeqAIJWithArrays In-Reply-To: References: Message-ID: <87ioxx1fyi.fsf@mcs.anl.gov> Chung-Kan Huang writes: > Hi, > > I am writing a testing code that read CRS matrix M and rhs vector B from > two files and solve M*B=X. > > I got error message indicating that I have segmentation fault but problem > disappeared after I removed MatDestroy(& M); Run in Valgrind to find the problem. > Another problem is that I am not sure if I should have > MatAssemblyBegin(M, MAT_FINAL_ASSEMBLY); > > MatAssemblyEnd(M, MAT_FINAL_ASSEMBLY); > > after > > MatCreateSeqAIJWithArrays(PETSC_COMM_WORLD, nrow, ncol, rowptr, colptr, > mat, &M); No, the last two lines of MatCreateSeqAIJWithArrays is MatAssemblyBegin and MatAssemblyEnd. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From bisheshkh at gmail.com Thu Sep 19 05:57:28 2013 From: bisheshkh at gmail.com (Bishesh Khanal) Date: Thu, 19 Sep 2013 12:57:28 +0200 Subject: [petsc-users] accessing individual elements of a global vector created from dmda In-Reply-To: References: Message-ID: On Tue, Sep 17, 2013 at 6:15 PM, Barry Smith wrote: > > Bishesh, > > Do you need off process values? That is, might one process ask for a > value that is not in the local part of the vector and belongs on a > different process? If you know that the values accessed are always local > then you can just use sol[k][j][i].vx the k,j,i are the global indices > over the entire mesh. > > If any process might ask for any value on the entire mesh then you > need to use DMDAGlobalToNaturalBegin/End() followed by > VecScatterCreateToAll() and scatter all the values to all processes then > you access them locally. Note this is not scalable. > > Thanks Barry. Any process might ask for any value on the entire mesh since this is a function I need to provide to the part of the code that "sees" my solver as an external object and that part does not use petsc explicity and not necessarily parallel. I did as you suggested and it works, thanks. > > Barry > > Note that though with sol[k][j][i].vx it looks like you can access any > value for k,j,i if you use a k, j, i that do not correspond to a value that > belongs to this process you will get a memory error. > > > > > On Sep 17, 2013, at 11:06 AM, Bishesh Khanal wrote: > > > Dear all, > > I have a vector say sol which is a vector created from a dmda with dof = > 3. > > Now I need to provide an interface to the part of my code which does not > use petsc but needs access to the values of sol. > > > > I was thinking of providing an interface such as: > > double solver::getSolutionAtPosition(int x, int y, int z, int field); > > which should return a value of the vector sol at the grid position > (x,y,z) and dof = field. > > > > I know how to get the array and loop through all the values such as: > > DMDALocalInfo info; > > ierr = DMDAGetLocalInfo(da,&info);CHKERRQ(ierr); > > > > Field ***sol; > > ierr = DMDAVecGetArray(da,b,&rhs);CHKERRQ(ierr); > > > > for (PetscInt k = info.zs; k > for (PetscInt j = info.ys; j < info.ys+info.ym; ++j) { > > for (PetscInt i = info.xs; i < info.xs+info.xm; ++i) { > > //access sol[k][j][i].vx or sol[k][j][i].vy or > sol[k][j][i].vz depending on field > > > > } > > } > > } > > > > But if I have a particular (x,y,z) based on global co-ordinate system, > how do I get the value for the given field ? > > > > Thanks, > > Bishesh > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From reza.yaghmaie2 at gmail.com Thu Sep 19 13:44:38 2013 From: reza.yaghmaie2 at gmail.com (Reza Yaghmaie) Date: Thu, 19 Sep 2013 14:44:38 -0400 Subject: [petsc-users] MATMPIBAIJ Message-ID: Dear Sir/Madam, I am very new to PETSC and I have some issues running my code. I want to create a global M*1 matrix (not a vector since I want to multiply this matrix with an other 1*M matrix and get a M*M matrix and I know that transpose function on vectors does not work so this is why I need to have a M*1 matrix) I have used the below function and it works well for a M*M matrix call MatCreateMPIBAIJ (PETSC_COMM_WORLD,6,6*numpn,6*numpn, & PETSC_DETERMINE, PETSC_DETERMINE, & PETSC_NULL_INTEGER, mr(np(246)), & PETSC_NULL_INTEGER, mr(np(247)), & matdummy2, ierr) I am trying to change it accordingly for the M*1 matrix So I use the following function call MatCreateMPIBAIJ (PETSC_COMM_WORLD,alpha,m,n, & PETSC_DETERMINE, PETSC_DETERMINE, & PETSC_NULL_INTEGER, mr(np(246)), & PETSC_NULL_INTEGER, mr(np(247)), & mvecdummy1, ierr) defining the following parameters respectively: alpha=number of degrees of freedom per node=1 m=each processor number of rows=6*numpn n=local number of columns=1 I think PETSC_DETERMINE represents the global number of rows and columns respectively. I think PETSC_NULL_INTEGER represents the number of nonzero terms in rows in the diagonal matrix and off diagonal matrix respectively and mr(np(246), mr(np(247) show the nnz terms in columns of diagonal and off diagonal matrices. mvecdummy1=the name of the matrix ierr is just an error compliment in the subroutine. The errors that I am getting are d_nnz can not be less than 0 o_nnz can not be less than -1 Please guide me through this issue. Thanks alot, -Reza -------------- next part -------------- An HTML attachment was scrubbed... URL: From petsc-maint at mcs.anl.gov Thu Sep 19 13:49:14 2013 From: petsc-maint at mcs.anl.gov (Matthew Knepley) Date: Thu, 19 Sep 2013 11:49:14 -0700 Subject: [petsc-users] MATMPIBAIJ In-Reply-To: References: Message-ID: On Thu, Sep 19, 2013 at 11:44 AM, Reza Yaghmaie wrote: > > Dear Sir/Madam, > > I am very new to PETSC and I have some issues running my code. > I want to create a global M*1 matrix (not a vector since I want to > multiply this matrix with an other 1*M matrix and get a M*M matrix and I > know that transpose function on vectors does not work so this is why I need > to have a M*1 matrix) > You never ever ever want to do this. This creates a dense matrix from 2 Vecs. You would instead create a MatShell() which holds the two vectors and applies them in sequence in its MatMult(). Matt > I have used the below function and it works well for a M*M matrix > call MatCreateMPIBAIJ (PETSC_COMM_WORLD,6,6*numpn,6*numpn, > & PETSC_DETERMINE, PETSC_DETERMINE, > & PETSC_NULL_INTEGER, mr(np(246)), > & PETSC_NULL_INTEGER, mr(np(247)), > & matdummy2, ierr) > > > I am trying to change it accordingly for the M*1 matrix > > So I use the following function > > call MatCreateMPIBAIJ (PETSC_COMM_WORLD,alpha,m,n, > & PETSC_DETERMINE, PETSC_DETERMINE, > & PETSC_NULL_INTEGER, mr(np(246)), > & PETSC_NULL_INTEGER, mr(np(247)), > & mvecdummy1, ierr) > > defining the following parameters respectively: > alpha=number of degrees of freedom per node=1 > m=each processor number of rows=6*numpn > n=local number of columns=1 > I think PETSC_DETERMINE represents the global number of rows and columns > respectively. > I think PETSC_NULL_INTEGER represents the number of nonzero terms in rows > in the diagonal matrix and off diagonal matrix respectively and mr(np(246), > mr(np(247) show the nnz terms in columns of diagonal and off diagonal > matrices. > mvecdummy1=the name of the matrix > ierr is just an error compliment in the subroutine. > > > The errors that I am getting are > d_nnz can not be less than 0 > o_nnz can not be less than -1 > > Please guide me through this issue. > > Thanks alot, > -Reza > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From reza.yaghmaie2 at gmail.com Thu Sep 19 14:01:00 2013 From: reza.yaghmaie2 at gmail.com (Reza Yaghmaie) Date: Thu, 19 Sep 2013 15:01:00 -0400 Subject: [petsc-users] MATMPIBAIJ In-Reply-To: References: Message-ID: Thanks Matthew for the reply. It would be time consuming for me to implement an other way for creating a matrix and doing matmult and I am not concerned with how much time this will take. So is there any way which you could possibly guide me through using matmpibaij? Thanks alot, -Reza On Thu, Sep 19, 2013 at 2:44 PM, Reza Yaghmaie wrote: > > Dear Sir/Madam, > > I am very new to PETSC and I have some issues running my code. > I want to create a global M*1 matrix (not a vector since I want to > multiply this matrix with an other 1*M matrix and get a M*M matrix and I > know that transpose function on vectors does not work so this is why I need > to have a M*1 matrix) > > I have used the below function and it works well for a M*M matrix > call MatCreateMPIBAIJ (PETSC_COMM_WORLD,6,6*numpn,6*numpn, > & PETSC_DETERMINE, PETSC_DETERMINE, > & PETSC_NULL_INTEGER, mr(np(246)), > & PETSC_NULL_INTEGER, mr(np(247)), > & matdummy2, ierr) > > > I am trying to change it accordingly for the M*1 matrix > > So I use the following function > > call MatCreateMPIBAIJ (PETSC_COMM_WORLD,alpha,m,n, > & PETSC_DETERMINE, PETSC_DETERMINE, > & PETSC_NULL_INTEGER, mr(np(246)), > & PETSC_NULL_INTEGER, mr(np(247)), > & mvecdummy1, ierr) > > defining the following parameters respectively: > alpha=number of degrees of freedom per node=1 > m=each processor number of rows=6*numpn > n=local number of columns=1 > I think PETSC_DETERMINE represents the global number of rows and columns > respectively. > I think PETSC_NULL_INTEGER represents the number of nonzero terms in rows > in the diagonal matrix and off diagonal matrix respectively and mr(np(246), > mr(np(247) show the nnz terms in columns of diagonal and off diagonal > matrices. > mvecdummy1=the name of the matrix > ierr is just an error compliment in the subroutine. > > > The errors that I am getting are > d_nnz can not be less than 0 > o_nnz can not be less than -1 > > Please guide me through this issue. > > Thanks alot, > -Reza > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Sep 19 14:12:03 2013 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 19 Sep 2013 12:12:03 -0700 Subject: [petsc-users] MATMPIBAIJ In-Reply-To: References: Message-ID: On Thu, Sep 19, 2013 at 12:01 PM, Reza Yaghmaie wrote: > > Thanks Matthew for the reply. > It would be time consuming for me to implement an other way for creating a > matrix and doing matmult and I am not concerned with how much time this > will take. > So is there any way which you could possibly guide me through using > matmpibaij? > Whatever you want to do for the parallel outer product will take longer and be worse than using MatShell. I am not saying this lightly. Matt > Thanks alot, > -Reza > > > > > > > > > > > > On Thu, Sep 19, 2013 at 2:44 PM, Reza Yaghmaie wrote: > >> >> Dear Sir/Madam, >> >> I am very new to PETSC and I have some issues running my code. >> I want to create a global M*1 matrix (not a vector since I want to >> multiply this matrix with an other 1*M matrix and get a M*M matrix and I >> know that transpose function on vectors does not work so this is why I need >> to have a M*1 matrix) >> >> I have used the below function and it works well for a M*M matrix >> call MatCreateMPIBAIJ (PETSC_COMM_WORLD,6,6*numpn,6*numpn, >> & PETSC_DETERMINE, PETSC_DETERMINE, >> & PETSC_NULL_INTEGER, mr(np(246)), >> & PETSC_NULL_INTEGER, mr(np(247)), >> & matdummy2, ierr) >> >> >> I am trying to change it accordingly for the M*1 matrix >> >> So I use the following function >> >> call MatCreateMPIBAIJ (PETSC_COMM_WORLD,alpha,m,n, >> & PETSC_DETERMINE, PETSC_DETERMINE, >> & PETSC_NULL_INTEGER, mr(np(246)), >> & PETSC_NULL_INTEGER, mr(np(247)), >> & mvecdummy1, ierr) >> >> defining the following parameters respectively: >> alpha=number of degrees of freedom per node=1 >> m=each processor number of rows=6*numpn >> n=local number of columns=1 >> I think PETSC_DETERMINE represents the global number of rows and columns >> respectively. >> I think PETSC_NULL_INTEGER represents the number of nonzero terms in rows >> in the diagonal matrix and off diagonal matrix respectively and mr(np(246), >> mr(np(247) show the nnz terms in columns of diagonal and off diagonal >> matrices. >> mvecdummy1=the name of the matrix >> ierr is just an error compliment in the subroutine. >> >> >> The errors that I am getting are >> d_nnz can not be less than 0 >> o_nnz can not be less than -1 >> >> Please guide me through this issue. >> >> Thanks alot, >> -Reza >> >> >> >> >> >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From rlmackie862 at gmail.com Thu Sep 19 14:15:33 2013 From: rlmackie862 at gmail.com (Randall Mackie) Date: Thu, 19 Sep 2013 12:15:33 -0700 Subject: [petsc-users] MATMPIBAIJ In-Reply-To: References: Message-ID: <064C27D4-553F-4F6A-80B4-BF6C5DFE9551@gmail.com> I'm not sure I understand your statement about matrix shells being time consuming to implement. They are easy, see this example: http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex14f.F.html Randy M. On Sep 19, 2013, at 12:01 PM, Reza Yaghmaie wrote: > > Thanks Matthew for the reply. > It would be time consuming for me to implement an other way for creating a matrix and doing matmult and I am not concerned with how much time this will take. > So is there any way which you could possibly guide me through using matmpibaij? > > Thanks alot, > -Reza > > > > > > > > > > > > On Thu, Sep 19, 2013 at 2:44 PM, Reza Yaghmaie wrote: > > Dear Sir/Madam, > > I am very new to PETSC and I have some issues running my code. > I want to create a global M*1 matrix (not a vector since I want to multiply this matrix with an other 1*M matrix and get a M*M matrix and I know that transpose function on vectors does not work so this is why I need to have a M*1 matrix) > > I have used the below function and it works well for a M*M matrix > call MatCreateMPIBAIJ (PETSC_COMM_WORLD,6,6*numpn,6*numpn, > & PETSC_DETERMINE, PETSC_DETERMINE, > & PETSC_NULL_INTEGER, mr(np(246)), > & PETSC_NULL_INTEGER, mr(np(247)), > & matdummy2, ierr) > > > I am trying to change it accordingly for the M*1 matrix > > So I use the following function > > call MatCreateMPIBAIJ (PETSC_COMM_WORLD,alpha,m,n, > & PETSC_DETERMINE, PETSC_DETERMINE, > & PETSC_NULL_INTEGER, mr(np(246)), > & PETSC_NULL_INTEGER, mr(np(247)), > & mvecdummy1, ierr) > > defining the following parameters respectively: > alpha=number of degrees of freedom per node=1 > m=each processor number of rows=6*numpn > n=local number of columns=1 > I think PETSC_DETERMINE represents the global number of rows and columns respectively. > I think PETSC_NULL_INTEGER represents the number of nonzero terms in rows in the diagonal matrix and off diagonal matrix respectively and mr(np(246), mr(np(247) show the nnz terms in columns of diagonal and off diagonal matrices. > mvecdummy1=the name of the matrix > ierr is just an error compliment in the subroutine. > > > The errors that I am getting are > d_nnz can not be less than 0 > o_nnz can not be less than -1 > > Please guide me through this issue. > > Thanks alot, > -Reza > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fangxingjun0319 at gmail.com Thu Sep 19 22:10:11 2013 From: fangxingjun0319 at gmail.com (Frank) Date: Thu, 19 Sep 2013 22:10:11 -0500 Subject: [petsc-users] Dense matrix solver Message-ID: <523BBC93.4040704@gmail.com> Hi, I am thinking of solving a linear equations, whose coefficient matrix has 27 nonzero diagonal bands(diagonal dominant). Does any body have any idea about how this will perform? Do you have any recommendation about which solver to choose? I have solved a 11 nonzero diagonal bands matrix equation with "boomeramg", which is 50% solwer than 7 nonzero diagonal bands. Thanks. From jedbrown at mcs.anl.gov Thu Sep 19 22:30:49 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 19 Sep 2013 22:30:49 -0500 Subject: [petsc-users] Dense matrix solver In-Reply-To: <523BBC93.4040704@gmail.com> References: <523BBC93.4040704@gmail.com> Message-ID: <87zjr8w4nq.fsf@mcs.anl.gov> Frank writes: > Hi, > > I am thinking of solving a linear equations, whose coefficient matrix > has 27 nonzero diagonal bands(diagonal dominant). Does any body have any > idea about how this will perform? Do you have any recommendation about > which solver to choose? How far away from the diagonal are those bands, as a function of the grid size (if relevant)? http://scicomp.stackexchange.com/a/880/119 http://scicomp.stackexchange.com/a/869/119 > I have solved a 11 nonzero diagonal bands matrix equation with > "boomeramg", which is 50% solwer than 7 nonzero diagonal bands. This doesn't mean much of anything. Send -log_summary if you have performance questions. Why does the subject of this email say "dense"? -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From knepley at gmail.com Fri Sep 20 05:41:10 2013 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 20 Sep 2013 03:41:10 -0700 Subject: [petsc-users] Dense matrix solver In-Reply-To: <87zjr8w4nq.fsf@mcs.anl.gov> References: <523BBC93.4040704@gmail.com> <87zjr8w4nq.fsf@mcs.anl.gov> Message-ID: On Thu, Sep 19, 2013 at 8:30 PM, Jed Brown wrote: > Frank writes: > > > Hi, > > > > I am thinking of solving a linear equations, whose coefficient matrix > > has 27 nonzero diagonal bands(diagonal dominant). Does any body have any > > idea about how this will perform? Do you have any recommendation about > > which solver to choose? > > How far away from the diagonal are those bands, as a function of the > grid size (if relevant)? > > http://scicomp.stackexchange.com/a/880/119 > http://scicomp.stackexchange.com/a/869/119 If your matrix/preconditioner has bandwidth K which is small and is diagonally dominant, you can solve it efficiently using http://en.wikipedia.org/wiki/SPIKE_algorithm. Thanks, Matt > > I have solved a 11 nonzero diagonal bands matrix equation with > > "boomeramg", which is 50% solwer than 7 nonzero diagonal bands. > > This doesn't mean much of anything. Send -log_summary if you have > performance questions. > > Why does the subject of this email say "dense"? > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Sep 20 08:17:19 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 20 Sep 2013 08:17:19 -0500 Subject: [petsc-users] PETSc-dev and OpenMP In-Reply-To: <5238AE11.6050400@gmail.com> References: <52389171.1090600@gmail.com> <7AB30D0B-6ADA-427A-9CDE-A8D6B84E04E8@mcs.anl.gov> <5238AE11.6050400@gmail.com> Message-ID: Sorry I did not respond sooner. I've been trying to interface PETSc to superlu_mt for a couple of days but haven't completed it yet. I'll let you know if I get it working. Barry On Sep 17, 2013, at 2:31 PM, Danyang Su wrote: > Hi Barry, > > Thanks so much. > Which solvers are current OpenMP parallel currently? Can I use the external package (e.g., SuperLU OpenMP version) with OpenMP through PETSc? > Is there any document or examples available for the usage of PETSc with OpenMP? > > Thanks again, > > Danyang > > On 17/09/2013 11:21 AM, Barry Smith wrote: >> You can utilize the "OpenMP version of PETSc" with your OpenMP program but not all the solvers are OpenMP parallel currently. >> >> Use petsc-dev and configure PETSc with ./configure --with-mpi=0 --with-openmp see http://www.mcs.anl.gov/petsc/developers/index.html >> >> Barry >> >> >> On Sep 17, 2013, at 12:29 PM, Danyang Su wrote: >> >>> Hi All, >>> >>> I have a parallel Fortran project utilizing OpenMP parallelism. Before porting to PETSc utilizing MPICH, I want to know if it is possible to use PETSc solver with OpenMP. There is some information for petsc-dev openmp benchmarking as can be found here, http://lists.mcs.anl.gov/pipermail/petsc-dev/2012-March/007817.html. If this is possible, I would like to have a OpenMP version of PETSc solver first, otherwise, the project will be ported to PETSc utilizing MPI. Could anybody give me some advice on this? >>> >>> Thanks and regards, >>> >>> Danyang > From danyang.su at gmail.com Fri Sep 20 17:01:07 2013 From: danyang.su at gmail.com (Danyang Su) Date: Fri, 20 Sep 2013 15:01:07 -0700 Subject: [petsc-users] PETSc-dev and OpenMP In-Reply-To: References: <52389171.1090600@gmail.com> <7AB30D0B-6ADA-427A-9CDE-A8D6B84E04E8@mcs.anl.gov> <5238AE11.6050400@gmail.com> Message-ID: <523CC5A3.6010106@gmail.com> Hi Barry, Thanks for following up. I just want some PETSc OpenMP example or user guide. Looking forward to your work on superlu_mt. Regards, Danyang On 20/09/2013 6:17 AM, Barry Smith wrote: > Sorry I did not respond sooner. I've been trying to interface PETSc to superlu_mt for a couple of days but haven't completed it yet. I'll let you know if I get it working. > > Barry > > On Sep 17, 2013, at 2:31 PM, Danyang Su wrote: > >> Hi Barry, >> >> Thanks so much. >> Which solvers are current OpenMP parallel currently? Can I use the external package (e.g., SuperLU OpenMP version) with OpenMP through PETSc? >> Is there any document or examples available for the usage of PETSc with OpenMP? >> >> Thanks again, >> >> Danyang >> >> On 17/09/2013 11:21 AM, Barry Smith wrote: >>> You can utilize the "OpenMP version of PETSc" with your OpenMP program but not all the solvers are OpenMP parallel currently. >>> >>> Use petsc-dev and configure PETSc with ./configure --with-mpi=0 --with-openmp see http://www.mcs.anl.gov/petsc/developers/index.html >>> >>> Barry >>> >>> >>> On Sep 17, 2013, at 12:29 PM, Danyang Su wrote: >>> >>>> Hi All, >>>> >>>> I have a parallel Fortran project utilizing OpenMP parallelism. Before porting to PETSc utilizing MPICH, I want to know if it is possible to use PETSc solver with OpenMP. There is some information for petsc-dev openmp benchmarking as can be found here, http://lists.mcs.anl.gov/pipermail/petsc-dev/2012-March/007817.html. If this is possible, I would like to have a OpenMP version of PETSc solver first, otherwise, the project will be ported to PETSc utilizing MPI. Could anybody give me some advice on this? >>>> >>>> Thanks and regards, >>>> >>>> Danyang From mrosso at uci.edu Fri Sep 20 17:55:30 2013 From: mrosso at uci.edu (Michele Rosso) Date: Fri, 20 Sep 2013 15:55:30 -0700 Subject: [petsc-users] GAMG speed In-Reply-To: <874nasld2x.fsf@mcs.anl.gov> References: <51FAA56D.60106@uci.edu> <51FAE69E.4020108@uci.edu> <51FAEF43.2050207@uci.edu> <4B76E0BB-D8A1-4F38-A8D1-0174801E0AD9@mcs.anl.gov> <51FC2A38.5000000@uci.edu> <406E1FE4-5CE5-4840-BD77-DDB5830AD520@mcs.anl.gov> <520A88F6.9070603@uci.edu> <520A9815.7030400@mcs.anl.gov> <520ABC8E.3040204@uci.edu> <520AC9DE.1050508@uci.edu> <520ADCA4.3030902@uci.edu> <520AE004.1010803@uci.edu> <87y585ko2v.fsf@mcs.anl.gov> <520AF220.3070900@uci.edu> <87k3jpkko7.fsf@mcs.anl.gov> <520B2743.7020405@uci.edu> <874nasld2x.fsf@mcs.anl.gov> Message-ID: <523CD262.5000108@uci.edu> Hi Jed, I set-up and tested the solve for the multi-phase test case (liquid droplet with surface tension falling under the effect of gravity in a quiescent fluid). The solution of the Poisson Equation via multigrid is correct but it becomes progressively slower and slower as the simulation progresses (I am performing successive solves). Could you please help me out in speeding this up? I run the test case with the options: -pc_type mg -pc_mg_galerkin -pc_mg_levels 5 -mg_levels_ksp_type richardson -mg_levels_ksp_max_it 1 -mg_coarse_pc_type lu -mg_coarse_pc_factor_mat_solver_package superlu_dist -log_summary -ksp_view -ksp_monitor_true_residual -options_left Please find the diagnostic for the final solve in the attached file "final.txt'. Thank you, Michele On 08/14/2013 04:35 AM, Jed Brown wrote: > Please always use "reply-all" so that your messages go to the list. > This is standard mailing list etiquette. It is important to preserve > threading for people who find this discussion later and so that we do > not waste our time re-answering the same questions that have already > been answered in private side-conversations. You'll likely get an > answer faster that way too. > > Michele Rosso writes: > >> Jed, >> >> thank you very much for the detailed analysis. >> I confirm that >> >> -log_summary -ksp_monitor -ksp_view -ksp_converged_reason -pc_type mg >> -pc_mg_galerkin -pc_mg_levels 5 -mg_levels_ksp_type richardson >> -mg_levels_ksp_max_it 1 >> >> results in a faster solve with the 256^3 grid (it finally beats CG + ICC). >> >> Yes, I perform different solves: I setup matrix and KSP only once at the beginning of the run and then I re-use them >> at each time step. The rhs term changes during the simulation though. For now (single phase flow) the matrix does not change, >> but I will be dealing soon with a multiphase flow and thus not even the matrix values will be constant in time (it will be a variable coefficients Poisson Equation). > Okay, the coefficient variation from the multiphase flow can drastically > change the characteristics of the solve. > >> I need to solve up to the discretization error, so maybe FMG is worth a try. >> The matrix-free approach is appealing given that the Poisson solver is really mission critical (basically it accounts for most of the simulation time). >> I will use the level set method and ghost fluid method in order to account for the discontinuities at the interface between phases: the computation of the matrix and rhs >> values will be influenced by such methods so my only concern is to be sure matrix-free can be used in these circumstances. > Matrix-free can be used in principle, but those problems can be several > orders of magnitude more ill-conditioned, so don't invest any more time > on it right now. Get the discretization set up using assembled > matrices, then go through the options we've tried to find an efficient > solver. The best choice will likely depend on the details of the > formulation, the types of fluids involved, and the geometric > configuration of the fluids. > >> I do not have any prior experience with matrix-free methods so I will have to rely on your assistance for this. >> Thank you very much. >> >> Michele >> >> >> >> >> On 08/13/2013 08:36 PM, Jed Brown wrote: >>> Michele Rosso writes: >>> >>>> Hi Jed, >>>> >>>> I attached the output for both the runs you suggested. At the beginning >>>> of each file I included the options I used. >>>> >>>> On a side note, I tried to run with a grid of 256^3 (exactly as before) >>>> but with less levels, i.e. 3 instead of 4 or 5. >>>> My system stops the run because of an Out Of Memory condition. It is >>>> really odd since I have not changed anything except >>>> - pc_mg_levels. I cannot send you any output since there is none. Do >>>> you have any guess where the problem comes from? >>> The selected algorithm does a direct solve on the coarse grid. Each >>> time you reduce the number of levels, the coarse grid size grows by a >>> factor of 8. Going from 5 to 3 levels is going from a 16^3 coarse grid >>> to a 64^3 coarse grid. Applying a direct solver to the latter ends up >>> using a lot of memory. I think this is not worth bothering with and it >>> might even be (slightly) faster to use 6 levels. That is not where the >>> time is being spent. >>> >>>> -log_summary -ksp_monitor -ksp_view -ksp_converged_reason -pc_type mg >>>> -pc_mg_galerkin -pc_mg_levels 5 -mg_levels_ksp_type richardson >>>> -mg_levels_ksp_max_it 1 >>>> >>>> >>>> >>>> 0 KSP Residual norm 3.653965664551e-05 >>>> 1 KSP Residual norm 1.910638846094e-06 >>>> 2 KSP Residual norm 8.690440116045e-08 >>>> 3 KSP Residual norm 3.732213639394e-09 >>>> 4 KSP Residual norm 1.964855338020e-10 >>> This converges well. >>> >>>> Max Max/Min Avg Total >>>> Time (sec): 4.048e+00 1.00012 4.048e+00 >>>> Objects: 2.490e+02 1.00000 2.490e+02 >>>> Flops: 2.663e+08 1.00000 2.663e+08 2.130e+09 >>>> Flops/sec: 6.579e+07 1.00012 6.579e+07 5.263e+08 >>>> MPI Messages: 6.820e+02 1.00000 6.820e+02 5.456e+03 >>>> MPI Message Lengths: 8.245e+06 1.00000 1.209e+04 6.596e+07 >>>> MPI Reductions: 4.580e+02 1.00000 >>>> VecTDot 12 1.0 2.9428e-02 1.2 6.29e+06 1.0 0.0e+00 0.0e+00 1.2e+01 1 2 0 0 3 1 2 0 0 3 1710 >>>> VecNorm 9 1.0 1.0796e-02 1.2 4.72e+06 1.0 0.0e+00 0.0e+00 9.0e+00 0 2 0 0 2 0 2 0 0 2 3497 >>>> VecScale 24 1.0 2.4652e-04 1.1 1.99e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 6442 >>>> VecCopy 3 1.0 5.0740e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> VecSet 116 1.0 1.4349e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> VecAXPY 12 1.0 2.8027e-02 1.0 6.29e+06 1.0 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 1796 >>>> VecAYPX 29 1.0 3.0655e-02 1.4 4.16e+06 1.0 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 1085 >>>> VecScatterBegin 123 1.0 3.5391e-02 1.1 0.00e+00 0.0 3.5e+03 1.2e+04 0.0e+00 1 0 65 66 0 1 0 65 66 0 0 >>>> VecScatterEnd 123 1.0 2.5395e-02 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> MatMult 31 1.0 2.3556e-01 1.0 5.62e+07 1.0 1.0e+03 2.3e+04 0.0e+00 6 21 19 36 0 6 21 19 36 0 1908 >>>> MatMultAdd 24 1.0 5.9044e-02 1.0 1.21e+07 1.0 5.8e+02 2.8e+03 0.0e+00 1 5 11 2 0 1 5 11 2 0 1644 >>>> MatMultTranspose 28 1.0 7.4601e-02 1.1 1.42e+07 1.0 6.7e+02 2.8e+03 0.0e+00 2 5 12 3 0 2 5 12 3 0 1518 >>>> MatSolve 6 1.0 3.8311e-03 1.0 1.44e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 3006 >>>> MatSOR 48 1.0 5.8050e-01 1.0 1.01e+08 1.0 8.6e+02 1.5e+04 4.8e+01 14 38 16 19 10 14 38 16 19 11 1390 >>> Most of the solve time is in MatSOR and MatMult. That's expected since >>> the subdomains are pretty big. >>> >>>> MatLUFactorSym 1 1.0 3.0620e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 1 0 0 0 0 1 0 >>>> MatLUFactorNum 1 1.0 2.4665e-02 1.0 1.95e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 7 0 0 0 1 7 0 0 0 6329 >>>> MatAssemblyBegin 20 1.0 2.4351e-02 6.7 0.00e+00 0.0 0.0e+00 0.0e+00 2.2e+01 0 0 0 0 5 0 0 0 0 5 0 >>>> MatAssemblyEnd 20 1.0 1.3176e-01 1.0 0.00e+00 0.0 5.6e+02 2.1e+03 7.2e+01 3 0 10 2 16 3 0 10 2 16 0 >>>> MatGetRowIJ 1 1.0 1.1516e-04 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> MatGetOrdering 1 1.0 4.1008e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> MatView 16 1.3 1.0209e-03 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01 0 0 0 0 3 0 0 0 0 3 0 >>>> MatPtAP 4 1.0 6.4001e-01 1.0 4.06e+07 1.0 1.1e+03 1.7e+04 1.0e+02 16 15 21 30 22 16 15 21 30 22 507 >>> MatPtAP dominates the setup time. For profiling, you could register a >>> stage (PetscLogStageRegister) and time the setup separately from the >>> solve. >>> >>>> MatPtAPSymbolic 4 1.0 3.7003e-01 1.0 0.00e+00 0.0 7.2e+02 2.0e+04 6.0e+01 9 0 13 22 13 9 0 13 22 13 0 >>>> MatPtAPNumeric 4 1.0 2.7004e-01 1.0 4.06e+07 1.0 4.2e+02 1.2e+04 4.0e+01 7 15 8 8 9 7 15 8 8 9 1202 >>>> MatGetRedundant 1 1.0 7.9393e-04 1.0 0.00e+00 0.0 1.7e+02 7.1e+03 4.0e+00 0 0 3 2 1 0 0 3 2 1 0 >>>> MatGetLocalMat 4 1.0 3.9521e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00 1 0 0 0 2 1 0 0 0 2 0 >>>> MatGetBrAoCol 4 1.0 1.7719e-02 1.0 0.00e+00 0.0 4.3e+02 2.7e+04 8.0e+00 0 0 8 18 2 0 0 8 18 2 0 >>>> MatGetSymTrans 8 1.0 1.3007e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> KSPSetUp 7 1.0 1.3097e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.4e+01 0 0 0 0 5 0 0 0 0 5 0 >>>> KSPSolve 2 1.0 1.0450e+00 1.0 2.04e+08 1.0 3.4e+03 1.2e+04 7.5e+01 26 77 62 60 16 26 77 62 60 16 1563 >>>> PCSetUp 1 1.0 8.6248e-01 1.0 6.21e+07 1.0 1.9e+03 1.1e+04 3.2e+02 21 23 35 32 69 21 23 35 32 69 576 >>>> PCApply 6 1.0 8.4384e-01 1.0 1.61e+08 1.0 3.2e+03 9.0e+03 4.8e+01 21 60 59 44 10 21 60 59 44 11 1523 >>> Do you know why there are 6 PCApply events? With four iterations of the >>> Krylov method, there should be only 5 events. Oh, it looks like you do >>> two solves. Is one of those with a different system? >>> >>> Recall that the old KSPSolve time was over 3.35 seconds. >>> >>>> -log_summary -ksp_monitor -ksp_view -ksp_converged_reason -pc_type mg >>>> -pc_mg_galerkin -pc_mg_levels 5 -mg_levels_ksp_type richardson >>>> -mg_levels_ksp_max_it 1 -pc_mg_type full >>>> >>>> 0 KSP Residual norm 3.654533581988e-05 >>>> 1 KSP Residual norm 8.730776244351e-07 >>>> 2 KSP Residual norm 3.474626061661e-08 >>>> 3 KSP Residual norm 1.813665557493e-09 >>> This converges slightly faster, but ends up not paying off. >>> >>>> Time (sec): 4.261e+00 1.00012 4.261e+00 >>>> Objects: 2.950e+02 1.00000 2.950e+02 >>>> Flops: 3.322e+08 1.00000 3.322e+08 2.658e+09 >>>> Flops/sec: 7.797e+07 1.00012 7.796e+07 6.237e+08 >>>> MPI Messages: 1.442e+03 1.00000 1.442e+03 1.154e+04 >>>> MPI Message Lengths: 1.018e+07 1.00000 7.057e+03 8.141e+07 >>>> MPI Reductions: 5.460e+02 1.00000 >>> More messages, more work, etc., so not better. >>> >>>> KSPSolve 2 1.0 1.2287e+00 1.0 2.70e+08 1.0 9.5e+03 5.8e+03 1.6e+02 29 81 82 68 30 29 81 82 68 30 1758 >>>> PCSetUp 1 1.0 8.6414e-01 1.0 6.21e+07 1.0 1.9e+03 1.1e+04 3.2e+02 20 19 17 26 58 20 19 17 26 58 575 >>>> PCApply 5 1.0 1.0571e+00 1.0 2.33e+08 1.0 9.3e+03 4.9e+03 1.4e+02 24 70 81 56 26 24 70 81 56 26 1764 >>> It's still entirely possible that you can make Full MG beat V-cycles, >>> especially if you only need to converge up to discretization error. By >>> my figures, your good solver takes 12 work units to converge well below >>> discretization error (after Galerkin setup, but maybe you only need to >>> do that once?). If you only need to equal truncation error, this can be >>> brought down to about 5 (probably at best a 2x speedup in parallel). >>> This would involve a high-order (cubic) FMG prolongation. >>> >>> Alternatively, you can speed up the implementation (and significantly >>> reduce memory usage) by creating geometric coarse levels and a >>> matrix-free implementation of MatSOR and MatMult. (The matrices are >>> great for experimenting, but if this solver is mission critical and >>> still a bottleneck, the matrix is an inefficient way to represent the >>> operator since it has very low arithmetic intensity/requires a lot of >>> memory bandwidth.) I predict you can probably speed up the solve by >>> perhaps another factor of 2 with a good matrix-free FMG implementation. >>> Do you want to go down this path? -------------- next part -------------- 0 KSP unpreconditioned resid norm 8.471082897815e-07 true resid norm 8.471082897815e-07 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 5.889629222055e-07 true resid norm 5.889629222055e-07 ||r(i)||/||b|| 6.952628481034e-01 2 KSP unpreconditioned resid norm 4.929221177373e-07 true resid norm 4.929221177373e-07 ||r(i)||/||b|| 5.818879636563e-01 3 KSP unpreconditioned resid norm 4.077087883813e-07 true resid norm 4.077087883813e-07 ||r(i)||/||b|| 4.812947686847e-01 4 KSP unpreconditioned resid norm 3.532777909896e-07 true resid norm 3.532777909896e-07 ||r(i)||/||b|| 4.170397046649e-01 5 KSP unpreconditioned resid norm 3.133843759979e-07 true resid norm 3.133843759979e-07 ||r(i)||/||b|| 3.699460621247e-01 6 KSP unpreconditioned resid norm 2.664884465415e-07 true resid norm 2.664884465415e-07 ||r(i)||/||b|| 3.145860449675e-01 7 KSP unpreconditioned resid norm 2.266816102745e-07 true resid norm 2.266816102745e-07 ||r(i)||/||b|| 2.675946074532e-01 8 KSP unpreconditioned resid norm 1.909025017919e-07 true resid norm 1.909025017919e-07 ||r(i)||/||b|| 2.253578486892e-01 9 KSP unpreconditioned resid norm 1.635648456077e-07 true resid norm 1.635648456077e-07 ||r(i)||/||b|| 1.930861113989e-01 10 KSP unpreconditioned resid norm 1.434980659348e-07 true resid norm 1.434980659348e-07 ||r(i)||/||b|| 1.693975465307e-01 11 KSP unpreconditioned resid norm 1.271661365357e-07 true resid norm 1.271661365357e-07 ||r(i)||/||b|| 1.501179224306e-01 12 KSP unpreconditioned resid norm 1.132794401526e-07 true resid norm 1.132794401526e-07 ||r(i)||/||b|| 1.337248631835e-01 13 KSP unpreconditioned resid norm 1.002788428906e-07 true resid norm 1.002788428906e-07 ||r(i)||/||b|| 1.183778320909e-01 14 KSP unpreconditioned resid norm 8.753317847137e-08 true resid norm 8.753317847137e-08 ||r(i)||/||b|| 1.033317458078e-01 15 KSP unpreconditioned resid norm 7.770742874496e-08 true resid norm 7.770742874496e-08 ||r(i)||/||b|| 9.173257974492e-02 16 KSP unpreconditioned resid norm 6.961935247483e-08 true resid norm 6.961935247483e-08 ||r(i)||/||b|| 8.218471394346e-02 17 KSP unpreconditioned resid norm 6.471790768651e-08 true resid norm 6.471790768651e-08 ||r(i)||/||b|| 7.639862396248e-02 18 KSP unpreconditioned resid norm 6.162477500458e-08 true resid norm 6.162477500458e-08 ||r(i)||/||b|| 7.274722222407e-02 19 KSP unpreconditioned resid norm 6.116147101803e-08 true resid norm 6.116147101803e-08 ||r(i)||/||b|| 7.220029806792e-02 20 KSP unpreconditioned resid norm 6.307214323139e-08 true resid norm 6.307214323139e-08 ||r(i)||/||b|| 7.445582104698e-02 21 KSP unpreconditioned resid norm 6.569260578724e-08 true resid norm 6.569260578724e-08 ||r(i)||/||b|| 7.754924202688e-02 22 KSP unpreconditioned resid norm 7.002116374918e-08 true resid norm 7.002116374918e-08 ||r(i)||/||b|| 8.265904677576e-02 23 KSP unpreconditioned resid norm 7.648471462197e-08 true resid norm 7.648471462197e-08 ||r(i)||/||b|| 9.028918208521e-02 24 KSP unpreconditioned resid norm 8.218354530044e-08 true resid norm 8.218354530044e-08 ||r(i)||/||b|| 9.701657543883e-02 25 KSP unpreconditioned resid norm 8.777006719029e-08 true resid norm 8.777006719029e-08 ||r(i)||/||b|| 1.036113897704e-01 26 KSP unpreconditioned resid norm 9.074938460336e-08 true resid norm 9.074938460336e-08 ||r(i)||/||b|| 1.071284341070e-01 27 KSP unpreconditioned resid norm 9.311175500020e-08 true resid norm 9.311175500020e-08 ||r(i)||/||b|| 1.099171807470e-01 28 KSP unpreconditioned resid norm 9.391328578136e-08 true resid norm 9.391328578136e-08 ||r(i)||/||b|| 1.108633771080e-01 29 KSP unpreconditioned resid norm 9.397746582366e-08 true resid norm 9.397746582366e-08 ||r(i)||/||b|| 1.109391407891e-01 30 KSP unpreconditioned resid norm 9.269761294239e-08 true resid norm 9.269761294239e-08 ||r(i)||/||b|| 1.094282915899e-01 31 KSP unpreconditioned resid norm 9.037592803258e-08 true resid norm 9.037592803258e-08 ||r(i)||/||b|| 1.066875736228e-01 32 KSP unpreconditioned resid norm 8.739586906294e-08 true resid norm 8.739586906294e-08 ||r(i)||/||b|| 1.031696538886e-01 33 KSP unpreconditioned resid norm 8.294737580326e-08 true resid norm 8.294737580326e-08 ||r(i)||/||b|| 9.791826712575e-02 34 KSP unpreconditioned resid norm 7.825811344542e-08 true resid norm 7.825811344542e-08 ||r(i)||/||b|| 9.238265566450e-02 35 KSP unpreconditioned resid norm 7.370584204445e-08 true resid norm 7.370584204445e-08 ||r(i)||/||b|| 8.700876019459e-02 36 KSP unpreconditioned resid norm 6.944350218542e-08 true resid norm 6.944350218542e-08 ||r(i)||/||b|| 8.197712503006e-02 37 KSP unpreconditioned resid norm 6.611084651652e-08 true resid norm 6.611084651652e-08 ||r(i)||/||b|| 7.804296961086e-02 38 KSP unpreconditioned resid norm 6.155389358744e-08 true resid norm 6.155389358744e-08 ||r(i)||/||b|| 7.266354765968e-02 39 KSP unpreconditioned resid norm 5.877895869357e-08 true resid norm 5.877895869357e-08 ||r(i)||/||b|| 6.938777415191e-02 40 KSP unpreconditioned resid norm 5.544264371508e-08 true resid norm 5.544264371508e-08 ||r(i)||/||b|| 6.544929896671e-02 41 KSP unpreconditioned resid norm 5.273241362566e-08 true resid norm 5.273241362566e-08 ||r(i)||/||b|| 6.224990861471e-02 42 KSP unpreconditioned resid norm 5.119250885272e-08 true resid norm 5.119250885272e-08 ||r(i)||/||b|| 6.043207163741e-02 43 KSP unpreconditioned resid norm 4.982771093310e-08 true resid norm 4.982771093310e-08 ||r(i)||/||b|| 5.882094595716e-02 44 KSP unpreconditioned resid norm 4.811237834099e-08 true resid norm 4.811237834099e-08 ||r(i)||/||b|| 5.679601878692e-02 45 KSP unpreconditioned resid norm 4.628304375229e-08 true resid norm 4.628304375229e-08 ||r(i)||/||b|| 5.463651378530e-02 46 KSP unpreconditioned resid norm 4.476871541552e-08 true resid norm 4.476871541552e-08 ||r(i)||/||b|| 5.284886944864e-02 47 KSP unpreconditioned resid norm 4.419875774021e-08 true resid norm 4.419875774021e-08 ||r(i)||/||b|| 5.217604204016e-02 48 KSP unpreconditioned resid norm 4.359181679552e-08 true resid norm 4.359181679552e-08 ||r(i)||/||b|| 5.145955637710e-02 49 KSP unpreconditioned resid norm 4.304314318039e-08 true resid norm 4.304314318039e-08 ||r(i)||/||b|| 5.081185451685e-02 50 KSP unpreconditioned resid norm 4.347264822128e-08 true resid norm 4.347264822128e-08 ||r(i)||/||b|| 5.131887946993e-02 51 KSP unpreconditioned resid norm 4.378084979329e-08 true resid norm 4.378084979329e-08 ||r(i)||/||b|| 5.168270730132e-02 52 KSP unpreconditioned resid norm 4.459872124486e-08 true resid norm 4.459872124486e-08 ||r(i)||/||b|| 5.264819360505e-02 53 KSP unpreconditioned resid norm 4.586835225867e-08 true resid norm 4.586835225867e-08 ||r(i)||/||b|| 5.414697602652e-02 54 KSP unpreconditioned resid norm 4.759826693288e-08 true resid norm 4.759826693288e-08 ||r(i)||/||b|| 5.618911714955e-02 55 KSP unpreconditioned resid norm 4.951983455569e-08 true resid norm 4.951983455569e-08 ||r(i)||/||b|| 5.845750201366e-02 56 KSP unpreconditioned resid norm 5.077348189601e-08 true resid norm 5.077348189601e-08 ||r(i)||/||b|| 5.993741592247e-02 57 KSP unpreconditioned resid norm 5.187532881197e-08 true resid norm 5.187532881197e-08 ||r(i)||/||b|| 6.123813146174e-02 58 KSP unpreconditioned resid norm 5.194414285622e-08 true resid norm 5.194414285622e-08 ||r(i)||/||b|| 6.131936552011e-02 59 KSP unpreconditioned resid norm 5.114127723640e-08 true resid norm 5.114127723640e-08 ||r(i)||/||b|| 6.037159339994e-02 60 KSP unpreconditioned resid norm 4.987798575055e-08 true resid norm 4.987798575055e-08 ||r(i)||/||b|| 5.888029470638e-02 61 KSP unpreconditioned resid norm 4.837751000107e-08 true resid norm 4.837751000107e-08 ||r(i)||/||b|| 5.710900316363e-02 62 KSP unpreconditioned resid norm 4.735998413707e-08 true resid norm 4.735998413707e-08 ||r(i)||/||b|| 5.590782749782e-02 63 KSP unpreconditioned resid norm 4.609537746550e-08 true resid norm 4.609537746550e-08 ||r(i)||/||b|| 5.441497624511e-02 64 KSP unpreconditioned resid norm 4.470951308242e-08 true resid norm 4.470951308242e-08 ||r(i)||/||b|| 5.277898188667e-02 65 KSP unpreconditioned resid norm 4.325085133153e-08 true resid norm 4.325085133153e-08 ||r(i)||/||b|| 5.105705120970e-02 66 KSP unpreconditioned resid norm 4.181200034623e-08 true resid norm 4.181200034623e-08 ||r(i)||/||b|| 4.935850687640e-02 67 KSP unpreconditioned resid norm 4.054904820539e-08 true resid norm 4.054904820539e-08 ||r(i)||/||b|| 4.786760877508e-02 68 KSP unpreconditioned resid norm 3.941638106292e-08 true resid norm 3.941638106292e-08 ||r(i)||/||b|| 4.653051037086e-02 69 KSP unpreconditioned resid norm 3.818351979063e-08 true resid norm 3.818351979063e-08 ||r(i)||/||b|| 4.507513413720e-02 70 KSP unpreconditioned resid norm 3.732269485449e-08 true resid norm 3.732269485449e-08 ||r(i)||/||b|| 4.405894181973e-02 71 KSP unpreconditioned resid norm 3.638709503712e-08 true resid norm 3.638709503712e-08 ||r(i)||/||b|| 4.295447875562e-02 72 KSP unpreconditioned resid norm 3.507340393789e-08 true resid norm 3.507340393789e-08 ||r(i)||/||b|| 4.140368399292e-02 73 KSP unpreconditioned resid norm 3.378527900460e-08 true resid norm 3.378527900460e-08 ||r(i)||/||b|| 3.988306974699e-02 74 KSP unpreconditioned resid norm 3.213894671931e-08 true resid norm 3.213894671931e-08 ||r(i)||/||b|| 3.793959651558e-02 75 KSP unpreconditioned resid norm 3.071349886547e-08 true resid norm 3.071349886547e-08 ||r(i)||/||b|| 3.625687439960e-02 76 KSP unpreconditioned resid norm 2.982607199289e-08 true resid norm 2.982607199289e-08 ||r(i)||/||b|| 3.520927885216e-02 77 KSP unpreconditioned resid norm 2.896588396864e-08 true resid norm 2.896588396864e-08 ||r(i)||/||b|| 3.419383840065e-02 78 KSP unpreconditioned resid norm 2.849350659098e-08 true resid norm 2.849350659098e-08 ||r(i)||/||b|| 3.363620322772e-02 79 KSP unpreconditioned resid norm 2.839644438060e-08 true resid norm 2.839644438060e-08 ||r(i)||/||b|| 3.352162258726e-02 80 KSP unpreconditioned resid norm 2.878125026793e-08 true resid norm 2.878125026793e-08 ||r(i)||/||b|| 3.397588078775e-02 81 KSP unpreconditioned resid norm 2.927107179400e-08 true resid norm 2.927107179400e-08 ||r(i)||/||b|| 3.455410854444e-02 82 KSP unpreconditioned resid norm 2.995930520842e-08 true resid norm 2.995930520842e-08 ||r(i)||/||b|| 3.536655888016e-02 83 KSP unpreconditioned resid norm 3.045914214927e-08 true resid norm 3.045914214927e-08 ||r(i)||/||b|| 3.595660969995e-02 84 KSP unpreconditioned resid norm 3.095556579777e-08 true resid norm 3.095556579777e-08 ||r(i)||/||b|| 3.654263117382e-02 85 KSP unpreconditioned resid norm 3.108374161273e-08 true resid norm 3.108374161273e-08 ||r(i)||/||b|| 3.669394100812e-02 86 KSP unpreconditioned resid norm 3.098332255348e-08 true resid norm 3.098332255348e-08 ||r(i)||/||b|| 3.657539765249e-02 87 KSP unpreconditioned resid norm 3.090888002411e-08 true resid norm 3.090888002411e-08 ||r(i)||/||b|| 3.648751924276e-02 88 KSP unpreconditioned resid norm 3.060254382784e-08 true resid norm 3.060254382784e-08 ||r(i)||/||b|| 3.612589346250e-02 89 KSP unpreconditioned resid norm 3.054287138188e-08 true resid norm 3.054287138188e-08 ||r(i)||/||b|| 3.605545093858e-02 90 KSP unpreconditioned resid norm 3.019326857855e-08 true resid norm 3.019326857855e-08 ||r(i)||/||b|| 3.564274950767e-02 91 KSP unpreconditioned resid norm 2.981415246582e-08 true resid norm 2.981415246582e-08 ||r(i)||/||b|| 3.519520800996e-02 92 KSP unpreconditioned resid norm 2.934425853265e-08 true resid norm 2.934425853265e-08 ||r(i)||/||b|| 3.464050451002e-02 93 KSP unpreconditioned resid norm 2.892868376837e-08 true resid norm 2.892868376837e-08 ||r(i)||/||b|| 3.414992406205e-02 94 KSP unpreconditioned resid norm 2.807160358194e-08 true resid norm 2.807160358194e-08 ||r(i)||/||b|| 3.313815237150e-02 95 KSP unpreconditioned resid norm 2.733196031734e-08 true resid norm 2.733196031734e-08 ||r(i)||/||b|| 3.226501339561e-02 96 KSP unpreconditioned resid norm 2.674172249913e-08 true resid norm 2.674172249913e-08 ||r(i)||/||b|| 3.156824554984e-02 97 KSP unpreconditioned resid norm 2.570861397552e-08 true resid norm 2.570861397552e-08 ||r(i)||/||b|| 3.034867476288e-02 98 KSP unpreconditioned resid norm 2.480290643102e-08 true resid norm 2.480290643102e-08 ||r(i)||/||b|| 2.927949912687e-02 99 KSP unpreconditioned resid norm 2.393472264734e-08 true resid norm 2.393472264734e-08 ||r(i)||/||b|| 2.825461978836e-02 100 KSP unpreconditioned resid norm 2.275435374520e-08 true resid norm 2.275435374520e-08 ||r(i)||/||b|| 2.686121009520e-02 101 KSP unpreconditioned resid norm 2.178902400109e-08 true resid norm 2.178902400109e-08 ||r(i)||/||b|| 2.572165125041e-02 102 KSP unpreconditioned resid norm 2.056965586266e-08 true resid norm 2.056965586266e-08 ||r(i)||/||b|| 2.428220348069e-02 103 KSP unpreconditioned resid norm 1.971345133315e-08 true resid norm 1.971345133315e-08 ||r(i)||/||b|| 2.327146549143e-02 104 KSP unpreconditioned resid norm 1.874079176158e-08 true resid norm 1.874079176158e-08 ||r(i)||/||b|| 2.212325388341e-02 105 KSP unpreconditioned resid norm 1.786435683475e-08 true resid norm 1.786435683475e-08 ||r(i)||/||b|| 2.108863418083e-02 106 KSP unpreconditioned resid norm 1.706457796840e-08 true resid norm 1.706457796840e-08 ||r(i)||/||b|| 2.014450593183e-02 107 KSP unpreconditioned resid norm 1.630473998948e-08 true resid norm 1.630473998948e-08 ||r(i)||/||b|| 1.924752736594e-02 108 KSP unpreconditioned resid norm 1.574283867690e-08 true resid norm 1.574283867690e-08 ||r(i)||/||b|| 1.858421038585e-02 109 KSP unpreconditioned resid norm 1.556968009361e-08 true resid norm 1.556968009361e-08 ||r(i)||/||b|| 1.837979899551e-02 110 KSP unpreconditioned resid norm 1.523973136297e-08 true resid norm 1.523973136297e-08 ||r(i)||/||b|| 1.799029893439e-02 111 KSP unpreconditioned resid norm 1.488557248475e-08 true resid norm 1.488557248475e-08 ||r(i)||/||b|| 1.757221911804e-02 112 KSP unpreconditioned resid norm 1.449864006948e-08 true resid norm 1.449864006948e-08 ||r(i)||/||b|| 1.711545057978e-02 113 KSP unpreconditioned resid norm 1.432126672372e-08 true resid norm 1.432126672372e-08 ||r(i)||/||b|| 1.690606371874e-02 114 KSP unpreconditioned resid norm 1.415386315852e-08 true resid norm 1.415386315852e-08 ||r(i)||/||b|| 1.670844605023e-02 115 KSP unpreconditioned resid norm 1.382237564251e-08 true resid norm 1.382237564251e-08 ||r(i)||/||b|| 1.631712947358e-02 116 KSP unpreconditioned resid norm 1.352938132525e-08 true resid norm 1.352938132525e-08 ||r(i)||/||b|| 1.597125360294e-02 117 KSP unpreconditioned resid norm 1.332023721901e-08 true resid norm 1.332023721901e-08 ||r(i)||/||b|| 1.572436178431e-02 118 KSP unpreconditioned resid norm 1.294810005035e-08 true resid norm 1.294810005035e-08 ||r(i)||/||b|| 1.528505883668e-02 119 KSP unpreconditioned resid norm 1.260835085133e-08 true resid norm 1.260835085133e-08 ||r(i)||/||b|| 1.488398945380e-02 120 KSP unpreconditioned resid norm 1.232815412303e-08 true resid norm 1.232815412303e-08 ||r(i)||/||b|| 1.455322096566e-02 121 KSP unpreconditioned resid norm 1.200898521152e-08 true resid norm 1.200898521152e-08 ||r(i)||/||b|| 1.417644633677e-02 122 KSP unpreconditioned resid norm 1.180686235502e-08 true resid norm 1.180686235502e-08 ||r(i)||/||b|| 1.393784300949e-02 123 KSP unpreconditioned resid norm 1.152359469099e-08 true resid norm 1.152359469099e-08 ||r(i)||/||b|| 1.360344932282e-02 124 KSP unpreconditioned resid norm 1.129846801969e-08 true resid norm 1.129846801969e-08 ||r(i)||/||b|| 1.333769030003e-02 125 KSP unpreconditioned resid norm 1.081947011589e-08 true resid norm 1.081947011589e-08 ||r(i)||/||b|| 1.277223968459e-02 126 KSP unpreconditioned resid norm 1.036954512419e-08 true resid norm 1.036954512419e-08 ||r(i)||/||b|| 1.224110925283e-02 127 KSP unpreconditioned resid norm 9.744507014406e-09 true resid norm 9.744507014406e-09 ||r(i)||/||b|| 1.150326012855e-02 128 KSP unpreconditioned resid norm 9.160148503681e-09 true resid norm 9.160148503681e-09 ||r(i)||/||b|| 1.081343272658e-02 129 KSP unpreconditioned resid norm 8.713396021123e-09 true resid norm 8.713396021123e-09 ||r(i)||/||b|| 1.028604739941e-02 130 KSP unpreconditioned resid norm 8.065538786935e-09 true resid norm 8.065538786935e-09 ||r(i)||/||b|| 9.521260604138e-03 131 KSP unpreconditioned resid norm 7.600710595219e-09 true resid norm 7.600710595219e-09 ||r(i)||/||b|| 8.972537144194e-03 132 KSP unpreconditioned resid norm 7.042499013017e-09 true resid norm 7.042499013017e-09 ||r(i)||/||b|| 8.313575841447e-03 133 KSP unpreconditioned resid norm 6.488293059262e-09 true resid norm 6.488293059262e-09 ||r(i)||/||b|| 7.659343129478e-03 134 KSP unpreconditioned resid norm 6.095008004956e-09 true resid norm 6.095008004956e-09 ||r(i)||/||b|| 7.195075385850e-03 135 KSP unpreconditioned resid norm 5.689629867887e-09 true resid norm 5.689629867887e-09 ||r(i)||/||b|| 6.716531919850e-03 136 KSP unpreconditioned resid norm 5.350918350727e-09 true resid norm 5.350918350727e-09 ||r(i)||/||b|| 6.316687506514e-03 137 KSP unpreconditioned resid norm 5.077782670826e-09 true resid norm 5.077782670826e-09 ||r(i)||/||b|| 5.994254491520e-03 138 KSP unpreconditioned resid norm 4.693456261445e-09 true resid norm 4.693456261445e-09 ||r(i)||/||b|| 5.540562308339e-03 139 KSP unpreconditioned resid norm 4.327021695310e-09 true resid norm 4.327021695310e-09 ||r(i)||/||b|| 5.107991206681e-03 140 KSP unpreconditioned resid norm 4.022960135943e-09 true resid norm 4.022960135943e-09 ||r(i)||/||b|| 4.749050604830e-03 141 KSP unpreconditioned resid norm 3.803745635547e-09 true resid norm 3.803745635547e-09 ||r(i)||/||b|| 4.490270820662e-03 142 KSP unpreconditioned resid norm 3.581362358291e-09 true resid norm 3.581362358291e-09 ||r(i)||/||b|| 4.227750337817e-03 143 KSP unpreconditioned resid norm 3.408132039231e-09 true resid norm 3.408132039231e-09 ||r(i)||/||b|| 4.023254264351e-03 144 KSP unpreconditioned resid norm 3.344832032939e-09 true resid norm 3.344832032939e-09 ||r(i)||/||b|| 3.948529454011e-03 145 KSP unpreconditioned resid norm 3.208107857745e-09 true resid norm 3.208107857745e-09 ||r(i)||/||b|| 3.787128394851e-03 146 KSP unpreconditioned resid norm 3.087343355611e-09 true resid norm 3.087343355611e-09 ||r(i)||/||b|| 3.644567516164e-03 147 KSP unpreconditioned resid norm 3.022538018497e-09 true resid norm 3.022538018497e-09 ||r(i)||/||b|| 3.568065682932e-03 148 KSP unpreconditioned resid norm 2.923230289033e-09 true resid norm 2.923230289033e-09 ||r(i)||/||b|| 3.450834237246e-03 149 KSP unpreconditioned resid norm 2.833109002568e-09 true resid norm 2.833109002568e-09 ||r(i)||/||b|| 3.344447264586e-03 150 KSP unpreconditioned resid norm 2.753952392615e-09 true resid norm 2.753952392615e-09 ||r(i)||/||b|| 3.251003945818e-03 151 KSP unpreconditioned resid norm 2.677124193740e-09 true resid norm 2.677124193740e-09 ||r(i)||/||b|| 3.160309285168e-03 152 KSP unpreconditioned resid norm 2.578412706791e-09 true resid norm 2.578412706791e-09 ||r(i)||/||b|| 3.043781695792e-03 153 KSP unpreconditioned resid norm 2.449680459273e-09 true resid norm 2.449680459273e-09 ||r(i)||/||b|| 2.891815000305e-03 154 KSP unpreconditioned resid norm 2.350739446063e-09 true resid norm 2.350739446063e-09 ||r(i)||/||b|| 2.775016458249e-03 155 KSP unpreconditioned resid norm 2.230331194218e-09 true resid norm 2.230331194218e-09 ||r(i)||/||b|| 2.632876128261e-03 156 KSP unpreconditioned resid norm 2.095716328619e-09 true resid norm 2.095716328619e-09 ||r(i)||/||b|| 2.473965080851e-03 157 KSP unpreconditioned resid norm 1.967063208093e-09 true resid norm 1.967063208093e-09 ||r(i)||/||b|| 2.322091793720e-03 158 KSP unpreconditioned resid norm 1.861768684115e-09 true resid norm 1.861768684115e-09 ||r(i)||/||b|| 2.197793017225e-03 159 KSP unpreconditioned resid norm 1.789027091537e-09 true resid norm 1.789027091537e-09 ||r(i)||/||b|| 2.111922540622e-03 160 KSP unpreconditioned resid norm 1.684922987165e-09 true resid norm 1.684922987165e-09 ||r(i)||/||b|| 1.989029038542e-03 161 KSP unpreconditioned resid norm 1.582609227035e-09 true resid norm 1.582609227035e-09 ||r(i)||/||b|| 1.868249013882e-03 162 KSP unpreconditioned resid norm 1.501597611234e-09 true resid norm 1.501597611234e-09 ||r(i)||/||b|| 1.772615885534e-03 163 KSP unpreconditioned resid norm 1.395819010765e-09 true resid norm 1.395819010765e-09 ||r(i)||/||b|| 1.647745663219e-03 164 KSP unpreconditioned resid norm 1.271906790435e-09 true resid norm 1.271906790435e-09 ||r(i)||/||b|| 1.501468945326e-03 165 KSP unpreconditioned resid norm 1.160968729258e-09 true resid norm 1.160968729258e-09 ||r(i)||/||b|| 1.370508048690e-03 166 KSP unpreconditioned resid norm 1.086312221042e-09 true resid norm 1.086312221042e-09 ||r(i)||/||b|| 1.282377039802e-03 167 KSP unpreconditioned resid norm 1.010733613894e-09 true resid norm 1.010733613894e-09 ||r(i)||/||b|| 1.193157505465e-03 168 KSP unpreconditioned resid norm 9.659506407625e-10 true resid norm 9.659506407625e-10 ||r(i)||/||b|| 1.140291804973e-03 169 KSP unpreconditioned resid norm 9.142920667557e-10 true resid norm 9.142920667557e-10 ||r(i)||/||b|| 1.079309549658e-03 170 KSP unpreconditioned resid norm 8.777007956409e-10 true resid norm 8.777007956409e-10 ||r(i)||/||b|| 1.036114043775e-03 171 KSP unpreconditioned resid norm 8.651938742751e-10 true resid norm 8.651938742751e-10 ||r(i)||/||b|| 1.021349790472e-03 172 KSP unpreconditioned resid norm 8.449669697471e-10 true resid norm 8.449669697471e-10 ||r(i)||/||b|| 9.974722003548e-04 173 KSP unpreconditioned resid norm 8.226753443690e-10 true resid norm 8.226753443690e-10 ||r(i)||/||b|| 9.711572349047e-04 174 KSP unpreconditioned resid norm 8.119036890805e-10 true resid norm 8.119036890805e-10 ||r(i)||/||b|| 9.584414399840e-04 175 KSP unpreconditioned resid norm 7.969494091924e-10 true resid norm 7.969494091924e-10 ||r(i)||/||b|| 9.407881126958e-04 176 KSP unpreconditioned resid norm 7.927024508324e-10 true resid norm 7.927024508324e-10 ||r(i)||/||b|| 9.357746351850e-04 177 KSP unpreconditioned resid norm 7.828725447824e-10 true resid norm 7.828725447824e-10 ||r(i)||/||b|| 9.241705626377e-04 178 KSP unpreconditioned resid norm 7.494508082515e-10 true resid norm 7.494508082515e-10 ||r(i)||/||b|| 8.847166499160e-04 179 KSP unpreconditioned resid norm 7.231444464622e-10 true resid norm 7.231444464622e-10 ||r(i)||/||b|| 8.536623418581e-04 180 KSP unpreconditioned resid norm 7.011791261722e-10 true resid norm 7.011791261722e-10 ||r(i)||/||b|| 8.277325751978e-04 181 KSP unpreconditioned resid norm 6.696139551270e-10 true resid norm 6.696139551271e-10 ||r(i)||/||b|| 7.904703131872e-04 182 KSP unpreconditioned resid norm 6.451235118753e-10 true resid norm 6.451235118753e-10 ||r(i)||/||b|| 7.615596726621e-04 183 KSP unpreconditioned resid norm 6.306916481370e-10 true resid norm 6.306916481370e-10 ||r(i)||/||b|| 7.445230506476e-04 184 KSP unpreconditioned resid norm 6.100227035205e-10 true resid norm 6.100227035205e-10 ||r(i)||/||b|| 7.201236381217e-04 185 KSP unpreconditioned resid norm 5.919504539386e-10 true resid norm 5.919504539386e-10 ||r(i)||/||b|| 6.987895893349e-04 186 KSP unpreconditioned resid norm 5.856700542909e-10 true resid norm 5.856700542909e-10 ||r(i)||/||b|| 6.913756615957e-04 187 KSP unpreconditioned resid norm 5.715591865793e-10 true resid norm 5.715591865793e-10 ||r(i)||/||b|| 6.747179710952e-04 188 KSP unpreconditioned resid norm 5.547066799722e-10 true resid norm 5.547066799722e-10 ||r(i)||/||b|| 6.548238125674e-04 189 KSP unpreconditioned resid norm 5.480604885087e-10 true resid norm 5.480604885087e-10 ||r(i)||/||b|| 6.469780724848e-04 190 KSP unpreconditioned resid norm 5.296236594065e-10 true resid norm 5.296236594065e-10 ||r(i)||/||b|| 6.252136424531e-04 191 KSP unpreconditioned resid norm 5.105824359200e-10 true resid norm 5.105824359201e-10 ||r(i)||/||b|| 6.027357329389e-04 192 KSP unpreconditioned resid norm 4.831649558134e-10 true resid norm 4.831649558134e-10 ||r(i)||/||b|| 5.703697645764e-04 193 KSP unpreconditioned resid norm 4.690710598203e-10 true resid norm 4.690710598203e-10 ||r(i)||/||b|| 5.537321089625e-04 194 KSP unpreconditioned resid norm 4.547545034138e-10 true resid norm 4.547545034138e-10 ||r(i)||/||b|| 5.368316057102e-04 195 KSP unpreconditioned resid norm 4.328632397413e-10 true resid norm 4.328632397413e-10 ||r(i)||/||b|| 5.109892618959e-04 196 KSP unpreconditioned resid norm 4.079089522157e-10 true resid norm 4.079089522157e-10 ||r(i)||/||b|| 4.815310594126e-04 197 KSP unpreconditioned resid norm 3.873059797615e-10 true resid norm 3.873059797615e-10 ||r(i)||/||b|| 4.572095261415e-04 198 KSP unpreconditioned resid norm 3.729969299802e-10 true resid norm 3.729969299802e-10 ||r(i)||/||b|| 4.403178843598e-04 199 KSP unpreconditioned resid norm 3.477970037895e-10 true resid norm 3.477970037895e-10 ||r(i)||/||b|| 4.105697087196e-04 200 KSP unpreconditioned resid norm 3.282214610955e-10 true resid norm 3.282214610955e-10 ||r(i)||/||b|| 3.874610425312e-04 201 KSP unpreconditioned resid norm 3.120709893337e-10 true resid norm 3.120709893337e-10 ||r(i)||/||b|| 3.683956267436e-04 202 KSP unpreconditioned resid norm 2.888671401828e-10 true resid norm 2.888671401828e-10 ||r(i)||/||b|| 3.410037933371e-04 203 KSP unpreconditioned resid norm 2.762162725563e-10 true resid norm 2.762162725563e-10 ||r(i)||/||b|| 3.260696133992e-04 204 KSP unpreconditioned resid norm 2.633334545155e-10 true resid norm 2.633334545155e-10 ||r(i)||/||b|| 3.108616191011e-04 205 KSP unpreconditioned resid norm 2.491344685484e-10 true resid norm 2.491344685484e-10 ||r(i)||/||b|| 2.940999061792e-04 206 KSP unpreconditioned resid norm 2.386964029507e-10 true resid norm 2.386964029507e-10 ||r(i)||/||b|| 2.817779094244e-04 207 KSP unpreconditioned resid norm 2.274820848858e-10 true resid norm 2.274820848858e-10 ||r(i)||/||b|| 2.685395570199e-04 208 KSP unpreconditioned resid norm 2.212409903975e-10 true resid norm 2.212409903975e-10 ||r(i)||/||b|| 2.611720284954e-04 209 KSP unpreconditioned resid norm 2.114227895993e-10 true resid norm 2.114227895993e-10 ||r(i)||/||b|| 2.495817738413e-04 210 KSP unpreconditioned resid norm 2.048332706014e-10 true resid norm 2.048332706014e-10 ||r(i)||/||b|| 2.418029348459e-04 211 KSP unpreconditioned resid norm 1.975994847684e-10 true resid norm 1.975994847684e-10 ||r(i)||/||b|| 2.332635474732e-04 212 KSP unpreconditioned resid norm 1.907641063078e-10 true resid norm 1.907641063078e-10 ||r(i)||/||b|| 2.251944746722e-04 213 KSP unpreconditioned resid norm 1.835317861753e-10 true resid norm 1.835317861753e-10 ||r(i)||/||b|| 2.166568175394e-04 214 KSP unpreconditioned resid norm 1.783343277288e-10 true resid norm 1.783343277288e-10 ||r(i)||/||b|| 2.105212873962e-04 215 KSP unpreconditioned resid norm 1.737252962117e-10 true resid norm 1.737252962116e-10 ||r(i)||/||b|| 2.050803873687e-04 216 KSP unpreconditioned resid norm 1.681090858385e-10 true resid norm 1.681090858385e-10 ||r(i)||/||b|| 1.984505261799e-04 217 KSP unpreconditioned resid norm 1.634109844504e-10 true resid norm 1.634109844504e-10 ||r(i)||/||b|| 1.929044803617e-04 218 KSP unpreconditioned resid norm 1.596828500439e-10 true resid norm 1.596828500438e-10 ||r(i)||/||b|| 1.885034675850e-04 219 KSP unpreconditioned resid norm 1.559655758914e-10 true resid norm 1.559655758914e-10 ||r(i)||/||b|| 1.841152751930e-04 220 KSP unpreconditioned resid norm 1.497441081261e-10 true resid norm 1.497441081261e-10 ||r(i)||/||b|| 1.767709157523e-04 221 KSP unpreconditioned resid norm 1.440543864995e-10 true resid norm 1.440543864995e-10 ||r(i)||/||b|| 1.700542755126e-04 222 KSP unpreconditioned resid norm 1.406455332877e-10 true resid norm 1.406455332877e-10 ||r(i)||/||b|| 1.660301699136e-04 223 KSP unpreconditioned resid norm 1.352855224988e-10 true resid norm 1.352855224988e-10 ||r(i)||/||b|| 1.597027489055e-04 224 KSP unpreconditioned resid norm 1.318566764812e-10 true resid norm 1.318566764812e-10 ||r(i)||/||b|| 1.556550420670e-04 225 KSP unpreconditioned resid norm 1.247714207143e-10 true resid norm 1.247714207143e-10 ||r(i)||/||b|| 1.472909924497e-04 226 KSP unpreconditioned resid norm 1.170326734811e-10 true resid norm 1.170326734811e-10 ||r(i)||/||b|| 1.381555049016e-04 227 KSP unpreconditioned resid norm 1.104850176393e-10 true resid norm 1.104850176393e-10 ||r(i)||/||b|| 1.304260848017e-04 228 KSP unpreconditioned resid norm 1.057860311059e-10 true resid norm 1.057860311059e-10 ||r(i)||/||b|| 1.248789940814e-04 229 KSP unpreconditioned resid norm 1.010024307373e-10 true resid norm 1.010024307373e-10 ||r(i)||/||b|| 1.192320178609e-04 230 KSP unpreconditioned resid norm 9.517109911279e-11 true resid norm 9.517109911278e-11 ||r(i)||/||b|| 1.123482089136e-04 231 KSP unpreconditioned resid norm 8.906657903584e-11 true resid norm 8.906657903583e-11 ||r(i)||/||b|| 1.051419046540e-04 232 KSP unpreconditioned resid norm 8.571475780780e-11 true resid norm 8.571475780779e-11 ||r(i)||/||b|| 1.011851245487e-04 233 KSP unpreconditioned resid norm 8.105732120348e-11 true resid norm 8.105732120347e-11 ||r(i)||/||b|| 9.568708296360e-05 234 KSP unpreconditioned resid norm 7.805220078644e-11 true resid norm 7.805220078644e-11 ||r(i)||/||b|| 9.213957852611e-05 235 KSP unpreconditioned resid norm 7.445788494535e-11 true resid norm 7.445788494536e-11 ||r(i)||/||b|| 8.789653677520e-05 236 KSP unpreconditioned resid norm 7.257750273074e-11 true resid norm 7.257750273076e-11 ||r(i)||/||b|| 8.567677073433e-05 237 KSP unpreconditioned resid norm 7.062425062277e-11 true resid norm 7.062425062278e-11 ||r(i)||/||b|| 8.337098275948e-05 238 KSP unpreconditioned resid norm 6.715189342905e-11 true resid norm 6.715189342907e-11 ||r(i)||/||b|| 7.927191155973e-05 239 KSP unpreconditioned resid norm 6.623125011338e-11 true resid norm 6.623125011340e-11 ||r(i)||/||b|| 7.818510444572e-05 240 KSP unpreconditioned resid norm 6.442916815518e-11 true resid norm 6.442916815520e-11 ||r(i)||/||b|| 7.605777080971e-05 241 KSP unpreconditioned resid norm 6.297584870411e-11 true resid norm 6.297584870414e-11 ||r(i)||/||b|| 7.434214664619e-05 242 KSP unpreconditioned resid norm 6.218304192586e-11 true resid norm 6.218304192588e-11 ||r(i)||/||b|| 7.340624885388e-05 243 KSP unpreconditioned resid norm 6.119793248448e-11 true resid norm 6.119793248450e-11 ||r(i)||/||b|| 7.224334034115e-05 244 KSP unpreconditioned resid norm 6.039481879983e-11 true resid norm 6.039481879984e-11 ||r(i)||/||b|| 7.129527538376e-05 245 KSP unpreconditioned resid norm 6.025966944200e-11 true resid norm 6.025966944201e-11 ||r(i)||/||b|| 7.113573337542e-05 246 KSP unpreconditioned resid norm 5.948714963964e-11 true resid norm 5.948714963965e-11 ||r(i)||/||b|| 7.022378408667e-05 247 KSP unpreconditioned resid norm 5.875133821595e-11 true resid norm 5.875133821596e-11 ||r(i)||/||b|| 6.935516854771e-05 248 KSP unpreconditioned resid norm 5.865559932945e-11 true resid norm 5.865559932945e-11 ||r(i)||/||b|| 6.924215007337e-05 249 KSP unpreconditioned resid norm 5.807527161438e-11 true resid norm 5.807527161438e-11 ||r(i)||/||b|| 6.855708097174e-05 250 KSP unpreconditioned resid norm 5.729567969355e-11 true resid norm 5.729567969356e-11 ||r(i)||/||b|| 6.763678314178e-05 251 KSP unpreconditioned resid norm 5.638458710165e-11 true resid norm 5.638458710166e-11 ||r(i)||/||b|| 6.656125052938e-05 252 KSP unpreconditioned resid norm 5.502345910189e-11 true resid norm 5.502345910190e-11 ||r(i)||/||b|| 6.495445714041e-05 253 KSP unpreconditioned resid norm 5.380650850111e-11 true resid norm 5.380650850112e-11 ||r(i)||/||b|| 6.351786324155e-05 254 KSP unpreconditioned resid norm 5.122861296752e-11 true resid norm 5.122861296753e-11 ||r(i)||/||b|| 6.047469206179e-05 255 KSP unpreconditioned resid norm 4.940773406195e-11 true resid norm 4.940773406196e-11 ||r(i)||/||b|| 5.832516888095e-05 256 KSP unpreconditioned resid norm 4.709810110736e-11 true resid norm 4.709810110736e-11 ||r(i)||/||b|| 5.559867808578e-05 257 KSP unpreconditioned resid norm 4.445230653863e-11 true resid norm 4.445230653862e-11 ||r(i)||/||b|| 5.247535300367e-05 258 KSP unpreconditioned resid norm 4.227887565137e-11 true resid norm 4.227887565136e-11 ||r(i)||/||b|| 4.990964692632e-05 259 KSP unpreconditioned resid norm 3.927450241359e-11 true resid norm 3.927450241358e-11 ||r(i)||/||b|| 4.636302452395e-05 260 KSP unpreconditioned resid norm 3.754948266816e-11 true resid norm 3.754948266814e-11 ||r(i)||/||b|| 4.432666179884e-05 261 KSP unpreconditioned resid norm 3.517864153420e-11 true resid norm 3.517864153417e-11 ||r(i)||/||b|| 4.152791556703e-05 262 KSP unpreconditioned resid norm 3.347259122434e-11 true resid norm 3.347259122432e-11 ||r(i)||/||b|| 3.951394600678e-05 263 KSP unpreconditioned resid norm 3.122604584625e-11 true resid norm 3.122604584623e-11 ||r(i)||/||b|| 3.686192925144e-05 264 KSP unpreconditioned resid norm 2.921844421531e-11 true resid norm 2.921844421529e-11 ||r(i)||/||b|| 3.449198239204e-05 265 KSP unpreconditioned resid norm 2.807535371206e-11 true resid norm 2.807535371204e-11 ||r(i)||/||b|| 3.314257934990e-05 266 KSP unpreconditioned resid norm 2.679326847398e-11 true resid norm 2.679326847397e-11 ||r(i)||/||b|| 3.162909488335e-05 267 KSP unpreconditioned resid norm 2.620100211750e-11 true resid norm 2.620100211750e-11 ||r(i)||/||b|| 3.092993237530e-05 268 KSP unpreconditioned resid norm 2.480717416211e-11 true resid norm 2.480717416212e-11 ||r(i)||/||b|| 2.928453712632e-05 269 KSP unpreconditioned resid norm 2.421629839252e-11 true resid norm 2.421629839252e-11 ||r(i)||/||b|| 2.858701618747e-05 270 KSP unpreconditioned resid norm 2.361009921624e-11 true resid norm 2.361009921625e-11 ||r(i)||/||b|| 2.787140617210e-05 271 KSP unpreconditioned resid norm 2.251561534423e-11 true resid norm 2.251561534425e-11 ||r(i)||/||b|| 2.657938260769e-05 272 KSP unpreconditioned resid norm 2.149855566100e-11 true resid norm 2.149855566102e-11 ||r(i)||/||b|| 2.537875726203e-05 273 KSP unpreconditioned resid norm 2.108747915310e-11 true resid norm 2.108747915312e-11 ||r(i)||/||b|| 2.489348694552e-05 274 KSP unpreconditioned resid norm 2.035331893385e-11 true resid norm 2.035331893387e-11 ||r(i)||/||b|| 2.402682063130e-05 275 KSP unpreconditioned resid norm 1.941734891236e-11 true resid norm 1.941734891238e-11 ||r(i)||/||b|| 2.292192054618e-05 276 KSP unpreconditioned resid norm 1.853704982428e-11 true resid norm 1.853704982430e-11 ||r(i)||/||b|| 2.188273925295e-05 277 KSP unpreconditioned resid norm 1.792368946256e-11 true resid norm 1.792368946257e-11 ||r(i)||/||b|| 2.115867555398e-05 278 KSP unpreconditioned resid norm 1.760944257573e-11 true resid norm 1.760944257573e-11 ||r(i)||/||b|| 2.078771130935e-05 279 KSP unpreconditioned resid norm 1.714545149771e-11 true resid norm 1.714545149771e-11 ||r(i)||/||b|| 2.023997605092e-05 280 KSP unpreconditioned resid norm 1.671909267963e-11 true resid norm 1.671909267963e-11 ||r(i)||/||b|| 1.973666517175e-05 281 KSP unpreconditioned resid norm 1.609444716094e-11 true resid norm 1.609444716094e-11 ||r(i)||/||b|| 1.899927949600e-05 282 KSP unpreconditioned resid norm 1.566178459113e-11 true resid norm 1.566178459112e-11 ||r(i)||/||b|| 1.848852712227e-05 283 KSP unpreconditioned resid norm 1.514203649442e-11 true resid norm 1.514203649441e-11 ||r(i)||/||b|| 1.787497144942e-05 284 KSP unpreconditioned resid norm 1.456362863785e-11 true resid norm 1.456362863785e-11 ||r(i)||/||b|| 1.719216871506e-05 285 KSP unpreconditioned resid norm 1.376505865010e-11 true resid norm 1.376505865009e-11 ||r(i)||/||b|| 1.624946753105e-05 286 KSP unpreconditioned resid norm 1.303680225659e-11 true resid norm 1.303680225658e-11 ||r(i)||/||b|| 1.538977060411e-05 287 KSP unpreconditioned resid norm 1.225258977795e-11 true resid norm 1.225258977795e-11 ||r(i)||/||b|| 1.446401826750e-05 288 KSP unpreconditioned resid norm 1.121424657129e-11 true resid norm 1.121424657129e-11 ||r(i)||/||b|| 1.323826800725e-05 289 KSP unpreconditioned resid norm 1.054203483048e-11 true resid norm 1.054203483047e-11 ||r(i)||/||b|| 1.244473104282e-05 290 KSP unpreconditioned resid norm 9.820019392645e-12 true resid norm 9.820019392636e-12 ||r(i)||/||b|| 1.159240148054e-05 291 KSP unpreconditioned resid norm 9.223807475811e-12 true resid norm 9.223807475802e-12 ||r(i)||/||b|| 1.088858129128e-05 292 KSP unpreconditioned resid norm 8.482817720280e-12 true resid norm 8.482817720271e-12 ||r(i)||/||b|| 1.001385280087e-05 293 KSP unpreconditioned resid norm 8.015330479160e-12 true resid norm 8.015330479152e-12 ||r(i)||/||b|| 9.461990368692e-06 KSP Object: 128 MPI processes type: cg maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning has attached null space using UNPRECONDITIONED norm type for convergence test PC Object: 128 MPI processes type: mg MG: type is MULTIPLICATIVE, levels=5 cycles=v Cycles per PCApply=1 Using Galerkin computed coarse grid matrices Coarse grid solver -- level ------------------------------- KSP Object: (mg_coarse_) 128 MPI processes type: preonly maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (mg_coarse_) 128 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: natural factor fill ratio given 0, needed 0 Factored matrix follows: Matrix Object: 128 MPI processes type: mpiaij rows=1024, cols=1024 package used to perform factorization: superlu_dist total: nonzeros=0, allocated nonzeros=0 total number of mallocs used during MatSetValues calls =0 SuperLU_DIST run parameters: Process grid nprow 16 x npcol 8 Equilibrate matrix TRUE Matrix input mode 1 Replace tiny pivots TRUE Use iterative refinement FALSE Processors in row 16 col partition 8 Row permutation LargeDiag Column permutation METIS_AT_PLUS_A Parallel symbolic factorization FALSE Repeated factorization SamePattern_SameRowPerm linear system matrix = precond matrix: Matrix Object: 128 MPI processes type: mpiaij rows=1024, cols=1024 total: nonzeros=27648, allocated nonzeros=27648 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Down solver (pre-smoother) on level 1 ------------------------------- KSP Object: (mg_levels_1_) 128 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (mg_levels_1_) 128 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 128 MPI processes type: mpiaij rows=8192, cols=8192 total: nonzeros=221184, allocated nonzeros=221184 total number of mallocs used during MatSetValues calls =0 using I-node (on process 0) routines: found 16 nodes, limit used is 5 Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 2 ------------------------------- KSP Object: (mg_levels_2_) 128 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (mg_levels_2_) 128 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 128 MPI processes type: mpiaij rows=65536, cols=65536 total: nonzeros=1769472, allocated nonzeros=1769472 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 3 ------------------------------- KSP Object: (mg_levels_3_) 128 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (mg_levels_3_) 128 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 128 MPI processes type: mpiaij rows=524288, cols=524288 total: nonzeros=14155776, allocated nonzeros=14155776 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 4 ------------------------------- KSP Object: (mg_levels_4_) 128 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (mg_levels_4_) 128 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 128 MPI processes type: mpiaij rows=4194304, cols=4194304 total: nonzeros=29360128, allocated nonzeros=29360128 total number of mallocs used during MatSetValues calls =0 Up solver (post-smoother) same as down solver (pre-smoother) linear system matrix = precond matrix: Matrix Object: 128 MPI processes type: mpiaij rows=4194304, cols=4194304 total: nonzeros=29360128, allocated nonzeros=29360128 total number of mallocs used during MatSetValues calls =0 ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./hit on a interlagos-64idx-pgi-opt named nid25475 with 128 processors, by Unknown Fri Sep 20 12:56:00 2013 Using Petsc Release Version 3.4.2, Jul, 02, 2013 Max Max/Min Avg Total Time (sec): 3.240e+03 1.00000 3.240e+03 Objects: 5.352e+05 1.00000 5.352e+05 Flops: 4.067e+11 1.00000 4.067e+11 5.206e+13 Flops/sec: 1.255e+08 1.00000 1.255e+08 1.606e+10 MPI Messages: 7.811e+06 1.00000 7.811e+06 9.998e+08 MPI Message Lengths: 2.990e+10 1.00000 3.827e+03 3.827e+12 MPI Reductions: 1.522e+06 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 3.2403e+03 100.0% 5.2056e+13 100.0% 9.998e+08 100.0% 3.827e+03 100.0% 1.522e+06 100.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %f - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage VecTDot 177386 1.0 5.7980e+01 2.1 1.16e+10 1.0 0.0e+00 0.0e+00 1.8e+05 1 3 0 0 12 1 3 0 0 12 25664 VecNorm 179519 1.0 4.6141e+01 3.5 1.18e+10 1.0 0.0e+00 0.0e+00 1.8e+05 1 3 0 0 12 1 3 0 0 12 32637 VecScale 354772 1.0 2.3496e+00 1.5 1.00e+09 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 54733 VecCopy 179519 1.0 8.4071e+00 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 1243857 1.0 1.8819e+01 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 VecAXPY 177386 1.0 2.6131e+01 1.2 1.16e+10 1.0 0.0e+00 0.0e+00 0.0e+00 1 3 0 0 0 1 3 0 0 0 56944 VecAYPX 532158 1.0 4.1760e+01 1.3 1.20e+10 1.0 0.0e+00 0.0e+00 0.0e+00 1 3 0 0 0 1 3 0 0 0 36832 VecScatterBegin 1598611 1.0 9.1558e+01 1.3 0.00e+00 0.0 1.0e+09 3.8e+03 0.0e+00 3 0100100 0 3 0100100 0 0 VecScatterEnd 1598611 1.0 2.5825e+02 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 6 0 0 0 0 6 0 0 0 0 0 MatMult 532869 1.0 1.0230e+03 1.1 1.36e+11 1.0 4.1e+08 6.2e+03 0.0e+00 29 33 41 66 0 29 33 41 66 0 16969 MatMultAdd 354772 1.0 1.6270e+02 1.1 2.24e+10 1.0 1.4e+08 9.9e+02 0.0e+00 5 6 14 4 0 5 6 14 4 0 17634 MatMultTranspose 354776 1.0 1.8670e+02 1.3 2.24e+10 1.0 1.4e+08 9.9e+02 0.0e+00 5 6 14 4 0 5 6 14 4 0 15367 MatSolve 88693 1.0 1.6765e+02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 4 0 0 0 0 4 0 0 0 0 0 MatSOR 709544 1.0 1.3796e+03 1.1 1.79e+11 1.0 3.2e+08 3.2e+03 7.1e+05 41 44 32 27 47 41 44 32 27 47 16627 MatLUFactorSym 1 1.0 6.4993e-04 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatLUFactorNum 1 1.0 1.2938e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 728 1.0 2.8366e+0018.5 0.00e+00 0.0 0.0e+00 0.0e+00 1.4e+03 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 728 1.0 2.5708e+00 1.2 0.00e+00 0.0 1.2e+04 1.1e+03 7.2e+01 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 1 1.0 5.0068e-06 5.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 2.0981e-05 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatView 4977 1.0 1.3075e+00 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 5.0e+03 0 0 0 0 0 0 0 0 0 0 0 MatPtAP 4 1.0 2.1368e-01 1.0 5.11e+06 1.0 2.5e+04 6.0e+03 1.0e+02 0 0 0 0 0 0 0 0 0 0 3058 MatPtAPSymbolic 4 1.0 1.5158e-01 1.1 0.00e+00 0.0 1.5e+04 7.8e+03 6.0e+01 0 0 0 0 0 0 0 0 0 0 0 MatPtAPNumeric 4 1.0 7.0144e-02 1.1 5.11e+06 1.0 9.7e+03 3.1e+03 4.0e+01 0 0 0 0 0 0 0 0 0 0 9316 MatGetLocalMat 4 1.0 2.1817e-02 3.3 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetBrAoCol 4 1.0 2.9885e-02 3.2 0.00e+00 0.0 1.1e+04 8.4e+03 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetSymTrans 8 1.0 9.0678e-03 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSetUp 6 1.0 1.0205e-02 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 2.2e+01 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 711 1.0 3.0209e+03 1.0 4.07e+11 1.0 1.0e+09 3.8e+03 1.5e+06 93100100100 99 93100100100 99 17232 PCSetUp 1 1.0 4.2324e-01 1.0 5.36e+06 1.0 3.4e+04 4.6e+03 3.0e+02 0 0 0 0 0 0 0 0 0 0 1620 PCApply 88693 1.0 2.2119e+03 1.0 2.87e+11 1.0 9.1e+08 2.6e+03 7.1e+05 67 71 91 61 47 67 71 91 61 47 16614 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Vector 535068 535068 74824032416 0 Vector Scatter 19 19 22572 0 Matrix 38 38 14004608 0 Matrix Null Space 1 1 652 0 Distributed Mesh 5 5 830792 0 Bipartite Graph 10 10 8560 0 Index Set 47 47 534480 0 IS L to G Mapping 5 5 405756 0 Krylov Solver 7 7 9536 0 DMKSP interface 3 3 2088 0 Preconditioner 7 7 7352 0 Viewer 1 0 0 0 ======================================================================================================================== Average time to get PetscTime(): 4.05312e-07 Average time for MPI_Barrier(): 1.32084e-05 Average time for zero size MPI_Send(): 2.75671e-06 #PETSc Option Table entries: -ksp_monitor_true_residual -ksp_view -log_summary -mg_coarse_pc_factor_mat_solver_package superlu_dist -mg_coarse_pc_type lu -mg_levels_ksp_max_it 1 -mg_levels_ksp_type richardson -options_left -pc_mg_galerkin -pc_mg_levels 5 -pc_type mg #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 8 Configure run at: Wed Aug 28 23:25:43 2013 Configure options: --known-level1-dcache-size=16384 --known-level1-dcache-linesize=64 --known-level1-dcache-assoc=4 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=0 --known-mpi-c-double-complex=0 --with-batch="1 " --known-mpi-shared="0 " --known-memcmp-ok --with-blas-lapack-lib="-L/opt/acml/5.3.0/pgi64/lib -lacml" --COPTFLAGS="-O3 -fastsse" --FOPTFLAGS="-O3 -fastsse" --CXXOPTFLAGS="-O3 -fastsse" --with-x="0 " --with-debugging="0 " --with-clib-autodetect="0 " --with-cxxlib-autodetect="0 " --with-fortranlib-autodetect="0 " --with-shared-libraries=0 --with-dynamic-loading=0 --with-mpi-compilers="1 " --known-mpi-shared-libraries=0 --with-64-bit-indices --download-blacs="1 " --download-scalapack="1 " --download-superlu_dist="1 " --download-metis="1 " --download-parmetis="1 " --with-cc=cc --with-cxx=CC --with-fc=ftn PETSC_ARCH=interlagos-64idx-pgi-opt ----------------------------------------- Libraries compiled on Wed Aug 28 23:25:43 2013 on h2ologin3 Machine characteristics: Linux-2.6.32.59-0.7-default-x86_64-with-SuSE-11-x86_64 Using PETSc directory: /mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2 Using PETSc arch: interlagos-64idx-pgi-opt ----------------------------------------- Using C compiler: cc -O3 -fastsse ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: ftn -O3 -fastsse ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/include -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/include -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/include -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/include -I/opt/cray/udreg/2.3.2-1.0402.7311.2.1.gem/include -I/opt/cray/ugni/5.0-1.0402.7128.7.6.gem/include -I/opt/cray/pmi/4.0.1-1.0000.9421.73.3.gem/include -I/opt/cray/dmapp/4.0.1-1.0402.7439.5.1.gem/include -I/opt/cray/gni-headers/2.1-1.0402.7082.6.2.gem/include -I/opt/cray/xpmem/0.1-2.0402.44035.2.1.gem/include -I/opt/cray/rca/1.0.0-2.0402.42153.2.106.gem/include -I/opt/cray-hss-devel/7.0.0/include -I/opt/cray/krca/1.0.0-2.0402.42157.2.94.gem/include -I/opt/cray/mpt/6.0.1/gni/mpich2-pgi/121/include -I/opt/acml/5.3.0/pgi64_fma4/include -I/opt/cray/libsci/12.1.01/pgi/121/interlagos/include -I/opt/fftw/3.3.0.3/interlagos/include -I/usr/include/alps -I/opt/pgi/13.6.0/linux86-64/13.6/include -I/opt/cray/xe-sysroot/4.2.24/usr/include ----------------------------------------- Using C linker: cc Using Fortran linker: ftn Using libraries: -Wl,-rpath,/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -L/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -lpetsc -Wl,-rpath,/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -L/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -lsuperlu_dist_3.3 -L/opt/acml/5.3.0/pgi64/lib -lacml -lpthread -lparmetis -lmetis -ldl ----------------------------------------- #PETSc Option Table entries: -ksp_monitor_true_residual -ksp_view -log_summary -mg_coarse_pc_factor_mat_solver_package superlu_dist -mg_coarse_pc_type lu -mg_levels_ksp_max_it 1 -mg_levels_ksp_type richardson -options_left -pc_mg_galerkin -pc_mg_levels 5 -pc_type mg #End of PETSc Option Table entries There are no unused options. From bsmith at mcs.anl.gov Fri Sep 20 20:37:33 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 20 Sep 2013 20:37:33 -0500 Subject: [petsc-users] PETSc-dev and OpenMP In-Reply-To: <523CC5A3.6010106@gmail.com> References: <52389171.1090600@gmail.com> <7AB30D0B-6ADA-427A-9CDE-A8D6B84E04E8@mcs.anl.gov> <5238AE11.6050400@gmail.com> <523CC5A3.6010106@gmail.com> Message-ID: <009FFA25-C4AC-4680-A6CD-D16FABF42B14@mcs.anl.gov> You don't need any special code to use the OpenMP inside some of the PETSc routines. Just ./configure PETSc with --with-openmp --with-mpi=0 then at runtime use -threadcomm_nthreads where n is for example for when you run the code. When you run with -log_summary it will print how many threads it is using. You are free to use OpenMP in your code. Barry On Sep 20, 2013, at 5:01 PM, Danyang Su wrote: > Hi Barry, > > Thanks for following up. I just want some PETSc OpenMP example or user guide. Looking forward to your work on superlu_mt. > > Regards, > > Danyang > > On 20/09/2013 6:17 AM, Barry Smith wrote: >> Sorry I did not respond sooner. I've been trying to interface PETSc to superlu_mt for a couple of days but haven't completed it yet. I'll let you know if I get it working. >> >> Barry >> >> On Sep 17, 2013, at 2:31 PM, Danyang Su wrote: >> >>> Hi Barry, >>> >>> Thanks so much. >>> Which solvers are current OpenMP parallel currently? Can I use the external package (e.g., SuperLU OpenMP version) with OpenMP through PETSc? >>> Is there any document or examples available for the usage of PETSc with OpenMP? >>> >>> Thanks again, >>> >>> Danyang >>> >>> On 17/09/2013 11:21 AM, Barry Smith wrote: >>>> You can utilize the "OpenMP version of PETSc" with your OpenMP program but not all the solvers are OpenMP parallel currently. >>>> >>>> Use petsc-dev and configure PETSc with ./configure --with-mpi=0 --with-openmp see http://www.mcs.anl.gov/petsc/developers/index.html >>>> >>>> Barry >>>> >>>> >>>> On Sep 17, 2013, at 12:29 PM, Danyang Su wrote: >>>> >>>>> Hi All, >>>>> >>>>> I have a parallel Fortran project utilizing OpenMP parallelism. Before porting to PETSc utilizing MPICH, I want to know if it is possible to use PETSc solver with OpenMP. There is some information for petsc-dev openmp benchmarking as can be found here, http://lists.mcs.anl.gov/pipermail/petsc-dev/2012-March/007817.html. If this is possible, I would like to have a OpenMP version of PETSc solver first, otherwise, the project will be ported to PETSc utilizing MPI. Could anybody give me some advice on this? >>>>> >>>>> Thanks and regards, >>>>> >>>>> Danyang > From mirzadeh at gmail.com Fri Sep 20 21:34:46 2013 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Fri, 20 Sep 2013 19:34:46 -0700 Subject: [petsc-users] ghost size of a ghosted vector Message-ID: Hi guys, Is there any function that would return the ghost (or ghost+local preferably) size of a ghosted vector? VecGetSize returns the total global size and VecGetLocalSize returns only the local part. I need this size for a error-checking guard i'm adding in my code. Thanks, Mohammad From bsmith at mcs.anl.gov Fri Sep 20 21:40:47 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 20 Sep 2013 21:40:47 -0500 Subject: [petsc-users] ghost size of a ghosted vector In-Reply-To: References: Message-ID: <4D5FC60A-786D-4ED5-B5E9-4001D83EA3EF@mcs.anl.gov> You need to call VecGhostGetLocalForm() and call VecGetSize() on that. Don't worry, its cost is only the function calls. Barry On Sep 20, 2013, at 9:34 PM, Mohammad Mirzadeh wrote: > Hi guys, > > Is there any function that would return the ghost (or ghost+local > preferably) size of a ghosted vector? VecGetSize returns the total > global size and VecGetLocalSize returns only the local part. > > I need this size for a error-checking guard i'm adding in my code. > > Thanks, > Mohammad From mirzadeh at gmail.com Fri Sep 20 21:44:50 2013 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Fri, 20 Sep 2013 19:44:50 -0700 Subject: [petsc-users] ghost size of a ghosted vector In-Reply-To: <4D5FC60A-786D-4ED5-B5E9-4001D83EA3EF@mcs.anl.gov> References: <4D5FC60A-786D-4ED5-B5E9-4001D83EA3EF@mcs.anl.gov> Message-ID: sweet! Thanks Barry :) On Fri, Sep 20, 2013 at 7:40 PM, Barry Smith wrote: > > You need to call VecGhostGetLocalForm() and call VecGetSize() on that. Don't worry, its cost is only the function calls. > > Barry > > On Sep 20, 2013, at 9:34 PM, Mohammad Mirzadeh wrote: > >> Hi guys, >> >> Is there any function that would return the ghost (or ghost+local >> preferably) size of a ghosted vector? VecGetSize returns the total >> global size and VecGetLocalSize returns only the local part. >> >> I need this size for a error-checking guard i'm adding in my code. >> >> Thanks, >> Mohammad > From potaman at outlook.com Sat Sep 21 22:25:42 2013 From: potaman at outlook.com (subramanya sadasiva) Date: Sat, 21 Sep 2013 23:25:42 -0400 Subject: [petsc-users] Tutorials for using DMPlex Message-ID: Hi, Are there any tutorials, presentations that can help me get started with DMPlex? Thanks, Subramanya -------------- next part -------------- An HTML attachment was scrubbed... URL: From danyang.su at gmail.com Sat Sep 21 23:06:07 2013 From: danyang.su at gmail.com (Danyang Su) Date: Sat, 21 Sep 2013 21:06:07 -0700 Subject: [petsc-users] Strange efficiency in PETSc-dev using OpenMP In-Reply-To: <933EF7F4-598A-4F88-8902-0613E27F0679@mcs.anl.gov> References: <523D482D.5030409@gmail.com> <101244E8-FFFA-4F3F-A4BC-CC47F68D93B7@mcs.anl.gov> <523E059D.6080901@gmail.com> <933EF7F4-598A-4F88-8902-0613E27F0679@mcs.anl.gov> Message-ID: <523E6CAF.9020805@gmail.com> Hi Shri, Thanks for your info. It can work with the option -threadcomm_type openmp. But another problem arises, as described as follows. The sparse matrix is 53760*53760 with 1067392 non-zero entries. If the codes is compiled using PETSc-3.4.2, it works fine, the equations can be solved quickly and I can see the speedup. But if the code is compiled using PETSc-dev with OpenMP option, it takes a long time in solving the equations and I cannot see any speedup when more processors are used. For PETSc-3.4.2, run by "mpiexec -n 4 ksp_inhm_d -log_summary log_mpi4_petsc3.4.2.log", the iteration and runtime are: Iterations 6 *time_assembly 0.4137E-01* time_ksp 0.9296E-01 For PETSc-dev, run by "mpiexec -n 1 ksp_inhm_d -threadcomm_type openmp -threadcomm_nthreads 4 -log_summary log_openmp_petsc_dev.log", the iteration and runtime are: Iterations 6 *time_assembly 0.3595E+03* time_ksp 0.2907E+00 Most of the time *'time_assembly 0.3595E+03*' is spent on the following codes do i = istart, iend - 1 ii = ia_in(i+1) jj = ia_in(i+2) call MatSetValues(a, ione, i, jj-ii, ja_in(ii:jj-1)-1, a_in(ii:jj-1), Insert_Values, ierr) end do The log files for both PETSc-3.4.2 and PETSc-dev are attached. Is there anything wrong with my codes or with running option? The above codes works fine when using MPICH. Thanks and regards, Danyang On 21/09/2013 2:09 PM, Shri wrote: > There are three thread communicator types in PETSc. The default is "no > thread" which is basically a non-threaded version. The other two types > are "openmp" and "pthread". If you want to use OpenMP then use the > option -threadcomm_type openmp. > > Shri > > On Sep 21, 2013, at 3:46 PM, Danyang Su > wrote: > >> Hi Barry, >> >> Thanks for the quick reply. >> >> After changing >> /#if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) / >> to >> /#if defined(PETSC_HAVE_PTHREADCLASSES)/ >> and comment out >> /#elif defined(PETSC_HAVE_OPENMP)// >> //PETSC_EXTERN PetscStack *petscstack;/ >> >> It can be compiled and validated with "make test". >> >> But I still have questions on running the examples. After rebuild the >> codes (e.g., ksp_ex2f.f), I can run it with "mpiexec -n 1 ksp_ex2f", >> or "mpiexec -n 4 ksp_ex2f", or "mpiexec -n 1 ksp_ex2f >> -threadcomm_nthreads 1", but if I run it with "mpiexec -n 1 ksp_ex2f >> -threadcomm_nthreads 4", there will be a lot of error information >> (attached). >> >> The codes is not modified and there is no OpenMP routines in it. For >> the current development in my project, I want to keep the OpenMP >> codes in calculating matrix values, but want to solve it with PETSc >> (OpenMP). Is it possible? >> >> Thanks and regards, >> >> Danyang >> >> >> >> On 21/09/2013 7:26 AM, Barry Smith wrote: >>> Danyang, >>> >>> I don't think the || defined (PETSC_HAVE_OPENMP) belongs in the code below. >>> >>> /* Linux functions CPU_SET and others don't work if sched.h is not included before >>> including pthread.h. Also, these functions are active only if either _GNU_SOURCE >>> or __USE_GNU is not set (see /usr/include/sched.h and /usr/include/features.h), hence >>> set these first. >>> */ >>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>> >>> Edit include/petscerror.h and locate these lines and remove that part and then rerun make all. Let us know if it works or not. >>> >>> Barry >>> >>> i.e. replace >>> >>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>> >>> with >>> >>> #if defined(PETSC_HAVE_PTHREADCLASSES) >>> >>> On Sep 21, 2013, at 6:53 AM, Matthew Knepley wrote: >>> >>>> On Sat, Sep 21, 2013 at 12:18 AM, Danyang Su wrote: >>>> Hi All, >>>> >>>> I got error information in compiling petsc-dev with openmp in cygwin. Before, I have successfully compiled petsc-3.4.2 and it works fine. >>>> The log files have been attached. >>>> >>>> The OpenMP configure test is wrong. It clearly fails to find pthread.h, but the test passes. Then in petscerror.h >>>> we guard pthread.h using PETSC_HAVE_OPENMP. Can someone who knows OpenMP fix this? >>>> >>>> Matt >>>> >>>> Thanks, >>>> >>>> Danyang >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>> -- Norbert Wiener >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: log_mpi4_petsc3.4.2.log Type: application/octet-stream Size: 12168 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: log_openmp_petsc_dev.log Type: application/octet-stream Size: 11076 bytes Desc: not available URL: From knepley at gmail.com Sun Sep 22 06:19:04 2013 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 22 Sep 2013 04:19:04 -0700 Subject: [petsc-users] Tutorials for using DMPlex In-Reply-To: References: Message-ID: On Sat, Sep 21, 2013 at 8:25 PM, subramanya sadasiva wrote: > Hi, > Are there any tutorials, presentations that can help me get started with > DMPlex? > There is the latest tutorial from Paris on the Tutorials Page. There are two examples (SNES ex12, and 62). There are sections of the manual,and there are these mailing lists. Thanks, Matt > Thanks, > Subramanya > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hariseldon99 at gmail.com Sun Sep 22 10:50:02 2013 From: hariseldon99 at gmail.com (Analabha Roy) Date: Sun, 22 Sep 2013 21:20:02 +0530 Subject: [petsc-users] make all freezes In-Reply-To: References: Message-ID: Strangely enough, make all is now freezing in my desktop. Running "make chk_upgrade" also hangs. Running "chk_makej" ok no problem. This is the configure.log from my desktop Does "make chk_upgrade" run "${PETSC_DIR}/bin/petscnagupgrade.py" ? That's what ${PETSC_DIR}/conf/rules says, and this file is transcluded in the root makefile Running "petscnagupgrade.py" in the bin dir from shell also hangs with no output. I removed "chk_upgrade" from the root makefile and the compile worked! I reinserted chk_upgrade and commented out the nag upgrade file in conf/rules and "make all " worked. I commented out all instances of "chk_upgrade" from /conf/rules and all compiles work. Maybe a network problem??? On Wed, Sep 18, 2013 at 9:56 AM, Satish Balay wrote: > On Wed, 18 Sep 2013, Analabha Roy wrote: > > > On Sep 17, 2013 9:39 PM, "Satish Balay" wrote: > > > > > > This doesn't make any sense.. > > > > > > If the build is hanging - we'd like to know exactly where its hanging. > > > [perhaps copy/paste] > > > > > > > It didn't dump any output, either to stdout or Stderr. There was nothing > to > > copy/paste. > > so you type in 'make all' and then see nothing on the screen? > > >>>>>>> > all: chk_makej > @${OMAKE} PETSC_ARCH=${PETSC_ARCH} PETSC_DIR=${PETSC_DIR} > chk_petscdir chk_upgrade | tee ${PETSC_ARCH}/conf/make.log > <<<<<< > > The only thing I can think off is chk_upgrade perhaps misconfigured > network is affecting it. [if the behavior is reproduceable - perhaps > we can narrow it down by invoking these commands one at a time]. > > > > > If the run is hanging - we'd like to see the stack trace [via gdb] of > the > > > location of the hang. > > > > > > xorg-devel should'nt make a difference [from what we know].. > > > > > > > Doesn't it need xorg headers for rendering graphic output via petscdraw? > > > > Maybe it was one of the dependencies of xorg devel that did the trick. > > If x11 includes/libs are not found - then petsc is configured without X11. > > > https://dl.dropboxusercontent.com/u/2769707/configure.log>(dropbox) > > The configure.log above confirms that it did not detect X11. [so none > of the X11 stuff was enabled]. > > Even if X11 was enabled - 'make test' does not run any X11 related tests. > > If some make target caused hang for both 'make all' and 'make test' - > then running the examples manually might confirm this. [and then > invoking the steps in 'make test' one at a time might narrow down to > the location of the hang]. > > Satish > > -- --- *Analabha Roy* C.S.I.R Senior Research Associate Saha Institute of Nuclear Physics Section 1, Block AF Bidhannagar, Calcutta 700064 India *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com *Webpage*: http://www.ph.utexas.edu/~daneel/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Sep 22 11:53:04 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 22 Sep 2013 11:53:04 -0500 Subject: [petsc-users] Strange efficiency in PETSc-dev using OpenMP In-Reply-To: <523E6CAF.9020805@gmail.com> References: <523D482D.5030409@gmail.com> <101244E8-FFFA-4F3F-A4BC-CC47F68D93B7@mcs.anl.gov> <523E059D.6080901@gmail.com> <933EF7F4-598A-4F88-8902-0613E27F0679@mcs.anl.gov> <523E6CAF.9020805@gmail.com> Message-ID: 1) # WARNING!!! # # # # This code was compiled with a debugging option, # # To get timing results run ./configure # # using --with-debugging=no, the performance will # # be generally two or three times faster. # Never time without optimization, it can give very misleading information because different parts of the code speedup very differently when optimized 2) Where are the 4 MPI processes being placed on your system? Are they being placed on 4 cores on the same CPU (as with the OpenMP run) or possibly on different CPUs? 3) Do you have any OpenMP pragmas in your code. Make a run where you take them all out 4) Both runs are actually taking very little time in the solver; KSPSolve 1 1.0 9.2897e-002 KSPSolve 1 1.0 2.9056e-001 How are you getting your matrix? From a file? Barry On Sep 21, 2013, at 11:06 PM, Danyang Su wrote: > Hi Shri, > > Thanks for your info. It can work with the option -threadcomm_type openmp. But another problem arises, as described as follows. > > The sparse matrix is 53760*53760 with 1067392 non-zero entries. If the codes is compiled using PETSc-3.4.2, it works fine, the equations can be solved quickly and I can see the speedup. But if the code is compiled using PETSc-dev with OpenMP option, it takes a long time in solving the equations and I cannot see any speedup when more processors are used. > > For PETSc-3.4.2, run by "mpiexec -n 4 ksp_inhm_d -log_summary log_mpi4_petsc3.4.2.log", the iteration and runtime are: > Iterations 6 time_assembly 0.4137E-01 time_ksp 0.9296E-01 > > For PETSc-dev, run by "mpiexec -n 1 ksp_inhm_d -threadcomm_type openmp -threadcomm_nthreads 4 -log_summary log_openmp_petsc_dev.log", the iteration and runtime are: > Iterations 6 time_assembly 0.3595E+03 time_ksp 0.2907E+00 > > Most of the time 'time_assembly 0.3595E+03' is spent on the following codes > do i = istart, iend - 1 > ii = ia_in(i+1) > jj = ia_in(i+2) > call MatSetValues(a, ione, i, jj-ii, ja_in(ii:jj-1)-1, a_in(ii:jj-1), Insert_Values, ierr) > end do > > The log files for both PETSc-3.4.2 and PETSc-dev are attached. > > Is there anything wrong with my codes or with running option? The above codes works fine when using MPICH. > > Thanks and regards, > > Danyang > > On 21/09/2013 2:09 PM, Shri wrote: >> There are three thread communicator types in PETSc. The default is "no thread" which is basically a non-threaded version. The other two types are "openmp" and "pthread". If you want to use OpenMP then use the option -threadcomm_type openmp. >> >> Shri >> >> On Sep 21, 2013, at 3:46 PM, Danyang Su wrote: >> >>> Hi Barry, >>> >>> Thanks for the quick reply. >>> >>> After changing >>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>> to >>> #if defined(PETSC_HAVE_PTHREADCLASSES) >>> and comment out >>> #elif defined(PETSC_HAVE_OPENMP) >>> PETSC_EXTERN PetscStack *petscstack; >>> >>> It can be compiled and validated with "make test". >>> >>> But I still have questions on running the examples. After rebuild the codes (e.g., ksp_ex2f.f), I can run it with "mpiexec -n 1 ksp_ex2f", or "mpiexec -n 4 ksp_ex2f", or "mpiexec -n 1 ksp_ex2f -threadcomm_nthreads 1", but if I run it with "mpiexec -n 1 ksp_ex2f -threadcomm_nthreads 4", there will be a lot of error information (attached). >>> >>> The codes is not modified and there is no OpenMP routines in it. For the current development in my project, I want to keep the OpenMP codes in calculating matrix values, but want to solve it with PETSc (OpenMP). Is it possible? >>> >>> Thanks and regards, >>> >>> Danyang >>> >>> >>> >>> On 21/09/2013 7:26 AM, Barry Smith wrote: >>>> Danyang, >>>> >>>> I don't think the || defined (PETSC_HAVE_OPENMP) belongs in the code below. >>>> >>>> /* Linux functions CPU_SET and others don't work if sched.h is not included before >>>> including pthread.h. Also, these functions are active only if either _GNU_SOURCE >>>> or __USE_GNU is not set (see /usr/include/sched.h and /usr/include/features.h), hence >>>> set these first. >>>> */ >>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>> >>>> Edit include/petscerror.h and locate these lines and remove that part and then rerun make all. Let us know if it works or not. >>>> >>>> Barry >>>> >>>> i.e. replace >>>> >>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>> >>>> with >>>> >>>> #if defined(PETSC_HAVE_PTHREADCLASSES) >>>> >>>> On Sep 21, 2013, at 6:53 AM, Matthew Knepley >>>> >>>> wrote: >>>> >>>> >>>>> On Sat, Sep 21, 2013 at 12:18 AM, Danyang Su >>>>> wrote: >>>>> Hi All, >>>>> >>>>> I got error information in compiling petsc-dev with openmp in cygwin. Before, I have successfully compiled petsc-3.4.2 and it works fine. >>>>> The log files have been attached. >>>>> >>>>> The OpenMP configure test is wrong. It clearly fails to find pthread.h, but the test passes. Then in petscerror.h >>>>> we guard pthread.h using PETSC_HAVE_OPENMP. Can someone who knows OpenMP fix this? >>>>> >>>>> Matt >>>>> >>>>> Thanks, >>>>> >>>>> Danyang >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>> -- Norbert Wiener >>>>> >>> >>> > > From bsmith at mcs.anl.gov Sun Sep 22 12:06:26 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 22 Sep 2013 12:06:26 -0500 Subject: [petsc-users] make all freezes In-Reply-To: References: Message-ID: Thanks for figuring this out. Normally if there is no network available then the code below should just return quickly after trying to connect import urllib fd = urllib.urlopen("http://www.mcs.anl.gov/petsc/petsc-dev/include/petscversion.h") pv = fd.read() fd.close() but for some reason it is hanging for you. Could you try those lines in python and see if it hangs? Then you could try import requests pv = requests.get("http://www.mcs.anl.gov/petsc/petsc-dev/include/petscversion.h") Does that hang? Barry On Sep 22, 2013, at 10:50 AM, Analabha Roy wrote: > Strangely enough, make all is now freezing in my desktop. Running "make chk_upgrade" also hangs. Running "chk_makej" ok no problem. > > This is the configure.log from my desktop > > > Does "make chk_upgrade" run "${PETSC_DIR}/bin/petscnagupgrade.py" ? > That's what ${PETSC_DIR}/conf/rules says, and this file is transcluded in the root makefile > > Running "petscnagupgrade.py" in the bin dir from shell also hangs with no output. > > I removed "chk_upgrade" from the root makefile and the compile worked! > > > I reinserted chk_upgrade and commented out the nag upgrade file in conf/rules and "make all " worked. I commented out all instances of "chk_upgrade" from /conf/rules and all compiles work. > > > Maybe a network problem??? > > > > On Wed, Sep 18, 2013 at 9:56 AM, Satish Balay wrote: > On Wed, 18 Sep 2013, Analabha Roy wrote: > > > On Sep 17, 2013 9:39 PM, "Satish Balay" wrote: > > > > > > This doesn't make any sense.. > > > > > > If the build is hanging - we'd like to know exactly where its hanging. > > > [perhaps copy/paste] > > > > > > > It didn't dump any output, either to stdout or Stderr. There was nothing to > > copy/paste. > > so you type in 'make all' and then see nothing on the screen? > > >>>>>>> > all: chk_makej > @${OMAKE} PETSC_ARCH=${PETSC_ARCH} PETSC_DIR=${PETSC_DIR} chk_petscdir chk_upgrade | tee ${PETSC_ARCH}/conf/make.log > <<<<<< > > The only thing I can think off is chk_upgrade perhaps misconfigured > network is affecting it. [if the behavior is reproduceable - perhaps > we can narrow it down by invoking these commands one at a time]. > > > > > If the run is hanging - we'd like to see the stack trace [via gdb] of the > > > location of the hang. > > > > > > xorg-devel should'nt make a difference [from what we know].. > > > > > > > Doesn't it need xorg headers for rendering graphic output via petscdraw? > > > > Maybe it was one of the dependencies of xorg devel that did the trick. > > If x11 includes/libs are not found - then petsc is configured without X11. > > > https://dl.dropboxusercontent.com/u/2769707/configure.log>(dropbox) > > The configure.log above confirms that it did not detect X11. [so none > of the X11 stuff was enabled]. > > Even if X11 was enabled - 'make test' does not run any X11 related tests. > > If some make target caused hang for both 'make all' and 'make test' - > then running the examples manually might confirm this. [and then > invoking the steps in 'make test' one at a time might narrow down to > the location of the hang]. > > Satish > > > > > -- > --- > Analabha Roy > C.S.I.R Senior Research Associate > Saha Institute of Nuclear Physics > Section 1, Block AF > Bidhannagar, Calcutta 700064 > India > Emails: daneel at physics.utexas.edu, hariseldon99 at gmail.com > Webpage: http://www.ph.utexas.edu/~daneel/ From danyang.su at gmail.com Sun Sep 22 14:43:45 2013 From: danyang.su at gmail.com (Danyang Su) Date: Sun, 22 Sep 2013 12:43:45 -0700 Subject: [petsc-users] Strange efficiency in PETSc-dev using OpenMP In-Reply-To: References: <523D482D.5030409@gmail.com> <101244E8-FFFA-4F3F-A4BC-CC47F68D93B7@mcs.anl.gov> <523E059D.6080901@gmail.com> <933EF7F4-598A-4F88-8902-0613E27F0679@mcs.anl.gov> <523E6CAF.9020805@gmail.com> Message-ID: <523F4871.6000803@gmail.com> Hi Barry, Thanks, please find the answer bellow. On 22/09/2013 9:53 AM, Barry Smith wrote: > 1) # WARNING!!! # > # # > # This code was compiled with a debugging option, # > # To get timing results run ./configure # > # using --with-debugging=no, the performance will # > # be generally two or three times faster. # > > Never time without optimization, it can give very misleading information because different parts of the code speedup very differently when optimized With optimization, the problem still exists. See attached log, using 1 thread and 4 threads. > 2) Where are the 4 MPI processes being placed on your system? Are they being placed on 4 cores on the same CPU (as with the OpenMP run) or possibly on different CPUs? Yes, the system information are as follows OS: Windows 7 X64 Pro, CYGWIN Processor: Intel Xeon E5-2620 2.0GHz, 6cores/12threads Memory: 16GB Compiler: Intel Visual Fortran V13.1. > 3) Do you have any OpenMP pragmas in your code. Make a run where you take them all out For the current, I have no OpenMP pragmas in the code. The code is just the same as I used for PETSc MPI version and it works fine when using MPI. > > 4) Both runs are actually taking very little time in the solver; > > KSPSolve 1 1.0 9.2897e-002 > KSPSolve 1 1.0 2.9056e-001 > > How are you getting your matrix? From a file? Yes, the matrix are currently from the files. Should this be the problem? The timing is started after the reading matrix. And I didn't see the speedup for the solver, the runtime for kspsolve are almost the same. nthreads = 1, KSPSolve 1 1.0 6.2800e-002 nthreads = 4, KSPSolve 1 1.0 5.5090e-002 The main question is that the program is stuck in the following codes when run with openmp, but no problem when run with mpi. do i = istart, iend - 1 ii = ia_in(i+1) jj = ia_in(i+2) call MatSetValues(a, ione, i, jj-ii, ja_in(ii:jj-1)-1, a_in(ii:jj-1), Insert_Values, ierr) end do The testing codes has also been attached. I have removed some unnecessary codes, but some are still there. > > Barry > > > On Sep 21, 2013, at 11:06 PM, Danyang Su wrote: > >> Hi Shri, >> >> Thanks for your info. It can work with the option -threadcomm_type openmp. But another problem arises, as described as follows. >> >> The sparse matrix is 53760*53760 with 1067392 non-zero entries. If the codes is compiled using PETSc-3.4.2, it works fine, the equations can be solved quickly and I can see the speedup. But if the code is compiled using PETSc-dev with OpenMP option, it takes a long time in solving the equations and I cannot see any speedup when more processors are used. >> >> For PETSc-3.4.2, run by "mpiexec -n 4 ksp_inhm_d -log_summary log_mpi4_petsc3.4.2.log", the iteration and runtime are: >> Iterations 6 time_assembly 0.4137E-01 time_ksp 0.9296E-01 >> >> For PETSc-dev, run by "mpiexec -n 1 ksp_inhm_d -threadcomm_type openmp -threadcomm_nthreads 4 -log_summary log_openmp_petsc_dev.log", the iteration and runtime are: >> Iterations 6 time_assembly 0.3595E+03 time_ksp 0.2907E+00 >> >> Most of the time 'time_assembly 0.3595E+03' is spent on the following codes >> do i = istart, iend - 1 >> ii = ia_in(i+1) >> jj = ia_in(i+2) >> call MatSetValues(a, ione, i, jj-ii, ja_in(ii:jj-1)-1, a_in(ii:jj-1), Insert_Values, ierr) >> end do >> >> The log files for both PETSc-3.4.2 and PETSc-dev are attached. >> >> Is there anything wrong with my codes or with running option? The above codes works fine when using MPICH. >> >> Thanks and regards, >> >> Danyang >> >> On 21/09/2013 2:09 PM, Shri wrote: >>> There are three thread communicator types in PETSc. The default is "no thread" which is basically a non-threaded version. The other two types are "openmp" and "pthread". If you want to use OpenMP then use the option -threadcomm_type openmp. >>> >>> Shri >>> >>> On Sep 21, 2013, at 3:46 PM, Danyang Su wrote: >>> >>>> Hi Barry, >>>> >>>> Thanks for the quick reply. >>>> >>>> After changing >>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>> to >>>> #if defined(PETSC_HAVE_PTHREADCLASSES) >>>> and comment out >>>> #elif defined(PETSC_HAVE_OPENMP) >>>> PETSC_EXTERN PetscStack *petscstack; >>>> >>>> It can be compiled and validated with "make test". >>>> >>>> But I still have questions on running the examples. After rebuild the codes (e.g., ksp_ex2f.f), I can run it with "mpiexec -n 1 ksp_ex2f", or "mpiexec -n 4 ksp_ex2f", or "mpiexec -n 1 ksp_ex2f -threadcomm_nthreads 1", but if I run it with "mpiexec -n 1 ksp_ex2f -threadcomm_nthreads 4", there will be a lot of error information (attached). >>>> >>>> The codes is not modified and there is no OpenMP routines in it. For the current development in my project, I want to keep the OpenMP codes in calculating matrix values, but want to solve it with PETSc (OpenMP). Is it possible? >>>> >>>> Thanks and regards, >>>> >>>> Danyang >>>> >>>> >>>> >>>> On 21/09/2013 7:26 AM, Barry Smith wrote: >>>>> Danyang, >>>>> >>>>> I don't think the || defined (PETSC_HAVE_OPENMP) belongs in the code below. >>>>> >>>>> /* Linux functions CPU_SET and others don't work if sched.h is not included before >>>>> including pthread.h. Also, these functions are active only if either _GNU_SOURCE >>>>> or __USE_GNU is not set (see /usr/include/sched.h and /usr/include/features.h), hence >>>>> set these first. >>>>> */ >>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>> >>>>> Edit include/petscerror.h and locate these lines and remove that part and then rerun make all. Let us know if it works or not. >>>>> >>>>> Barry >>>>> >>>>> i.e. replace >>>>> >>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>> >>>>> with >>>>> >>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) >>>>> >>>>> On Sep 21, 2013, at 6:53 AM, Matthew Knepley >>>>> >>>>> wrote: >>>>> >>>>> >>>>>> On Sat, Sep 21, 2013 at 12:18 AM, Danyang Su >>>>>> wrote: >>>>>> Hi All, >>>>>> >>>>>> I got error information in compiling petsc-dev with openmp in cygwin. Before, I have successfully compiled petsc-3.4.2 and it works fine. >>>>>> The log files have been attached. >>>>>> >>>>>> The OpenMP configure test is wrong. It clearly fails to find pthread.h, but the test passes. Then in petscerror.h >>>>>> we guard pthread.h using PETSC_HAVE_OPENMP. Can someone who knows OpenMP fix this? >>>>>> >>>>>> Matt >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Danyang >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>> >> -------------- next part -------------- A non-text attachment was scrubbed... Name: log_openmp_petsc_dev_opt_1.log Type: application/octet-stream Size: 10116 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: log_openmp_petsc_dev_opt_4.log Type: application/octet-stream Size: 10145 bytes Desc: not available URL: -------------- next part -------------- program petsc_unsym_f implicit none ! The following include statements are required for KSP Fortran programs: ! petscsys.h - base PETSc routines ! petscvec.h - vectors ! petscmat.h - matrices ! petscpc.h - preconditioners ! petscksp.h - Krylov subspace methods ! Additional include statements may be needed if using additional ! PETSc routines in a Fortran program, e.g., ! petscviewer.h - viewers ! petscis.h - index sets ! #include #include #include #include #include #include #include #include #include ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ! Variable declarations ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ! ! Variables: ! ksp - linear solver context ! ksp - Krylov subspace method context ! pc - preconditioner context ! x, b, u - approx solution, right-hand-side, exact solution vectors ! A - matrix that defines linear system ! its - iterations for convergence ! norm - norm of error in solution ! rctx - random number generator context ! ! Note that vectors are declared as PETSc "Vec" objects. These vectors ! are mathematical objects that contain more than just an array of ! double precision numbers. I.e., vectors in PETSc are not just ! double precision x(*). ! However, local vector data can be easily accessed via VecGetArray(). ! See the Fortran section of the PETSc users manual for details. ! #define xx_a(ib) xx_v(xx_i + (ib)) PetscReal :: norm, norm1, norm2, norm1_ref, norm2_ref PetscInt :: i,j,k,ii,jj,kk,m,n,its PetscInt :: istart,iend,ione PetscErrorCode :: ierr PetscMPIInt :: rank,nprcs PetscBool :: flg PetscScalar :: v,one,neg_one Vec :: x,b,u, x_ref, u_ref Mat :: a KSP :: ksp PetscRandom :: rctx PetscLogDouble :: tstart,tend PetscReal :: ttotal_assembly, ttotal_ksp PetscViewer :: viewer ! Convergence parameters PetscReal :: rtol !default 1.0E-5 PetscReal :: abstol !default 1.0E-50 PetscReal :: dtol !default 1.0E5 PetscInt :: maxits !default 1.0E5 !c.. all other variables integer, allocatable :: ia_in(:) integer, allocatable :: ja_in(:) PetscScalar, allocatable :: a_in(:) PetscScalar, allocatable :: b_in(:) PetscScalar, allocatable :: x_out(:) PetscReal, allocatable :: x_ref_in(:) PetscInt, allocatable :: ix(:), d_nnz(:), o_nnz(:) PetscInt :: nia, nb, nb_sqrt, nnz, nd_nzrow, nlocal PetscReal :: err_max, err_min, err_norm PetscScalar :: xx_v(1) PetscOffset :: xx_i !PetscInt :: ixs(4) !PetscInt :: ix_in(2) !PetscInt :: ix_out(2) PetscInt :: ng PetscInt, allocatable :: gindices(:) ISLocalToGlobalMapping :: mapping PetscBool :: flgUseDMDA !Check if use DMDA, only valid for structured mesh 1D, 2D and 3D. PetscInt :: nDimDMDA DM :: da PetscInt :: xs, ys, zs, xm, ym, zm, mx, my, mz, nstep PetscScalar, pointer :: vecpointer(:) MatPartitioning :: part IS :: is, isg character(256) :: strFolder(100) character(256) :: strXOut character(256) :: strNameAppend, strResultRefAppend character(16) :: strKSPType character(12) :: strnum integer :: iunit, ifolder, imatrix ! These variables are not currently used. ! PC pc ! PCType ptype ! double precision tol ! Note: Any user-defined Fortran routines (such as MyKSPMonitor) ! MUST be declared as external. !external MyKSPMonitor,MyKSPConverged ione = 1 one = 1.0 neg_one = -1.0 ! Convergence parameters rtol = 1.0E-10 !default 1.0E-5 abstol = 1.0E-8 !default 1.0E-50 dtol = 1.0E5 !default 1.0E5 maxits = 100 !default 1.0E5 strKSPType = "KSPGMRES" !strKSPType = "KSPBCGS" flgUseDMDA = .false. nDimDMDA = 1 ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ! Beginning of program ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - call PetscInitialize(Petsc_Null_Character,ierr) call MPI_Comm_rank(Petsc_Comm_World,rank,ierr) call MPI_Comm_size(Petsc_Comm_World,nprcs,ierr) !read data strFolder(1) = "D:\dsu\ResearchAtUBC\Min3P_Parallel_Test\Solver_Matrix_Test\chrom-bio" strFolder(2) = "D:\dsu\ResearchAtUBC\Min3P_Parallel_Test\Solver_Matrix_Test\chrom-bio-restart" strFolder(3) = "D:\dsu\ResearchAtUBC\Min3P_Parallel_Test\Solver_Matrix_Test\Decal-concrete-level2" nd_nzrow = 0 !number of non-zero elements in a row, maximum. !strNameAppend = "_" strNameAppend = "_reactran_rt_" strResultRefAppend = "" iunit = 21 do ifolder = 3, 3 if(rank == 0) then write(*,"(a)") trim(strFolder(ifolder)) end if !read sparse matrix structure ia open(unit = iunit, file = trim(adjustl(strFolder(ifolder))) // "\ia" //trim(strNameAppend)//"1.txt") read(iunit,*) nia if(.not. allocated(ia_in)) then allocate( ia_in ( nia ) ) end if if(size(ia_in,1) /= nia) then deallocate(ia_in) allocate(ia_in(nia)) end if read(iunit,*) ia_in(1) nd_nzrow = ia_in(1) do i = 2, nia read(iunit,*) ia_in(i) if(ia_in(i) - ia_in(i-1) > nd_nzrow) then nd_nzrow = ia_in(i) - ia_in(i-1) end if end do close(iunit) write(*,*) "read ia finished, size of ia", size(ia_in,1) !read sparse matrix structure ja open(unit = iunit, file = trim(adjustl(strFolder(ifolder))) // "\ja" //trim(strNameAppend)//"1.txt") read(iunit,*) nnz if(.not. allocated(ja_in)) then allocate( ja_in( nnz) ) end if if(size(ja_in,1) /= nnz) then deallocate(ja_in) allocate(ja_in(nnz)) end if do i = 1, nnz read(iunit,*) ja_in(i) end do close(iunit) write(*,*) "read ja finished. size of ja", size(ja_in,1) nb = nia - 1 !initialize indices if(.not. allocated(ix)) then allocate( ix( nb) ) end if if(size(ix,1) /= nb) then deallocate(ix) allocate(ix(nb)) end if do i = 1, nb ix(i)=i-1 end do !initialize if(.not. allocated(x_out)) then allocate( x_out( nb) ) end if if(size(x_out,1) /= nb) then deallocate(x_out) allocate(x_out(nb)) end if !initialize d_nnz, o_nnz if(.not. allocated(d_nnz)) then allocate( d_nnz( nb) ) end if if(size(d_nnz,1) /= nb) then deallocate(d_nnz) allocate(d_nnz(nb)) end if d_nnz = 0 if(.not. allocated(o_nnz)) then allocate( o_nnz( nb) ) end if if(size(o_nnz,1) /= nb) then deallocate(o_nnz) allocate(o_nnz(nb)) end if o_nnz = 0 call MatCreate(Petsc_Comm_World, a, ierr) call MatSetSizes(a, Petsc_Decide, Petsc_Decide, nb, nb, ierr) call MatSetFromOptions(a, ierr) call MatSetUp(a, ierr) call MatGetOwnershipRange(a,istart,iend,ierr) do i = istart + 1, iend do j = 1, ia_in(i+1) - ia_in(i) k = ja_in(ia_in(i)+j-1) if(k>istart .and. k<=iend) then d_nnz(i) = d_nnz(i) + 1 else o_nnz(i) = o_nnz(i) + 1 end if end do end do call MatMPIAIJSetPreallocation(a,Petsc_Null_Integer,d_nnz(istart+1:iend), Petsc_Null_Integer,o_nnz(istart+1:iend), ierr) !!Create stiffness matrix !Currently, all PETSc parallel matrix formats are partitioned by !contiguous chunks of rows across the processors. Determine which !rows of the matrix are locally owned. !call MatGetOwnershipRange(a,istart,iend,ierr) !Create local element variables and RHS call VecCreateMPI(Petsc_Comm_World, Petsc_Decide, nb, b, ierr) call VecDuplicate(b, x, ierr) call VecDuplicate(x, u, ierr) call VecDuplicate(u, x_ref, ierr) call VecDuplicate(x_ref, u_ref, ierr) write(*,*) "Mat preallocation finished." !!local to global mapping call VecGetSize(b, m, ierr) call VecGetOwnershipRange(b,istart,iend,ierr) !!Method 3, consider ghost node index ng = iend-istart + 2 if(.not. allocated(gindices)) then allocate( gindices(ng) ) end if if(size(gindices,1) /= ng) then deallocate(gindices) allocate(gindices(ng)) end if gindices(1)=istart-1 do i = 1, ng-1 gindices(i+1)=gindices(i)+1 end do !map the first and last point as periodic if(gindices(1) == -1) then gindices(1) = m-1 end if if(gindices(ng) == m) then gindices(ng) = 0 end if call ISLocalToGlobalMappingCreate(Petsc_Comm_Self,ng,gindices,Petsc_Copy_Values,mapping,ierr) call VecSetLocalToGlobalMapping(b,mapping,ierr); call ISLocalToGlobalMappingDestroy(mapping,ierr) write(*,*) "Local global mapping finished." !Create linear solver context call KSPCreate(Petsc_Comm_World,ksp,ierr) !Set KSP type if(trim(strKSPType) == "KSPGMRES") then call KSPSetType(ksp, KSPGMRES, ierr) else if(trim(strKSPType) == "KSPBCGS") then call KSPSetType(ksp, KSPBCGS, ierr) end if !Set linear solver defaults for this problem (optional). call KSPSetTolerances(ksp,rtol, abstol, dtol, maxits,ierr) call KSPSetFromOptions(ksp,ierr) write(*,*) "Create KSP solver finished." nstep = 0 !Read matrix value and right hand side, contructure linear equation do imatrix = 1, 10 nstep = nstep + 1 write(*,*) "Begin matrix ", imatrix write(strnum, *) imatrix strXOut = trim(adjustl(strFolder(ifolder))) // "\x_PETSc"// trim(strNameAppend)//trim(adjustl(strnum))//".txt" !read matrix value a open(unit = iunit, file = trim(adjustl(strFolder(ifolder))) // "\a" // trim(strNameAppend)//trim(adjustl(strnum))//".txt") read(iunit,*) nnz if(.not. allocated(a_in)) then allocate( a_in(nnz) ) end if if(size(a_in,1) /= nnz) then deallocate(a_in) allocate(a_in(nnz)) end if do i = 1, nnz read(iunit,*) a_in(i) end do close(iunit) write(*,*) "read a finished, size of a", size(a_in,1) !read right hand side b open(unit = iunit, file = trim(adjustl(strFolder(ifolder))) // "\b"//trim(strNameAppend)//trim(adjustl(strnum))//".txt") read(iunit,*) nb if(.not. allocated(b_in)) then allocate(b_in(nb)) end if if(size(b_in,1) /= nb ) then deallocate(b_in) allocate(b_in(nb)) end if do i = 1, nb read(iunit,*) b_in(i) end do close(iunit) write(*,*) "read b finished, size of b", size(b_in,1) !referenced result open(unit = iunit, file = trim(adjustl(strFolder(ifolder))) // "\x"//trim(strResultRefAppend)//trim(strNameAppend)//trim(adjustl(strnum))//".txt") read(iunit,*) nb if(.not. allocated(x_ref_in)) then allocate(x_ref_in(nb)) end if if(size(x_ref_in,1)/=nb) then deallocate(x_ref_in) allocate(x_ref_in(nb)) end if do i = 1, nb read(iunit,*) x_ref_in(i) end do close(iunit) write(*,*) "read x_ref finished, size of x_ref", size(x_ref_in,1) !Set the solution from WatSolv (ws209) call VecGetOwnershipRange(x_ref,istart,iend,ierr) do i = istart, iend - 1 call VecSetValue(x_ref, i, x_ref_in(i+1), Insert_Values, ierr) end do call VecAssemblyBegin(x_ref,ierr) call VecAssemblyEnd(x_ref,ierr) write(*,*) "Assembly reference result finished" call PetscTime(tstart, ierr) !Set matrix elements ! - Each processor needs to insert only elements that it owns ! locally (but any non-local elements will be sent to the ! appropriate processor during matrix assembly). ! - Always specify global row and columns of matrix entries. ! - Note that MatSetValues() uses 0-based row and column numbers ! in Fortran as well as in C. call MatZeroEntries(a, ierr) write(*,*) "mark 1" call MatGetOwnershipRange(a,istart,iend,ierr) write(*,*) "mark 2" do i = istart, iend - 1 ii = ia_in(i+1) jj = ia_in(i+2) call MatSetValues(a, ione, i, jj-ii, ja_in(ii:jj-1)-1, a_in(ii:jj-1), Insert_Values, ierr) end do write(*,*) "mark 3" call MatAssemblyBegin(a, Mat_Final_Assembly, ierr) call MatAssemblyEnd(a, Mat_Final_Assembly, ierr) write(*,*) "mark 4" if(nstep == 1) then call MatSetOption(a,Mat_New_Nonzero_location_Err,Petsc_True,ierr) !!Test matrix partition using ParMetis !call MatPartitioningCreate(Petsc_Comm_World, part, ierr) !call MatPartitioningSetAdjacency(part, a, ierr) !call MatPartitioningSetFromOptions(part, ierr) !call MatPartitioningApply(part, is, ierr) !call ISPartitioningToNumbering(is, isg, ierr) !call ISView(isg, PETSC_VIEWER_STDOUT_WORLD, ierr) !call ISDestroy(is, ierr) !call ISDestroy(isg, ierr) !call MatPartitioningDestroy(part, ierr) end if !call MatView(a, PETSC_VIEWER_STDOUT_WORLD, ierr) write(*,*) "Assembly matrix a finished" !!Set right hand side b call VecGetOwnershipRange(b,istart,iend,ierr) !!Method 1 !do i = istart, iend - 1 ! call VecSetValue(b, i, b_in(i+1), Insert_Values, ierr) !end do !!Method 2 !!Use VecSetValues instead, it is faster. !call VecSetValues(b, iend-istart, ix(istart+1:iend), b_in(istart+1:iend), Insert_Values, ierr) !!Method 3-1, set the values using the local ordering, without inserting ghost values !do i = 2, ng-1 ! call VecSetValuesLocal(b,1,i-1,b_in(istart+i-1:istart+i-1),Insert_Values,ierr) !end do !!Method 3-2, set the values using the local ordering, without inserting ghost values call VecSetValuesLocal(b,ng-2,ix(2:ng-1),b_in(istart+1:istart+ng-2),Insert_Values,ierr) !!Assemble matrix call VecAssemblyBegin(b,ierr) call VecAssemblyEnd(b,ierr) !call VecView(b,PETSC_VIEWER_STDOUT_WORLD,ierr) write(*,*) "Assembly right-hand-side b finished" call PetscTime(tend, ierr) ttotal_assembly = tend - tstart call PetscTime(tstart, ierr) !call KSPSetOperators(ksp,a,a,SAME_PRECONDITIONER,ierr) !Fail for NWMO_Basin call KSPSetOperators(ksp,a,a,SAME_NONZERO_PATTERN,ierr) !call KSPSetOperators(ksp,a,a,DIFFERENT_NONZERO_PATTERN,ierr) call KSPSolve(ksp,b,x,ierr) !call VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr) call PetscTime(tend, ierr) ttotal_ksp = tend - tstart write(*,*) "KSP solver finished" !!calculate residual r=ax-b, not required call MatMult(a, x, u, ierr) call VecAXPY(u, neg_one, b, ierr) !!calculate norm of residual a, not required call VecNormBegin(u, Norm_1, norm1, ierr) call VecNormBegin(u, Norm_2, norm2, ierr) call VecNormEnd(u, Norm_1, norm1, ierr) call VecNormEnd(u, Norm_2, norm2, ierr) !!Calculate norm of PETSc solution and compare solution PETSc vs WatSolv (ws209) call MatMult(a,x_ref,u_ref,ierr) call VecAXPY(u_ref, neg_one, b, ierr) call VecNormBegin(u_ref, Norm_1, norm1_ref, ierr) call VecNormBegin(u_ref, Norm_2, norm2_ref, ierr) call VecNormEnd(u_ref, Norm_1, norm1_ref, ierr) call VecNormEnd(u_ref, Norm_2, norm2_ref, ierr) call VecAXPY(x_ref, neg_one, x, ierr) call VecMax(x_ref, PETSC_NULL_INTEGER, err_max, ierr) call VecMin(x_ref, PETSC_NULL_INTEGER, err_min, ierr) call VecNorm(x_ref, Norm_2, err_norm, ierr) !!get solver residual call KSPGetResidualNorm(ksp,norm,ierr) call KSPGetIterationNumber(ksp,its,ierr) call KSPGetErrorIfNotConverged(ksp,flg,ierr) !!Get solution vector to array data !call VecGetOwnershipRange(x,istart,iend,ierr) !call VecGetValues(x, iend-istart,ix(istart+1:iend),x_out(istart+1:iend), ierr) !write(*, 90) istart+1, x_out(istart+1), iend, x_out(iend) !!Get solution vector by pointer, !call VecGetArray(x, xx_v, xx_i, ierr) !call VecGetLocalSize(x,nlocal,ierr) !you can not use ownership range istart, iend here, it's buggy. !write(*, 90) istart+1, xx_a(1), iend, xx_a(nlocal) !call VecRestoreArray(x, xx_v, xx_i, ierr) 90 format ('x(', i6, ') = ',e11.4, ' x(', i6, ') = ',e11.4) write(*,*) "Get solution residual finished" !!Output the solution solved by PETSc call PetscViewerASCIIOpen(Petsc_Comm_World, trim(strXOut) , viewer, ierr) call VecView(x, viewer, ierr) call PetscViewerDestroy(viewer, ierr) if (rank == 0) then if(flg) then write(*,*) "Solver does not converge" else write(*,100) its, norm, norm1, norm2, ttotal_assembly, ttotal_ksp end if end if 100 format('Iterations ',i5, ' norm ', e11.4, ' norm1 ', e11.4, ' norm2 ', e11.4, & ' time_assembly ', e11.4, ' time_ksp ', e11.4) end do !release local arrays if (allocated(ia_in)) deallocate(ia_in) if (allocated(ja_in)) deallocate(ja_in) if (allocated(a_in)) deallocate(a_in) if (allocated(b_in)) deallocate(b_in) if (allocated(x_out)) deallocate(x_out) if (allocated(x_ref_in)) deallocate(x_ref_in) if (allocated(ix)) deallocate(ix) if (allocated(gindices)) deallocate(gindices) if (allocated(d_nnz)) deallocate(d_nnz) if (allocated(o_nnz)) deallocate(o_nnz) !release Petsc objects call KSPDestroy(ksp,ierr) call VecDestroy(b, ierr) call VecDestroy(x, ierr) call VecDestroy(u, ierr) call VecDestroy(x_ref, ierr) call VecDestroy(u_ref, ierr) call MatDestroy(a, ierr) if(flgUseDMDA) then call DMDestroy(da,ierr) end if end do call PetscFinalize(ierr) end program From bsmith at mcs.anl.gov Sun Sep 22 16:22:27 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 22 Sep 2013 16:22:27 -0500 Subject: [petsc-users] Strange efficiency in PETSc-dev using OpenMP In-Reply-To: <523F4871.6000803@gmail.com> References: <523D482D.5030409@gmail.com> <101244E8-FFFA-4F3F-A4BC-CC47F68D93B7@mcs.anl.gov> <523E059D.6080901@gmail.com> <933EF7F4-598A-4F88-8902-0613E27F0679@mcs.anl.gov> <523E6CAF.9020805@gmail.com> <523F4871.6000803@gmail.com> Message-ID: Please send the data files so I can run it and reproduce and debug the problem. Barry On Sep 22, 2013, at 2:43 PM, Danyang Su wrote: > Hi Barry, > > Thanks, please find the answer bellow. > > On 22/09/2013 9:53 AM, Barry Smith wrote: >> 1) # WARNING!!! # >> # # >> # This code was compiled with a debugging option, # >> # To get timing results run ./configure # >> # using --with-debugging=no, the performance will # >> # be generally two or three times faster. # >> >> Never time without optimization, it can give very misleading information because different parts of the code speedup very differently when optimized > With optimization, the problem still exists. See attached log, using 1 thread and 4 threads. >> 2) Where are the 4 MPI processes being placed on your system? Are they being placed on 4 cores on the same CPU (as with the OpenMP run) or possibly on different CPUs? > Yes, the system information are as follows > OS: Windows 7 X64 Pro, CYGWIN > Processor: Intel Xeon E5-2620 2.0GHz, 6cores/12threads > Memory: 16GB > Compiler: Intel Visual Fortran V13.1. >> 3) Do you have any OpenMP pragmas in your code. Make a run where you take them all out > For the current, I have no OpenMP pragmas in the code. The code is just the same as I used for PETSc MPI version and it works fine when using MPI. >> >> 4) Both runs are actually taking very little time in the solver; >> >> KSPSolve 1 1.0 9.2897e-002 >> KSPSolve 1 1.0 2.9056e-001 >> >> How are you getting your matrix? From a file? > Yes, the matrix are currently from the files. Should this be the problem? The timing is started after the reading matrix. And I didn't see the speedup for the solver, the runtime for kspsolve are almost the same. > > nthreads = 1, KSPSolve 1 1.0 6.2800e-002 > > nthreads = 4, KSPSolve 1 1.0 5.5090e-002 > > The main question is that the program is stuck in the following codes when run with openmp, but no problem when run with mpi. > > do i = istart, iend - 1 > ii = ia_in(i+1) > jj = ia_in(i+2) > call MatSetValues(a, ione, i, jj-ii, ja_in(ii:jj-1)-1, a_in(ii:jj-1), Insert_Values, ierr) > end do > > The testing codes has also been attached. I have removed some unnecessary codes, but some are still there. >> >> Barry >> >> >> On Sep 21, 2013, at 11:06 PM, Danyang Su wrote: >> >>> Hi Shri, >>> >>> Thanks for your info. It can work with the option -threadcomm_type openmp. But another problem arises, as described as follows. >>> >>> The sparse matrix is 53760*53760 with 1067392 non-zero entries. If the codes is compiled using PETSc-3.4.2, it works fine, the equations can be solved quickly and I can see the speedup. But if the code is compiled using PETSc-dev with OpenMP option, it takes a long time in solving the equations and I cannot see any speedup when more processors are used. >>> >>> For PETSc-3.4.2, run by "mpiexec -n 4 ksp_inhm_d -log_summary log_mpi4_petsc3.4.2.log", the iteration and runtime are: >>> Iterations 6 time_assembly 0.4137E-01 time_ksp 0.9296E-01 >>> >>> For PETSc-dev, run by "mpiexec -n 1 ksp_inhm_d -threadcomm_type openmp -threadcomm_nthreads 4 -log_summary log_openmp_petsc_dev.log", the iteration and runtime are: >>> Iterations 6 time_assembly 0.3595E+03 time_ksp 0.2907E+00 >>> >>> Most of the time 'time_assembly 0.3595E+03' is spent on the following codes >>> do i = istart, iend - 1 >>> ii = ia_in(i+1) >>> jj = ia_in(i+2) >>> call MatSetValues(a, ione, i, jj-ii, ja_in(ii:jj-1)-1, a_in(ii:jj-1), Insert_Values, ierr) >>> end do >>> >>> The log files for both PETSc-3.4.2 and PETSc-dev are attached. >>> >>> Is there anything wrong with my codes or with running option? The above codes works fine when using MPICH. >>> >>> Thanks and regards, >>> >>> Danyang >>> >>> On 21/09/2013 2:09 PM, Shri wrote: >>>> There are three thread communicator types in PETSc. The default is "no thread" which is basically a non-threaded version. The other two types are "openmp" and "pthread". If you want to use OpenMP then use the option -threadcomm_type openmp. >>>> >>>> Shri >>>> >>>> On Sep 21, 2013, at 3:46 PM, Danyang Su wrote: >>>> >>>>> Hi Barry, >>>>> >>>>> Thanks for the quick reply. >>>>> >>>>> After changing >>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>> to >>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) >>>>> and comment out >>>>> #elif defined(PETSC_HAVE_OPENMP) >>>>> PETSC_EXTERN PetscStack *petscstack; >>>>> >>>>> It can be compiled and validated with "make test". >>>>> >>>>> But I still have questions on running the examples. After rebuild the codes (e.g., ksp_ex2f.f), I can run it with "mpiexec -n 1 ksp_ex2f", or "mpiexec -n 4 ksp_ex2f", or "mpiexec -n 1 ksp_ex2f -threadcomm_nthreads 1", but if I run it with "mpiexec -n 1 ksp_ex2f -threadcomm_nthreads 4", there will be a lot of error information (attached). >>>>> >>>>> The codes is not modified and there is no OpenMP routines in it. For the current development in my project, I want to keep the OpenMP codes in calculating matrix values, but want to solve it with PETSc (OpenMP). Is it possible? >>>>> >>>>> Thanks and regards, >>>>> >>>>> Danyang >>>>> >>>>> >>>>> >>>>> On 21/09/2013 7:26 AM, Barry Smith wrote: >>>>>> Danyang, >>>>>> >>>>>> I don't think the || defined (PETSC_HAVE_OPENMP) belongs in the code below. >>>>>> >>>>>> /* Linux functions CPU_SET and others don't work if sched.h is not included before >>>>>> including pthread.h. Also, these functions are active only if either _GNU_SOURCE >>>>>> or __USE_GNU is not set (see /usr/include/sched.h and /usr/include/features.h), hence >>>>>> set these first. >>>>>> */ >>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>>> >>>>>> Edit include/petscerror.h and locate these lines and remove that part and then rerun make all. Let us know if it works or not. >>>>>> >>>>>> Barry >>>>>> >>>>>> i.e. replace >>>>>> >>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>>> >>>>>> with >>>>>> >>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) >>>>>> >>>>>> On Sep 21, 2013, at 6:53 AM, Matthew Knepley >>>>>> >>>>>> wrote: >>>>>> >>>>>> >>>>>>> On Sat, Sep 21, 2013 at 12:18 AM, Danyang Su >>>>>>> wrote: >>>>>>> Hi All, >>>>>>> >>>>>>> I got error information in compiling petsc-dev with openmp in cygwin. Before, I have successfully compiled petsc-3.4.2 and it works fine. >>>>>>> The log files have been attached. >>>>>>> >>>>>>> The OpenMP configure test is wrong. It clearly fails to find pthread.h, but the test passes. Then in petscerror.h >>>>>>> we guard pthread.h using PETSC_HAVE_OPENMP. Can someone who knows OpenMP fix this? >>>>>>> >>>>>>> Matt >>>>>>> Thanks, >>>>>>> >>>>>>> Danyang >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>> >>> > > From bsmith at mcs.anl.gov Sun Sep 22 16:24:33 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 22 Sep 2013 16:24:33 -0500 Subject: [petsc-users] Strange efficiency in PETSc-dev using OpenMP In-Reply-To: <523F4871.6000803@gmail.com> References: <523D482D.5030409@gmail.com> <101244E8-FFFA-4F3F-A4BC-CC47F68D93B7@mcs.anl.gov> <523E059D.6080901@gmail.com> <933EF7F4-598A-4F88-8902-0613E27F0679@mcs.anl.gov> <523E6CAF.9020805@gmail.com> <523F4871.6000803@gmail.com> Message-ID: <76020DE9-2154-484F-9031-952E3EB323C2@mcs.anl.gov> If you run the Openmp compiled version WITHOUT the -threadcomm_nthreads 1 -threadcomm_type openmp command line options is it still slow? I want to understand if the MPI compile options triggering the much slower run. Barry On Sep 22, 2013, at 2:43 PM, Danyang Su wrote: > Hi Barry, > > Thanks, please find the answer bellow. > > On 22/09/2013 9:53 AM, Barry Smith wrote: >> 1) # WARNING!!! # >> # # >> # This code was compiled with a debugging option, # >> # To get timing results run ./configure # >> # using --with-debugging=no, the performance will # >> # be generally two or three times faster. # >> >> Never time without optimization, it can give very misleading information because different parts of the code speedup very differently when optimized > With optimization, the problem still exists. See attached log, using 1 thread and 4 threads. >> 2) Where are the 4 MPI processes being placed on your system? Are they being placed on 4 cores on the same CPU (as with the OpenMP run) or possibly on different CPUs? > Yes, the system information are as follows > OS: Windows 7 X64 Pro, CYGWIN > Processor: Intel Xeon E5-2620 2.0GHz, 6cores/12threads > Memory: 16GB > Compiler: Intel Visual Fortran V13.1. >> 3) Do you have any OpenMP pragmas in your code. Make a run where you take them all out > For the current, I have no OpenMP pragmas in the code. The code is just the same as I used for PETSc MPI version and it works fine when using MPI. >> >> 4) Both runs are actually taking very little time in the solver; >> >> KSPSolve 1 1.0 9.2897e-002 >> KSPSolve 1 1.0 2.9056e-001 >> >> How are you getting your matrix? From a file? > Yes, the matrix are currently from the files. Should this be the problem? The timing is started after the reading matrix. And I didn't see the speedup for the solver, the runtime for kspsolve are almost the same. > > nthreads = 1, KSPSolve 1 1.0 6.2800e-002 > > nthreads = 4, KSPSolve 1 1.0 5.5090e-002 > > The main question is that the program is stuck in the following codes when run with openmp, but no problem when run with mpi. > > do i = istart, iend - 1 > ii = ia_in(i+1) > jj = ia_in(i+2) > call MatSetValues(a, ione, i, jj-ii, ja_in(ii:jj-1)-1, a_in(ii:jj-1), Insert_Values, ierr) > end do > > The testing codes has also been attached. I have removed some unnecessary codes, but some are still there. >> >> Barry >> >> >> On Sep 21, 2013, at 11:06 PM, Danyang Su wrote: >> >>> Hi Shri, >>> >>> Thanks for your info. It can work with the option -threadcomm_type openmp. But another problem arises, as described as follows. >>> >>> The sparse matrix is 53760*53760 with 1067392 non-zero entries. If the codes is compiled using PETSc-3.4.2, it works fine, the equations can be solved quickly and I can see the speedup. But if the code is compiled using PETSc-dev with OpenMP option, it takes a long time in solving the equations and I cannot see any speedup when more processors are used. >>> >>> For PETSc-3.4.2, run by "mpiexec -n 4 ksp_inhm_d -log_summary log_mpi4_petsc3.4.2.log", the iteration and runtime are: >>> Iterations 6 time_assembly 0.4137E-01 time_ksp 0.9296E-01 >>> >>> For PETSc-dev, run by "mpiexec -n 1 ksp_inhm_d -threadcomm_type openmp -threadcomm_nthreads 4 -log_summary log_openmp_petsc_dev.log", the iteration and runtime are: >>> Iterations 6 time_assembly 0.3595E+03 time_ksp 0.2907E+00 >>> >>> Most of the time 'time_assembly 0.3595E+03' is spent on the following codes >>> do i = istart, iend - 1 >>> ii = ia_in(i+1) >>> jj = ia_in(i+2) >>> call MatSetValues(a, ione, i, jj-ii, ja_in(ii:jj-1)-1, a_in(ii:jj-1), Insert_Values, ierr) >>> end do >>> >>> The log files for both PETSc-3.4.2 and PETSc-dev are attached. >>> >>> Is there anything wrong with my codes or with running option? The above codes works fine when using MPICH. >>> >>> Thanks and regards, >>> >>> Danyang >>> >>> On 21/09/2013 2:09 PM, Shri wrote: >>>> There are three thread communicator types in PETSc. The default is "no thread" which is basically a non-threaded version. The other two types are "openmp" and "pthread". If you want to use OpenMP then use the option -threadcomm_type openmp. >>>> >>>> Shri >>>> >>>> On Sep 21, 2013, at 3:46 PM, Danyang Su wrote: >>>> >>>>> Hi Barry, >>>>> >>>>> Thanks for the quick reply. >>>>> >>>>> After changing >>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>> to >>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) >>>>> and comment out >>>>> #elif defined(PETSC_HAVE_OPENMP) >>>>> PETSC_EXTERN PetscStack *petscstack; >>>>> >>>>> It can be compiled and validated with "make test". >>>>> >>>>> But I still have questions on running the examples. After rebuild the codes (e.g., ksp_ex2f.f), I can run it with "mpiexec -n 1 ksp_ex2f", or "mpiexec -n 4 ksp_ex2f", or "mpiexec -n 1 ksp_ex2f -threadcomm_nthreads 1", but if I run it with "mpiexec -n 1 ksp_ex2f -threadcomm_nthreads 4", there will be a lot of error information (attached). >>>>> >>>>> The codes is not modified and there is no OpenMP routines in it. For the current development in my project, I want to keep the OpenMP codes in calculating matrix values, but want to solve it with PETSc (OpenMP). Is it possible? >>>>> >>>>> Thanks and regards, >>>>> >>>>> Danyang >>>>> >>>>> >>>>> >>>>> On 21/09/2013 7:26 AM, Barry Smith wrote: >>>>>> Danyang, >>>>>> >>>>>> I don't think the || defined (PETSC_HAVE_OPENMP) belongs in the code below. >>>>>> >>>>>> /* Linux functions CPU_SET and others don't work if sched.h is not included before >>>>>> including pthread.h. Also, these functions are active only if either _GNU_SOURCE >>>>>> or __USE_GNU is not set (see /usr/include/sched.h and /usr/include/features.h), hence >>>>>> set these first. >>>>>> */ >>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>>> >>>>>> Edit include/petscerror.h and locate these lines and remove that part and then rerun make all. Let us know if it works or not. >>>>>> >>>>>> Barry >>>>>> >>>>>> i.e. replace >>>>>> >>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>>> >>>>>> with >>>>>> >>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) >>>>>> >>>>>> On Sep 21, 2013, at 6:53 AM, Matthew Knepley >>>>>> >>>>>> wrote: >>>>>> >>>>>> >>>>>>> On Sat, Sep 21, 2013 at 12:18 AM, Danyang Su >>>>>>> wrote: >>>>>>> Hi All, >>>>>>> >>>>>>> I got error information in compiling petsc-dev with openmp in cygwin. Before, I have successfully compiled petsc-3.4.2 and it works fine. >>>>>>> The log files have been attached. >>>>>>> >>>>>>> The OpenMP configure test is wrong. It clearly fails to find pthread.h, but the test passes. Then in petscerror.h >>>>>>> we guard pthread.h using PETSC_HAVE_OPENMP. Can someone who knows OpenMP fix this? >>>>>>> >>>>>>> Matt >>>>>>> Thanks, >>>>>>> >>>>>>> Danyang >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>> >>> > > From abhyshr at mcs.anl.gov Sun Sep 22 22:00:52 2013 From: abhyshr at mcs.anl.gov (Shri) Date: Sun, 22 Sep 2013 22:00:52 -0500 Subject: [petsc-users] Strange efficiency in PETSc-dev using OpenMP In-Reply-To: <523E6CAF.9020805@gmail.com> References: <523D482D.5030409@gmail.com> <101244E8-FFFA-4F3F-A4BC-CC47F68D93B7@mcs.anl.gov> <523E059D.6080901@gmail.com> <933EF7F4-598A-4F88-8902-0613E27F0679@mcs.anl.gov> <523E6CAF.9020805@gmail.com> Message-ID: I think this is definitely an issue with setting the affinities for threads, i.e., the assignment of threads to cores. Ideally each thread should be assigned to a distinct core but in your case all the 4 threads are getting pinned to the same core resulting in such a massive slowdown. Unfortunately, the thread affinities for OpenMP are set through environment variables. For Intel's OpenMP one needs to define the thread affinities through the environment variable KMP_AFFINITY. See this document here http://software.intel.com/sites/products/documentation/studio/composer/en-us/2011Update/compiler_c/optaps/common/optaps_openmp_thread_affinity.htm. Try setting the affinities via KMP_AFFINITY and let us know if it works. Shri On Sep 21, 2013, at 11:06 PM, Danyang Su wrote: > Hi Shri, > > Thanks for your info. It can work with the option -threadcomm_type openmp. But another problem arises, as described as follows. > > The sparse matrix is 53760*53760 with 1067392 non-zero entries. If the codes is compiled using PETSc-3.4.2, it works fine, the equations can be solved quickly and I can see the speedup. But if the code is compiled using PETSc-dev with OpenMP option, it takes a long time in solving the equations and I cannot see any speedup when more processors are used. > > For PETSc-3.4.2, run by "mpiexec -n 4 ksp_inhm_d -log_summary log_mpi4_petsc3.4.2.log", the iteration and runtime are: > Iterations 6 time_assembly 0.4137E-01 time_ksp 0.9296E-01 > > For PETSc-dev, run by "mpiexec -n 1 ksp_inhm_d -threadcomm_type openmp -threadcomm_nthreads 4 -log_summary log_openmp_petsc_dev.log", the iteration and runtime are: > Iterations 6 time_assembly 0.3595E+03 time_ksp 0.2907E+00 > > Most of the time 'time_assembly 0.3595E+03' is spent on the following codes > do i = istart, iend - 1 > ii = ia_in(i+1) > jj = ia_in(i+2) > call MatSetValues(a, ione, i, jj-ii, ja_in(ii:jj-1)-1, a_in(ii:jj-1), Insert_Values, ierr) > end do > > The log files for both PETSc-3.4.2 and PETSc-dev are attached. > > Is there anything wrong with my codes or with running option? The above codes works fine when using MPICH. > > Thanks and regards, > > Danyang > > On 21/09/2013 2:09 PM, Shri wrote: >> There are three thread communicator types in PETSc. The default is "no thread" which is basically a non-threaded version. The other two types are "openmp" and "pthread". If you want to use OpenMP then use the option -threadcomm_type openmp. >> >> Shri >> >> On Sep 21, 2013, at 3:46 PM, Danyang Su wrote: >> >>> Hi Barry, >>> >>> Thanks for the quick reply. >>> >>> After changing >>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>> to >>> #if defined(PETSC_HAVE_PTHREADCLASSES) >>> and comment out >>> #elif defined(PETSC_HAVE_OPENMP) >>> PETSC_EXTERN PetscStack *petscstack; >>> >>> It can be compiled and validated with "make test". >>> >>> But I still have questions on running the examples. After rebuild the codes (e.g., ksp_ex2f.f), I can run it with "mpiexec -n 1 ksp_ex2f", or "mpiexec -n 4 ksp_ex2f", or "mpiexec -n 1 ksp_ex2f -threadcomm_nthreads 1", but if I run it with "mpiexec -n 1 ksp_ex2f -threadcomm_nthreads 4", there will be a lot of error information (attached). >>> >>> The codes is not modified and there is no OpenMP routines in it. For the current development in my project, I want to keep the OpenMP codes in calculating matrix values, but want to solve it with PETSc (OpenMP). Is it possible? >>> >>> Thanks and regards, >>> >>> Danyang >>> >>> >>> >>> On 21/09/2013 7:26 AM, Barry Smith wrote: >>>> Danyang, >>>> >>>> I don't think the || defined (PETSC_HAVE_OPENMP) belongs in the code below. >>>> >>>> /* Linux functions CPU_SET and others don't work if sched.h is not included before >>>> including pthread.h. Also, these functions are active only if either _GNU_SOURCE >>>> or __USE_GNU is not set (see /usr/include/sched.h and /usr/include/features.h), hence >>>> set these first. >>>> */ >>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>> >>>> Edit include/petscerror.h and locate these lines and remove that part and then rerun make all. Let us know if it works or not. >>>> >>>> Barry >>>> >>>> i.e. replace >>>> >>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>> >>>> with >>>> >>>> #if defined(PETSC_HAVE_PTHREADCLASSES) >>>> >>>> On Sep 21, 2013, at 6:53 AM, Matthew Knepley wrote: >>>> >>>>> On Sat, Sep 21, 2013 at 12:18 AM, Danyang Su wrote: >>>>> Hi All, >>>>> >>>>> I got error information in compiling petsc-dev with openmp in cygwin. Before, I have successfully compiled petsc-3.4.2 and it works fine. >>>>> The log files have been attached. >>>>> >>>>> The OpenMP configure test is wrong. It clearly fails to find pthread.h, but the test passes. Then in petscerror.h >>>>> we guard pthread.h using PETSC_HAVE_OPENMP. Can someone who knows OpenMP fix this? >>>>> >>>>> Matt >>>>> >>>>> Thanks, >>>>> >>>>> Danyang >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>> -- Norbert Wiener >>> >>> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hariseldon99 at gmail.com Mon Sep 23 09:07:59 2013 From: hariseldon99 at gmail.com (Analabha Roy) Date: Mon, 23 Sep 2013 19:37:59 +0530 Subject: [petsc-users] make all freezes In-Reply-To: References: Message-ID: On Sun, Sep 22, 2013 at 10:36 PM, Barry Smith wrote: > > Thanks for figuring this out. Normally if there is no network > available then the code below should just return quickly after trying to > connect > > Fresh download of petsc-lite-3.4.2.tar.gz. Unpack and set PETSC_DIR to the location $PETSC_DIR=$PWD ./configure --with-scalar-type=complex works no problem $make PETSC_DIR=$PWD all works no problem (???) > import urllib > fd = urllib.urlopen(" > http://www.mcs.anl.gov/petsc/petsc-dev/include/petscversion.h") > pv = fd.read() > fd.close() > > Executes without errors.on python 2.7 On python 3.3, the second line yields >>> fd = urllib.urlopen(" http://www.mcs.anl.gov/petsc/petsc-dev/include/petscversion.h") Traceback (most recent call last): File "", line 1, in AttributeError: 'module' object has no attribute 'urlopen' > but for some reason it is hanging for you. Could you try those lines in > python and see if it hangs? > > Then you could try > > import requests > pv = requests.get(" > http://www.mcs.anl.gov/petsc/petsc-dev/include/petscversion.h") > > Does that hang? > > On python2.7, first line yields Traceback (most recent call last): File "", line 1, in ImportError: No module named requests On Python3.3, no errors. Ubuntu repos do not seem to have python requests for 2.7. They just have python3-requests. the default python links to 2.7 Barry > > > On Sep 22, 2013, at 10:50 AM, Analabha Roy wrote: > > > Strangely enough, make all is now freezing in my desktop. Running "make > chk_upgrade" also hangs. Running "chk_makej" ok no problem. > > > > This is the configure.log from my desktop > > > > > > Does "make chk_upgrade" run "${PETSC_DIR}/bin/petscnagupgrade.py" ? > > That's what ${PETSC_DIR}/conf/rules says, and this file is transcluded > in the root makefile > > > > Running "petscnagupgrade.py" in the bin dir from shell also hangs with > no output. > > > > I removed "chk_upgrade" from the root makefile and the compile worked! > > > > > > I reinserted chk_upgrade and commented out the nag upgrade file in > conf/rules and "make all " worked. I commented out all instances of > "chk_upgrade" from /conf/rules and all compiles work. > > > > > > Maybe a network problem??? > > > > > > > > On Wed, Sep 18, 2013 at 9:56 AM, Satish Balay wrote: > > On Wed, 18 Sep 2013, Analabha Roy wrote: > > > > > On Sep 17, 2013 9:39 PM, "Satish Balay" wrote: > > > > > > > > This doesn't make any sense.. > > > > > > > > If the build is hanging - we'd like to know exactly where its > hanging. > > > > [perhaps copy/paste] > > > > > > > > > > It didn't dump any output, either to stdout or Stderr. There was > nothing to > > > copy/paste. > > > > so you type in 'make all' and then see nothing on the screen? > > > > >>>>>>> > > all: chk_makej > > @${OMAKE} PETSC_ARCH=${PETSC_ARCH} PETSC_DIR=${PETSC_DIR} > chk_petscdir chk_upgrade | tee ${PETSC_ARCH}/conf/make.log > > <<<<<< > > > > The only thing I can think off is chk_upgrade perhaps misconfigured > > network is affecting it. [if the behavior is reproduceable - perhaps > > we can narrow it down by invoking these commands one at a time]. > > > > > > > > If the run is hanging - we'd like to see the stack trace [via gdb] > of the > > > > location of the hang. > > > > > > > > xorg-devel should'nt make a difference [from what we know].. > > > > > > > > > > Doesn't it need xorg headers for rendering graphic output via > petscdraw? > > > > > > Maybe it was one of the dependencies of xorg devel that did the trick. > > > > If x11 includes/libs are not found - then petsc is configured without > X11. > > > > > https://dl.dropboxusercontent.com/u/2769707/configure.log>(dropbox) > > > > The configure.log above confirms that it did not detect X11. [so none > > of the X11 stuff was enabled]. > > > > Even if X11 was enabled - 'make test' does not run any X11 related tests. > > > > If some make target caused hang for both 'make all' and 'make test' - > > then running the examples manually might confirm this. [and then > > invoking the steps in 'make test' one at a time might narrow down to > > the location of the hang]. > > > > Satish > > > > > > > > > > -- > > --- > > Analabha Roy > > C.S.I.R Senior Research Associate > > Saha Institute of Nuclear Physics > > Section 1, Block AF > > Bidhannagar, Calcutta 700064 > > India > > Emails: daneel at physics.utexas.edu, hariseldon99 at gmail.com > > Webpage: http://www.ph.utexas.edu/~daneel/ > > -- --- *Analabha Roy* C.S.I.R Senior Research Associate Saha Institute of Nuclear Physics Section 1, Block AF Bidhannagar, Calcutta 700064 India *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com *Webpage*: http://www.ph.utexas.edu/~daneel/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrosso at uci.edu Mon Sep 23 10:55:12 2013 From: mrosso at uci.edu (Michele Rosso) Date: Mon, 23 Sep 2013 08:55:12 -0700 Subject: [petsc-users] How to speed up geometric multigrid Message-ID: <52406460.3060905@uci.edu> Hi, I am successfully using PETSc to solve a 3D Poisson's equation with CG + MG . Such equation arises from a projection algorithm for a multiphase incompressible flow simulation. I set up the solver as I was suggested to do in a previous thread (title: "GAMG speed") and run a test case (liquid droplet with surface tension falling under the effect of gravity in a quiescent fluid). The solution of the Poisson Equation via multigrid is correct but it becomes progressively slower and slower as the simulation progresses (I am performing successive solves) due to an increase in the number of iterations. Since the solution of the Poisson equation is mission-critical, I need to speed it up as much as I can. Could you please help me out with this? I run the test case with the following options: -pc_type mg -pc_mg_galerkin -pc_mg_levels 5 -mg_levels_ksp_type richardson -mg_levels_ksp_max_it 1 -mg_coarse_pc_type lu -mg_coarse_pc_factor_mat_solver_package superlu_dist -log_summary -ksp_view -ksp_monitor_true_residual -options_left Please find the diagnostic for the final solve in the attached file "final.txt'. Thank you, Michele -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- 0 KSP unpreconditioned resid norm 8.471082897815e-07 true resid norm 8.471082897815e-07 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 5.889629222055e-07 true resid norm 5.889629222055e-07 ||r(i)||/||b|| 6.952628481034e-01 2 KSP unpreconditioned resid norm 4.929221177373e-07 true resid norm 4.929221177373e-07 ||r(i)||/||b|| 5.818879636563e-01 3 KSP unpreconditioned resid norm 4.077087883813e-07 true resid norm 4.077087883813e-07 ||r(i)||/||b|| 4.812947686847e-01 4 KSP unpreconditioned resid norm 3.532777909896e-07 true resid norm 3.532777909896e-07 ||r(i)||/||b|| 4.170397046649e-01 5 KSP unpreconditioned resid norm 3.133843759979e-07 true resid norm 3.133843759979e-07 ||r(i)||/||b|| 3.699460621247e-01 6 KSP unpreconditioned resid norm 2.664884465415e-07 true resid norm 2.664884465415e-07 ||r(i)||/||b|| 3.145860449675e-01 7 KSP unpreconditioned resid norm 2.266816102745e-07 true resid norm 2.266816102745e-07 ||r(i)||/||b|| 2.675946074532e-01 8 KSP unpreconditioned resid norm 1.909025017919e-07 true resid norm 1.909025017919e-07 ||r(i)||/||b|| 2.253578486892e-01 9 KSP unpreconditioned resid norm 1.635648456077e-07 true resid norm 1.635648456077e-07 ||r(i)||/||b|| 1.930861113989e-01 10 KSP unpreconditioned resid norm 1.434980659348e-07 true resid norm 1.434980659348e-07 ||r(i)||/||b|| 1.693975465307e-01 11 KSP unpreconditioned resid norm 1.271661365357e-07 true resid norm 1.271661365357e-07 ||r(i)||/||b|| 1.501179224306e-01 12 KSP unpreconditioned resid norm 1.132794401526e-07 true resid norm 1.132794401526e-07 ||r(i)||/||b|| 1.337248631835e-01 13 KSP unpreconditioned resid norm 1.002788428906e-07 true resid norm 1.002788428906e-07 ||r(i)||/||b|| 1.183778320909e-01 14 KSP unpreconditioned resid norm 8.753317847137e-08 true resid norm 8.753317847137e-08 ||r(i)||/||b|| 1.033317458078e-01 15 KSP unpreconditioned resid norm 7.770742874496e-08 true resid norm 7.770742874496e-08 ||r(i)||/||b|| 9.173257974492e-02 16 KSP unpreconditioned resid norm 6.961935247483e-08 true resid norm 6.961935247483e-08 ||r(i)||/||b|| 8.218471394346e-02 17 KSP unpreconditioned resid norm 6.471790768651e-08 true resid norm 6.471790768651e-08 ||r(i)||/||b|| 7.639862396248e-02 18 KSP unpreconditioned resid norm 6.162477500458e-08 true resid norm 6.162477500458e-08 ||r(i)||/||b|| 7.274722222407e-02 19 KSP unpreconditioned resid norm 6.116147101803e-08 true resid norm 6.116147101803e-08 ||r(i)||/||b|| 7.220029806792e-02 20 KSP unpreconditioned resid norm 6.307214323139e-08 true resid norm 6.307214323139e-08 ||r(i)||/||b|| 7.445582104698e-02 21 KSP unpreconditioned resid norm 6.569260578724e-08 true resid norm 6.569260578724e-08 ||r(i)||/||b|| 7.754924202688e-02 22 KSP unpreconditioned resid norm 7.002116374918e-08 true resid norm 7.002116374918e-08 ||r(i)||/||b|| 8.265904677576e-02 23 KSP unpreconditioned resid norm 7.648471462197e-08 true resid norm 7.648471462197e-08 ||r(i)||/||b|| 9.028918208521e-02 24 KSP unpreconditioned resid norm 8.218354530044e-08 true resid norm 8.218354530044e-08 ||r(i)||/||b|| 9.701657543883e-02 25 KSP unpreconditioned resid norm 8.777006719029e-08 true resid norm 8.777006719029e-08 ||r(i)||/||b|| 1.036113897704e-01 26 KSP unpreconditioned resid norm 9.074938460336e-08 true resid norm 9.074938460336e-08 ||r(i)||/||b|| 1.071284341070e-01 27 KSP unpreconditioned resid norm 9.311175500020e-08 true resid norm 9.311175500020e-08 ||r(i)||/||b|| 1.099171807470e-01 28 KSP unpreconditioned resid norm 9.391328578136e-08 true resid norm 9.391328578136e-08 ||r(i)||/||b|| 1.108633771080e-01 29 KSP unpreconditioned resid norm 9.397746582366e-08 true resid norm 9.397746582366e-08 ||r(i)||/||b|| 1.109391407891e-01 30 KSP unpreconditioned resid norm 9.269761294239e-08 true resid norm 9.269761294239e-08 ||r(i)||/||b|| 1.094282915899e-01 31 KSP unpreconditioned resid norm 9.037592803258e-08 true resid norm 9.037592803258e-08 ||r(i)||/||b|| 1.066875736228e-01 32 KSP unpreconditioned resid norm 8.739586906294e-08 true resid norm 8.739586906294e-08 ||r(i)||/||b|| 1.031696538886e-01 33 KSP unpreconditioned resid norm 8.294737580326e-08 true resid norm 8.294737580326e-08 ||r(i)||/||b|| 9.791826712575e-02 34 KSP unpreconditioned resid norm 7.825811344542e-08 true resid norm 7.825811344542e-08 ||r(i)||/||b|| 9.238265566450e-02 35 KSP unpreconditioned resid norm 7.370584204445e-08 true resid norm 7.370584204445e-08 ||r(i)||/||b|| 8.700876019459e-02 36 KSP unpreconditioned resid norm 6.944350218542e-08 true resid norm 6.944350218542e-08 ||r(i)||/||b|| 8.197712503006e-02 37 KSP unpreconditioned resid norm 6.611084651652e-08 true resid norm 6.611084651652e-08 ||r(i)||/||b|| 7.804296961086e-02 38 KSP unpreconditioned resid norm 6.155389358744e-08 true resid norm 6.155389358744e-08 ||r(i)||/||b|| 7.266354765968e-02 39 KSP unpreconditioned resid norm 5.877895869357e-08 true resid norm 5.877895869357e-08 ||r(i)||/||b|| 6.938777415191e-02 40 KSP unpreconditioned resid norm 5.544264371508e-08 true resid norm 5.544264371508e-08 ||r(i)||/||b|| 6.544929896671e-02 41 KSP unpreconditioned resid norm 5.273241362566e-08 true resid norm 5.273241362566e-08 ||r(i)||/||b|| 6.224990861471e-02 42 KSP unpreconditioned resid norm 5.119250885272e-08 true resid norm 5.119250885272e-08 ||r(i)||/||b|| 6.043207163741e-02 43 KSP unpreconditioned resid norm 4.982771093310e-08 true resid norm 4.982771093310e-08 ||r(i)||/||b|| 5.882094595716e-02 44 KSP unpreconditioned resid norm 4.811237834099e-08 true resid norm 4.811237834099e-08 ||r(i)||/||b|| 5.679601878692e-02 45 KSP unpreconditioned resid norm 4.628304375229e-08 true resid norm 4.628304375229e-08 ||r(i)||/||b|| 5.463651378530e-02 46 KSP unpreconditioned resid norm 4.476871541552e-08 true resid norm 4.476871541552e-08 ||r(i)||/||b|| 5.284886944864e-02 47 KSP unpreconditioned resid norm 4.419875774021e-08 true resid norm 4.419875774021e-08 ||r(i)||/||b|| 5.217604204016e-02 48 KSP unpreconditioned resid norm 4.359181679552e-08 true resid norm 4.359181679552e-08 ||r(i)||/||b|| 5.145955637710e-02 49 KSP unpreconditioned resid norm 4.304314318039e-08 true resid norm 4.304314318039e-08 ||r(i)||/||b|| 5.081185451685e-02 50 KSP unpreconditioned resid norm 4.347264822128e-08 true resid norm 4.347264822128e-08 ||r(i)||/||b|| 5.131887946993e-02 51 KSP unpreconditioned resid norm 4.378084979329e-08 true resid norm 4.378084979329e-08 ||r(i)||/||b|| 5.168270730132e-02 52 KSP unpreconditioned resid norm 4.459872124486e-08 true resid norm 4.459872124486e-08 ||r(i)||/||b|| 5.264819360505e-02 53 KSP unpreconditioned resid norm 4.586835225867e-08 true resid norm 4.586835225867e-08 ||r(i)||/||b|| 5.414697602652e-02 54 KSP unpreconditioned resid norm 4.759826693288e-08 true resid norm 4.759826693288e-08 ||r(i)||/||b|| 5.618911714955e-02 55 KSP unpreconditioned resid norm 4.951983455569e-08 true resid norm 4.951983455569e-08 ||r(i)||/||b|| 5.845750201366e-02 56 KSP unpreconditioned resid norm 5.077348189601e-08 true resid norm 5.077348189601e-08 ||r(i)||/||b|| 5.993741592247e-02 57 KSP unpreconditioned resid norm 5.187532881197e-08 true resid norm 5.187532881197e-08 ||r(i)||/||b|| 6.123813146174e-02 58 KSP unpreconditioned resid norm 5.194414285622e-08 true resid norm 5.194414285622e-08 ||r(i)||/||b|| 6.131936552011e-02 59 KSP unpreconditioned resid norm 5.114127723640e-08 true resid norm 5.114127723640e-08 ||r(i)||/||b|| 6.037159339994e-02 60 KSP unpreconditioned resid norm 4.987798575055e-08 true resid norm 4.987798575055e-08 ||r(i)||/||b|| 5.888029470638e-02 61 KSP unpreconditioned resid norm 4.837751000107e-08 true resid norm 4.837751000107e-08 ||r(i)||/||b|| 5.710900316363e-02 62 KSP unpreconditioned resid norm 4.735998413707e-08 true resid norm 4.735998413707e-08 ||r(i)||/||b|| 5.590782749782e-02 63 KSP unpreconditioned resid norm 4.609537746550e-08 true resid norm 4.609537746550e-08 ||r(i)||/||b|| 5.441497624511e-02 64 KSP unpreconditioned resid norm 4.470951308242e-08 true resid norm 4.470951308242e-08 ||r(i)||/||b|| 5.277898188667e-02 65 KSP unpreconditioned resid norm 4.325085133153e-08 true resid norm 4.325085133153e-08 ||r(i)||/||b|| 5.105705120970e-02 66 KSP unpreconditioned resid norm 4.181200034623e-08 true resid norm 4.181200034623e-08 ||r(i)||/||b|| 4.935850687640e-02 67 KSP unpreconditioned resid norm 4.054904820539e-08 true resid norm 4.054904820539e-08 ||r(i)||/||b|| 4.786760877508e-02 68 KSP unpreconditioned resid norm 3.941638106292e-08 true resid norm 3.941638106292e-08 ||r(i)||/||b|| 4.653051037086e-02 69 KSP unpreconditioned resid norm 3.818351979063e-08 true resid norm 3.818351979063e-08 ||r(i)||/||b|| 4.507513413720e-02 70 KSP unpreconditioned resid norm 3.732269485449e-08 true resid norm 3.732269485449e-08 ||r(i)||/||b|| 4.405894181973e-02 71 KSP unpreconditioned resid norm 3.638709503712e-08 true resid norm 3.638709503712e-08 ||r(i)||/||b|| 4.295447875562e-02 72 KSP unpreconditioned resid norm 3.507340393789e-08 true resid norm 3.507340393789e-08 ||r(i)||/||b|| 4.140368399292e-02 73 KSP unpreconditioned resid norm 3.378527900460e-08 true resid norm 3.378527900460e-08 ||r(i)||/||b|| 3.988306974699e-02 74 KSP unpreconditioned resid norm 3.213894671931e-08 true resid norm 3.213894671931e-08 ||r(i)||/||b|| 3.793959651558e-02 75 KSP unpreconditioned resid norm 3.071349886547e-08 true resid norm 3.071349886547e-08 ||r(i)||/||b|| 3.625687439960e-02 76 KSP unpreconditioned resid norm 2.982607199289e-08 true resid norm 2.982607199289e-08 ||r(i)||/||b|| 3.520927885216e-02 77 KSP unpreconditioned resid norm 2.896588396864e-08 true resid norm 2.896588396864e-08 ||r(i)||/||b|| 3.419383840065e-02 78 KSP unpreconditioned resid norm 2.849350659098e-08 true resid norm 2.849350659098e-08 ||r(i)||/||b|| 3.363620322772e-02 79 KSP unpreconditioned resid norm 2.839644438060e-08 true resid norm 2.839644438060e-08 ||r(i)||/||b|| 3.352162258726e-02 80 KSP unpreconditioned resid norm 2.878125026793e-08 true resid norm 2.878125026793e-08 ||r(i)||/||b|| 3.397588078775e-02 81 KSP unpreconditioned resid norm 2.927107179400e-08 true resid norm 2.927107179400e-08 ||r(i)||/||b|| 3.455410854444e-02 82 KSP unpreconditioned resid norm 2.995930520842e-08 true resid norm 2.995930520842e-08 ||r(i)||/||b|| 3.536655888016e-02 83 KSP unpreconditioned resid norm 3.045914214927e-08 true resid norm 3.045914214927e-08 ||r(i)||/||b|| 3.595660969995e-02 84 KSP unpreconditioned resid norm 3.095556579777e-08 true resid norm 3.095556579777e-08 ||r(i)||/||b|| 3.654263117382e-02 85 KSP unpreconditioned resid norm 3.108374161273e-08 true resid norm 3.108374161273e-08 ||r(i)||/||b|| 3.669394100812e-02 86 KSP unpreconditioned resid norm 3.098332255348e-08 true resid norm 3.098332255348e-08 ||r(i)||/||b|| 3.657539765249e-02 87 KSP unpreconditioned resid norm 3.090888002411e-08 true resid norm 3.090888002411e-08 ||r(i)||/||b|| 3.648751924276e-02 88 KSP unpreconditioned resid norm 3.060254382784e-08 true resid norm 3.060254382784e-08 ||r(i)||/||b|| 3.612589346250e-02 89 KSP unpreconditioned resid norm 3.054287138188e-08 true resid norm 3.054287138188e-08 ||r(i)||/||b|| 3.605545093858e-02 90 KSP unpreconditioned resid norm 3.019326857855e-08 true resid norm 3.019326857855e-08 ||r(i)||/||b|| 3.564274950767e-02 91 KSP unpreconditioned resid norm 2.981415246582e-08 true resid norm 2.981415246582e-08 ||r(i)||/||b|| 3.519520800996e-02 92 KSP unpreconditioned resid norm 2.934425853265e-08 true resid norm 2.934425853265e-08 ||r(i)||/||b|| 3.464050451002e-02 93 KSP unpreconditioned resid norm 2.892868376837e-08 true resid norm 2.892868376837e-08 ||r(i)||/||b|| 3.414992406205e-02 94 KSP unpreconditioned resid norm 2.807160358194e-08 true resid norm 2.807160358194e-08 ||r(i)||/||b|| 3.313815237150e-02 95 KSP unpreconditioned resid norm 2.733196031734e-08 true resid norm 2.733196031734e-08 ||r(i)||/||b|| 3.226501339561e-02 96 KSP unpreconditioned resid norm 2.674172249913e-08 true resid norm 2.674172249913e-08 ||r(i)||/||b|| 3.156824554984e-02 97 KSP unpreconditioned resid norm 2.570861397552e-08 true resid norm 2.570861397552e-08 ||r(i)||/||b|| 3.034867476288e-02 98 KSP unpreconditioned resid norm 2.480290643102e-08 true resid norm 2.480290643102e-08 ||r(i)||/||b|| 2.927949912687e-02 99 KSP unpreconditioned resid norm 2.393472264734e-08 true resid norm 2.393472264734e-08 ||r(i)||/||b|| 2.825461978836e-02 100 KSP unpreconditioned resid norm 2.275435374520e-08 true resid norm 2.275435374520e-08 ||r(i)||/||b|| 2.686121009520e-02 101 KSP unpreconditioned resid norm 2.178902400109e-08 true resid norm 2.178902400109e-08 ||r(i)||/||b|| 2.572165125041e-02 102 KSP unpreconditioned resid norm 2.056965586266e-08 true resid norm 2.056965586266e-08 ||r(i)||/||b|| 2.428220348069e-02 103 KSP unpreconditioned resid norm 1.971345133315e-08 true resid norm 1.971345133315e-08 ||r(i)||/||b|| 2.327146549143e-02 104 KSP unpreconditioned resid norm 1.874079176158e-08 true resid norm 1.874079176158e-08 ||r(i)||/||b|| 2.212325388341e-02 105 KSP unpreconditioned resid norm 1.786435683475e-08 true resid norm 1.786435683475e-08 ||r(i)||/||b|| 2.108863418083e-02 106 KSP unpreconditioned resid norm 1.706457796840e-08 true resid norm 1.706457796840e-08 ||r(i)||/||b|| 2.014450593183e-02 107 KSP unpreconditioned resid norm 1.630473998948e-08 true resid norm 1.630473998948e-08 ||r(i)||/||b|| 1.924752736594e-02 108 KSP unpreconditioned resid norm 1.574283867690e-08 true resid norm 1.574283867690e-08 ||r(i)||/||b|| 1.858421038585e-02 109 KSP unpreconditioned resid norm 1.556968009361e-08 true resid norm 1.556968009361e-08 ||r(i)||/||b|| 1.837979899551e-02 110 KSP unpreconditioned resid norm 1.523973136297e-08 true resid norm 1.523973136297e-08 ||r(i)||/||b|| 1.799029893439e-02 111 KSP unpreconditioned resid norm 1.488557248475e-08 true resid norm 1.488557248475e-08 ||r(i)||/||b|| 1.757221911804e-02 112 KSP unpreconditioned resid norm 1.449864006948e-08 true resid norm 1.449864006948e-08 ||r(i)||/||b|| 1.711545057978e-02 113 KSP unpreconditioned resid norm 1.432126672372e-08 true resid norm 1.432126672372e-08 ||r(i)||/||b|| 1.690606371874e-02 114 KSP unpreconditioned resid norm 1.415386315852e-08 true resid norm 1.415386315852e-08 ||r(i)||/||b|| 1.670844605023e-02 115 KSP unpreconditioned resid norm 1.382237564251e-08 true resid norm 1.382237564251e-08 ||r(i)||/||b|| 1.631712947358e-02 116 KSP unpreconditioned resid norm 1.352938132525e-08 true resid norm 1.352938132525e-08 ||r(i)||/||b|| 1.597125360294e-02 117 KSP unpreconditioned resid norm 1.332023721901e-08 true resid norm 1.332023721901e-08 ||r(i)||/||b|| 1.572436178431e-02 118 KSP unpreconditioned resid norm 1.294810005035e-08 true resid norm 1.294810005035e-08 ||r(i)||/||b|| 1.528505883668e-02 119 KSP unpreconditioned resid norm 1.260835085133e-08 true resid norm 1.260835085133e-08 ||r(i)||/||b|| 1.488398945380e-02 120 KSP unpreconditioned resid norm 1.232815412303e-08 true resid norm 1.232815412303e-08 ||r(i)||/||b|| 1.455322096566e-02 121 KSP unpreconditioned resid norm 1.200898521152e-08 true resid norm 1.200898521152e-08 ||r(i)||/||b|| 1.417644633677e-02 122 KSP unpreconditioned resid norm 1.180686235502e-08 true resid norm 1.180686235502e-08 ||r(i)||/||b|| 1.393784300949e-02 123 KSP unpreconditioned resid norm 1.152359469099e-08 true resid norm 1.152359469099e-08 ||r(i)||/||b|| 1.360344932282e-02 124 KSP unpreconditioned resid norm 1.129846801969e-08 true resid norm 1.129846801969e-08 ||r(i)||/||b|| 1.333769030003e-02 125 KSP unpreconditioned resid norm 1.081947011589e-08 true resid norm 1.081947011589e-08 ||r(i)||/||b|| 1.277223968459e-02 126 KSP unpreconditioned resid norm 1.036954512419e-08 true resid norm 1.036954512419e-08 ||r(i)||/||b|| 1.224110925283e-02 127 KSP unpreconditioned resid norm 9.744507014406e-09 true resid norm 9.744507014406e-09 ||r(i)||/||b|| 1.150326012855e-02 128 KSP unpreconditioned resid norm 9.160148503681e-09 true resid norm 9.160148503681e-09 ||r(i)||/||b|| 1.081343272658e-02 129 KSP unpreconditioned resid norm 8.713396021123e-09 true resid norm 8.713396021123e-09 ||r(i)||/||b|| 1.028604739941e-02 130 KSP unpreconditioned resid norm 8.065538786935e-09 true resid norm 8.065538786935e-09 ||r(i)||/||b|| 9.521260604138e-03 131 KSP unpreconditioned resid norm 7.600710595219e-09 true resid norm 7.600710595219e-09 ||r(i)||/||b|| 8.972537144194e-03 132 KSP unpreconditioned resid norm 7.042499013017e-09 true resid norm 7.042499013017e-09 ||r(i)||/||b|| 8.313575841447e-03 133 KSP unpreconditioned resid norm 6.488293059262e-09 true resid norm 6.488293059262e-09 ||r(i)||/||b|| 7.659343129478e-03 134 KSP unpreconditioned resid norm 6.095008004956e-09 true resid norm 6.095008004956e-09 ||r(i)||/||b|| 7.195075385850e-03 135 KSP unpreconditioned resid norm 5.689629867887e-09 true resid norm 5.689629867887e-09 ||r(i)||/||b|| 6.716531919850e-03 136 KSP unpreconditioned resid norm 5.350918350727e-09 true resid norm 5.350918350727e-09 ||r(i)||/||b|| 6.316687506514e-03 137 KSP unpreconditioned resid norm 5.077782670826e-09 true resid norm 5.077782670826e-09 ||r(i)||/||b|| 5.994254491520e-03 138 KSP unpreconditioned resid norm 4.693456261445e-09 true resid norm 4.693456261445e-09 ||r(i)||/||b|| 5.540562308339e-03 139 KSP unpreconditioned resid norm 4.327021695310e-09 true resid norm 4.327021695310e-09 ||r(i)||/||b|| 5.107991206681e-03 140 KSP unpreconditioned resid norm 4.022960135943e-09 true resid norm 4.022960135943e-09 ||r(i)||/||b|| 4.749050604830e-03 141 KSP unpreconditioned resid norm 3.803745635547e-09 true resid norm 3.803745635547e-09 ||r(i)||/||b|| 4.490270820662e-03 142 KSP unpreconditioned resid norm 3.581362358291e-09 true resid norm 3.581362358291e-09 ||r(i)||/||b|| 4.227750337817e-03 143 KSP unpreconditioned resid norm 3.408132039231e-09 true resid norm 3.408132039231e-09 ||r(i)||/||b|| 4.023254264351e-03 144 KSP unpreconditioned resid norm 3.344832032939e-09 true resid norm 3.344832032939e-09 ||r(i)||/||b|| 3.948529454011e-03 145 KSP unpreconditioned resid norm 3.208107857745e-09 true resid norm 3.208107857745e-09 ||r(i)||/||b|| 3.787128394851e-03 146 KSP unpreconditioned resid norm 3.087343355611e-09 true resid norm 3.087343355611e-09 ||r(i)||/||b|| 3.644567516164e-03 147 KSP unpreconditioned resid norm 3.022538018497e-09 true resid norm 3.022538018497e-09 ||r(i)||/||b|| 3.568065682932e-03 148 KSP unpreconditioned resid norm 2.923230289033e-09 true resid norm 2.923230289033e-09 ||r(i)||/||b|| 3.450834237246e-03 149 KSP unpreconditioned resid norm 2.833109002568e-09 true resid norm 2.833109002568e-09 ||r(i)||/||b|| 3.344447264586e-03 150 KSP unpreconditioned resid norm 2.753952392615e-09 true resid norm 2.753952392615e-09 ||r(i)||/||b|| 3.251003945818e-03 151 KSP unpreconditioned resid norm 2.677124193740e-09 true resid norm 2.677124193740e-09 ||r(i)||/||b|| 3.160309285168e-03 152 KSP unpreconditioned resid norm 2.578412706791e-09 true resid norm 2.578412706791e-09 ||r(i)||/||b|| 3.043781695792e-03 153 KSP unpreconditioned resid norm 2.449680459273e-09 true resid norm 2.449680459273e-09 ||r(i)||/||b|| 2.891815000305e-03 154 KSP unpreconditioned resid norm 2.350739446063e-09 true resid norm 2.350739446063e-09 ||r(i)||/||b|| 2.775016458249e-03 155 KSP unpreconditioned resid norm 2.230331194218e-09 true resid norm 2.230331194218e-09 ||r(i)||/||b|| 2.632876128261e-03 156 KSP unpreconditioned resid norm 2.095716328619e-09 true resid norm 2.095716328619e-09 ||r(i)||/||b|| 2.473965080851e-03 157 KSP unpreconditioned resid norm 1.967063208093e-09 true resid norm 1.967063208093e-09 ||r(i)||/||b|| 2.322091793720e-03 158 KSP unpreconditioned resid norm 1.861768684115e-09 true resid norm 1.861768684115e-09 ||r(i)||/||b|| 2.197793017225e-03 159 KSP unpreconditioned resid norm 1.789027091537e-09 true resid norm 1.789027091537e-09 ||r(i)||/||b|| 2.111922540622e-03 160 KSP unpreconditioned resid norm 1.684922987165e-09 true resid norm 1.684922987165e-09 ||r(i)||/||b|| 1.989029038542e-03 161 KSP unpreconditioned resid norm 1.582609227035e-09 true resid norm 1.582609227035e-09 ||r(i)||/||b|| 1.868249013882e-03 162 KSP unpreconditioned resid norm 1.501597611234e-09 true resid norm 1.501597611234e-09 ||r(i)||/||b|| 1.772615885534e-03 163 KSP unpreconditioned resid norm 1.395819010765e-09 true resid norm 1.395819010765e-09 ||r(i)||/||b|| 1.647745663219e-03 164 KSP unpreconditioned resid norm 1.271906790435e-09 true resid norm 1.271906790435e-09 ||r(i)||/||b|| 1.501468945326e-03 165 KSP unpreconditioned resid norm 1.160968729258e-09 true resid norm 1.160968729258e-09 ||r(i)||/||b|| 1.370508048690e-03 166 KSP unpreconditioned resid norm 1.086312221042e-09 true resid norm 1.086312221042e-09 ||r(i)||/||b|| 1.282377039802e-03 167 KSP unpreconditioned resid norm 1.010733613894e-09 true resid norm 1.010733613894e-09 ||r(i)||/||b|| 1.193157505465e-03 168 KSP unpreconditioned resid norm 9.659506407625e-10 true resid norm 9.659506407625e-10 ||r(i)||/||b|| 1.140291804973e-03 169 KSP unpreconditioned resid norm 9.142920667557e-10 true resid norm 9.142920667557e-10 ||r(i)||/||b|| 1.079309549658e-03 170 KSP unpreconditioned resid norm 8.777007956409e-10 true resid norm 8.777007956409e-10 ||r(i)||/||b|| 1.036114043775e-03 171 KSP unpreconditioned resid norm 8.651938742751e-10 true resid norm 8.651938742751e-10 ||r(i)||/||b|| 1.021349790472e-03 172 KSP unpreconditioned resid norm 8.449669697471e-10 true resid norm 8.449669697471e-10 ||r(i)||/||b|| 9.974722003548e-04 173 KSP unpreconditioned resid norm 8.226753443690e-10 true resid norm 8.226753443690e-10 ||r(i)||/||b|| 9.711572349047e-04 174 KSP unpreconditioned resid norm 8.119036890805e-10 true resid norm 8.119036890805e-10 ||r(i)||/||b|| 9.584414399840e-04 175 KSP unpreconditioned resid norm 7.969494091924e-10 true resid norm 7.969494091924e-10 ||r(i)||/||b|| 9.407881126958e-04 176 KSP unpreconditioned resid norm 7.927024508324e-10 true resid norm 7.927024508324e-10 ||r(i)||/||b|| 9.357746351850e-04 177 KSP unpreconditioned resid norm 7.828725447824e-10 true resid norm 7.828725447824e-10 ||r(i)||/||b|| 9.241705626377e-04 178 KSP unpreconditioned resid norm 7.494508082515e-10 true resid norm 7.494508082515e-10 ||r(i)||/||b|| 8.847166499160e-04 179 KSP unpreconditioned resid norm 7.231444464622e-10 true resid norm 7.231444464622e-10 ||r(i)||/||b|| 8.536623418581e-04 180 KSP unpreconditioned resid norm 7.011791261722e-10 true resid norm 7.011791261722e-10 ||r(i)||/||b|| 8.277325751978e-04 181 KSP unpreconditioned resid norm 6.696139551270e-10 true resid norm 6.696139551271e-10 ||r(i)||/||b|| 7.904703131872e-04 182 KSP unpreconditioned resid norm 6.451235118753e-10 true resid norm 6.451235118753e-10 ||r(i)||/||b|| 7.615596726621e-04 183 KSP unpreconditioned resid norm 6.306916481370e-10 true resid norm 6.306916481370e-10 ||r(i)||/||b|| 7.445230506476e-04 184 KSP unpreconditioned resid norm 6.100227035205e-10 true resid norm 6.100227035205e-10 ||r(i)||/||b|| 7.201236381217e-04 185 KSP unpreconditioned resid norm 5.919504539386e-10 true resid norm 5.919504539386e-10 ||r(i)||/||b|| 6.987895893349e-04 186 KSP unpreconditioned resid norm 5.856700542909e-10 true resid norm 5.856700542909e-10 ||r(i)||/||b|| 6.913756615957e-04 187 KSP unpreconditioned resid norm 5.715591865793e-10 true resid norm 5.715591865793e-10 ||r(i)||/||b|| 6.747179710952e-04 188 KSP unpreconditioned resid norm 5.547066799722e-10 true resid norm 5.547066799722e-10 ||r(i)||/||b|| 6.548238125674e-04 189 KSP unpreconditioned resid norm 5.480604885087e-10 true resid norm 5.480604885087e-10 ||r(i)||/||b|| 6.469780724848e-04 190 KSP unpreconditioned resid norm 5.296236594065e-10 true resid norm 5.296236594065e-10 ||r(i)||/||b|| 6.252136424531e-04 191 KSP unpreconditioned resid norm 5.105824359200e-10 true resid norm 5.105824359201e-10 ||r(i)||/||b|| 6.027357329389e-04 192 KSP unpreconditioned resid norm 4.831649558134e-10 true resid norm 4.831649558134e-10 ||r(i)||/||b|| 5.703697645764e-04 193 KSP unpreconditioned resid norm 4.690710598203e-10 true resid norm 4.690710598203e-10 ||r(i)||/||b|| 5.537321089625e-04 194 KSP unpreconditioned resid norm 4.547545034138e-10 true resid norm 4.547545034138e-10 ||r(i)||/||b|| 5.368316057102e-04 195 KSP unpreconditioned resid norm 4.328632397413e-10 true resid norm 4.328632397413e-10 ||r(i)||/||b|| 5.109892618959e-04 196 KSP unpreconditioned resid norm 4.079089522157e-10 true resid norm 4.079089522157e-10 ||r(i)||/||b|| 4.815310594126e-04 197 KSP unpreconditioned resid norm 3.873059797615e-10 true resid norm 3.873059797615e-10 ||r(i)||/||b|| 4.572095261415e-04 198 KSP unpreconditioned resid norm 3.729969299802e-10 true resid norm 3.729969299802e-10 ||r(i)||/||b|| 4.403178843598e-04 199 KSP unpreconditioned resid norm 3.477970037895e-10 true resid norm 3.477970037895e-10 ||r(i)||/||b|| 4.105697087196e-04 200 KSP unpreconditioned resid norm 3.282214610955e-10 true resid norm 3.282214610955e-10 ||r(i)||/||b|| 3.874610425312e-04 201 KSP unpreconditioned resid norm 3.120709893337e-10 true resid norm 3.120709893337e-10 ||r(i)||/||b|| 3.683956267436e-04 202 KSP unpreconditioned resid norm 2.888671401828e-10 true resid norm 2.888671401828e-10 ||r(i)||/||b|| 3.410037933371e-04 203 KSP unpreconditioned resid norm 2.762162725563e-10 true resid norm 2.762162725563e-10 ||r(i)||/||b|| 3.260696133992e-04 204 KSP unpreconditioned resid norm 2.633334545155e-10 true resid norm 2.633334545155e-10 ||r(i)||/||b|| 3.108616191011e-04 205 KSP unpreconditioned resid norm 2.491344685484e-10 true resid norm 2.491344685484e-10 ||r(i)||/||b|| 2.940999061792e-04 206 KSP unpreconditioned resid norm 2.386964029507e-10 true resid norm 2.386964029507e-10 ||r(i)||/||b|| 2.817779094244e-04 207 KSP unpreconditioned resid norm 2.274820848858e-10 true resid norm 2.274820848858e-10 ||r(i)||/||b|| 2.685395570199e-04 208 KSP unpreconditioned resid norm 2.212409903975e-10 true resid norm 2.212409903975e-10 ||r(i)||/||b|| 2.611720284954e-04 209 KSP unpreconditioned resid norm 2.114227895993e-10 true resid norm 2.114227895993e-10 ||r(i)||/||b|| 2.495817738413e-04 210 KSP unpreconditioned resid norm 2.048332706014e-10 true resid norm 2.048332706014e-10 ||r(i)||/||b|| 2.418029348459e-04 211 KSP unpreconditioned resid norm 1.975994847684e-10 true resid norm 1.975994847684e-10 ||r(i)||/||b|| 2.332635474732e-04 212 KSP unpreconditioned resid norm 1.907641063078e-10 true resid norm 1.907641063078e-10 ||r(i)||/||b|| 2.251944746722e-04 213 KSP unpreconditioned resid norm 1.835317861753e-10 true resid norm 1.835317861753e-10 ||r(i)||/||b|| 2.166568175394e-04 214 KSP unpreconditioned resid norm 1.783343277288e-10 true resid norm 1.783343277288e-10 ||r(i)||/||b|| 2.105212873962e-04 215 KSP unpreconditioned resid norm 1.737252962117e-10 true resid norm 1.737252962116e-10 ||r(i)||/||b|| 2.050803873687e-04 216 KSP unpreconditioned resid norm 1.681090858385e-10 true resid norm 1.681090858385e-10 ||r(i)||/||b|| 1.984505261799e-04 217 KSP unpreconditioned resid norm 1.634109844504e-10 true resid norm 1.634109844504e-10 ||r(i)||/||b|| 1.929044803617e-04 218 KSP unpreconditioned resid norm 1.596828500439e-10 true resid norm 1.596828500438e-10 ||r(i)||/||b|| 1.885034675850e-04 219 KSP unpreconditioned resid norm 1.559655758914e-10 true resid norm 1.559655758914e-10 ||r(i)||/||b|| 1.841152751930e-04 220 KSP unpreconditioned resid norm 1.497441081261e-10 true resid norm 1.497441081261e-10 ||r(i)||/||b|| 1.767709157523e-04 221 KSP unpreconditioned resid norm 1.440543864995e-10 true resid norm 1.440543864995e-10 ||r(i)||/||b|| 1.700542755126e-04 222 KSP unpreconditioned resid norm 1.406455332877e-10 true resid norm 1.406455332877e-10 ||r(i)||/||b|| 1.660301699136e-04 223 KSP unpreconditioned resid norm 1.352855224988e-10 true resid norm 1.352855224988e-10 ||r(i)||/||b|| 1.597027489055e-04 224 KSP unpreconditioned resid norm 1.318566764812e-10 true resid norm 1.318566764812e-10 ||r(i)||/||b|| 1.556550420670e-04 225 KSP unpreconditioned resid norm 1.247714207143e-10 true resid norm 1.247714207143e-10 ||r(i)||/||b|| 1.472909924497e-04 226 KSP unpreconditioned resid norm 1.170326734811e-10 true resid norm 1.170326734811e-10 ||r(i)||/||b|| 1.381555049016e-04 227 KSP unpreconditioned resid norm 1.104850176393e-10 true resid norm 1.104850176393e-10 ||r(i)||/||b|| 1.304260848017e-04 228 KSP unpreconditioned resid norm 1.057860311059e-10 true resid norm 1.057860311059e-10 ||r(i)||/||b|| 1.248789940814e-04 229 KSP unpreconditioned resid norm 1.010024307373e-10 true resid norm 1.010024307373e-10 ||r(i)||/||b|| 1.192320178609e-04 230 KSP unpreconditioned resid norm 9.517109911279e-11 true resid norm 9.517109911278e-11 ||r(i)||/||b|| 1.123482089136e-04 231 KSP unpreconditioned resid norm 8.906657903584e-11 true resid norm 8.906657903583e-11 ||r(i)||/||b|| 1.051419046540e-04 232 KSP unpreconditioned resid norm 8.571475780780e-11 true resid norm 8.571475780779e-11 ||r(i)||/||b|| 1.011851245487e-04 233 KSP unpreconditioned resid norm 8.105732120348e-11 true resid norm 8.105732120347e-11 ||r(i)||/||b|| 9.568708296360e-05 234 KSP unpreconditioned resid norm 7.805220078644e-11 true resid norm 7.805220078644e-11 ||r(i)||/||b|| 9.213957852611e-05 235 KSP unpreconditioned resid norm 7.445788494535e-11 true resid norm 7.445788494536e-11 ||r(i)||/||b|| 8.789653677520e-05 236 KSP unpreconditioned resid norm 7.257750273074e-11 true resid norm 7.257750273076e-11 ||r(i)||/||b|| 8.567677073433e-05 237 KSP unpreconditioned resid norm 7.062425062277e-11 true resid norm 7.062425062278e-11 ||r(i)||/||b|| 8.337098275948e-05 238 KSP unpreconditioned resid norm 6.715189342905e-11 true resid norm 6.715189342907e-11 ||r(i)||/||b|| 7.927191155973e-05 239 KSP unpreconditioned resid norm 6.623125011338e-11 true resid norm 6.623125011340e-11 ||r(i)||/||b|| 7.818510444572e-05 240 KSP unpreconditioned resid norm 6.442916815518e-11 true resid norm 6.442916815520e-11 ||r(i)||/||b|| 7.605777080971e-05 241 KSP unpreconditioned resid norm 6.297584870411e-11 true resid norm 6.297584870414e-11 ||r(i)||/||b|| 7.434214664619e-05 242 KSP unpreconditioned resid norm 6.218304192586e-11 true resid norm 6.218304192588e-11 ||r(i)||/||b|| 7.340624885388e-05 243 KSP unpreconditioned resid norm 6.119793248448e-11 true resid norm 6.119793248450e-11 ||r(i)||/||b|| 7.224334034115e-05 244 KSP unpreconditioned resid norm 6.039481879983e-11 true resid norm 6.039481879984e-11 ||r(i)||/||b|| 7.129527538376e-05 245 KSP unpreconditioned resid norm 6.025966944200e-11 true resid norm 6.025966944201e-11 ||r(i)||/||b|| 7.113573337542e-05 246 KSP unpreconditioned resid norm 5.948714963964e-11 true resid norm 5.948714963965e-11 ||r(i)||/||b|| 7.022378408667e-05 247 KSP unpreconditioned resid norm 5.875133821595e-11 true resid norm 5.875133821596e-11 ||r(i)||/||b|| 6.935516854771e-05 248 KSP unpreconditioned resid norm 5.865559932945e-11 true resid norm 5.865559932945e-11 ||r(i)||/||b|| 6.924215007337e-05 249 KSP unpreconditioned resid norm 5.807527161438e-11 true resid norm 5.807527161438e-11 ||r(i)||/||b|| 6.855708097174e-05 250 KSP unpreconditioned resid norm 5.729567969355e-11 true resid norm 5.729567969356e-11 ||r(i)||/||b|| 6.763678314178e-05 251 KSP unpreconditioned resid norm 5.638458710165e-11 true resid norm 5.638458710166e-11 ||r(i)||/||b|| 6.656125052938e-05 252 KSP unpreconditioned resid norm 5.502345910189e-11 true resid norm 5.502345910190e-11 ||r(i)||/||b|| 6.495445714041e-05 253 KSP unpreconditioned resid norm 5.380650850111e-11 true resid norm 5.380650850112e-11 ||r(i)||/||b|| 6.351786324155e-05 254 KSP unpreconditioned resid norm 5.122861296752e-11 true resid norm 5.122861296753e-11 ||r(i)||/||b|| 6.047469206179e-05 255 KSP unpreconditioned resid norm 4.940773406195e-11 true resid norm 4.940773406196e-11 ||r(i)||/||b|| 5.832516888095e-05 256 KSP unpreconditioned resid norm 4.709810110736e-11 true resid norm 4.709810110736e-11 ||r(i)||/||b|| 5.559867808578e-05 257 KSP unpreconditioned resid norm 4.445230653863e-11 true resid norm 4.445230653862e-11 ||r(i)||/||b|| 5.247535300367e-05 258 KSP unpreconditioned resid norm 4.227887565137e-11 true resid norm 4.227887565136e-11 ||r(i)||/||b|| 4.990964692632e-05 259 KSP unpreconditioned resid norm 3.927450241359e-11 true resid norm 3.927450241358e-11 ||r(i)||/||b|| 4.636302452395e-05 260 KSP unpreconditioned resid norm 3.754948266816e-11 true resid norm 3.754948266814e-11 ||r(i)||/||b|| 4.432666179884e-05 261 KSP unpreconditioned resid norm 3.517864153420e-11 true resid norm 3.517864153417e-11 ||r(i)||/||b|| 4.152791556703e-05 262 KSP unpreconditioned resid norm 3.347259122434e-11 true resid norm 3.347259122432e-11 ||r(i)||/||b|| 3.951394600678e-05 263 KSP unpreconditioned resid norm 3.122604584625e-11 true resid norm 3.122604584623e-11 ||r(i)||/||b|| 3.686192925144e-05 264 KSP unpreconditioned resid norm 2.921844421531e-11 true resid norm 2.921844421529e-11 ||r(i)||/||b|| 3.449198239204e-05 265 KSP unpreconditioned resid norm 2.807535371206e-11 true resid norm 2.807535371204e-11 ||r(i)||/||b|| 3.314257934990e-05 266 KSP unpreconditioned resid norm 2.679326847398e-11 true resid norm 2.679326847397e-11 ||r(i)||/||b|| 3.162909488335e-05 267 KSP unpreconditioned resid norm 2.620100211750e-11 true resid norm 2.620100211750e-11 ||r(i)||/||b|| 3.092993237530e-05 268 KSP unpreconditioned resid norm 2.480717416211e-11 true resid norm 2.480717416212e-11 ||r(i)||/||b|| 2.928453712632e-05 269 KSP unpreconditioned resid norm 2.421629839252e-11 true resid norm 2.421629839252e-11 ||r(i)||/||b|| 2.858701618747e-05 270 KSP unpreconditioned resid norm 2.361009921624e-11 true resid norm 2.361009921625e-11 ||r(i)||/||b|| 2.787140617210e-05 271 KSP unpreconditioned resid norm 2.251561534423e-11 true resid norm 2.251561534425e-11 ||r(i)||/||b|| 2.657938260769e-05 272 KSP unpreconditioned resid norm 2.149855566100e-11 true resid norm 2.149855566102e-11 ||r(i)||/||b|| 2.537875726203e-05 273 KSP unpreconditioned resid norm 2.108747915310e-11 true resid norm 2.108747915312e-11 ||r(i)||/||b|| 2.489348694552e-05 274 KSP unpreconditioned resid norm 2.035331893385e-11 true resid norm 2.035331893387e-11 ||r(i)||/||b|| 2.402682063130e-05 275 KSP unpreconditioned resid norm 1.941734891236e-11 true resid norm 1.941734891238e-11 ||r(i)||/||b|| 2.292192054618e-05 276 KSP unpreconditioned resid norm 1.853704982428e-11 true resid norm 1.853704982430e-11 ||r(i)||/||b|| 2.188273925295e-05 277 KSP unpreconditioned resid norm 1.792368946256e-11 true resid norm 1.792368946257e-11 ||r(i)||/||b|| 2.115867555398e-05 278 KSP unpreconditioned resid norm 1.760944257573e-11 true resid norm 1.760944257573e-11 ||r(i)||/||b|| 2.078771130935e-05 279 KSP unpreconditioned resid norm 1.714545149771e-11 true resid norm 1.714545149771e-11 ||r(i)||/||b|| 2.023997605092e-05 280 KSP unpreconditioned resid norm 1.671909267963e-11 true resid norm 1.671909267963e-11 ||r(i)||/||b|| 1.973666517175e-05 281 KSP unpreconditioned resid norm 1.609444716094e-11 true resid norm 1.609444716094e-11 ||r(i)||/||b|| 1.899927949600e-05 282 KSP unpreconditioned resid norm 1.566178459113e-11 true resid norm 1.566178459112e-11 ||r(i)||/||b|| 1.848852712227e-05 283 KSP unpreconditioned resid norm 1.514203649442e-11 true resid norm 1.514203649441e-11 ||r(i)||/||b|| 1.787497144942e-05 284 KSP unpreconditioned resid norm 1.456362863785e-11 true resid norm 1.456362863785e-11 ||r(i)||/||b|| 1.719216871506e-05 285 KSP unpreconditioned resid norm 1.376505865010e-11 true resid norm 1.376505865009e-11 ||r(i)||/||b|| 1.624946753105e-05 286 KSP unpreconditioned resid norm 1.303680225659e-11 true resid norm 1.303680225658e-11 ||r(i)||/||b|| 1.538977060411e-05 287 KSP unpreconditioned resid norm 1.225258977795e-11 true resid norm 1.225258977795e-11 ||r(i)||/||b|| 1.446401826750e-05 288 KSP unpreconditioned resid norm 1.121424657129e-11 true resid norm 1.121424657129e-11 ||r(i)||/||b|| 1.323826800725e-05 289 KSP unpreconditioned resid norm 1.054203483048e-11 true resid norm 1.054203483047e-11 ||r(i)||/||b|| 1.244473104282e-05 290 KSP unpreconditioned resid norm 9.820019392645e-12 true resid norm 9.820019392636e-12 ||r(i)||/||b|| 1.159240148054e-05 291 KSP unpreconditioned resid norm 9.223807475811e-12 true resid norm 9.223807475802e-12 ||r(i)||/||b|| 1.088858129128e-05 292 KSP unpreconditioned resid norm 8.482817720280e-12 true resid norm 8.482817720271e-12 ||r(i)||/||b|| 1.001385280087e-05 293 KSP unpreconditioned resid norm 8.015330479160e-12 true resid norm 8.015330479152e-12 ||r(i)||/||b|| 9.461990368692e-06 KSP Object: 128 MPI processes type: cg maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning has attached null space using UNPRECONDITIONED norm type for convergence test PC Object: 128 MPI processes type: mg MG: type is MULTIPLICATIVE, levels=5 cycles=v Cycles per PCApply=1 Using Galerkin computed coarse grid matrices Coarse grid solver -- level ------------------------------- KSP Object: (mg_coarse_) 128 MPI processes type: preonly maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (mg_coarse_) 128 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: natural factor fill ratio given 0, needed 0 Factored matrix follows: Matrix Object: 128 MPI processes type: mpiaij rows=1024, cols=1024 package used to perform factorization: superlu_dist total: nonzeros=0, allocated nonzeros=0 total number of mallocs used during MatSetValues calls =0 SuperLU_DIST run parameters: Process grid nprow 16 x npcol 8 Equilibrate matrix TRUE Matrix input mode 1 Replace tiny pivots TRUE Use iterative refinement FALSE Processors in row 16 col partition 8 Row permutation LargeDiag Column permutation METIS_AT_PLUS_A Parallel symbolic factorization FALSE Repeated factorization SamePattern_SameRowPerm linear system matrix = precond matrix: Matrix Object: 128 MPI processes type: mpiaij rows=1024, cols=1024 total: nonzeros=27648, allocated nonzeros=27648 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Down solver (pre-smoother) on level 1 ------------------------------- KSP Object: (mg_levels_1_) 128 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (mg_levels_1_) 128 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 128 MPI processes type: mpiaij rows=8192, cols=8192 total: nonzeros=221184, allocated nonzeros=221184 total number of mallocs used during MatSetValues calls =0 using I-node (on process 0) routines: found 16 nodes, limit used is 5 Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 2 ------------------------------- KSP Object: (mg_levels_2_) 128 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (mg_levels_2_) 128 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 128 MPI processes type: mpiaij rows=65536, cols=65536 total: nonzeros=1769472, allocated nonzeros=1769472 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 3 ------------------------------- KSP Object: (mg_levels_3_) 128 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (mg_levels_3_) 128 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 128 MPI processes type: mpiaij rows=524288, cols=524288 total: nonzeros=14155776, allocated nonzeros=14155776 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 4 ------------------------------- KSP Object: (mg_levels_4_) 128 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (mg_levels_4_) 128 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 128 MPI processes type: mpiaij rows=4194304, cols=4194304 total: nonzeros=29360128, allocated nonzeros=29360128 total number of mallocs used during MatSetValues calls =0 Up solver (post-smoother) same as down solver (pre-smoother) linear system matrix = precond matrix: Matrix Object: 128 MPI processes type: mpiaij rows=4194304, cols=4194304 total: nonzeros=29360128, allocated nonzeros=29360128 total number of mallocs used during MatSetValues calls =0 ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./hit on a interlagos-64idx-pgi-opt named nid25475 with 128 processors, by Unknown Fri Sep 20 12:56:00 2013 Using Petsc Release Version 3.4.2, Jul, 02, 2013 Max Max/Min Avg Total Time (sec): 3.240e+03 1.00000 3.240e+03 Objects: 5.352e+05 1.00000 5.352e+05 Flops: 4.067e+11 1.00000 4.067e+11 5.206e+13 Flops/sec: 1.255e+08 1.00000 1.255e+08 1.606e+10 MPI Messages: 7.811e+06 1.00000 7.811e+06 9.998e+08 MPI Message Lengths: 2.990e+10 1.00000 3.827e+03 3.827e+12 MPI Reductions: 1.522e+06 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 3.2403e+03 100.0% 5.2056e+13 100.0% 9.998e+08 100.0% 3.827e+03 100.0% 1.522e+06 100.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %f - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage VecTDot 177386 1.0 5.7980e+01 2.1 1.16e+10 1.0 0.0e+00 0.0e+00 1.8e+05 1 3 0 0 12 1 3 0 0 12 25664 VecNorm 179519 1.0 4.6141e+01 3.5 1.18e+10 1.0 0.0e+00 0.0e+00 1.8e+05 1 3 0 0 12 1 3 0 0 12 32637 VecScale 354772 1.0 2.3496e+00 1.5 1.00e+09 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 54733 VecCopy 179519 1.0 8.4071e+00 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 1243857 1.0 1.8819e+01 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 VecAXPY 177386 1.0 2.6131e+01 1.2 1.16e+10 1.0 0.0e+00 0.0e+00 0.0e+00 1 3 0 0 0 1 3 0 0 0 56944 VecAYPX 532158 1.0 4.1760e+01 1.3 1.20e+10 1.0 0.0e+00 0.0e+00 0.0e+00 1 3 0 0 0 1 3 0 0 0 36832 VecScatterBegin 1598611 1.0 9.1558e+01 1.3 0.00e+00 0.0 1.0e+09 3.8e+03 0.0e+00 3 0100100 0 3 0100100 0 0 VecScatterEnd 1598611 1.0 2.5825e+02 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 6 0 0 0 0 6 0 0 0 0 0 MatMult 532869 1.0 1.0230e+03 1.1 1.36e+11 1.0 4.1e+08 6.2e+03 0.0e+00 29 33 41 66 0 29 33 41 66 0 16969 MatMultAdd 354772 1.0 1.6270e+02 1.1 2.24e+10 1.0 1.4e+08 9.9e+02 0.0e+00 5 6 14 4 0 5 6 14 4 0 17634 MatMultTranspose 354776 1.0 1.8670e+02 1.3 2.24e+10 1.0 1.4e+08 9.9e+02 0.0e+00 5 6 14 4 0 5 6 14 4 0 15367 MatSolve 88693 1.0 1.6765e+02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 4 0 0 0 0 4 0 0 0 0 0 MatSOR 709544 1.0 1.3796e+03 1.1 1.79e+11 1.0 3.2e+08 3.2e+03 7.1e+05 41 44 32 27 47 41 44 32 27 47 16627 MatLUFactorSym 1 1.0 6.4993e-04 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatLUFactorNum 1 1.0 1.2938e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 728 1.0 2.8366e+0018.5 0.00e+00 0.0 0.0e+00 0.0e+00 1.4e+03 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 728 1.0 2.5708e+00 1.2 0.00e+00 0.0 1.2e+04 1.1e+03 7.2e+01 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 1 1.0 5.0068e-06 5.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 2.0981e-05 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatView 4977 1.0 1.3075e+00 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 5.0e+03 0 0 0 0 0 0 0 0 0 0 0 MatPtAP 4 1.0 2.1368e-01 1.0 5.11e+06 1.0 2.5e+04 6.0e+03 1.0e+02 0 0 0 0 0 0 0 0 0 0 3058 MatPtAPSymbolic 4 1.0 1.5158e-01 1.1 0.00e+00 0.0 1.5e+04 7.8e+03 6.0e+01 0 0 0 0 0 0 0 0 0 0 0 MatPtAPNumeric 4 1.0 7.0144e-02 1.1 5.11e+06 1.0 9.7e+03 3.1e+03 4.0e+01 0 0 0 0 0 0 0 0 0 0 9316 MatGetLocalMat 4 1.0 2.1817e-02 3.3 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetBrAoCol 4 1.0 2.9885e-02 3.2 0.00e+00 0.0 1.1e+04 8.4e+03 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetSymTrans 8 1.0 9.0678e-03 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSetUp 6 1.0 1.0205e-02 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 2.2e+01 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 711 1.0 3.0209e+03 1.0 4.07e+11 1.0 1.0e+09 3.8e+03 1.5e+06 93100100100 99 93100100100 99 17232 PCSetUp 1 1.0 4.2324e-01 1.0 5.36e+06 1.0 3.4e+04 4.6e+03 3.0e+02 0 0 0 0 0 0 0 0 0 0 1620 PCApply 88693 1.0 2.2119e+03 1.0 2.87e+11 1.0 9.1e+08 2.6e+03 7.1e+05 67 71 91 61 47 67 71 91 61 47 16614 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Vector 535068 535068 74824032416 0 Vector Scatter 19 19 22572 0 Matrix 38 38 14004608 0 Matrix Null Space 1 1 652 0 Distributed Mesh 5 5 830792 0 Bipartite Graph 10 10 8560 0 Index Set 47 47 534480 0 IS L to G Mapping 5 5 405756 0 Krylov Solver 7 7 9536 0 DMKSP interface 3 3 2088 0 Preconditioner 7 7 7352 0 Viewer 1 0 0 0 ======================================================================================================================== Average time to get PetscTime(): 4.05312e-07 Average time for MPI_Barrier(): 1.32084e-05 Average time for zero size MPI_Send(): 2.75671e-06 #PETSc Option Table entries: -ksp_monitor_true_residual -ksp_view -log_summary -mg_coarse_pc_factor_mat_solver_package superlu_dist -mg_coarse_pc_type lu -mg_levels_ksp_max_it 1 -mg_levels_ksp_type richardson -options_left -pc_mg_galerkin -pc_mg_levels 5 -pc_type mg #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 8 Configure run at: Wed Aug 28 23:25:43 2013 Configure options: --known-level1-dcache-size=16384 --known-level1-dcache-linesize=64 --known-level1-dcache-assoc=4 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=0 --known-mpi-c-double-complex=0 --with-batch="1 " --known-mpi-shared="0 " --known-memcmp-ok --with-blas-lapack-lib="-L/opt/acml/5.3.0/pgi64/lib -lacml" --COPTFLAGS="-O3 -fastsse" --FOPTFLAGS="-O3 -fastsse" --CXXOPTFLAGS="-O3 -fastsse" --with-x="0 " --with-debugging="0 " --with-clib-autodetect="0 " --with-cxxlib-autodetect="0 " --with-fortranlib-autodetect="0 " --with-shared-libraries=0 --with-dynamic-loading=0 --with-mpi-compilers="1 " --known-mpi-shared-libraries=0 --with-64-bit-indices --download-blacs="1 " --download-scalapack="1 " --download-superlu_dist="1 " --download-metis="1 " --download-parmetis="1 " --with-cc=cc --with-cxx=CC --with-fc=ftn PETSC_ARCH=interlagos-64idx-pgi-opt ----------------------------------------- Libraries compiled on Wed Aug 28 23:25:43 2013 on h2ologin3 Machine characteristics: Linux-2.6.32.59-0.7-default-x86_64-with-SuSE-11-x86_64 Using PETSc directory: /mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2 Using PETSc arch: interlagos-64idx-pgi-opt ----------------------------------------- Using C compiler: cc -O3 -fastsse ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: ftn -O3 -fastsse ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/include -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/include -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/include -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/include -I/opt/cray/udreg/2.3.2-1.0402.7311.2.1.gem/include -I/opt/cray/ugni/5.0-1.0402.7128.7.6.gem/include -I/opt/cray/pmi/4.0.1-1.0000.9421.73.3.gem/include -I/opt/cray/dmapp/4.0.1-1.0402.7439.5.1.gem/include -I/opt/cray/gni-headers/2.1-1.0402.7082.6.2.gem/include -I/opt/cray/xpmem/0.1-2.0402.44035.2.1.gem/include -I/opt/cray/rca/1.0.0-2.0402.42153.2.106.gem/include -I/opt/cray-hss-devel/7.0.0/include -I/opt/cray/krca/1.0.0-2.0402.42157.2.94.gem/include -I/opt/cray/mpt/6.0.1/gni/mpich2-pgi/121/include -I/opt/acml/5.3.0/pgi64_fma4/include -I/opt/cray/libsci/12.1.01/pgi/121/interlagos/include -I/opt/fftw/3.3.0.3/interlagos/include -I/usr/include/alps -I/opt/pgi/13.6.0/linux86-64/13.6/include -I/opt/cray/xe-sysroot/4.2.24/usr/include ----------------------------------------- Using C linker: cc Using Fortran linker: ftn Using libraries: -Wl,-rpath,/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -L/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -lpetsc -Wl,-rpath,/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -L/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -lsuperlu_dist_3.3 -L/opt/acml/5.3.0/pgi64/lib -lacml -lpthread -lparmetis -lmetis -ldl ----------------------------------------- #PETSc Option Table entries: -ksp_monitor_true_residual -ksp_view -log_summary -mg_coarse_pc_factor_mat_solver_package superlu_dist -mg_coarse_pc_type lu -mg_levels_ksp_max_it 1 -mg_levels_ksp_type richardson -options_left -pc_mg_galerkin -pc_mg_levels 5 -pc_type mg #End of PETSc Option Table entries There are no unused options. From knepley at gmail.com Mon Sep 23 11:11:00 2013 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 23 Sep 2013 09:11:00 -0700 Subject: [petsc-users] How to speed up geometric multigrid In-Reply-To: <52406460.3060905@uci.edu> References: <52406460.3060905@uci.edu> Message-ID: On Mon, Sep 23, 2013 at 8:55 AM, Michele Rosso wrote: > Hi, > > I am successfully using PETSc to solve a 3D Poisson's equation with CG + > MG . Such equation arises from a projection algorithm for a multiphase > incompressible flow simulation. > I set up the solver as I was suggested to do in a previous thread(title: "GAMG speed") and run a test case (liquid droplet with surface > tension falling under the effect of gravity in a quiescent fluid). > The solution of the Poisson Equation via multigrid is correct but it > becomes progressively slower and slower as the simulation progresses (I am > performing successive solves) due to an increase in the number of > iterations. > Since the solution of the Poisson equation is mission-critical, I need to > speed it up as much as I can. > Could you please help me out with this? > First, what does the coefficient look like? Second, what are the boundary conditions? Matt > I run the test case with the following options: > > -pc_type mg -pc_mg_galerkin -pc_mg_levels 5 -mg_levels_ksp_type > richardson -mg_levels_ksp_max_it 1 > -mg_coarse_pc_type lu -mg_coarse_pc_factor_mat_solver_package > superlu_dist > -log_summary -ksp_view -ksp_monitor_true_residual -options_left > > Please find the diagnostic for the final solve in the attached file > "final.txt'. > Thank you, > > Michele > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Mon Sep 23 11:24:40 2013 From: mfadams at lbl.gov (Mark F. Adams) Date: Mon, 23 Sep 2013 12:24:40 -0400 Subject: [petsc-users] How to speed up geometric multigrid In-Reply-To: <52406460.3060905@uci.edu> References: <52406460.3060905@uci.edu> Message-ID: On Sep 23, 2013, at 11:55 AM, Michele Rosso wrote: > Hi, > > I am successfully using PETSc to solve a 3D Poisson's equation with CG + MG . Such equation arises from a projection algorithm for a multiphase incompressible flow simulation. > I set up the solver as I was suggested to do in a previous thread (title: "GAMG speed") and run a test case (liquid droplet with surface tension falling under the effect of gravity in a quiescent fluid). > The solution of the Poisson Equation via multigrid is correct but it becomes progressively slower and slower as the simulation progresses (I am performing successive solves) due to an increase in the number of iterations. > Since the solution of the Poisson equation is mission-critical, I need to speed it up as much as I can. > Could you please help me out with this? > Just to add, it is not unusual as structures develop in your simulation to see an increase in iterations. If you plot the material coefficients I suspect that you will see sharper structures developing in time. Can you look at this in some way or generate some statistics like (max) gradient of material coefficients or even just max & min coefficients? > I run the test case with the following options: > > -pc_type mg -pc_mg_galerkin -pc_mg_levels 5 -mg_levels_ksp_type richardson -mg_levels_ksp_max_it 1 > -mg_coarse_pc_type lu -mg_coarse_pc_factor_mat_solver_package superlu_dist > -log_summary -ksp_view -ksp_monitor_true_residual -options_left > > Please find the diagnostic for the final solve in the attached file "final.txt'. > Thank you, > > Michele > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrosso at uci.edu Mon Sep 23 11:27:45 2013 From: mrosso at uci.edu (Michele Rosso) Date: Mon, 23 Sep 2013 09:27:45 -0700 Subject: [petsc-users] How to speed up geometric multigrid In-Reply-To: References: <52406460.3060905@uci.edu> Message-ID: <52406C01.5090804@uci.edu> The boundary conditions are periodic. The equation I am solving is: div(beta*grad(u))= f where beta is 1 inside the gas phase, 0.001 inside the liquid phase and a value in between for the nodes close to the interface. The system matrix is built so to remain symmetric positive defined despite the coefficients. Michele On 09/23/2013 09:11 AM, Matthew Knepley wrote: > On Mon, Sep 23, 2013 at 8:55 AM, Michele Rosso > wrote: > > Hi, > > I am successfully using PETSc to solve a 3D Poisson's equation > with CG + MG . Such equation arises from a projection algorithm > for a multiphase incompressible flow simulation. > I set up the solver as I was suggested to do in a previous thread > (title: "GAMG speed") and run a test case (liquid droplet with > surface tension falling under the effect of gravity in a quiescent > fluid). > The solution of the Poisson Equation via multigrid is correct but > it becomes progressively slower and slower as the simulation > progresses (I am performing successive solves) due to an increase > in the number of iterations. > Since the solution of the Poisson equation is mission-critical, I > need to speed it up as much as I can. > Could you please help me out with this? > > > First, what does the coefficient look like? > > Second, what are the boundary conditions? > > Matt > > I run the test case with the following options: > > -pc_type mg -pc_mg_galerkin -pc_mg_levels 5 -mg_levels_ksp_type > richardson -mg_levels_ksp_max_it 1 > -mg_coarse_pc_type lu -mg_coarse_pc_factor_mat_solver_package > superlu_dist > -log_summary -ksp_view -ksp_monitor_true_residual -options_left > > Please find the diagnostic for the final solve in the attached > file "final.txt'. > Thank you, > > Michele > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From danyang.su at gmail.com Mon Sep 23 11:29:08 2013 From: danyang.su at gmail.com (Danyang Su) Date: Mon, 23 Sep 2013 09:29:08 -0700 Subject: [petsc-users] Strange efficiency in PETSc-dev using OpenMP In-Reply-To: <76020DE9-2154-484F-9031-952E3EB323C2@mcs.anl.gov> References: <523D482D.5030409@gmail.com> <101244E8-FFFA-4F3F-A4BC-CC47F68D93B7@mcs.anl.gov> <523E059D.6080901@gmail.com> <933EF7F4-598A-4F88-8902-0613E27F0679@mcs.anl.gov> <523E6CAF.9020805@gmail.com> <523F4871.6000803@gmail.com> <76020DE9-2154-484F-9031-952E3EB323C2@mcs.anl.gov> Message-ID: <52406C54.2060709@gmail.com> Hi Barry, Please find the data file in the attachment. The fist value in a*.txt, b*.txt, ia*.txt and ja*.txt is the number of data in this file. I will test with Shri's suggestion "setting the affinities via KMP_AFFINITY" to see if it works. Thanks, Danyang On 22/09/2013 2:24 PM, Barry Smith wrote: > If you run the Openmp compiled version WITHOUT the > > -threadcomm_nthreads 1 > -threadcomm_type openmp > > command line options is it still slow? > > I want to understand if the MPI compile options triggering the much slower run. > > Barry > > On Sep 22, 2013, at 2:43 PM, Danyang Su wrote: > >> Hi Barry, >> >> Thanks, please find the answer bellow. >> >> On 22/09/2013 9:53 AM, Barry Smith wrote: >>> 1) # WARNING!!! # >>> # # >>> # This code was compiled with a debugging option, # >>> # To get timing results run ./configure # >>> # using --with-debugging=no, the performance will # >>> # be generally two or three times faster. # >>> >>> Never time without optimization, it can give very misleading information because different parts of the code speedup very differently when optimized >> With optimization, the problem still exists. See attached log, using 1 thread and 4 threads. >>> 2) Where are the 4 MPI processes being placed on your system? Are they being placed on 4 cores on the same CPU (as with the OpenMP run) or possibly on different CPUs? >> Yes, the system information are as follows >> OS: Windows 7 X64 Pro, CYGWIN >> Processor: Intel Xeon E5-2620 2.0GHz, 6cores/12threads >> Memory: 16GB >> Compiler: Intel Visual Fortran V13.1. >>> 3) Do you have any OpenMP pragmas in your code. Make a run where you take them all out >> For the current, I have no OpenMP pragmas in the code. The code is just the same as I used for PETSc MPI version and it works fine when using MPI. >>> 4) Both runs are actually taking very little time in the solver; >>> >>> KSPSolve 1 1.0 9.2897e-002 >>> KSPSolve 1 1.0 2.9056e-001 >>> >>> How are you getting your matrix? From a file? >> Yes, the matrix are currently from the files. Should this be the problem? The timing is started after the reading matrix. And I didn't see the speedup for the solver, the runtime for kspsolve are almost the same. >> >> nthreads = 1, KSPSolve 1 1.0 6.2800e-002 >> >> nthreads = 4, KSPSolve 1 1.0 5.5090e-002 >> >> The main question is that the program is stuck in the following codes when run with openmp, but no problem when run with mpi. >> >> do i = istart, iend - 1 >> ii = ia_in(i+1) >> jj = ia_in(i+2) >> call MatSetValues(a, ione, i, jj-ii, ja_in(ii:jj-1)-1, a_in(ii:jj-1), Insert_Values, ierr) >> end do >> >> The testing codes has also been attached. I have removed some unnecessary codes, but some are still there. >>> Barry >>> >>> >>> On Sep 21, 2013, at 11:06 PM, Danyang Su wrote: >>> >>>> Hi Shri, >>>> >>>> Thanks for your info. It can work with the option -threadcomm_type openmp. But another problem arises, as described as follows. >>>> >>>> The sparse matrix is 53760*53760 with 1067392 non-zero entries. If the codes is compiled using PETSc-3.4.2, it works fine, the equations can be solved quickly and I can see the speedup. But if the code is compiled using PETSc-dev with OpenMP option, it takes a long time in solving the equations and I cannot see any speedup when more processors are used. >>>> >>>> For PETSc-3.4.2, run by "mpiexec -n 4 ksp_inhm_d -log_summary log_mpi4_petsc3.4.2.log", the iteration and runtime are: >>>> Iterations 6 time_assembly 0.4137E-01 time_ksp 0.9296E-01 >>>> >>>> For PETSc-dev, run by "mpiexec -n 1 ksp_inhm_d -threadcomm_type openmp -threadcomm_nthreads 4 -log_summary log_openmp_petsc_dev.log", the iteration and runtime are: >>>> Iterations 6 time_assembly 0.3595E+03 time_ksp 0.2907E+00 >>>> >>>> Most of the time 'time_assembly 0.3595E+03' is spent on the following codes >>>> do i = istart, iend - 1 >>>> ii = ia_in(i+1) >>>> jj = ia_in(i+2) >>>> call MatSetValues(a, ione, i, jj-ii, ja_in(ii:jj-1)-1, a_in(ii:jj-1), Insert_Values, ierr) >>>> end do >>>> >>>> The log files for both PETSc-3.4.2 and PETSc-dev are attached. >>>> >>>> Is there anything wrong with my codes or with running option? The above codes works fine when using MPICH. >>>> >>>> Thanks and regards, >>>> >>>> Danyang >>>> >>>> On 21/09/2013 2:09 PM, Shri wrote: >>>>> There are three thread communicator types in PETSc. The default is "no thread" which is basically a non-threaded version. The other two types are "openmp" and "pthread". If you want to use OpenMP then use the option -threadcomm_type openmp. >>>>> >>>>> Shri >>>>> >>>>> On Sep 21, 2013, at 3:46 PM, Danyang Su wrote: >>>>> >>>>>> Hi Barry, >>>>>> >>>>>> Thanks for the quick reply. >>>>>> >>>>>> After changing >>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>>> to >>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) >>>>>> and comment out >>>>>> #elif defined(PETSC_HAVE_OPENMP) >>>>>> PETSC_EXTERN PetscStack *petscstack; >>>>>> >>>>>> It can be compiled and validated with "make test". >>>>>> >>>>>> But I still have questions on running the examples. After rebuild the codes (e.g., ksp_ex2f.f), I can run it with "mpiexec -n 1 ksp_ex2f", or "mpiexec -n 4 ksp_ex2f", or "mpiexec -n 1 ksp_ex2f -threadcomm_nthreads 1", but if I run it with "mpiexec -n 1 ksp_ex2f -threadcomm_nthreads 4", there will be a lot of error information (attached). >>>>>> >>>>>> The codes is not modified and there is no OpenMP routines in it. For the current development in my project, I want to keep the OpenMP codes in calculating matrix values, but want to solve it with PETSc (OpenMP). Is it possible? >>>>>> >>>>>> Thanks and regards, >>>>>> >>>>>> Danyang >>>>>> >>>>>> >>>>>> >>>>>> On 21/09/2013 7:26 AM, Barry Smith wrote: >>>>>>> Danyang, >>>>>>> >>>>>>> I don't think the || defined (PETSC_HAVE_OPENMP) belongs in the code below. >>>>>>> >>>>>>> /* Linux functions CPU_SET and others don't work if sched.h is not included before >>>>>>> including pthread.h. Also, these functions are active only if either _GNU_SOURCE >>>>>>> or __USE_GNU is not set (see /usr/include/sched.h and /usr/include/features.h), hence >>>>>>> set these first. >>>>>>> */ >>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>>>> >>>>>>> Edit include/petscerror.h and locate these lines and remove that part and then rerun make all. Let us know if it works or not. >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>> i.e. replace >>>>>>> >>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>>>> >>>>>>> with >>>>>>> >>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) >>>>>>> >>>>>>> On Sep 21, 2013, at 6:53 AM, Matthew Knepley >>>>>>> >>>>>>> wrote: >>>>>>> >>>>>>> >>>>>>>> On Sat, Sep 21, 2013 at 12:18 AM, Danyang Su >>>>>>>> wrote: >>>>>>>> Hi All, >>>>>>>> >>>>>>>> I got error information in compiling petsc-dev with openmp in cygwin. Before, I have successfully compiled petsc-3.4.2 and it works fine. >>>>>>>> The log files have been attached. >>>>>>>> >>>>>>>> The OpenMP configure test is wrong. It clearly fails to find pthread.h, but the test passes. Then in petscerror.h >>>>>>>> we guard pthread.h using PETSC_HAVE_OPENMP. Can someone who knows OpenMP fix this? >>>>>>>> >>>>>>>> Matt >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Danyang >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>> >>>> >> -------------- next part -------------- A non-text attachment was scrubbed... Name: matrix.7z Type: application/octet-stream Size: 3078758 bytes Desc: not available URL: From bsmith at mcs.anl.gov Mon Sep 23 11:35:07 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 23 Sep 2013 11:35:07 -0500 Subject: [petsc-users] How to speed up geometric multigrid In-Reply-To: <52406460.3060905@uci.edu> References: <52406460.3060905@uci.edu> Message-ID: <4499926F-40CE-462C-BFD6-8798A02126FB@mcs.anl.gov> Run with the additional option -pc_mg_log and send us the log file. Barry Maybe we should make this the default somehow. On Sep 23, 2013, at 10:55 AM, Michele Rosso wrote: > Hi, > > I am successfully using PETSc to solve a 3D Poisson's equation with CG + MG . Such equation arises from a projection algorithm for a multiphase incompressible flow simulation. > I set up the solver as I was suggested to do in a previous thread (title: "GAMG speed") and run a test case (liquid droplet with surface tension falling under the effect of gravity in a quiescent fluid). > The solution of the Poisson Equation via multigrid is correct but it becomes progressively slower and slower as the simulation progresses (I am performing successive solves) due to an increase in the number of iterations. > Since the solution of the Poisson equation is mission-critical, I need to speed it up as much as I can. > Could you please help me out with this? > > I run the test case with the following options: > > -pc_type mg -pc_mg_galerkin -pc_mg_levels 5 -mg_levels_ksp_type richardson -mg_levels_ksp_max_it 1 > -mg_coarse_pc_type lu -mg_coarse_pc_factor_mat_solver_package superlu_dist > -log_summary -ksp_view -ksp_monitor_true_residual -options_left > > Please find the diagnostic for the final solve in the attached file "final.txt'. > Thank you, > > Michele > From mrosso at uci.edu Mon Sep 23 11:34:35 2013 From: mrosso at uci.edu (Michele Rosso) Date: Mon, 23 Sep 2013 09:34:35 -0700 Subject: [petsc-users] How to speed up geometric multigrid In-Reply-To: References: <52406460.3060905@uci.edu> Message-ID: <52406D9B.3020203@uci.edu> On 09/23/2013 09:24 AM, Mark F. Adams wrote: > > On Sep 23, 2013, at 11:55 AM, Michele Rosso > wrote: > >> Hi, >> >> I am successfully using PETSc to solve a 3D Poisson's equation with >> CG + MG . Such equation arises from a projection algorithm for a >> multiphase incompressible flow simulation. >> I set up the solver as I was suggested to do in a previous thread >> (title: "GAMG speed") and run a test case (liquid droplet with >> surface tension falling under the effect of gravity in a quiescent >> fluid). >> The solution of the Poisson Equation via multigrid is correct but it >> becomes progressively slower and slower as the simulation progresses >> (I am performing successive solves) due to an increase in the number >> of iterations. >> Since the solution of the Poisson equation is mission-critical, I >> need to speed it up as much as I can. >> Could you please help me out with this? >> > > Just to add, it is not unusual as structures develop in your > simulation to see an increase in iterations. If you plot the material > coefficients I suspect that you will see sharper structures developing > in time. Can you look at this in some way or generate some statistics > like (max) gradient of material coefficients or even just max & min > coefficients? Yes, I suspected that this may have been a reason. Initially both phases are at rest, so basically no fluid structures are present, thus the convergence is very fast(6 iterations maximum). Then vortices start developing both inside and outside the droplet and at that point the number of iteration increases. I noticed that the convergence rate becomes almost steady at a certain point, I guess when the flow is fully developed. I would like to speed up the solve for this last scenario. > >> I run the test case with the following options: >> >> -pc_type mg -pc_mg_galerkin -pc_mg_levels 5 -mg_levels_ksp_type >> richardson -mg_levels_ksp_max_it 1 >> -mg_coarse_pc_type lu -mg_coarse_pc_factor_mat_solver_package >> superlu_dist >> -log_summary -ksp_view -ksp_monitor_true_residual -options_left >> >> Please find the diagnostic for the final solve in the attached file >> "final.txt'. >> Thank you, >> >> Michele >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Sep 23 11:36:39 2013 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 23 Sep 2013 09:36:39 -0700 Subject: [petsc-users] How to speed up geometric multigrid In-Reply-To: <52406D9B.3020203@uci.edu> References: <52406460.3060905@uci.edu> <52406D9B.3020203@uci.edu> Message-ID: On Mon, Sep 23, 2013 at 9:34 AM, Michele Rosso wrote: > > On 09/23/2013 09:24 AM, Mark F. Adams wrote: > > > On Sep 23, 2013, at 11:55 AM, Michele Rosso wrote: > > Hi, > > I am successfully using PETSc to solve a 3D Poisson's equation with CG + > MG . Such equation arises from a projection algorithm for a multiphase > incompressible flow simulation. > I set up the solver as I was suggested to do in a previous thread(title: "GAMG speed") and run a test case (liquid droplet with surface > tension falling under the effect of gravity in a quiescent fluid). > The solution of the Poisson Equation via multigrid is correct but it > becomes progressively slower and slower as the simulation progresses (I am > performing successive solves) due to an increase in the number of > iterations. > Since the solution of the Poisson equation is mission-critical, I need to > speed it up as much as I can. > Could you please help me out with this? > > > Just to add, it is not unusual as structures develop in your simulation > to see an increase in iterations. If you plot the material coefficients I > suspect that you will see sharper structures developing in time. Can you > look at this in some way or generate some statistics like (max) gradient of > material coefficients or even just max & min coefficients? > > Yes, I suspected that this may have been a reason. Initially both phases > are at rest, so basically no fluid structures are present, thus the > convergence is very fast(6 iterations maximum). Then vortices start > developing both inside and outside the droplet and at that point the number > of iteration increases. I noticed that the convergence rate becomes almost > steady at a certain point, I guess when the flow is fully developed. I > would like to speed up the solve for this last scenario. > Mark, is this where we do Bootstrap? :) Thanks, Matt > I run the test case with the following options: > > -pc_type mg -pc_mg_galerkin -pc_mg_levels 5 -mg_levels_ksp_type > richardson -mg_levels_ksp_max_it 1 > -mg_coarse_pc_type lu -mg_coarse_pc_factor_mat_solver_package > superlu_dist > -log_summary -ksp_view -ksp_monitor_true_residual -options_left > > Please find the diagnostic for the final solve in the attached file > "final.txt'. > Thank you, > > Michele > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Mon Sep 23 11:48:55 2013 From: mfadams at lbl.gov (Mark F. Adams) Date: Mon, 23 Sep 2013 12:48:55 -0400 Subject: [petsc-users] How to speed up geometric multigrid In-Reply-To: <52406C01.5090804@uci.edu> References: <52406460.3060905@uci.edu> <52406C01.5090804@uci.edu> Message-ID: On Sep 23, 2013, at 12:27 PM, Michele Rosso wrote: > The boundary conditions are periodic. > The equation I am solving is: > > div(beta*grad(u))= f > > where beta is 1 inside the gas phase, 0.001 inside the liquid phase and a value in between for the nodes close to the interface. This is a pretty big jump for geometric MG. You might try AMG. I suspect that the geometry is getting more complex as the simulation progresses. Does the simulation start with both phases? Also this problem is singular. You might try projecting out the constant. It could be that as the geometry gets more complex floating point errors are creeping in and you are getting an effective constant component to your RHS. > The system matrix is built so to remain symmetric positive defined despite the coefficients. > > Michele > > > On 09/23/2013 09:11 AM, Matthew Knepley wrote: >> On Mon, Sep 23, 2013 at 8:55 AM, Michele Rosso wrote: >> Hi, >> >> I am successfully using PETSc to solve a 3D Poisson's equation with CG + MG . Such equation arises from a projection algorithm for a multiphase incompressible flow simulation. >> I set up the solver as I was suggested to do in a previous thread (title: "GAMG speed") and run a test case (liquid droplet with surface tension falling under the effect of gravity in a quiescent fluid). >> The solution of the Poisson Equation via multigrid is correct but it becomes progressively slower and slower as the simulation progresses (I am performing successive solves) due to an increase in the number of iterations. >> Since the solution of the Poisson equation is mission-critical, I need to speed it up as much as I can. >> Could you please help me out with this? >> >> First, what does the coefficient look like? >> >> Second, what are the boundary conditions? >> >> Matt >> >> I run the test case with the following options: >> >> -pc_type mg -pc_mg_galerkin -pc_mg_levels 5 -mg_levels_ksp_type richardson -mg_levels_ksp_max_it 1 >> -mg_coarse_pc_type lu -mg_coarse_pc_factor_mat_solver_package superlu_dist >> -log_summary -ksp_view -ksp_monitor_true_residual -options_left >> >> Please find the diagnostic for the final solve in the attached file "final.txt'. >> Thank you, >> >> Michele >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrosso at uci.edu Mon Sep 23 12:02:45 2013 From: mrosso at uci.edu (Michele Rosso) Date: Mon, 23 Sep 2013 10:02:45 -0700 Subject: [petsc-users] How to speed up geometric multigrid In-Reply-To: References: <52406460.3060905@uci.edu> <52406C01.5090804@uci.edu> Message-ID: <52407435.6080302@uci.edu> On 09/23/2013 09:48 AM, Mark F. Adams wrote: > > On Sep 23, 2013, at 12:27 PM, Michele Rosso > wrote: > >> The boundary conditions are periodic. >> The equation I am solving is: >> >> div(beta*grad(u))= f >> >> where beta is 1 inside the gas phase, 0.001 inside the liquid phase >> and a value in between for the nodes close to the interface. > > This is a pretty big jump for geometric MG. You might try AMG. I > suspect that the geometry is getting more complex as the simulation > progresses. Does the simulation start with both phases? Also this > problem is singular. You might try projecting out the constant. It > could be that as the geometry gets more complex floating point errors > are creeping in and you are getting an effective constant component to > your RHS. > The simulation does start with both phases and the geometry is supposed to become more complex as the simulation progresses. But so far the run is stopped before there are significant changes in the shape of the droplet. I can give a shot to AMG: which options would you suggest to use. Also, how can I project out the constant from the rhs? Thanks a lot! Michele >> The system matrix is built so to remain symmetric positive defined >> despite the coefficients. >> >> Michele >> >> >> On 09/23/2013 09:11 AM, Matthew Knepley wrote: >>> On Mon, Sep 23, 2013 at 8:55 AM, Michele Rosso >> > wrote: >>> >>> Hi, >>> >>> I am successfully using PETSc to solve a 3D Poisson's equation >>> with CG + MG . Such equation arises from a projection algorithm >>> for a multiphase incompressible flow simulation. >>> I set up the solver as I was suggested to do in a previous >>> thread (title: "GAMG speed") and run a test case (liquid droplet >>> with surface tension falling under the effect of gravity in a >>> quiescent fluid). >>> The solution of the Poisson Equation via multigrid is correct >>> but it becomes progressively slower and slower as the simulation >>> progresses (I am performing successive solves) due to an >>> increase in the number of iterations. >>> Since the solution of the Poisson equation is mission-critical, >>> I need to speed it up as much as I can. >>> Could you please help me out with this? >>> >>> >>> First, what does the coefficient look like? >>> >>> Second, what are the boundary conditions? >>> >>> Matt >>> >>> I run the test case with the following options: >>> >>> -pc_type mg -pc_mg_galerkin -pc_mg_levels 5 >>> -mg_levels_ksp_type richardson -mg_levels_ksp_max_it 1 >>> -mg_coarse_pc_type lu -mg_coarse_pc_factor_mat_solver_package >>> superlu_dist >>> -log_summary -ksp_view -ksp_monitor_true_residual -options_left >>> >>> Please find the diagnostic for the final solve in the attached >>> file "final.txt'. >>> Thank you, >>> >>> Michele >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which >>> their experiments lead. >>> -- Norbert Wiener >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mpovolot at purdue.edu Mon Sep 23 12:30:57 2013 From: mpovolot at purdue.edu (Michael Povolotskyi) Date: Mon, 23 Sep 2013 13:30:57 -0400 Subject: [petsc-users] Configuring petsc with Intel MPI Message-ID: <52407AD1.3060800@purdue.edu> Dear Petsc developers and users, we are building petsc with intel compiler and intel mpi. In doing so we provide to the petsc configuration script a directory where the MPI is located. The problem is that petsc takes the non-thread safe mpi library (libmpi.so), but we need a threadsafe one (libmpi_mt.so) We had to specify the exact library to solve the issue. Question: is there any better way to insure that petsc picks the thread safe mpi library during configuration? Thank you, Michael. -- Michael Povolotskyi, PhD Research Assistant Professor Network for Computational Nanotechnology 207 S Martin Jischke Drive Purdue University, DLR, room 441-10 West Lafayette, Indiana 47907 phone: +1-765-494-9396 fax: +1-765-496-6026 From balay at mcs.anl.gov Mon Sep 23 12:38:01 2013 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 23 Sep 2013 12:38:01 -0500 (CDT) Subject: [petsc-users] Configuring petsc with Intel MPI In-Reply-To: <52407AD1.3060800@purdue.edu> References: <52407AD1.3060800@purdue.edu> Message-ID: On Mon, 23 Sep 2013, Michael Povolotskyi wrote: > Dear Petsc developers and users, > we are building petsc with intel compiler and intel mpi. > In doing so we provide to the petsc configuration script a directory where the > MPI is located. > > The problem is that petsc takes the non-thread safe mpi library (libmpi.so), > but we need a threadsafe one (libmpi_mt.so) > We had to specify the exact library to solve the issue. > Question: is there any better way to insure that petsc picks the thread safe > mpi library during configuration? What is the recommended method [by Intel MPI] to specify libmpi_mt.so? Use that method to specify things to petsc configure. [i.e you might have to specify the correct compilers, and the correct compiler options using CC, CFLAGS CPPPFLAGS etc options to configure] If you are having trouble tell us the recommended way - and what you get when you try it [i.e configure.log] Satish From danyang.su at gmail.com Mon Sep 23 13:33:45 2013 From: danyang.su at gmail.com (Danyang Su) Date: Mon, 23 Sep 2013 11:33:45 -0700 Subject: [petsc-users] Strange efficiency in PETSc-dev using OpenMP In-Reply-To: References: <523D482D.5030409@gmail.com> <101244E8-FFFA-4F3F-A4BC-CC47F68D93B7@mcs.anl.gov> <523E059D.6080901@gmail.com> <933EF7F4-598A-4F88-8902-0613E27F0679@mcs.anl.gov> <523E6CAF.9020805@gmail.com> Message-ID: <52408989.7090001@gmail.com> Hi Shri, It seems that the problem does not result from the affinities setting for threads. I have tried several settings, the threads are set to different cores, but there is no improvement. Here is the information of package, core and thread maps OMP: Info #204: KMP_AFFINITY: decoding x2APIC ids. OMP: Info #202: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: {0,1,2,3,4,5,6,7,8,9,10,11} OMP: Info #156: KMP_AFFINITY: 12 available OS procs OMP: Info #157: KMP_AFFINITY: Uniform topology OMP: Info #179: KMP_AFFINITY: 1 packages x 6 cores/pkg x 2 threads/core (6 total cores) OMP: Info #206: KMP_AFFINITY: OS proc to physical thread map: OMP: Info #171: KMP_AFFINITY: OS proc 0 maps to package 0 core 0 thread 0 OMP: Info #171: KMP_AFFINITY: OS proc 1 maps to package 0 core 0 thread 1 OMP: Info #171: KMP_AFFINITY: OS proc 2 maps to package 0 core 1 thread 0 OMP: Info #171: KMP_AFFINITY: OS proc 3 maps to package 0 core 1 thread 1 OMP: Info #171: KMP_AFFINITY: OS proc 4 maps to package 0 core 2 thread 0 OMP: Info #171: KMP_AFFINITY: OS proc 5 maps to package 0 core 2 thread 1 OMP: Info #171: KMP_AFFINITY: OS proc 6 maps to package 0 core 3 thread 0 OMP: Info #171: KMP_AFFINITY: OS proc 7 maps to package 0 core 3 thread 1 OMP: Info #171: KMP_AFFINITY: OS proc 8 maps to package 0 core 4 thread 0 OMP: Info #171: KMP_AFFINITY: OS proc 9 maps to package 0 core 4 thread 1 OMP: Info #171: KMP_AFFINITY: OS proc 10 maps to package 0 core 5 thread 0 OMP: Info #171: KMP_AFFINITY: OS proc 11 maps to package 0 core 5 thread 1 OMP: Info #144: KMP_AFFINITY: Threads may migrate across 1 innermost levels of machine And here is the internal thread bounding with different kmp_affinity settings: 1. KMP_AFFINITY=verbose,granularity=thread,compact OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0} OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {1} OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2} OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {3} 2. KMP_AFFINITY=verbose,granularity=fine,compact OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0} OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {1} OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2} OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {3} 3. KMP_AFFINITY=verbose,granularity=fine,compact,1,0 OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0} OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {2} OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {4} OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {6} 4. KMP_AFFINITY=verbose,scatter OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,1} OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {2,3} OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {4,5} OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {6,7} 5. KMP_AFFINITY=verbose,compact (For this setting, two threads are assigned to the same core) OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,1} OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {0,1} OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2,3} OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {2,3} 6. KMP_AFFINITY=verbose,granularity=core,compact (For this setting, two threads are assigned to the same core) OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,1} OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {0,1} OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2,3} OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {2,3} The first 4 settings can assign threads to a distinct core, but the problem is not solved. Thanks, Danyang On 22/09/2013 8:00 PM, Shri wrote: > I think this is definitely an issue with setting the affinities for > threads, i.e., the assignment of threads to cores. Ideally each thread > should be assigned to a distinct core but in your case all the 4 > threads are getting pinned to the same core resulting in such a > massive slowdown. Unfortunately, the thread affinities for OpenMP are > set through environment variables. For Intel's OpenMP one needs to > define the thread affinities through the environment variable > KMP_AFFINITY. See this document here > http://software.intel.com/sites/products/documentation/studio/composer/en-us/2011Update/compiler_c/optaps/common/optaps_openmp_thread_affinity.htm. > Try setting the affinities via KMP_AFFINITY and let us know if it works. > > Shri > On Sep 21, 2013, at 11:06 PM, Danyang Su wrote: > >> Hi Shri, >> >> Thanks for your info. It can work with the option -threadcomm_type >> openmp. But another problem arises, as described as follows. >> >> The sparse matrix is 53760*53760 with 1067392 non-zero entries. If >> the codes is compiled using PETSc-3.4.2, it works fine, the equations >> can be solved quickly and I can see the speedup. But if the code is >> compiled using PETSc-dev with OpenMP option, it takes a long time in >> solving the equations and I cannot see any speedup when more >> processors are used. >> >> For PETSc-3.4.2, run by "mpiexec -n 4 ksp_inhm_d -log_summary >> log_mpi4_petsc3.4.2.log", the iteration and runtime are: >> Iterations 6 *time_assembly 0.4137E-01* time_ksp 0.9296E-01 >> >> For PETSc-dev, run by "mpiexec -n 1 ksp_inhm_d -threadcomm_type >> openmp -threadcomm_nthreads 4 -log_summary log_openmp_petsc_dev.log", >> the iteration and runtime are: >> Iterations 6 *time_assembly 0.3595E+03* time_ksp 0.2907E+00 >> >> Most of the time *'time_assembly 0.3595E+03*' is spent on the >> following codes >> do i = istart, iend - 1 >> ii = ia_in(i+1) >> jj = ia_in(i+2) >> call MatSetValues(a, ione, i, jj-ii, >> ja_in(ii:jj-1)-1, a_in(ii:jj-1), Insert_Values, ierr) >> end do >> >> The log files for both PETSc-3.4.2 and PETSc-dev are attached. >> >> Is there anything wrong with my codes or with running option? The >> above codes works fine when using MPICH. >> >> Thanks and regards, >> >> Danyang >> >> On 21/09/2013 2:09 PM, Shri wrote: >>> There are three thread communicator types in PETSc. The default is >>> "no thread" which is basically a non-threaded version. The other two >>> types are "openmp" and "pthread". If you want to use OpenMP then use >>> the option -threadcomm_type openmp. >>> >>> Shri >>> >>> On Sep 21, 2013, at 3:46 PM, Danyang Su >> > wrote: >>> >>>> Hi Barry, >>>> >>>> Thanks for the quick reply. >>>> >>>> After changing >>>> /#if defined(PETSC_HAVE_PTHREADCLASSES) || defined >>>> (PETSC_HAVE_OPENMP) / >>>> to >>>> /#if defined(PETSC_HAVE_PTHREADCLASSES)/ >>>> and comment out >>>> /#elif defined(PETSC_HAVE_OPENMP)// >>>> //PETSC_EXTERN PetscStack *petscstack;/ >>>> >>>> It can be compiled and validated with "make test". >>>> >>>> But I still have questions on running the examples. After rebuild >>>> the codes (e.g., ksp_ex2f.f), I can run it with "mpiexec -n 1 >>>> ksp_ex2f", or "mpiexec -n 4 ksp_ex2f", or "mpiexec -n 1 ksp_ex2f >>>> -threadcomm_nthreads 1", but if I run it with "mpiexec -n 1 >>>> ksp_ex2f -threadcomm_nthreads 4", there will be a lot of error >>>> information (attached). >>>> >>>> The codes is not modified and there is no OpenMP routines in it. >>>> For the current development in my project, I want to keep the >>>> OpenMP codes in calculating matrix values, but want to solve it >>>> with PETSc (OpenMP). Is it possible? >>>> >>>> Thanks and regards, >>>> >>>> Danyang >>>> >>>> >>>> >>>> On 21/09/2013 7:26 AM, Barry Smith wrote: >>>>> Danyang, >>>>> >>>>> I don't think the || defined (PETSC_HAVE_OPENMP) belongs in the code below. >>>>> >>>>> /* Linux functions CPU_SET and others don't work if sched.h is not included before >>>>> including pthread.h. Also, these functions are active only if either _GNU_SOURCE >>>>> or __USE_GNU is not set (see /usr/include/sched.h and /usr/include/features.h), hence >>>>> set these first. >>>>> */ >>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>> >>>>> Edit include/petscerror.h and locate these lines and remove that part and then rerun make all. Let us know if it works or not. >>>>> >>>>> Barry >>>>> >>>>> i.e. replace >>>>> >>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>> >>>>> with >>>>> >>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) >>>>> >>>>> On Sep 21, 2013, at 6:53 AM, Matthew Knepley wrote: >>>>> >>>>>> On Sat, Sep 21, 2013 at 12:18 AM, Danyang Su wrote: >>>>>> Hi All, >>>>>> >>>>>> I got error information in compiling petsc-dev with openmp in cygwin. Before, I have successfully compiled petsc-3.4.2 and it works fine. >>>>>> The log files have been attached. >>>>>> >>>>>> The OpenMP configure test is wrong. It clearly fails to find pthread.h, but the test passes. Then in petscerror.h >>>>>> we guard pthread.h using PETSC_HAVE_OPENMP. Can someone who knows OpenMP fix this? >>>>>> >>>>>> Matt >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Danyang >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>> -- Norbert Wiener >>>> >>>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Sep 23 13:43:17 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 23 Sep 2013 13:43:17 -0500 Subject: [petsc-users] Strange efficiency in PETSc-dev using OpenMP In-Reply-To: <52408989.7090001@gmail.com> References: <523D482D.5030409@gmail.com> <101244E8-FFFA-4F3F-A4BC-CC47F68D93B7@mcs.anl.gov> <523E059D.6080901@gmail.com> <933EF7F4-598A-4F88-8902-0613E27F0679@mcs.anl.gov> <523E6CAF.9020805@gmail.com> <52408989.7090001@gmail.com> Message-ID: You did not answer my question from yesterday: If you run the Openmp compiled version WITHOUT the -threadcomm_nthreads 1 -threadcomm_type openmp command line options is it still slow? On Sep 23, 2013, at 1:33 PM, Danyang Su wrote: > Hi Shri, > > It seems that the problem does not result from the affinities setting for threads. I have tried several settings, the threads are set to different cores, but there is no improvement. > > Here is the information of package, core and thread maps > > OMP: Info #204: KMP_AFFINITY: decoding x2APIC ids. > OMP: Info #202: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info > OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: {0,1,2,3,4,5,6,7,8,9,10,11} > OMP: Info #156: KMP_AFFINITY: 12 available OS procs > OMP: Info #157: KMP_AFFINITY: Uniform topology > OMP: Info #179: KMP_AFFINITY: 1 packages x 6 cores/pkg x 2 threads/core (6 total cores) > OMP: Info #206: KMP_AFFINITY: OS proc to physical thread map: > OMP: Info #171: KMP_AFFINITY: OS proc 0 maps to package 0 core 0 thread 0 > OMP: Info #171: KMP_AFFINITY: OS proc 1 maps to package 0 core 0 thread 1 > OMP: Info #171: KMP_AFFINITY: OS proc 2 maps to package 0 core 1 thread 0 > OMP: Info #171: KMP_AFFINITY: OS proc 3 maps to package 0 core 1 thread 1 > OMP: Info #171: KMP_AFFINITY: OS proc 4 maps to package 0 core 2 thread 0 > OMP: Info #171: KMP_AFFINITY: OS proc 5 maps to package 0 core 2 thread 1 > OMP: Info #171: KMP_AFFINITY: OS proc 6 maps to package 0 core 3 thread 0 > OMP: Info #171: KMP_AFFINITY: OS proc 7 maps to package 0 core 3 thread 1 > OMP: Info #171: KMP_AFFINITY: OS proc 8 maps to package 0 core 4 thread 0 > OMP: Info #171: KMP_AFFINITY: OS proc 9 maps to package 0 core 4 thread 1 > OMP: Info #171: KMP_AFFINITY: OS proc 10 maps to package 0 core 5 thread 0 > OMP: Info #171: KMP_AFFINITY: OS proc 11 maps to package 0 core 5 thread 1 > OMP: Info #144: KMP_AFFINITY: Threads may migrate across 1 innermost levels of machine > > > And here is the internal thread bounding with different kmp_affinity settings: > > 1. KMP_AFFINITY=verbose,granularity=thread,compact > > OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0} > OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {1} > OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2} > OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {3} > > 2. KMP_AFFINITY=verbose,granularity=fine,compact > > OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0} > OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {1} > OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2} > OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {3} > > 3. KMP_AFFINITY=verbose,granularity=fine,compact,1,0 > > OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0} > OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {2} > OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {4} > OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {6} > > 4. KMP_AFFINITY=verbose,scatter > > OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,1} > OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {2,3} > OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {4,5} > OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {6,7} > > 5. KMP_AFFINITY=verbose,compact (For this setting, two threads are assigned to the same core) > > OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,1} > OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {0,1} > OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2,3} > OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {2,3} > > 6. KMP_AFFINITY=verbose,granularity=core,compact (For this setting, two threads are assigned to the same core) > > OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,1} > OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {0,1} > OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2,3} > OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {2,3} > > The first 4 settings can assign threads to a distinct core, but the problem is not solved. > > Thanks, > > Danyang > > > > On 22/09/2013 8:00 PM, Shri wrote: >> I think this is definitely an issue with setting the affinities for threads, i.e., the assignment of threads to cores. Ideally each thread should be assigned to a distinct core but in your case all the 4 threads are getting pinned to the same core resulting in such a massive slowdown. Unfortunately, the thread affinities for OpenMP are set through environment variables. For Intel's OpenMP one needs to define the thread affinities through the environment variable KMP_AFFINITY. See this document here http://software.intel.com/sites/products/documentation/studio/composer/en-us/2011Update/compiler_c/optaps/common/optaps_openmp_thread_affinity.htm. Try setting the affinities via KMP_AFFINITY and let us know if it works. >> >> Shri >> On Sep 21, 2013, at 11:06 PM, Danyang Su wrote: >> >>> Hi Shri, >>> >>> Thanks for your info. It can work with the option -threadcomm_type openmp. But another problem arises, as described as follows. >>> >>> The sparse matrix is 53760*53760 with 1067392 non-zero entries. If the codes is compiled using PETSc-3.4.2, it works fine, the equations can be solved quickly and I can see the speedup. But if the code is compiled using PETSc-dev with OpenMP option, it takes a long time in solving the equations and I cannot see any speedup when more processors are used. >>> >>> For PETSc-3.4.2, run by "mpiexec -n 4 ksp_inhm_d -log_summary log_mpi4_petsc3.4.2.log", the iteration and runtime are: >>> Iterations 6 time_assembly 0.4137E-01 time_ksp 0.9296E-01 >>> >>> For PETSc-dev, run by "mpiexec -n 1 ksp_inhm_d -threadcomm_type openmp -threadcomm_nthreads 4 -log_summary log_openmp_petsc_dev.log", the iteration and runtime are: >>> Iterations 6 time_assembly 0.3595E+03 time_ksp 0.2907E+00 >>> >>> Most of the time 'time_assembly 0.3595E+03' is spent on the following codes >>> do i = istart, iend - 1 >>> ii = ia_in(i+1) >>> jj = ia_in(i+2) >>> call MatSetValues(a, ione, i, jj-ii, ja_in(ii:jj-1)-1, a_in(ii:jj-1), Insert_Values, ierr) >>> end do >>> >>> The log files for both PETSc-3.4.2 and PETSc-dev are attached. >>> >>> Is there anything wrong with my codes or with running option? The above codes works fine when using MPICH. >>> >>> Thanks and regards, >>> >>> Danyang >>> >>> On 21/09/2013 2:09 PM, Shri wrote: >>>> There are three thread communicator types in PETSc. The default is "no thread" which is basically a non-threaded version. The other two types are "openmp" and "pthread". If you want to use OpenMP then use the option -threadcomm_type openmp. >>>> >>>> Shri >>>> >>>> On Sep 21, 2013, at 3:46 PM, Danyang Su wrote: >>>> >>>>> Hi Barry, >>>>> >>>>> Thanks for the quick reply. >>>>> >>>>> After changing >>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>> to >>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) >>>>> and comment out >>>>> #elif defined(PETSC_HAVE_OPENMP) >>>>> PETSC_EXTERN PetscStack *petscstack; >>>>> >>>>> It can be compiled and validated with "make test". >>>>> >>>>> But I still have questions on running the examples. After rebuild the codes (e.g., ksp_ex2f.f), I can run it with "mpiexec -n 1 ksp_ex2f", or "mpiexec -n 4 ksp_ex2f", or "mpiexec -n 1 ksp_ex2f -threadcomm_nthreads 1", but if I run it with "mpiexec -n 1 ksp_ex2f -threadcomm_nthreads 4", there will be a lot of error information (attached). >>>>> >>>>> The codes is not modified and there is no OpenMP routines in it. For the current development in my project, I want to keep the OpenMP codes in calculating matrix values, but want to solve it with PETSc (OpenMP). Is it possible? >>>>> >>>>> Thanks and regards, >>>>> >>>>> Danyang >>>>> >>>>> >>>>> >>>>> On 21/09/2013 7:26 AM, Barry Smith wrote: >>>>>> Danyang, >>>>>> >>>>>> I don't think the || defined (PETSC_HAVE_OPENMP) belongs in the code below. >>>>>> >>>>>> /* Linux functions CPU_SET and others don't work if sched.h is not included before >>>>>> including pthread.h. Also, these functions are active only if either _GNU_SOURCE >>>>>> or __USE_GNU is not set (see /usr/include/sched.h and /usr/include/features.h), hence >>>>>> set these first. >>>>>> */ >>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>>> >>>>>> Edit include/petscerror.h and locate these lines and remove that part and then rerun make all. Let us know if it works or not. >>>>>> >>>>>> Barry >>>>>> >>>>>> i.e. replace >>>>>> >>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>>> >>>>>> with >>>>>> >>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) >>>>>> >>>>>> On Sep 21, 2013, at 6:53 AM, Matthew Knepley >>>>>> >>>>>> wrote: >>>>>> >>>>>> >>>>>>> On Sat, Sep 21, 2013 at 12:18 AM, Danyang Su >>>>>>> wrote: >>>>>>> Hi All, >>>>>>> >>>>>>> I got error information in compiling petsc-dev with openmp in cygwin. Before, I have successfully compiled petsc-3.4.2 and it works fine. >>>>>>> The log files have been attached. >>>>>>> >>>>>>> The OpenMP configure test is wrong. It clearly fails to find pthread.h, but the test passes. Then in petscerror.h >>>>>>> we guard pthread.h using PETSC_HAVE_OPENMP. Can someone who knows OpenMP fix this? >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Danyang >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>> >>>>> >>> >>> >> > From mfadams at lbl.gov Mon Sep 23 13:52:25 2013 From: mfadams at lbl.gov (Mark F. Adams) Date: Mon, 23 Sep 2013 14:52:25 -0400 Subject: [petsc-users] How to speed up geometric multigrid In-Reply-To: <52407435.6080302@uci.edu> References: <52406460.3060905@uci.edu> <52406C01.5090804@uci.edu> <52407435.6080302@uci.edu> Message-ID: <4F04E2EB-90FC-4EAD-B33A-9510AC8593F7@lbl.gov> >> > The simulation does start with both phases and the geometry is supposed to become more complex as the simulation progresses. > But so far the run is stopped before there are significant changes in the shape of the droplet. Humm, not sure why you are seeing degradation then. I imagine the initial geometry is not grid aligned so I'm not sure why you are seeing degradation in convergence rate. > I can give a shot to AMG: which options would you suggest to use. -pc_type gamg -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 > Also, how can I project out the constant from the rhs? Thanks a lot! http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPSetNullSpace.html#KSPSetNullSpace From danyang.su at gmail.com Mon Sep 23 14:01:29 2013 From: danyang.su at gmail.com (Danyang Su) Date: Mon, 23 Sep 2013 12:01:29 -0700 Subject: [petsc-users] Strange efficiency in PETSc-dev using OpenMP In-Reply-To: References: <523D482D.5030409@gmail.com> <101244E8-FFFA-4F3F-A4BC-CC47F68D93B7@mcs.anl.gov> <523E059D.6080901@gmail.com> <933EF7F4-598A-4F88-8902-0613E27F0679@mcs.anl.gov> <523E6CAF.9020805@gmail.com> <52408989.7090001@gmail.com> Message-ID: <52409009.2000506@gmail.com> Hi Barry, Sorry I forgot the message in the previous email. It is still slow when run without the "-threadcomm_type openmp -threadcomm_nthreads 1" Thanks, Danyang On 23/09/2013 11:43 AM, Barry Smith wrote: > You did not answer my question from yesterday: > > If you run the Openmp compiled version WITHOUT the > > -threadcomm_nthreads 1 > -threadcomm_type openmp > > command line options is it still slow? > > > On Sep 23, 2013, at 1:33 PM, Danyang Su wrote: > >> Hi Shri, >> >> It seems that the problem does not result from the affinities setting for threads. I have tried several settings, the threads are set to different cores, but there is no improvement. >> >> Here is the information of package, core and thread maps >> >> OMP: Info #204: KMP_AFFINITY: decoding x2APIC ids. >> OMP: Info #202: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info >> OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: {0,1,2,3,4,5,6,7,8,9,10,11} >> OMP: Info #156: KMP_AFFINITY: 12 available OS procs >> OMP: Info #157: KMP_AFFINITY: Uniform topology >> OMP: Info #179: KMP_AFFINITY: 1 packages x 6 cores/pkg x 2 threads/core (6 total cores) >> OMP: Info #206: KMP_AFFINITY: OS proc to physical thread map: >> OMP: Info #171: KMP_AFFINITY: OS proc 0 maps to package 0 core 0 thread 0 >> OMP: Info #171: KMP_AFFINITY: OS proc 1 maps to package 0 core 0 thread 1 >> OMP: Info #171: KMP_AFFINITY: OS proc 2 maps to package 0 core 1 thread 0 >> OMP: Info #171: KMP_AFFINITY: OS proc 3 maps to package 0 core 1 thread 1 >> OMP: Info #171: KMP_AFFINITY: OS proc 4 maps to package 0 core 2 thread 0 >> OMP: Info #171: KMP_AFFINITY: OS proc 5 maps to package 0 core 2 thread 1 >> OMP: Info #171: KMP_AFFINITY: OS proc 6 maps to package 0 core 3 thread 0 >> OMP: Info #171: KMP_AFFINITY: OS proc 7 maps to package 0 core 3 thread 1 >> OMP: Info #171: KMP_AFFINITY: OS proc 8 maps to package 0 core 4 thread 0 >> OMP: Info #171: KMP_AFFINITY: OS proc 9 maps to package 0 core 4 thread 1 >> OMP: Info #171: KMP_AFFINITY: OS proc 10 maps to package 0 core 5 thread 0 >> OMP: Info #171: KMP_AFFINITY: OS proc 11 maps to package 0 core 5 thread 1 >> OMP: Info #144: KMP_AFFINITY: Threads may migrate across 1 innermost levels of machine >> >> >> And here is the internal thread bounding with different kmp_affinity settings: >> >> 1. KMP_AFFINITY=verbose,granularity=thread,compact >> >> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0} >> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {1} >> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2} >> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {3} >> >> 2. KMP_AFFINITY=verbose,granularity=fine,compact >> >> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0} >> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {1} >> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2} >> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {3} >> >> 3. KMP_AFFINITY=verbose,granularity=fine,compact,1,0 >> >> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0} >> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {2} >> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {4} >> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {6} >> >> 4. KMP_AFFINITY=verbose,scatter >> >> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,1} >> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {2,3} >> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {4,5} >> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {6,7} >> >> 5. KMP_AFFINITY=verbose,compact (For this setting, two threads are assigned to the same core) >> >> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,1} >> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {0,1} >> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2,3} >> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {2,3} >> >> 6. KMP_AFFINITY=verbose,granularity=core,compact (For this setting, two threads are assigned to the same core) >> >> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,1} >> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {0,1} >> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2,3} >> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {2,3} >> >> The first 4 settings can assign threads to a distinct core, but the problem is not solved. >> >> Thanks, >> >> Danyang >> >> >> >> On 22/09/2013 8:00 PM, Shri wrote: >>> I think this is definitely an issue with setting the affinities for threads, i.e., the assignment of threads to cores. Ideally each thread should be assigned to a distinct core but in your case all the 4 threads are getting pinned to the same core resulting in such a massive slowdown. Unfortunately, the thread affinities for OpenMP are set through environment variables. For Intel's OpenMP one needs to define the thread affinities through the environment variable KMP_AFFINITY. See this document here http://software.intel.com/sites/products/documentation/studio/composer/en-us/2011Update/compiler_c/optaps/common/optaps_openmp_thread_affinity.htm. Try setting the affinities via KMP_AFFINITY and let us know if it works. >>> >>> Shri >>> On Sep 21, 2013, at 11:06 PM, Danyang Su wrote: >>> >>>> Hi Shri, >>>> >>>> Thanks for your info. It can work with the option -threadcomm_type openmp. But another problem arises, as described as follows. >>>> >>>> The sparse matrix is 53760*53760 with 1067392 non-zero entries. If the codes is compiled using PETSc-3.4.2, it works fine, the equations can be solved quickly and I can see the speedup. But if the code is compiled using PETSc-dev with OpenMP option, it takes a long time in solving the equations and I cannot see any speedup when more processors are used. >>>> >>>> For PETSc-3.4.2, run by "mpiexec -n 4 ksp_inhm_d -log_summary log_mpi4_petsc3.4.2.log", the iteration and runtime are: >>>> Iterations 6 time_assembly 0.4137E-01 time_ksp 0.9296E-01 >>>> >>>> For PETSc-dev, run by "mpiexec -n 1 ksp_inhm_d -threadcomm_type openmp -threadcomm_nthreads 4 -log_summary log_openmp_petsc_dev.log", the iteration and runtime are: >>>> Iterations 6 time_assembly 0.3595E+03 time_ksp 0.2907E+00 >>>> >>>> Most of the time 'time_assembly 0.3595E+03' is spent on the following codes >>>> do i = istart, iend - 1 >>>> ii = ia_in(i+1) >>>> jj = ia_in(i+2) >>>> call MatSetValues(a, ione, i, jj-ii, ja_in(ii:jj-1)-1, a_in(ii:jj-1), Insert_Values, ierr) >>>> end do >>>> >>>> The log files for both PETSc-3.4.2 and PETSc-dev are attached. >>>> >>>> Is there anything wrong with my codes or with running option? The above codes works fine when using MPICH. >>>> >>>> Thanks and regards, >>>> >>>> Danyang >>>> >>>> On 21/09/2013 2:09 PM, Shri wrote: >>>>> There are three thread communicator types in PETSc. The default is "no thread" which is basically a non-threaded version. The other two types are "openmp" and "pthread". If you want to use OpenMP then use the option -threadcomm_type openmp. >>>>> >>>>> Shri >>>>> >>>>> On Sep 21, 2013, at 3:46 PM, Danyang Su wrote: >>>>> >>>>>> Hi Barry, >>>>>> >>>>>> Thanks for the quick reply. >>>>>> >>>>>> After changing >>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>>> to >>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) >>>>>> and comment out >>>>>> #elif defined(PETSC_HAVE_OPENMP) >>>>>> PETSC_EXTERN PetscStack *petscstack; >>>>>> >>>>>> It can be compiled and validated with "make test". >>>>>> >>>>>> But I still have questions on running the examples. After rebuild the codes (e.g., ksp_ex2f.f), I can run it with "mpiexec -n 1 ksp_ex2f", or "mpiexec -n 4 ksp_ex2f", or "mpiexec -n 1 ksp_ex2f -threadcomm_nthreads 1", but if I run it with "mpiexec -n 1 ksp_ex2f -threadcomm_nthreads 4", there will be a lot of error information (attached). >>>>>> >>>>>> The codes is not modified and there is no OpenMP routines in it. For the current development in my project, I want to keep the OpenMP codes in calculating matrix values, but want to solve it with PETSc (OpenMP). Is it possible? >>>>>> >>>>>> Thanks and regards, >>>>>> >>>>>> Danyang >>>>>> >>>>>> >>>>>> >>>>>> On 21/09/2013 7:26 AM, Barry Smith wrote: >>>>>>> Danyang, >>>>>>> >>>>>>> I don't think the || defined (PETSC_HAVE_OPENMP) belongs in the code below. >>>>>>> >>>>>>> /* Linux functions CPU_SET and others don't work if sched.h is not included before >>>>>>> including pthread.h. Also, these functions are active only if either _GNU_SOURCE >>>>>>> or __USE_GNU is not set (see /usr/include/sched.h and /usr/include/features.h), hence >>>>>>> set these first. >>>>>>> */ >>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>>>> >>>>>>> Edit include/petscerror.h and locate these lines and remove that part and then rerun make all. Let us know if it works or not. >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>> i.e. replace >>>>>>> >>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>>>> >>>>>>> with >>>>>>> >>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) >>>>>>> >>>>>>> On Sep 21, 2013, at 6:53 AM, Matthew Knepley >>>>>>> >>>>>>> wrote: >>>>>>> >>>>>>> >>>>>>>> On Sat, Sep 21, 2013 at 12:18 AM, Danyang Su >>>>>>>> wrote: >>>>>>>> Hi All, >>>>>>>> >>>>>>>> I got error information in compiling petsc-dev with openmp in cygwin. Before, I have successfully compiled petsc-3.4.2 and it works fine. >>>>>>>> The log files have been attached. >>>>>>>> >>>>>>>> The OpenMP configure test is wrong. It clearly fails to find pthread.h, but the test passes. Then in petscerror.h >>>>>>>> we guard pthread.h using PETSC_HAVE_OPENMP. Can someone who knows OpenMP fix this? >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Danyang >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>> >>>> From danyang.su at gmail.com Mon Sep 23 14:13:48 2013 From: danyang.su at gmail.com (Danyang Su) Date: Mon, 23 Sep 2013 12:13:48 -0700 Subject: [petsc-users] Strange efficiency in PETSc-dev using OpenMP In-Reply-To: <52409009.2000506@gmail.com> References: <523D482D.5030409@gmail.com> <101244E8-FFFA-4F3F-A4BC-CC47F68D93B7@mcs.anl.gov> <523E059D.6080901@gmail.com> <933EF7F4-598A-4F88-8902-0613E27F0679@mcs.anl.gov> <523E6CAF.9020805@gmail.com> <52408989.7090001@gmail.com> <52409009.2000506@gmail.com> Message-ID: <524092EC.6080605@gmail.com> Hi Barry, Another strange problem: Currently I have PETSc-3.4.2 MPI version and PETSc-dev OpenMP version on my computer, with different environment variable of PETSC_ARCH and PETSC_DIR. Before installation of PETSc-dev OpenMP version, the PETSc-3.4.2 MPI version works fine. But after installation of PETSc-dev OpenMP version, the same problem exist in PETSc-3.4.2 MPI version if run with 1 processor, but no problem with 2 or more processors. Thanks, Danyang On 23/09/2013 12:01 PM, Danyang Su wrote: > Hi Barry, > > Sorry I forgot the message in the previous email. It is still slow > when run without the "-threadcomm_type openmp -threadcomm_nthreads 1" > > Thanks, > > Danyang > > On 23/09/2013 11:43 AM, Barry Smith wrote: >> You did not answer my question from yesterday: >> >> If you run the Openmp compiled version WITHOUT the >> >> -threadcomm_nthreads 1 >> -threadcomm_type openmp >> >> command line options is it still slow? >> >> >> On Sep 23, 2013, at 1:33 PM, Danyang Su wrote: >> >>> Hi Shri, >>> >>> It seems that the problem does not result from the affinities >>> setting for threads. I have tried several settings, the threads are >>> set to different cores, but there is no improvement. >>> >>> Here is the information of package, core and thread maps >>> >>> OMP: Info #204: KMP_AFFINITY: decoding x2APIC ids. >>> OMP: Info #202: KMP_AFFINITY: Affinity capable, using global cpuid >>> leaf 11 info >>> OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: >>> {0,1,2,3,4,5,6,7,8,9,10,11} >>> OMP: Info #156: KMP_AFFINITY: 12 available OS procs >>> OMP: Info #157: KMP_AFFINITY: Uniform topology >>> OMP: Info #179: KMP_AFFINITY: 1 packages x 6 cores/pkg x 2 >>> threads/core (6 total cores) >>> OMP: Info #206: KMP_AFFINITY: OS proc to physical thread map: >>> OMP: Info #171: KMP_AFFINITY: OS proc 0 maps to package 0 core 0 >>> thread 0 >>> OMP: Info #171: KMP_AFFINITY: OS proc 1 maps to package 0 core 0 >>> thread 1 >>> OMP: Info #171: KMP_AFFINITY: OS proc 2 maps to package 0 core 1 >>> thread 0 >>> OMP: Info #171: KMP_AFFINITY: OS proc 3 maps to package 0 core 1 >>> thread 1 >>> OMP: Info #171: KMP_AFFINITY: OS proc 4 maps to package 0 core 2 >>> thread 0 >>> OMP: Info #171: KMP_AFFINITY: OS proc 5 maps to package 0 core 2 >>> thread 1 >>> OMP: Info #171: KMP_AFFINITY: OS proc 6 maps to package 0 core 3 >>> thread 0 >>> OMP: Info #171: KMP_AFFINITY: OS proc 7 maps to package 0 core 3 >>> thread 1 >>> OMP: Info #171: KMP_AFFINITY: OS proc 8 maps to package 0 core 4 >>> thread 0 >>> OMP: Info #171: KMP_AFFINITY: OS proc 9 maps to package 0 core 4 >>> thread 1 >>> OMP: Info #171: KMP_AFFINITY: OS proc 10 maps to package 0 core 5 >>> thread 0 >>> OMP: Info #171: KMP_AFFINITY: OS proc 11 maps to package 0 core 5 >>> thread 1 >>> OMP: Info #144: KMP_AFFINITY: Threads may migrate across 1 innermost >>> levels of machine >>> >>> >>> And here is the internal thread bounding with different kmp_affinity >>> settings: >>> >>> 1. KMP_AFFINITY=verbose,granularity=thread,compact >>> >>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set >>> {0} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set >>> {1} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set >>> {2} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set >>> {3} >>> >>> 2. KMP_AFFINITY=verbose,granularity=fine,compact >>> >>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set >>> {0} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set >>> {1} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set >>> {2} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set >>> {3} >>> >>> 3. KMP_AFFINITY=verbose,granularity=fine,compact,1,0 >>> >>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set >>> {0} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set >>> {2} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set >>> {4} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set >>> {6} >>> >>> 4. KMP_AFFINITY=verbose,scatter >>> >>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set >>> {0,1} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set >>> {2,3} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set >>> {4,5} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set >>> {6,7} >>> >>> 5. KMP_AFFINITY=verbose,compact (For this setting, two threads are >>> assigned to the same core) >>> >>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set >>> {0,1} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set >>> {0,1} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set >>> {2,3} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set >>> {2,3} >>> >>> 6. KMP_AFFINITY=verbose,granularity=core,compact (For this setting, >>> two threads are assigned to the same core) >>> >>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set >>> {0,1} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set >>> {0,1} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set >>> {2,3} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set >>> {2,3} >>> >>> The first 4 settings can assign threads to a distinct core, but the >>> problem is not solved. >>> >>> Thanks, >>> >>> Danyang >>> >>> >>> >>> On 22/09/2013 8:00 PM, Shri wrote: >>>> I think this is definitely an issue with setting the affinities for >>>> threads, i.e., the assignment of threads to cores. Ideally each >>>> thread should be assigned to a distinct core but in your case all >>>> the 4 threads are getting pinned to the same core resulting in such >>>> a massive slowdown. Unfortunately, the thread affinities for OpenMP >>>> are set through environment variables. For Intel's OpenMP one needs >>>> to define the thread affinities through the environment variable >>>> KMP_AFFINITY. See this document here >>>> http://software.intel.com/sites/products/documentation/studio/composer/en-us/2011Update/compiler_c/optaps/common/optaps_openmp_thread_affinity.htm. >>>> Try setting the affinities via KMP_AFFINITY and let us know if it >>>> works. >>>> >>>> Shri >>>> On Sep 21, 2013, at 11:06 PM, Danyang Su wrote: >>>> >>>>> Hi Shri, >>>>> >>>>> Thanks for your info. It can work with the option -threadcomm_type >>>>> openmp. But another problem arises, as described as follows. >>>>> >>>>> The sparse matrix is 53760*53760 with 1067392 non-zero entries. >>>>> If the codes is compiled using PETSc-3.4.2, it works fine, the >>>>> equations can be solved quickly and I can see the speedup. But if >>>>> the code is compiled using PETSc-dev with OpenMP option, it takes >>>>> a long time in solving the equations and I cannot see any speedup >>>>> when more processors are used. >>>>> >>>>> For PETSc-3.4.2, run by "mpiexec -n 4 ksp_inhm_d -log_summary >>>>> log_mpi4_petsc3.4.2.log", the iteration and runtime are: >>>>> Iterations 6 time_assembly 0.4137E-01 time_ksp 0.9296E-01 >>>>> >>>>> For PETSc-dev, run by "mpiexec -n 1 ksp_inhm_d -threadcomm_type >>>>> openmp -threadcomm_nthreads 4 -log_summary >>>>> log_openmp_petsc_dev.log", the iteration and runtime are: >>>>> Iterations 6 time_assembly 0.3595E+03 time_ksp 0.2907E+00 >>>>> >>>>> Most of the time 'time_assembly 0.3595E+03' is spent on the >>>>> following codes >>>>> do i = istart, iend - 1 >>>>> ii = ia_in(i+1) >>>>> jj = ia_in(i+2) >>>>> call MatSetValues(a, ione, i, jj-ii, >>>>> ja_in(ii:jj-1)-1, a_in(ii:jj-1), Insert_Values, ierr) >>>>> end do >>>>> >>>>> The log files for both PETSc-3.4.2 and PETSc-dev are attached. >>>>> >>>>> Is there anything wrong with my codes or with running option? The >>>>> above codes works fine when using MPICH. >>>>> >>>>> Thanks and regards, >>>>> >>>>> Danyang >>>>> >>>>> On 21/09/2013 2:09 PM, Shri wrote: >>>>>> There are three thread communicator types in PETSc. The default >>>>>> is "no thread" which is basically a non-threaded version. The >>>>>> other two types are "openmp" and "pthread". If you want to use >>>>>> OpenMP then use the option -threadcomm_type openmp. >>>>>> >>>>>> Shri >>>>>> >>>>>> On Sep 21, 2013, at 3:46 PM, Danyang Su >>>>>> wrote: >>>>>> >>>>>>> Hi Barry, >>>>>>> >>>>>>> Thanks for the quick reply. >>>>>>> >>>>>>> After changing >>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined >>>>>>> (PETSC_HAVE_OPENMP) >>>>>>> to >>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) >>>>>>> and comment out >>>>>>> #elif defined(PETSC_HAVE_OPENMP) >>>>>>> PETSC_EXTERN PetscStack *petscstack; >>>>>>> >>>>>>> It can be compiled and validated with "make test". >>>>>>> >>>>>>> But I still have questions on running the examples. After >>>>>>> rebuild the codes (e.g., ksp_ex2f.f), I can run it with "mpiexec >>>>>>> -n 1 ksp_ex2f", or "mpiexec -n 4 ksp_ex2f", or "mpiexec -n 1 >>>>>>> ksp_ex2f -threadcomm_nthreads 1", but if I run it with "mpiexec >>>>>>> -n 1 ksp_ex2f -threadcomm_nthreads 4", there will be a lot of >>>>>>> error information (attached). >>>>>>> >>>>>>> The codes is not modified and there is no OpenMP routines in it. >>>>>>> For the current development in my project, I want to keep the >>>>>>> OpenMP codes in calculating matrix values, but want to solve it >>>>>>> with PETSc (OpenMP). Is it possible? >>>>>>> >>>>>>> Thanks and regards, >>>>>>> >>>>>>> Danyang >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 21/09/2013 7:26 AM, Barry Smith wrote: >>>>>>>> Danyang, >>>>>>>> >>>>>>>> I don't think the || defined (PETSC_HAVE_OPENMP) >>>>>>>> belongs in the code below. >>>>>>>> >>>>>>>> /* Linux functions CPU_SET and others don't work if sched.h is >>>>>>>> not included before >>>>>>>> including pthread.h. Also, these functions are active only >>>>>>>> if either _GNU_SOURCE >>>>>>>> or __USE_GNU is not set (see /usr/include/sched.h and >>>>>>>> /usr/include/features.h), hence >>>>>>>> set these first. >>>>>>>> */ >>>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined >>>>>>>> (PETSC_HAVE_OPENMP) >>>>>>>> >>>>>>>> Edit include/petscerror.h and locate these lines and remove >>>>>>>> that part and then rerun make all. Let us know if it works or >>>>>>>> not. >>>>>>>> >>>>>>>> Barry >>>>>>>> >>>>>>>> i.e. replace >>>>>>>> >>>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined >>>>>>>> (PETSC_HAVE_OPENMP) >>>>>>>> >>>>>>>> with >>>>>>>> >>>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) >>>>>>>> >>>>>>>> On Sep 21, 2013, at 6:53 AM, Matthew Knepley >>>>>>>> >>>>>>>> wrote: >>>>>>>> >>>>>>>> >>>>>>>>> On Sat, Sep 21, 2013 at 12:18 AM, Danyang Su >>>>>>>>> >>>>>>>>> wrote: >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> I got error information in compiling petsc-dev with openmp in >>>>>>>>> cygwin. Before, I have successfully compiled petsc-3.4.2 and >>>>>>>>> it works fine. >>>>>>>>> The log files have been attached. >>>>>>>>> >>>>>>>>> The OpenMP configure test is wrong. It clearly fails to find >>>>>>>>> pthread.h, but the test passes. Then in petscerror.h >>>>>>>>> we guard pthread.h using PETSC_HAVE_OPENMP. Can someone who >>>>>>>>> knows OpenMP fix this? >>>>>>>>> >>>>>>>>> Matt >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Danyang >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>> their experiments is infinitely more interesting than any >>>>>>>>> results to which their experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>> >>>>> > From bsmith at mcs.anl.gov Mon Sep 23 14:15:31 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 23 Sep 2013 14:15:31 -0500 Subject: [petsc-users] Strange efficiency in PETSc-dev using OpenMP In-Reply-To: <52409009.2000506@gmail.com> References: <523D482D.5030409@gmail.com> <101244E8-FFFA-4F3F-A4BC-CC47F68D93B7@mcs.anl.gov> <523E059D.6080901@gmail.com> <933EF7F4-598A-4F88-8902-0613E27F0679@mcs.anl.gov> <523E6CAF.9020805@gmail.com> <52408989.7090001@gmail.com> <52409009.2000506@gmail.com> Message-ID: On Sep 23, 2013, at 2:01 PM, Danyang Su wrote: > Hi Barry, > > Sorry I forgot the message in the previous email. It is still slow when run without the "-threadcomm_type openmp -threadcomm_nthreads 1" > So when you compile the software to use OpenMP it is slow; regardless of whether you use OpenMP explicitly or not. When you compile the software to NOT use OpenMP then it is much faster? If this is the case they there is nothing we can do to help you. Something is wrong with the OpenMP compiler option or the libraries it links it. You might consider Linux and the gnu compilers. Barry > Thanks, > > Danyang > > On 23/09/2013 11:43 AM, Barry Smith wrote: >> You did not answer my question from yesterday: >> >> If you run the Openmp compiled version WITHOUT the >> >> -threadcomm_nthreads 1 >> -threadcomm_type openmp >> >> command line options is it still slow? >> >> >> On Sep 23, 2013, at 1:33 PM, Danyang Su wrote: >> >>> Hi Shri, >>> >>> It seems that the problem does not result from the affinities setting for threads. I have tried several settings, the threads are set to different cores, but there is no improvement. >>> >>> Here is the information of package, core and thread maps >>> >>> OMP: Info #204: KMP_AFFINITY: decoding x2APIC ids. >>> OMP: Info #202: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info >>> OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: {0,1,2,3,4,5,6,7,8,9,10,11} >>> OMP: Info #156: KMP_AFFINITY: 12 available OS procs >>> OMP: Info #157: KMP_AFFINITY: Uniform topology >>> OMP: Info #179: KMP_AFFINITY: 1 packages x 6 cores/pkg x 2 threads/core (6 total cores) >>> OMP: Info #206: KMP_AFFINITY: OS proc to physical thread map: >>> OMP: Info #171: KMP_AFFINITY: OS proc 0 maps to package 0 core 0 thread 0 >>> OMP: Info #171: KMP_AFFINITY: OS proc 1 maps to package 0 core 0 thread 1 >>> OMP: Info #171: KMP_AFFINITY: OS proc 2 maps to package 0 core 1 thread 0 >>> OMP: Info #171: KMP_AFFINITY: OS proc 3 maps to package 0 core 1 thread 1 >>> OMP: Info #171: KMP_AFFINITY: OS proc 4 maps to package 0 core 2 thread 0 >>> OMP: Info #171: KMP_AFFINITY: OS proc 5 maps to package 0 core 2 thread 1 >>> OMP: Info #171: KMP_AFFINITY: OS proc 6 maps to package 0 core 3 thread 0 >>> OMP: Info #171: KMP_AFFINITY: OS proc 7 maps to package 0 core 3 thread 1 >>> OMP: Info #171: KMP_AFFINITY: OS proc 8 maps to package 0 core 4 thread 0 >>> OMP: Info #171: KMP_AFFINITY: OS proc 9 maps to package 0 core 4 thread 1 >>> OMP: Info #171: KMP_AFFINITY: OS proc 10 maps to package 0 core 5 thread 0 >>> OMP: Info #171: KMP_AFFINITY: OS proc 11 maps to package 0 core 5 thread 1 >>> OMP: Info #144: KMP_AFFINITY: Threads may migrate across 1 innermost levels of machine >>> >>> >>> And here is the internal thread bounding with different kmp_affinity settings: >>> >>> 1. KMP_AFFINITY=verbose,granularity=thread,compact >>> >>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {1} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {3} >>> >>> 2. KMP_AFFINITY=verbose,granularity=fine,compact >>> >>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {1} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {3} >>> >>> 3. KMP_AFFINITY=verbose,granularity=fine,compact,1,0 >>> >>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {2} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {4} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {6} >>> >>> 4. KMP_AFFINITY=verbose,scatter >>> >>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,1} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {2,3} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {4,5} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {6,7} >>> >>> 5. KMP_AFFINITY=verbose,compact (For this setting, two threads are assigned to the same core) >>> >>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,1} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {0,1} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2,3} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {2,3} >>> >>> 6. KMP_AFFINITY=verbose,granularity=core,compact (For this setting, two threads are assigned to the same core) >>> >>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,1} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {0,1} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2,3} >>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {2,3} >>> >>> The first 4 settings can assign threads to a distinct core, but the problem is not solved. >>> >>> Thanks, >>> >>> Danyang >>> >>> >>> >>> On 22/09/2013 8:00 PM, Shri wrote: >>>> I think this is definitely an issue with setting the affinities for threads, i.e., the assignment of threads to cores. Ideally each thread should be assigned to a distinct core but in your case all the 4 threads are getting pinned to the same core resulting in such a massive slowdown. Unfortunately, the thread affinities for OpenMP are set through environment variables. For Intel's OpenMP one needs to define the thread affinities through the environment variable KMP_AFFINITY. See this document here http://software.intel.com/sites/products/documentation/studio/composer/en-us/2011Update/compiler_c/optaps/common/optaps_openmp_thread_affinity.htm. Try setting the affinities via KMP_AFFINITY and let us know if it works. >>>> >>>> Shri >>>> On Sep 21, 2013, at 11:06 PM, Danyang Su wrote: >>>> >>>>> Hi Shri, >>>>> >>>>> Thanks for your info. It can work with the option -threadcomm_type openmp. But another problem arises, as described as follows. >>>>> >>>>> The sparse matrix is 53760*53760 with 1067392 non-zero entries. If the codes is compiled using PETSc-3.4.2, it works fine, the equations can be solved quickly and I can see the speedup. But if the code is compiled using PETSc-dev with OpenMP option, it takes a long time in solving the equations and I cannot see any speedup when more processors are used. >>>>> >>>>> For PETSc-3.4.2, run by "mpiexec -n 4 ksp_inhm_d -log_summary log_mpi4_petsc3.4.2.log", the iteration and runtime are: >>>>> Iterations 6 time_assembly 0.4137E-01 time_ksp 0.9296E-01 >>>>> >>>>> For PETSc-dev, run by "mpiexec -n 1 ksp_inhm_d -threadcomm_type openmp -threadcomm_nthreads 4 -log_summary log_openmp_petsc_dev.log", the iteration and runtime are: >>>>> Iterations 6 time_assembly 0.3595E+03 time_ksp 0.2907E+00 >>>>> >>>>> Most of the time 'time_assembly 0.3595E+03' is spent on the following codes >>>>> do i = istart, iend - 1 >>>>> ii = ia_in(i+1) >>>>> jj = ia_in(i+2) >>>>> call MatSetValues(a, ione, i, jj-ii, ja_in(ii:jj-1)-1, a_in(ii:jj-1), Insert_Values, ierr) >>>>> end do >>>>> >>>>> The log files for both PETSc-3.4.2 and PETSc-dev are attached. >>>>> >>>>> Is there anything wrong with my codes or with running option? The above codes works fine when using MPICH. >>>>> >>>>> Thanks and regards, >>>>> >>>>> Danyang >>>>> >>>>> On 21/09/2013 2:09 PM, Shri wrote: >>>>>> There are three thread communicator types in PETSc. The default is "no thread" which is basically a non-threaded version. The other two types are "openmp" and "pthread". If you want to use OpenMP then use the option -threadcomm_type openmp. >>>>>> >>>>>> Shri >>>>>> >>>>>> On Sep 21, 2013, at 3:46 PM, Danyang Su wrote: >>>>>> >>>>>>> Hi Barry, >>>>>>> >>>>>>> Thanks for the quick reply. >>>>>>> >>>>>>> After changing >>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>>>> to >>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) >>>>>>> and comment out >>>>>>> #elif defined(PETSC_HAVE_OPENMP) >>>>>>> PETSC_EXTERN PetscStack *petscstack; >>>>>>> >>>>>>> It can be compiled and validated with "make test". >>>>>>> >>>>>>> But I still have questions on running the examples. After rebuild the codes (e.g., ksp_ex2f.f), I can run it with "mpiexec -n 1 ksp_ex2f", or "mpiexec -n 4 ksp_ex2f", or "mpiexec -n 1 ksp_ex2f -threadcomm_nthreads 1", but if I run it with "mpiexec -n 1 ksp_ex2f -threadcomm_nthreads 4", there will be a lot of error information (attached). >>>>>>> >>>>>>> The codes is not modified and there is no OpenMP routines in it. For the current development in my project, I want to keep the OpenMP codes in calculating matrix values, but want to solve it with PETSc (OpenMP). Is it possible? >>>>>>> >>>>>>> Thanks and regards, >>>>>>> >>>>>>> Danyang >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 21/09/2013 7:26 AM, Barry Smith wrote: >>>>>>>> Danyang, >>>>>>>> >>>>>>>> I don't think the || defined (PETSC_HAVE_OPENMP) belongs in the code below. >>>>>>>> >>>>>>>> /* Linux functions CPU_SET and others don't work if sched.h is not included before >>>>>>>> including pthread.h. Also, these functions are active only if either _GNU_SOURCE >>>>>>>> or __USE_GNU is not set (see /usr/include/sched.h and /usr/include/features.h), hence >>>>>>>> set these first. >>>>>>>> */ >>>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>>>>> >>>>>>>> Edit include/petscerror.h and locate these lines and remove that part and then rerun make all. Let us know if it works or not. >>>>>>>> >>>>>>>> Barry >>>>>>>> >>>>>>>> i.e. replace >>>>>>>> >>>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>>>>> >>>>>>>> with >>>>>>>> >>>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) >>>>>>>> >>>>>>>> On Sep 21, 2013, at 6:53 AM, Matthew Knepley >>>>>>>> >>>>>>>> wrote: >>>>>>>> >>>>>>>> >>>>>>>>> On Sat, Sep 21, 2013 at 12:18 AM, Danyang Su >>>>>>>>> wrote: >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> I got error information in compiling petsc-dev with openmp in cygwin. Before, I have successfully compiled petsc-3.4.2 and it works fine. >>>>>>>>> The log files have been attached. >>>>>>>>> >>>>>>>>> The OpenMP configure test is wrong. It clearly fails to find pthread.h, but the test passes. Then in petscerror.h >>>>>>>>> we guard pthread.h using PETSC_HAVE_OPENMP. Can someone who knows OpenMP fix this? >>>>>>>>> >>>>>>>>> Matt >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Danyang >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>> >>>>> > From bsmith at mcs.anl.gov Mon Sep 23 14:16:37 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 23 Sep 2013 14:16:37 -0500 Subject: [petsc-users] Strange efficiency in PETSc-dev using OpenMP In-Reply-To: <524092EC.6080605@gmail.com> References: <523D482D.5030409@gmail.com> <101244E8-FFFA-4F3F-A4BC-CC47F68D93B7@mcs.anl.gov> <523E059D.6080901@gmail.com> <933EF7F4-598A-4F88-8902-0613E27F0679@mcs.anl.gov> <523E6CAF.9020805@gmail.com> <52408989.7090001@gmail.com> <52409009.2000506@gmail.com> <524092EC.6080605@gmail.com> Message-ID: <47AF7C50-1533-4298-A579-901CE3AABEAC@mcs.anl.gov> On Sep 23, 2013, at 2:13 PM, Danyang Su wrote: > Hi Barry, > > Another strange problem: > > Currently I have PETSc-3.4.2 MPI version and PETSc-dev OpenMP version on my computer, with different environment variable of PETSC_ARCH and PETSC_DIR. Before installation of PETSc-dev OpenMP version, the PETSc-3.4.2 MPI version works fine. But after installation of PETSc-dev OpenMP version, the same problem exist in PETSc-3.4.2 MPI version if run with 1 processor, but no problem with 2 or more processors. Are you sure some environmental variable is not set? Related to OpenMP? What if you run on one process with mpiexec and then immediately after on two process with mpiexec in the same window? Barry > > Thanks, > > Danyang > > On 23/09/2013 12:01 PM, Danyang Su wrote: >> Hi Barry, >> >> Sorry I forgot the message in the previous email. It is still slow when run without the "-threadcomm_type openmp -threadcomm_nthreads 1" >> >> Thanks, >> >> Danyang >> >> On 23/09/2013 11:43 AM, Barry Smith wrote: >>> You did not answer my question from yesterday: >>> >>> If you run the Openmp compiled version WITHOUT the >>> >>> -threadcomm_nthreads 1 >>> -threadcomm_type openmp >>> >>> command line options is it still slow? >>> >>> >>> On Sep 23, 2013, at 1:33 PM, Danyang Su wrote: >>> >>>> Hi Shri, >>>> >>>> It seems that the problem does not result from the affinities setting for threads. I have tried several settings, the threads are set to different cores, but there is no improvement. >>>> >>>> Here is the information of package, core and thread maps >>>> >>>> OMP: Info #204: KMP_AFFINITY: decoding x2APIC ids. >>>> OMP: Info #202: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info >>>> OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: {0,1,2,3,4,5,6,7,8,9,10,11} >>>> OMP: Info #156: KMP_AFFINITY: 12 available OS procs >>>> OMP: Info #157: KMP_AFFINITY: Uniform topology >>>> OMP: Info #179: KMP_AFFINITY: 1 packages x 6 cores/pkg x 2 threads/core (6 total cores) >>>> OMP: Info #206: KMP_AFFINITY: OS proc to physical thread map: >>>> OMP: Info #171: KMP_AFFINITY: OS proc 0 maps to package 0 core 0 thread 0 >>>> OMP: Info #171: KMP_AFFINITY: OS proc 1 maps to package 0 core 0 thread 1 >>>> OMP: Info #171: KMP_AFFINITY: OS proc 2 maps to package 0 core 1 thread 0 >>>> OMP: Info #171: KMP_AFFINITY: OS proc 3 maps to package 0 core 1 thread 1 >>>> OMP: Info #171: KMP_AFFINITY: OS proc 4 maps to package 0 core 2 thread 0 >>>> OMP: Info #171: KMP_AFFINITY: OS proc 5 maps to package 0 core 2 thread 1 >>>> OMP: Info #171: KMP_AFFINITY: OS proc 6 maps to package 0 core 3 thread 0 >>>> OMP: Info #171: KMP_AFFINITY: OS proc 7 maps to package 0 core 3 thread 1 >>>> OMP: Info #171: KMP_AFFINITY: OS proc 8 maps to package 0 core 4 thread 0 >>>> OMP: Info #171: KMP_AFFINITY: OS proc 9 maps to package 0 core 4 thread 1 >>>> OMP: Info #171: KMP_AFFINITY: OS proc 10 maps to package 0 core 5 thread 0 >>>> OMP: Info #171: KMP_AFFINITY: OS proc 11 maps to package 0 core 5 thread 1 >>>> OMP: Info #144: KMP_AFFINITY: Threads may migrate across 1 innermost levels of machine >>>> >>>> >>>> And here is the internal thread bounding with different kmp_affinity settings: >>>> >>>> 1. KMP_AFFINITY=verbose,granularity=thread,compact >>>> >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {1} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {3} >>>> >>>> 2. KMP_AFFINITY=verbose,granularity=fine,compact >>>> >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {1} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {3} >>>> >>>> 3. KMP_AFFINITY=verbose,granularity=fine,compact,1,0 >>>> >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {2} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {4} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {6} >>>> >>>> 4. KMP_AFFINITY=verbose,scatter >>>> >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,1} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {2,3} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {4,5} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {6,7} >>>> >>>> 5. KMP_AFFINITY=verbose,compact (For this setting, two threads are assigned to the same core) >>>> >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,1} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {0,1} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2,3} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {2,3} >>>> >>>> 6. KMP_AFFINITY=verbose,granularity=core,compact (For this setting, two threads are assigned to the same core) >>>> >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,1} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {0,1} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2,3} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {2,3} >>>> >>>> The first 4 settings can assign threads to a distinct core, but the problem is not solved. >>>> >>>> Thanks, >>>> >>>> Danyang >>>> >>>> >>>> >>>> On 22/09/2013 8:00 PM, Shri wrote: >>>>> I think this is definitely an issue with setting the affinities for threads, i.e., the assignment of threads to cores. Ideally each thread should be assigned to a distinct core but in your case all the 4 threads are getting pinned to the same core resulting in such a massive slowdown. Unfortunately, the thread affinities for OpenMP are set through environment variables. For Intel's OpenMP one needs to define the thread affinities through the environment variable KMP_AFFINITY. See this document here http://software.intel.com/sites/products/documentation/studio/composer/en-us/2011Update/compiler_c/optaps/common/optaps_openmp_thread_affinity.htm. Try setting the affinities via KMP_AFFINITY and let us know if it works. >>>>> >>>>> Shri >>>>> On Sep 21, 2013, at 11:06 PM, Danyang Su wrote: >>>>> >>>>>> Hi Shri, >>>>>> >>>>>> Thanks for your info. It can work with the option -threadcomm_type openmp. But another problem arises, as described as follows. >>>>>> >>>>>> The sparse matrix is 53760*53760 with 1067392 non-zero entries. If the codes is compiled using PETSc-3.4.2, it works fine, the equations can be solved quickly and I can see the speedup. But if the code is compiled using PETSc-dev with OpenMP option, it takes a long time in solving the equations and I cannot see any speedup when more processors are used. >>>>>> >>>>>> For PETSc-3.4.2, run by "mpiexec -n 4 ksp_inhm_d -log_summary log_mpi4_petsc3.4.2.log", the iteration and runtime are: >>>>>> Iterations 6 time_assembly 0.4137E-01 time_ksp 0.9296E-01 >>>>>> >>>>>> For PETSc-dev, run by "mpiexec -n 1 ksp_inhm_d -threadcomm_type openmp -threadcomm_nthreads 4 -log_summary log_openmp_petsc_dev.log", the iteration and runtime are: >>>>>> Iterations 6 time_assembly 0.3595E+03 time_ksp 0.2907E+00 >>>>>> >>>>>> Most of the time 'time_assembly 0.3595E+03' is spent on the following codes >>>>>> do i = istart, iend - 1 >>>>>> ii = ia_in(i+1) >>>>>> jj = ia_in(i+2) >>>>>> call MatSetValues(a, ione, i, jj-ii, ja_in(ii:jj-1)-1, a_in(ii:jj-1), Insert_Values, ierr) >>>>>> end do >>>>>> >>>>>> The log files for both PETSc-3.4.2 and PETSc-dev are attached. >>>>>> >>>>>> Is there anything wrong with my codes or with running option? The above codes works fine when using MPICH. >>>>>> >>>>>> Thanks and regards, >>>>>> >>>>>> Danyang >>>>>> >>>>>> On 21/09/2013 2:09 PM, Shri wrote: >>>>>>> There are three thread communicator types in PETSc. The default is "no thread" which is basically a non-threaded version. The other two types are "openmp" and "pthread". If you want to use OpenMP then use the option -threadcomm_type openmp. >>>>>>> >>>>>>> Shri >>>>>>> >>>>>>> On Sep 21, 2013, at 3:46 PM, Danyang Su wrote: >>>>>>> >>>>>>>> Hi Barry, >>>>>>>> >>>>>>>> Thanks for the quick reply. >>>>>>>> >>>>>>>> After changing >>>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>>>>> to >>>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) >>>>>>>> and comment out >>>>>>>> #elif defined(PETSC_HAVE_OPENMP) >>>>>>>> PETSC_EXTERN PetscStack *petscstack; >>>>>>>> >>>>>>>> It can be compiled and validated with "make test". >>>>>>>> >>>>>>>> But I still have questions on running the examples. After rebuild the codes (e.g., ksp_ex2f.f), I can run it with "mpiexec -n 1 ksp_ex2f", or "mpiexec -n 4 ksp_ex2f", or "mpiexec -n 1 ksp_ex2f -threadcomm_nthreads 1", but if I run it with "mpiexec -n 1 ksp_ex2f -threadcomm_nthreads 4", there will be a lot of error information (attached). >>>>>>>> >>>>>>>> The codes is not modified and there is no OpenMP routines in it. For the current development in my project, I want to keep the OpenMP codes in calculating matrix values, but want to solve it with PETSc (OpenMP). Is it possible? >>>>>>>> >>>>>>>> Thanks and regards, >>>>>>>> >>>>>>>> Danyang >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 21/09/2013 7:26 AM, Barry Smith wrote: >>>>>>>>> Danyang, >>>>>>>>> >>>>>>>>> I don't think the || defined (PETSC_HAVE_OPENMP) belongs in the code below. >>>>>>>>> >>>>>>>>> /* Linux functions CPU_SET and others don't work if sched.h is not included before >>>>>>>>> including pthread.h. Also, these functions are active only if either _GNU_SOURCE >>>>>>>>> or __USE_GNU is not set (see /usr/include/sched.h and /usr/include/features.h), hence >>>>>>>>> set these first. >>>>>>>>> */ >>>>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>>>>>> >>>>>>>>> Edit include/petscerror.h and locate these lines and remove that part and then rerun make all. Let us know if it works or not. >>>>>>>>> >>>>>>>>> Barry >>>>>>>>> >>>>>>>>> i.e. replace >>>>>>>>> >>>>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>>>>>> >>>>>>>>> with >>>>>>>>> >>>>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) >>>>>>>>> >>>>>>>>> On Sep 21, 2013, at 6:53 AM, Matthew Knepley >>>>>>>>> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>>> On Sat, Sep 21, 2013 at 12:18 AM, Danyang Su >>>>>>>>>> wrote: >>>>>>>>>> Hi All, >>>>>>>>>> >>>>>>>>>> I got error information in compiling petsc-dev with openmp in cygwin. Before, I have successfully compiled petsc-3.4.2 and it works fine. >>>>>>>>>> The log files have been attached. >>>>>>>>>> >>>>>>>>>> The OpenMP configure test is wrong. It clearly fails to find pthread.h, but the test passes. Then in petscerror.h >>>>>>>>>> we guard pthread.h using PETSC_HAVE_OPENMP. Can someone who knows OpenMP fix this? >>>>>>>>>> >>>>>>>>>> Matt >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Danyang >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>>>>> -- Norbert Wiener >>>>>>>>>> >>>>>>>> >>>>>> >> > From jedbrown at mcs.anl.gov Mon Sep 23 14:33:30 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 23 Sep 2013 20:33:30 +0100 Subject: [petsc-users] Strange efficiency in PETSc-dev using OpenMP In-Reply-To: References: <523D482D.5030409@gmail.com> <101244E8-FFFA-4F3F-A4BC-CC47F68D93B7@mcs.anl.gov> <523E059D.6080901@gmail.com> <933EF7F4-598A-4F88-8902-0613E27F0679@mcs.anl.gov> <523E6CAF.9020805@gmail.com> <52408989.7090001@gmail.com> <52409009.2000506@gmail.com> Message-ID: <8738ov49kl.fsf@mcs.anl.gov> Barry Smith writes: > So when you compile the software to use OpenMP it is slow; > regardless of whether you use OpenMP explicitly or not. When you > compile the software to NOT use OpenMP then it is much faster? Are you running multiple MPI processes? How are the affinities set? Are any other jobs competing for the same pinned core as your application? -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From dave.mayhem23 at gmail.com Mon Sep 23 14:36:31 2013 From: dave.mayhem23 at gmail.com (Dave May) Date: Mon, 23 Sep 2013 21:36:31 +0200 Subject: [petsc-users] How to speed up geometric multigrid In-Reply-To: <4F04E2EB-90FC-4EAD-B33A-9510AC8593F7@lbl.gov> References: <52406460.3060905@uci.edu> <52406C01.5090804@uci.edu> <52407435.6080302@uci.edu> <4F04E2EB-90FC-4EAD-B33A-9510AC8593F7@lbl.gov> Message-ID: Hey Michele, I think Galerkin MG, even with a jump in the coefficient of 10^3, can be made more robust and faster for your problem. I'd start by trying different smoothers. Try cheby/(bjacobi or asm), or gmres/(bjacobi or asm). When using cheby, make sure you ask for the spectrum estimate. Definitely do more than one application of KSP on each level. I don't think you should switch to AMG without first trying more robust smoothers and profiling how much time is spent in the coarse grid solve. I solve similar problems (geometry and size of the jump) in connection with variable visocsity stokes problems on structured grids, and I'm yet to see a case with AMG is more robust and or faster than Galerkin MG. Cheers, Dave On 23 September 2013 20:52, Mark F. Adams wrote: > > >> > > The simulation does start with both phases and the geometry is supposed > to become more complex as the simulation progresses. > > But so far the run is stopped before there are significant changes in > the shape of the droplet. > > Humm, not sure why you are seeing degradation then. I imagine the > initial geometry is not grid aligned so I'm not sure why you are seeing > degradation in convergence rate. > > > I can give a shot to AMG: which options would you suggest to use. > > -pc_type gamg -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 > > > Also, how can I project out the constant from the rhs? Thanks a lot! > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPSetNullSpace.html#KSPSetNullSpace > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From garnet.vaz at gmail.com Mon Sep 23 15:00:28 2013 From: garnet.vaz at gmail.com (Garnet Vaz) Date: Mon, 23 Sep 2013 13:00:28 -0700 Subject: [petsc-users] Writing VTK output for vectors derived from DM Global vector Message-ID: Hi, I would like to write out my solution to a VTK file where I use unstructured grids in DMPlex. The solution corresponds to harmonic frequencies (~ 5-10 distinct) which are currently saved into a Vec* object. These vectors were created by duplicating the global vector for the DM object. (I know that they correspond to the right cell.) Is it possible to have one file which would contain the mesh from the DM object and all the Vec* objects? Would I have to clone the original DM and use http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerVTKAddField.html to include all the vectors? In the above function does PetscViewerVTKWriteFunction just need to be myWriteFunc(vec,viewer) { VecView(vec,viewer) } if I do not need to make any changes to the vector? Thanks. -- Regards, Garnet -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Mon Sep 23 15:24:05 2013 From: mfadams at lbl.gov (Mark F. Adams) Date: Mon, 23 Sep 2013 16:24:05 -0400 Subject: [petsc-users] How to speed up geometric multigrid In-Reply-To: References: <52406460.3060905@uci.edu> <52406C01.5090804@uci.edu> <52407435.6080302@uci.edu> <4F04E2EB-90FC-4EAD-B33A-9510AC8593F7@lbl.gov> Message-ID: Ah yes, I agree, geometric MG with Galerkin is worth trying. On Sep 23, 2013, at 3:36 PM, Dave May wrote: > Hey Michele, > > I think Galerkin MG, even with a jump in the coefficient of 10^3, can be made more robust and faster for your problem. > I'd start by trying different smoothers. Try cheby/(bjacobi or asm), or gmres/(bjacobi or asm). When using cheby, make sure you ask for the spectrum estimate. > Definitely do more than one application of KSP on each level. > > I don't think you should switch to AMG without first trying more robust smoothers and profiling how much time is spent in the coarse grid solve. > I solve similar problems (geometry and size of the jump) in connection with variable visocsity stokes problems on structured grids, and I'm yet to see a case with AMG is more robust and or faster than Galerkin MG. > > > Cheers, > Dave > > > > On 23 September 2013 20:52, Mark F. Adams wrote: > > >> > > The simulation does start with both phases and the geometry is supposed to become more complex as the simulation progresses. > > But so far the run is stopped before there are significant changes in the shape of the droplet. > > Humm, not sure why you are seeing degradation then. I imagine the initial geometry is not grid aligned so I'm not sure why you are seeing degradation in convergence rate. > > > I can give a shot to AMG: which options would you suggest to use. > > -pc_type gamg -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 > > > Also, how can I project out the constant from the rhs? Thanks a lot! > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPSetNullSpace.html#KSPSetNullSpace > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From danyang.su at gmail.com Mon Sep 23 15:25:38 2013 From: danyang.su at gmail.com (Danyang Su) Date: Mon, 23 Sep 2013 13:25:38 -0700 Subject: [petsc-users] Strange efficiency in PETSc-dev using OpenMP In-Reply-To: References: <523D482D.5030409@gmail.com> <101244E8-FFFA-4F3F-A4BC-CC47F68D93B7@mcs.anl.gov> <523E059D.6080901@gmail.com> <933EF7F4-598A-4F88-8902-0613E27F0679@mcs.anl.gov> <523E6CAF.9020805@gmail.com> <52408989.7090001@gmail.com> <52409009.2000506@gmail.com> Message-ID: <5240A3C2.7010207@gmail.com> Hi Barry, Yes, it works fine when OpenMP is not used. I will try GNU compiler. Thanks, Danyang On 23/09/2013 12:15 PM, Barry Smith wrote: > On Sep 23, 2013, at 2:01 PM, Danyang Su wrote: > >> Hi Barry, >> >> Sorry I forgot the message in the previous email. It is still slow when run without the "-threadcomm_type openmp -threadcomm_nthreads 1" >> > So when you compile the software to use OpenMP it is slow; regardless of whether you use OpenMP explicitly or not. When you compile the software to NOT use OpenMP then it is much faster? > > If this is the case they there is nothing we can do to help you. Something is wrong with the OpenMP compiler option or the libraries it links it. You might consider Linux and the gnu compilers. > > > Barry > > >> Thanks, >> >> Danyang >> >> On 23/09/2013 11:43 AM, Barry Smith wrote: >>> You did not answer my question from yesterday: >>> >>> If you run the Openmp compiled version WITHOUT the >>> >>> -threadcomm_nthreads 1 >>> -threadcomm_type openmp >>> >>> command line options is it still slow? >>> >>> >>> On Sep 23, 2013, at 1:33 PM, Danyang Su wrote: >>> >>>> Hi Shri, >>>> >>>> It seems that the problem does not result from the affinities setting for threads. I have tried several settings, the threads are set to different cores, but there is no improvement. >>>> >>>> Here is the information of package, core and thread maps >>>> >>>> OMP: Info #204: KMP_AFFINITY: decoding x2APIC ids. >>>> OMP: Info #202: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info >>>> OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: {0,1,2,3,4,5,6,7,8,9,10,11} >>>> OMP: Info #156: KMP_AFFINITY: 12 available OS procs >>>> OMP: Info #157: KMP_AFFINITY: Uniform topology >>>> OMP: Info #179: KMP_AFFINITY: 1 packages x 6 cores/pkg x 2 threads/core (6 total cores) >>>> OMP: Info #206: KMP_AFFINITY: OS proc to physical thread map: >>>> OMP: Info #171: KMP_AFFINITY: OS proc 0 maps to package 0 core 0 thread 0 >>>> OMP: Info #171: KMP_AFFINITY: OS proc 1 maps to package 0 core 0 thread 1 >>>> OMP: Info #171: KMP_AFFINITY: OS proc 2 maps to package 0 core 1 thread 0 >>>> OMP: Info #171: KMP_AFFINITY: OS proc 3 maps to package 0 core 1 thread 1 >>>> OMP: Info #171: KMP_AFFINITY: OS proc 4 maps to package 0 core 2 thread 0 >>>> OMP: Info #171: KMP_AFFINITY: OS proc 5 maps to package 0 core 2 thread 1 >>>> OMP: Info #171: KMP_AFFINITY: OS proc 6 maps to package 0 core 3 thread 0 >>>> OMP: Info #171: KMP_AFFINITY: OS proc 7 maps to package 0 core 3 thread 1 >>>> OMP: Info #171: KMP_AFFINITY: OS proc 8 maps to package 0 core 4 thread 0 >>>> OMP: Info #171: KMP_AFFINITY: OS proc 9 maps to package 0 core 4 thread 1 >>>> OMP: Info #171: KMP_AFFINITY: OS proc 10 maps to package 0 core 5 thread 0 >>>> OMP: Info #171: KMP_AFFINITY: OS proc 11 maps to package 0 core 5 thread 1 >>>> OMP: Info #144: KMP_AFFINITY: Threads may migrate across 1 innermost levels of machine >>>> >>>> >>>> And here is the internal thread bounding with different kmp_affinity settings: >>>> >>>> 1. KMP_AFFINITY=verbose,granularity=thread,compact >>>> >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {1} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {3} >>>> >>>> 2. KMP_AFFINITY=verbose,granularity=fine,compact >>>> >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {1} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {3} >>>> >>>> 3. KMP_AFFINITY=verbose,granularity=fine,compact,1,0 >>>> >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {2} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {4} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {6} >>>> >>>> 4. KMP_AFFINITY=verbose,scatter >>>> >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,1} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {2,3} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {4,5} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {6,7} >>>> >>>> 5. KMP_AFFINITY=verbose,compact (For this setting, two threads are assigned to the same core) >>>> >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,1} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {0,1} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2,3} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {2,3} >>>> >>>> 6. KMP_AFFINITY=verbose,granularity=core,compact (For this setting, two threads are assigned to the same core) >>>> >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,1} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {0,1} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2,3} >>>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {2,3} >>>> >>>> The first 4 settings can assign threads to a distinct core, but the problem is not solved. >>>> >>>> Thanks, >>>> >>>> Danyang >>>> >>>> >>>> >>>> On 22/09/2013 8:00 PM, Shri wrote: >>>>> I think this is definitely an issue with setting the affinities for threads, i.e., the assignment of threads to cores. Ideally each thread should be assigned to a distinct core but in your case all the 4 threads are getting pinned to the same core resulting in such a massive slowdown. Unfortunately, the thread affinities for OpenMP are set through environment variables. For Intel's OpenMP one needs to define the thread affinities through the environment variable KMP_AFFINITY. See this document here http://software.intel.com/sites/products/documentation/studio/composer/en-us/2011Update/compiler_c/optaps/common/optaps_openmp_thread_affinity.htm. Try setting the affinities via KMP_AFFINITY and let us know if it works. >>>>> >>>>> Shri >>>>> On Sep 21, 2013, at 11:06 PM, Danyang Su wrote: >>>>> >>>>>> Hi Shri, >>>>>> >>>>>> Thanks for your info. It can work with the option -threadcomm_type openmp. But another problem arises, as described as follows. >>>>>> >>>>>> The sparse matrix is 53760*53760 with 1067392 non-zero entries. If the codes is compiled using PETSc-3.4.2, it works fine, the equations can be solved quickly and I can see the speedup. But if the code is compiled using PETSc-dev with OpenMP option, it takes a long time in solving the equations and I cannot see any speedup when more processors are used. >>>>>> >>>>>> For PETSc-3.4.2, run by "mpiexec -n 4 ksp_inhm_d -log_summary log_mpi4_petsc3.4.2.log", the iteration and runtime are: >>>>>> Iterations 6 time_assembly 0.4137E-01 time_ksp 0.9296E-01 >>>>>> >>>>>> For PETSc-dev, run by "mpiexec -n 1 ksp_inhm_d -threadcomm_type openmp -threadcomm_nthreads 4 -log_summary log_openmp_petsc_dev.log", the iteration and runtime are: >>>>>> Iterations 6 time_assembly 0.3595E+03 time_ksp 0.2907E+00 >>>>>> >>>>>> Most of the time 'time_assembly 0.3595E+03' is spent on the following codes >>>>>> do i = istart, iend - 1 >>>>>> ii = ia_in(i+1) >>>>>> jj = ia_in(i+2) >>>>>> call MatSetValues(a, ione, i, jj-ii, ja_in(ii:jj-1)-1, a_in(ii:jj-1), Insert_Values, ierr) >>>>>> end do >>>>>> >>>>>> The log files for both PETSc-3.4.2 and PETSc-dev are attached. >>>>>> >>>>>> Is there anything wrong with my codes or with running option? The above codes works fine when using MPICH. >>>>>> >>>>>> Thanks and regards, >>>>>> >>>>>> Danyang >>>>>> >>>>>> On 21/09/2013 2:09 PM, Shri wrote: >>>>>>> There are three thread communicator types in PETSc. The default is "no thread" which is basically a non-threaded version. The other two types are "openmp" and "pthread". If you want to use OpenMP then use the option -threadcomm_type openmp. >>>>>>> >>>>>>> Shri >>>>>>> >>>>>>> On Sep 21, 2013, at 3:46 PM, Danyang Su wrote: >>>>>>> >>>>>>>> Hi Barry, >>>>>>>> >>>>>>>> Thanks for the quick reply. >>>>>>>> >>>>>>>> After changing >>>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>>>>> to >>>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) >>>>>>>> and comment out >>>>>>>> #elif defined(PETSC_HAVE_OPENMP) >>>>>>>> PETSC_EXTERN PetscStack *petscstack; >>>>>>>> >>>>>>>> It can be compiled and validated with "make test". >>>>>>>> >>>>>>>> But I still have questions on running the examples. After rebuild the codes (e.g., ksp_ex2f.f), I can run it with "mpiexec -n 1 ksp_ex2f", or "mpiexec -n 4 ksp_ex2f", or "mpiexec -n 1 ksp_ex2f -threadcomm_nthreads 1", but if I run it with "mpiexec -n 1 ksp_ex2f -threadcomm_nthreads 4", there will be a lot of error information (attached). >>>>>>>> >>>>>>>> The codes is not modified and there is no OpenMP routines in it. For the current development in my project, I want to keep the OpenMP codes in calculating matrix values, but want to solve it with PETSc (OpenMP). Is it possible? >>>>>>>> >>>>>>>> Thanks and regards, >>>>>>>> >>>>>>>> Danyang >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 21/09/2013 7:26 AM, Barry Smith wrote: >>>>>>>>> Danyang, >>>>>>>>> >>>>>>>>> I don't think the || defined (PETSC_HAVE_OPENMP) belongs in the code below. >>>>>>>>> >>>>>>>>> /* Linux functions CPU_SET and others don't work if sched.h is not included before >>>>>>>>> including pthread.h. Also, these functions are active only if either _GNU_SOURCE >>>>>>>>> or __USE_GNU is not set (see /usr/include/sched.h and /usr/include/features.h), hence >>>>>>>>> set these first. >>>>>>>>> */ >>>>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>>>>>> >>>>>>>>> Edit include/petscerror.h and locate these lines and remove that part and then rerun make all. Let us know if it works or not. >>>>>>>>> >>>>>>>>> Barry >>>>>>>>> >>>>>>>>> i.e. replace >>>>>>>>> >>>>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>>>>>> >>>>>>>>> with >>>>>>>>> >>>>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) >>>>>>>>> >>>>>>>>> On Sep 21, 2013, at 6:53 AM, Matthew Knepley >>>>>>>>> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>>> On Sat, Sep 21, 2013 at 12:18 AM, Danyang Su >>>>>>>>>> wrote: >>>>>>>>>> Hi All, >>>>>>>>>> >>>>>>>>>> I got error information in compiling petsc-dev with openmp in cygwin. Before, I have successfully compiled petsc-3.4.2 and it works fine. >>>>>>>>>> The log files have been attached. >>>>>>>>>> >>>>>>>>>> The OpenMP configure test is wrong. It clearly fails to find pthread.h, but the test passes. Then in petscerror.h >>>>>>>>>> we guard pthread.h using PETSC_HAVE_OPENMP. Can someone who knows OpenMP fix this? >>>>>>>>>> >>>>>>>>>> Matt >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Danyang >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>>>>> -- Norbert Wiener >>>>>>>>>> >>>>>>>> >>>>>> From danyang.su at gmail.com Mon Sep 23 15:32:22 2013 From: danyang.su at gmail.com (Danyang Su) Date: Mon, 23 Sep 2013 13:32:22 -0700 Subject: [petsc-users] Strange efficiency in PETSc-dev using OpenMP In-Reply-To: <47AF7C50-1533-4298-A579-901CE3AABEAC@mcs.anl.gov> References: <523D482D.5030409@gmail.com> <101244E8-FFFA-4F3F-A4BC-CC47F68D93B7@mcs.anl.gov> <523E059D.6080901@gmail.com> <933EF7F4-598A-4F88-8902-0613E27F0679@mcs.anl.gov> <523E6CAF.9020805@gmail.com> <52408989.7090001@gmail.com> <52409009.2000506@gmail.com> <524092EC.6080605@gmail.com> <47AF7C50-1533-4298-A579-901CE3AABEAC@mcs.anl.gov> Message-ID: <5240A556.9010501@gmail.com> On 23/09/2013 12:16 PM, Barry Smith wrote: > On Sep 23, 2013, at 2:13 PM, Danyang Su wrote: > >> Hi Barry, >> >> Another strange problem: >> >> Currently I have PETSc-3.4.2 MPI version and PETSc-dev OpenMP version on my computer, with different environment variable of PETSC_ARCH and PETSC_DIR. Before installation of PETSc-dev OpenMP version, the PETSc-3.4.2 MPI version works fine. But after installation of PETSc-dev OpenMP version, the same problem exist in PETSc-3.4.2 MPI version if run with 1 processor, but no problem with 2 or more processors. > Are you sure some environmental variable is not set? Related to OpenMP? What if you run on one process with mpiexec and then immediately after on two process with mpiexec in the same window? No special environmental variable is set. Run on two or more processors with mpiexec in the same windows immediately after works fine. The problem only exist when I run one processor with mpiexec. Danyang > > Barry > >> Thanks, >> >> Danyang >> >> On 23/09/2013 12:01 PM, Danyang Su wrote: >>> Hi Barry, >>> >>> Sorry I forgot the message in the previous email. It is still slow when run without the "-threadcomm_type openmp -threadcomm_nthreads 1" >>> >>> Thanks, >>> >>> Danyang >>> >>> On 23/09/2013 11:43 AM, Barry Smith wrote: >>>> You did not answer my question from yesterday: >>>> >>>> If you run the Openmp compiled version WITHOUT the >>>> >>>> -threadcomm_nthreads 1 >>>> -threadcomm_type openmp >>>> >>>> command line options is it still slow? >>>> >>>> >>>> On Sep 23, 2013, at 1:33 PM, Danyang Su wrote: >>>> >>>>> Hi Shri, >>>>> >>>>> It seems that the problem does not result from the affinities setting for threads. I have tried several settings, the threads are set to different cores, but there is no improvement. >>>>> >>>>> Here is the information of package, core and thread maps >>>>> >>>>> OMP: Info #204: KMP_AFFINITY: decoding x2APIC ids. >>>>> OMP: Info #202: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info >>>>> OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: {0,1,2,3,4,5,6,7,8,9,10,11} >>>>> OMP: Info #156: KMP_AFFINITY: 12 available OS procs >>>>> OMP: Info #157: KMP_AFFINITY: Uniform topology >>>>> OMP: Info #179: KMP_AFFINITY: 1 packages x 6 cores/pkg x 2 threads/core (6 total cores) >>>>> OMP: Info #206: KMP_AFFINITY: OS proc to physical thread map: >>>>> OMP: Info #171: KMP_AFFINITY: OS proc 0 maps to package 0 core 0 thread 0 >>>>> OMP: Info #171: KMP_AFFINITY: OS proc 1 maps to package 0 core 0 thread 1 >>>>> OMP: Info #171: KMP_AFFINITY: OS proc 2 maps to package 0 core 1 thread 0 >>>>> OMP: Info #171: KMP_AFFINITY: OS proc 3 maps to package 0 core 1 thread 1 >>>>> OMP: Info #171: KMP_AFFINITY: OS proc 4 maps to package 0 core 2 thread 0 >>>>> OMP: Info #171: KMP_AFFINITY: OS proc 5 maps to package 0 core 2 thread 1 >>>>> OMP: Info #171: KMP_AFFINITY: OS proc 6 maps to package 0 core 3 thread 0 >>>>> OMP: Info #171: KMP_AFFINITY: OS proc 7 maps to package 0 core 3 thread 1 >>>>> OMP: Info #171: KMP_AFFINITY: OS proc 8 maps to package 0 core 4 thread 0 >>>>> OMP: Info #171: KMP_AFFINITY: OS proc 9 maps to package 0 core 4 thread 1 >>>>> OMP: Info #171: KMP_AFFINITY: OS proc 10 maps to package 0 core 5 thread 0 >>>>> OMP: Info #171: KMP_AFFINITY: OS proc 11 maps to package 0 core 5 thread 1 >>>>> OMP: Info #144: KMP_AFFINITY: Threads may migrate across 1 innermost levels of machine >>>>> >>>>> >>>>> And here is the internal thread bounding with different kmp_affinity settings: >>>>> >>>>> 1. KMP_AFFINITY=verbose,granularity=thread,compact >>>>> >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {1} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {3} >>>>> >>>>> 2. KMP_AFFINITY=verbose,granularity=fine,compact >>>>> >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {1} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {3} >>>>> >>>>> 3. KMP_AFFINITY=verbose,granularity=fine,compact,1,0 >>>>> >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {2} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {4} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {6} >>>>> >>>>> 4. KMP_AFFINITY=verbose,scatter >>>>> >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,1} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {2,3} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {4,5} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {6,7} >>>>> >>>>> 5. KMP_AFFINITY=verbose,compact (For this setting, two threads are assigned to the same core) >>>>> >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,1} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {0,1} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2,3} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {2,3} >>>>> >>>>> 6. KMP_AFFINITY=verbose,granularity=core,compact (For this setting, two threads are assigned to the same core) >>>>> >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,1} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {0,1} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2,3} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {2,3} >>>>> >>>>> The first 4 settings can assign threads to a distinct core, but the problem is not solved. >>>>> >>>>> Thanks, >>>>> >>>>> Danyang >>>>> >>>>> >>>>> >>>>> On 22/09/2013 8:00 PM, Shri wrote: >>>>>> I think this is definitely an issue with setting the affinities for threads, i.e., the assignment of threads to cores. Ideally each thread should be assigned to a distinct core but in your case all the 4 threads are getting pinned to the same core resulting in such a massive slowdown. Unfortunately, the thread affinities for OpenMP are set through environment variables. For Intel's OpenMP one needs to define the thread affinities through the environment variable KMP_AFFINITY. See this document here http://software.intel.com/sites/products/documentation/studio/composer/en-us/2011Update/compiler_c/optaps/common/optaps_openmp_thread_affinity.htm. Try setting the affinities via KMP_AFFINITY and let us know if it works. >>>>>> >>>>>> Shri >>>>>> On Sep 21, 2013, at 11:06 PM, Danyang Su wrote: >>>>>> >>>>>>> Hi Shri, >>>>>>> >>>>>>> Thanks for your info. It can work with the option -threadcomm_type openmp. But another problem arises, as described as follows. >>>>>>> >>>>>>> The sparse matrix is 53760*53760 with 1067392 non-zero entries. If the codes is compiled using PETSc-3.4.2, it works fine, the equations can be solved quickly and I can see the speedup. But if the code is compiled using PETSc-dev with OpenMP option, it takes a long time in solving the equations and I cannot see any speedup when more processors are used. >>>>>>> >>>>>>> For PETSc-3.4.2, run by "mpiexec -n 4 ksp_inhm_d -log_summary log_mpi4_petsc3.4.2.log", the iteration and runtime are: >>>>>>> Iterations 6 time_assembly 0.4137E-01 time_ksp 0.9296E-01 >>>>>>> >>>>>>> For PETSc-dev, run by "mpiexec -n 1 ksp_inhm_d -threadcomm_type openmp -threadcomm_nthreads 4 -log_summary log_openmp_petsc_dev.log", the iteration and runtime are: >>>>>>> Iterations 6 time_assembly 0.3595E+03 time_ksp 0.2907E+00 >>>>>>> >>>>>>> Most of the time 'time_assembly 0.3595E+03' is spent on the following codes >>>>>>> do i = istart, iend - 1 >>>>>>> ii = ia_in(i+1) >>>>>>> jj = ia_in(i+2) >>>>>>> call MatSetValues(a, ione, i, jj-ii, ja_in(ii:jj-1)-1, a_in(ii:jj-1), Insert_Values, ierr) >>>>>>> end do >>>>>>> >>>>>>> The log files for both PETSc-3.4.2 and PETSc-dev are attached. >>>>>>> >>>>>>> Is there anything wrong with my codes or with running option? The above codes works fine when using MPICH. >>>>>>> >>>>>>> Thanks and regards, >>>>>>> >>>>>>> Danyang >>>>>>> >>>>>>> On 21/09/2013 2:09 PM, Shri wrote: >>>>>>>> There are three thread communicator types in PETSc. The default is "no thread" which is basically a non-threaded version. The other two types are "openmp" and "pthread". If you want to use OpenMP then use the option -threadcomm_type openmp. >>>>>>>> >>>>>>>> Shri >>>>>>>> >>>>>>>> On Sep 21, 2013, at 3:46 PM, Danyang Su wrote: >>>>>>>> >>>>>>>>> Hi Barry, >>>>>>>>> >>>>>>>>> Thanks for the quick reply. >>>>>>>>> >>>>>>>>> After changing >>>>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>>>>>> to >>>>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) >>>>>>>>> and comment out >>>>>>>>> #elif defined(PETSC_HAVE_OPENMP) >>>>>>>>> PETSC_EXTERN PetscStack *petscstack; >>>>>>>>> >>>>>>>>> It can be compiled and validated with "make test". >>>>>>>>> >>>>>>>>> But I still have questions on running the examples. After rebuild the codes (e.g., ksp_ex2f.f), I can run it with "mpiexec -n 1 ksp_ex2f", or "mpiexec -n 4 ksp_ex2f", or "mpiexec -n 1 ksp_ex2f -threadcomm_nthreads 1", but if I run it with "mpiexec -n 1 ksp_ex2f -threadcomm_nthreads 4", there will be a lot of error information (attached). >>>>>>>>> >>>>>>>>> The codes is not modified and there is no OpenMP routines in it. For the current development in my project, I want to keep the OpenMP codes in calculating matrix values, but want to solve it with PETSc (OpenMP). Is it possible? >>>>>>>>> >>>>>>>>> Thanks and regards, >>>>>>>>> >>>>>>>>> Danyang >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 21/09/2013 7:26 AM, Barry Smith wrote: >>>>>>>>>> Danyang, >>>>>>>>>> >>>>>>>>>> I don't think the || defined (PETSC_HAVE_OPENMP) belongs in the code below. >>>>>>>>>> >>>>>>>>>> /* Linux functions CPU_SET and others don't work if sched.h is not included before >>>>>>>>>> including pthread.h. Also, these functions are active only if either _GNU_SOURCE >>>>>>>>>> or __USE_GNU is not set (see /usr/include/sched.h and /usr/include/features.h), hence >>>>>>>>>> set these first. >>>>>>>>>> */ >>>>>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>>>>>>> >>>>>>>>>> Edit include/petscerror.h and locate these lines and remove that part and then rerun make all. Let us know if it works or not. >>>>>>>>>> >>>>>>>>>> Barry >>>>>>>>>> >>>>>>>>>> i.e. replace >>>>>>>>>> >>>>>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>>>>>>> >>>>>>>>>> with >>>>>>>>>> >>>>>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) >>>>>>>>>> >>>>>>>>>> On Sep 21, 2013, at 6:53 AM, Matthew Knepley >>>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On Sat, Sep 21, 2013 at 12:18 AM, Danyang Su >>>>>>>>>>> wrote: >>>>>>>>>>> Hi All, >>>>>>>>>>> >>>>>>>>>>> I got error information in compiling petsc-dev with openmp in cygwin. Before, I have successfully compiled petsc-3.4.2 and it works fine. >>>>>>>>>>> The log files have been attached. >>>>>>>>>>> >>>>>>>>>>> The OpenMP configure test is wrong. It clearly fails to find pthread.h, but the test passes. Then in petscerror.h >>>>>>>>>>> we guard pthread.h using PETSC_HAVE_OPENMP. Can someone who knows OpenMP fix this? >>>>>>>>>>> >>>>>>>>>>> Matt >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Danyang >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>> >>>>>>>>> >>>>>>> From danyang.su at gmail.com Mon Sep 23 15:40:29 2013 From: danyang.su at gmail.com (Danyang Su) Date: Mon, 23 Sep 2013 13:40:29 -0700 Subject: [petsc-users] Strange efficiency in PETSc-dev using OpenMP In-Reply-To: <8738ov49kl.fsf@mcs.anl.gov> References: <523D482D.5030409@gmail.com> <101244E8-FFFA-4F3F-A4BC-CC47F68D93B7@mcs.anl.gov> <523E059D.6080901@gmail.com> <933EF7F4-598A-4F88-8902-0613E27F0679@mcs.anl.gov> <523E6CAF.9020805@gmail.com> <52408989.7090001@gmail.com> <52409009.2000506@gmail.com> <8738ov49kl.fsf@mcs.anl.gov> Message-ID: <5240A73D.9030206@gmail.com> On 23/09/2013 12:33 PM, Jed Brown wrote: > Barry Smith writes: >> So when you compile the software to use OpenMP it is slow; >> regardless of whether you use OpenMP explicitly or not. When you >> compile the software to NOT use OpenMP then it is much faster? > Are you running multiple MPI processes? How are the affinities set? > Are any other jobs competing for the same pinned core as your > application? No. Only one MPI processes is running. And no other application is competing for the cores. The affinities and threads mapping are as follows. All the threads are assigned to a distinct core. KMP_AFFINITY=verbose,granularity=fine,compact OMP: Info #204: KMP_AFFINITY: decoding x2APIC ids. OMP: Info #202: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: {0,1,2,3,4,5,6,7,8,9,10,11} OMP: Info #156: KMP_AFFINITY: 12 available OS procs OMP: Info #157: KMP_AFFINITY: Uniform topology OMP: Info #179: KMP_AFFINITY: 1 packages x 6 cores/pkg x 2 threads/core (6 total cores) OMP: Info #206: KMP_AFFINITY: OS proc to physical thread map: OMP: Info #171: KMP_AFFINITY: OS proc 0 maps to package 0 core 0 thread 0 OMP: Info #171: KMP_AFFINITY: OS proc 1 maps to package 0 core 0 thread 1 OMP: Info #171: KMP_AFFINITY: OS proc 2 maps to package 0 core 1 thread 0 OMP: Info #171: KMP_AFFINITY: OS proc 3 maps to package 0 core 1 thread 1 OMP: Info #171: KMP_AFFINITY: OS proc 4 maps to package 0 core 2 thread 0 OMP: Info #171: KMP_AFFINITY: OS proc 5 maps to package 0 core 2 thread 1 OMP: Info #171: KMP_AFFINITY: OS proc 6 maps to package 0 core 3 thread 0 OMP: Info #171: KMP_AFFINITY: OS proc 7 maps to package 0 core 3 thread 1 OMP: Info #171: KMP_AFFINITY: OS proc 8 maps to package 0 core 4 thread 0 OMP: Info #171: KMP_AFFINITY: OS proc 9 maps to package 0 core 4 thread 1 OMP: Info #171: KMP_AFFINITY: OS proc 10 maps to package 0 core 5 thread 0 OMP: Info #171: KMP_AFFINITY: OS proc 11 maps to package 0 core 5 thread 1 OMP: Info #144: KMP_AFFINITY: Threads may migrate across 1 innermost levels of machine OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0} OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {1} OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2} OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {3} From mrosso at uci.edu Mon Sep 23 17:37:04 2013 From: mrosso at uci.edu (Michele Rosso) Date: Mon, 23 Sep 2013 15:37:04 -0700 Subject: [petsc-users] How to speed up geometric multigrid In-Reply-To: References: <52406460.3060905@uci.edu> <52406C01.5090804@uci.edu> <52407435.6080302@uci.edu> <4F04E2EB-90FC-4EAD-B33A-9510AC8593F7@lbl.gov> Message-ID: <5240C290.2070608@uci.edu> Thank you very much! I will make some tests and let you know how it goes. Michele On 09/23/2013 01:24 PM, Mark F. Adams wrote: > Ah yes, I agree, geometric MG with Galerkin is worth trying. > > > On Sep 23, 2013, at 3:36 PM, Dave May > wrote: > >> Hey Michele, >> >> I think Galerkin MG, even with a jump in the coefficient of 10^3, can >> be made more robust and faster for your problem. >> I'd start by trying different smoothers. Try cheby/(bjacobi or asm), >> or gmres/(bjacobi or asm). When using cheby, make sure you ask for >> the spectrum estimate. >> Definitely do more than one application of KSP on each level. >> >> I don't think you should switch to AMG without first trying more >> robust smoothers and profiling how much time is spent in the coarse >> grid solve. >> I solve similar problems (geometry and size of the jump) in >> connection with variable visocsity stokes problems on structured >> grids, and I'm yet to see a case with AMG is more robust and or >> faster than Galerkin MG. >> >> >> Cheers, >> Dave >> >> >> >> On 23 September 2013 20:52, Mark F. Adams > > wrote: >> >> >> >> >> > The simulation does start with both phases and the geometry is >> supposed to become more complex as the simulation progresses. >> > But so far the run is stopped before there are significant >> changes in the shape of the droplet. >> >> Humm, not sure why you are seeing degradation then. I imagine >> the initial geometry is not grid aligned so I'm not sure why you >> are seeing degradation in convergence rate. >> >> > I can give a shot to AMG: which options would you suggest to use. >> >> -pc_type gamg -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 >> >> > Also, how can I project out the constant from the rhs? Thanks a >> lot! >> >> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPSetNullSpace.html#KSPSetNullSpace >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Sep 23 17:30:30 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 23 Sep 2013 23:30:30 +0100 Subject: [petsc-users] Writing VTK output for vectors derived from DM Global vector In-Reply-To: References: Message-ID: <87li2n2mt5.fsf@mcs.anl.gov> Garnet Vaz writes: > Hi, > > I would like to write out my solution to a VTK file where I use unstructured > grids in DMPlex. > > The solution corresponds to harmonic frequencies (~ 5-10 distinct) which > are > currently saved into a Vec* object. These vectors were created by > duplicating the > global vector for the DM object. (I know that they correspond to the right > cell.) > > Is it possible to have one file which would contain the mesh from the DM > object and all the Vec* objects? > Would I have to clone the original DM and use > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerVTKAddField.html > to include all the vectors? This is a private function. > In the above function does > PetscViewerVTKWriteFunction > just need to be > > myWriteFunc(vec,viewer) > { > VecView(vec,viewer) > } > if I do not need to make any changes to the vector? This should work, but you may want to use PetscObjectSetName() so that the names of the different fields are sensible. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From hariseldon99 at gmail.com Tue Sep 24 00:36:34 2013 From: hariseldon99 at gmail.com (Analabha Roy) Date: Tue, 24 Sep 2013 11:06:34 +0530 Subject: [petsc-users] Building the same petsc matrix with different numprocs gives different results! Message-ID: Hi all, Compiling and running this codethat builds a petsc matrix gives different results when run with different number of processors. The code is large and complex, so I have created a smaller program with the same loop structure here. Compile it and run it with "mpirun -np $N ./test -draw_pause -1" gives different results for different values of N even though it's not supposed to. Here is a sample output for N=1,2,3,4 from left to right. Can anyone guide me as to what I'm doing wrong? Are any of the petssc routines used not parallelizable? Thanks in advance, Regards. -- --- *Analabha Roy* C.S.I.R Senior Research Associate Saha Institute of Nuclear Physics Section 1, Block AF Bidhannagar, Calcutta 700064 India *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com *Webpage*: http://www.ph.utexas.edu/~daneel/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Sep 24 03:12:06 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 24 Sep 2013 09:12:06 +0100 Subject: [petsc-users] Building the same petsc matrix with different numprocs gives different results! In-Reply-To: References: Message-ID: <87zjr2zlih.fsf@mcs.anl.gov> Analabha Roy writes: > Hi all, > > > Compiling and running this > codethat > builds a petsc matrix gives different results when run with different > number of processors. Uh, if you call rand() on different processors, why would you expect it to give the same results? for (sitestride1 = Istart; sitestride1 < Iend; sitestride1++) { for (sitestride2 = 0; sitestride2 < matsize; sitestride2++) { for (alpha = 0; alpha < dim; alpha++) { for (mu = 0; mu < dim; mu++) for (lambda = 0; lambda < dim; lambda++) { vecval = rand () / rand (); } VecSetValue (BDB_AA, alpha, vecval, INSERT_VALUES); } VecAssemblyBegin (BDB_AA); VecAssemblyEnd (BDB_AA); VecSum (BDB_AA, &element); colvalues[sitestride2] = element; } //Insert the array of colvalues to the sitestride1^th row of H MatSetValues (AVG_BDIBJ, 1, &sitestride1, matsize, idx, colvalues, INSERT_VALUES); } > The code is large and complex, so I have created a smaller program > with the same > loop structure here. > > Compile it and run it with "mpirun -np $N ./test -draw_pause -1" gives > different results for different values of N even though it's not supposed > to. What do you expect to see? > Here is a sample output for N=1,2,3,4 > from left to right. > > Can anyone guide me as to what I'm doing wrong? Are any of the petssc > routines used not parallelizable? > > Thanks in advance, > > Regards. > > -- > --- > *Analabha Roy* > C.S.I.R Senior Research > Associate > Saha Institute of Nuclear Physics > Section 1, Block AF > Bidhannagar, Calcutta 700064 > India > *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com > *Webpage*: http://www.ph.utexas.edu/~daneel/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From garnet.vaz at gmail.com Tue Sep 24 07:54:25 2013 From: garnet.vaz at gmail.com (Garnet Vaz) Date: Tue, 24 Sep 2013 05:54:25 -0700 Subject: [petsc-users] Writing VTK output for vectors derived from DM Global vector In-Reply-To: <87li2n2mt5.fsf@mcs.anl.gov> References: <87li2n2mt5.fsf@mcs.anl.gov> Message-ID: Hi, Thanks Jed. I will set the field names before I write them. I need to create a clone and then change the default section to have dof to 1 before I write them. I am not able to use clone with DMClone(originaldm,&newdm); The error I get when compiling is "'DMClone' was not declared in this scope". I do have #include which includes the petscdm.h header. Other than clone I am able to use the other functions. Thanks. - Garnet On Mon, Sep 23, 2013 at 3:30 PM, Jed Brown wrote: > Garnet Vaz writes: > > > Hi, > > > > I would like to write out my solution to a VTK file where I use > unstructured > > grids in DMPlex. > > > > The solution corresponds to harmonic frequencies (~ 5-10 distinct) which > > are > > currently saved into a Vec* object. These vectors were created by > > duplicating the > > global vector for the DM object. (I know that they correspond to the > right > > cell.) > > > > Is it possible to have one file which would contain the mesh from the DM > > object and all the Vec* objects? > > Would I have to clone the original DM and use > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerVTKAddField.html > > to include all the vectors? > > This is a private function. > > > In the above function does > > PetscViewerVTKWriteFunction > > just need to be > > > > myWriteFunc(vec,viewer) > > { > > VecView(vec,viewer) > > } > > if I do not need to make any changes to the vector? > > This should work, but you may want to use PetscObjectSetName() so that > the names of the different fields are sensible. > -- Regards, Garnet -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Sep 24 08:08:26 2013 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 24 Sep 2013 06:08:26 -0700 Subject: [petsc-users] Writing VTK output for vectors derived from DM Global vector In-Reply-To: References: <87li2n2mt5.fsf@mcs.anl.gov> Message-ID: On Tue, Sep 24, 2013 at 5:54 AM, Garnet Vaz wrote: > Hi, > > Thanks Jed. I will set the field names before I write them. > I need to create a clone and then change the default section > to have dof to 1 before I write them. I am not able to use clone with > > DMClone(originaldm,&newdm); > > The error I get when compiling is > "'DMClone' was not declared in this scope". > > I do have #include which includes the > petscdm.h header. Other than clone I am able to use the > other functions. > Have you updated? I think I forgot the declaration initially. Matt > Thanks. > - > Garnet > > > On Mon, Sep 23, 2013 at 3:30 PM, Jed Brown wrote: > >> Garnet Vaz writes: >> >> > Hi, >> > >> > I would like to write out my solution to a VTK file where I use >> unstructured >> > grids in DMPlex. >> > >> > The solution corresponds to harmonic frequencies (~ 5-10 distinct) >> which >> > are >> > currently saved into a Vec* object. These vectors were created by >> > duplicating the >> > global vector for the DM object. (I know that they correspond to the >> right >> > cell.) >> > >> > Is it possible to have one file which would contain the mesh from the DM >> > object and all the Vec* objects? >> > Would I have to clone the original DM and use >> > >> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerVTKAddField.html >> > to include all the vectors? >> >> This is a private function. >> >> > In the above function does >> > PetscViewerVTKWriteFunction >> > just need to be >> > >> > myWriteFunc(vec,viewer) >> > { >> > VecView(vec,viewer) >> > } >> > if I do not need to make any changes to the vector? >> >> This should work, but you may want to use PetscObjectSetName() so that >> the names of the different fields are sensible. >> > > > > -- > Regards, > Garnet > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From garnet.vaz at gmail.com Tue Sep 24 08:16:40 2013 From: garnet.vaz at gmail.com (Garnet Vaz) Date: Tue, 24 Sep 2013 06:16:40 -0700 Subject: [petsc-users] Writing VTK output for vectors derived from DM Global vector In-Reply-To: References: <87li2n2mt5.fsf@mcs.anl.gov> Message-ID: Hi Matt, I did pull yesterday. My configure.log file has Defined "VERSION_GIT" to ""b4e904d1358c92c43842a940440f995d2117e40b"" Defined "VERSION_DATE_GIT" to ""2013-09-23 12:36:08 +0100"" Is the change added after this? - Garnet On Tue, Sep 24, 2013 at 6:08 AM, Matthew Knepley wrote: > On Tue, Sep 24, 2013 at 5:54 AM, Garnet Vaz wrote: > >> Hi, >> >> Thanks Jed. I will set the field names before I write them. >> I need to create a clone and then change the default section >> to have dof to 1 before I write them. I am not able to use clone with >> >> DMClone(originaldm,&newdm); >> >> The error I get when compiling is >> "'DMClone' was not declared in this scope". >> >> I do have #include which includes the >> petscdm.h header. Other than clone I am able to use the >> other functions. >> > > Have you updated? I think I forgot the declaration initially. > > Matt > > >> Thanks. >> - >> Garnet >> >> >> On Mon, Sep 23, 2013 at 3:30 PM, Jed Brown wrote: >> >>> Garnet Vaz writes: >>> >>> > Hi, >>> > >>> > I would like to write out my solution to a VTK file where I use >>> unstructured >>> > grids in DMPlex. >>> > >>> > The solution corresponds to harmonic frequencies (~ 5-10 distinct) >>> which >>> > are >>> > currently saved into a Vec* object. These vectors were created by >>> > duplicating the >>> > global vector for the DM object. (I know that they correspond to the >>> right >>> > cell.) >>> > >>> > Is it possible to have one file which would contain the mesh from the >>> DM >>> > object and all the Vec* objects? >>> > Would I have to clone the original DM and use >>> > >>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerVTKAddField.html >>> > to include all the vectors? >>> >>> This is a private function. >>> >>> > In the above function does >>> > PetscViewerVTKWriteFunction >>> > just need to be >>> > >>> > myWriteFunc(vec,viewer) >>> > { >>> > VecView(vec,viewer) >>> > } >>> > if I do not need to make any changes to the vector? >>> >>> This should work, but you may want to use PetscObjectSetName() so that >>> the names of the different fields are sensible. >>> >> >> >> >> -- >> Regards, >> Garnet >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Regards, Garnet -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Sep 24 08:26:06 2013 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 24 Sep 2013 06:26:06 -0700 Subject: [petsc-users] Writing VTK output for vectors derived from DM Global vector In-Reply-To: References: <87li2n2mt5.fsf@mcs.anl.gov> Message-ID: On Tue, Sep 24, 2013 at 6:16 AM, Garnet Vaz wrote: > Hi Matt, > > I did pull yesterday. My configure.log file has > Defined "VERSION_GIT" to > ""b4e904d1358c92c43842a940440f995d2117e40b"" > Defined "VERSION_DATE_GIT" to ""2013-09-23 12:36:08 +0100"" > > Is the change added after this? > I see it here: https://bitbucket.org/petsc/petsc/src/33dd83b134909f5b4acb842818b1e0de2db2f4eb/include/petscdm.h?at=master#cl-35 and it has been there since this commit in July: https://bitbucket.org/petsc/petsc/commits/38221697d93bc7c9a6149445938a5de22e85bc83 Matt > - > Garnet > > > > On Tue, Sep 24, 2013 at 6:08 AM, Matthew Knepley wrote: > >> On Tue, Sep 24, 2013 at 5:54 AM, Garnet Vaz wrote: >> >>> Hi, >>> >>> Thanks Jed. I will set the field names before I write them. >>> I need to create a clone and then change the default section >>> to have dof to 1 before I write them. I am not able to use clone with >>> >>> DMClone(originaldm,&newdm); >>> >>> The error I get when compiling is >>> "'DMClone' was not declared in this scope". >>> >>> I do have #include which includes the >>> petscdm.h header. Other than clone I am able to use the >>> other functions. >>> >> >> Have you updated? I think I forgot the declaration initially. >> >> Matt >> >> >>> Thanks. >>> - >>> Garnet >>> >>> >>> On Mon, Sep 23, 2013 at 3:30 PM, Jed Brown wrote: >>> >>>> Garnet Vaz writes: >>>> >>>> > Hi, >>>> > >>>> > I would like to write out my solution to a VTK file where I use >>>> unstructured >>>> > grids in DMPlex. >>>> > >>>> > The solution corresponds to harmonic frequencies (~ 5-10 distinct) >>>> which >>>> > are >>>> > currently saved into a Vec* object. These vectors were created by >>>> > duplicating the >>>> > global vector for the DM object. (I know that they correspond to the >>>> right >>>> > cell.) >>>> > >>>> > Is it possible to have one file which would contain the mesh from the >>>> DM >>>> > object and all the Vec* objects? >>>> > Would I have to clone the original DM and use >>>> > >>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerVTKAddField.html >>>> > to include all the vectors? >>>> >>>> This is a private function. >>>> >>>> > In the above function does >>>> > PetscViewerVTKWriteFunction >>>> > just need to be >>>> > >>>> > myWriteFunc(vec,viewer) >>>> > { >>>> > VecView(vec,viewer) >>>> > } >>>> > if I do not need to make any changes to the vector? >>>> >>>> This should work, but you may want to use PetscObjectSetName() so that >>>> the names of the different fields are sensible. >>>> >>> >>> >>> >>> -- >>> Regards, >>> Garnet >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > Regards, > Garnet > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From garnet.vaz at gmail.com Tue Sep 24 08:44:59 2013 From: garnet.vaz at gmail.com (Garnet Vaz) Date: Tue, 24 Sep 2013 06:44:59 -0700 Subject: [petsc-users] Writing VTK output for vectors derived from DM Global vector In-Reply-To: References: <87li2n2mt5.fsf@mcs.anl.gov> Message-ID: Hi Matt, Yes. Thanks. I got it to work. The shell on emacs had not updated $PETSC_DIR and was using 3.4.2. - Garnet On Tue, Sep 24, 2013 at 6:26 AM, Matthew Knepley wrote: > On Tue, Sep 24, 2013 at 6:16 AM, Garnet Vaz wrote: > >> Hi Matt, >> >> I did pull yesterday. My configure.log file has >> Defined "VERSION_GIT" to >> ""b4e904d1358c92c43842a940440f995d2117e40b"" >> Defined "VERSION_DATE_GIT" to ""2013-09-23 12:36:08 +0100"" >> >> Is the change added after this? >> > > I see it here: > > > https://bitbucket.org/petsc/petsc/src/33dd83b134909f5b4acb842818b1e0de2db2f4eb/include/petscdm.h?at=master#cl-35 > > and it has been there since this commit in July: > > > https://bitbucket.org/petsc/petsc/commits/38221697d93bc7c9a6149445938a5de22e85bc83 > > Matt > > >> - >> Garnet >> >> >> >> On Tue, Sep 24, 2013 at 6:08 AM, Matthew Knepley wrote: >> >>> On Tue, Sep 24, 2013 at 5:54 AM, Garnet Vaz wrote: >>> >>>> Hi, >>>> >>>> Thanks Jed. I will set the field names before I write them. >>>> I need to create a clone and then change the default section >>>> to have dof to 1 before I write them. I am not able to use clone with >>>> >>>> DMClone(originaldm,&newdm); >>>> >>>> The error I get when compiling is >>>> "'DMClone' was not declared in this scope". >>>> >>>> I do have #include which includes the >>>> petscdm.h header. Other than clone I am able to use the >>>> other functions. >>>> >>> >>> Have you updated? I think I forgot the declaration initially. >>> >>> Matt >>> >>> >>>> Thanks. >>>> - >>>> Garnet >>>> >>>> >>>> On Mon, Sep 23, 2013 at 3:30 PM, Jed Brown wrote: >>>> >>>>> Garnet Vaz writes: >>>>> >>>>> > Hi, >>>>> > >>>>> > I would like to write out my solution to a VTK file where I use >>>>> unstructured >>>>> > grids in DMPlex. >>>>> > >>>>> > The solution corresponds to harmonic frequencies (~ 5-10 distinct) >>>>> which >>>>> > are >>>>> > currently saved into a Vec* object. These vectors were created by >>>>> > duplicating the >>>>> > global vector for the DM object. (I know that they correspond to the >>>>> right >>>>> > cell.) >>>>> > >>>>> > Is it possible to have one file which would contain the mesh from >>>>> the DM >>>>> > object and all the Vec* objects? >>>>> > Would I have to clone the original DM and use >>>>> > >>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerVTKAddField.html >>>>> > to include all the vectors? >>>>> >>>>> This is a private function. >>>>> >>>>> > In the above function does >>>>> > PetscViewerVTKWriteFunction >>>>> > just need to be >>>>> > >>>>> > myWriteFunc(vec,viewer) >>>>> > { >>>>> > VecView(vec,viewer) >>>>> > } >>>>> > if I do not need to make any changes to the vector? >>>>> >>>>> This should work, but you may want to use PetscObjectSetName() so that >>>>> the names of the different fields are sensible. >>>>> >>>> >>>> >>>> >>>> -- >>>> Regards, >>>> Garnet >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> >> >> -- >> Regards, >> Garnet >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Regards, Garnet -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominik at itis.ethz.ch Tue Sep 24 09:50:39 2013 From: dominik at itis.ethz.ch (Dominik Szczerba) Date: Tue, 24 Sep 2013 16:50:39 +0200 Subject: [petsc-users] VecCreateGhost Message-ID: As per http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecCreateGhost.html is it legal to pass NULL for PetscInt ghosts[] argument in case no ghosts are present (running in sequential mode)? Thanks for any clarifications. Dominik From knepley at gmail.com Tue Sep 24 09:53:21 2013 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 24 Sep 2013 07:53:21 -0700 Subject: [petsc-users] VecCreateGhost In-Reply-To: References: Message-ID: On Tue, Sep 24, 2013 at 7:50 AM, Dominik Szczerba wrote: > As per > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecCreateGhost.html > > is it legal to pass NULL for PetscInt ghosts[] argument in case no > ghosts are present (running in sequential mode)? > Yes Matt > Thanks for any clarifications. > > Dominik > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hariseldon99 at gmail.com Tue Sep 24 10:08:16 2013 From: hariseldon99 at gmail.com (Analabha Roy) Date: Tue, 24 Sep 2013 20:38:16 +0530 Subject: [petsc-users] Fwd: Building the same petsc matrix with different numprocs gives different results! In-Reply-To: References: <87zjr2zlih.fsf@mcs.anl.gov> Message-ID: On Tue, Sep 24, 2013 at 1:42 PM, Jed Brown wrote: > Analabha Roy writes: > > > Hi all, > > > > > > Compiling and running this > > code< > https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c>that > > builds a petsc matrix gives different results when run with different > > number of processors. > > Thanks for the reply. > Uh, if you call rand() on different processors, why would you expect it > to give the same results? > > Right, I get that. The rand() was a placeholder. This original much larger codereplicates the same loop structure and runs the same Petsc subroutines, but running it by mpirun -np $N ./eth -lattice_size 5 -vector_size 1 -repulsion 0.0 -draw_out -draw_pause -1 with N=1,2,3,4 gives different results for the matrix dumped out by lines 514-519. The matrix itself is evaluated in parallel, created in lines 263-275 and evaluated in lines 294-356 (you can click on the line numbers above to navigate directly to them) Here is a sample of the output of lines 514-519 for N=1,2,3,4 procs left to right. Thty're different for different procs. They should be the same, since none of my input parameters are numprocs dependent, and I don't explicitly use the size or rank anywhere in the code. > for (sitestride1 = Istart; sitestride1 < Iend; sitestride1++) > { > for (sitestride2 = 0; sitestride2 < matsize; sitestride2++) > { > for (alpha = 0; alpha < dim; alpha++) > { > for (mu = 0; mu < dim; mu++) > for (lambda = 0; lambda < dim; lambda++) > { > vecval = rand () / rand (); > } > > VecSetValue (BDB_AA, alpha, vecval, INSERT_VALUES); > > } > VecAssemblyBegin (BDB_AA); > VecAssemblyEnd (BDB_AA); > VecSum (BDB_AA, &element); > colvalues[sitestride2] = element; > > } > //Insert the array of colvalues to the sitestride1^th row of H > MatSetValues (AVG_BDIBJ, 1, &sitestride1, matsize, idx, colvalues, > INSERT_VALUES); > > } > > > The code is large and complex, so I have created a smaller program > > with the same > > loop structure here. > > > > Compile it and run it with "mpirun -np $N ./test -draw_pause -1" gives > > different results for different values of N even though it's not supposed > > to. > > What do you expect to see? > > > Here is a sample output for > N=1,2,3,4 > > from left to right. > > > > Can anyone guide me as to what I'm doing wrong? Are any of the petssc > > routines used not parallelizable? > > > > Thanks in advance, > > > > Regards. > > > > -- > > --- > > *Analabha Roy* > > C.S.I.R Senior Research > > Associate > > Saha Institute of Nuclear Physics > > Section 1, Block AF > > Bidhannagar, Calcutta 700064 > > India > > *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com > > *Webpage*: http://www.ph.utexas.edu/~daneel/ > -- --- *Analabha Roy* C.S.I.R Senior Research Associate Saha Institute of Nuclear Physics Section 1, Block AF Bidhannagar, Calcutta 700064 India *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com *Webpage*: http://www.ph.utexas.edu/~daneel/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Sep 24 10:11:21 2013 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 24 Sep 2013 08:11:21 -0700 Subject: [petsc-users] Fwd: Building the same petsc matrix with different numprocs gives different results! In-Reply-To: References: <87zjr2zlih.fsf@mcs.anl.gov> Message-ID: On Tue, Sep 24, 2013 at 8:08 AM, Analabha Roy wrote: > > On Tue, Sep 24, 2013 at 1:42 PM, Jed Brown wrote: > >> Analabha Roy writes: >> >> > Hi all, >> > >> > >> > Compiling and running this >> > code< >> https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c >> >that >> > builds a petsc matrix gives different results when run with different >> > number of processors. >> >> > Thanks for the reply. > > >> Uh, if you call rand() on different processors, why would you expect it >> to give the same results? >> >> Right, I get that. The rand() was a placeholder. > > This original much larger codereplicates the same loop structure and runs the same Petsc subroutines, but > running it by > > mpirun -np $N ./eth -lattice_size 5 -vector_size 1 -repulsion 0.0 > -draw_out -draw_pause -1 > > with N=1,2,3,4 gives different results for the matrix dumped out by lines > 514-519. > The matrix itself is evaluated in parallel, created in lines 263-275 > and > evaluated in lines 294-356 > > (you can click on the line numbers above to navigate directly to them) > > Here is a sample of the output of > lines 514-519 for N=1,2,3,4 procs left to right. > > Thty're different for different procs. They should be the same, since none > of my input parameters are numprocs dependent, and I don't explicitly use > the size or rank anywhere in the code. > You are likely not dividing the rows you loop over so you are redundantly computing. Matt > > >> for (sitestride1 = Istart; sitestride1 < Iend; sitestride1++) >> { >> for (sitestride2 = 0; sitestride2 < matsize; sitestride2++) >> { >> for (alpha = 0; alpha < dim; alpha++) >> { >> for (mu = 0; mu < dim; mu++) >> for (lambda = 0; lambda < dim; lambda++) >> { >> vecval = rand () / rand (); >> } >> >> VecSetValue (BDB_AA, alpha, vecval, INSERT_VALUES); >> >> } >> VecAssemblyBegin (BDB_AA); >> VecAssemblyEnd (BDB_AA); >> VecSum (BDB_AA, &element); >> colvalues[sitestride2] = element; >> >> } >> //Insert the array of colvalues to the sitestride1^th row of H >> MatSetValues (AVG_BDIBJ, 1, &sitestride1, matsize, idx, colvalues, >> INSERT_VALUES); >> >> } >> >> > The code is large and complex, so I have created a smaller program >> > with the same >> > loop structure here. >> > >> > Compile it and run it with "mpirun -np $N ./test -draw_pause -1" gives >> > different results for different values of N even though it's not >> supposed >> > to. >> >> What do you expect to see? >> >> > Here is a sample output for >> N=1,2,3,4 >> > from left to right. >> > >> > Can anyone guide me as to what I'm doing wrong? Are any of the petssc >> > routines used not parallelizable? >> > >> > Thanks in advance, >> > >> > Regards. >> > >> > -- >> > --- >> > *Analabha Roy* >> > C.S.I.R Senior Research >> > Associate >> > Saha Institute of Nuclear Physics >> > Section 1, Block AF >> > Bidhannagar, Calcutta 700064 >> > India >> > *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >> > *Webpage*: http://www.ph.utexas.edu/~daneel/ >> > > > > -- > --- > *Analabha Roy* > C.S.I.R Senior Research Associate > Saha Institute of Nuclear Physics > Section 1, Block AF > Bidhannagar, Calcutta 700064 > India > *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com > *Webpage*: http://www.ph.utexas.edu/~daneel/ > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hariseldon99 at gmail.com Tue Sep 24 10:35:38 2013 From: hariseldon99 at gmail.com (Analabha Roy) Date: Tue, 24 Sep 2013 21:05:38 +0530 Subject: [petsc-users] Fwd: Building the same petsc matrix with different numprocs gives different results! In-Reply-To: References: <87zjr2zlih.fsf@mcs.anl.gov> Message-ID: On Tue, Sep 24, 2013 at 8:41 PM, Matthew Knepley wrote: > On Tue, Sep 24, 2013 at 8:08 AM, Analabha Roy wrote: > >> >> On Tue, Sep 24, 2013 at 1:42 PM, Jed Brown wrote: >> >>> Analabha Roy writes: >>> >>> > Hi all, >>> > >>> > >>> > Compiling and running this >>> > code< >>> https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c >>> >that >>> > builds a petsc matrix gives different results when run with different >>> > number of processors. >>> >>> >> Thanks for the reply. >> >> >>> Uh, if you call rand() on different processors, why would you expect it >>> to give the same results? >>> >>> Right, I get that. The rand() was a placeholder. >> >> This original much larger codereplicates the same loop structure and runs the same Petsc subroutines, but >> running it by >> >> mpirun -np $N ./eth -lattice_size 5 -vector_size 1 -repulsion 0.0 >> -draw_out -draw_pause -1 >> >> with N=1,2,3,4 gives different results for the matrix dumped out by lines >> 514-519. >> The matrix itself is evaluated in parallel, created in lines 263-275 >> and >> evaluated in lines 294-356 >> >> (you can click on the line numbers above to navigate directly to them) >> >> Here is a sample of the output of >> lines 514-519 for N=1,2,3,4 procs left to right. >> >> Thty're different for different procs. They should be the same, since >> none of my input parameters are numprocs dependent, and I don't explicitly >> use the size or rank anywhere in the code. >> > > You are likely not dividing the rows you loop over so you are redundantly > computing. > Thanks for the reply. Line 274gets the local row indices of Petsc Matrix AVG_BDIBJ Line 295 iterates over the local rows and the lines below get the column elements. For each row, the column elements are assigned by the lines up to Line 344and stored locally in colvalues[]. Dunno if the details are relevant. Line 347inserts the sitestride1^th row into the matrix Line 353+does the mat assembly Then, after a lot of currently irrelevant code, Line 514+dumps the mat plot to graphics Different numprocs give different matrices. Can somebody suggest what I did wrong (or didn't do)? > > Matt > > >> >> >>> for (sitestride1 = Istart; sitestride1 < Iend; sitestride1++) >>> { >>> for (sitestride2 = 0; sitestride2 < matsize; sitestride2++) >>> { >>> for (alpha = 0; alpha < dim; alpha++) >>> { >>> for (mu = 0; mu < dim; mu++) >>> for (lambda = 0; lambda < dim; lambda++) >>> { >>> vecval = rand () / rand (); >>> } >>> >>> VecSetValue (BDB_AA, alpha, vecval, INSERT_VALUES); >>> >>> } >>> VecAssemblyBegin (BDB_AA); >>> VecAssemblyEnd (BDB_AA); >>> VecSum (BDB_AA, &element); >>> colvalues[sitestride2] = element; >>> >>> } >>> //Insert the array of colvalues to the sitestride1^th row of H >>> MatSetValues (AVG_BDIBJ, 1, &sitestride1, matsize, idx, colvalues, >>> INSERT_VALUES); >>> >>> } >>> >>> > The code is large and complex, so I have created a smaller program >>> > with the same >>> > loop structure here. >>> > >>> > Compile it and run it with "mpirun -np $N ./test -draw_pause -1" gives >>> > different results for different values of N even though it's not >>> supposed >>> > to. >>> >>> What do you expect to see? >>> >>> > Here is a sample output for >>> N=1,2,3,4 >>> > from left to right. >>> > >>> > Can anyone guide me as to what I'm doing wrong? Are any of the petssc >>> > routines used not parallelizable? >>> > >>> > Thanks in advance, >>> > >>> > Regards. >>> > >>> > -- >>> > --- >>> > *Analabha Roy* >>> > C.S.I.R Senior Research >>> > Associate >>> > Saha Institute of Nuclear Physics >>> > Section 1, Block AF >>> > Bidhannagar, Calcutta 700064 >>> > India >>> > *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >>> > *Webpage*: http://www.ph.utexas.edu/~daneel/ >>> >> >> >> >> -- >> --- >> *Analabha Roy* >> C.S.I.R Senior Research Associate >> Saha Institute of Nuclear Physics >> Section 1, Block AF >> Bidhannagar, Calcutta 700064 >> India >> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >> *Webpage*: http://www.ph.utexas.edu/~daneel/ >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- --- *Analabha Roy* C.S.I.R Senior Research Associate Saha Institute of Nuclear Physics Section 1, Block AF Bidhannagar, Calcutta 700064 India *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com *Webpage*: http://www.ph.utexas.edu/~daneel/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Sep 24 11:03:13 2013 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 24 Sep 2013 09:03:13 -0700 Subject: [petsc-users] Fwd: Building the same petsc matrix with different numprocs gives different results! In-Reply-To: References: <87zjr2zlih.fsf@mcs.anl.gov> Message-ID: On Tue, Sep 24, 2013 at 8:35 AM, Analabha Roy wrote: > > > > On Tue, Sep 24, 2013 at 8:41 PM, Matthew Knepley wrote: > >> On Tue, Sep 24, 2013 at 8:08 AM, Analabha Roy wrote: >> >>> >>> On Tue, Sep 24, 2013 at 1:42 PM, Jed Brown wrote: >>> >>>> Analabha Roy writes: >>>> >>>> > Hi all, >>>> > >>>> > >>>> > Compiling and running this >>>> > code< >>>> https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c >>>> >that >>>> > builds a petsc matrix gives different results when run with different >>>> > number of processors. >>>> >>>> >>> Thanks for the reply. >>> >>> >>>> Uh, if you call rand() on different processors, why would you expect >>>> it >>>> to give the same results? >>>> >>>> Right, I get that. The rand() was a placeholder. >>> >>> This original much larger codereplicates the same loop structure and runs the same Petsc subroutines, but >>> running it by >>> >>> mpirun -np $N ./eth -lattice_size 5 -vector_size 1 -repulsion 0.0 >>> -draw_out -draw_pause -1 >>> >>> with N=1,2,3,4 gives different results for the matrix dumped out by >>> lines 514-519. >>> The matrix itself is evaluated in parallel, created in lines 263-275 >>> and >>> evaluated in lines 294-356 >>> >>> (you can click on the line numbers above to navigate directly to them) >>> >>> Here is a sample of the output of >>> lines 514-519 for N=1,2,3,4 procs left to right. >>> >>> Thty're different for different procs. They should be the same, since >>> none of my input parameters are numprocs dependent, and I don't explicitly >>> use the size or rank anywhere in the code. >>> >> >> You are likely not dividing the rows you loop over so you are redundantly >> computing. >> > > Thanks for the reply. > > Line 274gets the local row indices of Petsc Matrix > AVG_BDIBJ > > Line 295 > iterates > over the local rows and the lines below get the column > elements. For each row, the column elements are assigned by the lines up > to Line 344and stored locally in colvalues[]. Dunno if the details are relevant. > > Line 347inserts the sitestride1^th row into the matrix > > Line 353+does the mat assembly > > Then, after a lot of currently irrelevant code, > > Line 514+dumps the mat plot to graphics > > > Different numprocs give different matrices. > > Can somebody suggest what I did wrong (or didn't do)? > Different values are being given to MatSetValues() for different numbers of processes. So 1) Reduce this to the smallest problem size possible 2) Print out all rows/cols/values for each call 3) Compare 2 procs to the serial case Matt > >> Matt >> >> >>> >>> >>>> for (sitestride1 = Istart; sitestride1 < Iend; sitestride1++) >>>> { >>>> for (sitestride2 = 0; sitestride2 < matsize; sitestride2++) >>>> { >>>> for (alpha = 0; alpha < dim; alpha++) >>>> { >>>> for (mu = 0; mu < dim; mu++) >>>> for (lambda = 0; lambda < dim; lambda++) >>>> { >>>> vecval = rand () / rand (); >>>> } >>>> >>>> VecSetValue (BDB_AA, alpha, vecval, INSERT_VALUES); >>>> >>>> } >>>> VecAssemblyBegin (BDB_AA); >>>> VecAssemblyEnd (BDB_AA); >>>> VecSum (BDB_AA, &element); >>>> colvalues[sitestride2] = element; >>>> >>>> } >>>> //Insert the array of colvalues to the sitestride1^th row of H >>>> MatSetValues (AVG_BDIBJ, 1, &sitestride1, matsize, idx, colvalues, >>>> INSERT_VALUES); >>>> >>>> } >>>> >>>> > The code is large and complex, so I have created a smaller program >>>> > with the same >>>> > loop structure here. >>>> > >>>> > Compile it and run it with "mpirun -np $N ./test -draw_pause -1" gives >>>> > different results for different values of N even though it's not >>>> supposed >>>> > to. >>>> >>>> What do you expect to see? >>>> >>>> > Here is a sample output for >>>> N=1,2,3,4 >>>> > from left to right. >>>> > >>>> > Can anyone guide me as to what I'm doing wrong? Are any of the petssc >>>> > routines used not parallelizable? >>>> > >>>> > Thanks in advance, >>>> > >>>> > Regards. >>>> > >>>> > -- >>>> > --- >>>> > *Analabha Roy* >>>> > C.S.I.R Senior Research >>>> > Associate >>>> > Saha Institute of Nuclear Physics >>>> > Section 1, Block AF >>>> > Bidhannagar, Calcutta 700064 >>>> > India >>>> > *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >>>> > *Webpage*: http://www.ph.utexas.edu/~daneel/ >>>> >>> >>> >>> >>> -- >>> --- >>> *Analabha Roy* >>> C.S.I.R Senior Research Associate >>> Saha Institute of Nuclear Physics >>> Section 1, Block AF >>> Bidhannagar, Calcutta 700064 >>> India >>> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >>> *Webpage*: http://www.ph.utexas.edu/~daneel/ >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > --- > *Analabha Roy* > C.S.I.R Senior Research Associate > Saha Institute of Nuclear Physics > Section 1, Block AF > Bidhannagar, Calcutta 700064 > India > *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com > *Webpage*: http://www.ph.utexas.edu/~daneel/ > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hariseldon99 at gmail.com Tue Sep 24 12:39:10 2013 From: hariseldon99 at gmail.com (Analabha Roy) Date: Tue, 24 Sep 2013 23:09:10 +0530 Subject: [petsc-users] Fwd: Building the same petsc matrix with different numprocs gives different results! In-Reply-To: References: <87zjr2zlih.fsf@mcs.anl.gov> Message-ID: Hi, On Tue, Sep 24, 2013 at 9:33 PM, Matthew Knepley wrote: > On Tue, Sep 24, 2013 at 8:35 AM, Analabha Roy wrote: > >> >> >> >> On Tue, Sep 24, 2013 at 8:41 PM, Matthew Knepley wrote: >> >>> On Tue, Sep 24, 2013 at 8:08 AM, Analabha Roy wrote: >>> >>>> >>>> On Tue, Sep 24, 2013 at 1:42 PM, Jed Brown wrote: >>>> >>>>> Analabha Roy writes: >>>>> >>>>> > Hi all, >>>>> > >>>>> > >>>>> > Compiling and running this >>>>> > code< >>>>> https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c >>>>> >that >>>>> > builds a petsc matrix gives different results when run with different >>>>> > number of processors. >>>>> >>>>> >>>> Thanks for the reply. >>>> >>>> >>>>> Uh, if you call rand() on different processors, why would you expect >>>>> it >>>>> to give the same results? >>>>> >>>>> Right, I get that. The rand() was a placeholder. >>>> >>>> This original much larger codereplicates the same loop structure and runs the same Petsc subroutines, but >>>> running it by >>>> >>>> mpirun -np $N ./eth -lattice_size 5 -vector_size 1 -repulsion 0.0 >>>> -draw_out -draw_pause -1 >>>> >>>> with N=1,2,3,4 gives different results for the matrix dumped out by >>>> lines 514-519. >>>> The matrix itself is evaluated in parallel, created in lines 263-275 >>>> and >>>> evaluated in lines 294-356 >>>> >>>> (you can click on the line numbers above to navigate directly to them) >>>> >>>> Here is a sample of the output of >>>> lines 514-519 for N=1,2,3,4 procs left to right. >>>> >>>> Thty're different for different procs. They should be the same, since >>>> none of my input parameters are numprocs dependent, and I don't explicitly >>>> use the size or rank anywhere in the code. >>>> >>> >>> You are likely not dividing the rows you loop over so you are >>> redundantly computing. >>> >> >> Thanks for the reply. >> >> Line 274gets the local row indices of Petsc Matrix >> AVG_BDIBJ >> >> Line 295 >> iterates >> over the local rows and the lines below get the column >> elements. For each row, the column elements are assigned by the lines up >> to Line 344and stored locally in colvalues[]. Dunno if the details are relevant. >> >> Line 347inserts the sitestride1^th row into the matrix >> >> Line 353+does the mat assembly >> >> Then, after a lot of currently irrelevant code, >> >> Line 514+dumps the mat plot to graphics >> >> >> Different numprocs give different matrices. >> >> Can somebody suggest what I did wrong (or didn't do)? >> > > Different values are being given to MatSetValues() for different numbers > of processes. So > > 1) Reduce this to the smallest problem size possible > > 2) Print out all rows/cols/values for each call > > 3) Compare 2 procs to the serial case > > Thanks for your excellent suggestion. I modified my codeto dump the matrix in binary Then I used this python script I hadto convert to ascii Here are the values of AVG_BDIBJ, a 9X9 matrix (the smallest possible problem size) run with the exact same input parameters with 1,2,3 and 4 procs As you can see, the 1 and 2 procs match up, but the 3 and 4 procs do not. Serious wierdness. > Matt > > >> >>> Matt >>> >>> >>>> >>>> >>>>> for (sitestride1 = Istart; sitestride1 < Iend; sitestride1++) >>>>> { >>>>> for (sitestride2 = 0; sitestride2 < matsize; sitestride2++) >>>>> { >>>>> for (alpha = 0; alpha < dim; alpha++) >>>>> { >>>>> for (mu = 0; mu < dim; mu++) >>>>> for (lambda = 0; lambda < dim; lambda++) >>>>> { >>>>> vecval = rand () / rand (); >>>>> } >>>>> >>>>> VecSetValue (BDB_AA, alpha, vecval, INSERT_VALUES); >>>>> >>>>> } >>>>> VecAssemblyBegin (BDB_AA); >>>>> VecAssemblyEnd (BDB_AA); >>>>> VecSum (BDB_AA, &element); >>>>> colvalues[sitestride2] = element; >>>>> >>>>> } >>>>> //Insert the array of colvalues to the sitestride1^th row of H >>>>> MatSetValues (AVG_BDIBJ, 1, &sitestride1, matsize, idx, >>>>> colvalues, >>>>> INSERT_VALUES); >>>>> >>>>> } >>>>> >>>>> > The code is large and complex, so I have created a smaller program >>>>> > with the same >>>>> > loop structure here. >>>>> > >>>>> > Compile it and run it with "mpirun -np $N ./test -draw_pause -1" >>>>> gives >>>>> > different results for different values of N even though it's not >>>>> supposed >>>>> > to. >>>>> >>>>> What do you expect to see? >>>>> >>>>> > Here is a sample output for >>>>> N=1,2,3,4 >>>>> > from left to right. >>>>> > >>>>> > Can anyone guide me as to what I'm doing wrong? Are any of the petssc >>>>> > routines used not parallelizable? >>>>> > >>>>> > Thanks in advance, >>>>> > >>>>> > Regards. >>>>> > >>>>> > -- >>>>> > --- >>>>> > *Analabha Roy* >>>>> > C.S.I.R Senior Research >>>>> > Associate >>>>> > Saha Institute of Nuclear Physics >>>>> > Section 1, Block AF >>>>> > Bidhannagar, Calcutta 700064 >>>>> > India >>>>> > *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >>>>> > *Webpage*: http://www.ph.utexas.edu/~daneel/ >>>>> >>>> >>>> >>>> >>>> -- >>>> --- >>>> *Analabha Roy* >>>> C.S.I.R Senior Research Associate >>>> Saha Institute of Nuclear Physics >>>> Section 1, Block AF >>>> Bidhannagar, Calcutta 700064 >>>> India >>>> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >>>> *Webpage*: http://www.ph.utexas.edu/~daneel/ >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> >> >> -- >> --- >> *Analabha Roy* >> C.S.I.R Senior Research Associate >> Saha Institute of Nuclear Physics >> Section 1, Block AF >> Bidhannagar, Calcutta 700064 >> India >> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >> *Webpage*: http://www.ph.utexas.edu/~daneel/ >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- --- *Analabha Roy* C.S.I.R Senior Research Associate Saha Institute of Nuclear Physics Section 1, Block AF Bidhannagar, Calcutta 700064 India *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com *Webpage*: http://www.ph.utexas.edu/~daneel/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Sep 24 12:42:02 2013 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 24 Sep 2013 10:42:02 -0700 Subject: [petsc-users] Fwd: Building the same petsc matrix with different numprocs gives different results! In-Reply-To: References: <87zjr2zlih.fsf@mcs.anl.gov> Message-ID: On Tue, Sep 24, 2013 at 10:39 AM, Analabha Roy wrote: > Hi, > > > > On Tue, Sep 24, 2013 at 9:33 PM, Matthew Knepley wrote: > >> On Tue, Sep 24, 2013 at 8:35 AM, Analabha Roy wrote: >> >>> >>> >>> >>> On Tue, Sep 24, 2013 at 8:41 PM, Matthew Knepley wrote: >>> >>>> On Tue, Sep 24, 2013 at 8:08 AM, Analabha Roy wrote: >>>> >>>>> >>>>> On Tue, Sep 24, 2013 at 1:42 PM, Jed Brown wrote: >>>>> >>>>>> Analabha Roy writes: >>>>>> >>>>>> > Hi all, >>>>>> > >>>>>> > >>>>>> > Compiling and running this >>>>>> > code< >>>>>> https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c >>>>>> >that >>>>>> > builds a petsc matrix gives different results when run with >>>>>> different >>>>>> > number of processors. >>>>>> >>>>>> >>>>> Thanks for the reply. >>>>> >>>>> >>>>>> Uh, if you call rand() on different processors, why would you >>>>>> expect it >>>>>> to give the same results? >>>>>> >>>>>> Right, I get that. The rand() was a placeholder. >>>>> >>>>> This original much larger codereplicates the same loop structure and runs the same Petsc subroutines, but >>>>> running it by >>>>> >>>>> mpirun -np $N ./eth -lattice_size 5 -vector_size 1 -repulsion 0.0 >>>>> -draw_out -draw_pause -1 >>>>> >>>>> with N=1,2,3,4 gives different results for the matrix dumped out by >>>>> lines 514-519. >>>>> The matrix itself is evaluated in parallel, created in lines 263-275 >>>>> and >>>>> evaluated in lines 294-356 >>>>> >>>>> (you can click on the line numbers above to navigate directly to them) >>>>> >>>>> Here is a sample of the output >>>>> of lines 514-519 for N=1,2,3,4 procs left to right. >>>>> >>>>> Thty're different for different procs. They should be the same, since >>>>> none of my input parameters are numprocs dependent, and I don't explicitly >>>>> use the size or rank anywhere in the code. >>>>> >>>> >>>> You are likely not dividing the rows you loop over so you are >>>> redundantly computing. >>>> >>> >>> Thanks for the reply. >>> >>> Line 274gets the local row indices of Petsc Matrix >>> AVG_BDIBJ >>> >>> Line 295 >>> iterates >>> over the local rows and the lines below get the column >>> elements. For each row, the column elements are assigned by the lines up >>> to Line 344and stored locally in colvalues[]. Dunno if the details are relevant. >>> >>> Line 347inserts the sitestride1^th row into the matrix >>> >>> Line 353+does the mat assembly >>> >>> Then, after a lot of currently irrelevant code, >>> >>> Line 514+dumps the mat plot to graphics >>> >>> >>> Different numprocs give different matrices. >>> >>> Can somebody suggest what I did wrong (or didn't do)? >>> >> >> Different values are being given to MatSetValues() for different numbers >> of processes. So >> >> 1) Reduce this to the smallest problem size possible >> >> 2) Print out all rows/cols/values for each call >> >> 3) Compare 2 procs to the serial case >> >> > > Thanks for your excellent suggestion. > > I modified my codeto dump the matrix in binary > > Then I used this python script I hadto convert to ascii > Do not print the matrix, print the data you are passing to MatSetValues(). MatSetValues() is not likely to be broken. Every PETSc code in the world calls this many times on every simulation. Matt > > Here are the values of AVG_BDIBJ, > a 9X9 matrix (the smallest possible problem size) run with the exact same > input parameters with 1,2,3 and 4 procs > > As you can see, the 1 and 2 procs match up, but the 3 and 4 procs do not. > > Serious wierdness. > > > >> Matt >> >> >>> >>>> Matt >>>> >>>> >>>>> >>>>> >>>>>> for (sitestride1 = Istart; sitestride1 < Iend; sitestride1++) >>>>>> { >>>>>> for (sitestride2 = 0; sitestride2 < matsize; sitestride2++) >>>>>> { >>>>>> for (alpha = 0; alpha < dim; alpha++) >>>>>> { >>>>>> for (mu = 0; mu < dim; mu++) >>>>>> for (lambda = 0; lambda < dim; lambda++) >>>>>> { >>>>>> vecval = rand () / rand (); >>>>>> } >>>>>> >>>>>> VecSetValue (BDB_AA, alpha, vecval, INSERT_VALUES); >>>>>> >>>>>> } >>>>>> VecAssemblyBegin (BDB_AA); >>>>>> VecAssemblyEnd (BDB_AA); >>>>>> VecSum (BDB_AA, &element); >>>>>> colvalues[sitestride2] = element; >>>>>> >>>>>> } >>>>>> //Insert the array of colvalues to the sitestride1^th row of H >>>>>> MatSetValues (AVG_BDIBJ, 1, &sitestride1, matsize, idx, >>>>>> colvalues, >>>>>> INSERT_VALUES); >>>>>> >>>>>> } >>>>>> >>>>>> > The code is large and complex, so I have created a smaller program >>>>>> > with the same >>>>>> > loop structure here. >>>>>> > >>>>>> > Compile it and run it with "mpirun -np $N ./test -draw_pause -1" >>>>>> gives >>>>>> > different results for different values of N even though it's not >>>>>> supposed >>>>>> > to. >>>>>> >>>>>> What do you expect to see? >>>>>> >>>>>> > Here is a sample output for >>>>>> N=1,2,3,4 >>>>>> > from left to right. >>>>>> > >>>>>> > Can anyone guide me as to what I'm doing wrong? Are any of the >>>>>> petssc >>>>>> > routines used not parallelizable? >>>>>> > >>>>>> > Thanks in advance, >>>>>> > >>>>>> > Regards. >>>>>> > >>>>>> > -- >>>>>> > --- >>>>>> > *Analabha Roy* >>>>>> > C.S.I.R Senior Research >>>>>> > Associate >>>>>> > Saha Institute of Nuclear Physics >>>>>> > Section 1, Block AF >>>>>> > Bidhannagar, Calcutta 700064 >>>>>> > India >>>>>> > *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >>>>>> > *Webpage*: http://www.ph.utexas.edu/~daneel/ >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> --- >>>>> *Analabha Roy* >>>>> C.S.I.R Senior Research Associate >>>>> Saha Institute of Nuclear Physics >>>>> Section 1, Block AF >>>>> Bidhannagar, Calcutta 700064 >>>>> India >>>>> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >>>>> *Webpage*: http://www.ph.utexas.edu/~daneel/ >>>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >>> >>> -- >>> --- >>> *Analabha Roy* >>> C.S.I.R Senior Research Associate >>> Saha Institute of Nuclear Physics >>> Section 1, Block AF >>> Bidhannagar, Calcutta 700064 >>> India >>> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >>> *Webpage*: http://www.ph.utexas.edu/~daneel/ >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > --- > *Analabha Roy* > C.S.I.R Senior Research Associate > Saha Institute of Nuclear Physics > Section 1, Block AF > Bidhannagar, Calcutta 700064 > India > *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com > *Webpage*: http://www.ph.utexas.edu/~daneel/ > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hariseldon99 at gmail.com Tue Sep 24 12:58:33 2013 From: hariseldon99 at gmail.com (Analabha Roy) Date: Tue, 24 Sep 2013 23:28:33 +0530 Subject: [petsc-users] Fwd: Building the same petsc matrix with different numprocs gives different results! In-Reply-To: References: <87zjr2zlih.fsf@mcs.anl.gov> Message-ID: Hi, Sorry for misunderstanding I modified my source thus so that the rows/cols/values for each call are printed before inserting into MatSetValues() Then ran it with 1,2 processors Here are the outputs Strange! Running it with 2 procs and only half the values show up!!!!! And even those do not match!!!! On Tue, Sep 24, 2013 at 11:12 PM, Matthew Knepley wrote: > On Tue, Sep 24, 2013 at 10:39 AM, Analabha Roy wrote: > >> Hi, >> >> >> >> On Tue, Sep 24, 2013 at 9:33 PM, Matthew Knepley wrote: >> >>> On Tue, Sep 24, 2013 at 8:35 AM, Analabha Roy wrote: >>> >>>> >>>> >>>> >>>> On Tue, Sep 24, 2013 at 8:41 PM, Matthew Knepley wrote: >>>> >>>>> On Tue, Sep 24, 2013 at 8:08 AM, Analabha Roy wrote: >>>>> >>>>>> >>>>>> On Tue, Sep 24, 2013 at 1:42 PM, Jed Brown wrote: >>>>>> >>>>>>> Analabha Roy writes: >>>>>>> >>>>>>> > Hi all, >>>>>>> > >>>>>>> > >>>>>>> > Compiling and running this >>>>>>> > code< >>>>>>> https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c >>>>>>> >that >>>>>>> > builds a petsc matrix gives different results when run with >>>>>>> different >>>>>>> > number of processors. >>>>>>> >>>>>>> >>>>>> Thanks for the reply. >>>>>> >>>>>> >>>>>>> Uh, if you call rand() on different processors, why would you >>>>>>> expect it >>>>>>> to give the same results? >>>>>>> >>>>>>> Right, I get that. The rand() was a placeholder. >>>>>> >>>>>> This original much larger codereplicates the same loop structure and runs the same Petsc subroutines, but >>>>>> running it by >>>>>> >>>>>> mpirun -np $N ./eth -lattice_size 5 -vector_size 1 -repulsion 0.0 >>>>>> -draw_out -draw_pause -1 >>>>>> >>>>>> with N=1,2,3,4 gives different results for the matrix dumped out by >>>>>> lines 514-519. >>>>>> The matrix itself is evaluated in parallel, created in lines 263-275 >>>>>> and >>>>>> evaluated in lines 294-356 >>>>>> >>>>>> (you can click on the line numbers above to navigate directly to them) >>>>>> >>>>>> Here is a sample of the output >>>>>> of lines 514-519 for N=1,2,3,4 procs left to right. >>>>>> >>>>>> Thty're different for different procs. They should be the same, since >>>>>> none of my input parameters are numprocs dependent, and I don't explicitly >>>>>> use the size or rank anywhere in the code. >>>>>> >>>>> >>>>> You are likely not dividing the rows you loop over so you are >>>>> redundantly computing. >>>>> >>>> >>>> Thanks for the reply. >>>> >>>> Line 274gets the local row indices of Petsc Matrix >>>> AVG_BDIBJ >>>> >>>> Line 295 >>>> iterates >>>> over the local rows and the lines below get the column >>>> elements. For each row, the column elements are assigned by the lines >>>> up to Line 344and stored locally in colvalues[]. Dunno if the details are relevant. >>>> >>>> Line 347inserts the sitestride1^th row into the matrix >>>> >>>> Line 353+does the mat assembly >>>> >>>> Then, after a lot of currently irrelevant code, >>>> >>>> Line 514+dumps the mat plot to graphics >>>> >>>> >>>> Different numprocs give different matrices. >>>> >>>> Can somebody suggest what I did wrong (or didn't do)? >>>> >>> >>> Different values are being given to MatSetValues() for different numbers >>> of processes. So >>> >>> 1) Reduce this to the smallest problem size possible >>> >>> 2) Print out all rows/cols/values for each call >>> >>> 3) Compare 2 procs to the serial case >>> >>> >> >> Thanks for your excellent suggestion. >> >> I modified my codeto dump the matrix in binary >> >> Then I used this python script I hadto convert to ascii >> > > Do not print the matrix, print the data you are passing to MatSetValues(). > > MatSetValues() is not likely to be broken. Every PETSc code in the world > calls this many times on every simulation. > > Matt > > >> >> Here are the values of AVG_BDIBJ, >> a 9X9 matrix (the smallest possible problem size) run with the exact same >> input parameters with 1,2,3 and 4 procs >> >> As you can see, the 1 and 2 procs match up, but the 3 and 4 procs do not. >> >> Serious wierdness. >> >> >> >>> Matt >>> >>> >>>> >>>>> Matt >>>>> >>>>> >>>>>> >>>>>> >>>>>>> for (sitestride1 = Istart; sitestride1 < Iend; sitestride1++) >>>>>>> { >>>>>>> for (sitestride2 = 0; sitestride2 < matsize; sitestride2++) >>>>>>> { >>>>>>> for (alpha = 0; alpha < dim; alpha++) >>>>>>> { >>>>>>> for (mu = 0; mu < dim; mu++) >>>>>>> for (lambda = 0; lambda < dim; lambda++) >>>>>>> { >>>>>>> vecval = rand () / rand (); >>>>>>> } >>>>>>> >>>>>>> VecSetValue (BDB_AA, alpha, vecval, INSERT_VALUES); >>>>>>> >>>>>>> } >>>>>>> VecAssemblyBegin (BDB_AA); >>>>>>> VecAssemblyEnd (BDB_AA); >>>>>>> VecSum (BDB_AA, &element); >>>>>>> colvalues[sitestride2] = element; >>>>>>> >>>>>>> } >>>>>>> //Insert the array of colvalues to the sitestride1^th row of H >>>>>>> MatSetValues (AVG_BDIBJ, 1, &sitestride1, matsize, idx, >>>>>>> colvalues, >>>>>>> INSERT_VALUES); >>>>>>> >>>>>>> } >>>>>>> >>>>>>> > The code is large and complex, so I have created a smaller program >>>>>>> > with the same >>>>>>> > loop structure here. >>>>>>> > >>>>>>> > Compile it and run it with "mpirun -np $N ./test -draw_pause -1" >>>>>>> gives >>>>>>> > different results for different values of N even though it's not >>>>>>> supposed >>>>>>> > to. >>>>>>> >>>>>>> What do you expect to see? >>>>>>> >>>>>>> > Here is a sample output for >>>>>>> N=1,2,3,4 >>>>>>> > from left to right. >>>>>>> > >>>>>>> > Can anyone guide me as to what I'm doing wrong? Are any of the >>>>>>> petssc >>>>>>> > routines used not parallelizable? >>>>>>> > >>>>>>> > Thanks in advance, >>>>>>> > >>>>>>> > Regards. >>>>>>> > >>>>>>> > -- >>>>>>> > --- >>>>>>> > *Analabha Roy* >>>>>>> > C.S.I.R Senior Research >>>>>>> > Associate >>>>>>> > Saha Institute of Nuclear Physics >>>>>>> > Section 1, Block AF >>>>>>> > Bidhannagar, Calcutta 700064 >>>>>>> > India >>>>>>> > *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >>>>>>> > *Webpage*: http://www.ph.utexas.edu/~daneel/ >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> --- >>>>>> *Analabha Roy* >>>>>> C.S.I.R Senior Research Associate >>>>>> Saha Institute of Nuclear Physics >>>>>> Section 1, Block AF >>>>>> Bidhannagar, Calcutta 700064 >>>>>> India >>>>>> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >>>>>> *Webpage*: http://www.ph.utexas.edu/~daneel/ >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>> >>>> >>>> >>>> -- >>>> --- >>>> *Analabha Roy* >>>> C.S.I.R Senior Research Associate >>>> Saha Institute of Nuclear Physics >>>> Section 1, Block AF >>>> Bidhannagar, Calcutta 700064 >>>> India >>>> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >>>> *Webpage*: http://www.ph.utexas.edu/~daneel/ >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> >> >> -- >> --- >> *Analabha Roy* >> C.S.I.R Senior Research Associate >> Saha Institute of Nuclear Physics >> Section 1, Block AF >> Bidhannagar, Calcutta 700064 >> India >> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >> *Webpage*: http://www.ph.utexas.edu/~daneel/ >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- --- *Analabha Roy* C.S.I.R Senior Research Associate Saha Institute of Nuclear Physics Section 1, Block AF Bidhannagar, Calcutta 700064 India *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com *Webpage*: http://www.ph.utexas.edu/~daneel/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Sep 24 13:05:04 2013 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 24 Sep 2013 11:05:04 -0700 Subject: [petsc-users] Fwd: Building the same petsc matrix with different numprocs gives different results! In-Reply-To: References: <87zjr2zlih.fsf@mcs.anl.gov> Message-ID: On Tue, Sep 24, 2013 at 10:58 AM, Analabha Roy wrote: > Hi, > > Sorry for misunderstanding > > I modified my source thus so that the > rows/cols/values for each call are printed before inserting into > MatSetValues() > > > Then ran it with 1,2 processors > > > Here are the outputs > > > Strange! Running it with 2 procs and only half the values show up!!!!! > PetscPrintf() only prints from rank 0. Use PETSC_COMM_SELF. Matt > And even those do not match!!!! > > > > > On Tue, Sep 24, 2013 at 11:12 PM, Matthew Knepley wrote: > >> On Tue, Sep 24, 2013 at 10:39 AM, Analabha Roy wrote: >> >>> Hi, >>> >>> >>> >>> On Tue, Sep 24, 2013 at 9:33 PM, Matthew Knepley wrote: >>> >>>> On Tue, Sep 24, 2013 at 8:35 AM, Analabha Roy wrote: >>>> >>>>> >>>>> >>>>> >>>>> On Tue, Sep 24, 2013 at 8:41 PM, Matthew Knepley wrote: >>>>> >>>>>> On Tue, Sep 24, 2013 at 8:08 AM, Analabha Roy >>>>> > wrote: >>>>>> >>>>>>> >>>>>>> On Tue, Sep 24, 2013 at 1:42 PM, Jed Brown wrote: >>>>>>> >>>>>>>> Analabha Roy writes: >>>>>>>> >>>>>>>> > Hi all, >>>>>>>> > >>>>>>>> > >>>>>>>> > Compiling and running this >>>>>>>> > code< >>>>>>>> https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c >>>>>>>> >that >>>>>>>> > builds a petsc matrix gives different results when run with >>>>>>>> different >>>>>>>> > number of processors. >>>>>>>> >>>>>>>> >>>>>>> Thanks for the reply. >>>>>>> >>>>>>> >>>>>>>> Uh, if you call rand() on different processors, why would you >>>>>>>> expect it >>>>>>>> to give the same results? >>>>>>>> >>>>>>>> Right, I get that. The rand() was a placeholder. >>>>>>> >>>>>>> This original much larger codereplicates the same loop structure and runs the same Petsc subroutines, but >>>>>>> running it by >>>>>>> >>>>>>> mpirun -np $N ./eth -lattice_size 5 -vector_size 1 -repulsion 0.0 >>>>>>> -draw_out -draw_pause -1 >>>>>>> >>>>>>> with N=1,2,3,4 gives different results for the matrix dumped out by >>>>>>> lines 514-519. >>>>>>> The matrix itself is evaluated in parallel, created in lines263-275 >>>>>>> and >>>>>>> evaluated in lines 294-356 >>>>>>> >>>>>>> (you can click on the line numbers above to navigate directly to >>>>>>> them) >>>>>>> >>>>>>> Here is a sample of the output >>>>>>> of lines 514-519 for N=1,2,3,4 procs left to right. >>>>>>> >>>>>>> Thty're different for different procs. They should be the same, >>>>>>> since none of my input parameters are numprocs dependent, and I don't >>>>>>> explicitly use the size or rank anywhere in the code. >>>>>>> >>>>>> >>>>>> You are likely not dividing the rows you loop over so you are >>>>>> redundantly computing. >>>>>> >>>>> >>>>> Thanks for the reply. >>>>> >>>>> Line 274gets the local row indices of Petsc Matrix >>>>> AVG_BDIBJ >>>>> >>>>> Line 295 >>>>> iterates >>>>> over the local rows and the lines below get the column >>>>> elements. For each row, the column elements are assigned by the lines >>>>> up to Line 344and stored locally in colvalues[]. Dunno if the details are relevant. >>>>> >>>>> Line 347inserts the sitestride1^th row into the matrix >>>>> >>>>> Line 353+does the mat assembly >>>>> >>>>> Then, after a lot of currently irrelevant code, >>>>> >>>>> Line 514+dumps the mat plot to graphics >>>>> >>>>> >>>>> Different numprocs give different matrices. >>>>> >>>>> Can somebody suggest what I did wrong (or didn't do)? >>>>> >>>> >>>> Different values are being given to MatSetValues() for different >>>> numbers of processes. So >>>> >>>> 1) Reduce this to the smallest problem size possible >>>> >>>> 2) Print out all rows/cols/values for each call >>>> >>>> 3) Compare 2 procs to the serial case >>>> >>>> >>> >>> Thanks for your excellent suggestion. >>> >>> I modified my codeto dump the matrix in binary >>> >>> Then I used this python script I hadto convert to ascii >>> >> >> Do not print the matrix, print the data you are passing to MatSetValues(). >> >> MatSetValues() is not likely to be broken. Every PETSc code in the world >> calls this many times on every simulation. >> >> Matt >> >> >>> >>> Here are the values of AVG_BDIBJ, >>> a 9X9 matrix (the smallest possible problem size) run with the exact same >>> input parameters with 1,2,3 and 4 procs >>> >>> As you can see, the 1 and 2 procs match up, but the 3 and 4 procs do not. >>> >>> Serious wierdness. >>> >>> >>> >>>> Matt >>>> >>>> >>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>>> for (sitestride1 = Istart; sitestride1 < Iend; sitestride1++) >>>>>>>> { >>>>>>>> for (sitestride2 = 0; sitestride2 < matsize; sitestride2++) >>>>>>>> { >>>>>>>> for (alpha = 0; alpha < dim; alpha++) >>>>>>>> { >>>>>>>> for (mu = 0; mu < dim; mu++) >>>>>>>> for (lambda = 0; lambda < dim; lambda++) >>>>>>>> { >>>>>>>> vecval = rand () / rand (); >>>>>>>> } >>>>>>>> >>>>>>>> VecSetValue (BDB_AA, alpha, vecval, INSERT_VALUES); >>>>>>>> >>>>>>>> } >>>>>>>> VecAssemblyBegin (BDB_AA); >>>>>>>> VecAssemblyEnd (BDB_AA); >>>>>>>> VecSum (BDB_AA, &element); >>>>>>>> colvalues[sitestride2] = element; >>>>>>>> >>>>>>>> } >>>>>>>> //Insert the array of colvalues to the sitestride1^th row of H >>>>>>>> MatSetValues (AVG_BDIBJ, 1, &sitestride1, matsize, idx, >>>>>>>> colvalues, >>>>>>>> INSERT_VALUES); >>>>>>>> >>>>>>>> } >>>>>>>> >>>>>>>> > The code is large and complex, so I have created a smaller program >>>>>>>> > with the same >>>>>>>> > loop structure here. >>>>>>>> > >>>>>>>> > Compile it and run it with "mpirun -np $N ./test -draw_pause -1" >>>>>>>> gives >>>>>>>> > different results for different values of N even though it's not >>>>>>>> supposed >>>>>>>> > to. >>>>>>>> >>>>>>>> What do you expect to see? >>>>>>>> >>>>>>>> > Here is a sample output for >>>>>>>> N=1,2,3,4 >>>>>>>> > from left to right. >>>>>>>> > >>>>>>>> > Can anyone guide me as to what I'm doing wrong? Are any of the >>>>>>>> petssc >>>>>>>> > routines used not parallelizable? >>>>>>>> > >>>>>>>> > Thanks in advance, >>>>>>>> > >>>>>>>> > Regards. >>>>>>>> > >>>>>>>> > -- >>>>>>>> > --- >>>>>>>> > *Analabha Roy* >>>>>>>> > C.S.I.R Senior Research >>>>>>>> > Associate >>>>>>>> > Saha Institute of Nuclear Physics >>>>>>>> > Section 1, Block AF >>>>>>>> > Bidhannagar, Calcutta 700064 >>>>>>>> > India >>>>>>>> > *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >>>>>>>> > *Webpage*: http://www.ph.utexas.edu/~daneel/ >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> --- >>>>>>> *Analabha Roy* >>>>>>> C.S.I.R Senior Research Associate >>>>>>> Saha Institute of Nuclear Physics >>>>>>> Section 1, Block AF >>>>>>> Bidhannagar, Calcutta 700064 >>>>>>> India >>>>>>> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >>>>>>> *Webpage*: http://www.ph.utexas.edu/~daneel/ >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> --- >>>>> *Analabha Roy* >>>>> C.S.I.R Senior Research Associate >>>>> Saha Institute of Nuclear Physics >>>>> Section 1, Block AF >>>>> Bidhannagar, Calcutta 700064 >>>>> India >>>>> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >>>>> *Webpage*: http://www.ph.utexas.edu/~daneel/ >>>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >>> >>> -- >>> --- >>> *Analabha Roy* >>> C.S.I.R Senior Research Associate >>> Saha Institute of Nuclear Physics >>> Section 1, Block AF >>> Bidhannagar, Calcutta 700064 >>> India >>> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >>> *Webpage*: http://www.ph.utexas.edu/~daneel/ >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > --- > *Analabha Roy* > C.S.I.R Senior Research Associate > Saha Institute of Nuclear Physics > Section 1, Block AF > Bidhannagar, Calcutta 700064 > India > *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com > *Webpage*: http://www.ph.utexas.edu/~daneel/ > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hariseldon99 at gmail.com Tue Sep 24 13:18:30 2013 From: hariseldon99 at gmail.com (Analabha Roy) Date: Tue, 24 Sep 2013 23:48:30 +0530 Subject: [petsc-users] Fwd: Building the same petsc matrix with different numprocs gives different results! In-Reply-To: References: <87zjr2zlih.fsf@mcs.anl.gov> Message-ID: On Tue, Sep 24, 2013 at 11:35 PM, Matthew Knepley wrote: > On Tue, Sep 24, 2013 at 10:58 AM, Analabha Roy wrote: > >> Hi, >> >> Sorry for misunderstanding >> >> I modified my source thus so that the >> rows/cols/values for each call are printed before inserting into >> MatSetValues() >> >> >> Then ran it with 1,2 processors >> >> >> Here are the outputs >> >> >> Strange! Running it with 2 procs and only half the values show up!!!!! >> > > PetscPrintf() only prints from rank 0. Use PETSC_COMM_SELF. > > Sorry. Modified accordingly and here is new output(I manually reordered the output of the 2 procs case since the order in which it was printed was haphazard) All the elements do not match. > > Matt > > >> And even those do not match!!!! >> >> >> >> >> On Tue, Sep 24, 2013 at 11:12 PM, Matthew Knepley wrote: >> >>> On Tue, Sep 24, 2013 at 10:39 AM, Analabha Roy wrote: >>> >>>> Hi, >>>> >>>> >>>> >>>> On Tue, Sep 24, 2013 at 9:33 PM, Matthew Knepley wrote: >>>> >>>>> On Tue, Sep 24, 2013 at 8:35 AM, Analabha Roy wrote: >>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Sep 24, 2013 at 8:41 PM, Matthew Knepley wrote: >>>>>> >>>>>>> On Tue, Sep 24, 2013 at 8:08 AM, Analabha Roy < >>>>>>> hariseldon99 at gmail.com> wrote: >>>>>>> >>>>>>>> >>>>>>>> On Tue, Sep 24, 2013 at 1:42 PM, Jed Brown wrote: >>>>>>>> >>>>>>>>> Analabha Roy writes: >>>>>>>>> >>>>>>>>> > Hi all, >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > Compiling and running this >>>>>>>>> > code< >>>>>>>>> https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c >>>>>>>>> >that >>>>>>>>> > builds a petsc matrix gives different results when run with >>>>>>>>> different >>>>>>>>> > number of processors. >>>>>>>>> >>>>>>>>> >>>>>>>> Thanks for the reply. >>>>>>>> >>>>>>>> >>>>>>>>> Uh, if you call rand() on different processors, why would you >>>>>>>>> expect it >>>>>>>>> to give the same results? >>>>>>>>> >>>>>>>>> Right, I get that. The rand() was a placeholder. >>>>>>>> >>>>>>>> This original much larger codereplicates the same loop structure and runs the same Petsc subroutines, but >>>>>>>> running it by >>>>>>>> >>>>>>>> mpirun -np $N ./eth -lattice_size 5 -vector_size 1 -repulsion 0.0 >>>>>>>> -draw_out -draw_pause -1 >>>>>>>> >>>>>>>> with N=1,2,3,4 gives different results for the matrix dumped out by >>>>>>>> lines 514-519. >>>>>>>> The matrix itself is evaluated in parallel, created in lines263-275 >>>>>>>> and >>>>>>>> evaluated in lines 294-356 >>>>>>>> >>>>>>>> (you can click on the line numbers above to navigate directly to >>>>>>>> them) >>>>>>>> >>>>>>>> Here is a sample of the output >>>>>>>> of lines 514-519 for N=1,2,3,4 procs left to right. >>>>>>>> >>>>>>>> Thty're different for different procs. They should be the same, >>>>>>>> since none of my input parameters are numprocs dependent, and I don't >>>>>>>> explicitly use the size or rank anywhere in the code. >>>>>>>> >>>>>>> >>>>>>> You are likely not dividing the rows you loop over so you are >>>>>>> redundantly computing. >>>>>>> >>>>>> >>>>>> Thanks for the reply. >>>>>> >>>>>> Line 274gets the local row indices of Petsc Matrix >>>>>> AVG_BDIBJ >>>>>> >>>>>> Line 295 >>>>>> iterates >>>>>> over the local rows and the lines below get the column >>>>>> elements. For each row, the column elements are assigned by the lines >>>>>> up to Line 344and stored locally in colvalues[]. Dunno if the details are relevant. >>>>>> >>>>>> Line 347inserts the sitestride1^th row into the matrix >>>>>> >>>>>> Line 353+does the mat assembly >>>>>> >>>>>> Then, after a lot of currently irrelevant code, >>>>>> >>>>>> Line 514+dumps the mat plot to graphics >>>>>> >>>>>> >>>>>> Different numprocs give different matrices. >>>>>> >>>>>> Can somebody suggest what I did wrong (or didn't do)? >>>>>> >>>>> >>>>> Different values are being given to MatSetValues() for different >>>>> numbers of processes. So >>>>> >>>>> 1) Reduce this to the smallest problem size possible >>>>> >>>>> 2) Print out all rows/cols/values for each call >>>>> >>>>> 3) Compare 2 procs to the serial case >>>>> >>>>> >>>> >>>> Thanks for your excellent suggestion. >>>> >>>> I modified my codeto dump the matrix in binary >>>> >>>> Then I used this python script I hadto convert to ascii >>>> >>> >>> Do not print the matrix, print the data you are passing to >>> MatSetValues(). >>> >>> MatSetValues() is not likely to be broken. Every PETSc code in the world >>> calls this many times on every simulation. >>> >>> Matt >>> >>> >>>> >>>> Here are the values of AVG_BDIBJ, >>>> a 9X9 matrix (the smallest possible problem size) run with the exact same >>>> input parameters with 1,2,3 and 4 procs >>>> >>>> As you can see, the 1 and 2 procs match up, but the 3 and 4 procs do >>>> not. >>>> >>>> Serious wierdness. >>>> >>>> >>>> >>>>> Matt >>>>> >>>>> >>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> for (sitestride1 = Istart; sitestride1 < Iend; sitestride1++) >>>>>>>>> { >>>>>>>>> for (sitestride2 = 0; sitestride2 < matsize; sitestride2++) >>>>>>>>> { >>>>>>>>> for (alpha = 0; alpha < dim; alpha++) >>>>>>>>> { >>>>>>>>> for (mu = 0; mu < dim; mu++) >>>>>>>>> for (lambda = 0; lambda < dim; lambda++) >>>>>>>>> { >>>>>>>>> vecval = rand () / rand (); >>>>>>>>> } >>>>>>>>> >>>>>>>>> VecSetValue (BDB_AA, alpha, vecval, INSERT_VALUES); >>>>>>>>> >>>>>>>>> } >>>>>>>>> VecAssemblyBegin (BDB_AA); >>>>>>>>> VecAssemblyEnd (BDB_AA); >>>>>>>>> VecSum (BDB_AA, &element); >>>>>>>>> colvalues[sitestride2] = element; >>>>>>>>> >>>>>>>>> } >>>>>>>>> //Insert the array of colvalues to the sitestride1^th row of >>>>>>>>> H >>>>>>>>> MatSetValues (AVG_BDIBJ, 1, &sitestride1, matsize, idx, >>>>>>>>> colvalues, >>>>>>>>> INSERT_VALUES); >>>>>>>>> >>>>>>>>> } >>>>>>>>> >>>>>>>>> > The code is large and complex, so I have created a smaller >>>>>>>>> program >>>>>>>>> > with the same >>>>>>>>> > loop structure here. >>>>>>>>> > >>>>>>>>> > Compile it and run it with "mpirun -np $N ./test -draw_pause -1" >>>>>>>>> gives >>>>>>>>> > different results for different values of N even though it's not >>>>>>>>> supposed >>>>>>>>> > to. >>>>>>>>> >>>>>>>>> What do you expect to see? >>>>>>>>> >>>>>>>>> > Here is a sample output >>>>>>>>> for N=1,2,3,4 >>>>>>>>> > from left to right. >>>>>>>>> > >>>>>>>>> > Can anyone guide me as to what I'm doing wrong? Are any of the >>>>>>>>> petssc >>>>>>>>> > routines used not parallelizable? >>>>>>>>> > >>>>>>>>> > Thanks in advance, >>>>>>>>> > >>>>>>>>> > Regards. >>>>>>>>> > >>>>>>>>> > -- >>>>>>>>> > --- >>>>>>>>> > *Analabha Roy* >>>>>>>>> > C.S.I.R Senior Research >>>>>>>>> > Associate >>>>>>>>> > Saha Institute of Nuclear Physics >>>>>>>>> > Section 1, Block AF >>>>>>>>> > Bidhannagar, Calcutta 700064 >>>>>>>>> > India >>>>>>>>> > *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >>>>>>>>> > *Webpage*: http://www.ph.utexas.edu/~daneel/ >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> --- >>>>>>>> *Analabha Roy* >>>>>>>> C.S.I.R Senior Research Associate >>>>>>>> Saha Institute of Nuclear Physics >>>>>>>> Section 1, Block AF >>>>>>>> Bidhannagar, Calcutta 700064 >>>>>>>> India >>>>>>>> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >>>>>>>> *Webpage*: http://www.ph.utexas.edu/~daneel/ >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> --- >>>>>> *Analabha Roy* >>>>>> C.S.I.R Senior Research Associate >>>>>> Saha Institute of Nuclear Physics >>>>>> Section 1, Block AF >>>>>> Bidhannagar, Calcutta 700064 >>>>>> India >>>>>> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >>>>>> *Webpage*: http://www.ph.utexas.edu/~daneel/ >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>> >>>> >>>> >>>> -- >>>> --- >>>> *Analabha Roy* >>>> C.S.I.R Senior Research Associate >>>> Saha Institute of Nuclear Physics >>>> Section 1, Block AF >>>> Bidhannagar, Calcutta 700064 >>>> India >>>> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >>>> *Webpage*: http://www.ph.utexas.edu/~daneel/ >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> >> >> -- >> --- >> *Analabha Roy* >> C.S.I.R Senior Research Associate >> Saha Institute of Nuclear Physics >> Section 1, Block AF >> Bidhannagar, Calcutta 700064 >> India >> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >> *Webpage*: http://www.ph.utexas.edu/~daneel/ >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- --- *Analabha Roy* C.S.I.R Senior Research Associate Saha Institute of Nuclear Physics Section 1, Block AF Bidhannagar, Calcutta 700064 India *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com *Webpage*: http://www.ph.utexas.edu/~daneel/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrosso at uci.edu Tue Sep 24 16:29:45 2013 From: mrosso at uci.edu (Michele Rosso) Date: Tue, 24 Sep 2013 14:29:45 -0700 Subject: [petsc-users] How to speed up geometric multigrid In-Reply-To: <4499926F-40CE-462C-BFD6-8798A02126FB@mcs.anl.gov> References: <52406460.3060905@uci.edu> <4499926F-40CE-462C-BFD6-8798A02126FB@mcs.anl.gov> Message-ID: <52420449.4010603@uci.edu> Barry, I re-rerun the test case with the option -pc_mg_log as you suggested. I attached the new output ("final_new.txt'). Thanks for your help. Michele On 09/23/2013 09:35 AM, Barry Smith wrote: > Run with the additional option -pc_mg_log and send us the log file. > > Barry > > Maybe we should make this the default somehow. > > > On Sep 23, 2013, at 10:55 AM, Michele Rosso wrote: > >> Hi, >> >> I am successfully using PETSc to solve a 3D Poisson's equation with CG + MG . Such equation arises from a projection algorithm for a multiphase incompressible flow simulation. >> I set up the solver as I was suggested to do in a previous thread (title: "GAMG speed") and run a test case (liquid droplet with surface tension falling under the effect of gravity in a quiescent fluid). >> The solution of the Poisson Equation via multigrid is correct but it becomes progressively slower and slower as the simulation progresses (I am performing successive solves) due to an increase in the number of iterations. >> Since the solution of the Poisson equation is mission-critical, I need to speed it up as much as I can. >> Could you please help me out with this? >> >> I run the test case with the following options: >> >> -pc_type mg -pc_mg_galerkin -pc_mg_levels 5 -mg_levels_ksp_type richardson -mg_levels_ksp_max_it 1 >> -mg_coarse_pc_type lu -mg_coarse_pc_factor_mat_solver_package superlu_dist >> -log_summary -ksp_view -ksp_monitor_true_residual -options_left >> >> Please find the diagnostic for the final solve in the attached file "final.txt'. >> Thank you, >> >> Michele >> > -------------- next part -------------- 0 KSP unpreconditioned resid norm 6.668649980607e-07 true resid norm 6.668649980607e-07 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 5.553470942050e-07 true resid norm 5.553470942050e-07 ||r(i)||/||b|| 8.327728937941e-01 2 KSP unpreconditioned resid norm 4.864455176402e-07 true resid norm 4.864455176402e-07 ||r(i)||/||b|| 7.294512668305e-01 3 KSP unpreconditioned resid norm 4.052735454511e-07 true resid norm 4.052735454511e-07 ||r(i)||/||b|| 6.077295204122e-01 4 KSP unpreconditioned resid norm 3.521525854995e-07 true resid norm 3.521525854995e-07 ||r(i)||/||b|| 5.280717784314e-01 5 KSP unpreconditioned resid norm 3.127341886521e-07 true resid norm 3.127341886521e-07 ||r(i)||/||b|| 4.689617682163e-01 6 KSP unpreconditioned resid norm 2.661343212148e-07 true resid norm 2.661343212148e-07 ||r(i)||/||b|| 3.990827558632e-01 7 KSP unpreconditioned resid norm 2.264743335680e-07 true resid norm 2.264743335680e-07 ||r(i)||/||b|| 3.396104672259e-01 8 KSP unpreconditioned resid norm 1.907585902453e-07 true resid norm 1.907585902453e-07 ||r(i)||/||b|| 2.860527854964e-01 9 KSP unpreconditioned resid norm 1.634189868677e-07 true resid norm 1.634189868677e-07 ||r(i)||/||b|| 2.450555769803e-01 10 KSP unpreconditioned resid norm 1.433245085298e-07 true resid norm 1.433245085298e-07 ||r(i)||/||b|| 2.149228238797e-01 11 KSP unpreconditioned resid norm 1.269502496716e-07 true resid norm 1.269502496716e-07 ||r(i)||/||b|| 1.903687403609e-01 12 KSP unpreconditioned resid norm 1.130211959415e-07 true resid norm 1.130211959415e-07 ||r(i)||/||b|| 1.694813736966e-01 13 KSP unpreconditioned resid norm 9.996952262259e-08 true resid norm 9.996952262259e-08 ||r(i)||/||b|| 1.499096862383e-01 14 KSP unpreconditioned resid norm 8.716350099438e-08 true resid norm 8.716350099438e-08 ||r(i)||/||b|| 1.307063667277e-01 15 KSP unpreconditioned resid norm 7.728115085097e-08 true resid norm 7.728115085097e-08 ||r(i)||/||b|| 1.158872501566e-01 16 KSP unpreconditioned resid norm 6.913427522680e-08 true resid norm 6.913427522680e-08 ||r(i)||/||b|| 1.036705711469e-01 17 KSP unpreconditioned resid norm 6.419022635565e-08 true resid norm 6.419022635565e-08 ||r(i)||/||b|| 9.625670344421e-02 18 KSP unpreconditioned resid norm 6.106997635578e-08 true resid norm 6.106997635578e-08 ||r(i)||/||b|| 9.157772042824e-02 19 KSP unpreconditioned resid norm 6.060486457187e-08 true resid norm 6.060486457187e-08 ||r(i)||/||b|| 9.088026024475e-02 20 KSP unpreconditioned resid norm 6.253815762889e-08 true resid norm 6.253815762889e-08 ||r(i)||/||b|| 9.377933736328e-02 21 KSP unpreconditioned resid norm 6.519301631838e-08 true resid norm 6.519301631838e-08 ||r(i)||/||b|| 9.776044103075e-02 22 KSP unpreconditioned resid norm 6.957117380581e-08 true resid norm 6.957117380581e-08 ||r(i)||/||b|| 1.043257241093e-01 23 KSP unpreconditioned resid norm 7.610073544020e-08 true resid norm 7.610073544020e-08 ||r(i)||/||b|| 1.141171536391e-01 24 KSP unpreconditioned resid norm 8.186698411243e-08 true resid norm 8.186698411243e-08 ||r(i)||/||b|| 1.227639542494e-01 25 KSP unpreconditioned resid norm 8.752525135530e-08 true resid norm 8.752525135530e-08 ||r(i)||/||b|| 1.312488308876e-01 26 KSP unpreconditioned resid norm 9.056156051616e-08 true resid norm 9.056156051616e-08 ||r(i)||/||b|| 1.358019400921e-01 27 KSP unpreconditioned resid norm 9.298785441037e-08 true resid norm 9.298785441037e-08 ||r(i)||/||b|| 1.394402985324e-01 28 KSP unpreconditioned resid norm 9.384690986352e-08 true resid norm 9.384690986352e-08 ||r(i)||/||b|| 1.407284984764e-01 29 KSP unpreconditioned resid norm 9.396046040085e-08 true resid norm 9.396046040085e-08 ||r(i)||/||b|| 1.408987736260e-01 30 KSP unpreconditioned resid norm 9.271312762560e-08 true resid norm 9.271312762560e-08 ||r(i)||/||b|| 1.390283309144e-01 31 KSP unpreconditioned resid norm 9.041515489292e-08 true resid norm 9.041515489292e-08 ||r(i)||/||b|| 1.355823969707e-01 32 KSP unpreconditioned resid norm 8.744080535140e-08 true resid norm 8.744080535140e-08 ||r(i)||/||b|| 1.311221995542e-01 33 KSP unpreconditioned resid norm 8.298300549188e-08 true resid norm 8.298300549188e-08 ||r(i)||/||b|| 1.244374884470e-01 34 KSP unpreconditioned resid norm 7.827292133955e-08 true resid norm 7.827292133955e-08 ||r(i)||/||b|| 1.173744634479e-01 35 KSP unpreconditioned resid norm 7.369219142361e-08 true resid norm 7.369219142361e-08 ||r(i)||/||b|| 1.105054120968e-01 36 KSP unpreconditioned resid norm 6.939855306900e-08 true resid norm 6.939855306900e-08 ||r(i)||/||b|| 1.040668700124e-01 37 KSP unpreconditioned resid norm 6.603717997371e-08 true resid norm 6.603717997371e-08 ||r(i)||/||b|| 9.902630992143e-02 38 KSP unpreconditioned resid norm 6.143846086712e-08 true resid norm 6.143846086712e-08 ||r(i)||/||b|| 9.213028280955e-02 39 KSP unpreconditioned resid norm 5.862855945694e-08 true resid norm 5.862855945694e-08 ||r(i)||/||b|| 8.791668422760e-02 40 KSP unpreconditioned resid norm 5.525007018405e-08 true resid norm 5.525007018405e-08 ||r(i)||/||b|| 8.285045750598e-02 41 KSP unpreconditioned resid norm 5.249856083862e-08 true resid norm 5.249856083862e-08 ||r(i)||/||b|| 7.872442097170e-02 42 KSP unpreconditioned resid norm 5.092834924822e-08 true resid norm 5.092834924822e-08 ||r(i)||/||b|| 7.636980407778e-02 43 KSP unpreconditioned resid norm 4.953977568220e-08 true resid norm 4.953977568220e-08 ||r(i)||/||b|| 7.428756318935e-02 44 KSP unpreconditioned resid norm 4.780756932577e-08 true resid norm 4.780756932577e-08 ||r(i)||/||b|| 7.169002641435e-02 45 KSP unpreconditioned resid norm 4.596369150694e-08 true resid norm 4.596369150694e-08 ||r(i)||/||b|| 6.892503226382e-02 46 KSP unpreconditioned resid norm 4.444012557313e-08 true resid norm 4.444012557313e-08 ||r(i)||/||b|| 6.664036304555e-02 47 KSP unpreconditioned resid norm 4.386987929605e-08 true resid norm 4.386987929605e-08 ||r(i)||/||b|| 6.578524802416e-02 48 KSP unpreconditioned resid norm 4.327010140460e-08 true resid norm 4.327010140460e-08 ||r(i)||/||b|| 6.488584875565e-02 49 KSP unpreconditioned resid norm 4.273337623498e-08 true resid norm 4.273337623498e-08 ||r(i)||/||b|| 6.408100044124e-02 50 KSP unpreconditioned resid norm 4.318588012407e-08 true resid norm 4.318588012407e-08 ||r(i)||/||b|| 6.475955440706e-02 51 KSP unpreconditioned resid norm 4.352727563068e-08 true resid norm 4.352727563068e-08 ||r(i)||/||b|| 6.527149536602e-02 52 KSP unpreconditioned resid norm 4.440414830419e-08 true resid norm 4.440414830419e-08 ||r(i)||/||b|| 6.658641319206e-02 53 KSP unpreconditioned resid norm 4.599990690361e-08 true resid norm 4.599990690361e-08 ||r(i)||/||b|| 6.897933920267e-02 54 KSP unpreconditioned resid norm 4.818235033087e-08 true resid norm 4.818235033087e-08 ||r(i)||/||b|| 7.225203072734e-02 55 KSP unpreconditioned resid norm 4.982635628148e-08 true resid norm 4.982635628148e-08 ||r(i)||/||b|| 7.471730624096e-02 56 KSP unpreconditioned resid norm 5.062956137068e-08 true resid norm 5.062956137068e-08 ||r(i)||/||b|| 7.592175555461e-02 57 KSP unpreconditioned resid norm 5.178154503258e-08 true resid norm 5.178154503258e-08 ||r(i)||/||b|| 7.764921713266e-02 58 KSP unpreconditioned resid norm 5.192348330208e-08 true resid norm 5.192348330208e-08 ||r(i)||/||b|| 7.786206121642e-02 59 KSP unpreconditioned resid norm 5.116016560240e-08 true resid norm 5.116016560240e-08 ||r(i)||/||b|| 7.671742519278e-02 60 KSP unpreconditioned resid norm 4.991678465592e-08 true resid norm 4.991678465592e-08 ||r(i)||/||b|| 7.485290846136e-02 61 KSP unpreconditioned resid norm 4.843006194965e-08 true resid norm 4.843006194965e-08 ||r(i)||/||b|| 7.262348764816e-02 62 KSP unpreconditioned resid norm 4.741529986557e-08 true resid norm 4.741529986557e-08 ||r(i)||/||b|| 7.110179722052e-02 63 KSP unpreconditioned resid norm 4.614609418591e-08 true resid norm 4.614609418591e-08 ||r(i)||/||b|| 6.919855491008e-02 64 KSP unpreconditioned resid norm 4.474933115979e-08 true resid norm 4.474933115979e-08 ||r(i)||/||b|| 6.710403348494e-02 65 KSP unpreconditioned resid norm 4.327061845197e-08 true resid norm 4.327061845197e-08 ||r(i)||/||b|| 6.488662409604e-02 66 KSP unpreconditioned resid norm 4.181101077793e-08 true resid norm 4.181101077793e-08 ||r(i)||/||b|| 6.269786373482e-02 67 KSP unpreconditioned resid norm 4.052042782582e-08 true resid norm 4.052042782582e-08 ||r(i)||/||b|| 6.076256505238e-02 68 KSP unpreconditioned resid norm 3.934547509174e-08 true resid norm 3.934547509174e-08 ||r(i)||/||b|| 5.900066011287e-02 69 KSP unpreconditioned resid norm 3.825955794222e-08 true resid norm 3.825955794222e-08 ||r(i)||/||b|| 5.737226883025e-02 70 KSP unpreconditioned resid norm 3.743025940470e-08 true resid norm 3.743025940470e-08 ||r(i)||/||b|| 5.612869098476e-02 71 KSP unpreconditioned resid norm 3.629579364234e-08 true resid norm 3.629579364234e-08 ||r(i)||/||b|| 5.442749844106e-02 72 KSP unpreconditioned resid norm 3.492644577559e-08 true resid norm 3.492644577559e-08 ||r(i)||/||b|| 5.237408752471e-02 73 KSP unpreconditioned resid norm 3.360610823449e-08 true resid norm 3.360610823449e-08 ||r(i)||/||b|| 5.039417023269e-02 74 KSP unpreconditioned resid norm 3.192823945022e-08 true resid norm 3.192823945022e-08 ||r(i)||/||b|| 4.787811557523e-02 75 KSP unpreconditioned resid norm 3.052725874579e-08 true resid norm 3.052725874579e-08 ||r(i)||/||b|| 4.577726951417e-02 76 KSP unpreconditioned resid norm 2.969172524325e-08 true resid norm 2.969172524325e-08 ||r(i)||/||b|| 4.452434200264e-02 77 KSP unpreconditioned resid norm 2.872752108342e-08 true resid norm 2.872752108342e-08 ||r(i)||/||b|| 4.307846590684e-02 78 KSP unpreconditioned resid norm 2.823034410194e-08 true resid norm 2.823034410194e-08 ||r(i)||/||b|| 4.233292223170e-02 79 KSP unpreconditioned resid norm 2.813464203018e-08 true resid norm 2.813464203018e-08 ||r(i)||/||b|| 4.218941181798e-02 80 KSP unpreconditioned resid norm 2.853366096663e-08 true resid norm 2.853366096663e-08 ||r(i)||/||b|| 4.278776221516e-02 81 KSP unpreconditioned resid norm 2.904485150986e-08 true resid norm 2.904485150986e-08 ||r(i)||/||b|| 4.355431998130e-02 82 KSP unpreconditioned resid norm 2.976425305819e-08 true resid norm 2.976425305819e-08 ||r(i)||/||b|| 4.463310136947e-02 83 KSP unpreconditioned resid norm 3.030617696330e-08 true resid norm 3.030617696330e-08 ||r(i)||/||b|| 4.544574546788e-02 84 KSP unpreconditioned resid norm 3.085032763639e-08 true resid norm 3.085032763639e-08 ||r(i)||/||b|| 4.626172872487e-02 85 KSP unpreconditioned resid norm 3.106948896112e-08 true resid norm 3.106948896112e-08 ||r(i)||/||b|| 4.659037294126e-02 86 KSP unpreconditioned resid norm 3.097264721696e-08 true resid norm 3.097264721696e-08 ||r(i)||/||b|| 4.644515352737e-02 87 KSP unpreconditioned resid norm 3.073121354901e-08 true resid norm 3.073121354901e-08 ||r(i)||/||b|| 4.608311073213e-02 88 KSP unpreconditioned resid norm 3.057805509019e-08 true resid norm 3.057805509019e-08 ||r(i)||/||b|| 4.585344136986e-02 89 KSP unpreconditioned resid norm 3.055953721811e-08 true resid norm 3.055953721811e-08 ||r(i)||/||b|| 4.582567282280e-02 90 KSP unpreconditioned resid norm 3.021043511164e-08 true resid norm 3.021043511164e-08 ||r(i)||/||b|| 4.530217540206e-02 91 KSP unpreconditioned resid norm 2.985779907968e-08 true resid norm 2.985779907968e-08 ||r(i)||/||b|| 4.477337866962e-02 92 KSP unpreconditioned resid norm 2.940558495203e-08 true resid norm 2.940558495203e-08 ||r(i)||/||b|| 4.409525921669e-02 93 KSP unpreconditioned resid norm 2.898874635244e-08 true resid norm 2.898874635244e-08 ||r(i)||/||b|| 4.347018727440e-02 94 KSP unpreconditioned resid norm 2.815214923051e-08 true resid norm 2.815214923051e-08 ||r(i)||/||b|| 4.221566480829e-02 95 KSP unpreconditioned resid norm 2.722338445462e-08 true resid norm 2.722338445462e-08 ||r(i)||/||b|| 4.082293197842e-02 96 KSP unpreconditioned resid norm 2.632669802720e-08 true resid norm 2.632669802720e-08 ||r(i)||/||b|| 3.947830236069e-02 97 KSP unpreconditioned resid norm 2.543770053490e-08 true resid norm 2.543770053490e-08 ||r(i)||/||b|| 3.814520271551e-02 98 KSP unpreconditioned resid norm 2.472705425508e-08 true resid norm 2.472705425508e-08 ||r(i)||/||b|| 3.707955032426e-02 99 KSP unpreconditioned resid norm 2.393324291382e-08 true resid norm 2.393324291382e-08 ||r(i)||/||b|| 3.588918744186e-02 100 KSP unpreconditioned resid norm 2.320986855424e-08 true resid norm 2.320986855424e-08 ||r(i)||/||b|| 3.480444860914e-02 101 KSP unpreconditioned resid norm 2.215004767384e-08 true resid norm 2.215004767384e-08 ||r(i)||/||b|| 3.321519008833e-02 102 KSP unpreconditioned resid norm 2.113255545307e-08 true resid norm 2.113255545307e-08 ||r(i)||/||b|| 3.168940567361e-02 103 KSP unpreconditioned resid norm 1.994148992056e-08 true resid norm 1.994148992056e-08 ||r(i)||/||b|| 2.990333872456e-02 104 KSP unpreconditioned resid norm 1.865844893196e-08 true resid norm 1.865844893196e-08 ||r(i)||/||b|| 2.797934962282e-02 105 KSP unpreconditioned resid norm 1.771157759657e-08 true resid norm 1.771157759657e-08 ||r(i)||/||b|| 2.655946503127e-02 106 KSP unpreconditioned resid norm 1.688339886106e-08 true resid norm 1.688339886106e-08 ||r(i)||/||b|| 2.531756638924e-02 107 KSP unpreconditioned resid norm 1.611686466306e-08 true resid norm 1.611686466306e-08 ||r(i)||/||b|| 2.416810705304e-02 108 KSP unpreconditioned resid norm 1.559361130979e-08 true resid norm 1.559361130979e-08 ||r(i)||/||b|| 2.338346045322e-02 109 KSP unpreconditioned resid norm 1.526356424250e-08 true resid norm 1.526356424250e-08 ||r(i)||/||b|| 2.288853709055e-02 110 KSP unpreconditioned resid norm 1.491635379995e-08 true resid norm 1.491635379995e-08 ||r(i)||/||b|| 2.236787632178e-02 111 KSP unpreconditioned resid norm 1.467514600440e-08 true resid norm 1.467514600440e-08 ||r(i)||/||b|| 2.200617223438e-02 112 KSP unpreconditioned resid norm 1.432366873409e-08 true resid norm 1.432366873409e-08 ||r(i)||/||b|| 2.147911312747e-02 113 KSP unpreconditioned resid norm 1.416510304700e-08 true resid norm 1.416510304700e-08 ||r(i)||/||b|| 2.124133533502e-02 114 KSP unpreconditioned resid norm 1.402261903746e-08 true resid norm 1.402261903746e-08 ||r(i)||/||b|| 2.102767288468e-02 115 KSP unpreconditioned resid norm 1.372225731655e-08 true resid norm 1.372225731655e-08 ||r(i)||/||b|| 2.057726429856e-02 116 KSP unpreconditioned resid norm 1.345487140441e-08 true resid norm 1.345487140441e-08 ||r(i)||/||b|| 2.017630471465e-02 117 KSP unpreconditioned resid norm 1.312343543903e-08 true resid norm 1.312343543903e-08 ||r(i)||/||b|| 1.967929862445e-02 118 KSP unpreconditioned resid norm 1.288538974812e-08 true resid norm 1.288538974812e-08 ||r(i)||/||b|| 1.932233628335e-02 119 KSP unpreconditioned resid norm 1.247478398268e-08 true resid norm 1.247478398268e-08 ||r(i)||/||b|| 1.870661081172e-02 120 KSP unpreconditioned resid norm 1.223766740298e-08 true resid norm 1.223766740298e-08 ||r(i)||/||b|| 1.835104172294e-02 121 KSP unpreconditioned resid norm 1.198487578389e-08 true resid norm 1.198487578389e-08 ||r(i)||/||b|| 1.797196706791e-02 122 KSP unpreconditioned resid norm 1.180587844720e-08 true resid norm 1.180587844720e-08 ||r(i)||/||b|| 1.770355091591e-02 123 KSP unpreconditioned resid norm 1.151117400425e-08 true resid norm 1.151117400425e-08 ||r(i)||/||b|| 1.726162572293e-02 124 KSP unpreconditioned resid norm 1.131476344543e-08 true resid norm 1.131476344543e-08 ||r(i)||/||b|| 1.696709750599e-02 125 KSP unpreconditioned resid norm 1.086471401690e-08 true resid norm 1.086471401690e-08 ||r(i)||/||b|| 1.629222413606e-02 126 KSP unpreconditioned resid norm 1.047697215777e-08 true resid norm 1.047697215777e-08 ||r(i)||/||b|| 1.571078432402e-02 127 KSP unpreconditioned resid norm 1.005580230205e-08 true resid norm 1.005580230205e-08 ||r(i)||/||b|| 1.507921742976e-02 128 KSP unpreconditioned resid norm 9.472352797371e-09 true resid norm 9.472352797371e-09 ||r(i)||/||b|| 1.420430345710e-02 129 KSP unpreconditioned resid norm 8.836154294435e-09 true resid norm 8.836154294435e-09 ||r(i)||/||b|| 1.325028951907e-02 130 KSP unpreconditioned resid norm 8.256641591379e-09 true resid norm 8.256641591379e-09 ||r(i)||/||b|| 1.238127899258e-02 131 KSP unpreconditioned resid norm 7.680921164226e-09 true resid norm 7.680921164226e-09 ||r(i)||/||b|| 1.151795518818e-02 132 KSP unpreconditioned resid norm 7.027758084531e-09 true resid norm 7.027758084531e-09 ||r(i)||/||b|| 1.053850195312e-02 133 KSP unpreconditioned resid norm 6.471402181571e-09 true resid norm 6.471402181571e-09 ||r(i)||/||b|| 9.704216296238e-03 134 KSP unpreconditioned resid norm 6.063001199783e-09 true resid norm 6.063001199783e-09 ||r(i)||/||b|| 9.091797016510e-03 135 KSP unpreconditioned resid norm 5.597958558477e-09 true resid norm 5.597958558477e-09 ||r(i)||/||b|| 8.394440516081e-03 136 KSP unpreconditioned resid norm 5.184530292842e-09 true resid norm 5.184530292842e-09 ||r(i)||/||b|| 7.774482553319e-03 137 KSP unpreconditioned resid norm 4.776269184098e-09 true resid norm 4.776269184098e-09 ||r(i)||/||b|| 7.162273020759e-03 138 KSP unpreconditioned resid norm 4.469219353275e-09 true resid norm 4.469219353275e-09 ||r(i)||/||b|| 6.701835253420e-03 139 KSP unpreconditioned resid norm 4.197298649093e-09 true resid norm 4.197298649093e-09 ||r(i)||/||b|| 6.294075504487e-03 140 KSP unpreconditioned resid norm 3.939702510329e-09 true resid norm 3.939702510329e-09 ||r(i)||/||b|| 5.907796213307e-03 141 KSP unpreconditioned resid norm 3.796476096109e-09 true resid norm 3.796476096109e-09 ||r(i)||/||b|| 5.693020487130e-03 142 KSP unpreconditioned resid norm 3.607984323478e-09 true resid norm 3.607984323478e-09 ||r(i)||/||b|| 5.410366916798e-03 143 KSP unpreconditioned resid norm 3.401485649235e-09 true resid norm 3.401485649235e-09 ||r(i)||/||b|| 5.100711027159e-03 144 KSP unpreconditioned resid norm 3.274097905105e-09 true resid norm 3.274097905105e-09 ||r(i)||/||b|| 4.909686240283e-03 145 KSP unpreconditioned resid norm 3.125638802147e-09 true resid norm 3.125638802147e-09 ||r(i)||/||b|| 4.687063815370e-03 146 KSP unpreconditioned resid norm 3.061111335524e-09 true resid norm 3.061111335524e-09 ||r(i)||/||b|| 4.590301401972e-03 147 KSP unpreconditioned resid norm 2.985684800902e-09 true resid norm 2.985684800902e-09 ||r(i)||/||b|| 4.477195248791e-03 148 KSP unpreconditioned resid norm 2.862674471503e-09 true resid norm 2.862674471503e-09 ||r(i)||/||b|| 4.292734631189e-03 149 KSP unpreconditioned resid norm 2.763637251201e-09 true resid norm 2.763637251201e-09 ||r(i)||/||b|| 4.144222982520e-03 150 KSP unpreconditioned resid norm 2.694548414669e-09 true resid norm 2.694548414669e-09 ||r(i)||/||b|| 4.040620549144e-03 151 KSP unpreconditioned resid norm 2.639492053311e-09 true resid norm 2.639492053311e-09 ||r(i)||/||b|| 3.958060568461e-03 152 KSP unpreconditioned resid norm 2.547902787256e-09 true resid norm 2.547902787256e-09 ||r(i)||/||b|| 3.820717528534e-03 153 KSP unpreconditioned resid norm 2.434901309797e-09 true resid norm 2.434901309797e-09 ||r(i)||/||b|| 3.651265723765e-03 154 KSP unpreconditioned resid norm 2.355118711798e-09 true resid norm 2.355118711798e-09 ||r(i)||/||b|| 3.531627418813e-03 155 KSP unpreconditioned resid norm 2.229813711959e-09 true resid norm 2.229813711959e-09 ||r(i)||/||b|| 3.343725819235e-03 156 KSP unpreconditioned resid norm 2.105745185624e-09 true resid norm 2.105745185624e-09 ||r(i)||/||b|| 3.157678378305e-03 157 KSP unpreconditioned resid norm 2.029966988265e-09 true resid norm 2.029966988265e-09 ||r(i)||/||b|| 3.044044887898e-03 158 KSP unpreconditioned resid norm 1.925518329336e-09 true resid norm 1.925518329336e-09 ||r(i)||/||b|| 2.887418495401e-03 159 KSP unpreconditioned resid norm 1.804401885691e-09 true resid norm 1.804401885691e-09 ||r(i)||/||b|| 2.705797861544e-03 160 KSP unpreconditioned resid norm 1.713290467595e-09 true resid norm 1.713290467595e-09 ||r(i)||/||b|| 2.569171380381e-03 161 KSP unpreconditioned resid norm 1.635294322724e-09 true resid norm 1.635294322724e-09 ||r(i)||/||b|| 2.452211958162e-03 162 KSP unpreconditioned resid norm 1.518468455862e-09 true resid norm 1.518468455862e-09 ||r(i)||/||b|| 2.277025275397e-03 163 KSP unpreconditioned resid norm 1.402499202597e-09 true resid norm 1.402499202597e-09 ||r(i)||/||b|| 2.103123130883e-03 164 KSP unpreconditioned resid norm 1.303779077546e-09 true resid norm 1.303779077546e-09 ||r(i)||/||b|| 1.955086983629e-03 165 KSP unpreconditioned resid norm 1.206014419382e-09 true resid norm 1.206014419382e-09 ||r(i)||/||b|| 1.808483610460e-03 166 KSP unpreconditioned resid norm 1.098568247745e-09 true resid norm 1.098568247745e-09 ||r(i)||/||b|| 1.647362286129e-03 167 KSP unpreconditioned resid norm 1.041644979554e-09 true resid norm 1.041644979554e-09 ||r(i)||/||b|| 1.562002778048e-03 168 KSP unpreconditioned resid norm 9.618756695271e-10 true resid norm 9.618756695271e-10 ||r(i)||/||b|| 1.442384399128e-03 169 KSP unpreconditioned resid norm 9.040827145172e-10 true resid norm 9.040827145172e-10 ||r(i)||/||b|| 1.355720748797e-03 170 KSP unpreconditioned resid norm 8.778242178263e-10 true resid norm 8.778242178263e-10 ||r(i)||/||b|| 1.316344718015e-03 171 KSP unpreconditioned resid norm 8.565215814244e-10 true resid norm 8.565215814244e-10 ||r(i)||/||b|| 1.284400266794e-03 172 KSP unpreconditioned resid norm 8.334887162606e-10 true resid norm 8.334887162607e-10 ||r(i)||/||b|| 1.249861244307e-03 173 KSP unpreconditioned resid norm 8.115645794799e-10 true resid norm 8.115645794799e-10 ||r(i)||/||b|| 1.216984819776e-03 174 KSP unpreconditioned resid norm 8.021212899395e-10 true resid norm 8.021212899395e-10 ||r(i)||/||b|| 1.202824098239e-03 175 KSP unpreconditioned resid norm 7.814018306714e-10 true resid norm 7.814018306715e-10 ||r(i)||/||b|| 1.171754152555e-03 176 KSP unpreconditioned resid norm 7.810010815154e-10 true resid norm 7.810010815154e-10 ||r(i)||/||b|| 1.171153207601e-03 177 KSP unpreconditioned resid norm 7.712833003442e-10 true resid norm 7.712833003442e-10 ||r(i)||/||b|| 1.156580871072e-03 178 KSP unpreconditioned resid norm 7.368308755917e-10 true resid norm 7.368308755917e-10 ||r(i)||/||b|| 1.104917603615e-03 179 KSP unpreconditioned resid norm 7.097782690911e-10 true resid norm 7.097782690911e-10 ||r(i)||/||b|| 1.064350762381e-03 180 KSP unpreconditioned resid norm 6.997135865790e-10 true resid norm 6.997135865790e-10 ||r(i)||/||b|| 1.049258228598e-03 181 KSP unpreconditioned resid norm 6.672026050919e-10 true resid norm 6.672026050919e-10 ||r(i)||/||b|| 1.000506259936e-03 182 KSP unpreconditioned resid norm 6.355831541638e-10 true resid norm 6.355831541638e-10 ||r(i)||/||b|| 9.530911893894e-04 183 KSP unpreconditioned resid norm 6.202489874036e-10 true resid norm 6.202489874036e-10 ||r(i)||/||b|| 9.300967800190e-04 184 KSP unpreconditioned resid norm 6.036442758510e-10 true resid norm 6.036442758510e-10 ||r(i)||/||b|| 9.051971202664e-04 185 KSP unpreconditioned resid norm 5.880268827444e-10 true resid norm 5.880268827444e-10 ||r(i)||/||b|| 8.817779977274e-04 186 KSP unpreconditioned resid norm 5.769956320566e-10 true resid norm 5.769956320566e-10 ||r(i)||/||b|| 8.652360428791e-04 187 KSP unpreconditioned resid norm 5.663593263381e-10 true resid norm 5.663593263381e-10 ||r(i)||/||b|| 8.492863292948e-04 188 KSP unpreconditioned resid norm 5.497132345835e-10 true resid norm 5.497132345835e-10 ||r(i)||/||b|| 8.243246176994e-04 189 KSP unpreconditioned resid norm 5.436252239895e-10 true resid norm 5.436252239895e-10 ||r(i)||/||b|| 8.151953177486e-04 190 KSP unpreconditioned resid norm 5.282097875316e-10 true resid norm 5.282097875316e-10 ||r(i)||/||b|| 7.920790400872e-04 191 KSP unpreconditioned resid norm 5.101630032669e-10 true resid norm 5.101630032669e-10 ||r(i)||/||b|| 7.650169145937e-04 192 KSP unpreconditioned resid norm 4.844602064548e-10 true resid norm 4.844602064548e-10 ||r(i)||/||b|| 7.264741857253e-04 193 KSP unpreconditioned resid norm 4.687980122470e-10 true resid norm 4.687980122470e-10 ||r(i)||/||b|| 7.029878815207e-04 194 KSP unpreconditioned resid norm 4.549021590181e-10 true resid norm 4.549021590181e-10 ||r(i)||/||b|| 6.821503007970e-04 195 KSP unpreconditioned resid norm 4.326734267876e-10 true resid norm 4.326734267876e-10 ||r(i)||/||b|| 6.488171189759e-04 196 KSP unpreconditioned resid norm 4.071928064828e-10 true resid norm 4.071928064828e-10 ||r(i)||/||b|| 6.106075557527e-04 197 KSP unpreconditioned resid norm 3.848523194464e-10 true resid norm 3.848523194464e-10 ||r(i)||/||b|| 5.771067915779e-04 198 KSP unpreconditioned resid norm 3.656089471441e-10 true resid norm 3.656089471441e-10 ||r(i)||/||b|| 5.482503178414e-04 199 KSP unpreconditioned resid norm 3.443600386388e-10 true resid norm 3.443600386388e-10 ||r(i)||/||b|| 5.163864344960e-04 200 KSP unpreconditioned resid norm 3.213308702898e-10 true resid norm 3.213308702898e-10 ||r(i)||/||b|| 4.818529555822e-04 201 KSP unpreconditioned resid norm 3.061822520171e-10 true resid norm 3.061822520171e-10 ||r(i)||/||b|| 4.591367861674e-04 202 KSP unpreconditioned resid norm 2.840513737626e-10 true resid norm 2.840513737626e-10 ||r(i)||/||b|| 4.259503416563e-04 203 KSP unpreconditioned resid norm 2.655386048214e-10 true resid norm 2.655386048214e-10 ||r(i)||/||b|| 3.981894470300e-04 204 KSP unpreconditioned resid norm 2.545477706690e-10 true resid norm 2.545477706690e-10 ||r(i)||/||b|| 3.817080989545e-04 205 KSP unpreconditioned resid norm 2.416392955554e-10 true resid norm 2.416392955554e-10 ||r(i)||/||b|| 3.623511449215e-04 206 KSP unpreconditioned resid norm 2.316888370466e-10 true resid norm 2.316888370466e-10 ||r(i)||/||b|| 3.474298961865e-04 207 KSP unpreconditioned resid norm 2.201886183217e-10 true resid norm 2.201886183217e-10 ||r(i)||/||b|| 3.301846984952e-04 208 KSP unpreconditioned resid norm 2.101280809866e-10 true resid norm 2.101280809865e-10 ||r(i)||/||b|| 3.150983806282e-04 209 KSP unpreconditioned resid norm 2.034372617530e-10 true resid norm 2.034372617530e-10 ||r(i)||/||b|| 3.050651366387e-04 210 KSP unpreconditioned resid norm 1.951518823483e-10 true resid norm 1.951518823483e-10 ||r(i)||/||b|| 2.926407637466e-04 211 KSP unpreconditioned resid norm 1.889694723467e-10 true resid norm 1.889694723467e-10 ||r(i)||/||b|| 2.833699067971e-04 212 KSP unpreconditioned resid norm 1.846046008460e-10 true resid norm 1.846046008460e-10 ||r(i)||/||b|| 2.768245467716e-04 213 KSP unpreconditioned resid norm 1.807735570045e-10 true resid norm 1.807735570045e-10 ||r(i)||/||b|| 2.710796900875e-04 214 KSP unpreconditioned resid norm 1.747248902848e-10 true resid norm 1.747248902848e-10 ||r(i)||/||b|| 2.620093883964e-04 215 KSP unpreconditioned resid norm 1.688026185022e-10 true resid norm 1.688026185022e-10 ||r(i)||/||b|| 2.531286227244e-04 216 KSP unpreconditioned resid norm 1.660155376002e-10 true resid norm 1.660155376002e-10 ||r(i)||/||b|| 2.489492447242e-04 217 KSP unpreconditioned resid norm 1.605106667902e-10 true resid norm 1.605106667902e-10 ||r(i)||/||b|| 2.406943943032e-04 218 KSP unpreconditioned resid norm 1.573672080294e-10 true resid norm 1.573672080294e-10 ||r(i)||/||b|| 2.359806084995e-04 219 KSP unpreconditioned resid norm 1.537783647014e-10 true resid norm 1.537783647014e-10 ||r(i)||/||b|| 2.305989445370e-04 220 KSP unpreconditioned resid norm 1.481945075699e-10 true resid norm 1.481945075700e-10 ||r(i)||/||b|| 2.222256498706e-04 221 KSP unpreconditioned resid norm 1.445378181616e-10 true resid norm 1.445378181616e-10 ||r(i)||/||b|| 2.167422470545e-04 222 KSP unpreconditioned resid norm 1.397744140894e-10 true resid norm 1.397744140894e-10 ||r(i)||/||b|| 2.095992659622e-04 223 KSP unpreconditioned resid norm 1.379541038307e-10 true resid norm 1.379541038307e-10 ||r(i)||/||b|| 2.068696126381e-04 224 KSP unpreconditioned resid norm 1.323264758884e-10 true resid norm 1.323264758884e-10 ||r(i)||/||b|| 1.984306812821e-04 225 KSP unpreconditioned resid norm 1.273432382473e-10 true resid norm 1.273432382474e-10 ||r(i)||/||b|| 1.909580479073e-04 226 KSP unpreconditioned resid norm 1.221326127921e-10 true resid norm 1.221326127921e-10 ||r(i)||/||b|| 1.831444342517e-04 227 KSP unpreconditioned resid norm 1.144060397745e-10 true resid norm 1.144060397745e-10 ||r(i)||/||b|| 1.715580216494e-04 228 KSP unpreconditioned resid norm 1.095304463108e-10 true resid norm 1.095304463108e-10 ||r(i)||/||b|| 1.642468065191e-04 229 KSP unpreconditioned resid norm 1.037480236151e-10 true resid norm 1.037480236151e-10 ||r(i)||/||b|| 1.555757520890e-04 230 KSP unpreconditioned resid norm 9.711743749788e-11 true resid norm 9.711743749788e-11 ||r(i)||/||b|| 1.456328309033e-04 231 KSP unpreconditioned resid norm 9.249562432431e-11 true resid norm 9.249562432431e-11 ||r(i)||/||b|| 1.387021729935e-04 232 KSP unpreconditioned resid norm 8.650359783535e-11 true resid norm 8.650359783535e-11 ||r(i)||/||b|| 1.297168063805e-04 233 KSP unpreconditioned resid norm 8.384337667719e-11 true resid norm 8.384337667719e-11 ||r(i)||/||b|| 1.257276614023e-04 234 KSP unpreconditioned resid norm 7.923821265547e-11 true resid norm 7.923821265548e-11 ||r(i)||/||b|| 1.188219697929e-04 235 KSP unpreconditioned resid norm 7.544526999352e-11 true resid norm 7.544526999353e-11 ||r(i)||/||b|| 1.131342478807e-04 236 KSP unpreconditioned resid norm 7.281429708004e-11 true resid norm 7.281429708005e-11 ||r(i)||/||b|| 1.091889622214e-04 237 KSP unpreconditioned resid norm 7.004299078371e-11 true resid norm 7.004299078372e-11 ||r(i)||/||b|| 1.050332390925e-04 238 KSP unpreconditioned resid norm 6.818817867592e-11 true resid norm 6.818817867591e-11 ||r(i)||/||b|| 1.022518483864e-04 239 KSP unpreconditioned resid norm 6.556448835210e-11 true resid norm 6.556448835209e-11 ||r(i)||/||b|| 9.831748336284e-05 240 KSP unpreconditioned resid norm 6.317051072221e-11 true resid norm 6.317051072220e-11 ||r(i)||/||b|| 9.472758490236e-05 241 KSP unpreconditioned resid norm 6.203479739022e-11 true resid norm 6.203479739022e-11 ||r(i)||/||b|| 9.302452156077e-05 242 KSP unpreconditioned resid norm 6.081127204964e-11 true resid norm 6.081127204963e-11 ||r(i)||/||b|| 9.118977938035e-05 243 KSP unpreconditioned resid norm 5.980425034319e-11 true resid norm 5.980425034319e-11 ||r(i)||/||b|| 8.967969606608e-05 244 KSP unpreconditioned resid norm 5.924780734871e-11 true resid norm 5.924780734870e-11 ||r(i)||/||b|| 8.884527981076e-05 245 KSP unpreconditioned resid norm 5.889713074895e-11 true resid norm 5.889713074894e-11 ||r(i)||/||b|| 8.831942135249e-05 246 KSP unpreconditioned resid norm 5.825603685666e-11 true resid norm 5.825603685665e-11 ||r(i)||/||b|| 8.735806651432e-05 247 KSP unpreconditioned resid norm 5.701780435624e-11 true resid norm 5.701780435624e-11 ||r(i)||/||b|| 8.550127015521e-05 248 KSP unpreconditioned resid norm 5.742094328471e-11 true resid norm 5.742094328471e-11 ||r(i)||/||b|| 8.610579870244e-05 249 KSP unpreconditioned resid norm 5.690167124326e-11 true resid norm 5.690167124326e-11 ||r(i)||/||b|| 8.532712229423e-05 250 KSP unpreconditioned resid norm 5.729142147412e-11 true resid norm 5.729142147411e-11 ||r(i)||/||b|| 8.591157376789e-05 251 KSP unpreconditioned resid norm 5.588301692231e-11 true resid norm 5.588301692231e-11 ||r(i)||/||b|| 8.379959524765e-05 252 KSP unpreconditioned resid norm 5.496143343772e-11 true resid norm 5.496143343772e-11 ||r(i)||/||b|| 8.241763115107e-05 253 KSP unpreconditioned resid norm 5.360155316398e-11 true resid norm 5.360155316398e-11 ||r(i)||/||b|| 8.037841740061e-05 254 KSP unpreconditioned resid norm 5.111714300080e-11 true resid norm 5.111714300080e-11 ||r(i)||/||b|| 7.665291048332e-05 255 KSP unpreconditioned resid norm 4.913138711663e-11 true resid norm 4.913138711663e-11 ||r(i)||/||b|| 7.367516252842e-05 256 KSP unpreconditioned resid norm 4.683086623154e-11 true resid norm 4.683086623156e-11 ||r(i)||/||b|| 7.022540749289e-05 257 KSP unpreconditioned resid norm 4.429151937402e-11 true resid norm 4.429151937403e-11 ||r(i)||/||b|| 6.641752004204e-05 258 KSP unpreconditioned resid norm 4.218203543106e-11 true resid norm 4.218203543107e-11 ||r(i)||/||b|| 6.325423519565e-05 259 KSP unpreconditioned resid norm 3.919814151602e-11 true resid norm 3.919814151603e-11 ||r(i)||/||b|| 5.877972547671e-05 260 KSP unpreconditioned resid norm 3.755684833052e-11 true resid norm 3.755684833052e-11 ||r(i)||/||b|| 5.631851790054e-05 261 KSP unpreconditioned resid norm 3.489946224172e-11 true resid norm 3.489946224172e-11 ||r(i)||/||b|| 5.233362426161e-05 262 KSP unpreconditioned resid norm 3.330600965236e-11 true resid norm 3.330600965237e-11 ||r(i)||/||b|| 4.994415623736e-05 263 KSP unpreconditioned resid norm 3.114834935352e-11 true resid norm 3.114834935353e-11 ||r(i)||/||b|| 4.670862834923e-05 264 KSP unpreconditioned resid norm 2.955675347019e-11 true resid norm 2.955675347019e-11 ||r(i)||/||b|| 4.432194455571e-05 265 KSP unpreconditioned resid norm 2.800280306837e-11 true resid norm 2.800280306837e-11 ||r(i)||/||b|| 4.199171219033e-05 266 KSP unpreconditioned resid norm 2.641330971692e-11 true resid norm 2.641330971691e-11 ||r(i)||/||b|| 3.960818125666e-05 267 KSP unpreconditioned resid norm 2.582570881596e-11 true resid norm 2.582570881595e-11 ||r(i)||/||b|| 3.872704204157e-05 268 KSP unpreconditioned resid norm 2.446997332262e-11 true resid norm 2.446997332261e-11 ||r(i)||/||b|| 3.669404361269e-05 269 KSP unpreconditioned resid norm 2.402233706978e-11 true resid norm 2.402233706977e-11 ||r(i)||/||b|| 3.602278892973e-05 270 KSP unpreconditioned resid norm 2.327147953669e-11 true resid norm 2.327147953668e-11 ||r(i)||/||b|| 3.489683759736e-05 271 KSP unpreconditioned resid norm 2.222032490472e-11 true resid norm 2.222032490471e-11 ||r(i)||/||b|| 3.332057458305e-05 272 KSP unpreconditioned resid norm 2.180011314458e-11 true resid norm 2.180011314457e-11 ||r(i)||/||b|| 3.269044440474e-05 273 KSP unpreconditioned resid norm 2.099596075971e-11 true resid norm 2.099596075970e-11 ||r(i)||/||b|| 3.148457457021e-05 274 KSP unpreconditioned resid norm 2.034091756086e-11 true resid norm 2.034091756086e-11 ||r(i)||/||b|| 3.050230199518e-05 275 KSP unpreconditioned resid norm 1.906005782131e-11 true resid norm 1.906005782130e-11 ||r(i)||/||b|| 2.858158379392e-05 276 KSP unpreconditioned resid norm 1.837712668014e-11 true resid norm 1.837712668014e-11 ||r(i)||/||b|| 2.755749174657e-05 277 KSP unpreconditioned resid norm 1.782164784199e-11 true resid norm 1.782164784198e-11 ||r(i)||/||b|| 2.672452129563e-05 278 KSP unpreconditioned resid norm 1.726951539224e-11 true resid norm 1.726951539223e-11 ||r(i)||/||b|| 2.589656893443e-05 279 KSP unpreconditioned resid norm 1.666433804438e-11 true resid norm 1.666433804438e-11 ||r(i)||/||b|| 2.498907289007e-05 280 KSP unpreconditioned resid norm 1.622753878977e-11 true resid norm 1.622753878977e-11 ||r(i)||/||b|| 2.433406886995e-05 281 KSP unpreconditioned resid norm 1.573254775328e-11 true resid norm 1.573254775328e-11 ||r(i)||/||b|| 2.359180313711e-05 282 KSP unpreconditioned resid norm 1.552412747263e-11 true resid norm 1.552412747262e-11 ||r(i)||/||b|| 2.327926569511e-05 283 KSP unpreconditioned resid norm 1.475437057857e-11 true resid norm 1.475437057856e-11 ||r(i)||/||b|| 2.212497375251e-05 284 KSP unpreconditioned resid norm 1.430642953266e-11 true resid norm 1.430642953265e-11 ||r(i)||/||b|| 2.145326201593e-05 285 KSP unpreconditioned resid norm 1.367828633121e-11 true resid norm 1.367828633118e-11 ||r(i)||/||b|| 2.051132743653e-05 286 KSP unpreconditioned resid norm 1.296838514211e-11 true resid norm 1.296838514209e-11 ||r(i)||/||b|| 1.944679234898e-05 287 KSP unpreconditioned resid norm 1.218327263661e-11 true resid norm 1.218327263660e-11 ||r(i)||/||b|| 1.826947383958e-05 288 KSP unpreconditioned resid norm 1.142259366824e-11 true resid norm 1.142259366823e-11 ||r(i)||/||b|| 1.712879473574e-05 289 KSP unpreconditioned resid norm 1.054990782334e-11 true resid norm 1.054990782333e-11 ||r(i)||/||b|| 1.582015528482e-05 290 KSP unpreconditioned resid norm 9.906690120787e-12 true resid norm 9.906690120776e-12 ||r(i)||/||b|| 1.485561567872e-05 291 KSP unpreconditioned resid norm 9.368888463731e-12 true resid norm 9.368888463718e-12 ||r(i)||/||b|| 1.404915311340e-05 292 KSP unpreconditioned resid norm 8.624978989462e-12 true resid norm 8.624978989449e-12 ||r(i)||/||b|| 1.293362076962e-05 293 KSP unpreconditioned resid norm 8.262924522708e-12 true resid norm 8.262924522695e-12 ||r(i)||/||b|| 1.239070058666e-05 294 KSP unpreconditioned resid norm 7.640482509450e-12 true resid norm 7.640482509441e-12 ||r(i)||/||b|| 1.145731524620e-05 295 KSP unpreconditioned resid norm 7.212352955020e-12 true resid norm 7.212352955018e-12 ||r(i)||/||b|| 1.081531190870e-05 296 KSP unpreconditioned resid norm 6.680200564344e-12 true resid norm 6.680200564353e-12 ||r(i)||/||b|| 1.001732072275e-05 297 KSP unpreconditioned resid norm 6.271514053301e-12 true resid norm 6.271514053312e-12 ||r(i)||/||b|| 9.404473276525e-06 KSP Object: 128 MPI processes type: cg maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning has attached null space using UNPRECONDITIONED norm type for convergence test PC Object: 128 MPI processes type: mg MG: type is MULTIPLICATIVE, levels=5 cycles=v Cycles per PCApply=1 Using Galerkin computed coarse grid matrices Coarse grid solver -- level ------------------------------- KSP Object: (mg_coarse_) 128 MPI processes type: preonly maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (mg_coarse_) 128 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: natural factor fill ratio given 0, needed 0 Factored matrix follows: Matrix Object: 128 MPI processes type: mpiaij rows=1024, cols=1024 package used to perform factorization: superlu_dist total: nonzeros=0, allocated nonzeros=0 total number of mallocs used during MatSetValues calls =0 SuperLU_DIST run parameters: Process grid nprow 16 x npcol 8 Equilibrate matrix TRUE Matrix input mode 1 Replace tiny pivots TRUE Use iterative refinement FALSE Processors in row 16 col partition 8 Row permutation LargeDiag Column permutation METIS_AT_PLUS_A Parallel symbolic factorization FALSE Repeated factorization SamePattern_SameRowPerm linear system matrix = precond matrix: Matrix Object: 128 MPI processes type: mpiaij rows=1024, cols=1024 total: nonzeros=27648, allocated nonzeros=27648 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Down solver (pre-smoother) on level 1 ------------------------------- KSP Object: (mg_levels_1_) 128 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (mg_levels_1_) 128 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 128 MPI processes type: mpiaij rows=8192, cols=8192 total: nonzeros=221184, allocated nonzeros=221184 total number of mallocs used during MatSetValues calls =0 using I-node (on process 0) routines: found 16 nodes, limit used is 5 Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 2 ------------------------------- KSP Object: (mg_levels_2_) 128 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (mg_levels_2_) 128 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 128 MPI processes type: mpiaij rows=65536, cols=65536 total: nonzeros=1769472, allocated nonzeros=1769472 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 3 ------------------------------- KSP Object: (mg_levels_3_) 128 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (mg_levels_3_) 128 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 128 MPI processes type: mpiaij rows=524288, cols=524288 total: nonzeros=14155776, allocated nonzeros=14155776 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 4 ------------------------------- KSP Object: (mg_levels_4_) 128 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1 tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using nonzero initial guess using NONE norm type for convergence test PC Object: (mg_levels_4_) 128 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 128 MPI processes type: mpiaij rows=4194304, cols=4194304 total: nonzeros=29360128, allocated nonzeros=29360128 total number of mallocs used during MatSetValues calls =0 Up solver (post-smoother) same as down solver (pre-smoother) linear system matrix = precond matrix: Matrix Object: 128 MPI processes type: mpiaij rows=4194304, cols=4194304 total: nonzeros=29360128, allocated nonzeros=29360128 total number of mallocs used during MatSetValues calls =0 ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./hit on a interlagos-64idx-pgi-opt named nid22041 with 128 processors, by Unknown Tue Sep 24 13:37:33 2013 Using Petsc Release Version 3.4.2, Jul, 02, 2013 Max Max/Min Avg Total Time (sec): 3.240e+03 1.00001 3.240e+03 Objects: 5.351e+05 1.00000 5.351e+05 Flops: 4.066e+11 1.00000 4.066e+11 5.205e+13 Flops/sec: 1.255e+08 1.00001 1.255e+08 1.606e+10 MPI Messages: 7.810e+06 1.00000 7.810e+06 9.996e+08 MPI Message Lengths: 2.989e+10 1.00000 3.827e+03 3.826e+12 MPI Reductions: 1.522e+06 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 1.0613e+03 32.8% 1.5304e+13 29.4% 9.157e+07 9.2% 1.500e+03 39.2% 8.127e+05 53.4% 1: MG Apply: 2.1785e+03 67.2% 3.6743e+13 70.6% 9.081e+08 90.8% 2.327e+03 60.8% 7.094e+05 46.6% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %f - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage VecTDot 177358 1.0 6.0650e+01 2.0 1.16e+10 1.0 0.0e+00 0.0e+00 1.8e+05 1 3 0 0 12 4 10 0 0 22 24530 VecNorm 179491 1.0 4.7175e+01 3.2 1.18e+10 1.0 0.0e+00 0.0e+00 1.8e+05 1 3 0 0 12 3 10 0 0 22 31917 VecCopy 90812 1.0 7.8039e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecSet 2155 1.0 4.0682e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 177358 1.0 2.6918e+01 1.2 1.16e+10 1.0 0.0e+00 0.0e+00 0.0e+00 1 3 0 0 0 2 10 0 0 0 55270 VecAYPX 177358 1.0 2.6435e+01 1.3 8.69e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 2 7 0 0 0 42098 VecScatterBegin 179495 1.0 2.1807e+01 1.3 0.00e+00 0.0 9.2e+07 1.6e+04 0.0e+00 1 0 9 39 0 2 0100100 0 0 VecScatterEnd 179495 1.0 8.7667e+01 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 5 0 0 0 0 0 MatMult 178069 1.0 6.3495e+02 1.1 7.59e+10 1.0 9.1e+07 1.6e+04 0.0e+00 19 19 9 39 0 57 63100100 0 15291 MatMultTranspose 4 1.0 2.2790e-03 1.1 2.53e+05 1.0 1.5e+03 9.9e+02 0.0e+00 0 0 0 0 0 0 0 0 0 0 14194 MatLUFactorSym 1 1.0 4.7803e-04 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatLUFactorNum 1 1.0 1.1906e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 728 1.0 2.9026e+0014.8 0.00e+00 0.0 0.0e+00 0.0e+00 1.4e+03 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 728 1.0 2.6438e+00 1.2 0.00e+00 0.0 1.2e+04 1.1e+03 7.2e+01 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 1 1.0 4.0531e-06 4.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 1.7166e-05 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatView 4977 1.0 1.2820e+00 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 5.0e+03 0 0 0 0 0 0 0 0 0 1 0 MatPtAP 4 1.0 2.1180e-01 1.0 5.11e+06 1.0 2.5e+04 6.0e+03 1.0e+02 0 0 0 0 0 0 0 0 0 0 3085 MatPtAPSymbolic 4 1.0 1.4372e-01 1.1 0.00e+00 0.0 1.5e+04 7.8e+03 6.0e+01 0 0 0 0 0 0 0 0 0 0 0 MatPtAPNumeric 4 1.0 7.5788e-02 1.1 5.11e+06 1.0 9.7e+03 3.1e+03 4.0e+01 0 0 0 0 0 0 0 0 0 0 8622 MatGetLocalMat 4 1.0 2.0369e-02 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetBrAoCol 4 1.0 2.8472e-02 3.0 0.00e+00 0.0 1.1e+04 8.4e+03 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetSymTrans 8 1.0 9.3176e-03 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSetUp 6 1.0 1.0706e-02 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.2e+01 0 0 0 0 0 0 0 0 0 0 0 Warning -- total time of even greater than time of entire stage -- something is wrong with the timer KSPSolve 711 1.0 3.0211e+03 1.0 4.07e+11 1.0 1.0e+09 3.8e+03 1.5e+06 93100100100 99 2853401091255186 17228 PCSetUp 1 1.0 4.0822e-01 1.0 5.36e+06 1.0 3.4e+04 4.6e+03 3.0e+02 0 0 0 0 0 0 0 0 0 0 1680 Warning -- total time of even greater than time of entire stage -- something is wrong with the timer PCApply 88679 1.0 2.2076e+03 1.0 2.87e+11 1.0 9.1e+08 2.6e+03 7.1e+05 67 71 91 61 47 205240992155 87 16644 MGSetup Level 0 1 1.0 1.2105e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01 0 0 0 0 0 0 0 0 0 0 0 MGSetup Level 1 1 1.0 2.0220e-0321.5 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0 MGSetup Level 2 1 1.0 2.7680e-04 2.8 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0 MGSetup Level 3 1 1.0 1.4710e-04 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0 MGSetup Level 4 1 1.0 1.6508e-03 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0 --- Event Stage 1: MG Apply VecScale 354716 1.0 2.2152e+00 1.4 1.00e+09 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 58045 VecCopy 88679 1.0 8.2399e-02 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 1241506 1.0 1.8962e+01 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecAYPX 354716 1.0 1.5960e+01 1.5 3.32e+09 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 1 1 0 0 0 26627 VecScatterBegin 1418864 1.0 6.7544e+01 1.3 0.00e+00 0.0 9.1e+08 2.6e+03 0.0e+00 2 0 91 61 0 3 0100100 0 0 VecScatterEnd 1418864 1.0 1.7830e+02 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 4 0 0 0 0 6 0 0 0 0 0 MatMult 354716 1.0 4.1424e+02 1.4 5.97e+10 1.0 3.2e+08 3.2e+03 0.0e+00 11 15 32 27 0 16 21 35 44 0 18458 MatMultAdd 354716 1.0 1.6205e+02 1.1 2.24e+10 1.0 1.4e+08 9.9e+02 0.0e+00 5 6 14 4 0 7 8 15 6 0 17702 MatMultTranspose 354716 1.0 1.9092e+02 1.4 2.24e+10 1.0 1.4e+08 9.9e+02 0.0e+00 5 6 14 4 0 8 8 15 6 0 15025 MatSolve 88679 1.0 1.6955e+02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 4 0 0 0 0 6 0 0 0 0 0 MatSOR 709432 1.0 1.3774e+03 1.1 1.79e+11 1.0 3.2e+08 3.2e+03 7.1e+05 41 44 32 27 47 60 62 35 44100 16651 KSPSolve 798111 1.0 1.5030e+03 1.1 1.79e+11 1.0 3.2e+08 3.2e+03 7.1e+05 45 44 32 27 47 67 62 35 44100 15259 PCApply 88679 1.0 1.6968e+02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 4 0 0 0 0 6 0 0 0 0 0 MGSmooth Level 0 88679 1.0 1.7100e+02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 4 0 0 0 0 6 0 0 0 0 0 MGSmooth Level 1 177358 1.0 2.1576e+01 1.2 5.96e+08 1.0 9.1e+07 1.9e+02 1.8e+05 1 0 9 0 12 1 0 10 1 25 3535 MGResid Level 1 88679 1.0 4.6774e+01 8.6 3.06e+08 1.0 9.1e+07 1.9e+02 0.0e+00 0 0 9 0 0 1 0 10 1 0 839 MGInterp Level 1 177358 1.0 2.6532e+01 8.3 7.66e+07 1.0 6.8e+07 6.4e+01 0.0e+00 0 0 7 0 0 0 0 8 0 0 370 MGSmooth Level 2 177358 1.0 3.5809e+01 1.2 5.91e+09 1.0 9.1e+07 6.4e+02 1.8e+05 1 1 9 2 12 2 2 10 2 25 21139 MGResid Level 2 88679 1.0 5.0840e+01 5.0 2.45e+09 1.0 9.1e+07 6.4e+02 0.0e+00 1 1 9 2 0 1 1 10 2 0 6173 MGInterp Level 2 177358 1.0 3.3743e+01 5.4 6.13e+08 1.0 6.8e+07 2.1e+02 0.0e+00 0 0 7 0 0 1 0 8 1 0 2325 MGSmooth Level 3 177358 1.0 2.6888e+02 1.2 5.28e+10 1.0 9.1e+07 2.3e+03 1.8e+05 8 13 9 5 12 12 18 10 9 25 25126 MGResid Level 3 88679 1.0 7.7293e+01 1.5 1.96e+10 1.0 9.1e+07 2.3e+03 0.0e+00 2 5 9 5 0 3 7 10 9 0 32482 MGInterp Level 3 177358 1.0 5.8751e+01 2.2 4.90e+09 1.0 6.8e+07 7.7e+02 0.0e+00 1 1 7 1 0 2 2 8 2 0 10683 MGSmooth Level 4 177358 1.0 1.1098e+03 1.1 1.20e+11 1.0 4.5e+07 1.6e+04 1.8e+05 32 29 5 19 12 47 42 5 32 25 13828 MGResid Level 4 88679 1.0 3.4958e+02 1.5 4.07e+10 1.0 4.5e+07 1.6e+04 0.0e+00 8 10 5 19 0 12 14 5 32 0 14896 MGInterp Level 4 177358 1.0 2.8219e+02 1.1 3.92e+10 1.0 6.8e+07 2.9e+03 0.0e+00 8 10 7 5 0 12 14 8 8 0 17794 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Vector 180268 180268 47652462624 0 Vector Scatter 19 19 22572 0 Matrix 38 38 14004608 0 Matrix Null Space 1 1 652 0 Distributed Mesh 5 5 830792 0 Bipartite Graph 10 10 8560 0 Index Set 47 47 534480 0 IS L to G Mapping 5 5 405756 0 Krylov Solver 7 7 9536 0 DMKSP interface 3 3 2088 0 Preconditioner 7 7 7352 0 Viewer 1 0 0 0 --- Event Stage 1: MG Apply Vector 354716 354716 27159894688 0 ======================================================================================================================== Average time to get PetscTime(): 1.90735e-07 Average time for MPI_Barrier(): 1.28269e-05 Average time for zero size MPI_Send(): 2.74926e-06 #PETSc Option Table entries: -ksp_monitor_true_residual -ksp_view -log_summary -mg_coarse_pc_factor_mat_solver_package superlu_dist -mg_coarse_pc_type lu -mg_levels_ksp_max_it 1 -mg_levels_ksp_type richardson -options_left -pc_mg_galerkin -pc_mg_levels 5 -pc_mg_log -pc_type mg #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 8 Configure run at: Wed Aug 28 23:25:43 2013 Configure options: --known-level1-dcache-size=16384 --known-level1-dcache-linesize=64 --known-level1-dcache-assoc=4 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=0 --known-mpi-c-double-complex=0 --with-batch="1 " --known-mpi-shared="0 " --known-memcmp-ok --with-blas-lapack-lib="-L/opt/acml/5.3.0/pgi64/lib -lacml" --COPTFLAGS="-O3 -fastsse" --FOPTFLAGS="-O3 -fastsse" --CXXOPTFLAGS="-O3 -fastsse" --with-x="0 " --with-debugging="0 " --with-clib-autodetect="0 " --with-cxxlib-autodetect="0 " --with-fortranlib-autodetect="0 " --with-shared-libraries=0 --with-dynamic-loading=0 --with-mpi-compilers="1 " --known-mpi-shared-libraries=0 --with-64-bit-indices --download-blacs="1 " --download-scalapack="1 " --download-superlu_dist="1 " --download-metis="1 " --download-parmetis="1 " --with-cc=cc --with-cxx=CC --with-fc=ftn PETSC_ARCH=interlagos-64idx-pgi-opt ----------------------------------------- Libraries compiled on Wed Aug 28 23:25:43 2013 on h2ologin3 Machine characteristics: Linux-2.6.32.59-0.7-default-x86_64-with-SuSE-11-x86_64 Using PETSc directory: /mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2 Using PETSc arch: interlagos-64idx-pgi-opt ----------------------------------------- Using C compiler: cc -O3 -fastsse ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: ftn -O3 -fastsse ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/include -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/include -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/include -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/include -I/opt/cray/udreg/2.3.2-1.0402.7311.2.1.gem/include -I/opt/cray/ugni/5.0-1.0402.7128.7.6.gem/include -I/opt/cray/pmi/4.0.1-1.0000.9421.73.3.gem/include -I/opt/cray/dmapp/4.0.1-1.0402.7439.5.1.gem/include -I/opt/cray/gni-headers/2.1-1.0402.7082.6.2.gem/include -I/opt/cray/xpmem/0.1-2.0402.44035.2.1.gem/include -I/opt/cray/rca/1.0.0-2.0402.42153.2.106.gem/include -I/opt/cray-hss-devel/7.0.0/include -I/opt/cray/krca/1.0.0-2.0402.42157.2.94.gem/include -I/opt/cray/mpt/6.0.1/gni/mpich2-pgi/121/include -I/opt/acml/5.3.0/pgi64_fma4/include -I/opt/cray/libsci/12.1.01/pgi/121/interlagos/include -I/opt/fftw/3.3.0.3/interlagos/include -I/usr/include/alps -I/opt/pgi/13.6.0/linux86-64/13.6/include -I/opt/cray/xe-sysroot/4.2.24/usr/include ----------------------------------------- Using C linker: cc Using Fortran linker: ftn Using libraries: -Wl,-rpath,/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -L/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -lpetsc -Wl,-rpath,/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -L/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -lsuperlu_dist_3.3 -L/opt/acml/5.3.0/pgi64/lib -lacml -lpthread -lparmetis -lmetis -ldl ----------------------------------------- #PETSc Option Table entries: -ksp_monitor_true_residual -ksp_view -log_summary -mg_coarse_pc_factor_mat_solver_package superlu_dist -mg_coarse_pc_type lu -mg_levels_ksp_max_it 1 -mg_levels_ksp_type richardson -options_left -pc_mg_galerkin -pc_mg_levels 5 -pc_mg_log -pc_type mg #End of PETSc Option Table entries There are no unused options. Application 2156772 resources: utime ~413688s, stime ~1158s, Rss ~62444, inblocks ~140390, outblocks ~13085369 From bsmith at mcs.anl.gov Tue Sep 24 19:08:20 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 24 Sep 2013 19:08:20 -0500 Subject: [petsc-users] How to speed up geometric multigrid In-Reply-To: <52420449.4010603@uci.edu> References: <52406460.3060905@uci.edu> <4499926F-40CE-462C-BFD6-8798A02126FB@mcs.anl.gov> <52420449.4010603@uci.edu> Message-ID: <60076751-799B-410A-9779-D697677A869D@mcs.anl.gov> Thanks. The balance of work on the different levels and across processes looks ok. So it is is a matter of improving the convergence rate. The initial residual norm is very small. Are you sure you need to decrease it to 10^-12 ???? Start with a really robust multigrid smoother use -pc_mg_type full -ksp_type fgmres -mg_levels_pc_type bjacobi -mg_levels_ksp_type gmres -mg_levels_ksp_max_it 3 PLUS -mg_coarse_pc_factor_mat_solver_package superlu_dist -mg_coarse_pc_type lu -pc_mg_galerkin -pc_mg_levels 5 -pc_mg_log -pc_type mg run with the -log_summary and -pc_mg_log Now back off a little on the smoother and use -mg_levels_pc_type sor instead how does that change the convergence and time. Back off even more an replace the -ksp_type fgmres with -ksp_type cg and the -mg_levels_ksp_type gmres with -mg_levels_ksp_type richardson how does that change the convergence and the time? There are some additional variants we can try based on the results from above. Barry On Sep 24, 2013, at 4:29 PM, Michele Rosso wrote: > Barry, > > I re-rerun the test case with the option -pc_mg_log as you suggested. > I attached the new output ("final_new.txt'). > Thanks for your help. > > Michele > > On 09/23/2013 09:35 AM, Barry Smith wrote: >> Run with the additional option -pc_mg_log and send us the log file. >> >> Barry >> >> Maybe we should make this the default somehow. >> >> >> On Sep 23, 2013, at 10:55 AM, Michele Rosso wrote: >> >>> Hi, >>> >>> I am successfully using PETSc to solve a 3D Poisson's equation with CG + MG . Such equation arises from a projection algorithm for a multiphase incompressible flow simulation. >>> I set up the solver as I was suggested to do in a previous thread (title: "GAMG speed") and run a test case (liquid droplet with surface tension falling under the effect of gravity in a quiescent fluid). >>> The solution of the Poisson Equation via multigrid is correct but it becomes progressively slower and slower as the simulation progresses (I am performing successive solves) due to an increase in the number of iterations. >>> Since the solution of the Poisson equation is mission-critical, I need to speed it up as much as I can. >>> Could you please help me out with this? >>> >>> I run the test case with the following options: >>> >>> -pc_type mg -pc_mg_galerkin -pc_mg_levels 5 -mg_levels_ksp_type richardson -mg_levels_ksp_max_it 1 >>> -mg_coarse_pc_type lu -mg_coarse_pc_factor_mat_solver_package superlu_dist >>> -log_summary -ksp_view -ksp_monitor_true_residual -options_left >>> >>> Please find the diagnostic for the final solve in the attached file "final.txt'. >>> Thank you, >>> >>> Michele >>> >> > > From mrosso at uci.edu Tue Sep 24 19:43:10 2013 From: mrosso at uci.edu (Michele Rosso) Date: Tue, 24 Sep 2013 17:43:10 -0700 Subject: [petsc-users] How to speed up geometric multigrid In-Reply-To: <60076751-799B-410A-9779-D697677A869D@mcs.anl.gov> References: <52406460.3060905@uci.edu> <4499926F-40CE-462C-BFD6-8798A02126FB@mcs.anl.gov> <52420449.4010603@uci.edu> <60076751-799B-410A-9779-D697677A869D@mcs.anl.gov> Message-ID: <5242319E.3060003@uci.edu> Barry, thank you! I absolutely do not need such accuracy!!! In my code I set an accuracy of 10^(-5) for the rtol instead of atol. Thanks for pointing that out. I only need to solve up to discretization error. Anyway I will try all the variants (this time by setting the proper tolerance!) you suggested and let you know. Thank you very much, Michele On 09/24/2013 05:08 PM, Barry Smith wrote: > Thanks. The balance of work on the different levels and across processes looks ok. So it is is a matter of improving the convergence rate. > > The initial residual norm is very small. Are you sure you need to decrease it to 10^-12 ???? > > Start with a really robust multigrid smoother use > > -pc_mg_type full -ksp_type fgmres -mg_levels_pc_type bjacobi -mg_levels_ksp_type gmres -mg_levels_ksp_max_it 3 PLUS -mg_coarse_pc_factor_mat_solver_package superlu_dist > -mg_coarse_pc_type lu -pc_mg_galerkin -pc_mg_levels 5 -pc_mg_log -pc_type mg > > run with the -log_summary and -pc_mg_log > > Now back off a little on the smoother and use -mg_levels_pc_type sor instead how does that change the convergence and time. > > Back off even more an replace the -ksp_type fgmres with -ksp_type cg and the -mg_levels_ksp_type gmres with -mg_levels_ksp_type richardson how does that change the convergence and the time? > > There are some additional variants we can try based on the results from above. > > Barry > > > > On Sep 24, 2013, at 4:29 PM, Michele Rosso wrote: > >> Barry, >> >> I re-rerun the test case with the option -pc_mg_log as you suggested. >> I attached the new output ("final_new.txt'). >> Thanks for your help. >> >> Michele >> >> On 09/23/2013 09:35 AM, Barry Smith wrote: >>> Run with the additional option -pc_mg_log and send us the log file. >>> >>> Barry >>> >>> Maybe we should make this the default somehow. >>> >>> >>> On Sep 23, 2013, at 10:55 AM, Michele Rosso wrote: >>> >>>> Hi, >>>> >>>> I am successfully using PETSc to solve a 3D Poisson's equation with CG + MG . Such equation arises from a projection algorithm for a multiphase incompressible flow simulation. >>>> I set up the solver as I was suggested to do in a previous thread (title: "GAMG speed") and run a test case (liquid droplet with surface tension falling under the effect of gravity in a quiescent fluid). >>>> The solution of the Poisson Equation via multigrid is correct but it becomes progressively slower and slower as the simulation progresses (I am performing successive solves) due to an increase in the number of iterations. >>>> Since the solution of the Poisson equation is mission-critical, I need to speed it up as much as I can. >>>> Could you please help me out with this? >>>> >>>> I run the test case with the following options: >>>> >>>> -pc_type mg -pc_mg_galerkin -pc_mg_levels 5 -mg_levels_ksp_type richardson -mg_levels_ksp_max_it 1 >>>> -mg_coarse_pc_type lu -mg_coarse_pc_factor_mat_solver_package superlu_dist >>>> -log_summary -ksp_view -ksp_monitor_true_residual -options_left >>>> >>>> Please find the diagnostic for the final solve in the attached file "final.txt'. >>>> Thank you, >>>> >>>> Michele >>>> >> > From hariseldon99 at gmail.com Wed Sep 25 00:01:24 2013 From: hariseldon99 at gmail.com (Analabha Roy) Date: Wed, 25 Sep 2013 10:31:24 +0530 Subject: [petsc-users] Fwd: Building the same petsc matrix with different numprocs gives different results! In-Reply-To: References: <87zjr2zlih.fsf@mcs.anl.gov> Message-ID: There is one thing In the code, the evaluation of each element of AVG_BIBJ requires a read-only matrix U_parallel that I input from another program, and a writeable sequential vector BDB_AA that is different for each element. I sequentiate U_parallel to U_seq by using MatCopy here in lines 242+ and each process is supposed to update its copy of BDB_AA at every loop iteration here in line 347+ Is this right? Or are sequential vectors/matrices handled by the root process only? I know how to scatter a parallel vector to all processes using PetSc scatter contexts but don't see any way to do so to a matrix other than MatCopy. How do I ensure that each process has its own private writeable copy of a sequential vector? On Tue, Sep 24, 2013 at 11:48 PM, Analabha Roy wrote: > > > > On Tue, Sep 24, 2013 at 11:35 PM, Matthew Knepley wrote: > >> On Tue, Sep 24, 2013 at 10:58 AM, Analabha Roy wrote: >> >>> Hi, >>> >>> Sorry for misunderstanding >>> >>> I modified my source thus so that the >>> rows/cols/values for each call are printed before inserting into >>> MatSetValues() >>> >>> >>> Then ran it with 1,2 processors >>> >>> >>> Here are the outputs >>> >>> >>> Strange! Running it with 2 procs and only half the values show up!!!!! >>> >> >> PetscPrintf() only prints from rank 0. Use PETSC_COMM_SELF. >> >> > > > Sorry. Modified accordingly and here is new output(I manually reordered the output of the 2 procs case since the order in > which it was printed was haphazard) > > > All the elements do not match. > > > >> >> Matt >> >> >>> And even those do not match!!!! >>> >>> >>> >>> >>> On Tue, Sep 24, 2013 at 11:12 PM, Matthew Knepley wrote: >>> >>>> On Tue, Sep 24, 2013 at 10:39 AM, Analabha Roy wrote: >>>> >>>>> Hi, >>>>> >>>>> >>>>> >>>>> On Tue, Sep 24, 2013 at 9:33 PM, Matthew Knepley wrote: >>>>> >>>>>> On Tue, Sep 24, 2013 at 8:35 AM, Analabha Roy >>>>> > wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Sep 24, 2013 at 8:41 PM, Matthew Knepley wrote: >>>>>>> >>>>>>>> On Tue, Sep 24, 2013 at 8:08 AM, Analabha Roy < >>>>>>>> hariseldon99 at gmail.com> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Sep 24, 2013 at 1:42 PM, Jed Brown wrote: >>>>>>>>> >>>>>>>>>> Analabha Roy writes: >>>>>>>>>> >>>>>>>>>> > Hi all, >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > Compiling and running this >>>>>>>>>> > code< >>>>>>>>>> https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c >>>>>>>>>> >that >>>>>>>>>> > builds a petsc matrix gives different results when run with >>>>>>>>>> different >>>>>>>>>> > number of processors. >>>>>>>>>> >>>>>>>>>> >>>>>>>>> Thanks for the reply. >>>>>>>>> >>>>>>>>> >>>>>>>>>> Uh, if you call rand() on different processors, why would you >>>>>>>>>> expect it >>>>>>>>>> to give the same results? >>>>>>>>>> >>>>>>>>>> Right, I get that. The rand() was a placeholder. >>>>>>>>> >>>>>>>>> This original much larger codereplicates the same loop structure and runs the same Petsc subroutines, but >>>>>>>>> running it by >>>>>>>>> >>>>>>>>> mpirun -np $N ./eth -lattice_size 5 -vector_size 1 -repulsion 0.0 >>>>>>>>> -draw_out -draw_pause -1 >>>>>>>>> >>>>>>>>> with N=1,2,3,4 gives different results for the matrix dumped out >>>>>>>>> by lines 514-519. >>>>>>>>> The matrix itself is evaluated in parallel, created in lines263-275 >>>>>>>>> and >>>>>>>>> evaluated in lines 294-356 >>>>>>>>> >>>>>>>>> (you can click on the line numbers above to navigate directly to >>>>>>>>> them) >>>>>>>>> >>>>>>>>> Here is a sample of the >>>>>>>>> output of lines 514-519 for N=1,2,3,4 procs left to right. >>>>>>>>> >>>>>>>>> Thty're different for different procs. They should be the same, >>>>>>>>> since none of my input parameters are numprocs dependent, and I don't >>>>>>>>> explicitly use the size or rank anywhere in the code. >>>>>>>>> >>>>>>>> >>>>>>>> You are likely not dividing the rows you loop over so you are >>>>>>>> redundantly computing. >>>>>>>> >>>>>>> >>>>>>> Thanks for the reply. >>>>>>> >>>>>>> Line 274gets the local row indices of Petsc Matrix >>>>>>> AVG_BDIBJ >>>>>>> >>>>>>> Line 295 >>>>>>> iterates >>>>>>> over the local rows and the lines below get the column >>>>>>> elements. For each row, the column elements are assigned by the >>>>>>> lines up to Line 344and stored locally in colvalues[]. Dunno if the details are relevant. >>>>>>> >>>>>>> Line 347inserts the sitestride1^th row into the matrix >>>>>>> >>>>>>> Line 353+does the mat assembly >>>>>>> >>>>>>> Then, after a lot of currently irrelevant code, >>>>>>> >>>>>>> Line 514+dumps the mat plot to graphics >>>>>>> >>>>>>> >>>>>>> Different numprocs give different matrices. >>>>>>> >>>>>>> Can somebody suggest what I did wrong (or didn't do)? >>>>>>> >>>>>> >>>>>> Different values are being given to MatSetValues() for different >>>>>> numbers of processes. So >>>>>> >>>>>> 1) Reduce this to the smallest problem size possible >>>>>> >>>>>> 2) Print out all rows/cols/values for each call >>>>>> >>>>>> 3) Compare 2 procs to the serial case >>>>>> >>>>>> >>>>> >>>>> Thanks for your excellent suggestion. >>>>> >>>>> I modified my codeto dump the matrix in binary >>>>> >>>>> Then I used this python script I hadto convert to ascii >>>>> >>>> >>>> Do not print the matrix, print the data you are passing to >>>> MatSetValues(). >>>> >>>> MatSetValues() is not likely to be broken. Every PETSc code in the >>>> world calls this many times on every simulation. >>>> >>>> Matt >>>> >>>> >>>>> >>>>> Here are the values of AVG_BDIBJ, >>>>> a 9X9 matrix (the smallest possible problem size) run with the exact same >>>>> input parameters with 1,2,3 and 4 procs >>>>> >>>>> As you can see, the 1 and 2 procs match up, but the 3 and 4 procs do >>>>> not. >>>>> >>>>> Serious wierdness. >>>>> >>>>> >>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> for (sitestride1 = Istart; sitestride1 < Iend; sitestride1++) >>>>>>>>>> { >>>>>>>>>> for (sitestride2 = 0; sitestride2 < matsize; sitestride2++) >>>>>>>>>> { >>>>>>>>>> for (alpha = 0; alpha < dim; alpha++) >>>>>>>>>> { >>>>>>>>>> for (mu = 0; mu < dim; mu++) >>>>>>>>>> for (lambda = 0; lambda < dim; lambda++) >>>>>>>>>> { >>>>>>>>>> vecval = rand () / rand (); >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> VecSetValue (BDB_AA, alpha, vecval, INSERT_VALUES); >>>>>>>>>> >>>>>>>>>> } >>>>>>>>>> VecAssemblyBegin (BDB_AA); >>>>>>>>>> VecAssemblyEnd (BDB_AA); >>>>>>>>>> VecSum (BDB_AA, &element); >>>>>>>>>> colvalues[sitestride2] = element; >>>>>>>>>> >>>>>>>>>> } >>>>>>>>>> //Insert the array of colvalues to the sitestride1^th row >>>>>>>>>> of H >>>>>>>>>> MatSetValues (AVG_BDIBJ, 1, &sitestride1, matsize, idx, >>>>>>>>>> colvalues, >>>>>>>>>> INSERT_VALUES); >>>>>>>>>> >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> > The code is large and complex, so I have created a smaller >>>>>>>>>> program >>>>>>>>>> > with the same >>>>>>>>>> > loop structure here. >>>>>>>>>> > >>>>>>>>>> > Compile it and run it with "mpirun -np $N ./test -draw_pause >>>>>>>>>> -1" gives >>>>>>>>>> > different results for different values of N even though it's >>>>>>>>>> not supposed >>>>>>>>>> > to. >>>>>>>>>> >>>>>>>>>> What do you expect to see? >>>>>>>>>> >>>>>>>>>> > Here is a sample output >>>>>>>>>> for N=1,2,3,4 >>>>>>>>>> > from left to right. >>>>>>>>>> > >>>>>>>>>> > Can anyone guide me as to what I'm doing wrong? Are any of the >>>>>>>>>> petssc >>>>>>>>>> > routines used not parallelizable? >>>>>>>>>> > >>>>>>>>>> > Thanks in advance, >>>>>>>>>> > >>>>>>>>>> > Regards. >>>>>>>>>> > >>>>>>>>>> > -- >>>>>>>>>> > --- >>>>>>>>>> > *Analabha Roy* >>>>>>>>>> > C.S.I.R Senior Research >>>>>>>>>> > Associate >>>>>>>>>> > Saha Institute of Nuclear Physics >>>>>>>>>> > Section 1, Block AF >>>>>>>>>> > Bidhannagar, Calcutta 700064 >>>>>>>>>> > India >>>>>>>>>> > *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >>>>>>>>>> > *Webpage*: http://www.ph.utexas.edu/~daneel/ >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> --- >>>>>>>>> *Analabha Roy* >>>>>>>>> C.S.I.R Senior Research Associate >>>>>>>>> Saha Institute of Nuclear Physics >>>>>>>>> Section 1, Block AF >>>>>>>>> Bidhannagar, Calcutta 700064 >>>>>>>>> India >>>>>>>>> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >>>>>>>>> *Webpage*: http://www.ph.utexas.edu/~daneel/ >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their >>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>> experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> --- >>>>>>> *Analabha Roy* >>>>>>> C.S.I.R Senior Research Associate >>>>>>> Saha Institute of Nuclear Physics >>>>>>> Section 1, Block AF >>>>>>> Bidhannagar, Calcutta 700064 >>>>>>> India >>>>>>> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >>>>>>> *Webpage*: http://www.ph.utexas.edu/~daneel/ >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> --- >>>>> *Analabha Roy* >>>>> C.S.I.R Senior Research Associate >>>>> Saha Institute of Nuclear Physics >>>>> Section 1, Block AF >>>>> Bidhannagar, Calcutta 700064 >>>>> India >>>>> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >>>>> *Webpage*: http://www.ph.utexas.edu/~daneel/ >>>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >>> >>> -- >>> --- >>> *Analabha Roy* >>> C.S.I.R Senior Research Associate >>> Saha Institute of Nuclear Physics >>> Section 1, Block AF >>> Bidhannagar, Calcutta 700064 >>> India >>> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com >>> *Webpage*: http://www.ph.utexas.edu/~daneel/ >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > --- > *Analabha Roy* > C.S.I.R Senior Research Associate > Saha Institute of Nuclear Physics > Section 1, Block AF > Bidhannagar, Calcutta 700064 > India > *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com > *Webpage*: http://www.ph.utexas.edu/~daneel/ > -- --- *Analabha Roy* C.S.I.R Senior Research Associate Saha Institute of Nuclear Physics Section 1, Block AF Bidhannagar, Calcutta 700064 India *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com *Webpage*: http://www.ph.utexas.edu/~daneel/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Sep 24 19:59:14 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 25 Sep 2013 01:59:14 +0100 Subject: [petsc-users] Fwd: Building the same petsc matrix with different numprocs gives different results! In-Reply-To: References: <87zjr2zlih.fsf@mcs.anl.gov> Message-ID: <87zjr1n2cd.fsf@mcs.anl.gov> Analabha Roy writes: > Sorry. Modified accordingly and here is new > output(I manually reordered the output of > the 2 procs case since the order in > which it was printed was haphazard) There is no automatic ordering of output among processes. You can use PetscSynchronizedPrintf and PetscSynchronizedFlush if you want collated output. > All the elements do not match. Sounds like you are inserting different values when you run in parallel. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From jedbrown at mcs.anl.gov Wed Sep 25 01:42:39 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 25 Sep 2013 07:42:39 +0100 Subject: [petsc-users] Fwd: Building the same petsc matrix with different numprocs gives different results! In-Reply-To: References: <87zjr2zlih.fsf@mcs.anl.gov> Message-ID: <87wqm5mmg0.fsf@mcs.anl.gov> Analabha Roy writes: > There is one thing > > In the code, the evaluation of each element of AVG_BIBJ requires a > read-only matrix U_parallel that I input from another program, and a > writeable sequential vector BDB_AA that is different for each element. > > I sequentiate U_parallel to U_seq by using MatCopy here in lines > 242+ /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Create a sequential matrix and copy U_parallel to it so that each processor has complete local copy of U - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */ ierr = MatCreateSeqAIJ (PETSC_COMM_SELF, dim, dim, 0, PETSC_NULL, &U_seq); CHKERRQ (ierr); ierr = MatSetOption (U_seq, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE); CHKERRQ (ierr); ierr = MatCopy (U_parallel, U_seq, DIFFERENT_NONZERO_PATTERN); This is wrong and should raise an error (definitely in a debugging build). Are you intending to create a redundant matrix containing all of U_parallel replicated onto each process? If so, you may as well assemble the whole thing redundantly, but if you'd rather distribute the computation of entries, you can use MatGetRedundantMatrix(). If you want something else, what is it? CHKERRQ (ierr); ierr = MatAssemblyBegin (U_seq, MAT_FINAL_ASSEMBLY); CHKERRQ (ierr); ierr = MatAssemblyEnd (U_seq, MAT_FINAL_ASSEMBLY); CHKERRQ (ierr); > and each process is supposed to update its copy of BDB_AA at every loop > iteration here in line > 347+ > > > Is this right? Or are sequential vectors/matrices handled by the root > process only? PETSC_COMM_SELF means the current process, so every process does that stuff. > I know how to scatter a parallel vector to all processes using PetSc > scatter contexts but don't see any way to do so to a matrix other than > MatCopy. How do I ensure that each process has its own private > writeable copy of a sequential vector? Is the last sentence supposed to say "vector"? If so, see VecScatterCreateToAll. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From hariseldon99 at gmail.com Wed Sep 25 02:25:12 2013 From: hariseldon99 at gmail.com (Analabha Roy) Date: Wed, 25 Sep 2013 12:55:12 +0530 Subject: [petsc-users] Fwd: Building the same petsc matrix with different numprocs gives different results! In-Reply-To: <87wqm5mmg0.fsf@mcs.anl.gov> References: <87zjr2zlih.fsf@mcs.anl.gov> <87wqm5mmg0.fsf@mcs.anl.gov> Message-ID: On Wed, Sep 25, 2013 at 12:12 PM, Jed Brown wrote: > Analabha Roy writes: > > > There is one thing > > > > In the code, the evaluation of each element of AVG_BIBJ requires a > > read-only matrix U_parallel that I input from another program, and a > > writeable sequential vector BDB_AA that is different for each element. > > > > I sequentiate U_parallel to U_seq by using MatCopy here in lines > > 242+< > https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c#242> > > /* > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > - - - - - - - - - - - - > Create a sequential matrix and copy U_parallel to it > so that each processor has complete local copy of U > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > - - - - - - - - - - - - > */ > ierr = MatCreateSeqAIJ (PETSC_COMM_SELF, dim, dim, 0, PETSC_NULL, > &U_seq); > CHKERRQ (ierr); > ierr = MatSetOption (U_seq, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE); > CHKERRQ (ierr); > ierr = MatCopy (U_parallel, U_seq, DIFFERENT_NONZERO_PATTERN); > > This is wrong and should raise an error (definitely in a debugging > build). Are you intending to create a redundant matrix containing all > of U_parallel replicated onto each process? If so, you may as well > assemble the whole thing redundantly, but if you'd rather distribute the > computation of entries, you can use MatGetRedundantMatrix(). If you > want something else, what is it? > > Thanks for the informative post. You're right in that I need to distribute the computation of entries, but need the data scattered/gathered to all procs afterwards. I thought MatCopy would do this. I'm not surprised that this is wrong. This part is probably what's wrong with the code. I'm looking at MatGetRedundantMatrix() docs and using it to fix the problem. Will post results as soon as I'm done... > CHKERRQ (ierr); > ierr = MatAssemblyBegin (U_seq, MAT_FINAL_ASSEMBLY); > CHKERRQ (ierr); > ierr = MatAssemblyEnd (U_seq, MAT_FINAL_ASSEMBLY); > CHKERRQ (ierr); > > > > and each process is supposed to update its copy of BDB_AA at every loop > > iteration here in line > > 347+< > https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c#347> > > > > > > Is this right? Or are sequential vectors/matrices handled by the root > > process only? > > PETSC_COMM_SELF means the current process, so every process does that > stuff. > > > I know how to scatter a parallel vector to all processes using PetSc > > scatter contexts but don't see any way to do so to a matrix other than > > MatCopy. How do I ensure that each process has its own private > > writeable copy of a sequential vector? > > Is the last sentence supposed to say "vector"? If so, see > VecScatterCreateToAll. > -- --- *Analabha Roy* C.S.I.R Senior Research Associate Saha Institute of Nuclear Physics Section 1, Block AF Bidhannagar, Calcutta 700064 India *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com *Webpage*: http://www.ph.utexas.edu/~daneel/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From hariseldon99 at gmail.com Wed Sep 25 06:13:52 2013 From: hariseldon99 at gmail.com (Analabha Roy) Date: Wed, 25 Sep 2013 16:43:52 +0530 Subject: [petsc-users] Fwd: Building the same petsc matrix with different numprocs gives different results! In-Reply-To: References: <87zjr2zlih.fsf@mcs.anl.gov> <87wqm5mmg0.fsf@mcs.anl.gov> Message-ID: Progress Report: So I modified my code to remove all the wrong usages of MatCopy where I was trying to AllScatter/AllGather a parallel matrix, and replaced with usage of MatGetRedundantMatrix(). The highlighted diff is here . Now, running it with 1 proc crashes, I guess because the documentation saysthat only parallel matrices are supported by MatGetRedundantMatrix(), and running with 1 proc means the parallel matrices are sequantial (from what I read in the PetSc users manual). However, running it with 2,3,4,5+ procs now yields identical and consistent results (unlike before), so the problem appears to have been solved. I would be very grateful if an extra pair of eyes could take a quick gander at the diff and confirm if my changes are correct. Thanks a bunch and Regards, On Wed, Sep 25, 2013 at 12:55 PM, Analabha Roy wrote: > > > > On Wed, Sep 25, 2013 at 12:12 PM, Jed Brown wrote: > >> Analabha Roy writes: >> >> > There is one thing >> > >> > In the code, the evaluation of each element of AVG_BIBJ requires a >> > read-only matrix U_parallel that I input from another program, and a >> > writeable sequential vector BDB_AA that is different for each element. >> > >> > I sequentiate U_parallel to U_seq by using MatCopy here in lines >> > 242+< >> https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c#242 >> > >> >> /* >> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - >> - - - - - - - - - - - - - >> Create a sequential matrix and copy U_parallel to it >> so that each processor has complete local copy of U >> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - >> - - - - - - - - - - - - - >> */ >> ierr = MatCreateSeqAIJ (PETSC_COMM_SELF, dim, dim, 0, PETSC_NULL, >> &U_seq); >> CHKERRQ (ierr); >> ierr = MatSetOption (U_seq, MAT_NEW_NONZERO_ALLOCATION_ERR, >> PETSC_FALSE); >> CHKERRQ (ierr); >> ierr = MatCopy (U_parallel, U_seq, DIFFERENT_NONZERO_PATTERN); >> >> This is wrong and should raise an error (definitely in a debugging >> build). Are you intending to create a redundant matrix containing all >> of U_parallel replicated onto each process? If so, you may as well >> assemble the whole thing redundantly, but if you'd rather distribute the >> computation of entries, you can use MatGetRedundantMatrix(). If you >> want something else, what is it? >> >> > > Thanks for the informative post. You're right in that I need to distribute > the computation of entries, but need the data scattered/gathered to all > procs afterwards. I thought MatCopy would do this. I'm not surprised that > this is wrong. This part is probably what's wrong with the code. > > I'm looking at MatGetRedundantMatrix() docs and using it to fix the > problem. Will post results as soon as I'm done... > > > >> CHKERRQ (ierr); >> ierr = MatAssemblyBegin (U_seq, MAT_FINAL_ASSEMBLY); >> CHKERRQ (ierr); >> ierr = MatAssemblyEnd (U_seq, MAT_FINAL_ASSEMBLY); >> CHKERRQ (ierr); >> >> >> > and each process is supposed to update its copy of BDB_AA at every loop >> > iteration here in line >> > 347+< >> https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c#347 >> > >> > >> > >> > Is this right? Or are sequential vectors/matrices handled by the root >> > process only? >> >> PETSC_COMM_SELF means the current process, so every process does that >> stuff. >> >> > I know how to scatter a parallel vector to all processes using PetSc >> > scatter contexts but don't see any way to do so to a matrix other than >> > MatCopy. How do I ensure that each process has its own private >> > writeable copy of a sequential vector? >> >> Is the last sentence supposed to say "vector"? If so, see >> VecScatterCreateToAll. >> > > > > -- > --- > *Analabha Roy* > C.S.I.R Senior Research Associate > Saha Institute of Nuclear Physics > Section 1, Block AF > Bidhannagar, Calcutta 700064 > India > *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com > *Webpage*: http://www.ph.utexas.edu/~daneel/ > -- --- *Analabha Roy* C.S.I.R Senior Research Associate Saha Institute of Nuclear Physics Section 1, Block AF Bidhannagar, Calcutta 700064 India *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com *Webpage*: http://www.ph.utexas.edu/~daneel/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From christophe.ortiz at ciemat.es Wed Sep 25 06:31:12 2013 From: christophe.ortiz at ciemat.es (Christophe Ortiz) Date: Wed, 25 Sep 2013 13:31:12 +0200 Subject: [petsc-users] How to construct Jacobian for multicomponent problem ? Message-ID: Hi guys, Please apologize if this question has already been raised in previous posts. I am developing a code in fortran 90 to solve a 1D multicomponent problem. For instance, something like u_t = u_xx + R(u,v) v_t = v_xx + R(u,v) I have already implemented this for one component (u) and using DMDA. So far no problem. It works fine. Now I am trying to implement several components per node. Following example ex22f.F, for the residual function I would have for node i (1D) F(1,i) = .... F(2,i) = .... Now, I am not quite sure to understand how to construct the Jacobian when there are various components. In this case, how is the vector in PETSc ? Is it a long vector like (u0, u1, ..., umx-1 , v0, v1, ..., vmx-1) ? Or is it (u0, v0, u1, v1, u2, v2, ....., umx-1, vmx-1) ? I don't understand the order of the components of the Jacobian and how to fill the matrix in that case. For instance, for the first row of the Jacobian, should I calculate dFu0/du0, dFu0/dFu1, ...., dFu0/dumx-1, dFu0/dv0, dFu0/dv1..... or dFu0/du0, dFu0/dv0, dFu0/du1, dFu0/dv1, .... And then, which subroutine should I use to fill in the Matrix ? Simply MatSetValues ? Many thanks in advance for your help. Christophe -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Sep 25 06:34:50 2013 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 25 Sep 2013 04:34:50 -0700 Subject: [petsc-users] How to construct Jacobian for multicomponent problem ? In-Reply-To: References: Message-ID: On Wed, Sep 25, 2013 at 4:31 AM, Christophe Ortiz < christophe.ortiz at ciemat.es> wrote: > Hi guys, > > Please apologize if this question has already been raised in previous > posts. > > I am developing a code in fortran 90 to solve a 1D multicomponent problem. > For instance, something like > > u_t = u_xx + R(u,v) > v_t = v_xx + R(u,v) > > I have already implemented this for one component (u) and using DMDA. So > far no problem. It works fine. > > Now I am trying to implement several components per node. Following > example ex22f.F, for the residual function I would have for node i (1D) > > F(1,i) = .... > F(2,i) = .... > > Now, I am not quite sure to understand how to construct the Jacobian when > there are various components. In this case, how is the vector in PETSc ? > The Jacobian is a matrix, and the rows are ordered in the same way as in the residual function. The columns are the same as the rows. Thanks, Matt > Is it a long vector like (u0, u1, ..., umx-1 , v0, v1, ..., vmx-1) ? > Or is it (u0, v0, u1, v1, u2, v2, ....., umx-1, vmx-1) ? > > I don't understand the order of the components of the Jacobian and how to > fill the matrix in that case. > > For instance, for the first row of the Jacobian, should I calculate > dFu0/du0, dFu0/dFu1, ...., dFu0/dumx-1, dFu0/dv0, dFu0/dv1..... > > or > > dFu0/du0, dFu0/dv0, dFu0/du1, dFu0/dv1, .... > > And then, which subroutine should I use to fill in the Matrix ? Simply > MatSetValues ? > > Many thanks in advance for your help. > > Christophe > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From christophe.ortiz at ciemat.es Wed Sep 25 06:40:53 2013 From: christophe.ortiz at ciemat.es (Christophe Ortiz) Date: Wed, 25 Sep 2013 13:40:53 +0200 Subject: [petsc-users] How to construct Jacobian for multicomponent problem ? In-Reply-To: References: Message-ID: Hi Matt, Thanks for your prompt reply. On Wed, Sep 25, 2013 at 1:34 PM, Matthew Knepley wrote: > On Wed, Sep 25, 2013 at 4:31 AM, Christophe Ortiz < > christophe.ortiz at ciemat.es> wrote: > >> Hi guys, >> >> Please apologize if this question has already been raised in previous >> posts. >> >> I am developing a code in fortran 90 to solve a 1D multicomponent problem. >> For instance, something like >> >> u_t = u_xx + R(u,v) >> v_t = v_xx + R(u,v) >> >> I have already implemented this for one component (u) and using DMDA. So >> far no problem. It works fine. >> >> Now I am trying to implement several components per node. Following >> example ex22f.F, for the residual function I would have for node i (1D) >> >> F(1,i) = .... >> F(2,i) = .... >> >> Now, I am not quite sure to understand how to construct the Jacobian when >> there are various components. In this case, how is the vector in PETSc ? >> > > The Jacobian is a matrix, and the rows are ordered in the same way as in > the residual function. The columns are the same as the rows. > Yes, I understand this for one component. My question is when you have dof > 1. For a multicomponent problem you can write the residual function as an array: F(1,i) =... (component 1) F(2,i) = ... (component 2) Since there is only one Jacobian for the whole system, how are the components of the Jacobian ordered in that case ? Is it first the components of u, then the components of v ? Christophe > > Thanks, > > Matt > > >> Is it a long vector like (u0, u1, ..., umx-1 , v0, v1, ..., vmx-1) ? >> Or is it (u0, v0, u1, v1, u2, v2, ....., umx-1, vmx-1) ? >> >> I don't understand the order of the components of the Jacobian and how to >> fill the matrix in that case. >> >> For instance, for the first row of the Jacobian, should I calculate >> dFu0/du0, dFu0/dFu1, ...., dFu0/dumx-1, dFu0/dv0, dFu0/dv1..... >> >> or >> >> dFu0/du0, dFu0/dv0, dFu0/du1, dFu0/dv1, .... >> >> And then, which subroutine should I use to fill in the Matrix ? Simply >> MatSetValues ? >> >> Many thanks in advance for your help. >> >> Christophe >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Sep 25 06:46:31 2013 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 25 Sep 2013 04:46:31 -0700 Subject: [petsc-users] How to construct Jacobian for multicomponent problem ? In-Reply-To: References: Message-ID: On Wed, Sep 25, 2013 at 4:40 AM, Christophe Ortiz < christophe.ortiz at ciemat.es> wrote: > Hi Matt, > Thanks for your prompt reply. > > On Wed, Sep 25, 2013 at 1:34 PM, Matthew Knepley wrote: > >> On Wed, Sep 25, 2013 at 4:31 AM, Christophe Ortiz < >> christophe.ortiz at ciemat.es> wrote: >> >>> Hi guys, >>> >>> Please apologize if this question has already been raised in previous >>> posts. >>> >>> I am developing a code in fortran 90 to solve a 1D multicomponent >>> problem. >>> For instance, something like >>> >>> u_t = u_xx + R(u,v) >>> v_t = v_xx + R(u,v) >>> >>> I have already implemented this for one component (u) and using DMDA. So >>> far no problem. It works fine. >>> >>> Now I am trying to implement several components per node. Following >>> example ex22f.F, for the residual function I would have for node i (1D) >>> >>> F(1,i) = .... >>> F(2,i) = .... >>> >>> Now, I am not quite sure to understand how to construct the Jacobian >>> when there are various components. In this case, how is the vector in PETSc >>> ? >>> >> >> The Jacobian is a matrix, and the rows are ordered in the same way as in >> the residual function. The columns are the same as the rows. >> > > Yes, I understand this for one component. My question is when you have dof > > 1. For a multicomponent problem you can write the residual function as an > array: > > F(1,i) =... (component 1) > F(2,i) = ... (component 2) > > Since there is only one Jacobian for the whole system, how are the > components of the Jacobian ordered in that case ? Is it first the > components of u, then the components of v ? > The ordering of the rows (and columns) of the Jacobian match the rows of the residual EVEN in the multicomponent case. Matt > Christophe > > >> >> Thanks, >> >> Matt >> >> >>> Is it a long vector like (u0, u1, ..., umx-1 , v0, v1, ..., vmx-1) ? >>> Or is it (u0, v0, u1, v1, u2, v2, ....., umx-1, vmx-1) ? >>> >>> I don't understand the order of the components of the Jacobian and how >>> to fill the matrix in that case. >>> >>> For instance, for the first row of the Jacobian, should I calculate >>> dFu0/du0, dFu0/dFu1, ...., dFu0/dumx-1, dFu0/dv0, dFu0/dv1..... >>> >>> or >>> >>> dFu0/du0, dFu0/dv0, dFu0/du1, dFu0/dv1, .... >>> >>> And then, which subroutine should I use to fill in the Matrix ? Simply >>> MatSetValues ? >>> >>> Many thanks in advance for your help. >>> >>> Christophe >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From christophe.ortiz at ciemat.es Wed Sep 25 06:55:12 2013 From: christophe.ortiz at ciemat.es (Christophe Ortiz) Date: Wed, 25 Sep 2013 13:55:12 +0200 Subject: [petsc-users] How to construct Jacobian for multicomponent problem ? In-Reply-To: References: Message-ID: On Wed, Sep 25, 2013 at 1:46 PM, Matthew Knepley wrote: > On Wed, Sep 25, 2013 at 4:40 AM, Christophe Ortiz < > christophe.ortiz at ciemat.es> wrote: > >> Hi Matt, >> Thanks for your prompt reply. >> >> On Wed, Sep 25, 2013 at 1:34 PM, Matthew Knepley wrote: >> >>> On Wed, Sep 25, 2013 at 4:31 AM, Christophe Ortiz < >>> christophe.ortiz at ciemat.es> wrote: >>> >>>> Hi guys, >>>> >>>> Please apologize if this question has already been raised in previous >>>> posts. >>>> >>>> I am developing a code in fortran 90 to solve a 1D multicomponent >>>> problem. >>>> For instance, something like >>>> >>>> u_t = u_xx + R(u,v) >>>> v_t = v_xx + R(u,v) >>>> >>>> I have already implemented this for one component (u) and using DMDA. >>>> So far no problem. It works fine. >>>> >>>> Now I am trying to implement several components per node. Following >>>> example ex22f.F, for the residual function I would have for node i (1D) >>>> >>>> F(1,i) = .... >>>> F(2,i) = .... >>>> >>>> Now, I am not quite sure to understand how to construct the Jacobian >>>> when there are various components. In this case, how is the vector in PETSc >>>> ? >>>> >>> >>> The Jacobian is a matrix, and the rows are ordered in the same way as in >>> the residual function. The columns are the same as the rows. >>> >> >> Yes, I understand this for one component. My question is when you have >> dof > 1. For a multicomponent problem you can write the residual function >> as an array: >> >> F(1,i) =... (component 1) >> F(2,i) = ... (component 2) >> >> Since there is only one Jacobian for the whole system, how are the >> components of the Jacobian ordered in that case ? Is it first the >> components of u, then the components of v ? >> > > The ordering of the rows (and columns) of the Jacobian match the rows of > the residual EVEN in the multicomponent case. > > Matt > > Ok. But what I still don't understand is the ordering of the residual in the multicomponent case. Let's say dof=2, components u and v. Which is the correct ordering in the following ? Fu0, Fu1, Fu2,...Fumx-1, Fv0, Fv1, Fv2, ...Fvmx-1 or Fu0, Fv0, Fu1, Fv1, Fu2, Fv2, ... ? It is this point that is not clear to me. Also, which subroutine do you advice to use to fill in the Matrix ? MatSetValues ? MatSetValuesBlocked ? Christophe -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Sep 25 07:15:36 2013 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 25 Sep 2013 05:15:36 -0700 Subject: [petsc-users] How to construct Jacobian for multicomponent problem ? In-Reply-To: References: Message-ID: On Wed, Sep 25, 2013 at 4:55 AM, Christophe Ortiz < christophe.ortiz at ciemat.es> wrote: > > > > On Wed, Sep 25, 2013 at 1:46 PM, Matthew Knepley wrote: > >> On Wed, Sep 25, 2013 at 4:40 AM, Christophe Ortiz < >> christophe.ortiz at ciemat.es> wrote: >> >>> Hi Matt, >>> Thanks for your prompt reply. >>> >>> On Wed, Sep 25, 2013 at 1:34 PM, Matthew Knepley wrote: >>> >>>> On Wed, Sep 25, 2013 at 4:31 AM, Christophe Ortiz < >>>> christophe.ortiz at ciemat.es> wrote: >>>> >>>>> Hi guys, >>>>> >>>>> Please apologize if this question has already been raised in previous >>>>> posts. >>>>> >>>>> I am developing a code in fortran 90 to solve a 1D multicomponent >>>>> problem. >>>>> For instance, something like >>>>> >>>>> u_t = u_xx + R(u,v) >>>>> v_t = v_xx + R(u,v) >>>>> >>>>> I have already implemented this for one component (u) and using DMDA. >>>>> So far no problem. It works fine. >>>>> >>>>> Now I am trying to implement several components per node. Following >>>>> example ex22f.F, for the residual function I would have for node i (1D) >>>>> >>>>> F(1,i) = .... >>>>> F(2,i) = .... >>>>> >>>>> Now, I am not quite sure to understand how to construct the Jacobian >>>>> when there are various components. In this case, how is the vector in PETSc >>>>> ? >>>>> >>>> >>>> The Jacobian is a matrix, and the rows are ordered in the same way as >>>> in the residual function. The columns are the same as the rows. >>>> >>> >>> Yes, I understand this for one component. My question is when you have >>> dof > 1. For a multicomponent problem you can write the residual function >>> as an array: >>> >>> F(1,i) =... (component 1) >>> F(2,i) = ... (component 2) >>> >>> Since there is only one Jacobian for the whole system, how are the >>> components of the Jacobian ordered in that case ? Is it first the >>> components of u, then the components of v ? >>> >> >> The ordering of the rows (and columns) of the Jacobian match the rows of >> the residual EVEN in the multicomponent case. >> >> Matt >> >> > > Ok. But what I still don't understand is the ordering of the residual in > the multicomponent case. Let's say dof=2, components u and v. Which is the > correct ordering in the following ? > > Fu0, Fu1, Fu2,...Fumx-1, Fv0, Fv1, Fv2, ...Fvmx-1 > > or > > Fu0, Fv0, Fu1, Fv1, Fu2, Fv2, ... ? > This one. > It is this point that is not clear to me. > > Also, which subroutine do you advice to use to fill in the Matrix ? > MatSetValues ? MatSetValuesBlocked ? > MatSetValuesStencil. Matt > Christophe > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From christophe.ortiz at ciemat.es Wed Sep 25 07:24:30 2013 From: christophe.ortiz at ciemat.es (Christophe Ortiz) Date: Wed, 25 Sep 2013 14:24:30 +0200 Subject: [petsc-users] How to construct Jacobian for multicomponent problem ? In-Reply-To: References: Message-ID: > > >>> >>> >> >> Ok. But what I still don't understand is the ordering of the residual in >> the multicomponent case. Let's say dof=2, components u and v. Which is the >> correct ordering in the following ? >> >> Fu0, Fu1, Fu2,...Fumx-1, Fv0, Fv1, Fv2, ...Fvmx-1 >> >> or >> >> Fu0, Fv0, Fu1, Fv1, Fu2, Fv2, ... ? >> > > This one. > Ok. > > >> It is this point that is not clear to me. >> >> Also, which subroutine do you advice to use to fill in the Matrix ? >> MatSetValues ? MatSetValuesBlocked ? >> > > MatSetValuesStencil. > Thanks a lot Matt for your prompt replies ! Christophe > > Matt > > >> Christophe >> >> >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Sep 25 06:33:15 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 25 Sep 2013 12:33:15 +0100 Subject: [petsc-users] Fwd: Building the same petsc matrix with different numprocs gives different results! In-Reply-To: References: <87zjr2zlih.fsf@mcs.anl.gov> <87wqm5mmg0.fsf@mcs.anl.gov> Message-ID: <87mwn1m8zo.fsf@mcs.anl.gov> Analabha Roy writes: > Progress Report: > > So I modified my code to remove all the wrong usages of MatCopy where I > was trying to AllScatter/AllGather a parallel matrix, and replaced with > usage of MatGetRedundantMatrix(). The highlighted diff is > here Looks fine. > Now, running it with 1 proc crashes, I guess because the documentation > saysthat > only parallel matrices are supported by MatGetRedundantMatrix(), and > running with 1 proc means the parallel matrices are sequantial (from what I > read in the PetSc users manual). You can use this to run on a single process. MPI_Comm_size(PetscObjectComm(PetscObject)mat,&size); if (size > 1) { MatGetRedundantMatrix(mat,&rmat); } else { rmat = mat; PetscObjectReference((PetscObject)rmat); } (In parallel, rmat is a copy, but in serial, rmat is the same matrix.) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From petr.kotas at gmail.com Wed Sep 25 10:08:55 2013 From: petr.kotas at gmail.com (Petr Kotas) Date: Wed, 25 Sep 2013 17:08:55 +0200 Subject: [petsc-users] How to get processes layout from already created DMDA object? In-Reply-To: References: Message-ID: <5242FC87.3010501@gmail.com> Hi guys, I have probably an easy question. How do I extract distribution of processes from already created DM object. Which was created with call to ierr = DMDACreate3d(PETSC_COMM_WORLD, boundaryType, boundaryType, boundaryType, DMDA_STENCIL_BOX, M[0], M[1], M[2], PETSC_DECIDE, PETSC_DECIDE, PETSC_DECIDE, dof, stencilSize, PETSC_NULL, PETSC_NULL, PETSC_NULL, &da); where /boundaryType/, /M[i], dof, stencilSize /are my variables. Simply, I would like to to know how PETSC_DECIDE(ed) about the layout. I need it for my other part of the code. Thank You, Petr Kotas -------------- next part -------------- An HTML attachment was scrubbed... URL: From prbrune at gmail.com Wed Sep 25 10:12:01 2013 From: prbrune at gmail.com (Peter Brune) Date: Wed, 25 Sep 2013 10:12:01 -0500 Subject: [petsc-users] How to get processes layout from already created DMDA object? In-Reply-To: <5242FC87.3010501@gmail.com> References: <5242FC87.3010501@gmail.com> Message-ID: http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMDAGetOwnershipRanges.html will give you the ownership ranges in each dimension. - Peter On Wed, Sep 25, 2013 at 10:08 AM, Petr Kotas wrote: > Hi guys, > > I have probably an easy question. How do I extract distribution of > processes from already created DM object. Which was created with call to > > ierr = DMDACreate3d(PETSC_COMM_WORLD, boundaryType, boundaryType, > boundaryType, DMDA_STENCIL_BOX, > M[0], M[1], M[2], PETSC_DECIDE, > PETSC_DECIDE, PETSC_DECIDE, dof, stencilSize, > PETSC_NULL, PETSC_NULL, PETSC_NULL, > &da); > > where *boundaryType*, *M[i], dof, stencilSize *are my variables. > > Simply, I would like to to know how PETSC_DECIDE(ed) about the layout. I > need it for my other part of the code. > > Thank You, > Petr Kotas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From danyang.su at gmail.com Wed Sep 25 17:59:30 2013 From: danyang.su at gmail.com (Danyang Su) Date: Wed, 25 Sep 2013 15:59:30 -0700 Subject: [petsc-users] Strange efficiency in PETSc-dev using OpenMP In-Reply-To: <47AF7C50-1533-4298-A579-901CE3AABEAC@mcs.anl.gov> References: <523D482D.5030409@gmail.com> <101244E8-FFFA-4F3F-A4BC-CC47F68D93B7@mcs.anl.gov> <523E059D.6080901@gmail.com> <933EF7F4-598A-4F88-8902-0613E27F0679@mcs.anl.gov> <523E6CAF.9020805@gmail.com> <52408989.7090001@gmail.com> <52409009.2000506@gmail.com> <524092EC.6080605@gmail.com> <47AF7C50-1533-4298-A579-901CE3AABEAC@mcs.anl.gov> Message-ID: <52436AD2.5030406@gmail.com> Hi All, The time-consuming problem has been solved but the efficiency problem is still stuck. I am sorry to say that the time-consuming problem is due to my codes, not compiler nor PETSc version. For the previous codes (bellow), I use the following commands to create matrix A and it takes a lot of time when running with OpenMP or running with MPI with 1 processor. call MatCreate(Petsc_Comm_World, a, ierr) call MatSetSizes(a, Petsc_Decide, Petsc_Decide, nb, nb, ierr) call MatSetFromOptions(a, ierr) call MatSetUp(a, ierr) call MatGetOwnershipRange(a,istart,iend,ierr) call MatMPIAIJSetPreallocation(a,Petsc_Null_Integer,d_nnz(istart+1:iend), Petsc_Null_Integer,o_nnz(istart+1:iend), ierr) After changing to the codes bellow, this problem is solved. call MatCreateAIJ(Petsc_Comm_World, Petsc_Decide, Petsc_Decide, nb, nb, Petsc_Null_Integer, & d_nnz(jj:kk), Petsc_Null_Integer, o_nnz(jj:kk), a, ierr) But the efficiency problem is still not solved, I can get a very good speed in solving the linear equation with MPI, but almost no speedup with OpenMP. I tried many matrices, from small size to big size. The log_summary files have been attached. Can anybody help to check my log to see what causes this problem? Thanks and regards, Danyang On 23/09/2013 12:16 PM, Barry Smith wrote: > On Sep 23, 2013, at 2:13 PM, Danyang Su wrote: > >> Hi Barry, >> >> Another strange problem: >> >> Currently I have PETSc-3.4.2 MPI version and PETSc-dev OpenMP version on my computer, with different environment variable of PETSC_ARCH and PETSC_DIR. Before installation of PETSc-dev OpenMP version, the PETSc-3.4.2 MPI version works fine. But after installation of PETSc-dev OpenMP version, the same problem exist in PETSc-3.4.2 MPI version if run with 1 processor, but no problem with 2 or more processors. > Are you sure some environmental variable is not set? Related to OpenMP? What if you run on one process with mpiexec and then immediately after on two process with mpiexec in the same window? > > Barry > >> Thanks, >> >> Danyang >> >> On 23/09/2013 12:01 PM, Danyang Su wrote: >>> Hi Barry, >>> >>> Sorry I forgot the message in the previous email. It is still slow when run without the "-threadcomm_type openmp -threadcomm_nthreads 1" >>> >>> Thanks, >>> >>> Danyang >>> >>> On 23/09/2013 11:43 AM, Barry Smith wrote: >>>> You did not answer my question from yesterday: >>>> >>>> If you run the Openmp compiled version WITHOUT the >>>> >>>> -threadcomm_nthreads 1 >>>> -threadcomm_type openmp >>>> >>>> command line options is it still slow? >>>> >>>> >>>> On Sep 23, 2013, at 1:33 PM, Danyang Su wrote: >>>> >>>>> Hi Shri, >>>>> >>>>> It seems that the problem does not result from the affinities setting for threads. I have tried several settings, the threads are set to different cores, but there is no improvement. >>>>> >>>>> Here is the information of package, core and thread maps >>>>> >>>>> OMP: Info #204: KMP_AFFINITY: decoding x2APIC ids. >>>>> OMP: Info #202: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info >>>>> OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: {0,1,2,3,4,5,6,7,8,9,10,11} >>>>> OMP: Info #156: KMP_AFFINITY: 12 available OS procs >>>>> OMP: Info #157: KMP_AFFINITY: Uniform topology >>>>> OMP: Info #179: KMP_AFFINITY: 1 packages x 6 cores/pkg x 2 threads/core (6 total cores) >>>>> OMP: Info #206: KMP_AFFINITY: OS proc to physical thread map: >>>>> OMP: Info #171: KMP_AFFINITY: OS proc 0 maps to package 0 core 0 thread 0 >>>>> OMP: Info #171: KMP_AFFINITY: OS proc 1 maps to package 0 core 0 thread 1 >>>>> OMP: Info #171: KMP_AFFINITY: OS proc 2 maps to package 0 core 1 thread 0 >>>>> OMP: Info #171: KMP_AFFINITY: OS proc 3 maps to package 0 core 1 thread 1 >>>>> OMP: Info #171: KMP_AFFINITY: OS proc 4 maps to package 0 core 2 thread 0 >>>>> OMP: Info #171: KMP_AFFINITY: OS proc 5 maps to package 0 core 2 thread 1 >>>>> OMP: Info #171: KMP_AFFINITY: OS proc 6 maps to package 0 core 3 thread 0 >>>>> OMP: Info #171: KMP_AFFINITY: OS proc 7 maps to package 0 core 3 thread 1 >>>>> OMP: Info #171: KMP_AFFINITY: OS proc 8 maps to package 0 core 4 thread 0 >>>>> OMP: Info #171: KMP_AFFINITY: OS proc 9 maps to package 0 core 4 thread 1 >>>>> OMP: Info #171: KMP_AFFINITY: OS proc 10 maps to package 0 core 5 thread 0 >>>>> OMP: Info #171: KMP_AFFINITY: OS proc 11 maps to package 0 core 5 thread 1 >>>>> OMP: Info #144: KMP_AFFINITY: Threads may migrate across 1 innermost levels of machine >>>>> >>>>> >>>>> And here is the internal thread bounding with different kmp_affinity settings: >>>>> >>>>> 1. KMP_AFFINITY=verbose,granularity=thread,compact >>>>> >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {1} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {3} >>>>> >>>>> 2. KMP_AFFINITY=verbose,granularity=fine,compact >>>>> >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {1} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {3} >>>>> >>>>> 3. KMP_AFFINITY=verbose,granularity=fine,compact,1,0 >>>>> >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {2} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {4} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {6} >>>>> >>>>> 4. KMP_AFFINITY=verbose,scatter >>>>> >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,1} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {2,3} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {4,5} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {6,7} >>>>> >>>>> 5. KMP_AFFINITY=verbose,compact (For this setting, two threads are assigned to the same core) >>>>> >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,1} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {0,1} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2,3} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {2,3} >>>>> >>>>> 6. KMP_AFFINITY=verbose,granularity=core,compact (For this setting, two threads are assigned to the same core) >>>>> >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,1} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {0,1} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {2,3} >>>>> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {2,3} >>>>> >>>>> The first 4 settings can assign threads to a distinct core, but the problem is not solved. >>>>> >>>>> Thanks, >>>>> >>>>> Danyang >>>>> >>>>> >>>>> >>>>> On 22/09/2013 8:00 PM, Shri wrote: >>>>>> I think this is definitely an issue with setting the affinities for threads, i.e., the assignment of threads to cores. Ideally each thread should be assigned to a distinct core but in your case all the 4 threads are getting pinned to the same core resulting in such a massive slowdown. Unfortunately, the thread affinities for OpenMP are set through environment variables. For Intel's OpenMP one needs to define the thread affinities through the environment variable KMP_AFFINITY. See this document here http://software.intel.com/sites/products/documentation/studio/composer/en-us/2011Update/compiler_c/optaps/common/optaps_openmp_thread_affinity.htm. Try setting the affinities via KMP_AFFINITY and let us know if it works. >>>>>> >>>>>> Shri >>>>>> On Sep 21, 2013, at 11:06 PM, Danyang Su wrote: >>>>>> >>>>>>> Hi Shri, >>>>>>> >>>>>>> Thanks for your info. It can work with the option -threadcomm_type openmp. But another problem arises, as described as follows. >>>>>>> >>>>>>> The sparse matrix is 53760*53760 with 1067392 non-zero entries. If the codes is compiled using PETSc-3.4.2, it works fine, the equations can be solved quickly and I can see the speedup. But if the code is compiled using PETSc-dev with OpenMP option, it takes a long time in solving the equations and I cannot see any speedup when more processors are used. >>>>>>> >>>>>>> For PETSc-3.4.2, run by "mpiexec -n 4 ksp_inhm_d -log_summary log_mpi4_petsc3.4.2.log", the iteration and runtime are: >>>>>>> Iterations 6 time_assembly 0.4137E-01 time_ksp 0.9296E-01 >>>>>>> >>>>>>> For PETSc-dev, run by "mpiexec -n 1 ksp_inhm_d -threadcomm_type openmp -threadcomm_nthreads 4 -log_summary log_openmp_petsc_dev.log", the iteration and runtime are: >>>>>>> Iterations 6 time_assembly 0.3595E+03 time_ksp 0.2907E+00 >>>>>>> >>>>>>> Most of the time 'time_assembly 0.3595E+03' is spent on the following codes >>>>>>> do i = istart, iend - 1 >>>>>>> ii = ia_in(i+1) >>>>>>> jj = ia_in(i+2) >>>>>>> call MatSetValues(a, ione, i, jj-ii, ja_in(ii:jj-1)-1, a_in(ii:jj-1), Insert_Values, ierr) >>>>>>> end do >>>>>>> >>>>>>> The log files for both PETSc-3.4.2 and PETSc-dev are attached. >>>>>>> >>>>>>> Is there anything wrong with my codes or with running option? The above codes works fine when using MPICH. >>>>>>> >>>>>>> Thanks and regards, >>>>>>> >>>>>>> Danyang >>>>>>> >>>>>>> On 21/09/2013 2:09 PM, Shri wrote: >>>>>>>> There are three thread communicator types in PETSc. The default is "no thread" which is basically a non-threaded version. The other two types are "openmp" and "pthread". If you want to use OpenMP then use the option -threadcomm_type openmp. >>>>>>>> >>>>>>>> Shri >>>>>>>> >>>>>>>> On Sep 21, 2013, at 3:46 PM, Danyang Su wrote: >>>>>>>> >>>>>>>>> Hi Barry, >>>>>>>>> >>>>>>>>> Thanks for the quick reply. >>>>>>>>> >>>>>>>>> After changing >>>>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>>>>>> to >>>>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) >>>>>>>>> and comment out >>>>>>>>> #elif defined(PETSC_HAVE_OPENMP) >>>>>>>>> PETSC_EXTERN PetscStack *petscstack; >>>>>>>>> >>>>>>>>> It can be compiled and validated with "make test". >>>>>>>>> >>>>>>>>> But I still have questions on running the examples. After rebuild the codes (e.g., ksp_ex2f.f), I can run it with "mpiexec -n 1 ksp_ex2f", or "mpiexec -n 4 ksp_ex2f", or "mpiexec -n 1 ksp_ex2f -threadcomm_nthreads 1", but if I run it with "mpiexec -n 1 ksp_ex2f -threadcomm_nthreads 4", there will be a lot of error information (attached). >>>>>>>>> >>>>>>>>> The codes is not modified and there is no OpenMP routines in it. For the current development in my project, I want to keep the OpenMP codes in calculating matrix values, but want to solve it with PETSc (OpenMP). Is it possible? >>>>>>>>> >>>>>>>>> Thanks and regards, >>>>>>>>> >>>>>>>>> Danyang >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 21/09/2013 7:26 AM, Barry Smith wrote: >>>>>>>>>> Danyang, >>>>>>>>>> >>>>>>>>>> I don't think the || defined (PETSC_HAVE_OPENMP) belongs in the code below. >>>>>>>>>> >>>>>>>>>> /* Linux functions CPU_SET and others don't work if sched.h is not included before >>>>>>>>>> including pthread.h. Also, these functions are active only if either _GNU_SOURCE >>>>>>>>>> or __USE_GNU is not set (see /usr/include/sched.h and /usr/include/features.h), hence >>>>>>>>>> set these first. >>>>>>>>>> */ >>>>>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>>>>>>> >>>>>>>>>> Edit include/petscerror.h and locate these lines and remove that part and then rerun make all. Let us know if it works or not. >>>>>>>>>> >>>>>>>>>> Barry >>>>>>>>>> >>>>>>>>>> i.e. replace >>>>>>>>>> >>>>>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) >>>>>>>>>> >>>>>>>>>> with >>>>>>>>>> >>>>>>>>>> #if defined(PETSC_HAVE_PTHREADCLASSES) >>>>>>>>>> >>>>>>>>>> On Sep 21, 2013, at 6:53 AM, Matthew Knepley >>>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On Sat, Sep 21, 2013 at 12:18 AM, Danyang Su >>>>>>>>>>> wrote: >>>>>>>>>>> Hi All, >>>>>>>>>>> >>>>>>>>>>> I got error information in compiling petsc-dev with openmp in cygwin. Before, I have successfully compiled petsc-3.4.2 and it works fine. >>>>>>>>>>> The log files have been attached. >>>>>>>>>>> >>>>>>>>>>> The OpenMP configure test is wrong. It clearly fails to find pthread.h, but the test passes. Then in petscerror.h >>>>>>>>>>> we guard pthread.h using PETSC_HAVE_OPENMP. Can someone who knows OpenMP fix this? >>>>>>>>>>> >>>>>>>>>>> Matt >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Danyang >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>> >>>>>>>>> >>>>>>> -------------- next part -------------- A non-text attachment was scrubbed... Name: basin_log_openmp_1.log Type: application/octet-stream Size: 10098 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: basin_log_openmp_4.log Type: application/octet-stream Size: 10137 bytes Desc: not available URL: From hariseldon99 at gmail.com Wed Sep 25 22:25:49 2013 From: hariseldon99 at gmail.com (Analabha Roy) Date: Thu, 26 Sep 2013 08:55:49 +0530 Subject: [petsc-users] Fwd: Building the same petsc matrix with different numprocs gives different results! In-Reply-To: <87mwn1m8zo.fsf@mcs.anl.gov> References: <87zjr2zlih.fsf@mcs.anl.gov> <87wqm5mmg0.fsf@mcs.anl.gov> <87mwn1m8zo.fsf@mcs.anl.gov> Message-ID: On Sep 25, 2013 7:28 PM, "Jed Brown" wrote: > > Analabha Roy writes: > > > Progress Report: > > > > So I modified my code to remove all the wrong usages of MatCopy where I > > was trying to AllScatter/AllGather a parallel matrix, and replaced with > > usage of MatGetRedundantMatrix(). The highlighted diff is > > here > > Looks fine. > > > Now, running it with 1 proc crashes, I guess because the documentation > > says< http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetRedundantMatrix.html >that > > only parallel matrices are supported by MatGetRedundantMatrix(), and > > running with 1 proc means the parallel matrices are sequantial (from what I > > read in the PetSc users manual). > > You can use this to run on a single process. > > MPI_Comm_size(PetscObjectComm(PetscObject)mat,&size); > if (size > 1) { > MatGetRedundantMatrix(mat,&rmat); > } else { > rmat = mat; > PetscObjectReference((PetscObject)rmat); > } > > (In parallel, rmat is a copy, but in serial, rmat is the same matrix.) Thanks very much. -------------- next part -------------- An HTML attachment was scrubbed... URL: From christophe.ortiz at ciemat.es Thu Sep 26 03:23:55 2013 From: christophe.ortiz at ciemat.es (Christophe Ortiz) Date: Thu, 26 Sep 2013 10:23:55 +0200 Subject: [petsc-users] Simple diffusion problem: TSARKIMEX vs TSROSW vs TSBEULER Message-ID: Hi all, I have implemented some simple diffusion problem in 1D using PETSc. I use TSSetIFunction and TSSetIJacobian. My initial solution is a simple gaussian with a maximum peak at 1e18. - When I use TSROSW, everything's fine. Timestep is adaptive and the solution is correct (a gaussian) - When I use TSARKIMEX, the timestep is adaptive but final solution is completely wrong. It gives some strange profiles on boundaries with very high values. Any idea why ? - When I use TSBEULER, timestep remains constant. It takes the initial timestep value given with TSSetInitialTimeStep and stays like that for ever. This makes simulations never finish. Actually, I observed this behavior with TSBEULER with the different examples given in tutorials. Why is the timestep not adaptive with TSBEULER ? Is there a way to have it adaptive ? Many thanks in advance ! Christophe -------------- next part -------------- An HTML attachment was scrubbed... URL: From choi240 at purdue.edu Thu Sep 26 03:31:41 2013 From: choi240 at purdue.edu (Joon Hee Choi) Date: Thu, 26 Sep 2013 04:31:41 -0400 (EDT) Subject: [petsc-users] About MatTransposeMatMult In-Reply-To: <1418450617.145987.1380183952313.JavaMail.root@mailhub028.itcs.purdue.edu> Message-ID: <725208471.145989.1380184301294.JavaMail.root@mailhub028.itcs.purdue.edu> Hello all, I am trying to calculate M = X^T * B. X is a seqaij and B is a seqdense matrix. However, I used MatTransposeMatMult() but I got an error. Can someone why I got the error and how I should modify my code? I am attaching my simple code, error message, and result. Thank you very much. Joon code: PetscInt m1, m2, n1, n2; MatType t1, t2; ierr = MatGetType(X, &t1); CHKERRQ(ierr); ierr = MatGetType(B, &t2); CHKERRQ(ierr); ierr = MatGetSize(X, &m1, &n1); CHKERRQ(ierr); ierr = MatGetSize(B, &m2, &n2); CHKERRQ(ierr); PetscPrintf(PETSC_COMM_SELF, "X: %s \n", t1); PetscPrintf(PETSC_COMM_SELF, "B: %s \n", t2); PetscPrintf(PETSC_COMM_SELF, "X: %d x %d \n", m1, n1); PetscPrintf(PETSC_COMM_SELF, "B: %d x %d \n", m2, n2); ierr = MatTransposeMatMult(X, B, MAT_INITIAL_MATRIX, PETSC_DEFAULT, &matM); CHKERRQ(ierr); error: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Arguments are incompatible! [0]PETSC ERROR: MatTransposeMatMult requires A, seqaij, to be compatible with B, seqdense! result: X: seqaij B: seqdense X: 3 x 12 B: 3 x 2 From christophe.ortiz at ciemat.es Thu Sep 26 04:41:39 2013 From: christophe.ortiz at ciemat.es (Christophe Ortiz) Date: Thu, 26 Sep 2013 11:41:39 +0200 Subject: [petsc-users] Back to struct in Fortran to represent field with dof > 1 Message-ID: Hi all, Me again ! I have read in previous posts that it is hard in Fortran to declare something similar to a typedef struct in C to manage a multicomponent problem. Is it still the case ? Has the problem been solved ? I am asking because my plan is to implement a multicomponent problem (in 1D), with many components that will be organized in arrays of two dimensions. In C I could define typedef struct{ PetscScalar U; PetscScalar V; PetscScalar A[N][N]; } Field; and then I could calculate the residual function with F[i].U = ... F[i].V = ... F[i].A[k][n] = ... which is quite convenient. If in Fortran it is not possible to use a struct as in C, I am afraid I'll have to deal with F(jdof,i) = ... where I will have only jdof to address U, V and A[ ][ ], which can be difficult and not very convenient I guess. Before I move all my code to C, does anyone have an alternative idea to manage this multi(many)component problem in Fortran ? Many thanks in advance for your help and suggestion ! Christophe -------------- next part -------------- An HTML attachment was scrubbed... URL: From rupp at mcs.anl.gov Thu Sep 26 04:55:24 2013 From: rupp at mcs.anl.gov (Karl Rupp) Date: Thu, 26 Sep 2013 11:55:24 +0200 Subject: [petsc-users] About MatTransposeMatMult In-Reply-To: <725208471.145989.1380184301294.JavaMail.root@mailhub028.itcs.purdue.edu> References: <725208471.145989.1380184301294.JavaMail.root@mailhub028.itcs.purdue.edu> Message-ID: <5244048C.6010705@mcs.anl.gov> Hi, > I am trying to calculate M = X^T * B. X is a seqaij and B is a seqdense matrix. This is not yet allowed in the latest 3.4. release. > However, I used MatTransposeMatMult() but I got an error. Can someone why I got the error and how I should modify my code? I am attaching my simple code, error message, and result. Thank you very much. Do you use the master branch from our Bitbucket repository? It contains the commit where Hong added the respective functionality: https://bitbucket.org/petsc/petsc/commits/8949adfd119cb1d44c9fb5cf3dc01f4b6a02252d Best regards, Karli From john.fettig at gmail.com Thu Sep 26 07:52:29 2013 From: john.fettig at gmail.com (John Fettig) Date: Thu, 26 Sep 2013 08:52:29 -0400 Subject: [petsc-users] On "very poor scalability" on multicore architectures Message-ID: http://onlinelibrary.wiley.com/doi/10.1002/fld.3827/abstract They do shift the point where strong scalability breaks down, which is notable, but their claim that "employing PETSc in its elemental form for high performance computing (HPC) applications would yield very poor scalability" seems exaggerated to put it politely. Is there anything we can take away from this paper? Regards, John -------------- next part -------------- An HTML attachment was scrubbed... URL: From olivier.bonnefon at avignon.inra.fr Thu Sep 26 08:04:50 2013 From: olivier.bonnefon at avignon.inra.fr (Olivier Bonnefon) Date: Thu, 26 Sep 2013 15:04:50 +0200 Subject: [petsc-users] ex12 with Neumann BC Message-ID: <524430F2.2020507@avignon.inra.fr> Hello, I have implemented my own system from ex12.c. It works with Dirichlet BC, but failed with Neumann one. So, I'm came back to the example /src/snes/example/tutorial/ex12.c, and I tried with Neumann BC: ./ex12 -bc_type NEUMANN This leads to the following crach: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: No support for this operation for this object type! [0]PETSC ERROR: Unsupported number of vertices 0 in cell 8 for element geometry computation! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./ex12 on a arch-linux2-c-debug named pcbiom38 by olivierb Thu Sep 26 14:53:32 2013 [0]PETSC ERROR: Libraries linked from /home/olivierb/SOFT/petsc-3.4.2/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Thu Sep 26 14:44:42 2013 [0]PETSC ERROR: Configure options --with-debugging=1 --download-fiat --download-scientificpython --download-generator --download-triangle --download-ctetgen --download-chaco --download-netcdf --download-hdf5 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: DMPlexComputeCellGeometry() line 732 in /home/olivierb/SOFT/petsc-3.4.2/src/dm/impls/plex/plexgeometry.c [0]PETSC ERROR: DMPlexComputeResidualFEM() line 558 in /home/olivierb/SOFT/petsc-3.4.2/src/dm/impls/plex/plexfem.c [0]PETSC ERROR: SNESComputeFunction_DMLocal() line 75 in /home/olivierb/SOFT/petsc-3.4.2/src/snes/utils/dmlocalsnes.c [0]PETSC ERROR: SNESComputeFunction() line 1988 in /home/olivierb/SOFT/petsc-3.4.2/src/snes/interface/snes.c [0]PETSC ERROR: SNESSolve_NEWTONLS() line 162 in /home/olivierb/SOFT/petsc-3.4.2/src/snes/impls/ls/ls.c [0]PETSC ERROR: SNESSolve() line 3636 in /home/olivierb/SOFT/petsc-3.4.2/src/snes/interface/snes.c [0]PETSC ERROR: main() line 582 in "unknowndirectory/"/home/olivierb/solvers/trunk/SandBox/PETSC/LANDSCAPE/REF/ex12.c -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 56. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- With Gbd, I saw that DMPlexGetConeSize is 0 for the last point. Do I have forget a step to use Neumann BC ? Thanks Olivier Bonnefon -- Olivier Bonnefon INRA PACA-Avignon, Unit? BioSP Tel: +33 (0)4 32 72 21 58 From knepley at gmail.com Thu Sep 26 08:24:24 2013 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 26 Sep 2013 06:24:24 -0700 Subject: [petsc-users] On "very poor scalability" on multicore architectures In-Reply-To: References: Message-ID: On Thu, Sep 26, 2013 at 5:52 AM, John Fettig wrote: > http://onlinelibrary.wiley.com/doi/10.1002/fld.3827/abstract > > They do shift the point where strong scalability breaks down, which is > notable, but their claim that "employing PETSc in its elemental form for > high performance computing (HPC) applications would yield very poor > scalability" seems exaggerated to put it politely. > > Is there anything we can take away from this paper? > I think I would have to get out my novelty microscope. Matt > Regards, > John > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Sep 26 09:09:48 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 26 Sep 2013 09:09:48 -0500 Subject: [petsc-users] Back to struct in Fortran to represent field with dof > 1 In-Reply-To: References: Message-ID: Christophe, Despite my LinkIn Endorsement for expertise in Fortran :-) I cannot pretend to be an expert in FortranXX but I have cooked up an example demonstrating accessing the Vec entries as if they are in an array of derived types. I've attached the example code; there needs to be a small C stub that defines the functions for your specific derived type name. Note that it will only work I think if your N is a compile time constant. -------------- next part -------------- A non-text attachment was scrubbed... Name: ex1.c Type: application/octet-stream Size: 1378 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex1f90.F Type: application/octet-stream Size: 7274 bytes Desc: not available URL: -------------- next part -------------- It worked with ~/Downloads$ gfortran --version GNU Fortran (GCC) 4.8.1 20130404 (prerelease) I do not understand exactly why it works since it uses F90Array1dCreate(fa,PETSC_SCALAR,1,len,ptr PETSC_F90_2PTR_PARAM(ptrd)); which has a single PETSC_SCALAR as a building block but ? I hope it works for you. If it doesn't, let us know the compiler you are using and we may be able to get it working for that compiler. Barry On Sep 26, 2013, at 4:41 AM, Christophe Ortiz wrote: > Hi all, > > Me again ! > > I have read in previous posts that it is hard in Fortran to declare something similar to a typedef struct in C to manage a multicomponent problem. > > Is it still the case ? Has the problem been solved ? > > I am asking because my plan is to implement a multicomponent problem (in 1D), with many components that will be organized in arrays of two dimensions. In C I could define > > typedef struct{ > PetscScalar U; > PetscScalar V; > PetscScalar A[N][N]; > } Field; > > and then I could calculate the residual function with > > F[i].U = ... > F[i].V = ... > F[i].A[k][n] = ... > > which is quite convenient. > > If in Fortran it is not possible to use a struct as in C, I am afraid I'll have to deal with > > F(jdof,i) = ... > > where I will have only jdof to address U, V and A[ ][ ], which can be difficult and not very convenient I guess. Before I move all my code to C, does anyone have an alternative idea to manage this multi(many)component problem in Fortran ? > > Many thanks in advance for your help and suggestion ! > > Christophe From hzhang at mcs.anl.gov Thu Sep 26 09:50:54 2013 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Thu, 26 Sep 2013 09:50:54 -0500 Subject: [petsc-users] About MatTransposeMatMult In-Reply-To: <5244048C.6010705@mcs.anl.gov> References: <725208471.145989.1380184301294.JavaMail.root@mailhub028.itcs.purdue.edu> <5244048C.6010705@mcs.anl.gov> Message-ID: Joon, Please send ENTIRE error message when reporting problems, from which we would know which petsc version and specific line in the petsc library that produced error. Hong On Thu, Sep 26, 2013 at 4:55 AM, Karl Rupp wrote: > Hi, > > > I am trying to calculate M = X^T * B. X is a seqaij and B is a seqdense >> matrix. >> > > This is not yet allowed in the latest 3.4. release. > > > However, I used MatTransposeMatMult() but I got an error. Can someone why >> I got the error and how I should modify my code? I am attaching my simple >> code, error message, and result. Thank you very much. >> > > Do you use the master branch from our Bitbucket repository? It contains > the commit where Hong added the respective functionality: > https://bitbucket.org/petsc/**petsc/commits/** > 8949adfd119cb1d44c9fb5cf3dc01f**4b6a02252d > > Best regards, > Karli > -------------- next part -------------- An HTML attachment was scrubbed... URL: From christophe.ortiz at ciemat.es Thu Sep 26 09:51:53 2013 From: christophe.ortiz at ciemat.es (Christophe Ortiz) Date: Thu, 26 Sep 2013 16:51:53 +0200 Subject: [petsc-users] Back to struct in Fortran to represent field with dof > 1 In-Reply-To: References: Message-ID: Hi Barry, Thanks a lot for the codes ! I will test it and let you know if it works. Christophe CIEMAT Laboratorio Nacional de Fusi?n por Confinamiento Magn?tico Unidad de Materiales Edificio 2 - Planta 0 - Despacho 28m Avenida Complutense 40, 28040 Madrid, Spain Tel: +34 91496 2582 Fax: +34 91346 6442 -- Q Por favor, piense en el medio ambiente antes de imprimir este mensaje. Please consider the environment before printing this email. On Thu, Sep 26, 2013 at 4:09 PM, Barry Smith wrote: > > Christophe, > > Despite my LinkIn Endorsement for expertise in Fortran :-) I cannot > pretend to be an expert in FortranXX but I have cooked up an example > demonstrating accessing the Vec entries as if they are in an array of > derived types. I've attached the example code; there needs to be a small C > stub that defines the functions for your specific derived type name. > Note that it will only work I think if your N is a compile time constant. > > > > > It worked with > ~/Downloads$ gfortran --version > GNU Fortran (GCC) 4.8.1 20130404 (prerelease) > > > I do not understand exactly why it works since it uses > F90Array1dCreate(fa,PETSC_SCALAR,1,len,ptr PETSC_F90_2PTR_PARAM(ptrd)); > which has a single PETSC_SCALAR as a building block but ? I hope it works > for you. If it doesn't, let us know the compiler you are using and we may > be able to get it working for that compiler. > > Barry > > > > On Sep 26, 2013, at 4:41 AM, Christophe Ortiz > wrote: > > > Hi all, > > > > Me again ! > > > > I have read in previous posts that it is hard in Fortran to declare > something similar to a typedef struct in C to manage a multicomponent > problem. > > > > Is it still the case ? Has the problem been solved ? > > > > I am asking because my plan is to implement a multicomponent problem (in > 1D), with many components that will be organized in arrays of two > dimensions. In C I could define > > > > typedef struct{ > > PetscScalar U; > > PetscScalar V; > > PetscScalar A[N][N]; > > } Field; > > > > and then I could calculate the residual function with > > > > F[i].U = ... > > F[i].V = ... > > F[i].A[k][n] = ... > > > > which is quite convenient. > > > > If in Fortran it is not possible to use a struct as in C, I am afraid > I'll have to deal with > > > > F(jdof,i) = ... > > > > where I will have only jdof to address U, V and A[ ][ ], which can be > difficult and not very convenient I guess. Before I move all my code to C, > does anyone have an alternative idea to manage this multi(many)component > problem in Fortran ? > > > > Many thanks in advance for your help and suggestion ! > > > > Christophe > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Sep 26 10:01:42 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 26 Sep 2013 16:01:42 +0100 Subject: [petsc-users] About MatTransposeMatMult In-Reply-To: <5244048C.6010705@mcs.anl.gov> References: <725208471.145989.1380184301294.JavaMail.root@mailhub028.itcs.purdue.edu> <5244048C.6010705@mcs.anl.gov> Message-ID: <8738or8w4p.fsf@mcs.anl.gov> Karl Rupp writes: > Hi, > >> I am trying to calculate M = X^T * B. X is a seqaij and B is a seqdense matrix. > > This is not yet allowed in the latest 3.4. release. You can explicitly transpose X: MatTranspose(X,MAT_INITIAL_MATRIX,&Xt); and then use MatMatMult. >> However, I used MatTransposeMatMult() but I got an error. Can someone why I got the error and how I should modify my code? I am attaching my simple code, error message, and result. Thank you very much. > > Do you use the master branch from our Bitbucket repository? It contains > the commit where Hong added the respective functionality: > https://bitbucket.org/petsc/petsc/commits/8949adfd119cb1d44c9fb5cf3dc01f4b6a02252d This is only for MPI. Hong, can you expose MatTransposeMatMult_SeqAIJ_SeqDense? -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From msoda_1414 at naver.com Thu Sep 26 10:10:31 2013 From: msoda_1414 at naver.com (M . S) Date: Fri, 27 Sep 2013 00:10:31 +0900 (KST) Subject: [petsc-users] =?utf-8?q?Hello=2E?= Message-ID: Hello. I assume you and your family are in good health. I am the Foreign Operations Manager at one of the leading generation bank here in Burkina Faso West Africa. This being a wide world in which it can be difficult to make new acquaintances and because it is virtually impossible to know who is trustworthy and who can be believed, i have decided to repose confidence in you after much fasting and prayer. It is only because of this that I have decided to confide in you and to share with you this confidential business. In my bank; there resides an overdue and abandoned huge amount of money. When the account holder suddenly passed on, he left no beneficiary who would be entitled to the receipt of the amount. For this reason, I have found it expedient to transfer this currency to a trustworthy individual with capacity to act as foreign business partner. Thus i humbly request your assistance to claim this amount. Upon the transfer of this amount in your account, you will take 45% as your share from the total currency, 10% will be shared to Charity Organizations in both country and 45% will be for me. Please if you are really sure you can handle this project, contact me immediately for details of the amount involved.I am looking forward to read from you soon. Thanks Your Good friend, -------------- next part -------------- An HTML attachment was scrubbed... URL: From christophe.ortiz at ciemat.es Thu Sep 26 10:14:59 2013 From: christophe.ortiz at ciemat.es (Christophe Ortiz) Date: Thu, 26 Sep 2013 17:14:59 +0200 Subject: [petsc-users] Back to struct in Fortran to represent field with dof > 1 In-Reply-To: References: Message-ID: By the way, what should I do with the small .c code ? Where should I put it ? Christophe > > On Thu, Sep 26, 2013 at 4:09 PM, Barry Smith wrote: > >> >> Christophe, >> >> Despite my LinkIn Endorsement for expertise in Fortran :-) I cannot >> pretend to be an expert in FortranXX but I have cooked up an example >> demonstrating accessing the Vec entries as if they are in an array of >> derived types. I've attached the example code; there needs to be a small C >> stub that defines the functions for your specific derived type name. >> Note that it will only work I think if your N is a compile time constant. >> >> >> >> >> It worked with >> ~/Downloads$ gfortran --version >> GNU Fortran (GCC) 4.8.1 20130404 (prerelease) >> >> >> I do not understand exactly why it works since it uses >> F90Array1dCreate(fa,PETSC_SCALAR,1,len,ptr PETSC_F90_2PTR_PARAM(ptrd)); >> which has a single PETSC_SCALAR as a building block but ? I hope it works >> for you. If it doesn't, let us know the compiler you are using and we may >> be able to get it working for that compiler. >> >> Barry >> >> >> >> >> On Sep 26, 2013, at 4:41 AM, Christophe Ortiz >> wrote: >> >> > Hi all, >> > >> > Me again ! >> > >> > I have read in previous posts that it is hard in Fortran to declare >> something similar to a typedef struct in C to manage a multicomponent >> problem. >> > >> > Is it still the case ? Has the problem been solved ? >> > >> > I am asking because my plan is to implement a multicomponent problem >> (in 1D), with many components that will be organized in arrays of two >> dimensions. In C I could define >> > >> > typedef struct{ >> > PetscScalar U; >> > PetscScalar V; >> > PetscScalar A[N][N]; >> > } Field; >> > >> > and then I could calculate the residual function with >> > >> > F[i].U = ... >> > F[i].V = ... >> > F[i].A[k][n] = ... >> > >> > which is quite convenient. >> > >> > If in Fortran it is not possible to use a struct as in C, I am afraid >> I'll have to deal with >> > >> > F(jdof,i) = ... >> > >> > where I will have only jdof to address U, V and A[ ][ ], which can be >> difficult and not very convenient I guess. Before I move all my code to C, >> does anyone have an alternative idea to manage this multi(many)component >> problem in Fortran ? >> > >> > Many thanks in advance for your help and suggestion ! >> > >> > Christophe >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Sep 26 10:45:42 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 26 Sep 2013 10:45:42 -0500 Subject: [petsc-users] Back to struct in Fortran to represent field with dof > 1 In-Reply-To: References: Message-ID: <3524B3B9-C5FE-46A7-8556-F36DEEC287BD@mcs.anl.gov> I just put it with the Fortran source code and compile it with the rest of the application code; here is the makefile I used -------------- next part -------------- A non-text attachment was scrubbed... Name: makefile Type: application/octet-stream Size: 13358 bytes Desc: not available URL: -------------- next part -------------- On Sep 26, 2013, at 10:14 AM, Christophe Ortiz wrote: > By the way, what should I do with the small .c code ? Where should I put it ? > > Christophe > > > > On Thu, Sep 26, 2013 at 4:09 PM, Barry Smith wrote: > > Christophe, > > Despite my LinkIn Endorsement for expertise in Fortran :-) I cannot pretend to be an expert in FortranXX but I have cooked up an example demonstrating accessing the Vec entries as if they are in an array of derived types. I've attached the example code; there needs to be a small C stub that defines the functions for your specific derived type name. > Note that it will only work I think if your N is a compile time constant. > > > > > It worked with > ~/Downloads$ gfortran --version > GNU Fortran (GCC) 4.8.1 20130404 (prerelease) > > > I do not understand exactly why it works since it uses F90Array1dCreate(fa,PETSC_SCALAR,1,len,ptr PETSC_F90_2PTR_PARAM(ptrd)); which has a single PETSC_SCALAR as a building block but ? I hope it works for you. If it doesn't, let us know the compiler you are using and we may be able to get it working for that compiler. > > Barry > > > > > On Sep 26, 2013, at 4:41 AM, Christophe Ortiz wrote: > > > Hi all, > > > > Me again ! > > > > I have read in previous posts that it is hard in Fortran to declare something similar to a typedef struct in C to manage a multicomponent problem. > > > > Is it still the case ? Has the problem been solved ? > > > > I am asking because my plan is to implement a multicomponent problem (in 1D), with many components that will be organized in arrays of two dimensions. In C I could define > > > > typedef struct{ > > PetscScalar U; > > PetscScalar V; > > PetscScalar A[N][N]; > > } Field; > > > > and then I could calculate the residual function with > > > > F[i].U = ... > > F[i].V = ... > > F[i].A[k][n] = ... > > > > which is quite convenient. > > > > If in Fortran it is not possible to use a struct as in C, I am afraid I'll have to deal with > > > > F(jdof,i) = ... > > > > where I will have only jdof to address U, V and A[ ][ ], which can be difficult and not very convenient I guess. Before I move all my code to C, does anyone have an alternative idea to manage this multi(many)component problem in Fortran ? > > > > Many thanks in advance for your help and suggestion ! > > > > Christophe > > > > From hzhang at mcs.anl.gov Thu Sep 26 11:12:53 2013 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Thu, 26 Sep 2013 11:12:53 -0500 Subject: [petsc-users] About MatTransposeMatMult In-Reply-To: <8738or8w4p.fsf@mcs.anl.gov> References: <725208471.145989.1380184301294.JavaMail.root@mailhub028.itcs.purdue.edu> <5244048C.6010705@mcs.anl.gov> <8738or8w4p.fsf@mcs.anl.gov> Message-ID: Jed, > > > > > Do you use the master branch from our Bitbucket repository? It contains > > the commit where Hong added the respective functionality: > > > https://bitbucket.org/petsc/petsc/commits/8949adfd119cb1d44c9fb5cf3dc01f4b6a02252d > > This is only for MPI. Hong, can you expose > MatTransposeMatMult_SeqAIJ_SeqDense? > What do you mean "expose MatTransposeMatMult_SeqAIJ_SeqDense"? MatTransposeMatMult_SeqAIJ_SeqDense() is being called for sequential runs. Joon, Can you run petsc/src/mat/examples/tests/ex163.c sequentially? Hong -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Sep 26 11:54:27 2013 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 26 Sep 2013 09:54:27 -0700 Subject: [petsc-users] ex12 with Neumann BC In-Reply-To: <524430F2.2020507@avignon.inra.fr> References: <524430F2.2020507@avignon.inra.fr> Message-ID: On Thu, Sep 26, 2013 at 6:04 AM, Olivier Bonnefon < olivier.bonnefon at avignon.inra.fr> wrote: > Hello, > > I have implemented my own system from ex12.c. It works with Dirichlet BC, > but failed with Neumann one. > > So, I'm came back to the example /src/snes/example/tutorial/**ex12.c, and > I tried with Neumann BC: > > ./ex12 -bc_type NEUMANN > Here is the full list of tests I run (just checked that it passes in 'next'): https://bitbucket.org/petsc/petsc/src/f34a81fe8510aa025c9247a5b14f0fe30e3c0bed/config/builder.py?at=master#cl-175 Make sure you use an interpolated mesh with Neumann conditions since you need faces. Matt > This leads to the following crach: > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------**------ > [0]PETSC ERROR: No support for this operation for this object type! > [0]PETSC ERROR: Unsupported number of vertices 0 in cell 8 for element > geometry computation! > [0]PETSC ERROR: ------------------------------** > ------------------------------**------------ > [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------** > ------------------------------**------------ > [0]PETSC ERROR: ./ex12 on a arch-linux2-c-debug named pcbiom38 by olivierb > Thu Sep 26 14:53:32 2013 > [0]PETSC ERROR: Libraries linked from /home/olivierb/SOFT/petsc-3.4.** > 2/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Thu Sep 26 14:44:42 2013 > [0]PETSC ERROR: Configure options --with-debugging=1 --download-fiat > --download-scientificpython --download-generator --download-triangle > --download-ctetgen --download-chaco --download-netcdf --download-hdf5 > [0]PETSC ERROR: ------------------------------** > ------------------------------**------------ > [0]PETSC ERROR: DMPlexComputeCellGeometry() line 732 in > /home/olivierb/SOFT/petsc-3.4.**2/src/dm/impls/plex/**plexgeometry.c > [0]PETSC ERROR: DMPlexComputeResidualFEM() line 558 in > /home/olivierb/SOFT/petsc-3.4.**2/src/dm/impls/plex/plexfem.c > [0]PETSC ERROR: SNESComputeFunction_DMLocal() line 75 in > /home/olivierb/SOFT/petsc-3.4.**2/src/snes/utils/dmlocalsnes.c > [0]PETSC ERROR: SNESComputeFunction() line 1988 in > /home/olivierb/SOFT/petsc-3.4.**2/src/snes/interface/snes.c > [0]PETSC ERROR: SNESSolve_NEWTONLS() line 162 in > /home/olivierb/SOFT/petsc-3.4.**2/src/snes/impls/ls/ls.c > [0]PETSC ERROR: SNESSolve() line 3636 in /home/olivierb/SOFT/petsc-3.4.** > 2/src/snes/interface/snes.c > [0]PETSC ERROR: main() line 582 in "unknowndirectory/"/home/** > olivierb/solvers/trunk/**SandBox/PETSC/LANDSCAPE/REF/**ex12.c > ------------------------------**------------------------------** > -------------- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > with errorcode 56. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > ------------------------------**------------------------------** > -------------- > > With Gbd, I saw that DMPlexGetConeSize is 0 for the last point. > > Do I have forget a step to use Neumann BC ? > > Thanks > Olivier Bonnefon > > -- > Olivier Bonnefon > INRA PACA-Avignon, Unit? BioSP > Tel: +33 (0)4 32 72 21 58 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From pengxwang at hotmail.com Thu Sep 26 13:34:40 2013 From: pengxwang at hotmail.com (Roc Wang) Date: Thu, 26 Sep 2013 13:34:40 -0500 Subject: [petsc-users] compiling error after re-installation with --download_hypre Message-ID: Hello, I re-installed the petsc with --download_hypre and --download_ml. The two test examples ex19 and ex5f run successfully. I added the libraries link in the makefile like: -Wl,-rpath,${PETSC_DIR}/${PETSC_ARCH}/lib -L${PETSC_DIR}/${PETSC_ARCH}/lib -lHYPRE -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -lmpichcxx -lstdc++ \ -Wl,-rpath,${PETSC_DIR}/${PETSC_ARCH}/lib -L${PETSC_DIR}/${PETSC_ARCH}/lib -lml -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -lmpichcxx -lstdc++ \ However, the compiling errors occurred: /home/pwang/soft/petsc-3.3-p6/arch-linux2-c-debug/lib/libmpich.a(init.o): In function `PMPI_Init': /home/pwang/soft/petsc-3.3-p6/externalpackages/mpich2-1.4.1p1/src/mpi/init/init.c:106: undefined reference to `MPL_env2str' /home/pwang/soft/petsc-3.3-p6/externalpackages/mpich2-1.4.1p1/src/mpi/init/init.c:132: undefined reference to `MPL_env2bool' /home/pwang/soft/petsc-3.3-p6/arch-linux2-c-debug/lib/libmpich.a(initthread.o): In function `PMPI_Init_thread': /home/pwang/soft/petsc-3.3-p6/externalpackages/mpich2-1.4.1p1/src/mpi/init/initthread.c:535: undefined reference to `MPL_env2bool' /home/pwang/soft/petsc-3.3-p6/arch-linux2-c-debug/lib/libmpich.a(param_vals.o): In function `MPIR_Param_init_params': /home/pwang/soft/petsc-3.3-p6/externalpackages/mpich2-1.4.1p1/src/util/param/param_vals.c:249: undefined reference to `MPL_env2int' /home/pwang/soft/petsc-3.3-p6/externalpackages/mpich2-1.4.1p1/src/util/param/param_vals.c:251: undefined reference to `MPL_env2int' /home/pwang/soft/petsc-3.3-p6/externalpackages/mpich2-1.4.1p1/src/util/param/param_vals.c:254: undefined reference to `MPL_env2int' /home/pwang/soft/petsc-3.3-p6/externalpackages/mpich2-1.4.1p1/src/util/param/param_vals.c:256: undefined reference to `MPL_env2int' /home/pwang/soft/petsc-3.3-p6/externalpackages/mpich2-1.4.1p1/src/util/param/param_vals.c:259: undefined reference to `MPL_env2int' /home/pwang/soft/petsc-3.3-p6/arch-linux2-c-debug/lib/libmpich.a(param_vals.o):/home/pwang/soft/petsc-3.3-p6/externalpackages/mpich2-1.4.1p1/src/util/param/param_vals.c:261: more undefined references to `MPL_env2int' follow ... ... It seems there is some library link I missed in the makefile. What should I add? Why it didn't happen before I re-install? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Sep 26 13:40:48 2013 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 26 Sep 2013 11:40:48 -0700 Subject: [petsc-users] compiling error after re-installation with --download_hypre In-Reply-To: References: Message-ID: On Thu, Sep 26, 2013 at 11:34 AM, Roc Wang wrote: > Hello, > > I re-installed the petsc with --download_hypre and --download_ml. The > two test examples ex19 and ex5f run successfully. > > I added the libraries link in the makefile like: > > -Wl,-rpath,${PETSC_DIR}/${PETSC_ARCH}/lib > -L${PETSC_DIR}/${PETSC_ARCH}/lib -lHYPRE > -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -lmpichcxx -lstdc++ \ > -Wl,-rpath,${PETSC_DIR}/${PETSC_ARCH}/lib > -L${PETSC_DIR}/${PETSC_ARCH}/lib -lml > -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -lmpichcxx -lstdc++ \ > > However, the compiling errors occurred: > > /home/pwang/soft/petsc-3.3-p6/arch-linux2-c-debug/lib/libmpich.a(init.o): > In function `PMPI_Init': > /home/pwang/soft/petsc-3.3-p6/externalpackages/mpich2-1.4.1p1/src/mpi/init/init.c:106: > undefined reference to `MPL_env2str' > /home/pwang/soft/petsc-3.3-p6/externalpackages/mpich2-1.4.1p1/src/mpi/init/init.c:132: > undefined reference to `MPL_env2bool' > /home/pwang/soft/petsc-3.3-p6/arch-linux2-c-debug/lib/libmpich.a(initthread.o): > In function `PMPI_Init_thread': > /home/pwang/soft/petsc-3.3-p6/externalpackages/mpich2-1.4.1p1/src/mpi/init/initthread.c:535: > undefined reference to `MPL_env2bool' > /home/pwang/soft/petsc-3.3-p6/arch-linux2-c-debug/lib/libmpich.a(param_vals.o): > In function `MPIR_Param_init_params': > /home/pwang/soft/petsc-3.3-p6/externalpackages/mpich2-1.4.1p1/src/util/param/param_vals.c:249: > undefined reference to `MPL_env2int' > /home/pwang/soft/petsc-3.3-p6/externalpackages/mpich2-1.4.1p1/src/util/param/param_vals.c:251: > undefined reference to `MPL_env2int' > /home/pwang/soft/petsc-3.3-p6/externalpackages/mpich2-1.4.1p1/src/util/param/param_vals.c:254: > undefined reference to `MPL_env2int' > /home/pwang/soft/petsc-3.3-p6/externalpackages/mpich2-1.4.1p1/src/util/param/param_vals.c:256: > undefined reference to `MPL_env2int' > /home/pwang/soft/petsc-3.3-p6/externalpackages/mpich2-1.4.1p1/src/util/param/param_vals.c:259: > undefined reference to `MPL_env2int' > /home/pwang/soft/petsc-3.3-p6/arch-linux2-c-debug/lib/libmpich.a(param_vals.o):/home/pwang/soft/petsc-3.3-p6/externalpackages/mpich2-1.4.1p1/src/util/param/param_vals.c:261: > more undefined references to `MPL_env2int' follow > ... ... > > It seems there is some library link I missed in the makefile. What should > I add? Why it didn't happen before I re-install? Thanks. > make getlinklibs Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From choi240 at purdue.edu Thu Sep 26 13:41:14 2013 From: choi240 at purdue.edu (Joon Hee Choi) Date: Thu, 26 Sep 2013 14:41:14 -0400 (EDT) Subject: [petsc-users] About MatTransposeMatMult In-Reply-To: Message-ID: <343513237.147671.1380220874947.JavaMail.root@mailhub028.itcs.purdue.edu> Hello Hong, I am attaching the whole error messages. I am using Petsc3.4.1. Thank you. Joon [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Arguments are incompatible! [0]PETSC ERROR: MatTransposeMatMult requires A, seqaij, to be compatible with B, seqdense! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.4.1, Jun, 10, 2013 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./tenfac on a linux named rossmann-b000.rcac.purdue.edu by choi240 Thu Sep 26 14:39:01 2013 [0]PETSC ERROR: Libraries linked from /apps/rhel5/petsc-3.4.1/64/impi-4.1.0.030_intel-13.0.1.117_ind64/linux/lib [0]PETSC ERROR: Configure run at Sun Jun 23 10:20:51 2013 [0]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --download-sowing --with-scalar-type=real --with-shared-libraries=1 --with-pic=1 --with-clanguage=C++ --with-fortran --with-fortran-kernels=1 --with-64-bit-indices=1 --with-debugging=0 --with-blas-lapack-dir=/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64 --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --download-hdf5=no --download-metis=no --download-parmetis=no --download-superlu_dist=no --download-mumps=no --download-scalapack=yes --download-blacs=no --download-hypre=no --download-spooles=no [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatTransposeMatMult() line 8850 in /apps/rhel5/petsc-3.4.1/64/impi-4.1.0.030_intel-13.0.1.117_ind64/src/mat/interface/matrix.c [0]PETSC ERROR: cal_M1() line 36 in tenfac.cpp [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.4.1, Jun, 10, 2013 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./tenfac on a linux named rossmann-b000.rcac.purdue.edu by choi240 Thu Sep 26 14:39:01 2013 [0]PETSC ERROR: Libraries linked from /apps/rhel5/petsc-3.4.1/64/impi-4.1.0.030_intel-13.0.1.117_ind64/linux/lib [0]PETSC ERROR: Configure run at Sun Jun 23 10:20:51 2013 [0]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --download-sowing --with-scalar-type=real --with-shared-libraries=1 --with-pic=1 --with-clanguage=C++ --with-fortran --with-fortran-kernels=1 --with-64-bit-indices=1 --with-debugging=0 --with-blas-lapack-dir=/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64 --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --download-hdf5=no --download-metis=no --download-parmetis=no --download-superlu_dist=no --download-mumps=no --download-scalapack=yes --download-blacs=no --download-hypre=no --download-spooles=no [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 ----- Original Message ----- From: "Hong Zhang" To: "Karl Rupp" Cc: "Joon Hee Choi" , "PETSc users list" Sent: Thursday, September 26, 2013 10:50:54 AM Subject: Re: [petsc-users] About MatTransposeMatMult Joon, Please send ENTIRE error message when reporting problems, from which we would know which petsc version and specific line in the petsc library that produced error. Hong On Thu, Sep 26, 2013 at 4:55 AM, Karl Rupp < rupp at mcs.anl.gov > wrote: Hi, I am trying to calculate M = X^T * B. X is a seqaij and B is a seqdense matrix. This is not yet allowed in the latest 3.4. release. However, I used MatTransposeMatMult() but I got an error. Can someone why I got the error and how I should modify my code? I am attaching my simple code, error message, and result. Thank you very much. Do you use the master branch from our Bitbucket repository? It contains the commit where Hong added the respective functionality: https://bitbucket.org/petsc/ petsc/commits/ 8949adfd119cb1d44c9fb5cf3dc01f 4b6a02252d Best regards, Karli From choi240 at purdue.edu Thu Sep 26 13:43:17 2013 From: choi240 at purdue.edu (Joon Hee Choi) Date: Thu, 26 Sep 2013 14:43:17 -0400 (EDT) Subject: [petsc-users] About MatTransposeMatMult In-Reply-To: Message-ID: <805162833.147677.1380220997897.JavaMail.root@mailhub028.itcs.purdue.edu> Hello Hong, I don't have time now. So I try to run petsc/src/mat/examples/tests/ex163.c sequentially tonight and let you know. Joon ----- Original Message ----- From: "Hong Zhang" To: "Jed Brown" Cc: "Karl Rupp" , "Joon Hee Choi" , "PETSc users list" Sent: Thursday, September 26, 2013 12:12:53 PM Subject: Re: [petsc-users] About MatTransposeMatMult Jed, > > Do you use the master branch from our Bitbucket repository? It contains > the commit where Hong added the respective functionality: > https://bitbucket.org/petsc/petsc/commits/8949adfd119cb1d44c9fb5cf3dc01f4b6a02252d This is only for MPI. Hong, can you expose MatTransposeMatMult_SeqAIJ_SeqDense? What do you mean "expose MatTransposeMatMult_SeqAIJ_SeqDense"? MatTransposeMatMult_SeqAIJ_SeqDense() is being called for sequential runs. Joon, Can you run petsc/src/mat/examples/tests/ex163.c sequentially? Hong From choi240 at purdue.edu Thu Sep 26 13:45:38 2013 From: choi240 at purdue.edu (Joon Hee Choi) Date: Thu, 26 Sep 2013 14:45:38 -0400 (EDT) Subject: [petsc-users] About MatTransposeMatMult In-Reply-To: <8738or8w4p.fsf@mcs.anl.gov> Message-ID: <301155309.147696.1380221138859.JavaMail.root@mailhub028.itcs.purdue.edu> Thank you. I think MatMatMult runs well for multiplication between seqaij and seqdense. Joon ----- Original Message ----- From: "Jed Brown" To: "Karl Rupp" , "Joon Hee Choi" , petsc-users at mcs.anl.gov Cc: "Hong Zhang" Sent: Thursday, September 26, 2013 11:01:42 AM Subject: Re: [petsc-users] About MatTransposeMatMult > Karl Rupp writes: >> Hi, >> >> I am trying to calculate M = X^T * B. X is a seqaij and B is a seqdense matrix. >> >> This is not yet allowed in the latest 3.4. release. > You can explicitly transpose X: > MatTranspose(X,MAT_INITIAL_MATRIX,&Xt); > and then use MatMatMult. >> However, I used MatTransposeMatMult() but I got an error. Can someone why I got the >> error and how I should modify my code? I am attaching my simple code, error message, >> and result. Thank you very much. >> >> Do you use the master branch from our Bitbucket repository? It contains >> the commit where Hong added the respective functionality: >> https://bitbucket.org/petsc/petsc/commits/8949adfd119cb1d44c9fb5cf3dc01f4b6a02252d > This is only for MPI. Hong, can you expose MatTransposeMatMult_SeqAIJ_SeqDense? From hzhang at mcs.anl.gov Thu Sep 26 13:50:03 2013 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Thu, 26 Sep 2013 13:50:03 -0500 Subject: [petsc-users] About MatTransposeMatMult In-Reply-To: <805162833.147677.1380220997897.JavaMail.root@mailhub028.itcs.purdue.edu> References: <805162833.147677.1380220997897.JavaMail.root@mailhub028.itcs.purdue.edu> Message-ID: Joon : > > I don't have time now. So I try to run > petsc/src/mat/examples/tests/ex163.c sequentially tonight and let you know. > ex163.c requires a matrix data file, I'm attaching one for your convenience. Run it with ./ex163 -f small or you may send me your code for testing. Hong > > ----- Original Message ----- > From: "Hong Zhang" > To: "Jed Brown" > Cc: "Karl Rupp" , "Joon Hee Choi" , > "PETSc users list" > Sent: Thursday, September 26, 2013 12:12:53 PM > Subject: Re: [petsc-users] About MatTransposeMatMult > > > Jed, > > > > > > > > > > Do you use the master branch from our Bitbucket repository? It contains > > the commit where Hong added the respective functionality: > > > https://bitbucket.org/petsc/petsc/commits/8949adfd119cb1d44c9fb5cf3dc01f4b6a02252d > > This is only for MPI. Hong, can you expose > MatTransposeMatMult_SeqAIJ_SeqDense? > > > > What do you mean "expose MatTransposeMatMult_SeqAIJ_SeqDense"? > MatTransposeMatMult_SeqAIJ_SeqDense() is being called for sequential runs. > > > > Joon, > Can you run petsc/src/mat/examples/tests/ex163.c sequentially? > > > Hong > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: small Type: application/octet-stream Size: 2328 bytes Desc: not available URL: From pengxwang at hotmail.com Thu Sep 26 13:57:47 2013 From: pengxwang at hotmail.com (Roc Wang) Date: Thu, 26 Sep 2013 13:57:47 -0500 Subject: [petsc-users] compiling error after re-installation with --download_hypre In-Reply-To: References: , Message-ID: It works. Thanks a lot. Date: Thu, 26 Sep 2013 11:40:48 -0700 Subject: Re: [petsc-users] compiling error after re-installation with --download_hypre From: knepley at gmail.com To: pengxwang at hotmail.com CC: petsc-users at mcs.anl.gov On Thu, Sep 26, 2013 at 11:34 AM, Roc Wang wrote: Hello, I re-installed the petsc with --download_hypre and --download_ml. The two test examples ex19 and ex5f run successfully. I added the libraries link in the makefile like: -Wl,-rpath,${PETSC_DIR}/${PETSC_ARCH}/lib -L${PETSC_DIR}/${PETSC_ARCH}/lib -lHYPRE -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -lmpichcxx -lstdc++ \ -Wl,-rpath,${PETSC_DIR}/${PETSC_ARCH}/lib -L${PETSC_DIR}/${PETSC_ARCH}/lib -lml -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -lmpichcxx -lstdc++ \ However, the compiling errors occurred: /home/pwang/soft/petsc-3.3-p6/arch-linux2-c-debug/lib/libmpich.a(init.o): In function `PMPI_Init': /home/pwang/soft/petsc-3.3-p6/externalpackages/mpich2-1.4.1p1/src/mpi/init/init.c:106: undefined reference to `MPL_env2str' /home/pwang/soft/petsc-3.3-p6/externalpackages/mpich2-1.4.1p1/src/mpi/init/init.c:132: undefined reference to `MPL_env2bool' /home/pwang/soft/petsc-3.3-p6/arch-linux2-c-debug/lib/libmpich.a(initthread.o): In function `PMPI_Init_thread': /home/pwang/soft/petsc-3.3-p6/externalpackages/mpich2-1.4.1p1/src/mpi/init/initthread.c:535: undefined reference to `MPL_env2bool' /home/pwang/soft/petsc-3.3-p6/arch-linux2-c-debug/lib/libmpich.a(param_vals.o): In function `MPIR_Param_init_params': /home/pwang/soft/petsc-3.3-p6/externalpackages/mpich2-1.4.1p1/src/util/param/param_vals.c:249: undefined reference to `MPL_env2int' /home/pwang/soft/petsc-3.3-p6/externalpackages/mpich2-1.4.1p1/src/util/param/param_vals.c:251: undefined reference to `MPL_env2int' /home/pwang/soft/petsc-3.3-p6/externalpackages/mpich2-1.4.1p1/src/util/param/param_vals.c:254: undefined reference to `MPL_env2int' /home/pwang/soft/petsc-3.3-p6/externalpackages/mpich2-1.4.1p1/src/util/param/param_vals.c:256: undefined reference to `MPL_env2int' /home/pwang/soft/petsc-3.3-p6/externalpackages/mpich2-1.4.1p1/src/util/param/param_vals.c:259: undefined reference to `MPL_env2int' /home/pwang/soft/petsc-3.3-p6/arch-linux2-c-debug/lib/libmpich.a(param_vals.o):/home/pwang/soft/petsc-3.3-p6/externalpackages/mpich2-1.4.1p1/src/util/param/param_vals.c:261: more undefined references to `MPL_env2int' follow ... ... It seems there is some library link I missed in the makefile. What should I add? Why it didn't happen before I re-install? Thanks. make getlinklibs Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From choi240 at purdue.edu Thu Sep 26 16:17:31 2013 From: choi240 at purdue.edu (Joon Hee Choi) Date: Thu, 26 Sep 2013 17:17:31 -0400 (EDT) Subject: [petsc-users] About MatTransposeMatMult In-Reply-To: <301155309.147696.1380221138859.JavaMail.root@mailhub028.itcs.purdue.edu> Message-ID: <486227904.148296.1380230251928.JavaMail.root@mailhub028.itcs.purdue.edu> Dear Jed, Hello. I modified my code using MatTranspose and MatMatMult, and it works well. BTW, the result of the multiplication of seqaij and seqdense was seqdense. I thought C should be seqaij when A is seqaij and B is seqdense in C=A*B. Is the way to get seqaij just seqaij*seqaij? Joon ----- Original Message ----- From: "Joon Hee Choi" To: "Jed Brown" Cc: petsc-users at mcs.anl.gov Sent: Thursday, September 26, 2013 2:45:38 PM Subject: Re: [petsc-users] About MatTransposeMatMult > Thank you. I think MatMatMult runs well for multiplication between seqaij and seqdense. > Joon ----- Original Message ----- From: "Jed Brown" To: "Karl Rupp" , "Joon Hee Choi" , petsc-users at mcs.anl.gov Cc: "Hong Zhang" Sent: Thursday, September 26, 2013 11:01:42 AM Subject: Re: [petsc-users] About MatTransposeMatMult >> Karl Rupp writes: >>> Hi, >>> >>> I am trying to calculate M = X^T * B. X is a seqaij and B is a seqdense matrix. >>> >>> This is not yet allowed in the latest 3.4. release. >> You can explicitly transpose X: >> MatTranspose(X,MAT_INITIAL_MATRIX,&Xt); >> and then use MatMatMult. >>> However, I used MatTransposeMatMult() but I got an error. Can someone why I got the >>> error and how I should modify my code? I am attaching my simple code, error message, >>> and result. Thank you very much. >>> >>> Do you use the master branch from our Bitbucket repository? It contains >>> the commit where Hong added the respective functionality: >>> https://bitbucket.org/petsc/petsc/commits/8949adfd119cb1d44c9fb5cf3dc01f4b6a02252d >> This is only for MPI. Hong, can you expose MatTransposeMatMult_SeqAIJ_SeqDense? From william.perkins at pnnl.gov Thu Sep 26 16:37:30 2013 From: william.perkins at pnnl.gov (William A. Perkins) Date: Thu, 26 Sep 2013 14:37:30 -0700 Subject: [petsc-users] Cannot use KSP if I choose local Matrix/Vector ownership Message-ID: <21060.43290.839444.337643@pnl.gov> Hello, I have run into a problem using KSP. Attached is a hopefully short, simple example program (which has nothing to do with my actual application) demonstrating the problem. Also attached is an output log from several runs of the program. The program runs fine if I choose local ownership according to PetscSplitOwnership(). If, however, I slightly modify that local ownership scheme, the program will not run in parallel. In my view, KSP appears to make its own coefficient (or preconditioner, or whatever) matrix that has different local ownership than the one I give to KSPSetOperators(), and so cannot do matrix/vector multiplies. I would like to be able to choose my own local ownership. Any help would be appreciated. Hopefully, I'm not doing something stupid. Thanks. Bill -- Bill Perkins Research Engineer Hydrology Group Pacific Northwest National Laboratory 902 Battelle Boulevard P.O. Box 999, MSIN K9-36 Richland, WA 99352 USA Tel: 509-372-6131 Fax: 509-372-6089 william.perkins at pnnl.gov www.pnnl.gov -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: versteeg.cpp URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: versteeg.log URL: From bsmith at mcs.anl.gov Thu Sep 26 16:43:49 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 26 Sep 2013 16:43:49 -0500 Subject: [petsc-users] Cannot use KSP if I choose local Matrix/Vector ownership In-Reply-To: <21060.43290.839444.337643@pnl.gov> References: <21060.43290.839444.337643@pnl.gov> Message-ID: Bill, When you call ierr = MatSetSizes(A,local_size, PETSC_DECIDE,imax*jmax, imax*jmax); CHKERRXX(ierr); you should pass local_size as the third argument as well instead of PETSC_DECIDE. PETSc is making a choice here based only on the total number of columns imax*jmax and doesn't take into account what you provided for local row size. Essentially you can either let PETSc decide all the local sizes of the Vecs and Mats or you decide all of them; but mixing that will usually not work. Barry On Sep 26, 2013, at 4:37 PM, "William A. Perkins" wrote: > > Hello, > > I have run into a problem using KSP. Attached is a hopefully short, > simple example program (which has nothing to do with my actual > application) demonstrating the problem. Also attached is an output > log from several runs of the program. The program runs fine if I > choose local ownership according to PetscSplitOwnership(). If, > however, I slightly modify that local ownership scheme, the program > will not run in parallel. In my view, KSP appears to make its own > coefficient (or preconditioner, or whatever) matrix that has different > local ownership than the one I give to KSPSetOperators(), and so > cannot do matrix/vector multiplies. > > I would like to be able to choose my own local ownership. Any help > would be appreciated. Hopefully, I'm not doing something stupid. > > Thanks. > > Bill > > -- > Bill Perkins > Research Engineer > Hydrology Group > > Pacific Northwest National Laboratory > 902 Battelle Boulevard > P.O. Box 999, MSIN K9-36 > Richland, WA 99352 USA > Tel: 509-372-6131 > Fax: 509-372-6089 > william.perkins at pnnl.gov > www.pnnl.gov > /* ------------------------------------------------------------- > file: versteeg.cpp > > This is a simple test of PETSC Matrix operations and Linear > equation solvers. This problem comes from Example 7.2 in > > Versteeg, H.K. and W. Malalasekera, 1995. An introduction to > computational fluid dynamics, the finite volume method. Prentice > Hall. 257 pp. > > > ------------------------------------------------------------- */ > /* ------------------------------------------------------------- > Created October 10, 2002 by William A. Perkins > Last Change: 2013-09-26 14:32:33 d3g096 > ------------------------------------------------------------- */ > > > static const char* SCCS_ID = "$Id: versteeg.c,v 1.3 2008/05/24 23:04:39 d3g096 Exp $ Battelle PNNL"; > > static char help[] = > "Solves an example heat transfer problem from Versteeg and Malalasekera with SLES.\n\ > The following options are available:\n\ > -view_solution yes/no: write the initial and final solution to stdout\n\ > -problem yes/no: cause problem by choosing local ownership\n\n"; > > #include > #include > > /* ------------------------------------------------------------- > assemble > ------------------------------------------------------------- */ > int > assemble(const int imax, const int jmax, const double scale, Mat A, Vec b) > { > static const float k = 1000; /* conductivity, W/m/K */ > static const float t = 0.01; /* plate thickness, m */ > static const float W = 0.3; /* plate width, m */ > static const float H = 0.4; /* plate height, m */ > > PetscErrorCode ierr; > > int i, j; > float ap, aw, ae, as, an, bp; > > float dx = W/(float)imax; > float dy = H/(float)jmax; > > int iP, iN, iS, iE, iW; > > PetscScalar v; > > PetscInt lo, hi; > > ierr = MatGetOwnershipRange(A, &lo, &hi); CHKERRQ(ierr); > > for (i = 0; i < imax; i++) { > for (j = 0; j < jmax; j++) { > iP = i*jmax + j; > if (! (lo <= iP && iP < hi) ) continue; > iE = (i+1)*jmax + j; > iW = (i-1)*jmax + j; > iN = i*jmax + (j+1); > iS = i*jmax + (j-1); > > bp = 0.0; > ap = 0.0; > if (j == 0) { /* insulated south boundary */ > as = 0.0; > bp += 0.0; > ap -= 0.0; > } else { > as = (k/dx)*(dx*t); > } > > if (j == jmax - 1) { /* constant tempurature (100C) north boundary */ > an = 0.0; > bp += 2*k/dy*(dy*t)*100.0; > ap -= -2*k/dy*(dy*t); > } else { > an = (k/dx)*(dx*t); > } > > if (i == 0) { /* constant flux (500kw/m2) west boundary */ > aw = 0.0; > bp += 500000.0*dy*t; > ap -= 0.0; > } else { > aw = (k/dx)*(dx*t); > } > > if (i == imax - 1) { /* insulated east boundary */ > ae = 0.0; > bp += 0.0; > ap -= 0.0; > } else { > ae = (k/dx)*(dx*t); > } > > ap += as + an + aw + ae; > > v = ap*scale; > ierr = MatSetValues(A,1,&iP,1,&iP,&v,INSERT_VALUES); CHKERRQ(ierr); > > if (an != 0.0) { > v = -an*scale; > ierr = MatSetValues(A,1,&iP,1,&iN,&v,INSERT_VALUES); CHKERRQ(ierr); > } > if (as != 0.0) { > v = -as*scale; > ierr = MatSetValues(A,1,&iP,1,&iS,&v,INSERT_VALUES); CHKERRQ(ierr); > } > if (ae != 0.0) { > v = -ae*scale; > ierr = MatSetValues(A,1,&iP,1,&iE,&v,INSERT_VALUES); CHKERRQ(ierr); > } > if (aw != 0.0) { > v = -aw*scale; > ierr = MatSetValues(A,1,&iP,1,&iW,&v,INSERT_VALUES); CHKERRQ(ierr); > } > > v = bp*scale; > ierr = VecSetValues(b, 1, &iP, &v, INSERT_VALUES); CHKERRQ(ierr); > > } > } > > ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > > ierr = VecAssemblyBegin(b); CHKERRQ(ierr); > ierr = VecAssemblyEnd(b); CHKERRQ(ierr); > > return(0); > } > > /* ------------------------------------------------------------- > Main Program > ------------------------------------------------------------- */ > int > main(int argc, char **args) > { > > static const int imax = 3; > static const int jmax = 4; > > PetscBool print_solution(PETSC_FALSE); > PetscBool set_local_size(PETSC_FALSE); > int its; > > PetscErrorCode ierr; > > Vec x, b; > Mat A; > PetscScalar v; > KSP ksp; > VecScatter ctx; > int rank; > int nproc; > PetscInt local_size(PETSC_DECIDE); > PetscInt global_size(imax*jmax); > > PetscInitialize(&argc,&args,(char *)0,help); > > PetscOptionsGetBool(NULL, "-print_solution", &print_solution, NULL); > PetscOptionsGetBool(NULL, "-problem", &set_local_size, NULL); > > > ierr = MPI_Comm_size(PETSC_COMM_WORLD,&nproc); > ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank); > > ierr = PetscSplitOwnership(PETSC_COMM_WORLD, &local_size, &global_size); CHKERRQ(ierr); > if (set_local_size) { > > // If there is more than one processor, alter the default local > // ownership so that process 0 has one less row and the last > // process has more row. > > if (nproc > 1) { > if (rank == 0) { > local_size -= 1; > } else if (rank == nproc - 1) { > local_size += 1; > } > } > } > > ierr = MatCreate(PETSC_COMM_WORLD, &A); CHKERRQ(ierr); > > ierr = MatSetSizes(A, > local_size, PETSC_DECIDE, > imax*jmax, imax*jmax); CHKERRXX(ierr); > ierr = MatSetFromOptions(A); CHKERRQ(ierr); > ierr = MatSetUp(A); CHKERRQ(ierr); > > ierr = VecCreate(PETSC_COMM_WORLD,&x);CHKERRQ(ierr); > ierr = VecSetSizes(x,local_size,imax*jmax);CHKERRQ(ierr); > ierr = VecSetFromOptions(x);CHKERRQ(ierr); > ierr = VecDuplicate(x,&b);CHKERRQ(ierr); > > ierr = assemble(imax, jmax, 1.0, A, b); CHKERRQ(ierr); > > v = 0.0; > ierr = VecSet(x,v);CHKERRQ(ierr); > if (print_solution) { > ierr = VecView(x, PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr); > } > > ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr); > ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr); > ierr = KSPSetInitialGuessNonzero(ksp,PETSC_TRUE); CHKERRQ(ierr); > ierr = KSPSetTolerances(ksp, 1e-06, 1e-12, PETSC_DEFAULT, PETSC_DEFAULT); CHKERRQ(ierr); > > ierr = KSPSetOperators(ksp,A,A,SAME_NONZERO_PATTERN);CHKERRQ(ierr); > ierr = KSPSetUp(ksp); CHKERRQ(ierr); > ierr = KSPSolve(ksp, b, x);CHKERRQ(ierr); > ierr = KSPGetIterationNumber(ksp,&its);CHKERRQ(ierr); > ierr = PetscPrintf(PETSC_COMM_WORLD,"iterations %d\n",its);CHKERRQ(ierr); > > if (print_solution) { > ierr = VecView(x, PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr); > } > > ierr = KSPDestroy(&ksp);CHKERRQ(ierr); > ierr = VecDestroy(&x);CHKERRQ(ierr); > ierr = VecDestroy(&b);CHKERRQ(ierr); > ierr = MatDestroy(&A);CHKERRQ(ierr); > > ierr = PetscFinalize(); > return(ierr); > } >>>> mpirun -np 1 versteeg -print_solution > Vector Object: 1 MPI processes > type: seq > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > iterations 8 > Vector Object: 1 MPI processes > type: seq > 260.037 > 242.275 > 205.592 > 146.322 > 227.799 > 211.195 > 178.178 > 129.696 > 212.164 > 196.53 > 166.23 > 123.982 >>>> mpirun -np 2 versteeg -print_solution > Vector Object: 2 MPI processes > type: mpi > Process [0] > 0 > 0 > 0 > 0 > 0 > 0 > Process [1] > 0 > 0 > 0 > 0 > 0 > 0 > iterations 9 > Vector Object: 2 MPI processes > type: mpi > Process [0] > 260.037 > 242.275 > 205.592 > 146.322 > 227.799 > 211.195 > Process [1] > 178.178 > 129.696 > 212.164 > 196.53 > 166.23 > 123.982 >>>> mpirun -np 4 versteeg -print_solution > Vector Object: 4 MPI processes > type: mpi > Process [0] > 0 > 0 > 0 > Process [1] > 0 > 0 > 0 > Process [2] > 0 > 0 > 0 > Process [3] > 0 > 0 > 0 > iterations 12 > Vector Object: 4 MPI processes > type: mpi > Process [0] > 260.037 > 242.275 > 205.592 > Process [1] > 146.322 > 227.799 > 211.195 > Process [2] > 178.178 > 129.696 > 212.164 > Process [3] > 196.53 > 166.23 > 123.982 >>>> mpirun -np 2 versteeg -print_solution -problem > Vector Object: 2 MPI processes > type: mpi > Process [0] > 0 > 0 > 0 > 0 > 0 > Process [1] > 0 > 0 > 0 > 0 > 0 > 0 > 0 > [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message ------------------------------------ > [1]PETSC ERROR: Invalid argument! > [1]PETSC ERROR: Must be square matrix, rows 7 columns 6! > [1]PETSC ERROR: ------------------------------------------------------------------------ > [1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 > [1]PETSC ERROR: See docs/changes/index.html for recent updates. > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [1]PETSC ERROR: See docs/index.html for manual pages. > [1]PETSC ERROR: ------------------------------------------------------------------------ > [1]PETSC ERROR: versteeg on a arch-complex named flophouse by d3g096 Thu Sep 26 14:30:33 2013 > [1]PETSC ERROR: Libraries linked from /net/flophouse/files0/perksoft/petsc-3.4.2/arch-complex/lib > [1]PETSC ERROR: Configure run at Wed Sep 11 06:43:12 2013 > [1]PETSC ERROR: Configure options --with-scalar-type=complex --with-clanguage=C++ PETSC_ARCH=arch-complex --with-fortran-kernels=generic --download-superlu_dist --download-mumps --download-scalapack --download-parmetis --download-metis --download-elemental --with-mpi=1 --download-blacs > [1]PETSC ERROR: ------------------------------------------------------------------------ > [1]PETSC ERROR: MatGetOrdering() line 257 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/mat/order/sorder.c > [1]PETSC ERROR: PCSetUp_ILU() line 194 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/impls/factor/ilu/ilu.c > [1]PETSC ERROR: PCSetUp() line 890 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/interface/precon.c > [1]PETSC ERROR: KSPSetUp() line 278 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c > [1]PETSC ERROR: PCSetUpOnBlocks_BJacobi_Singleblock() line 646 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/impls/bjacobi/bjacobi.c > [1]PETSC ERROR: PCSetUpOnBlocks() line 924 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/interface/precon.c > [1]PETSC ERROR: KSPSetUpOnBlocks() line 165 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c > [1]PETSC ERROR: KSPSolve() line 400 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c > [1]PETSC ERROR: main() line 216 in /home/d3g096/Projects/PerformanceLDRD/src/sandbox/petsc/versteeg.cpp > --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Invalid argument! > [0]PETSC ERROR: Must be square matrix, rows 5 columns 6! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: versteeg on a arch-complex named flophouse by d3g096 Thu Sep 26 14:30:33 2013 > [0]PETSC ERROR: Libraries linked from /net/flophouse/files0/perksoft/petsc-3.4.2/arch-complex/lib > [0]PETSC ERROR: Configure run at Wed Sep 11 06:43:12 2013 > [0]PETSC ERROR: Configure options --with-scalar-type=complex --with-clanguage=C++ PETSC_ARCH=arch-complex --with-fortran-kernels=generic --download-superlu_dist --download-mumps --download-scalapack --download-parmetis --download-metis --download-elemental --with-mpi=1 --download-blacs > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatGetOrdering() line 257 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/mat/order/sorder.c > [0]PETSC ERROR: PCSetUp_ILU() line 194 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/impls/factor/ilu/ilu.c > [0]PETSC ERROR: PCSetUp() line 890 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPSetUp() line 278 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: PCSetUpOnBlocks_BJacobi_Singleblock() line 646 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/impls/bjacobi/bjacobi.c > [0]PETSC ERROR: PCSetUpOnBlocks() line 924 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPSetUpOnBlocks() line 165 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: KSPSolve() line 400 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: main() line 216 in /home/d3g096/Projects/PerformanceLDRD/src/sandbox/petsc/versteeg.cpp > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD > with errorcode 62. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > -------------------------------------------------------------------------- > mpirun has exited due to process rank 1 with PID 12459 on > node flophouse exiting without calling "finalize". This may > have caused other processes in the application to be > terminated by signals sent by mpirun (as reported here). > -------------------------------------------------------------------------- > [flophouse:12457] 1 more process has sent help message help-mpi-api.txt / mpi-abort > [flophouse:12457] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages >>>> mpirun -np 4 versteeg -print_solution -problem > Vector Object: 4 MPI processes > type: mpi > Process [0] > 0 > 0 > Process [1] > 0 > 0 > 0 > Process [2] > 0 > 0 > 0 > Process [3] > 0 > 0 > 0 > 0 > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Invalid argument! > [0]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message ------------------------------------ > [3]PETSC ERROR: Invalid argument! > [3]PETSC ERROR: Must be square matrix, rows 4 columns 3! > [3]PETSC ERROR: ------------------------------------------------------------------------ > [3]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 > [3]PETSC ERROR: See docs/changes/index.html for recent updates. > [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [3]PETSC ERROR: See docs/index.html for manual pages. > [3]PETSC ERROR: ------------------------------------------------------------------------ > [3]PETSC ERROR: versteeg on a arch-complex named flophouse by d3g096 Thu Sep 26 14:30:51 2013 > [3]PETSC ERROR: Libraries linked from /net/flophouse/files0/perksoft/petsc-3.4.2/arch-complex/lib > [3]PETSC ERROR: Configure run at Wed Sep 11 06:43:12 2013 > [3]PETSC ERROR: Configure options --with-scalar-type=complex --with-clanguage=C++ PETSC_ARCH=arch-complex --with-fortran-kernels=generic --download-superlu_dist --download-mumps --download-scalapack --download-parmetis --download-metis --download-elemental --with-mpi=1 --download-blacs > [3]PETSC ERROR: ------------------------------------------------------------------------ > [3]PETSC ERROR: MatGetOrdering() line 257 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/mat/order/sorder.c > [3]PETSC ERROR: PCSetUp_ILU() line 194 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/impls/factor/ilu/ilu.c > [3]PETSC ERROR: PCSetUp() line 890 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/interface/precon.c > [3]PETSC ERROR: KSPSetUp() line 278 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c > [3]PETSC ERROR: PCSetUpOnBlocks_BJacobi_Singleblock() line 646 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/impls/bjacobi/bjacobi.c > [3]PETSC ERROR: PCSetUpOnBlocks() line 924 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/interface/precon.c > [3]PETSC ERROR: KSPSetUpOnBlocks() line 165 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c > [3]PETSC ERROR: KSPSolve() line 400 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c > [3]PETSC ERROR: main() line 216 in /home/d3g096/Projects/PerformanceLDRD/src/sandbox/petsc/versteeg.cpp > Must be square matrix, rows 2 columns 3! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: versteeg on a arch-complex named flophouse by d3g096 Thu Sep 26 14:30:51 2013 > [0]PETSC ERROR: Libraries linked from /net/flophouse/files0/perksoft/petsc-3.4.2/arch-complex/lib > [0]PETSC ERROR: Configure run at Wed Sep 11 06:43:12 2013 > [0]PETSC ERROR: Configure options --with-scalar-type=complex --with-clanguage=C++ PETSC_ARCH=arch-complex --with-fortran-kernels=generic --download-superlu_dist --download-mumps --download-scalapack --download-parmetis --download-metis --download-elemental --with-mpi=1 --download-blacs > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatGetOrdering() line 257 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/mat/order/sorder.c > [0]PETSC ERROR: PCSetUp_ILU() line 194 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/impls/factor/ilu/ilu.c > [0]PETSC ERROR: PCSetUp() line 890 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPSetUp() line 278 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: PCSetUpOnBlocks_BJacobi_Singleblock() line 646 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/impls/bjacobi/bjacobi.c > [0]PETSC ERROR: PCSetUpOnBlocks() line 924 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPSetUpOnBlocks() line 165 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: KSPSolve() line 400 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: main() line 216 in /home/d3g096/Projects/PerformanceLDRD/src/sandbox/petsc/versteeg.cpp > [2]PETSC ERROR: --------------------- Error Message ------------------------------------ > [2]PETSC ERROR: Object is in wrong state! > [2]PETSC ERROR: Matrix is missing diagonal entry 2! > [2]PETSC ERROR: ------------------------------------------------------------------------ > [2]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 > [2]PETSC ERROR: -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD > with errorcode 62. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > See docs/changes/index.html for recent updates. > [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [2]PETSC ERROR: See docs/index.html for manual pages. > [2]PETSC ERROR: ------------------------------------------------------------------------ > [2]PETSC ERROR: versteeg on a arch-complex named flophouse by d3g096 Thu Sep 26 14:30:51 2013 > [2]PETSC ERROR: Libraries linked from /net/flophouse/files0/perksoft/petsc-3.4.2/arch-complex/lib > [2]PETSC ERROR: Configure run at Wed Sep 11 06:43:12 2013 > [2]PETSC ERROR: Configure options --with-scalar-type=complex --with-clanguage=C++ PETSC_ARCH=arch-complex --with-fortran-kernels=generic --download-superlu_dist --download-mumps --download-scalapack --download-parmetis --download-metis --download-elemental --with-mpi=1 --download-blacs > [2]PETSC ERROR: ------------------------------------------------------------------------ > [2]PETSC ERROR: MatILUFactorSymbolic_SeqAIJ_ilu0() line 1653 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/mat/impls/aij/seq/aijfact.c > [2]PETSC ERROR: MatILUFactorSymbolic_SeqAIJ() line 1756 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/mat/impls/aij/seq/aijfact.c > [2]PETSC ERROR: MatILUFactorSymbolic() line 6238 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/mat/interface/matrix.c > [2]PETSC ERROR: PCSetUp_ILU() line 204 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/impls/factor/ilu/ilu.c > [2]PETSC ERROR: PCSetUp() line 890 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/interface/precon.c > [2]PETSC ERROR: KSPSetUp() line 278 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c > [2]PETSC ERROR: PCSetUpOnBlocks_BJacobi_Singleblock() line 646 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/impls/bjacobi/bjacobi.c > [2]PETSC ERROR: PCSetUpOnBlocks() line 924 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/interface/precon.c > [2]PETSC ERROR: KSPSetUpOnBlocks() line 165 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c > [2]PETSC ERROR: KSPSolve() line 400 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c > [2]PETSC ERROR: main() line 216 in /home/d3g096/Projects/PerformanceLDRD/src/sandbox/petsc/versteeg.cpp > [1]PETSC ERROR: --------------------- Error Message ------------------------------------ > [1]PETSC ERROR: Argument out of range! > [1]PETSC ERROR: Cannot log negative flops! > [1]PETSC ERROR: ------------------------------------------------------------------------ > [1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 > [1]PETSC ERROR: See docs/changes/index.html for recent updates. > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [1]PETSC ERROR: See docs/index.html for manual pages. > [1]PETSC ERROR: ------------------------------------------------------------------------ > [1]PETSC ERROR: versteeg on a arch-complex named flophouse by d3g096 Thu Sep 26 14:30:51 2013 > [1]PETSC ERROR: Libraries linked from /net/flophouse/files0/perksoft/petsc-3.4.2/arch-complex/lib > [1]PETSC ERROR: Configure run at Wed Sep 11 06:43:12 2013 > [1]PETSC ERROR: Configure options --with-scalar-type=complex --with-clanguage=C++ PETSC_ARCH=arch-complex --with-fortran-kernels=generic --download-superlu_dist --download-mumps --download-scalapack --download-parmetis --download-metis --download-elemental --with-mpi=1 --download-blacs > [1]PETSC ERROR: ------------------------------------------------------------------------ > [1]PETSC ERROR: PetscLogFlops() line 204 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/include/petsclog.h > [1]PETSC ERROR: MatLUFactorNumeric_SeqAIJ_Inode() line 1312 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/mat/impls/aij/seq/inode.c > [1]PETSC ERROR: MatLUFactorNumeric() line 2889 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/mat/interface/matrix.c > [1]PETSC ERROR: PCSetUp_ILU() line 232 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/impls/factor/ilu/ilu.c > [1]PETSC ERROR: PCSetUp() line 890 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/interface/precon.c > [1]PETSC ERROR: KSPSetUp() line 278 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c > [1]PETSC ERROR: PCSetUpOnBlocks_BJacobi_Singleblock() line 646 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/impls/bjacobi/bjacobi.c > [1]PETSC ERROR: PCSetUpOnBlocks() line 924 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/interface/precon.c > [1]PETSC ERROR: KSPSetUpOnBlocks() line 165 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c > [1]PETSC ERROR: KSPSolve() line 400 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c > [1]PETSC ERROR: main() line 216 in /home/d3g096/Projects/PerformanceLDRD/src/sandbox/petsc/versteeg.cpp > -------------------------------------------------------------------------- > mpirun has exited due to process rank 3 with PID 12483 on > node flophouse exiting without calling "finalize". This may > have caused other processes in the application to be > terminated by signals sent by mpirun (as reported here). > -------------------------------------------------------------------------- > [flophouse:12479] 3 more processes have sent help message help-mpi-api.txt / mpi-abort > [flophouse:12479] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages From hsahasra at purdue.edu Thu Sep 26 17:40:59 2013 From: hsahasra at purdue.edu (Harshad Sahasrabudhe) Date: Thu, 26 Sep 2013 18:40:59 -0400 (EDT) Subject: [petsc-users] cudaSetDevice Message-ID: <1200617869.185648.1380235259121.JavaMail.root@mailhub027.itcs.purdue.edu> Hi, In our code, PetscInitialize() is called initially. Then, depending on whether we need libmesh or not, LibMeshInit() is called. This works fine when we are not using CUDA in PETSc. But when we are using CUDA, cudaSetDevice() gets called twice which throws an error: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Error in system call! [0]PETSC ERROR: error in cudaSetDevice cannot set while device is active in this process! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Development GIT revision: GIT Date: [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./nemo on a arch-linux2-cxx-opt named nid20743 by sahasrab Thu Sep 26 17:26:43 2013 [0]PETSC ERROR: Libraries linked from /u/sciteam/sahasrab/NEMO5/libs/petsc/build-real/arch-linux2-cxx-opt/lib [0]PETSC ERROR: Configure run at Mon Sep 23 18:59:51 2013 [0]PETSC ERROR: Configure options --with-x=0 --with-hdf5 --with-hdf5-dir=/opt/cray/hdf5-parallel/1.8.11/GNU/48 --with-scalar-type=real --with-single-library=0 --with-pic=1 --with-shared-libraries=0 --with-mpiexec=aprun -lsci --with-clanguage=C++ --with-fortran --with-debugging=1 --with-cc="cc " --with-fc="ftn " --with-cxx="CC " COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 --download-mumps=1 --download-scalapack=1 --download-blacs=1 --download-magma=1 --with-cuda-dir=/opt/nvidia/cudatoolkit/5.0.35 --with-cusp-dir=/u/sciteam/sahasrab/NEMO5/libs/petsc/cusplibrary-master --with-cuda=1 --with-thrust=1 --with-mpi-dir=/opt/cray/mpt/6.0.1/gni/mpich2-cray/81 --with-mpi=1 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: PetscOptionsCheckInitial_Private() line 575 in /u/sciteam/sahasrab/NEMO5/libs/petsc/build-real/src/sys/objects/init.c [0]PETSC ERROR: PetscInitialize() line 788 in /u/sciteam/sahasrab/NEMO5/libs/petsc/build-real/src/sys/objects/pinit.c [0]PETSC ERROR: LibMeshInit() line 457 in "unknowndirectory/"src/base/libmesh.C Can we tell PETSc when to run cudaSetDevice()? Or is there any other way to avoid this? Thanks, Harshad From jedbrown at mcs.anl.gov Thu Sep 26 17:46:47 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 26 Sep 2013 23:46:47 +0100 Subject: [petsc-users] About MatTransposeMatMult In-Reply-To: References: <725208471.145989.1380184301294.JavaMail.root@mailhub028.itcs.purdue.edu> <5244048C.6010705@mcs.anl.gov> <8738or8w4p.fsf@mcs.anl.gov> Message-ID: <87txh7kxpk.fsf@mcs.anl.gov> Hong Zhang writes: > What do you mean "expose MatTransposeMatMult_SeqAIJ_SeqDense"? > MatTransposeMatMult_SeqAIJ_SeqDense() is being called for sequential runs. Oops, stale tags in the branch when I checked. Looks like everything is implemented. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From knepley at gmail.com Thu Sep 26 19:23:14 2013 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 26 Sep 2013 17:23:14 -0700 Subject: [petsc-users] cudaSetDevice In-Reply-To: <1200617869.185648.1380235259121.JavaMail.root@mailhub027.itcs.purdue.edu> References: <1200617869.185648.1380235259121.JavaMail.root@mailhub027.itcs.purdue.edu> Message-ID: On Thu, Sep 26, 2013 at 3:40 PM, Harshad Sahasrabudhe wrote: > Hi, > > In our code, PetscInitialize() is called initially. Then, depending on > whether we need libmesh or not, LibMeshInit() is called. This works fine > when we are not using CUDA in PETSc. But when we are using CUDA, > cudaSetDevice() gets called twice which throws an error: > So you are saying cudaSetDevice() gets called before PetscInit? What would you propose for us? The problem is that we have no abstraction (like the communicator) which can tell us that you have called it. Matt > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Error in system call! > [0]PETSC ERROR: error in cudaSetDevice cannot set while device is active > in this process! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Development GIT revision: GIT Date: > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./nemo on a arch-linux2-cxx-opt named nid20743 by sahasrab > Thu Sep 26 17:26:43 2013 > [0]PETSC ERROR: Libraries linked from > /u/sciteam/sahasrab/NEMO5/libs/petsc/build-real/arch-linux2-cxx-opt/lib > [0]PETSC ERROR: Configure run at Mon Sep 23 18:59:51 2013 > [0]PETSC ERROR: Configure options --with-x=0 --with-hdf5 > --with-hdf5-dir=/opt/cray/hdf5-parallel/1.8.11/GNU/48 > --with-scalar-type=real --with-single-library=0 --with-pic=1 > --with-shared-libraries=0 --with-mpiexec=aprun -lsci --with-clanguage=C++ > --with-fortran --with-debugging=1 --with-cc="cc " --with-fc="ftn " > --with-cxx="CC " COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 > --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 > --download-mumps=1 --download-scalapack=1 --download-blacs=1 > --download-magma=1 --with-cuda-dir=/opt/nvidia/cudatoolkit/5.0.35 > --with-cusp-dir=/u/sciteam/sahasrab/NEMO5/libs/petsc/cusplibrary-master > --with-cuda=1 --with-thrust=1 > --with-mpi-dir=/opt/cray/mpt/6.0.1/gni/mpich2-cray/81 --with-mpi=1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: PetscOptionsCheckInitial_Private() line 575 in > /u/sciteam/sahasrab/NEMO5/libs/petsc/build-real/src/sys/objects/init.c > [0]PETSC ERROR: PetscInitialize() line 788 in > /u/sciteam/sahasrab/NEMO5/libs/petsc/build-real/src/sys/objects/pinit.c > [0]PETSC ERROR: LibMeshInit() line 457 in > "unknowndirectory/"src/base/libmesh.C > > > Can we tell PETSc when to run cudaSetDevice()? Or is there any other way > to avoid this? > > Thanks, > Harshad > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hsahasra at purdue.edu Thu Sep 26 19:26:54 2013 From: hsahasra at purdue.edu (Harshad Sahasrabudhe) Date: Thu, 26 Sep 2013 20:26:54 -0400 (EDT) Subject: [petsc-users] cudaSetDevice In-Reply-To: Message-ID: <527021386.185984.1380241614466.JavaMail.root@mailhub027.itcs.purdue.edu> Basically, PetscInitialize is called 2 times, the second time from libmesh. PetscInitialize calls cudaSetDevice(), which fails the second time. ----- Original Message ----- From: "Matthew Knepley" To: "Harshad Sahasrabudhe" Cc: petsc-users at mcs.anl.gov Sent: Thursday, September 26, 2013 8:23:14 PM Subject: Re: [petsc-users] cudaSetDevice On Thu, Sep 26, 2013 at 3:40 PM, Harshad Sahasrabudhe < hsahasra at purdue.edu > wrote: Hi, In our code, PetscInitialize() is called initially. Then, depending on whether we need libmesh or not, LibMeshInit() is called. This works fine when we are not using CUDA in PETSc. But when we are using CUDA, cudaSetDevice() gets called twice which throws an error: So you are saying cudaSetDevice() gets called before PetscInit? What would you propose for us? The problem is that we have no abstraction (like the communicator) which can tell us that you have called it. Matt [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Error in system call! [0]PETSC ERROR: error in cudaSetDevice cannot set while device is active in this process! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Development GIT revision: GIT Date: [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./nemo on a arch-linux2-cxx-opt named nid20743 by sahasrab Thu Sep 26 17:26:43 2013 [0]PETSC ERROR: Libraries linked from /u/sciteam/sahasrab/NEMO5/libs/petsc/build-real/arch-linux2-cxx-opt/lib [0]PETSC ERROR: Configure run at Mon Sep 23 18:59:51 2013 [0]PETSC ERROR: Configure options --with-x=0 --with-hdf5 --with-hdf5-dir=/opt/cray/hdf5-parallel/1.8.11/GNU/48 --with-scalar-type=real --with-single-library=0 --with-pic=1 --with-shared-libraries=0 --with-mpiexec=aprun -lsci --with-clanguage=C++ --with-fortran --with-debugging=1 --with-cc="cc " --with-fc="ftn " --with-cxx="CC " COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 --download-mumps=1 --download-scalapack=1 --download-blacs=1 --download-magma=1 --with-cuda-dir=/opt/nvidia/cudatoolkit/5.0.35 --with-cusp-dir=/u/sciteam/sahasrab/NEMO5/libs/petsc/cusplibrary-master --with-cuda=1 --with-thrust=1 --with-mpi-dir=/opt/cray/mpt/6.0.1/gni/mpich2-cray/81 --with-mpi=1 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: PetscOptionsCheckInitial_Private() line 575 in /u/sciteam/sahasrab/NEMO5/libs/petsc/build-real/src/sys/objects/init.c [0]PETSC ERROR: PetscInitialize() line 788 in /u/sciteam/sahasrab/NEMO5/libs/petsc/build-real/src/sys/objects/pinit.c [0]PETSC ERROR: LibMeshInit() line 457 in "unknowndirectory/"src/base/libmesh.C Can we tell PETSc when to run cudaSetDevice()? Or is there any other way to avoid this? Thanks, Harshad -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener From jedbrown at mcs.anl.gov Thu Sep 26 19:37:36 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 27 Sep 2013 01:37:36 +0100 Subject: [petsc-users] cudaSetDevice In-Reply-To: <527021386.185984.1380241614466.JavaMail.root@mailhub027.itcs.purdue.edu> References: <527021386.185984.1380241614466.JavaMail.root@mailhub027.itcs.purdue.edu> Message-ID: <877ge3kskv.fsf@mcs.anl.gov> Harshad Sahasrabudhe writes: > Basically, PetscInitialize is called 2 times, the second time from > libmesh. It should not be. $ git grep -C3 PetscInitialize src/base/libmesh.C src/base/libmesh.C- // Check whether the calling program has already initialized src/base/libmesh.C- // PETSc, and avoid duplicate Initialize/Finalize src/base/libmesh.C- PetscBool petsc_already_initialized; src/base/libmesh.C: ierr = PetscInitialized(&petsc_already_initialized); src/base/libmesh.C- CHKERRABORT(libMesh::COMM_WORLD,ierr); src/base/libmesh.C- if (petsc_already_initialized != PETSC_TRUE) src/base/libmesh.C- libmesh_initialized_petsc = true; -- src/base/libmesh.C-# else src/base/libmesh.C- if (libmesh_initialized_petsc) src/base/libmesh.C- { src/base/libmesh.C: ierr = PetscInitialize (&argc, const_cast(&argv), NULL, NULL); src/base/libmesh.C- CHKERRABORT(libMesh::COMM_WORLD,ierr); src/base/libmesh.C- } src/base/libmesh.C-# endif -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From hzhang at mcs.anl.gov Thu Sep 26 19:56:39 2013 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Thu, 26 Sep 2013 19:56:39 -0500 Subject: [petsc-users] About MatTransposeMatMult In-Reply-To: <486227904.148296.1380230251928.JavaMail.root@mailhub028.itcs.purdue.edu> References: <301155309.147696.1380221138859.JavaMail.root@mailhub028.itcs.purdue.edu> <486227904.148296.1380230251928.JavaMail.root@mailhub028.itcs.purdue.edu> Message-ID: Joon : > Dear Jed, > > Hello. I modified my code using MatTranspose and MatMatMult, and it works > well. > BTW, the result of the multiplication of seqaij and seqdense was seqdense. > I thought C should be seqaij when A is seqaij and B is seqdense in C=A*B. > When B is dense, C=A*B is a dense matrix. Is the way to get seqaij just seqaij*seqaij? > Why do you want store a dense matrix in sparse format? Hong > > ----- Original Message ----- > From: "Joon Hee Choi" > To: "Jed Brown" > Cc: petsc-users at mcs.anl.gov > Sent: Thursday, September 26, 2013 2:45:38 PM > Subject: Re: [petsc-users] About MatTransposeMatMult > > > Thank you. I think MatMatMult runs well for multiplication between > seqaij and seqdense. > > > Joon > > ----- Original Message ----- > From: "Jed Brown" > To: "Karl Rupp" , "Joon Hee Choi" , > petsc-users at mcs.anl.gov > Cc: "Hong Zhang" > Sent: Thursday, September 26, 2013 11:01:42 AM > Subject: Re: [petsc-users] About MatTransposeMatMult > > >> Karl Rupp writes: > > >>> Hi, > >>> > >>> I am trying to calculate M = X^T * B. X is a seqaij and B is a > seqdense matrix. > >>> > >>> This is not yet allowed in the latest 3.4. release. > > >> You can explicitly transpose X: > > >> MatTranspose(X,MAT_INITIAL_MATRIX,&Xt); > > >> and then use MatMatMult. > > >>> However, I used MatTransposeMatMult() but I got an error. Can someone > why I got the >>> error and how I should modify my code? I am attaching my > simple code, error message, >>> and result. Thank you very much. > >>> > >>> Do you use the master branch from our Bitbucket repository? It contains > >>> the commit where Hong added the respective functionality: > >>> > https://bitbucket.org/petsc/petsc/commits/8949adfd119cb1d44c9fb5cf3dc01f4b6a02252d > > >> This is only for MPI. Hong, can you expose > MatTransposeMatMult_SeqAIJ_SeqDense? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From choi240 at purdue.edu Thu Sep 26 21:41:49 2013 From: choi240 at purdue.edu (Joon Hee Choi) Date: Thu, 26 Sep 2013 22:41:49 -0400 (EDT) Subject: [petsc-users] About MatTransposeMatMult In-Reply-To: Message-ID: <1061024055.149405.1380249709883.JavaMail.root@mailhub028.itcs.purdue.edu> Dear Hong, Thank you for your reply. I think it reduces the number of flops to use a seqaij matrix C because I will use very large sparse matrix for A and also multiply C and another large matrix. If so, then I might create B as a seqaij matrix. Thanks, Joon ----- Original Message ----- From: "Hong Zhang" To: "Joon Hee Choi" Cc: "Jed Brown" , "PETSc users list" Sent: Thursday, September 26, 2013 8:56:39 PM Subject: Re: [petsc-users] About MatTransposeMatMult Joon : Dear Jed, Hello. I modified my code using MatTranspose and MatMatMult, and it works well. BTW, the result of the multiplication of seqaij and seqdense was seqdense. I thought C should be seqaij when A is seqaij and B is seqdense in C=A*B. When B is dense, C=A*B is a dense matrix. Is the way to get seqaij just seqaij*seqaij? Why do you want store a dense matrix in sparse format? Hong ----- Original Message ----- From: "Joon Hee Choi" < choi240 at purdue.edu > To: "Jed Brown" < jedbrown at mcs.anl.gov > Cc: petsc-users at mcs.anl.gov Sent: Thursday, September 26, 2013 2:45:38 PM Subject: Re: [petsc-users] About MatTransposeMatMult > Thank you. I think MatMatMult runs well for multiplication between seqaij and seqdense. > Joon ----- Original Message ----- From: "Jed Brown" < jedbrown at mcs.anl.gov > To: "Karl Rupp" < rupp at mcs.anl.gov >, "Joon Hee Choi" < choi240 at purdue.edu >, petsc-users at mcs.anl.gov Cc: "Hong Zhang" < hzhang at mcs.anl.gov > Sent: Thursday, September 26, 2013 11:01:42 AM Subject: Re: [petsc-users] About MatTransposeMatMult >> Karl Rupp < rupp at mcs.anl.gov > writes: >>> Hi, >>> >>> I am trying to calculate M = X^T * B. X is a seqaij and B is a seqdense matrix. >>> >>> This is not yet allowed in the latest 3.4. release. >> You can explicitly transpose X: >> MatTranspose(X,MAT_INITIAL_MATRIX,&Xt); >> and then use MatMatMult. >>> However, I used MatTransposeMatMult() but I got an error. Can someone why I got the >>> error and how I should modify my code? I am attaching my simple code, error message, >>> and result. Thank you very much. >>> >>> Do you use the master branch from our Bitbucket repository? It contains >>> the commit where Hong added the respective functionality: >>> https://bitbucket.org/petsc/petsc/commits/8949adfd119cb1d44c9fb5cf3dc01f4b6a02252d >> This is only for MPI. Hong, can you expose MatTransposeMatMult_SeqAIJ_SeqDense? From jedbrown at mcs.anl.gov Fri Sep 27 03:00:10 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 27 Sep 2013 09:00:10 +0100 Subject: [petsc-users] About MatTransposeMatMult In-Reply-To: <1061024055.149405.1380249709883.JavaMail.root@mailhub028.itcs.purdue.edu> References: <1061024055.149405.1380249709883.JavaMail.root@mailhub028.itcs.purdue.edu> Message-ID: <87siwqk839.fsf@mcs.anl.gov> Joon Hee Choi writes: > Dear Hong, > > Thank you for your reply. I think it reduces the number of flops to > use a seqaij matrix C because I will use very large sparse matrix for > A and also multiply C and another large matrix. Joon, if A is sparse with at least one nonzero per row, and B is a dense matrix, then C = A*B is not sparse in the sense that literally every entry is nonzero. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From christophe.ortiz at ciemat.es Fri Sep 27 05:40:41 2013 From: christophe.ortiz at ciemat.es (Christophe Ortiz) Date: Fri, 27 Sep 2013 12:40:41 +0200 Subject: [petsc-users] Back to struct in Fortran to represent field with dof > 1 In-Reply-To: <3524B3B9-C5FE-46A7-8556-F36DEEC287BD@mcs.anl.gov> References: <3524B3B9-C5FE-46A7-8556-F36DEEC287BD@mcs.anl.gov> Message-ID: Hi Barry, I made some tests with your code that allows to use struct in Fortran90. It compiles and does not complain with the struct with 2 components but...it exhibits some strange behaviour. Here are the behaviours I observed. To make it simple, I used n=10. - After using call VecSet(x,one,ierr), I checked the values of each component of the vector using call VecGetArrayMyStruct(x,xarray,ierr) do i = 0 , n write(*,*) i , xarray(i)%a, xarray(i)%b end do call VecRestoreArrayMyStruct(x,xarray,ierr) I checked the values from i= 0 to n since I was not sure what was the convention (0- or 1-based). I obtained: 0 0.000000000000000E+000 1.00000000000000 1 1.00000000000000 1.00000000000000 2 1.00000000000000 1.00000000000000 3 1.00000000000000 1.00000000000000 4 1.00000000000000 1.00000000000000 5 1.00000000000000 1.00000000000000 6 1.00000000000000 1.00000000000000 7 1.00000000000000 1.00000000000000 8 1.00000000000000 1.00000000000000 9 1.00000000000000 1.00000000000000 10 1.00000000000000 1.996650376645798E-314 As you can see, for component a, there is a 0 for i=0, then the values are correct. For component b, correct values go from i=1 to 9.There is garbage in second component for i=10 . I checked that if you define a third component c, its values go from i=-1 to n-2. It's like if values of second component started at the end of first component, and same thing for third component. Then, instead of using VecSet(), I tried to assign values directly to each component of the vector with call VecGetArrayMyStruct(x,xarray,ierr) do i = 0 , n-1 xarray(i)%a = 2.0 xarray(i)%b = 3.0 end do call VecRestoreArrayMyStruct(x,xarray,ierr) Here I checked that indices actually start at 0 and not at 1. With a loop from i=1 to n I got a Memory corruption message. and I checked that the values are correctly assigned with call VecGetArrayMyStruct(x,xarray,ierr) do i = 0 , n-1 write(*,*) i, xarray(i)%a, xarray(i)%b end do call VecRestoreArrayMyStruct(x,xarray,ierr) Then, I obtained: 0 2.00000000000000 2.00000000000000 1 2.00000000000000 2.00000000000000 2 2.00000000000000 2.00000000000000 3 2.00000000000000 2.00000000000000 4 2.00000000000000 2.00000000000000 5 2.00000000000000 2.00000000000000 6 2.00000000000000 2.00000000000000 7 2.00000000000000 2.00000000000000 8 2.00000000000000 2.00000000000000 9 2.00000000000000 3.00000000000000 Here the problem seems more severe. Values are not correctly assigned to the second component. There should be 3.0 in the second column. It seems values of the first component are copied to the second one. Only for i=10 the value of xarray(i)%b is correct (3.0). Any idea where it could come from ? I guess it needs some fixes here and there but I think your idea is good and that it could work. Christophe On Thu, Sep 26, 2013 at 5:45 PM, Barry Smith wrote: > > I just put it with the Fortran source code and compile it with the rest > of the application code; here is the makefile I used > > > > > On Sep 26, 2013, at 10:14 AM, Christophe Ortiz > wrote: > > > By the way, what should I do with the small .c code ? Where should I put > it ? > > > > Christophe > > > > > > > > On Thu, Sep 26, 2013 at 4:09 PM, Barry Smith wrote: > > > > Christophe, > > > > Despite my LinkIn Endorsement for expertise in Fortran :-) I cannot > pretend to be an expert in FortranXX but I have cooked up an example > demonstrating accessing the Vec entries as if they are in an array of > derived types. I've attached the example code; there needs to be a small C > stub that defines the functions for your specific derived type name. > > Note that it will only work I think if your N is a compile time constant. > > > > > > > > > > It worked with > > ~/Downloads$ gfortran --version > > GNU Fortran (GCC) 4.8.1 20130404 (prerelease) > > > > > > I do not understand exactly why it works since it uses > F90Array1dCreate(fa,PETSC_SCALAR,1,len,ptr PETSC_F90_2PTR_PARAM(ptrd)); > which has a single PETSC_SCALAR as a building block but ? I hope it works > for you. If it doesn't, let us know the compiler you are using and we may > be able to get it working for that compiler. > > > > Barry > > > > > > > > > > On Sep 26, 2013, at 4:41 AM, Christophe Ortiz < > christophe.ortiz at ciemat.es> wrote: > > > > > Hi all, > > > > > > Me again ! > > > > > > I have read in previous posts that it is hard in Fortran to declare > something similar to a typedef struct in C to manage a multicomponent > problem. > > > > > > Is it still the case ? Has the problem been solved ? > > > > > > I am asking because my plan is to implement a multicomponent problem > (in 1D), with many components that will be organized in arrays of two > dimensions. In C I could define > > > > > > typedef struct{ > > > PetscScalar U; > > > PetscScalar V; > > > PetscScalar A[N][N]; > > > } Field; > > > > > > and then I could calculate the residual function with > > > > > > F[i].U = ... > > > F[i].V = ... > > > F[i].A[k][n] = ... > > > > > > which is quite convenient. > > > > > > If in Fortran it is not possible to use a struct as in C, I am afraid > I'll have to deal with > > > > > > F(jdof,i) = ... > > > > > > where I will have only jdof to address U, V and A[ ][ ], which can be > difficult and not very convenient I guess. Before I move all my code to C, > does anyone have an alternative idea to manage this multi(many)component > problem in Fortran ? > > > > > > Many thanks in advance for your help and suggestion ! > > > > > > Christophe > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From choi240 at purdue.edu Fri Sep 27 07:21:29 2013 From: choi240 at purdue.edu (Joon Hee Choi) Date: Fri, 27 Sep 2013 08:21:29 -0400 (EDT) Subject: [petsc-users] The inverse of seqaij matrix In-Reply-To: <793563206.149895.1380283760525.JavaMail.root@mailhub028.itcs.purdue.edu> Message-ID: <137548802.149917.1380284489278.JavaMail.root@mailhub028.itcs.purdue.edu> Hi. I am trying to get the inverse matrix of seqaij. I followed FAQ of petsc manual, but I got error. Could someone tell me what I am missing? I am attaching my code, results, and error message. Thank you. Joon PetscPrintf(PETSC_COMM_WORLD, "I \n"); MatView(I, PETSC_VIEWER_STDOUT_WORLD); PetscPrintf(PETSC_COMM_WORLD, "invM2T \n"); MatView(invM2T, PETSC_VIEWER_STDOUT_WORLD); ierr = MatCreateSeqDense(PETSC_COMM_SELF, R, R, NULL, &seqM2T); CHKERRQ(ierr); ierr = MatGetOrdering(invM2T, MATORDERINGNATURAL, &isr, &isc); CHKERRQ(ierr); ierr = MatFactorInfoInitialize(&info); CHKERRQ(ierr); ierr = MatLUFactor(invM2T, isr, isc, &info); CHKERRQ(ierr); ierr = MatMatSolve(invM2T, I, seqM2T); CHKERRQ(ierr); PetscPrintf(PETSC_COMM_WORLD, "invM2T \n"); MatView(invM2T, PETSC_VIEWER_STDOUT_WORLD); Matrix Object: 1 MPI processes type: seqdense 1.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 1.0000000000000000e+00 invM2T Matrix Object: 1 MPI processes type: seqaij row 0: (0, 3.22689) (1, 2.47603) row 1: (0, 2.47603) (1, 2.25324) invM2T Matrix Object: 1 MPI processes type: seqaij row 0: (0, 3.22689) (1, 2.47603) row 1: (0, 0.76731) (1, 0.353361) [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Detected zero pivot in LU factorization: see http://www.mcs.anl.gov/petsc/documentation/faq.html#ZeroPivot! [1]PETSC ERROR: Zero pivot row 0 value 0 tolerance 0! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Release Version 3.4.1, Jun, 10, 2013 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: ./tenfac on a linux named rossmann-b000.rcac.purdue.edu by choi240 Fri Sep 27 08:01:31 2013 [1]PETSC ERROR: Libraries linked from /apps/rhel5/petsc-3.4.1/64/impi-4.1.0.030_intel-13.0.1.117_ind64/linux/lib [1]PETSC ERROR: Configure run at Sun Jun 23 10:20:51 2013 [1]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --download-sowing --with-scalar-type=real --with-shared-libraries=1 --with-pic=1 --with-clanguage=C++ --with-fortran --with-fortran-kernels=1 --with-64-bit-indices=1 --with-debugging=0 --with-blas-lapack-dir=/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64 --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --download-hdf5=no --download-metis=no --download-parmetis=no --download-superlu_dist=no --download-mumps=no --download-scalapack=yes --download-blacs=no --download-hypre=no --download-spooles=no [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: MatPivotCheck_none() line 589 in /apps/rhel5/petsc-3.4.1/64/impi-4.1.0.030_intel-13.0.1.117_ind64/include/petsc-private/matimpl.h [1]PETSC ERROR: MatPivotCheck() line 608 in /apps/rhel5/petsc-3.4.1/64/impi-4.1.0.030_intel-13.0.1.117_ind64/include/petsc-private/matimpl.h [1]PETSC ERROR: MatLUFactorNumeric_SeqAIJ_Inode() line 1422 in /apps/rhel5/petsc-3.4.1/64/impi-4.1.0.030_intel-13.0.1.117_ind64/src/mat/impls/aij/seq/inode.c [1]PETSC ERROR: MatLUFactorNumeric() line 2889 in /apps/rhel5/petsc-3.4.1/64/impi-4.1.0.030_intel-13.0.1.117_ind64/src/mat/interface/matrix.c [1]PETSC ERROR: MatLUFactor_SeqAIJ() line 971 in /apps/rhel5/petsc-3.4.1/64/impi-4.1.0.030_intel-13.0.1.117_ind64/src/mat/impls/aij/seq/aijfact.c [1]PETSC ERROR: MatLUFactor() line 2716 in /apps/rhel5/petsc-3.4.1/64/impi-4.1.0.030_intel-13.0.1.117_ind64/src/mat/interface/matrix.c [1]PETSC ERROR: cal_M2() line 83 in tenfac.cpp [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [1]PETSC ERROR: to get more information on the crash. [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Signal received! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Release Version 3.4.1, Jun, 10, 2013 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: ./tenfac on a linux named rossmann-b000.rcac.purdue.edu by choi240 Fri Sep 27 08:01:31 2013 [1]PETSC ERROR: Libraries linked from /apps/rhel5/petsc-3.4.1/64/impi-4.1.0.030_intel-13.0.1.117_ind64/linux/lib [1]PETSC ERROR: Configure run at Sun Jun 23 10:20:51 2013 [1]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --download-sowing --with-scalar-type=real --with-shared-libraries=1 --with-pic=1 --with-clanguage=C++ --with-fortran --with-fortran-kernels=1 --with-64-bit-indices=1 --with-debugging=0 --with-blas-lapack-dir=/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64 --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --download-hdf5=no --download-metis=no --download-parmetis=no --download-superlu_dist=no --download-mumps=no --download-scalapack=yes --download-blacs=no --download-hypre=no --download-spooles=no [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1 From bsmith at mcs.anl.gov Fri Sep 27 07:47:33 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 27 Sep 2013 07:47:33 -0500 Subject: [petsc-users] Back to struct in Fortran to represent field with dof > 1 In-Reply-To: References: <3524B3B9-C5FE-46A7-8556-F36DEEC287BD@mcs.anl.gov> Message-ID: Indices start in fortran arrays from 1 by default so don't even try zero I've attached a new version that has three components, -------------- next part -------------- A non-text attachment was scrubbed... Name: ex1f90.F Type: application/octet-stream Size: 7281 bytes Desc: not available URL: -------------- next part -------------- 30 TOTAL vector entries and 10 for each component. Please run it and send the results back. Barry On Sep 27, 2013, at 5:40 AM, Christophe Ortiz wrote: > Hi Barry, > > I made some tests with your code that allows to use struct in Fortran90. It compiles and does not complain with the struct with 2 components but...it exhibits some strange behaviour. Here are the behaviours I observed. To make it simple, I used n=10. > > - After using call VecSet(x,one,ierr), I checked the values of each component of the vector using > > call VecGetArrayMyStruct(x,xarray,ierr) > do i = 0 , n > write(*,*) i , xarray(i)%a, xarray(i)%b > end do > call VecRestoreArrayMyStruct(x,xarray,ierr) > > I checked the values from i= 0 to n since I was not sure what was the convention (0- or 1-based). I obtained: > > 0 0.000000000000000E+000 1.00000000000000 > 1 1.00000000000000 1.00000000000000 > 2 1.00000000000000 1.00000000000000 > 3 1.00000000000000 1.00000000000000 > 4 1.00000000000000 1.00000000000000 > 5 1.00000000000000 1.00000000000000 > 6 1.00000000000000 1.00000000000000 > 7 1.00000000000000 1.00000000000000 > 8 1.00000000000000 1.00000000000000 > 9 1.00000000000000 1.00000000000000 > 10 1.00000000000000 1.996650376645798E-314 > > > As you can see, for component a, there is a 0 for i=0, then the values are correct. For component b, correct values go from i=1 to 9.There is garbage in second component for i=10 . I checked that if you define a third component c, its values go from i=-1 to n-2. It's like if values of second component started at the end of first component, and same thing for third component. > > Then, instead of using VecSet(), I tried to assign values directly to each component of the vector with > > call VecGetArrayMyStruct(x,xarray,ierr) > do i = 0 , n-1 > xarray(i)%a = 2.0 > xarray(i)%b = 3.0 > end do > call VecRestoreArrayMyStruct(x,xarray,ierr) > > Here I checked that indices actually start at 0 and not at 1. With a loop from i=1 to n I got a Memory corruption message. > > and I checked that the values are correctly assigned with > > call VecGetArrayMyStruct(x,xarray,ierr) > do i = 0 , n-1 > write(*,*) i, xarray(i)%a, xarray(i)%b > end do > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > Then, I obtained: > > 0 2.00000000000000 2.00000000000000 > 1 2.00000000000000 2.00000000000000 > 2 2.00000000000000 2.00000000000000 > 3 2.00000000000000 2.00000000000000 > 4 2.00000000000000 2.00000000000000 > 5 2.00000000000000 2.00000000000000 > 6 2.00000000000000 2.00000000000000 > 7 2.00000000000000 2.00000000000000 > 8 2.00000000000000 2.00000000000000 > 9 2.00000000000000 3.00000000000000 > > > Here the problem seems more severe. Values are not correctly assigned to the second component. There should be 3.0 in the second column. It seems values of the first component are copied to the second one. Only for i=10 the value of xarray(i)%b is correct (3.0). > > Any idea where it could come from ? > I guess it needs some fixes here and there but I think your idea is good and that it could work. > > Christophe > > > > On Thu, Sep 26, 2013 at 5:45 PM, Barry Smith wrote: > > I just put it with the Fortran source code and compile it with the rest of the application code; here is the makefile I used > > > > > On Sep 26, 2013, at 10:14 AM, Christophe Ortiz wrote: > > > By the way, what should I do with the small .c code ? Where should I put it ? > > > > Christophe > > > > > > > > On Thu, Sep 26, 2013 at 4:09 PM, Barry Smith wrote: > > > > Christophe, > > > > Despite my LinkIn Endorsement for expertise in Fortran :-) I cannot pretend to be an expert in FortranXX but I have cooked up an example demonstrating accessing the Vec entries as if they are in an array of derived types. I've attached the example code; there needs to be a small C stub that defines the functions for your specific derived type name. > > Note that it will only work I think if your N is a compile time constant. > > > > > > > > > > It worked with > > ~/Downloads$ gfortran --version > > GNU Fortran (GCC) 4.8.1 20130404 (prerelease) > > > > > > I do not understand exactly why it works since it uses F90Array1dCreate(fa,PETSC_SCALAR,1,len,ptr PETSC_F90_2PTR_PARAM(ptrd)); which has a single PETSC_SCALAR as a building block but ? I hope it works for you. If it doesn't, let us know the compiler you are using and we may be able to get it working for that compiler. > > > > Barry > > > > > > > > > > On Sep 26, 2013, at 4:41 AM, Christophe Ortiz wrote: > > > > > Hi all, > > > > > > Me again ! > > > > > > I have read in previous posts that it is hard in Fortran to declare something similar to a typedef struct in C to manage a multicomponent problem. > > > > > > Is it still the case ? Has the problem been solved ? > > > > > > I am asking because my plan is to implement a multicomponent problem (in 1D), with many components that will be organized in arrays of two dimensions. In C I could define > > > > > > typedef struct{ > > > PetscScalar U; > > > PetscScalar V; > > > PetscScalar A[N][N]; > > > } Field; > > > > > > and then I could calculate the residual function with > > > > > > F[i].U = ... > > > F[i].V = ... > > > F[i].A[k][n] = ... > > > > > > which is quite convenient. > > > > > > If in Fortran it is not possible to use a struct as in C, I am afraid I'll have to deal with > > > > > > F(jdof,i) = ... > > > > > > where I will have only jdof to address U, V and A[ ][ ], which can be difficult and not very convenient I guess. Before I move all my code to C, does anyone have an alternative idea to manage this multi(many)component problem in Fortran ? > > > > > > Many thanks in advance for your help and suggestion ! > > > > > > Christophe > > > > > > > > > > > From bsmith at mcs.anl.gov Fri Sep 27 08:05:58 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 27 Sep 2013 08:05:58 -0500 Subject: [petsc-users] The inverse of seqaij matrix In-Reply-To: <137548802.149917.1380284489278.JavaMail.root@mailhub028.itcs.purdue.edu> References: <137548802.149917.1380284489278.JavaMail.root@mailhub028.itcs.purdue.edu> Message-ID: Based on the output below it appears that the code block you sent was completely successfully since it printed all three matrices you tried to proint. So it is something else in your code that is generating the zero pivot. BTW: it should not be MatView(invM2T, PETSC_VIEWER_STDOUT_WORLD); it should be MatView(seqM2T, PETSC_VIEWER_STDOUT_WORLD); to display the inverse And how is it possible that the error occurred on MPI rank 1 but the matrices are sequential matrices viewed on PETSC_VIEWER_STDOUT_WORLD? A zero privet means that either the matrix is singular or the ordering used to factor the matrix is not a good ordering. Perhaps the matrix was generated incorrectly. The code crashed because likely your code returned an error code returned from PETSc and tried to keep running. For example if the factorization generated an error code but you called the MatSolveSolve on the result anyway, you cannot do that. On Sep 27, 2013, at 7:21 AM, Joon Hee Choi wrote: > Hi. > I am trying to get the inverse matrix of seqaij. I followed FAQ of petsc manual, but I got error. > Could someone tell me what I am missing? I am attaching my code, results, and error message. > Thank you. > > Joon > > > > PetscPrintf(PETSC_COMM_WORLD, "I \n"); > MatView(I, PETSC_VIEWER_STDOUT_WORLD); > PetscPrintf(PETSC_COMM_WORLD, "invM2T \n"); > MatView(invM2T, PETSC_VIEWER_STDOUT_WORLD); > > ierr = MatCreateSeqDense(PETSC_COMM_SELF, R, R, NULL, &seqM2T); CHKERRQ(ierr); > ierr = MatGetOrdering(invM2T, MATORDERINGNATURAL, &isr, &isc); CHKERRQ(ierr); > ierr = MatFactorInfoInitialize(&info); CHKERRQ(ierr); > ierr = MatLUFactor(invM2T, isr, isc, &info); CHKERRQ(ierr); > > ierr = MatMatSolve(invM2T, I, seqM2T); CHKERRQ(ierr); > PetscPrintf(PETSC_COMM_WORLD, "invM2T \n"); > MatView(invM2T, PETSC_VIEWER_STDOUT_WORLD); > > > Matrix Object: 1 MPI processes > type: seqdense > 1.0000000000000000e+00 0.0000000000000000e+00 > 0.0000000000000000e+00 1.0000000000000000e+00 > invM2T > Matrix Object: 1 MPI processes > type: seqaij > row 0: (0, 3.22689) (1, 2.47603) > row 1: (0, 2.47603) (1, 2.25324) > invM2T > Matrix Object: 1 MPI processes > type: seqaij > row 0: (0, 3.22689) (1, 2.47603) > row 1: (0, 0.76731) (1, 0.353361) > > > [1]PETSC ERROR: --------------------- Error Message ------------------------------------ > [1]PETSC ERROR: Detected zero pivot in LU factorization: > see http://www.mcs.anl.gov/petsc/documentation/faq.html#ZeroPivot! > [1]PETSC ERROR: Zero pivot row 0 value 0 tolerance 0! > [1]PETSC ERROR: ------------------------------------------------------------------------ > [1]PETSC ERROR: Petsc Release Version 3.4.1, Jun, 10, 2013 > [1]PETSC ERROR: See docs/changes/index.html for recent updates. > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [1]PETSC ERROR: See docs/index.html for manual pages. > [1]PETSC ERROR: ------------------------------------------------------------------------ > [1]PETSC ERROR: ./tenfac on a linux named rossmann-b000.rcac.purdue.edu by choi240 Fri Sep 27 08:01:31 2013 > [1]PETSC ERROR: Libraries linked from /apps/rhel5/petsc-3.4.1/64/impi-4.1.0.030_intel-13.0.1.117_ind64/linux/lib > [1]PETSC ERROR: Configure run at Sun Jun 23 10:20:51 2013 > [1]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --download-sowing --with-scalar-type=real --with-shared-libraries=1 --with-pic=1 --with-clanguage=C++ --with-fortran --with-fortran-kernels=1 --with-64-bit-indices=1 --with-debugging=0 --with-blas-lapack-dir=/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64 --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --download-hdf5=no --download-metis=no --download-parmetis=no --download-superlu_dist=no --download-mumps=no --download-scalapack=yes --download-blacs=no --download-hypre=no --download-spooles=no > [1]PETSC ERROR: ------------------------------------------------------------------------ > [1]PETSC ERROR: MatPivotCheck_none() line 589 in /apps/rhel5/petsc-3.4.1/64/impi-4.1.0.030_intel-13.0.1.117_ind64/include/petsc-private/matimpl.h > [1]PETSC ERROR: MatPivotCheck() line 608 in /apps/rhel5/petsc-3.4.1/64/impi-4.1.0.030_intel-13.0.1.117_ind64/include/petsc-private/matimpl.h > [1]PETSC ERROR: MatLUFactorNumeric_SeqAIJ_Inode() line 1422 in /apps/rhel5/petsc-3.4.1/64/impi-4.1.0.030_intel-13.0.1.117_ind64/src/mat/impls/aij/seq/inode.c > [1]PETSC ERROR: MatLUFactorNumeric() line 2889 in /apps/rhel5/petsc-3.4.1/64/impi-4.1.0.030_intel-13.0.1.117_ind64/src/mat/interface/matrix.c > [1]PETSC ERROR: MatLUFactor_SeqAIJ() line 971 in /apps/rhel5/petsc-3.4.1/64/impi-4.1.0.030_intel-13.0.1.117_ind64/src/mat/impls/aij/seq/aijfact.c > [1]PETSC ERROR: MatLUFactor() line 2716 in /apps/rhel5/petsc-3.4.1/64/impi-4.1.0.030_intel-13.0.1.117_ind64/src/mat/interface/matrix.c > [1]PETSC ERROR: cal_M2() line 83 in tenfac.cpp > [1]PETSC ERROR: ------------------------------------------------------------------------ > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [1]PETSC ERROR: to get more information on the crash. > [1]PETSC ERROR: --------------------- Error Message ------------------------------------ > [1]PETSC ERROR: Signal received! > [1]PETSC ERROR: ------------------------------------------------------------------------ > [1]PETSC ERROR: Petsc Release Version 3.4.1, Jun, 10, 2013 > [1]PETSC ERROR: See docs/changes/index.html for recent updates. > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [1]PETSC ERROR: See docs/index.html for manual pages. > [1]PETSC ERROR: ------------------------------------------------------------------------ > [1]PETSC ERROR: ./tenfac on a linux named rossmann-b000.rcac.purdue.edu by choi240 Fri Sep 27 08:01:31 2013 > [1]PETSC ERROR: Libraries linked from /apps/rhel5/petsc-3.4.1/64/impi-4.1.0.030_intel-13.0.1.117_ind64/linux/lib > [1]PETSC ERROR: Configure run at Sun Jun 23 10:20:51 2013 > [1]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --download-sowing --with-scalar-type=real --with-shared-libraries=1 --with-pic=1 --with-clanguage=C++ --with-fortran --with-fortran-kernels=1 --with-64-bit-indices=1 --with-debugging=0 --with-blas-lapack-dir=/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64 --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --download-hdf5=no --download-metis=no --download-parmetis=no --download-superlu_dist=no --download-mumps=no --download-scalapack=yes --download-blacs=no --download-hypre=no --download-spooles=no > [1]PETSC ERROR: ------------------------------------------------------------------------ > [1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1 From christophe.ortiz at ciemat.es Fri Sep 27 08:22:29 2013 From: christophe.ortiz at ciemat.es (Christophe Ortiz) Date: Fri, 27 Sep 2013 15:22:29 +0200 Subject: [petsc-users] Back to struct in Fortran to represent field with dof > 1 In-Reply-To: References: <3524B3B9-C5FE-46A7-8556-F36DEEC287BD@mcs.anl.gov> Message-ID: There was a misunderstanding. There are n entries in total in the vector. I thought it was a vector of n points, with 3 components associated to each point, ie 3*n entries. This is clear now. I've tested your example and I checked the values assigned to the components with your piece of code: call VecGetArrayMyStruct(x,xarray,ierr) do i=1,10 xarray(i)%a = i xarray(i)%b = 100*i xarray(i)%c = 10000*i enddo call VecRestoreArrayMyStruct(x,xarray,ierr) I did it with: call VecGetArrayMyStruct(x,xarray,ierr) do i = 1 , 10 write(*,*) xarray(i)%a,xarray(i)%b,xarray(i)%c end do call VecRestoreArrayMyStruct(x,xarray,ierr) I obtained the following: 1.00000000000000 2.00000000000000 3.00000000000000 2.00000000000000 3.00000000000000 4.00000000000000 3.00000000000000 4.00000000000000 5.00000000000000 4.00000000000000 5.00000000000000 6.00000000000000 5.00000000000000 6.00000000000000 7.00000000000000 6.00000000000000 7.00000000000000 8.00000000000000 7.00000000000000 8.00000000000000 9.00000000000000 8.00000000000000 9.00000000000000 10.0000000000000 9.00000000000000 10.0000000000000 1000.00000000000 10.0000000000000 1000.00000000000 100000.000000000 First column if ok (1st component), but the other two columns show wrong numbers. For component 2 and 3, only the very last value is ok. It seems values from component b are copied from a, and those from component c from b, with an offset. What do you think ? Christophe On Fri, Sep 27, 2013 at 2:47 PM, Barry Smith wrote: > > Indices start in fortran arrays from 1 by default so don't even try zero > > I've attached a new version that has three components, > 30 TOTAL vector entries and 10 for each component. Please run it and send > the results back. > > Barry > > On Sep 27, 2013, at 5:40 AM, Christophe Ortiz > wrote: > > > Hi Barry, > > > > I made some tests with your code that allows to use struct in Fortran90. > It compiles and does not complain with the struct with 2 components > but...it exhibits some strange behaviour. Here are the behaviours I > observed. To make it simple, I used n=10. > > > > - After using call VecSet(x,one,ierr), I checked the values of each > component of the vector using > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > do i = 0 , n > > write(*,*) i , xarray(i)%a, xarray(i)%b > > end do > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > I checked the values from i= 0 to n since I was not sure what was the > convention (0- or 1-based). I obtained: > > > > 0 0.000000000000000E+000 1.00000000000000 > > 1 1.00000000000000 1.00000000000000 > > 2 1.00000000000000 1.00000000000000 > > 3 1.00000000000000 1.00000000000000 > > 4 1.00000000000000 1.00000000000000 > > 5 1.00000000000000 1.00000000000000 > > 6 1.00000000000000 1.00000000000000 > > 7 1.00000000000000 1.00000000000000 > > 8 1.00000000000000 1.00000000000000 > > 9 1.00000000000000 1.00000000000000 > > 10 1.00000000000000 1.996650376645798E-314 > > > > > > As you can see, for component a, there is a 0 for i=0, then the values > are correct. For component b, correct values go from i=1 to 9.There is > garbage in second component for i=10 . I checked that if you define a third > component c, its values go from i=-1 to n-2. It's like if values of second > component started at the end of first component, and same thing for third > component. > > > > Then, instead of using VecSet(), I tried to assign values directly to > each component of the vector with > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > do i = 0 , n-1 > > xarray(i)%a = 2.0 > > xarray(i)%b = 3.0 > > end do > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > Here I checked that indices actually start at 0 and not at 1. With a > loop from i=1 to n I got a Memory corruption message. > > > > and I checked that the values are correctly assigned with > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > do i = 0 , n-1 > > write(*,*) i, xarray(i)%a, xarray(i)%b > > end do > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > > > > > Then, I obtained: > > > > 0 2.00000000000000 2.00000000000000 > > 1 2.00000000000000 2.00000000000000 > > 2 2.00000000000000 2.00000000000000 > > 3 2.00000000000000 2.00000000000000 > > 4 2.00000000000000 2.00000000000000 > > 5 2.00000000000000 2.00000000000000 > > 6 2.00000000000000 2.00000000000000 > > 7 2.00000000000000 2.00000000000000 > > 8 2.00000000000000 2.00000000000000 > > 9 2.00000000000000 3.00000000000000 > > > > > > Here the problem seems more severe. Values are not correctly assigned to > the second component. There should be 3.0 in the second column. It seems > values of the first component are copied to the second one. Only for i=10 > the value of xarray(i)%b is correct (3.0). > > > > Any idea where it could come from ? > > I guess it needs some fixes here and there but I think your idea is good > and that it could work. > > > > Christophe > > > > > > > > On Thu, Sep 26, 2013 at 5:45 PM, Barry Smith wrote: > > > > I just put it with the Fortran source code and compile it with the > rest of the application code; here is the makefile I used > > > > > > > > > > On Sep 26, 2013, at 10:14 AM, Christophe Ortiz < > christophe.ortiz at ciemat.es> wrote: > > > > > By the way, what should I do with the small .c code ? Where should I > put it ? > > > > > > Christophe > > > > > > > > > > > > On Thu, Sep 26, 2013 at 4:09 PM, Barry Smith > wrote: > > > > > > Christophe, > > > > > > Despite my LinkIn Endorsement for expertise in Fortran :-) I > cannot pretend to be an expert in FortranXX but I have cooked up an example > demonstrating accessing the Vec entries as if they are in an array of > derived types. I've attached the example code; there needs to be a small C > stub that defines the functions for your specific derived type name. > > > Note that it will only work I think if your N is a compile time > constant. > > > > > > > > > > > > > > > It worked with > > > ~/Downloads$ gfortran --version > > > GNU Fortran (GCC) 4.8.1 20130404 (prerelease) > > > > > > > > > I do not understand exactly why it works since it uses > F90Array1dCreate(fa,PETSC_SCALAR,1,len,ptr PETSC_F90_2PTR_PARAM(ptrd)); > which has a single PETSC_SCALAR as a building block but ? I hope it works > for you. If it doesn't, let us know the compiler you are using and we may > be able to get it working for that compiler. > > > > > > Barry > > > > > > > > > > > > > > > On Sep 26, 2013, at 4:41 AM, Christophe Ortiz < > christophe.ortiz at ciemat.es> wrote: > > > > > > > Hi all, > > > > > > > > Me again ! > > > > > > > > I have read in previous posts that it is hard in Fortran to declare > something similar to a typedef struct in C to manage a multicomponent > problem. > > > > > > > > Is it still the case ? Has the problem been solved ? > > > > > > > > I am asking because my plan is to implement a multicomponent problem > (in 1D), with many components that will be organized in arrays of two > dimensions. In C I could define > > > > > > > > typedef struct{ > > > > PetscScalar U; > > > > PetscScalar V; > > > > PetscScalar A[N][N]; > > > > } Field; > > > > > > > > and then I could calculate the residual function with > > > > > > > > F[i].U = ... > > > > F[i].V = ... > > > > F[i].A[k][n] = ... > > > > > > > > which is quite convenient. > > > > > > > > If in Fortran it is not possible to use a struct as in C, I am > afraid I'll have to deal with > > > > > > > > F(jdof,i) = ... > > > > > > > > where I will have only jdof to address U, V and A[ ][ ], which can > be difficult and not very convenient I guess. Before I move all my code to > C, does anyone have an alternative idea to manage this multi(many)component > problem in Fortran ? > > > > > > > > Many thanks in advance for your help and suggestion ! > > > > > > > > Christophe > > > > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From choi240 at purdue.edu Fri Sep 27 08:50:52 2013 From: choi240 at purdue.edu (Joon Hee Choi) Date: Fri, 27 Sep 2013 09:50:52 -0400 (EDT) Subject: [petsc-users] =?utf-8?b?64u17J6lOiBSZTogIFRoZSBpbnZlcnNlIG9mIHNl?= =?utf-8?q?qaij_matrix?= In-Reply-To: Message-ID: <1492695855.150162.1380289852504.JavaMail.root@mailhub028.itcs.purdue.edu> Dear Brry, Thank you for your fast reply. I generated a SEQAIJ matrix in MPI rank 1 from a MPIAIJ matrix, because I needed the inverse matrix of the MPI matrix. Actually, the furmula is very simple, but my code is very complex. What I want to get is the inverse of (temp1 * temp2). temp1 and temp2 are mpiaij and * is hadamard product. Also, I want to get mpiaij matrix as the result. I think the ordering used to factor the matrix was not good because seqM2 is not singular. From my code, could you let me know why the ordering is wrong? I am sorry, but I am attaching my very long code. Thank you again. Joon MPI_Comm_size(PETSC_COMM_WORLD, &size); MPI_Comm_rank(MPI_COMM_SELF, &myrank); ierr = MatCreate(PETSC_COMM_WORLD, &invM2); CHKERRQ(ierr); ierr = MatSetType(invM2, MATDENSE); CHKERRQ(ierr); ierr = MatSetSizes(invM2, PETSC_DECIDE, PETSC_DECIDE, R, R); CHKERRQ(ierr); ierr = MatSeqDenseSetPreallocation(invM2, NULL); CHKERRQ(ierr); ierr = MatMPIDenseSetPreallocation(invM2, NULL); CHKERRQ(ierr); ierr = MatGetOwnershipRange(temp1, &rstart, &rend); CHKERRQ(ierr); ierr = PetscMalloc((rend-rstart)*R*sizeof(PetscScalar), &cell); CHKERRQ(ierr); for (i=rstart; i ?? ??: Joon Hee Choi ??: petsc-users at mcs.anl.gov ??: Fri, 27 Sep 2013 09:05:58 -0400 (EDT) ??: Re: [petsc-users] The inverse of seqaij matrix > Based on the output below it appears that the code block you sent was completely successfully since it printed all three matrices you tried to proint. So it is something else in your code that is generating the zero pivot. > BTW: it should not be >MatView(invM2T, PETSC_VIEWER_STDOUT_WORLD); >it should be >MatView(seqM2T, PETSC_VIEWER_STDOUT_WORLD); >to display the inverse >And how is it possible that the error occurred on MPI rank 1 but the matrices are sequential matrices viewed on PETSC_VIEWER_STDOUT_WORLD? >A zero privet means that either the matrix is singular or the ordering used to factor the matrix is not a good ordering. Perhaps the matrix was generated incorrectly. >The code crashed because likely your code returned an error code returned from PETSc and tried to keep running. For example if the factorization generated an error code but you called the MatSolveSolve on the result anyway, you cannot do that. On Sep 27, 2013, at 7:21 AM, Joon Hee Choi wrote: >> Hi. >> I am trying to get the inverse matrix of seqaij. I followed FAQ of petsc manual, but I got error. >> Could someone tell me what I am missing? I am attaching my code, results, and error message. >> Thank you. >> >> Joon >> >> >> >> PetscPrintf(PETSC_COMM_WORLD, "I \n"); >> MatView(I, PETSC_VIEWER_STDOUT_WORLD); >> PetscPrintf(PETSC_COMM_WORLD, "invM2T \n"); >> MatView(invM2T, PETSC_VIEWER_STDOUT_WORLD); >> >> ierr = MatCreateSeqDense(PETSC_COMM_SELF, R, R, NULL, &seqM2T); CHKERRQ(ierr); >> ierr = MatGetOrdering(invM2T, MATORDERINGNATURAL, &isr, &isc); CHKERRQ(ierr); >> ierr = MatFactorInfoInitialize(&info); CHKERRQ(ierr); >> ierr = MatLUFactor(invM2T, isr, isc, &info); CHKERRQ(ierr); >> >> ierr = MatMatSolve(invM2T, I, seqM2T); CHKERRQ(ierr); >> PetscPrintf(PETSC_COMM_WORLD, "invM2T \n"); >> MatView(invM2T, PETSC_VIEWER_STDOUT_WORLD); >> >> >> Matrix Object: 1 MPI processes >> type: seqdense >> 1.0000000000000000e+00 0.0000000000000000e+00 >> 0.0000000000000000e+00 1.0000000000000000e+00 >> invM2T >> Matrix Object: 1 MPI processes >> type: seqaij >> row 0: (0, 3.22689) (1, 2.47603) >> row 1: (0, 2.47603) (1, 2.25324) >> invM2T >> Matrix Object: 1 MPI processes >> type: seqaij >> row 0: (0, 3.22689) (1, 2.47603) >> row 1: (0, 0.76731) (1, 0.353361) >> >> >> [1]PETSC ERROR: --------------------- Error Message ------------------------------------ >> [1]PETSC ERROR: Detected zero pivot in LU factorization: >> see http://www.mcs.anl.gov/petsc/documentation/faq.html#ZeroPivot! >> [1]PETSC ERROR: Zero pivot row 0 value 0 tolerance 0! >> [1]PETSC ERROR: ------------------------------------------------------------------------ >> [1]PETSC ERROR: Petsc Release Version 3.4.1, Jun, 10, 2013 >> [1]PETSC ERROR: See docs/changes/index.html for recent updates. >> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [1]PETSC ERROR: See docs/index.html for manual pages. >> [1]PETSC ERROR: ------------------------------------------------------------------------ >> [1]PETSC ERROR: ./tenfac on a linux named rossmann-b000.rcac.purdue.edu by choi240 Fri Sep 27 08:01:31 2013 >> [1]PETSC ERROR: Libraries linked from /apps/rhel5/petsc-3.4.1/64/impi-4.1.0.030_intel-13.0.1.117_ind64/linux/lib >> [1]PETSC ERROR: Configure run at Sun Jun 23 10:20:51 2013 >> [1]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --download-sowing --with-scalar-type=real --with-shared-libraries=1 --with-pic=1 --with-clanguage=C++ --with-fortran --with-fortran-kernels=1 --with-64-bit-indices=1 --with-debugging=0 --with-blas-lapack-dir=/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64 --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --download-hdf5=no --download-metis=no --download-parmetis=no --download-superlu_dist=no --download-mumps=no --download-scalapack=yes --download-blacs=no --download-hypre=no --download-spooles=no >> [1]PETSC ERROR: ------------------------------------------------------------------------ >> [1]PETSC ERROR: MatPivotCheck_none() line 589 in /apps/rhel5/petsc-3.4.1/64/impi-4.1.0.030_intel-13.0.1.117_ind64/include/petsc-private/matimpl.h >> [1]PETSC ERROR: MatPivotCheck() line 608 in /apps/rhel5/petsc-3.4.1/64/impi-4.1.0.030_intel-13.0.1.117_ind64/include/petsc-private/matimpl.h >> [1]PETSC ERROR: MatLUFactorNumeric_SeqAIJ_Inode() line 1422 in /apps/rhel5/petsc-3.4.1/64/impi-4.1.0.030_intel-13.0.1.117_ind64/src/mat/impls/aij/seq/inode.c >> [1]PETSC ERROR: MatLUFactorNumeric() line 2889 in /apps/rhel5/petsc-3.4.1/64/impi-4.1.0.030_intel-13.0.1.117_ind64/src/mat/interface/matrix.c >> [1]PETSC ERROR: MatLUFactor_SeqAIJ() line 971 in /apps/rhel5/petsc-3.4.1/64/impi-4.1.0.030_intel-13.0.1.117_ind64/src/mat/impls/aij/seq/aijfact.c >> [1]PETSC ERROR: MatLUFactor() line 2716 in /apps/rhel5/petsc-3.4.1/64/impi-4.1.0.030_intel-13.0.1.117_ind64/src/mat/interface/matrix.c >> [1]PETSC ERROR: cal_M2() line 83 in tenfac.cpp >> [1]PETSC ERROR: ------------------------------------------------------------------------ >> [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >> [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >> [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run >> [1]PETSC ERROR: to get more information on the crash. >> [1]PETSC ERROR: --------------------- Error Message ------------------------------------ >> [1]PETSC ERROR: Signal received! >> [1]PETSC ERROR: ------------------------------------------------------------------------ >> [1]PETSC ERROR: Petsc Release Version 3.4.1, Jun, 10, 2013 >> [1]PETSC ERROR: See docs/changes/index.html for recent updates. >> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [1]PETSC ERROR: See docs/index.html for manual pages. >> [1]PETSC ERROR: ------------------------------------------------------------------------ >> [1]PETSC ERROR: ./tenfac on a linux named rossmann-b000.rcac.purdue.edu by choi240 Fri Sep 27 08:01:31 2013 >> [1]PETSC ERROR: Libraries linked from /apps/rhel5/petsc-3.4.1/64/impi-4.1.0.030_intel-13.0.1.117_ind64/linux/lib >> [1]PETSC ERROR: Configure run at Sun Jun 23 10:20:51 2013 >> [1]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --download-sowing --with-scalar-type=real --with-shared-libraries=1 --with-pic=1 --with-clanguage=C++ --with-fortran --with-fortran-kernels=1 --with-64-bit-indices=1 --with-debugging=0 --with-blas-lapack-dir=/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64 --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --download-hdf5=no --download-metis=no --download-parmetis=no --download-superlu_dist=no --download-mumps=no --download-scalapack=yes --download-blacs=no --download-hypre=no --download-spooles=no >> [1]PETSC ERROR: ------------------------------------------------------------------------ >> [1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1 From garnet.vaz at gmail.com Fri Sep 27 12:12:26 2013 From: garnet.vaz at gmail.com (Garnet Vaz) Date: Fri, 27 Sep 2013 10:12:26 -0700 Subject: [petsc-users] Writing VTK output for vectors derived from DM Global vector In-Reply-To: References: <87li2n2mt5.fsf@mcs.anl.gov> Message-ID: Hi, Does the vtk output write complex vectors? I wrote an example to test the usage but only the real part of the vector is written to the file. Is complex output supported? Thanks. - Garnet On Tue, Sep 24, 2013 at 6:44 AM, Garnet Vaz wrote: > Hi Matt, > > Yes. Thanks. I got it to work. The shell on emacs had not > updated $PETSC_DIR and was using 3.4.2. > > - > Garnet > > > On Tue, Sep 24, 2013 at 6:26 AM, Matthew Knepley wrote: > >> On Tue, Sep 24, 2013 at 6:16 AM, Garnet Vaz wrote: >> >>> Hi Matt, >>> >>> I did pull yesterday. My configure.log file has >>> Defined "VERSION_GIT" to >>> ""b4e904d1358c92c43842a940440f995d2117e40b"" >>> Defined "VERSION_DATE_GIT" to ""2013-09-23 12:36:08 +0100"" >>> >>> Is the change added after this? >>> >> >> I see it here: >> >> >> https://bitbucket.org/petsc/petsc/src/33dd83b134909f5b4acb842818b1e0de2db2f4eb/include/petscdm.h?at=master#cl-35 >> >> and it has been there since this commit in July: >> >> >> https://bitbucket.org/petsc/petsc/commits/38221697d93bc7c9a6149445938a5de22e85bc83 >> >> Matt >> >> >>> - >>> Garnet >>> >>> >>> >>> On Tue, Sep 24, 2013 at 6:08 AM, Matthew Knepley wrote: >>> >>>> On Tue, Sep 24, 2013 at 5:54 AM, Garnet Vaz wrote: >>>> >>>>> Hi, >>>>> >>>>> Thanks Jed. I will set the field names before I write them. >>>>> I need to create a clone and then change the default section >>>>> to have dof to 1 before I write them. I am not able to use clone with >>>>> >>>>> DMClone(originaldm,&newdm); >>>>> >>>>> The error I get when compiling is >>>>> "'DMClone' was not declared in this scope". >>>>> >>>>> I do have #include which includes the >>>>> petscdm.h header. Other than clone I am able to use the >>>>> other functions. >>>>> >>>> >>>> Have you updated? I think I forgot the declaration initially. >>>> >>>> Matt >>>> >>>> >>>>> Thanks. >>>>> - >>>>> Garnet >>>>> >>>>> >>>>> On Mon, Sep 23, 2013 at 3:30 PM, Jed Brown wrote: >>>>> >>>>>> Garnet Vaz writes: >>>>>> >>>>>> > Hi, >>>>>> > >>>>>> > I would like to write out my solution to a VTK file where I use >>>>>> unstructured >>>>>> > grids in DMPlex. >>>>>> > >>>>>> > The solution corresponds to harmonic frequencies (~ 5-10 distinct) >>>>>> which >>>>>> > are >>>>>> > currently saved into a Vec* object. These vectors were created by >>>>>> > duplicating the >>>>>> > global vector for the DM object. (I know that they correspond to >>>>>> the right >>>>>> > cell.) >>>>>> > >>>>>> > Is it possible to have one file which would contain the mesh from >>>>>> the DM >>>>>> > object and all the Vec* objects? >>>>>> > Would I have to clone the original DM and use >>>>>> > >>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerVTKAddField.html >>>>>> > to include all the vectors? >>>>>> >>>>>> This is a private function. >>>>>> >>>>>> > In the above function does >>>>>> > PetscViewerVTKWriteFunction >>>>>> > just need to be >>>>>> > >>>>>> > myWriteFunc(vec,viewer) >>>>>> > { >>>>>> > VecView(vec,viewer) >>>>>> > } >>>>>> > if I do not need to make any changes to the vector? >>>>>> >>>>>> This should work, but you may want to use PetscObjectSetName() so that >>>>>> the names of the different fields are sensible. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> Garnet >>>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >>> >>> -- >>> Regards, >>> Garnet >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > Regards, > Garnet > -- Regards, Garnet -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Sep 27 12:14:30 2013 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 27 Sep 2013 10:14:30 -0700 Subject: [petsc-users] Writing VTK output for vectors derived from DM Global vector In-Reply-To: References: <87li2n2mt5.fsf@mcs.anl.gov> Message-ID: On Fri, Sep 27, 2013 at 10:12 AM, Garnet Vaz wrote: > Hi, > > Does the vtk output write complex vectors? I wrote an example > to test the usage but only the real part of the vector is written > to the file. Is complex output supported? > I had never considered that since I did not know how VTK would plot it. Does VTK support it? If so, I can change the code to output what you want. Thanks, Matt > Thanks. > - > Garnet > > > On Tue, Sep 24, 2013 at 6:44 AM, Garnet Vaz wrote: > >> Hi Matt, >> >> Yes. Thanks. I got it to work. The shell on emacs had not >> updated $PETSC_DIR and was using 3.4.2. >> >> - >> Garnet >> >> >> On Tue, Sep 24, 2013 at 6:26 AM, Matthew Knepley wrote: >> >>> On Tue, Sep 24, 2013 at 6:16 AM, Garnet Vaz wrote: >>> >>>> Hi Matt, >>>> >>>> I did pull yesterday. My configure.log file has >>>> Defined "VERSION_GIT" to >>>> ""b4e904d1358c92c43842a940440f995d2117e40b"" >>>> Defined "VERSION_DATE_GIT" to ""2013-09-23 12:36:08 +0100"" >>>> >>>> Is the change added after this? >>>> >>> >>> I see it here: >>> >>> >>> https://bitbucket.org/petsc/petsc/src/33dd83b134909f5b4acb842818b1e0de2db2f4eb/include/petscdm.h?at=master#cl-35 >>> >>> and it has been there since this commit in July: >>> >>> >>> https://bitbucket.org/petsc/petsc/commits/38221697d93bc7c9a6149445938a5de22e85bc83 >>> >>> Matt >>> >>> >>>> - >>>> Garnet >>>> >>>> >>>> >>>> On Tue, Sep 24, 2013 at 6:08 AM, Matthew Knepley wrote: >>>> >>>>> On Tue, Sep 24, 2013 at 5:54 AM, Garnet Vaz wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Thanks Jed. I will set the field names before I write them. >>>>>> I need to create a clone and then change the default section >>>>>> to have dof to 1 before I write them. I am not able to use clone with >>>>>> >>>>>> DMClone(originaldm,&newdm); >>>>>> >>>>>> The error I get when compiling is >>>>>> "'DMClone' was not declared in this scope". >>>>>> >>>>>> I do have #include which includes the >>>>>> petscdm.h header. Other than clone I am able to use the >>>>>> other functions. >>>>>> >>>>> >>>>> Have you updated? I think I forgot the declaration initially. >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Thanks. >>>>>> - >>>>>> Garnet >>>>>> >>>>>> >>>>>> On Mon, Sep 23, 2013 at 3:30 PM, Jed Brown wrote: >>>>>> >>>>>>> Garnet Vaz writes: >>>>>>> >>>>>>> > Hi, >>>>>>> > >>>>>>> > I would like to write out my solution to a VTK file where I use >>>>>>> unstructured >>>>>>> > grids in DMPlex. >>>>>>> > >>>>>>> > The solution corresponds to harmonic frequencies (~ 5-10 >>>>>>> distinct) which >>>>>>> > are >>>>>>> > currently saved into a Vec* object. These vectors were created by >>>>>>> > duplicating the >>>>>>> > global vector for the DM object. (I know that they correspond to >>>>>>> the right >>>>>>> > cell.) >>>>>>> > >>>>>>> > Is it possible to have one file which would contain the mesh from >>>>>>> the DM >>>>>>> > object and all the Vec* objects? >>>>>>> > Would I have to clone the original DM and use >>>>>>> > >>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerVTKAddField.html >>>>>>> > to include all the vectors? >>>>>>> >>>>>>> This is a private function. >>>>>>> >>>>>>> > In the above function does >>>>>>> > PetscViewerVTKWriteFunction >>>>>>> > just need to be >>>>>>> > >>>>>>> > myWriteFunc(vec,viewer) >>>>>>> > { >>>>>>> > VecView(vec,viewer) >>>>>>> > } >>>>>>> > if I do not need to make any changes to the vector? >>>>>>> >>>>>>> This should work, but you may want to use PetscObjectSetName() so >>>>>>> that >>>>>>> the names of the different fields are sensible. >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Regards, >>>>>> Garnet >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>> >>>> >>>> >>>> -- >>>> Regards, >>>> Garnet >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> >> >> -- >> Regards, >> Garnet >> > > > > -- > Regards, > Garnet > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From garnet.vaz at gmail.com Fri Sep 27 12:20:04 2013 From: garnet.vaz at gmail.com (Garnet Vaz) Date: Fri, 27 Sep 2013 10:20:04 -0700 Subject: [petsc-users] Writing VTK output for vectors derived from DM Global vector In-Reply-To: References: <87li2n2mt5.fsf@mcs.anl.gov> Message-ID: Hi Matt, I did run into some documentation that said that complex output should use complex instead of scalars and have Real[0] Imag[0] ... Real[N] Imag[N]. I will try to find the documentation. - Garnet On Fri, Sep 27, 2013 at 10:14 AM, Matthew Knepley wrote: > On Fri, Sep 27, 2013 at 10:12 AM, Garnet Vaz wrote: > >> Hi, >> >> Does the vtk output write complex vectors? I wrote an example >> to test the usage but only the real part of the vector is written >> to the file. Is complex output supported? >> > > I had never considered that since I did not know how VTK would plot it. > Does VTK support it? > If so, I can change the code to output what you want. > > Thanks, > > Matt > > >> Thanks. >> - >> Garnet >> >> >> On Tue, Sep 24, 2013 at 6:44 AM, Garnet Vaz wrote: >> >>> Hi Matt, >>> >>> Yes. Thanks. I got it to work. The shell on emacs had not >>> updated $PETSC_DIR and was using 3.4.2. >>> >>> - >>> Garnet >>> >>> >>> On Tue, Sep 24, 2013 at 6:26 AM, Matthew Knepley wrote: >>> >>>> On Tue, Sep 24, 2013 at 6:16 AM, Garnet Vaz wrote: >>>> >>>>> Hi Matt, >>>>> >>>>> I did pull yesterday. My configure.log file has >>>>> Defined "VERSION_GIT" to >>>>> ""b4e904d1358c92c43842a940440f995d2117e40b"" >>>>> Defined "VERSION_DATE_GIT" to ""2013-09-23 12:36:08 +0100"" >>>>> >>>>> Is the change added after this? >>>>> >>>> >>>> I see it here: >>>> >>>> >>>> https://bitbucket.org/petsc/petsc/src/33dd83b134909f5b4acb842818b1e0de2db2f4eb/include/petscdm.h?at=master#cl-35 >>>> >>>> and it has been there since this commit in July: >>>> >>>> >>>> https://bitbucket.org/petsc/petsc/commits/38221697d93bc7c9a6149445938a5de22e85bc83 >>>> >>>> Matt >>>> >>>> >>>>> - >>>>> Garnet >>>>> >>>>> >>>>> >>>>> On Tue, Sep 24, 2013 at 6:08 AM, Matthew Knepley wrote: >>>>> >>>>>> On Tue, Sep 24, 2013 at 5:54 AM, Garnet Vaz wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Thanks Jed. I will set the field names before I write them. >>>>>>> I need to create a clone and then change the default section >>>>>>> to have dof to 1 before I write them. I am not able to use clone with >>>>>>> >>>>>>> DMClone(originaldm,&newdm); >>>>>>> >>>>>>> The error I get when compiling is >>>>>>> "'DMClone' was not declared in this scope". >>>>>>> >>>>>>> I do have #include which includes the >>>>>>> petscdm.h header. Other than clone I am able to use the >>>>>>> other functions. >>>>>>> >>>>>> >>>>>> Have you updated? I think I forgot the declaration initially. >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Thanks. >>>>>>> - >>>>>>> Garnet >>>>>>> >>>>>>> >>>>>>> On Mon, Sep 23, 2013 at 3:30 PM, Jed Brown wrote: >>>>>>> >>>>>>>> Garnet Vaz writes: >>>>>>>> >>>>>>>> > Hi, >>>>>>>> > >>>>>>>> > I would like to write out my solution to a VTK file where I use >>>>>>>> unstructured >>>>>>>> > grids in DMPlex. >>>>>>>> > >>>>>>>> > The solution corresponds to harmonic frequencies (~ 5-10 >>>>>>>> distinct) which >>>>>>>> > are >>>>>>>> > currently saved into a Vec* object. These vectors were created by >>>>>>>> > duplicating the >>>>>>>> > global vector for the DM object. (I know that they correspond to >>>>>>>> the right >>>>>>>> > cell.) >>>>>>>> > >>>>>>>> > Is it possible to have one file which would contain the mesh from >>>>>>>> the DM >>>>>>>> > object and all the Vec* objects? >>>>>>>> > Would I have to clone the original DM and use >>>>>>>> > >>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerVTKAddField.html >>>>>>>> > to include all the vectors? >>>>>>>> >>>>>>>> This is a private function. >>>>>>>> >>>>>>>> > In the above function does >>>>>>>> > PetscViewerVTKWriteFunction >>>>>>>> > just need to be >>>>>>>> > >>>>>>>> > myWriteFunc(vec,viewer) >>>>>>>> > { >>>>>>>> > VecView(vec,viewer) >>>>>>>> > } >>>>>>>> > if I do not need to make any changes to the vector? >>>>>>>> >>>>>>>> This should work, but you may want to use PetscObjectSetName() so >>>>>>>> that >>>>>>>> the names of the different fields are sensible. >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Regards, >>>>>>> Garnet >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> Garnet >>>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >>> >>> -- >>> Regards, >>> Garnet >>> >> >> >> >> -- >> Regards, >> Garnet >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Regards, Garnet -------------- next part -------------- An HTML attachment was scrubbed... URL: From parsani.matteo at gmail.com Fri Sep 27 12:20:09 2013 From: parsani.matteo at gmail.com (Matteo Parsani) Date: Fri, 27 Sep 2013 13:20:09 -0400 Subject: [petsc-users] MatMPIBAIJSetPreallocationCSR i and j indices Message-ID: Dear PETSc users and developers, I am assembling a PC matrix in parallel. For instance, with just one element in the mesh the structure of the matrix is *[5x5 nonzero] [5x5 nonzero] [5x5 nonzero] [5x5 nonzero]* [5x5 ZEROs] [5x5 ZEROs]* [5x5 nonzero]* [5x5 ZEROs] [5x5 ZEROs]* [5x5 nonzero] [5x5 nonzero] [5x5 nonzero] *[5x5 ZEROs] *[5x5 nonzero]* [5x5 ZEROs] [5x5 ZEROs]* [5x5 nonzero] *[5x5 ZEROs] *[5x5 nonzero] [5x5 nonzero] [5x5 nonzero]* [5x5 ZEROs] [5x5 ZEROs] *[5x5 nonzero]* [5x5 ZEROs] [5x5 ZEROs] * [5x5 nonzero]* *[5x5 nonzero]* [5x5 ZEROs] [5x5 ZEROs]* [5x5 nonzero] [5x5 nonzero] [5x5 nonzero]** [5x5 nonzero]* [5x5 ZEROs] [5x5 ZEROs] [5x5 ZEROs] *[5x5 nonzero]* [5x5 ZEROs]* ** [5x5 nonzero] [5x5 nonzero] [5x5 nonzero] *[5x5 ZEROs] *[5x5 nonzero]* [5x5 ZEROs] [5x5 ZEROs] [5x5 ZEROs]* [5x5 nonzero]** ** [5x5 nonzero] [5x5 nonzero] [5x5 nonzero]* [5x5 ZEROs] [5x5 ZEROs] *[5x5 nonzero]* *[5x5 nonzero]* [5x5 ZEROs] [5x5 ZEROs]* [5x5 nonzero]* [5x5 ZEROs] [5x5 ZEROs]* ** [5x5 nonzero] [5x5 nonzero] [5x5 nonzero]* [5x5 ZEROs] *[5x5 nonzero]* [5x5 ZEROs]* *[5x5 ZEROs] *[5x5 nonzero]*[5x5 ZEROs]* ** [5x5 nonzero] [5x5 nonzero] [5x5 nonzero]* [5x5 ZEROs]* *[5x5 ZEROs] *[5x5 nonzero]* [5x5 ZEROs] [5x5 ZEROs] *[5x5 nonzero] [5x5 nonzero] [5x5 nonzero]** [5x5 nonzero]* With multiple elements in the mesh, the above matrix will be the structure of the diagonal blocks of the PC matrix. If I call the block above [A], then the full PC matrix (assemble through all the processors) will look like [A] 0 0 0 0 0 0 0 0 [A] 0 0 0 0 0 0 0 0 [A] 0 0 0 0 0 0 0 0 [A] 0 0 0 0 PC= 0 0 0 0 [A] 0 0 0 -> PC matrix if I have 8 element in my mesh. 0 0 0 0 0 [A] 0 0 0 0 0 0 0 0 [A] 0 0 0 0 0 0 0 0 [A] Now in my code I have constructed the CSR format which gives the position of the blue nonzero blocks in the PC matrix constructed by each processor. I am calling the following PETSc functions: call MatCreate(PETSC_COMM_WORLD,petsc_pc_mat,i_err) call check_petsc_error(i_err) call MatSetType(petsc_pc_mat,MATMPIBAIJ,i_err) call check_petsc_error(i_err) call MatSetSizes(petsc_pc_mat,nprows,nprows,ngrows,ngrows,i_err) ! nprowsis the number of local rows which is equal to the number of local columns call check_petsc_error(i_err) call MatSetFromOptions(petsc_pc_mat,i_err) call check_petsc_error(i_err) call MatSetUp(petsc_pc_mat,i_err) call check_petsc_error(i_err) block_size = nequations call MatMPIBAIJSetPreallocationCSR(petsc_pc_mat,block_size,ia, ja, PETSC_NULL_SCALAR i_err) call check_petsc_error(i_err) 1) In the latter call, should ia and ja be the CSR vectors which gives the position of the blue blocks in the matrix or the ia and ja be the CSRvectors which gives the position of the nnz scalar terms in the PC matrix? 2) My blue block are stored in an array of 3 dimensions: a (1:5,1:5, Number_of_blocks). I am thinking to try to use 2 approaches: a) Looping over the number of blocks and at each time set a single block in the matrix. However, I have seen that I should flatten the array before to pass the values. Is there a way to avoid to flatten the array and just pass it as a(1:5,1:5,block_index)? b) If I am not wrong, having the CSR structure I could call just 1 time MatSetValueBlocked and set the entire matrix owned by one process. Am I right? Thank you. Best Regards -- Matteo -------------- next part -------------- An HTML attachment was scrubbed... URL: From garnet.vaz at gmail.com Fri Sep 27 12:35:41 2013 From: garnet.vaz at gmail.com (Garnet Vaz) Date: Fri, 27 Sep 2013 10:35:41 -0700 Subject: [petsc-users] Writing VTK output for vectors derived from DM Global vector In-Reply-To: References: <87li2n2mt5.fsf@mcs.anl.gov> Message-ID: Hi Matt, I can just as easily write the real and imaginary parts into two components. This would be more flexible as well. I do not know yet if I can select only the real/imaginary parts of the solution in VisIT. Thanks. - Garnet On Fri, Sep 27, 2013 at 10:20 AM, Garnet Vaz wrote: > Hi Matt, > > I did run into some documentation that said that complex output > should use complex instead of scalars and have > Real[0] > Imag[0] > ... > Real[N] > Imag[N]. > > > I will try to find the documentation. > > - > Garnet > > > On Fri, Sep 27, 2013 at 10:14 AM, Matthew Knepley wrote: > >> On Fri, Sep 27, 2013 at 10:12 AM, Garnet Vaz wrote: >> >>> Hi, >>> >>> Does the vtk output write complex vectors? I wrote an example >>> to test the usage but only the real part of the vector is written >>> to the file. Is complex output supported? >>> >> >> I had never considered that since I did not know how VTK would plot it. >> Does VTK support it? >> If so, I can change the code to output what you want. >> >> Thanks, >> >> Matt >> >> >>> Thanks. >>> - >>> Garnet >>> >>> >>> On Tue, Sep 24, 2013 at 6:44 AM, Garnet Vaz wrote: >>> >>>> Hi Matt, >>>> >>>> Yes. Thanks. I got it to work. The shell on emacs had not >>>> updated $PETSC_DIR and was using 3.4.2. >>>> >>>> - >>>> Garnet >>>> >>>> >>>> On Tue, Sep 24, 2013 at 6:26 AM, Matthew Knepley wrote: >>>> >>>>> On Tue, Sep 24, 2013 at 6:16 AM, Garnet Vaz wrote: >>>>> >>>>>> Hi Matt, >>>>>> >>>>>> I did pull yesterday. My configure.log file has >>>>>> Defined "VERSION_GIT" to >>>>>> ""b4e904d1358c92c43842a940440f995d2117e40b"" >>>>>> Defined "VERSION_DATE_GIT" to ""2013-09-23 12:36:08 +0100"" >>>>>> >>>>>> Is the change added after this? >>>>>> >>>>> >>>>> I see it here: >>>>> >>>>> >>>>> https://bitbucket.org/petsc/petsc/src/33dd83b134909f5b4acb842818b1e0de2db2f4eb/include/petscdm.h?at=master#cl-35 >>>>> >>>>> and it has been there since this commit in July: >>>>> >>>>> >>>>> https://bitbucket.org/petsc/petsc/commits/38221697d93bc7c9a6149445938a5de22e85bc83 >>>>> >>>>> Matt >>>>> >>>>> >>>>>> - >>>>>> Garnet >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Sep 24, 2013 at 6:08 AM, Matthew Knepley wrote: >>>>>> >>>>>>> On Tue, Sep 24, 2013 at 5:54 AM, Garnet Vaz wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> Thanks Jed. I will set the field names before I write them. >>>>>>>> I need to create a clone and then change the default section >>>>>>>> to have dof to 1 before I write them. I am not able to use clone >>>>>>>> with >>>>>>>> >>>>>>>> DMClone(originaldm,&newdm); >>>>>>>> >>>>>>>> The error I get when compiling is >>>>>>>> "'DMClone' was not declared in this scope". >>>>>>>> >>>>>>>> I do have #include which includes the >>>>>>>> petscdm.h header. Other than clone I am able to use the >>>>>>>> other functions. >>>>>>>> >>>>>>> >>>>>>> Have you updated? I think I forgot the declaration initially. >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> Thanks. >>>>>>>> - >>>>>>>> Garnet >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Sep 23, 2013 at 3:30 PM, Jed Brown wrote: >>>>>>>> >>>>>>>>> Garnet Vaz writes: >>>>>>>>> >>>>>>>>> > Hi, >>>>>>>>> > >>>>>>>>> > I would like to write out my solution to a VTK file where I use >>>>>>>>> unstructured >>>>>>>>> > grids in DMPlex. >>>>>>>>> > >>>>>>>>> > The solution corresponds to harmonic frequencies (~ 5-10 >>>>>>>>> distinct) which >>>>>>>>> > are >>>>>>>>> > currently saved into a Vec* object. These vectors were created by >>>>>>>>> > duplicating the >>>>>>>>> > global vector for the DM object. (I know that they correspond to >>>>>>>>> the right >>>>>>>>> > cell.) >>>>>>>>> > >>>>>>>>> > Is it possible to have one file which would contain the mesh >>>>>>>>> from the DM >>>>>>>>> > object and all the Vec* objects? >>>>>>>>> > Would I have to clone the original DM and use >>>>>>>>> > >>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerVTKAddField.html >>>>>>>>> > to include all the vectors? >>>>>>>>> >>>>>>>>> This is a private function. >>>>>>>>> >>>>>>>>> > In the above function does >>>>>>>>> > PetscViewerVTKWriteFunction >>>>>>>>> > just need to be >>>>>>>>> > >>>>>>>>> > myWriteFunc(vec,viewer) >>>>>>>>> > { >>>>>>>>> > VecView(vec,viewer) >>>>>>>>> > } >>>>>>>>> > if I do not need to make any changes to the vector? >>>>>>>>> >>>>>>>>> This should work, but you may want to use PetscObjectSetName() so >>>>>>>>> that >>>>>>>>> the names of the different fields are sensible. >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Regards, >>>>>>>> Garnet >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Regards, >>>>>> Garnet >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>> >>>> >>>> >>>> -- >>>> Regards, >>>> Garnet >>>> >>> >>> >>> >>> -- >>> Regards, >>> Garnet >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > Regards, > Garnet > -- Regards, Garnet -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Sep 27 14:23:09 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 27 Sep 2013 14:23:09 -0500 Subject: [petsc-users] MatMPIBAIJSetPreallocationCSR i and j indices In-Reply-To: References: Message-ID: Just use MatCreateMPIBAIJWithArrays() saves thinking and work. The i and j indices are with respect to blocks, not the individual entries in the matrix. Run one one process with just a couple of grid points and then use MatView() to look at the matrix and make sure it is correct On Sep 27, 2013, at 12:20 PM, Matteo Parsani wrote: > Dear PETSc users and developers, > I am assembling a PC matrix in parallel. For instance, with just one element in the mesh the structure of the matrix is > > [5x5 nonzero] [5x5 nonzero] [5x5 nonzero] [5x5 nonzero] [5x5 ZEROs] [5x5 ZEROs] [5x5 nonzero] [5x5 ZEROs] [5x5 ZEROs] > [5x5 nonzero] [5x5 nonzero] [5x5 nonzero] [5x5 ZEROs] [5x5 nonzero] [5x5 ZEROs] [5x5 ZEROs] [5x5 nonzero] [5x5 ZEROs] > [5x5 nonzero] [5x5 nonzero] [5x5 nonzero] [5x5 ZEROs] [5x5 ZEROs] [5x5 nonzero] [5x5 ZEROs] [5x5 ZEROs] [5x5 nonzero] > [5x5 nonzero] [5x5 ZEROs] [5x5 ZEROs] [5x5 nonzero] [5x5 nonzero] [5x5 nonzero] [5x5 nonzero] [5x5 ZEROs] [5x5 ZEROs] > [5x5 ZEROs] [5x5 nonzero] [5x5 ZEROs] [5x5 nonzero] [5x5 nonzero] [5x5 nonzero] [5x5 ZEROs] [5x5 nonzero] [5x5 ZEROs] > [5x5 ZEROs] [5x5 ZEROs] [5x5 nonzero] [5x5 nonzero] [5x5 nonzero] [5x5 nonzero] [5x5 ZEROs] [5x5 ZEROs] [5x5 nonzero] > [5x5 nonzero] [5x5 ZEROs] [5x5 ZEROs] [5x5 nonzero] [5x5 ZEROs] [5x5 ZEROs] [5x5 nonzero] [5x5 nonzero] [5x5 nonzero] > [5x5 ZEROs] [5x5 nonzero] [5x5 ZEROs] [5x5 ZEROs] [5x5 nonzero] [5x5 ZEROs] [5x5 nonzero] [5x5 nonzero] [5x5 nonzero] > [5x5 ZEROs] [5x5 ZEROs] [5x5 nonzero] [5x5 ZEROs] [5x5 ZEROs] [5x5 nonzero] [5x5 nonzero] [5x5 nonzero] [5x5 nonzero] > > > With multiple elements in the mesh, the above matrix will be the structure of the diagonal blocks of the PC matrix. If I call the block above [A], then the full PC matrix (assemble through all the processors) will look like > > [A] 0 0 0 0 0 0 0 > 0 [A] 0 0 0 0 0 0 > 0 0 [A] 0 0 0 0 0 > 0 0 0 [A] 0 0 0 0 > PC= 0 0 0 0 [A] 0 0 0 -> PC matrix if I have 8 element in my mesh. > 0 0 0 0 0 [A] 0 0 > 0 0 0 0 0 0 [A] 0 > 0 0 0 0 0 0 0 [A] > > Now in my code I have constructed the CSR format which gives the position of the blue nonzero blocks in the PC matrix constructed by each processor. > > I am calling the following PETSc functions: > > call MatCreate(PETSC_COMM_WORLD,petsc_pc_mat,i_err) > call check_petsc_error(i_err) > > call MatSetType(petsc_pc_mat,MATMPIBAIJ,i_err) > call check_petsc_error(i_err) > > call MatSetSizes(petsc_pc_mat,nprows,nprows,ngrows,ngrows,i_err) ! nprows is the number of local rows which is equal to the number of local columns > call check_petsc_error(i_err) > > call MatSetFromOptions(petsc_pc_mat,i_err) > call check_petsc_error(i_err) > > call MatSetUp(petsc_pc_mat,i_err) > call check_petsc_error(i_err) > > block_size = nequations > call MatMPIBAIJSetPreallocationCSR(petsc_pc_mat,block_size,ia, ja, PETSC_NULL_SCALAR i_err) > call check_petsc_error(i_err) > > 1) In the latter call, should ia and ja be the CSR vectors which gives the position of the blue blocks in the matrix or the ia and ja be the CSR vectors which gives the position of the nnz scalar terms in the PC matrix? > > 2) My blue block are stored in an array of 3 dimensions: a (1:5,1:5, Number_of_blocks). I am thinking to try to use 2 approaches: > a) Looping over the number of blocks and at each time set a single block in the matrix. However, I have seen that I should flatten the array before to pass the values. Is there a way to avoid to flatten the array and just pass it as a(1:5,1:5,block_index)? > > b) If I am not wrong, having the CSR structure I could call just 1 time MatSetValueBlocked and set the entire matrix owned by one process. Am I right? > > Thank you. > > Best Regards > > > > > > > > -- > Matteo From bsmith at mcs.anl.gov Fri Sep 27 14:28:01 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 27 Sep 2013 14:28:01 -0500 Subject: [petsc-users] Back to struct in Fortran to represent field with dof > 1 In-Reply-To: References: <3524B3B9-C5FE-46A7-8556-F36DEEC287BD@mcs.anl.gov> Message-ID: Please find attached a new version of the code. ex1.c and ex1f90.F please run it as is and send me all the output. It works correctly with gfortran and the intel 12.x ifort compiler, I believe it should now work for all fortran compilers. If it does not work as expected please send me the name and version of your fortran compiler. Barry On Sep 27, 2013, at 8:22 AM, Christophe Ortiz wrote: > There was a misunderstanding. There are n entries in total in the vector. I thought it was a vector of n points, with 3 components associated to each point, ie 3*n entries. This is clear now. > > I've tested your example and I checked the values assigned to the components with your piece of code: > > call VecGetArrayMyStruct(x,xarray,ierr) > do i=1,10 > xarray(i)%a = i > xarray(i)%b = 100*i > xarray(i)%c = 10000*i > enddo > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > I did it with: > > call VecGetArrayMyStruct(x,xarray,ierr) > do i = 1 , 10 > write(*,*) xarray(i)%a,xarray(i)%b,xarray(i)%c > end do > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > I obtained the following: > > 1.00000000000000 2.00000000000000 3.00000000000000 > 2.00000000000000 3.00000000000000 4.00000000000000 > 3.00000000000000 4.00000000000000 5.00000000000000 > 4.00000000000000 5.00000000000000 6.00000000000000 > 5.00000000000000 6.00000000000000 7.00000000000000 > 6.00000000000000 7.00000000000000 8.00000000000000 > 7.00000000000000 8.00000000000000 9.00000000000000 > 8.00000000000000 9.00000000000000 10.0000000000000 > 9.00000000000000 10.0000000000000 1000.00000000000 > 10.0000000000000 1000.00000000000 100000.000000000 > > First column if ok (1st component), but the other two columns show wrong numbers. For component 2 and 3, only the very last value is ok. It seems values from component b are copied from a, and those from component c from b, with an offset. > > What do you think ? > > Christophe > > > On Fri, Sep 27, 2013 at 2:47 PM, Barry Smith wrote: > > Indices start in fortran arrays from 1 by default so don't even try zero > > I've attached a new version that has three components, > 30 TOTAL vector entries and 10 for each component. Please run it and send the results back. > > Barry > > On Sep 27, 2013, at 5:40 AM, Christophe Ortiz wrote: > > > Hi Barry, > > > > I made some tests with your code that allows to use struct in Fortran90. It compiles and does not complain with the struct with 2 components but...it exhibits some strange behaviour. Here are the behaviours I observed. To make it simple, I used n=10. > > > > - After using call VecSet(x,one,ierr), I checked the values of each component of the vector using > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > do i = 0 , n > > write(*,*) i , xarray(i)%a, xarray(i)%b > > end do > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > I checked the values from i= 0 to n since I was not sure what was the convention (0- or 1-based). I obtained: > > > > 0 0.000000000000000E+000 1.00000000000000 > > 1 1.00000000000000 1.00000000000000 > > 2 1.00000000000000 1.00000000000000 > > 3 1.00000000000000 1.00000000000000 > > 4 1.00000000000000 1.00000000000000 > > 5 1.00000000000000 1.00000000000000 > > 6 1.00000000000000 1.00000000000000 > > 7 1.00000000000000 1.00000000000000 > > 8 1.00000000000000 1.00000000000000 > > 9 1.00000000000000 1.00000000000000 > > 10 1.00000000000000 1.996650376645798E-314 > > > > > > As you can see, for component a, there is a 0 for i=0, then the values are correct. For component b, correct values go from i=1 to 9.There is garbage in second component for i=10 . I checked that if you define a third component c, its values go from i=-1 to n-2. It's like if values of second component started at the end of first component, and same thing for third component. > > > > Then, instead of using VecSet(), I tried to assign values directly to each component of the vector with > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > do i = 0 , n-1 > > xarray(i)%a = 2.0 > > xarray(i)%b = 3.0 > > end do > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > Here I checked that indices actually start at 0 and not at 1. With a loop from i=1 to n I got a Memory corruption message. > > > > and I checked that the values are correctly assigned with > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > do i = 0 , n-1 > > write(*,*) i, xarray(i)%a, xarray(i)%b > > end do > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > > > > > Then, I obtained: > > > > 0 2.00000000000000 2.00000000000000 > > 1 2.00000000000000 2.00000000000000 > > 2 2.00000000000000 2.00000000000000 > > 3 2.00000000000000 2.00000000000000 > > 4 2.00000000000000 2.00000000000000 > > 5 2.00000000000000 2.00000000000000 > > 6 2.00000000000000 2.00000000000000 > > 7 2.00000000000000 2.00000000000000 > > 8 2.00000000000000 2.00000000000000 > > 9 2.00000000000000 3.00000000000000 > > > > > > Here the problem seems more severe. Values are not correctly assigned to the second component. There should be 3.0 in the second column. It seems values of the first component are copied to the second one. Only for i=10 the value of xarray(i)%b is correct (3.0). > > > > Any idea where it could come from ? > > I guess it needs some fixes here and there but I think your idea is good and that it could work. > > > > Christophe > > > > > > > > On Thu, Sep 26, 2013 at 5:45 PM, Barry Smith wrote: > > > > I just put it with the Fortran source code and compile it with the rest of the application code; here is the makefile I used > > > > > > > > > > On Sep 26, 2013, at 10:14 AM, Christophe Ortiz wrote: > > > > > By the way, what should I do with the small .c code ? Where should I put it ? > > > > > > Christophe > > > > > > > > > > > > On Thu, Sep 26, 2013 at 4:09 PM, Barry Smith wrote: > > > > > > Christophe, > > > > > > Despite my LinkIn Endorsement for expertise in Fortran :-) I cannot pretend to be an expert in FortranXX but I have cooked up an example demonstrating accessing the Vec entries as if they are in an array of derived types. I've attached the example code; there needs to be a small C stub that defines the functions for your specific derived type name. > > > Note that it will only work I think if your N is a compile time constant. > > > > > > > > > > > > > > > It worked with > > > ~/Downloads$ gfortran --version > > > GNU Fortran (GCC) 4.8.1 20130404 (prerelease) > > > > > > > > > I do not understand exactly why it works since it uses F90Array1dCreate(fa,PETSC_SCALAR,1,len,ptr PETSC_F90_2PTR_PARAM(ptrd)); which has a single PETSC_SCALAR as a building block but ? I hope it works for you. If it doesn't, let us know the compiler you are using and we may be able to get it working for that compiler. > > > > > > Barry > > > > > > > > > > > > > > > On Sep 26, 2013, at 4:41 AM, Christophe Ortiz wrote: > > > > > > > Hi all, > > > > > > > > Me again ! > > > > > > > > I have read in previous posts that it is hard in Fortran to declare something similar to a typedef struct in C to manage a multicomponent problem. > > > > > > > > Is it still the case ? Has the problem been solved ? > > > > > > > > I am asking because my plan is to implement a multicomponent problem (in 1D), with many components that will be organized in arrays of two dimensions. In C I could define > > > > > > > > typedef struct{ > > > > PetscScalar U; > > > > PetscScalar V; > > > > PetscScalar A[N][N]; > > > > } Field; > > > > > > > > and then I could calculate the residual function with > > > > > > > > F[i].U = ... > > > > F[i].V = ... > > > > F[i].A[k][n] = ... > > > > > > > > which is quite convenient. > > > > > > > > If in Fortran it is not possible to use a struct as in C, I am afraid I'll have to deal with > > > > > > > > F(jdof,i) = ... > > > > > > > > where I will have only jdof to address U, V and A[ ][ ], which can be difficult and not very convenient I guess. Before I move all my code to C, does anyone have an alternative idea to manage this multi(many)component problem in Fortran ? > > > > > > > > Many thanks in advance for your help and suggestion ! > > > > > > > > Christophe > > > > > > > > > > > > > > > > > > > > > From christophe.ortiz at ciemat.es Fri Sep 27 15:06:04 2013 From: christophe.ortiz at ciemat.es (Christophe Ortiz) Date: Fri, 27 Sep 2013 22:06:04 +0200 Subject: [petsc-users] Back to struct in Fortran to represent field with dof > 1 In-Reply-To: References: <3524B3B9-C5FE-46A7-8556-F36DEEC287BD@mcs.anl.gov> Message-ID: The files were not attached. Could you please send them again ? Thanks. Christophe El 27/09/2013 21:28, "Barry Smith" escribi?: > > Please find attached a new version of the code. ex1.c and ex1f90.F > please run it as is and send me all the output. > > It works correctly with gfortran and the intel 12.x ifort compiler, I > believe it should now work for all fortran compilers. If it does not work > as expected please send me the name and version of your fortran compiler. > > Barry > > On Sep 27, 2013, at 8:22 AM, Christophe Ortiz > wrote: > > > There was a misunderstanding. There are n entries in total in the > vector. I thought it was a vector of n points, with 3 components associated > to each point, ie 3*n entries. This is clear now. > > > > I've tested your example and I checked the values assigned to the > components with your piece of code: > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > do i=1,10 > > xarray(i)%a = i > > xarray(i)%b = 100*i > > xarray(i)%c = 10000*i > > enddo > > > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > > > I did it with: > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > do i = 1 , 10 > > write(*,*) xarray(i)%a,xarray(i)%b,xarray(i)%c > > end do > > > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > I obtained the following: > > > > 1.00000000000000 2.00000000000000 3.00000000000000 > > 2.00000000000000 3.00000000000000 4.00000000000000 > > 3.00000000000000 4.00000000000000 5.00000000000000 > > 4.00000000000000 5.00000000000000 6.00000000000000 > > 5.00000000000000 6.00000000000000 7.00000000000000 > > 6.00000000000000 7.00000000000000 8.00000000000000 > > 7.00000000000000 8.00000000000000 9.00000000000000 > > 8.00000000000000 9.00000000000000 10.0000000000000 > > 9.00000000000000 10.0000000000000 1000.00000000000 > > 10.0000000000000 1000.00000000000 100000.000000000 > > > > First column if ok (1st component), but the other two columns show wrong > numbers. For component 2 and 3, only the very last value is ok. It seems > values from component b are copied from a, and those from component c from > b, with an offset. > > > > What do you think ? > > > > Christophe > > > > > > On Fri, Sep 27, 2013 at 2:47 PM, Barry Smith wrote: > > > > Indices start in fortran arrays from 1 by default so don't even try > zero > > > > I've attached a new version that has three components, > > 30 TOTAL vector entries and 10 for each component. Please run it and > send the results back. > > > > Barry > > > > On Sep 27, 2013, at 5:40 AM, Christophe Ortiz < > christophe.ortiz at ciemat.es> wrote: > > > > > Hi Barry, > > > > > > I made some tests with your code that allows to use struct in > Fortran90. It compiles and does not complain with the struct with 2 > components but...it exhibits some strange behaviour. Here are the > behaviours I observed. To make it simple, I used n=10. > > > > > > - After using call VecSet(x,one,ierr), I checked the values of each > component of the vector using > > > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > > do i = 0 , n > > > write(*,*) i , xarray(i)%a, xarray(i)%b > > > end do > > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > > > I checked the values from i= 0 to n since I was not sure what was the > convention (0- or 1-based). I obtained: > > > > > > 0 0.000000000000000E+000 1.00000000000000 > > > 1 1.00000000000000 1.00000000000000 > > > 2 1.00000000000000 1.00000000000000 > > > 3 1.00000000000000 1.00000000000000 > > > 4 1.00000000000000 1.00000000000000 > > > 5 1.00000000000000 1.00000000000000 > > > 6 1.00000000000000 1.00000000000000 > > > 7 1.00000000000000 1.00000000000000 > > > 8 1.00000000000000 1.00000000000000 > > > 9 1.00000000000000 1.00000000000000 > > > 10 1.00000000000000 1.996650376645798E-314 > > > > > > > > > As you can see, for component a, there is a 0 for i=0, then the values > are correct. For component b, correct values go from i=1 to 9.There is > garbage in second component for i=10 . I checked that if you define a third > component c, its values go from i=-1 to n-2. It's like if values of second > component started at the end of first component, and same thing for third > component. > > > > > > Then, instead of using VecSet(), I tried to assign values directly to > each component of the vector with > > > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > > do i = 0 , n-1 > > > xarray(i)%a = 2.0 > > > xarray(i)%b = 3.0 > > > end do > > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > > > Here I checked that indices actually start at 0 and not at 1. With a > loop from i=1 to n I got a Memory corruption message. > > > > > > and I checked that the values are correctly assigned with > > > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > > do i = 0 , n-1 > > > write(*,*) i, xarray(i)%a, xarray(i)%b > > > end do > > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > > > > > > > > > Then, I obtained: > > > > > > 0 2.00000000000000 2.00000000000000 > > > 1 2.00000000000000 2.00000000000000 > > > 2 2.00000000000000 2.00000000000000 > > > 3 2.00000000000000 2.00000000000000 > > > 4 2.00000000000000 2.00000000000000 > > > 5 2.00000000000000 2.00000000000000 > > > 6 2.00000000000000 2.00000000000000 > > > 7 2.00000000000000 2.00000000000000 > > > 8 2.00000000000000 2.00000000000000 > > > 9 2.00000000000000 3.00000000000000 > > > > > > > > > Here the problem seems more severe. Values are not correctly assigned > to the second component. There should be 3.0 in the second column. It seems > values of the first component are copied to the second one. Only for i=10 > the value of xarray(i)%b is correct (3.0). > > > > > > Any idea where it could come from ? > > > I guess it needs some fixes here and there but I think your idea is > good and that it could work. > > > > > > Christophe > > > > > > > > > > > > On Thu, Sep 26, 2013 at 5:45 PM, Barry Smith > wrote: > > > > > > I just put it with the Fortran source code and compile it with the > rest of the application code; here is the makefile I used > > > > > > > > > > > > > > > On Sep 26, 2013, at 10:14 AM, Christophe Ortiz < > christophe.ortiz at ciemat.es> wrote: > > > > > > > By the way, what should I do with the small .c code ? Where should I > put it ? > > > > > > > > Christophe > > > > > > > > > > > > > > > > On Thu, Sep 26, 2013 at 4:09 PM, Barry Smith > wrote: > > > > > > > > Christophe, > > > > > > > > Despite my LinkIn Endorsement for expertise in Fortran :-) I > cannot pretend to be an expert in FortranXX but I have cooked up an example > demonstrating accessing the Vec entries as if they are in an array of > derived types. I've attached the example code; there needs to be a small C > stub that defines the functions for your specific derived type name. > > > > Note that it will only work I think if your N is a compile time > constant. > > > > > > > > > > > > > > > > > > > > It worked with > > > > ~/Downloads$ gfortran --version > > > > GNU Fortran (GCC) 4.8.1 20130404 (prerelease) > > > > > > > > > > > > I do not understand exactly why it works since it uses > F90Array1dCreate(fa,PETSC_SCALAR,1,len,ptr PETSC_F90_2PTR_PARAM(ptrd)); > which has a single PETSC_SCALAR as a building block but ? I hope it works > for you. If it doesn't, let us know the compiler you are using and we may > be able to get it working for that compiler. > > > > > > > > Barry > > > > > > > > > > > > > > > > > > > > On Sep 26, 2013, at 4:41 AM, Christophe Ortiz < > christophe.ortiz at ciemat.es> wrote: > > > > > > > > > Hi all, > > > > > > > > > > Me again ! > > > > > > > > > > I have read in previous posts that it is hard in Fortran to > declare something similar to a typedef struct in C to manage a > multicomponent problem. > > > > > > > > > > Is it still the case ? Has the problem been solved ? > > > > > > > > > > I am asking because my plan is to implement a multicomponent > problem (in 1D), with many components that will be organized in arrays of > two dimensions. In C I could define > > > > > > > > > > typedef struct{ > > > > > PetscScalar U; > > > > > PetscScalar V; > > > > > PetscScalar A[N][N]; > > > > > } Field; > > > > > > > > > > and then I could calculate the residual function with > > > > > > > > > > F[i].U = ... > > > > > F[i].V = ... > > > > > F[i].A[k][n] = ... > > > > > > > > > > which is quite convenient. > > > > > > > > > > If in Fortran it is not possible to use a struct as in C, I am > afraid I'll have to deal with > > > > > > > > > > F(jdof,i) = ... > > > > > > > > > > where I will have only jdof to address U, V and A[ ][ ], which can > be difficult and not very convenient I guess. Before I move all my code to > C, does anyone have an alternative idea to manage this multi(many)component > problem in Fortran ? > > > > > > > > > > Many thanks in advance for your help and suggestion ! > > > > > > > > > > Christophe > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalcinl at gmail.com Fri Sep 27 16:02:19 2013 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 27 Sep 2013 23:02:19 +0200 Subject: [petsc-users] MatMPIBAIJSetPreallocationCSR i and j indices In-Reply-To: References: Message-ID: On 27 September 2013 21:23, Barry Smith wrote: > Just use MatCreateMPIBAIJWithArrays() saves thinking and work. BTW, Have you ever thought about how hard to use is this API if you want to provide the matrix values? How are users supposed to fill the "values" array for a matrix with bs>1 ? This API is certainly not Fortran-friendly (and I think it not even C-friendly). Mateo had the CSR indices as well as the block values, but I recommended to use the CSR indices to perform proper matrix preallocation, then loop over cells and call MatSetValuesBlocked(). Otherwise, he would need to create an new intermediate array and fill it the right way for MatCreateMPIBAIJWithArrays() to work as expected. The root of the problem is how MatSetValuesBlocked() expects the array of values to be layout in memory. While I understand that the current convention make it compatible with MatSetValues(), I'm always wondering if a layout like values[nblockrows][nblockcols][bs][bs] would not be much more convenient for users. -- Lisandro Dalcin --------------- CIMEC (UNL/CONICET) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1016) Tel/Fax: +54-342-4511169 From bsmith at mcs.anl.gov Fri Sep 27 16:19:30 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 27 Sep 2013 16:19:30 -0500 Subject: [petsc-users] Back to struct in Fortran to represent field with dof > 1 In-Reply-To: References: <3524B3B9-C5FE-46A7-8556-F36DEEC287BD@mcs.anl.gov> Message-ID: A non-text attachment was scrubbed... Name: ex1f.F Type: application/octet-stream Size: 21787 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex1.c Type: application/octet-stream Size: 2276 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: makefile Type: application/octet-stream Size: 13358 bytes Desc: not available URL: -------------- next part -------------- On Sep 27, 2013, at 3:06 PM, Christophe Ortiz wrote: > The files were not attached. Could you please send them again ? > > Thanks. > Christophe > > El 27/09/2013 21:28, "Barry Smith" escribi?: > > Please find attached a new version of the code. ex1.c and ex1f90.F please run it as is and send me all the output. > > It works correctly with gfortran and the intel 12.x ifort compiler, I believe it should now work for all fortran compilers. If it does not work as expected please send me the name and version of your fortran compiler. > > Barry > > On Sep 27, 2013, at 8:22 AM, Christophe Ortiz wrote: > > > There was a misunderstanding. There are n entries in total in the vector. I thought it was a vector of n points, with 3 components associated to each point, ie 3*n entries. This is clear now. > > > > I've tested your example and I checked the values assigned to the components with your piece of code: > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > do i=1,10 > > xarray(i)%a = i > > xarray(i)%b = 100*i > > xarray(i)%c = 10000*i > > enddo > > > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > > > I did it with: > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > do i = 1 , 10 > > write(*,*) xarray(i)%a,xarray(i)%b,xarray(i)%c > > end do > > > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > I obtained the following: > > > > 1.00000000000000 2.00000000000000 3.00000000000000 > > 2.00000000000000 3.00000000000000 4.00000000000000 > > 3.00000000000000 4.00000000000000 5.00000000000000 > > 4.00000000000000 5.00000000000000 6.00000000000000 > > 5.00000000000000 6.00000000000000 7.00000000000000 > > 6.00000000000000 7.00000000000000 8.00000000000000 > > 7.00000000000000 8.00000000000000 9.00000000000000 > > 8.00000000000000 9.00000000000000 10.0000000000000 > > 9.00000000000000 10.0000000000000 1000.00000000000 > > 10.0000000000000 1000.00000000000 100000.000000000 > > > > First column if ok (1st component), but the other two columns show wrong numbers. For component 2 and 3, only the very last value is ok. It seems values from component b are copied from a, and those from component c from b, with an offset. > > > > What do you think ? > > > > Christophe > > > > > > On Fri, Sep 27, 2013 at 2:47 PM, Barry Smith wrote: > > > > Indices start in fortran arrays from 1 by default so don't even try zero > > > > I've attached a new version that has three components, > > 30 TOTAL vector entries and 10 for each component. Please run it and send the results back. > > > > Barry > > > > On Sep 27, 2013, at 5:40 AM, Christophe Ortiz wrote: > > > > > Hi Barry, > > > > > > I made some tests with your code that allows to use struct in Fortran90. It compiles and does not complain with the struct with 2 components but...it exhibits some strange behaviour. Here are the behaviours I observed. To make it simple, I used n=10. > > > > > > - After using call VecSet(x,one,ierr), I checked the values of each component of the vector using > > > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > > do i = 0 , n > > > write(*,*) i , xarray(i)%a, xarray(i)%b > > > end do > > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > > > I checked the values from i= 0 to n since I was not sure what was the convention (0- or 1-based). I obtained: > > > > > > 0 0.000000000000000E+000 1.00000000000000 > > > 1 1.00000000000000 1.00000000000000 > > > 2 1.00000000000000 1.00000000000000 > > > 3 1.00000000000000 1.00000000000000 > > > 4 1.00000000000000 1.00000000000000 > > > 5 1.00000000000000 1.00000000000000 > > > 6 1.00000000000000 1.00000000000000 > > > 7 1.00000000000000 1.00000000000000 > > > 8 1.00000000000000 1.00000000000000 > > > 9 1.00000000000000 1.00000000000000 > > > 10 1.00000000000000 1.996650376645798E-314 > > > > > > > > > As you can see, for component a, there is a 0 for i=0, then the values are correct. For component b, correct values go from i=1 to 9.There is garbage in second component for i=10 . I checked that if you define a third component c, its values go from i=-1 to n-2. It's like if values of second component started at the end of first component, and same thing for third component. > > > > > > Then, instead of using VecSet(), I tried to assign values directly to each component of the vector with > > > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > > do i = 0 , n-1 > > > xarray(i)%a = 2.0 > > > xarray(i)%b = 3.0 > > > end do > > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > > > Here I checked that indices actually start at 0 and not at 1. With a loop from i=1 to n I got a Memory corruption message. > > > > > > and I checked that the values are correctly assigned with > > > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > > do i = 0 , n-1 > > > write(*,*) i, xarray(i)%a, xarray(i)%b > > > end do > > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > > > > > > > > > Then, I obtained: > > > > > > 0 2.00000000000000 2.00000000000000 > > > 1 2.00000000000000 2.00000000000000 > > > 2 2.00000000000000 2.00000000000000 > > > 3 2.00000000000000 2.00000000000000 > > > 4 2.00000000000000 2.00000000000000 > > > 5 2.00000000000000 2.00000000000000 > > > 6 2.00000000000000 2.00000000000000 > > > 7 2.00000000000000 2.00000000000000 > > > 8 2.00000000000000 2.00000000000000 > > > 9 2.00000000000000 3.00000000000000 > > > > > > > > > Here the problem seems more severe. Values are not correctly assigned to the second component. There should be 3.0 in the second column. It seems values of the first component are copied to the second one. Only for i=10 the value of xarray(i)%b is correct (3.0). > > > > > > Any idea where it could come from ? > > > I guess it needs some fixes here and there but I think your idea is good and that it could work. > > > > > > Christophe > > > > > > > > > > > > On Thu, Sep 26, 2013 at 5:45 PM, Barry Smith wrote: > > > > > > I just put it with the Fortran source code and compile it with the rest of the application code; here is the makefile I used > > > > > > > > > > > > > > > On Sep 26, 2013, at 10:14 AM, Christophe Ortiz wrote: > > > > > > > By the way, what should I do with the small .c code ? Where should I put it ? > > > > > > > > Christophe > > > > > > > > > > > > > > > > On Thu, Sep 26, 2013 at 4:09 PM, Barry Smith wrote: > > > > > > > > Christophe, > > > > > > > > Despite my LinkIn Endorsement for expertise in Fortran :-) I cannot pretend to be an expert in FortranXX but I have cooked up an example demonstrating accessing the Vec entries as if they are in an array of derived types. I've attached the example code; there needs to be a small C stub that defines the functions for your specific derived type name. > > > > Note that it will only work I think if your N is a compile time constant. > > > > > > > > > > > > > > > > > > > > It worked with > > > > ~/Downloads$ gfortran --version > > > > GNU Fortran (GCC) 4.8.1 20130404 (prerelease) > > > > > > > > > > > > I do not understand exactly why it works since it uses F90Array1dCreate(fa,PETSC_SCALAR,1,len,ptr PETSC_F90_2PTR_PARAM(ptrd)); which has a single PETSC_SCALAR as a building block but ? I hope it works for you. If it doesn't, let us know the compiler you are using and we may be able to get it working for that compiler. > > > > > > > > Barry > > > > > > > > > > > > > > > > > > > > On Sep 26, 2013, at 4:41 AM, Christophe Ortiz wrote: > > > > > > > > > Hi all, > > > > > > > > > > Me again ! > > > > > > > > > > I have read in previous posts that it is hard in Fortran to declare something similar to a typedef struct in C to manage a multicomponent problem. > > > > > > > > > > Is it still the case ? Has the problem been solved ? > > > > > > > > > > I am asking because my plan is to implement a multicomponent problem (in 1D), with many components that will be organized in arrays of two dimensions. In C I could define > > > > > > > > > > typedef struct{ > > > > > PetscScalar U; > > > > > PetscScalar V; > > > > > PetscScalar A[N][N]; > > > > > } Field; > > > > > > > > > > and then I could calculate the residual function with > > > > > > > > > > F[i].U = ... > > > > > F[i].V = ... > > > > > F[i].A[k][n] = ... > > > > > > > > > > which is quite convenient. > > > > > > > > > > If in Fortran it is not possible to use a struct as in C, I am afraid I'll have to deal with > > > > > > > > > > F(jdof,i) = ... > > > > > > > > > > where I will have only jdof to address U, V and A[ ][ ], which can be difficult and not very convenient I guess. Before I move all my code to C, does anyone have an alternative idea to manage this multi(many)component problem in Fortran ? > > > > > > > > > > Many thanks in advance for your help and suggestion ! > > > > > > > > > > Christophe > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From bsmith at mcs.anl.gov Fri Sep 27 16:34:40 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 27 Sep 2013 16:34:40 -0500 Subject: [petsc-users] MatMPIBAIJSetPreallocationCSR i and j indices In-Reply-To: References: Message-ID: I don't understand the (mild) rant below, the expected format of the values is suppose to be EXACTLY what one puts in the double *array for block CSR format. That is the values are (formally) a 1 dimensional array) containing each block of values (in column oriented storage, that is the first column of the values in the first block, followed by the next column in the first block, etc etc) in the same order as the j indices. Note that this means that in Fortran the values array could be thought of as double A(bs,bs,nnz) Note that this is exactly how Matteo has his stuff organized so "it should just work". If it does not then please submit a bug report with a very small matrix that reproduces the problem. This function is exactly what it says it is, it takes in exactly a matrix in block CSR format. > wondering if a layout like values[nblockrows][nblockcols][bs][bs] > would not be much more convenient for users. I don't understand this. It is a dense matrix as big as the entire sparse matrix. Barry On Sep 27, 2013, at 4:02 PM, Lisandro Dalcin wrote: > On 27 September 2013 21:23, Barry Smith wrote: >> Just use MatCreateMPIBAIJWithArrays() saves thinking and work. > > BTW, Have you ever thought about how hard to use is this API if you > want to provide the matrix values? How are users supposed to fill the > "values" array for a matrix with bs>1 ? This API is certainly not > Fortran-friendly (and I think it not even C-friendly). Mateo had the > CSR indices as well as the block values, but I recommended to use the > CSR indices to perform proper matrix preallocation, then loop over > cells and call MatSetValuesBlocked(). Otherwise, he would need to > create an new intermediate array and fill it the right way for > MatCreateMPIBAIJWithArrays() to work as expected. > > The root of the problem is how MatSetValuesBlocked() expects the array > of values to be layout in memory. While I understand that the current > convention make it compatible with MatSetValues(), I'm always > wondering if a layout like values[nblockrows][nblockcols][bs][bs] > would not be much more convenient for users. > > > > -- > Lisandro Dalcin > --------------- > CIMEC (UNL/CONICET) > Predio CONICET-Santa Fe > Colectora RN 168 Km 472, Paraje El Pozo > 3000 Santa Fe, Argentina > Tel: +54-342-4511594 (ext 1016) > Tel/Fax: +54-342-4511169 From parsani.matteo at gmail.com Fri Sep 27 20:01:14 2013 From: parsani.matteo at gmail.com (Matteo Parsani) Date: Fri, 27 Sep 2013 21:01:14 -0400 Subject: [petsc-users] MatMPIBAIJSetPreallocationCSR i and j indices In-Reply-To: References: Message-ID: Hi Barry, thank you for your feedback. I could be wrong but it seems to me that MatSetValuesBlocked has to be called passing a 1D array, as you said. In fact, From http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages /Mat/MatSetValuesBlocked.html Suppose m=n=2 and block size(bs) = 2 The array is 1 2 | 3 4 5 6 | 7 8 - - - | - - - 9 10 | 11 12 13 14 | 15 16 v[] should be passed in like v[] = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16] If you are not using row oriented storage of v (that is you called MatSetOption (mat,MAT_ROW_ORIENTED,PETSC_FALSE )) then v[] = [1,5,9,13,2,6,10,14,3,7,11,15,4,8,12,16] So, from above I can see that v[] is a 1D array. My problem or question is that my small block are stored in memory as A = [a11, a12 ; a21, a22] B = [b11, b12 ; b21, b22] etc. (My actual blocks are 5x5 as indicated in my first e-mail but here I made them smaller to type less) They are actually 2D arrays, i.e with two indices. Therefore before to pass them to MatSetValuesBlocked I have to concatenate them in row oriented storage or column oriented storage, i.e v = [a11, a12, b11, b12, a21, a22, b21, b22] or v = [a11, a21, a12, a22, b11, b21, b12, b22] For each element in my mesh I construct N [5x5] blocks (N 2D arrays) which, once inserted in the PC matrix, will form the element contribution to the PC matrix. **The 2D arrays are dense matrix and they are the output of the multiplication between a dense matrix and a matrix in CSR which is done internally to my Fortran code. Thus I will loop over all elements owned by a processor, and each time I will construct the N 2D arrays of 1 element and insert them in the PC matrix. I would like to know if I can avoid to concatenate them in row oriented storage or column oriented storage and pass them directly as 2D arrays. Thus, if I am not wrong, it would be nice if the user could have the option to pass to MatSetValuesBlocked each block as: 1- values[bs][bs][1] (e.g. A = [a11, a12 ; a21, a22] ) values[bs][bs][2] (e.g. B = [b11, b12 ; b21, b22] ) plus the two vectors that complete the CSR format, i.e. global block row and column indices (i and j indices). Maybe the last indices that define the blocks (i.e. [1] and [2] in the examples can be avoided) or 2- values[nblockrows][nblockcols][bs][bs] Thanks, On Fri, Sep 27, 2013 at 5:34 PM, Barry Smith wrote: > > I don't understand the (mild) rant below, the expected format of the > values is suppose to be EXACTLY what one puts in the double *array for > block CSR format. That is the values are (formally) a 1 dimensional array) > containing each block of values (in column oriented storage, that is the > first column of the values in the first block, followed by the next column > in the first block, etc etc) in the same order as the j indices. Note > that this means that in Fortran the values array could be thought of as > double A(bs,bs,nnz) Note that this is exactly how Matteo has his stuff > organized so "it should just work". If it does not then please submit a bug > report with a very small matrix that reproduces the problem. > > This function is exactly what it says it is, it takes in exactly a > matrix in block CSR format. > > > wondering if a layout like values[nblockrows][nblockcols][bs][bs] > > would not be much more convenient for users. > > I don't understand this. It is a dense matrix as big as the entire > sparse matrix. > > Barry > > > On Sep 27, 2013, at 4:02 PM, Lisandro Dalcin wrote: > > > On 27 September 2013 21:23, Barry Smith wrote: > >> Just use MatCreateMPIBAIJWithArrays() saves thinking and work. > > > > BTW, Have you ever thought about how hard to use is this API if you > > want to provide the matrix values? How are users supposed to fill the > > "values" array for a matrix with bs>1 ? This API is certainly not > > Fortran-friendly (and I think it not even C-friendly). Mateo had the > > CSR indices as well as the block values, but I recommended to use the > > CSR indices to perform proper matrix preallocation, then loop over > > cells and call MatSetValuesBlocked(). Otherwise, he would need to > > create an new intermediate array and fill it the right way for > > MatCreateMPIBAIJWithArrays() to work as expected. > > > > The root of the problem is how MatSetValuesBlocked() expects the array > > of values to be layout in memory. While I understand that the current > > convention make it compatible with MatSetValues(), I'm always > > wondering if a layout like values[nblockrows][nblockcols][bs][bs] > > would not be much more convenient for users. > > > > > > > > -- > > Lisandro Dalcin > > --------------- > > CIMEC (UNL/CONICET) > > Predio CONICET-Santa Fe > > Colectora RN 168 Km 472, Paraje El Pozo > > 3000 Santa Fe, Argentina > > Tel: +54-342-4511594 (ext 1016) > > Tel/Fax: +54-342-4511169 > > -- Matteo -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Sep 27 21:34:57 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 27 Sep 2013 21:34:57 -0500 Subject: [petsc-users] MatMPIBAIJSetPreallocationCSR i and j indices In-Reply-To: References: Message-ID: <7858A699-38DB-427B-8FD4-A67BD17E8589@mcs.anl.gov> Mattero Multi-dimensional arrays in Fortran are are a sham. That is, a multi-dimensional array in fortran is simply a one dimensional array with a little additional information. If you have a three dimensional array in Fortran that is 1 3 2 4 5 7 6 8 9 11 10 12 where above I have indicated A(:,:,1) A(:,:,2) A(:,:,3) it is in actual memory stored in consecutive memory locations as 1 2 3 4 5 6 7 8 9 10 11 12 this is called column major format because the columns are listed in consecutive memory before rows. When you call MatCreateMPIBAIJWithArrays() it does not expect the values in row-major order, it expects them in column major order per block, which, as I understand it, is exactly what you have already computed. Please just try what I suggest and see what happens. Send all the output if it goes wrong, for a tiny problem. The beautiful thing which I love about programming is no one punishes you for trying something, even it if it is wrong. As I said before it should just work, if it doesn't then let us know EXACTLY what goes wrong and we'll see if there is a bug on our end or a misunderstanding. We are talking too much and not trying enough. Barry On Sep 27, 2013, at 8:01 PM, Matteo Parsani wrote: > Hi Barry, > thank you for your feedback. I could be wrong but it seems to me that MatSetValuesBlocked has to be called passing a 1D array, as you said. > > In fact, From http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetValuesBlocked.html > Suppose m=n=2 and block size(bs > ) = 2 The array is > > 1 2 | 3 4 > > 5 6 | 7 8 > > - - - | - - - > > 9 10 | 11 12 > > 13 14 | 15 16 > > v[] should be passed in like > > v[] = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16] > > If you are not using row oriented storage of v (that is you called MatSetOption(mat,MAT_ROW_ORIENTED,PETSC_FALSE > )) then > > v[] = [1,5,9,13,2,6,10,14,3,7,11,15,4,8,12,16] > > > > So, from above I can see that v[] is a 1D array. My problem or question is that my small block are stored in memory as > > A = [a11, a12 ; a21, a22] > B = [b11, b12 ; b21, b22] > etc. > (My actual blocks are 5x5 as indicated in my first e-mail but here I made them smaller to type less) > > > They are actually 2D arrays, i.e with two indices. Therefore before to pass them to MatSetValuesBlocked I have to concatenate them in row oriented storage or column oriented storage, i.e > > v = [a11, a12, b11, b12, a21, a22, b21, b22] > > or > > v = [a11, a21, a12, a22, b11, b21, b12, b22] > > For each element in my mesh I construct N [5x5] blocks (N 2D arrays) which, once inserted in the PC matrix, will form the element contribution to the PC matrix. The 2D arrays are dense matrix and they are the output of the multiplication between a dense matrix and a matrix in CSR which is done internally to my Fortran code. Thus I will loop over all elements owned by a processor, and each time I will construct the N 2D arrays of 1 element and insert them in the PC matrix. > > I would like to know if I can avoid to concatenate them in row oriented storage or column oriented storage and pass them directly as 2D arrays. > > > Thus, if I am not wrong, it would be nice if the user could have the option to pass to MatSetValuesBlocked each block as: > > 1- values[bs][bs][1] (e.g. A = [a11, a12 ; a21, a22] ) > values[bs][bs][2] (e.g. B = [b11, b12 ; b21, b22] ) > > plus the two vectors that complete the CSR format, i.e. global block row and column indices (i and j indices). Maybe the last indices that define the blocks (i.e. [1] and [2] in the examples can be avoided) > > or > > 2- values[nblockrows][nblockcols][bs][bs] > > Thanks, > > > > On Fri, Sep 27, 2013 at 5:34 PM, Barry Smith wrote: > > I don't understand the (mild) rant below, the expected format of the values is suppose to be EXACTLY what one puts in the double *array for block CSR format. That is the values are (formally) a 1 dimensional array) containing each block of values (in column oriented storage, that is the first column of the values in the first block, followed by the next column in the first block, etc etc) in the same order as the j indices. Note that this means that in Fortran the values array could be thought of as double A(bs,bs,nnz) Note that this is exactly how Matteo has his stuff organized so "it should just work". If it does not then please submit a bug report with a very small matrix that reproduces the problem. > > This function is exactly what it says it is, it takes in exactly a matrix in block CSR format. > > > wondering if a layout like values[nblockrows][nblockcols][bs][bs] > > would not be much more convenient for users. > > I don't understand this. It is a dense matrix as big as the entire sparse matrix. > > Barry > > > On Sep 27, 2013, at 4:02 PM, Lisandro Dalcin wrote: > > > On 27 September 2013 21:23, Barry Smith wrote: > >> Just use MatCreateMPIBAIJWithArrays() saves thinking and work. > > > > BTW, Have you ever thought about how hard to use is this API if you > > want to provide the matrix values? How are users supposed to fill the > > "values" array for a matrix with bs>1 ? This API is certainly not > > Fortran-friendly (and I think it not even C-friendly). Mateo had the > > CSR indices as well as the block values, but I recommended to use the > > CSR indices to perform proper matrix preallocation, then loop over > > cells and call MatSetValuesBlocked(). Otherwise, he would need to > > create an new intermediate array and fill it the right way for > > MatCreateMPIBAIJWithArrays() to work as expected. > > > > The root of the problem is how MatSetValuesBlocked() expects the array > > of values to be layout in memory. While I understand that the current > > convention make it compatible with MatSetValues(), I'm always > > wondering if a layout like values[nblockrows][nblockcols][bs][bs] > > would not be much more convenient for users. > > > > > > > > -- > > Lisandro Dalcin > > --------------- > > CIMEC (UNL/CONICET) > > Predio CONICET-Santa Fe > > Colectora RN 168 Km 472, Paraje El Pozo > > 3000 Santa Fe, Argentina > > Tel: +54-342-4511594 (ext 1016) > > Tel/Fax: +54-342-4511169 > > > > > -- > Matteo From jedbrown at mcs.anl.gov Sat Sep 28 21:12:45 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 28 Sep 2013 21:12:45 -0500 Subject: [petsc-users] MatMPIBAIJSetPreallocationCSR i and j indices In-Reply-To: References: Message-ID: <87had4e5pe.fsf@mcs.anl.gov> Barry Smith writes: > I don't understand the (mild) rant below, the expected format of > the values is suppose to be EXACTLY what one puts in the double > *array for block CSR format. That is the values are (formally) a 1 > dimensional array) containing each block of values (in column > oriented storage, that is the first column of the values in the > first block, followed by the next column in the first block, etc > etc) in the same order as the j indices. Note that this means that > in Fortran the values array could be thought of as double > A(bs,bs,nnz) No, this is not how it works. The ordering is values[nblockrows][bs][nblockcols][bs] or, in Fortran values(bs,nblockcols,bs,nblockrows) where nblockrows and nblockcols are the number of rows and columns being inserted in this all to MatSetValuesBlocked. this layout means that the entries are packed in exactly the same way as if the block indices were blown up to scalar indices and then plain MatSetValues was called. > Note that this is exactly how Matteo has his stuff organized so "it > should just work". If it does not then please submit a bug report > with a very small matrix that reproduces the problem. > > This function is exactly what it says it is, it takes in exactly a matrix in block CSR format. > >> wondering if a layout like values[nblockrows][nblockcols][bs][bs] >> would not be much more convenient for users. > > I don't understand this. It is a dense matrix as big as the entire sparse matrix. Lisandro means the number of block rows and cols being inserted in the call, not the dimension of the matrix. > > Barry > > > On Sep 27, 2013, at 4:02 PM, Lisandro Dalcin wrote: > >> On 27 September 2013 21:23, Barry Smith wrote: >>> Just use MatCreateMPIBAIJWithArrays() saves thinking and work. >> >> BTW, Have you ever thought about how hard to use is this API if you >> want to provide the matrix values? How are users supposed to fill the >> "values" array for a matrix with bs>1 ? This API is certainly not >> Fortran-friendly (and I think it not even C-friendly). Mateo had the >> CSR indices as well as the block values, but I recommended to use the >> CSR indices to perform proper matrix preallocation, then loop over >> cells and call MatSetValuesBlocked(). Otherwise, he would need to >> create an new intermediate array and fill it the right way for >> MatCreateMPIBAIJWithArrays() to work as expected. >> >> The root of the problem is how MatSetValuesBlocked() expects the array >> of values to be layout in memory. While I understand that the current >> convention make it compatible with MatSetValues(), I'm always >> wondering if a layout like values[nblockrows][nblockcols][bs][bs] >> would not be much more convenient for users. >> >> >> >> -- >> Lisandro Dalcin >> --------------- >> CIMEC (UNL/CONICET) >> Predio CONICET-Santa Fe >> Colectora RN 168 Km 472, Paraje El Pozo >> 3000 Santa Fe, Argentina >> Tel: +54-342-4511594 (ext 1016) >> Tel/Fax: +54-342-4511169 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From christophe.ortiz at ciemat.es Sun Sep 29 06:16:44 2013 From: christophe.ortiz at ciemat.es (Christophe Ortiz) Date: Sun, 29 Sep 2013 13:16:44 +0200 Subject: [petsc-users] Back to struct in Fortran to represent field with dof > 1 In-Reply-To: References: <3524B3B9-C5FE-46A7-8556-F36DEEC287BD@mcs.anl.gov> Message-ID: Hi Barry, I just looked at exf1.F. It seems you sent me a wrong file. It does not correspond to ex1f90. There is no vecgetarraymystruct in the file you sent me. Could you please send me the correct version of ex1f90 so I can test it ? Many thanks in advance. Christophe CIEMAT Laboratorio Nacional de Fusi?n por Confinamiento Magn?tico Unidad de Materiales Edificio 2 - Planta 0 - Despacho 28m Avenida Complutense 40, 28040 Madrid, Spain Tel: +34 91496 2582 Fax: +34 91346 6442 -- Q Por favor, piense en el medio ambiente antes de imprimir este mensaje. Please consider the environment before printing this email. 2013/9/27 Barry Smith > > On Sep 27, 2013, at 3:06 PM, Christophe Ortiz > wrote: > > > The files were not attached. Could you please send them again ? > > > > Thanks. > > Christophe > > > > El 27/09/2013 21:28, "Barry Smith" escribi?: > > > > Please find attached a new version of the code. ex1.c and ex1f90.F > please run it as is and send me all the output. > > > > It works correctly with gfortran and the intel 12.x ifort compiler, I > believe it should now work for all fortran compilers. If it does not work > as expected please send me the name and version of your fortran compiler. > > > > Barry > > > > On Sep 27, 2013, at 8:22 AM, Christophe Ortiz < > christophe.ortiz at ciemat.es> wrote: > > > > > There was a misunderstanding. There are n entries in total in the > vector. I thought it was a vector of n points, with 3 components associated > to each point, ie 3*n entries. This is clear now. > > > > > > I've tested your example and I checked the values assigned to the > components with your piece of code: > > > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > > do i=1,10 > > > xarray(i)%a = i > > > xarray(i)%b = 100*i > > > xarray(i)%c = 10000*i > > > enddo > > > > > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > > > > > > I did it with: > > > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > > do i = 1 , 10 > > > write(*,*) xarray(i)%a,xarray(i)%b,xarray(i)%c > > > end do > > > > > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > > > I obtained the following: > > > > > > 1.00000000000000 2.00000000000000 3.00000000000000 > > > 2.00000000000000 3.00000000000000 4.00000000000000 > > > 3.00000000000000 4.00000000000000 5.00000000000000 > > > 4.00000000000000 5.00000000000000 6.00000000000000 > > > 5.00000000000000 6.00000000000000 7.00000000000000 > > > 6.00000000000000 7.00000000000000 8.00000000000000 > > > 7.00000000000000 8.00000000000000 9.00000000000000 > > > 8.00000000000000 9.00000000000000 10.0000000000000 > > > 9.00000000000000 10.0000000000000 1000.00000000000 > > > 10.0000000000000 1000.00000000000 100000.000000000 > > > > > > First column if ok (1st component), but the other two columns show > wrong numbers. For component 2 and 3, only the very last value is ok. It > seems values from component b are copied from a, and those from component c > from b, with an offset. > > > > > > What do you think ? > > > > > > Christophe > > > > > > > > > On Fri, Sep 27, 2013 at 2:47 PM, Barry Smith > wrote: > > > > > > Indices start in fortran arrays from 1 by default so don't even try > zero > > > > > > I've attached a new version that has three components, > > > 30 TOTAL vector entries and 10 for each component. Please run it and > send the results back. > > > > > > Barry > > > > > > On Sep 27, 2013, at 5:40 AM, Christophe Ortiz < > christophe.ortiz at ciemat.es> wrote: > > > > > > > Hi Barry, > > > > > > > > I made some tests with your code that allows to use struct in > Fortran90. It compiles and does not complain with the struct with 2 > components but...it exhibits some strange behaviour. Here are the > behaviours I observed. To make it simple, I used n=10. > > > > > > > > - After using call VecSet(x,one,ierr), I checked the values of each > component of the vector using > > > > > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > > > do i = 0 , n > > > > write(*,*) i , xarray(i)%a, xarray(i)%b > > > > end do > > > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > > > > > I checked the values from i= 0 to n since I was not sure what was > the convention (0- or 1-based). I obtained: > > > > > > > > 0 0.000000000000000E+000 1.00000000000000 > > > > 1 1.00000000000000 1.00000000000000 > > > > 2 1.00000000000000 1.00000000000000 > > > > 3 1.00000000000000 1.00000000000000 > > > > 4 1.00000000000000 1.00000000000000 > > > > 5 1.00000000000000 1.00000000000000 > > > > 6 1.00000000000000 1.00000000000000 > > > > 7 1.00000000000000 1.00000000000000 > > > > 8 1.00000000000000 1.00000000000000 > > > > 9 1.00000000000000 1.00000000000000 > > > > 10 1.00000000000000 1.996650376645798E-314 > > > > > > > > > > > > As you can see, for component a, there is a 0 for i=0, then the > values are correct. For component b, correct values go from i=1 to 9.There > is garbage in second component for i=10 . I checked that if you define a > third component c, its values go from i=-1 to n-2. It's like if values of > second component started at the end of first component, and same thing for > third component. > > > > > > > > Then, instead of using VecSet(), I tried to assign values directly > to each component of the vector with > > > > > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > > > do i = 0 , n-1 > > > > xarray(i)%a = 2.0 > > > > xarray(i)%b = 3.0 > > > > end do > > > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > > > > > Here I checked that indices actually start at 0 and not at 1. With a > loop from i=1 to n I got a Memory corruption message. > > > > > > > > and I checked that the values are correctly assigned with > > > > > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > > > do i = 0 , n-1 > > > > write(*,*) i, xarray(i)%a, xarray(i)%b > > > > end do > > > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > > > > > > > > > > > > > Then, I obtained: > > > > > > > > 0 2.00000000000000 2.00000000000000 > > > > 1 2.00000000000000 2.00000000000000 > > > > 2 2.00000000000000 2.00000000000000 > > > > 3 2.00000000000000 2.00000000000000 > > > > 4 2.00000000000000 2.00000000000000 > > > > 5 2.00000000000000 2.00000000000000 > > > > 6 2.00000000000000 2.00000000000000 > > > > 7 2.00000000000000 2.00000000000000 > > > > 8 2.00000000000000 2.00000000000000 > > > > 9 2.00000000000000 3.00000000000000 > > > > > > > > > > > > Here the problem seems more severe. Values are not correctly > assigned to the second component. There should be 3.0 in the second column. > It seems values of the first component are copied to the second one. Only > for i=10 the value of xarray(i)%b is correct (3.0). > > > > > > > > Any idea where it could come from ? > > > > I guess it needs some fixes here and there but I think your idea is > good and that it could work. > > > > > > > > Christophe > > > > > > > > > > > > > > > > On Thu, Sep 26, 2013 at 5:45 PM, Barry Smith > wrote: > > > > > > > > I just put it with the Fortran source code and compile it with > the rest of the application code; here is the makefile I used > > > > > > > > > > > > > > > > > > > > On Sep 26, 2013, at 10:14 AM, Christophe Ortiz < > christophe.ortiz at ciemat.es> wrote: > > > > > > > > > By the way, what should I do with the small .c code ? Where should > I put it ? > > > > > > > > > > Christophe > > > > > > > > > > > > > > > > > > > > On Thu, Sep 26, 2013 at 4:09 PM, Barry Smith > wrote: > > > > > > > > > > Christophe, > > > > > > > > > > Despite my LinkIn Endorsement for expertise in Fortran :-) I > cannot pretend to be an expert in FortranXX but I have cooked up an example > demonstrating accessing the Vec entries as if they are in an array of > derived types. I've attached the example code; there needs to be a small C > stub that defines the functions for your specific derived type name. > > > > > Note that it will only work I think if your N is a compile time > constant. > > > > > > > > > > > > > > > > > > > > > > > > > It worked with > > > > > ~/Downloads$ gfortran --version > > > > > GNU Fortran (GCC) 4.8.1 20130404 (prerelease) > > > > > > > > > > > > > > > I do not understand exactly why it works since it uses > F90Array1dCreate(fa,PETSC_SCALAR,1,len,ptr PETSC_F90_2PTR_PARAM(ptrd)); > which has a single PETSC_SCALAR as a building block but ? I hope it works > for you. If it doesn't, let us know the compiler you are using and we may > be able to get it working for that compiler. > > > > > > > > > > Barry > > > > > > > > > > > > > > > > > > > > > > > > > On Sep 26, 2013, at 4:41 AM, Christophe Ortiz < > christophe.ortiz at ciemat.es> wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > Me again ! > > > > > > > > > > > > I have read in previous posts that it is hard in Fortran to > declare something similar to a typedef struct in C to manage a > multicomponent problem. > > > > > > > > > > > > Is it still the case ? Has the problem been solved ? > > > > > > > > > > > > I am asking because my plan is to implement a multicomponent > problem (in 1D), with many components that will be organized in arrays of > two dimensions. In C I could define > > > > > > > > > > > > typedef struct{ > > > > > > PetscScalar U; > > > > > > PetscScalar V; > > > > > > PetscScalar A[N][N]; > > > > > > } Field; > > > > > > > > > > > > and then I could calculate the residual function with > > > > > > > > > > > > F[i].U = ... > > > > > > F[i].V = ... > > > > > > F[i].A[k][n] = ... > > > > > > > > > > > > which is quite convenient. > > > > > > > > > > > > If in Fortran it is not possible to use a struct as in C, I am > afraid I'll have to deal with > > > > > > > > > > > > F(jdof,i) = ... > > > > > > > > > > > > where I will have only jdof to address U, V and A[ ][ ], which > can be difficult and not very convenient I guess. Before I move all my code > to C, does anyone have an alternative idea to manage this > multi(many)component problem in Fortran ? > > > > > > > > > > > > Many thanks in advance for your help and suggestion ! > > > > > > > > > > > > Christophe > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Sep 29 10:31:00 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 29 Sep 2013 10:31:00 -0500 Subject: [petsc-users] MatMPIBAIJSetPreallocationCSR i and j indices In-Reply-To: <87had4e5pe.fsf@mcs.anl.gov> References: <87had4e5pe.fsf@mcs.anl.gov> Message-ID: <8405066E-DD41-4D9E-9D96-1826F094D4A4@mcs.anl.gov> I forgot that MatSetValuesBlocked() takes row oriented input even within the blocks. So if we simply call MatSetOption(mat,MAT_ROW_ORIENTED,PETSC_FALSE) on the matrix after it is created then the storage should work as I originally planned. Barry I found this discussion values(bs,nblockcols,bs,nblockrows) very confusing. MatMPIBAIJSetPreallocationCSR() doesn't take a square block of values so I didn't see what you are talking about. You must be talking about each internal call to MatSetValuesBlocked_MPIBAIJ(B,1,&row,ncols,icols,svals,INSERT_VALUES) where it puts in one block row of values with each call. On Sep 28, 2013, at 9:12 PM, Jed Brown wrote: > Barry Smith writes: > >> I don't understand the (mild) rant below, the expected format of >> the values is suppose to be EXACTLY what one puts in the double >> *array for block CSR format. That is the values are (formally) a 1 >> dimensional array) containing each block of values (in column >> oriented storage, that is the first column of the values in the >> first block, followed by the next column in the first block, etc >> etc) in the same order as the j indices. Note that this means that >> in Fortran the values array could be thought of as double >> A(bs,bs,nnz) > > No, this is not how it works. The ordering is > > values[nblockrows][bs][nblockcols][bs] > > or, in Fortran > > values(bs,nblockcols,bs,nblockrows) > > where nblockrows and nblockcols are the number of rows and columns being > inserted in this all to MatSetValuesBlocked. > > this layout means that the entries are packed in exactly the same way as > if the block indices were blown up to scalar indices and then plain > MatSetValues was called. > >> Note that this is exactly how Matteo has his stuff organized so "it >> should just work". If it does not then please submit a bug report >> with a very small matrix that reproduces the problem. >> >> This function is exactly what it says it is, it takes in exactly a matrix in block CSR format. >> >>> wondering if a layout like values[nblockrows][nblockcols][bs][bs] >>> would not be much more convenient for users. >> >> I don't understand this. It is a dense matrix as big as the entire sparse matrix. > > Lisandro means the number of block rows and cols being inserted in the > call, not the dimension of the matrix. > >> >> Barry >> >> >> On Sep 27, 2013, at 4:02 PM, Lisandro Dalcin wrote: >> >>> On 27 September 2013 21:23, Barry Smith wrote: >>>> Just use MatCreateMPIBAIJWithArrays() saves thinking and work. >>> >>> BTW, Have you ever thought about how hard to use is this API if you >>> want to provide the matrix values? How are users supposed to fill the >>> "values" array for a matrix with bs>1 ? This API is certainly not >>> Fortran-friendly (and I think it not even C-friendly). Mateo had the >>> CSR indices as well as the block values, but I recommended to use the >>> CSR indices to perform proper matrix preallocation, then loop over >>> cells and call MatSetValuesBlocked(). Otherwise, he would need to >>> create an new intermediate array and fill it the right way for >>> MatCreateMPIBAIJWithArrays() to work as expected. >>> >>> The root of the problem is how MatSetValuesBlocked() expects the array >>> of values to be layout in memory. While I understand that the current >>> convention make it compatible with MatSetValues(), I'm always >>> wondering if a layout like values[nblockrows][nblockcols][bs][bs] >>> would not be much more convenient for users. >>> >>> >>> >>> -- >>> Lisandro Dalcin >>> --------------- >>> CIMEC (UNL/CONICET) >>> Predio CONICET-Santa Fe >>> Colectora RN 168 Km 472, Paraje El Pozo >>> 3000 Santa Fe, Argentina >>> Tel: +54-342-4511594 (ext 1016) >>> Tel/Fax: +54-342-4511169 From bsmith at mcs.anl.gov Sun Sep 29 10:33:37 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 29 Sep 2013 10:33:37 -0500 Subject: [petsc-users] Back to struct in Fortran to represent field with dof > 1 In-Reply-To: References: <3524B3B9-C5FE-46A7-8556-F36DEEC287BD@mcs.anl.gov> Message-ID: <33F8252E-1F7C-487E-9EFC-83CC60852471@mcs.anl.gov> A non-text attachment was scrubbed... Name: ex1f90.F Type: application/octet-stream Size: 4049 bytes Desc: not available URL: -------------- next part -------------- Sorry, that is what I get for having a ex1.F and ex1F90.F in the same directory. Barry On Sep 29, 2013, at 6:16 AM, Christophe Ortiz wrote: > Hi Barry, > > I just looked at exf1.F. It seems you sent me a wrong file. It does not correspond to ex1f90. There is no vecgetarraymystruct in the file you sent me. Could you please send me the correct version of ex1f90 so I can test it ? Many thanks in advance. > > Christophe > > CIEMAT > Laboratorio Nacional de Fusi?n por Confinamiento Magn?tico > Unidad de Materiales > Edificio 2 - Planta 0 - Despacho 28m > Avenida Complutense 40, > 28040 Madrid, Spain > Tel: +34 91496 2582 > Fax: +34 91346 6442 > > -- > Q > Por favor, piense en el medio ambiente antes de imprimir este mensaje. > Please consider the environment before printing this email. > > > 2013/9/27 Barry Smith > > On Sep 27, 2013, at 3:06 PM, Christophe Ortiz wrote: > > > The files were not attached. Could you please send them again ? > > > > Thanks. > > Christophe > > > > El 27/09/2013 21:28, "Barry Smith" escribi?: > > > > Please find attached a new version of the code. ex1.c and ex1f90.F please run it as is and send me all the output. > > > > It works correctly with gfortran and the intel 12.x ifort compiler, I believe it should now work for all fortran compilers. If it does not work as expected please send me the name and version of your fortran compiler. > > > > Barry > > > > On Sep 27, 2013, at 8:22 AM, Christophe Ortiz wrote: > > > > > There was a misunderstanding. There are n entries in total in the vector. I thought it was a vector of n points, with 3 components associated to each point, ie 3*n entries. This is clear now. > > > > > > I've tested your example and I checked the values assigned to the components with your piece of code: > > > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > > do i=1,10 > > > xarray(i)%a = i > > > xarray(i)%b = 100*i > > > xarray(i)%c = 10000*i > > > enddo > > > > > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > > > > > > I did it with: > > > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > > do i = 1 , 10 > > > write(*,*) xarray(i)%a,xarray(i)%b,xarray(i)%c > > > end do > > > > > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > > > I obtained the following: > > > > > > 1.00000000000000 2.00000000000000 3.00000000000000 > > > 2.00000000000000 3.00000000000000 4.00000000000000 > > > 3.00000000000000 4.00000000000000 5.00000000000000 > > > 4.00000000000000 5.00000000000000 6.00000000000000 > > > 5.00000000000000 6.00000000000000 7.00000000000000 > > > 6.00000000000000 7.00000000000000 8.00000000000000 > > > 7.00000000000000 8.00000000000000 9.00000000000000 > > > 8.00000000000000 9.00000000000000 10.0000000000000 > > > 9.00000000000000 10.0000000000000 1000.00000000000 > > > 10.0000000000000 1000.00000000000 100000.000000000 > > > > > > First column if ok (1st component), but the other two columns show wrong numbers. For component 2 and 3, only the very last value is ok. It seems values from component b are copied from a, and those from component c from b, with an offset. > > > > > > What do you think ? > > > > > > Christophe > > > > > > > > > On Fri, Sep 27, 2013 at 2:47 PM, Barry Smith wrote: > > > > > > Indices start in fortran arrays from 1 by default so don't even try zero > > > > > > I've attached a new version that has three components, > > > 30 TOTAL vector entries and 10 for each component. Please run it and send the results back. > > > > > > Barry > > > > > > On Sep 27, 2013, at 5:40 AM, Christophe Ortiz wrote: > > > > > > > Hi Barry, > > > > > > > > I made some tests with your code that allows to use struct in Fortran90. It compiles and does not complain with the struct with 2 components but...it exhibits some strange behaviour. Here are the behaviours I observed. To make it simple, I used n=10. > > > > > > > > - After using call VecSet(x,one,ierr), I checked the values of each component of the vector using > > > > > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > > > do i = 0 , n > > > > write(*,*) i , xarray(i)%a, xarray(i)%b > > > > end do > > > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > > > > > I checked the values from i= 0 to n since I was not sure what was the convention (0- or 1-based). I obtained: > > > > > > > > 0 0.000000000000000E+000 1.00000000000000 > > > > 1 1.00000000000000 1.00000000000000 > > > > 2 1.00000000000000 1.00000000000000 > > > > 3 1.00000000000000 1.00000000000000 > > > > 4 1.00000000000000 1.00000000000000 > > > > 5 1.00000000000000 1.00000000000000 > > > > 6 1.00000000000000 1.00000000000000 > > > > 7 1.00000000000000 1.00000000000000 > > > > 8 1.00000000000000 1.00000000000000 > > > > 9 1.00000000000000 1.00000000000000 > > > > 10 1.00000000000000 1.996650376645798E-314 > > > > > > > > > > > > As you can see, for component a, there is a 0 for i=0, then the values are correct. For component b, correct values go from i=1 to 9.There is garbage in second component for i=10 . I checked that if you define a third component c, its values go from i=-1 to n-2. It's like if values of second component started at the end of first component, and same thing for third component. > > > > > > > > Then, instead of using VecSet(), I tried to assign values directly to each component of the vector with > > > > > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > > > do i = 0 , n-1 > > > > xarray(i)%a = 2.0 > > > > xarray(i)%b = 3.0 > > > > end do > > > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > > > > > Here I checked that indices actually start at 0 and not at 1. With a loop from i=1 to n I got a Memory corruption message. > > > > > > > > and I checked that the values are correctly assigned with > > > > > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > > > do i = 0 , n-1 > > > > write(*,*) i, xarray(i)%a, xarray(i)%b > > > > end do > > > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > > > > > > > > > > > > > Then, I obtained: > > > > > > > > 0 2.00000000000000 2.00000000000000 > > > > 1 2.00000000000000 2.00000000000000 > > > > 2 2.00000000000000 2.00000000000000 > > > > 3 2.00000000000000 2.00000000000000 > > > > 4 2.00000000000000 2.00000000000000 > > > > 5 2.00000000000000 2.00000000000000 > > > > 6 2.00000000000000 2.00000000000000 > > > > 7 2.00000000000000 2.00000000000000 > > > > 8 2.00000000000000 2.00000000000000 > > > > 9 2.00000000000000 3.00000000000000 > > > > > > > > > > > > Here the problem seems more severe. Values are not correctly assigned to the second component. There should be 3.0 in the second column. It seems values of the first component are copied to the second one. Only for i=10 the value of xarray(i)%b is correct (3.0). > > > > > > > > Any idea where it could come from ? > > > > I guess it needs some fixes here and there but I think your idea is good and that it could work. > > > > > > > > Christophe > > > > > > > > > > > > > > > > On Thu, Sep 26, 2013 at 5:45 PM, Barry Smith wrote: > > > > > > > > I just put it with the Fortran source code and compile it with the rest of the application code; here is the makefile I used > > > > > > > > > > > > > > > > > > > > On Sep 26, 2013, at 10:14 AM, Christophe Ortiz wrote: > > > > > > > > > By the way, what should I do with the small .c code ? Where should I put it ? > > > > > > > > > > Christophe > > > > > > > > > > > > > > > > > > > > On Thu, Sep 26, 2013 at 4:09 PM, Barry Smith wrote: > > > > > > > > > > Christophe, > > > > > > > > > > Despite my LinkIn Endorsement for expertise in Fortran :-) I cannot pretend to be an expert in FortranXX but I have cooked up an example demonstrating accessing the Vec entries as if they are in an array of derived types. I've attached the example code; there needs to be a small C stub that defines the functions for your specific derived type name. > > > > > Note that it will only work I think if your N is a compile time constant. > > > > > > > > > > > > > > > > > > > > > > > > > It worked with > > > > > ~/Downloads$ gfortran --version > > > > > GNU Fortran (GCC) 4.8.1 20130404 (prerelease) > > > > > > > > > > > > > > > I do not understand exactly why it works since it uses F90Array1dCreate(fa,PETSC_SCALAR,1,len,ptr PETSC_F90_2PTR_PARAM(ptrd)); which has a single PETSC_SCALAR as a building block but ? I hope it works for you. If it doesn't, let us know the compiler you are using and we may be able to get it working for that compiler. > > > > > > > > > > Barry > > > > > > > > > > > > > > > > > > > > > > > > > On Sep 26, 2013, at 4:41 AM, Christophe Ortiz wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > Me again ! > > > > > > > > > > > > I have read in previous posts that it is hard in Fortran to declare something similar to a typedef struct in C to manage a multicomponent problem. > > > > > > > > > > > > Is it still the case ? Has the problem been solved ? > > > > > > > > > > > > I am asking because my plan is to implement a multicomponent problem (in 1D), with many components that will be organized in arrays of two dimensions. In C I could define > > > > > > > > > > > > typedef struct{ > > > > > > PetscScalar U; > > > > > > PetscScalar V; > > > > > > PetscScalar A[N][N]; > > > > > > } Field; > > > > > > > > > > > > and then I could calculate the residual function with > > > > > > > > > > > > F[i].U = ... > > > > > > F[i].V = ... > > > > > > F[i].A[k][n] = ... > > > > > > > > > > > > which is quite convenient. > > > > > > > > > > > > If in Fortran it is not possible to use a struct as in C, I am afraid I'll have to deal with > > > > > > > > > > > > F(jdof,i) = ... > > > > > > > > > > > > where I will have only jdof to address U, V and A[ ][ ], which can be difficult and not very convenient I guess. Before I move all my code to C, does anyone have an alternative idea to manage this multi(many)component problem in Fortran ? > > > > > > > > > > > > Many thanks in advance for your help and suggestion ! > > > > > > > > > > > > Christophe > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From dalcinl at gmail.com Sun Sep 29 11:22:44 2013 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sun, 29 Sep 2013 18:22:44 +0200 Subject: [petsc-users] MatMPIBAIJSetPreallocationCSR i and j indices In-Reply-To: <8405066E-DD41-4D9E-9D96-1826F094D4A4@mcs.anl.gov> References: <87had4e5pe.fsf@mcs.anl.gov> <8405066E-DD41-4D9E-9D96-1826F094D4A4@mcs.anl.gov> Message-ID: On 29 September 2013 17:31, Barry Smith wrote: > > I forgot that MatSetValuesBlocked() takes row oriented input even within the blocks. So if we simply call MatSetOption(mat,MAT_ROW_ORIENTED,PETSC_FALSE) on the matrix after it is created then the storage should work as I originally planned. > > Barry > > I found this discussion values(bs,nblockcols,bs,nblockrows) very confusing. MatMPIBAIJSetPreallocationCSR() doesn't take a square block of values so I didn't see what you are talking about. You must be talking about each internal call to MatSetValuesBlocked_MPIBAIJ(B,1,&row,ncols,icols,svals,INSERT_VALUES) where it puts in one block row of values with each call. > Yes, that's what I was talking about. And yes, if you have column-oriented block values [e.g, in fotran you use an array VALUES(bs,bs,nnz) ] and use MatSetOption(), you are fine and MatMPIBAIJSetPreallocationCSR() will do the right thing. Otherwise (e.g, you are in C and use VALUES[nnz][bs][bs]), feeding MatMPIBAIJSetPreallocationCSR() with these values will not work. I guess we have to live with that. -- Lisandro Dalcin --------------- CIMEC (UNL/CONICET) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1016) Tel/Fax: +54-342-4511169 From jedbrown at mcs.anl.gov Sun Sep 29 11:56:46 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 29 Sep 2013 11:56:46 -0500 Subject: [petsc-users] MatMPIBAIJSetPreallocationCSR i and j indices In-Reply-To: <8405066E-DD41-4D9E-9D96-1826F094D4A4@mcs.anl.gov> References: <87had4e5pe.fsf@mcs.anl.gov> <8405066E-DD41-4D9E-9D96-1826F094D4A4@mcs.anl.gov> Message-ID: <87fvsnd0s1.fsf@mcs.anl.gov> Barry Smith writes: > I found this discussion values(bs,nblockcols,bs,nblockrows) very > confusing. MatMPIBAIJSetPreallocationCSR() doesn't take a square block > of values so I didn't see what you are talking about. You must be > talking about each internal call to > MatSetValuesBlocked_MPIBAIJ(B,1,&row,ncols,icols,svals,INSERT_VALUES) > where it puts in one block row of values with each call. Yeah, or inserting an element matrix where you have several block rows. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From bsmith at mcs.anl.gov Sun Sep 29 12:05:30 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 29 Sep 2013 12:05:30 -0500 Subject: [petsc-users] MatMPIBAIJSetPreallocationCSR i and j indices In-Reply-To: <87fvsnd0s1.fsf@mcs.anl.gov> References: <87had4e5pe.fsf@mcs.anl.gov> <8405066E-DD41-4D9E-9D96-1826F094D4A4@mcs.anl.gov> <87fvsnd0s1.fsf@mcs.anl.gov> Message-ID: <36DCFD46-D654-40F2-B763-6A7AB7470111@mcs.anl.gov> On Sep 29, 2013, at 11:56 AM, Jed Brown wrote: > Barry Smith writes: >> I found this discussion values(bs,nblockcols,bs,nblockrows) very >> confusing. MatMPIBAIJSetPreallocationCSR() doesn't take a square block >> of values so I didn't see what you are talking about. You must be >> talking about each internal call to >> MatSetValuesBlocked_MPIBAIJ(B,1,&row,ncols,icols,svals,INSERT_VALUES) >> where it puts in one block row of values with each call. > > Yeah, or inserting an element matrix where you have several block rows. But that has absolutely nothing to do with MatMPIBAIJSetPreallocationCSR(). From jedbrown at mcs.anl.gov Sun Sep 29 12:09:32 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 29 Sep 2013 12:09:32 -0500 Subject: [petsc-users] MatMPIBAIJSetPreallocationCSR i and j indices In-Reply-To: <36DCFD46-D654-40F2-B763-6A7AB7470111@mcs.anl.gov> References: <87had4e5pe.fsf@mcs.anl.gov> <8405066E-DD41-4D9E-9D96-1826F094D4A4@mcs.anl.gov> <87fvsnd0s1.fsf@mcs.anl.gov> <36DCFD46-D654-40F2-B763-6A7AB7470111@mcs.anl.gov> Message-ID: <87a9ivd06r.fsf@mcs.anl.gov> Barry Smith writes: >> Yeah, or inserting an element matrix where you have several block rows. > > But that has absolutely nothing to do with MatMPIBAIJSetPreallocationCSR(). The second paragraph of Lisandro's message was specifically about MatSetValuesBlocked, and that is what I was responding to. I don't use MatMPIBAIJSetPreallocationCSR, but it has a different layout from MatSetValuesBlocked, so you can't easily switch between them. (Of course I just recommend calling MatXAIJSetPreallocation and then MatSetValuesBlocked.) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From bsmith at mcs.anl.gov Sun Sep 29 12:30:51 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 29 Sep 2013 12:30:51 -0500 Subject: [petsc-users] MatMPIBAIJSetPreallocationCSR i and j indices In-Reply-To: References: <87had4e5pe.fsf@mcs.anl.gov> <8405066E-DD41-4D9E-9D96-1826F094D4A4@mcs.anl.gov> Message-ID: <6B15898A-6086-4437-90B4-C84E3AD4746F@mcs.anl.gov> Fixed and accidentally pushed directly to maint at bitbucket On Sep 29, 2013, at 11:22 AM, Lisandro Dalcin wrote: > On 29 September 2013 17:31, Barry Smith wrote: >> >> I forgot that MatSetValuesBlocked() takes row oriented input even within the blocks. So if we simply call MatSetOption(mat,MAT_ROW_ORIENTED,PETSC_FALSE) on the matrix after it is created then the storage should work as I originally planned. >> >> Barry >> >> I found this discussion values(bs,nblockcols,bs,nblockrows) very confusing. MatMPIBAIJSetPreallocationCSR() doesn't take a square block of values so I didn't see what you are talking about. You must be talking about each internal call to MatSetValuesBlocked_MPIBAIJ(B,1,&row,ncols,icols,svals,INSERT_VALUES) where it puts in one block row of values with each call. >> > > Yes, that's what I was talking about. And yes, if you have > column-oriented block values [e.g, in fotran you use an array > VALUES(bs,bs,nnz) ] and use MatSetOption(), you are fine and > MatMPIBAIJSetPreallocationCSR() will do the right thing. Otherwise > (e.g, you are in C and use VALUES[nnz][bs][bs]), feeding > MatMPIBAIJSetPreallocationCSR() with these values will not work. I > guess we have to live with that. > > > -- > Lisandro Dalcin > --------------- > CIMEC (UNL/CONICET) > Predio CONICET-Santa Fe > Colectora RN 168 Km 472, Paraje El Pozo > 3000 Santa Fe, Argentina > Tel: +54-342-4511594 (ext 1016) > Tel/Fax: +54-342-4511169 From jedbrown at mcs.anl.gov Sun Sep 29 12:41:12 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 29 Sep 2013 12:41:12 -0500 Subject: [petsc-users] MatMPIBAIJSetPreallocationCSR i and j indices In-Reply-To: <6B15898A-6086-4437-90B4-C84E3AD4746F@mcs.anl.gov> References: <87had4e5pe.fsf@mcs.anl.gov> <8405066E-DD41-4D9E-9D96-1826F094D4A4@mcs.anl.gov> <6B15898A-6086-4437-90B4-C84E3AD4746F@mcs.anl.gov> Message-ID: <877gdzcypz.fsf@mcs.anl.gov> Barry Smith writes: > Fixed and accidentally pushed directly to maint at bitbucket Should warn that this function permanently changes MAT_ROW_ORIENTED and thus will affect subsequent calls to MatSetValues. Or (better, in my opinion) restore the original value of that option before returning. On the Reported-by tags, could you do them in a structured way as Reported-by: Matteo Parsani Reported-by: Lisandro Dalcin instead of this Reported by Matteo Parsani and Lisandro Dalcin so that it is easier to extract statistics at the end of a release cycle, maybe to explicitly name people that have frequently reported bugs and ways to improve the library. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From jedbrown at mcs.anl.gov Sun Sep 29 13:13:22 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 29 Sep 2013 13:13:22 -0500 Subject: [petsc-users] MatMPIBAIJSetPreallocationCSR i and j indices In-Reply-To: <877gdzcypz.fsf@mcs.anl.gov> References: <87had4e5pe.fsf@mcs.anl.gov> <8405066E-DD41-4D9E-9D96-1826F094D4A4@mcs.anl.gov> <6B15898A-6086-4437-90B4-C84E3AD4746F@mcs.anl.gov> <877gdzcypz.fsf@mcs.anl.gov> Message-ID: <8738oncx8d.fsf@mcs.anl.gov> Jed Brown writes: > Barry Smith writes: > >> Fixed and accidentally pushed directly to maint at bitbucket > > Should warn that this function permanently changes MAT_ROW_ORIENTED and > thus will affect subsequent calls to MatSetValues. Or (better, in my > opinion) restore the original value of that option before returning. On second thought, why don't we just document that MatMPIBAIJSetPreallocationCSR respects MAT_ROW_ORIENTED within each block and let the user choose which they want to use? -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From bsmith at mcs.anl.gov Sun Sep 29 14:45:18 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 29 Sep 2013 14:45:18 -0500 Subject: [petsc-users] MatMPIBAIJSetPreallocationCSR i and j indices In-Reply-To: <8738oncx8d.fsf@mcs.anl.gov> References: <87had4e5pe.fsf@mcs.anl.gov> <8405066E-DD41-4D9E-9D96-1826F094D4A4@mcs.anl.gov> <6B15898A-6086-4437-90B4-C84E3AD4746F@mcs.anl.gov> <877gdzcypz.fsf@mcs.anl.gov> <8738oncx8d.fsf@mcs.anl.gov> Message-ID: <2756C588-A8CE-415E-8D69-C3D100EBC53B@mcs.anl.gov> We can do all this, but how much effort do we really want to make for supporting these inconvenient "convenience routines"? BTW: We should really label these function with Level: Recommended only for legacy applications being quickly ported to PETSc I really only want people who already have a working non-PETSc code that uses CSR directly already to use these routines so they can quickly try out PETSc, I never want someone writing fresh code or doing a large refactorization to EVER use these routines. > On second thought, why don't we just document that > MatMPIBAIJSetPreallocationCSR respects MAT_ROW_ORIENTED within each > block and let the user choose which they want to use? Also there is no way to get this flag into MatCreateMPIBAIJWithArrays() Barry On Sep 29, 2013, at 1:13 PM, Jed Brown wrote: > Jed Brown writes: > >> Barry Smith writes: >> >>> Fixed and accidentally pushed directly to maint at bitbucket >> >> Should warn that this function permanently changes MAT_ROW_ORIENTED and >> thus will affect subsequent calls to MatSetValues. Or (better, in my >> opinion) restore the original value of that option before returning. > > On second thought, why don't we just document that > MatMPIBAIJSetPreallocationCSR respects MAT_ROW_ORIENTED within each > block and let the user choose which they want to use? From jedbrown at mcs.anl.gov Sun Sep 29 14:52:12 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 29 Sep 2013 14:52:12 -0500 Subject: [petsc-users] MatMPIBAIJSetPreallocationCSR i and j indices In-Reply-To: <2756C588-A8CE-415E-8D69-C3D100EBC53B@mcs.anl.gov> References: <87had4e5pe.fsf@mcs.anl.gov> <8405066E-DD41-4D9E-9D96-1826F094D4A4@mcs.anl.gov> <6B15898A-6086-4437-90B4-C84E3AD4746F@mcs.anl.gov> <877gdzcypz.fsf@mcs.anl.gov> <8738oncx8d.fsf@mcs.anl.gov> <2756C588-A8CE-415E-8D69-C3D100EBC53B@mcs.anl.gov> Message-ID: <87ob7bbe37.fsf@mcs.anl.gov> Barry Smith writes: > We can do all this, but how much effort do we really want to make > for supporting these inconvenient "convenience routines"? Not much, but this is a tiny amount of code. > BTW: We should really label these function with > > Level: Recommended only for legacy applications being quickly ported to PETSc > > I really only want people who already have a working non-PETSc > code that uses CSR directly already to use these routines so they > can quickly try out PETSc, I never want someone writing fresh code > or doing a large refactorization to EVER use these routines. Agreed. >> On second thought, why don't we just document that >> MatMPIBAIJSetPreallocationCSR respects MAT_ROW_ORIENTED within each >> block and let the user choose which they want to use? > > Also there is no way to get this flag into MatCreateMPIBAIJWithArrays() The user calls MatSetOption before making that call and changes it afterward if they want. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From bsmith at mcs.anl.gov Sun Sep 29 16:14:19 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 29 Sep 2013 16:14:19 -0500 Subject: [petsc-users] MatMPIBAIJSetPreallocationCSR i and j indices In-Reply-To: <87ob7bbe37.fsf@mcs.anl.gov> References: <87had4e5pe.fsf@mcs.anl.gov> <8405066E-DD41-4D9E-9D96-1826F094D4A4@mcs.anl.gov> <6B15898A-6086-4437-90B4-C84E3AD4746F@mcs.anl.gov> <877gdzcypz.fsf@mcs.anl.gov> <8738oncx8d.fsf@mcs.anl.gov> <2756C588-A8CE-415E-8D69-C3D100EBC53B@mcs.anl.gov> <87ob7bbe37.fsf@mcs.anl.gov> Message-ID: <1B9C04B6-A5F1-4B2E-A731-BBBEE7FCD954@mcs.anl.gov> On Sep 29, 2013, at 2:52 PM, Jed Brown wrote: > Barry Smith writes: >> >> Also there is no way to get this flag into MatCreateMPIBAIJWithArrays() > > The user calls MatSetOption before making that call and changes it > afterward if they want. Huh? I can't do MatSetOption(?, MAT_ROW_ORIENTED,something) MatCreateMPIBAIJWithArrays(??, &mat); because mat doesn't exist before the MatCreate? () call. Now obviously they could call the several routines instead of MatCreateMPIBAIJWithArrays() but then we still need to decide what format MatCreateMPIBAIJWithArrays() should use since the user cannot choose from several. I conclude the best thing to do is to just use the default column oriented as the only possibility. If, by chance, the user built their block CSR by row (in the block) they could easily change that around at the same time they "ported" to PETSc, this change would presumably be trivial in their code and matches our mantra that PETSc will provide some tools for porting legacy CSR based codes to PETSc but won't make it a rich flexible development environment for working with legacy CSR codes. Barry From jedbrown at mcs.anl.gov Sun Sep 29 16:20:24 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 29 Sep 2013 16:20:24 -0500 Subject: [petsc-users] MatMPIBAIJSetPreallocationCSR i and j indices In-Reply-To: <1B9C04B6-A5F1-4B2E-A731-BBBEE7FCD954@mcs.anl.gov> References: <87had4e5pe.fsf@mcs.anl.gov> <8405066E-DD41-4D9E-9D96-1826F094D4A4@mcs.anl.gov> <6B15898A-6086-4437-90B4-C84E3AD4746F@mcs.anl.gov> <877gdzcypz.fsf@mcs.anl.gov> <8738oncx8d.fsf@mcs.anl.gov> <2756C588-A8CE-415E-8D69-C3D100EBC53B@mcs.anl.gov> <87ob7bbe37.fsf@mcs.anl.gov> <1B9C04B6-A5F1-4B2E-A731-BBBEE7FCD954@mcs.anl.gov> Message-ID: <87d2nrba07.fsf@mcs.anl.gov> Barry Smith writes: > On Sep 29, 2013, at 2:52 PM, Jed Brown wrote: > >> Barry Smith writes: >>> >>> Also there is no way to get this flag into MatCreateMPIBAIJWithArrays() >> >> The user calls MatSetOption before making that call and changes it >> afterward if they want. > > Huh? I can't do > > MatSetOption(?, MAT_ROW_ORIENTED,something) > MatCreateMPIBAIJWithArrays(??, &mat); > > because mat doesn't exist before the MatCreate? () call. Now > obviously they could call the several routines instead of > MatCreateMPIBAIJWithArrays() Exactly. > but then we still need to decide what format > MatCreateMPIBAIJWithArrays() should use since the user cannot > choose from several. I conclude the best thing to do is to just use > the default column oriented as the only possibility. Agreed, but I would do that by calling MatSetOption(A,MAT_ROW_ORIENTED,PETSC_FALSE) inside MatCreateMPIBAIJWithArrays and restoring it afterward, so that MatMPIBAIJSetPreallocationCSR does not change that option. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From bsmith at mcs.anl.gov Sun Sep 29 17:13:52 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 29 Sep 2013 17:13:52 -0500 Subject: [petsc-users] MatMPIBAIJSetPreallocationCSR i and j indices In-Reply-To: <87d2nrba07.fsf@mcs.anl.gov> References: <87had4e5pe.fsf@mcs.anl.gov> <8405066E-DD41-4D9E-9D96-1826F094D4A4@mcs.anl.gov> <6B15898A-6086-4437-90B4-C84E3AD4746F@mcs.anl.gov> <877gdzcypz.fsf@mcs.anl.gov> <8738oncx8d.fsf@mcs.anl.gov> <2756C588-A8CE-415E-8D69-C3D100EBC53B@mcs.anl.gov> <87ob7bbe37.fsf@mcs.anl.gov> <1B9C04B6-A5F1-4B2E-A731-BBBEE7FCD954@mcs.anl.gov> <87d2nrba07.fsf@mcs.anl.gov> Message-ID: <32D95A3D-D3A7-4D7D-B058-34AA24089689@mcs.anl.gov> On Sep 29, 2013, at 4:20 PM, Jed Brown wrote: > Barry Smith writes: > > >> but then we still need to decide what format >> MatCreateMPIBAIJWithArrays() should use since the user cannot >> choose from several. I conclude the best thing to do is to just use >> the default column oriented as the only possibility. > > Agreed, but I would do that by calling > MatSetOption(A,MAT_ROW_ORIENTED,PETSC_FALSE) inside > MatCreateMPIBAIJWithArrays and restoring it afterward, so that > MatMPIBAIJSetPreallocationCSR does not change that option. I am fine with this. In fact I would go ahead and make the change in the branch I already made exactly for this work. Sadly, git allowed me to fuck up the branch so I cannot easily work on that branch since it empty, if only git could have warned me I was fucking up?.. Barry I actually got to the stage of pushing my branch to the repository, then I went to bitbucket.org/petsc/petsc to make the pull request of my branch to next (where you could have made all these suggestions in an organized way) when I realized the branch wasn't in the repository because I forgot to check it out after making it. If this sad story doesn't make you realize git having optional enforceable policies and warnings are a good thing then I guess nothing will and we'll continue to limp along with constant mistakes, not caught by simple tests, slowing down our productivity. From parsani.matteo at gmail.com Sun Sep 29 17:22:47 2013 From: parsani.matteo at gmail.com (Matteo Parsani) Date: Sun, 29 Sep 2013 18:22:47 -0400 Subject: [petsc-users] MatMPIBAIJSetPreallocationCSR i and j indices In-Reply-To: <87ob7bbe37.fsf@mcs.anl.gov> References: <87had4e5pe.fsf@mcs.anl.gov> <8405066E-DD41-4D9E-9D96-1826F094D4A4@mcs.anl.gov> <6B15898A-6086-4437-90B4-C84E3AD4746F@mcs.anl.gov> <877gdzcypz.fsf@mcs.anl.gov> <8738oncx8d.fsf@mcs.anl.gov> <2756C588-A8CE-415E-8D69-C3D100EBC53B@mcs.anl.gov> <87ob7bbe37.fsf@mcs.anl.gov> Message-ID: Hi Barry and Jed, just a curiosity and maybe a good point to learn something: 1- why are those* inconvenient *"convenience routines" ? 2 - why those routines *should not be used* when writing a fresh code or a large re-factoring? Thanks in advance. On Sun, Sep 29, 2013 at 3:52 PM, Jed Brown wrote: > Barry Smith writes: > > > We can do all this, but how much effort do we really want to make > > for supporting these inconvenient "convenience routines"? > > Not much, but this is a tiny amount of code. > > > BTW: We should really label these function with > > > > Level: Recommended only for legacy applications being quickly ported > to PETSc > > > > I really only want people who already have a working non-PETSc > > code that uses CSR directly already to use these routines so they > > can quickly try out PETSc, I never want someone writing fresh code > > or doing a large refactorization to EVER use these routines. > > Agreed. > > >> On second thought, why don't we just document that > >> MatMPIBAIJSetPreallocationCSR respects MAT_ROW_ORIENTED within each > >> block and let the user choose which they want to use? > > > > Also there is no way to get this flag into > MatCreateMPIBAIJWithArrays() > > The user calls MatSetOption before making that call and changes it > afterward if they want. > -- Matteo -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sun Sep 29 18:01:43 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 29 Sep 2013 18:01:43 -0500 Subject: [petsc-users] MatMPIBAIJSetPreallocationCSR i and j indices In-Reply-To: <32D95A3D-D3A7-4D7D-B058-34AA24089689@mcs.anl.gov> References: <87had4e5pe.fsf@mcs.anl.gov> <8405066E-DD41-4D9E-9D96-1826F094D4A4@mcs.anl.gov> <6B15898A-6086-4437-90B4-C84E3AD4746F@mcs.anl.gov> <877gdzcypz.fsf@mcs.anl.gov> <8738oncx8d.fsf@mcs.anl.gov> <2756C588-A8CE-415E-8D69-C3D100EBC53B@mcs.anl.gov> <87ob7bbe37.fsf@mcs.anl.gov> <1B9C04B6-A5F1-4B2E-A731-BBBEE7FCD954@mcs.anl.gov> <87d2nrba07.fsf@mcs.anl.gov> <32D95A3D-D3A7-4D7D-B058-34AA24089689@mcs.anl.gov> Message-ID: <874n93b5bc.fsf@mcs.anl.gov> Barry Smith writes: > I am fine with this. In fact I would go ahead and make the change > in the branch I already made exactly for this work. Sadly, git > allowed me to fuck up the branch so I cannot easily work on that > branch since it empty, if only git could have warned me I was > fucking up?.. You can still create a new branch and we can merge later so that it looks like your current commit had been in the branch all along, provided we merge it to 'maint' before merging something else there. (maint) $ git checkout -b barry/new-branch ... make changes, commit ... discuss, merge to 'next', decide branch is good $ git checkout maint (maint) $ git reset --hard HEAD^ # move local 'maint' back to before this commit (maint) $ git merge --no-ff barry/new-branch The original commit is still an ancestor of the merge (just not via first-parent) so the resulting push will still fast-forward. Hold off on the above if you're not confident about what the steps are doing. > I actually got to the stage of pushing my branch to the repository, > then I went to bitbucket.org/petsc/petsc to make the pull request of > my branch to next (where you could have made all these suggestions in > an organized way) when I realized the branch wasn't in the repository > because I forgot to check it out after making it. You can reset the branch marker at that time. I would have done (maint) $ git branch barry/the-branch # create branch, but forget to check it out ... edit ... (maint) $ git commit -am'commit intended for barry/the-branch, but made in maint' (maint) $ git push -u origin barry/the-branch ... look on bitbucket and notice barry/the-branch is at an old commit ... (maint) $ git branch -f barry/the-branch HEAD # move barry/the-branch to where I am now (maint) $ git reset --hard HEAD^ # move (unpushed) 'maint' back before this commit $ git push origin barry/the-branch > If this sad story doesn't make you realize git having optional > enforceable policies and warnings are a good thing then I guess > nothing will and we'll continue to limp along with constant mistakes, > not caught by simple tests, slowing down our productivity. I'm just not convinced the tests are simple, and maintaining buggy and incomplete validators sounds more confusing than using the tools right the first time. If you know how to do it reliably and simply enough that the failure modes are reasonable, then let's write these things. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From bsmith at mcs.anl.gov Sun Sep 29 18:35:47 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 29 Sep 2013 18:35:47 -0500 Subject: [petsc-users] MatMPIBAIJSetPreallocationCSR i and j indices In-Reply-To: References: <87had4e5pe.fsf@mcs.anl.gov> <8405066E-DD41-4D9E-9D96-1826F094D4A4@mcs.anl.gov> <6B15898A-6086-4437-90B4-C84E3AD4746F@mcs.anl.gov> <877gdzcypz.fsf@mcs.anl.gov> <8738oncx8d.fsf@mcs.anl.gov> <2756C588-A8CE-415E-8D69-C3D100EBC53B@mcs.anl.gov> <87ob7bbe37.fsf@mcs.anl.gov> Message-ID: Matteo, This is 20+ years of PETSc history, it centers around the imperfect*, yet extremely important, principle of "data encapsulation". In our case, to first order, this means that scientific simulation codes that need solvers should NOT have to know about the specific storage formats formats of the sparse matrices and should not be written around any particular storage format. The compressed sparse row (CSR) and block CSR formats are specific storage formats, useful for some operations (like matrix vector product) but cumbersome for other operations (like inserting a value into the matrix). If you are calling MatCreateMPIBAIJWithArrays() or MatMPIBAIJSetPreallocationCSR() this means that YOUR application code is directly storing the sparse matrix in a particular data format; hence no data encapsulation. The only reason the routines MatMPIBAIJSetPreallocationCSR() and friends and relations exist in PETSc is because some potential users begged us to add them to PETSc so they could take an old code that already built the matrix in CSR format and use the PETSc solvers without having to rewrite a bunch of their code. Being nice guys (except to each other) and desperate for users :-) we provided these convenience routines. We don't want people who are writing new codes, or seriously modifying old codes, to use these routines because we don't think that users should be building matrices directly in (block) CSR, or any other specific, format. Rather we think they should use one of the many MatSetValues friends and relations to pass the matrix values to PETSc, or doing matrix-free etc. Partly this ideological, but it is also practical, I claim building parallel matrices from, say finite element methods, with CSR directly is painful and not needed since PETSc can handle the "merging" of the element stiffness matrices into the "global matrix" for you, we already wrote the code, why should each person write the code themselves? In addition, if another format is best for implementation (say in multicore or on GPUs) by abstracting away the specific storage from the user code, they can switch to the new format without changing their code. If you are going to write this CSR code, why not write your own GMRES, your own Newton's method, your own ODE integrator? We believe in numerical libraries to gather up all the code that yes, in theory, each person can write themselves, but write it only once in the best way we know how and continue to improve it based on input from all new users. This allows people to make far more scientific simulation progress instead of spending a lot of time reinventing the wheel. Now since PETSc is a library, you can choose to use whatever part you want and to write the other parts yourself, so you can do your own assembly to block CSR yourself if you want, it is just something we don't think would be the most productive use of your time. Barry * No computer science abstraction/idea is perfect, but some are useful. On Sep 29, 2013, at 5:22 PM, Matteo Parsani wrote: > Hi Barry and Jed, > just a curiosity and maybe a good point to learn something: > > 1- why are those inconvenient "convenience routines" ? > > 2 - why those routines should not be used when writing a fresh code or a large re-factoring? > > > Thanks in advance. > > > > On Sun, Sep 29, 2013 at 3:52 PM, Jed Brown wrote: > Barry Smith writes: > > > We can do all this, but how much effort do we really want to make > > for supporting these inconvenient "convenience routines"? > > Not much, but this is a tiny amount of code. > > > BTW: We should really label these function with > > > > Level: Recommended only for legacy applications being quickly ported to PETSc > > > > I really only want people who already have a working non-PETSc > > code that uses CSR directly already to use these routines so they > > can quickly try out PETSc, I never want someone writing fresh code > > or doing a large refactorization to EVER use these routines. > > Agreed. > > >> On second thought, why don't we just document that > >> MatMPIBAIJSetPreallocationCSR respects MAT_ROW_ORIENTED within each > >> block and let the user choose which they want to use? > > > > Also there is no way to get this flag into MatCreateMPIBAIJWithArrays() > > The user calls MatSetOption before making that call and changes it > afterward if they want. > > > > -- > Matteo From jedbrown at mcs.anl.gov Sun Sep 29 18:46:13 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 29 Sep 2013 18:46:13 -0500 Subject: [petsc-users] MatMPIBAIJSetPreallocationCSR i and j indices In-Reply-To: References: <87had4e5pe.fsf@mcs.anl.gov> <8405066E-DD41-4D9E-9D96-1826F094D4A4@mcs.anl.gov> <6B15898A-6086-4437-90B4-C84E3AD4746F@mcs.anl.gov> <877gdzcypz.fsf@mcs.anl.gov> <8738oncx8d.fsf@mcs.anl.gov> <2756C588-A8CE-415E-8D69-C3D100EBC53B@mcs.anl.gov> <87ob7bbe37.fsf@mcs.anl.gov> Message-ID: <87y56f9ooq.fsf@mcs.anl.gov> Matteo Parsani writes: > 2 - why those routines *should not be used* when writing a fresh code or a > large re-factoring? Barry wrote a great rationale, but consider an example: almost all third-party linear algebra libraries (and some special features in PETSc) require scalar (non-blocked) matrix formats, but PETSc has good implementations of other operations like matrix multiplication and incomplete factorization using BAIJ format. Thus with some solution methods, you want a BAIJ matrix and with other methods, you need an AIJ matrix. If you try to build the backend data format directly, you'll either need multiple versions to specifically support each format or there will need to be a copy, which doubles the peak memory usage required for the matrix. If you just use MatXAIJSetPreallocation and MatSetValuesBlocked, then you can choose the matrix format at run-time, in which case all solvers become available without the extra memory usage. There are many more examples. It is bad software design for a component to depend on more than it must. Depend on generic interfaces instead of internal formats. It will make your software more powerful and more maintainable. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From bsmith at mcs.anl.gov Sun Sep 29 18:47:20 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 29 Sep 2013 18:47:20 -0500 Subject: [petsc-users] MatMPIBAIJSetPreallocationCSR i and j indices In-Reply-To: <874n93b5bc.fsf@mcs.anl.gov> References: <87had4e5pe.fsf@mcs.anl.gov> <8405066E-DD41-4D9E-9D96-1826F094D4A4@mcs.anl.gov> <6B15898A-6086-4437-90B4-C84E3AD4746F@mcs.anl.gov> <877gdzcypz.fsf@mcs.anl.gov> <8738oncx8d.fsf@mcs.anl.gov> <2756C588-A8CE-415E-8D69-C3D100EBC53B@mcs.anl.gov> <87ob7bbe37.fsf@mcs.anl.gov> <1B9C04B6-A5F1-4B2E-A731-BBBEE7FCD954@mcs.anl.gov> <87d2nrba07.fsf@mcs.anl.gov> <32D95A3D-D3A7-4D7D-B058-34AA24089689@mcs.anl.gov> <874n93b5bc.fsf@mcs.anl.gov> Message-ID: <264FC64D-0E90-42E1-874F-9A420ED83B58@mcs.anl.gov> Looks like adding tests for checking out and committing is easy but sadly it doesn't look like there is a pre-branch hook? http://git-scm.com/book/en/Customizing-Git-Git-Hooks Also doesn't look like there is a trivial way to have both "my" additional hooks that are specialized to me, plus a bunch of hooks that should be always run for everyone using the repository. Well, I'll play around with the hooks and see if they are useful for what I think is needed. Barry On Sep 29, 2013, at 6:01 PM, Jed Brown wrote: > Barry Smith writes: > > > I'm just not convinced the tests are simple, and maintaining buggy and > incomplete validators sounds more confusing than using the tools right > the first time. If you know how to do it reliably and simply enough > that the failure modes are reasonable, then let's write these things. From jedbrown at mcs.anl.gov Sun Sep 29 21:31:25 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 29 Sep 2013 21:31:25 -0500 Subject: [petsc-users] MatMPIBAIJSetPreallocationCSR i and j indices In-Reply-To: <874n93b5bc.fsf@mcs.anl.gov> References: <87had4e5pe.fsf@mcs.anl.gov> <8405066E-DD41-4D9E-9D96-1826F094D4A4@mcs.anl.gov> <6B15898A-6086-4437-90B4-C84E3AD4746F@mcs.anl.gov> <877gdzcypz.fsf@mcs.anl.gov> <8738oncx8d.fsf@mcs.anl.gov> <2756C588-A8CE-415E-8D69-C3D100EBC53B@mcs.anl.gov> <87ob7bbe37.fsf@mcs.anl.gov> <1B9C04B6-A5F1-4B2E-A731-BBBEE7FCD954@mcs.anl.gov> <87d2nrba07.fsf@mcs.anl.gov> <32D95A3D-D3A7-4D7D-B058-34AA24089689@mcs.anl.gov> <874n93b5bc.fsf@mcs.anl.gov> Message-ID: <87li2f9h1e.fsf@mcs.anl.gov> Jed Brown writes: > Barry Smith writes: > >> I am fine with this. In fact I would go ahead and make the change >> in the branch I already made exactly for this work. Sadly, git >> allowed me to fuck up the branch so I cannot easily work on that >> branch since it empty, if only git could have warned me I was >> fucking up?.. > > You can still create a new branch and we can merge later so that it > looks like your current commit had been in the branch all along, > provided we merge it to 'maint' before merging something else there. On second thought, I think this should be reverted in 'maint' because it would break existing code that calls the *SetPreallocationCSR functions. Interface "improvements" belong in 'master' where they can be documented in release notes and people know that some things might change. I'm preparing a branch that respects MAT_ROW_ORIENTED and keeps MatSeqBAIJSetPreallocationCSR consistent. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From bsmith at mcs.anl.gov Sun Sep 29 21:50:29 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 29 Sep 2013 21:50:29 -0500 Subject: [petsc-users] MatMPIBAIJSetPreallocationCSR i and j indices In-Reply-To: <87li2f9h1e.fsf@mcs.anl.gov> References: <87had4e5pe.fsf@mcs.anl.gov> <8405066E-DD41-4D9E-9D96-1826F094D4A4@mcs.anl.gov> <6B15898A-6086-4437-90B4-C84E3AD4746F@mcs.anl.gov> <877gdzcypz.fsf@mcs.anl.gov> <8738oncx8d.fsf@mcs.anl.gov> <2756C588-A8CE-415E-8D69-C3D100EBC53B@mcs.anl.gov> <87ob7bbe37.fsf@mcs.anl.gov> <1B9C04B6-A5F1-4B2E-A731-BBBEE7FCD954@mcs.anl.gov> <87d2nrba07.fsf@mcs.anl.gov> <32D95A3D-D3A7-4D7D-B058-34AA24089689@mcs.anl.gov> <874n93b5bc.fsf@mcs.anl.gov> <87li2f9h1e.fsf@mcs.anl.gov> Message-ID: On Sep 29, 2013, at 9:31 PM, Jed Brown wrote: > Jed Brown writes: > >> Barry Smith writes: >> >>> I am fine with this. In fact I would go ahead and make the change >>> in the branch I already made exactly for this work. Sadly, git >>> allowed me to fuck up the branch so I cannot easily work on that >>> branch since it empty, if only git could have warned me I was >>> fucking up?.. >> >> You can still create a new branch and we can merge later so that it >> looks like your current commit had been in the branch all along, >> provided we merge it to 'maint' before merging something else there. > > On second thought, I think this should be reverted in 'maint' because it > would break existing code that calls the *SetPreallocationCSR functions. > Interface "improvements" belong in 'master' where they can be documented > in release notes and people know that some things might change. It actually isn't an improvement, it is a bug fix; the current code was broken (though I am not sure if it was always broken). But I am fine with removing my "fix". > > I'm preparing a branch that respects MAT_ROW_ORIENTED and keeps > MatSeqBAIJSetPreallocationCSR consistent. That's fine. From jedbrown at mcs.anl.gov Sun Sep 29 22:07:59 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 29 Sep 2013 22:07:59 -0500 Subject: [petsc-users] MatMPIBAIJSetPreallocationCSR i and j indices In-Reply-To: References: <87had4e5pe.fsf@mcs.anl.gov> <8405066E-DD41-4D9E-9D96-1826F094D4A4@mcs.anl.gov> <6B15898A-6086-4437-90B4-C84E3AD4746F@mcs.anl.gov> <877gdzcypz.fsf@mcs.anl.gov> <8738oncx8d.fsf@mcs.anl.gov> <2756C588-A8CE-415E-8D69-C3D100EBC53B@mcs.anl.gov> <87ob7bbe37.fsf@mcs.anl.gov> <1B9C04B6-A5F1-4B2E-A731-BBBEE7FCD954@mcs.anl.gov> <87d2nrba07.fsf@mcs.anl.gov> <32D95A3D-D3A7-4D7D-B058-34AA24089689@mcs.anl.gov> <874n93b5bc.fsf@mcs.anl.gov> <87li2f9h1e.fsf@mcs.anl.gov> Message-ID: <87fvsn9fcg.fsf@mcs.anl.gov> Barry Smith writes: > It actually isn't an improvement, it is a bug fix; the current code > was broken (though I am not sure if it was always broken). But I am > fine with removing my "fix". "broken" in the sense of "doesn't work" or broken in the sense that it's confusing and doesn't match the (indirect) documentation? I don't think that behavior should change in 'maint' unless we can be confident that nobody was using the function, in which case it wouldn't be urgent to fix in 'maint'. (And if after all these years, _nobody_ uses it, shouldn't we remove it?) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From christophe.ortiz at ciemat.es Mon Sep 30 03:01:20 2013 From: christophe.ortiz at ciemat.es (Christophe Ortiz) Date: Mon, 30 Sep 2013 10:01:20 +0200 Subject: [petsc-users] Back to struct in Fortran to represent field with dof > 1 In-Reply-To: <33F8252E-1F7C-487E-9EFC-83CC60852471@mcs.anl.gov> References: <3524B3B9-C5FE-46A7-8556-F36DEEC287BD@mcs.anl.gov> <33F8252E-1F7C-487E-9EFC-83CC60852471@mcs.anl.gov> Message-ID: Hi Barry, It works ! Great job ! Here is the output: Vector Object: 1 MPI processes type: seq 1 100 10000 2 200 20000 3 300 30000 4 400 40000 5 500 50000 6 600 60000 7 700 70000 8 800 80000 9 900 90000 10 1000 100000 1.00000000000000 100.000000000000 10000.0000000000 2.00000000000000 200.000000000000 20000.0000000000 3.00000000000000 300.000000000000 30000.0000000000 4.00000000000000 400.000000000000 40000.0000000000 5.00000000000000 500.000000000000 50000.0000000000 6.00000000000000 600.000000000000 60000.0000000000 7.00000000000000 700.000000000000 70000.0000000000 8.00000000000000 800.000000000000 80000.0000000000 9.00000000000000 900.000000000000 90000.0000000000 10.0000000000000 1000.00000000000 100000.000000000 Components have the right values. Do you think it can be adapted to work with DMDAVecGetArrayF90(da,...) / DMDAVecRestoreArrayF90(da,...) in order to work with distributed arrays ? Best regards, Christophe 2013/9/29 Barry Smith > > > Sorry, that is what I get for having a ex1.F and ex1F90.F in the same > directory. > > Barry > > On Sep 29, 2013, at 6:16 AM, Christophe Ortiz > wrote: > > > Hi Barry, > > > > I just looked at exf1.F. It seems you sent me a wrong file. It does not > correspond to ex1f90. There is no vecgetarraymystruct in the file you sent > me. Could you please send me the correct version of ex1f90 so I can test it > ? Many thanks in advance. > > > > Christophe > > > > CIEMAT > > Laboratorio Nacional de Fusi?n por Confinamiento Magn?tico > > Unidad de Materiales > > Edificio 2 - Planta 0 - Despacho 28m > > Avenida Complutense 40, > > 28040 Madrid, Spain > > Tel: +34 91496 2582 > > Fax: +34 91346 6442 > > > > -- > > Q > > Por favor, piense en el medio ambiente antes de imprimir este mensaje. > > Please consider the environment before printing this email. > > > > > > 2013/9/27 Barry Smith > > > > On Sep 27, 2013, at 3:06 PM, Christophe Ortiz < > christophe.ortiz at ciemat.es> wrote: > > > > > The files were not attached. Could you please send them again ? > > > > > > Thanks. > > > Christophe > > > > > > El 27/09/2013 21:28, "Barry Smith" escribi?: > > > > > > Please find attached a new version of the code. ex1.c and ex1f90.F > please run it as is and send me all the output. > > > > > > It works correctly with gfortran and the intel 12.x ifort compiler, > I believe it should now work for all fortran compilers. If it does not work > as expected please send me the name and version of your fortran compiler. > > > > > > Barry > > > > > > On Sep 27, 2013, at 8:22 AM, Christophe Ortiz < > christophe.ortiz at ciemat.es> wrote: > > > > > > > There was a misunderstanding. There are n entries in total in the > vector. I thought it was a vector of n points, with 3 components associated > to each point, ie 3*n entries. This is clear now. > > > > > > > > I've tested your example and I checked the values assigned to the > components with your piece of code: > > > > > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > > > do i=1,10 > > > > xarray(i)%a = i > > > > xarray(i)%b = 100*i > > > > xarray(i)%c = 10000*i > > > > enddo > > > > > > > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > > > > > > > > > I did it with: > > > > > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > > > do i = 1 , 10 > > > > write(*,*) xarray(i)%a,xarray(i)%b,xarray(i)%c > > > > end do > > > > > > > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > > > > > I obtained the following: > > > > > > > > 1.00000000000000 2.00000000000000 3.00000000000000 > > > > 2.00000000000000 3.00000000000000 4.00000000000000 > > > > 3.00000000000000 4.00000000000000 5.00000000000000 > > > > 4.00000000000000 5.00000000000000 6.00000000000000 > > > > 5.00000000000000 6.00000000000000 7.00000000000000 > > > > 6.00000000000000 7.00000000000000 8.00000000000000 > > > > 7.00000000000000 8.00000000000000 9.00000000000000 > > > > 8.00000000000000 9.00000000000000 10.0000000000000 > > > > 9.00000000000000 10.0000000000000 1000.00000000000 > > > > 10.0000000000000 1000.00000000000 100000.000000000 > > > > > > > > First column if ok (1st component), but the other two columns show > wrong numbers. For component 2 and 3, only the very last value is ok. It > seems values from component b are copied from a, and those from component c > from b, with an offset. > > > > > > > > What do you think ? > > > > > > > > Christophe > > > > > > > > > > > > On Fri, Sep 27, 2013 at 2:47 PM, Barry Smith > wrote: > > > > > > > > Indices start in fortran arrays from 1 by default so don't even > try zero > > > > > > > > I've attached a new version that has three components, > > > > 30 TOTAL vector entries and 10 for each component. Please run it > and send the results back. > > > > > > > > Barry > > > > > > > > On Sep 27, 2013, at 5:40 AM, Christophe Ortiz < > christophe.ortiz at ciemat.es> wrote: > > > > > > > > > Hi Barry, > > > > > > > > > > I made some tests with your code that allows to use struct in > Fortran90. It compiles and does not complain with the struct with 2 > components but...it exhibits some strange behaviour. Here are the > behaviours I observed. To make it simple, I used n=10. > > > > > > > > > > - After using call VecSet(x,one,ierr), I checked the values of > each component of the vector using > > > > > > > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > > > > do i = 0 , n > > > > > write(*,*) i , xarray(i)%a, xarray(i)%b > > > > > end do > > > > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > > > > > > > I checked the values from i= 0 to n since I was not sure what was > the convention (0- or 1-based). I obtained: > > > > > > > > > > 0 0.000000000000000E+000 1.00000000000000 > > > > > 1 1.00000000000000 1.00000000000000 > > > > > 2 1.00000000000000 1.00000000000000 > > > > > 3 1.00000000000000 1.00000000000000 > > > > > 4 1.00000000000000 1.00000000000000 > > > > > 5 1.00000000000000 1.00000000000000 > > > > > 6 1.00000000000000 1.00000000000000 > > > > > 7 1.00000000000000 1.00000000000000 > > > > > 8 1.00000000000000 1.00000000000000 > > > > > 9 1.00000000000000 1.00000000000000 > > > > > 10 1.00000000000000 1.996650376645798E-314 > > > > > > > > > > > > > > > As you can see, for component a, there is a 0 for i=0, then the > values are correct. For component b, correct values go from i=1 to 9.There > is garbage in second component for i=10 . I checked that if you define a > third component c, its values go from i=-1 to n-2. It's like if values of > second component started at the end of first component, and same thing for > third component. > > > > > > > > > > Then, instead of using VecSet(), I tried to assign values directly > to each component of the vector with > > > > > > > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > > > > do i = 0 , n-1 > > > > > xarray(i)%a = 2.0 > > > > > xarray(i)%b = 3.0 > > > > > end do > > > > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > > > > > > > Here I checked that indices actually start at 0 and not at 1. With > a loop from i=1 to n I got a Memory corruption message. > > > > > > > > > > and I checked that the values are correctly assigned with > > > > > > > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > > > > do i = 0 , n-1 > > > > > write(*,*) i, xarray(i)%a, xarray(i)%b > > > > > end do > > > > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > > > > > > > > > > > > > > > > > Then, I obtained: > > > > > > > > > > 0 2.00000000000000 2.00000000000000 > > > > > 1 2.00000000000000 2.00000000000000 > > > > > 2 2.00000000000000 2.00000000000000 > > > > > 3 2.00000000000000 2.00000000000000 > > > > > 4 2.00000000000000 2.00000000000000 > > > > > 5 2.00000000000000 2.00000000000000 > > > > > 6 2.00000000000000 2.00000000000000 > > > > > 7 2.00000000000000 2.00000000000000 > > > > > 8 2.00000000000000 2.00000000000000 > > > > > 9 2.00000000000000 3.00000000000000 > > > > > > > > > > > > > > > Here the problem seems more severe. Values are not correctly > assigned to the second component. There should be 3.0 in the second column. > It seems values of the first component are copied to the second one. Only > for i=10 the value of xarray(i)%b is correct (3.0). > > > > > > > > > > Any idea where it could come from ? > > > > > I guess it needs some fixes here and there but I think your idea > is good and that it could work. > > > > > > > > > > Christophe > > > > > > > > > > > > > > > > > > > > On Thu, Sep 26, 2013 at 5:45 PM, Barry Smith > wrote: > > > > > > > > > > I just put it with the Fortran source code and compile it with > the rest of the application code; here is the makefile I used > > > > > > > > > > > > > > > > > > > > > > > > > On Sep 26, 2013, at 10:14 AM, Christophe Ortiz < > christophe.ortiz at ciemat.es> wrote: > > > > > > > > > > > By the way, what should I do with the small .c code ? Where > should I put it ? > > > > > > > > > > > > Christophe > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 26, 2013 at 4:09 PM, Barry Smith > wrote: > > > > > > > > > > > > Christophe, > > > > > > > > > > > > Despite my LinkIn Endorsement for expertise in Fortran :-) > I cannot pretend to be an expert in FortranXX but I have cooked up an > example demonstrating accessing the Vec entries as if they are in an array > of derived types. I've attached the example code; there needs to be a small > C stub that defines the functions for your specific derived type name. > > > > > > Note that it will only work I think if your N is a compile time > constant. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It worked with > > > > > > ~/Downloads$ gfortran --version > > > > > > GNU Fortran (GCC) 4.8.1 20130404 (prerelease) > > > > > > > > > > > > > > > > > > I do not understand exactly why it works since it uses > F90Array1dCreate(fa,PETSC_SCALAR,1,len,ptr PETSC_F90_2PTR_PARAM(ptrd)); > which has a single PETSC_SCALAR as a building block but ? I hope it works > for you. If it doesn't, let us know the compiler you are using and we may > be able to get it working for that compiler. > > > > > > > > > > > > Barry > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sep 26, 2013, at 4:41 AM, Christophe Ortiz < > christophe.ortiz at ciemat.es> wrote: > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > Me again ! > > > > > > > > > > > > > > I have read in previous posts that it is hard in Fortran to > declare something similar to a typedef struct in C to manage a > multicomponent problem. > > > > > > > > > > > > > > Is it still the case ? Has the problem been solved ? > > > > > > > > > > > > > > I am asking because my plan is to implement a multicomponent > problem (in 1D), with many components that will be organized in arrays of > two dimensions. In C I could define > > > > > > > > > > > > > > typedef struct{ > > > > > > > PetscScalar U; > > > > > > > PetscScalar V; > > > > > > > PetscScalar A[N][N]; > > > > > > > } Field; > > > > > > > > > > > > > > and then I could calculate the residual function with > > > > > > > > > > > > > > F[i].U = ... > > > > > > > F[i].V = ... > > > > > > > F[i].A[k][n] = ... > > > > > > > > > > > > > > which is quite convenient. > > > > > > > > > > > > > > If in Fortran it is not possible to use a struct as in C, I am > afraid I'll have to deal with > > > > > > > > > > > > > > F(jdof,i) = ... > > > > > > > > > > > > > > where I will have only jdof to address U, V and A[ ][ ], which > can be difficult and not very convenient I guess. Before I move all my code > to C, does anyone have an alternative idea to manage this > multi(many)component problem in Fortran ? > > > > > > > > > > > > > > Many thanks in advance for your help and suggestion ! > > > > > > > > > > > > > > Christophe > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Sep 30 07:04:01 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 30 Sep 2013 07:04:01 -0500 Subject: [petsc-users] Back to struct in Fortran to represent field with dof > 1 In-Reply-To: References: <3524B3B9-C5FE-46A7-8556-F36DEEC287BD@mcs.anl.gov> <33F8252E-1F7C-487E-9EFC-83CC60852471@mcs.anl.gov> Message-ID: <214E26C8-54A1-4E4B-B7A4-5EBBC4E50B40@mcs.anl.gov> On Sep 30, 2013, at 3:01 AM, Christophe Ortiz wrote: > Hi Barry, > > It works ! Great job ! Here is the output: > > > Components have the right values. > > Do you think it can be adapted to work with DMDAVecGetArrayF90(da,...) / DMDAVecRestoreArrayF90(da,...) in order to work with distributed arrays ? What dimension do you need? For one dimension, which I think you mentioned before it is almost identical to what I sent you PETSC_EXTERN void PETSC_STDCALL dmdavecgetarrayf901_(DM *da,Vec *v,F90Array1d *a,PetscErrorCode *ierr PETSC_F90_2PTR_PROTO(ptrd)) { PetscInt xs,ys,zs,xm,ym,zm,gxs,gys,gzs,gxm,gym,gzm,N,dim,dof; PetscScalar *aa; *ierr = DMDAGetCorners(*da,&xs,&ys,&zs,&xm,&ym,&zm);if (*ierr) return; *ierr = DMDAGetGhostCorners(*da,&gxs,&gys,&gzs,&gxm,&gym,&gzm);if (*ierr) return; *ierr = DMDAGetInfo(*da,&dim,0,0,0,0,0,0,&dof,0,0,0,0,0);if (*ierr) return; /* Handle case where user passes in global vector as opposed to local */ *ierr = VecGetLocalSize(*v,&N);if (*ierr) return; if (N == xm*ym*zm*dof) { gxm = xm; gym = ym; gzm = zm; gxs = xs; gys = ys; gzs = zs; } else if (N != gxm*gym*gzm*dof) { *ierr = PETSC_ERR_ARG_INCOMP; } *ierr = VecGetArray(*v,&aa);if (*ierr) return; *ierr = F90Array1dCreate(aa,PETSC_SCALAR,gxs,gxm,a PETSC_F90_2PTR_PARAM(ptrd));if (*ierr) return; } except it (1) handles both the local vector or global vector version (the if above) and (2) handles that start index at gxs instead of 0. All you should have to do is rename the beast above, provide an FORTRAN 90 interface for it, have it call the specialized version of F90Array1dCreate I already provided. Two and three dimensions are a bit more involved since you need to copy over and modify more functions but I don't think there are any tricky things. Barry > > Best regards, > Christophe > > > 2013/9/29 Barry Smith > > > Sorry, that is what I get for having a ex1.F and ex1F90.F in the same directory. > > Barry > > On Sep 29, 2013, at 6:16 AM, Christophe Ortiz wrote: > > > Hi Barry, > > > > I just looked at exf1.F. It seems you sent me a wrong file. It does not correspond to ex1f90. There is no vecgetarraymystruct in the file you sent me. Could you please send me the correct version of ex1f90 so I can test it ? Many thanks in advance. > > > > Christophe > > > > CIEMAT > > Laboratorio Nacional de Fusi?n por Confinamiento Magn?tico > > Unidad de Materiales > > Edificio 2 - Planta 0 - Despacho 28m > > Avenida Complutense 40, > > 28040 Madrid, Spain > > Tel: +34 91496 2582 > > Fax: +34 91346 6442 > > > > -- > > Q > > Por favor, piense en el medio ambiente antes de imprimir este mensaje. > > Please consider the environment before printing this email. > > > > > > 2013/9/27 Barry Smith > > > > On Sep 27, 2013, at 3:06 PM, Christophe Ortiz wrote: > > > > > The files were not attached. Could you please send them again ? > > > > > > Thanks. > > > Christophe > > > > > > El 27/09/2013 21:28, "Barry Smith" escribi?: > > > > > > Please find attached a new version of the code. ex1.c and ex1f90.F please run it as is and send me all the output. > > > > > > It works correctly with gfortran and the intel 12.x ifort compiler, I believe it should now work for all fortran compilers. If it does not work as expected please send me the name and version of your fortran compiler. > > > > > > Barry > > > > > > On Sep 27, 2013, at 8:22 AM, Christophe Ortiz wrote: > > > > > > > There was a misunderstanding. There are n entries in total in the vector. I thought it was a vector of n points, with 3 components associated to each point, ie 3*n entries. This is clear now. > > > > > > > > I've tested your example and I checked the values assigned to the components with your piece of code: > > > > > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > > > do i=1,10 > > > > xarray(i)%a = i > > > > xarray(i)%b = 100*i > > > > xarray(i)%c = 10000*i > > > > enddo > > > > > > > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > > > > > > > > > I did it with: > > > > > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > > > do i = 1 , 10 > > > > write(*,*) xarray(i)%a,xarray(i)%b,xarray(i)%c > > > > end do > > > > > > > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > > > > > I obtained the following: > > > > > > > > 1.00000000000000 2.00000000000000 3.00000000000000 > > > > 2.00000000000000 3.00000000000000 4.00000000000000 > > > > 3.00000000000000 4.00000000000000 5.00000000000000 > > > > 4.00000000000000 5.00000000000000 6.00000000000000 > > > > 5.00000000000000 6.00000000000000 7.00000000000000 > > > > 6.00000000000000 7.00000000000000 8.00000000000000 > > > > 7.00000000000000 8.00000000000000 9.00000000000000 > > > > 8.00000000000000 9.00000000000000 10.0000000000000 > > > > 9.00000000000000 10.0000000000000 1000.00000000000 > > > > 10.0000000000000 1000.00000000000 100000.000000000 > > > > > > > > First column if ok (1st component), but the other two columns show wrong numbers. For component 2 and 3, only the very last value is ok. It seems values from component b are copied from a, and those from component c from b, with an offset. > > > > > > > > What do you think ? > > > > > > > > Christophe > > > > > > > > > > > > On Fri, Sep 27, 2013 at 2:47 PM, Barry Smith wrote: > > > > > > > > Indices start in fortran arrays from 1 by default so don't even try zero > > > > > > > > I've attached a new version that has three components, > > > > 30 TOTAL vector entries and 10 for each component. Please run it and send the results back. > > > > > > > > Barry > > > > > > > > On Sep 27, 2013, at 5:40 AM, Christophe Ortiz wrote: > > > > > > > > > Hi Barry, > > > > > > > > > > I made some tests with your code that allows to use struct in Fortran90. It compiles and does not complain with the struct with 2 components but...it exhibits some strange behaviour. Here are the behaviours I observed. To make it simple, I used n=10. > > > > > > > > > > - After using call VecSet(x,one,ierr), I checked the values of each component of the vector using > > > > > > > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > > > > do i = 0 , n > > > > > write(*,*) i , xarray(i)%a, xarray(i)%b > > > > > end do > > > > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > > > > > > > I checked the values from i= 0 to n since I was not sure what was the convention (0- or 1-based). I obtained: > > > > > > > > > > 0 0.000000000000000E+000 1.00000000000000 > > > > > 1 1.00000000000000 1.00000000000000 > > > > > 2 1.00000000000000 1.00000000000000 > > > > > 3 1.00000000000000 1.00000000000000 > > > > > 4 1.00000000000000 1.00000000000000 > > > > > 5 1.00000000000000 1.00000000000000 > > > > > 6 1.00000000000000 1.00000000000000 > > > > > 7 1.00000000000000 1.00000000000000 > > > > > 8 1.00000000000000 1.00000000000000 > > > > > 9 1.00000000000000 1.00000000000000 > > > > > 10 1.00000000000000 1.996650376645798E-314 > > > > > > > > > > > > > > > As you can see, for component a, there is a 0 for i=0, then the values are correct. For component b, correct values go from i=1 to 9.There is garbage in second component for i=10 . I checked that if you define a third component c, its values go from i=-1 to n-2. It's like if values of second component started at the end of first component, and same thing for third component. > > > > > > > > > > Then, instead of using VecSet(), I tried to assign values directly to each component of the vector with > > > > > > > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > > > > do i = 0 , n-1 > > > > > xarray(i)%a = 2.0 > > > > > xarray(i)%b = 3.0 > > > > > end do > > > > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > > > > > > > Here I checked that indices actually start at 0 and not at 1. With a loop from i=1 to n I got a Memory corruption message. > > > > > > > > > > and I checked that the values are correctly assigned with > > > > > > > > > > call VecGetArrayMyStruct(x,xarray,ierr) > > > > > do i = 0 , n-1 > > > > > write(*,*) i, xarray(i)%a, xarray(i)%b > > > > > end do > > > > > call VecRestoreArrayMyStruct(x,xarray,ierr) > > > > > > > > > > > > > > > > > > > > Then, I obtained: > > > > > > > > > > 0 2.00000000000000 2.00000000000000 > > > > > 1 2.00000000000000 2.00000000000000 > > > > > 2 2.00000000000000 2.00000000000000 > > > > > 3 2.00000000000000 2.00000000000000 > > > > > 4 2.00000000000000 2.00000000000000 > > > > > 5 2.00000000000000 2.00000000000000 > > > > > 6 2.00000000000000 2.00000000000000 > > > > > 7 2.00000000000000 2.00000000000000 > > > > > 8 2.00000000000000 2.00000000000000 > > > > > 9 2.00000000000000 3.00000000000000 > > > > > > > > > > > > > > > Here the problem seems more severe. Values are not correctly assigned to the second component. There should be 3.0 in the second column. It seems values of the first component are copied to the second one. Only for i=10 the value of xarray(i)%b is correct (3.0). > > > > > > > > > > Any idea where it could come from ? > > > > > I guess it needs some fixes here and there but I think your idea is good and that it could work. > > > > > > > > > > Christophe > > > > > > > > > > > > > > > > > > > > On Thu, Sep 26, 2013 at 5:45 PM, Barry Smith wrote: > > > > > > > > > > I just put it with the Fortran source code and compile it with the rest of the application code; here is the makefile I used > > > > > > > > > > > > > > > > > > > > > > > > > On Sep 26, 2013, at 10:14 AM, Christophe Ortiz wrote: > > > > > > > > > > > By the way, what should I do with the small .c code ? Where should I put it ? > > > > > > > > > > > > Christophe > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 26, 2013 at 4:09 PM, Barry Smith wrote: > > > > > > > > > > > > Christophe, > > > > > > > > > > > > Despite my LinkIn Endorsement for expertise in Fortran :-) I cannot pretend to be an expert in FortranXX but I have cooked up an example demonstrating accessing the Vec entries as if they are in an array of derived types. I've attached the example code; there needs to be a small C stub that defines the functions for your specific derived type name. > > > > > > Note that it will only work I think if your N is a compile time constant. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It worked with > > > > > > ~/Downloads$ gfortran --version > > > > > > GNU Fortran (GCC) 4.8.1 20130404 (prerelease) > > > > > > > > > > > > > > > > > > I do not understand exactly why it works since it uses F90Array1dCreate(fa,PETSC_SCALAR,1,len,ptr PETSC_F90_2PTR_PARAM(ptrd)); which has a single PETSC_SCALAR as a building block but ? I hope it works for you. If it doesn't, let us know the compiler you are using and we may be able to get it working for that compiler. > > > > > > > > > > > > Barry > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sep 26, 2013, at 4:41 AM, Christophe Ortiz wrote: > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > Me again ! > > > > > > > > > > > > > > I have read in previous posts that it is hard in Fortran to declare something similar to a typedef struct in C to manage a multicomponent problem. > > > > > > > > > > > > > > Is it still the case ? Has the problem been solved ? > > > > > > > > > > > > > > I am asking because my plan is to implement a multicomponent problem (in 1D), with many components that will be organized in arrays of two dimensions. In C I could define > > > > > > > > > > > > > > typedef struct{ > > > > > > > PetscScalar U; > > > > > > > PetscScalar V; > > > > > > > PetscScalar A[N][N]; > > > > > > > } Field; > > > > > > > > > > > > > > and then I could calculate the residual function with > > > > > > > > > > > > > > F[i].U = ... > > > > > > > F[i].V = ... > > > > > > > F[i].A[k][n] = ... > > > > > > > > > > > > > > which is quite convenient. > > > > > > > > > > > > > > If in Fortran it is not possible to use a struct as in C, I am afraid I'll have to deal with > > > > > > > > > > > > > > F(jdof,i) = ... > > > > > > > > > > > > > > where I will have only jdof to address U, V and A[ ][ ], which can be difficult and not very convenient I guess. Before I move all my code to C, does anyone have an alternative idea to manage this multi(many)component problem in Fortran ? > > > > > > > > > > > > > > Many thanks in advance for your help and suggestion ! > > > > > > > > > > > > > > Christophe > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From popov at uni-mainz.de Mon Sep 30 08:12:06 2013 From: popov at uni-mainz.de (Anton Popov) Date: Mon, 30 Sep 2013 15:12:06 +0200 Subject: [petsc-users] PetscViewerBinaryMatlabOutputVecDA In-Reply-To: References: <87li2n2mt5.fsf@mcs.anl.gov> Message-ID: <524978A6.1020203@uni-mainz.de> Hi guys, just a quick question. I'm upgrading to 3.4.2 and it complains about PetscViewerBinaryMatlabOutputVecDA What to use instead? I couldn't find it in Changes 3.4 Thanks, Anton -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Sep 30 08:37:32 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 30 Sep 2013 08:37:32 -0500 Subject: [petsc-users] PetscViewerBinaryMatlabOutputVecDA In-Reply-To: <524978A6.1020203@uni-mainz.de> References: <87li2n2mt5.fsf@mcs.anl.gov> <524978A6.1020203@uni-mainz.de> Message-ID: <60F02DCC-576F-4420-9970-BF5A4A7ECA3A@mcs.anl.gov> On Sep 30, 2013, at 8:12 AM, Anton Popov wrote: > Hi guys, > > just a quick question. I'm upgrading to 3.4.2 and it complains about PetscViewerBinaryMatlabOutputVecDA > > What to use instead? I couldn't find it in Changes 3.4 To get the effect you just call PetscViewerBinaryOpen() followed by PetscViewerSetFormat(viewer,PETSC_FORMAT_BINARY_MATLAB); the names of the resulting MATLAB variables are obtained by the names of the PETSc objects which may be set with PetscObjectSetName(). You can use VecView() on a Vec from a DA and PetscBagView() etc and the Matlab code that is needed to reshape the vectors and load the bags is generated automatically as before. Please let us know if you have any trouble, Barry > > Thanks, > > Anton From jedbrown at mcs.anl.gov Mon Sep 30 08:53:17 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 30 Sep 2013 08:53:17 -0500 Subject: [petsc-users] PetscViewerBinaryMatlabOutputVecDA In-Reply-To: <524978A6.1020203@uni-mainz.de> References: <87li2n2mt5.fsf@mcs.anl.gov> <524978A6.1020203@uni-mainz.de> Message-ID: <87siwm8lgy.fsf@mcs.anl.gov> Anton Popov writes: > Hi guys, > > just a quick question. I'm upgrading to 3.4.2 and it complains about > PetscViewerBinaryMatlabOutputVecDA > > What to use instead? I couldn't find it in Changes 3.4 That's the "and friends" in: * PetscViewerBinaryMatlabOpen() and friends are all removed, simply use PetscViewerBinaryOpen() followed by PetscViewerSetFormat(viewer,PETSC_FORMAT_BINARY_MATLAB) commit a261c58fdc00b5e279fd9d92904f5cf5eafe466b Author: Barry Smith Date: Wed Feb 13 16:13:47 2013 -0600 removed PetscViewerBinaryMatlabOpen() and friends, incorporated functionality into binary PETSc viewer Hg-commit: 1676211ea1509edcdd7a044ca5b8b543f443b8ab include/finclude/petscviewer.h | 6 +- include/petsc.h | 1 + include/petscdm.h | 8 - include/petscviewer.h | 1 + src/dm/impls/da/gr2.c | 54 ++++- src/dm/impls/da/makefile | 2 +- src/dm/impls/da/utils/binarymatlab.c | 198 ----------------- src/dm/impls/da/utils/makefile | 17 -- .../website/documentation/changes/dev.html | 1 + src/sys/classes/bag/bag.c | 17 +- src/sys/classes/viewer/impls/binary/binv.c | 39 +++- src/sys/classes/viewer/interface/viewa.c | 1 + src/vec/vec/impls/mpi/pdvec.c | 14 ++ src/vec/vec/impls/seq/bvec2.c | 14 ++ 14 files changed, 128 insertions(+), 245 deletions(-) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From william.perkins at pnnl.gov Mon Sep 30 09:36:29 2013 From: william.perkins at pnnl.gov (William A. Perkins) Date: Mon, 30 Sep 2013 07:36:29 -0700 Subject: [petsc-users] Cannot use KSP if I choose local Matrix/Vector ownership In-Reply-To: References: <21060.43290.839444.337643@pnl.gov> Message-ID: <21065.35949.596494.279021@pnl.gov> Barry, Thanks. I should have known that. Bill >>>>> "Barry" == Barry Smith writes: Barry> Bill, Barry> When you call Barry> ierr = MatSetSizes(A,local_size, PETSC_DECIDE,imax*jmax, imax*jmax); CHKERRXX(ierr); Barry> you should pass local_size as the third argument as well Barry> instead of PETSC_DECIDE. PETSc is making a choice here Barry> based only on the total number of columns imax*jmax and Barry> doesn't take into account what you provided for local row Barry> size. Barry> Essentially you can either let PETSc decide all the Barry> local sizes of the Vecs and Mats or you decide all of Barry> them; but mixing that will usually not work. Barry> Barry Barry> On Sep 26, 2013, at 4:37 PM, "William A. Perkins" wrote: >> >> Hello, >> >> I have run into a problem using KSP. Attached is a hopefully short, >> simple example program (which has nothing to do with my actual >> application) demonstrating the problem. Also attached is an output >> log from several runs of the program. The program runs fine if I >> choose local ownership according to PetscSplitOwnership(). If, >> however, I slightly modify that local ownership scheme, the program >> will not run in parallel. In my view, KSP appears to make its own >> coefficient (or preconditioner, or whatever) matrix that has different >> local ownership than the one I give to KSPSetOperators(), and so >> cannot do matrix/vector multiplies. >> >> I would like to be able to choose my own local ownership. Any help >> would be appreciated. Hopefully, I'm not doing something stupid. >> >> Thanks. >> >> Bill >> >> -- >> Bill Perkins >> Research Engineer >> Hydrology Group >> >> Pacific Northwest National Laboratory >> 902 Battelle Boulevard >> P.O. Box 999, MSIN K9-36 >> Richland, WA 99352 USA >> Tel: 509-372-6131 >> Fax: 509-372-6089 >> william.perkins at pnnl.gov >> www.pnnl.gov >> /* ------------------------------------------------------------- >> file: versteeg.cpp >> >> This is a simple test of PETSC Matrix operations and Linear >> equation solvers. This problem comes from Example 7.2 in >> >> Versteeg, H.K. and W. Malalasekera, 1995. An introduction to >> computational fluid dynamics, the finite volume method. Prentice >> Hall. 257 pp. >> >> >> ------------------------------------------------------------- */ >> /* ------------------------------------------------------------- >> Created October 10, 2002 by William A. Perkins >> Last Change: 2013-09-26 14:32:33 d3g096 >> ------------------------------------------------------------- */ >> >> >> static const char* SCCS_ID = "$Id: versteeg.c,v 1.3 2008/05/24 23:04:39 d3g096 Exp $ Battelle PNNL"; >> >> static char help[] = >> "Solves an example heat transfer problem from Versteeg and Malalasekera with SLES.\n\ >> The following options are available:\n\ >> -view_solution yes/no: write the initial and final solution to stdout\n\ >> -problem yes/no: cause problem by choosing local ownership\n\n"; >> >> #include >> #include >> >> /* ------------------------------------------------------------- >> assemble >> ------------------------------------------------------------- */ >> int >> assemble(const int imax, const int jmax, const double scale, Mat A, Vec b) >> { >> static const float k = 1000; /* conductivity, W/m/K */ >> static const float t = 0.01; /* plate thickness, m */ >> static const float W = 0.3; /* plate width, m */ >> static const float H = 0.4; /* plate height, m */ >> >> PetscErrorCode ierr; >> >> int i, j; >> float ap, aw, ae, as, an, bp; >> >> float dx = W/(float)imax; >> float dy = H/(float)jmax; >> >> int iP, iN, iS, iE, iW; >> >> PetscScalar v; >> >> PetscInt lo, hi; >> >> ierr = MatGetOwnershipRange(A, &lo, &hi); CHKERRQ(ierr); >> >> for (i = 0; i < imax; i++) { >> for (j = 0; j < jmax; j++) { >> iP = i*jmax + j; >> if (! (lo <= iP && iP < hi) ) continue; >> iE = (i+1)*jmax + j; >> iW = (i-1)*jmax + j; >> iN = i*jmax + (j+1); >> iS = i*jmax + (j-1); >> >> bp = 0.0; >> ap = 0.0; >> if (j == 0) { /* insulated south boundary */ >> as = 0.0; >> bp += 0.0; >> ap -= 0.0; >> } else { >> as = (k/dx)*(dx*t); >> } >> >> if (j == jmax - 1) { /* constant tempurature (100C) north boundary */ >> an = 0.0; >> bp += 2*k/dy*(dy*t)*100.0; >> ap -= -2*k/dy*(dy*t); >> } else { >> an = (k/dx)*(dx*t); >> } >> >> if (i == 0) { /* constant flux (500kw/m2) west boundary */ >> aw = 0.0; >> bp += 500000.0*dy*t; >> ap -= 0.0; >> } else { >> aw = (k/dx)*(dx*t); >> } >> >> if (i == imax - 1) { /* insulated east boundary */ >> ae = 0.0; >> bp += 0.0; >> ap -= 0.0; >> } else { >> ae = (k/dx)*(dx*t); >> } >> >> ap += as + an + aw + ae; >> >> v = ap*scale; >> ierr = MatSetValues(A,1,&iP,1,&iP,&v,INSERT_VALUES); CHKERRQ(ierr); >> >> if (an != 0.0) { >> v = -an*scale; >> ierr = MatSetValues(A,1,&iP,1,&iN,&v,INSERT_VALUES); CHKERRQ(ierr); >> } >> if (as != 0.0) { >> v = -as*scale; >> ierr = MatSetValues(A,1,&iP,1,&iS,&v,INSERT_VALUES); CHKERRQ(ierr); >> } >> if (ae != 0.0) { >> v = -ae*scale; >> ierr = MatSetValues(A,1,&iP,1,&iE,&v,INSERT_VALUES); CHKERRQ(ierr); >> } >> if (aw != 0.0) { >> v = -aw*scale; >> ierr = MatSetValues(A,1,&iP,1,&iW,&v,INSERT_VALUES); CHKERRQ(ierr); >> } >> >> v = bp*scale; >> ierr = VecSetValues(b, 1, &iP, &v, INSERT_VALUES); CHKERRQ(ierr); >> >> } >> } >> >> ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >> ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); >> >> ierr = VecAssemblyBegin(b); CHKERRQ(ierr); >> ierr = VecAssemblyEnd(b); CHKERRQ(ierr); >> >> return(0); >> } >> >> /* ------------------------------------------------------------- >> Main Program >> ------------------------------------------------------------- */ >> int >> main(int argc, char **args) >> { >> >> static const int imax = 3; >> static const int jmax = 4; >> >> PetscBool print_solution(PETSC_FALSE); >> PetscBool set_local_size(PETSC_FALSE); >> int its; >> >> PetscErrorCode ierr; >> >> Vec x, b; >> Mat A; >> PetscScalar v; >> KSP ksp; >> VecScatter ctx; >> int rank; >> int nproc; >> PetscInt local_size(PETSC_DECIDE); >> PetscInt global_size(imax*jmax); >> >> PetscInitialize(&argc,&args,(char *)0,help); >> >> PetscOptionsGetBool(NULL, "-print_solution", &print_solution, NULL); >> PetscOptionsGetBool(NULL, "-problem", &set_local_size, NULL); >> >> >> ierr = MPI_Comm_size(PETSC_COMM_WORLD,&nproc); >> ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank); >> >> ierr = PetscSplitOwnership(PETSC_COMM_WORLD, &local_size, &global_size); CHKERRQ(ierr); >> if (set_local_size) { >> >> // If there is more than one processor, alter the default local >> // ownership so that process 0 has one less row and the last >> // process has more row. >> >> if (nproc > 1) { >> if (rank == 0) { >> local_size -= 1; >> } else if (rank == nproc - 1) { >> local_size += 1; >> } >> } >> } >> >> ierr = MatCreate(PETSC_COMM_WORLD, &A); CHKERRQ(ierr); >> >> ierr = MatSetSizes(A, >> local_size, PETSC_DECIDE, >> imax*jmax, imax*jmax); CHKERRXX(ierr); >> ierr = MatSetFromOptions(A); CHKERRQ(ierr); >> ierr = MatSetUp(A); CHKERRQ(ierr); >> >> ierr = VecCreate(PETSC_COMM_WORLD,&x);CHKERRQ(ierr); >> ierr = VecSetSizes(x,local_size,imax*jmax);CHKERRQ(ierr); >> ierr = VecSetFromOptions(x);CHKERRQ(ierr); >> ierr = VecDuplicate(x,&b);CHKERRQ(ierr); >> >> ierr = assemble(imax, jmax, 1.0, A, b); CHKERRQ(ierr); >> >> v = 0.0; >> ierr = VecSet(x,v);CHKERRQ(ierr); >> if (print_solution) { >> ierr = VecView(x, PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr); >> } >> >> ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr); >> ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr); >> ierr = KSPSetInitialGuessNonzero(ksp,PETSC_TRUE); CHKERRQ(ierr); >> ierr = KSPSetTolerances(ksp, 1e-06, 1e-12, PETSC_DEFAULT, PETSC_DEFAULT); CHKERRQ(ierr); >> >> ierr = KSPSetOperators(ksp,A,A,SAME_NONZERO_PATTERN);CHKERRQ(ierr); >> ierr = KSPSetUp(ksp); CHKERRQ(ierr); >> ierr = KSPSolve(ksp, b, x);CHKERRQ(ierr); >> ierr = KSPGetIterationNumber(ksp,&its);CHKERRQ(ierr); >> ierr = PetscPrintf(PETSC_COMM_WORLD,"iterations %d\n",its);CHKERRQ(ierr); >> >> if (print_solution) { >> ierr = VecView(x, PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr); >> } >> >> ierr = KSPDestroy(&ksp);CHKERRQ(ierr); >> ierr = VecDestroy(&x);CHKERRQ(ierr); >> ierr = VecDestroy(&b);CHKERRQ(ierr); >> ierr = MatDestroy(&A);CHKERRQ(ierr); >> >> ierr = PetscFinalize(); >> return(ierr); >> } >>>>> mpirun -np 1 versteeg -print_solution >> Vector Object: 1 MPI processes >> type: seq >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> iterations 8 >> Vector Object: 1 MPI processes >> type: seq >> 260.037 >> 242.275 >> 205.592 >> 146.322 >> 227.799 >> 211.195 >> 178.178 >> 129.696 >> 212.164 >> 196.53 >> 166.23 >> 123.982 >>>>> mpirun -np 2 versteeg -print_solution >> Vector Object: 2 MPI processes >> type: mpi >> Process [0] >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> Process [1] >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> iterations 9 >> Vector Object: 2 MPI processes >> type: mpi >> Process [0] >> 260.037 >> 242.275 >> 205.592 >> 146.322 >> 227.799 >> 211.195 >> Process [1] >> 178.178 >> 129.696 >> 212.164 >> 196.53 >> 166.23 >> 123.982 >>>>> mpirun -np 4 versteeg -print_solution >> Vector Object: 4 MPI processes >> type: mpi >> Process [0] >> 0 >> 0 >> 0 >> Process [1] >> 0 >> 0 >> 0 >> Process [2] >> 0 >> 0 >> 0 >> Process [3] >> 0 >> 0 >> 0 >> iterations 12 >> Vector Object: 4 MPI processes >> type: mpi >> Process [0] >> 260.037 >> 242.275 >> 205.592 >> Process [1] >> 146.322 >> 227.799 >> 211.195 >> Process [2] >> 178.178 >> 129.696 >> 212.164 >> Process [3] >> 196.53 >> 166.23 >> 123.982 >>>>> mpirun -np 2 versteeg -print_solution -problem >> Vector Object: 2 MPI processes >> type: mpi >> Process [0] >> 0 >> 0 >> 0 >> 0 >> 0 >> Process [1] >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message ------------------------------------ >> [1]PETSC ERROR: Invalid argument! >> [1]PETSC ERROR: Must be square matrix, rows 7 columns 6! >> [1]PETSC ERROR: ------------------------------------------------------------------------ >> [1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 >> [1]PETSC ERROR: See docs/changes/index.html for recent updates. >> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [1]PETSC ERROR: See docs/index.html for manual pages. >> [1]PETSC ERROR: ------------------------------------------------------------------------ >> [1]PETSC ERROR: versteeg on a arch-complex named flophouse by d3g096 Thu Sep 26 14:30:33 2013 >> [1]PETSC ERROR: Libraries linked from /net/flophouse/files0/perksoft/petsc-3.4.2/arch-complex/lib >> [1]PETSC ERROR: Configure run at Wed Sep 11 06:43:12 2013 >> [1]PETSC ERROR: Configure options --with-scalar-type=complex --with-clanguage=C++ PETSC_ARCH=arch-complex --with-fortran-kernels=generic --download-superlu_dist --download-mumps --download-scalapack --download-parmetis --download-metis --download-elemental --with-mpi=1 --download-blacs >> [1]PETSC ERROR: ------------------------------------------------------------------------ >> [1]PETSC ERROR: MatGetOrdering() line 257 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/mat/order/sorder.c >> [1]PETSC ERROR: PCSetUp_ILU() line 194 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/impls/factor/ilu/ilu.c >> [1]PETSC ERROR: PCSetUp() line 890 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/interface/precon.c >> [1]PETSC ERROR: KSPSetUp() line 278 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c >> [1]PETSC ERROR: PCSetUpOnBlocks_BJacobi_Singleblock() line 646 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/impls/bjacobi/bjacobi.c >> [1]PETSC ERROR: PCSetUpOnBlocks() line 924 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/interface/precon.c >> [1]PETSC ERROR: KSPSetUpOnBlocks() line 165 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c >> [1]PETSC ERROR: KSPSolve() line 400 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c >> [1]PETSC ERROR: main() line 216 in /home/d3g096/Projects/PerformanceLDRD/src/sandbox/petsc/versteeg.cpp >> --------------------- Error Message ------------------------------------ >> [0]PETSC ERROR: Invalid argument! >> [0]PETSC ERROR: Must be square matrix, rows 5 columns 6! >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: versteeg on a arch-complex named flophouse by d3g096 Thu Sep 26 14:30:33 2013 >> [0]PETSC ERROR: Libraries linked from /net/flophouse/files0/perksoft/petsc-3.4.2/arch-complex/lib >> [0]PETSC ERROR: Configure run at Wed Sep 11 06:43:12 2013 >> [0]PETSC ERROR: Configure options --with-scalar-type=complex --with-clanguage=C++ PETSC_ARCH=arch-complex --with-fortran-kernels=generic --download-superlu_dist --download-mumps --download-scalapack --download-parmetis --download-metis --download-elemental --with-mpi=1 --download-blacs >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: MatGetOrdering() line 257 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/mat/order/sorder.c >> [0]PETSC ERROR: PCSetUp_ILU() line 194 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/impls/factor/ilu/ilu.c >> [0]PETSC ERROR: PCSetUp() line 890 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: KSPSetUp() line 278 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: PCSetUpOnBlocks_BJacobi_Singleblock() line 646 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/impls/bjacobi/bjacobi.c >> [0]PETSC ERROR: PCSetUpOnBlocks() line 924 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: KSPSetUpOnBlocks() line 165 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: KSPSolve() line 400 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: main() line 216 in /home/d3g096/Projects/PerformanceLDRD/src/sandbox/petsc/versteeg.cpp >> -------------------------------------------------------------------------- >> MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD >> with errorcode 62. >> >> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >> You may or may not see output from other processes, depending on >> exactly when Open MPI kills them. >> -------------------------------------------------------------------------- >> -------------------------------------------------------------------------- >> mpirun has exited due to process rank 1 with PID 12459 on >> node flophouse exiting without calling "finalize". This may >> have caused other processes in the application to be >> terminated by signals sent by mpirun (as reported here). >> -------------------------------------------------------------------------- >> [flophouse:12457] 1 more process has sent help message help-mpi-api.txt / mpi-abort >> [flophouse:12457] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages >>>>> mpirun -np 4 versteeg -print_solution -problem >> Vector Object: 4 MPI processes >> type: mpi >> Process [0] >> 0 >> 0 >> Process [1] >> 0 >> 0 >> 0 >> Process [2] >> 0 >> 0 >> 0 >> Process [3] >> 0 >> 0 >> 0 >> 0 >> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >> [0]PETSC ERROR: Invalid argument! >> [0]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message ------------------------------------ >> [3]PETSC ERROR: Invalid argument! >> [3]PETSC ERROR: Must be square matrix, rows 4 columns 3! >> [3]PETSC ERROR: ------------------------------------------------------------------------ >> [3]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 >> [3]PETSC ERROR: See docs/changes/index.html for recent updates. >> [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [3]PETSC ERROR: See docs/index.html for manual pages. >> [3]PETSC ERROR: ------------------------------------------------------------------------ >> [3]PETSC ERROR: versteeg on a arch-complex named flophouse by d3g096 Thu Sep 26 14:30:51 2013 >> [3]PETSC ERROR: Libraries linked from /net/flophouse/files0/perksoft/petsc-3.4.2/arch-complex/lib >> [3]PETSC ERROR: Configure run at Wed Sep 11 06:43:12 2013 >> [3]PETSC ERROR: Configure options --with-scalar-type=complex --with-clanguage=C++ PETSC_ARCH=arch-complex --with-fortran-kernels=generic --download-superlu_dist --download-mumps --download-scalapack --download-parmetis --download-metis --download-elemental --with-mpi=1 --download-blacs >> [3]PETSC ERROR: ------------------------------------------------------------------------ >> [3]PETSC ERROR: MatGetOrdering() line 257 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/mat/order/sorder.c >> [3]PETSC ERROR: PCSetUp_ILU() line 194 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/impls/factor/ilu/ilu.c >> [3]PETSC ERROR: PCSetUp() line 890 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/interface/precon.c >> [3]PETSC ERROR: KSPSetUp() line 278 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c >> [3]PETSC ERROR: PCSetUpOnBlocks_BJacobi_Singleblock() line 646 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/impls/bjacobi/bjacobi.c >> [3]PETSC ERROR: PCSetUpOnBlocks() line 924 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/interface/precon.c >> [3]PETSC ERROR: KSPSetUpOnBlocks() line 165 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c >> [3]PETSC ERROR: KSPSolve() line 400 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c >> [3]PETSC ERROR: main() line 216 in /home/d3g096/Projects/PerformanceLDRD/src/sandbox/petsc/versteeg.cpp >> Must be square matrix, rows 2 columns 3! >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: versteeg on a arch-complex named flophouse by d3g096 Thu Sep 26 14:30:51 2013 >> [0]PETSC ERROR: Libraries linked from /net/flophouse/files0/perksoft/petsc-3.4.2/arch-complex/lib >> [0]PETSC ERROR: Configure run at Wed Sep 11 06:43:12 2013 >> [0]PETSC ERROR: Configure options --with-scalar-type=complex --with-clanguage=C++ PETSC_ARCH=arch-complex --with-fortran-kernels=generic --download-superlu_dist --download-mumps --download-scalapack --download-parmetis --download-metis --download-elemental --with-mpi=1 --download-blacs >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: MatGetOrdering() line 257 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/mat/order/sorder.c >> [0]PETSC ERROR: PCSetUp_ILU() line 194 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/impls/factor/ilu/ilu.c >> [0]PETSC ERROR: PCSetUp() line 890 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: KSPSetUp() line 278 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: PCSetUpOnBlocks_BJacobi_Singleblock() line 646 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/impls/bjacobi/bjacobi.c >> [0]PETSC ERROR: PCSetUpOnBlocks() line 924 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: KSPSetUpOnBlocks() line 165 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: KSPSolve() line 400 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: main() line 216 in /home/d3g096/Projects/PerformanceLDRD/src/sandbox/petsc/versteeg.cpp >> [2]PETSC ERROR: --------------------- Error Message ------------------------------------ >> [2]PETSC ERROR: Object is in wrong state! >> [2]PETSC ERROR: Matrix is missing diagonal entry 2! >> [2]PETSC ERROR: ------------------------------------------------------------------------ >> [2]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 >> [2]PETSC ERROR: -------------------------------------------------------------------------- >> MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD >> with errorcode 62. >> >> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >> You may or may not see output from other processes, depending on >> exactly when Open MPI kills them. >> -------------------------------------------------------------------------- >> See docs/changes/index.html for recent updates. >> [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [2]PETSC ERROR: See docs/index.html for manual pages. >> [2]PETSC ERROR: ------------------------------------------------------------------------ >> [2]PETSC ERROR: versteeg on a arch-complex named flophouse by d3g096 Thu Sep 26 14:30:51 2013 >> [2]PETSC ERROR: Libraries linked from /net/flophouse/files0/perksoft/petsc-3.4.2/arch-complex/lib >> [2]PETSC ERROR: Configure run at Wed Sep 11 06:43:12 2013 >> [2]PETSC ERROR: Configure options --with-scalar-type=complex --with-clanguage=C++ PETSC_ARCH=arch-complex --with-fortran-kernels=generic --download-superlu_dist --download-mumps --download-scalapack --download-parmetis --download-metis --download-elemental --with-mpi=1 --download-blacs >> [2]PETSC ERROR: ------------------------------------------------------------------------ >> [2]PETSC ERROR: MatILUFactorSymbolic_SeqAIJ_ilu0() line 1653 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/mat/impls/aij/seq/aijfact.c >> [2]PETSC ERROR: MatILUFactorSymbolic_SeqAIJ() line 1756 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/mat/impls/aij/seq/aijfact.c >> [2]PETSC ERROR: MatILUFactorSymbolic() line 6238 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/mat/interface/matrix.c >> [2]PETSC ERROR: PCSetUp_ILU() line 204 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/impls/factor/ilu/ilu.c >> [2]PETSC ERROR: PCSetUp() line 890 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/interface/precon.c >> [2]PETSC ERROR: KSPSetUp() line 278 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c >> [2]PETSC ERROR: PCSetUpOnBlocks_BJacobi_Singleblock() line 646 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/impls/bjacobi/bjacobi.c >> [2]PETSC ERROR: PCSetUpOnBlocks() line 924 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/interface/precon.c >> [2]PETSC ERROR: KSPSetUpOnBlocks() line 165 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c >> [2]PETSC ERROR: KSPSolve() line 400 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c >> [2]PETSC ERROR: main() line 216 in /home/d3g096/Projects/PerformanceLDRD/src/sandbox/petsc/versteeg.cpp >> [1]PETSC ERROR: --------------------- Error Message ------------------------------------ >> [1]PETSC ERROR: Argument out of range! >> [1]PETSC ERROR: Cannot log negative flops! >> [1]PETSC ERROR: ------------------------------------------------------------------------ >> [1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 >> [1]PETSC ERROR: See docs/changes/index.html for recent updates. >> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [1]PETSC ERROR: See docs/index.html for manual pages. >> [1]PETSC ERROR: ------------------------------------------------------------------------ >> [1]PETSC ERROR: versteeg on a arch-complex named flophouse by d3g096 Thu Sep 26 14:30:51 2013 >> [1]PETSC ERROR: Libraries linked from /net/flophouse/files0/perksoft/petsc-3.4.2/arch-complex/lib >> [1]PETSC ERROR: Configure run at Wed Sep 11 06:43:12 2013 >> [1]PETSC ERROR: Configure options --with-scalar-type=complex --with-clanguage=C++ PETSC_ARCH=arch-complex --with-fortran-kernels=generic --download-superlu_dist --download-mumps --download-scalapack --download-parmetis --download-metis --download-elemental --with-mpi=1 --download-blacs >> [1]PETSC ERROR: ------------------------------------------------------------------------ >> [1]PETSC ERROR: PetscLogFlops() line 204 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/include/petsclog.h >> [1]PETSC ERROR: MatLUFactorNumeric_SeqAIJ_Inode() line 1312 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/mat/impls/aij/seq/inode.c >> [1]PETSC ERROR: MatLUFactorNumeric() line 2889 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/mat/interface/matrix.c >> [1]PETSC ERROR: PCSetUp_ILU() line 232 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/impls/factor/ilu/ilu.c >> [1]PETSC ERROR: PCSetUp() line 890 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/interface/precon.c >> [1]PETSC ERROR: KSPSetUp() line 278 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c >> [1]PETSC ERROR: PCSetUpOnBlocks_BJacobi_Singleblock() line 646 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/impls/bjacobi/bjacobi.c >> [1]PETSC ERROR: PCSetUpOnBlocks() line 924 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/pc/interface/precon.c >> [1]PETSC ERROR: KSPSetUpOnBlocks() line 165 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c >> [1]PETSC ERROR: KSPSolve() line 400 in /net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/net/flophouse/files0/perksoft/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c >> [1]PETSC ERROR: main() line 216 in /home/d3g096/Projects/PerformanceLDRD/src/sandbox/petsc/versteeg.cpp >> -------------------------------------------------------------------------- >> mpirun has exited due to process rank 3 with PID 12483 on >> node flophouse exiting without calling "finalize". This may >> have caused other processes in the application to be >> terminated by signals sent by mpirun (as reported here). >> -------------------------------------------------------------------------- >> [flophouse:12479] 3 more processes have sent help message help-mpi-api.txt / mpi-abort >> [flophouse:12479] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages -- Bill Perkins Research Engineer Hydrology Group Pacific Northwest National Laboratory 902 Battelle Boulevard P.O. Box 999, MSIN K9-36 Richland, WA 99352 USA Tel: 509-372-6131 Fax: 509-372-6089 william.perkins at pnnl.gov www.pnnl.gov From parsani.matteo at gmail.com Mon Sep 30 11:38:53 2013 From: parsani.matteo at gmail.com (Matteo Parsani) Date: Mon, 30 Sep 2013 12:38:53 -0400 Subject: [petsc-users] MatMPIBAIJSetPreallocationCSR i and j indices In-Reply-To: <87y56f9ooq.fsf@mcs.anl.gov> References: <87had4e5pe.fsf@mcs.anl.gov> <8405066E-DD41-4D9E-9D96-1826F094D4A4@mcs.anl.gov> <6B15898A-6086-4437-90B4-C84E3AD4746F@mcs.anl.gov> <877gdzcypz.fsf@mcs.anl.gov> <8738oncx8d.fsf@mcs.anl.gov> <2756C588-A8CE-415E-8D69-C3D100EBC53B@mcs.anl.gov> <87ob7bbe37.fsf@mcs.anl.gov> <87y56f9ooq.fsf@mcs.anl.gov> Message-ID: Thank you Barry and Jed for the explanation! On Sun, Sep 29, 2013 at 7:46 PM, Jed Brown wrote: > Matteo Parsani writes: > > 2 - why those routines *should not be used* when writing a fresh code or > a > > large re-factoring? > > Barry wrote a great rationale, but consider an example: almost all > third-party linear algebra libraries (and some special features in > PETSc) require scalar (non-blocked) matrix formats, but PETSc has good > implementations of other operations like matrix multiplication and > incomplete factorization using BAIJ format. Thus with some solution > methods, you want a BAIJ matrix and with other methods, you need an AIJ > matrix. If you try to build the backend data format directly, you'll > either need multiple versions to specifically support each format or > there will need to be a copy, which doubles the peak memory usage > required for the matrix. If you just use MatXAIJSetPreallocation and > MatSetValuesBlocked, then you can choose the matrix format at run-time, > in which case all solvers become available without the extra memory > usage. > > There are many more examples. It is bad software design for a component > to depend on more than it must. Depend on generic interfaces instead of > internal formats. It will make your software more powerful and more > maintainable. > -- Matteo -------------- next part -------------- An HTML attachment was scrubbed... URL: From choi240 at purdue.edu Mon Sep 30 12:28:34 2013 From: choi240 at purdue.edu (Joon Hee Choi) Date: Mon, 30 Sep 2013 13:28:34 -0400 (EDT) Subject: [petsc-users] The product of two MPIDENSE matrices In-Reply-To: Message-ID: <1528036471.157490.1380562114604.JavaMail.root@mailhub028.itcs.purdue.edu> Hello Hong, I tried to get Elemental but it does not support 64-bit integer now. I need to use 64-bit integer because the matrix I am using is very large. I need just MatMatMult() for MPIDENSE matrices, but it also didn't support MPIDENSE. Is there any method to multiply MPIDENSE and MPIDENSE? Thank you. Joon ----- Original Message ----- From: "Hong Zhang" To: "Joon Hee Choi" Cc: "PETSc users list" Sent: Saturday, September 14, 2013 9:18:44 PM Subject: Re: [petsc-users] The product of two MPIDENSE matrices Petsc does not support C=A x B^T for MPIDENSE_ MPIDENSE matrices, but supports it via elemental interface, see http://code.google.com/p/elemental/ and ~petsc//src/mat/impls/elemental/matelem.cxx. Hong On Sat, Sep 14, 2013 at 12:47 AM, Joon Hee Choi < choi240 at purdue.edu > wrote: Hello, I want to calculate M = C x C^T. C is a MPIDENSE matrix. I tried the calculation with MatTransposeMatMult (C, C, MAT_INITIAL_MATRIX, PETSC_DEFAULT, &M), but it did not support MPIDENSE. Is there any method to calculate the product of two MPIDENSE matrices? Thanks, Joon From jedbrown at mcs.anl.gov Mon Sep 30 13:36:26 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 30 Sep 2013 13:36:26 -0500 Subject: [petsc-users] The product of two MPIDENSE matrices In-Reply-To: <1528036471.157490.1380562114604.JavaMail.root@mailhub028.itcs.purdue.edu> References: <1528036471.157490.1380562114604.JavaMail.root@mailhub028.itcs.purdue.edu> Message-ID: <87bo3a88d1.fsf@mcs.anl.gov> Joon Hee Choi writes: > Hello Hong, > > I tried to get Elemental but it does not support 64-bit integer now. I > need to use 64-bit integer because the matrix I am using is very > large. I need just MatMatMult() for MPIDENSE matrices, but it also > didn't support MPIDENSE. Is there any method to multiply MPIDENSE and > MPIDENSE? I take it that you need 64-bit indices because your problems are that big already, not that you're worried they will eventually be that large? I assume Jack will reply to your other message which was cross-posted to dev at libelemental.org, but I'm Cc'ing him here as well. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From poulson at stanford.edu Mon Sep 30 13:45:48 2013 From: poulson at stanford.edu (Jack Poulson) Date: Mon, 30 Sep 2013 11:45:48 -0700 Subject: [petsc-users] The product of two MPIDENSE matrices In-Reply-To: <87bo3a88d1.fsf@mcs.anl.gov> References: <1528036471.157490.1380562114604.JavaMail.root@mailhub028.itcs.purdue.edu> <87bo3a88d1.fsf@mcs.anl.gov> Message-ID: <5249C6DC.4080503@stanford.edu> On 09/30/2013 11:36 AM, Jed Brown wrote: > Joon Hee Choi writes: > >> Hello Hong, >> >> I tried to get Elemental but it does not support 64-bit integer >> now. I need to use 64-bit integer because the matrix I am using >> is very large. I need just MatMatMult() for MPIDENSE matrices, >> but it also didn't support MPIDENSE. Is there any method to >> multiply MPIDENSE and MPIDENSE? > > I take it that you need 64-bit indices because your problems are > that big already, not that you're worried they will eventually be > that large? > > I assume Jack will reply to your other message which was > cross-posted to dev at libelemental.org, but I'm Cc'ing him here as > well. > It hasn't yet been rigorously tested, but support for 64-bit integers should be possible by simply changing the typedefs for elem::Int and elem::Unsigned from 'int' -> 'long long int' and 'unsigned' to 'long long unsigned'. These are guaranteed to be 64-bit by C++11. https://github.com/elemental/Elemental/blob/f288c2b50b0ad2c95c99591918d8fa27990167d2/include/elemental/core/types_decl.hpp#L17 Elemental's Python interface does not yet support this modification, but I would expect it to be straightforward to extend the PETSc interface to do so. I don't believe that this change made it into version 0.81 of Elemental, but 0.82 will be released as soon as I finish rounding the edges of a new Bunch-Kaufman implementation, which would also be good to expose an interface for in PETSc. Jack From jedbrown at mcs.anl.gov Mon Sep 30 14:26:56 2013 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 30 Sep 2013 14:26:56 -0500 Subject: [petsc-users] The product of two MPIDENSE matrices In-Reply-To: <5249C6DC.4080503@stanford.edu> References: <1528036471.157490.1380562114604.JavaMail.root@mailhub028.itcs.purdue.edu> <87bo3a88d1.fsf@mcs.anl.gov> <5249C6DC.4080503@stanford.edu> Message-ID: <8761ti860v.fsf@mcs.anl.gov> Jack Poulson writes: > It hasn't yet been rigorously tested, but support for 64-bit integers > should be possible by simply changing the typedefs for elem::Int and > elem::Unsigned from 'int' -> 'long long int' and 'unsigned' to 'long > long unsigned'. These are guaranteed to be 64-bit by C++11. Can you add a CMake configure option for these? I'd rather --download-elemental use an interface other than stream-editing headers. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From zhenglun.wei at gmail.com Mon Sep 30 17:40:14 2013 From: zhenglun.wei at gmail.com (Alan Z. Wei) Date: Mon, 30 Sep 2013 17:40:14 -0500 Subject: [petsc-users] Out of memory while using /src/src/ksp/example/tutorial/ex45.c Message-ID: <5249FDCE.4070503@gmail.com> Dear all, I hope you're having a nice day. I came across a memory problem while using ex45.c to solve 3D Poisson equation. As recommended while using 2D Poisson solver (e.g /src/src/ksp/example/tutorial/ex29.c), I used the multigrid preconditioner and the setting is listed here: mpiexec -np 8 ./ex45 -pc_type mg -ksp_type cg -da_refine 2 -pc_mg_galerkin -ksp_rtol 1.0e-7 -mg_levels_pc_type jacobi -mg_levels_ksp_type chebyshev If I apply the same settings to ex45 with np = 32 and it results a 'out of memory' error. The case has grid size = 801*385*385. Of course, if I reduced the grid size to very small, it would work well. For the test case, I tried to play with 1, doubleing the number of process (by using np =64, total avaiable memory should be aroud 128G) 2, reducing the grid size to half 3, reducing the level of the MG to 1 They do not help at all. What should I output in order to detect the problem? BTW, the ex45 works well with the same amount of DoF with GAMG and the following settings however it is slow, mpiexec -np 32 ./ex45 -pc_type gamg -ksp_type cg -pc_gamg_agg_nsmooths 1 -mg_levels_ksp_max_it 1 -mg_levels_ksp_type richardson -ksp_rtol 1.0e-7. I understand that MG requires more memory than GAMG; yet, is there any way I can approximately calculate the memory required? Thanks in advance, Alan From poulson at stanford.edu Mon Sep 30 18:59:22 2013 From: poulson at stanford.edu (Jack Poulson) Date: Mon, 30 Sep 2013 16:59:22 -0700 Subject: [petsc-users] The product of two MPIDENSE matrices In-Reply-To: <8761ti860v.fsf@mcs.anl.gov> References: <1528036471.157490.1380562114604.JavaMail.root@mailhub028.itcs.purdue.edu> <87bo3a88d1.fsf@mcs.anl.gov> <5249C6DC.4080503@stanford.edu> <8761ti860v.fsf@mcs.anl.gov> Message-ID: <524A105A.6030303@stanford.edu> On 09/30/2013 12:26 PM, Jed Brown wrote: > Jack Poulson writes: >> It hasn't yet been rigorously tested, but support for 64-bit >> integers should be possible by simply changing the typedefs for >> elem::Int and elem::Unsigned from 'int' -> 'long long int' and >> 'unsigned' to 'long long unsigned'. These are guaranteed to be >> 64-bit by C++11. > > Can you add a CMake configure option for these? I'd rather > --download-elemental use an interface other than stream-editing > headers. > Done: https://github.com/elemental/Elemental/commit/03d864e2a8ae01877194bbb572144568454f7d05 Configuring Elemental with the "-D USE_64BIT_INTS=ON" option will now use 'long long' integers. Also, the preceding commit essentially finished the Bunch-Kaufman support: https://github.com/elemental/Elemental/commit/d2c79c6cfddab6c2ad9c84c49a0ee23b8fe27893 Jack From mrosso at uci.edu Mon Sep 30 20:00:18 2013 From: mrosso at uci.edu (Michele Rosso) Date: Mon, 30 Sep 2013 18:00:18 -0700 Subject: [petsc-users] How to speed up geometric multigrid In-Reply-To: <60076751-799B-410A-9779-D697677A869D@mcs.anl.gov> References: <52406460.3060905@uci.edu> <4499926F-40CE-462C-BFD6-8798A02126FB@mcs.anl.gov> <52420449.4010603@uci.edu> <60076751-799B-410A-9779-D697677A869D@mcs.anl.gov> Message-ID: <524A1EA2.8090606@uci.edu> Barry, sorry again for the very late answer. I tried all the variations you proposed: all of them converge very slow , except the last one (CG instead of fgmres) diverges. I attached the diagnostics for the two options that convergence: each of the attached files starts with a list of the options I used for the run. As you pointed out earlier, the residual I used to require was too small, therefore I increased atol to e-9. After some tests, I noticed that any further increase of the absolute tolerance changes significantly the solution. What would you suggest to try next? Thank you very much, Michele On 09/24/2013 05:08 PM, Barry Smith wrote: > Thanks. The balance of work on the different levels and across processes looks ok. So it is is a matter of improving the convergence rate. > > The initial residual norm is very small. Are you sure you need to decrease it to 10^-12 ???? > > Start with a really robust multigrid smoother use > > -pc_mg_type full -ksp_type fgmres -mg_levels_pc_type bjacobi -mg_levels_ksp_type gmres -mg_levels_ksp_max_it 3 PLUS -mg_coarse_pc_factor_mat_solver_package superlu_dist > -mg_coarse_pc_type lu -pc_mg_galerkin -pc_mg_levels 5 -pc_mg_log -pc_type mg > > run with the -log_summary and -pc_mg_log > > Now back off a little on the smoother and use -mg_levels_pc_type sor instead how does that change the convergence and time. > > Back off even more an replace the -ksp_type fgmres with -ksp_type cg and the -mg_levels_ksp_type gmres with -mg_levels_ksp_type richardson how does that change the convergence and the time? > > There are some additional variants we can try based on the results from above. > > Barry > > > > On Sep 24, 2013, at 4:29 PM, Michele Rosso wrote: > >> Barry, >> >> I re-rerun the test case with the option -pc_mg_log as you suggested. >> I attached the new output ("final_new.txt'). >> Thanks for your help. >> >> Michele >> >> On 09/23/2013 09:35 AM, Barry Smith wrote: >>> Run with the additional option -pc_mg_log and send us the log file. >>> >>> Barry >>> >>> Maybe we should make this the default somehow. >>> >>> >>> On Sep 23, 2013, at 10:55 AM, Michele Rosso wrote: >>> >>>> Hi, >>>> >>>> I am successfully using PETSc to solve a 3D Poisson's equation with CG + MG . Such equation arises from a projection algorithm for a multiphase incompressible flow simulation. >>>> I set up the solver as I was suggested to do in a previous thread (title: "GAMG speed") and run a test case (liquid droplet with surface tension falling under the effect of gravity in a quiescent fluid). >>>> The solution of the Poisson Equation via multigrid is correct but it becomes progressively slower and slower as the simulation progresses (I am performing successive solves) due to an increase in the number of iterations. >>>> Since the solution of the Poisson equation is mission-critical, I need to speed it up as much as I can. >>>> Could you please help me out with this? >>>> >>>> I run the test case with the following options: >>>> >>>> -pc_type mg -pc_mg_galerkin -pc_mg_levels 5 -mg_levels_ksp_type richardson -mg_levels_ksp_max_it 1 >>>> -mg_coarse_pc_type lu -mg_coarse_pc_factor_mat_solver_package superlu_dist >>>> -log_summary -ksp_view -ksp_monitor_true_residual -options_left >>>> >>>> Please find the diagnostic for the final solve in the attached file "final.txt'. >>>> Thank you, >>>> >>>> Michele >>>> >> > -------------- next part -------------- -pc_mg_type full -ksp_type fgmres -mg_levels_pc_type bjacobi -mg_levels_ksp_type gmres -mg_levels_ksp_max_it 3 -mg_coarse_pc_factor_mat_solver_package superlu_dist -mg_coarse_pc_type lu -pc_mg_galerkin -pc_mg_levels 5 -pc_type mg -log_summary -pc_mg_log -ksp_monitor_true_residual -options_left 0 KSP unpreconditioned resid norm 2.452212453860e-07 true resid norm 2.452212453860e-07 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.419822158980e-07 true resid norm 2.419822158980e-07 ||r(i)||/||b|| 9.867913994036e-01 2 KSP unpreconditioned resid norm 1.667048646853e-07 true resid norm 1.667048646853e-07 ||r(i)||/||b|| 6.798141181563e-01 3 KSP unpreconditioned resid norm 1.358535510529e-07 true resid norm 1.358535510529e-07 ||r(i)||/||b|| 5.540040009136e-01 4 KSP unpreconditioned resid norm 1.354895642983e-07 true resid norm 1.354895642983e-07 ||r(i)||/||b|| 5.525196810946e-01 5 KSP unpreconditioned resid norm 1.244851791906e-07 true resid norm 1.244851791906e-07 ||r(i)||/||b|| 5.076443478402e-01 6 KSP unpreconditioned resid norm 1.173350897701e-07 true resid norm 1.173350897701e-07 ||r(i)||/||b|| 4.784866400357e-01 7 KSP unpreconditioned resid norm 1.173017401976e-07 true resid norm 1.173017401976e-07 ||r(i)||/||b|| 4.783506421433e-01 8 KSP unpreconditioned resid norm 1.105615956213e-07 true resid norm 1.105615956213e-07 ||r(i)||/||b|| 4.508646689531e-01 9 KSP unpreconditioned resid norm 9.643586842608e-08 true resid norm 9.643586842608e-08 ||r(i)||/||b|| 3.932606584485e-01 10 KSP unpreconditioned resid norm 9.612358457319e-08 true resid norm 9.612358457319e-08 ||r(i)||/||b|| 3.919871804821e-01 11 KSP unpreconditioned resid norm 8.229327150626e-08 true resid norm 8.229327150626e-08 ||r(i)||/||b|| 3.355878540487e-01 12 KSP unpreconditioned resid norm 7.938773453245e-08 true resid norm 7.938773453245e-08 ||r(i)||/||b|| 3.237392192812e-01 13 KSP unpreconditioned resid norm 7.935125220727e-08 true resid norm 7.935125220727e-08 ||r(i)||/||b|| 3.235904461799e-01 14 KSP unpreconditioned resid norm 7.335324537068e-08 true resid norm 7.335324537068e-08 ||r(i)||/||b|| 2.991308736533e-01 15 KSP unpreconditioned resid norm 7.197373101672e-08 true resid norm 7.197373101672e-08 ||r(i)||/||b|| 2.935052829677e-01 16 KSP unpreconditioned resid norm 7.016384861756e-08 true resid norm 7.016384861756e-08 ||r(i)||/||b|| 2.861246728729e-01 17 KSP unpreconditioned resid norm 5.944871806930e-08 true resid norm 5.944871806929e-08 ||r(i)||/||b|| 2.424289052757e-01 18 KSP unpreconditioned resid norm 5.922418197663e-08 true resid norm 5.922418197663e-08 ||r(i)||/||b|| 2.415132582962e-01 19 KSP unpreconditioned resid norm 5.614577237297e-08 true resid norm 5.614577237297e-08 ||r(i)||/||b|| 2.289596575720e-01 20 KSP unpreconditioned resid norm 5.543683039338e-08 true resid norm 5.543683039338e-08 ||r(i)||/||b|| 2.260686275616e-01 21 KSP unpreconditioned resid norm 5.537009509960e-08 true resid norm 5.537009509960e-08 ||r(i)||/||b|| 2.257964843643e-01 22 KSP unpreconditioned resid norm 5.440757674516e-08 true resid norm 5.440757674516e-08 ||r(i)||/||b|| 2.218713825529e-01 23 KSP unpreconditioned resid norm 5.262955197017e-08 true resid norm 5.262955197017e-08 ||r(i)||/||b|| 2.146206862595e-01 24 KSP unpreconditioned resid norm 5.185946887388e-08 true resid norm 5.185946887388e-08 ||r(i)||/||b|| 2.114803258267e-01 25 KSP unpreconditioned resid norm 4.821365539193e-08 true resid norm 4.821365539193e-08 ||r(i)||/||b|| 1.966128804053e-01 26 KSP unpreconditioned resid norm 4.809210785109e-08 true resid norm 4.809210785109e-08 ||r(i)||/||b|| 1.961172156001e-01 27 KSP unpreconditioned resid norm 4.785689974287e-08 true resid norm 4.785689974287e-08 ||r(i)||/||b|| 1.951580486737e-01 28 KSP unpreconditioned resid norm 4.699105156134e-08 true resid norm 4.699105156134e-08 ||r(i)||/||b|| 1.916271630028e-01 29 KSP unpreconditioned resid norm 4.667740913574e-08 true resid norm 4.667740913574e-08 ||r(i)||/||b|| 1.903481448447e-01 30 KSP unpreconditioned resid norm 4.495092087718e-08 true resid norm 4.495092087718e-08 ||r(i)||/||b|| 1.833076118931e-01 31 KSP unpreconditioned resid norm 4.485095979916e-08 true resid norm 4.485095979916e-08 ||r(i)||/||b|| 1.828999756059e-01 32 KSP unpreconditioned resid norm 4.433072229482e-08 true resid norm 4.433072229482e-08 ||r(i)||/||b|| 1.807784730276e-01 33 KSP unpreconditioned resid norm 4.381505863646e-08 true resid norm 4.381505863646e-08 ||r(i)||/||b|| 1.786756223650e-01 34 KSP unpreconditioned resid norm 4.367664103883e-08 true resid norm 4.367664103883e-08 ||r(i)||/||b|| 1.781111623101e-01 35 KSP unpreconditioned resid norm 4.341427566282e-08 true resid norm 4.341427566282e-08 ||r(i)||/||b|| 1.770412494010e-01 36 KSP unpreconditioned resid norm 4.330641141605e-08 true resid norm 4.330641141605e-08 ||r(i)||/||b|| 1.766013843861e-01 37 KSP unpreconditioned resid norm 4.318853001779e-08 true resid norm 4.318853001779e-08 ||r(i)||/||b|| 1.761206699273e-01 38 KSP unpreconditioned resid norm 4.250324334124e-08 true resid norm 4.250324334124e-08 ||r(i)||/||b|| 1.733261050621e-01 39 KSP unpreconditioned resid norm 4.232592940157e-08 true resid norm 4.232592940157e-08 ||r(i)||/||b|| 1.726030276656e-01 40 KSP unpreconditioned resid norm 4.223089367169e-08 true resid norm 4.223089367169e-08 ||r(i)||/||b|| 1.722154767023e-01 41 KSP unpreconditioned resid norm 4.186456912966e-08 true resid norm 4.186456912966e-08 ||r(i)||/||b|| 1.707216235027e-01 42 KSP unpreconditioned resid norm 4.179398678087e-08 true resid norm 4.179398678087e-08 ||r(i)||/||b|| 1.704337922070e-01 43 KSP unpreconditioned resid norm 4.167238168835e-08 true resid norm 4.167238168835e-08 ||r(i)||/||b|| 1.699378927089e-01 44 KSP unpreconditioned resid norm 4.090901142391e-08 true resid norm 4.090901142391e-08 ||r(i)||/||b|| 1.668249068693e-01 45 KSP unpreconditioned resid norm 3.975611201184e-08 true resid norm 3.975611201184e-08 ||r(i)||/||b|| 1.621234406067e-01 46 KSP unpreconditioned resid norm 3.924101286981e-08 true resid norm 3.924101286981e-08 ||r(i)||/||b|| 1.600228920134e-01 47 KSP unpreconditioned resid norm 3.521904644775e-08 true resid norm 3.521904644775e-08 ||r(i)||/||b|| 1.436215136756e-01 48 KSP unpreconditioned resid norm 3.498324602837e-08 true resid norm 3.498324602837e-08 ||r(i)||/||b|| 1.426599313338e-01 49 KSP unpreconditioned resid norm 3.459067870011e-08 true resid norm 3.459067870011e-08 ||r(i)||/||b|| 1.410590613617e-01 50 KSP unpreconditioned resid norm 3.399462215977e-08 true resid norm 3.399462215977e-08 ||r(i)||/||b|| 1.386283725387e-01 51 KSP unpreconditioned resid norm 3.399430398445e-08 true resid norm 3.399430398445e-08 ||r(i)||/||b|| 1.386270750356e-01 52 KSP unpreconditioned resid norm 3.346869309028e-08 true resid norm 3.346869309028e-08 ||r(i)||/||b|| 1.364836600417e-01 53 KSP unpreconditioned resid norm 3.138744643238e-08 true resid norm 3.138744643238e-08 ||r(i)||/||b|| 1.279964400432e-01 54 KSP unpreconditioned resid norm 3.128257969232e-08 true resid norm 3.128257969232e-08 ||r(i)||/||b|| 1.275687987110e-01 55 KSP unpreconditioned resid norm 2.821479391120e-08 true resid norm 2.821479391120e-08 ||r(i)||/||b|| 1.150585214050e-01 56 KSP unpreconditioned resid norm 2.564797554339e-08 true resid norm 2.564797554339e-08 ||r(i)||/||b|| 1.045911642077e-01 57 KSP unpreconditioned resid norm 2.504481002459e-08 true resid norm 2.504481002459e-08 ||r(i)||/||b|| 1.021314853253e-01 58 KSP unpreconditioned resid norm 2.242451149879e-08 true resid norm 2.242451149879e-08 ||r(i)||/||b|| 9.144603871285e-02 59 KSP unpreconditioned resid norm 2.093393512438e-08 true resid norm 2.093393512438e-08 ||r(i)||/||b|| 8.536754265081e-02 60 KSP unpreconditioned resid norm 2.047371164412e-08 true resid norm 2.047371164412e-08 ||r(i)||/||b|| 8.349077426749e-02 61 KSP unpreconditioned resid norm 2.017207880642e-08 true resid norm 2.017207880642e-08 ||r(i)||/||b|| 8.226073060948e-02 62 KSP unpreconditioned resid norm 1.996642911112e-08 true resid norm 1.996642911112e-08 ||r(i)||/||b|| 8.142210141575e-02 63 KSP unpreconditioned resid norm 1.949671274709e-08 true resid norm 1.949671274709e-08 ||r(i)||/||b|| 7.950662152618e-02 64 KSP unpreconditioned resid norm 1.908868151489e-08 true resid norm 1.908868151489e-08 ||r(i)||/||b|| 7.784269052561e-02 65 KSP unpreconditioned resid norm 1.835547412807e-08 true resid norm 1.835547412807e-08 ||r(i)||/||b|| 7.485270739562e-02 66 KSP unpreconditioned resid norm 1.699752112694e-08 true resid norm 1.699752112694e-08 ||r(i)||/||b|| 6.931504283073e-02 67 KSP unpreconditioned resid norm 1.699326819014e-08 true resid norm 1.699326819014e-08 ||r(i)||/||b|| 6.929769956673e-02 68 KSP unpreconditioned resid norm 1.676964398385e-08 true resid norm 1.676964398385e-08 ||r(i)||/||b|| 6.838577121428e-02 69 KSP unpreconditioned resid norm 1.668869881419e-08 true resid norm 1.668869881419e-08 ||r(i)||/||b|| 6.805568085228e-02 70 KSP unpreconditioned resid norm 1.667200369803e-08 true resid norm 1.667200369803e-08 ||r(i)||/||b|| 6.798759900182e-02 71 KSP unpreconditioned resid norm 1.656208779511e-08 true resid norm 1.656208779511e-08 ||r(i)||/||b|| 6.753936743546e-02 72 KSP unpreconditioned resid norm 1.626050460200e-08 true resid norm 1.626050460200e-08 ||r(i)||/||b|| 6.630952622560e-02 73 KSP unpreconditioned resid norm 1.624495256069e-08 true resid norm 1.624495256069e-08 ||r(i)||/||b|| 6.624610577731e-02 74 KSP unpreconditioned resid norm 1.580439129234e-08 true resid norm 1.580439129234e-08 ||r(i)||/||b|| 6.444951891286e-02 75 KSP unpreconditioned resid norm 1.528210351844e-08 true resid norm 1.528210351844e-08 ||r(i)||/||b|| 6.231965543763e-02 76 KSP unpreconditioned resid norm 1.521637918567e-08 true resid norm 1.521637918567e-08 ||r(i)||/||b|| 6.205163488878e-02 77 KSP unpreconditioned resid norm 1.510418394672e-08 true resid norm 1.510418394672e-08 ||r(i)||/||b|| 6.159410830387e-02 78 KSP unpreconditioned resid norm 1.448733751222e-08 true resid norm 1.448733751222e-08 ||r(i)||/||b|| 5.907863933003e-02 79 KSP unpreconditioned resid norm 1.440897586029e-08 true resid norm 1.440897586029e-08 ||r(i)||/||b|| 5.875908442440e-02 80 KSP unpreconditioned resid norm 1.432288207531e-08 true resid norm 1.432288207531e-08 ||r(i)||/||b|| 5.840799826605e-02 81 KSP unpreconditioned resid norm 1.397436573103e-08 true resid norm 1.397436573103e-08 ||r(i)||/||b|| 5.698676600810e-02 82 KSP unpreconditioned resid norm 1.388657061475e-08 true resid norm 1.388657061475e-08 ||r(i)||/||b|| 5.662874190564e-02 83 KSP unpreconditioned resid norm 1.359722455182e-08 true resid norm 1.359722455182e-08 ||r(i)||/||b|| 5.544880310191e-02 84 KSP unpreconditioned resid norm 1.265365292676e-08 true resid norm 1.265365292676e-08 ||r(i)||/||b|| 5.160096510739e-02 85 KSP unpreconditioned resid norm 1.259945408065e-08 true resid norm 1.259945408065e-08 ||r(i)||/||b|| 5.137994491795e-02 86 KSP unpreconditioned resid norm 1.201823818499e-08 true resid norm 1.201823818499e-08 ||r(i)||/||b|| 4.900977550321e-02 87 KSP unpreconditioned resid norm 1.189035093877e-08 true resid norm 1.189035093877e-08 ||r(i)||/||b|| 4.848825769583e-02 88 KSP unpreconditioned resid norm 1.158499778932e-08 true resid norm 1.158499778932e-08 ||r(i)||/||b|| 4.724304279216e-02 89 KSP unpreconditioned resid norm 1.133107493622e-08 true resid norm 1.133107493622e-08 ||r(i)||/||b|| 4.620755807019e-02 90 KSP unpreconditioned resid norm 1.125101113680e-08 true resid norm 1.125101113680e-08 ||r(i)||/||b|| 4.588106189205e-02 91 KSP unpreconditioned resid norm 1.117730224530e-08 true resid norm 1.117730224530e-08 ||r(i)||/||b|| 4.558048071123e-02 92 KSP unpreconditioned resid norm 1.098216364313e-08 true resid norm 1.098216364313e-08 ||r(i)||/||b|| 4.478471523068e-02 93 KSP unpreconditioned resid norm 1.081908478633e-08 true resid norm 1.081908478633e-08 ||r(i)||/||b|| 4.411968779173e-02 94 KSP unpreconditioned resid norm 1.074175388349e-08 true resid norm 1.074175388349e-08 ||r(i)||/||b|| 4.380433622941e-02 95 KSP unpreconditioned resid norm 1.044866647328e-08 true resid norm 1.044866647328e-08 ||r(i)||/||b|| 4.260914039822e-02 96 KSP unpreconditioned resid norm 1.043552109077e-08 true resid norm 1.043552109077e-08 ||r(i)||/||b|| 4.255553418443e-02 97 KSP unpreconditioned resid norm 9.830235360062e-09 true resid norm 9.830235360062e-09 ||r(i)||/||b|| 4.008720918363e-02 98 KSP unpreconditioned resid norm 9.521119236860e-09 true resid norm 9.521119236860e-09 ||r(i)||/||b|| 3.882664906081e-02 99 KSP unpreconditioned resid norm 9.520911974750e-09 true resid norm 9.520911974750e-09 ||r(i)||/||b|| 3.882580385628e-02 100 KSP unpreconditioned resid norm 9.376716676094e-09 true resid norm 9.376716676094e-09 ||r(i)||/||b|| 3.823778262496e-02 101 KSP unpreconditioned resid norm 9.206346340775e-09 true resid norm 9.206346340775e-09 ||r(i)||/||b|| 3.754302090050e-02 102 KSP unpreconditioned resid norm 9.147193139051e-09 true resid norm 9.147193139051e-09 ||r(i)||/||b|| 3.730179709613e-02 103 KSP unpreconditioned resid norm 9.056811926003e-09 true resid norm 9.056811926003e-09 ||r(i)||/||b|| 3.693322702013e-02 104 KSP unpreconditioned resid norm 8.936944824040e-09 true resid norm 8.936944824040e-09 ||r(i)||/||b|| 3.644441496075e-02 105 KSP unpreconditioned resid norm 8.934739148676e-09 true resid norm 8.934739148676e-09 ||r(i)||/||b|| 3.643542032669e-02 106 KSP unpreconditioned resid norm 8.850749678925e-09 true resid norm 8.850749678925e-09 ||r(i)||/||b|| 3.609291546086e-02 107 KSP unpreconditioned resid norm 8.679435160414e-09 true resid norm 8.679435160414e-09 ||r(i)||/||b|| 3.539430340446e-02 108 KSP unpreconditioned resid norm 8.657752776426e-09 true resid norm 8.657752776426e-09 ||r(i)||/||b|| 3.530588372470e-02 109 KSP unpreconditioned resid norm 8.514323923495e-09 true resid norm 8.514323923495e-09 ||r(i)||/||b|| 3.472098802080e-02 110 KSP unpreconditioned resid norm 8.281813089698e-09 true resid norm 8.281813089698e-09 ||r(i)||/||b|| 3.377282044490e-02 111 KSP unpreconditioned resid norm 8.256092263525e-09 true resid norm 8.256092263525e-09 ||r(i)||/||b|| 3.366793219946e-02 112 KSP unpreconditioned resid norm 8.249978289270e-09 true resid norm 8.249978289270e-09 ||r(i)||/||b|| 3.364299971759e-02 113 KSP unpreconditioned resid norm 8.018171173819e-09 true resid norm 8.018171173819e-09 ||r(i)||/||b|| 3.269770186998e-02 114 KSP unpreconditioned resid norm 7.989855578933e-09 true resid norm 7.989855578933e-09 ||r(i)||/||b|| 3.258223228724e-02 115 KSP unpreconditioned resid norm 7.459893032882e-09 true resid norm 7.459893032882e-09 ||r(i)||/||b|| 3.042107147420e-02 116 KSP unpreconditioned resid norm 7.202291666167e-09 true resid norm 7.202291666167e-09 ||r(i)||/||b|| 2.937058595730e-02 117 KSP unpreconditioned resid norm 7.170313473198e-09 true resid norm 7.170313473198e-09 ||r(i)||/||b|| 2.924018048236e-02 118 KSP unpreconditioned resid norm 6.972218265192e-09 true resid norm 6.972218265192e-09 ||r(i)||/||b|| 2.843235811080e-02 119 KSP unpreconditioned resid norm 6.635512376660e-09 true resid norm 6.635512376660e-09 ||r(i)||/||b|| 2.705928830193e-02 120 KSP unpreconditioned resid norm 6.549929915762e-09 true resid norm 6.549929915762e-09 ||r(i)||/||b|| 2.671028729770e-02 121 KSP unpreconditioned resid norm 6.470318250528e-09 true resid norm 6.470318250528e-09 ||r(i)||/||b|| 2.638563490020e-02 122 KSP unpreconditioned resid norm 6.241392498551e-09 true resid norm 6.241392498551e-09 ||r(i)||/||b|| 2.545208710903e-02 123 KSP unpreconditioned resid norm 6.128754651656e-09 true resid norm 6.128754651656e-09 ||r(i)||/||b|| 2.499275559101e-02 124 KSP unpreconditioned resid norm 6.110276397128e-09 true resid norm 6.110276397128e-09 ||r(i)||/||b|| 2.491740219127e-02 125 KSP unpreconditioned resid norm 5.954851894569e-09 true resid norm 5.954851894569e-09 ||r(i)||/||b|| 2.428358882688e-02 126 KSP unpreconditioned resid norm 5.797967509678e-09 true resid norm 5.797967509678e-09 ||r(i)||/||b|| 2.364382213519e-02 127 KSP unpreconditioned resid norm 5.767558700754e-09 true resid norm 5.767558700754e-09 ||r(i)||/||b|| 2.351981653007e-02 128 KSP unpreconditioned resid norm 5.639121572750e-09 true resid norm 5.639121572750e-09 ||r(i)||/||b|| 2.299605633220e-02 129 KSP unpreconditioned resid norm 5.624366897720e-09 true resid norm 5.624366897720e-09 ||r(i)||/||b|| 2.293588750381e-02 130 KSP unpreconditioned resid norm 5.608499860222e-09 true resid norm 5.608499860222e-09 ||r(i)||/||b|| 2.287118251681e-02 131 KSP unpreconditioned resid norm 5.303626608640e-09 true resid norm 5.303626608640e-09 ||r(i)||/||b|| 2.162792461269e-02 132 KSP unpreconditioned resid norm 5.207681722116e-09 true resid norm 5.207681722116e-09 ||r(i)||/||b|| 2.123666615394e-02 133 KSP unpreconditioned resid norm 5.201502631855e-09 true resid norm 5.201502631855e-09 ||r(i)||/||b|| 2.121146813224e-02 134 KSP unpreconditioned resid norm 5.131488055171e-09 true resid norm 5.131488055171e-09 ||r(i)||/||b|| 2.092595218286e-02 135 KSP unpreconditioned resid norm 4.975360345852e-09 true resid norm 4.975360345852e-09 ||r(i)||/||b|| 2.028927117640e-02 136 KSP unpreconditioned resid norm 4.952631636049e-09 true resid norm 4.952631636048e-09 ||r(i)||/||b|| 2.019658463218e-02 137 KSP unpreconditioned resid norm 4.910518190967e-09 true resid norm 4.910518190967e-09 ||r(i)||/||b|| 2.002484810497e-02 138 KSP unpreconditioned resid norm 4.828869410474e-09 true resid norm 4.828869410474e-09 ||r(i)||/||b|| 1.969188845311e-02 139 KSP unpreconditioned resid norm 4.828706549393e-09 true resid norm 4.828706549393e-09 ||r(i)||/||b|| 1.969122431375e-02 140 KSP unpreconditioned resid norm 4.786379052905e-09 true resid norm 4.786379052905e-09 ||r(i)||/||b|| 1.951861489558e-02 141 KSP unpreconditioned resid norm 4.681977202199e-09 true resid norm 4.681977202199e-09 ||r(i)||/||b|| 1.909286935897e-02 142 KSP unpreconditioned resid norm 4.662789271598e-09 true resid norm 4.662789271598e-09 ||r(i)||/||b|| 1.901462193562e-02 143 KSP unpreconditioned resid norm 4.522398642038e-09 true resid norm 4.522398642038e-09 ||r(i)||/||b|| 1.844211595500e-02 144 KSP unpreconditioned resid norm 4.274697680856e-09 true resid norm 4.274697680855e-09 ||r(i)||/||b|| 1.743200379774e-02 145 KSP unpreconditioned resid norm 4.265555182135e-09 true resid norm 4.265555182135e-09 ||r(i)||/||b|| 1.739472114425e-02 146 KSP unpreconditioned resid norm 4.207412975160e-09 true resid norm 4.207412975160e-09 ||r(i)||/||b|| 1.715762012601e-02 147 KSP unpreconditioned resid norm 4.154016179231e-09 true resid norm 4.154016179231e-09 ||r(i)||/||b|| 1.693987065718e-02 148 KSP unpreconditioned resid norm 4.131742489350e-09 true resid norm 4.131742489350e-09 ||r(i)||/||b|| 1.684903966149e-02 149 KSP unpreconditioned resid norm 4.043172138715e-09 true resid norm 4.043172138715e-09 ||r(i)||/||b|| 1.648785419204e-02 150 KSP unpreconditioned resid norm 3.939168395943e-09 true resid norm 3.939168395943e-09 ||r(i)||/||b|| 1.606373211971e-02 151 KSP unpreconditioned resid norm 3.857606362584e-09 true resid norm 3.857606362584e-09 ||r(i)||/||b|| 1.573112621833e-02 152 KSP unpreconditioned resid norm 3.793151011702e-09 true resid norm 3.793151011702e-09 ||r(i)||/||b|| 1.546828051432e-02 153 KSP unpreconditioned resid norm 3.771214716922e-09 true resid norm 3.771214716922e-09 ||r(i)||/||b|| 1.537882539902e-02 154 KSP unpreconditioned resid norm 3.744927773353e-09 true resid norm 3.744927773353e-09 ||r(i)||/||b|| 1.527162855509e-02 155 KSP unpreconditioned resid norm 3.699554989589e-09 true resid norm 3.699554989589e-09 ||r(i)||/||b|| 1.508660060740e-02 156 KSP unpreconditioned resid norm 3.699553806166e-09 true resid norm 3.699553806166e-09 ||r(i)||/||b|| 1.508659578146e-02 157 KSP unpreconditioned resid norm 3.578721024518e-09 true resid norm 3.578721024518e-09 ||r(i)||/||b|| 1.459384572851e-02 158 KSP unpreconditioned resid norm 3.468771260320e-09 true resid norm 3.468771260320e-09 ||r(i)||/||b|| 1.414547607757e-02 159 KSP unpreconditioned resid norm 3.467025485554e-09 true resid norm 3.467025485554e-09 ||r(i)||/||b|| 1.413835689520e-02 160 KSP unpreconditioned resid norm 3.400003173641e-09 true resid norm 3.400003173641e-09 ||r(i)||/||b|| 1.386504325222e-02 161 KSP unpreconditioned resid norm 3.357061147998e-09 true resid norm 3.357061147998e-09 ||r(i)||/||b|| 1.368992781483e-02 162 KSP unpreconditioned resid norm 3.356049647642e-09 true resid norm 3.356049647642e-09 ||r(i)||/||b|| 1.368580296686e-02 163 KSP unpreconditioned resid norm 3.284283436860e-09 true resid norm 3.284283436860e-09 ||r(i)||/||b|| 1.339314394105e-02 164 KSP unpreconditioned resid norm 3.236517913866e-09 true resid norm 3.236517913866e-09 ||r(i)||/||b|| 1.319835852220e-02 165 KSP unpreconditioned resid norm 3.226209835937e-09 true resid norm 3.226209835937e-09 ||r(i)||/||b|| 1.315632269487e-02 166 KSP unpreconditioned resid norm 3.153468551104e-09 true resid norm 3.153468551104e-09 ||r(i)||/||b|| 1.285968736575e-02 167 KSP unpreconditioned resid norm 3.090746655863e-09 true resid norm 3.090746655863e-09 ||r(i)||/||b|| 1.260391060733e-02 168 KSP unpreconditioned resid norm 3.066178249083e-09 true resid norm 3.066178249083e-09 ||r(i)||/||b|| 1.250372187065e-02 169 KSP unpreconditioned resid norm 3.030296737504e-09 true resid norm 3.030296737504e-09 ||r(i)||/||b|| 1.235739885724e-02 170 KSP unpreconditioned resid norm 2.920555814502e-09 true resid norm 2.920555814502e-09 ||r(i)||/||b|| 1.190988085027e-02 171 KSP unpreconditioned resid norm 2.919920023392e-09 true resid norm 2.919920023392e-09 ||r(i)||/||b|| 1.190728812585e-02 172 KSP unpreconditioned resid norm 2.899203585400e-09 true resid norm 2.899203585400e-09 ||r(i)||/||b|| 1.182280752565e-02 173 KSP unpreconditioned resid norm 2.853525738559e-09 true resid norm 2.853525738559e-09 ||r(i)||/||b|| 1.163653554596e-02 174 KSP unpreconditioned resid norm 2.848599432980e-09 true resid norm 2.848599432980e-09 ||r(i)||/||b|| 1.161644631767e-02 175 KSP unpreconditioned resid norm 2.670567714904e-09 true resid norm 2.670567714905e-09 ||r(i)||/||b|| 1.089044185670e-02 176 KSP unpreconditioned resid norm 2.574460860036e-09 true resid norm 2.574460860036e-09 ||r(i)||/||b|| 1.049852289912e-02 177 KSP unpreconditioned resid norm 2.566251090751e-09 true resid norm 2.566251090751e-09 ||r(i)||/||b|| 1.046504386972e-02 178 KSP unpreconditioned resid norm 2.521090771205e-09 true resid norm 2.521090771205e-09 ||r(i)||/||b|| 1.028088234050e-02 179 KSP unpreconditioned resid norm 2.483916357622e-09 true resid norm 2.483916357622e-09 ||r(i)||/||b|| 1.012928693724e-02 180 KSP unpreconditioned resid norm 2.421953972199e-09 true resid norm 2.421953972199e-09 ||r(i)||/||b|| 9.876607421949e-03 181 KSP unpreconditioned resid norm 2.373332058238e-09 true resid norm 2.373332058238e-09 ||r(i)||/||b|| 9.678329683472e-03 182 KSP unpreconditioned resid norm 2.362195619604e-09 true resid norm 2.362195619604e-09 ||r(i)||/||b|| 9.632915842531e-03 183 KSP unpreconditioned resid norm 2.321212697334e-09 true resid norm 2.321212697334e-09 ||r(i)||/||b|| 9.465789530918e-03 184 KSP unpreconditioned resid norm 2.269721903308e-09 true resid norm 2.269721903308e-09 ||r(i)||/||b|| 9.255812642721e-03 185 KSP unpreconditioned resid norm 2.179999635151e-09 true resid norm 2.179999635151e-09 ||r(i)||/||b|| 8.889929711103e-03 186 KSP unpreconditioned resid norm 2.048752349177e-09 true resid norm 2.048752349178e-09 ||r(i)||/||b|| 8.354709829290e-03 187 KSP unpreconditioned resid norm 2.046743929968e-09 true resid norm 2.046743929968e-09 ||r(i)||/||b|| 8.346519595991e-03 188 KSP unpreconditioned resid norm 2.036272520522e-09 true resid norm 2.036272520522e-09 ||r(i)||/||b|| 8.303817710886e-03 189 KSP unpreconditioned resid norm 1.994115099266e-09 true resid norm 1.994115099266e-09 ||r(i)||/||b|| 8.131901851029e-03 190 KSP unpreconditioned resid norm 1.977275642988e-09 true resid norm 1.977275642988e-09 ||r(i)||/||b|| 8.063231388764e-03 191 KSP unpreconditioned resid norm 1.955324293191e-09 true resid norm 1.955324293191e-09 ||r(i)||/||b|| 7.973714879855e-03 192 KSP unpreconditioned resid norm 1.935585651100e-09 true resid norm 1.935585651100e-09 ||r(i)||/||b|| 7.893221682537e-03 193 KSP unpreconditioned resid norm 1.934506635159e-09 true resid norm 1.934506635159e-09 ||r(i)||/||b|| 7.888821509382e-03 194 KSP unpreconditioned resid norm 1.922907675017e-09 true resid norm 1.922907675017e-09 ||r(i)||/||b|| 7.841521528814e-03 195 KSP unpreconditioned resid norm 1.824598688118e-09 true resid norm 1.824598688118e-09 ||r(i)||/||b|| 7.440622386716e-03 196 KSP unpreconditioned resid norm 1.817944587173e-09 true resid norm 1.817944587173e-09 ||r(i)||/||b|| 7.413487295162e-03 197 KSP unpreconditioned resid norm 1.807090795128e-09 true resid norm 1.807090795128e-09 ||r(i)||/||b|| 7.369226072902e-03 198 KSP unpreconditioned resid norm 1.753669753179e-09 true resid norm 1.753669753179e-09 ||r(i)||/||b|| 7.151377730010e-03 199 KSP unpreconditioned resid norm 1.710838452332e-09 true resid norm 1.710838452332e-09 ||r(i)||/||b|| 6.976713822811e-03 200 KSP unpreconditioned resid norm 1.710770440488e-09 true resid norm 1.710770440488e-09 ||r(i)||/||b|| 6.976436473905e-03 201 KSP unpreconditioned resid norm 1.670865227850e-09 true resid norm 1.670865227850e-09 ||r(i)||/||b|| 6.813705008389e-03 202 KSP unpreconditioned resid norm 1.665513805615e-09 true resid norm 1.665513805615e-09 ||r(i)||/||b|| 6.791882175596e-03 203 KSP unpreconditioned resid norm 1.639873964672e-09 true resid norm 1.639873964672e-09 ||r(i)||/||b|| 6.687324183885e-03 204 KSP unpreconditioned resid norm 1.589069902779e-09 true resid norm 1.589069902779e-09 ||r(i)||/||b|| 6.480147755050e-03 205 KSP unpreconditioned resid norm 1.588252119589e-09 true resid norm 1.588252119589e-09 ||r(i)||/||b|| 6.476812876016e-03 206 KSP unpreconditioned resid norm 1.567411584020e-09 true resid norm 1.567411584020e-09 ||r(i)||/||b|| 6.391826212090e-03 207 KSP unpreconditioned resid norm 1.495962924807e-09 true resid norm 1.495962924807e-09 ||r(i)||/||b|| 6.100462145733e-03 208 KSP unpreconditioned resid norm 1.495926920942e-09 true resid norm 1.495926920942e-09 ||r(i)||/||b|| 6.100315323768e-03 209 KSP unpreconditioned resid norm 1.407870398173e-09 true resid norm 1.407870398173e-09 ||r(i)||/||b|| 5.741225218707e-03 210 KSP unpreconditioned resid norm 1.396979823489e-09 true resid norm 1.396979823489e-09 ||r(i)||/||b|| 5.696813998679e-03 211 KSP unpreconditioned resid norm 1.385830225181e-09 true resid norm 1.385830225181e-09 ||r(i)||/||b|| 5.651346493244e-03 212 KSP unpreconditioned resid norm 1.330892709317e-09 true resid norm 1.330892709317e-09 ||r(i)||/||b|| 5.427314045413e-03 213 KSP unpreconditioned resid norm 1.328397746461e-09 true resid norm 1.328397746461e-09 ||r(i)||/||b|| 5.417139711405e-03 214 KSP unpreconditioned resid norm 1.306080253239e-09 true resid norm 1.306080253239e-09 ||r(i)||/||b|| 5.326130087883e-03 215 KSP unpreconditioned resid norm 1.240539718062e-09 true resid norm 1.240539718062e-09 ||r(i)||/||b|| 5.058859056478e-03 216 KSP unpreconditioned resid norm 1.240539252753e-09 true resid norm 1.240539252753e-09 ||r(i)||/||b|| 5.058857158970e-03 217 KSP unpreconditioned resid norm 1.196342468783e-09 true resid norm 1.196342468783e-09 ||r(i)||/||b|| 4.878624879746e-03 218 KSP unpreconditioned resid norm 1.183315169922e-09 true resid norm 1.183315169922e-09 ||r(i)||/||b|| 4.825500205169e-03 219 KSP unpreconditioned resid norm 1.183257111912e-09 true resid norm 1.183257111912e-09 ||r(i)||/||b|| 4.825263447502e-03 220 KSP unpreconditioned resid norm 1.158502689777e-09 true resid norm 1.158502689777e-09 ||r(i)||/||b|| 4.724316149496e-03 221 KSP unpreconditioned resid norm 1.141467176795e-09 true resid norm 1.141467176795e-09 ||r(i)||/||b|| 4.654846177776e-03 222 KSP unpreconditioned resid norm 1.125380759183e-09 true resid norm 1.125380759183e-09 ||r(i)||/||b|| 4.589246569608e-03 223 KSP unpreconditioned resid norm 1.049751865544e-09 true resid norm 1.049751865544e-09 ||r(i)||/||b|| 4.280835715890e-03 224 KSP unpreconditioned resid norm 1.038873206649e-09 true resid norm 1.038873206649e-09 ||r(i)||/||b|| 4.236473087857e-03 225 KSP unpreconditioned resid norm 1.037106322062e-09 true resid norm 1.037106322062e-09 ||r(i)||/||b|| 4.229267820694e-03 226 KSP unpreconditioned resid norm 1.023165629117e-09 true resid norm 1.023165629117e-09 ||r(i)||/||b|| 4.172418370629e-03 227 KSP unpreconditioned resid norm 1.012513033752e-09 true resid norm 1.012513033752e-09 ||r(i)||/||b|| 4.128977618385e-03 228 KSP unpreconditioned resid norm 1.011709961023e-09 true resid norm 1.011709961023e-09 ||r(i)||/||b|| 4.125702727880e-03 229 KSP unpreconditioned resid norm 1.000890151107e-09 true resid norm 1.000890151107e-09 ||r(i)||/||b|| 4.081580083044e-03 230 KSP unpreconditioned resid norm 9.932322546904e-10 true resid norm 9.932322546904e-10 ||r(i)||/||b|| 4.050351563654e-03 231 KSP unpreconditioned resid norm 9.815667443562e-10 true resid norm 9.815667443562e-10 ||r(i)||/||b|| 4.002780194722e-03 232 KSP unpreconditioned resid norm 9.700314124060e-10 true resid norm 9.700314124060e-10 ||r(i)||/||b|| 3.955739686743e-03 233 KSP unpreconditioned resid norm 9.574610866687e-10 true resid norm 9.574610866687e-10 ||r(i)||/||b|| 3.904478525755e-03 234 KSP unpreconditioned resid norm 9.280928856868e-10 true resid norm 9.280928856868e-10 ||r(i)||/||b|| 3.784716467881e-03 235 KSP unpreconditioned resid norm 9.010773908713e-10 true resid norm 9.010773908713e-10 ||r(i)||/||b|| 3.674548628333e-03 236 KSP unpreconditioned resid norm 8.803465259674e-10 true resid norm 8.803465259674e-10 ||r(i)||/||b|| 3.590009195906e-03 237 KSP unpreconditioned resid norm 8.798665240877e-10 true resid norm 8.798665240877e-10 ||r(i)||/||b|| 3.588051772197e-03 238 KSP unpreconditioned resid norm 8.400926267700e-10 true resid norm 8.400926267700e-10 ||r(i)||/||b|| 3.425855804001e-03 239 KSP unpreconditioned resid norm 8.373642559707e-10 true resid norm 8.373642559707e-10 ||r(i)||/||b|| 3.414729644051e-03 240 KSP unpreconditioned resid norm 7.987805643411e-10 true resid norm 7.987805643411e-10 ||r(i)||/||b|| 3.257387275249e-03 241 KSP unpreconditioned resid norm 7.781091552377e-10 true resid norm 7.781091552377e-10 ||r(i)||/||b|| 3.173090300609e-03 242 KSP unpreconditioned resid norm 7.781050658407e-10 true resid norm 7.781050658407e-10 ||r(i)||/||b|| 3.173073624252e-03 243 KSP unpreconditioned resid norm 7.445999948358e-10 true resid norm 7.445999948358e-10 ||r(i)||/||b|| 3.036441616890e-03 244 KSP unpreconditioned resid norm 7.441159890920e-10 true resid norm 7.441159890920e-10 ||r(i)||/||b|| 3.034467865623e-03 245 KSP unpreconditioned resid norm 7.163771304103e-10 true resid norm 7.163771304103e-10 ||r(i)||/||b|| 2.921350184331e-03 246 KSP unpreconditioned resid norm 7.085036867050e-10 true resid norm 7.085036867050e-10 ||r(i)||/||b|| 2.889242673854e-03 247 KSP unpreconditioned resid norm 7.083620085774e-10 true resid norm 7.083620085774e-10 ||r(i)||/||b|| 2.888664917521e-03 248 KSP unpreconditioned resid norm 6.947432439294e-10 true resid norm 6.947432439294e-10 ||r(i)||/||b|| 2.833128274982e-03 249 KSP unpreconditioned resid norm 6.823952950716e-10 true resid norm 6.823952950716e-10 ||r(i)||/||b|| 2.782773955810e-03 250 KSP unpreconditioned resid norm 6.796826154377e-10 true resid norm 6.796826154377e-10 ||r(i)||/||b|| 2.771711783649e-03 251 KSP unpreconditioned resid norm 6.718040441435e-10 true resid norm 6.718040441435e-10 ||r(i)||/||b|| 2.739583363121e-03 252 KSP unpreconditioned resid norm 6.686245721995e-10 true resid norm 6.686245721995e-10 ||r(i)||/||b|| 2.726617635217e-03 253 KSP unpreconditioned resid norm 6.683255976770e-10 true resid norm 6.683255976770e-10 ||r(i)||/||b|| 2.725398432036e-03 254 KSP unpreconditioned resid norm 6.424717983573e-10 true resid norm 6.424717983573e-10 ||r(i)||/||b|| 2.619967928741e-03 255 KSP unpreconditioned resid norm 6.249393762433e-10 true resid norm 6.249393762433e-10 ||r(i)||/||b|| 2.548471586381e-03 256 KSP unpreconditioned resid norm 6.241944596833e-10 true resid norm 6.241944596833e-10 ||r(i)||/||b|| 2.545433853827e-03 257 KSP unpreconditioned resid norm 6.187343723218e-10 true resid norm 6.187343723218e-10 ||r(i)||/||b|| 2.523167890073e-03 258 KSP unpreconditioned resid norm 6.070150446634e-10 true resid norm 6.070150446634e-10 ||r(i)||/||b|| 2.475377056779e-03 259 KSP unpreconditioned resid norm 6.050327025112e-10 true resid norm 6.050327025112e-10 ||r(i)||/||b|| 2.467293164419e-03 260 KSP unpreconditioned resid norm 5.863857468676e-10 true resid norm 5.863857468677e-10 ||r(i)||/||b|| 2.391251809950e-03 261 KSP unpreconditioned resid norm 5.800609942453e-10 true resid norm 5.800609942453e-10 ||r(i)||/||b|| 2.365459784417e-03 262 KSP unpreconditioned resid norm 5.635375866246e-10 true resid norm 5.635375866246e-10 ||r(i)||/||b|| 2.298078152802e-03 263 KSP unpreconditioned resid norm 5.425626596200e-10 true resid norm 5.425626596200e-10 ||r(i)||/||b|| 2.212543447310e-03 264 KSP unpreconditioned resid norm 5.351709834533e-10 true resid norm 5.351709834533e-10 ||r(i)||/||b|| 2.182400560811e-03 265 KSP unpreconditioned resid norm 5.350898708340e-10 true resid norm 5.350898708340e-10 ||r(i)||/||b|| 2.182069787598e-03 266 KSP unpreconditioned resid norm 5.132862441853e-10 true resid norm 5.132862441853e-10 ||r(i)||/||b|| 2.093155686316e-03 267 KSP unpreconditioned resid norm 5.124896691838e-10 true resid norm 5.124896691838e-10 ||r(i)||/||b|| 2.089907293216e-03 268 KSP unpreconditioned resid norm 5.037801763816e-10 true resid norm 5.037801763816e-10 ||r(i)||/||b|| 2.054390416248e-03 269 KSP unpreconditioned resid norm 4.925535549039e-10 true resid norm 4.925535549040e-10 ||r(i)||/||b|| 2.008608814169e-03 270 KSP unpreconditioned resid norm 4.898675799887e-10 true resid norm 4.898675799887e-10 ||r(i)||/||b|| 1.997655542518e-03 271 KSP unpreconditioned resid norm 4.875059607231e-10 true resid norm 4.875059607231e-10 ||r(i)||/||b|| 1.988024977019e-03 272 KSP unpreconditioned resid norm 4.837449656585e-10 true resid norm 4.837449656585e-10 ||r(i)||/||b|| 1.972687826852e-03 273 KSP unpreconditioned resid norm 4.720657497394e-10 true resid norm 4.720657497394e-10 ||r(i)||/||b|| 1.925060567229e-03 274 KSP unpreconditioned resid norm 4.715724551130e-10 true resid norm 4.715724551130e-10 ||r(i)||/||b|| 1.923048936362e-03 275 KSP unpreconditioned resid norm 4.556552568181e-10 true resid norm 4.556552568181e-10 ||r(i)||/||b|| 1.858139396123e-03 276 KSP unpreconditioned resid norm 4.546658619224e-10 true resid norm 4.546658619224e-10 ||r(i)||/||b|| 1.854104693118e-03 277 KSP unpreconditioned resid norm 4.492436702595e-10 true resid norm 4.492436702595e-10 ||r(i)||/||b|| 1.831993266131e-03 278 KSP unpreconditioned resid norm 4.374310091973e-10 true resid norm 4.374310091973e-10 ||r(i)||/||b|| 1.783821823874e-03 279 KSP unpreconditioned resid norm 4.303070397063e-10 true resid norm 4.303070397063e-10 ||r(i)||/||b|| 1.754770631839e-03 280 KSP unpreconditioned resid norm 4.302972725838e-10 true resid norm 4.302972725838e-10 ||r(i)||/||b|| 1.754730802001e-03 281 KSP unpreconditioned resid norm 4.125645901087e-10 true resid norm 4.125645901087e-10 ||r(i)||/||b|| 1.682417807883e-03 282 KSP unpreconditioned resid norm 4.103036443391e-10 true resid norm 4.103036443391e-10 ||r(i)||/||b|| 1.673197783875e-03 283 KSP unpreconditioned resid norm 3.929667750546e-10 true resid norm 3.929667750546e-10 ||r(i)||/||b|| 1.602498896195e-03 284 KSP unpreconditioned resid norm 3.873776455578e-10 true resid norm 3.873776455578e-10 ||r(i)||/||b|| 1.579706705053e-03 285 KSP unpreconditioned resid norm 3.872816271266e-10 true resid norm 3.872816271266e-10 ||r(i)||/||b|| 1.579315146683e-03 286 KSP unpreconditioned resid norm 3.830696799350e-10 true resid norm 3.830696799350e-10 ||r(i)||/||b|| 1.562139036248e-03 287 KSP unpreconditioned resid norm 3.696542291520e-10 true resid norm 3.696542291520e-10 ||r(i)||/||b|| 1.507431497504e-03 288 KSP unpreconditioned resid norm 3.641272853703e-10 true resid norm 3.641272853703e-10 ||r(i)||/||b|| 1.484892896605e-03 289 KSP unpreconditioned resid norm 3.635260937116e-10 true resid norm 3.635260937116e-10 ||r(i)||/||b|| 1.482441267025e-03 290 KSP unpreconditioned resid norm 3.587436182869e-10 true resid norm 3.587436182869e-10 ||r(i)||/||b|| 1.462938570931e-03 291 KSP unpreconditioned resid norm 3.584310825884e-10 true resid norm 3.584310825884e-10 ||r(i)||/||b|| 1.461664065951e-03 292 KSP unpreconditioned resid norm 3.573721244021e-10 true resid norm 3.573721244021e-10 ||r(i)||/||b|| 1.457345687318e-03 293 KSP unpreconditioned resid norm 3.468100005226e-10 true resid norm 3.468100005226e-10 ||r(i)||/||b|| 1.414273873280e-03 294 KSP unpreconditioned resid norm 3.457583459976e-10 true resid norm 3.457583459976e-10 ||r(i)||/||b|| 1.409985278613e-03 295 KSP unpreconditioned resid norm 3.387076173157e-10 true resid norm 3.387076173157e-10 ||r(i)||/||b|| 1.381232758942e-03 296 KSP unpreconditioned resid norm 3.324320252350e-10 true resid norm 3.324320252350e-10 ||r(i)||/||b|| 1.355641207643e-03 297 KSP unpreconditioned resid norm 3.320316935344e-10 true resid norm 3.320316935344e-10 ||r(i)||/||b|| 1.354008674949e-03 298 KSP unpreconditioned resid norm 3.199433868021e-10 true resid norm 3.199433868021e-10 ||r(i)||/||b|| 1.304713163407e-03 299 KSP unpreconditioned resid norm 3.186963397244e-10 true resid norm 3.186963397244e-10 ||r(i)||/||b|| 1.299627767662e-03 300 KSP unpreconditioned resid norm 3.038254414015e-10 true resid norm 3.038254414015e-10 ||r(i)||/||b|| 1.238984986489e-03 301 KSP unpreconditioned resid norm 2.957613786677e-10 true resid norm 2.957613786677e-10 ||r(i)||/||b|| 1.206100141128e-03 302 KSP unpreconditioned resid norm 2.948798167727e-10 true resid norm 2.948798167727e-10 ||r(i)||/||b|| 1.202505175718e-03 303 KSP unpreconditioned resid norm 2.871808272344e-10 true resid norm 2.871808272344e-10 ||r(i)||/||b|| 1.171109080627e-03 304 KSP unpreconditioned resid norm 2.833844512083e-10 true resid norm 2.833844512083e-10 ||r(i)||/||b|| 1.155627648665e-03 305 KSP unpreconditioned resid norm 2.774102056635e-10 true resid norm 2.774102056635e-10 ||r(i)||/||b|| 1.131264973501e-03 306 KSP unpreconditioned resid norm 2.700213061379e-10 true resid norm 2.700213061379e-10 ||r(i)||/||b|| 1.101133410006e-03 307 KSP unpreconditioned resid norm 2.695724613761e-10 true resid norm 2.695724613761e-10 ||r(i)||/||b|| 1.099303043469e-03 308 KSP unpreconditioned resid norm 2.664577097467e-10 true resid norm 2.664577097467e-10 ||r(i)||/||b|| 1.086601241778e-03 309 KSP unpreconditioned resid norm 2.551357253957e-10 true resid norm 2.551357253957e-10 ||r(i)||/||b|| 1.040430754660e-03 310 KSP unpreconditioned resid norm 2.551346590051e-10 true resid norm 2.551346590051e-10 ||r(i)||/||b|| 1.040426405972e-03 311 KSP unpreconditioned resid norm 2.519231006984e-10 true resid norm 2.519231006984e-10 ||r(i)||/||b|| 1.027329831482e-03 312 KSP unpreconditioned resid norm 2.493737722590e-10 true resid norm 2.493737722590e-10 ||r(i)||/||b|| 1.016933797340e-03 313 KSP unpreconditioned resid norm 2.481881884201e-10 true resid norm 2.481881884201e-10 ||r(i)||/||b|| 1.012099045617e-03 314 KSP unpreconditioned resid norm 2.442954977149e-10 true resid norm 2.442954977149e-10 ||r(i)||/||b|| 9.962248471999e-04 ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./hit on a interlagos-64idx-pgi-opt named nid24651 with 128 processors, by Unknown Mon Sep 30 12:07:56 2013 Using Petsc Release Version 3.4.2, Jul, 02, 2013 Max Max/Min Avg Total Time (sec): 5.495e+01 1.00032 5.494e+01 Objects: 2.012e+03 1.00000 2.012e+03 Flops: 7.393e+09 1.00000 7.393e+09 9.463e+11 Flops/sec: 1.346e+08 1.00032 1.346e+08 1.723e+10 MPI Messages: 2.139e+05 1.00000 2.139e+05 2.738e+07 MPI Message Lengths: 4.189e+08 1.00000 1.958e+03 5.361e+10 MPI Reductions: 6.444e+03 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 1.9671e+01 35.8% 3.8477e+11 40.7% 8.837e+05 3.2% 5.137e+02 26.2% 6.429e+03 99.8% 1: MG Apply: 3.5264e+01 64.2% 5.6152e+11 59.3% 2.649e+07 96.8% 1.445e+03 73.8% 1.400e+01 0.2% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %f - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage VecMDot 784 1.0 2.3676e+00 1.4 6.93e+08 1.0 0.0e+00 0.0e+00 7.8e+02 4 9 0 0 12 10 23 0 0 12 37453 VecNorm 1690 1.0 7.5013e-01 3.6 1.11e+08 1.0 0.0e+00 0.0e+00 1.7e+03 1 1 0 0 26 3 4 0 0 26 18899 VecScale 838 1.0 4.0263e-02 1.1 2.75e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 87296 VecCopy 852 1.0 1.2544e-01 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 942 1.0 1.3346e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecAXPY 818 1.0 1.3729e-01 1.4 5.36e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 1 2 0 0 0 49979 VecAYPX 818 1.0 1.1893e-01 1.9 2.68e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 28849 VecWAXPY 20 1.0 5.6469e-03 1.9 6.55e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 14855 VecMAXPY 1602 1.0 3.4485e+00 1.2 1.40e+09 1.0 0.0e+00 0.0e+00 0.0e+00 6 19 0 0 0 16 46 0 0 0 51877 VecScatterBegin 1694 1.0 2.3263e-01 1.3 0.00e+00 0.0 8.5e+05 1.6e+04 0.0e+00 0 0 3 26 0 1 0 96 99 0 0 VecScatterEnd 1694 1.0 1.1359e+00 3.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 3 0 0 0 0 0 MatMult 1622 1.0 5.9099e+00 1.1 6.91e+08 1.0 8.3e+05 1.6e+04 0.0e+00 10 9 3 25 0 28 23 94 97 0 14965 MatMultTranspose 4 1.0 2.2697e-03 1.1 2.53e+05 1.0 1.5e+03 9.9e+02 0.0e+00 0 0 0 0 0 0 0 0 0 0 14252 MatLUFactorSym 1 1.0 5.1498e-04 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatLUFactorNum 1 1.0 1.3133e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 MatAssemblyBegin 51 1.0 1.6192e-01 5.5 0.00e+00 0.0 0.0e+00 0.0e+00 8.6e+01 0 0 0 0 1 1 0 0 0 1 0 MatAssemblyEnd 51 1.0 1.6754e-01 1.2 0.00e+00 0.0 1.2e+04 1.1e+03 7.2e+01 0 0 0 0 1 1 0 1 0 1 0 MatGetRowIJ 1 1.0 4.0531e-06 4.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 1.5974e-05 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatPtAP 4 1.0 2.1024e-01 1.0 5.11e+06 1.0 2.5e+04 6.0e+03 1.0e+02 0 0 0 0 2 1 0 3 1 2 3108 MatPtAPSymbolic 4 1.0 1.4685e-01 1.1 0.00e+00 0.0 1.5e+04 7.8e+03 6.0e+01 0 0 0 0 1 1 0 2 1 1 0 MatPtAPNumeric 4 1.0 7.1755e-02 1.1 5.11e+06 1.0 9.7e+03 3.1e+03 4.0e+01 0 0 0 0 1 0 0 1 0 1 9107 MatGetLocalMat 4 1.0 1.9876e-02 3.0 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetBrAoCol 4 1.0 2.8243e-02 3.0 0.00e+00 0.0 1.1e+04 8.4e+03 8.0e+00 0 0 0 0 0 0 0 1 1 0 0 MatGetSymTrans 8 1.0 9.5758e-03 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPGMRESOrthog 784 1.0 3.7788e+00 1.2 1.39e+09 1.0 0.0e+00 0.0e+00 7.8e+02 6 19 0 0 12 18 46 0 0 12 46933 KSPSetUp 6 1.0 2.6372e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 3.6e+01 0 0 0 0 1 0 0 0 0 1 0 Warning -- total time of even greater than time of entire stage -- something is wrong with the timer KSPSolve 34 1.0 4.8017e+01 1.0 7.39e+09 1.0 2.7e+07 1.9e+03 6.2e+03 87100100 99 96 2442463096379 97 19707 PCSetUp 1 1.0 4.2120e-01 1.0 5.36e+06 1.0 3.4e+04 4.6e+03 3.1e+02 1 0 0 0 5 2 0 4 1 5 1628 Warning -- total time of even greater than time of entire stage -- something is wrong with the timer PCApply 784 1.0 3.5575e+01 1.0 4.39e+09 1.0 2.6e+07 1.5e+03 1.4e+01 64 59 97 74 0 1791462998281 0 15784 MGSetup Level 0 1 1.0 1.3333e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01 0 0 0 0 0 1 0 0 0 0 0 MGSetup Level 1 1 1.0 2.0680e-03 7.6 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 0 0 0 0 0 0 0 MGSetup Level 2 1 1.0 3.9411e-04 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 0 0 0 0 0 0 0 MGSetup Level 3 1 1.0 2.9707e-04 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 0 0 0 0 0 0 0 MGSetup Level 4 1 1.0 1.7860e-03 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 0 0 0 0 0 0 0 --- Event Stage 1: MG Apply VecCopy 8624 1.0 2.4634e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 50176 1.0 3.2026e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 VecAXPY 15680 1.0 3.5027e-01 1.4 1.34e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 3 0 0 0 48993 VecAYPX 18816 1.0 4.3705e-01 1.2 9.64e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 2 0 0 0 28229 VecScatterBegin 40768 1.0 1.3572e+00 1.8 0.00e+00 0.0 2.6e+07 1.5e+03 0.0e+00 2 0 97 74 0 3 0100100 0 0 VecScatterEnd 40768 1.0 3.4473e+00 2.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 4 0 0 0 0 7 0 0 0 0 0 MatMult 21952 1.0 1.5737e+01 1.1 2.19e+09 1.0 1.9e+07 1.8e+03 0.0e+00 27 30 70 66 0 43 50 73 89 0 17850 MatMultAdd 7840 1.0 1.7853e+00 1.2 2.26e+08 1.0 3.0e+06 5.3e+02 0.0e+00 3 3 11 3 0 4 5 11 4 0 16221 MatMultTranspose 10976 1.0 4.0456e+00 1.3 4.24e+08 1.0 4.2e+06 6.6e+02 0.0e+00 6 6 15 5 0 10 10 16 7 0 13427 MatSolve 19600 1.0 1.3496e+01 1.1 1.31e+09 1.0 0.0e+00 0.0e+00 0.0e+00 24 18 0 0 0 37 30 0 0 0 12419 MatLUFactorNum 4 1.0 1.0570e-02 1.2 1.86e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 22564 MatILUFactorSym 4 1.0 5.1000e-02 3.9 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 29 0 MatGetRowIJ 4 1.0 9.7752e-0610.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 4 1.0 5.0209e-0244.6 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01 0 0 0 0 0 0 0 0 0 71 0 KSPSetUp 4 1.0 4.2915e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 19600 1.0 2.3593e+01 1.0 2.79e+09 1.0 1.0e+07 2.1e+03 1.4e+01 42 38 38 40 0 65 63 39 54100 15111 PCSetUp 4 1.0 7.3869e-02 1.8 1.86e+06 1.0 0.0e+00 0.0e+00 1.4e+01 0 0 0 0 0 0 0 0 0100 3229 PCSetUpOnBlocks 15680 1.0 8.8366e-02 1.7 1.86e+06 1.0 0.0e+00 0.0e+00 1.4e+01 0 0 0 0 0 0 0 0 0100 2699 PCApply 19600 1.0 1.4283e+01 1.1 1.31e+09 1.0 0.0e+00 0.0e+00 0.0e+00 25 18 0 0 0 39 30 0 0 0 11734 MGSmooth Level 0 3920 1.0 6.4374e+00 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 11 0 0 0 0 17 0 0 0 0 0 MGSmooth Level 1 6272 1.0 5.7907e-01 1.1 2.36e+07 1.0 4.0e+06 1.9e+02 5.0e+00 1 0 15 1 0 2 1 15 2 36 5214 MGResid Level 1 3136 1.0 7.1739e-01 5.5 1.08e+07 1.0 3.2e+06 1.9e+02 0.0e+00 1 0 12 1 0 1 0 12 2 0 1934 MGInterp Level 1 7840 1.0 5.6076e-01 4.3 3.34e+06 1.0 3.0e+06 6.4e+01 0.0e+00 0 0 11 0 0 1 0 11 0 0 762 MGSmooth Level 2 4704 1.0 1.0847e+00 1.2 1.80e+08 1.0 3.2e+06 6.4e+02 3.0e+00 2 2 12 4 0 3 4 12 5 21 21185 MGResid Level 2 2352 1.0 9.1275e-01 2.8 6.50e+07 1.0 2.4e+06 6.4e+02 0.0e+00 1 1 9 3 0 1 1 9 4 0 9119 MGInterp Level 2 6272 1.0 7.0760e-01 3.4 2.13e+07 1.0 2.4e+06 2.1e+02 0.0e+00 1 0 9 1 0 1 0 9 1 0 3848 MGSmooth Level 3 3136 1.0 6.2875e+00 1.1 1.12e+09 1.0 2.4e+06 2.3e+03 3.0e+00 11 15 9 10 0 17 25 9 14 21 22740 MGResid Level 3 1568 1.0 1.9152e+00 1.2 3.47e+08 1.0 1.6e+06 2.3e+03 0.0e+00 3 5 6 7 0 5 8 6 9 0 23179 MGInterp Level 3 4704 1.0 1.4607e+00 1.9 1.27e+08 1.0 1.8e+06 7.7e+02 0.0e+00 2 2 7 3 0 3 3 7 4 0 11116 MGSmooth Level 4 1568 1.0 1.0432e+01 1.1 1.47e+09 1.0 8.0e+05 1.6e+04 3.0e+00 18 20 3 25 0 29 33 3 33 21 17976 MGResid Level 4 784 1.0 2.9461e+00 1.1 3.60e+08 1.0 4.0e+05 1.6e+04 0.0e+00 5 5 1 12 0 8 8 2 17 0 15626 MGInterp Level 4 3136 1.0 5.2866e+00 1.1 6.68e+08 1.0 1.2e+06 2.9e+03 0.0e+00 9 9 4 7 0 14 15 5 9 0 16172 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Vector 1843 1843 474947688 0 Vector Scatter 19 19 22572 0 Matrix 38 42 19508928 0 Matrix Null Space 1 1 652 0 Distributed Mesh 5 5 830792 0 Bipartite Graph 10 10 8560 0 Index Set 47 59 844496 0 IS L to G Mapping 5 5 405756 0 Krylov Solver 11 11 32800 0 DMKSP interface 3 3 2088 0 Preconditioner 11 11 11864 0 Viewer 1 0 0 0 --- Event Stage 1: MG Apply Matrix 4 0 0 0 Index Set 14 2 1792 0 ======================================================================================================================== Average time to get PetscTime(): 1.90735e-07 Average time for MPI_Barrier(): 1.52111e-05 Average time for zero size MPI_Send(): 2.83681e-06 #PETSc Option Table entries: -ksp_monitor_true_residual -ksp_type fgmres -log_summary -mg_coarse_pc_factor_mat_solver_package superlu_dist -mg_coarse_pc_type lu -mg_levels_ksp_max_it 1 -mg_levels_ksp_type richardson -mg_levels_pc_type bjacobi -options_left -pc_mg_galerkin -pc_mg_levels 5 -pc_mg_log -pc_mg_type full -pc_type mg #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 8 Configure run at: Wed Aug 28 23:25:43 2013 Configure options: --known-level1-dcache-size=16384 --known-level1-dcache-linesize=64 --known-level1-dcache-assoc=4 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=0 --known-mpi-c-double-complex=0 --with-batch="1 " --known-mpi-shared="0 " --known-memcmp-ok --with-blas-lapack-lib="-L/opt/acml/5.3.0/pgi64/lib -lacml" --COPTFLAGS="-O3 -fastsse" --FOPTFLAGS="-O3 -fastsse" --CXXOPTFLAGS="-O3 -fastsse" --with-x="0 " --with-debugging="0 " --with-clib-autodetect="0 " --with-cxxlib-autodetect="0 " --with-fortranlib-autodetect="0 " --with-shared-libraries=0 --with-dynamic-loading=0 --with-mpi-compilers="1 " --known-mpi-shared-libraries=0 --with-64-bit-indices --download-blacs="1 " --download-scalapack="1 " --download-superlu_dist="1 " --download-metis="1 " --download-parmetis="1 " --with-cc=cc --with-cxx=CC --with-fc=ftn PETSC_ARCH=interlagos-64idx-pgi-opt ----------------------------------------- Libraries compiled on Wed Aug 28 23:25:43 2013 on h2ologin3 Machine characteristics: Linux-2.6.32.59-0.7-default-x86_64-with-SuSE-11-x86_64 Using PETSc directory: /mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2 Using PETSc arch: interlagos-64idx-pgi-opt ----------------------------------------- Using C compiler: cc -O3 -fastsse ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: ftn -O3 -fastsse ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/include -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/include -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/include -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/include -I/opt/cray/udreg/2.3.2-1.0402.7311.2.1.gem/include -I/opt/cray/ugni/5.0-1.0402.7128.7.6.gem/include -I/opt/cray/pmi/4.0.1-1.0000.9421.73.3.gem/include -I/opt/cray/dmapp/4.0.1-1.0402.7439.5.1.gem/include -I/opt/cray/gni-headers/2.1-1.0402.7082.6.2.gem/include -I/opt/cray/xpmem/0.1-2.0402.44035.2.1.gem/include -I/opt/cray/rca/1.0.0-2.0402.42153.2.106.gem/include -I/opt/cray-hss-devel/7.0.0/include -I/opt/cray/krca/1.0.0-2.0402.42157.2.94.gem/include -I/opt/cray/mpt/6.0.1/gni/mpich2-pgi/121/include -I/opt/acml/5.3.0/pgi64_fma4/include -I/opt/cray/libsci/12.1.01/pgi/121/interlagos/include -I/opt/fftw/3.3.0.3/interlagos/include -I/usr/include/alps -I/opt/pgi/13.6.0/linux86-64/13.6/include -I/opt/cray/xe-sysroot/4.2.24/usr/include ----------------------------------------- Using C linker: cc Using Fortran linker: ftn Using libraries: -Wl,-rpath,/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -L/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -lpetsc -Wl,-rpath,/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -L/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -lsuperlu_dist_3.3 -L/opt/acml/5.3.0/pgi64/lib -lacml -lpthread -lparmetis -lmetis -ldl ----------------------------------------- #PETSc Option Table entries: -ksp_monitor_true_residual -ksp_type fgmres -log_summary -mg_coarse_pc_factor_mat_solver_package superlu_dist -mg_coarse_pc_type lu -mg_levels_ksp_max_it 1 -mg_levels_ksp_type richardson -mg_levels_pc_type bjacobi -options_left -pc_mg_galerkin -pc_mg_levels 5 -pc_mg_log -pc_mg_type full -pc_type mg #End of PETSc Option Table entries There are no unused options. -------------- next part -------------- -pc_mg_type full -ksp_type fgmres -mg_levels_pc_type sor -mg_levels_ksp_type gmres -mg_levels_ksp_max_it 3 -mg_coarse_pc_factor_mat_solver_package superlu_dist -mg_coarse_pc_type lu -pc_mg_galerkin -pc_mg_levels 5 -pc_type mg -log_summary -pc_mg_log -ksp_monitor_true_residual -options_left 9979 KSP unpreconditioned resid norm 1.205238163959e-08 true resid norm 1.205238163959e-08 ||r(i)||/||b|| 6.439485962882e-02 9980 KSP unpreconditioned resid norm 1.205238163959e-08 true resid norm 1.205238163959e-08 ||r(i)||/||b|| 6.439485962882e-02 9981 KSP unpreconditioned resid norm 1.205238163959e-08 true resid norm 1.205238163959e-08 ||r(i)||/||b|| 6.439485962882e-02 9982 KSP unpreconditioned resid norm 1.205238163959e-08 true resid norm 1.205238163959e-08 ||r(i)||/||b|| 6.439485962882e-02 9983 KSP unpreconditioned resid norm 1.205238163959e-08 true resid norm 1.205238163959e-08 ||r(i)||/||b|| 6.439485962882e-02 9984 KSP unpreconditioned resid norm 1.205238163959e-08 true resid norm 1.205238163959e-08 ||r(i)||/||b|| 6.439485962882e-02 9985 KSP unpreconditioned resid norm 1.205238163959e-08 true resid norm 1.205238163959e-08 ||r(i)||/||b|| 6.439485962882e-02 9986 KSP unpreconditioned resid norm 1.205238163959e-08 true resid norm 1.205238163959e-08 ||r(i)||/||b|| 6.439485962882e-02 9987 KSP unpreconditioned resid norm 1.205238163959e-08 true resid norm 1.205238163959e-08 ||r(i)||/||b|| 6.439485962882e-02 9988 KSP unpreconditioned resid norm 1.205238163959e-08 true resid norm 1.205238163959e-08 ||r(i)||/||b|| 6.439485962882e-02 9989 KSP unpreconditioned resid norm 1.205238163959e-08 true resid norm 1.205238163959e-08 ||r(i)||/||b|| 6.439485962882e-02 9990 KSP unpreconditioned resid norm 1.205238163959e-08 true resid norm 1.205238163959e-08 ||r(i)||/||b|| 6.439485962882e-02 9991 KSP unpreconditioned resid norm 1.205238163959e-08 true resid norm 1.205238163959e-08 ||r(i)||/||b|| 6.439485962882e-02 9992 KSP unpreconditioned resid norm 1.205238163959e-08 true resid norm 1.205238163959e-08 ||r(i)||/||b|| 6.439485962882e-02 9993 KSP unpreconditioned resid norm 1.205238163959e-08 true resid norm 1.205238163959e-08 ||r(i)||/||b|| 6.439485962882e-02 9994 KSP unpreconditioned resid norm 1.205238163959e-08 true resid norm 1.205238163959e-08 ||r(i)||/||b|| 6.439485962882e-02 9995 KSP unpreconditioned resid norm 1.205238163959e-08 true resid norm 1.205238163959e-08 ||r(i)||/||b|| 6.439485962882e-02 9996 KSP unpreconditioned resid norm 1.205238163959e-08 true resid norm 1.205238163959e-08 ||r(i)||/||b|| 6.439485962882e-02 9997 KSP unpreconditioned resid norm 1.205238163959e-08 true resid norm 1.205238163959e-08 ||r(i)||/||b|| 6.439485962882e-02 9998 KSP unpreconditioned resid norm 1.205238163959e-08 true resid norm 1.205238163959e-08 ||r(i)||/||b|| 6.439485962882e-02 9999 KSP unpreconditioned resid norm 1.205238163959e-08 true resid norm 1.205238163959e-08 ||r(i)||/||b|| 6.439485962882e-02 10000 KSP unpreconditioned resid norm 1.205238163959e-08 true resid norm 1.205238163959e-08 ||r(i)||/||b|| 6.439485962882e-02 ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./hit on a interlagos-64idx-pgi-opt named nid00193 with 128 processors, by Unknown Mon Sep 30 12:37:25 2013 Using Petsc Release Version 3.4.2, Jul, 02, 2013 Max Max/Min Avg Total Time (sec): 1.632e+03 1.00002 1.632e+03 Objects: 4.184e+05 1.00000 4.184e+05 Flops: 2.514e+11 1.00000 2.514e+11 3.218e+13 Flops/sec: 1.540e+08 1.00002 1.540e+08 1.972e+10 MPI Messages: 7.101e+06 1.00000 7.101e+06 9.089e+08 MPI Message Lengths: 1.378e+10 1.00000 1.941e+03 1.764e+12 MPI Reductions: 9.168e+05 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 4.6064e+02 28.2% 1.3696e+13 42.6% 2.736e+07 3.0% 4.928e+02 25.4% 1.864e+05 20.3% 1: MG Apply: 1.1712e+03 71.8% 1.8482e+13 57.4% 8.816e+08 97.0% 1.448e+03 74.6% 7.305e+05 79.7% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %f - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage VecMDot 26088 1.0 7.9541e+01 1.2 2.57e+10 1.0 0.0e+00 0.0e+00 2.6e+04 4 10 0 0 3 15 24 0 0 14 41427 VecNorm 53560 1.0 2.3821e+01 2.7 3.51e+09 1.0 0.0e+00 0.0e+00 5.4e+04 1 1 0 0 6 4 3 0 0 29 18861 VecScale 27094 1.0 1.2548e+00 1.1 8.88e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 90567 VecCopy 26466 1.0 3.6450e+00 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecSet 26866 1.0 3.6354e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecAXPY 26277 1.0 3.9569e+00 1.3 1.72e+09 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 1 2 0 0 0 55707 VecAYPX 26277 1.0 3.4483e+00 1.5 8.61e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 1 0 0 0 31962 VecWAXPY 819 1.0 2.3387e-01 1.9 2.68e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 14688 VecMAXPY 52365 1.0 1.2676e+02 1.2 5.16e+10 1.0 0.0e+00 0.0e+00 0.0e+00 7 21 0 0 0 26 48 0 0 0 52096 VecScatterBegin 53566 1.0 7.2821e+00 1.3 0.00e+00 0.0 2.7e+07 1.6e+04 0.0e+00 0 0 3 25 0 1 0100100 0 0 VecScatterEnd 53566 1.0 3.9626e+01 3.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 5 0 0 0 0 0 MatMult 53184 1.0 1.8661e+02 1.2 2.27e+10 1.0 2.7e+07 1.6e+04 0.0e+00 11 9 3 25 0 38 21100100 0 15540 MatMultTranspose 4 1.0 2.2471e-03 1.1 2.53e+05 1.0 1.5e+03 9.9e+02 0.0e+00 0 0 0 0 0 0 0 0 0 0 14396 MatLUFactorSym 1 1.0 4.9806e-04 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatLUFactorNum 1 1.0 1.2223e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 206 1.0 6.9716e-0111.8 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+02 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 206 1.0 7.1718e-01 1.2 0.00e+00 0.0 1.2e+04 1.1e+03 7.2e+01 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 1 1.0 5.0068e-06 5.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 1.9073e-05 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatPtAP 4 1.0 2.0369e-01 1.0 5.11e+06 1.0 2.5e+04 6.0e+03 1.0e+02 0 0 0 0 0 0 0 0 0 0 3208 MatPtAPSymbolic 4 1.0 1.4161e-01 1.1 0.00e+00 0.0 1.5e+04 7.8e+03 6.0e+01 0 0 0 0 0 0 0 0 0 0 0 MatPtAPNumeric 4 1.0 7.0208e-02 1.1 5.11e+06 1.0 9.7e+03 3.1e+03 4.0e+01 0 0 0 0 0 0 0 0 0 0 9307 MatGetLocalMat 4 1.0 1.8704e-02 3.0 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetBrAoCol 4 1.0 2.6713e-02 2.8 0.00e+00 0.0 1.1e+04 8.4e+03 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetSymTrans 8 1.0 9.3246e-03 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPGMRESOrthog 26088 1.0 1.3317e+02 1.1 5.15e+10 1.0 0.0e+00 0.0e+00 2.6e+04 8 20 0 0 3 27 48 0 0 14 49486 KSPSetUp 6 1.0 2.7412e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 3.6e+01 0 0 0 0 0 0 0 0 0 0 0 Warning -- total time of even greater than time of entire stage -- something is wrong with the timer KSPSolve 189 1.0 1.5818e+03 1.0 2.51e+11 1.0 9.1e+08 1.9e+03 9.2e+05 97100100100100 3432353321393491 20342 PCSetUp 1 1.0 4.0787e-01 1.0 5.36e+06 1.0 3.4e+04 4.6e+03 3.0e+02 0 0 0 0 0 0 0 0 0 0 1681 Warning -- total time of even greater than time of entire stage -- something is wrong with the timer PCApply 26088 1.0 1.1782e+03 1.0 1.44e+11 1.0 8.8e+08 1.5e+03 7.3e+05 72 57 97 75 80 2541353222294392 15686 MGSetup Level 0 1 1.0 1.2428e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01 0 0 0 0 0 0 0 0 0 0 0 MGSetup Level 1 1 1.0 1.9360e-0319.3 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0 MGSetup Level 2 1 1.0 2.7823e-04 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0 MGSetup Level 3 1 1.0 1.4997e-04 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0 MGSetup Level 4 1 1.0 1.6689e-03 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0 --- Event Stage 1: MG Apply VecScale 365232 1.0 1.5631e+00 1.3 7.00e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 57287 VecCopy 130440 1.0 1.0518e-01 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 1147872 1.0 2.7147e+00 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAYPX 260880 1.0 4.9379e+00 1.2 1.12e+09 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 28911 VecScatterBegin 1356576 1.0 7.7021e+01 1.9 0.00e+00 0.0 8.8e+08 1.5e+03 0.0e+00 3 0 97 75 0 5 0100100 0 0 VecScatterEnd 1356576 1.0 1.1642e+02 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 6 0 0 0 0 8 0 0 0 0 0 MatMult 365232 1.0 2.0117e+02 1.1 3.05e+10 1.0 2.9e+08 1.5e+03 0.0e+00 12 12 32 26 0 16 21 33 34 0 19428 MatMultAdd 260880 1.0 6.2891e+01 1.2 7.53e+09 1.0 1.0e+08 5.3e+02 0.0e+00 3 3 11 3 0 5 5 11 4 0 15322 MatMultTranspose 365232 1.0 1.3004e+02 1.2 1.41e+10 1.0 1.4e+08 6.6e+02 0.0e+00 7 6 15 5 0 10 10 16 7 0 13900 MatSolve 130440 1.0 2.2788e+02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 13 0 0 0 0 18 0 0 0 0 0 MatSOR 521760 1.0 5.7865e+02 1.1 9.11e+10 1.0 3.5e+08 2.1e+03 7.3e+05 35 36 38 41 80 48 63 39 54100 20149 KSPSolve 652200 1.0 7.9658e+02 1.0 9.11e+10 1.0 3.5e+08 2.1e+03 7.3e+05 48 36 38 41 80 67 63 39 54100 14637 PCApply 130440 1.0 2.2803e+02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 13 0 0 0 0 18 0 0 0 0 0 MGSmooth Level 0 130440 1.0 2.3022e+02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 13 0 0 0 0 19 0 0 0 0 0 MGSmooth Level 1 208704 1.0 3.2475e+01 1.2 7.96e+08 1.0 1.3e+08 1.9e+02 2.6e+05 2 0 15 1 28 3 1 15 2 36 3139 MGResid Level 1 104352 1.0 1.4487e+01 2.4 3.61e+08 1.0 1.1e+08 1.9e+02 0.0e+00 1 0 12 1 0 1 0 12 2 0 3186 MGInterp Level 1 260880 1.0 1.7836e+01 3.7 1.11e+08 1.0 1.0e+08 6.4e+01 0.0e+00 1 0 11 0 0 1 0 11 0 0 797 MGSmooth Level 2 156528 1.0 4.2359e+01 1.1 5.96e+09 1.0 1.1e+08 6.4e+02 2.1e+05 2 2 12 4 23 3 4 12 5 29 18002 MGResid Level 2 78264 1.0 2.3577e+01 1.9 2.16e+09 1.0 8.0e+07 6.4e+02 0.0e+00 1 1 9 3 0 1 1 9 4 0 11747 MGInterp Level 2 208704 1.0 2.1986e+01 2.5 7.08e+08 1.0 8.0e+07 2.1e+02 0.0e+00 1 0 9 1 0 1 0 9 1 0 4121 MGSmooth Level 3 104352 1.0 1.8000e+02 1.1 3.69e+10 1.0 8.0e+07 2.3e+03 1.6e+05 11 15 9 10 17 15 26 9 14 21 26230 MGResid Level 3 52176 1.0 4.4884e+01 1.2 1.15e+10 1.0 5.3e+07 2.3e+03 0.0e+00 3 5 6 7 0 4 8 6 9 0 32911 MGInterp Level 3 156528 1.0 4.0354e+01 1.5 4.22e+09 1.0 6.0e+07 7.7e+02 0.0e+00 2 2 7 3 0 3 3 7 4 0 13388 MGSmooth Level 4 52176 1.0 3.4639e+02 1.1 4.75e+10 1.0 2.7e+07 1.6e+04 1.0e+05 21 19 3 25 11 29 33 3 33 14 17534 MGResid Level 4 26088 1.0 8.7990e+01 1.2 1.20e+10 1.0 1.3e+07 1.6e+04 0.0e+00 5 5 1 12 0 7 8 2 17 0 17410 MGInterp Level 4 104352 1.0 1.7549e+02 1.1 2.22e+10 1.0 4.0e+07 2.9e+03 0.0e+00 10 9 4 7 0 14 15 5 9 0 16211 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Vector 53063 53063 14013688680 0 Vector Scatter 19 19 22572 0 Matrix 38 38 14004608 0 Matrix Null Space 1 1 652 0 Distributed Mesh 5 5 830792 0 Bipartite Graph 10 10 8560 0 Index Set 47 47 534480 0 IS L to G Mapping 5 5 405756 0 Krylov Solver 7 7 27744 0 DMKSP interface 3 3 2088 0 Preconditioner 7 7 7352 0 Viewer 1 0 0 0 --- Event Stage 1: MG Apply Vector 365232 365232 17352902784 0 ======================================================================================================================== Average time to get PetscTime(): 9.53674e-08 Average time for MPI_Barrier(): 1.28269e-05 Average time for zero size MPI_Send(): 2.4531e-06 #PETSc Option Table entries: -ksp_monitor_true_residual -ksp_type fgmres -log_summary -mg_coarse_pc_factor_mat_solver_package superlu_dist -mg_coarse_pc_type lu -mg_levels_ksp_max_it 1 -mg_levels_ksp_type richardson -mg_levels_pc_type sor -options_left -pc_mg_galerkin -pc_mg_levels 5 -pc_mg_log -pc_mg_type full -pc_type mg #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 8 Configure run at: Wed Aug 28 23:25:43 2013 Configure options: --known-level1-dcache-size=16384 --known-level1-dcache-linesize=64 --known-level1-dcache-assoc=4 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=0 --known-mpi-c-double-complex=0 --with-batch="1 " --known-mpi-shared="0 " --known-memcmp-ok --with-blas-lapack-lib="-L/opt/acml/5.3.0/pgi64/lib -lacml" --COPTFLAGS="-O3 -fastsse" --FOPTFLAGS="-O3 -fastsse" --CXXOPTFLAGS="-O3 -fastsse" --with-x="0 " --with-debugging="0 " --with-clib-autodetect="0 " --with-cxxlib-autodetect="0 " --with-fortranlib-autodetect="0 " --with-shared-libraries=0 --with-dynamic-loading=0 --with-mpi-compilers="1 " --known-mpi-shared-libraries=0 --with-64-bit-indices --download-blacs="1 " --download-scalapack="1 " --download-superlu_dist="1 " --download-metis="1 " --download-parmetis="1 " --with-cc=cc --with-cxx=CC --with-fc=ftn PETSC_ARCH=interlagos-64idx-pgi-opt ----------------------------------------- Libraries compiled on Wed Aug 28 23:25:43 2013 on h2ologin3 Machine characteristics: Linux-2.6.32.59-0.7-default-x86_64-with-SuSE-11-x86_64 Using PETSc directory: /mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2 Using PETSc arch: interlagos-64idx-pgi-opt ----------------------------------------- Using C compiler: cc -O3 -fastsse ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: ftn -O3 -fastsse ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/include -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/include -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/include -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/include -I/opt/cray/udreg/2.3.2-1.0402.7311.2.1.gem/include -I/opt/cray/ugni/5.0-1.0402.7128.7.6.gem/include -I/opt/cray/pmi/4.0.1-1.0000.9421.73.3.gem/include -I/opt/cray/dmapp/4.0.1-1.0402.7439.5.1.gem/include -I/opt/cray/gni-headers/2.1-1.0402.7082.6.2.gem/include -I/opt/cray/xpmem/0.1-2.0402.44035.2.1.gem/include -I/opt/cray/rca/1.0.0-2.0402.42153.2.106.gem/include -I/opt/cray-hss-devel/7.0.0/include -I/opt/cray/krca/1.0.0-2.0402.42157.2.94.gem/include -I/opt/cray/mpt/6.0.1/gni/mpich2-pgi/121/include -I/opt/acml/5.3.0/pgi64_fma4/include -I/opt/cray/libsci/12.1.01/pgi/121/interlagos/include -I/opt/fftw/3.3.0.3/interlagos/include -I/usr/include/alps -I/opt/pgi/13.6.0/linux86-64/13.6/include -I/opt/cray/xe-sysroot/4.2.24/usr/include ----------------------------------------- Using C linker: cc Using Fortran linker: ftn Using libraries: -Wl,-rpath,/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -L/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -lpetsc -Wl,-rpath,/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -L/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -lsuperlu_dist_3.3 -L/opt/acml/5.3.0/pgi64/lib -lacml -lpthread -lparmetis -lmetis -ldl ----------------------------------------- #PETSc Option Table entries: -ksp_monitor_true_residual -ksp_type fgmres -log_summary -mg_coarse_pc_factor_mat_solver_package superlu_dist -mg_coarse_pc_type lu -mg_levels_ksp_max_it 1 -mg_levels_ksp_type richardson -mg_levels_pc_type sor -options_left -pc_mg_galerkin -pc_mg_levels 5 -pc_mg_log -pc_mg_type full -pc_type mg #End of PETSc Option Table entries There are no unused options. From bsmith at mcs.anl.gov Mon Sep 30 21:17:12 2013 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 30 Sep 2013 21:17:12 -0500 Subject: [petsc-users] How to speed up geometric multigrid In-Reply-To: <524A1EA2.8090606@uci.edu> References: <52406460.3060905@uci.edu> <4499926F-40CE-462C-BFD6-8798A02126FB@mcs.anl.gov> <52420449.4010603@uci.edu> <60076751-799B-410A-9779-D697677A869D@mcs.anl.gov> <524A1EA2.8090606@uci.edu> Message-ID: <9079A4B7-C533-4F90-BA8E-C163761A4A99@mcs.anl.gov> I wasn't expecting this. Try -pc_mg_type full -ksp_type richardson -mg_levels_pc_type bjacobi -mg_levels_ksp_type gmres -mg_levels_ksp_max_it 3 -mg_coarse_pc_factor_mat_solver_package superlu_dist -mg_coarse_pc_type lu -pc_mg_galerkin -pc_mg_levels 5 -pc_type mg -log_summary -pc_mg_log -ksp_monitor_true_residual -options_left -ksp_view -ksp_max_it 30 On Sep 30, 2013, at 8:00 PM, Michele Rosso wrote: > Barry, > > sorry again for the very late answer. I tried all the variations you proposed: all of them converge very slow , except the last one (CG instead of fgmres) diverges. > I attached the diagnostics for the two options that convergence: each of the attached files starts with a list of the options I used for the run. > As you pointed out earlier, the residual I used to require was too small, therefore I increased atol to e-9. > After some tests, I noticed that any further increase of the absolute tolerance changes significantly the solution. > What would you suggest to try next? > Thank you very much, > > Michele > > > > > > > On 09/24/2013 05:08 PM, Barry Smith wrote: >> Thanks. The balance of work on the different levels and across processes looks ok. So it is is a matter of improving the convergence rate. >> >> The initial residual norm is very small. Are you sure you need to decrease it to 10^-12 ???? >> >> Start with a really robust multigrid smoother use >> >> -pc_mg_type full -ksp_type fgmres -mg_levels_pc_type bjacobi -mg_levels_ksp_type gmres -mg_levels_ksp_max_it 3 PLUS -mg_coarse_pc_factor_mat_solver_package superlu_dist >> -mg_coarse_pc_type lu -pc_mg_galerkin -pc_mg_levels 5 -pc_mg_log -pc_type mg >> >> run with the -log_summary and -pc_mg_log >> >> Now back off a little on the smoother and use -mg_levels_pc_type sor instead how does that change the convergence and time. >> >> Back off even more an replace the -ksp_type fgmres with -ksp_type cg and the -mg_levels_ksp_type gmres with -mg_levels_ksp_type richardson how does that change the convergence and the time? >> >> There are some additional variants we can try based on the results from above. >> >> Barry >> >> >> >> On Sep 24, 2013, at 4:29 PM, Michele Rosso wrote: >> >>> Barry, >>> >>> I re-rerun the test case with the option -pc_mg_log as you suggested. >>> I attached the new output ("final_new.txt'). >>> Thanks for your help. >>> >>> Michele >>> >>> On 09/23/2013 09:35 AM, Barry Smith wrote: >>>> Run with the additional option -pc_mg_log and send us the log file. >>>> >>>> Barry >>>> >>>> Maybe we should make this the default somehow. >>>> >>>> >>>> On Sep 23, 2013, at 10:55 AM, Michele Rosso wrote: >>>> >>>>> Hi, >>>>> >>>>> I am successfully using PETSc to solve a 3D Poisson's equation with CG + MG . Such equation arises from a projection algorithm for a multiphase incompressible flow simulation. >>>>> I set up the solver as I was suggested to do in a previous thread (title: "GAMG speed") and run a test case (liquid droplet with surface tension falling under the effect of gravity in a quiescent fluid). >>>>> The solution of the Poisson Equation via multigrid is correct but it becomes progressively slower and slower as the simulation progresses (I am performing successive solves) due to an increase in the number of iterations. >>>>> Since the solution of the Poisson equation is mission-critical, I need to speed it up as much as I can. >>>>> Could you please help me out with this? >>>>> >>>>> I run the test case with the following options: >>>>> >>>>> -pc_type mg -pc_mg_galerkin -pc_mg_levels 5 -mg_levels_ksp_type richardson -mg_levels_ksp_max_it 1 >>>>> -mg_coarse_pc_type lu -mg_coarse_pc_factor_mat_solver_package superlu_dist >>>>> -log_summary -ksp_view -ksp_monitor_true_residual -options_left >>>>> >>>>> Please find the diagnostic for the final solve in the attached file "final.txt'. >>>>> Thank you, >>>>> >>>>> Michele >>>>> >>> >> > > From choi240 at purdue.edu Mon Sep 30 21:25:46 2013 From: choi240 at purdue.edu (Joon Hee Choi) Date: Mon, 30 Sep 2013 22:25:46 -0400 (EDT) Subject: [petsc-users] The product of two MPIDENSE matrices In-Reply-To: <524A105A.6030303@stanford.edu> Message-ID: <85419958.159902.1380594346277.JavaMail.root@mailhub028.itcs.purdue.edu> Dear all, I tried to compile elemental with 64 bit support by modifying the PETSc build script (see attached elemental.py). I modified the typedefs in Elemental and pointed the build script to a location on my home page. I also needed to add -std=c++11 to the CMake arguments lists. This allows Elemental to be built without errors. However, the PETSc build fails with a strange configuration error, which seems to be related to C++11. Note that I have included --CXXFLAGS="-std=c++11" in the configure options to PETSc. The configure.log file can be downloaded from http://www.stat.purdue.edu/~vishy/configure.log any advice on how to workaround this issue is greatly appreciated. thanks ----- ?? ??? ----- ?? ??: "Jack Poulson" ?? ??: "Jed Brown" , "Joon Hee Choi" , "Hong Zhang" ??: "PETSc users list" , "" ?? ??: 2013? 9? 30?, ??? ?? 7:59:22 ??: Re: [petsc-users] The product of two MPIDENSE matrices On 09/30/2013 12:26 PM, Jed Brown wrote: >>> Jack Poulson writes: >>> It hasn't yet been rigorously tested, but support for 64-bit >>> integers should be possible by simply changing the typedefs for >>> elem::Int and elem::Unsigned from 'int' -> 'long long int' and >>> 'unsigned' to 'long long unsigned'. These are guaranteed to be >>> 64-bit by C++11. >>> >> Can you add a CMake configure option for these? I'd rather >> --download-elemental use an interface other than stream-editing >> headers. >> > Done: > https://github.com/elemental/Elemental/commit/03d864e2a8ae01877194bbb572144568454f7d05 > Configuring Elemental with the "-D USE_64BIT_INTS=ON" option will now > use 'long long' integers. > Also, the preceding commit essentially finished the Bunch-Kaufman support: > https://github.com/elemental/Elemental/commit/d2c79c6cfddab6c2ad9c84c49a0ee23b8fe27893 > Jack -------------- next part -------------- A non-text attachment was scrubbed... Name: elemental.py Type: text/x-python Size: 3926 bytes Desc: not available URL: