From er.bhullersingh at gmail.com Thu Jan 4 00:12:29 2018 From: er.bhullersingh at gmail.com (Avtar Singh) Date: Thu, 4 Jan 2018 11:42:29 +0530 Subject: [petsc-users] Solvers for phase field fracture Message-ID: Hello Petsc Users, I am solving a fortran framework to simulate phase-field fracture problem. Upto the failure point the code is working fine. But as the crack-phase starts to propagate, there is abrupt changes in the Stiffness Matrix, so determinant becomes nearly equal to zero. Hence, the code crashes. I tried with mumps package with pc lu. Also tried, superlu, gmres and fgmres with jacobi, bjacobi, lu. But the problem still persist. Can anyone suggest, Which solver and preconditioner should i use? Thank you -- Avtar Singh Research Scholar Multiscale Mechanics and Multiphysics Lab IIT Roorkee, Roorkee -------------- next part -------------- An HTML attachment was scrubbed... URL: From s_g at berkeley.edu Thu Jan 4 01:35:38 2018 From: s_g at berkeley.edu (Sanjay Govindjee) Date: Wed, 3 Jan 2018 23:35:38 -0800 Subject: [petsc-users] Solvers for phase field fracture In-Reply-To: References: Message-ID: <40590950-980a-5141-b01d-1c6d031472c6@berkeley.edu> Are you sure your problem is stable?? Also have you tried adding mass? On 1/3/18 10:12 PM, Avtar Singh wrote: > Hello Petsc Users, > > I am solving a fortran?framework to simulate phase-field fracture > problem. Upto the failure point the code is working fine. But as the > crack-phase starts to propagate, there?is abrupt changes in the > Stiffness Matrix, so determinant becomes nearly equal to zero.? Hence, > the code crashes. > I tried with mumps package with pc lu. Also tried, superlu, gmres?and > fgmres with jacobi, bjacobi, lu. But the problem still persist. > > Can anyone suggest, Which solver and preconditioner should?i?use? > > Thank you > -- > Avtar Singh > Research Scholar > Multiscale Mechanics and Multiphysics Lab > IIT Roorkee, Roorkee -------------- next part -------------- An HTML attachment was scrubbed... URL: From niko.karin at gmail.com Thu Jan 4 02:36:14 2018 From: niko.karin at gmail.com (Karin&NiKo) Date: Thu, 4 Jan 2018 09:36:14 +0100 Subject: [petsc-users] Solvers for phase field fracture In-Reply-To: <40590950-980a-5141-b01d-1c6d031472c6@berkeley.edu> References: <40590950-980a-5141-b01d-1c6d031472c6@berkeley.edu> Message-ID: What you describe is typical of strain softening. In the solid mechanics community, arc length methods are often prefered to solve this kind of problems. The classical references are Riks and Crisfield. You can also take a look at this technique : http://onlinelibrary.wiley.com/doi/10.1002/nme.971/abstract. But I am not aware of the avalability of such approach in the TS objects of PETSc. Nicolas 2018-01-04 8:35 GMT+01:00 Sanjay Govindjee : > Are you sure your problem is stable? Also have you tried adding mass? > > > On 1/3/18 10:12 PM, Avtar Singh wrote: > > Hello Petsc Users, > > I am solving a fortran framework to simulate phase-field fracture problem. > Upto the failure point the code is working fine. But as the crack-phase > starts to propagate, there is abrupt changes in the Stiffness Matrix, so > determinant becomes nearly equal to zero. Hence, the code crashes. > > I tried with mumps package with pc lu. Also tried, superlu, gmres and > fgmres with jacobi, bjacobi, lu. But the problem still persist. > > Can anyone suggest, Which solver and preconditioner should i use? > > Thank you > > -- > Avtar Singh > Research Scholar > Multiscale Mechanics and Multiphysics Lab > IIT Roorkee, Roorkee > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jan 4 04:50:37 2018 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 4 Jan 2018 05:50:37 -0500 Subject: [petsc-users] Solvers for phase field fracture In-Reply-To: References: Message-ID: On Thu, Jan 4, 2018 at 1:12 AM, Avtar Singh wrote: > Hello Petsc Users, > > I am solving a fortran framework to simulate phase-field fracture problem. > Upto the failure point the code is working fine. But as the crack-phase > starts to propagate, there is abrupt changes in the Stiffness Matrix, so > determinant becomes nearly equal to zero. Hence, the code crashes. > I am not sure what formulation you are using, but that does not happen with the Francfort-Marrigo-Bourdin formulation I am familiar with: https://bitbucket.org/litianyi/dynamic-gradient-damage Matt > > I tried with mumps package with pc lu. Also tried, superlu, gmres and > fgmres with jacobi, bjacobi, lu. But the problem still persist. > > Can anyone suggest, Which solver and preconditioner should i use? > > Thank you > > -- > Avtar Singh > Research Scholar > Multiscale Mechanics and Multiphysics Lab > IIT Roorkee, Roorkee > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bourdin at lsu.edu Thu Jan 4 10:32:25 2018 From: bourdin at lsu.edu (Blaise A Bourdin) Date: Thu, 4 Jan 2018 16:32:25 +0000 Subject: [petsc-users] Solvers for phase field fracture In-Reply-To: References: Message-ID: <8C4514F3-D43B-4766-8CC0-C0028B2AE0EF@lsu.edu> Hi Avtar, I am assuming that you are referring to spatial variations of the phase-field variable and that the fracture problem that you are solving is well posed (i.e. displacement controlled, thermal loads, work controlled). As we explained in our 2008 book, force driven problems are often ill posed. You may use an arc-length formulation, of minimize the fracture energy subject to prescribed work, and go back to a force-drive evolution (subject to a safe-load type of condition). In a staggered scheme (alternate minimizations, block Gauss Seidel, adjoint-based quasi-Newton, call it whatever you like), the elasticity problem can become stiff, but remains really manageable. Feel free to try my implementation https://bitbucket.org/bourdin/mef90-sieve for comparison. Blaise On Jan 4, 2018, at 12:12 AM, Avtar Singh > wrote: Hello Petsc Users, I am solving a fortran framework to simulate phase-field fracture problem. Upto the failure point the code is working fine. But as the crack-phase starts to propagate, there is abrupt changes in the Stiffness Matrix, so determinant becomes nearly equal to zero. Hence, the code crashes. I tried with mumps package with pc lu. Also tried, superlu, gmres and fgmres with jacobi, bjacobi, lu. But the problem still persist. Can anyone suggest, Which solver and preconditioner should i use? Thank you -- Avtar Singh Research Scholar Multiscale Mechanics and Multiphysics Lab IIT Roorkee, Roorkee -- Department of Mathematics and Center for Computation & Technology Louisiana State University, Baton Rouge, LA 70803, USA Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin -------------- next part -------------- An HTML attachment was scrubbed... URL: From hbcbh1999 at gmail.com Fri Jan 5 21:43:03 2018 From: hbcbh1999 at gmail.com (Hao Zhang) Date: Fri, 5 Jan 2018 22:43:03 -0500 Subject: [petsc-users] PETSC ERROR: Argument out of range New nonzero at (55, 157) caused a malloc Message-ID: when PETSc crashed and reported this message. it also gives the following message matsetoption(a mat_new_nonzero_allocation_err petsc_false) to turn off this check is this a workaround or? Thanks! -- Hao Zhang Dept. of Applid Mathematics and Statistics, Stony Brook University, Stony Brook, New York, 11790 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jan 5 21:45:06 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 5 Jan 2018 22:45:06 -0500 Subject: [petsc-users] PETSC ERROR: Argument out of range New nonzero at (55, 157) caused a malloc In-Reply-To: References: Message-ID: On Fri, Jan 5, 2018 at 10:43 PM, Hao Zhang wrote: > when PETSc crashed and reported this message. it also gives the following > message > > matsetoption(a mat_new_nonzero_allocation_err petsc_false) to turn off > this check > > > is this a workaround or? Thanks! > 1) Your preallocation is wrong for your matrix 2) If you do not care about performance, then you can use the call to MatSetOption() to remove the error. If you do, please preallocate the matrix correctly. Thanks, Matt > -- > Hao Zhang > Dept. of Applid Mathematics and Statistics, > Stony Brook University, > Stony Brook, New York, 11790 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Jan 6 00:16:44 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Sat, 6 Jan 2018 06:16:44 +0000 Subject: [petsc-users] PETSC ERROR: Argument out of range New nonzero at (55, 157) caused a malloc In-Reply-To: References: Message-ID: <7A6AE834-7466-4C90-B347-E77D307F6BF9@anl.gov> > On Jan 5, 2018, at 9:43 PM, Hao Zhang wrote: > > when PETSc crashed and reported this message. it also gives the following message Correction. PETSC did not "crash" it detected an error condition and stopped, this is very different than a program crashing. > > matsetoption(a mat_new_nonzero_allocation_err petsc_false) to turn off this check Does you code have on particular time when the matrix nonzero structure will change, or will it keep changing once in a while forever as it has more and more time steps? If it only changes once then you can just use the option, if it changes multiple times regularly you will need to understand the mechanism of why the nonzero structure changes before you do anything. For example, do you have moving meshes that form new connections and hence new nonzeros in the matrix. Does the physics change that equations that previously where not coupled become coupled later etc. Barry > > > is this a workaround or? Thanks! > > -- > Hao Zhang > Dept. of Applid Mathematics and Statistics, > Stony Brook University, > Stony Brook, New York, 11790 From f.el-haddad at imperial.ac.uk Mon Jan 8 08:14:30 2018 From: f.el-haddad at imperial.ac.uk (El Haddad, Fadi) Date: Mon, 8 Jan 2018 14:14:30 +0000 Subject: [petsc-users] Convergence test norm Message-ID: Hello Petsc Users, I am using the Fortran framework to create a Jacobian Free solver with SNES. Is there any way to do the convergence test using the infinity Norm instead of the L2 one? Thank you, -- Dr. Fadi EL HADDAD Research Associate Rolls-Royce VUTC team Vibration University Technology Centre City and Guilds Building, room 562 Department of Mechanical Engineering Tel: +44(0)7577171774 Imperial College London Exhibition Road London SW7 2AZ, UK Please consider the environment before printing this e-mail. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Mon Jan 8 10:06:08 2018 From: mfadams at lbl.gov (Mark Adams) Date: Mon, 8 Jan 2018 11:06:08 -0500 Subject: [petsc-users] Convergence test norm In-Reply-To: References: Message-ID: On Mon, Jan 8, 2018 at 9:14 AM, El Haddad, Fadi wrote: > Hello Petsc Users, > > I am using the Fortran framework to create a Jacobian Free solver with > SNES. > > Is there any way to do the convergence test using the infinity Norm > instead of the L2 one? > I don't think so. You can get the linear solver to use other norms, but I don't see a hook for SNES. > > Thank you, > > -- > > Dr. Fadi EL HADDAD > > Research Associate > > > Rolls-Royce VUTC team > Vibration University Technology Centre > City and Guilds Building, room 562 > Department of Mechanical Engineering > > Tel: +44(0)7577171774 <+44%207577%20171774> > > Imperial College London > Exhibition Road > London SW7 2AZ, UK > > Please consider the environment before printing this e-mail. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Jan 8 10:39:47 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Mon, 8 Jan 2018 16:39:47 +0000 Subject: [petsc-users] Convergence test norm In-Reply-To: References: Message-ID: <853E3A2F-DEAC-45D1-94BF-054FCD6C6E08@anl.gov> I have added a branch in the PETSc git repository barry/add-ftn-linfinity-conv-test where I add such a test to src/snes/examples/tutorials/ex5f.F90 Barry Note that I had to fix one thing in PETSc for this to work so you cannot just take the example and use it with the current release, you need to use the branch I provided. Soon that branch will go into the master branch and then eventually the next release. > On Jan 8, 2018, at 8:14 AM, El Haddad, Fadi wrote: > > Hello Petsc Users, > > I am using the Fortran framework to create a Jacobian Free solver with SNES. > > Is there any way to do the convergence test using the infinity Norm instead of the L2 one? > > Thank you, > > -- > Dr. Fadi EL HADDAD > Research Associate > > Rolls-Royce VUTC team > Vibration University Technology Centre > City and Guilds Building, room 562 > Department of Mechanical Engineering > > Tel: +44(0)7577171774 > > Imperial College London > Exhibition Road > London SW7 2AZ, UK > > Please consider the environment before printing this e-mail. From alexlindsay239 at gmail.com Mon Jan 8 14:59:43 2018 From: alexlindsay239 at gmail.com (Alexander Lindsay) Date: Mon, 8 Jan 2018 13:59:43 -0700 Subject: [petsc-users] Context behind SNESComputeFunction call Message-ID: Is there any elegant way to tell whether SNESComputeFunction is being called under different conceptual contexts? E.g. non-linear residual evaluation vs. Jacobian formation from finite differencing vs. Jacobian-vector products from finite differencing? Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Jan 8 15:15:17 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Mon, 8 Jan 2018 21:15:17 +0000 Subject: [petsc-users] Context behind SNESComputeFunction call In-Reply-To: References: Message-ID: > On Jan 8, 2018, at 2:59 PM, Alexander Lindsay wrote: > > Is there any elegant way to tell whether SNESComputeFunction is being called under different conceptual contexts? > > E.g. non-linear residual evaluation vs. Jacobian formation from finite differencing vs. Jacobian-vector products from finite differencing? Under normal usage with the options database no. If you have some reason to know you could write three functions and provide them to SNESSetFunction(), MatMFFDSetFunction(), and MatFDColoringSetFunction(). Note that these functions have slightly different calling sequences but you can have all of them call the same underlying common function if you are only interested in, for example, how many times the function is used for each purpose. Barry > > Alex From matteo.semplice at unito.it Tue Jan 9 00:53:58 2018 From: matteo.semplice at unito.it (Matteo Semplice) Date: Tue, 9 Jan 2018 07:53:58 +0100 Subject: [petsc-users] local/global Vec for SNES function/jacobian Message-ID: Dear all, ??? I could not find info on the manual but from the examples and my own trying I guess that the vectors involved in the SNESfunction should be global ones and that, if one needs ghosts values to compute the function and/or the jacobian, should do a GlobalToLocal communication inside the function. Correct? (I get errors of inconsistent dimensions between Vecs and Mats if I call SNESSolve(snes,b, localU)...) ??? Since I need the ghost values both in the function and in the jacobian evaluation, would it be safe (and worthwhile) to do define a local vector in the ctx, perform the communication in the function and using the vec in the ctx in the jacobian evaluation (without performing again the communication in the SNESJacobianFunction?)... Put in other words, is it guaranteed that each call to the jacobian function is preceded by a call to the function with the same x argument? Thanks, ??? Matteo Semplice From bsmith at mcs.anl.gov Tue Jan 9 00:59:42 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 9 Jan 2018 06:59:42 +0000 Subject: [petsc-users] local/global Vec for SNES function/jacobian In-Reply-To: References: Message-ID: <35FF96ED-574C-4954-8045-492871D2C8F2@anl.gov> > On Jan 9, 2018, at 12:53 AM, Matteo Semplice wrote: > > Dear all, > > I could not find info on the manual but from the examples and my own trying I guess that the vectors involved in the SNESfunction should be global ones and that, if one needs ghosts values to compute the function and/or the jacobian, should do a GlobalToLocal communication inside the function. Correct? Absolutely > (I get errors of inconsistent dimensions between Vecs and Mats if I call SNESSolve(snes,b, localU)...) > > Since I need the ghost values both in the function and in the jacobian evaluation, would it be safe (and worthwhile) to do define a local vector in the ctx, perform the communication in the function and using the vec in the ctx in the jacobian evaluation (without performing again the communication in the SNESJacobianFunction?)... Put in other words, is it guaranteed that each call to the jacobian function is preceded by a call to the function with the same x argument? I would not rely on this. It is an unnecessary and risky optimization. Just treat the function and Jacobian evaluations as independent operations. Barry > > Thanks, > Matteo Semplice From hbuesing at eonerc.rwth-aachen.de Tue Jan 9 07:59:47 2018 From: hbuesing at eonerc.rwth-aachen.de (Buesing, Henrik) Date: Tue, 9 Jan 2018 13:59:47 +0000 Subject: [petsc-users] Geometric multigrid with a cell-centered discretization Message-ID: Dear all, I am using DMDACreate3d to generate my domain and a cell-centered two-point flux approximation as discretization. I use geometric Multigrid (-pc_type mg) with DMDASetInterpolationType(petsc_da, DMDA_Q0, petsc_ierr). With DM_BOUNDARY_GHOSTED as boundary type in DMDACreate3d, I get [0]PETSC ERROR: Arguments are incompatible [0]PETSC ERROR: Ratio between levels: (mx - 1)/(Mx - 1) must be integer: mx 1536 Mx 768 With DM_BOUNDARY_PERIODIC, I get invalid accesses when I assemble the matrix. Can I somehow tell him, that I have a cell-centered grid, or do I have to resort to using N+1 in my dimensions? Thank you! Henrik -- Dipl.-Math. Henrik B?sing Institute for Applied Geophysics and Geothermal Energy E.ON Energy Research Center RWTH Aachen University ------------------------------------------------------ Mathieustr. 10 | Tel +49 (0)241 80 49907 52074 Aachen, Germany | Fax +49 (0)241 80 49889 ------------------------------------------------------ http://www.eonerc.rwth-aachen.de/GGE hbuesing at eonerc.rwth-aachen.de ------------------------------------------------------ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jan 9 08:10:01 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 9 Jan 2018 14:10:01 +0000 Subject: [petsc-users] Geometric multigrid with a cell-centered discretization In-Reply-To: References: Message-ID: <8BDBA8E7-50C2-4A15-8114-C8991CCD62B9@anl.gov> With cell centered elements each element is sliced in half in each of the three directions for refinement (or you can refine only in 1 or 2 of the dimensions if you like). This means that the ratio of mx to MX needs to be 2 (or 4).So your numbers are fine. > On Jan 9, 2018, at 7:59 AM, Buesing, Henrik wrote: > > Dear all, > > I am using DMDACreate3d to generate my domain and a cell-centered two-point flux approximation as discretization. I use geometric Multigrid (-pc_type mg) with DMDASetInterpolationType(petsc_da, DMDA_Q0, petsc_ierr). With DM_BOUNDARY_GHOSTED as boundary type in DMDACreate3d, I get > > [0]PETSC ERROR: Arguments are incompatible > [0]PETSC ERROR: Ratio between levels: (mx - 1)/(Mx - 1) must be integer: mx 1536 Mx 768 We need ALL the output from the error (always). It should never be doing this check in your case so something is wrong with the order of your code perhaps. Barry > > With DM_BOUNDARY_PERIODIC, I get invalid accesses when I assemble the matrix. > > Can I somehow tell him, that I have a cell-centered grid, or do I have to resort to using N+1 in my dimensions? > > Thank you! > Henrik > > > -- > Dipl.-Math. Henrik B?sing > Institute for Applied Geophysics and Geothermal Energy > E.ON Energy Research Center > RWTH Aachen University > ------------------------------------------------------ > Mathieustr. 10 | Tel +49 (0)241 80 49907 > 52074 Aachen, Germany | Fax +49 (0)241 80 49889 > ------------------------------------------------------ > http://www.eonerc.rwth-aachen.de/GGE > hbuesing at eonerc.rwth-aachen.de > ------------------------------------------------------ From hbuesing at eonerc.rwth-aachen.de Tue Jan 9 09:12:49 2018 From: hbuesing at eonerc.rwth-aachen.de (Buesing, Henrik) Date: Tue, 9 Jan 2018 15:12:49 +0000 Subject: [petsc-users] Geometric multigrid with a cell-centered discretization In-Reply-To: <8BDBA8E7-50C2-4A15-8114-C8991CCD62B9@anl.gov> References: <8BDBA8E7-50C2-4A15-8114-C8991CCD62B9@anl.gov> Message-ID: Sorry (again), for posting only part of the error message... So, here it comes. I will first [1] show the code snippets I am using, then second [2] I will show the PETSc options, and finally [3] the error message I get. The grid is 1536 x 1 x 256 in x,y,z-direction. Thus, we see the first level of coarsening in x-direction here. Thank you! Henrik [1] CALL DMDACreate3d(PETSC_COMM_WORLD,DM_BOUNDARY_GHOSTED,DM_BOUNDARY_GHOSTED,DM_BOUNDARY_GHOSTED,& DMDA_STENCIL_STAR, I0g,J0g,K0g, PETSC_DECIDE, PETSC_DECIDE,PETSC_DECIDE, petsc_two, petsc_one, & PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER, petsc_da, petsc_ierr) CALL DMSetUp(petsc_da,petsc_ierr) ... CALL DMSetFromOptions(petsc_da, petsc_ierr) CALL DMDASetInterpolationType(petsc_da, DMDA_Q0, petsc_ierr) CALL SNESSetDM(petsc_snes, petsc_da, petsc_ierr) CALL SNESSetFromOptions(petsc_snes,petsc_ierr) [2] -snes_monitor -snes_linesearch_monitor -snes_type newtonls -snes_linesearch_type basic -ksp_type fgmres -ksp_max_it 1000 -snes_stol 1e-06 -snes_rtol 1e-06 -snes_atol 1e-6 -ksp_gmres_restart 100 -ksp_monitor -pc_type mg -da_refine_y 1 -ksp_converged_reason -snes_converged_reason -ksp_monitor -ksp_view -pc_mg_levels 2 -options_left [3] [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Arguments are incompatible [0]PETSC ERROR: Ratio between levels: (mx - 1)/(Mx - 1) must be integer: mx 1536 Mx 768 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.8.2-48-g851ec02 GIT Date: 2017-12-05 09:52:17 -0600 [0]PETSC ERROR: /rwthfs/rz/cluster/work/hb111949/EoCoE/aixcelerate17/gge_gnu/bench_run/000023/000000_copy_sub_launch/work/shem_fw64gnu_const.x on a gnu_openmpi named linuxbmc0008.rz.RWTH-Aachen.DE by hb111949 Tue Jan 9 16:04:39 2018 [0]PETSC ERROR: Configure options --download-fblaslapack --with-cc=mpicc -with-fc=mpif90 --with-cxx=mpicxx --download-hypre --download-superlu_dist --download-suitesparse --download-scalapack --download-blacs --download-hdf5 --download-parmetis --download-metis --with-debugging=0 --download-mumps [0]PETSC ERROR: #1 DMCreateInjection_DA_3D() line 1195 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dainterp.c [0]PETSC ERROR: #2 DMCreateInjection_DA() line 1283 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dainterp.c [0]PETSC ERROR: #3 DMCreateInjection() line 1082 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c [0]PETSC ERROR: #4 DMCoarsen_DA() line 1219 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/da.c [0]PETSC ERROR: #5 DMCoarsen() line 2427 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c [0]PETSC ERROR: #6 PCSetUp_MG() line 618 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: #7 PCSetUp() line 924 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #8 KSPSetUp() line 381 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #9 KSPSolve() line 612 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- -- Dipl.-Math. Henrik B?sing Institute for Applied Geophysics and Geothermal Energy E.ON Energy Research Center RWTH Aachen University ------------------------------------------------------ Mathieustr. 10 | Tel +49 (0)241 80 49907 52074 Aachen, Germany | Fax +49 (0)241 80 49889 ------------------------------------------------------ http://www.eonerc.rwth-aachen.de/GGE hbuesing at eonerc.rwth-aachen.de ------------------------------------------------------ > -----Urspr?ngliche Nachricht----- > Von: Smith, Barry F. [mailto:bsmith at mcs.anl.gov] > Gesendet: 09 January 2018 15:10 > An: Buesing, Henrik > Cc: petsc-users > Betreff: Re: [petsc-users] Geometric multigrid with a cell-centered > discretization > > > With cell centered elements each element is sliced in half in each of > the three directions for refinement (or you can refine only in 1 or 2 of > the dimensions if you like). This means that the ratio of mx to MX needs > to be 2 (or 4).So your numbers are fine. > > > On Jan 9, 2018, at 7:59 AM, Buesing, Henrik aachen.de> wrote: > > > > Dear all, > > > > I am using DMDACreate3d to generate my domain and a cell-centered > > two-point flux approximation as discretization. I use geometric > > Multigrid (-pc_type mg) with DMDASetInterpolationType(petsc_da, > > DMDA_Q0, petsc_ierr). With DM_BOUNDARY_GHOSTED as boundary type in > > DMDACreate3d, I get > > > > [0]PETSC ERROR: Arguments are incompatible [0]PETSC ERROR: Ratio > > between levels: (mx - 1)/(Mx - 1) must be integer: mx 1536 Mx 768 > > We need ALL the output from the error (always). It should never be > doing this check in your case so something is wrong with the order of > your code perhaps. > > Barry > > > > > With DM_BOUNDARY_PERIODIC, I get invalid accesses when I assemble the > matrix. > > > > Can I somehow tell him, that I have a cell-centered grid, or do I have > to resort to using N+1 in my dimensions? > > > > Thank you! > > Henrik > > > > > > -- > > Dipl.-Math. Henrik B?sing > > Institute for Applied Geophysics and Geothermal Energy E.ON Energy > > Research Center RWTH Aachen University > > ------------------------------------------------------ > > Mathieustr. 10 | Tel +49 (0)241 80 49907 > > 52074 Aachen, Germany | Fax +49 (0)241 80 49889 > > ------------------------------------------------------ > > http://www.eonerc.rwth-aachen.de/GGE > > hbuesing at eonerc.rwth-aachen.de > > ------------------------------------------------------ From mfadams at lbl.gov Tue Jan 9 09:20:32 2018 From: mfadams at lbl.gov (Mark Adams) Date: Tue, 9 Jan 2018 10:20:32 -0500 Subject: [petsc-users] Geometric multigrid with a cell-centered discretization In-Reply-To: References: <8BDBA8E7-50C2-4A15-8114-C8991CCD62B9@anl.gov> Message-ID: On Tue, Jan 9, 2018 at 10:12 AM, Buesing, Henrik < hbuesing at eonerc.rwth-aachen.de> wrote: > Sorry (again), for posting only part of the error message... > > So, here it comes. I will first [1] show the code snippets I am using, > then second [2] I will show the PETSc options, and finally [3] the error > message I get. > The grid is 1536 x 1 x 256 in x,y,z-direction. Thus, we see the first > level of coarsening in x-direction here. > > Thank you! > Henrik > > [1] > CALL DMDACreate3d(PETSC_COMM_WORLD,DM_BOUNDARY_GHOSTED,DM_ > BOUNDARY_GHOSTED,DM_BOUNDARY_GHOSTED,& > DMDA_STENCIL_STAR, I0g,J0g,K0g, PETSC_DECIDE, > PETSC_DECIDE,PETSC_DECIDE, petsc_two, petsc_one, & > PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER, > petsc_da, petsc_ierr) > CALL DMSetUp(petsc_da,petsc_ierr) > ... > CALL DMSetFromOptions(petsc_da, petsc_ierr) > CALL DMDASetInterpolationType(petsc_da, DMDA_Q0, petsc_ierr) > CALL SNESSetDM(petsc_snes, petsc_da, petsc_ierr) > CALL SNESSetFromOptions(petsc_snes,petsc_ierr) > > [2] -snes_monitor -snes_linesearch_monitor -snes_type newtonls > -snes_linesearch_type basic -ksp_type fgmres -ksp_max_it 1000 -snes_stol > 1e-06 -snes_rtol 1e-06 -snes_atol 1e-6 -ksp_gmres_restart 100 -ksp_monitor > -pc_type mg -da_refine_y 1 -ksp_converged_reason -snes_converged_reason > -ksp_monitor -ksp_view -pc_mg_levels 2 -options_left > > [3] > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Arguments are incompatible > [0]PETSC ERROR: Ratio between levels: (mx - 1)/(Mx - 1) must be integer: > mx 1536 Mx 768 > It looks like you give it the number of vertices, not cells, so +1 to your grid size. > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.8.2-48-g851ec02 GIT > Date: 2017-12-05 09:52:17 -0600 > [0]PETSC ERROR: /rwthfs/rz/cluster/work/hb111949/EoCoE/aixcelerate17/ > gge_gnu/bench_run/000023/000000_copy_sub_launch/work/shem_fw64gnu_const.x > on a gnu_openmpi named linuxbmc0008.rz.RWTH-Aachen.DE by hb111949 Tue > Jan 9 16:04:39 2018 > [0]PETSC ERROR: Configure options --download-fblaslapack --with-cc=mpicc > -with-fc=mpif90 --with-cxx=mpicxx --download-hypre --download-superlu_dist > --download-suitesparse --download-scalapack --download-blacs > --download-hdf5 --download-parmetis --download-metis --with-debugging=0 > --download-mumps > [0]PETSC ERROR: #1 DMCreateInjection_DA_3D() line 1195 in > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dainterp.c > [0]PETSC ERROR: #2 DMCreateInjection_DA() line 1283 in > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dainterp.c > [0]PETSC ERROR: #3 DMCreateInjection() line 1082 in > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c > [0]PETSC ERROR: #4 DMCoarsen_DA() line 1219 in /rwthfs/rz/cluster/work/ > hb111949/Code/petsc/src/dm/impls/da/da.c > [0]PETSC ERROR: #5 DMCoarsen() line 2427 in /rwthfs/rz/cluster/work/ > hb111949/Code/petsc/src/dm/interface/dm.c > [0]PETSC ERROR: #6 PCSetUp_MG() line 618 in /rwthfs/rz/cluster/work/ > hb111949/Code/petsc/src/ksp/pc/impls/mg/mg.c > [0]PETSC ERROR: #7 PCSetUp() line 924 in /rwthfs/rz/cluster/work/ > hb111949/Code/petsc/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #8 KSPSetUp() line 381 in /rwthfs/rz/cluster/work/ > hb111949/Code/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #9 KSPSolve() line 612 in /rwthfs/rz/cluster/work/ > hb111949/Code/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > > -- > Dipl.-Math. Henrik B?sing > Institute for Applied Geophysics and Geothermal Energy > E.ON Energy Research Center > RWTH Aachen University > ------------------------------------------------------ > Mathieustr. 10 | Tel +49 (0)241 80 49907 > 52074 Aachen, Germany | Fax +49 (0)241 80 49889 > ------------------------------------------------------ > http://www.eonerc.rwth-aachen.de/GGE > hbuesing at eonerc.rwth-aachen.de > ------------------------------------------------------ > > > -----Urspr?ngliche Nachricht----- > > Von: Smith, Barry F. [mailto:bsmith at mcs.anl.gov] > > Gesendet: 09 January 2018 15:10 > > An: Buesing, Henrik > > Cc: petsc-users > > Betreff: Re: [petsc-users] Geometric multigrid with a cell-centered > > discretization > > > > > > With cell centered elements each element is sliced in half in each of > > the three directions for refinement (or you can refine only in 1 or 2 of > > the dimensions if you like). This means that the ratio of mx to MX needs > > to be 2 (or 4).So your numbers are fine. > > > > > On Jan 9, 2018, at 7:59 AM, Buesing, Henrik > aachen.de> wrote: > > > > > > Dear all, > > > > > > I am using DMDACreate3d to generate my domain and a cell-centered > > > two-point flux approximation as discretization. I use geometric > > > Multigrid (-pc_type mg) with DMDASetInterpolationType(petsc_da, > > > DMDA_Q0, petsc_ierr). With DM_BOUNDARY_GHOSTED as boundary type in > > > DMDACreate3d, I get > > > > > > [0]PETSC ERROR: Arguments are incompatible [0]PETSC ERROR: Ratio > > > between levels: (mx - 1)/(Mx - 1) must be integer: mx 1536 Mx 768 > > > > We need ALL the output from the error (always). It should never be > > doing this check in your case so something is wrong with the order of > > your code perhaps. > > > > Barry > > > > > > > > With DM_BOUNDARY_PERIODIC, I get invalid accesses when I assemble the > > matrix. > > > > > > Can I somehow tell him, that I have a cell-centered grid, or do I have > > to resort to using N+1 in my dimensions? > > > > > > Thank you! > > > Henrik > > > > > > > > > -- > > > Dipl.-Math. Henrik B?sing > > > Institute for Applied Geophysics and Geothermal Energy E.ON Energy > > > Research Center RWTH Aachen University > > > ------------------------------------------------------ > > > Mathieustr. 10 | Tel +49 (0)241 80 49907 > > > 52074 Aachen, Germany | Fax +49 (0)241 80 49889 > > > ------------------------------------------------------ > > > http://www.eonerc.rwth-aachen.de/GGE > > > hbuesing at eonerc.rwth-aachen.de > > > ------------------------------------------------------ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hbuesing at eonerc.rwth-aachen.de Tue Jan 9 09:28:02 2018 From: hbuesing at eonerc.rwth-aachen.de (Buesing, Henrik) Date: Tue, 9 Jan 2018 15:28:02 +0000 Subject: [petsc-users] Geometric multigrid with a cell-centered discretization In-Reply-To: References: <8BDBA8E7-50C2-4A15-8114-C8991CCD62B9@anl.gov> Message-ID: [3] [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Arguments are incompatible [0]PETSC ERROR: Ratio between levels: (mx - 1)/(Mx - 1) must be integer: mx 1536 Mx 768 It looks like you give it the number of vertices, not cells, so +1 to your grid size. [Buesing, Henrik] Yes! But, I do not want that. I want to give the cells? [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.8.2-48-g851ec02 GIT Date: 2017-12-05 09:52:17 -0600 [0]PETSC ERROR: /rwthfs/rz/cluster/work/hb111949/EoCoE/aixcelerate17/gge_gnu/bench_run/000023/000000_copy_sub_launch/work/shem_fw64gnu_const.x on a gnu_openmpi named linuxbmc0008.rz.RWTH-Aachen.DE by hb111949 Tue Jan 9 16:04:39 2018 [0]PETSC ERROR: Configure options --download-fblaslapack --with-cc=mpicc -with-fc=mpif90 --with-cxx=mpicxx --download-hypre --download-superlu_dist --download-suitesparse --download-scalapack --download-blacs --download-hdf5 --download-parmetis --download-metis --with-debugging=0 --download-mumps [0]PETSC ERROR: #1 DMCreateInjection_DA_3D() line 1195 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dainterp.c [0]PETSC ERROR: #2 DMCreateInjection_DA() line 1283 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dainterp.c [0]PETSC ERROR: #3 DMCreateInjection() line 1082 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c [0]PETSC ERROR: #4 DMCoarsen_DA() line 1219 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/da.c [0]PETSC ERROR: #5 DMCoarsen() line 2427 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c [0]PETSC ERROR: #6 PCSetUp_MG() line 618 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: #7 PCSetUp() line 924 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #8 KSPSetUp() line 381 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #9 KSPSolve() line 612 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- -- Dipl.-Math. Henrik B?sing Institute for Applied Geophysics and Geothermal Energy E.ON Energy Research Center RWTH Aachen University ------------------------------------------------------ Mathieustr. 10 | Tel +49 (0)241 80 49907 52074 Aachen, Germany | Fax +49 (0)241 80 49889 ------------------------------------------------------ http://www.eonerc.rwth-aachen.de/GGE hbuesing at eonerc.rwth-aachen.de ------------------------------------------------------ > -----Urspr?ngliche Nachricht----- > Von: Smith, Barry F. [mailto:bsmith at mcs.anl.gov] > Gesendet: 09 January 2018 15:10 > An: Buesing, Henrik > > Cc: petsc-users > > Betreff: Re: [petsc-users] Geometric multigrid with a cell-centered > discretization > > > With cell centered elements each element is sliced in half in each of > the three directions for refinement (or you can refine only in 1 or 2 of > the dimensions if you like). This means that the ratio of mx to MX needs > to be 2 (or 4).So your numbers are fine. > > > On Jan 9, 2018, at 7:59 AM, Buesing, Henrik > aachen.de> wrote: > > > > Dear all, > > > > I am using DMDACreate3d to generate my domain and a cell-centered > > two-point flux approximation as discretization. I use geometric > > Multigrid (-pc_type mg) with DMDASetInterpolationType(petsc_da, > > DMDA_Q0, petsc_ierr). With DM_BOUNDARY_GHOSTED as boundary type in > > DMDACreate3d, I get > > > > [0]PETSC ERROR: Arguments are incompatible [0]PETSC ERROR: Ratio > > between levels: (mx - 1)/(Mx - 1) must be integer: mx 1536 Mx 768 > > We need ALL the output from the error (always). It should never be > doing this check in your case so something is wrong with the order of > your code perhaps. > > Barry > > > > > With DM_BOUNDARY_PERIODIC, I get invalid accesses when I assemble the > matrix. > > > > Can I somehow tell him, that I have a cell-centered grid, or do I have > to resort to using N+1 in my dimensions? > > > > Thank you! > > Henrik > > > > > > -- > > Dipl.-Math. Henrik B?sing > > Institute for Applied Geophysics and Geothermal Energy E.ON Energy > > Research Center RWTH Aachen University > > ------------------------------------------------------ > > Mathieustr. 10 | Tel +49 (0)241 80 49907 > > 52074 Aachen, Germany | Fax +49 (0)241 80 49889 > > ------------------------------------------------------ > > http://www.eonerc.rwth-aachen.de/GGE > > hbuesing at eonerc.rwth-aachen.de > > ------------------------------------------------------ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Tue Jan 9 09:29:33 2018 From: mfadams at lbl.gov (Mark Adams) Date: Tue, 9 Jan 2018 10:29:33 -0500 Subject: [petsc-users] Geometric multigrid with a cell-centered discretization In-Reply-To: References: <8BDBA8E7-50C2-4A15-8114-C8991CCD62B9@anl.gov> Message-ID: On Tue, Jan 9, 2018 at 10:28 AM, Buesing, Henrik < hbuesing at eonerc.rwth-aachen.de> wrote: > [3] > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Arguments are incompatible > [0]PETSC ERROR: Ratio between levels: (mx - 1)/(Mx - 1) must be integer: > mx 1536 Mx 768 > > > > It looks like you give it the number of vertices, not cells, so +1 to your > grid size. > > > [Buesing, Henrik] Yes! But, I do not want that. I want to give the cells? > That is the interface. You will get m-1 cells so give it m = num_cells + 1. > > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.8.2-48-g851ec02 GIT > Date: 2017-12-05 09:52:17 -0600 > [0]PETSC ERROR: /rwthfs/rz/cluster/work/hb111949/EoCoE/aixcelerate17/ > gge_gnu/bench_run/000023/000000_copy_sub_launch/work/shem_fw64gnu_const.x > on a gnu_openmpi named linuxbmc0008.rz.RWTH-Aachen.DE by hb111949 Tue > Jan 9 16:04:39 2018 > [0]PETSC ERROR: Configure options --download-fblaslapack --with-cc=mpicc > -with-fc=mpif90 --with-cxx=mpicxx --download-hypre --download-superlu_dist > --download-suitesparse --download-scalapack --download-blacs > --download-hdf5 --download-parmetis --download-metis --with-debugging=0 > --download-mumps > [0]PETSC ERROR: #1 DMCreateInjection_DA_3D() line 1195 in > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dainterp.c > [0]PETSC ERROR: #2 DMCreateInjection_DA() line 1283 in > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dainterp.c > [0]PETSC ERROR: #3 DMCreateInjection() line 1082 in > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c > [0]PETSC ERROR: #4 DMCoarsen_DA() line 1219 in /rwthfs/rz/cluster/work/ > hb111949/Code/petsc/src/dm/impls/da/da.c > [0]PETSC ERROR: #5 DMCoarsen() line 2427 in /rwthfs/rz/cluster/work/ > hb111949/Code/petsc/src/dm/interface/dm.c > [0]PETSC ERROR: #6 PCSetUp_MG() line 618 in /rwthfs/rz/cluster/work/ > hb111949/Code/petsc/src/ksp/pc/impls/mg/mg.c > [0]PETSC ERROR: #7 PCSetUp() line 924 in /rwthfs/rz/cluster/work/ > hb111949/Code/petsc/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #8 KSPSetUp() line 381 in /rwthfs/rz/cluster/work/ > hb111949/Code/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #9 KSPSolve() line 612 in /rwthfs/rz/cluster/work/ > hb111949/Code/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > > -- > Dipl.-Math. Henrik B?sing > Institute for Applied Geophysics and Geothermal Energy > E.ON Energy Research Center > RWTH Aachen University > ------------------------------------------------------ > Mathieustr. 10 | Tel +49 (0)241 80 49907 > 52074 Aachen, Germany | Fax +49 (0)241 80 49889 > ------------------------------------------------------ > http://www.eonerc.rwth-aachen.de/GGE > hbuesing at eonerc.rwth-aachen.de > ------------------------------------------------------ > > > -----Urspr?ngliche Nachricht----- > > Von: Smith, Barry F. [mailto:bsmith at mcs.anl.gov] > > Gesendet: 09 January 2018 15:10 > > An: Buesing, Henrik > > Cc: petsc-users > > Betreff: Re: [petsc-users] Geometric multigrid with a cell-centered > > discretization > > > > > > > With cell centered elements each element is sliced in half in each of > > the three directions for refinement (or you can refine only in 1 or 2 of > > the dimensions if you like). This means that the ratio of mx to MX needs > > to be 2 (or 4).So your numbers are fine. > > > > > On Jan 9, 2018, at 7:59 AM, Buesing, Henrik > aachen.de> wrote: > > > > > > Dear all, > > > > > > I am using DMDACreate3d to generate my domain and a cell-centered > > > two-point flux approximation as discretization. I use geometric > > > Multigrid (-pc_type mg) with DMDASetInterpolationType(petsc_da, > > > DMDA_Q0, petsc_ierr). With DM_BOUNDARY_GHOSTED as boundary type in > > > DMDACreate3d, I get > > > > > > [0]PETSC ERROR: Arguments are incompatible [0]PETSC ERROR: Ratio > > > between levels: (mx - 1)/(Mx - 1) must be integer: mx 1536 Mx 768 > > > > We need ALL the output from the error (always). It should never be > > doing this check in your case so something is wrong with the order of > > your code perhaps. > > > > Barry > > > > > > > > With DM_BOUNDARY_PERIODIC, I get invalid accesses when I assemble the > > matrix. > > > > > > Can I somehow tell him, that I have a cell-centered grid, or do I have > > to resort to using N+1 in my dimensions? > > > > > > Thank you! > > > Henrik > > > > > > > > > -- > > > Dipl.-Math. Henrik B?sing > > > Institute for Applied Geophysics and Geothermal Energy E.ON Energy > > > Research Center RWTH Aachen University > > > ------------------------------------------------------ > > > Mathieustr. 10 | Tel +49 (0)241 80 49907 > <+49%20241%208049907> > > > 52074 Aachen, Germany | Fax +49 (0)241 80 49889 > <+49%20241%208049889> > > > ------------------------------------------------------ > > > http://www.eonerc.rwth-aachen.de/GGE > > > hbuesing at eonerc.rwth-aachen.de > > > ------------------------------------------------------ > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hbuesing at eonerc.rwth-aachen.de Tue Jan 9 09:30:11 2018 From: hbuesing at eonerc.rwth-aachen.de (Buesing, Henrik) Date: Tue, 9 Jan 2018 15:30:11 +0000 Subject: [petsc-users] Geometric multigrid with a cell-centered discretization In-Reply-To: References: <8BDBA8E7-50C2-4A15-8114-C8991CCD62B9@anl.gov> Message-ID: Correction to [3]. This actually is the whole stack down to SNESSolve... [3] [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Arguments are incompatible [0]PETSC ERROR: Ratio between levels: (mx - 1)/(Mx - 1) must be integer: mx 1536 Mx 768 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.8.2-48-g851ec02 GIT Date: 2017-12-05 09:52:17 -0600 [0]PETSC ERROR: shem_fw64gnu_const.x on a gnu_openmpi named cluster-x.rz.RWTH-Aachen.DE by hb111949 Tue Jan 9 16:26:56 2018 [0]PETSC ERROR: Configure options --download-fblaslapack --with-cc=mpicc -with-fc=mpif90 --with-cxx=mpicxx --download-hypre --download-superlu_dist --download-suitesparse --download-scalapack --download-blacs --download-hdf5 --download-parmetis --download-metis --with-debugging=0 --download-mumps [0]PETSC ERROR: #1 DMCreateInjection_DA_3D() line 1195 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dainterp.c [0]PETSC ERROR: #2 DMCreateInjection_DA() line 1283 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dainterp.c [0]PETSC ERROR: #3 DMCreateInjection() line 1082 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c [0]PETSC ERROR: #4 DMCoarsen_DA() line 1219 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/da.c [0]PETSC ERROR: #5 DMCoarsen() line 2427 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c [0]PETSC ERROR: #6 PCSetUp_MG() line 618 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: #7 PCSetUp() line 924 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #8 KSPSetUp() line 381 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #9 KSPSolve() line 612 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #10 SNESSolve_NEWTONLS() line 224 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/snes/impls/ls/ls.c [0]PETSC ERROR: #11 SNESSolve() line 4179 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/snes/interface/snes.c [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.8.2-48-g851ec02 GIT Date: 2017-12-05 09:52:17 -0600 [0]PETSC ERROR: shem_fw64gnu_const.x on a gnu_openmpi named cluster-x.rz.RWTH-Aachen.DE by hb111949 Tue Jan 9 16:26:56 2018 [0]PETSC ERROR: Configure options --download-fblaslapack --with-cc=mpicc -with-fc=mpif90 --with-cxx=mpicxx --download-hypre --download-superlu_dist --download-suitesparse --download-scalapack --download-blacs --download-hdf5 --download-parmetis --download-metis --with-debugging=0 --download-mumps [0]PETSC ERROR: #12 User provided function() line 0 in unknown file -------------------------------------------------------------------------- -- Dipl.-Math. Henrik B?sing Institute for Applied Geophysics and Geothermal Energy E.ON Energy Research Center RWTH Aachen University ------------------------------------------------------ Mathieustr. 10 | Tel +49 (0)241 80 49907 52074 Aachen, Germany | Fax +49 (0)241 80 49889 ------------------------------------------------------ http://www.eonerc.rwth-aachen.de/GGE hbuesing at eonerc.rwth-aachen.de ------------------------------------------------------ > -----Urspr?ngliche Nachricht----- > Von: petsc-users [mailto:petsc-users-bounces at mcs.anl.gov] Im Auftrag von > Buesing, Henrik > Gesendet: 09 January 2018 16:13 > An: 'Smith, Barry F.' > Cc: petsc-users > Betreff: Re: [petsc-users] Geometric multigrid with a cell-centered > discretization > > Sorry (again), for posting only part of the error message... > > So, here it comes. I will first [1] show the code snippets I am using, > then second [2] I will show the PETSc options, and finally [3] the error > message I get. > The grid is 1536 x 1 x 256 in x,y,z-direction. Thus, we see the first > level of coarsening in x-direction here. > > Thank you! > Henrik > > [1] > CALL > DMDACreate3d(PETSC_COMM_WORLD,DM_BOUNDARY_GHOSTED,DM_BOUNDARY_GHOSTED,DM > _BOUNDARY_GHOSTED,& > DMDA_STENCIL_STAR, I0g,J0g,K0g, PETSC_DECIDE, > PETSC_DECIDE,PETSC_DECIDE, petsc_two, petsc_one, & > > PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER, petsc_da, > petsc_ierr) > CALL DMSetUp(petsc_da,petsc_ierr) ... > CALL DMSetFromOptions(petsc_da, petsc_ierr) > CALL DMDASetInterpolationType(petsc_da, DMDA_Q0, petsc_ierr) > CALL SNESSetDM(petsc_snes, petsc_da, petsc_ierr) > CALL SNESSetFromOptions(petsc_snes,petsc_ierr) > > [2] -snes_monitor -snes_linesearch_monitor -snes_type newtonls - > snes_linesearch_type basic -ksp_type fgmres -ksp_max_it 1000 -snes_stol > 1e-06 -snes_rtol 1e-06 -snes_atol 1e-6 -ksp_gmres_restart 100 - > ksp_monitor -pc_type mg -da_refine_y 1 -ksp_converged_reason - > snes_converged_reason -ksp_monitor -ksp_view -pc_mg_levels 2 - > options_left > > [3] > [0]PETSC ERROR: --------------------- Error Message -------------------- > ------------------------------------------ > [0]PETSC ERROR: Arguments are incompatible [0]PETSC ERROR: Ratio between > levels: (mx - 1)/(Mx - 1) must be integer: mx 1536 Mx 768 [0]PETSC > ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for > trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.8.2-48-g851ec02 GIT > Date: 2017-12-05 09:52:17 -0600 [0]PETSC ERROR: > /rwthfs/rz/cluster/work/hb111949/EoCoE/aixcelerate17/gge_gnu/bench_run/0 > 00023/000000_copy_sub_launch/work/shem_fw64gnu_const.x on a gnu_openmpi > named linuxbmc0008.rz.RWTH-Aachen.DE by hb111949 Tue Jan 9 16:04:39 > 2018 [0]PETSC ERROR: Configure options --download-fblaslapack --with- > cc=mpicc -with-fc=mpif90 --with-cxx=mpicxx --download-hypre --download- > superlu_dist --download-suitesparse --download-scalapack --download- > blacs --download-hdf5 --download-parmetis --download-metis --with- > debugging=0 --download-mumps [0]PETSC ERROR: #1 > DMCreateInjection_DA_3D() line 1195 in > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dainterp.c > [0]PETSC ERROR: #2 DMCreateInjection_DA() line 1283 in > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dainterp.c > [0]PETSC ERROR: #3 DMCreateInjection() line 1082 in > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c > [0]PETSC ERROR: #4 DMCoarsen_DA() line 1219 in > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/da.c > [0]PETSC ERROR: #5 DMCoarsen() line 2427 in > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c > [0]PETSC ERROR: #6 PCSetUp_MG() line 618 in > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/pc/impls/mg/mg.c > [0]PETSC ERROR: #7 PCSetUp() line 924 in > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/pc/interface/precon. > c > [0]PETSC ERROR: #8 KSPSetUp() line 381 in > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/ksp/interface/itfunc > .c > [0]PETSC ERROR: #9 KSPSolve() line 612 in > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/ksp/interface/itfunc > .c > [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message ---- > ---------------------------------------------------------- > > > -- > Dipl.-Math. Henrik B?sing > Institute for Applied Geophysics and Geothermal Energy E.ON Energy > Research Center RWTH Aachen University > ------------------------------------------------------ > Mathieustr. 10 | Tel +49 (0)241 80 49907 > 52074 Aachen, Germany | Fax +49 (0)241 80 49889 > ------------------------------------------------------ > http://www.eonerc.rwth-aachen.de/GGE > hbuesing at eonerc.rwth-aachen.de > ------------------------------------------------------ > > > -----Urspr?ngliche Nachricht----- > > Von: Smith, Barry F. [mailto:bsmith at mcs.anl.gov] > > Gesendet: 09 January 2018 15:10 > > An: Buesing, Henrik > > Cc: petsc-users > > Betreff: Re: [petsc-users] Geometric multigrid with a cell-centered > > discretization > > > > > > With cell centered elements each element is sliced in half in each > > of the three directions for refinement (or you can refine only in 1 or > > 2 of the dimensions if you like). This means that the ratio of mx to > > MX needs to be 2 (or 4).So your numbers are fine. > > > > > On Jan 9, 2018, at 7:59 AM, Buesing, Henrik > aachen.de> wrote: > > > > > > Dear all, > > > > > > I am using DMDACreate3d to generate my domain and a cell-centered > > > two-point flux approximation as discretization. I use geometric > > > Multigrid (-pc_type mg) with DMDASetInterpolationType(petsc_da, > > > DMDA_Q0, petsc_ierr). With DM_BOUNDARY_GHOSTED as boundary type in > > > DMDACreate3d, I get > > > > > > [0]PETSC ERROR: Arguments are incompatible [0]PETSC ERROR: Ratio > > > between levels: (mx - 1)/(Mx - 1) must be integer: mx 1536 Mx 768 > > > > We need ALL the output from the error (always). It should never be > > doing this check in your case so something is wrong with the order of > > your code perhaps. > > > > Barry > > > > > > > > With DM_BOUNDARY_PERIODIC, I get invalid accesses when I assemble > > > the > > matrix. > > > > > > Can I somehow tell him, that I have a cell-centered grid, or do I > > > have > > to resort to using N+1 in my dimensions? > > > > > > Thank you! > > > Henrik > > > > > > > > > -- > > > Dipl.-Math. Henrik B?sing > > > Institute for Applied Geophysics and Geothermal Energy E.ON Energy > > > Research Center RWTH Aachen University > > > ------------------------------------------------------ > > > Mathieustr. 10 | Tel +49 (0)241 80 49907 > > > 52074 Aachen, Germany | Fax +49 (0)241 80 49889 > > > ------------------------------------------------------ > > > http://www.eonerc.rwth-aachen.de/GGE > > > hbuesing at eonerc.rwth-aachen.de > > > ------------------------------------------------------ From hbuesing at eonerc.rwth-aachen.de Tue Jan 9 09:32:59 2018 From: hbuesing at eonerc.rwth-aachen.de (Buesing, Henrik) Date: Tue, 9 Jan 2018 15:32:59 +0000 Subject: [petsc-users] Geometric multigrid with a cell-centered discretization In-Reply-To: References: <8BDBA8E7-50C2-4A15-8114-C8991CCD62B9@anl.gov> Message-ID: [3] [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Arguments are incompatible [0]PETSC ERROR: Ratio between levels: (mx - 1)/(Mx - 1) must be integer: mx 1536 Mx 768 It looks like you give it the number of vertices, not cells, so +1 to your grid size. [Buesing, Henrik] Yes! But, I do not want that. I want to give the cells? That is the interface. You will get m-1 cells so give it m = num_cells + 1. [Buesing, Henrik] Ok, I can do that. But, does the space discretization matter for the coarsening and refining. With DMDA_Q0, I tell him it is cell-centered two-point flux approximation, right? [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.8.2-48-g851ec02 GIT Date: 2017-12-05 09:52:17 -0600 [0]PETSC ERROR: /rwthfs/rz/cluster/work/hb111949/EoCoE/aixcelerate17/gge_gnu/bench_run/000023/000000_copy_sub_launch/work/shem_fw64gnu_const.x on a gnu_openmpi named linuxbmc0008.rz.RWTH-Aachen.DE by hb111949 Tue Jan 9 16:04:39 2018 [0]PETSC ERROR: Configure options --download-fblaslapack --with-cc=mpicc -with-fc=mpif90 --with-cxx=mpicxx --download-hypre --download-superlu_dist --download-suitesparse --download-scalapack --download-blacs --download-hdf5 --download-parmetis --download-metis --with-debugging=0 --download-mumps [0]PETSC ERROR: #1 DMCreateInjection_DA_3D() line 1195 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dainterp.c [0]PETSC ERROR: #2 DMCreateInjection_DA() line 1283 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dainterp.c [0]PETSC ERROR: #3 DMCreateInjection() line 1082 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c [0]PETSC ERROR: #4 DMCoarsen_DA() line 1219 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/da.c [0]PETSC ERROR: #5 DMCoarsen() line 2427 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c [0]PETSC ERROR: #6 PCSetUp_MG() line 618 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: #7 PCSetUp() line 924 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #8 KSPSetUp() line 381 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #9 KSPSolve() line 612 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- -- Dipl.-Math. Henrik B?sing Institute for Applied Geophysics and Geothermal Energy E.ON Energy Research Center RWTH Aachen University ------------------------------------------------------ Mathieustr. 10 | Tel +49 (0)241 80 49907 52074 Aachen, Germany | Fax +49 (0)241 80 49889 ------------------------------------------------------ http://www.eonerc.rwth-aachen.de/GGE hbuesing at eonerc.rwth-aachen.de ------------------------------------------------------ > -----Urspr?ngliche Nachricht----- > Von: Smith, Barry F. [mailto:bsmith at mcs.anl.gov] > Gesendet: 09 January 2018 15:10 > An: Buesing, Henrik > > Cc: petsc-users > > Betreff: Re: [petsc-users] Geometric multigrid with a cell-centered > discretization > > > With cell centered elements each element is sliced in half in each of > the three directions for refinement (or you can refine only in 1 or 2 of > the dimensions if you like). This means that the ratio of mx to MX needs > to be 2 (or 4).So your numbers are fine. > > > On Jan 9, 2018, at 7:59 AM, Buesing, Henrik > aachen.de> wrote: > > > > Dear all, > > > > I am using DMDACreate3d to generate my domain and a cell-centered > > two-point flux approximation as discretization. I use geometric > > Multigrid (-pc_type mg) with DMDASetInterpolationType(petsc_da, > > DMDA_Q0, petsc_ierr). With DM_BOUNDARY_GHOSTED as boundary type in > > DMDACreate3d, I get > > > > [0]PETSC ERROR: Arguments are incompatible [0]PETSC ERROR: Ratio > > between levels: (mx - 1)/(Mx - 1) must be integer: mx 1536 Mx 768 > > We need ALL the output from the error (always). It should never be > doing this check in your case so something is wrong with the order of > your code perhaps. > > Barry > > > > > With DM_BOUNDARY_PERIODIC, I get invalid accesses when I assemble the > matrix. > > > > Can I somehow tell him, that I have a cell-centered grid, or do I have > to resort to using N+1 in my dimensions? > > > > Thank you! > > Henrik > > > > > > -- > > Dipl.-Math. Henrik B?sing > > Institute for Applied Geophysics and Geothermal Energy E.ON Energy > > Research Center RWTH Aachen University > > ------------------------------------------------------ > > Mathieustr. 10 | Tel +49 (0)241 80 49907 > > 52074 Aachen, Germany | Fax +49 (0)241 80 49889 > > ------------------------------------------------------ > > http://www.eonerc.rwth-aachen.de/GGE > > hbuesing at eonerc.rwth-aachen.de > > ------------------------------------------------------ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Tue Jan 9 09:37:33 2018 From: mfadams at lbl.gov (Mark Adams) Date: Tue, 9 Jan 2018 10:37:33 -0500 Subject: [petsc-users] Geometric multigrid with a cell-centered discretization In-Reply-To: References: <8BDBA8E7-50C2-4A15-8114-C8991CCD62B9@anl.gov> Message-ID: On Tue, Jan 9, 2018 at 10:32 AM, Buesing, Henrik < hbuesing at eonerc.rwth-aachen.de> wrote: > [3] > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Arguments are incompatible > [0]PETSC ERROR: Ratio between levels: (mx - 1)/(Mx - 1) must be integer: > mx 1536 Mx 768 > > > > It looks like you give it the number of vertices, not cells, so +1 to your > grid size. > > > [Buesing, Henrik] Yes! But, I do not want that. I want to give the cells? > > > > That is the interface. You will get m-1 cells so give it m = num_cells + 1. > > > [Buesing, Henrik] Ok, I can do that. But, does the space discretization > matter for the coarsening and refining. > No. We are careful to separate topology from the numerics. > With DMDA_Q0, I tell him it is cell-centered two-point flux approximation, > right? > Not sure. You specific that you want one variable per cell someplace (not edge or vertex), look at examples and/or documentation. > > > > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.8.2-48-g851ec02 GIT > Date: 2017-12-05 09:52:17 -0600 > [0]PETSC ERROR: /rwthfs/rz/cluster/work/hb111949/EoCoE/aixcelerate17/ > gge_gnu/bench_run/000023/000000_copy_sub_launch/work/shem_fw64gnu_const.x > on a gnu_openmpi named linuxbmc0008.rz.RWTH-Aachen.DE by hb111949 Tue > Jan 9 16:04:39 2018 > [0]PETSC ERROR: Configure options --download-fblaslapack --with-cc=mpicc > -with-fc=mpif90 --with-cxx=mpicxx --download-hypre --download-superlu_dist > --download-suitesparse --download-scalapack --download-blacs > --download-hdf5 --download-parmetis --download-metis --with-debugging=0 > --download-mumps > [0]PETSC ERROR: #1 DMCreateInjection_DA_3D() line 1195 in > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dainterp.c > [0]PETSC ERROR: #2 DMCreateInjection_DA() line 1283 in > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dainterp.c > [0]PETSC ERROR: #3 DMCreateInjection() line 1082 in > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c > [0]PETSC ERROR: #4 DMCoarsen_DA() line 1219 in /rwthfs/rz/cluster/work/ > hb111949/Code/petsc/src/dm/impls/da/da.c > [0]PETSC ERROR: #5 DMCoarsen() line 2427 in /rwthfs/rz/cluster/work/ > hb111949/Code/petsc/src/dm/interface/dm.c > [0]PETSC ERROR: #6 PCSetUp_MG() line 618 in /rwthfs/rz/cluster/work/ > hb111949/Code/petsc/src/ksp/pc/impls/mg/mg.c > [0]PETSC ERROR: #7 PCSetUp() line 924 in /rwthfs/rz/cluster/work/ > hb111949/Code/petsc/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #8 KSPSetUp() line 381 in /rwthfs/rz/cluster/work/ > hb111949/Code/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #9 KSPSolve() line 612 in /rwthfs/rz/cluster/work/ > hb111949/Code/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > > -- > Dipl.-Math. Henrik B?sing > Institute for Applied Geophysics and Geothermal Energy > E.ON Energy Research Center > RWTH Aachen University > ------------------------------------------------------ > Mathieustr. 10 > > | Tel +49 (0)241 80 49907 > 52074 Aachen, Germany | Fax +49 (0)241 80 49889 > ------------------------------------------------------ > http://www.eonerc.rwth-aachen.de/GGE > hbuesing at eonerc.rwth-aachen.de > ------------------------------------------------------ > > > -----Urspr?ngliche Nachricht----- > > Von: Smith, Barry F. [mailto:bsmith at mcs.anl.gov] > > Gesendet: 09 January 2018 15:10 > > An: Buesing, Henrik > > Cc: petsc-users > > Betreff: Re: [petsc-users] Geometric multigrid with a cell-centered > > discretization > > > > > > > With cell centered elements each element is sliced in half in each of > > the three directions for refinement (or you can refine only in 1 or 2 of > > the dimensions if you like). This means that the ratio of mx to MX needs > > to be 2 (or 4).So your numbers are fine. > > > > > On Jan 9, 2018, at 7:59 AM, Buesing, Henrik > aachen.de> wrote: > > > > > > Dear all, > > > > > > I am using DMDACreate3d to generate my domain and a cell-centered > > > two-point flux approximation as discretization. I use geometric > > > Multigrid (-pc_type mg) with DMDASetInterpolationType(petsc_da, > > > DMDA_Q0, petsc_ierr). With DM_BOUNDARY_GHOSTED as boundary type in > > > DMDACreate3d, I get > > > > > > [0]PETSC ERROR: Arguments are incompatible [0]PETSC ERROR: Ratio > > > between levels: (mx - 1)/(Mx - 1) must be integer: mx 1536 Mx 768 > > > > We need ALL the output from the error (always). It should never be > > doing this check in your case so something is wrong with the order of > > your code perhaps. > > > > Barry > > > > > > > > With DM_BOUNDARY_PERIODIC, I get invalid accesses when I assemble the > > matrix. > > > > > > Can I somehow tell him, that I have a cell-centered grid, or do I have > > to resort to using N+1 in my dimensions? > > > > > > Thank you! > > > Henrik > > > > > > > > > -- > > > Dipl.-Math. Henrik B?sing > > > Institute for Applied Geophysics and Geothermal Energy E.ON Energy > > > Research Center RWTH Aachen University > > > ------------------------------------------------------ > > > Mathieustr. 10 > > | Tel +49 (0)241 80 49907 <+49%20241%208049907> > > > 52074 Aachen, Germany | Fax +49 (0)241 80 49889 > <+49%20241%208049889> > > > ------------------------------------------------------ > > > http://www.eonerc.rwth-aachen.de/GGE > > > hbuesing at eonerc.rwth-aachen.de > > > ------------------------------------------------------ > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hbuesing at eonerc.rwth-aachen.de Tue Jan 9 09:44:53 2018 From: hbuesing at eonerc.rwth-aachen.de (Buesing, Henrik) Date: Tue, 9 Jan 2018 15:44:53 +0000 Subject: [petsc-users] Geometric multigrid with a cell-centered discretization In-Reply-To: References: <8BDBA8E7-50C2-4A15-8114-C8991CCD62B9@anl.gov> Message-ID: With DMDA_Q0, I tell him it is cell-centered two-point flux approximation, right? Not sure. You specific that you want one variable per cell someplace (not edge or vertex), look at examples and/or documentation. [Buesing, Henrik] I look at KSP example 32. There it says, functions are cell-centered. In the documentation to DMDASetInterpolationType it just says there exists Q0 and Q1. So, I assume this just means constant and linear interpolation on cuboids. [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.8.2-48-g851ec02 GIT Date: 2017-12-05 09:52:17 -0600 [0]PETSC ERROR: /rwthfs/rz/cluster/work/hb111949/EoCoE/aixcelerate17/gge_gnu/bench_run/000023/000000_copy_sub_launch/work/shem_fw64gnu_const.x on a gnu_openmpi named linuxbmc0008.rz.RWTH-Aachen.DE by hb111949 Tue Jan 9 16:04:39 2018 [0]PETSC ERROR: Configure options --download-fblaslapack --with-cc=mpicc -with-fc=mpif90 --with-cxx=mpicxx --download-hypre --download-superlu_dist --download-suitesparse --download-scalapack --download-blacs --download-hdf5 --download-parmetis --download-metis --with-debugging=0 --download-mumps [0]PETSC ERROR: #1 DMCreateInjection_DA_3D() line 1195 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dainterp.c [0]PETSC ERROR: #2 DMCreateInjection_DA() line 1283 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dainterp.c [0]PETSC ERROR: #3 DMCreateInjection() line 1082 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c [0]PETSC ERROR: #4 DMCoarsen_DA() line 1219 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/da.c [0]PETSC ERROR: #5 DMCoarsen() line 2427 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c [0]PETSC ERROR: #6 PCSetUp_MG() line 618 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: #7 PCSetUp() line 924 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #8 KSPSetUp() line 381 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #9 KSPSolve() line 612 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- -- Dipl.-Math. Henrik B?sing Institute for Applied Geophysics and Geothermal Energy E.ON Energy Research Center RWTH Aachen University ------------------------------------------------------ Mathieustr. 10 | Tel +49 (0)241 80 49907 52074 Aachen, Germany | Fax +49 (0)241 80 49889 ------------------------------------------------------ http://www.eonerc.rwth-aachen.de/GGE hbuesing at eonerc.rwth-aachen.de ------------------------------------------------------ > -----Urspr?ngliche Nachricht----- > Von: Smith, Barry F. [mailto:bsmith at mcs.anl.gov] > Gesendet: 09 January 2018 15:10 > An: Buesing, Henrik > > Cc: petsc-users > > Betreff: Re: [petsc-users] Geometric multigrid with a cell-centered > discretization > > > With cell centered elements each element is sliced in half in each of > the three directions for refinement (or you can refine only in 1 or 2 of > the dimensions if you like). This means that the ratio of mx to MX needs > to be 2 (or 4).So your numbers are fine. > > > On Jan 9, 2018, at 7:59 AM, Buesing, Henrik > aachen.de> wrote: > > > > Dear all, > > > > I am using DMDACreate3d to generate my domain and a cell-centered > > two-point flux approximation as discretization. I use geometric > > Multigrid (-pc_type mg) with DMDASetInterpolationType(petsc_da, > > DMDA_Q0, petsc_ierr). With DM_BOUNDARY_GHOSTED as boundary type in > > DMDACreate3d, I get > > > > [0]PETSC ERROR: Arguments are incompatible [0]PETSC ERROR: Ratio > > between levels: (mx - 1)/(Mx - 1) must be integer: mx 1536 Mx 768 > > We need ALL the output from the error (always). It should never be > doing this check in your case so something is wrong with the order of > your code perhaps. > > Barry > > > > > With DM_BOUNDARY_PERIODIC, I get invalid accesses when I assemble the > matrix. > > > > Can I somehow tell him, that I have a cell-centered grid, or do I have > to resort to using N+1 in my dimensions? > > > > Thank you! > > Henrik > > > > > > -- > > Dipl.-Math. Henrik B?sing > > Institute for Applied Geophysics and Geothermal Energy E.ON Energy > > Research Center RWTH Aachen University > > ------------------------------------------------------ > > Mathieustr. 10 | Tel +49 (0)241 80 49907 > > 52074 Aachen, Germany | Fax +49 (0)241 80 49889 > > ------------------------------------------------------ > > http://www.eonerc.rwth-aachen.de/GGE > > hbuesing at eonerc.rwth-aachen.de > > ------------------------------------------------------ -------------- next part -------------- An HTML attachment was scrubbed... URL: From hbuesing at eonerc.rwth-aachen.de Tue Jan 9 09:55:59 2018 From: hbuesing at eonerc.rwth-aachen.de (Buesing, Henrik) Date: Tue, 9 Jan 2018 15:55:59 +0000 Subject: [petsc-users] Geometric multigrid with a cell-centered discretization In-Reply-To: References: <8BDBA8E7-50C2-4A15-8114-C8991CCD62B9@anl.gov> Message-ID: [3] [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Arguments are incompatible [0]PETSC ERROR: Ratio between levels: (mx - 1)/(Mx - 1) must be integer: mx 1536 Mx 768 It looks like you give it the number of vertices, not cells, so +1 to your grid size. [Buesing, Henrik] Yes! But, I do not want that. I want to give the cells? That is the interface. You will get m-1 cells so give it m = num_cells + 1. [Buesing, Henrik] Looking at KSP example 32, the example gives a 12x12 grid in x-y-direction. So, it is possible to give even numbers. Thus, something is wrong in my code? [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.8.2-48-g851ec02 GIT Date: 2017-12-05 09:52:17 -0600 [0]PETSC ERROR: /rwthfs/rz/cluster/work/hb111949/EoCoE/aixcelerate17/gge_gnu/bench_run/000023/000000_copy_sub_launch/work/shem_fw64gnu_const.x on a gnu_openmpi named linuxbmc0008.rz.RWTH-Aachen.DE by hb111949 Tue Jan 9 16:04:39 2018 [0]PETSC ERROR: Configure options --download-fblaslapack --with-cc=mpicc -with-fc=mpif90 --with-cxx=mpicxx --download-hypre --download-superlu_dist --download-suitesparse --download-scalapack --download-blacs --download-hdf5 --download-parmetis --download-metis --with-debugging=0 --download-mumps [0]PETSC ERROR: #1 DMCreateInjection_DA_3D() line 1195 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dainterp.c [0]PETSC ERROR: #2 DMCreateInjection_DA() line 1283 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dainterp.c [0]PETSC ERROR: #3 DMCreateInjection() line 1082 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c [0]PETSC ERROR: #4 DMCoarsen_DA() line 1219 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/da.c [0]PETSC ERROR: #5 DMCoarsen() line 2427 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c [0]PETSC ERROR: #6 PCSetUp_MG() line 618 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: #7 PCSetUp() line 924 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #8 KSPSetUp() line 381 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #9 KSPSolve() line 612 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- -- Dipl.-Math. Henrik B?sing Institute for Applied Geophysics and Geothermal Energy E.ON Energy Research Center RWTH Aachen University ------------------------------------------------------ Mathieustr. 10 | Tel +49 (0)241 80 49907 52074 Aachen, Germany | Fax +49 (0)241 80 49889 ------------------------------------------------------ http://www.eonerc.rwth-aachen.de/GGE hbuesing at eonerc.rwth-aachen.de ------------------------------------------------------ > -----Urspr?ngliche Nachricht----- > Von: Smith, Barry F. [mailto:bsmith at mcs.anl.gov] > Gesendet: 09 January 2018 15:10 > An: Buesing, Henrik > > Cc: petsc-users > > Betreff: Re: [petsc-users] Geometric multigrid with a cell-centered > discretization > > > With cell centered elements each element is sliced in half in each of > the three directions for refinement (or you can refine only in 1 or 2 of > the dimensions if you like). This means that the ratio of mx to MX needs > to be 2 (or 4).So your numbers are fine. > > > On Jan 9, 2018, at 7:59 AM, Buesing, Henrik > aachen.de> wrote: > > > > Dear all, > > > > I am using DMDACreate3d to generate my domain and a cell-centered > > two-point flux approximation as discretization. I use geometric > > Multigrid (-pc_type mg) with DMDASetInterpolationType(petsc_da, > > DMDA_Q0, petsc_ierr). With DM_BOUNDARY_GHOSTED as boundary type in > > DMDACreate3d, I get > > > > [0]PETSC ERROR: Arguments are incompatible [0]PETSC ERROR: Ratio > > between levels: (mx - 1)/(Mx - 1) must be integer: mx 1536 Mx 768 > > We need ALL the output from the error (always). It should never be > doing this check in your case so something is wrong with the order of > your code perhaps. > > Barry > > > > > With DM_BOUNDARY_PERIODIC, I get invalid accesses when I assemble the > matrix. > > > > Can I somehow tell him, that I have a cell-centered grid, or do I have > to resort to using N+1 in my dimensions? > > > > Thank you! > > Henrik > > > > > > -- > > Dipl.-Math. Henrik B?sing > > Institute for Applied Geophysics and Geothermal Energy E.ON Energy > > Research Center RWTH Aachen University > > ------------------------------------------------------ > > Mathieustr. 10 | Tel +49 (0)241 80 49907 > > 52074 Aachen, Germany | Fax +49 (0)241 80 49889 > > ------------------------------------------------------ > > http://www.eonerc.rwth-aachen.de/GGE > > hbuesing at eonerc.rwth-aachen.de > > ------------------------------------------------------ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jan 9 10:14:46 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 9 Jan 2018 16:14:46 +0000 Subject: [petsc-users] Geometric multigrid with a cell-centered discretization In-Reply-To: References: <8BDBA8E7-50C2-4A15-8114-C8991CCD62B9@anl.gov> Message-ID: <64F4ACE6-B934-4C20-87C1-FD71CCFA32DA@anl.gov> Ok, so the problem is that for Q0 we do not handle injection correctly (it is not really injection with cell centered it is an average over multiple class) and hence we cannot handle coordinates correctly you need to not attach any coordinates to your DMDA and it should work. See for example src/ksp/ksp/examples/tutorials/ex32.c Barry Sorry about this, someone needs to fix the injection for Q0 case. > On Jan 9, 2018, at 9:12 AM, Buesing, Henrik wrote: > > Sorry (again), for posting only part of the error message... > > So, here it comes. I will first [1] show the code snippets I am using, then second [2] I will show the PETSc options, and finally [3] the error message I get. > The grid is 1536 x 1 x 256 in x,y,z-direction. Thus, we see the first level of coarsening in x-direction here. > > Thank you! > Henrik > > [1] > CALL DMDACreate3d(PETSC_COMM_WORLD,DM_BOUNDARY_GHOSTED,DM_BOUNDARY_GHOSTED,DM_BOUNDARY_GHOSTED,& > DMDA_STENCIL_STAR, I0g,J0g,K0g, PETSC_DECIDE, PETSC_DECIDE,PETSC_DECIDE, petsc_two, petsc_one, & > PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER, petsc_da, petsc_ierr) > CALL DMSetUp(petsc_da,petsc_ierr) > ... > CALL DMSetFromOptions(petsc_da, petsc_ierr) > CALL DMDASetInterpolationType(petsc_da, DMDA_Q0, petsc_ierr) > CALL SNESSetDM(petsc_snes, petsc_da, petsc_ierr) > CALL SNESSetFromOptions(petsc_snes,petsc_ierr) > > [2] -snes_monitor -snes_linesearch_monitor -snes_type newtonls -snes_linesearch_type basic -ksp_type fgmres -ksp_max_it 1000 -snes_stol 1e-06 -snes_rtol 1e-06 -snes_atol 1e-6 -ksp_gmres_restart 100 -ksp_monitor -pc_type mg -da_refine_y 1 -ksp_converged_reason -snes_converged_reason -ksp_monitor -ksp_view -pc_mg_levels 2 -options_left > > [3] > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Arguments are incompatible > [0]PETSC ERROR: Ratio between levels: (mx - 1)/(Mx - 1) must be integer: mx 1536 Mx 768 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.8.2-48-g851ec02 GIT Date: 2017-12-05 09:52:17 -0600 > [0]PETSC ERROR: /rwthfs/rz/cluster/work/hb111949/EoCoE/aixcelerate17/gge_gnu/bench_run/000023/000000_copy_sub_launch/work/shem_fw64gnu_const.x on a gnu_openmpi named linuxbmc0008.rz.RWTH-Aachen.DE by hb111949 Tue Jan 9 16:04:39 2018 > [0]PETSC ERROR: Configure options --download-fblaslapack --with-cc=mpicc -with-fc=mpif90 --with-cxx=mpicxx --download-hypre --download-superlu_dist --download-suitesparse --download-scalapack --download-blacs --download-hdf5 --download-parmetis --download-metis --with-debugging=0 --download-mumps > [0]PETSC ERROR: #1 DMCreateInjection_DA_3D() line 1195 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dainterp.c > [0]PETSC ERROR: #2 DMCreateInjection_DA() line 1283 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dainterp.c > [0]PETSC ERROR: #3 DMCreateInjection() line 1082 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c > [0]PETSC ERROR: #4 DMCoarsen_DA() line 1219 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/da.c > [0]PETSC ERROR: #5 DMCoarsen() line 2427 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c > [0]PETSC ERROR: #6 PCSetUp_MG() line 618 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/pc/impls/mg/mg.c > [0]PETSC ERROR: #7 PCSetUp() line 924 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #8 KSPSetUp() line 381 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #9 KSPSolve() line 612 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > > -- > Dipl.-Math. Henrik B?sing > Institute for Applied Geophysics and Geothermal Energy > E.ON Energy Research Center > RWTH Aachen University > ------------------------------------------------------ > Mathieustr. 10 | Tel +49 (0)241 80 49907 > 52074 Aachen, Germany | Fax +49 (0)241 80 49889 > ------------------------------------------------------ > http://www.eonerc.rwth-aachen.de/GGE > hbuesing at eonerc.rwth-aachen.de > ------------------------------------------------------ > >> -----Urspr?ngliche Nachricht----- >> Von: Smith, Barry F. [mailto:bsmith at mcs.anl.gov] >> Gesendet: 09 January 2018 15:10 >> An: Buesing, Henrik >> Cc: petsc-users >> Betreff: Re: [petsc-users] Geometric multigrid with a cell-centered >> discretization >> >> >> With cell centered elements each element is sliced in half in each of >> the three directions for refinement (or you can refine only in 1 or 2 of >> the dimensions if you like). This means that the ratio of mx to MX needs >> to be 2 (or 4).So your numbers are fine. >> >>> On Jan 9, 2018, at 7:59 AM, Buesing, Henrik > aachen.de> wrote: >>> >>> Dear all, >>> >>> I am using DMDACreate3d to generate my domain and a cell-centered >>> two-point flux approximation as discretization. I use geometric >>> Multigrid (-pc_type mg) with DMDASetInterpolationType(petsc_da, >>> DMDA_Q0, petsc_ierr). With DM_BOUNDARY_GHOSTED as boundary type in >>> DMDACreate3d, I get >>> >>> [0]PETSC ERROR: Arguments are incompatible [0]PETSC ERROR: Ratio >>> between levels: (mx - 1)/(Mx - 1) must be integer: mx 1536 Mx 768 >> >> We need ALL the output from the error (always). It should never be >> doing this check in your case so something is wrong with the order of >> your code perhaps. >> >> Barry >> >>> >>> With DM_BOUNDARY_PERIODIC, I get invalid accesses when I assemble the >> matrix. >>> >>> Can I somehow tell him, that I have a cell-centered grid, or do I have >> to resort to using N+1 in my dimensions? >>> >>> Thank you! >>> Henrik >>> >>> >>> -- >>> Dipl.-Math. Henrik B?sing >>> Institute for Applied Geophysics and Geothermal Energy E.ON Energy >>> Research Center RWTH Aachen University >>> ------------------------------------------------------ >>> Mathieustr. 10 | Tel +49 (0)241 80 49907 >>> 52074 Aachen, Germany | Fax +49 (0)241 80 49889 >>> ------------------------------------------------------ >>> http://www.eonerc.rwth-aachen.de/GGE >>> hbuesing at eonerc.rwth-aachen.de >>> ------------------------------------------------------ > From hbuesing at eonerc.rwth-aachen.de Tue Jan 9 16:15:28 2018 From: hbuesing at eonerc.rwth-aachen.de (Buesing, Henrik) Date: Tue, 9 Jan 2018 22:15:28 +0000 Subject: [petsc-users] Geometric multigrid with a cell-centered discretization In-Reply-To: <64F4ACE6-B934-4C20-87C1-FD71CCFA32DA@anl.gov> References: <8BDBA8E7-50C2-4A15-8114-C8991CCD62B9@anl.gov> <64F4ACE6-B934-4C20-87C1-FD71CCFA32DA@anl.gov> Message-ID: > you need to not attach any > coordinates to your DMDA and it should work. See for example > src/ksp/ksp/examples/tutorials/ex32.c You mean, I should omit the call to DMDASetUniformCoordinates, right? This means, that my Paraview files will have no domain size info anymore. I will try it tomorrow and come back to you with the results. Thank you! Henrik > > Barry > > Sorry about this, someone needs to fix the injection for Q0 case. > > > > On Jan 9, 2018, at 9:12 AM, Buesing, Henrik aachen.de> wrote: > > > > Sorry (again), for posting only part of the error message... > > > > So, here it comes. I will first [1] show the code snippets I am using, then > second [2] I will show the PETSc options, and finally [3] the error message I > get. > > The grid is 1536 x 1 x 256 in x,y,z-direction. Thus, we see the first level of > coarsening in x-direction here. > > > > Thank you! > > Henrik > > > > [1] > > CALL > DMDACreate3d(PETSC_COMM_WORLD,DM_BOUNDARY_GHOSTED,DM_BO > UNDARY_GHOSTED,DM_BOUNDARY_GHOSTED,& > > DMDA_STENCIL_STAR, I0g,J0g,K0g, PETSC_DECIDE, > PETSC_DECIDE,PETSC_DECIDE, petsc_two, petsc_one, & > > > PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER, > petsc_da, petsc_ierr) > > CALL DMSetUp(petsc_da,petsc_ierr) ... > > CALL DMSetFromOptions(petsc_da, petsc_ierr) > > CALL DMDASetInterpolationType(petsc_da, DMDA_Q0, petsc_ierr) > > CALL SNESSetDM(petsc_snes, petsc_da, petsc_ierr) > > CALL SNESSetFromOptions(petsc_snes,petsc_ierr) > > > > [2] -snes_monitor -snes_linesearch_monitor -snes_type newtonls > > -snes_linesearch_type basic -ksp_type fgmres -ksp_max_it 1000 > > -snes_stol 1e-06 -snes_rtol 1e-06 -snes_atol 1e-6 -ksp_gmres_restart > > 100 -ksp_monitor -pc_type mg -da_refine_y 1 -ksp_converged_reason > > -snes_converged_reason -ksp_monitor -ksp_view -pc_mg_levels 2 > > -options_left > > > > [3] > > [0]PETSC ERROR: --------------------- Error Message > > -------------------------------------------------------------- > > [0]PETSC ERROR: Arguments are incompatible [0]PETSC ERROR: Ratio > > between levels: (mx - 1)/(Mx - 1) must be integer: mx 1536 Mx 768 > > [0]PETSC ERROR: See > http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > > [0]PETSC ERROR: Petsc Development GIT revision: v3.8.2-48-g851ec02 > > GIT Date: 2017-12-05 09:52:17 -0600 [0]PETSC ERROR: > > > /rwthfs/rz/cluster/work/hb111949/EoCoE/aixcelerate17/gge_gnu/bench_ru > n > > /000023/000000_copy_sub_launch/work/shem_fw64gnu_const.x on a > > gnu_openmpi named linuxbmc0008.rz.RWTH-Aachen.DE by hb111949 Tue > Jan > > 9 16:04:39 2018 [0]PETSC ERROR: Configure options > > --download-fblaslapack --with-cc=mpicc -with-fc=mpif90 > > --with-cxx=mpicxx --download-hypre --download-superlu_dist > > --download-suitesparse --download-scalapack --download-blacs > > --download-hdf5 --download-parmetis --download-metis > > --with-debugging=0 --download-mumps [0]PETSC ERROR: #1 > > DMCreateInjection_DA_3D() line 1195 in > > > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dainterp.c > > [0]PETSC ERROR: #2 DMCreateInjection_DA() line 1283 in > > > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dainterp.c > > [0]PETSC ERROR: #3 DMCreateInjection() line 1082 in > > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c > > [0]PETSC ERROR: #4 DMCoarsen_DA() line 1219 in > > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/da.c > > [0]PETSC ERROR: #5 DMCoarsen() line 2427 in > > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c > > [0]PETSC ERROR: #6 PCSetUp_MG() line 618 in > > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/pc/impls/mg/mg.c > > [0]PETSC ERROR: #7 PCSetUp() line 924 in > > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/pc/interface/preco > > n.c [0]PETSC ERROR: #8 KSPSetUp() line 381 in > > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/ksp/interface/itfu > > nc.c [0]PETSC ERROR: #9 KSPSolve() line 612 in > > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/ksp/interface/itfu > > nc.c [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error > > Message -------------------------------------------------------------- > > > > > > -- > > Dipl.-Math. Henrik B?sing > > Institute for Applied Geophysics and Geothermal Energy E.ON Energy > > Research Center RWTH Aachen University > > ------------------------------------------------------ > > Mathieustr. 10 | Tel +49 (0)241 80 49907 > > 52074 Aachen, Germany | Fax +49 (0)241 80 49889 > > ------------------------------------------------------ > > http://www.eonerc.rwth-aachen.de/GGE > > hbuesing at eonerc.rwth-aachen.de > > ------------------------------------------------------ > > > >> -----Urspr?ngliche Nachricht----- > >> Von: Smith, Barry F. [mailto:bsmith at mcs.anl.gov] > >> Gesendet: 09 January 2018 15:10 > >> An: Buesing, Henrik > >> Cc: petsc-users > >> Betreff: Re: [petsc-users] Geometric multigrid with a cell-centered > >> discretization > >> > >> > >> With cell centered elements each element is sliced in half in each > >> of the three directions for refinement (or you can refine only in 1 > >> or 2 of the dimensions if you like). This means that the ratio of mx > >> to MX needs to be 2 (or 4).So your numbers are fine. > >> > >>> On Jan 9, 2018, at 7:59 AM, Buesing, Henrik >> aachen.de> wrote: > >>> > >>> Dear all, > >>> > >>> I am using DMDACreate3d to generate my domain and a cell-centered > >>> two-point flux approximation as discretization. I use geometric > >>> Multigrid (-pc_type mg) with DMDASetInterpolationType(petsc_da, > >>> DMDA_Q0, petsc_ierr). With DM_BOUNDARY_GHOSTED as boundary > type in > >>> DMDACreate3d, I get > >>> > >>> [0]PETSC ERROR: Arguments are incompatible [0]PETSC ERROR: Ratio > >>> between levels: (mx - 1)/(Mx - 1) must be integer: mx 1536 Mx 768 > >> > >> We need ALL the output from the error (always). It should never be > >> doing this check in your case so something is wrong with the order of > >> your code perhaps. > >> > >> Barry > >> > >>> > >>> With DM_BOUNDARY_PERIODIC, I get invalid accesses when I assemble > >>> the > >> matrix. > >>> > >>> Can I somehow tell him, that I have a cell-centered grid, or do I > >>> have > >> to resort to using N+1 in my dimensions? > >>> > >>> Thank you! > >>> Henrik > >>> > >>> > >>> -- > >>> Dipl.-Math. Henrik B?sing > >>> Institute for Applied Geophysics and Geothermal Energy E.ON Energy > >>> Research Center RWTH Aachen University > >>> ------------------------------------------------------ > >>> Mathieustr. 10 | Tel +49 (0)241 80 49907 > >>> 52074 Aachen, Germany | Fax +49 (0)241 80 49889 > >>> ------------------------------------------------------ > >>> http://www.eonerc.rwth-aachen.de/GGE > >>> hbuesing at eonerc.rwth-aachen.de > >>> ------------------------------------------------------ > > From bsmith at mcs.anl.gov Tue Jan 9 16:38:58 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 9 Jan 2018 22:38:58 +0000 Subject: [petsc-users] Geometric multigrid with a cell-centered discretization In-Reply-To: References: <8BDBA8E7-50C2-4A15-8114-C8991CCD62B9@anl.gov> <64F4ACE6-B934-4C20-87C1-FD71CCFA32DA@anl.gov> Message-ID: <6B144260-4282-4662-9785-46D60005677B@anl.gov> > On Jan 9, 2018, at 4:15 PM, Buesing, Henrik wrote: > >> you need to not attach any >> coordinates to your DMDA and it should work. See for example >> src/ksp/ksp/examples/tutorials/ex32.c > > You mean, I should omit the call to DMDASetUniformCoordinates, right? I guess. > This means, that my Paraview files will have no domain size info anymore. Presumably. But you may be able to work around the issue some way, for example setting the coordinates after the solvers are complete. Barry > > I will try it tomorrow and come back to you with the results. > > Thank you! > Henrik > >> >> Barry >> >> Sorry about this, someone needs to fix the injection for Q0 case. >> >> >>> On Jan 9, 2018, at 9:12 AM, Buesing, Henrik > aachen.de> wrote: >>> >>> Sorry (again), for posting only part of the error message... >>> >>> So, here it comes. I will first [1] show the code snippets I am using, then >> second [2] I will show the PETSc options, and finally [3] the error message I >> get. >>> The grid is 1536 x 1 x 256 in x,y,z-direction. Thus, we see the first level of >> coarsening in x-direction here. >>> >>> Thank you! >>> Henrik >>> >>> [1] >>> CALL >> DMDACreate3d(PETSC_COMM_WORLD,DM_BOUNDARY_GHOSTED,DM_BO >> UNDARY_GHOSTED,DM_BOUNDARY_GHOSTED,& >>> DMDA_STENCIL_STAR, I0g,J0g,K0g, PETSC_DECIDE, >> PETSC_DECIDE,PETSC_DECIDE, petsc_two, petsc_one, & >>> >> PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER, >> petsc_da, petsc_ierr) >>> CALL DMSetUp(petsc_da,petsc_ierr) ... >>> CALL DMSetFromOptions(petsc_da, petsc_ierr) >>> CALL DMDASetInterpolationType(petsc_da, DMDA_Q0, petsc_ierr) >>> CALL SNESSetDM(petsc_snes, petsc_da, petsc_ierr) >>> CALL SNESSetFromOptions(petsc_snes,petsc_ierr) >>> >>> [2] -snes_monitor -snes_linesearch_monitor -snes_type newtonls >>> -snes_linesearch_type basic -ksp_type fgmres -ksp_max_it 1000 >>> -snes_stol 1e-06 -snes_rtol 1e-06 -snes_atol 1e-6 -ksp_gmres_restart >>> 100 -ksp_monitor -pc_type mg -da_refine_y 1 -ksp_converged_reason >>> -snes_converged_reason -ksp_monitor -ksp_view -pc_mg_levels 2 >>> -options_left >>> >>> [3] >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Arguments are incompatible [0]PETSC ERROR: Ratio >>> between levels: (mx - 1)/(Mx - 1) must be integer: mx 1536 Mx 768 >>> [0]PETSC ERROR: See >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >> shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.2-48-g851ec02 >>> GIT Date: 2017-12-05 09:52:17 -0600 [0]PETSC ERROR: >>> >> /rwthfs/rz/cluster/work/hb111949/EoCoE/aixcelerate17/gge_gnu/bench_ru >> n >>> /000023/000000_copy_sub_launch/work/shem_fw64gnu_const.x on a >>> gnu_openmpi named linuxbmc0008.rz.RWTH-Aachen.DE by hb111949 Tue >> Jan >>> 9 16:04:39 2018 [0]PETSC ERROR: Configure options >>> --download-fblaslapack --with-cc=mpicc -with-fc=mpif90 >>> --with-cxx=mpicxx --download-hypre --download-superlu_dist >>> --download-suitesparse --download-scalapack --download-blacs >>> --download-hdf5 --download-parmetis --download-metis >>> --with-debugging=0 --download-mumps [0]PETSC ERROR: #1 >>> DMCreateInjection_DA_3D() line 1195 in >>> >> /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dainterp.c >>> [0]PETSC ERROR: #2 DMCreateInjection_DA() line 1283 in >>> >> /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dainterp.c >>> [0]PETSC ERROR: #3 DMCreateInjection() line 1082 in >>> /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c >>> [0]PETSC ERROR: #4 DMCoarsen_DA() line 1219 in >>> /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/da.c >>> [0]PETSC ERROR: #5 DMCoarsen() line 2427 in >>> /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c >>> [0]PETSC ERROR: #6 PCSetUp_MG() line 618 in >>> /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/pc/impls/mg/mg.c >>> [0]PETSC ERROR: #7 PCSetUp() line 924 in >>> /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/pc/interface/preco >>> n.c [0]PETSC ERROR: #8 KSPSetUp() line 381 in >>> /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/ksp/interface/itfu >>> nc.c [0]PETSC ERROR: #9 KSPSolve() line 612 in >>> /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/ksp/interface/itfu >>> nc.c [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error >>> Message -------------------------------------------------------------- >>> >>> >>> -- >>> Dipl.-Math. Henrik B?sing >>> Institute for Applied Geophysics and Geothermal Energy E.ON Energy >>> Research Center RWTH Aachen University >>> ------------------------------------------------------ >>> Mathieustr. 10 | Tel +49 (0)241 80 49907 >>> 52074 Aachen, Germany | Fax +49 (0)241 80 49889 >>> ------------------------------------------------------ >>> http://www.eonerc.rwth-aachen.de/GGE >>> hbuesing at eonerc.rwth-aachen.de >>> ------------------------------------------------------ >>> >>>> -----Urspr?ngliche Nachricht----- >>>> Von: Smith, Barry F. [mailto:bsmith at mcs.anl.gov] >>>> Gesendet: 09 January 2018 15:10 >>>> An: Buesing, Henrik >>>> Cc: petsc-users >>>> Betreff: Re: [petsc-users] Geometric multigrid with a cell-centered >>>> discretization >>>> >>>> >>>> With cell centered elements each element is sliced in half in each >>>> of the three directions for refinement (or you can refine only in 1 >>>> or 2 of the dimensions if you like). This means that the ratio of mx >>>> to MX needs to be 2 (or 4).So your numbers are fine. >>>> >>>>> On Jan 9, 2018, at 7:59 AM, Buesing, Henrik >>> aachen.de> wrote: >>>>> >>>>> Dear all, >>>>> >>>>> I am using DMDACreate3d to generate my domain and a cell-centered >>>>> two-point flux approximation as discretization. I use geometric >>>>> Multigrid (-pc_type mg) with DMDASetInterpolationType(petsc_da, >>>>> DMDA_Q0, petsc_ierr). With DM_BOUNDARY_GHOSTED as boundary >> type in >>>>> DMDACreate3d, I get >>>>> >>>>> [0]PETSC ERROR: Arguments are incompatible [0]PETSC ERROR: Ratio >>>>> between levels: (mx - 1)/(Mx - 1) must be integer: mx 1536 Mx 768 >>>> >>>> We need ALL the output from the error (always). It should never be >>>> doing this check in your case so something is wrong with the order of >>>> your code perhaps. >>>> >>>> Barry >>>> >>>>> >>>>> With DM_BOUNDARY_PERIODIC, I get invalid accesses when I assemble >>>>> the >>>> matrix. >>>>> >>>>> Can I somehow tell him, that I have a cell-centered grid, or do I >>>>> have >>>> to resort to using N+1 in my dimensions? >>>>> >>>>> Thank you! >>>>> Henrik >>>>> >>>>> >>>>> -- >>>>> Dipl.-Math. Henrik B?sing >>>>> Institute for Applied Geophysics and Geothermal Energy E.ON Energy >>>>> Research Center RWTH Aachen University >>>>> ------------------------------------------------------ >>>>> Mathieustr. 10 | Tel +49 (0)241 80 49907 >>>>> 52074 Aachen, Germany | Fax +49 (0)241 80 49889 >>>>> ------------------------------------------------------ >>>>> http://www.eonerc.rwth-aachen.de/GGE >>>>> hbuesing at eonerc.rwth-aachen.de >>>>> ------------------------------------------------------ >>> > From hbuesing at eonerc.rwth-aachen.de Wed Jan 10 04:22:27 2018 From: hbuesing at eonerc.rwth-aachen.de (Buesing, Henrik) Date: Wed, 10 Jan 2018 10:22:27 +0000 Subject: [petsc-users] Geometric multigrid with a cell-centered discretization In-Reply-To: <6B144260-4282-4662-9785-46D60005677B@anl.gov> References: <8BDBA8E7-50C2-4A15-8114-C8991CCD62B9@anl.gov> <64F4ACE6-B934-4C20-87C1-FD71CCFA32DA@anl.gov> <6B144260-4282-4662-9785-46D60005677B@anl.gov> Message-ID: > >> you need to not attach any > >> coordinates to your DMDA and it should work. See for example > >> src/ksp/ksp/examples/tutorials/ex32.c > > > > You mean, I should omit the call to DMDASetUniformCoordinates, right? > > I guess. > > > This means, that my Paraview files will have no domain size info > anymore. > > Presumably. But you may be able to work around the issue some way, for > example setting the coordinates after the solvers are complete. [Buesing, Henrik] Good idea! I moved the DMDASetUniformCoordinates to after the SNESSolve call. This works! With this, I stepped over a weird bug with my own convergence test function. It appears only in optimized mode and not in debug mode. Anyhow, I will send another message for this. Thank you for your help! Henrik > > Barry > > > > > I will try it tomorrow and come back to you with the results. > > > > Thank you! > > Henrik > > > >> > >> Barry > >> > >> Sorry about this, someone needs to fix the injection for Q0 case. > >> > >> > >>> On Jan 9, 2018, at 9:12 AM, Buesing, Henrik >> aachen.de> wrote: > >>> > >>> Sorry (again), for posting only part of the error message... > >>> > >>> So, here it comes. I will first [1] show the code snippets I am > >>> using, then > >> second [2] I will show the PETSc options, and finally [3] the error > >> message I get. > >>> The grid is 1536 x 1 x 256 in x,y,z-direction. Thus, we see the > >>> first level of > >> coarsening in x-direction here. > >>> > >>> Thank you! > >>> Henrik > >>> > >>> [1] > >>> CALL > >> DMDACreate3d(PETSC_COMM_WORLD,DM_BOUNDARY_GHOSTED,DM_BO > >> UNDARY_GHOSTED,DM_BOUNDARY_GHOSTED,& > >>> DMDA_STENCIL_STAR, I0g,J0g,K0g, PETSC_DECIDE, > >> PETSC_DECIDE,PETSC_DECIDE, petsc_two, petsc_one, & > >>> > >> PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER, > >> petsc_da, petsc_ierr) > >>> CALL DMSetUp(petsc_da,petsc_ierr) ... > >>> CALL DMSetFromOptions(petsc_da, petsc_ierr) > >>> CALL DMDASetInterpolationType(petsc_da, DMDA_Q0, petsc_ierr) > >>> CALL SNESSetDM(petsc_snes, petsc_da, petsc_ierr) > >>> CALL SNESSetFromOptions(petsc_snes,petsc_ierr) > >>> > >>> [2] -snes_monitor -snes_linesearch_monitor -snes_type newtonls > >>> -snes_linesearch_type basic -ksp_type fgmres -ksp_max_it 1000 > >>> -snes_stol 1e-06 -snes_rtol 1e-06 -snes_atol 1e-6 -ksp_gmres_restart > >>> 100 -ksp_monitor -pc_type mg -da_refine_y 1 -ksp_converged_reason > >>> -snes_converged_reason -ksp_monitor -ksp_view -pc_mg_levels 2 > >>> -options_left > >>> > >>> [3] > >>> [0]PETSC ERROR: --------------------- Error Message > >>> -------------------------------------------------------------- > >>> [0]PETSC ERROR: Arguments are incompatible [0]PETSC ERROR: Ratio > >>> between levels: (mx - 1)/(Mx - 1) must be integer: mx 1536 Mx 768 > >>> [0]PETSC ERROR: See > >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > >> shooting. > >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.2-48-g851ec02 > >>> GIT Date: 2017-12-05 09:52:17 -0600 [0]PETSC ERROR: > >>> > >> /rwthfs/rz/cluster/work/hb111949/EoCoE/aixcelerate17/gge_gnu/bench_ru > >> n > >>> /000023/000000_copy_sub_launch/work/shem_fw64gnu_const.x on a > >>> gnu_openmpi named linuxbmc0008.rz.RWTH-Aachen.DE by hb111949 Tue > >> Jan > >>> 9 16:04:39 2018 [0]PETSC ERROR: Configure options > >>> --download-fblaslapack --with-cc=mpicc -with-fc=mpif90 > >>> --with-cxx=mpicxx --download-hypre --download-superlu_dist > >>> --download-suitesparse --download-scalapack --download-blacs > >>> --download-hdf5 --download-parmetis --download-metis > >>> --with-debugging=0 --download-mumps [0]PETSC ERROR: #1 > >>> DMCreateInjection_DA_3D() line 1195 in > >>> > >> /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dainterp. > >> c > >>> [0]PETSC ERROR: #2 DMCreateInjection_DA() line 1283 in > >>> > >> /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dainterp. > >> c > >>> [0]PETSC ERROR: #3 DMCreateInjection() line 1082 in > >>> /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c > >>> [0]PETSC ERROR: #4 DMCoarsen_DA() line 1219 in > >>> /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/da.c > >>> [0]PETSC ERROR: #5 DMCoarsen() line 2427 in > >>> /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c > >>> [0]PETSC ERROR: #6 PCSetUp_MG() line 618 in > >>> /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/pc/impls/mg/mg.c > >>> [0]PETSC ERROR: #7 PCSetUp() line 924 in > >>> /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/pc/interface/pre > >>> co n.c [0]PETSC ERROR: #8 KSPSetUp() line 381 in > >>> /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/ksp/interface/it > >>> fu nc.c [0]PETSC ERROR: #9 KSPSolve() line 612 in > >>> /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/ksp/interface/it > >>> fu nc.c [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error > >>> Message > >>> -------------------------------------------------------------- > >>> > >>> > >>> -- > >>> Dipl.-Math. Henrik B?sing > >>> Institute for Applied Geophysics and Geothermal Energy E.ON Energy > >>> Research Center RWTH Aachen University > >>> ------------------------------------------------------ > >>> Mathieustr. 10 | Tel +49 (0)241 80 49907 > >>> 52074 Aachen, Germany | Fax +49 (0)241 80 49889 > >>> ------------------------------------------------------ > >>> http://www.eonerc.rwth-aachen.de/GGE > >>> hbuesing at eonerc.rwth-aachen.de > >>> ------------------------------------------------------ > >>> > >>>> -----Urspr?ngliche Nachricht----- > >>>> Von: Smith, Barry F. [mailto:bsmith at mcs.anl.gov] > >>>> Gesendet: 09 January 2018 15:10 > >>>> An: Buesing, Henrik > >>>> Cc: petsc-users > >>>> Betreff: Re: [petsc-users] Geometric multigrid with a cell-centered > >>>> discretization > >>>> > >>>> > >>>> With cell centered elements each element is sliced in half in each > >>>> of the three directions for refinement (or you can refine only in 1 > >>>> or 2 of the dimensions if you like). This means that the ratio of > >>>> mx to MX needs to be 2 (or 4).So your numbers are fine. > >>>> > >>>>> On Jan 9, 2018, at 7:59 AM, Buesing, Henrik >>>> aachen.de> wrote: > >>>>> > >>>>> Dear all, > >>>>> > >>>>> I am using DMDACreate3d to generate my domain and a cell-centered > >>>>> two-point flux approximation as discretization. I use geometric > >>>>> Multigrid (-pc_type mg) with DMDASetInterpolationType(petsc_da, > >>>>> DMDA_Q0, petsc_ierr). With DM_BOUNDARY_GHOSTED as boundary > >> type in > >>>>> DMDACreate3d, I get > >>>>> > >>>>> [0]PETSC ERROR: Arguments are incompatible [0]PETSC ERROR: Ratio > >>>>> between levels: (mx - 1)/(Mx - 1) must be integer: mx 1536 Mx 768 > >>>> > >>>> We need ALL the output from the error (always). It should never be > >>>> doing this check in your case so something is wrong with the order > >>>> of your code perhaps. > >>>> > >>>> Barry > >>>> > >>>>> > >>>>> With DM_BOUNDARY_PERIODIC, I get invalid accesses when I assemble > >>>>> the > >>>> matrix. > >>>>> > >>>>> Can I somehow tell him, that I have a cell-centered grid, or do I > >>>>> have > >>>> to resort to using N+1 in my dimensions? > >>>>> > >>>>> Thank you! > >>>>> Henrik > >>>>> > >>>>> > >>>>> -- > >>>>> Dipl.-Math. Henrik B?sing > >>>>> Institute for Applied Geophysics and Geothermal Energy E.ON Energy > >>>>> Research Center RWTH Aachen University > >>>>> ------------------------------------------------------ > >>>>> Mathieustr. 10 | Tel +49 (0)241 80 49907 > >>>>> 52074 Aachen, Germany | Fax +49 (0)241 80 49889 > >>>>> ------------------------------------------------------ > >>>>> http://www.eonerc.rwth-aachen.de/GGE > >>>>> hbuesing at eonerc.rwth-aachen.de > >>>>> ------------------------------------------------------ > >>> > > From hbuesing at eonerc.rwth-aachen.de Wed Jan 10 05:25:11 2018 From: hbuesing at eonerc.rwth-aachen.de (Buesing, Henrik) Date: Wed, 10 Jan 2018 11:25:11 +0000 Subject: [petsc-users] Weak scaling test with geometric multigrid Message-ID: Dear all, I am doing a weak scaling test using geometric multigrid. With increasing the number of cells, and the number of processes, I also increase the number of multigrid levels. With 64 cores, 12288 cells in x-direction and 11 multigrid levels, I see error message [1]. Could you help me understand what is happening here? The characteristics of the weak scaling test are summarized in table [2]. Refinement level 7-9 went through fine. Thank you! Henrik [1] [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Argument out of range [0]PETSC ERROR: Partition in x direction is too fine! 12 16 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.8.2-48-g851ec02 GIT Date: 2017-12-05 09:52:17 -0600 [0]PETSC ERROR: shem_fw64gnu_const.x on a gnu_openmpi named linuxihfc033.rz.RWTH-Aachen.DE by hb111949 Wed Jan 10 11:48:09 2018 [0]PETSC ERROR: Configure options --download-fblaslapack --with-cc=mpicc -with-fc=mpif90 --with-cxx=mpicxx --download-hypre --download-superlu_dist --download-suitesparse --download-scalapack --download-blacs --download-hdf5 --download-parmetis --download-metis --with-debugging=0 --download-mumps [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 299 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/da3.c [0]PETSC ERROR: #2 DMSetUp_DA() line 25 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dareg.c [0]PETSC ERROR: #3 DMSetUp() line 720 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c [0]PETSC ERROR: #4 DMCoarsen_DA() line 1203 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/da.c [0]PETSC ERROR: #5 DMCoarsen() line 2427 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c [0]PETSC ERROR: #6 PCSetUp_MG() line 618 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: #7 PCSetUp() line 924 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #8 KSPSetUp() line 381 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #9 KSPSolve() line 612 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #10 SNESSolve_NEWTONLS() line 224 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/snes/impls/ls/ls.c [0]PETSC ERROR: #11 SNESSolve() line 4179 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/snes/interface/snes.c [2] # refinement level # cores # cells in x # cells in y # cells in z # mg levels 7 1 1536 1 256 8 8 4 3072 1 512 9 9 16 6144 1 1024 10 10 64 12288 1 2048 11 -- Dipl.-Math. Henrik B?sing Institute for Applied Geophysics and Geothermal Energy E.ON Energy Research Center RWTH Aachen University ------------------------------------------------------ Mathieustr. 10 | Tel +49 (0)241 80 49907 52074 Aachen, Germany | Fax +49 (0)241 80 49889 ------------------------------------------------------ http://www.eonerc.rwth-aachen.de/GGE hbuesing at eonerc.rwth-aachen.de ------------------------------------------------------ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Jan 10 09:34:34 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 10 Jan 2018 15:34:34 +0000 Subject: [petsc-users] Weak scaling test with geometric multigrid In-Reply-To: References: Message-ID: DMDA requires that there be at least 1 grid points in each direction on each process (this simplified the implementation a huge amount but gives up flexibility). In your case you have 16 processes in a particular direction but a total of only 12 grid points hence not enough grid points to have at least 1 per process. Barry > On Jan 10, 2018, at 5:25 AM, Buesing, Henrik wrote: > > Dear all, > > I am doing a weak scaling test using geometric multigrid. With increasing the number of cells, and the number of processes, I also increase the number of multigrid levels. With 64 cores, 12288 cells in x-direction and 11 multigrid levels, I see error message [1]. > > Could you help me understand what is happening here? > > The characteristics of the weak scaling test are summarized in table [2]. Refinement level 7-9 went through fine. > > Thank you! > Henrik > > [1] > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Argument out of range > [0]PETSC ERROR: Partition in x direction is too fine! 12 16 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.8.2-48-g851ec02 GIT Date: 2017-12-05 09:52:17 -0600 > [0]PETSC ERROR: shem_fw64gnu_const.x on a gnu_openmpi named linuxihfc033.rz.RWTH-Aachen.DE by hb111949 Wed Jan 10 11:48:09 2018 > [0]PETSC ERROR: Configure options --download-fblaslapack --with-cc=mpicc -with-fc=mpif90 --with-cxx=mpicxx --download-hypre --download-superlu_dist --download-suitesparse --download-scalapack --download-blacs --download-hdf5 --download-parmetis --download-metis --with-debugging=0 --download-mumps > [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 299 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/da3.c > [0]PETSC ERROR: #2 DMSetUp_DA() line 25 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dareg.c > [0]PETSC ERROR: #3 DMSetUp() line 720 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c > [0]PETSC ERROR: #4 DMCoarsen_DA() line 1203 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/da.c > [0]PETSC ERROR: #5 DMCoarsen() line 2427 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c > [0]PETSC ERROR: #6 PCSetUp_MG() line 618 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/pc/impls/mg/mg.c > [0]PETSC ERROR: #7 PCSetUp() line 924 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #8 KSPSetUp() line 381 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #9 KSPSolve() line 612 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #10 SNESSolve_NEWTONLS() line 224 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/snes/impls/ls/ls.c > [0]PETSC ERROR: #11 SNESSolve() line 4179 in /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/snes/interface/snes.c > > > [2] > > # refinement level > # cores > # cells in x > # cells in y > # cells in z > # mg levels > 7 > 1 > 1536 > 1 > 256 > 8 > 8 > 4 > 3072 > 1 > 512 > 9 > 9 > 16 > 6144 > 1 > 1024 > 10 > 10 > 64 > 12288 > 1 > 2048 > 11 > > -- > Dipl.-Math. Henrik B?sing > Institute for Applied Geophysics and Geothermal Energy > E.ON Energy Research Center > RWTH Aachen University > ------------------------------------------------------ > Mathieustr. 10 | Tel +49 (0)241 80 49907 > 52074 Aachen, Germany | Fax +49 (0)241 80 49889 > ------------------------------------------------------ > http://www.eonerc.rwth-aachen.de/GGE > hbuesing at eonerc.rwth-aachen.de > ------------------------------------------------------ From s.lanthaler at gmail.com Wed Jan 10 10:11:12 2018 From: s.lanthaler at gmail.com (Samuel Lanthaler) Date: Wed, 10 Jan 2018 17:11:12 +0100 Subject: [petsc-users] Using SLEPC: Fortran call to NEPNLEIGSSetSingularitiesFunction (similar to SNESSetFunction?) In-Reply-To: References: Message-ID: <5A563B20.9050407@gmail.com> Hi there, I'm stuck getting a call to the SLEPC routine "NEPNLEIGSSetSingularitiesFunction" to work from within Fortran. To be more precise: To get started with the solution of a non-linear eigenvalue-problem, I am first trying to write a Fortran version of slepc-3.8.1/.../nep/examples/tutorials/ex27.c, and then use it as a template for my own code. But I haven't gotten far... Let me attach the source of the current state of my attempt to translate ex27.c to Fortran code. When trying to compile the attached source, my compiler complains: nleigs.o: In function `MAIN__': nleigs.f90:(.text+0x213): undefined reference to `nepnleigssetsingularitiesfunction_' Does someone maybe see what I'm doing wrong? Do I maybe need to add a USE statement in addition to USE slepcsys USE slepcnep ? I'm quite confused about the C++ pointers and how to deal with them when calling C++ functions from Fortran, so I don't really understand what's going on here. I will greatly appreciate your help! Thanks, Samuel -------------- next part -------------- A non-text attachment was scrubbed... Name: nleigs.f90 Type: text/x-fortran Size: 2913 bytes Desc: not available URL: From jroman at dsic.upv.es Wed Jan 10 10:23:05 2018 From: jroman at dsic.upv.es (Jose E. Roman) Date: Wed, 10 Jan 2018 17:23:05 +0100 Subject: [petsc-users] Using SLEPC: Fortran call to NEPNLEIGSSetSingularitiesFunction (similar to SNESSetFunction?) In-Reply-To: <5A563B20.9050407@gmail.com> References: <5A563B20.9050407@gmail.com> Message-ID: This function is not implemented in Fortran. I will add it and let you know. Jose > El 10 ene 2018, a las 17:11, Samuel Lanthaler escribi?: > > Hi there, > > I'm stuck getting a call to the SLEPC routine "NEPNLEIGSSetSingularitiesFunction" to work from within Fortran. > To be more precise: To get started with the solution of a non-linear eigenvalue-problem, I am first trying to write a Fortran version of slepc-3.8.1/.../nep/examples/tutorials/ex27.c, and then use it as a template for my own code. But I haven't gotten far... > > Let me attach the source of the current state of my attempt to translate ex27.c to Fortran code. When trying to compile the attached source, my compiler complains: > > nleigs.o: In function `MAIN__': > nleigs.f90:(.text+0x213): undefined reference to `nepnleigssetsingularitiesfunction_' > > Does someone maybe see what I'm doing wrong? Do I maybe need to add a USE statement in addition to > > USE slepcsys > USE slepcnep > > ? > I'm quite confused about the C++ pointers and how to deal with them when calling C++ functions from Fortran, so I don't really understand what's going on here. I will greatly appreciate your help! > > Thanks, > Samuel > From s.lanthaler at gmail.com Wed Jan 10 10:29:04 2018 From: s.lanthaler at gmail.com (Samuel Lanthaler) Date: Wed, 10 Jan 2018 17:29:04 +0100 Subject: [petsc-users] Using SLEPC: Fortran call to NEPNLEIGSSetSingularitiesFunction (similar to SNESSetFunction?) In-Reply-To: References: <5A563B20.9050407@gmail.com> Message-ID: <5A563F50.8040501@gmail.com> Ah, I see. Great, thank you, Jose! Cheers, Samuel On 01/10/2018 05:23 PM, Jose E. Roman wrote: > This function is not implemented in Fortran. I will add it and let you know. > Jose > > >> El 10 ene 2018, a las 17:11, Samuel Lanthaler escribi?: >> >> Hi there, >> >> I'm stuck getting a call to the SLEPC routine "NEPNLEIGSSetSingularitiesFunction" to work from within Fortran. >> To be more precise: To get started with the solution of a non-linear eigenvalue-problem, I am first trying to write a Fortran version of slepc-3.8.1/.../nep/examples/tutorials/ex27.c, and then use it as a template for my own code. But I haven't gotten far... >> >> Let me attach the source of the current state of my attempt to translate ex27.c to Fortran code. When trying to compile the attached source, my compiler complains: >> >> nleigs.o: In function `MAIN__': >> nleigs.f90:(.text+0x213): undefined reference to `nepnleigssetsingularitiesfunction_' >> >> Does someone maybe see what I'm doing wrong? Do I maybe need to add a USE statement in addition to >> >> USE slepcsys >> USE slepcnep >> >> ? >> I'm quite confused about the C++ pointers and how to deal with them when calling C++ functions from Fortran, so I don't really understand what's going on here. I will greatly appreciate your help! >> >> Thanks, >> Samuel >> From gohardoust at email.arizona.edu Wed Jan 10 15:41:15 2018 From: gohardoust at email.arizona.edu (Mohammad R. Gohardoust) Date: Wed, 10 Jan 2018 14:41:15 -0700 Subject: [petsc-users] Issue with KSPSolve when the number of CPUs are larger than four Message-ID: Hi, I hope you are doing well. I am Mohammad, a PhD student of environmental sciences at the University of Arizona. I do appreciate any help that can shed some light on the issue I have: recently I have added a feature to an existing parallel code called 'parswms' which solves water and solute transport in soils. The code uses MPI and parmetis for the parallelization purposes and petsc package (KSPSolve) for solving linear systems. I have it installed on the UofA HPC : the issue is that it works well if the number of CPUs are up to 4! but when I add to this number the numerical linear solver (here it is 'KSPCGS') stops with the error of 'KSP_DIVERGED_NANORINF'. Would you mind please giving me some hints, suggestions or resources in this regard? Best Regards, Mohammad -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jan 10 18:04:23 2018 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 10 Jan 2018 19:04:23 -0500 Subject: [petsc-users] Issue with KSPSolve when the number of CPUs are larger than four In-Reply-To: References: Message-ID: On Wed, Jan 10, 2018 at 4:41 PM, Mohammad R. Gohardoust < gohardoust at email.arizona.edu> wrote: > Hi, > > I hope you are doing well. I am Mohammad, a PhD student of environmental > sciences at the University of Arizona. > > I do appreciate any help that can shed some light on the issue I have: > recently I have added a feature to an existing parallel code called > 'parswms' which solves water and solute transport in soils. The code uses > MPI and parmetis for the parallelization purposes and petsc package > (KSPSolve) for solving linear systems. I have it installed on the UofA > HPC : the issue is that it works well if the number of CPUs are up to 4! > but when I add to this number the numerical linear solver (here it is > 'KSPCGS') stops with the error of 'KSP_DIVERGED_NANORINF'. > > Would you mind please giving me some hints, suggestions or resources in > this regard? > It sounds like you might have a problem in matrix assembly. However, first use -pc_type gmres instead since it has nicer numerical behavior (I believe). Thanks, Matt > Best Regards, > Mohammad > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Jan 10 18:30:04 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 11 Jan 2018 00:30:04 +0000 Subject: [petsc-users] Issue with KSPSolve when the number of CPUs are larger than four In-Reply-To: References: Message-ID: <13F171E7-B6F3-4074-AD03-6A66EED40695@anl.gov> You can also run with -fp_trap to find where the nan/inf first appears as well as run with -ksp_monitor to see what is happening before the tragedy Barry > On Jan 10, 2018, at 3:41 PM, Mohammad R. Gohardoust wrote: > > Hi, > > I hope you are doing well. I am Mohammad, a PhD student of environmental sciences at the University of Arizona. > > I do appreciate any help that can shed some light on the issue I have: recently I have added a feature to an existing parallel code called 'parswms' which solves water and solute transport in soils. The code uses MPI and parmetis for the parallelization purposes and petsc package (KSPSolve) for solving linear systems. I have it installed on the UofA HPC : the issue is that it works well if the number of CPUs are up to 4! but when I add to this number the numerical linear solver (here it is 'KSPCGS') stops with the error of 'KSP_DIVERGED_NANORINF'. > > Would you mind please giving me some hints, suggestions or resources in this regard? > > > Best Regards, > Mohammad > > > From dave.mayhem23 at gmail.com Thu Jan 11 01:15:41 2018 From: dave.mayhem23 at gmail.com (Dave May) Date: Thu, 11 Jan 2018 07:15:41 +0000 Subject: [petsc-users] Weak scaling test with geometric multigrid In-Reply-To: References: Message-ID: Henrik, On Wed, 10 Jan 2018 at 16:39, Smith, Barry F. wrote: > > DMDA requires that there be at least 1 grid points in each direction on > each process (this simplified the implementation a huge amount but gives up > flexibility). In your case you have 16 processes in a particular direction > but a total of only 12 grid points hence not enough grid points to have at > least 1 per process. This particular implementation limitation can be overcome using PCTELESCOPE. It allows you to repartition the coarse levels onto fewer ranks. Cheers, Dave > > Barry > > > > On Jan 10, 2018, at 5:25 AM, Buesing, Henrik < > hbuesing at eonerc.rwth-aachen.de> wrote: > > > > Dear all, > > > > I am doing a weak scaling test using geometric multigrid. With > increasing the number of cells, and the number of processes, I also > increase the number of multigrid levels. With 64 cores, 12288 cells in > x-direction and 11 multigrid levels, I see error message [1]. > > > > Could you help me understand what is happening here? > > > > The characteristics of the weak scaling test are summarized in table > [2]. Refinement level 7-9 went through fine. > > > > Thank you! > > Henrik > > > > [1] > > > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [0]PETSC ERROR: Argument out of range > > [0]PETSC ERROR: Partition in x direction is too fine! 12 16 > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [0]PETSC ERROR: Petsc Development GIT revision: v3.8.2-48-g851ec02 GIT > Date: 2017-12-05 09:52:17 -0600 > > [0]PETSC ERROR: shem_fw64gnu_const.x on a gnu_openmpi named > linuxihfc033.rz.RWTH-Aachen.DE by hb111949 Wed Jan 10 11:48:09 2018 > > [0]PETSC ERROR: Configure options --download-fblaslapack --with-cc=mpicc > -with-fc=mpif90 --with-cxx=mpicxx --download-hypre --download-superlu_dist > --download-suitesparse --download-scalapack --download-blacs > --download-hdf5 --download-parmetis --download-metis --with-debugging=0 > --download-mumps > > [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 299 in > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/da3.c > > [0]PETSC ERROR: #2 DMSetUp_DA() line 25 in > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/dareg.c > > [0]PETSC ERROR: #3 DMSetUp() line 720 in > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c > > [0]PETSC ERROR: #4 DMCoarsen_DA() line 1203 in > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/impls/da/da.c > > [0]PETSC ERROR: #5 DMCoarsen() line 2427 in > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/dm/interface/dm.c > > [0]PETSC ERROR: #6 PCSetUp_MG() line 618 in > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/pc/impls/mg/mg.c > > [0]PETSC ERROR: #7 PCSetUp() line 924 in > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #8 KSPSetUp() line 381 in > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: #9 KSPSolve() line 612 in > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: #10 SNESSolve_NEWTONLS() line 224 in > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/snes/impls/ls/ls.c > > [0]PETSC ERROR: #11 SNESSolve() line 4179 in > /rwthfs/rz/cluster/work/hb111949/Code/petsc/src/snes/interface/snes.c > > > > > > [2] > > > > # refinement level > > # cores > > # cells in x > > # cells in y > > # cells in z > > # mg levels > > 7 > > 1 > > 1536 > > 1 > > 256 > > 8 > > 8 > > 4 > > 3072 > > 1 > > 512 > > 9 > > 9 > > 16 > > 6144 > > 1 > > 1024 > > 10 > > 10 > > 64 > > 12288 > > 1 > > 2048 > > 11 > > > > -- > > Dipl.-Math. Henrik B?sing > > Institute for Applied Geoph > ysics and > Geothermal Energy > > E.ON Energy Research Center > > RWTH Aachen University > > ------------------------------------------------------ > > Mathieustr. 10 | Tel +49 (0)241 80 49907 > > 52074 Aachen, Germany | Fax +49 (0)241 80 49889 > > ------------------------------------------------------ > > http://www.eonerc.rwth-aachen.de/GGE > > hbuesing at eonerc.rwth-aachen.de > > ------------------------------------------------------ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Thu Jan 11 07:48:06 2018 From: jroman at dsic.upv.es (Jose E. Roman) Date: Thu, 11 Jan 2018 14:48:06 +0100 Subject: [petsc-users] Using SLEPC: Fortran call to NEPNLEIGSSetSingularitiesFunction (similar to SNESSetFunction?) In-Reply-To: <5A563F50.8040501@gmail.com> References: <5A563B20.9050407@gmail.com> <5A563F50.8040501@gmail.com> Message-ID: The Fortran stub is now in master. I have also completed the example. https://bitbucket.org/slepc/slepc/commits/acaefe873 Jose > El 10 ene 2018, a las 17:29, Samuel Lanthaler escribi?: > > Ah, I see. > Great, thank you, Jose! > > Cheers, > Samuel > > > On 01/10/2018 05:23 PM, Jose E. Roman wrote: >> This function is not implemented in Fortran. I will add it and let you know. >> Jose >> >> >>> El 10 ene 2018, a las 17:11, Samuel Lanthaler escribi?: >>> >>> Hi there, >>> >>> I'm stuck getting a call to the SLEPC routine "NEPNLEIGSSetSingularitiesFunction" to work from within Fortran. >>> To be more precise: To get started with the solution of a non-linear eigenvalue-problem, I am first trying to write a Fortran version of slepc-3.8.1/.../nep/examples/tutorials/ex27.c, and then use it as a template for my own code. But I haven't gotten far... >>> >>> Let me attach the source of the current state of my attempt to translate ex27.c to Fortran code. When trying to compile the attached source, my compiler complains: >>> >>> nleigs.o: In function `MAIN__': >>> nleigs.f90:(.text+0x213): undefined reference to `nepnleigssetsingularitiesfunction_' >>> >>> Does someone maybe see what I'm doing wrong? Do I maybe need to add a USE statement in addition to >>> >>> USE slepcsys >>> USE slepcnep >>> >>> ? >>> I'm quite confused about the C++ pointers and how to deal with them when calling C++ functions from Fortran, so I don't really understand what's going on here. I will greatly appreciate your help! >>> >>> Thanks, >>> Samuel >>> > From s.lanthaler at gmail.com Thu Jan 11 08:21:57 2018 From: s.lanthaler at gmail.com (Samuel Lanthaler) Date: Thu, 11 Jan 2018 15:21:57 +0100 Subject: [petsc-users] Using SLEPC: Fortran call to NEPNLEIGSSetSingularitiesFunction (similar to SNESSetFunction?) In-Reply-To: References: <5A563B20.9050407@gmail.com> <5A563F50.8040501@gmail.com> Message-ID: <5A577305.70001@gmail.com> Amazing, thank you! Samuel On 01/11/2018 02:48 PM, Jose E. Roman wrote: > The Fortran stub is now in master. I have also completed the example. > https://bitbucket.org/slepc/slepc/commits/acaefe873 > Jose > >> El 10 ene 2018, a las 17:29, Samuel Lanthaler escribi?: >> >> Ah, I see. >> Great, thank you, Jose! >> >> Cheers, >> Samuel >> >> >> On 01/10/2018 05:23 PM, Jose E. Roman wrote: >>> This function is not implemented in Fortran. I will add it and let you know. >>> Jose >>> >>> >>>> El 10 ene 2018, a las 17:11, Samuel Lanthaler escribi?: >>>> >>>> Hi there, >>>> >>>> I'm stuck getting a call to the SLEPC routine "NEPNLEIGSSetSingularitiesFunction" to work from within Fortran. >>>> To be more precise: To get started with the solution of a non-linear eigenvalue-problem, I am first trying to write a Fortran version of slepc-3.8.1/.../nep/examples/tutorials/ex27.c, and then use it as a template for my own code. But I haven't gotten far... >>>> >>>> Let me attach the source of the current state of my attempt to translate ex27.c to Fortran code. When trying to compile the attached source, my compiler complains: >>>> >>>> nleigs.o: In function `MAIN__': >>>> nleigs.f90:(.text+0x213): undefined reference to `nepnleigssetsingularitiesfunction_' >>>> >>>> Does someone maybe see what I'm doing wrong? Do I maybe need to add a USE statement in addition to >>>> >>>> USE slepcsys >>>> USE slepcnep >>>> >>>> ? >>>> I'm quite confused about the C++ pointers and how to deal with them when calling C++ functions from Fortran, so I don't really understand what's going on here. I will greatly appreciate your help! >>>> >>>> Thanks, >>>> Samuel >>>> From jpi at informatik.uni-kiel.de Fri Jan 12 08:38:43 2018 From: jpi at informatik.uni-kiel.de (Jaroslaw Piwonski (CAU)) Date: Fri, 12 Jan 2018 15:38:43 +0100 Subject: [petsc-users] petsc4py, cython Message-ID: <781ADC9E-82A2-4FEB-8D03-15F733D89354@informatik.uni-kiel.de> Dear PETSc users, I would like to use petsc4py to create (say) PETSc vectors and then pass them to a routine from a module written in Cython, which in turn passes them to a routine written in C: main.py ======================================================== from petsc4py import PETSc import m3d4py if __name__ == "__main__": v = PETSc.Vec() v.create() v.setType("standard") v.setSizes(3) v.set(0.3) v.assemble() m3d4py.sub_pyx(v) m3d4py.pyx ======================================================== cimport petsc4py.PETSc as PETSc cdef extern from "sub.c": int sub_c(Vec) def sub_pyx(PETSc.Vec vec): vec.view() sub_c(vec.vec) sub.c ======================================================== #include "petsc.h" extern PetscErrorCode sub_c(Vec); PetscErrorCode sub_c(Vec vec) { PetscFunctionBegin; VecView(vec, PETSC_VIEWER_STDOUT_WORLD); PetscFunctionReturn(0); } ======================================================== I used (ana/mini)conda to install petsc4py and cython. When I try to compile the cython file I get the following error: $> cython -f m3d4py.pyx -I /Users/jpicau/miniconda3/lib/python3.6/site-packages/petsc4py/include/ Error compiling Cython file: ------------------------------------------------------------ ... cdef extern from "sub.c": int sub_c(Vec) def sub_pyx(PETSc.Vec vec): vec.view() sub_c(vec.vec) ^ ------------------------------------------------------------ m3d4py.pyx:8:13: Cannot convert 'PetscVec' to Python object make: *** [all] Error 1 ======================================================== I somehow realize that this is not correct. But comparing this to the petsc4py implementation ... petscvec.pxi: int VecView(PetscVec,PetscViewer) Vec.pyx: def view(self, Viewer viewer=None): cdef PetscViewer vwr = NULL if viewer is not None: vwr = viewer.vwr CHKERR( VecView(self.vec, vwr) ) ? I do not understand why. Can some explain this to me or just give me a hint where I can dig further, please? And just out of pure curiosity, why isn?t the return value of VecView() in petscvec.pxi defined as PetscErrorCode? The type seems to be defined: libpetsc4py.pyx: ctypedef int PetscErrorCode Thanks in advance, Jaroslaw ? Dr. Jaroslaw Piwonski Christian-Albrechts-Universit?t zu Kiel Institut f?r Informatik Algorithmische Optimale Steuerung ? CO2-Aufnahme des Meeres Christian-Albrechts-Platz 4 24118 Kiel Telefon: +49 431 880-7452 Telefax: +49 431 880-7618 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jan 12 09:35:00 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 12 Jan 2018 10:35:00 -0500 Subject: [petsc-users] petsc4py, cython In-Reply-To: <781ADC9E-82A2-4FEB-8D03-15F733D89354@informatik.uni-kiel.de> References: <781ADC9E-82A2-4FEB-8D03-15F733D89354@informatik.uni-kiel.de> Message-ID: On Fri, Jan 12, 2018 at 9:38 AM, Jaroslaw Piwonski < jpi at informatik.uni-kiel.de> wrote: > Dear PETSc users, > > I would like to use petsc4py to create (say) PETSc vectors and > then pass them to a routine from a module written in Cython, > which in turn passes them to a routine written in C: > > main.py ======================================================== > > from petsc4py import PETSc > import m3d4py > > if __name__ == "__main__": > > v = PETSc.Vec() > v.create() > v.setType("standard") > v.setSizes(3) > v.set(0.3) > v.assemble() > > m3d4py.sub_pyx(v) > > m3d4py.pyx ======================================================== > > cimport petsc4py.PETSc as PETSc > > cdef extern from "sub.c": > int sub_c(Vec) > I think the declaration above is wrong. 'Vec' looks like it is seeing the Python definition, and probably you want 'PetscVec'. Thanks, Matt > > def sub_pyx(PETSc.Vec vec): > vec.view() > sub_c(vec.vec) > > sub.c ======================================================== > > #include "petsc.h" > > extern PetscErrorCode sub_c(Vec); > > PetscErrorCode sub_c(Vec vec) { > PetscFunctionBegin; > VecView(vec, PETSC_VIEWER_STDOUT_WORLD); > PetscFunctionReturn(0); > } > > ======================================================== > > I used (ana/mini)conda to install petsc4py and cython. When I try to > compile the cython file I get the following error: > > $> cython -f m3d4py.pyx -I /Users/jpicau/miniconda3/lib/ > python3.6/site-packages/petsc4py/include/ > > Error compiling Cython file: > ------------------------------------------------------------ > ... > cdef extern from "sub.c": > int sub_c(Vec) > > def sub_pyx(PETSc.Vec vec): > vec.view() > sub_c(vec.vec) > ^ > ------------------------------------------------------------ > > m3d4py.pyx:8:13: Cannot convert 'PetscVec' to Python object > make: *** [all] Error 1 > > ======================================================== > > I somehow realize that this is not correct. But comparing this to the > petsc4py implementation ... > > petscvec.pxi: > int VecView(PetscVec,PetscViewer) > > Vec.pyx: > def view(self, Viewer viewer=None): > cdef PetscViewer vwr = NULL > if viewer is not None: vwr = viewer.vwr > CHKERR( VecView(self.vec, vwr) ) > > ? I do not understand why. > > Can some explain this to me or just give me a hint where I can dig > further, please? > > And just out of pure curiosity, why isn?t the return value of VecView() in > petscvec.pxi defined as PetscErrorCode? > The type seems to be defined: > > libpetsc4py.pyx: > ctypedef int PetscErrorCode > > Thanks in advance, > Jaroslaw > > ? > > Dr. Jaroslaw Piwonski > > Christian-Albrechts-Universit?t zu Kiel > Institut f?r Informatik > Algorithmische Optimale Steuerung ? CO2-Aufnahme des Meeres > Christian-Albrechts-Platz 4 > > 24118 Kiel > > > Telefon: +49 431 880-7452 <+49%20431%208807452> > Telefax: +49 431 880-7618 <+49%20431%208807618> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jpi at informatik.uni-kiel.de Fri Jan 12 10:13:47 2018 From: jpi at informatik.uni-kiel.de (Jaroslaw Piwonski (CAU)) Date: Fri, 12 Jan 2018 17:13:47 +0100 Subject: [petsc-users] petsc4py, cython In-Reply-To: References: <781ADC9E-82A2-4FEB-8D03-15F733D89354@informatik.uni-kiel.de> Message-ID: <183EC67F-4754-4146-A10C-841498BC9ABA@informatik.uni-kiel.de> > Am 12.01.2018 um 16:35 schrieb Matthew Knepley >: > > On Fri, Jan 12, 2018 at 9:38 AM, Jaroslaw Piwonski > wrote: > Dear PETSc users, > > I would like to use petsc4py to create (say) PETSc vectors and > then pass them to a routine from a module written in Cython, > which in turn passes them to a routine written in C: > > main.py ======================================================== > > from petsc4py import PETSc > import m3d4py > > if __name__ == "__main__": > > v = PETSc.Vec() > v.create() > v.setType("standard") > v.setSizes(3) > v.set(0.3) > v.assemble() > > m3d4py.sub_pyx(v) > > m3d4py.pyx ======================================================== > > cimport petsc4py.PETSc as PETSc > > cdef extern from "sub.c": > int sub_c(Vec) > > I think the declaration above is wrong. 'Vec' looks like it is seeing the Python definition, and > probably you want 'PetscVec?. I tried a lot in this regard. Unfortunately nothing worked: $> cython -f m3d4py.pyx -I /Users/jpicau/miniconda3/lib/python3.6/site-packages/petsc4py/include/ Error compiling Cython file: ------------------------------------------------------------ ... cdef extern from "sub.c": int sub_c(PetscVec) def sub_pyx(PETSc.Vec vec): vec.view() sub_c(vec.vec) ^ ------------------------------------------------------------ m3d4py.pyx:8:13: Cannot convert 'PetscVec' to Python object make: *** [all] Error 1 This seems not to be the point ? the compiler does not complain about this definition: $> cython -f m3d4py.pyx -I /Users/jpicau/miniconda3/lib/python3.6/site-packages/petsc4py/include/ Error compiling Cython file: ------------------------------------------------------------ ... cdef extern from "sub.c": int sub_c(i_can_define_here_what_i_want_it_wont_matter) def sub_pyx(PETSc.Vec vec): vec.view() sub_c(vec.vec) ^ ------------------------------------------------------------ m3d4py.pyx:8:13: Cannot convert 'PetscVec' to Python object make: *** [all] Error 1 Thanks, Jaroslaw @Matt: Sorry, the first answer went to the wrong address. > > Thanks, > > Matt > > > def sub_pyx(PETSc.Vec vec): > vec.view() > sub_c(vec.vec) > > sub.c ======================================================== > > #include "petsc.h" > > extern PetscErrorCode sub_c(Vec); > > PetscErrorCode sub_c(Vec vec) { > PetscFunctionBegin; > VecView(vec, PETSC_VIEWER_STDOUT_WORLD); > PetscFunctionReturn(0); > } > > ======================================================== > > I used (ana/mini)conda to install petsc4py and cython. When I try to > compile the cython file I get the following error: > > $> cython -f m3d4py.pyx -I /Users/jpicau/miniconda3/lib/python3.6/site-packages/petsc4py/include/ > > Error compiling Cython file: > ------------------------------------------------------------ > ... > cdef extern from "sub.c": > int sub_c(Vec) > > def sub_pyx(PETSc.Vec vec): > vec.view() > sub_c(vec.vec) > ^ > ------------------------------------------------------------ > > m3d4py.pyx:8:13: Cannot convert 'PetscVec' to Python object > make: *** [all] Error 1 > > ======================================================== > > I somehow realize that this is not correct. But comparing this to the petsc4py implementation ... > > petscvec.pxi: > int VecView(PetscVec,PetscViewer) > > Vec.pyx: > def view(self, Viewer viewer=None): > cdef PetscViewer vwr = NULL > if viewer is not None: vwr = viewer.vwr > CHKERR( VecView(self.vec, vwr) ) > > ? I do not understand why. > > Can some explain this to me or just give me a hint where I can dig further, please? > > And just out of pure curiosity, why isn?t the return value of VecView() in petscvec.pxi defined as PetscErrorCode? > The type seems to be defined: > > libpetsc4py.pyx: > ctypedef int PetscErrorCode > > Thanks in advance, > Jaroslaw > > ? > > Dr. Jaroslaw Piwonski > > Christian-Albrechts-Universit?t zu Kiel > Institut f?r Informatik > Algorithmische Optimale Steuerung ? CO2-Aufnahme des Meeres > Christian-Albrechts-Platz 4 > 24118 Kiel > > Telefon: +49 431 880-7452 > Telefax: +49 431 880-7618 > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ ? Dr. Jaroslaw Piwonski Christian-Albrechts-Universit?t zu Kiel Institut f?r Informatik Algorithmische Optimale Steuerung ? CO2-Aufnahme des Meeres Christian-Albrechts-Platz 4 24118 Kiel Telefon: +49 431 880-7452 Telefax: +49 431 880-7618 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistloin at unist.ac.kr Mon Jan 15 01:03:10 2018 From: mistloin at unist.ac.kr (=?ks_c_5601-1987?B?vK29wsH4ICix4rDox9ew+LnXv/jA2rfCsPjH0LrOKQ==?=) Date: Mon, 15 Jan 2018 07:03:10 +0000 Subject: [petsc-users] I have a question for PETSc example In-Reply-To: <888DF7F8-1A65-4343-AEE8-E4387F9FA716@mcs.anl.gov> References: , <888DF7F8-1A65-4343-AEE8-E4387F9FA716@mcs.anl.gov> Message-ID: Dear PETSc-user I am Seungjin Seo, researcher of Korean Advanced Institute of Science and Technology, South Korea. I am tring to understand the ex12.c in SNES examples, and I have a problem... Could you give me some advise? Is there any good description or materials for PetscFE and DMPlex? I will make a code with unstructured grid with PetscFE and SNES. My geometry is a rectangle mesh (the size is not fixed so I think this is unstructured grid). The equation is I will set boundry conditions as follow; 1. (Top-side boundary) Neumann boundary 2. (Bottom-side boundary) Neumann boundary 3. (Left-side boundary) Dirichlet boundary 4. (Right-side boundary) Dirichlet boundary In example ex12.c in SNES, load vector f is defiend as f_0, and neumann boundary condition is defined as f_bd_0. f_1 means the test function gradient term. The example set general type of neumann boundary condition on f_bd_0, but I want to set each boundary condition individually. Could you give me some advise? Thansk for reading my email. Best regard, Seungjin Seo ________________________________ ?? ??: Smith, Barry F. ?? ??: 2017? 12? 21? ??? ?? 3:46:42 ?? ??: ??? (???????????) ??: petsc-users at mcs.anl.gov ??: Re: [petsc-users] I have a question for PETSc example In deed you can, yes many PETSc examples only feature a single field for simplicity, but multiple fields are fine. You simple provide your FormFunction and FormJacobian to handle both degrees of freedom per cell/node at the same time. You use SNES or TS depending on if your problem is time dependent and then use -pc_type lu to start. Once you have the physics correct come back and ask us about optimizing the linear solver with preconditioners; but until you have the physics correct it is absurd to waste time worrying about making the linear solver efficient. What you want to do, many people do and is not a big deal. Barry > On Dec 21, 2017, at 12:27 AM, ??? (???????????) wrote: > > Dear PETSc-user > > > I am Seungjin Seo, researcher of Korean Advanced Institute of Science and Technology, South Korea. > > > > I am trying to solve thermal and fluid equations within a porous structure. > The thermal equation includes a non-linear term of pressure and the fluid equation has a boundary condition using temperature gradient. > Is there any way I can solve these two equations at the same time, instead of solving temperature first with previous pressure distribution and then pressure and update temperature? > Many examples solve only one PDF instead of coupling several physics. > Please recommend me which solver is appropriate for my case? > > Best regards, > Seungjin Seo -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Jan 15 08:06:18 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Mon, 15 Jan 2018 14:06:18 +0000 Subject: [petsc-users] I have a question for PETSc example In-Reply-To: References: <888DF7F8-1A65-4343-AEE8-E4387F9FA716@mcs.anl.gov> Message-ID: <3D59C605-93DD-4364-8C49-67E18CB96783@anl.gov> The term "rectangular mesh" usually refers to a "structured grid mesh" so it is not clear if you truly need an unstructured grid. Take a look at src/snes/examples/tutorials/ex19.c Barry > On Jan 15, 2018, at 1:03 AM, ??? (???????????) wrote: > > Dear PETSc-user > > I am Seungjin Seo, researcher of Korean Advanced Institute of Science and Technology, South Korea. > > I am tring to understand the ex12.c in SNES examples, and I have a problem... > Could you give me some advise? Is there any good description or materials for PetscFE and DMPlex? > > I will make a code with unstructured grid with PetscFE and SNES. My geometry is a rectangle mesh (the size is not fixed so I think this is unstructured grid). > The equation is > I will set boundry conditions as follow; > 1. (Top-side boundary) Neumann boundary > 2. (Bottom-side boundary) Neumann boundary > 3. (Left-side boundary) Dirichlet boundary > 4. (Right-side boundary) Dirichlet boundary > > In example ex12.c in SNES, load vector f is defiend as f_0, and neumann boundary condition is defined as f_bd_0. f_1 means the test function gradient term. The example set general type of neumann boundary condition on f_bd_0, but I want to set each boundary condition individually. Could you give me some advise? > > Thansk for reading my email. > > Best regard, > Seungjin Seo > > > > ?? ??: Smith, Barry F. > ?? ??: 2017? 12? 21? ??? ?? 3:46:42 > ?? ??: ??? (???????????) > ??: petsc-users at mcs.anl.gov > ??: Re: [petsc-users] I have a question for PETSc example > > > In deed you can, yes many PETSc examples only feature a single field for simplicity, but multiple fields are fine. You simple provide your FormFunction and FormJacobian to handle both degrees of freedom per cell/node at the same time. You use SNES or TS depending on if your problem is time dependent and then use -pc_type lu to start. Once you have the physics correct come back and ask us about optimizing the linear solver with preconditioners; but until you have the physics correct it is absurd to waste time worrying about making the linear solver efficient. What you want to do, many people do and is not a big deal. > > Barry > > > > On Dec 21, 2017, at 12:27 AM, ??? (???????????) wrote: > > > > Dear PETSc-user > > > > > > I am Seungjin Seo, researcher of Korean Advanced Institute of Science and Technology, South Korea. > > > > > > > > I am trying to solve thermal and fluid equations within a porous structure. > > The thermal equation includes a non-linear term of pressure and the fluid equation has a boundary condition using temperature gradient. > > Is there any way I can solve these two equations at the same time, instead of solving temperature first with previous pressure distribution and then pressure and update temperature? > > Many examples solve only one PDF instead of coupling several physics. > > Please recommend me which solver is appropriate for my case? > > > > Best regards, > > Seungjin Seo From mistloin at unist.ac.kr Mon Jan 15 19:28:37 2018 From: mistloin at unist.ac.kr (=?ks_c_5601-1987?B?vK29wsH4ICix4rDox9ew+LnXv/jA2rfCsPjH0LrOKQ==?=) Date: Tue, 16 Jan 2018 01:28:37 +0000 Subject: [petsc-users] I have a question for PETSc example In-Reply-To: <3D59C605-93DD-4364-8C49-67E18CB96783@anl.gov> References: <888DF7F8-1A65-4343-AEE8-E4387F9FA716@mcs.anl.gov> , <3D59C605-93DD-4364-8C49-67E18CB96783@anl.gov> Message-ID: sorry, I mean that My geometry is a rectangle. That means, geometry is just rectangle box shape with four boundary. My question is that, how can I set the four boundary condition individually in example 12 of SNES? (I think that the boundary condition should be added in right side of the weak form, where load vector f and test function term is included. However, in example of petsc, boundary condition is applied as general form.) Seungjin Seo ________________________________ ?? ??: Smith, Barry F. ?? ??: 2018? 1? 15? ??? ?? 11:06:18 ?? ??: ??? (???????????) ??: Smith, Barry F.; petsc-users at mcs.anl.gov ??: Re: [petsc-users] I have a question for PETSc example The term "rectangular mesh" usually refers to a "structured grid mesh" so it is not clear if you truly need an unstructured grid. Take a look at src/snes/examples/tutorials/ex19.c Barry > On Jan 15, 2018, at 1:03 AM, ??? (???????????) wrote: > > Dear PETSc-user > > I am Seungjin Seo, researcher of Korean Advanced Institute of Science and Technology, South Korea. > > I am tring to understand the ex12.c in SNES examples, and I have a problem... > Could you give me some advise? Is there any good description or materials for PetscFE and DMPlex? > > I will make a code with unstructured grid with PetscFE and SNES. My geometry is a rectangle mesh (the size is not fixed so I think this is unstructured grid). > The equation is > I will set boundry conditions as follow; > 1. (Top-side boundary) Neumann boundary > 2. (Bottom-side boundary) Neumann boundary > 3. (Left-side boundary) Dirichlet boundary > 4. (Right-side boundary) Dirichlet boundary > > In example ex12.c in SNES, load vector f is defiend as f_0, and neumann boundary condition is defined as f_bd_0. f_1 means the test function gradient term. The example set general type of neumann boundary condition on f_bd_0, but I want to set each boundary condition individually. Could you give me some advise? > > Thansk for reading my email. > > Best regard, > Seungjin Seo > > > > ?? ??: Smith, Barry F. > ?? ??: 2017? 12? 21? ??? ?? 3:46:42 > ?? ??: ??? (???????????) > ??: petsc-users at mcs.anl.gov > ??: Re: [petsc-users] I have a question for PETSc example > > > In deed you can, yes many PETSc examples only feature a single field for simplicity, but multiple fields are fine. You simple provide your FormFunction and FormJacobian to handle both degrees of freedom per cell/node at the same time. You use SNES or TS depending on if your problem is time dependent and then use -pc_type lu to start. Once you have the physics correct come back and ask us about optimizing the linear solver with preconditioners; but until you have the physics correct it is absurd to waste time worrying about making the linear solver efficient. What you want to do, many people do and is not a big deal. > > Barry > > > > On Dec 21, 2017, at 12:27 AM, ??? (???????????) wrote: > > > > Dear PETSc-user > > > > > > I am Seungjin Seo, researcher of Korean Advanced Institute of Science and Technology, South Korea. > > > > > > > > I am trying to solve thermal and fluid equations within a porous structure. > > The thermal equation includes a non-linear term of pressure and the fluid equation has a boundary condition using temperature gradient. > > Is there any way I can solve these two equations at the same time, instead of solving temperature first with previous pressure distribution and then pressure and update temperature? > > Many examples solve only one PDF instead of coupling several physics. > > Please recommend me which solver is appropriate for my case? > > > > Best regards, > > Seungjin Seo -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Jan 15 21:49:57 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 16 Jan 2018 03:49:57 +0000 Subject: [petsc-users] I have a question for PETSc example In-Reply-To: References: <888DF7F8-1A65-4343-AEE8-E4387F9FA716@mcs.anl.gov> <3D59C605-93DD-4364-8C49-67E18CB96783@anl.gov> Message-ID: > On Jan 15, 2018, at 7:28 PM, ??? (???????????) wrote: > > sorry, I mean that > My geometry is a rectangle. > That means, geometry is just rectangle box shape with four boundary. > > My question is that, how can I set the four boundary condition individually in example 12 of SNES? Of course. Ex19.c has a different boundary condition on the top than the sides. How you define and implement your boundary conditions is up to you. PETSc does't define it. Barry > > (I think that the boundary condition should be added in right side of the weak form, where load vector f and test function term is included. However, in example of petsc, boundary condition is applied as general form.) > > Seungjin Seo > ?? ??: Smith, Barry F. > ?? ??: 2018? 1? 15? ??? ?? 11:06:18 > ?? ??: ??? (???????????) > ??: Smith, Barry F.; petsc-users at mcs.anl.gov > ??: Re: [petsc-users] I have a question for PETSc example > > > The term "rectangular mesh" usually refers to a "structured grid mesh" so it is not clear if you truly need an unstructured grid. Take a look at src/snes/examples/tutorials/ex19.c > > Barry > > > > On Jan 15, 2018, at 1:03 AM, ??? (???????????) wrote: > > > > Dear PETSc-user > > > > I am Seungjin Seo, researcher of Korean Advanced Institute of Science and Technology, South Korea. > > > > I am tring to understand the ex12.c in SNES examples, and I have a problem... > > Could you give me some advise? Is there any good description or materials for PetscFE and DMPlex? > > > > I will make a code with unstructured grid with PetscFE and SNES. My geometry is a rectangle mesh (the size is not fixed so I think this is unstructured grid). > > The equation is > > I will set boundry conditions as follow; > > 1. (Top-side boundary) Neumann boundary > > 2. (Bottom-side boundary) Neumann boundary > > 3. (Left-side boundary) Dirichlet boundary > > 4. (Right-side boundary) Dirichlet boundary > > > > In example ex12.c in SNES, load vector f is defiend as f_0, and neumann boundary condition is defined as f_bd_0. f_1 means the test function gradient term. The example set general type of neumann boundary condition on f_bd_0, but I want to set each boundary condition individually. Could you give me some advise? > > > > Thansk for reading my email. > > > > Best regard, > > Seungjin Seo > > > > > > > > ?? ??: Smith, Barry F. > > ?? ??: 2017? 12? 21? ??? ?? 3:46:42 > > ?? ??: ??? (???????????) > > ??: petsc-users at mcs.anl.gov > > ??: Re: [petsc-users] I have a question for PETSc example > > > > > > In deed you can, yes many PETSc examples only feature a single field for simplicity, but multiple fields are fine. You simple provide your FormFunction and FormJacobian to handle both degrees of freedom per cell/node at the same time. You use SNES or TS depending on if your problem is time dependent and then use -pc_type lu to start. Once you have the physics correct come back and ask us about optimizing the linear solver with preconditioners; but until you have the physics correct it is absurd to waste time worrying about making the linear solver efficient. What you want to do, many people do and is not a big deal. > > > > Barry > > > > > > > On Dec 21, 2017, at 12:27 AM, ??? (???????????) wrote: > > > > > > Dear PETSc-user > > > > > > > > > I am Seungjin Seo, researcher of Korean Advanced Institute of Science and Technology, South Korea. > > > > > > > > > > > > I am trying to solve thermal and fluid equations within a porous structure. > > > The thermal equation includes a non-linear term of pressure and the fluid equation has a boundary condition using temperature gradient. > > > Is there any way I can solve these two equations at the same time, instead of solving temperature first with previous pressure distribution and then pressure and update temperature? > > > Many examples solve only one PDF instead of coupling several physics. > > > Please recommend me which solver is appropriate for my case? > > > > > > Best regards, > > > Seungjin Seo From knepley at gmail.com Tue Jan 16 05:48:16 2018 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 16 Jan 2018 06:48:16 -0500 Subject: [petsc-users] I have a question for PETSc example In-Reply-To: References: <888DF7F8-1A65-4343-AEE8-E4387F9FA716@mcs.anl.gov> <3D59C605-93DD-4364-8C49-67E18CB96783@anl.gov> Message-ID: On Mon, Jan 15, 2018 at 8:28 PM, ??? (???????????) wrote: > sorry, I mean that > > *My geometry is a rectangle*. > > That means, geometry is just rectangle box shape with four boundary. > > > My question is that, how can I set the four boundary condition > individually in example 12 of SNES? > You put a different marker on each side. The easiest way to do it is to give the -dm_plex_separate_marker option. This is done in SNES ex69. Thanks, Matt > (I think that the boundary condition should be added in right side of the > weak form, where load vector f and test function term is included. However, > in example of petsc, boundary condition is applied as general form.) > > > Seungjin Seo > ------------------------------ > *?? ??:* Smith, Barry F. > *?? ??:* 2018? 1? 15? ??? ?? 11:06:18 > *?? ??:* ??? (???????????) > *??:* Smith, Barry F.; petsc-users at mcs.anl.gov > > *??:* Re: [petsc-users] I have a question for PETSc example > > > The term "rectangular mesh" usually refers to a "structured grid mesh" > so it is not clear if you truly need an unstructured grid. Take a look at > src/snes/examples/tutorials/ex19.c > > Barry > > > > On Jan 15, 2018, at 1:03 AM, ??? (???????????) > wrote: > > > > Dear PETSc-user > > > > I am Seungjin Seo, researcher of Korean Advanced Institute of Science > and Technology, South Korea. > > > > I am tring to understand the ex12.c in SNES examples, and I have a > problem... > > Could you give me some advise? Is there any good description or > materials for PetscFE and DMPlex? > > > > I will make a code with unstructured grid with PetscFE and SNES. My > geometry is a rectangle mesh (the size is not fixed so I think this is > unstructured grid). > > The equation is > > I will set boundry conditions as follow; > > 1. (Top-side boundary) Neumann boundary > > 2. (Bottom-side boundary) Neumann boundary > > 3. (Left-side boundary) Dirichlet boundary > > 4. (Right-side boundary) Dirichlet boundary > > > > In example ex12.c in SNES, load vector f is defiend as f_0, and neumann > boundary condition is defined as f_bd_0. f_1 means the test function > gradient term. The example set general type of neumann boundary condition > on f_bd_0, but I want to set each boundary condition individually. Could > you give me some advise? > > > > Thansk for reading my email. > > > > Best regard, > > Seungjin Seo > > > > > > > > ?? ??: Smith, Barry F. > > ?? ??: 2017? 12? 21? ??? ?? 3:46:42 > > ?? ??: ??? (???????????) > > ??: petsc-users at mcs.anl.gov > > ??: Re: [petsc-users] I have a question for PETSc example > > > > > > In deed you can, yes many PETSc examples only feature a single field > for simplicity, but multiple fields are fine. You simple provide your > FormFunction and FormJacobian to handle both degrees of freedom per > cell/node at the same time. You use SNES or TS depending on if your problem > is time dependent and then use -pc_type lu to start. Once you have the > physics correct come back and ask us about optimizing the linear solver > with preconditioners; but until you have the physics correct it is absurd > to waste time worrying about making the linear solver efficient. What you > want to do, many people do and is not a big deal. > > > > Barry > > > > > > > On Dec 21, 2017, at 12:27 AM, ??? (???????????) > wrote: > > > > > > Dear PETSc-user > > > > > > > > > I am Seungjin Seo, researcher of Korean Advanced Institute of Science > and Technology, South Korea. > > > > > > > > > > > > I am trying to solve thermal and fluid equations within a porous > structure. > > > The thermal equation includes a non-linear term of pressure and the > fluid equation has a boundary condition using temperature gradient. > > > Is there any way I can solve these two equations at the same time, > instead of solving temperature first with previous pressure distribution > and then pressure and update temperature? > > > Many examples solve only one PDF instead of coupling several physics. > > > Please recommend me which solver is appropriate for my case? > > > > > > Best regards, > > > Seungjin Seo > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From f.el-haddad at imperial.ac.uk Tue Jan 16 12:26:03 2018 From: f.el-haddad at imperial.ac.uk (El Haddad, Fadi) Date: Tue, 16 Jan 2018 18:26:03 +0000 Subject: [petsc-users] matrix free non linear solver Fortran Message-ID: Hello Petsc Users, I am trying to set up a snes matrix free solver into an existing code. I obtained the following log error by petsc : [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: No support for this operation for this object type [0]PETSC ERROR: Mat type mffd [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: unknown GIT Date: unknown [0]PETSC ERROR: ../Forse/JM62-FORSE/petsc-infconv/forse on a arch-linux2-c-debug named me-felhadda by felhadda Tue Jan 16 12:49:53 2018 [0]PETSC ERROR: Configure options --with-cc=mpicc --with-cxx=mpic++ --with-fc=mpif90 --with-mpi-dir=/usr/lib/mpich --download-fblaslapack [0]PETSC ERROR: #1 MatZeroEntries() line 5617 in /opt/petsc/petsc-infconv/src/mat/interface/matrix.c [0]PETSC ERROR: #2 SNESComputeJacobianDefault() line 64 in /opt/petsc/petsc-infconv/src/snes/interface/snesj.c [0]PETSC ERROR: #3 oursnesjacobian() line 97 in /opt/petsc/petsc-infconv/src/snes/interface/ftn-custom/zsnesf.c [0]PETSC ERROR: #4 SNESComputeJacobian() line 2374 in /opt/petsc/petsc-infconv/src/snes/interface/snes.c [0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 222 in /opt/petsc/petsc-infconv/src/snes/impls/ls/ls.c [0]PETSC ERROR: #6 SNESSolve() line 4179 in /opt/petsc/petsc-infconv/src/snes/interface/snes.c Have anyone dealt with a similar error? I copied my snes matrix free set up here below : external :: FormFunction external :: MySNESConverged call PetscInitialize(PETSC_NULL_CHARACTER,ierr) if (ierr .ne. 0) then print*,'Unable to initialize PETSc' stop endif call MPI_Comm_size(PETSC_COMM_SELF,size,ierr) call MPI_Comm_rank(PETSC_COMM_SELF,rank,ierr) call VecCreateSeq(PETSC_COMM_SELF,n,x,ierr) call VecCreateSeq(PETSC_COMM_SELF,n,r,ierr) call SNESCreate(PETSC_COMM_SELF,snes,ierr) call SNESSetApplicationContext(snes,userCtxptrs,ierr);CHKERRA(ierr) call SNESSetFunction(snes,r,FormFunction,userCtxptrs,ierr);CHKERRA(ierr) call MatCreateSnesMF(snes,JMF,ierr) call MatMFFDWPSetComputeNormU(JMF,PETSC_FALSE,ierr) call SNESSetJacobian(snes,Jmf,Jmf,SNESComputeJacobianDefault,userCtxptrs,ierr) call SNESSetUseMatrixFree(self%snes,PETSC_TRUE,PETSC_TRUE,ierr) call SNESSetConvergenceTest(snes,MySNESConverged,userCtxptrs,PETSC_NULL_FUNCTION,ierr) call SNESSetConvergedReason(snes,2,ierr) call SNESSetNormSchedule(snes,SNES_NORM_ALWAYS,ierr) call SnesSetTolerances(snes,1.d-8,1.d-7,1.d-7,10,25,ierr) call SNESSetFromOptions(snes,ierr);CHKERRA(ierr) call SNESView(snes,PETSC_VIEWER_STDOUT_SELF,ierr) call getpointersUserCtx(ds,pt,TypeY,TypeF,userCtxptrs) call FormInitialGuess(x,pt) call SNESsolve(snes,PETSC_NULL_VEC,x,ierr) call SNESGetConvergedReason(snes,reason,ierr) IF (reason .eq. 2 ) then call VecGetArrayReadF90(x,xx,ierr) pt%Y = xx call VecRestoreArrayReadF90(x,xx,ierr) ConvFlag = True ELSE ConvFlag = False ENDIF Am I missing a crucial subroutine call? Thank you for the help Fadi -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jan 16 12:33:48 2018 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 16 Jan 2018 13:33:48 -0500 Subject: [petsc-users] matrix free non linear solver Fortran In-Reply-To: References: Message-ID: On Tue, Jan 16, 2018 at 1:26 PM, El Haddad, Fadi wrote: > Hello Petsc Users, > > I am trying to set up a snes matrix free solver into an existing code. > I obtained the following log error by petsc : > Setting it up is a little tricky, Why not just omit SNESSetJacobian() and it will automatically set it up? Thanks, Matt > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: No support for this operation for this object type > [0]PETSC ERROR: Mat type mffd > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: unknown GIT Date: unknown > [0]PETSC ERROR: ../Forse/JM62-FORSE/petsc-infconv/forse on a > arch-linux2-c-debug named me-felhadda by felhadda Tue Jan 16 12:49:53 2018 > [0]PETSC ERROR: Configure options --with-cc=mpicc --with-cxx=mpic++ > --with-fc=mpif90 --with-mpi-dir=/usr/lib/mpich --download-fblaslapack > [0]PETSC ERROR: #1 MatZeroEntries() line 5617 in > /opt/petsc/petsc-infconv/src/mat/interface/matrix.c > [0]PETSC ERROR: #2 SNESComputeJacobianDefault() line 64 in > /opt/petsc/petsc-infconv/src/snes/interface/snesj.c > [0]PETSC ERROR: #3 oursnesjacobian() line 97 in > /opt/petsc/petsc-infconv/src/snes/interface/ftn-custom/zsnesf.c > [0]PETSC ERROR: #4 SNESComputeJacobian() line 2374 in > /opt/petsc/petsc-infconv/src/snes/interface/snes.c > [0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 222 in > /opt/petsc/petsc-infconv/src/snes/impls/ls/ls.c > [0]PETSC ERROR: #6 SNESSolve() line 4179 in /opt/petsc/petsc-infconv/src/ > snes/interface/snes.c > > > Have anyone dealt with a similar error? I copied my snes matrix free set > up here below : > > > external :: FormFunction > external :: MySNESConverged > > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > if (ierr .ne. 0) then > print*,'Unable to initialize PETSc' > stop > endif > call MPI_Comm_size(PETSC_COMM_SELF,size,ierr) > call MPI_Comm_rank(PETSC_COMM_SELF,rank,ierr) > > > call VecCreateSeq(PETSC_COMM_SELF,n,x,ierr) > call VecCreateSeq(PETSC_COMM_SELF,n,r,ierr) > > call SNESCreate(PETSC_COMM_SELF,snes,ierr) > call SNESSetApplicationContext(snes,userCtxptrs,ierr); > CHKERRA(ierr) > > call SNESSetFunction(snes,r,FormFunction,userCtxptrs,ierr) > ;CHKERRA(ierr) > call MatCreateSnesMF(snes,JMF,ierr) > call MatMFFDWPSetComputeNormU(JMF,PETSC_FALSE,ierr) > > call SNESSetJacobian(snes,Jmf,Jmf, > SNESComputeJacobianDefault,userCtxptrs,ierr) > call SNESSetUseMatrixFree(self%snes,PETSC_TRUE,PETSC_TRUE, > ierr) > call SNESSetConvergenceTest(snes, > MySNESConverged,userCtxptrs,PETSC_NULL_FUNCTION,ierr) > > call SNESSetConvergedReason(snes,2,ierr) > call SNESSetNormSchedule(snes,SNES_NORM_ALWAYS,ierr) > > call SnesSetTolerances(snes,1.d-8,1.d-7,1.d-7,10,25,ierr) > call SNESSetFromOptions(snes,ierr);CHKERRA(ierr) > call SNESView(snes,PETSC_VIEWER_STDOUT_SELF,ierr) > > call getpointersUserCtx(ds,pt,TypeY,TypeF,userCtxptrs) > call FormInitialGuess(x,pt) > > call SNESsolve(snes,PETSC_NULL_VEC,x,ierr) > > call SNESGetConvergedReason(snes,reason,ierr) > IF (reason .eq. 2 ) then > call VecGetArrayReadF90(x,xx,ierr) > pt%Y = xx > call VecRestoreArrayReadF90(x,xx,ierr) > ConvFlag = True > ELSE > ConvFlag = False > ENDIF > > Am I missing a crucial subroutine call? > Thank you for the help > > Fadi > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jan 16 13:04:15 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 16 Jan 2018 19:04:15 +0000 Subject: [petsc-users] matrix free non linear solver Fortran In-Reply-To: References: Message-ID: <71F2D064-1DE1-4887-9467-97DF21696A0D@anl.gov> Also from the users manual for MatCreateSNESMF() Notes: You can call SNESSetJacobian() with MatMFFDComputeJacobian() if you are using matrix and not a different preconditioner matrix > On Jan 16, 2018, at 12:33 PM, Matthew Knepley wrote: > > On Tue, Jan 16, 2018 at 1:26 PM, El Haddad, Fadi wrote: > Hello Petsc Users, > > I am trying to set up a snes matrix free solver into an existing code. > I obtained the following log error by petsc : > > Setting it up is a little tricky, Why not just omit SNESSetJacobian() and it will automatically set it up? > > Thanks, > > Matt > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: No support for this operation for this object type > [0]PETSC ERROR: Mat type mffd > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: unknown GIT Date: unknown > [0]PETSC ERROR: ../Forse/JM62-FORSE/petsc-infconv/forse on a arch-linux2-c-debug named me-felhadda by felhadda Tue Jan 16 12:49:53 2018 > [0]PETSC ERROR: Configure options --with-cc=mpicc --with-cxx=mpic++ --with-fc=mpif90 --with-mpi-dir=/usr/lib/mpich --download-fblaslapack > [0]PETSC ERROR: #1 MatZeroEntries() line 5617 in /opt/petsc/petsc-infconv/src/mat/interface/matrix.c > [0]PETSC ERROR: #2 SNESComputeJacobianDefault() line 64 in /opt/petsc/petsc-infconv/src/snes/interface/snesj.c > [0]PETSC ERROR: #3 oursnesjacobian() line 97 in /opt/petsc/petsc-infconv/src/snes/interface/ftn-custom/zsnesf.c > [0]PETSC ERROR: #4 SNESComputeJacobian() line 2374 in /opt/petsc/petsc-infconv/src/snes/interface/snes.c > [0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 222 in /opt/petsc/petsc-infconv/src/snes/impls/ls/ls.c > [0]PETSC ERROR: #6 SNESSolve() line 4179 in /opt/petsc/petsc-infconv/src/snes/interface/snes.c > > > Have anyone dealt with a similar error? I copied my snes matrix free set up here below : > > > external :: FormFunction > external :: MySNESConverged > > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > if (ierr .ne. 0) then > print*,'Unable to initialize PETSc' > stop > endif > call MPI_Comm_size(PETSC_COMM_SELF,size,ierr) > call MPI_Comm_rank(PETSC_COMM_SELF,rank,ierr) > > > call VecCreateSeq(PETSC_COMM_SELF,n,x,ierr) > call VecCreateSeq(PETSC_COMM_SELF,n,r,ierr) > > call SNESCreate(PETSC_COMM_SELF,snes,ierr) > call SNESSetApplicationContext(snes,userCtxptrs,ierr);CHKERRA(ierr) > > call SNESSetFunction(snes,r,FormFunction,userCtxptrs,ierr);CHKERRA(ierr) > call MatCreateSnesMF(snes,JMF,ierr) > call MatMFFDWPSetComputeNormU(JMF,PETSC_FALSE,ierr) > > call SNESSetJacobian(snes,Jmf,Jmf,SNESComputeJacobianDefault,userCtxptrs,ierr) > call SNESSetUseMatrixFree(self%snes,PETSC_TRUE,PETSC_TRUE,ierr) > call SNESSetConvergenceTest(snes,MySNESConverged,userCtxptrs,PETSC_NULL_FUNCTION,ierr) > > call SNESSetConvergedReason(snes,2,ierr) > call SNESSetNormSchedule(snes,SNES_NORM_ALWAYS,ierr) > > call SnesSetTolerances(snes,1.d-8,1.d-7,1.d-7,10,25,ierr) > call SNESSetFromOptions(snes,ierr);CHKERRA(ierr) > call SNESView(snes,PETSC_VIEWER_STDOUT_SELF,ierr) > > call getpointersUserCtx(ds,pt,TypeY,TypeF,userCtxptrs) > call FormInitialGuess(x,pt) > > call SNESsolve(snes,PETSC_NULL_VEC,x,ierr) > > call SNESGetConvergedReason(snes,reason,ierr) > IF (reason .eq. 2 ) then > call VecGetArrayReadF90(x,xx,ierr) > pt%Y = xx > call VecRestoreArrayReadF90(x,xx,ierr) > ConvFlag = True > ELSE > ConvFlag = False > ENDIF > > Am I missing a crucial subroutine call? > Thank you for the help > > Fadi > > > > > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ From marek.pecha at vsb.cz Wed Jan 17 06:35:28 2018 From: marek.pecha at vsb.cz (Marek Pecha) Date: Wed, 17 Jan 2018 13:35:28 +0100 Subject: [petsc-users] HDF5 Write attribute Message-ID: <94318AE1-337A-4B96-8948-93D0A9EA2215@vsb.cz> Hello, I create very simple program for writing scalar to HDF5 file (it is just test-program, because I learn how HDF5 works), see below. #include int main(int argc, char *argv[]) { PetscErrorCode ierr; PetscScalar n = 5; PetscViewer viewer; PetscInitialize(&argc,&argv,(char*)0,NULL); PetscViewerHDF5Open(PETSC_COMM_WORLD, "test.hdf5", FILE_MODE_WRITE, &viewer); PetscViewerHDF5PushGroup(viewer, "/"); ierr = PetscViewerHDF5WriteAttribute(viewer, "/", "n", PETSC_SCALAR, &n); CHKERRQ(ierr); PetscViewerDestroy(&viewer); PetscFinalize(); return (0); } However, I got following error: HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 0: #000: H5D.c line 341 in H5Dopen2(): not a dataset major: Dataset minor: Inappropriate type [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Error in external library [0]PETSC ERROR: Error in HDF5 call H5Dopen2() Status -1 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.7.5, unknown [0]PETSC ERROR: ./aijmathdf5 on a arch-llvm-opt named sweety by dev Wed Jan 17 13:33:51 2018 [0]PETSC ERROR: Configure options --download-blacs --download-boost --download-fblaslapack --download-metis --download-mpich --download-mumps --download-parmetis --download-scalapack --download-superlu --download-superlu_dist --download-hdf5 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 PETSC_ARCH=arch-llvm-opt [0]PETSC ERROR: #1 PetscViewerHDF5WriteAttribute() line 653 in /opt/petsc/petsc-maint/src/sys/classes/viewer/impls/hdf5/hdf5v.c [0]PETSC ERROR: #2 main() line 13 in /Users/dev/Desktop/sparse_hdf5/main.cpp [0]PETSC ERROR: No PETSc Option Table entries [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- application called MPI_Abort(MPI_COMM_WORLD, 76) - process 0 [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=76 : system msg for write_line failure : Bad file descriptor I don?t know, what I am doing wrong. Thank you Marek From gohardoust at gmail.com Wed Jan 17 14:31:07 2018 From: gohardoust at gmail.com (Mohammad Gohardoust) Date: Wed, 17 Jan 2018 13:31:07 -0700 Subject: [petsc-users] Issue with KSPSolve when the number of CPUs are larger than four In-Reply-To: References: Message-ID: Hi Matt, Thanks for your response. You were right: the right hand side matrix has had some 'nan's which caused the solver to return error. Strangely this does not happen when #cpus are less than 4! I am going to search for the source of that. Best, Mohammad On Wed, Jan 10, 2018 at 5:04 PM, Matthew Knepley wrote: > On Wed, Jan 10, 2018 at 4:41 PM, Mohammad R. Gohardoust < > gohardoust at email.arizona.edu> wrote: > >> Hi, >> >> I hope you are doing well. I am Mohammad, a PhD student of environmental >> sciences at the University of Arizona. >> >> I do appreciate any help that can shed some light on the issue I have: >> recently I have added a feature to an existing parallel code called >> 'parswms' which solves water and solute transport in soils. The code uses >> MPI and parmetis for the parallelization purposes and petsc package >> (KSPSolve) for solving linear systems. I have it installed on the UofA >> HPC : the issue is that it works well if the number of CPUs are up to 4! >> but when I add to this number the numerical linear solver (here it is >> 'KSPCGS') stops with the error of 'KSP_DIVERGED_NANORINF'. >> >> Would you mind please giving me some hints, suggestions or resources in >> this regard? >> > > It sounds like you might have a problem in matrix assembly. However, first > use -pc_type gmres instead since > it has nicer numerical behavior (I believe). > > Thanks, > > Matt > > >> Best Regards, >> Mohammad >> >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gohardoust at gmail.com Wed Jan 17 14:33:14 2018 From: gohardoust at gmail.com (Mohammad Gohardoust) Date: Wed, 17 Jan 2018 13:33:14 -0700 Subject: [petsc-users] Issue with KSPSolve when the number of CPUs are larger than four In-Reply-To: <13F171E7-B6F3-4074-AD03-6A66EED40695@anl.gov> References: <13F171E7-B6F3-4074-AD03-6A66EED40695@anl.gov> Message-ID: Thanks Barry, some 'nan's in the rhs matrix caused the issue strangely when cpu numbers gets larger than 4. Best, Mohammad On Wed, Jan 10, 2018 at 5:30 PM, Smith, Barry F. wrote: > > You can also run with -fp_trap to find where the nan/inf first appears > as well as run with -ksp_monitor to see what is happening before the tragedy > > Barry > > > > On Jan 10, 2018, at 3:41 PM, Mohammad R. Gohardoust < > gohardoust at email.arizona.edu> wrote: > > > > Hi, > > > > I hope you are doing well. I am Mohammad, a PhD student of environmental > sciences at the University of Arizona. > > > > I do appreciate any help that can shed some light on the issue I have: > recently I have added a feature to an existing parallel code called > 'parswms' which solves water and solute transport in soils. The code uses > MPI and parmetis for the parallelization purposes and petsc package > (KSPSolve) for solving linear systems. I have it installed on the UofA HPC > : the issue is that it works well if the number of CPUs are up to 4! but > when I add to this number the numerical linear solver (here it is 'KSPCGS') > stops with the error of 'KSP_DIVERGED_NANORINF'. > > > > Would you mind please giving me some hints, suggestions or resources in > this regard? > > > > > > Best Regards, > > Mohammad > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistloin at unist.ac.kr Thu Jan 18 05:10:18 2018 From: mistloin at unist.ac.kr (=?ks_c_5601-1987?B?vK29wsH4ICix4rDox9ew+LnXv/jA2rfCsPjH0LrOKQ==?=) Date: Thu, 18 Jan 2018 11:10:18 +0000 Subject: [petsc-users] I have a question for PETSc example In-Reply-To: References: <888DF7F8-1A65-4343-AEE8-E4387F9FA716@mcs.anl.gov> <3D59C605-93DD-4364-8C49-67E18CB96783@anl.gov> , Message-ID: Dear petsc-user. Hi. I am trying several parts of example code, (SNES ex12.c : Poisson Problem in 2d and 3d with simplicial finite element) and I have several problems to understand. I appologize to sent a long and boring email to you... 1) About setting of geometry [cid:009d718e-c50b-4bd9-8d2c-714321b9fc46] [cid:7e0f732c-ecad-4ea7-a3f3-8d6a09bc5cc9] [cid:40e3a7a8-4191-4693-bc97-84a550d5152f] I referred the "DMPlexCreateBoxMesh" function to make simplicial box mesh. I found that lower[] and upper[] are factors to set the size of the box mesh. Is it right? Actually, I want to make a rectangular geometry with size of 3m x 1m, and four boundary (two dirichlet and two neumann condition). If I wrong, please give me some tip to make a good geometry. 2) About application boundary condition [cid:c6eed790-070c-4e66-8dbb-8b461fb06f10] [cid:37a6abda-13a5-45bd-b751-2d193f784071] I reffered the example "SNES ex69.c" to set a label and condition for the four boundarys. First part of code is about label of boundary condition, and second part is about setting of boundary condition. However, due to second part, error occurs. Is there any wrong argument? Please give me some tip to applied four boundary conditions on the mesh. 3) About input parameters of DSSetJacobian() In DSSetJacobian (prob, f, g, g0, g1, g2, g3), I have a problem to understand g0~g3 g0 : integrand for test function and basis function g1 : integrand for test function gradient and basis function g2 : integrand for test function and basis function gradient g3 : integrand for test function gradient and basis function gradient I have studied the FEM before one month, but I am very confused by above four terms. I should study this in my self, but I need to some tip or internet address which include a helpful description about above terms. (I think this issue is not good at this email, but I have been really suffered by above term...) 4) About useful tool to visualize the result and geometry [cid:80900b60-6b68-4c72-8f3e-df0d475795d7] I also have searched the tool which can express the visualized result and geometry. Could you give me a tip to select the visualizing tool for better analysis? Thanks you for reading my email. Best regards, Seungjin Seo ________________________________ ?? ??: Matthew Knepley ?? ??: 2018? 1? 16? ??? ?? 8:48:16 ?? ??: ??? (???????????) ??: Smith, Barry F.; petsc-users at mcs.anl.gov ??: Re: [petsc-users] I have a question for PETSc example On Mon, Jan 15, 2018 at 8:28 PM, ??? (???????????) > wrote: sorry, I mean that My geometry is a rectangle. That means, geometry is just rectangle box shape with four boundary. My question is that, how can I set the four boundary condition individually in example 12 of SNES? You put a different marker on each side. The easiest way to do it is to give the -dm_plex_separate_marker option. This is done in SNES ex69. Thanks, Matt (I think that the boundary condition should be added in right side of the weak form, where load vector f and test function term is included. However, in example of petsc, boundary condition is applied as general form.) Seungjin Seo ________________________________ ?? ??: Smith, Barry F. > ?? ??: 2018? 1? 15? ??? ?? 11:06:18 ?? ??: ??? (???????????) ??: Smith, Barry F.; petsc-users at mcs.anl.gov ??: Re: [petsc-users] I have a question for PETSc example The term "rectangular mesh" usually refers to a "structured grid mesh" so it is not clear if you truly need an unstructured grid. Take a look at src/snes/examples/tutorials/ex19.c Barry > On Jan 15, 2018, at 1:03 AM, ??? (???????????) > wrote: > > Dear PETSc-user > > I am Seungjin Seo, researcher of Korean Advanced Institute of Science and Technology, South Korea. > > I am tring to understand the ex12.c in SNES examples, and I have a problem... > Could you give me some advise? Is there any good description or materials for PetscFE and DMPlex? > > I will make a code with unstructured grid with PetscFE and SNES. My geometry is a rectangle mesh (the size is not fixed so I think this is unstructured grid). > The equation is > I will set boundry conditions as follow; > 1. (Top-side boundary) Neumann boundary > 2. (Bottom-side boundary) Neumann boundary > 3. (Left-side boundary) Dirichlet boundary > 4. (Right-side boundary) Dirichlet boundary > > In example ex12.c in SNES, load vector f is defiend as f_0, and neumann boundary condition is defined as f_bd_0. f_1 means the test function gradient term. The example set general type of neumann boundary condition on f_bd_0, but I want to set each boundary condition individually. Could you give me some advise? > > Thansk for reading my email. > > Best regard, > Seungjin Seo > > > > ?? ??: Smith, Barry F. > > ?? ??: 2017? 12? 21? ??? ?? 3:46:42 > ?? ??: ??? (???????????) > ??: petsc-users at mcs.anl.gov > ??: Re: [petsc-users] I have a question for PETSc example > > > In deed you can, yes many PETSc examples only feature a single field for simplicity, but multiple fields are fine. You simple provide your FormFunction and FormJacobian to handle both degrees of freedom per cell/node at the same time. You use SNES or TS depending on if your problem is time dependent and then use -pc_type lu to start. Once you have the physics correct come back and ask us about optimizing the linear solver with preconditioners; but until you have the physics correct it is absurd to waste time worrying about making the linear solver efficient. What you want to do, many people do and is not a big deal. > > Barry > > > > On Dec 21, 2017, at 12:27 AM, ??? (???????????) > wrote: > > > > Dear PETSc-user > > > > > > I am Seungjin Seo, researcher of Korean Advanced Institute of Science and Technology, South Korea. > > > > > > > > I am trying to solve thermal and fluid equations within a porous structure. > > The thermal equation includes a non-linear term of pressure and the fluid equation has a boundary condition using temperature gradient. > > Is there any way I can solve these two equations at the same time, instead of solving temperature first with previous pressure distribution and then pressure and update temperature? > > Many examples solve only one PDF instead of coupling several physics. > > Please recommend me which solver is appropriate for my case? > > > > Best regards, > > Seungjin Seo -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 109380 bytes Desc: pastedImage.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 134256 bytes Desc: pastedImage.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 74067 bytes Desc: pastedImage.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 43510 bytes Desc: pastedImage.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 24165 bytes Desc: pastedImage.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 8003 bytes Desc: pastedImage.png URL: From knepley at gmail.com Thu Jan 18 09:55:55 2018 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 18 Jan 2018 10:55:55 -0500 Subject: [petsc-users] HDF5 Write attribute In-Reply-To: <94318AE1-337A-4B96-8948-93D0A9EA2215@vsb.cz> References: <94318AE1-337A-4B96-8948-93D0A9EA2215@vsb.cz> Message-ID: On Wed, Jan 17, 2018 at 7:35 AM, Marek Pecha wrote: > Hello, > > I create very simple program for writing scalar to HDF5 file (it is just > test-program, because I learn how HDF5 works), see below. > > #include > > int main(int argc, char *argv[]) { > PetscErrorCode ierr; > PetscScalar n = 5; > PetscViewer viewer; > > PetscInitialize(&argc,&argv,(char*)0,NULL); > > PetscViewerHDF5Open(PETSC_COMM_WORLD, "test.hdf5", FILE_MODE_WRITE, > &viewer); > PetscViewerHDF5PushGroup(viewer, "/"); > ierr = PetscViewerHDF5WriteAttribute(viewer, "/", "n", PETSC_SCALAR, > &n); > CHKERRQ(ierr); > PetscViewerDestroy(&viewer); > > PetscFinalize(); > > return (0); > } > > However, I got following error: > > HDF5-DIAG: Error detected in HDF5 (1.8.12) MPI-process 0: > #000: H5D.c line 341 in H5Dopen2(): not a dataset > Sorry it took me so long to respond. I was on winter break. The problem is that HDF5 attributes can only be applied to datasets. '/' is not a dataset. For example, if you wrote a Vec into the file, you could apply an attribute to it, like 'axial vector'. Thanks, Matt > major: Dataset > minor: Inappropriate type > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Error in external library > [0]PETSC ERROR: Error in HDF5 call H5Dopen2() Status -1 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.7.5, unknown > [0]PETSC ERROR: ./aijmathdf5 on a arch-llvm-opt named sweety by dev Wed > Jan 17 13:33:51 2018 > [0]PETSC ERROR: Configure options --download-blacs --download-boost > --download-fblaslapack --download-metis --download-mpich --download-mumps > --download-parmetis --download-scalapack --download-superlu > --download-superlu_dist --download-hdf5 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 > FOPTFLAGS=-O3 PETSC_ARCH=arch-llvm-opt > [0]PETSC ERROR: #1 PetscViewerHDF5WriteAttribute() line 653 in > /opt/petsc/petsc-maint/src/sys/classes/viewer/impls/hdf5/hdf5v.c > [0]PETSC ERROR: #2 main() line 13 in /Users/dev/Desktop/sparse_ > hdf5/main.cpp > [0]PETSC ERROR: No PETSc Option Table entries > [0]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint at mcs.anl.gov---------- > application called MPI_Abort(MPI_COMM_WORLD, 76) - process 0 > [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=76 > : > system msg for write_line failure : Bad file descriptor > > I don?t know, what I am doing wrong. > > Thank you > Marek -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From aurelien.ponte at ifremer.fr Thu Jan 18 09:56:22 2018 From: aurelien.ponte at ifremer.fr (Aurelien Ponte) Date: Thu, 18 Jan 2018 16:56:22 +0100 Subject: [petsc-users] petsc4py f2py Message-ID: <4fa20faa-1110-b282-ee74-e19f13357818@ifremer.fr> Hi, I am trying to speed up with f2py part of a python code that depends on petsc4py and cannot compile the fortran library. The error message are at the end of this message. The code is found here: https://github.com/apatlpo/qgsolver/tree/fwrapping/qgsolver This attempt borrows heavily from petsc4py/demo/wrap-f2py. The error 'Unexpected MODULE statement' does not get lot's of replies on google unfortunately Would you have any suggestions? thanks aur?lien (petsc) br146-244:qgsolver aponte$ make PYTHON=/Users/aponte/.miniconda3/envs/petsc/bin/python build /Users/aponte/.miniconda3/envs/petsc/bin:/opt/local/bin:/opt/local/sbin:/Users/aponte/.miniconda3/bin:/opt/local/bin:/opt/local/sbin:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin:/Library/TeX/texbin:/usr/local/bin:/Users/aponte/Installed/f90tohtml:/usr/local/mysql/bin:/Users/aponte/Tools/ftagshtml-0.520/bin:/usr/local/ferret/bin:/usr/local/bin:/Users/aponte/Installed/f90tohtml:/usr/local/mysql/bin:/Users/aponte/Tools/ftagshtml-0.520/bin:/usr/local/ferret/bin /Users/aponte/.miniconda3/envs/petsc/bin/python /bin/sh env \ ??????? F77="/opt/local/bin/mpif90-openmpi-gcc49" F77FLAGS="-Os -Os? -I/opt/local/include" \ ??????? F90="/opt/local/bin/mpif90-openmpi-gcc49" F90FLAGS="-Os -Os? -I/opt/local/include" \ ??????? LDSHARED="/opt/local/bin/mpif90-openmpi-gcc49" \ ??? /Users/aponte/.miniconda3/envs/petsc/bin/python setup.py -q build_ext --inplace Reading .f2py_f2cmap ... ??? Mapping "integer(kind=HANDLE_KIND)" to "long" ??? Mapping "integer(kind=handle_kind)" to "long" Successfully applied user defined changes from .f2py_f2cmap In file included from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1809:0, ???????????????? from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18, ???????????????? from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4, ???????????????? from build/src.macosx-10.9-x86_64-3.6/./fortranobject.h:13, ???????????????? from ./timestepper_advmodule.c:18: /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] ?#warning "Using deprecated NumPy API, disable it by " \ ? ^ In file included from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1809:0, ???????????????? from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18, ???????????????? from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4, ???????????????? from build/src.macosx-10.9-x86_64-3.6/./fortranobject.h:13, ???????????????? from build/src.macosx-10.9-x86_64-3.6/./fortranobject.c:2: /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] ?#warning "Using deprecated NumPy API, disable it by " \ ? ^ build/src.macosx-10.9-x86_64-3.6/./fortranobject.c: In function 'format_def': build/src.macosx-10.9-x86_64-3.6/./fortranobject.c:138:18: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] ???????? if (size < sizeof(notalloc)) { ????????????????? ^ ./timestepper_adv.F90:9.22: module timestepper_adv ????????????????????? 1 Error: Unexpected MODULE statement at (1) ./timestepper_adv.F90:11.11: ? use petsc ?????????? 1 Error: Unexpected USE statement at (1) ./timestepper_adv.F90:12.15: ? implicit none ?????????????? 1 Error: Unexpected IMPLICIT NONE statement at (1) ./timestepper_adv.F90:14.15: ? type gridinfo ?????????????? 1 ... Error: Symbol 'grd' at (1) has no IMPLICIT type ./timestepper_adv.F90:59.18: ??? type(gridinfo) grd ????????????????? 1 Error: Derived type 'gridinfo' at (1) is being used before it is defined Fatal Error: Error count reached limit of 25. error: Command "/opt/local/bin/mpif90-openmpi-gcc49 -Os -Os -I/opt/local/include -arch x86_64 -fPIC -O3 -funroll-loops -DF2PY_REPORT_ON_ARRAY_COPY=1 -I/opt/local/lib/petsc/include -I/Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/petsc4py/include -I. -Ibuild/src.macosx-10.9-x86_64-3.6/. -I/Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include -I/Users/aponte/.miniconda3/envs/petsc/include/python3.6m -c -c ./timestepper_adv.F90 -o build/temp.macosx-10.9-x86_64-3.6/timestepper_adv.o -Jbuild/temp.macosx-10.9-x86_64-3.6/ -Ibuild/temp.macosx-10.9-x86_64-3.6/" failed with exit status 1 make: *** [timestepper_adv.so] Error 1 -- Aur?lien Ponte Tel: (+33) 2 98 22 40 73 Fax: (+33) 2 98 22 44 96 UMR 6523, IFREMER ZI de la Pointe du Diable CS 10070 29280 Plouzan? From bsmith at mcs.anl.gov Thu Jan 18 10:05:07 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 18 Jan 2018 16:05:07 +0000 Subject: [petsc-users] petsc4py f2py In-Reply-To: <4fa20faa-1110-b282-ee74-e19f13357818@ifremer.fr> References: <4fa20faa-1110-b282-ee74-e19f13357818@ifremer.fr> Message-ID: Maybe it is not processing the #include as expected? Could you run with the -E option to see what it puts in for the include file? Barry > On Jan 18, 2018, at 9:56 AM, Aurelien Ponte wrote: > > Hi, > I am trying to speed up with f2py part of a python code that depends on petsc4py > and cannot compile the fortran library. > The error message are at the end of this message. > The code is found here: > https://github.com/apatlpo/qgsolver/tree/fwrapping/qgsolver > This attempt borrows heavily from petsc4py/demo/wrap-f2py. > The error 'Unexpected MODULE statement' does not get lot's of replies on google unfortunately > Would you have any suggestions? > > thanks > aur?lien > > > (petsc) br146-244:qgsolver aponte$ make PYTHON=/Users/aponte/.miniconda3/envs/petsc/bin/python build > /Users/aponte/.miniconda3/envs/petsc/bin:/opt/local/bin:/opt/local/sbin:/Users/aponte/.miniconda3/bin:/opt/local/bin:/opt/local/sbin:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin:/Library/TeX/texbin:/usr/local/bin:/Users/aponte/Installed/f90tohtml:/usr/local/mysql/bin:/Users/aponte/Tools/ftagshtml-0.520/bin:/usr/local/ferret/bin:/usr/local/bin:/Users/aponte/Installed/f90tohtml:/usr/local/mysql/bin:/Users/aponte/Tools/ftagshtml-0.520/bin:/usr/local/ferret/bin > /Users/aponte/.miniconda3/envs/petsc/bin/python > /bin/sh > env \ > F77="/opt/local/bin/mpif90-openmpi-gcc49" F77FLAGS="-Os -Os -I/opt/local/include" \ > F90="/opt/local/bin/mpif90-openmpi-gcc49" F90FLAGS="-Os -Os -I/opt/local/include" \ > LDSHARED="/opt/local/bin/mpif90-openmpi-gcc49" \ > /Users/aponte/.miniconda3/envs/petsc/bin/python setup.py -q build_ext --inplace > Reading .f2py_f2cmap ... > Mapping "integer(kind=HANDLE_KIND)" to "long" > Mapping "integer(kind=handle_kind)" to "long" > Successfully applied user defined changes from .f2py_f2cmap > In file included from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1809:0, > from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18, > from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4, > from build/src.macosx-10.9-x86_64-3.6/./fortranobject.h:13, > from ./timestepper_advmodule.c:18: > /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] > #warning "Using deprecated NumPy API, disable it by " \ > ^ > In file included from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1809:0, > from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18, > from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4, > from build/src.macosx-10.9-x86_64-3.6/./fortranobject.h:13, > from build/src.macosx-10.9-x86_64-3.6/./fortranobject.c:2: > /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] > #warning "Using deprecated NumPy API, disable it by " \ > ^ > build/src.macosx-10.9-x86_64-3.6/./fortranobject.c: In function 'format_def': > build/src.macosx-10.9-x86_64-3.6/./fortranobject.c:138:18: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] > if (size < sizeof(notalloc)) { > ^ > ./timestepper_adv.F90:9.22: > > module timestepper_adv > 1 > Error: Unexpected MODULE statement at (1) > ./timestepper_adv.F90:11.11: > > use petsc > 1 > Error: Unexpected USE statement at (1) > ./timestepper_adv.F90:12.15: > > implicit none > 1 > Error: Unexpected IMPLICIT NONE statement at (1) > ./timestepper_adv.F90:14.15: > > type gridinfo > 1 > > ... > > Error: Symbol 'grd' at (1) has no IMPLICIT type > ./timestepper_adv.F90:59.18: > > type(gridinfo) grd > 1 > Error: Derived type 'gridinfo' at (1) is being used before it is defined > Fatal Error: Error count reached limit of 25. > error: Command "/opt/local/bin/mpif90-openmpi-gcc49 -Os -Os -I/opt/local/include -arch x86_64 -fPIC -O3 -funroll-loops -DF2PY_REPORT_ON_ARRAY_COPY=1 -I/opt/local/lib/petsc/include -I/Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/petsc4py/include -I. -Ibuild/src.macosx-10.9-x86_64-3.6/. -I/Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include -I/Users/aponte/.miniconda3/envs/petsc/include/python3.6m -c -c ./timestepper_adv.F90 -o build/temp.macosx-10.9-x86_64-3.6/timestepper_adv.o -Jbuild/temp.macosx-10.9-x86_64-3.6/ -Ibuild/temp.macosx-10.9-x86_64-3.6/" failed with exit status 1 > make: *** [timestepper_adv.so] Error 1 > > -- > Aur?lien Ponte > Tel: (+33) 2 98 22 40 73 > Fax: (+33) 2 98 22 44 96 > UMR 6523, IFREMER > ZI de la Pointe du Diable > CS 10070 > 29280 Plouzan? > From knepley at gmail.com Thu Jan 18 10:09:02 2018 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 18 Jan 2018 11:09:02 -0500 Subject: [petsc-users] I have a question for PETSc example In-Reply-To: References: <888DF7F8-1A65-4343-AEE8-E4387F9FA716@mcs.anl.gov> <3D59C605-93DD-4364-8C49-67E18CB96783@anl.gov> Message-ID: On Thu, Jan 18, 2018 at 6:10 AM, ??? (???????????) wrote: > Dear petsc-user. > > > Hi. I am trying several parts of example code, (SNES ex12.c : Poisson > Problem in 2d and 3d with simplicial finite element) > > > and I have several problems to understand. > > > I appologize to sent a long and boring email to you... > > > *1) About setting of geometry* > > I referred the "DMPlexCreateBoxMesh" function to make simplicial box mesh. > > I found that lower[] and upper[] are factors to set the size of the box > mesh. Is it right? > > Actually, I want to make a rectangular geometry with size of 3m x 1m, and > four boundary (two dirichlet and two neumann condition). > > If I wrong, please give me some tip to make a good geometry. > Yes, you can use DMPlexCreateBoxMesh() to make that. > > *2) About application boundary condition* > > I reffered the example "SNES ex69.c" to set a label and condition for the > four boundarys. > > First part of code is about label of boundary condition, > This is wrong. You should not have DMPlexMarkBoundaryFaces() in there. What did you want it to do? > and second part is about setting of boundary condition. > > However, due to second part, error occurs. > Always, always, always send the full error message. Otherwise, I am just guessing. > Is there any wrong argument? > If they are homogeneous Neumann conditions, there is no need to mark them. If they are inhomogeneous, you must specify a part of the weak form, as is done in SNES ex12 and ex77. Second, for the Dirichlet conditions, are you sure you only want to constrain the components that I am in ex69? What would help is to fully specify your problem, so that we know what you want to do. > Please give me some tip to applied four boundary conditions on the mesh. > > > *3) About input parameters of DSSetJacobian()* > > In DSSetJacobian (prob, f, g, g0, g1, g2, g3), I have a problem to > understand g0~g3 > > g0 : integrand for test function and basis function > > g1 : integrand for test function gradient and basis function > > g2 : integrand for test function and basis function gradient > > g3 : integrand for test function gradient and basis function gradient > > > I have studied the FEM before one month, but I am very confused by above > four terms. > There is a short explanation here: http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DT/PetscDSSetJacobian.html and here you can see the four terms which are being integrated to arrive at the element matrix: http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DT/PetscFEIntegrateJacobian.html > I should study this in my self, but I need to some tip or internet address > which include a helpful description about above terms. > > > (I think this issue is not good at this email, but I have been really > suffered by above term...) > > > > *4) About useful tool to visualize the result and geometry * > > > I also have searched the tool which can express the visualized result and > geometry. > I use Paraview from KitWare, which is free. I output HDF5, using the generic PETSc Viewer options detailed in the manual, -dm_view hdf5:lap.h5 -vec_view hdf5:lap.h5::append and then I automatically create an XDMF file lap.xmf, describing the HDF5, which can be read by Paraview, $PETSC_DIR/bin/petsc_gen_xdmf.py lap.h5 Thanks, Matt > Could you give me a tip to select the visualizing tool for better analysis? > > > > > Thanks you for reading my email. > > > Best regards, > > Seungjin Seo > > > > ------------------------------ > *?? ??:* Matthew Knepley > *?? ??:* 2018? 1? 16? ??? ?? 8:48:16 > *?? ??:* ??? (???????????) > *??:* Smith, Barry F.; petsc-users at mcs.anl.gov > *??:* Re: [petsc-users] I have a question for PETSc example > > On Mon, Jan 15, 2018 at 8:28 PM, ??? (???????????) > wrote: > > sorry, I mean that > > *My geometry is a rectangle*. > > That means, geometry is just rectangle box shape with four boundary. > > > My question is that, how can I set the four boundary condition > individually in example 12 of SNES? > > > You put a different marker on each side. The easiest way to do it is to > give the > > -dm_plex_separate_marker > > option. This is done in SNES ex69. > > Thanks, > > Matt > > > (I think that the boundary condition should be added in right side of the > weak form, where load vector f and test function term is included. However, > in example of petsc, boundary condition is applied as general form.) > > > Seungjin Seo > ------------------------------ > *?? ??:* Smith, Barry F. > *?? ??:* 2018? 1? 15? ??? ?? 11:06:18 > *?? ??:* ??? (???????????) > *??:* Smith, Barry F.; petsc-users at mcs.anl.gov > > *??:* Re: [petsc-users] I have a question for PETSc example > > > The term "rectangular mesh" usually refers to a "structured grid mesh" > so it is not clear if you truly need an unstructured grid. Take a look at > src/snes/examples/tutorials/ex19.c > > Barry > > > > On Jan 15, 2018, at 1:03 AM, ??? (???????????) > wrote: > > > > Dear PETSc-user > > > > I am Seungjin Seo, researcher of Korean Advanced Institute of Science > and Technology, South Korea. > > > > I am tring to understand the ex12.c in SNES examples, and I have a > problem... > > Could you give me some advise? Is there any good description or > materials for PetscFE and DMPlex? > > > > I will make a code with unstructured grid with PetscFE and SNES. My > geometry is a rectangle mesh (the size is not fixed so I think this is > unstructured grid). > > The equation is > > I will set boundry conditions as follow; > > 1. (Top-side boundary) Neumann boundary > > 2. (Bottom-side boundary) Neumann boundary > > 3. (Left-side boundary) Dirichlet boundary > > 4. (Right-side boundary) Dirichlet boundary > > > > In example ex12.c in SNES, load vector f is defiend as f_0, and neumann > boundary condition is defined as f_bd_0. f_1 means the test function > gradient term. The example set general type of neumann boundary condition > on f_bd_0, but I want to set each boundary condition individually. Could > you give me some advise? > > > > Thansk for reading my email. > > > > Best regard, > > Seungjin Seo > > > > > > > > ?? ??: Smith, Barry F. > > ?? ??: 2017? 12? 21? ??? ?? 3:46:42 > > ?? ??: ??? (???????????) > > ??: petsc-users at mcs.anl.gov > > ??: Re: [petsc-users] I have a question for PETSc example > > > > > > In deed you can, yes many PETSc examples only feature a single field > for simplicity, but multiple fields are fine. You simple provide your > FormFunction and FormJacobian to handle both degrees of freedom per > cell/node at the same time. You use SNES or TS depending on if your problem > is time dependent and then use -pc_type lu to start. Once you have the > physics correct come back and ask us about optimizing the linear solver > with preconditioners; but until you have the physics correct it is absurd > to waste time worrying about making the linear solver efficient. What you > want to do, many people do and is not a big deal. > > > > Barry > > > > > > > On Dec 21, 2017, at 12:27 AM, ??? (???????????) > wrote: > > > > > > Dear PETSc-user > > > > > > > > > I am Seungjin Seo, researcher of Korean Advanced Institute of Science > and Technology, South Korea. > > > > > > > > > > > > I am trying to solve thermal and fluid equations within a porous > structure. > > > The thermal equation includes a non-linear term of pressure and the > fluid equation has a boundary condition using temperature gradient. > > > Is there any way I can solve these two equations at the same time, > instead of solving temperature first with previous pressure distribution > and then pressure and update temperature? > > > Many examples solve only one PDF instead of coupling several physics. > > > Please recommend me which solver is appropriate for my case? > > > > > > Best regards, > > > Seungjin Seo > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 8003 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 109380 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 74067 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 24165 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 43510 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 134256 bytes Desc: not available URL: From dalcinl at gmail.com Thu Jan 18 10:13:03 2018 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 18 Jan 2018 19:13:03 +0300 Subject: [petsc-users] petsc4py, cython In-Reply-To: <183EC67F-4754-4146-A10C-841498BC9ABA@informatik.uni-kiel.de> References: <781ADC9E-82A2-4FEB-8D03-15F733D89354@informatik.uni-kiel.de> <183EC67F-4754-4146-A10C-841498BC9ABA@informatik.uni-kiel.de> Message-ID: In Cython code using C types from petsc4py, you should do: from petsc4py.PETSc cimport Vec, PetscVec from petsc4py.PETSc cimport Mat, PetscMat # etcetera now you can "cdef" either "PetscVec" (the low-level PETSc C type) and "Vec" (the higher-level petsc4py Python type) Or you can just from petsc4py.PETSc cimport * to get all definitions, or even from petsc4py cimport PETSc then use properly-namespaced definitions "PETSc.PetscVec" (C type) and "PETSc.Vec" (Python type). -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 0109 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 From dalcinl at gmail.com Thu Jan 18 12:59:13 2018 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 18 Jan 2018 21:59:13 +0300 Subject: [petsc-users] petsc4py f2py In-Reply-To: <4fa20faa-1110-b282-ee74-e19f13357818@ifremer.fr> References: <4fa20faa-1110-b282-ee74-e19f13357818@ifremer.fr> Message-ID: On 18 January 2018 at 18:56, Aurelien Ponte wrote: > The code is found here: > https://github.com/apatlpo/qgsolver/tree/fwrapping/qgsolver Your code is not valid Fortran, you cannot use brackets [] for indexing in lines 80-90, you have to use parentheses (). -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 0109 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 From dalcinl at gmail.com Thu Jan 18 13:01:03 2018 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 18 Jan 2018 22:01:03 +0300 Subject: [petsc-users] petsc4py f2py In-Reply-To: References: <4fa20faa-1110-b282-ee74-e19f13357818@ifremer.fr> Message-ID: Line 71 is likely wrong, too, you likely meant: do k=grd%zs,grd%ze On 18 January 2018 at 21:59, Lisandro Dalcin wrote: > On 18 January 2018 at 18:56, Aurelien Ponte wrote: >> The code is found here: >> https://github.com/apatlpo/qgsolver/tree/fwrapping/qgsolver > > Your code is not valid Fortran, you cannot use brackets [] for > indexing in lines 80-90, you have to use parentheses (). > > -- > Lisandro Dalcin > ============ > Research Scientist > Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) > Extreme Computing Research Center (ECRC) > King Abdullah University of Science and Technology (KAUST) > http://ecrc.kaust.edu.sa/ > > 4700 King Abdullah University of Science and Technology > al-Khawarizmi Bldg (Bldg 1), Office # 0109 > Thuwal 23955-6900, Kingdom of Saudi Arabia > http://www.kaust.edu.sa > > Office Phone: +966 12 808-0459 -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 0109 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 From aurelien.ponte at ifremer.fr Thu Jan 18 13:23:32 2018 From: aurelien.ponte at ifremer.fr (Aurelien Ponte) Date: Thu, 18 Jan 2018 20:23:32 +0100 Subject: [petsc-users] petsc4py f2py In-Reply-To: References: <4fa20faa-1110-b282-ee74-e19f13357818@ifremer.fr> Message-ID: Thanks, I corrected the two errors in the fortran code Lisandro spotted. I added the -E option in the makefile: ??????? F90="${FC}" F90FLAGS="${FC_FLAGS} -E" \ make now goes through and produce two files: timestepper_adv.cpython-36m-darwin.so.dSYM timestepper_adv.cpython-36m-darwin.so I have not tested these libraries yet but will do soon. Do you believe I am good to go? Finally, I have several warning that may be of some interest, if you have quick fixes, I am a taker ! /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] ?#warning "Using deprecated NumPy API, disable it by " \ ? ^ build/src.macosx-10.9-x86_64-3.6/./fortranobject.c: In function 'format_def': build/src.macosx-10.9-x86_64-3.6/./fortranobject.c:138:18: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] ???????? if (size < sizeof(notalloc)) { ????????????????? ^ ld: warning: ignoring file build/temp.macosx-10.9-x86_64-3.6/timestepper_adv.o, file was built for unsupported file format ( 0x23 0x20 0x31 0x20 0x22 0x2E 0x2F 0x74 0x69 0x6D 0x65 0x73 0x74 0x65 0x70 0x70 ) which is not the architecture being linked (x86_64): build/temp.macosx-10.9-x86_64-3.6/timestepper_adv.o Le 18/01/2018 ? 20:01, Lisandro Dalcin a ?crit?: > Line 71 is likely wrong, too, you likely meant: > > do k=grd%zs,grd%ze > > On 18 January 2018 at 21:59, Lisandro Dalcin wrote: >> On 18 January 2018 at 18:56, Aurelien Ponte wrote: >>> The code is found here: >>> https://github.com/apatlpo/qgsolver/tree/fwrapping/qgsolver >> Your code is not valid Fortran, you cannot use brackets [] for >> indexing in lines 80-90, you have to use parentheses (). >> >> -- >> Lisandro Dalcin >> ============ >> Research Scientist >> Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) >> Extreme Computing Research Center (ECRC) >> King Abdullah University of Science and Technology (KAUST) >> http://ecrc.kaust.edu.sa/ >> >> 4700 King Abdullah University of Science and Technology >> al-Khawarizmi Bldg (Bldg 1), Office # 0109 >> Thuwal 23955-6900, Kingdom of Saudi Arabia >> http://www.kaust.edu.sa >> >> Office Phone: +966 12 808-0459 > > -- Aur?lien Ponte Tel: (+33) 2 98 22 40 73 Fax: (+33) 2 98 22 44 96 UMR 6523, IFREMER ZI de la Pointe du Diable CS 10070 29280 Plouzan? From aurelien.ponte at ifremer.fr Thu Jan 18 13:34:13 2018 From: aurelien.ponte at ifremer.fr (Aurelien Ponte) Date: Thu, 18 Jan 2018 20:34:13 +0100 Subject: [petsc-users] petsc4py f2py In-Reply-To: References: <4fa20faa-1110-b282-ee74-e19f13357818@ifremer.fr> Message-ID: <35918a00-7d93-a6b1-542b-5cce4a547a78@ifremer.fr> Hum, after rereading Barry's email I feel I am confused about the point of adding the -E flag. So I am pasting the full log of the make in case it may be useful: (petsc) br146-244:qgsolver aponte$ make PYTHON=/Users/aponte/.miniconda3/envs/petsc/bin/python build /Users/aponte/.miniconda3/envs/petsc/bin:/opt/local/bin:/opt/local/sbin:/Users/aponte/.miniconda3/bin:/opt/local/bin:/opt/local/sbin:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin:/Library/TeX/texbin:/usr/local/bin:/Users/aponte/Installed/f90tohtml:/usr/local/mysql/bin:/Users/aponte/Tools/ftagshtml-0.520/bin:/usr/local/ferret/bin:/usr/local/bin:/Users/aponte/Installed/f90tohtml:/usr/local/mysql/bin:/Users/aponte/Tools/ftagshtml-0.520/bin:/usr/local/ferret/bin /Users/aponte/.miniconda3/envs/petsc/bin/python /bin/sh env \ ??????? F77="/opt/local/bin/mpif90-openmpi-gcc49" F77FLAGS="-Os -Os? -I/opt/local/include" \ ??????? F90="/opt/local/bin/mpif90-openmpi-gcc49" F90FLAGS="-Os -Os? -I/opt/local/include -E" \ ??????? LDSHARED="/opt/local/bin/mpif90-openmpi-gcc49" \ ??? /Users/aponte/.miniconda3/envs/petsc/bin/python setup.py -q build_ext --inplace Reading .f2py_f2cmap ... ??? Mapping "integer(kind=HANDLE_KIND)" to "long" ??? Mapping "integer(kind=handle_kind)" to "long" Successfully applied user defined changes from .f2py_f2cmap In file included from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1809:0, ???????????????? from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18, ???????????????? from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4, ???????????????? from build/src.macosx-10.9-x86_64-3.6/./fortranobject.h:13, ???????????????? from ./timestepper_advmodule.c:18: /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] ?#warning "Using deprecated NumPy API, disable it by " \ ? ^ In file included from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1809:0, ???????????????? from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18, ???????????????? from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4, ???????????????? from build/src.macosx-10.9-x86_64-3.6/./fortranobject.h:13, ???????????????? from build/src.macosx-10.9-x86_64-3.6/./fortranobject.c:2: /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] ?#warning "Using deprecated NumPy API, disable it by " \ ? ^ build/src.macosx-10.9-x86_64-3.6/./fortranobject.c: In function 'format_def': build/src.macosx-10.9-x86_64-3.6/./fortranobject.c:138:18: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] ???????? if (size < sizeof(notalloc)) { ????????????????? ^ ld: warning: ignoring file build/temp.macosx-10.9-x86_64-3.6/timestepper_adv.o, file was built for unsupported file format ( 0x23 0x20 0x31 0x20 0x22 0x2E 0x2F 0x74 0x69 0x6D 0x65 0x73 0x74 0x65 0x70 0x70 ) which is not the architecture being linked (x86_64): build/temp.macosx-10.9-x86_64-3.6/timestepper_adv.o /bin/rm -f -r build timestepper_advmodule.c (petsc) br146-244:qgsolver aponte$ Le 18/01/2018 ? 20:23, Aurelien Ponte a ?crit?: > Thanks, > > I corrected the two errors in the fortran code Lisandro spotted. > > I added the -E option in the makefile: > ??????? F90="${FC}" F90FLAGS="${FC_FLAGS} -E" \ > > make now goes through and produce two files: > timestepper_adv.cpython-36m-darwin.so.dSYM > timestepper_adv.cpython-36m-darwin.so > > I have not tested these libraries yet but will do soon. > Do you believe I am good to go? > > Finally, I have several warning that may be of some interest, if you > have quick fixes, I am a taker ! > > /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: > warning: #warning "Using deprecated NumPy API, disable it by " > "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] > ?#warning "Using deprecated NumPy API, disable it by " \ > ? ^ > build/src.macosx-10.9-x86_64-3.6/./fortranobject.c: In function > 'format_def': > build/src.macosx-10.9-x86_64-3.6/./fortranobject.c:138:18: warning: > comparison between signed and unsigned integer expressions > [-Wsign-compare] > ???????? if (size < sizeof(notalloc)) { > ????????????????? ^ > ld: warning: ignoring file > build/temp.macosx-10.9-x86_64-3.6/timestepper_adv.o, file was built > for unsupported file format ( 0x23 0x20 0x31 0x20 0x22 0x2E 0x2F 0x74 > 0x69 0x6D 0x65 0x73 0x74 0x65 0x70 0x70 ) which is not the > architecture being linked (x86_64): > build/temp.macosx-10.9-x86_64-3.6/timestepper_adv.o > > Le 18/01/2018 ? 20:01, Lisandro Dalcin a ?crit?: >> Line 71 is likely wrong, too, you likely meant: >> >> do k=grd%zs,grd%ze >> >> On 18 January 2018 at 21:59, Lisandro Dalcin wrote: >>> On 18 January 2018 at 18:56, Aurelien Ponte >>> wrote: >>>> The code is found here: >>>> https://github.com/apatlpo/qgsolver/tree/fwrapping/qgsolver >>> Your code is not valid Fortran, you cannot use brackets [] for >>> indexing in lines 80-90, you have to use parentheses (). >>> >>> -- >>> Lisandro Dalcin >>> ============ >>> Research Scientist >>> Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) >>> Extreme Computing Research Center (ECRC) >>> King Abdullah University of Science and Technology (KAUST) >>> http://ecrc.kaust.edu.sa/ >>> >>> 4700 King Abdullah University of Science and Technology >>> al-Khawarizmi Bldg (Bldg 1), Office # 0109 >>> Thuwal 23955-6900, Kingdom of Saudi Arabia >>> http://www.kaust.edu.sa >>> >>> Office Phone: +966 12 808-0459 >> >> > -- Aur?lien Ponte Tel: (+33) 2 98 22 40 73 Fax: (+33) 2 98 22 44 96 UMR 6523, IFREMER ZI de la Pointe du Diable CS 10070 29280 Plouzan? From rlmackie862 at gmail.com Thu Jan 18 16:11:15 2018 From: rlmackie862 at gmail.com (Randall Mackie) Date: Thu, 18 Jan 2018 14:11:15 -0800 Subject: [petsc-users] valgrind errors on vecscattercreatetoAll Message-ID: The very simple attached program throws lots of valgrind errors. I am using pets 3.8.3, compiled with the following options: ./configure \ --with-debugging=1 \ --with-fortran=1 \ --download-mpich=../mpich-3.3a2.tar.gz \ The makefile, run file, and valgrind output are also attached. Randy M. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: cmd_test Type: application/octet-stream Size: 87 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: makefile Type: application/octet-stream Size: 370 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test.F90 Type: application/octet-stream Size: 443 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: valgrind.txt URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Jan 18 17:00:08 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 18 Jan 2018 23:00:08 +0000 Subject: [petsc-users] valgrind errors on vecscattercreatetoAll In-Reply-To: References: Message-ID: <849C2523-6365-413B-AB28-5502A12C650C@anl.gov> This is a known issue without a perfect solution. The problem is that by default PETSc fortran types, like type(tKSP) are not initialized but we want to check if they are a special null character, for example PETSC_NULL_KSP. So we could initialize them but then they cannot be used in common blocks (since default initialized Fortran90 types cannot be used in common blocks; some strange feature of Fortran 90) In next you can configure with the option -with-fortran-type-initialize=1 and the valgrind errors will go away, but then there cannot be Fortran datatypes into common blocks. Barry It may be we should make -with-fortran-type-initialize=1 the default but this will break some PETSc Fortran examples that use common blocks. We should rewrite those examples not to use common blocks but who volunteers? > On Jan 18, 2018, at 4:11 PM, Randall Mackie wrote: > > The very simple attached program throws lots of valgrind errors. > I am using pets 3.8.3, compiled with the following options: > > > ./configure \ > --with-debugging=1 \ > --with-fortran=1 \ > --download-mpich=../mpich-3.3a2.tar.gz \ > > > The makefile, run file, and valgrind output are also attached. > > > Randy M. > > > > > From bsmith at mcs.anl.gov Thu Jan 18 17:08:17 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 18 Jan 2018 23:08:17 +0000 Subject: [petsc-users] valgrind errors on vecscattercreatetoAll In-Reply-To: <849C2523-6365-413B-AB28-5502A12C650C@anl.gov> References: <849C2523-6365-413B-AB28-5502A12C650C@anl.gov> Message-ID: <61BC59F3-E250-4AA4-BCB0-905F9C57E681@anl.gov> Sorry this is actually in master already > On Jan 18, 2018, at 5:00 PM, Smith, Barry F. wrote: > > > This is a known issue without a perfect solution. > > The problem is that by default PETSc fortran types, like type(tKSP) are not initialized but we want to check if they are a special null character, for example PETSC_NULL_KSP. So we could initialize them but then they cannot be used in common blocks (since default initialized Fortran90 types cannot be used in common blocks; some strange feature of Fortran 90) > > In next you can configure with the option -with-fortran-type-initialize=1 and the valgrind errors will go away, but then there cannot be Fortran datatypes into common blocks. > > Barry > > It may be we should make -with-fortran-type-initialize=1 the default but this will break some PETSc Fortran examples that use common blocks. We should rewrite those examples not to use common blocks but who volunteers? > > > > > > >> On Jan 18, 2018, at 4:11 PM, Randall Mackie wrote: >> >> The very simple attached program throws lots of valgrind errors. >> I am using pets 3.8.3, compiled with the following options: >> >> >> ./configure \ >> --with-debugging=1 \ >> --with-fortran=1 \ >> --download-mpich=../mpich-3.3a2.tar.gz \ >> >> >> The makefile, run file, and valgrind output are also attached. >> >> >> Randy M. >> >> >> >> >> > From mistloin at unist.ac.kr Thu Jan 18 21:46:38 2018 From: mistloin at unist.ac.kr (=?ks_c_5601-1987?B?vK29wsH4ICix4rDox9ew+LnXv/jA2rfCsPjH0LrOKQ==?=) Date: Fri, 19 Jan 2018 03:46:38 +0000 Subject: [petsc-users] I have a question for PETSc example In-Reply-To: References: <888DF7F8-1A65-4343-AEE8-E4387F9FA716@mcs.anl.gov> <3D59C605-93DD-4364-8C49-67E18CB96783@anl.gov> , Message-ID: Dear petsc-user. Thanks for your respond. 2) About application boundary condition 2-1) About my geometry The reason why I use DMPlexMarkBoundaryFaces() is to mark the boundary faces where dirichlet condition is applied. I accepted your advise, so there is no DMPlexMarkBoundaryFaces() in my code. I want to explain my problem with my geometry. I want to solve the heat transfer equation (only conduction with internal heat source Q) with four boundary condition. Governing equation is [cid:8a4a5d6c-1ce7-41da-8085-a846856847f4] Left and Top : Dirichlet condition (T=T1, T=T2) (T1 < T, T2 < T) Right and Bottom : Neumann condition (grad(T)=0 or -k*grad(T)=q) Heat is supplied from bottom boundary and generated in internal region, and it is transferred to left and top boundary. Internal heat source is function of temperature, so the problem is non-linear problem. [cid:5ba5a6d9-49e2-4e6b-9f27-5453037d2c1e] 2-2) About error message The error message which I mentioned in previous mail is that [cid:de6e76da-0661-4e8a-92f2-aec7512ea42b] 2-3) About inhomogeneous neumann boundary condition You kindly answer my question; "If they are inhomogeneous, you must specify a part of the weak form, as is done in SNES ex12 and ex77." Is that means I should specify terms "f0_bd_u" and "f1_bd_u" in weak form? (they are integrated along boundary line/surface in weak form) 2-4) About dirichlet boundary condition Simplicial finite element examples (ex12, ex77, ex56) set the exact function ( = exact solution, for example, u=x^2+y^2 in ex12) as dirichlet condition. Is the condition menas that we set the exact "u" values on the boundary (labeled as "marker")? Thanks for reading my email. Best regard, Seungjin Seo ________________________________ ?? ??: Matthew Knepley ?? ??: 2018? 1? 19? ??? ?? 1:09:02 ?? ??: ??? (???????????) ??: Smith, Barry F.; petsc-users at mcs.anl.gov ??: Re: [petsc-users] I have a question for PETSc example On Thu, Jan 18, 2018 at 6:10 AM, ??? (???????????) > wrote: Dear petsc-user. Hi. I am trying several parts of example code, (SNES ex12.c : Poisson Problem in 2d and 3d with simplicial finite element) and I have several problems to understand. I appologize to sent a long and boring email to you... 1) About setting of geometry [cid:009d718e-c50b-4bd9-8d2c-714321b9fc46] [cid:7e0f732c-ecad-4ea7-a3f3-8d6a09bc5cc9] [cid:40e3a7a8-4191-4693-bc97-84a550d5152f] I referred the "DMPlexCreateBoxMesh" function to make simplicial box mesh. I found that lower[] and upper[] are factors to set the size of the box mesh. Is it right? Actually, I want to make a rectangular geometry with size of 3m x 1m, and four boundary (two dirichlet and two neumann condition). If I wrong, please give me some tip to make a good geometry. Yes, you can use DMPlexCreateBoxMesh() to make that. 2) About application boundary condition [cid:c6eed790-070c-4e66-8dbb-8b461fb06f10] [cid:37a6abda-13a5-45bd-b751-2d193f784071] I reffered the example "SNES ex69.c" to set a label and condition for the four boundarys. First part of code is about label of boundary condition, This is wrong. You should not have DMPlexMarkBoundaryFaces() in there. What did you want it to do? and second part is about setting of boundary condition. However, due to second part, error occurs. Always, always, always send the full error message. Otherwise, I am just guessing. Is there any wrong argument? If they are homogeneous Neumann conditions, there is no need to mark them. If they are inhomogeneous, you must specify a part of the weak form, as is done in SNES ex12 and ex77. Second, for the Dirichlet conditions, are you sure you only want to constrain the components that I am in ex69? What would help is to fully specify your problem, so that we know what you want to do. Please give me some tip to applied four boundary conditions on the mesh. 3) About input parameters of DSSetJacobian() In DSSetJacobian (prob, f, g, g0, g1, g2, g3), I have a problem to understand g0~g3 g0 : integrand for test function and basis function g1 : integrand for test function gradient and basis function g2 : integrand for test function and basis function gradient g3 : integrand for test function gradient and basis function gradient I have studied the FEM before one month, but I am very confused by above four terms. There is a short explanation here: http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DT/PetscDSSetJacobian.html and here you can see the four terms which are being integrated to arrive at the element matrix: http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DT/PetscFEIntegrateJacobian.html I should study this in my self, but I need to some tip or internet address which include a helpful description about above terms. (I think this issue is not good at this email, but I have been really suffered by above term...) 4) About useful tool to visualize the result and geometry [cid:80900b60-6b68-4c72-8f3e-df0d475795d7] I also have searched the tool which can express the visualized result and geometry. I use Paraview from KitWare, which is free. I output HDF5, using the generic PETSc Viewer options detailed in the manual, -dm_view hdf5:lap.h5 -vec_view hdf5:lap.h5::append and then I automatically create an XDMF file lap.xmf, describing the HDF5, which can be read by Paraview, $PETSC_DIR/bin/petsc_gen_xdmf.py lap.h5 Thanks, Matt Could you give me a tip to select the visualizing tool for better analysis? Thanks you for reading my email. Best regards, Seungjin Seo ________________________________ ?? ??: Matthew Knepley > ?? ??: 2018? 1? 16? ??? ?? 8:48:16 ?? ??: ??? (???????????) ??: Smith, Barry F.; petsc-users at mcs.anl.gov ??: Re: [petsc-users] I have a question for PETSc example On Mon, Jan 15, 2018 at 8:28 PM, ??? (???????????) > wrote: sorry, I mean that My geometry is a rectangle. That means, geometry is just rectangle box shape with four boundary. My question is that, how can I set the four boundary condition individually in example 12 of SNES? You put a different marker on each side. The easiest way to do it is to give the -dm_plex_separate_marker option. This is done in SNES ex69. Thanks, Matt (I think that the boundary condition should be added in right side of the weak form, where load vector f and test function term is included. However, in example of petsc, boundary condition is applied as general form.) Seungjin Seo ________________________________ ?? ??: Smith, Barry F. > ?? ??: 2018? 1? 15? ??? ?? 11:06:18 ?? ??: ??? (???????????) ??: Smith, Barry F.; petsc-users at mcs.anl.gov ??: Re: [petsc-users] I have a question for PETSc example The term "rectangular mesh" usually refers to a "structured grid mesh" so it is not clear if you truly need an unstructured grid. Take a look at src/snes/examples/tutorials/ex19.c Barry > On Jan 15, 2018, at 1:03 AM, ??? (???????????) > wrote: > > Dear PETSc-user > > I am Seungjin Seo, researcher of Korean Advanced Institute of Science and Technology, South Korea. > > I am tring to understand the ex12.c in SNES examples, and I have a problem... > Could you give me some advise? Is there any good description or materials for PetscFE and DMPlex? > > I will make a code with unstructured grid with PetscFE and SNES. My geometry is a rectangle mesh (the size is not fixed so I think this is unstructured grid). > The equation is > I will set boundry conditions as follow; > 1. (Top-side boundary) Neumann boundary > 2. (Bottom-side boundary) Neumann boundary > 3. (Left-side boundary) Dirichlet boundary > 4. (Right-side boundary) Dirichlet boundary > > In example ex12.c in SNES, load vector f is defiend as f_0, and neumann boundary condition is defined as f_bd_0. f_1 means the test function gradient term. The example set general type of neumann boundary condition on f_bd_0, but I want to set each boundary condition individually. Could you give me some advise? > > Thansk for reading my email. > > Best regard, > Seungjin Seo > > > > ?? ??: Smith, Barry F. > > ?? ??: 2017? 12? 21? ??? ?? 3:46:42 > ?? ??: ??? (???????????) > ??: petsc-users at mcs.anl.gov > ??: Re: [petsc-users] I have a question for PETSc example > > > In deed you can, yes many PETSc examples only feature a single field for simplicity, but multiple fields are fine. You simple provide your FormFunction and FormJacobian to handle both degrees of freedom per cell/node at the same time. You use SNES or TS depending on if your problem is time dependent and then use -pc_type lu to start. Once you have the physics correct come back and ask us about optimizing the linear solver with preconditioners; but until you have the physics correct it is absurd to waste time worrying about making the linear solver efficient. What you want to do, many people do and is not a big deal. > > Barry > > > > On Dec 21, 2017, at 12:27 AM, ??? (???????????) > wrote: > > > > Dear PETSc-user > > > > > > I am Seungjin Seo, researcher of Korean Advanced Institute of Science and Technology, South Korea. > > > > > > > > I am trying to solve thermal and fluid equations within a porous structure. > > The thermal equation includes a non-linear term of pressure and the fluid equation has a boundary condition using temperature gradient. > > Is there any way I can solve these two equations at the same time, instead of solving temperature first with previous pressure distribution and then pressure and update temperature? > > Many examples solve only one PDF instead of coupling several physics. > > Please recommend me which solver is appropriate for my case? > > > > Best regards, > > Seungjin Seo -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 8003 bytes Desc: pastedImage.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 109380 bytes Desc: pastedImage.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 74067 bytes Desc: pastedImage.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 24165 bytes Desc: pastedImage.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 43510 bytes Desc: pastedImage.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 134256 bytes Desc: pastedImage.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 76848 bytes Desc: pastedImage.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 1135 bytes Desc: pastedImage.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 486428 bytes Desc: pastedImage.png URL: From aurelien.ponte at ifremer.fr Thu Jan 18 22:17:37 2018 From: aurelien.ponte at ifremer.fr (Aurelien Ponte) Date: Fri, 19 Jan 2018 05:17:37 +0100 Subject: [petsc-users] petsc4py f2py In-Reply-To: <35918a00-7d93-a6b1-542b-5cce4a547a78@ifremer.fr> References: <4fa20faa-1110-b282-ee74-e19f13357818@ifremer.fr> <35918a00-7d93-a6b1-542b-5cce4a547a78@ifremer.fr> Message-ID: Ok, I believe I made some progress. I found one more issue: PETSC_DIR was pointing toward an old macport install of petsc This is fixed now and here is what I have (without -E): (petsc) barracuda:qgsolver aponte$ make PYTHON=~/.miniconda3/envs/petsc/bin/python PETSC_DIR=~/.miniconda3/envs/petsc build env \ ??? F77="" F77FLAGS="" \ ??? F90="" F90FLAGS="" \ ??? LDSHARED="" \ ??? /Users/aponte/.miniconda3/envs/petsc/bin/python setup.py -q build_ext --inplace Reading .f2py_f2cmap ... ??? Mapping "integer(kind=HANDLE_KIND)" to "long" ??? Mapping "integer(kind=handle_kind)" to "long" Successfully applied user defined changes from .f2py_f2cmap In file included from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1809:0, ???????????????? from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18, ???????????????? from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4, ???????????????? from build/src.macosx-10.9-x86_64-3.6/./fortranobject.h:13, ???????????????? from ./timestepper_advmodule.c:18: /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] ?#warning "Using deprecated NumPy API, disable it by " \ ? ^ In file included from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1809:0, ???????????????? from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18, ???????????????? from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4, ???????????????? from build/src.macosx-10.9-x86_64-3.6/./fortranobject.h:13, ???????????????? from build/src.macosx-10.9-x86_64-3.6/./fortranobject.c:2: /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] ?#warning "Using deprecated NumPy API, disable it by " \ ? ^ build/src.macosx-10.9-x86_64-3.6/./fortranobject.c: In function 'format_def': build/src.macosx-10.9-x86_64-3.6/./fortranobject.c:138:18: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] ???????? if (size < sizeof(notalloc)) { ????????????????? ^ ./timestepper_adv.F90:11.6: ? use petsc ????? 1 Fatal Error: Can't open module file 'petsc.mod' for reading at (1): No such file or directory ./timestepper_adv.F90:11.6: ? use petsc ????? 1 Fatal Error: Can't open module file 'petsc.mod' for reading at (1): No such file or directory error: Command "/opt/local/bin/gfortran -Wall -g -fno-second-underscore -DF2PY_REPORT_ON_ARRAY_COPY=1 -I/Users/aponte/.miniconda3/envs/petsc/include -I/Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/petsc4py/include -I. -Ibuild/src.macosx-10.9-x86_64-3.6/. -I/Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include -I/Users/aponte/.miniconda3/envs/petsc/include/python3.6m -c -c ./timestepper_adv.F90 -o build/temp.macosx-10.9-x86_64-3.6/timestepper_adv.o -Jbuild/temp.macosx-10.9-x86_64-3.6/ -Ibuild/temp.macosx-10.9-x86_64-3.6/" failed with exit status 1 -- Aur?lien Ponte Tel: (+33) 2 98 22 40 73 Fax: (+33) 2 98 22 44 96 UMR 6523, IFREMER ZI de la Pointe du Diable CS 10070 29280 Plouzan? From knepley at gmail.com Thu Jan 18 22:37:38 2018 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 18 Jan 2018 23:37:38 -0500 Subject: [petsc-users] I have a question for PETSc example In-Reply-To: References: <888DF7F8-1A65-4343-AEE8-E4387F9FA716@mcs.anl.gov> <3D59C605-93DD-4364-8C49-67E18CB96783@anl.gov> Message-ID: On Thu, Jan 18, 2018 at 10:46 PM, ??? (???????????) wrote: > Dear petsc-user. > > > Thanks for your respond. > > > *2) About application boundary condition* > *2-1) About my geometry* > > The reason why I use DMPlexMarkBoundaryFaces() is to mark the boundary > faces where dirichlet condition is applied. > > I accepted your advise, so there is no DMPlexMarkBoundaryFaces() in > my code. > > > > I want to explain my problem with my geometry. I want to solve the heat > transfer equation (only conduction with internal heat source Q) with four > boundary condition. > > Governing equation is > > Left and Top : Dirichlet condition (T=T1, T=T2) (T1 < T, T2 < T) > > Right and Bottom : Neumann condition (grad(T)=0 or -k*grad(T)=q) > > Heat is supplied from bottom boundary and generated in internal region, > and it is transferred to left and top boundary. > > Internal heat source is function of temperature, so the problem is > non-linear problem. > I recommend using a constant source to start, so you can use MMS and look at the actual error. This is invaluable for debugging, and you will see that all my examples do it. > *2-2) About error message* > > The error message which I mentioned in previous mail is that > You are giving the name of a label that does not exist I think. You can check what labels are defined (and how many points for each value) using -dm_view. > > *2-3) About inhomogeneous neumann boundary condition * > > You kindly answer my question; "If they are inhomogeneous, you must > specify a part of the weak form, as is done in SNES ex12 and ex77." > > Is that means I should specify terms "f0_bd_u" and "f1_bd_u" in weak > form? (they are integrated along boundary line/surface in weak form) > Yes. > *2-4) About dirichlet boundary condition * > > Simplicial finite element examples (ex12, ex77, ex56) set the exact > function ( = exact solution, for example, u=x^2+y^2 in ex12) as dirichlet > condition. > > Is the condition menas that we set the exact "u" values on the boundary > (labeled as "marker")? > The Dirichlet conditions is always the exact value on the boundary. That is what it means. Thanks, Matt > Thanks for reading my email. > > > Best regard, > > Seungjin Seo > ------------------------------ > *?? ??:* Matthew Knepley > *?? ??:* 2018? 1? 19? ??? ?? 1:09:02 > *?? ??:* ??? (???????????) > *??:* Smith, Barry F.; petsc-users at mcs.anl.gov > *??:* Re: [petsc-users] I have a question for PETSc example > > On Thu, Jan 18, 2018 at 6:10 AM, ??? (???????????) > wrote: > > Dear petsc-user. > > > Hi. I am trying several parts of example code, (SNES ex12.c : Poisson > Problem in 2d and 3d with simplicial finite element) > > > and I have several problems to understand. > > > I appologize to sent a long and boring email to you... > > > *1) About setting of geometry* > > I referred the "DMPlexCreateBoxMesh" function to make simplicial box mesh. > > I found that lower[] and upper[] are factors to set the size of the box > mesh. Is it right? > > Actually, I want to make a rectangular geometry with size of 3m x 1m, and > four boundary (two dirichlet and two neumann condition). > > If I wrong, please give me some tip to make a good geometry. > > > Yes, you can use DMPlexCreateBoxMesh() to make that. > > > *2) About application boundary condition* > > I reffered the example "SNES ex69.c" to set a label and condition for the > four boundarys. > > First part of code is about label of boundary condition, > > > This is wrong. You should not have DMPlexMarkBoundaryFaces() in there. > What did you want it to do? > > > and second part is about setting of boundary condition. > > However, due to second part, error occurs. > > > Always, always, always send the full error message. Otherwise, I am just > guessing. > > > Is there any wrong argument? > > > If they are homogeneous Neumann conditions, there is no need to mark them. > If they are inhomogeneous, you > must specify a part of the weak form, as is done in SNES ex12 and ex77. > > Second, for the Dirichlet conditions, are you sure you only want to > constrain the components that I am in ex69? > What would help is to fully specify your problem, so that we know what you > want to do. > > > Please give me some tip to applied four boundary conditions on the mesh. > > > *3) About input parameters of DSSetJacobian()* > > In DSSetJacobian (prob, f, g, g0, g1, g2, g3), I have a problem to > understand g0~g3 > > g0 : integrand for test function and basis function > > g1 : integrand for test function gradient and basis function > > g2 : integrand for test function and basis function gradient > > g3 : integrand for test function gradient and basis function gradient > > > I have studied the FEM before one month, but I am very confused by above > four terms. > > > There is a short explanation here: > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DT/ > PetscDSSetJacobian.html > > and here you can see the four terms which are being integrated to arrive > at the element matrix: > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DT/ > PetscFEIntegrateJacobian.html > > > I should study this in my self, but I need to some tip or internet address > which include a helpful description about above terms. > > > (I think this issue is not good at this email, but I have been really > suffered by above term...) > > > > *4) About useful tool to visualize the result and geometry * > > > I also have searched the tool which can express the visualized result and > geometry. > > > I use Paraview from KitWare, which is free. I output HDF5, using the > generic PETSc Viewer options detailed in the manual, > > -dm_view hdf5:lap.h5 -vec_view hdf5:lap.h5::append > > and then I automatically create an XDMF file lap.xmf, describing the HDF5, > which can be read by Paraview, > > $PETSC_DIR/bin/petsc_gen_xdmf.py lap.h5 > > Thanks, > > Matt > > > Could you give me a tip to select the visualizing tool for better analysis? > > > > > Thanks you for reading my email. > > > Best regards, > > Seungjin Seo > > > > ------------------------------ > *?? ??:* Matthew Knepley > *?? ??:* 2018? 1? 16? ??? ?? 8:48:16 > *?? ??:* ??? (???????????) > *??:* Smith, Barry F.; petsc-users at mcs.anl.gov > *??:* Re: [petsc-users] I have a question for PETSc example > > On Mon, Jan 15, 2018 at 8:28 PM, ??? (???????????) > wrote: > > sorry, I mean that > > *My geometry is a rectangle*. > > That means, geometry is just rectangle box shape with four boundary. > > > My question is that, how can I set the four boundary condition > individually in example 12 of SNES? > > > You put a different marker on each side. The easiest way to do it is to > give the > > -dm_plex_separate_marker > > option. This is done in SNES ex69. > > Thanks, > > Matt > > > (I think that the boundary condition should be added in right side of the > weak form, where load vector f and test function term is included. However, > in example of petsc, boundary condition is applied as general form.) > > > Seungjin Seo > ------------------------------ > *?? ??:* Smith, Barry F. > *?? ??:* 2018? 1? 15? ??? ?? 11:06:18 > *?? ??:* ??? (???????????) > *??:* Smith, Barry F.; petsc-users at mcs.anl.gov > > *??:* Re: [petsc-users] I have a question for PETSc example > > > The term "rectangular mesh" usually refers to a "structured grid mesh" > so it is not clear if you truly need an unstructured grid. Take a look at > src/snes/examples/tutorials/ex19.c > > Barry > > > > On Jan 15, 2018, at 1:03 AM, ??? (???????????) > wrote: > > > > Dear PETSc-user > > > > I am Seungjin Seo, researcher of Korean Advanced Institute of Science > and Technology, South Korea. > > > > I am tring to understand the ex12.c in SNES examples, and I have a > problem... > > Could you give me some advise? Is there any good description or > materials for PetscFE and DMPlex? > > > > I will make a code with unstructured grid with PetscFE and SNES. My > geometry is a rectangle mesh (the size is not fixed so I think this is > unstructured grid). > > The equation is > > I will set boundry conditions as follow; > > 1. (Top-side boundary) Neumann boundary > > 2. (Bottom-side boundary) Neumann boundary > > 3. (Left-side boundary) Dirichlet boundary > > 4. (Right-side boundary) Dirichlet boundary > > > > In example ex12.c in SNES, load vector f is defiend as f_0, and neumann > boundary condition is defined as f_bd_0. f_1 means the test function > gradient term. The example set general type of neumann boundary condition > on f_bd_0, but I want to set each boundary condition individually. Could > you give me some advise? > > > > Thansk for reading my email. > > > > Best regard, > > Seungjin Seo > > > > > > > > ?? ??: Smith, Barry F. > > ?? ??: 2017? 12? 21? ??? ?? 3:46:42 > > ?? ??: ??? (???????????) > > ??: petsc-users at mcs.anl.gov > > ??: Re: [petsc-users] I have a question for PETSc example > > > > > > In deed you can, yes many PETSc examples only feature a single field > for simplicity, but multiple fields are fine. You simple provide your > FormFunction and FormJacobian to handle both degrees of freedom per > cell/node at the same time. You use SNES or TS depending on if your problem > is time dependent and then use -pc_type lu to start. Once you have the > physics correct come back and ask us about optimizing the linear solver > with preconditioners; but until you have the physics correct it is absurd > to waste time worrying about making the linear solver efficient. What you > want to do, many people do and is not a big deal. > > > > Barry > > > > > > > On Dec 21, 2017, at 12:27 AM, ??? (???????????) > wrote: > > > > > > Dear PETSc-user > > > > > > > > > I am Seungjin Seo, researcher of Korean Advanced Institute of Science > and Technology, South Korea. > > > > > > > > > > > > I am trying to solve thermal and fluid equations within a porous > structure. > > > The thermal equation includes a non-linear term of pressure and the > fluid equation has a boundary condition using temperature gradient. > > > Is there any way I can solve these two equations at the same time, > instead of solving temperature first with previous pressure distribution > and then pressure and update temperature? > > > Many examples solve only one PDF instead of coupling several physics. > > > Please recommend me which solver is appropriate for my case? > > > > > > Best regards, > > > Seungjin Seo > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 43510 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 74067 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 8003 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 1135 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 76848 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 109380 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 134256 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 486428 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 24165 bytes Desc: not available URL: From bsmith at mcs.anl.gov Thu Jan 18 22:43:24 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 19 Jan 2018 04:43:24 +0000 Subject: [petsc-users] petsc4py f2py In-Reply-To: References: <4fa20faa-1110-b282-ee74-e19f13357818@ifremer.fr> <35918a00-7d93-a6b1-542b-5cce4a547a78@ifremer.fr> Message-ID: Fatal Error: Can't open module file 'petsc.mod' for reading at (1): No such file or directory error: Command "/opt/local/bin/gfortran -Wall -g -fno-second-underscore -DF2PY_REPORT_ON_ARRAY_COPY=1 -I/Users/aponte/.miniconda3/envs/petsc/include -I/Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/petsc4py/include -I. -Ibuild/src.macosx-10.9-x86_64-3.6/. -I/Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include -I/Users/aponte/.miniconda3/envs/petsc/include/python3.6m -c -c ./timestepper_adv.F90 -o build/temp.macosx-10.9-x86_64-3.6/timestepper_adv.o -Jbuild/temp.macosx-10.9-x86_64-3.6/ -Ibuild/temp.macosx-10.9-x86_64-3.6/" Normally the module files like petsc.mod would be in the $PETSC_DIR/$PETSC_ARCH/include so in your case I'm guessing /Users/aponte/.miniconda3/envs/petsc/include so first do and ls and see if there are any *.mod files in that directory? Barry > On Jan 18, 2018, at 10:17 PM, Aurelien Ponte wrote: > > Ok, I believe I made some progress. > I found one more issue: PETSC_DIR was pointing toward an old macport install of petsc > This is fixed now and here is what I have (without -E): > > (petsc) barracuda:qgsolver aponte$ make PYTHON=~/.miniconda3/envs/petsc/bin/python PETSC_DIR=~/.miniconda3/envs/petsc build > env \ > F77="" F77FLAGS="" \ > F90="" F90FLAGS="" \ > LDSHARED="" \ > /Users/aponte/.miniconda3/envs/petsc/bin/python setup.py -q build_ext --inplace > Reading .f2py_f2cmap ... > Mapping "integer(kind=HANDLE_KIND)" to "long" > Mapping "integer(kind=handle_kind)" to "long" > Successfully applied user defined changes from .f2py_f2cmap > In file included from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1809:0, > from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18, > from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4, > from build/src.macosx-10.9-x86_64-3.6/./fortranobject.h:13, > from ./timestepper_advmodule.c:18: > /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] > #warning "Using deprecated NumPy API, disable it by " \ > ^ > In file included from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1809:0, > from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18, > from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4, > from build/src.macosx-10.9-x86_64-3.6/./fortranobject.h:13, > from build/src.macosx-10.9-x86_64-3.6/./fortranobject.c:2: > /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] > #warning "Using deprecated NumPy API, disable it by " \ > ^ > build/src.macosx-10.9-x86_64-3.6/./fortranobject.c: In function 'format_def': > build/src.macosx-10.9-x86_64-3.6/./fortranobject.c:138:18: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] > if (size < sizeof(notalloc)) { > ^ > ./timestepper_adv.F90:11.6: > > use petsc > 1 > Fatal Error: Can't open module file 'petsc.mod' for reading at (1): No such file or directory > ./timestepper_adv.F90:11.6: > > use petsc > 1 > Fatal Error: Can't open module file 'petsc.mod' for reading at (1): No such file or directory > error: Command "/opt/local/bin/gfortran -Wall -g -fno-second-underscore -DF2PY_REPORT_ON_ARRAY_COPY=1 -I/Users/aponte/.miniconda3/envs/petsc/include -I/Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/petsc4py/include -I. -Ibuild/src.macosx-10.9-x86_64-3.6/. -I/Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include -I/Users/aponte/.miniconda3/envs/petsc/include/python3.6m -c -c ./timestepper_adv.F90 -o build/temp.macosx-10.9-x86_64-3.6/timestepper_adv.o -Jbuild/temp.macosx-10.9-x86_64-3.6/ -Ibuild/temp.macosx-10.9-x86_64-3.6/" failed with exit status 1 > > -- > Aur?lien Ponte > Tel: (+33) 2 98 22 40 73 > Fax: (+33) 2 98 22 44 96 > UMR 6523, IFREMER > ZI de la Pointe du Diable > CS 10070 > 29280 Plouzan? > From aurelien.ponte at ifremer.fr Thu Jan 18 22:48:46 2018 From: aurelien.ponte at ifremer.fr (Aurelien Ponte) Date: Fri, 19 Jan 2018 05:48:46 +0100 Subject: [petsc-users] petsc4py f2py In-Reply-To: References: <4fa20faa-1110-b282-ee74-e19f13357818@ifremer.fr> <35918a00-7d93-a6b1-542b-5cce4a547a78@ifremer.fr> Message-ID: <98bdd64d-565b-0b5a-61ed-2220ea44e7ba@ifremer.fr> there are indeed not petsc related mod files in /Users/aponte/.miniconda3/envs/petsc/include I installed petsc4py via conda, may this be the issue? aur?lien Le 19/01/2018 ? 05:43, Smith, Barry F. a ?crit?: > Fatal Error: Can't open module file 'petsc.mod' for reading at (1): No such file or directory > error: Command "/opt/local/bin/gfortran -Wall -g -fno-second-underscore -DF2PY_REPORT_ON_ARRAY_COPY=1 -I/Users/aponte/.miniconda3/envs/petsc/include -I/Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/petsc4py/include -I. -Ibuild/src.macosx-10.9-x86_64-3.6/. -I/Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include -I/Users/aponte/.miniconda3/envs/petsc/include/python3.6m -c -c ./timestepper_adv.F90 -o build/temp.macosx-10.9-x86_64-3.6/timestepper_adv.o -Jbuild/temp.macosx-10.9-x86_64-3.6/ -Ibuild/temp.macosx-10.9-x86_64-3.6/" > > Normally the module files like petsc.mod would be in the $PETSC_DIR/$PETSC_ARCH/include so in your case I'm guessing /Users/aponte/.miniconda3/envs/petsc/include > > so first do and ls and see if there are any *.mod files in that directory? > > Barry > > >> On Jan 18, 2018, at 10:17 PM, Aurelien Ponte wrote: >> >> Ok, I believe I made some progress. >> I found one more issue: PETSC_DIR was pointing toward an old macport install of petsc >> This is fixed now and here is what I have (without -E): >> >> (petsc) barracuda:qgsolver aponte$ make PYTHON=~/.miniconda3/envs/petsc/bin/python PETSC_DIR=~/.miniconda3/envs/petsc build >> env \ >> F77="" F77FLAGS="" \ >> F90="" F90FLAGS="" \ >> LDSHARED="" \ >> /Users/aponte/.miniconda3/envs/petsc/bin/python setup.py -q build_ext --inplace >> Reading .f2py_f2cmap ... >> Mapping "integer(kind=HANDLE_KIND)" to "long" >> Mapping "integer(kind=handle_kind)" to "long" >> Successfully applied user defined changes from .f2py_f2cmap >> In file included from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1809:0, >> from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18, >> from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4, >> from build/src.macosx-10.9-x86_64-3.6/./fortranobject.h:13, >> from ./timestepper_advmodule.c:18: >> /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] >> #warning "Using deprecated NumPy API, disable it by " \ >> ^ >> In file included from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1809:0, >> from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18, >> from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4, >> from build/src.macosx-10.9-x86_64-3.6/./fortranobject.h:13, >> from build/src.macosx-10.9-x86_64-3.6/./fortranobject.c:2: >> /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] >> #warning "Using deprecated NumPy API, disable it by " \ >> ^ >> build/src.macosx-10.9-x86_64-3.6/./fortranobject.c: In function 'format_def': >> build/src.macosx-10.9-x86_64-3.6/./fortranobject.c:138:18: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] >> if (size < sizeof(notalloc)) { >> ^ >> ./timestepper_adv.F90:11.6: >> >> use petsc >> 1 >> Fatal Error: Can't open module file 'petsc.mod' for reading at (1): No such file or directory >> ./timestepper_adv.F90:11.6: >> >> use petsc >> 1 >> Fatal Error: Can't open module file 'petsc.mod' for reading at (1): No such file or directory >> error: Command "/opt/local/bin/gfortran -Wall -g -fno-second-underscore -DF2PY_REPORT_ON_ARRAY_COPY=1 -I/Users/aponte/.miniconda3/envs/petsc/include -I/Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/petsc4py/include -I. -Ibuild/src.macosx-10.9-x86_64-3.6/. -I/Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include -I/Users/aponte/.miniconda3/envs/petsc/include/python3.6m -c -c ./timestepper_adv.F90 -o build/temp.macosx-10.9-x86_64-3.6/timestepper_adv.o -Jbuild/temp.macosx-10.9-x86_64-3.6/ -Ibuild/temp.macosx-10.9-x86_64-3.6/" failed with exit status 1 >> >> -- >> Aur?lien Ponte >> Tel: (+33) 2 98 22 40 73 >> Fax: (+33) 2 98 22 44 96 >> UMR 6523, IFREMER >> ZI de la Pointe du Diable >> CS 10070 >> 29280 Plouzan? >> -- Aur?lien Ponte Tel: (+33) 2 98 22 40 73 Fax: (+33) 2 98 22 44 96 UMR 6523, IFREMER ZI de la Pointe du Diable CS 10070 29280 Plouzan? From luis.saturday at gmail.com Thu Jan 18 22:49:47 2018 From: luis.saturday at gmail.com (saturday luis) Date: Thu, 18 Jan 2018 20:49:47 -0800 Subject: [petsc-users] matrix preallocation for unstructured finite element Message-ID: Hi PETSc team: Is there a way to make precise matrix preallocation for an unstructured grid (I am not using the DMPlex object). Or do you have any example code for estimating the dnz & onz data? On top of my head, I can do a two-phase calculation. I can first give a rough estimate and do one finite element assembly, then I will call MatGetRow to locate the column indices for nonzero entries. Then in the second step, I will destroy the Mat and create a new Mat with this obtained sparsity pattern. Do you have any suggestions? Thanks! Luis -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Jan 18 22:50:22 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 19 Jan 2018 04:50:22 +0000 Subject: [petsc-users] petsc4py f2py In-Reply-To: <98bdd64d-565b-0b5a-61ed-2220ea44e7ba@ifremer.fr> References: <4fa20faa-1110-b282-ee74-e19f13357818@ifremer.fr> <35918a00-7d93-a6b1-542b-5cce4a547a78@ifremer.fr> <98bdd64d-565b-0b5a-61ed-2220ea44e7ba@ifremer.fr> Message-ID: > On Jan 18, 2018, at 10:48 PM, Aurelien Ponte wrote: > > there are indeed not petsc related mod files in /Users/aponte/.miniconda3/envs/petsc/include > > I installed petsc4py via conda, may this be the issue? I don't know. Does conda build the "Fortran" part of PETSc? If not that would explain the missing mod files. Barry > > aur?lien > > > Le 19/01/2018 ? 05:43, Smith, Barry F. a ?crit : >> Fatal Error: Can't open module file 'petsc.mod' for reading at (1): No such file or directory >> error: Command "/opt/local/bin/gfortran -Wall -g -fno-second-underscore -DF2PY_REPORT_ON_ARRAY_COPY=1 -I/Users/aponte/.miniconda3/envs/petsc/include -I/Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/petsc4py/include -I. -Ibuild/src.macosx-10.9-x86_64-3.6/. -I/Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include -I/Users/aponte/.miniconda3/envs/petsc/include/python3.6m -c -c ./timestepper_adv.F90 -o build/temp.macosx-10.9-x86_64-3.6/timestepper_adv.o -Jbuild/temp.macosx-10.9-x86_64-3.6/ -Ibuild/temp.macosx-10.9-x86_64-3.6/" >> >> Normally the module files like petsc.mod would be in the $PETSC_DIR/$PETSC_ARCH/include so in your case I'm guessing /Users/aponte/.miniconda3/envs/petsc/include >> >> so first do and ls and see if there are any *.mod files in that directory? >> >> Barry >> >> >>> On Jan 18, 2018, at 10:17 PM, Aurelien Ponte wrote: >>> >>> Ok, I believe I made some progress. >>> I found one more issue: PETSC_DIR was pointing toward an old macport install of petsc >>> This is fixed now and here is what I have (without -E): >>> >>> (petsc) barracuda:qgsolver aponte$ make PYTHON=~/.miniconda3/envs/petsc/bin/python PETSC_DIR=~/.miniconda3/envs/petsc build >>> env \ >>> F77="" F77FLAGS="" \ >>> F90="" F90FLAGS="" \ >>> LDSHARED="" \ >>> /Users/aponte/.miniconda3/envs/petsc/bin/python setup.py -q build_ext --inplace >>> Reading .f2py_f2cmap ... >>> Mapping "integer(kind=HANDLE_KIND)" to "long" >>> Mapping "integer(kind=handle_kind)" to "long" >>> Successfully applied user defined changes from .f2py_f2cmap >>> In file included from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1809:0, >>> from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18, >>> from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4, >>> from build/src.macosx-10.9-x86_64-3.6/./fortranobject.h:13, >>> from ./timestepper_advmodule.c:18: >>> /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] >>> #warning "Using deprecated NumPy API, disable it by " \ >>> ^ >>> In file included from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1809:0, >>> from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18, >>> from /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4, >>> from build/src.macosx-10.9-x86_64-3.6/./fortranobject.h:13, >>> from build/src.macosx-10.9-x86_64-3.6/./fortranobject.c:2: >>> /Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] >>> #warning "Using deprecated NumPy API, disable it by " \ >>> ^ >>> build/src.macosx-10.9-x86_64-3.6/./fortranobject.c: In function 'format_def': >>> build/src.macosx-10.9-x86_64-3.6/./fortranobject.c:138:18: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] >>> if (size < sizeof(notalloc)) { >>> ^ >>> ./timestepper_adv.F90:11.6: >>> >>> use petsc >>> 1 >>> Fatal Error: Can't open module file 'petsc.mod' for reading at (1): No such file or directory >>> ./timestepper_adv.F90:11.6: >>> >>> use petsc >>> 1 >>> Fatal Error: Can't open module file 'petsc.mod' for reading at (1): No such file or directory >>> error: Command "/opt/local/bin/gfortran -Wall -g -fno-second-underscore -DF2PY_REPORT_ON_ARRAY_COPY=1 -I/Users/aponte/.miniconda3/envs/petsc/include -I/Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/petsc4py/include -I. -Ibuild/src.macosx-10.9-x86_64-3.6/. -I/Users/aponte/.miniconda3/envs/petsc/lib/python3.6/site-packages/numpy/core/include -I/Users/aponte/.miniconda3/envs/petsc/include/python3.6m -c -c ./timestepper_adv.F90 -o build/temp.macosx-10.9-x86_64-3.6/timestepper_adv.o -Jbuild/temp.macosx-10.9-x86_64-3.6/ -Ibuild/temp.macosx-10.9-x86_64-3.6/" failed with exit status 1 >>> >>> -- >>> Aur?lien Ponte >>> Tel: (+33) 2 98 22 40 73 >>> Fax: (+33) 2 98 22 44 96 >>> UMR 6523, IFREMER >>> ZI de la Pointe du Diable >>> CS 10070 >>> 29280 Plouzan? >>> > > -- > Aur?lien Ponte > Tel: (+33) 2 98 22 40 73 > Fax: (+33) 2 98 22 44 96 > UMR 6523, IFREMER > ZI de la Pointe du Diable > CS 10070 > 29280 Plouzan? > From s_g at berkeley.edu Thu Jan 18 22:59:14 2018 From: s_g at berkeley.edu (Sanjay Govindjee) Date: Thu, 18 Jan 2018 20:59:14 -0800 Subject: [petsc-users] matrix preallocation for unstructured finite element In-Reply-To: References: Message-ID: <171d1a5e-0d0f-d014-2522-d56829e78559@berkeley.edu> If you are doing FEA, then you should have the element connectivity data available from the input.? From this it is an easy task to compute the the assembly pattern and hence the precise allocation data, dnz and onz.? There should be no need to perform a 'trial' assembly. -sanjay On 1/18/18 8:49 PM, saturday luis wrote: > Hi PETSc team: > > Is there a way to make precise matrix preallocation for an > unstructured grid (I am not using the?DMPlex object). Or do you have > any example code for estimating the?dnz?& onz?data? > > On top of my head, I can do a two-phase calculation. I can first give > a rough estimate and do one finite element assembly, then I will call > MatGetRow to locate the column indices for nonzero entries. Then in > the second step, I will destroy the Mat and create a new Mat with this > obtained sparsity pattern. > > Do you have any suggestions? > > Thanks! > > Luis From luis.saturday at gmail.com Thu Jan 18 23:04:00 2018 From: luis.saturday at gmail.com (saturday luis) Date: Thu, 18 Jan 2018 21:04:00 -0800 Subject: [petsc-users] matrix preallocation for unstructured finite element In-Reply-To: <171d1a5e-0d0f-d014-2522-d56829e78559@berkeley.edu> References: <171d1a5e-0d0f-d014-2522-d56829e78559@berkeley.edu> Message-ID: I am not exactly sure what you mean. Given an IEN array, how do you get dnz and onz? Do you have any algorithm or pseudocode for that? Any reference will be appreciated! Thanks, Luis 2018-01-18 20:59 GMT-08:00 Sanjay Govindjee : > If you are doing FEA, then you should have the element connectivity data > available from the input. From this it is an easy task to compute the the > assembly pattern and hence the precise allocation data, dnz and onz. There > should be no need to perform a 'trial' assembly. > > -sanjay > > > On 1/18/18 8:49 PM, saturday luis wrote: > >> Hi PETSc team: >> >> Is there a way to make precise matrix preallocation for an unstructured >> grid (I am not using the DMPlex object). Or do you have any example code >> for estimating the dnz & onz data? >> >> On top of my head, I can do a two-phase calculation. I can first give a >> rough estimate and do one finite element assembly, then I will call >> MatGetRow to locate the column indices for nonzero entries. Then in the >> second step, I will destroy the Mat and create a new Mat with this obtained >> sparsity pattern. >> >> Do you have any suggestions? >> >> Thanks! >> >> Luis >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Jan 18 23:37:51 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 19 Jan 2018 05:37:51 +0000 Subject: [petsc-users] valgrind errors on vecscattercreatetoAll In-Reply-To: <61BC59F3-E250-4AA4-BCB0-905F9C57E681@anl.gov> References: <849C2523-6365-413B-AB28-5502A12C650C@anl.gov> <61BC59F3-E250-4AA4-BCB0-905F9C57E681@anl.gov> Message-ID: Randall, I have resolved in the problem in branch barry/fortran-initialize-objects (now in next for testing) which defaults to initializing the PETSc objects in Fortran and removes the use of PetscObjects in Fortran common blocks to prevent compile errors. Soon will be in master Barry > On Jan 18, 2018, at 5:08 PM, Smith, Barry F. wrote: > > > Sorry this is actually in master already > > >> On Jan 18, 2018, at 5:00 PM, Smith, Barry F. wrote: >> >> >> This is a known issue without a perfect solution. >> >> The problem is that by default PETSc fortran types, like type(tKSP) are not initialized but we want to check if they are a special null character, for example PETSC_NULL_KSP. So we could initialize them but then they cannot be used in common blocks (since default initialized Fortran90 types cannot be used in common blocks; some strange feature of Fortran 90) >> >> In next you can configure with the option -with-fortran-type-initialize=1 and the valgrind errors will go away, but then there cannot be Fortran datatypes into common blocks. >> >> Barry >> >> It may be we should make -with-fortran-type-initialize=1 the default but this will break some PETSc Fortran examples that use common blocks. We should rewrite those examples not to use common blocks but who volunteers? >> >> >> >> >> >> >>> On Jan 18, 2018, at 4:11 PM, Randall Mackie wrote: >>> >>> The very simple attached program throws lots of valgrind errors. >>> I am using pets 3.8.3, compiled with the following options: >>> >>> >>> ./configure \ >>> --with-debugging=1 \ >>> --with-fortran=1 \ >>> --download-mpich=../mpich-3.3a2.tar.gz \ >>> >>> >>> The makefile, run file, and valgrind output are also attached. >>> >>> >>> Randy M. >>> >>> >>> >>> >>> >> > From knepley at gmail.com Fri Jan 19 06:00:58 2018 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 19 Jan 2018 07:00:58 -0500 Subject: [petsc-users] matrix preallocation for unstructured finite element In-Reply-To: References: <171d1a5e-0d0f-d014-2522-d56829e78559@berkeley.edu> Message-ID: On Fri, Jan 19, 2018 at 12:04 AM, saturday luis wrote: > I am not exactly sure what you mean. > > Given an IEN array, how do you get dnz and onz? Do you have any algorithm > or pseudocode for that? Any reference will be appreciated! > You use a hash table the size of the number of rows, and insert columns for each entry. Then you read out those columns, checking whether they are in the diagonal block or not. This is exactly the code I use in Plex. Matt > Thanks, > > Luis > > 2018-01-18 20:59 GMT-08:00 Sanjay Govindjee : > >> If you are doing FEA, then you should have the element connectivity data >> available from the input. From this it is an easy task to compute the the >> assembly pattern and hence the precise allocation data, dnz and onz. There >> should be no need to perform a 'trial' assembly. >> >> -sanjay >> >> >> On 1/18/18 8:49 PM, saturday luis wrote: >> >>> Hi PETSc team: >>> >>> Is there a way to make precise matrix preallocation for an unstructured >>> grid (I am not using the DMPlex object). Or do you have any example code >>> for estimating the dnz & onz data? >>> >>> On top of my head, I can do a two-phase calculation. I can first give a >>> rough estimate and do one finite element assembly, then I will call >>> MatGetRow to locate the column indices for nonzero entries. Then in the >>> second step, I will destroy the Mat and create a new Mat with this obtained >>> sparsity pattern. >>> >>> Do you have any suggestions? >>> >>> Thanks! >>> >>> Luis >>> >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Jan 19 10:08:12 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 19 Jan 2018 16:08:12 +0000 Subject: [petsc-users] Possible post-docs or job positions working with the PETSc team References: <2C58B347-A339-4DD2-8728-7C92419DEE4A@mcs.anl.gov> Message-ID: <4E224A5F-50E7-4A9C-BE25-FB8904AFB48C@anl.gov> PETSc users and developers, We have some open positions at Argonne National Laboratory to work with the PETSc team, mostly post-docs but if a more senior person applied we could consider them as well. Here is where you apply https://careers.peopleclick.com/careerscp/client_argonnelab/post_doc/en_US/gateway.do?functionName=viewFromLink&localeCode=en-us&jobPostId=5704&source=Twitter&sourceType=NETWORKING_SITE https://careers.peopleclick.com/careerscp/client_argonnelab/post_doc/en_US/gateway.do?functionName=viewFromLink&localeCode=en-us&jobPostId=5920&source=Twitter&sourceType=NETWORKING_SITE Don't bother sending use emails, we'll get the applications through the application process. Thanks Barry -------------- next part -------------- An HTML attachment was scrubbed... URL: From Anthony.J.Ruth.12 at nd.edu Fri Jan 19 13:25:25 2018 From: Anthony.J.Ruth.12 at nd.edu (Anthony Ruth) Date: Fri, 19 Jan 2018 14:25:25 -0500 Subject: [petsc-users] Combine distributed vector into local vector Message-ID: Hello, I am working on converting a serial code to use Petsc. To do this it starts in serial with the old code, creates some Petsc matrices/vectors and does the most critical calculations in Petsc, and then returns to serial to return to the old code. After this I was planning to expand the area of the code which uses Petsc, but starting with a small chunk seemed appropriate. To return to the serial code, I need to have every process receive a full copy of a vector. I believe VecScatters are intended for this, but I cannot get it to work. I have produced a minimal example which starts with a serial double array, creates a Vec which is successfully distributed, and then tries to use a VecScatter to aggregate the vector from each process. After the VecScatter step, each process only has the parts of the vector which were available to the process in the distributed vector. If you could please show how to aggregate the vector, it would be greatly appreciated. Use with: mpiexec -n 2 ./test: #include double primes[] = {2,3,5,7,11,13,17}; int nprimes = 7; int main(int argc,char **argv) { PetscInitialize(&argc,&argv, NULL,NULL); MPI_Comm comm=MPI_COMM_WORLD; Vec xpar,xseq; PetscInt low,high; IS index_set_global, index_set_local; const PetscInt *indices; VecScatter vc; PetscErrorCode ierr; //Set up parallel vector ierr = VecCreateMPI(comm, PETSC_DETERMINE, nprimes, &xpar); CHKERRQ(ierr); ierr = VecGetOwnershipRange(xpar, &low, &high); CHKERRQ(ierr); ierr = ISCreateStride(comm, high - low, low, 1, &index_set_global); CHKERRQ(ierr); ierr = ISGetIndices(index_set_global, &indices); CHKERRQ(ierr); ierr = ISView(index_set_global, PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr); ierr = VecSetValues(xpar, high - low, indices, primes + low, INSERT_VALUES);CHKERRQ(ierr); ierr = VecAssemblyBegin(xpar); CHKERRQ(ierr); ierr = VecAssemblyEnd(xpar); CHKERRQ(ierr); ierr = VecView(xpar, PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr); //Scatter parallel vector so all processes have full vector ierr = VecCreateSeq(PETSC_COMM_SELF, nprimes, &xseq); CHKERRQ(ierr); //ierr = VecCreateMPI(comm, high - low, nprimes, &xseq); CHKERRQ(ierr); ierr = ISCreateStride(comm, high - low, 0, 1, &index_set_local); CHKERRQ(ierr); ierr = VecScatterCreate(xpar, index_set_local, xseq, index_set_global, &vc); CHKERRQ(ierr); ierr = VecScatterBegin(vc, xpar, xseq, ADD_VALUES, SCATTER_FORWARD); CHKERRQ(ierr); ierr = VecScatterEnd(vc, xpar, xseq, ADD_VALUES, SCATTER_FORWARD); CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_SELF, "\nPrinting out scattered vector\n"); CHKERRQ(ierr); ierr = VecView(xseq, PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr); PetscFinalize(); } Output: IS Object: 2 MPI processes type: stride [0] Index set is permutation [0] Number of indices in (stride) set 4 [0] 0 0 [0] 1 1 [0] 2 2 [0] 3 3 [1] Number of indices in (stride) set 3 [1] 0 4 [1] 1 5 [1] 2 6 Vec Object: 2 MPI processes type: mpi Process [0] 2. 3. 5. 7. Process [1] 11. 13. 17. Printing out scattered vector Printing out scattered vector Vec Object: 1 MPI processes type: seq 2. 3. 5. 7. 0. 0. 0. Anthony Ruth Condensed Matter Theory University of Notre Dame -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Fri Jan 19 14:42:12 2018 From: hzhang at mcs.anl.gov (Hong) Date: Fri, 19 Jan 2018 14:42:12 -0600 Subject: [petsc-users] Combine distributed vector into local vector In-Reply-To: References: Message-ID: Anthony: You can use VecScatterCreateToAll http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecScatterCreateToAll.html See petsc/src/vec/vec/examples/tests/ex33.c Hong Hello, I am working on converting a serial code to use Petsc. To do this it > starts in serial with the old code, creates some Petsc matrices/vectors and > does the most critical calculations in Petsc, and then returns to serial to > return to the old code. After this I was planning to expand the area of the > code which uses Petsc, but starting with a small chunk seemed appropriate. > To return to the serial code, I need to have every process receive a full > copy of a vector. I believe VecScatters are intended for this, but I cannot > get it to work. I have produced a minimal example which starts with a > serial double array, creates a Vec which is successfully distributed, and > then tries to use a VecScatter to aggregate the vector from each process. > After the VecScatter step, each process only has the parts of the vector > which were available to the process in the distributed vector. If you could > please show how to aggregate the vector, it would be greatly appreciated. > > Use with: mpiexec -n 2 ./test: > > #include > > double primes[] = {2,3,5,7,11,13,17}; > int nprimes = 7; > > int main(int argc,char **argv) > { > PetscInitialize(&argc,&argv, NULL,NULL); > MPI_Comm comm=MPI_COMM_WORLD; > Vec xpar,xseq; > PetscInt low,high; > IS index_set_global, index_set_local; > const PetscInt *indices; > VecScatter vc; > PetscErrorCode ierr; > //Set up parallel vector > ierr = VecCreateMPI(comm, PETSC_DETERMINE, nprimes, &xpar); > CHKERRQ(ierr); > ierr = VecGetOwnershipRange(xpar, &low, &high); CHKERRQ(ierr); > ierr = ISCreateStride(comm, high - low, low, 1, &index_set_global); > CHKERRQ(ierr); > ierr = ISGetIndices(index_set_global, &indices); CHKERRQ(ierr); > ierr = ISView(index_set_global, PETSC_VIEWER_STDOUT_WORLD); > CHKERRQ(ierr); > ierr = VecSetValues(xpar, high - low, indices, primes + low, > INSERT_VALUES);CHKERRQ(ierr); > ierr = VecAssemblyBegin(xpar); CHKERRQ(ierr); > ierr = VecAssemblyEnd(xpar); CHKERRQ(ierr); > ierr = VecView(xpar, PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr); > //Scatter parallel vector so all processes have full vector > ierr = VecCreateSeq(PETSC_COMM_SELF, nprimes, &xseq); CHKERRQ(ierr); > //ierr = VecCreateMPI(comm, high - low, nprimes, &xseq); CHKERRQ(ierr); > ierr = ISCreateStride(comm, high - low, 0, 1, &index_set_local); > CHKERRQ(ierr); > ierr = VecScatterCreate(xpar, index_set_local, xseq, index_set_global, > &vc); CHKERRQ(ierr); > ierr = VecScatterBegin(vc, xpar, xseq, ADD_VALUES, SCATTER_FORWARD); > CHKERRQ(ierr); > ierr = VecScatterEnd(vc, xpar, xseq, ADD_VALUES, SCATTER_FORWARD); > CHKERRQ(ierr); > ierr = PetscPrintf(PETSC_COMM_SELF, "\nPrinting out scattered > vector\n"); CHKERRQ(ierr); > ierr = VecView(xseq, PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr); > > PetscFinalize(); > } > > > Output: > > IS Object: 2 MPI processes > type: stride > [0] Index set is permutation > [0] Number of indices in (stride) set 4 > [0] 0 0 > [0] 1 1 > [0] 2 2 > [0] 3 3 > [1] Number of indices in (stride) set 3 > [1] 0 4 > [1] 1 5 > [1] 2 6 > Vec Object: 2 MPI processes > type: mpi > Process [0] > 2. > 3. > 5. > 7. > Process [1] > 11. > 13. > 17. > > Printing out scattered vector > > Printing out scattered vector > Vec Object: 1 MPI processes > type: seq > 2. > 3. > 5. > 7. > 0. > 0. > 0. > > > > > Anthony Ruth > Condensed Matter Theory > University of Notre Dame > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mbuerkle at web.de Sat Jan 20 02:28:08 2018 From: mbuerkle at web.de (Marius Buerkle) Date: Sat, 20 Jan 2018 09:28:08 +0100 Subject: [petsc-users] petsc and mumps for compute user-specified set of entries in inv(A) Message-ID: An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Sat Jan 20 05:49:10 2018 From: mfadams at lbl.gov (Mark Adams) Date: Sat, 20 Jan 2018 06:49:10 -0500 Subject: [petsc-users] hypre with OMP Message-ID: We are trying to use hypre with OpenMP. Is there anything that we need to do to get hypre to use threads? We tried maint without doing anything and hypre did not seem to be using OMP so I just wanted to check on the status of this. I understand that hypre supports OMP as of last summer and this has been integrated in PETSc. Thanks, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 4075393 bytes Desc: not available URL: From bsmith at mcs.anl.gov Sat Jan 20 07:10:55 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Sat, 20 Jan 2018 13:10:55 +0000 Subject: [petsc-users] hypre with OMP In-Reply-To: References: Message-ID: <1756E16C-7F61-479D-BB28-720A305AAA61@anl.gov> --with-openmp Then control the number of threads via whatever horrible mechanism exists for that which likely is setting some environmental variable. Barry > On Jan 20, 2018, at 5:49 AM, Mark Adams wrote: > > We are trying to use hypre with OpenMP. Is there anything that we need to do to get hypre to use threads? We tried maint without doing anything and hypre did not seem to be using OMP so I just wanted to check on the status of this. I understand that hypre supports OMP as of last summer and this has been integrated in PETSc. > > Thanks, > Mark > From hzhang at mcs.anl.gov Sat Jan 20 11:42:36 2018 From: hzhang at mcs.anl.gov (Hong) Date: Sat, 20 Jan 2018 11:42:36 -0600 Subject: [petsc-users] petsc and mumps for compute user-specified set of entries in inv(A) In-Reply-To: References: Message-ID: Marius : Current petsc-mumps interface supports ICNTL(30), e.g., runtime option '-mat_mumps_icntl_30 0' as default. However, no user ever has tested it. I tested it using petsc/src/mat/examples/tests/ex130.c with '-mat_mumps_icntl_30 1' and got an error in MatSolve() -- additional work needs to be down here. I'll read mumps user manual and investigate it next week. Meanwhile, you may give it a try and figure out how to set other parameters, such as icntl_27 and allocate correct rhs and solution vectors. Hong > Hi, > > Is it possible to interface MUMPS to compute user-specified set of > entries in inv(A) (ICNTL(30)) using petsc ? > > best, > marius > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fdkong.jd at gmail.com Mon Jan 22 11:44:35 2018 From: fdkong.jd at gmail.com (Fande Kong) Date: Mon, 22 Jan 2018 10:44:35 -0700 Subject: [petsc-users] Time cost by Vec Assembly In-Reply-To: <87int3cvjg.fsf@jedbrown.org> References: <577C337B.60909@uci.edu> <5959F823-EDE5-4B34-84C2-271076977368@mcs.anl.gov> <0CFDEA05-2C49-4127-9F13-2B2DB71ADA77@mcs.anl.gov> <27f4756a-3c58-5c56-fd5b-000aac881a5b@uci.edu> <613e3c14-12f9-8ffe-8b61-58faf284f002@uci.edu> <87oa2vcxnz.fsf@jedbrown.org> <87int3cvjg.fsf@jedbrown.org> Message-ID: On Fri, Oct 7, 2016 at 10:30 PM, Jed Brown wrote: > Barry Smith writes: > > There is still something wonky here, whether it is the MPI > implementation or how PETSc handles the assembly. Without any values that > need to be communicated it is unacceptably that these calls take so long. > If we understood __exactly__ why the performance suddenly drops so > dramatically we could perhaps fix it. I do not understand why. > > I guess it's worth timing. If they don't have MPI_Reduce_scatter_block > then it falls back to a big MPI_Allreduce. After that, it's all > point-to-point messaging that shouldn't suck and there actually > shouldn't be anything to send or receive anyway. The BTS implementation > should be much smarter and literally reduces to a barrier in this case. > Hi Jed, How to use the BTS implementation for Vec. For mat, we may just use "-matstash_bts"? Fande, -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Mon Jan 22 12:49:33 2018 From: jed at jedbrown.org (Jed Brown) Date: Mon, 22 Jan 2018 11:49:33 -0700 Subject: [petsc-users] Time cost by Vec Assembly In-Reply-To: References: <577C337B.60909@uci.edu> <5959F823-EDE5-4B34-84C2-271076977368@mcs.anl.gov> <0CFDEA05-2C49-4127-9F13-2B2DB71ADA77@mcs.anl.gov> <27f4756a-3c58-5c56-fd5b-000aac881a5b@uci.edu> <613e3c14-12f9-8ffe-8b61-58faf284f002@uci.edu> <87oa2vcxnz.fsf@jedbrown.org> <87int3cvjg.fsf@jedbrown.org> Message-ID: <87a7x591aa.fsf@jedbrown.org> Fande Kong writes: > On Fri, Oct 7, 2016 at 10:30 PM, Jed Brown wrote: > >> Barry Smith writes: >> > There is still something wonky here, whether it is the MPI >> implementation or how PETSc handles the assembly. Without any values that >> need to be communicated it is unacceptably that these calls take so long. >> If we understood __exactly__ why the performance suddenly drops so >> dramatically we could perhaps fix it. I do not understand why. >> >> I guess it's worth timing. If they don't have MPI_Reduce_scatter_block >> then it falls back to a big MPI_Allreduce. After that, it's all >> point-to-point messaging that shouldn't suck and there actually >> shouldn't be anything to send or receive anyway. The BTS implementation >> should be much smarter and literally reduces to a barrier in this case. >> > > Hi Jed, > > How to use the BTS implementation for Vec. For mat, we may just use > "-matstash_bts"? Should be -vec_assembly_bts From yann.jobic at univ-amu.fr Tue Jan 23 10:14:11 2018 From: yann.jobic at univ-amu.fr (Yann Jobic) Date: Tue, 23 Jan 2018 17:14:11 +0100 Subject: [petsc-users] FEM & conformal mesh Message-ID: <6daf01cd-d224-62bb-70f6-44c5670451e5@univ-amu.fr> Hello, I'm trying to understand the numbering of quadrature points in order to solve the FEM system, and how you manage this numbering in order to allow conformal mesh. I looked in several files in order to understand. Here's what i understood so far (which is not far...) I took the example of the jacobian calculus. I found this comment in dmplexsnes.c, which explains the basic idea: 1725:?? /* 1: Get sizes from dm and dmAux */ 1726:?? /* 2: Get geometric data */ 1727:?? /* 3: Handle boundary values */ 1728:?? /* 4: Loop over domain */ 1729:?? /*?? Extract coefficients */ 1730:?? /* Loop over fields */ 1731:?? /*?? Set tiling for FE*/ 1732:?? /*?? Integrate FE residual to get elemVec */ [...] 1740:?? /* Loop over domain */ 1741:?? /*?? Add elemVec to locX */ I almost get that. The critical part should be : loop over fieldI 2434:???? PetscFEGetQuadrature(fe, &quad); 2435:???? PetscFEGetDimension(fe, &Nb); 2436:???? PetscFEGetTileSizes(fe, NULL, &numBlocks, NULL, &numBatches); 2437:???? PetscQuadratureGetData(quad, NULL, NULL, &numQuadPoints, NULL, NULL); 2438:???? blockSize = Nb*numQuadPoints; 2439:???? batchSize = numBlocks * blockSize; 2440:???? PetscFESetTileSizes(fe, blockSize, numBlocks, batchSize, numBatches); 2441:???? numChunks = numCells / (numBatches*batchSize); 2442:???? Ne??????? = numChunks*numBatches*batchSize; 2443:???? Nr??????? = numCells % (numBatches*batchSize); 2444:???? offset??? = numCells - Nr; 2445:???? for (fieldJ = 0; fieldJ < Nf; ++fieldJ) { From there, we can have the numbering with (in dtfe.c) basic idea : 6560: $?? Loop over element matrix entries (f,fc,g,gc --> i,j): Which leads to : 4511:?????? PetscPrintf(PETSC_COMM_SELF, "Element matrix for fields %d and %d\n", fieldI, fieldJ); 4512:?????? for (fc = 0; fc < NcI; ++fc) { 4513:???????? for (f = 0; f < NbI; ++f) { 4514:?????????? const PetscInt i = offsetI + f*NcI+fc; 4515:?????????? for (gc = 0; gc < NcJ; ++gc) { 4516:???????????? for (g = 0; g < NbJ; ++g) { 4517:?????????????? const PetscInt j = offsetJ + g*NcJ+gc; 4518:?????????????? PetscPrintf(PETSC_COMM_SELF, " elemMat[%d,%d,%d,%d]: %g\n", f, fc, g, gc, PetscRealPart(elemMat[eOffset+i*totDim+j])); [...] 4525:???? cOffset??? += totDim; 4526:???? cOffsetAux += totDimAux; 4527:???? eOffset??? += PetscSqr(totDim); 4528:?? } But i didn't get how you can find that there are duplicates quadrature nodes, and how you manage them. Maybe i looked at the wrong part of the code ? Thanks ! Best regards, Yann --- L'absence de virus dans ce courrier ?lectronique a ?t? v?rifi?e par le logiciel antivirus Avast. https://www.avast.com/antivirus From luis.saturday at gmail.com Tue Jan 23 13:13:24 2018 From: luis.saturday at gmail.com (saturday luis) Date: Tue, 23 Jan 2018 11:13:24 -0800 Subject: [petsc-users] Solve Schur complement problem Message-ID: Hi PETSc team: I am trying to solve a Stokes problem in a segregated manner, in which the pressure field is obtained by solving the pressure Schur complement. This is what I did. I created a Schur complement by calling Mat S; MatCreateSchurComplement(gloAssem_ptr->K_11, gloAssem_ptr->K_11, gloAssem_ptr->K_10, gloAssem_ptr->K_01, gloAssem_ptr->K_00, &S); Then I use the Mat S as an input for KSP: KSPSetOperators(ksp, S, S); I got error message as follows. [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for possible LU and Cholesky solvers [0]PETSC ERROR: Could not locate a solver package. Perhaps you must ./configure with --download- [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017 [0]PETSC ERROR: ./bipn3d on a arch-linux2-cxx-debug named bacon by luis Tue Jan 23 11:02:34 2018 [0]PETSC ERROR: Configure options --prefix=~/lib/petsc-3.8.3-debug --with-x=0 -with-pic --with-mpi-compilers=1 --with-mpi-dir=/home/lib/mpich-3.2/ --with-clanguage=C++ --with-scalar-type=real --with-shared-libraries=1 --with-precision=double --with-hypre=1 --download-hypre --with-spai=1 --download-spai --with-superlu_dist=1 --download-superlu_dist --with-superlu=1 --download-superlu --with-parmetis=1 --download-parmetis --with-metis=1 --download-metis --with-mumps=1 --download-mumps --with-parmetis=1 --download-parmetis --with-metis=1 --download-metis --with-scalapack=1 --download-scalapack --with-blacs=1 --download-blacs --with-spooles=1 --download-spooles --with-debugging=yes --download-fblaslapack [0]PETSC ERROR: #1 MatGetFactor() line 4346 in /home/lib/petsc-3.8.3/src/mat/interface/matrix.c [0]PETSC ERROR: #2 PCSetUp_ILU() line 142 in /home/lib/petsc-3.8.3/src/ksp/pc/impls/factor/ilu/ilu.c [0]PETSC ERROR: #3 PCSetUp() line 924 in /home/lib/petsc-3.8.3/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #4 KSPSetUp() line 381 in /home/lib/petsc-3.8.3/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #5 KSPSolve() line 612 in /home/lib/petsc-3.8.3/src/ksp/ksp/interface/itfunc.c Let me know if you can see any issue in the usage of Schur complement creation. Thanks, Luis -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jan 23 13:25:27 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 23 Jan 2018 19:25:27 +0000 Subject: [petsc-users] Solve Schur complement problem In-Reply-To: References: Message-ID: Luis, This doesn't really have any thing to do with the Schur complement per se, you are asking do an ILU on this abstract Schur complement matrix which is not possible. Almost for sure you want to use PCFIELDSPLIT and not directly create the MatCreateSchurComplement() yourself. PCFIELDSPLIT has many possible variants for solving systems with segregated Schur complement methods. Barry > On Jan 23, 2018, at 1:13 PM, saturday luis wrote: > > Hi PETSc team: > > I am trying to solve a Stokes problem in a segregated manner, in which the pressure field is obtained by solving the pressure Schur complement. This is what I did. > > I created a Schur complement by calling > > Mat S; > MatCreateSchurComplement(gloAssem_ptr->K_11, gloAssem_ptr->K_11, gloAssem_ptr->K_10, gloAssem_ptr->K_01, gloAssem_ptr->K_00, &S); > > Then I use the Mat S as an input for KSP: > KSPSetOperators(ksp, S, S); > > I got error message as follows. > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for possible LU and Cholesky solvers > [0]PETSC ERROR: Could not locate a solver package. Perhaps you must ./configure with --download- > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017 > [0]PETSC ERROR: ./bipn3d on a arch-linux2-cxx-debug named bacon by luis Tue Jan 23 11:02:34 2018 > [0]PETSC ERROR: Configure options --prefix=~/lib/petsc-3.8.3-debug --with-x=0 -with-pic --with-mpi-compilers=1 --with-mpi-dir=/home/lib/mpich-3.2/ --with-clanguage=C++ --with-scalar-type=real --with-shared-libraries=1 --with-precision=double --with-hypre=1 --download-hypre --with-spai=1 --download-spai --with-superlu_dist=1 --download-superlu_dist --with-superlu=1 --download-superlu --with-parmetis=1 --download-parmetis --with-metis=1 --download-metis --with-mumps=1 --download-mumps --with-parmetis=1 --download-parmetis --with-metis=1 --download-metis --with-scalapack=1 --download-scalapack --with-blacs=1 --download-blacs --with-spooles=1 --download-spooles --with-debugging=yes --download-fblaslapack > [0]PETSC ERROR: #1 MatGetFactor() line 4346 in /home/lib/petsc-3.8.3/src/mat/interface/matrix.c > [0]PETSC ERROR: #2 PCSetUp_ILU() line 142 in /home/lib/petsc-3.8.3/src/ksp/pc/impls/factor/ilu/ilu.c > [0]PETSC ERROR: #3 PCSetUp() line 924 in /home/lib/petsc-3.8.3/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #4 KSPSetUp() line 381 in /home/lib/petsc-3.8.3/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #5 KSPSolve() line 612 in /home/lib/petsc-3.8.3/src/ksp/ksp/interface/itfunc.c > > Let me know if you can see any issue in the usage of Schur complement creation. > > Thanks, > > Luis From luis.saturday at gmail.com Tue Jan 23 14:18:51 2018 From: luis.saturday at gmail.com (saturday luis) Date: Tue, 23 Jan 2018 12:18:51 -0800 Subject: [petsc-users] Solve Schur complement problem In-Reply-To: References: Message-ID: Barry: Thanks for the quick response! I assumed that the Mat object created by MatCreateSchurComplement can somehow be solved by a KSP option (like plain CG). I am aware of the fieldsplit pc. My situation is that I have to solve a problem with S = L - D K^-1 D^T, wherein L, D, K are all sparse. Of course, the last option for me is to write a simple CG solver for S myself. Do you have any recommendations? Thanks, Luis 2018-01-23 11:25 GMT-08:00 Smith, Barry F. : > > Luis, > > This doesn't really have any thing to do with the Schur complement per > se, you are asking do an ILU on this abstract Schur complement matrix which > is not possible. > > Almost for sure you want to use PCFIELDSPLIT and not directly create > the MatCreateSchurComplement() yourself. PCFIELDSPLIT has many possible > variants for solving systems with segregated Schur complement methods. > > Barry > > > > On Jan 23, 2018, at 1:13 PM, saturday luis > wrote: > > > > Hi PETSc team: > > > > I am trying to solve a Stokes problem in a segregated manner, in which > the pressure field is obtained by solving the pressure Schur complement. > This is what I did. > > > > I created a Schur complement by calling > > > > Mat S; > > MatCreateSchurComplement(gloAssem_ptr->K_11, gloAssem_ptr->K_11, > gloAssem_ptr->K_10, gloAssem_ptr->K_01, gloAssem_ptr->K_00, &S); > > > > Then I use the Mat S as an input for KSP: > > KSPSetOperators(ksp, S, S); > > > > I got error message as follows. > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/ > linearsolvertable.html for possible LU and Cholesky solvers > > [0]PETSC ERROR: Could not locate a solver package. Perhaps you must > ./configure with --download- > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017 > > [0]PETSC ERROR: ./bipn3d on a arch-linux2-cxx-debug named bacon by luis > Tue Jan 23 11:02:34 2018 > > [0]PETSC ERROR: Configure options --prefix=~/lib/petsc-3.8.3-debug > --with-x=0 -with-pic --with-mpi-compilers=1 --with-mpi-dir=/home/lib/mpich-3.2/ > --with-clanguage=C++ --with-scalar-type=real --with-shared-libraries=1 > --with-precision=double --with-hypre=1 --download-hypre --with-spai=1 > --download-spai --with-superlu_dist=1 --download-superlu_dist > --with-superlu=1 --download-superlu --with-parmetis=1 --download-parmetis > --with-metis=1 --download-metis --with-mumps=1 --download-mumps > --with-parmetis=1 --download-parmetis --with-metis=1 --download-metis > --with-scalapack=1 --download-scalapack --with-blacs=1 --download-blacs > --with-spooles=1 --download-spooles --with-debugging=yes > --download-fblaslapack > > [0]PETSC ERROR: #1 MatGetFactor() line 4346 in > /home/lib/petsc-3.8.3/src/mat/interface/matrix.c > > [0]PETSC ERROR: #2 PCSetUp_ILU() line 142 in > /home/lib/petsc-3.8.3/src/ksp/pc/impls/factor/ilu/ilu.c > > [0]PETSC ERROR: #3 PCSetUp() line 924 in /home/lib/petsc-3.8.3/src/ksp/ > pc/interface/precon.c > > [0]PETSC ERROR: #4 KSPSetUp() line 381 in /home/lib/petsc-3.8.3/src/ksp/ > ksp/interface/itfunc.c > > [0]PETSC ERROR: #5 KSPSolve() line 612 in /home/lib/petsc-3.8.3/src/ksp/ > ksp/interface/itfunc.c > > > > Let me know if you can see any issue in the usage of Schur complement > creation. > > > > Thanks, > > > > Luis > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jan 23 14:27:21 2018 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 24 Jan 2018 07:27:21 +1100 Subject: [petsc-users] Solve Schur complement problem In-Reply-To: References: Message-ID: On Wed, Jan 24, 2018 at 6:13 AM, saturday luis wrote: > Hi PETSc team: > > I am trying to solve a Stokes problem in a segregated manner, in which the > pressure field is obtained by solving the pressure Schur complement. This > is what I did. > > I created a Schur complement by calling > > Mat S; > MatCreateSchurComplement(gloAssem_ptr->K_11, gloAssem_ptr->K_11, > gloAssem_ptr->K_10, gloAssem_ptr->K_01, gloAssem_ptr->K_00, &S); > > Then I use the Mat S as an input for KSP: > KSPSetOperators(ksp, S, S); > > I got error message as follows. > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/ > linearsolvertable.html for possible LU and Cholesky solvers > [0]PETSC ERROR: Could not locate a solver package. Perhaps you must > ./configure with --download- > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017 > [0]PETSC ERROR: ./bipn3d on a arch-linux2-cxx-debug named bacon by luis > Tue Jan 23 11:02:34 2018 > [0]PETSC ERROR: Configure options --prefix=~/lib/petsc-3.8.3-debug > --with-x=0 -with-pic --with-mpi-compilers=1 --with-mpi-dir=/home/lib/mpich-3.2/ > --with-clanguage=C++ --with-scalar-type=real --with-shared-libraries=1 > --with-precision=double --with-hypre=1 --download-hypre --with-spai=1 > --download-spai --with-superlu_dist=1 --download-superlu_dist > --with-superlu=1 --download-superlu --with-parmetis=1 --download-parmetis > --with-metis=1 --download-metis --with-mumps=1 --download-mumps > --with-parmetis=1 --download-parmetis --with-metis=1 --download-metis > --with-scalapack=1 --download-scalapack --with-blacs=1 --download-blacs > --with-spooles=1 --download-spooles --with-debugging=yes > --download-fblaslapack > [0]PETSC ERROR: #1 MatGetFactor() line 4346 in > /home/lib/petsc-3.8.3/src/mat/interface/matrix.c > [0]PETSC ERROR: #2 PCSetUp_ILU() line 142 in /home/lib/petsc-3.8.3/src/ksp/ > pc/impls/factor/ilu/ilu.c > [0]PETSC ERROR: #3 PCSetUp() line 924 in /home/lib/petsc-3.8.3/src/ksp/ > pc/interface/precon.c > [0]PETSC ERROR: #4 KSPSetUp() line 381 in /home/lib/petsc-3.8.3/src/ksp/ > ksp/interface/itfunc.c > [0]PETSC ERROR: #5 KSPSolve() line 612 in /home/lib/petsc-3.8.3/src/ksp/ > ksp/interface/itfunc.c > > Let me know if you can see any issue in the usage of Schur complement > creation. > This will work if you give -pc_type none which will give you plain GMRES. As Barry says, a more sophisticated thing to do is to use PCFIELDSPLIT. I do this in the examples for SNES ex62. Thanks, Matt > Thanks, > > Luis > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From luis.saturday at gmail.com Tue Jan 23 14:31:12 2018 From: luis.saturday at gmail.com (saturday luis) Date: Tue, 23 Jan 2018 12:31:12 -0800 Subject: [petsc-users] Solve Schur complement problem In-Reply-To: References: Message-ID: Great! This is exactly what I want. Thanks! Luis 2018-01-23 12:27 GMT-08:00 Matthew Knepley : > On Wed, Jan 24, 2018 at 6:13 AM, saturday luis > wrote: > >> Hi PETSc team: >> >> I am trying to solve a Stokes problem in a segregated manner, in which >> the pressure field is obtained by solving the pressure Schur complement. >> This is what I did. >> >> I created a Schur complement by calling >> >> Mat S; >> MatCreateSchurComplement(gloAssem_ptr->K_11, gloAssem_ptr->K_11, >> gloAssem_ptr->K_10, gloAssem_ptr->K_01, gloAssem_ptr->K_00, &S); >> >> Then I use the Mat S as an input for KSP: >> KSPSetOperators(ksp, S, S); >> >> I got error message as follows. >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >> ocumentation/linearsolvertable.html for possible LU and Cholesky solvers >> [0]PETSC ERROR: Could not locate a solver package. Perhaps you must >> ./configure with --download- >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017 >> [0]PETSC ERROR: ./bipn3d on a arch-linux2-cxx-debug named bacon by luis >> Tue Jan 23 11:02:34 2018 >> [0]PETSC ERROR: Configure options --prefix=~/lib/petsc-3.8.3-debug >> --with-x=0 -with-pic --with-mpi-compilers=1 --with-mpi-dir=/home/lib/mpich-3.2/ >> --with-clanguage=C++ --with-scalar-type=real --with-shared-libraries=1 >> --with-precision=double --with-hypre=1 --download-hypre --with-spai=1 >> --download-spai --with-superlu_dist=1 --download-superlu_dist >> --with-superlu=1 --download-superlu --with-parmetis=1 --download-parmetis >> --with-metis=1 --download-metis --with-mumps=1 --download-mumps >> --with-parmetis=1 --download-parmetis --with-metis=1 --download-metis >> --with-scalapack=1 --download-scalapack --with-blacs=1 --download-blacs >> --with-spooles=1 --download-spooles --with-debugging=yes >> --download-fblaslapack >> [0]PETSC ERROR: #1 MatGetFactor() line 4346 in >> /home/lib/petsc-3.8.3/src/mat/interface/matrix.c >> [0]PETSC ERROR: #2 PCSetUp_ILU() line 142 in >> /home/lib/petsc-3.8.3/src/ksp/pc/impls/factor/ilu/ilu.c >> [0]PETSC ERROR: #3 PCSetUp() line 924 in /home/lib/petsc-3.8.3/src/ksp/ >> pc/interface/precon.c >> [0]PETSC ERROR: #4 KSPSetUp() line 381 in /home/lib/petsc-3.8.3/src/ksp/ >> ksp/interface/itfunc.c >> [0]PETSC ERROR: #5 KSPSolve() line 612 in /home/lib/petsc-3.8.3/src/ksp/ >> ksp/interface/itfunc.c >> >> Let me know if you can see any issue in the usage of Schur complement >> creation. >> > > This will work if you give > > -pc_type none > > which will give you plain GMRES. As Barry says, a more sophisticated thing > to do is to use PCFIELDSPLIT. I do this in the examples for SNES ex62. > > Thanks, > > Matt > > >> Thanks, >> >> Luis >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From j.wagner at rice.edu Tue Jan 23 22:17:30 2018 From: j.wagner at rice.edu (Jordan Wagner) Date: Tue, 23 Jan 2018 22:17:30 -0600 Subject: [petsc-users] Is there a partition section in a distributed DM object? Message-ID: Hi, At some point, I would like to export my distributed DM along with an associated scalar field corresponding to the process rank for which each point is owned. I know that what I need is the partition section, which is more or less identical to Fig. 6 in Knepley and Karpeev (2009). I was wondering if this particular section is already made behind the scenes when distributing and just hiding somewhere in the DM object. I know it shouldn't be difficult to just make the section myself, but it seems like something that would be needed a lot and already made. I haven't been successful in finding it, so I just wanted to make sure I am not missing anything. Thanks very much jw From bcyee at umich.edu Wed Jan 24 09:12:36 2018 From: bcyee at umich.edu (Ben Yee) Date: Wed, 24 Jan 2018 10:12:36 -0500 Subject: [petsc-users] Changing the operator for PCMG while leaving interpolation matrices unchanged Message-ID: Hi, I have multiple matrices (with different nonzero structures) for which I would like to use the same KSP operator (KSPRICHARDSON with PCMG, Galerkin process for generating coarse grid operator). I want to use the same PCMG structure for all the matrices (same interpolation, same restriction, same number of levels, same smoothing techniques, etc.), so I tried using KSPSetOperators to just change the operator. My interpolation matrices are MPIAIJ matrices that I have defined myself and provided to PCMG. My hope was that, with the new operator I provided, PETSc would recompute all of the coarse/intermediate grid operators using the Galerkin process with the old interpolation matrices and the new fine-grid operator. However, I get the wrong solution after changing the operator with KSPSetOperators. After some extensive digging around the PETSc source code, it seems that the operators in mg.c are not updated by KSPSetOperators. When I do a MatView on mglevels->A or the operator in mglevels->smoothu or mglevels->smoothd in PCMGMCycle_Private() of mg.c, it has the matrix values of the old matrix. From what I understand, the only place where mglevels->A (used to compute the residual) is updated is in PCApplyRIchardson_MG() of mg.c, from the following lines of code: 74: for (i=0; iA) { 76: KSPGetOperators (mglevels[i]->smoothu,&mglevels[i]->A,NULL); 77: PetscObjectReference ((PetscObject )mglevels[i]->A); 78: } 79: } It seems to me that the pointer mglevels[i]->A is only set once, when it is undefined at the beginning of the iteration scheme. I think there are more issues as well, but this is the easiest one to point out. (For example, it doesn't seem like it is possible to update the operator for mglevels[i]->smoothd from KSPSetOperators on the outermost KSP object, and, if I try to set it manually via PCMGGetSmoother, I get errors if the nonzero structures of the coarse-grid operators have changed, which tends to be the case when the fine-grid operator's nonzero structure changes). Is there something I'm doing wrong? Is there a way to do this so that I can reuse PCMG with a new operator? Thanks! -- Ben Yee NERS PhD Candidate, University of Michigan B.S. Mech. & Nuclear Eng., U.C. Berkeley -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Jan 24 09:41:20 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 24 Jan 2018 15:41:20 +0000 Subject: [petsc-users] Changing the operator for PCMG while leaving interpolation matrices unchanged In-Reply-To: References: Message-ID: Ben This is a good question, I'll take a look at how difficult it would be to add this feature. I take it the nonzero structure of the outer matrix changes over time? Barry > On Jan 24, 2018, at 9:12 AM, Ben Yee wrote: > > Hi, > > I have multiple matrices (with different nonzero structures) for which I would like to use the same KSP operator (KSPRICHARDSON with PCMG, Galerkin process for generating coarse grid operator). I want to use the same PCMG structure for all the matrices (same interpolation, same restriction, same number of levels, same smoothing techniques, etc.), so I tried using KSPSetOperators to just change the operator. My interpolation matrices are MPIAIJ matrices that I have defined myself and provided to PCMG. My hope was that, with the new operator I provided, PETSc would recompute all of the coarse/intermediate grid operators using the Galerkin process with the old interpolation matrices and the new fine-grid operator. > > However, I get the wrong solution after changing the operator with KSPSetOperators. After some extensive digging around the PETSc source code, it seems that the operators in mg.c are not updated by KSPSetOperators. When I do a MatView on mglevels->A or the operator in mglevels->smoothu or mglevels->smoothd in PCMGMCycle_Private() of mg.c, it has the matrix values of the old matrix. From what I understand, the only place where mglevels->A (used to compute the residual) is updated is in PCApplyRIchardson_MG() of mg.c, from the following lines of code: > 74: for > (i=0; i > 75: if > (!mglevels[i]->A) { > > 76: KSPGetOperators > (mglevels[i]->smoothu,&mglevels[i]->A,NULL); > > 77: PetscObjectReference((PetscObject > )mglevels[i]->A); > > 78: > } > > 79: } > It seems to me that the pointer mglevels[i]->A is only set once, when it is undefined at the beginning of the iteration scheme. I think there are more issues as well, but this is the easiest one to point out. (For example, it doesn't seem like it is possible to update the operator for mglevels[i]->smoothd from KSPSetOperators on the outermost KSP object, and, if I try to set it manually via PCMGGetSmoother, I get errors if the nonzero structures of the coarse-grid operators have changed, which tends to be the case when the fine-grid operator's nonzero structure changes). > > Is there something I'm doing wrong? Is there a way to do this so that I can reuse PCMG with a new operator? > > Thanks! > > -- > Ben Yee > > NERS PhD Candidate, University of Michigan > B.S. Mech. & Nuclear Eng., U.C. Berkeley From bcyee at umich.edu Wed Jan 24 09:53:46 2018 From: bcyee at umich.edu (Ben Yee) Date: Wed, 24 Jan 2018 10:53:46 -0500 Subject: [petsc-users] Changing the operator for PCMG while leaving interpolation matrices unchanged In-Reply-To: References: Message-ID: Yes, the nonzero structure changes in a way that cannot easily be predicted. I could overallocate, add some extra entries to the first matrix, and reuse that matrix, but I would have to roughly double the number of allocated entries, leading to an undesirable increase in memory. I think that one alternative would be to just reset the KSP object when I change the operator, but, if I'm not mistaken, this would destroy all the interpolation operators and I'd have to redefine all of them. On Wed, Jan 24, 2018 at 10:41 AM, Smith, Barry F. wrote: > > Ben > > This is a good question, I'll take a look at how difficult it would be > to add this feature. > > I take it the nonzero structure of the outer matrix changes over time? > > > Barry > > > > On Jan 24, 2018, at 9:12 AM, Ben Yee wrote: > > > > Hi, > > > > I have multiple matrices (with different nonzero structures) for which I > would like to use the same KSP operator (KSPRICHARDSON with PCMG, Galerkin > process for generating coarse grid operator). I want to use the same PCMG > structure for all the matrices (same interpolation, same restriction, same > number of levels, same smoothing techniques, etc.), so I tried using > KSPSetOperators to just change the operator. My interpolation matrices are > MPIAIJ matrices that I have defined myself and provided to PCMG. My hope > was that, with the new operator I provided, PETSc would recompute all of > the coarse/intermediate grid operators using the Galerkin process with the > old interpolation matrices and the new fine-grid operator. > > > > However, I get the wrong solution after changing the operator with > KSPSetOperators. After some extensive digging around the PETSc source > code, it seems that the operators in mg.c are not updated by > KSPSetOperators. When I do a MatView on mglevels->A or the operator in > mglevels->smoothu or mglevels->smoothd in PCMGMCycle_Private() of mg.c, it > has the matrix values of the old matrix. From what I understand, the only > place where mglevels->A (used to compute the residual) is updated is in > PCApplyRIchardson_MG() of mg.c, from the following lines of code: > > 74: for > > (i=0; i > > > 75: if > > (!mglevels[i]->A) { > > > > 76: KSPGetOperators > > (mglevels[i]->smoothu,&mglevels[i]->A,NULL); > > > > 77: PetscObjectReference((PetscObject > > )mglevels[i]->A); > > > > 78: > > } > > > > 79: } > > It seems to me that the pointer mglevels[i]->A is only set once, when it > is undefined at the beginning of the iteration scheme. I think there are > more issues as well, but this is the easiest one to point out. (For > example, it doesn't seem like it is possible to update the operator for > mglevels[i]->smoothd from KSPSetOperators on the outermost KSP object, and, > if I try to set it manually via PCMGGetSmoother, I get errors if the > nonzero structures of the coarse-grid operators have changed, which tends > to be the case when the fine-grid operator's nonzero structure changes). > > > > Is there something I'm doing wrong? Is there a way to do this so that I > can reuse PCMG with a new operator? > > > > Thanks! > > > > -- > > Ben Yee > > > > NERS PhD Candidate, University of Michigan > > B.S. Mech. & Nuclear Eng., U.C. Berkeley > > -- Ben Yee NERS PhD Candidate, University of Michigan B.S. Mech. & Nuclear Eng., U.C. Berkeley -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Jan 24 11:19:01 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 24 Jan 2018 17:19:01 +0000 Subject: [petsc-users] Changing the operator for PCMG while leaving interpolation matrices unchanged In-Reply-To: References: Message-ID: This is off the top of my head and my not be useful. Copy PCReset_MG() which is PetscErrorCode PCReset_MG(PC pc) { PC_MG *mg = (PC_MG*)pc->data; PC_MG_Levels **mglevels = mg->levels; PetscErrorCode ierr; PetscInt i,n; PetscFunctionBegin; if (mglevels) { n = mglevels[0]->levels; for (i=0; ir);CHKERRQ(ierr); ierr = VecDestroy(&mglevels[i]->b);CHKERRQ(ierr); ierr = VecDestroy(&mglevels[i]->x);CHKERRQ(ierr); ierr = MatDestroy(&mglevels[i+1]->restrct);CHKERRQ(ierr); ierr = MatDestroy(&mglevels[i+1]->interpolate);CHKERRQ(ierr); ierr = VecDestroy(&mglevels[i+1]->rscale);CHKERRQ(ierr); } for (i=0; iA);CHKERRQ(ierr); if (mglevels[i]->smoothd != mglevels[i]->smoothu) { ierr = KSPReset(mglevels[i]->smoothd);CHKERRQ(ierr); } ierr = KSPReset(mglevels[i]->smoothu);CHKERRQ(ierr); } } PetscFunctionReturn(0); } And change it to only destroy mats on each level ierr = MatDestroy(&mglevels[i]->A);CHKERRQ(ierr); Keep everything else. Not that you will need to include #include in your code. > On Jan 24, 2018, at 9:53 AM, Ben Yee wrote: > > Yes, the nonzero structure changes in a way that cannot easily be predicted. I could overallocate, add some extra entries to the first matrix, and reuse that matrix, but I would have to roughly double the number of allocated entries, leading to an undesirable increase in memory. I think that one alternative would be to just reset the KSP object when I change the operator, but, if I'm not mistaken, this would destroy all the interpolation operators and I'd have to redefine all of them. > > On Wed, Jan 24, 2018 at 10:41 AM, Smith, Barry F. wrote: > > Ben > > This is a good question, I'll take a look at how difficult it would be to add this feature. > > I take it the nonzero structure of the outer matrix changes over time? > > > Barry > > > > On Jan 24, 2018, at 9:12 AM, Ben Yee wrote: > > > > Hi, > > > > I have multiple matrices (with different nonzero structures) for which I would like to use the same KSP operator (KSPRICHARDSON with PCMG, Galerkin process for generating coarse grid operator). I want to use the same PCMG structure for all the matrices (same interpolation, same restriction, same number of levels, same smoothing techniques, etc.), so I tried using KSPSetOperators to just change the operator. My interpolation matrices are MPIAIJ matrices that I have defined myself and provided to PCMG. My hope was that, with the new operator I provided, PETSc would recompute all of the coarse/intermediate grid operators using the Galerkin process with the old interpolation matrices and the new fine-grid operator. > > > > However, I get the wrong solution after changing the operator with KSPSetOperators. After some extensive digging around the PETSc source code, it seems that the operators in mg.c are not updated by KSPSetOperators. When I do a MatView on mglevels->A or the operator in mglevels->smoothu or mglevels->smoothd in PCMGMCycle_Private() of mg.c, it has the matrix values of the old matrix. From what I understand, the only place where mglevels->A (used to compute the residual) is updated is in PCApplyRIchardson_MG() of mg.c, from the following lines of code: > > 74: for > > (i=0; i > > > 75: if > > (!mglevels[i]->A) { > > > > 76: KSPGetOperators > > (mglevels[i]->smoothu,&mglevels[i]->A,NULL); > > > > 77: PetscObjectReference((PetscObject > > )mglevels[i]->A); > > > > 78: > > } > > > > 79: } > > It seems to me that the pointer mglevels[i]->A is only set once, when it is undefined at the beginning of the iteration scheme. I think there are more issues as well, but this is the easiest one to point out. (For example, it doesn't seem like it is possible to update the operator for mglevels[i]->smoothd from KSPSetOperators on the outermost KSP object, and, if I try to set it manually via PCMGGetSmoother, I get errors if the nonzero structures of the coarse-grid operators have changed, which tends to be the case when the fine-grid operator's nonzero structure changes). > > > > Is there something I'm doing wrong? Is there a way to do this so that I can reuse PCMG with a new operator? > > > > Thanks! > > > > -- > > Ben Yee > > > > NERS PhD Candidate, University of Michigan > > B.S. Mech. & Nuclear Eng., U.C. Berkeley > > > > > -- > Ben Yee > > NERS PhD Candidate, University of Michigan > B.S. Mech. & Nuclear Eng., U.C. Berkeley From srahul_05 at yahoo.co.in Thu Jan 25 08:38:02 2018 From: srahul_05 at yahoo.co.in (Rahul Samala) Date: Thu, 25 Jan 2018 14:38:02 +0000 (UTC) Subject: [petsc-users] Jacobian matrix construction when dof > 1 References: <396277963.374555.1516891082387.ref@mail.yahoo.com> Message-ID: <396277963.374555.1516891082387@mail.yahoo.com> Hello PetSc users, I am a beginner to PetSc and I am trying to write Jacobian matrix for my problem involving 2 nonlinear equations in 2 unknown variables, solving in a 1D domain.Most of the petsc examples involve one variable and I am finding it difficult to understand how the Jacobian matrix is built when there are more than one variables. For my problem the DOF is 2 (since 2 unknown variables). I am using finite difference and my stencil is 3 (i-1,i,i+1).Jacobian matrix will have 2x2 submatrix elements and with stencil 3, we have to set 4x3=12 values for each grid point. Let the unknown variables are "u,v" and (F1_u)_i-1 represent "partial derivative of residual of equation 1 at grid point 'i' with respect to variable 'u' at i-1". Is the following pseudo-code the correct way of filling jacobian matrix? FormJacobianLocal(DMDALocalInfo *info,Field *sol,Mat J,Mat Jpre, AppCtx *user){? ....... ????MatStencil???? col[2][3],row;????PetscReal?????? v[2][3]; .........for(i=xs;i From bsmith at mcs.anl.gov Thu Jan 25 10:08:14 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Thu, 25 Jan 2018 16:08:14 +0000 Subject: [petsc-users] Jacobian matrix construction when dof > 1 In-Reply-To: <396277963.374555.1516891082387@mail.yahoo.com> References: <396277963.374555.1516891082387.ref@mail.yahoo.com> <396277963.374555.1516891082387@mail.yahoo.com> Message-ID: <6FD1421C-B9CB-40D8-A618-261ECF2DEA9E@anl.gov> > On Jan 25, 2018, at 8:38 AM, Rahul Samala wrote: > > Hello PetSc users, > > I am a beginner to PetSc and I am trying to write Jacobian matrix for my problem involving 2 nonlinear equations in 2 unknown variables, solving in a 1D domain. > Most of the petsc examples involve one variable and I am finding it difficult to understand how the Jacobian matrix is built when there are more than one variables. > For my problem the DOF is 2 (since 2 unknown variables). I am using finite difference and my stencil is 3 (i-1,i,i+1). > Jacobian matrix will have 2x2 submatrix elements and with stencil 3, we have to set 4x3=12 values for each grid point. > Let the unknown variables are "u,v" and (F1_u)_i-1 represent "partial derivative of residual of equation 1 at grid point 'i' with respect to variable 'u' at i-1". > > Is the following pseudo-code the correct way of filling jacobian matrix? > > FormJacobianLocal(DMDALocalInfo *info,Field *sol,Mat J,Mat Jpre, AppCtx *user) > { > ....... > MatStencil col[2][3],row; > PetscReal v[2][3]; > ......... > for(i=xs;i row.i = i; row.c=0; > { // Internal points > col[0][0].i = i-1; col[0][0].c = 0; v[0][0] = jacobian value = (F1_u)_i-1 > col[0][1].i = i; col[0][1].c = 0; v[0][1] = jacobian value = (F1_u)_i > col[0][2].i = i+1; col[0][2].c = 0; v[0][1] = jacobian value = (F1_u)_i+1 Final one should be v[0][2] same below > > col[1][0].i = i-1; col[1][0].c = 1; v[1][0] = jacobian value = (F1_v)_i-1 > col[1][1].i = i; col[1][1].c = 1; v[1][1] = jacobian value = (F1_v)_i > col[1][2].i = i+1; col[1][2].c = 1; v[1][1] = jacobian value = (F1_v)_i+1 > } > MatSetValuesStencil(Jpre,1,&row,6,col,v,INSERT_VALUES); > row.i = i; row.c=1; > { // Internal points > col[0][0].i = i-1; col[0][0].c = 0; v[0][0] = jacobian value = (F2_u)_i-1 > col[0][1].i = i; col[0][1].c = 0; v[0][1] = jacobian value = (F2_u)_i > col[0][2].i = i+1; col[0][2].c = 0; v[0][1] = jacobian value = (F2_u)_i+1 > > col[1][0].i = i-1; col[1][0].c = 1; v[1][0] = jacobian value = (F2_v)_i-1 > col[1][1].i = i; col[1][1].c = 1; v[1][1] = jacobian value = (F2_v)_i > col[1][2].i = i+1; col[1][2].c = 1; v[1][1] = jacobian value = (F2_v)_i+1 > } > MatSetValuesStencil(Jpre,1,&row,6,col,v,INSERT_VALUES); > } Note you can run with -mat_view for a problem with say 4 grid points and then manually check that the entires are correct, I usually do this when implementing a new derivative. Barry > ....... > } > > Thank you, > Rahul. From mfadams at lbl.gov Thu Jan 25 15:16:28 2018 From: mfadams at lbl.gov (Mark Adams) Date: Thu, 25 Jan 2018 16:16:28 -0500 Subject: [petsc-users] with-openmp error with hypre Message-ID: Baky (cc'ed) is getting a strange error on Cori/KNL at NERSC. Using maint it runs fine with -with-openmp=0, it runs fine with -with-openmp=1 and gamg, but with hypre and -with-openmp=1, even running with flat MPI, the solver seems flatline (see attached and notice that the residual starts to creep after a few time steps). Maybe you can suggest a hypre test that I can run? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: log_omp Type: application/octet-stream Size: 346555 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: log_ Type: application/octet-stream Size: 56100 bytes Desc: not available URL: From jhaase1 at nd.edu Thu Jan 25 17:53:53 2018 From: jhaase1 at nd.edu (John Haase) Date: Thu, 25 Jan 2018 18:53:53 -0500 Subject: [petsc-users] Install Petsc from repository "--download-fblaslapack libraries cannot be used" Message-ID: Hello all, I am trying to install an bleeding edge version of Petsc ( https://bitbucket.org/petsc/petsc/pull-requests/843/added-api-snesgetforceiteration/diff). However, when I try to install with the command ./configure --prefix=$PETSC_DIR/../installed --download-hypre=1 --with-ssl=0 --with-debugging=no --with-pic=1 --with-shared-libraries=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-fblaslapack=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 --download-scalapack=1 --download-mumps=1 CC=mpicc CXX=mpicxx FC=mpif90 F77=mpif77 F90=mpif90 CFLAGS='-fPIC -fopenmp' CXXFLAGS='-fPIC -fopenmp' FFLAGS='-fPIC -fopenmp' FCFLAGS='-fPIC -fopenmp' F90FLAGS='-fPIC -fopenmp' F77FLAGS='-fPIC -fopenmp' PETSC_DIR=`pwd` I get the error "--download-fblaslapack libraries cannot be used". Any suggestions? I've attached configure.log in case I'm not reporting something important. Thank you for your help John Regards, John R. Haase jhaase1 at nd.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 2148138 bytes Desc: not available URL: From balay at mcs.anl.gov Thu Jan 25 18:01:24 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 25 Jan 2018 18:01:24 -0600 Subject: [petsc-users] Install Petsc from repository "--download-fblaslapack libraries cannot be used" In-Reply-To: References: Message-ID: > Configure Options: --configModules=PETSc.Configure --optionsModule=config.compilerOptions --download-hypre=1 --with-ssl=0 --with-debugging=no --with-pic=1 --with-shared-libraries=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-fblaslapack=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 --download-scalapack=1 --download-mumps=1 CC=mpicc CXX=mpicxx FC=mpif90 F77=mpif77 F90=mpif90 CFLAGS="-fPIC -fopenmp" CXXFLAGS="-fPIC -fopenmp" FFLAGS="-fPIC -fopenmp" FCFLAGS="-fPIC -fopenmp" F90FLAGS="-fPIC -fopenmp" F77FLAGS="-fPIC -fopenmp" PETSC_DIR=~/project/petsc/petsc use ${HOME} instead of ~ above. > /usr/bin/ld: cannot find -lfblas And delete your build dir i.e 'rm -rf ~/project/petsc/petsc/arch-linux2-c-opt/' And see if the problem persists. Satish On Thu, 25 Jan 2018, John Haase wrote: > Hello all, > > I am trying to install an bleeding edge version of Petsc ( > https://bitbucket.org/petsc/petsc/pull-requests/843/added-api-snesgetforceiteration/diff). > However, when I try to install with the command > > ./configure --prefix=$PETSC_DIR/../installed --download-hypre=1 > --with-ssl=0 --with-debugging=no --with-pic=1 --with-shared-libraries=1 > --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-fblaslapack=1 > --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 > --download-scalapack=1 --download-mumps=1 CC=mpicc CXX=mpicxx FC=mpif90 > F77=mpif77 F90=mpif90 CFLAGS='-fPIC -fopenmp' CXXFLAGS='-fPIC -fopenmp' > FFLAGS='-fPIC -fopenmp' FCFLAGS='-fPIC -fopenmp' F90FLAGS='-fPIC -fopenmp' > F77FLAGS='-fPIC -fopenmp' PETSC_DIR=`pwd` > > I get the error "--download-fblaslapack libraries cannot be used". Any > suggestions? > > I've attached configure.log in case I'm not reporting something important. > > Thank you for your help > > John > > Regards, > > John R. Haase > jhaase1 at nd.edu > From jhaase1 at nd.edu Thu Jan 25 19:01:38 2018 From: jhaase1 at nd.edu (John Haase) Date: Thu, 25 Jan 2018 20:01:38 -0500 Subject: [petsc-users] Install Petsc from repository "--download-fblaslapack libraries cannot be used" In-Reply-To: References: Message-ID: Satish, I ran the command PETSC_DIR="${HOME}/projects/petsc/petsc" > PETSC_ARCH="gcc" > ./configure \ > --configModules=PETSc.Configure \ > --optionsModule=config.compilerOptions \ > --download-hypre=1 \ > --with-ssl=0 \ > --with-debugging=no \ > --with-pic=1 \ > --with-shared-libraries=1 \ > --with-cc=mpicc \ > --with-cxx=mpicxx \ > --with-fc=mpif90 \ > --download-fblaslapack=1 \ > --download-metis=1 \ > --download-parmetis=1 \ > --download-superlu_dist=1 \ > --download-scalapack=1 \ > --download-mumps=1 \ > CC=mpicc CXX=mpicxx FC=mpif90 F77=mpif77 F90=mpif90 \ > CFLAGS="-fPIC -fopenmp" CXXFLAGS="-fPIC -fopenmp" \ > FFLAGS="-fPIC -fopenmp" FCFLAGS="-fPIC -fopenmp" \ > F90FLAGS="-fPIC -fopenmp" F77FLAGS="-fPIC -fopenmp" \ > PETSC_DIR=${HOME}/projects/petsc/petsc The process threw errors after the 2nd make statement, but still gives the 3rd make statement. However, the third make statement throws a ton of errors. See attached console output and configure.log Regards, John R. Haase jhaase1 at nd.edu On Thu, Jan 25, 2018 at 7:01 PM, Satish Balay wrote: > > Configure Options: --configModules=PETSc.Configure > --optionsModule=config.compilerOptions --download-hypre=1 --with-ssl=0 > --with-debugging=no --with-pic=1 --with-shared-libraries=1 --with-cc=mpicc > --with-cxx=mpicxx --with-fc=mpif90 --download-fblaslapack=1 > --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 > --download-scalapack=1 --download-mumps=1 CC=mpicc CXX=mpicxx FC=mpif90 > F77=mpif77 F90=mpif90 CFLAGS="-fPIC -fopenmp" CXXFLAGS="-fPIC -fopenmp" > FFLAGS="-fPIC -fopenmp" FCFLAGS="-fPIC -fopenmp" F90FLAGS="-fPIC -fopenmp" > F77FLAGS="-fPIC -fopenmp" PETSC_DIR=~/project/petsc/petsc > > > use ${HOME} instead of ~ above. > > > /usr/bin/ld: cannot find -lfblas > > And delete your build dir i.e 'rm -rf ~/project/petsc/petsc/arch- > linux2-c-opt/' > > And see if the problem persists. > > Satish > > On Thu, 25 Jan 2018, John Haase wrote: > > > Hello all, > > > > I am trying to install an bleeding edge version of Petsc ( > > https://bitbucket.org/petsc/petsc/pull-requests/843/added- > api-snesgetforceiteration/diff). > > However, when I try to install with the command > > > > ./configure --prefix=$PETSC_DIR/../installed --download-hypre=1 > > --with-ssl=0 --with-debugging=no --with-pic=1 --with-shared-libraries=1 > > --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > --download-fblaslapack=1 > > --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 > > --download-scalapack=1 --download-mumps=1 CC=mpicc CXX=mpicxx FC=mpif90 > > F77=mpif77 F90=mpif90 CFLAGS='-fPIC -fopenmp' CXXFLAGS='-fPIC -fopenmp' > > FFLAGS='-fPIC -fopenmp' FCFLAGS='-fPIC -fopenmp' F90FLAGS='-fPIC > -fopenmp' > > F77FLAGS='-fPIC -fopenmp' PETSC_DIR=`pwd` > > > > I get the error "--download-fblaslapack libraries cannot be used". Any > > suggestions? > > > > I've attached configure.log in case I'm not reporting something > important. > > > > Thank you for your help > > > > John > > > > Regards, > > > > John R. Haase > > jhaase1 at nd.edu > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- petsc $ PETSC_DIR="${HOME}/projects/petsc/petsc" petsc $ PETSC_ARCH="gcc" petsc $ petsc $ ./configure \ > --configModules=PETSc.Configure \ > --optionsModule=config.compilerOptions \ > --download-hypre=1 \ > --with-ssl=0 \ > --with-debugging=no \ > --with-pic=1 \ > --with-shared-libraries=1 \ > --with-cc=mpicc \ > --with-cxx=mpicxx \ > --with-fc=mpif90 \ > --download-fblaslapack=1 \ > --download-metis=1 \ > --download-parmetis=1 \ > --download-superlu_dist=1 \ > --download-scalapack=1 \ > --download-mumps=1 \ > CC=mpicc CXX=mpicxx FC=mpif90 F77=mpif77 F90=mpif90 \ > CFLAGS="-fPIC -fopenmp" CXXFLAGS="-fPIC -fopenmp" \ > FFLAGS="-fPIC -fopenmp" FCFLAGS="-fPIC -fopenmp" \ > F90FLAGS="-fPIC -fopenmp" F77FLAGS="-fPIC -fopenmp" \ > PETSC_DIR=${HOME}/projects/petsc/petsc =============================================================================== Configuring PETSc to compile on your system =============================================================================== =============================================================================== ***** WARNING: Using default optimization C flags -g -O You might consider manually setting optimal optimization flags for your system with COPTFLAGS="optimization flags" see config/examples/arch-*-opt.py for examples =============================================================================== =============================================================================== ***** WARNING: Using default C++ optimization flags -g -O You might consider manually setting optimal optimization flags for your system with CXXOPTFLAGS="optimization flags" see config/examples/arch-*-opt.py for examples =============================================================================== =============================================================================== ***** WARNING: Using default FORTRAN optimization flags -g -O You might consider manually setting optimal optimization flags for your system with FOPTFLAGS="optimization flags" see config/examples/arch-*-opt.py for examples =============================================================================== =============================================================================== WARNING! Compiling PETSc with no debugging, this should only be done for timing and production runs. All development should be done when configured using --with-debugging=1 =============================================================================== =============================================================================== It appears you do not have valgrind installed on your system. We HIGHLY recommend you install it from www.valgrind.org Or install valgrind-devel or equivalent using your package manager. Then rerun ./configure =============================================================================== =============================================================================== Trying to download git://https://bitbucket.org/petsc/pkg-sowing.git for SOWING =============================================================================== =============================================================================== Running configure on SOWING; this may take several minutes =============================================================================== =============================================================================== Running make on SOWING; this may take several minutes =============================================================================== =============================================================================== Running make install on SOWING; this may take several minutes =============================================================================== =============================================================================== Trying to download git://https://bitbucket.org/petsc/pkg-metis.git for METIS =============================================================================== =============================================================================== Configuring METIS with cmake, this may take several minutes =============================================================================== =============================================================================== Compiling and installing METIS; this may take several minutes =============================================================================== =============================================================================== Trying to download git://https://bitbucket.org/petsc/pkg-parmetis.git for PARMETIS =============================================================================== =============================================================================== Configuring PARMETIS with cmake, this may take several minutes =============================================================================== =============================================================================== Compiling and installing PARMETIS; this may take several minutes =============================================================================== =============================================================================== Trying to download http://ftp.mcs.anl.gov/pub/petsc/externalpackages/fblaslapack-3.4.2.tar.gz for FBLASLAPACK =============================================================================== =============================================================================== Compiling FBLASLAPACK; this may take several minutes =============================================================================== =============================================================================== Trying to download git://https://github.com/LLNL/hypre for HYPRE =============================================================================== =============================================================================== Running configure on HYPRE; this may take several minutes =============================================================================== =============================================================================== Running make on HYPRE; this may take several minutes =============================================================================== =============================================================================== Running make install on HYPRE; this may take several minutes =============================================================================== =============================================================================== Trying to download git://https://github.com/xiaoyeli/superlu_dist for SUPERLU_DIST =============================================================================== =============================================================================== Configuring SUPERLU_DIST with cmake, this may take several minutes =============================================================================== =============================================================================== Compiling and installing SUPERLU_DIST; this may take several minutes =============================================================================== =============================================================================== Trying to download http://www.netlib.org/scalapack/scalapack-2.0.2.tgz for SCALAPACK =============================================================================== =============================================================================== Compiling and installing Scalapack; this may take several minutes =============================================================================== =============================================================================== Trying to download git://https://bitbucket.org/petsc/pkg-mumps.git for MUMPS =============================================================================== =============================================================================== Compiling Mumps; this may take several minutes =============================================================================== =============================================================================== Installing Mumps; this may take several minutes =============================================================================== Compilers: C Compiler: mpicc -fPIC -fopenmp -g -O C++ Compiler: mpicxx -fPIC -fopenmp -g -O Fortran Compiler: mpif90 -fPIC -fopenmp -g -O Linkers: Shared linker: mpicc -shared -fPIC -fopenmp -g -O Dynamic linker: mpicc -shared -fPIC -fopenmp -g -O make: BLAS/LAPACK: -Wl,-rpath,${HOME}/projects/petsc/petsc/arch-linux2-c-opt/lib -L${HOME}/projects/petsc/petsc/arch-linux2-c-opt/lib -lflapack -Wl,-rpath,${HOME}/projects/petsc/petsc/arch-linux2-c-opt/lib -L${HOME}/projects/petsc/petsc/arch-linux2-c-opt/lib -lfblas MPI: fblaslapack: cmake: Arch: metis: Includes: -I${HOME}/projects/petsc/petsc/arch-linux2-c-opt/include Library: -Wl,-rpath,${HOME}/projects/petsc/petsc/arch-linux2-c-opt/lib -L${HOME}/projects/petsc/petsc/arch-linux2-c-opt/lib -lmetis parmetis: Includes: -I${HOME}/projects/petsc/petsc/arch-linux2-c-opt/include Library: -Wl,-rpath,${HOME}/projects/petsc/petsc/arch-linux2-c-opt/lib -L${HOME}/projects/petsc/petsc/arch-linux2-c-opt/lib -lparmetis X: Library: -lX11 MUMPS: Includes: -I${HOME}/projects/petsc/petsc/arch-linux2-c-opt/include Library: -Wl,-rpath,${HOME}/projects/petsc/petsc/arch-linux2-c-opt/lib -L${HOME}/projects/petsc/petsc/arch-linux2-c-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord scalapack: Library: -Wl,-rpath,${HOME}/projects/petsc/petsc/arch-linux2-c-opt/lib -L${HOME}/projects/petsc/petsc/arch-linux2-c-opt/lib -lscalapack SuperLU_DIST: Includes: -I${HOME}/projects/petsc/petsc/arch-linux2-c-opt/include Library: -Wl,-rpath,${HOME}/projects/petsc/petsc/arch-linux2-c-opt/lib -L${HOME}/projects/petsc/petsc/arch-linux2-c-opt/lib -lsuperlu_dist hypre: Includes: -I${HOME}/projects/petsc/petsc/arch-linux2-c-opt/include Library: -Wl,-rpath,${HOME}/projects/petsc/petsc/arch-linux2-c-opt/lib -L${HOME}/projects/petsc/petsc/arch-linux2-c-opt/lib -lHYPRE pthread: sowing: PETSc: PETSC_ARCH: arch-linux2-c-opt PETSC_DIR: ${HOME}/projects/petsc/petsc Scalar type: real Precision: double Clanguage: C Integer size: 32 shared libraries: enabled Memory alignment: 16 xxx=========================================================================xxx Configure stage complete. Now build PETSc libraries with (gnumake build): make PETSC_DIR=${HOME}/projects/petsc/petsc PETSC_ARCH=arch-linux2-c-opt all xxx=========================================================================xxx petsc $ make PETSC_DIR=${HOME}/projects/petsc/petsc PETSC_ARCH=arch-linux2-c-opt all gmake[1]: Entering directory `${HOME}/projects/petsc/petsc' ========================================== See documentation/faq.html and documentation/bugreporting.html for help with installation problems. Please send EVERYTHING printed out below when reporting problems. Please check the mailing list archives and consider subscribing. http://www.mcs.anl.gov/petsc/miscellaneous/mailing-lists.html ========================================== Starting on crcfe01.crc.nd.edu at Thu Jan 25 19:45:31 EST 2018 Machine characteristics: Linux crcfe01.crc.nd.edu 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Dec 28 14:23:39 EST 2017 x86_64 x86_64 x86_64 GNU/Linux ----------------------------------------- Using PETSc directory: ${HOME}/projects/petsc/petsc Using PETSc arch: arch-linux2-c-opt ----------------------------------------- PETSC_VERSION_RELEASE 1 PETSC_VERSION_MAJOR 3 PETSC_VERSION_MINOR 8 PETSC_VERSION_SUBMINOR 3 PETSC_VERSION_PATCH 0 PETSC_VERSION_DATE "unknown" PETSC_VERSION_GIT "unknown" PETSC_VERSION_DATE_GIT "unknown" PETSC_VERSION_EQ(MAJOR,MINOR,SUBMINOR) \ PETSC_VERSION_ PETSC_VERSION_EQ PETSC_VERSION_LT(MAJOR,MINOR,SUBMINOR) \ PETSC_VERSION_LE(MAJOR,MINOR,SUBMINOR) \ PETSC_VERSION_GT(MAJOR,MINOR,SUBMINOR) \ PETSC_VERSION_GE(MAJOR,MINOR,SUBMINOR) \ ----------------------------------------- Using configure Options: --download-hypre=1 --with-ssl=0 --with-debugging=no --with-pic=1 --with-shared-libraries=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-fblaslapack=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 --download-scalapack=1 --download-mumps=1 CC=mpicc CXX=mpicxx FC=mpif90 F77=mpif77 F90=mpif90 CFLAGS="-fPIC -fopenmp" CXXFLAGS="-fPIC -fopenmp" FFLAGS="-fPIC -fopenmp" FCFLAGS="-fPIC -fopenmp" F90FLAGS="-fPIC -fopenmp" F77FLAGS="-fPIC -fopenmp" PETSC_DIR=${HOME}/projects/petsc/petsc Using configuration flags: #define INCLUDED_PETSCCONF_H #define IS_COLORING_MAX 65535 #define STDC_HEADERS 1 #define MPIU_COLORING_VALUE MPI_UNSIGNED_SHORT #define PETSC_HAVE_CXX 1 #define PETSC_UINTPTR_T uintptr_t #define PETSC_HAVE_PTHREAD 1 #define PETSC_DEPRECATED(why) __attribute((deprecated)) #define PETSC_REPLACE_DIR_SEPARATOR '\\' #define PETSC_HAVE_SO_REUSEADDR 1 #define PETSC_HAVE_MPI 1 #define PETSC_PREFETCH_HINT_T2 _MM_HINT_T2 #define PETSC_PREFETCH_HINT_T0 _MM_HINT_T0 #define PETSC_PREFETCH_HINT_T1 _MM_HINT_T1 #define PETSC_HAVE_FORTRAN 1 #define PETSC_DIR "${HOME}/projects/petsc/petsc" #define PETSC_HAVE_X 1 #define PETSC_HAVE_HYPRE 1 #define PETSC_LIB_DIR "${HOME}/projects/petsc/petsc/arch-linux2-c-opt/lib" #define PETSC_HAVE_SUPERLU_DIST 1 #define PETSC_HAVE_PARMETIS 1 #define PETSC_USE_SOCKET_VIEWER 1 #define PETSC_USE_ISATTY 1 #define PETSC_HAVE_SOWING 1 #define PETSC_SLSUFFIX "so" #define PETSC_FUNCTION_NAME_CXX __func__ #define PETSC_HAVE_MUMPS 1 #define PETSC_HAVE_FBLASLAPACK 1 #define PETSC_HAVE_ATOLL 1 #define PETSC_HAVE_ATTRIBUTEALIGNED 1 #define PETSC_HAVE_DOUBLE_ALIGN_MALLOC 1 #define PETSC_UNUSED __attribute((unused)) #define PETSC_ATTRIBUTEALIGNED(size) __attribute((aligned (size))) #define PETSC_FUNCTION_NAME_C __func__ #define PETSC_USE_SINGLE_LIBRARY 1 #define PETSC_HAVE_BUILTIN_EXPECT 1 #define PETSC_HAVE_METIS 1 #define PETSC_DIR_SEPARATOR '/' #define PETSC_PATH_SEPARATOR ':' #define PETSC_HAVE_XMMINTRIN_H 1 #define PETSC_PREFETCH_HINT_NTA _MM_HINT_NTA #define PETSC_Prefetch(a,b,c) _mm_prefetch((const char*)(a),(c)) #define PETSC_HAVE_BLASLAPACK 1 #define PETSC_HAVE_GZIP 1 #define PETSC_HAVE_STRING_H 1 #define PETSC_HAVE_SYS_TYPES_H 1 #define PETSC_HAVE_ENDIAN_H 1 #define PETSC_HAVE_SYS_PROCFS_H 1 #define PETSC_HAVE_DLFCN_H 1 #define PETSC_HAVE_SCHED_H 1 #define PETSC_HAVE_STDINT_H 1 #define PETSC_HAVE_LINUX_KERNEL_H 1 #define PETSC_HAVE_TIME_H 1 #define PETSC_HAVE_MATH_H 1 #define PETSC_HAVE_INTTYPES_H 1 #define PETSC_TIME_WITH_SYS_TIME 1 #define PETSC_HAVE_SYS_PARAM_H 1 #define PETSC_HAVE_PTHREAD_H 1 #define PETSC_HAVE_UNISTD_H 1 #define PETSC_HAVE_STDLIB_H 1 #define PETSC_HAVE_SYS_WAIT_H 1 #define PETSC_HAVE_SETJMP_H 1 #define PETSC_HAVE_LIMITS_H 1 #define PETSC_HAVE_SYS_UTSNAME_H 1 #define PETSC_HAVE_NETINET_IN_H 1 #define PETSC_HAVE_SYS_SOCKET_H 1 #define PETSC_HAVE_FLOAT_H 1 #define PETSC_HAVE_SEARCH_H 1 #define PETSC_HAVE_SYS_RESOURCE_H 1 #define PETSC_HAVE_SYS_TIMES_H 1 #define PETSC_HAVE_NETDB_H 1 #define PETSC_HAVE_MALLOC_H 1 #define PETSC_HAVE_PWD_H 1 #define PETSC_HAVE_FCNTL_H 1 #define PETSC_HAVE_STRINGS_H 1 #define PETSC_HAVE_SYS_SYSINFO_H 1 #define PETSC_HAVE_SYS_TIME_H 1 #define PETSC_USING_F90 1 #define PETSC_USING_F2003 1 #define PETSC_HAVE_RTLD_NOW 1 #define PETSC_HAVE_RTLD_LOCAL 1 #define PETSC_HAVE_RTLD_LAZY 1 #define PETSC_C_STATIC_INLINE static inline #define PETSC_HAVE_FORTRAN_UNDERSCORE 1 #define PETSC_HAVE_CXX_NAMESPACE 1 #define PETSC_HAVE_RTLD_GLOBAL 1 #define PETSC_C_RESTRICT __restrict #define PETSC_CXX_RESTRICT __restrict #define PETSC_CXX_STATIC_INLINE static inline #define PETSC_HAVE_LIBDL 1 #define PETSC_HAVE_LIBGFORTRAN 1 #define PETSC_HAVE_LIBSCALAPACK 1 #define PETSC_HAVE_LIBMETIS 1 #define PETSC_HAVE_LIBMPI_USEMPIF08 1 #define PETSC_HAVE_LIBM 1 #define PETSC_HAVE_LIBMPI_MPIFH 1 #define PETSC_HAVE_LIBDMUMPS 1 #define PETSC_HAVE_LIBMUMPS_COMMON 1 #define PETSC_HAVE_LIBSUPERLU_DIST 1 #define PETSC_HAVE_LIBPARMETIS 1 #define PETSC_HAVE_LIBMPI_USEMPI_IGNORE_TKR 1 #define PETSC_HAVE_LIBFBLAS 1 #define PETSC_HAVE_LIBZMUMPS 1 #define PETSC_HAVE_FENV_H 1 #define PETSC_HAVE_LIBHYPRE 1 #define PETSC_HAVE_LOG2 1 #define PETSC_HAVE_LIBPORD 1 #define PETSC_HAVE_LIBSMUMPS 1 #define PETSC_HAVE_LIBCMUMPS 1 #define PETSC_HAVE_LIBFLAPACK 1 #define PETSC_HAVE_ERF 1 #define PETSC_HAVE_LIBX11 1 #define PETSC_HAVE_LIBQUADMATH 1 #define PETSC_HAVE_TGAMMA 1 #define PETSC_ARCH "arch-linux2-c-opt" #define PETSC_USE_SCALAR_REAL 1 #define PETSC_HAVE_ISINF 1 #define PETSC_HAVE_ISNAN 1 #define PETSC_HAVE_ISNORMAL 1 #define PETSC_USE_REAL_DOUBLE 1 #define PETSC_SIZEOF_MPI_COMM 8 #define PETSC_BITS_PER_BYTE 8 #define PETSC_SIZEOF_MPI_FINT 4 #define PETSC_USE_VISIBILITY_C 1 #define PETSC_SIZEOF_VOID_P 8 #define PETSC_RETSIGTYPE void #define PETSC_HAVE_CXX_COMPLEX 1 #define PETSC_SIZEOF_LONG 8 #define PETSC_USE_FORTRANKIND 1 #define PETSC_USE_VISIBILITY_CXX 1 #define PETSC_SIZEOF_SIZE_T 8 #define PETSC_HAVE_SIGINFO_T 1 #define PETSC_SIZEOF_CHAR 1 #define PETSC_SIZEOF_DOUBLE 8 #define PETSC_SIZEOF_FLOAT 4 #define PETSC_HAVE_C99_COMPLEX 1 #define PETSC_SIZEOF_INT 4 #define PETSC_SIZEOF_LONG_LONG 8 #define PETSC_SIZEOF_SHORT 2 #define PETSC_CLANGUAGE_C 1 #define PETSC_USE_ERRORCHECKING 1 #define PETSC_HAVE_STRCASECMP 1 #define PETSC_HAVE_GET_NPROCS 1 #define PETSC_HAVE_POPEN 1 #define PETSC_HAVE_SIGSET 1 #define PETSC_HAVE_GETWD 1 #define PETSC_HAVE_VSNPRINTF 1 #define PETSC_HAVE_TIMES 1 #define PETSC_HAVE_DLSYM 1 #define PETSC_HAVE_SNPRINTF 1 #define PETSC_HAVE_GETHOSTBYNAME 1 #define PETSC_HAVE_GETCWD 1 #define PETSC_HAVE_DLERROR 1 #define PETSC_HAVE_FORK 1 #define PETSC_HAVE_RAND 1 #define PETSC_HAVE_GETTIMEOFDAY 1 #define PETSC_HAVE_DLCLOSE 1 #define PETSC_HAVE_UNAME 1 #define PETSC_HAVE_GETHOSTNAME 1 #define PETSC_HAVE_MKSTEMP 1 #define PETSC_HAVE_SIGACTION 1 #define PETSC_HAVE_DRAND48 1 #define PETSC_HAVE_MEMALIGN 1 #define PETSC_HAVE_VA_COPY 1 #define PETSC_HAVE_CLOCK 1 #define PETSC_HAVE_ACCESS 1 #define PETSC_HAVE_SIGNAL 1 #define PETSC_HAVE_USLEEP 1 #define PETSC_HAVE_GETRUSAGE 1 #define PETSC_HAVE_VFPRINTF 1 #define PETSC_HAVE_NANOSLEEP 1 #define PETSC_HAVE_GETDOMAINNAME 1 #define PETSC_HAVE_TIME 1 #define PETSC_HAVE_LSEEK 1 #define PETSC_HAVE_SOCKET 1 #define PETSC_HAVE_SYSINFO 1 #define PETSC_HAVE_READLINK 1 #define PETSC_HAVE_REALPATH 1 #define PETSC_HAVE_DLOPEN 1 #define PETSC_HAVE_MEMMOVE 1 #define PETSC_HAVE__GFORTRAN_IARGC 1 #define PETSC_SIGNAL_CAST #define PETSC_HAVE_SLEEP 1 #define PETSC_HAVE_VPRINTF 1 #define PETSC_HAVE_BZERO 1 #define PETSC_HAVE_GETPAGESIZE 1 #define PETSC_WRITE_MEMORY_BARRIER() asm volatile("sfence":::"memory") #define PETSC_MEMORY_BARRIER() asm volatile("mfence":::"memory") #define PETSC_READ_MEMORY_BARRIER() asm volatile("lfence":::"memory") #define PETSC_CPU_RELAX() asm volatile("rep; nop" ::: "memory") #define PETSC_HAVE_DYNAMIC_LIBRARIES 1 #define PETSC_HAVE_SHARED_LIBRARIES 1 #define PETSC_USE_SHARED_LIBRARIES 1 #define PETSC_USE_GDB_DEBUGGER 1 #define PETSC_VERSION_DATE_GIT "2018-01-23 10:47:40 -0600" #define PETSC_VERSION_BRANCH_GIT "maint" #define PETSC_VERSION_GIT "v3.8.3-25-g0e761c6" #define PETSC_BLASLAPACK_UNDERSCORE 1 #define PETSC_HAVE_MPI_F90MODULE 1 #define PETSC_HAVE_MPI_IALLREDUCE 1 #define PETSC_HAVE_OMPI_MINOR_VERSION 0 #define PETSC_HAVE_MPI_REDUCE_SCATTER_BLOCK 1 #define PETSC_HAVE_MPI_IN_PLACE 1 #define PETSC_HAVE_MPI_COMM_C2F 1 #define PETSC_HAVE_MPI_GET_LIBRARY_VERSION 1 #define PETSC_HAVE_MPI_COMBINER_CONTIGUOUS 1 #define PETSC_HAVE_MPI_INT64_T 1 #define PETSC_HAVE_MPI_TYPE_GET_EXTENT 1 #define PETSC_HAVE_MPI_WIN_CREATE 1 #define PETSC_HAVE_MPI_TYPE_DUP 1 #define PETSC_HAVE_MPI_INIT_THREAD 1 #define PETSC_HAVE_MPI_COMBINER_NAMED 1 #define PETSC_HAVE_MPI_LONG_DOUBLE 1 #define PETSC_HAVE_MPI_COMM_F2C 1 #define PETSC_HAVE_MPI_TYPE_GET_ENVELOPE 1 #define PETSC_HAVE_OMPI_MAJOR_VERSION 2 #define PETSC_HAVE_MPI_REDUCE_SCATTER 1 #define PETSC_HAVE_MPI_COMBINER_DUP 1 #define PETSC_HAVE_MPIIO 1 #define PETSC_HAVE_MPI_COMM_SPAWN 1 #define PETSC_HAVE_MPI_FINT 1 #define PETSC_HAVE_MPI_IBARRIER 1 #define PETSC_HAVE_MPI_ALLTOALLW 1 #define PETSC_HAVE_OMPI_RELEASE_VERSION 1 #define PETSC_HAVE_MPI_REDUCE_LOCAL 1 #define PETSC_HAVE_MPI_REPLACE 1 #define PETSC_HAVE_MPI_EXSCAN 1 #define PETSC_HAVE_MPI_C_DOUBLE_COMPLEX 1 #define PETSC_HAVE_MPI_FINALIZED 1 #define PETSC_USE_INFO 1 #define PETSC_Alignx(a,b) #define PETSC_USE_BACKWARD_LOOP 1 #define PETSC_USE_MALLOC_COALESCED 1 #define PETSC_USE_LOG 1 #define PETSC_IS_COLOR_VALUE_TYPE_F integer2 #define PETSC_IS_COLOR_VALUE_TYPE short #define PETSC_USE_CTABLE 1 #define PETSC_MEMALIGN 16 #define PETSC_LEVEL1_DCACHE_LINESIZE 64 #define PETSC_LEVEL1_DCACHE_SIZE 32768 #define PETSC_LEVEL1_DCACHE_ASSOC 8 #define PETSC__GNU_SOURCE 1 #define PETSC__BSD_SOURCE 1 #define PETSC__DEFAULT_SOURCE 1 #define PETSC_HAVE_FORTRAN_GET_COMMAND_ARGUMENT 1 #define PETSC_HAVE_GFORTRAN_IARGC 1 #define PETSC_USE_PROC_FOR_SIZE 1 #define PETSC_HAVE_SCHED_CPU_SET_T 1 #define PETSC_HAVE_PTHREAD_BARRIER_T 1 #define PETSC_HAVE_SYS_SYSCTL_H 1 ----------------------------------------- Using C/C++ compile: mpicc -c -fPIC -fopenmp -g -O -I${HOME}/projects/petsc/petsc/include -I${HOME}/projects/petsc/petsc/arch-linux2-c-opt/include mpicc -show: gcc -I/opt/crc/o/openmpi/2.0.1/gcc/6.2.0/eth/include -pthread -Wl,-rpath -Wl,/opt/crc/o/openmpi/2.0.1/gcc/6.2.0/eth/lib -Wl,--enable-new-dtags -L/opt/crc/o/openmpi/2.0.1/gcc/6.2.0/eth/lib -lmpi Using Fortran compile: mpif90 -c -fPIC -fopenmp -g -O -I${HOME}/projects/petsc/petsc/include -I${HOME}/projects/petsc/petsc/arch-linux2-c-opt/include mpif90 -show: gfortran -I/opt/crc/o/openmpi/2.0.1/gcc/6.2.0/eth/include -pthread -I/opt/crc/o/openmpi/2.0.1/gcc/6.2.0/eth/lib -Wl,-rpath -Wl,/opt/crc/o/openmpi/2.0.1/gcc/6.2.0/eth/lib -Wl,--enable-new-dtags -L/opt/crc/o/openmpi/2.0.1/gcc/6.2.0/eth/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi ----------------------------------------- Using C/C++ linker: mpicc Using C/C++ flags: -fPIC -fopenmp -g -O Using Fortran linker: mpif90 Using Fortran flags: -fPIC -fopenmp -g -O ----------------------------------------- Using system modules: CRC_default/1.0:gcc/6.2.0:ompi/2.0.1-gcc-6.2.0:git/2.8:python/2.7.13:cmake/3.9.4:paraview/5.3.0 Using mpi.h: # 1 "/opt/crc/o/openmpi/2.0.1/gcc/6.2.0/eth/include/mpi.h" 1 ----------------------------------------- Using libraries: -Wl,-rpath,${HOME}/projects/petsc/petsc/arch-linux2-c-opt/lib -L${HOME}/projects/petsc/petsc/arch-linux2-c-opt/lib -Wl,-rpath,${HOME}/projects/petsc/petsc/arch-linux2-c-opt/lib -Wl,-rpath,/opt/crc/o/openmpi/2.0.1/gcc/6.2.0/eth/lib -L/opt/crc/o/openmpi/2.0.1/gcc/6.2.0/eth/lib -Wl,-rpath,/afs/crc.nd.edu/x86_64_linux/g/gcc/6.2.0/lib/gcc/x86_64-pc-linux-gnu/6.2.0 -L/afs/crc.nd.edu/x86_64_linux/g/gcc/6.2.0/lib/gcc/x86_64-pc-linux-gnu/6.2.0 -Wl,-rpath,/afs/crc.nd.edu/x86_64_linux/g/gcc/6.2.0/lib/gcc -L/afs/crc.nd.edu/x86_64_linux/g/gcc/6.2.0/lib/gcc -Wl,-rpath,/afs/crc.nd.edu/x86_64_linux/g/gcc/6.2.0/lib64 -L/afs/crc.nd.edu/x86_64_linux/g/gcc/6.2.0/lib64 -Wl,-rpath,/afs/crc.nd.edu/x86_64_linux/g/gcc/6.2.0/lib -L/afs/crc.nd.edu/x86_64_linux/g/gcc/6.2.0/lib -lpetsc -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_dist -lHYPRE -lflapack -lfblas -lparmetis -lmetis -lX11 -lm -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lgfortran -lm -lgfortran -lm -lquadmath -lstdc++ -lm -Wl,-rpath,/opt/crc/o/openmpi/2.0.1/gcc/6.2.0/eth/lib -L/opt/crc/o/openmpi/2.0.1/gcc/6.2.0/eth/lib -Wl,-rpath,/afs/crc.nd.edu/x86_64_linux/g/gcc/6.2.0/lib/gcc/x86_64-pc-linux-gnu/6.2.0 -L/afs/crc.nd.edu/x86_64_linux/g/gcc/6.2.0/lib/gcc/x86_64-pc-linux-gnu/6.2.0 -Wl,-rpath,/afs/crc.nd.edu/x86_64_linux/g/gcc/6.2.0/lib/gcc -L/afs/crc.nd.edu/x86_64_linux/g/gcc/6.2.0/lib/gcc -Wl,-rpath,/afs/crc.nd.edu/x86_64_linux/g/gcc/6.2.0/lib64 -L/afs/crc.nd.edu/x86_64_linux/g/gcc/6.2.0/lib64 -Wl,-rpath,/afs/crc.nd.edu/x86_64_linux/g/gcc/6.2.0/lib -L/afs/crc.nd.edu/x86_64_linux/g/gcc/6.2.0/lib -ldl -Wl,-rpath,/opt/crc/o/openmpi/2.0.1/gcc/6.2.0/eth/lib -lmpi -lgomp -lgcc_s -lpthread -ldl ------------------------------------------ Using mpiexec: mpiexec ========================================== Building PETSc using GNU Make with 18 build threads ========================================== gmake[2]: Entering directory `${HOME}/projects/petsc/petsc' /opt/crc/p/python/2.7.13/gcc/6.2.0/bin/python ./config/gmakegen.py --petsc-arch=arch-linux2-c-opt /opt/crc/p/python/2.7.13/gcc/6.2.0/bin/python ./config/gmakegentest.py --petsc-arch=arch-linux2-c-opt gmake[2]: Leaving directory `${HOME}/projects/petsc/petsc' gmake[2]: Entering directory `${HOME}/projects/petsc/petsc' Use "/usr/bin/gmake V=1" to see verbose compile lines, "/usr/bin/gmake V=0" to suppress. FC arch-linux2-c-opt/obj/src/sys/f90-mod/petscsysmod.o FC arch-linux2-c-opt/obj/src/sys/fsrc/somefort.o CC arch-linux2-c-opt/obj/src/sys/info/ftn-custom/zverboseinfof.o CC arch-linux2-c-opt/obj/src/sys/info/verboseinfo.o FC arch-linux2-c-opt/obj/src/sys/f90-src/fsrc/f90_fwrap.o CC arch-linux2-c-opt/obj/src/sys/info/ftn-auto/verboseinfof.o CC arch-linux2-c-opt/obj/src/sys/ftn-custom/zsys.o CC arch-linux2-c-opt/obj/src/sys/ftn-custom/zutils.o CC arch-linux2-c-opt/obj/src/sys/logging/xmlviewer.o CC arch-linux2-c-opt/obj/src/sys/logging/xmllogevent.o CC arch-linux2-c-opt/obj/src/sys/logging/plog.o CC arch-linux2-c-opt/obj/src/sys/logging/ftn-auto/plogf.o CC arch-linux2-c-opt/obj/src/sys/logging/ftn-custom/zplogf.o CC arch-linux2-c-opt/obj/src/sys/logging/utils/classlog.o CC arch-linux2-c-opt/obj/src/sys/logging/utils/stagelog.o CC arch-linux2-c-opt/obj/src/sys/logging/utils/eventlog.o CC arch-linux2-c-opt/obj/src/sys/logging/utils/stack.o CC arch-linux2-c-opt/obj/src/sys/time/cputime.o CC arch-linux2-c-opt/obj/src/sys/time/fdate.o CC arch-linux2-c-opt/obj/src/sys/time/ftn-auto/cputimef.o CC arch-linux2-c-opt/obj/src/sys/time/ftn-custom/zptimef.o CC arch-linux2-c-opt/obj/src/sys/objects/version.o CC arch-linux2-c-opt/obj/src/sys/objects/gcomm.o CC arch-linux2-c-opt/obj/src/sys/objects/gtype.o CC arch-linux2-c-opt/obj/src/sys/objects/olist.o CC arch-linux2-c-opt/obj/src/sys/objects/pname.o CC arch-linux2-c-opt/obj/src/sys/objects/tagm.o CC arch-linux2-c-opt/obj/src/sys/objects/gcookie.o CC arch-linux2-c-opt/obj/src/sys/objects/destroy.o CC arch-linux2-c-opt/obj/src/sys/objects/inherit.o CC arch-linux2-c-opt/obj/src/sys/objects/pgname.o CC arch-linux2-c-opt/obj/src/sys/objects/options.o CC arch-linux2-c-opt/obj/src/sys/objects/prefix.o CC arch-linux2-c-opt/obj/src/sys/objects/init.o CC arch-linux2-c-opt/obj/src/sys/objects/pinit.o CC arch-linux2-c-opt/obj/src/sys/objects/ptype.o CC arch-linux2-c-opt/obj/src/sys/objects/state.o CC arch-linux2-c-opt/obj/src/sys/objects/aoptions.o CC arch-linux2-c-opt/obj/src/sys/objects/subcomm.o CC arch-linux2-c-opt/obj/src/sys/objects/fcallback.o CC arch-linux2-c-opt/obj/src/sys/objects/ftn-auto/destroyf.o CC arch-linux2-c-opt/obj/src/sys/objects/ftn-auto/gcommf.o CC arch-linux2-c-opt/obj/src/sys/objects/ftn-auto/gcookief.o CC arch-linux2-c-opt/obj/src/sys/objects/ftn-auto/inheritf.o CC arch-linux2-c-opt/obj/src/sys/objects/ftn-auto/optionsf.o CC arch-linux2-c-opt/obj/src/sys/objects/ftn-auto/pinitf.o CC arch-linux2-c-opt/obj/src/sys/objects/ftn-auto/statef.o CC arch-linux2-c-opt/obj/src/sys/objects/ftn-auto/tagmf.o CC arch-linux2-c-opt/obj/src/sys/objects/ftn-custom/zgcommf.o CC arch-linux2-c-opt/obj/src/sys/objects/ftn-custom/zgtype.o CC arch-linux2-c-opt/obj/src/sys/objects/ftn-custom/zoptionsf.o CC arch-linux2-c-opt/obj/src/sys/objects/ftn-custom/zpgnamef.o CC arch-linux2-c-opt/obj/src/sys/objects/ftn-custom/zpnamef.o CC arch-linux2-c-opt/obj/src/sys/objects/ftn-custom/zprefixf.o CC arch-linux2-c-opt/obj/src/sys/objects/ftn-custom/zdestroyf.o CC arch-linux2-c-opt/obj/src/sys/objects/ftn-custom/zstart.o CC arch-linux2-c-opt/obj/src/sys/objects/ftn-custom/zstartf.o CC arch-linux2-c-opt/obj/src/sys/objects/ftn-custom/zversionf.o CC arch-linux2-c-opt/obj/src/sys/objects/ftn-custom/zinheritf.o CC arch-linux2-c-opt/obj/src/sys/objects/ftn-custom/zptypef.o CC arch-linux2-c-opt/obj/src/sys/python/pythonsys.o CC arch-linux2-c-opt/obj/src/sys/python/ftn-custom/zpythonf.o CC arch-linux2-c-opt/obj/src/sys/utils/arch.o CC arch-linux2-c-opt/obj/src/sys/utils/fhost.o CC arch-linux2-c-opt/obj/src/sys/utils/fuser.o CC arch-linux2-c-opt/obj/src/sys/utils/memc.o CC arch-linux2-c-opt/obj/src/sys/utils/mpiu.o CC arch-linux2-c-opt/obj/src/sys/utils/psleep.o CC arch-linux2-c-opt/obj/src/sys/utils/sortd.o CC arch-linux2-c-opt/obj/src/sys/utils/sorti.o CC arch-linux2-c-opt/obj/src/sys/utils/str.o CC arch-linux2-c-opt/obj/src/sys/utils/sortip.o CC arch-linux2-c-opt/obj/src/sys/utils/pbarrier.o CC arch-linux2-c-opt/obj/src/sys/utils/pdisplay.o CC arch-linux2-c-opt/obj/src/sys/utils/ctable.o CC arch-linux2-c-opt/obj/src/sys/utils/psplit.o CC arch-linux2-c-opt/obj/src/sys/utils/select.o CC arch-linux2-c-opt/obj/src/sys/utils/mpimesg.o CC arch-linux2-c-opt/obj/src/sys/utils/sseenabled.o CC arch-linux2-c-opt/obj/src/sys/utils/mpitr.o CC arch-linux2-c-opt/obj/src/sys/utils/mpilong.o CC arch-linux2-c-opt/obj/src/sys/utils/mathinf.o CC arch-linux2-c-opt/obj/src/sys/utils/matheq.o CC arch-linux2-c-opt/obj/src/sys/utils/mpits.o CC arch-linux2-c-opt/obj/src/sys/utils/segbuffer.o CC arch-linux2-c-opt/obj/src/sys/utils/ftn-auto/memcf.o CC arch-linux2-c-opt/obj/src/sys/utils/ftn-auto/mpitsf.o CC arch-linux2-c-opt/obj/src/sys/utils/ftn-auto/mpiuf.o CC arch-linux2-c-opt/obj/src/sys/utils/ftn-auto/psleepf.o CC arch-linux2-c-opt/obj/src/sys/utils/ftn-auto/psplitf.o CC arch-linux2-c-opt/obj/src/sys/utils/ftn-auto/sortdf.o CC arch-linux2-c-opt/obj/src/sys/utils/ftn-auto/sortif.o CC arch-linux2-c-opt/obj/src/sys/utils/ftn-auto/sortipf.o CC arch-linux2-c-opt/obj/src/sys/utils/ftn-custom/zarchf.o CC arch-linux2-c-opt/obj/src/sys/utils/ftn-custom/zstrf.o CC arch-linux2-c-opt/obj/src/sys/utils/ftn-custom/zpbarrierf.o CC arch-linux2-c-opt/obj/src/sys/utils/ftn-custom/zfhostf.o CC arch-linux2-c-opt/obj/src/sys/memory/mal.o CC arch-linux2-c-opt/obj/src/sys/memory/mem.o CC arch-linux2-c-opt/obj/src/sys/memory/mtr.o CC arch-linux2-c-opt/obj/src/sys/memory/mhbw.o CC arch-linux2-c-opt/obj/src/sys/memory/ftn-auto/memf.o CC arch-linux2-c-opt/obj/src/sys/memory/ftn-auto/mtrf.o CC arch-linux2-c-opt/obj/src/sys/memory/ftn-custom/zmtrf.o CC arch-linux2-c-opt/obj/src/sys/dll/dlimpl.o CC arch-linux2-c-opt/obj/src/sys/dll/dl.o CC arch-linux2-c-opt/obj/src/sys/dll/reg.o CC arch-linux2-c-opt/obj/src/sys/dll/ftn-auto/regf.o CC arch-linux2-c-opt/obj/src/sys/totalview/tv_data_display.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/impls/tikz/tikz.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/impls/image/drawimage.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/impls/x/xinit.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/impls/x/ximage.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/impls/x/xcolor.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/impls/x/xops.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/impls/x/xioerr.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/impls/x/xtext.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/impls/x/xtone.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/impls/x/drawopenx.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/impls/x/ftn-custom/zdrawopenxf.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/impls/null/drawnull.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/impls/null/ftn-auto/drawnullf.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/draw.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/dtext.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/dcoor.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/dpoint.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/dmarker.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/dline.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/dpause.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/dflush.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/dsave.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/dclear.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/dmouse.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/dviewp.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/dtri.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/drect.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/dellipse.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/drawreg.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/drawregall.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/ftn-auto/dclearf.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/ftn-auto/dcoorf.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/ftn-auto/dellipsef.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/ftn-auto/dflushf.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/ftn-auto/dlinef.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/ftn-auto/dmarkerf.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/ftn-auto/dmousef.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/ftn-auto/dpausef.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/ftn-auto/dpointf.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/ftn-auto/drawf.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/ftn-auto/drawregf.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/ftn-auto/dsavef.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/ftn-auto/drectf.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/ftn-auto/dtextf.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/ftn-auto/dtrif.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/ftn-auto/dviewpf.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/ftn-custom/zdrawf.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/ftn-custom/zdrawregf.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/ftn-custom/zdtextf.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/interface/ftn-custom/zdtrif.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/utils/lg.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/utils/axis.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/utils/dscatter.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/utils/hists.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/utils/zoom.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/utils/cmap.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/utils/lgc.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/utils/axisc.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/utils/bars.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/utils/image.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/utils/ftn-auto/axiscf.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/utils/ftn-auto/barsf.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/utils/ftn-auto/dscatterf.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/utils/ftn-auto/histsf.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/utils/ftn-auto/lgf.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/utils/ftn-auto/lgcf.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/utils/ftn-custom/zaxisf.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/utils/ftn-custom/zlgcf.o CC arch-linux2-c-opt/obj/src/sys/classes/draw/utils/ftn-custom/zzoomf.o CC arch-linux2-c-opt/obj/src/sys/classes/gll/petscgll.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/impls/binary/binv.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/impls/binary/ftn-auto/binvf.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/impls/binary/f90-custom/zbinvf90.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/impls/binary/ftn-custom/zbinvf.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/impls/draw/drawv.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/impls/draw/ftn-auto/drawvf.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/impls/draw/ftn-custom/zdrawvf.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/impls/socket/send.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/impls/socket/ftn-custom/zsendf.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/impls/vu/petscvu.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/impls/vtk/ftn-custom/zvtkvf.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/impls/vtk/vtkv.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/impls/glvis/glvis.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/impls/glvis/ftn-auto/glvisf.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/impls/ascii/filev.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/impls/ascii/vcreatea.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/impls/ascii/ftn-auto/filevf.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/impls/ascii/ftn-custom/zfilevf.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/impls/ascii/ftn-custom/zvcreatef.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/impls/string/stringv.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/impls/string/ftn-custom/zstringvf.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/interface/view.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/interface/flush.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/interface/viewregall.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/interface/viewreg.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/interface/viewa.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/interface/dlregispetsc.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/interface/viewers.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/interface/dupl.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/interface/ftn-auto/duplf.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/interface/ftn-auto/flushf.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/interface/ftn-auto/viewf.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/interface/ftn-auto/viewregf.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/interface/ftn-custom/zviewaf.o CC arch-linux2-c-opt/obj/src/sys/classes/viewer/interface/ftn-custom/zviewasetf.o CC arch-linux2-c-opt/obj/src/sys/classes/random/impls/rander48/rander48.o CC arch-linux2-c-opt/obj/src/sys/classes/random/impls/rand/rand.o CC arch-linux2-c-opt/obj/src/sys/classes/random/impls/rand48/rand48.o CC arch-linux2-c-opt/obj/src/sys/classes/random/interface/random.o CC arch-linux2-c-opt/obj/src/sys/classes/random/interface/randreg.o CC arch-linux2-c-opt/obj/src/sys/classes/random/interface/dlregisrand.o CC arch-linux2-c-opt/obj/src/sys/classes/random/interface/randomc.o CC arch-linux2-c-opt/obj/src/sys/classes/random/interface/ftn-auto/randomf.o CC arch-linux2-c-opt/obj/src/sys/classes/random/interface/ftn-auto/randomcf.o CC arch-linux2-c-opt/obj/src/sys/classes/random/interface/ftn-custom/zrandomf.o CC arch-linux2-c-opt/obj/src/sys/classes/bag/bag.o CC arch-linux2-c-opt/obj/src/sys/classes/bag/ftn-auto/bagf.o CC arch-linux2-c-opt/obj/src/sys/classes/bag/f90-custom/zbagf90.o CC arch-linux2-c-opt/obj/src/sys/classes/bag/ftn-custom/zbagf.o CC arch-linux2-c-opt/obj/src/sys/fileio/ftest.o CC arch-linux2-c-opt/obj/src/sys/fileio/ghome.o CC arch-linux2-c-opt/obj/src/sys/fileio/mpiuopen.o CC arch-linux2-c-opt/obj/src/sys/fileio/rpath.o CC arch-linux2-c-opt/obj/src/sys/fileio/fpath.o CC arch-linux2-c-opt/obj/src/sys/fileio/fwd.o CC arch-linux2-c-opt/obj/src/sys/fileio/grpath.o CC arch-linux2-c-opt/obj/src/sys/fileio/mprint.o CC arch-linux2-c-opt/obj/src/sys/fileio/sysio.o CC arch-linux2-c-opt/obj/src/sys/fileio/fretrieve.o CC arch-linux2-c-opt/obj/src/sys/fileio/smatlab.o CC arch-linux2-c-opt/obj/src/sys/fileio/fdir.o CC arch-linux2-c-opt/obj/src/sys/fileio/ftn-auto/sysiof.o CC arch-linux2-c-opt/obj/src/sys/fileio/ftn-custom/zghomef.o CC arch-linux2-c-opt/obj/src/sys/fileio/ftn-custom/zmpiuopenf.o CC arch-linux2-c-opt/obj/src/sys/fileio/ftn-custom/zmprintf.o CC arch-linux2-c-opt/obj/src/sys/fileio/ftn-custom/zsysiof.o CC arch-linux2-c-opt/obj/src/sys/error/adebug.o CC arch-linux2-c-opt/obj/src/sys/error/err.o CC arch-linux2-c-opt/obj/src/sys/error/errtrace.o CC arch-linux2-c-opt/obj/src/sys/error/errabort.o CC arch-linux2-c-opt/obj/src/sys/error/errstop.o CC arch-linux2-c-opt/obj/src/sys/error/fp.o CC arch-linux2-c-opt/obj/src/sys/error/signal.o CC arch-linux2-c-opt/obj/src/sys/error/pstack.o CC arch-linux2-c-opt/obj/src/sys/error/checkptr.o CC arch-linux2-c-opt/obj/src/sys/error/ftn-auto/adebugf.o CC arch-linux2-c-opt/obj/src/sys/error/ftn-auto/checkptrf.o CC arch-linux2-c-opt/obj/src/sys/error/ftn-auto/errf.o CC arch-linux2-c-opt/obj/src/sys/error/ftn-auto/fpf.o CC arch-linux2-c-opt/obj/src/sys/error/ftn-auto/signalf.o CC arch-linux2-c-opt/obj/src/sys/error/ftn-custom/zerrf.o CC arch-linux2-c-opt/obj/src/sys/f90-src/f90_cwrap.o FC arch-linux2-c-opt/obj/src/vec/f90-mod/petscvecmod.o CC arch-linux2-c-opt/obj/src/vec/is/utils/isio.o CC arch-linux2-c-opt/obj/src/vec/is/utils/isltog.o CC arch-linux2-c-opt/obj/src/vec/is/utils/pmap.o CC arch-linux2-c-opt/obj/src/vec/is/utils/vsectionis.o CC arch-linux2-c-opt/obj/src/vec/is/utils/ftn-auto/isltogf.o CC arch-linux2-c-opt/obj/src/vec/is/utils/ftn-auto/pmapf.o CC arch-linux2-c-opt/obj/src/vec/is/utils/f90-custom/zisltogf90.o CC arch-linux2-c-opt/obj/src/vec/is/utils/ftn-auto/vsectionisf.o CC arch-linux2-c-opt/obj/src/vec/is/utils/f90-custom/zvsectionisf90.o CC arch-linux2-c-opt/obj/src/vec/is/utils/ftn-custom/zisltogf.o CC arch-linux2-c-opt/obj/src/vec/is/utils/ftn-custom/zvsectionisf.o CC arch-linux2-c-opt/obj/src/vec/is/is/impls/f90-custom/zblockf90.o CC arch-linux2-c-opt/obj/src/vec/is/is/impls/stride/stride.o CC arch-linux2-c-opt/obj/src/vec/is/is/impls/stride/ftn-auto/stridef.o CC arch-linux2-c-opt/obj/src/vec/is/is/impls/block/block.o CC arch-linux2-c-opt/obj/src/vec/is/is/impls/block/ftn-auto/blockf.o CC arch-linux2-c-opt/obj/src/vec/is/is/impls/block/ftn-custom/zblockf.o CC arch-linux2-c-opt/obj/src/vec/is/is/impls/general/general.o CC arch-linux2-c-opt/obj/src/vec/is/is/impls/general/ftn-auto/generalf.o CC arch-linux2-c-opt/obj/src/vec/is/is/interface/index.o CC arch-linux2-c-opt/obj/src/vec/is/is/interface/isregall.o CC arch-linux2-c-opt/obj/src/vec/is/is/interface/isreg.o CC arch-linux2-c-opt/obj/src/vec/is/is/interface/ftn-auto/indexf.o CC arch-linux2-c-opt/obj/src/vec/is/is/interface/ftn-auto/isregf.o CC arch-linux2-c-opt/obj/src/vec/is/is/interface/f90-custom/zindexf90.o CC arch-linux2-c-opt/obj/src/vec/is/is/interface/ftn-custom/zindexf.o CC arch-linux2-c-opt/obj/src/vec/is/is/utils/iscomp.o CC arch-linux2-c-opt/obj/src/vec/is/is/utils/iscoloring.o CC arch-linux2-c-opt/obj/src/vec/is/is/utils/isdiff.o CC arch-linux2-c-opt/obj/src/vec/is/is/utils/isblock.o CC arch-linux2-c-opt/obj/src/vec/is/is/utils/ftn-auto/isblockf.o CC arch-linux2-c-opt/obj/src/vec/is/is/utils/ftn-auto/iscoloringf.o CC arch-linux2-c-opt/obj/src/vec/is/is/utils/ftn-auto/iscompf.o CC arch-linux2-c-opt/obj/src/vec/is/is/utils/ftn-auto/isdifff.o CC arch-linux2-c-opt/obj/src/vec/is/is/utils/f90-custom/ziscoloringf90.o CC arch-linux2-c-opt/obj/src/vec/is/is/utils/ftn-custom/ziscoloringf.o CC arch-linux2-c-opt/obj/src/vec/is/ao/impls/memscalable/aomemscalable.o CC arch-linux2-c-opt/obj/src/vec/is/ao/impls/mapping/aomapping.o CC arch-linux2-c-opt/obj/src/vec/is/ao/impls/mapping/ftn-auto/aomappingf.o CC arch-linux2-c-opt/obj/src/vec/is/ao/impls/mapping/ftn-custom/zaomappingf.o CC arch-linux2-c-opt/obj/src/vec/is/ao/impls/basic/aobasic.o CC arch-linux2-c-opt/obj/src/vec/is/ao/impls/basic/ftn-custom/zaobasicf.o CC arch-linux2-c-opt/obj/src/vec/is/ao/interface/ao.o CC arch-linux2-c-opt/obj/src/vec/is/ao/interface/dlregisdm.o CC arch-linux2-c-opt/obj/src/vec/is/ao/interface/aoreg.o CC arch-linux2-c-opt/obj/src/vec/is/ao/interface/aoregall.o CC arch-linux2-c-opt/obj/src/vec/is/ao/interface/ftn-auto/aof.o CC arch-linux2-c-opt/obj/src/vec/is/ao/interface/ftn-custom/zaof.o CC arch-linux2-c-opt/obj/src/vec/is/sf/impls/window/sfwindow.o CC arch-linux2-c-opt/obj/src/vec/is/sf/impls/basic/sfbasic.o CC arch-linux2-c-opt/obj/src/vec/is/sf/interface/dlregissf.o CC arch-linux2-c-opt/obj/src/vec/is/sf/interface/sfregi.o CC arch-linux2-c-opt/obj/src/vec/is/sf/interface/sf.o CC arch-linux2-c-opt/obj/src/vec/is/sf/interface/sftype.o CC arch-linux2-c-opt/obj/src/vec/is/sf/interface/ftn-auto/sff.o CC arch-linux2-c-opt/obj/src/vec/is/sf/interface/ftn-custom/zsf.o CC arch-linux2-c-opt/obj/src/vec/pf/impls/constant/const.o CC arch-linux2-c-opt/obj/src/vec/pf/impls/string/cstring.o CC arch-linux2-c-opt/obj/src/vec/pf/interface/pf.o CC arch-linux2-c-opt/obj/src/vec/pf/interface/pfall.o CC arch-linux2-c-opt/obj/src/vec/pf/interface/ftn-auto/pff.o CC arch-linux2-c-opt/obj/src/vec/vec/impls/shared/shvec.o CC arch-linux2-c-opt/obj/src/vec/vec/impls/shared/ftn-auto/shvecf.o CC arch-linux2-c-opt/obj/src/vec/vec/impls/hypre/vhyp.o CC arch-linux2-c-opt/obj/src/vec/vec/impls/nest/vecnest.o CC arch-linux2-c-opt/obj/src/vec/vec/impls/nest/ftn-custom/zvecnestf.o CC arch-linux2-c-opt/obj/src/vec/vec/impls/nest/ftn-auto/vecnestf.o CC arch-linux2-c-opt/obj/src/vec/vec/impls/seq/bvec2.o CC arch-linux2-c-opt/obj/src/vec/vec/impls/seq/bvec1.o CC arch-linux2-c-opt/obj/src/vec/vec/impls/seq/vseqcr.o CC arch-linux2-c-opt/obj/src/vec/vec/impls/seq/dvec2.o CC arch-linux2-c-opt/obj/src/vec/vec/impls/seq/bvec3.o CC arch-linux2-c-opt/obj/src/vec/vec/impls/seq/ftn-custom/zbvec2f.o CC arch-linux2-c-opt/obj/src/vec/vec/impls/seq/ftn-custom/zvsectionf.o CC arch-linux2-c-opt/obj/src/vec/vec/impls/mpi/pbvec.o CC arch-linux2-c-opt/obj/src/vec/vec/impls/seq/ftn-auto/vseqcrf.o CC arch-linux2-c-opt/obj/src/vec/vec/impls/mpi/pdvec.o CC arch-linux2-c-opt/obj/src/vec/vec/impls/mpi/pvec2.o CC arch-linux2-c-opt/obj/src/vec/vec/impls/mpi/commonmpvec.o CC arch-linux2-c-opt/obj/src/vec/vec/impls/mpi/vmpicr.o CC arch-linux2-c-opt/obj/src/vec/vec/impls/mpi/ftn-custom/zpbvecf.o CC arch-linux2-c-opt/obj/src/vec/vec/impls/mpi/ftn-auto/commonmpvecf.o CC arch-linux2-c-opt/obj/src/vec/vec/impls/mpi/ftn-auto/pbvecf.o CC arch-linux2-c-opt/obj/src/vec/vec/impls/mpi/ftn-auto/vmpicrf.o CC arch-linux2-c-opt/obj/src/vec/vec/interface/vector.o CC arch-linux2-c-opt/obj/src/vec/vec/interface/veccreate.o CC arch-linux2-c-opt/obj/src/vec/vec/interface/vecregall.o CC arch-linux2-c-opt/obj/src/vec/vec/interface/vecreg.o CC arch-linux2-c-opt/obj/src/vec/vec/interface/dlregisvec.o CC arch-linux2-c-opt/obj/src/vec/vec/interface/rvector.o CC arch-linux2-c-opt/obj/src/vec/vec/interface/ftn-auto/rvectorf.o CC arch-linux2-c-opt/obj/src/vec/vec/interface/ftn-auto/veccreatef.o CC arch-linux2-c-opt/obj/src/vec/vec/interface/ftn-auto/vectorf.o CC arch-linux2-c-opt/obj/src/vec/vec/interface/f90-custom/zvectorf90.o CC arch-linux2-c-opt/obj/src/vec/vec/interface/ftn-custom/zvecregf.o CC arch-linux2-c-opt/obj/src/vec/vec/interface/ftn-custom/zvectorf.o CC arch-linux2-c-opt/obj/src/vec/vec/utils/vinv.o CC arch-linux2-c-opt/obj/src/vec/vec/utils/vscat.o CC arch-linux2-c-opt/obj/src/vec/vec/utils/vpscat.o CC arch-linux2-c-opt/obj/src/vec/vec/utils/vecio.o CC arch-linux2-c-opt/obj/src/vec/vec/utils/comb.o CC arch-linux2-c-opt/obj/src/vec/vec/utils/vecstash.o CC arch-linux2-c-opt/obj/src/vec/vec/utils/vecmpitoseq.o CC arch-linux2-c-opt/obj/src/vec/vec/utils/vecs.o CC arch-linux2-c-opt/obj/src/vec/vec/utils/vsection.o CC arch-linux2-c-opt/obj/src/vec/vec/utils/projection.o CC arch-linux2-c-opt/obj/src/vec/vec/utils/vecglvis.o CC arch-linux2-c-opt/obj/src/vec/vec/utils/ftn-auto/combf.o CC arch-linux2-c-opt/obj/src/vec/vec/utils/ftn-auto/projectionf.o CC arch-linux2-c-opt/obj/src/vec/vec/utils/ftn-auto/veciof.o CC arch-linux2-c-opt/obj/src/vec/vec/utils/ftn-auto/vinvf.o CC arch-linux2-c-opt/obj/src/vec/vec/utils/ftn-auto/vpscatf.o CC arch-linux2-c-opt/obj/src/vec/vec/utils/ftn-auto/vscatf.o CC arch-linux2-c-opt/obj/src/vec/vec/utils/f90-custom/zvsectionf90.o CC arch-linux2-c-opt/obj/src/vec/vec/utils/ftn-custom/zvscatf.o CC arch-linux2-c-opt/obj/src/vec/vec/utils/tagger/impls/simple.o CC arch-linux2-c-opt/obj/src/vec/vec/utils/tagger/impls/absolute.o CC arch-linux2-c-opt/obj/src/vec/vec/utils/tagger/impls/relative.o CC arch-linux2-c-opt/obj/src/vec/vec/utils/tagger/impls/cdf.o CC arch-linux2-c-opt/obj/src/vec/vec/utils/tagger/impls/andor.o CC arch-linux2-c-opt/obj/src/vec/vec/utils/tagger/impls/and.o CC arch-linux2-c-opt/obj/src/vec/vec/utils/tagger/impls/or.o CC arch-linux2-c-opt/obj/src/vec/vec/utils/tagger/interface/tagger.o CC arch-linux2-c-opt/obj/src/vec/vec/utils/tagger/interface/taggerregi.o CC arch-linux2-c-opt/obj/src/vec/vec/utils/tagger/interface/dlregistagger.o CC arch-linux2-c-opt/obj/src/vec/vec/utils/tagger/interface/ftn-auto/taggerf.o CC arch-linux2-c-opt/obj/src/mat/ftn-custom/zmat.o CC arch-linux2-c-opt/obj/src/mat/color/impls/minpack/color.o CC arch-linux2-c-opt/obj/src/mat/color/impls/minpack/degr.o CC arch-linux2-c-opt/obj/src/mat/color/impls/minpack/dsm.o CC arch-linux2-c-opt/obj/src/mat/color/impls/minpack/ido.o CC arch-linux2-c-opt/obj/src/mat/color/impls/minpack/numsrt.o CC arch-linux2-c-opt/obj/src/mat/color/impls/minpack/seq.o CC arch-linux2-c-opt/obj/src/mat/color/impls/minpack/setr.o CC arch-linux2-c-opt/obj/src/mat/color/impls/minpack/slo.o CC arch-linux2-c-opt/obj/src/mat/color/impls/natural/natural.o CC arch-linux2-c-opt/obj/src/mat/color/impls/jp/jp.o CC arch-linux2-c-opt/obj/src/mat/color/impls/power/power.o CC arch-linux2-c-opt/obj/src/mat/color/impls/greedy/greedy.o CC arch-linux2-c-opt/obj/src/mat/color/interface/matcoloring.o CC arch-linux2-c-opt/obj/src/mat/color/interface/ftn-auto/matcoloringf.o CC arch-linux2-c-opt/obj/src/mat/color/interface/matcoloringregi.o CC arch-linux2-c-opt/obj/src/mat/color/interface/ftn-custom/zmatcoloringf.o CC arch-linux2-c-opt/obj/src/mat/color/utils/bipartite.o CC arch-linux2-c-opt/obj/src/mat/color/utils/valid.o CC arch-linux2-c-opt/obj/src/mat/color/utils/weights.o CC arch-linux2-c-opt/obj/src/mat/partition/partition.o CC arch-linux2-c-opt/obj/src/mat/partition/spartition.o CC arch-linux2-c-opt/obj/src/mat/partition/impls/pmetis/pmetis.o CC arch-linux2-c-opt/obj/src/mat/partition/impls/pmetis/ftn-auto/pmetisf.o CC arch-linux2-c-opt/obj/src/mat/partition/impls/hierarchical/hierarchical.o CC arch-linux2-c-opt/obj/src/mat/partition/ftn-auto/partitionf.o CC arch-linux2-c-opt/obj/src/mat/partition/ftn-custom/zpartitionf.o CC arch-linux2-c-opt/obj/src/mat/matfd/fdmatrix.o CC arch-linux2-c-opt/obj/src/mat/matfd/ftn-auto/fdmatrixf.o CC arch-linux2-c-opt/obj/src/mat/matfd/ftn-custom/zfdmatrixf.o CC arch-linux2-c-opt/obj/src/mat/coarsen/coarsen.o CC arch-linux2-c-opt/obj/src/mat/coarsen/scoarsen.o CC arch-linux2-c-opt/obj/src/mat/coarsen/impls/hem/hem.o CC arch-linux2-c-opt/obj/src/mat/coarsen/impls/mis/mis.o CC arch-linux2-c-opt/obj/src/mat/coarsen/ftn-auto/coarsenf.o CC arch-linux2-c-opt/obj/src/mat/interface/matrix.o CC arch-linux2-c-opt/obj/src/mat/interface/mhas.o CC arch-linux2-c-opt/obj/src/mat/interface/matreg.o CC arch-linux2-c-opt/obj/src/mat/interface/matregis.o CC arch-linux2-c-opt/obj/src/mat/interface/matnull.o CC arch-linux2-c-opt/obj/src/mat/interface/dlregismat.o CC arch-linux2-c-opt/obj/src/mat/interface/ftn-auto/matnullf.o CC arch-linux2-c-opt/obj/src/mat/interface/ftn-auto/matrixf.o CC arch-linux2-c-opt/obj/src/mat/interface/ftn-auto/mhasf.o CC arch-linux2-c-opt/obj/src/mat/interface/f90-custom/zmatrixf90.o CC arch-linux2-c-opt/obj/src/mat/interface/ftn-custom/zmatregf.o CC arch-linux2-c-opt/obj/src/mat/interface/ftn-custom/zmatrixf.o CC arch-linux2-c-opt/obj/src/mat/interface/ftn-custom/zmatnullf.o CC arch-linux2-c-opt/obj/src/mat/impls/submat/submat.o CC arch-linux2-c-opt/obj/src/mat/impls/submat/ftn-auto/submatf.o CC arch-linux2-c-opt/obj/src/mat/impls/is/matis.o CC arch-linux2-c-opt/obj/src/mat/impls/is/ftn-auto/matisf.o CC arch-linux2-c-opt/obj/src/mat/impls/fft/fft.o CC arch-linux2-c-opt/obj/src/mat/impls/fft/ftn-custom/zfftf.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/seq/baij.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/seq/baij2.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/seq/baijfact.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/seq/baijfact2.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/seq/dgefa.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/seq/dgedi.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/seq/dgefa3.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/seq/dgefa4.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/seq/dgefa5.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/seq/dgefa2.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/seq/dgefa6.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/seq/dgefa7.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/seq/aijbaij.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/seq/baijfact3.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/seq/baijfact4.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/seq/baijfact5.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/seq/baijfact7.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/seq/baijfact9.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/seq/baijfact13.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/seq/baijfact11.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/seq/baijsolvtrannat.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/seq/baijsolvtran.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/seq/baijsolv.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/seq/baijsolvnat.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/seq/ftn-auto/baijf.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/seq/ftn-custom/zbaijf.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/mpi/mpibaij.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/mpi/mmbaij.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/mpi/baijov.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/mpi/mpb_baij.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/mpi/ftn-auto/mpibaijf.o CC arch-linux2-c-opt/obj/src/mat/impls/baij/mpi/ftn-custom/zmpibaijf.o CC arch-linux2-c-opt/obj/src/mat/impls/dense/seq/dense.o CC arch-linux2-c-opt/obj/src/mat/impls/dense/seq/ftn-custom/zdensef.o CC arch-linux2-c-opt/obj/src/mat/impls/dense/mpi/mpidense.o CC arch-linux2-c-opt/obj/src/mat/impls/dense/mpi/mmdense.o CC arch-linux2-c-opt/obj/src/mat/impls/dense/mpi/ftn-auto/mpidensef.o CC arch-linux2-c-opt/obj/src/mat/impls/dense/mpi/ftn-custom/zmpidensef.o CC arch-linux2-c-opt/obj/src/mat/impls/normal/normmh.o CC arch-linux2-c-opt/obj/src/mat/impls/normal/normm.o CC arch-linux2-c-opt/obj/src/mat/impls/normal/ftn-auto/normmf.o CC arch-linux2-c-opt/obj/src/mat/impls/normal/ftn-auto/normmhf.o CC arch-linux2-c-opt/obj/src/mat/impls/mffd/mffd.o CC arch-linux2-c-opt/obj/src/mat/impls/mffd/mffddef.o CC arch-linux2-c-opt/obj/src/mat/impls/mffd/mfregis.o CC arch-linux2-c-opt/obj/src/mat/impls/mffd/wp.o CC arch-linux2-c-opt/obj/src/mat/impls/mffd/ftn-auto/mffdf.o CC arch-linux2-c-opt/obj/src/mat/impls/mffd/ftn-auto/mffddeff.o CC arch-linux2-c-opt/obj/src/mat/impls/mffd/ftn-auto/wpf.o CC arch-linux2-c-opt/obj/src/mat/impls/mffd/ftn-custom/zmffdf.o CC arch-linux2-c-opt/obj/src/mat/impls/python/pythonmat.o CC arch-linux2-c-opt/obj/src/mat/impls/python/ftn-custom/zpythonmf.o CC arch-linux2-c-opt/obj/src/mat/impls/adj/mpi/mpiadj.o CC arch-linux2-c-opt/obj/src/mat/impls/adj/mpi/ftn-auto/mpiadjf.o CC arch-linux2-c-opt/obj/src/mat/impls/adj/mpi/ftn-custom/zmpiadjf.o CC arch-linux2-c-opt/obj/src/mat/impls/maij/maij.o CC arch-linux2-c-opt/obj/src/mat/impls/maij/ftn-auto/maijf.o CC arch-linux2-c-opt/obj/src/mat/impls/lrc/lrc.o CC arch-linux2-c-opt/obj/src/mat/impls/lrc/ftn-auto/lrcf.o CC arch-linux2-c-opt/obj/src/mat/impls/shell/shell.o CC arch-linux2-c-opt/obj/src/mat/impls/shell/shellcnv.o CC arch-linux2-c-opt/obj/src/mat/impls/shell/ftn-auto/shellf.o CC arch-linux2-c-opt/obj/src/mat/impls/shell/ftn-custom/zshellf.o CC arch-linux2-c-opt/obj/src/mat/impls/hypre/mhypre.o CC arch-linux2-c-opt/obj/src/mat/impls/composite/mcomposite.o CC arch-linux2-c-opt/obj/src/mat/impls/composite/ftn-auto/mcompositef.o CC arch-linux2-c-opt/obj/src/mat/impls/nest/matnest.o CC arch-linux2-c-opt/obj/src/mat/impls/nest/ftn-auto/matnestf.o CC arch-linux2-c-opt/obj/src/mat/impls/nest/ftn-custom/zmatnestf.o CC arch-linux2-c-opt/obj/src/mat/impls/transpose/transm.o CC arch-linux2-c-opt/obj/src/mat/impls/transpose/htransm.o CC arch-linux2-c-opt/obj/src/mat/impls/transpose/ftn-auto/htransmf.o CC arch-linux2-c-opt/obj/src/mat/impls/transpose/ftn-auto/transmf.o CC arch-linux2-c-opt/obj/src/mat/impls/localref/mlocalref.o CC arch-linux2-c-opt/obj/src/mat/impls/localref/ftn-auto/mlocalreff.o CC arch-linux2-c-opt/obj/src/mat/impls/dummy/matdummy.o CC arch-linux2-c-opt/obj/src/mat/impls/blockmat/seq/blockmat.o CC arch-linux2-c-opt/obj/src/mat/impls/sbaij/seq/sbaij.o CC arch-linux2-c-opt/obj/src/mat/impls/sbaij/seq/sbaij2.o CC arch-linux2-c-opt/obj/src/mat/impls/sbaij/seq/sbaijfact.o CC arch-linux2-c-opt/obj/src/mat/impls/sbaij/seq/sbaijfact2.o CC arch-linux2-c-opt/obj/src/mat/impls/sbaij/seq/sro.o CC arch-linux2-c-opt/obj/src/mat/impls/sbaij/seq/sbaijfact3.o CC arch-linux2-c-opt/obj/src/mat/impls/sbaij/seq/sbaijfact4.o CC arch-linux2-c-opt/obj/src/mat/impls/sbaij/seq/sbaijfact5.o CC arch-linux2-c-opt/obj/src/mat/impls/sbaij/seq/sbaijfact6.o CC arch-linux2-c-opt/obj/src/mat/impls/sbaij/seq/sbaijfact7.o CC arch-linux2-c-opt/obj/src/mat/impls/sbaij/seq/sbaijfact8.o CC arch-linux2-c-opt/obj/src/mat/impls/sbaij/seq/sbaijfact9.o CC arch-linux2-c-opt/obj/src/mat/impls/sbaij/seq/sbaijfact10.o CC arch-linux2-c-opt/obj/src/mat/impls/sbaij/seq/sbaijfact11.o CC arch-linux2-c-opt/obj/src/mat/impls/sbaij/seq/sbaijfact12.o CC arch-linux2-c-opt/obj/src/mat/impls/sbaij/seq/aijsbaij.o CC arch-linux2-c-opt/obj/src/mat/impls/sbaij/seq/ftn-auto/sbaijf.o CC arch-linux2-c-opt/obj/src/mat/impls/sbaij/seq/ftn-custom/zsbaijf.o CC arch-linux2-c-opt/obj/src/mat/impls/sbaij/mpi/mpisbaij.o CC arch-linux2-c-opt/obj/src/mat/impls/sbaij/mpi/mmsbaij.o CC arch-linux2-c-opt/obj/src/mat/impls/sbaij/mpi/sbaijov.o CC arch-linux2-c-opt/obj/src/mat/impls/sbaij/mpi/mpiaijsbaij.o CC arch-linux2-c-opt/obj/src/mat/impls/sbaij/mpi/ftn-auto/mpisbaijf.o CC arch-linux2-c-opt/obj/src/mat/impls/sbaij/mpi/ftn-custom/zmpisbaijf.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/seq/aij.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/seq/aijfact.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/seq/ij.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/seq/matmatmult.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/seq/fdaij.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/seq/symtranspose.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/seq/matptap.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/seq/matrart.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/seq/inode.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/seq/inode2.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/seq/matmatmatmult.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/seq/mattransposematmult.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/seq/bas/basfactor.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/seq/bas/spbas.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/seq/ftn-auto/aijf.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/seq/ftn-custom/zaijf.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/seq/csrperm/csrperm.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/seq/crl/crl.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/mpi/mpiaij.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/mpi/mmaij.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/mpi/mpiaijpc.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/mpi/mpiov.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/mpi/fdmpiaij.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/mpi/mpiptap.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/mpi/mpimatmatmult.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/mpi/mpb_aij.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/mpi/mpimatmatmatmult.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/mpi/mpimattransposematmult.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/mpi/ftn-custom/zmpiaijf.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/mpi/csrperm/mpicsrperm.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/mpi/mumps/mumps.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/mpi/mumps/ftn-auto/mumpsf.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/mpi/ftn-auto/mpiaijf.o CC arch-linux2-c-opt/obj/src/mat/impls/aij/mpi/crl/mcrl.o CC arch-linux2-c-opt/obj/src/mat/impls/scatter/mscatter.o CC arch-linux2-c-opt/obj/src/mat/impls/scatter/ftn-auto/mscatterf.o CC arch-linux2-c-opt/obj/src/mat/impls/preallocator/matpreallocator.o CC arch-linux2-c-opt/obj/src/mat/impls/preallocator/ftn-auto/matpreallocatorf.o CC arch-linux2-c-opt/obj/src/mat/order/sp1wd.o CC arch-linux2-c-opt/obj/src/mat/order/spnd.o CC arch-linux2-c-opt/obj/src/mat/order/spqmd.o CC arch-linux2-c-opt/obj/src/mat/order/sprcm.o CC arch-linux2-c-opt/obj/src/mat/order/sorder.o CC arch-linux2-c-opt/obj/src/mat/order/spectral.o CC arch-linux2-c-opt/obj/src/mat/order/sregis.o CC arch-linux2-c-opt/obj/src/mat/order/degree.o CC arch-linux2-c-opt/obj/src/mat/order/fnroot.o CC arch-linux2-c-opt/obj/src/mat/order/genqmd.o CC arch-linux2-c-opt/obj/src/mat/order/qmdqt.o CC arch-linux2-c-opt/obj/src/mat/order/rcm.o CC arch-linux2-c-opt/obj/src/mat/order/fn1wd.o CC arch-linux2-c-opt/obj/src/mat/order/gen1wd.o CC arch-linux2-c-opt/obj/src/mat/order/genrcm.o CC arch-linux2-c-opt/obj/src/mat/order/rootls.o CC arch-linux2-c-opt/obj/src/mat/order/qmdrch.o CC arch-linux2-c-opt/obj/src/mat/order/fndsep.o CC arch-linux2-c-opt/obj/src/mat/order/gennd.o CC arch-linux2-c-opt/obj/src/mat/order/qmdmrg.o CC arch-linux2-c-opt/obj/src/mat/order/qmdupd.o CC arch-linux2-c-opt/obj/src/mat/order/wbm.o CC arch-linux2-c-opt/obj/src/mat/order/ftn-auto/spectralf.o CC arch-linux2-c-opt/obj/src/mat/order/ftn-custom/zsorderf.o CC arch-linux2-c-opt/obj/src/mat/utils/convert.o CC arch-linux2-c-opt/obj/src/mat/utils/matstash.o CC arch-linux2-c-opt/obj/src/mat/utils/axpy.o CC arch-linux2-c-opt/obj/src/mat/utils/zerodiag.o CC arch-linux2-c-opt/obj/src/mat/utils/factorschur.o CC arch-linux2-c-opt/obj/src/mat/utils/getcolv.o CC arch-linux2-c-opt/obj/src/mat/utils/gcreate.o CC arch-linux2-c-opt/obj/src/mat/utils/freespace.o CC arch-linux2-c-opt/obj/src/mat/utils/compressedrow.o CC arch-linux2-c-opt/obj/src/mat/utils/multequal.o CC arch-linux2-c-opt/obj/src/mat/utils/matstashspace.o CC arch-linux2-c-opt/obj/src/mat/utils/pheap.o CC arch-linux2-c-opt/obj/src/mat/utils/bandwidth.o CC arch-linux2-c-opt/obj/src/mat/utils/overlapsplit.o CC arch-linux2-c-opt/obj/src/mat/utils/zerorows.o CC arch-linux2-c-opt/obj/src/mat/utils/ftn-auto/axpyf.o CC arch-linux2-c-opt/obj/src/mat/utils/ftn-auto/bandwidthf.o CC arch-linux2-c-opt/obj/src/mat/utils/ftn-auto/gcreatef.o CC arch-linux2-c-opt/obj/src/mat/utils/ftn-auto/getcolvf.o CC arch-linux2-c-opt/obj/src/mat/utils/ftn-auto/multequalf.o CC arch-linux2-c-opt/obj/src/mat/utils/ftn-auto/zerodiagf.o CC arch-linux2-c-opt/obj/src/dm/label/dmlabel.o CC arch-linux2-c-opt/obj/src/dm/label/ftn-auto/dmlabelf.o CC arch-linux2-c-opt/obj/src/dm/label/ftn-custom/zdmlabel.o CC arch-linux2-c-opt/obj/src/dm/interface/dm.o CC arch-linux2-c-opt/obj/src/dm/interface/dmget.o CC arch-linux2-c-opt/obj/src/dm/interface/dmregall.o CC arch-linux2-c-opt/obj/src/dm/interface/dmi.o CC arch-linux2-c-opt/obj/src/dm/interface/dlregisdmdm.o CC arch-linux2-c-opt/obj/src/dm/interface/ftn-auto/dmf.o CC arch-linux2-c-opt/obj/src/dm/interface/ftn-auto/dmgetf.o CC arch-linux2-c-opt/obj/src/dm/interface/ftn-custom/zdmf.o CC arch-linux2-c-opt/obj/src/dm/interface/ftn-custom/zdmgetf.o CC arch-linux2-c-opt/obj/src/dm/impls/shell/dmshell.o CC arch-linux2-c-opt/obj/src/dm/impls/shell/ftn-auto/dmshellf.o CC arch-linux2-c-opt/obj/src/dm/impls/shell/ftn-custom/zdmshellf.o CC arch-linux2-c-opt/obj/src/dm/impls/network/networkcreate.o CC arch-linux2-c-opt/obj/src/dm/impls/network/network.o CC arch-linux2-c-opt/obj/src/dm/impls/network/networkmonitor.o CC arch-linux2-c-opt/obj/src/dm/impls/network/ftn-auto/networkf.o CC arch-linux2-c-opt/obj/src/dm/impls/network/ftn-auto/networkcreatef.o CC arch-linux2-c-opt/obj/src/dm/impls/network/ftn-auto/networkmonitorf.o CC arch-linux2-c-opt/obj/src/dm/impls/composite/pack.o CC arch-linux2-c-opt/obj/src/dm/impls/composite/packm.o CC arch-linux2-c-opt/obj/src/dm/impls/composite/ftn-auto/packf.o CC arch-linux2-c-opt/obj/src/dm/impls/composite/f90-custom/zfddaf90.o CC arch-linux2-c-opt/obj/src/dm/impls/composite/ftn-custom/zfddaf.o CC arch-linux2-c-opt/obj/src/dm/impls/redundant/dmredundant.o CC arch-linux2-c-opt/obj/src/dm/impls/redundant/ftn-auto/dmredundantf.o CC arch-linux2-c-opt/obj/src/dm/impls/patch/patchcreate.o CC arch-linux2-c-opt/obj/src/dm/impls/patch/patch.o CC arch-linux2-c-opt/obj/src/dm/impls/patch/ftn-auto/patchcreatef.o CC arch-linux2-c-opt/obj/src/dm/impls/sliced/sliced.o CC arch-linux2-c-opt/obj/src/dm/impls/swarm/swarm.o CC arch-linux2-c-opt/obj/src/dm/impls/swarm/data_bucket.o CC arch-linux2-c-opt/obj/src/dm/impls/swarm/swarmpic.o CC arch-linux2-c-opt/obj/src/dm/impls/swarm/data_ex.o CC arch-linux2-c-opt/obj/src/dm/impls/swarm/swarm_migrate.o CC arch-linux2-c-opt/obj/src/dm/impls/swarm/swarmpic_da.o CC arch-linux2-c-opt/obj/src/dm/impls/swarm/swarmpic_plex.o CC arch-linux2-c-opt/obj/src/dm/impls/swarm/swarmpic_view.o CC arch-linux2-c-opt/obj/src/dm/impls/swarm/swarmpic_sort.o CC arch-linux2-c-opt/obj/src/dm/impls/forest/forest.o CC arch-linux2-c-opt/obj/src/dm/impls/forest/ftn-auto/forestf.o CC arch-linux2-c-opt/obj/src/dm/impls/da/da2.o CC arch-linux2-c-opt/obj/src/dm/impls/da/da1.o CC arch-linux2-c-opt/obj/src/dm/impls/da/daghost.o CC arch-linux2-c-opt/obj/src/dm/impls/da/da3.o CC arch-linux2-c-opt/obj/src/dm/impls/da/dacorn.o CC arch-linux2-c-opt/obj/src/dm/impls/da/dagtol.o CC arch-linux2-c-opt/obj/src/dm/impls/da/daltol.o CC arch-linux2-c-opt/obj/src/dm/impls/da/daindex.o CC arch-linux2-c-opt/obj/src/dm/impls/da/dascatter.o CC arch-linux2-c-opt/obj/src/dm/impls/da/dacreate.o CC arch-linux2-c-opt/obj/src/dm/impls/da/dalocal.o CC arch-linux2-c-opt/obj/src/dm/impls/da/dadestroy.o CC arch-linux2-c-opt/obj/src/dm/impls/da/dadist.o CC arch-linux2-c-opt/obj/src/dm/impls/da/daview.o CC arch-linux2-c-opt/obj/src/dm/impls/da/dasub.o CC arch-linux2-c-opt/obj/src/dm/impls/da/gr1.o CC arch-linux2-c-opt/obj/src/dm/impls/da/gr2.o CC arch-linux2-c-opt/obj/src/dm/impls/da/dagtona.o CC arch-linux2-c-opt/obj/src/dm/impls/da/dainterp.o CC arch-linux2-c-opt/obj/src/dm/impls/da/dapf.o CC arch-linux2-c-opt/obj/src/dm/impls/da/dagetarray.o CC arch-linux2-c-opt/obj/src/dm/impls/da/dagetelem.o CC arch-linux2-c-opt/obj/src/dm/impls/da/dareg.o CC arch-linux2-c-opt/obj/src/dm/impls/da/da.o CC arch-linux2-c-opt/obj/src/dm/impls/da/fdda.o CC arch-linux2-c-opt/obj/src/dm/impls/da/grvtk.o CC arch-linux2-c-opt/obj/src/dm/impls/da/dageometry.o CC arch-linux2-c-opt/obj/src/dm/impls/da/dadd.o CC arch-linux2-c-opt/obj/src/dm/impls/da/dapreallocate.o CC arch-linux2-c-opt/obj/src/dm/impls/da/grglvis.o CC arch-linux2-c-opt/obj/src/dm/impls/da/ftn-auto/daf.o CC arch-linux2-c-opt/obj/src/dm/impls/da/ftn-auto/dacornf.o CC arch-linux2-c-opt/obj/src/dm/impls/da/ftn-auto/dacreatef.o CC arch-linux2-c-opt/obj/src/dm/impls/da/ftn-auto/daddf.o CC arch-linux2-c-opt/obj/src/dm/impls/da/ftn-auto/dadistf.o CC arch-linux2-c-opt/obj/src/dm/impls/da/ftn-auto/dageometryf.o CC arch-linux2-c-opt/obj/src/dm/impls/da/ftn-auto/dagetarrayf.o CC arch-linux2-c-opt/obj/src/dm/impls/da/ftn-auto/dagtolf.o CC arch-linux2-c-opt/obj/src/dm/impls/da/ftn-auto/dagtonaf.o CC arch-linux2-c-opt/obj/src/dm/impls/da/ftn-auto/daindexf.o CC arch-linux2-c-opt/obj/src/dm/impls/da/ftn-auto/dainterpf.o CC arch-linux2-c-opt/obj/src/dm/impls/da/ftn-auto/dalocalf.o CC arch-linux2-c-opt/obj/src/dm/impls/da/ftn-auto/dapreallocatef.o CC arch-linux2-c-opt/obj/src/dm/impls/da/ftn-auto/dasubf.o CC arch-linux2-c-opt/obj/src/dm/impls/da/ftn-auto/fddaf.o CC arch-linux2-c-opt/obj/src/dm/impls/da/ftn-auto/gr1f.o CC arch-linux2-c-opt/obj/src/dm/impls/da/f90-custom/zda1f90.o CC arch-linux2-c-opt/obj/src/dm/impls/da/ftn-custom/zdaf.o CC arch-linux2-c-opt/obj/src/dm/impls/da/ftn-custom/zda1f.o CC arch-linux2-c-opt/obj/src/dm/impls/da/ftn-custom/zda2f.o CC arch-linux2-c-opt/obj/src/dm/impls/da/ftn-custom/zda3f.o CC arch-linux2-c-opt/obj/src/dm/impls/da/ftn-custom/zdaghostf.o CC arch-linux2-c-opt/obj/src/dm/impls/da/ftn-custom/zdacornf.o CC arch-linux2-c-opt/obj/src/dm/impls/da/ftn-custom/zdagetscatterf.o CC arch-linux2-c-opt/obj/src/dm/impls/da/ftn-custom/zdaviewf.o CC arch-linux2-c-opt/obj/src/dm/impls/da/ftn-custom/zdaindexf.o CC arch-linux2-c-opt/obj/src/dm/impls/da/ftn-custom/zdasubf.o CC arch-linux2-c-opt/obj/src/dm/impls/da/hypre/mhyp.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plexcreate.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plex.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plexpartition.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plexdistribute.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plexrefine.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plexadapt.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plexcoarsen.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plexinterpolate.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plexpreallocate.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plexreorder.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plexgeometry.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plexsubmesh.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plexhdf5.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plexexodusii.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plexfluent.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plexcgns.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plexgmsh.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plexmed.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plexply.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plexvtk.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plexpoint.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plexvtu.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plexfem.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plexfvm.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plexindices.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plextree.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plexgenerate.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plexorient.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plexnatural.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plexproject.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/plexglvis.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/ftn-auto/plexf.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/ftn-auto/plexcgnsf.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/ftn-auto/plexcreatef.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/ftn-auto/plexdistributef.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/ftn-auto/plexexodusiif.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/ftn-auto/plexfemf.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/ftn-auto/plexfvmf.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/ftn-auto/plexgeometryf.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/ftn-auto/plexgmshf.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/ftn-auto/plexindicesf.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/ftn-auto/plexinterpolatef.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/ftn-auto/plexnaturalf.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/ftn-auto/plexorientf.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/ftn-auto/plexpartitionf.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/ftn-auto/plexpointf.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/ftn-auto/plexrefinef.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/ftn-auto/plexreorderf.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/ftn-auto/plexsubmeshf.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/ftn-auto/plextreef.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/f90-custom/zplexf90.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/f90-custom/zplexgeometryf90.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/ftn-custom/zplex.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/ftn-custom/zplexcreate.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/ftn-custom/zplexdistribute.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/ftn-custom/zplexinterpolate.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/ftn-custom/zplexsubmesh.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/ftn-custom/zplexexodusii.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/ftn-custom/zplexgmsh.o CC arch-linux2-c-opt/obj/src/dm/impls/plex/ftn-custom/zplexfluent.o CC arch-linux2-c-opt/obj/src/dm/dt/interface/dt.o CC arch-linux2-c-opt/obj/src/dm/dt/interface/dtfe.o CC arch-linux2-c-opt/obj/src/dm/dt/interface/dtfv.o CC arch-linux2-c-opt/obj/src/dm/dt/interface/dtds.o CC arch-linux2-c-opt/obj/src/dm/dt/interface/ftn-auto/dtf.o CC arch-linux2-c-opt/obj/src/dm/dt/interface/ftn-auto/dtdsf.o CC arch-linux2-c-opt/obj/src/dm/dt/interface/ftn-auto/dtfef.o CC arch-linux2-c-opt/obj/src/dm/dt/interface/ftn-auto/dtfvf.o CC arch-linux2-c-opt/obj/src/dm/dt/interface/f90-custom/zdtf90.o CC arch-linux2-c-opt/obj/src/dm/dt/interface/f90-custom/zdtdsf90.o CC arch-linux2-c-opt/obj/src/dm/dt/interface/ftn-custom/zdtf.o CC arch-linux2-c-opt/obj/src/dm/dt/interface/ftn-custom/zdtfef.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/bddc/bddc.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/bddc/bddcprivate.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/bddc/bddcgraph.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/bddc/bddcscalingbasic.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/bddc/bddcnullspace.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/bddc/bddcfetidp.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/bddc/bddcschurs.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/lsc/lsc.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/bddc/ftn-auto/bddcf.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/bjacobi/bjacobi.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/bjacobi/ftn-auto/bjacobif.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/bjacobi/ftn-custom/zbjacobif.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/is/pcis.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/is/ftn-auto/pcisf.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/is/nn/nn.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/sor/sor.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/sor/ftn-auto/sorf.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/pbjacobi/pbjacobi.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/cp/cp.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/eisens/eisen.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/eisens/ftn-auto/eisenf.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/telescope/telescope.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/telescope/telescope_dmda.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/telescope/ftn-auto/telescopef.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/ksp/pcksp.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/ksp/ftn-auto/pckspf.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/python/pythonpc.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/python/ftn-custom/zpythonpcf.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/tfs/bitmask.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/tfs/comm.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/tfs/gs.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/tfs/ivec.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/tfs/xxt.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/tfs/xyt.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/tfs/tfs.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/factor/factor.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/factor/factimpl.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/factor/ftn-custom/zluf.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/factor/cholesky/cholesky.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/factor/cholesky/ftn-auto/choleskyf.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/factor/ilu/ilu.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/factor/icc/icc.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/factor/lu/lu.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/factor/ftn-auto/factorf.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/jacobi/jacobi.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/jacobi/ftn-auto/jacobif.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/kaczmarz/kaczmarz.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/galerkin/galerkin.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/galerkin/ftn-auto/galerkinf.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/wb/wb.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/wb/ftn-auto/wbf.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/shell/shellpc.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/shell/ftn-auto/shellpcf.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/shell/ftn-custom/zshellpcf.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/gasm/gasm.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/gasm/ftn-auto/gasmf.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/gasm/ftn-custom/zgasmf.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/mat/pcmat.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/composite/composite.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/composite/ftn-auto/compositef.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/composite/ftn-custom/zcompositef.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/none/none.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/redistribute/redistribute.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/redistribute/ftn-auto/redistributef.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/gamg/agg.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/gamg/gamg.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/gamg/geo.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/gamg/util.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/gamg/classical.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/gamg/ftn-auto/aggf.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/gamg/ftn-auto/gamgf.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/gamg/ftn-custom/zgamgf.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/fieldsplit/fieldsplit.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/fieldsplit/ftn-auto/fieldsplitf.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/fieldsplit/ftn-custom/zfieldsplitf.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/asm/asm.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/asm/ftn-auto/asmf.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/asm/ftn-custom/zasmf.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/hypre/hypre.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/hypre/ftn-auto/hypref.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/hypre/ftn-custom/zhypref.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/mg/mg.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/mg/fmg.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/mg/smg.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/mg/mgfunc.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/mg/ftn-auto/mgf.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/mg/ftn-auto/mgfuncf.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/mg/ftn-custom/zmgf.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/mg/ftn-custom/zmgfuncf.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/svd/svd.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/redundant/redundant.o CC arch-linux2-c-opt/obj/src/ksp/pc/impls/redundant/ftn-auto/redundantf.o CC arch-linux2-c-opt/obj/src/ksp/pc/interface/precon.o CC arch-linux2-c-opt/obj/src/ksp/pc/interface/pcset.o CC arch-linux2-c-opt/obj/src/ksp/pc/interface/pcregis.o CC arch-linux2-c-opt/obj/src/ksp/pc/interface/ftn-auto/pcsetf.o CC arch-linux2-c-opt/obj/src/ksp/pc/interface/ftn-auto/preconf.o CC arch-linux2-c-opt/obj/src/ksp/pc/interface/ftn-custom/zpcsetf.o CC arch-linux2-c-opt/obj/src/ksp/pc/interface/ftn-custom/zpreconf.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/minres/minres.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/bicg/bicg.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/fcg/fcg.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/fcg/ftn-auto/fcgf.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/fcg/pipefcg/pipefcg.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/fcg/pipefcg/ftn-auto/pipefcgf.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/cg/cg.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/cg/cgeig.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/cg/cgtype.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/cg/cgls.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/cg/gltr/gltr.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/cg/gltr/ftn-auto/gltrf.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/cg/cgne/cgne.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/cg/pipecgrr/pipecgrr.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/cg/stcg/stcg.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/cg/groppcg/groppcg.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/cg/ftn-auto/cgtypef.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/cg/nash/nash.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/cg/pipecg/pipecg.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/cr/cr.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/cr/pipecr/pipecr.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/fetidp/fetidp.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/fetidp/ftn-auto/fetidpf.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/ibcgs/ibcgs.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/python/pythonksp.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/python/ftn-custom/zpythonkspf.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/tsirm/tsirm.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/gcr/gcr.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/gcr/pipegcr/pipegcr.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/gcr/pipegcr/ftn-auto/pipegcrf.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/qcg/qcg.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/qcg/ftn-auto/qcgf.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/rich/rich.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/rich/richscale.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/rich/ftn-auto/richscalef.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/bcgsl/bcgsl.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/bcgsl/ftn-auto/bcgslf.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/lsqr/lsqr.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/lsqr/lsqr_monitor.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/lsqr/lsqr_converged.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/cheby/cheby.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/cheby/ftn-auto/chebyf.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/bcgs/bcgs.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/bcgs/pipebcgs/pipebcgs.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/bcgs/fbcgs/fbcgs.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/bcgs/fbcgsr/fbcgsr.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/lcd/lcd.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/preonly/preonly.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/gmres/gmres.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/gmres/borthog.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/gmres/borthog2.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/gmres/gmres2.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/gmres/gmreig.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/gmres/gmpre.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/gmres/dgmres/dgmres.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/gmres/pgmres/pgmres.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/gmres/pipefgmres/pipefgmres.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/gmres/pipefgmres/ftn-auto/pipefgmresf.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/gmres/lgmres/lgmres.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/gmres/fgmres/fgmres.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/gmres/fgmres/modpcf.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/gmres/fgmres/ftn-auto/modpcff.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/gmres/fgmres/ftn-custom/zmodpcff.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/gmres/agmres/agmres.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/gmres/agmres/agmresorthog.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/gmres/agmres/agmresleja.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/gmres/agmres/agmresdeflation.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/gmres/ftn-auto/gmresf.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/gmres/ftn-auto/gmpref.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/cgs/cgs.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/symmlq/symmlq.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/tfqmr/tfqmr.o CC arch-linux2-c-opt/obj/src/ksp/ksp/impls/tcqmr/tcqmr.o CC arch-linux2-c-opt/obj/src/ksp/ksp/interface/itcl.o CC arch-linux2-c-opt/obj/src/ksp/ksp/interface/itfunc.o CC arch-linux2-c-opt/obj/src/ksp/ksp/interface/iguess.o CC arch-linux2-c-opt/obj/src/ksp/ksp/interface/itcreate.o CC arch-linux2-c-opt/obj/src/ksp/ksp/interface/iterativ.o CC arch-linux2-c-opt/obj/src/ksp/ksp/interface/itres.o CC arch-linux2-c-opt/obj/src/ksp/ksp/interface/itregis.o CC arch-linux2-c-opt/obj/src/ksp/ksp/interface/xmon.o CC arch-linux2-c-opt/obj/src/ksp/ksp/interface/eige.o CC arch-linux2-c-opt/obj/src/ksp/ksp/interface/dlregisksp.o CC arch-linux2-c-opt/obj/src/ksp/ksp/interface/dmksp.o CC arch-linux2-c-opt/obj/src/ksp/ksp/interface/ftn-auto/eigef.o CC arch-linux2-c-opt/obj/src/ksp/ksp/interface/ftn-auto/iguessf.o CC arch-linux2-c-opt/obj/src/ksp/ksp/interface/ftn-auto/itclf.o CC arch-linux2-c-opt/obj/src/ksp/ksp/interface/ftn-auto/itcreatef.o CC arch-linux2-c-opt/obj/src/ksp/ksp/interface/ftn-auto/iterativf.o CC arch-linux2-c-opt/obj/src/ksp/ksp/interface/ftn-auto/itfuncf.o CC arch-linux2-c-opt/obj/src/ksp/ksp/interface/f90-custom/zitfuncf90.o CC arch-linux2-c-opt/obj/src/ksp/ksp/interface/ftn-auto/itresf.o CC arch-linux2-c-opt/obj/src/ksp/ksp/interface/ftn-custom/zitclf.o CC arch-linux2-c-opt/obj/src/ksp/ksp/interface/ftn-custom/zitcreatef.o CC arch-linux2-c-opt/obj/src/ksp/ksp/interface/ftn-custom/zitfuncf.o CC arch-linux2-c-opt/obj/src/ksp/ksp/interface/ftn-custom/zxonf.o CC arch-linux2-c-opt/obj/src/ksp/ksp/interface/ftn-custom/zdmkspf.o CC arch-linux2-c-opt/obj/src/ksp/ksp/interface/ftn-custom/ziguess.o CC arch-linux2-c-opt/obj/src/ksp/ksp/guess/impls/pod/pod.o CC arch-linux2-c-opt/obj/src/ksp/ksp/guess/impls/fischer/fischer.o CC arch-linux2-c-opt/obj/src/ksp/ksp/guess/impls/fischer/ftn-auto/fischerf.o CC arch-linux2-c-opt/obj/src/ksp/ksp/utils/schurm.o CC arch-linux2-c-opt/obj/src/ksp/ksp/utils/ftn-auto/schurmf.o CC arch-linux2-c-opt/obj/src/ksp/ksp/utils/dmproject.o CC arch-linux2-c-opt/obj/src/ksp/ksp/utils/ftn-auto/dmprojectf.o CC arch-linux2-c-opt/obj/src/snes/mf/snesmfj.o CC arch-linux2-c-opt/obj/src/snes/mf/ftn-auto/snesmfjf.o CC arch-linux2-c-opt/obj/src/snes/utils/dmsnes.o CC arch-linux2-c-opt/obj/src/snes/utils/dmdasnes.o CC arch-linux2-c-opt/obj/src/snes/utils/dmlocalsnes.o CC arch-linux2-c-opt/obj/src/snes/utils/dmplexsnes.o CC arch-linux2-c-opt/obj/src/snes/utils/convest.o CC arch-linux2-c-opt/obj/src/snes/utils/ftn-auto/convestf.o CC arch-linux2-c-opt/obj/src/snes/utils/ftn-auto/dmplexsnesf.o CC arch-linux2-c-opt/obj/src/snes/utils/f90-custom/zdmplexsnesf90.o CC arch-linux2-c-opt/obj/src/snes/utils/ftn-custom/zdmdasnesf.o CC arch-linux2-c-opt/obj/src/snes/utils/ftn-custom/zdmlocalsnesf.o CC arch-linux2-c-opt/obj/src/snes/utils/ftn-custom/zdmsnesf.o CC arch-linux2-c-opt/obj/src/snes/interface/snes.o CC arch-linux2-c-opt/obj/src/snes/interface/snesregi.o CC arch-linux2-c-opt/obj/src/snes/interface/snesj.o CC arch-linux2-c-opt/obj/src/snes/interface/snesut.o CC arch-linux2-c-opt/obj/src/snes/interface/snesj2.o CC arch-linux2-c-opt/obj/src/snes/interface/dlregissnes.o CC arch-linux2-c-opt/obj/src/snes/interface/snesob.o CC arch-linux2-c-opt/obj/src/snes/interface/ftn-auto/snesf.o CC arch-linux2-c-opt/obj/src/snes/interface/snespc.o CC arch-linux2-c-opt/obj/src/snes/interface/ftn-auto/snespcf.o CC arch-linux2-c-opt/obj/src/snes/interface/ftn-auto/snesutf.o CC arch-linux2-c-opt/obj/src/snes/interface/f90-custom/zsnesf90.o CC arch-linux2-c-opt/obj/src/snes/interface/ftn-custom/zsnesf.o CC arch-linux2-c-opt/obj/src/snes/interface/noise/snesmfj2.o CC arch-linux2-c-opt/obj/src/snes/interface/noise/snesnoise.o CC arch-linux2-c-opt/obj/src/snes/interface/noise/snesdnest.o CC arch-linux2-c-opt/obj/src/snes/impls/vi/vi.o CC arch-linux2-c-opt/obj/src/snes/impls/vi/ss/viss.o CC arch-linux2-c-opt/obj/src/snes/impls/vi/ftn-auto/vif.o CC arch-linux2-c-opt/obj/src/snes/impls/vi/rs/virs.o CC arch-linux2-c-opt/obj/src/snes/impls/shell/snesshell.o CC arch-linux2-c-opt/obj/src/snes/impls/shell/ftn-auto/snesshellf.o CC arch-linux2-c-opt/obj/src/snes/impls/shell/ftn-custom/zsnesshellf.o CC arch-linux2-c-opt/obj/src/snes/impls/gs/gssecant.o CC arch-linux2-c-opt/obj/src/snes/impls/gs/snesgs.o CC arch-linux2-c-opt/obj/src/snes/impls/gs/ftn-auto/snesgsf.o CC arch-linux2-c-opt/obj/src/snes/impls/ngmres/snesngmres.o CC arch-linux2-c-opt/obj/src/snes/impls/ngmres/ngmresfunc.o CC arch-linux2-c-opt/obj/src/snes/impls/ngmres/anderson.o CC arch-linux2-c-opt/obj/src/snes/impls/ngmres/ftn-auto/snesngmresf.o CC arch-linux2-c-opt/obj/src/snes/impls/composite/snescomposite.o CC arch-linux2-c-opt/obj/src/snes/impls/composite/ftn-auto/snescompositef.o CC arch-linux2-c-opt/obj/src/snes/impls/fas/fas.o CC arch-linux2-c-opt/obj/src/snes/impls/fas/fasgalerkin.o CC arch-linux2-c-opt/obj/src/snes/impls/fas/fasfunc.o CC arch-linux2-c-opt/obj/src/snes/impls/fas/ftn-auto/fasf.o CC arch-linux2-c-opt/obj/src/snes/impls/fas/ftn-auto/fasfuncf.o CC arch-linux2-c-opt/obj/src/snes/impls/fas/ftn-auto/fasgalerkinf.o CC arch-linux2-c-opt/obj/src/snes/impls/ksponly/ksponly.o CC arch-linux2-c-opt/obj/src/snes/impls/tr/tr.o CC arch-linux2-c-opt/obj/src/snes/impls/python/pythonsnes.o CC arch-linux2-c-opt/obj/src/snes/impls/python/ftn-custom/zpythonsf.o CC arch-linux2-c-opt/obj/src/snes/impls/ls/ls.o CC arch-linux2-c-opt/obj/src/snes/impls/ms/ms.o CC arch-linux2-c-opt/obj/src/snes/impls/qn/ftn-auto/qnf.o CC arch-linux2-c-opt/obj/src/snes/impls/qn/qn.o CC arch-linux2-c-opt/obj/src/snes/impls/test/snestest.o CC arch-linux2-c-opt/obj/src/snes/impls/richardson/snesrichardson.o CC arch-linux2-c-opt/obj/src/snes/impls/ncg/snesncg.o CC arch-linux2-c-opt/obj/src/snes/impls/ncg/ftn-auto/snesncgf.o CC arch-linux2-c-opt/obj/src/snes/impls/nasm/nasm.o CC arch-linux2-c-opt/obj/src/snes/impls/nasm/aspin.o CC arch-linux2-c-opt/obj/src/snes/impls/nasm/ftn-auto/nasmf.o CC arch-linux2-c-opt/obj/src/snes/linesearch/impls/shell/linesearchshell.o CC arch-linux2-c-opt/obj/src/snes/linesearch/impls/shell/ftn-custom/zlinesearchshellf.o CC arch-linux2-c-opt/obj/src/snes/linesearch/impls/bt/linesearchbt.o CC arch-linux2-c-opt/obj/src/snes/linesearch/impls/l2/linesearchl2.o CC arch-linux2-c-opt/obj/src/snes/linesearch/impls/bt/ftn-auto/linesearchbtf.o CC arch-linux2-c-opt/obj/src/snes/linesearch/impls/basic/linesearchbasic.o CC arch-linux2-c-opt/obj/src/snes/linesearch/impls/cp/linesearchcp.o CC arch-linux2-c-opt/obj/src/snes/linesearch/impls/nleqerr/linesearchnleqerr.o CC arch-linux2-c-opt/obj/src/snes/linesearch/interface/linesearchregi.o CC arch-linux2-c-opt/obj/src/snes/linesearch/interface/linesearch.o CC arch-linux2-c-opt/obj/src/snes/linesearch/interface/ftn-auto/linesearchf.o CC arch-linux2-c-opt/obj/src/snes/linesearch/interface/ftn-custom/zlinesearchf.o CC arch-linux2-c-opt/obj/src/ts/characteristic/impls/da/slda.o CC arch-linux2-c-opt/obj/src/ts/characteristic/interface/characteristic.o CC arch-linux2-c-opt/obj/src/ts/characteristic/interface/mocregis.o CC arch-linux2-c-opt/obj/src/ts/characteristic/interface/slregis.o CC arch-linux2-c-opt/obj/src/ts/characteristic/interface/ftn-auto/characteristicf.o CC arch-linux2-c-opt/obj/src/ts/utils/dmts.o CC arch-linux2-c-opt/obj/src/ts/utils/dmlocalts.o CC arch-linux2-c-opt/obj/src/ts/utils/dmdats.o CC arch-linux2-c-opt/obj/src/ts/utils/ftn-auto/dmplextsf.o CC arch-linux2-c-opt/obj/src/ts/utils/dmplexts.o CC arch-linux2-c-opt/obj/src/ts/event/tsevent.o CC arch-linux2-c-opt/obj/src/ts/event/ftn-auto/tseventf.o CC arch-linux2-c-opt/obj/src/ts/interface/ts.o CC arch-linux2-c-opt/obj/src/ts/interface/tscreate.o CC arch-linux2-c-opt/obj/src/ts/interface/tsreg.o CC arch-linux2-c-opt/obj/src/ts/interface/tsregall.o CC arch-linux2-c-opt/obj/src/ts/interface/dlregists.o CC arch-linux2-c-opt/obj/src/ts/interface/tseig.o CC arch-linux2-c-opt/obj/src/ts/interface/tsfwdsen.o CC arch-linux2-c-opt/obj/src/ts/interface/ftn-auto/tsf.o CC arch-linux2-c-opt/obj/src/ts/interface/ftn-auto/tsfwdsenf.o CC arch-linux2-c-opt/obj/src/ts/interface/ftn-custom/ztscreatef.o CC arch-linux2-c-opt/obj/src/ts/interface/ftn-custom/ztsregf.o CC arch-linux2-c-opt/obj/src/ts/interface/ftn-custom/ztsf.o CC arch-linux2-c-opt/obj/src/ts/trajectory/impls/visualization/trajvisualization.o CC arch-linux2-c-opt/obj/src/ts/trajectory/impls/memory/trajmemory.o CC arch-linux2-c-opt/obj/src/ts/trajectory/impls/singlefile/singlefile.o CC arch-linux2-c-opt/obj/src/ts/trajectory/impls/basic/trajbasic.o CC arch-linux2-c-opt/obj/src/ts/trajectory/interface/traj.o CC arch-linux2-c-opt/obj/src/ts/trajectory/interface/ftn-auto/trajf.o CC arch-linux2-c-opt/obj/src/ts/impls/bdf/bdf.o CC arch-linux2-c-opt/obj/src/ts/impls/bdf/ftn-auto/bdff.o CC arch-linux2-c-opt/obj/src/ts/impls/python/pythonts.o CC arch-linux2-c-opt/obj/src/ts/impls/python/ftn-custom/zpythontf.o CC arch-linux2-c-opt/obj/src/ts/impls/pseudo/posindep.o CC arch-linux2-c-opt/obj/src/ts/impls/pseudo/ftn-auto/posindepf.o CC arch-linux2-c-opt/obj/src/ts/impls/glee/glee.o CC arch-linux2-c-opt/obj/src/ts/impls/explicit/ssp/ssp.o CC arch-linux2-c-opt/obj/src/ts/impls/explicit/ssp/ftn-auto/sspf.o CC arch-linux2-c-opt/obj/src/ts/impls/explicit/ssp/ftn-custom/zsspf.o CC arch-linux2-c-opt/obj/src/ts/impls/explicit/euler/euler.o CC arch-linux2-c-opt/obj/src/ts/impls/explicit/rk/rk.o CC arch-linux2-c-opt/obj/src/ts/impls/explicit/rk/ftn-custom/zrkf.o CC arch-linux2-c-opt/obj/src/ts/impls/mimex/mimex.o CC arch-linux2-c-opt/obj/src/ts/impls/arkimex/arkimex.o CC arch-linux2-c-opt/obj/src/ts/impls/arkimex/ftn-auto/arkimexf.o CC arch-linux2-c-opt/obj/src/ts/impls/arkimex/ftn-custom/zarkimexf.o CC arch-linux2-c-opt/obj/src/ts/impls/eimex/eimex.o CC arch-linux2-c-opt/obj/src/ts/impls/rosw/rosw.o CC arch-linux2-c-opt/obj/src/ts/impls/rosw/ftn-custom/zroswf.o CC arch-linux2-c-opt/obj/src/ts/impls/implicit/alpha/alpha1.o CC arch-linux2-c-opt/obj/src/ts/impls/implicit/alpha/alpha2.o CC arch-linux2-c-opt/obj/src/ts/impls/implicit/alpha/ftn-auto/alpha1f.o CC arch-linux2-c-opt/obj/src/ts/impls/implicit/glle/glle.o CC arch-linux2-c-opt/obj/src/ts/impls/implicit/alpha/ftn-auto/alpha2f.o CC arch-linux2-c-opt/obj/src/ts/impls/implicit/glle/glleadapt.o CC arch-linux2-c-opt/obj/src/ts/impls/implicit/theta/theta.o CC arch-linux2-c-opt/obj/src/ts/impls/implicit/theta/ftn-auto/thetaf.o CC arch-linux2-c-opt/obj/src/ts/adapt/impls/cfl/adaptcfl.o CC arch-linux2-c-opt/obj/src/ts/adapt/impls/glee/adaptglee.o CC arch-linux2-c-opt/obj/src/ts/adapt/impls/none/adaptnone.o CC arch-linux2-c-opt/obj/src/ts/adapt/impls/dsp/adaptdsp.o CC arch-linux2-c-opt/obj/src/ts/adapt/impls/dsp/ftn-auto/adaptdspf.o CC arch-linux2-c-opt/obj/src/ts/adapt/impls/dsp/ftn-custom/zadaptdspf.o CC arch-linux2-c-opt/obj/src/ts/adapt/impls/basic/adaptbasic.o CC arch-linux2-c-opt/obj/src/ts/adapt/interface/ftn-auto/tsadaptf.o CC arch-linux2-c-opt/obj/src/ts/adapt/interface/tsadapt.o CC arch-linux2-c-opt/obj/src/ts/adapt/interface/ftn-custom/ztsadaptf.o CC arch-linux2-c-opt/obj/src/tao/matrix/lmvmmat.o CC arch-linux2-c-opt/obj/src/tao/matrix/adamat.o CC arch-linux2-c-opt/obj/src/tao/matrix/submatfree.o CC arch-linux2-c-opt/obj/src/tao/bound/impls/gpcg/gpcg.o CC arch-linux2-c-opt/obj/src/tao/bound/impls/tron/tron.o CC arch-linux2-c-opt/obj/src/tao/bound/impls/blmvm/blmvm.o CC arch-linux2-c-opt/obj/src/tao/bound/impls/bqpip/bqpip.o CC arch-linux2-c-opt/obj/src/tao/bound/utils/isutil.o CC arch-linux2-c-opt/obj/src/tao/util/tao_util.o CC arch-linux2-c-opt/obj/src/tao/util/ftn-auto/tao_utilf.o CC arch-linux2-c-opt/obj/src/tao/interface/taosolver.o CC arch-linux2-c-opt/obj/src/tao/interface/taosolver_fg.o CC arch-linux2-c-opt/obj/src/tao/interface/taosolverregi.o CC arch-linux2-c-opt/obj/src/tao/interface/taosolver_hj.o CC arch-linux2-c-opt/obj/src/tao/interface/taosolver_bounds.o CC arch-linux2-c-opt/obj/src/tao/interface/dlregistao.o CC arch-linux2-c-opt/obj/src/tao/interface/fdiff.o CC arch-linux2-c-opt/obj/src/tao/interface/fdtest.o CC arch-linux2-c-opt/obj/src/tao/interface/ftn-auto/taosolverf.o CC arch-linux2-c-opt/obj/src/tao/interface/ftn-auto/taosolver_boundsf.o CC arch-linux2-c-opt/obj/src/tao/interface/ftn-auto/taosolver_fgf.o CC arch-linux2-c-opt/obj/src/tao/interface/ftn-auto/taosolver_hjf.o CC arch-linux2-c-opt/obj/src/tao/interface/ftn-custom/ztaosolverf.o CC arch-linux2-c-opt/obj/src/tao/pde_constrained/impls/lcl/lcl.o CC arch-linux2-c-opt/obj/src/tao/complementarity/impls/ssls/ssls.o CC arch-linux2-c-opt/obj/src/tao/complementarity/impls/ssls/ssils.o CC arch-linux2-c-opt/obj/src/tao/complementarity/impls/ssls/ssfls.o CC arch-linux2-c-opt/obj/src/tao/complementarity/impls/asls/asils.o CC arch-linux2-c-opt/obj/src/tao/unconstrained/impls/nls/nls.o CC arch-linux2-c-opt/obj/src/tao/complementarity/impls/asls/asfls.o CC arch-linux2-c-opt/obj/src/tao/unconstrained/impls/neldermead/neldermead.o CC arch-linux2-c-opt/obj/src/tao/unconstrained/impls/ntr/ntr.o CC arch-linux2-c-opt/obj/src/tao/unconstrained/impls/lmvm/lmvm.o CC arch-linux2-c-opt/obj/src/tao/unconstrained/impls/cg/taocg.o CC arch-linux2-c-opt/obj/src/tao/unconstrained/impls/bmrm/bmrm.o CC arch-linux2-c-opt/obj/src/tao/unconstrained/impls/ntl/ntl.o CC arch-linux2-c-opt/obj/src/tao/unconstrained/impls/owlqn/owlqn.o CC arch-linux2-c-opt/obj/src/tao/constrained/impls/ipm/ipm.o CC arch-linux2-c-opt/obj/src/tao/linesearch/impls/armijo/armijo.o CC arch-linux2-c-opt/obj/src/tao/linesearch/impls/morethuente/morethuente.o CC arch-linux2-c-opt/obj/src/tao/linesearch/impls/owarmijo/owarmijo.o CC arch-linux2-c-opt/obj/src/tao/linesearch/impls/unit/unit.o CC arch-linux2-c-opt/obj/src/tao/linesearch/impls/gpcglinesearch/gpcglinesearch.o CC arch-linux2-c-opt/obj/src/tao/linesearch/interface/taolinesearch.o CC arch-linux2-c-opt/obj/src/tao/linesearch/interface/dlregis_taolinesearch.o CC arch-linux2-c-opt/obj/src/tao/linesearch/interface/ftn-auto/taolinesearchf.o CC arch-linux2-c-opt/obj/src/tao/linesearch/interface/ftn-custom/ztaolinesearchf.o CC arch-linux2-c-opt/obj/src/tao/leastsquares/impls/pounders/pounders.o CC arch-linux2-c-opt/obj/src/tao/leastsquares/impls/pounders/gqt.o FC arch-linux2-c-opt/obj/src/sys/objects/f2003-src/fsrc/optionenum.o FC arch-linux2-c-opt/obj/src/sys/classes/bag/f2003-src/fsrc/bagenum.o FC arch-linux2-c-opt/obj/src/mat/f90-mod/petscmatmod.o FC arch-linux2-c-opt/obj/src/dm/f90-mod/petscdmmod.o FC arch-linux2-c-opt/obj/src/ksp/f90-mod/petsckspmod.o FC arch-linux2-c-opt/obj/src/snes/f90-mod/petscsnesmod.o FC arch-linux2-c-opt/obj/src/ts/f90-mod/petsctsmod.o FC arch-linux2-c-opt/obj/src/tao/f90-mod/petsctaomod.o CLINKER ${HOME}/projects/petsc/petsc/arch-linux2-c-opt/lib/libpetsc.so.3.8.3 gmake[2]: Leaving directory `${HOME}/projects/petsc/petsc' ========================================= gmake[1]: Leaving directory `${HOME}/projects/petsc/petsc' Now to check if the libraries are working do: make PETSC_DIR=${HOME}/projects/petsc/petsc PETSC_ARCH=arch-linux2-c-opt test ========================================= petsc $ make PETSC_DIR=${HOME}/projects/petsc/petsc PETSC_ARCH=arch-linux2-c-opt test Running test examples to verify correct installation Using PETSC_DIR=${HOME}/projects/petsc/petsc and PETSC_ARCH=arch-linux2-c-opt Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process See http://www.mcs.anl.gov/petsc/documentation/faq.html [mpiexec at crcfe01.crc.nd.edu] HYDU_create_process (utils/launch/launch.c:75): execvp error on file /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/hydra_pmi_proxy (No such file or directory) [mpiexec at crcfe01.crc.nd.edu] HYD_pmcd_pmiserv_proxy_init_cb (pm/pmiserv/pmiserv_cb.c:444): assert (!closed) failed [mpiexec at crcfe01.crc.nd.edu] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec at crcfe01.crc.nd.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec at crcfe01.crc.nd.edu] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes See http://www.mcs.anl.gov/petsc/documentation/faq.html [mpiexec at crcfe01.crc.nd.edu] HYDU_create_process (utils/launch/launch.c:75): execvp error on file /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/hydra_pmi_proxy (No such file or directory) [mpiexec at crcfe01.crc.nd.edu] HYD_pmcd_pmiserv_proxy_init_cb (pm/pmiserv/pmiserv_cb.c:444): assert (!closed) failed [mpiexec at crcfe01.crc.nd.edu] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec at crcfe01.crc.nd.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec at crcfe01.crc.nd.edu] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion 1,5c1,5 < lid velocity = 0.0016, prandtl # = 1., grashof # = 1. < 0 SNES Function norm 0.0406612 < 1 SNES Function norm 4.12227e-06 < 2 SNES Function norm 6.098e-11 < Number of SNES iterations = 2 --- > [mpiexec at crcfe01.crc.nd.edu] HYDU_create_process (utils/launch/launch.c:75): execvp error on file /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/hydra_pmi_proxy (No such file or directory) > [mpiexec at crcfe01.crc.nd.edu] HYD_pmcd_pmiserv_proxy_init_cb (pm/pmiserv/pmiserv_cb.c:444): assert (!closed) failed > [mpiexec at crcfe01.crc.nd.edu] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status > [mpiexec at crcfe01.crc.nd.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event > [mpiexec at crcfe01.crc.nd.edu] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion ${HOME}/projects/petsc/petsc/src/snes/examples/tutorials Possible problem with ex19_hypre, diffs above ========================================= 1,9c1,5 < lid velocity = 0.0625, prandtl # = 1., grashof # = 1. < 0 SNES Function norm 0.239155 < 0 KSP Residual norm 0.239155 < 1 KSP Residual norm < 1.e-11 < 1 SNES Function norm 6.81968e-05 < 0 KSP Residual norm 6.81968e-05 < 1 KSP Residual norm < 1.e-11 < 2 SNES Function norm < 1.e-11 < Number of SNES iterations = 2 --- > [mpiexec at crcfe01.crc.nd.edu] HYDU_create_process (utils/launch/launch.c:75): execvp error on file /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/hydra_pmi_proxy (No such file or directory) > [mpiexec at crcfe01.crc.nd.edu] HYD_pmcd_pmiserv_proxy_init_cb (pm/pmiserv/pmiserv_cb.c:444): assert (!closed) failed > [mpiexec at crcfe01.crc.nd.edu] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status > [mpiexec at crcfe01.crc.nd.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event > [mpiexec at crcfe01.crc.nd.edu] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion ${HOME}/projects/petsc/petsc/src/snes/examples/tutorials Possible problem with ex19_fieldsplit_fieldsplit_mumps, diffs above ========================================= 1,2c1,5 < lid velocity = 0.0025, prandtl # = 1., grashof # = 1. < Number of SNES iterations = 2 --- > [mpiexec at crcfe01.crc.nd.edu] HYDU_create_process (utils/launch/launch.c:75): execvp error on file /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/hydra_pmi_proxy (No such file or directory) > [mpiexec at crcfe01.crc.nd.edu] HYD_pmcd_pmiserv_proxy_init_cb (pm/pmiserv/pmiserv_cb.c:444): assert (!closed) failed > [mpiexec at crcfe01.crc.nd.edu] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status > [mpiexec at crcfe01.crc.nd.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event > [mpiexec at crcfe01.crc.nd.edu] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion ${HOME}/projects/petsc/petsc/src/snes/examples/tutorials Possible problem with ex19_superlu_dist, diffs above ========================================= Possible error running Fortran example src/snes/examples/tutorials/ex5f with 1 MPI process See http://www.mcs.anl.gov/petsc/documentation/faq.html [mpiexec at crcfe01.crc.nd.edu] HYDU_create_process (utils/launch/launch.c:75): execvp error on file /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/hydra_pmi_proxy (No such file or directory) [mpiexec at crcfe01.crc.nd.edu] HYD_pmcd_pmiserv_proxy_init_cb (pm/pmiserv/pmiserv_cb.c:444): assert (!closed) failed [mpiexec at crcfe01.crc.nd.edu] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec at crcfe01.crc.nd.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec at crcfe01.crc.nd.edu] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion Completed test examples ========================================= Now to evaluate the computer systems you plan use - do: make PETSC_DIR=${HOME}/projects/petsc/petsc PETSC_ARCH=arch-linux2-c-opt streams petsc $ make PETSC_DIR=${HOME}/projects/petsc/petsc PETSC_ARCH=arch-linux2-c-opt streams cd src/benchmarks/streams; /usr/bin/gmake --no-print-directory PETSC_DIR=${HOME}/projects/petsc/petsc PETSC_ARCH=arch-linux2-c-opt streams mpicc -o MPIVersion.o -c -fPIC -fopenmp -g -O -I${HOME}/projects/petsc/petsc/include -I${HOME}/projects/petsc/petsc/arch-linux2-c-opt/include `pwd`/MPIVersion.c Running streams with 'mpiexec ' using 'NPMAX=24' [mpiexec at crcfe01.crc.nd.edu] HYDU_create_process (utils/launch/launch.c:75): execvp error on file /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/hydra_pmi_proxy (No such file or directory) [mpiexec at crcfe01.crc.nd.edu] HYD_pmcd_pmiserv_proxy_init_cb (pm/pmiserv/pmiserv_cb.c:444): assert (!closed) failed [mpiexec at crcfe01.crc.nd.edu] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec at crcfe01.crc.nd.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec at crcfe01.crc.nd.edu] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion [mpiexec at crcfe01.crc.nd.edu] HYDU_create_process (utils/launch/launch.c:75): execvp error on file /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/hydra_pmi_proxy (No such file or directory) [mpiexec at crcfe01.crc.nd.edu] HYD_pmcd_pmiserv_proxy_init_cb (pm/pmiserv/pmiserv_cb.c:444): assert (!closed) failed [mpiexec at crcfe01.crc.nd.edu] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec at crcfe01.crc.nd.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec at crcfe01.crc.nd.edu] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion [mpiexec at crcfe01.crc.nd.edu] HYDU_create_process (utils/launch/launch.c:75): execvp error on file /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/hydra_pmi_proxy (No such file or directory) [mpiexec at crcfe01.crc.nd.edu] HYD_pmcd_pmiserv_proxy_init_cb (pm/pmiserv/pmiserv_cb.c:444): assert (!closed) failed [mpiexec at crcfe01.crc.nd.edu] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec at crcfe01.crc.nd.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec at crcfe01.crc.nd.edu] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion [mpiexec at crcfe01.crc.nd.edu] HYDU_create_process (utils/launch/launch.c:75): execvp error on file /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/hydra_pmi_proxy (No such file or directory) [mpiexec at crcfe01.crc.nd.edu] HYD_pmcd_pmiserv_proxy_init_cb (pm/pmiserv/pmiserv_cb.c:444): assert (!closed) failed [mpiexec at crcfe01.crc.nd.edu] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec at crcfe01.crc.nd.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec at crcfe01.crc.nd.edu] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion [mpiexec at crcfe01.crc.nd.edu] HYDU_create_process (utils/launch/launch.c:75): execvp error on file /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/hydra_pmi_proxy (No such file or directory) [mpiexec at crcfe01.crc.nd.edu] HYD_pmcd_pmiserv_proxy_init_cb (pm/pmiserv/pmiserv_cb.c:444): assert (!closed) failed [mpiexec at crcfe01.crc.nd.edu] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec at crcfe01.crc.nd.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec at crcfe01.crc.nd.edu] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion [mpiexec at crcfe01.crc.nd.edu] HYDU_create_process (utils/launch/launch.c:75): execvp error on file /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/hydra_pmi_proxy (No such file or directory) [mpiexec at crcfe01.crc.nd.edu] HYD_pmcd_pmiserv_proxy_init_cb (pm/pmiserv/pmiserv_cb.c:444): assert (!closed) failed [mpiexec at crcfe01.crc.nd.edu] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec at crcfe01.crc.nd.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec at crcfe01.crc.nd.edu] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion [mpiexec at crcfe01.crc.nd.edu] HYDU_create_process (utils/launch/launch.c:75): execvp error on file /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/hydra_pmi_proxy (No such file or directory) [mpiexec at crcfe01.crc.nd.edu] HYD_pmcd_pmiserv_proxy_init_cb (pm/pmiserv/pmiserv_cb.c:444): assert (!closed) failed [mpiexec at crcfe01.crc.nd.edu] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec at crcfe01.crc.nd.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec at crcfe01.crc.nd.edu] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion [mpiexec at crcfe01.crc.nd.edu] HYDU_create_process (utils/launch/launch.c:75): execvp error on file /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/hydra_pmi_proxy (No such file or directory) [mpiexec at crcfe01.crc.nd.edu] HYD_pmcd_pmiserv_proxy_init_cb (pm/pmiserv/pmiserv_cb.c:444): assert (!closed) failed [mpiexec at crcfe01.crc.nd.edu] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec at crcfe01.crc.nd.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec at crcfe01.crc.nd.edu] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion [mpiexec at crcfe01.crc.nd.edu] HYDU_create_process (utils/launch/launch.c:75): execvp error on file /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/hydra_pmi_proxy (No such file or directory) [mpiexec at crcfe01.crc.nd.edu] HYD_pmcd_pmiserv_proxy_init_cb (pm/pmiserv/pmiserv_cb.c:444): assert (!closed) failed [mpiexec at crcfe01.crc.nd.edu] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec at crcfe01.crc.nd.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec at crcfe01.crc.nd.edu] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion [mpiexec at crcfe01.crc.nd.edu] HYDU_create_process (utils/launch/launch.c:75): execvp error on file /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/hydra_pmi_proxy (No such file or directory) [mpiexec at crcfe01.crc.nd.edu] HYD_pmcd_pmiserv_proxy_init_cb (pm/pmiserv/pmiserv_cb.c:444): assert (!closed) failed [mpiexec at crcfe01.crc.nd.edu] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec at crcfe01.crc.nd.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec at crcfe01.crc.nd.edu] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion [mpiexec at crcfe01.crc.nd.edu] HYDU_create_process (utils/launch/launch.c:75): execvp error on file /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/hydra_pmi_proxy (No such file or directory) [mpiexec at crcfe01.crc.nd.edu] HYD_pmcd_pmiserv_proxy_init_cb (pm/pmiserv/pmiserv_cb.c:444): assert (!closed) failed [mpiexec at crcfe01.crc.nd.edu] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec at crcfe01.crc.nd.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec at crcfe01.crc.nd.edu] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion [mpiexec at crcfe01.crc.nd.edu] HYDU_create_process (utils/launch/launch.c:75): execvp error on file /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/hydra_pmi_proxy (No such file or directory) [mpiexec at crcfe01.crc.nd.edu] HYD_pmcd_pmiserv_proxy_init_cb (pm/pmiserv/pmiserv_cb.c:444): assert (!closed) failed [mpiexec at crcfe01.crc.nd.edu] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec at crcfe01.crc.nd.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec at crcfe01.crc.nd.edu] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion [mpiexec at crcfe01.crc.nd.edu] HYDU_create_process (utils/launch/launch.c:75): execvp error on file /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/hydra_pmi_proxy (No such file or directory) [mpiexec at crcfe01.crc.nd.edu] HYD_pmcd_pmiserv_proxy_init_cb (pm/pmiserv/pmiserv_cb.c:444): assert (!closed) failed [mpiexec at crcfe01.crc.nd.edu] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec at crcfe01.crc.nd.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec at crcfe01.crc.nd.edu] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion [mpiexec at crcfe01.crc.nd.edu] HYDU_create_process (utils/launch/launch.c:75): execvp error on file /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/hydra_pmi_proxy (No such file or directory) [mpiexec at crcfe01.crc.nd.edu] HYD_pmcd_pmiserv_proxy_init_cb (pm/pmiserv/pmiserv_cb.c:444): assert (!closed) failed [mpiexec at crcfe01.crc.nd.edu] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec at crcfe01.crc.nd.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec at crcfe01.crc.nd.edu] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion [mpiexec at crcfe01.crc.nd.edu] HYDU_create_process (utils/launch/launch.c:75): execvp error on file /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/hydra_pmi_proxy (No such file or directory) [mpiexec at crcfe01.crc.nd.edu] HYD_pmcd_pmiserv_proxy_init_cb (pm/pmiserv/pmiserv_cb.c:444): assert (!closed) failed [mpiexec at crcfe01.crc.nd.edu] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec at crcfe01.crc.nd.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec at crcfe01.crc.nd.edu] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion [mpiexec at crcfe01.crc.nd.edu] HYDU_create_process (utils/launch/launch.c:75): execvp error on file /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/hydra_pmi_proxy (No such file or directory) [mpiexec at crcfe01.crc.nd.edu] HYD_pmcd_pmiserv_proxy_init_cb (pm/pmiserv/pmiserv_cb.c:444): assert (!closed) failed [mpiexec at crcfe01.crc.nd.edu] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec at crcfe01.crc.nd.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec at crcfe01.crc.nd.edu] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion [mpiexec at crcfe01.crc.nd.edu] HYDU_create_process (utils/launch/launch.c:75): execvp error on file /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/hydra_pmi_proxy (No such file or directory) [mpiexec at crcfe01.crc.nd.edu] HYD_pmcd_pmiserv_proxy_init_cb (pm/pmiserv/pmiserv_cb.c:444): assert (!closed) failed [mpiexec at crcfe01.crc.nd.edu] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec at crcfe01.crc.nd.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec at crcfe01.crc.nd.edu] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion [mpiexec at crcfe01.crc.nd.edu] HYDU_create_process (utils/launch/launch.c:75): execvp error on file /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/hydra_pmi_proxy (No such file or directory) [mpiexec at crcfe01.crc.nd.edu] HYD_pmcd_pmiserv_proxy_init_cb (pm/pmiserv/pmiserv_cb.c:444): assert (!closed) failed [mpiexec at crcfe01.crc.nd.edu] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec at crcfe01.crc.nd.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec at crcfe01.crc.nd.edu] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion [mpiexec at crcfe01.crc.nd.edu] HYDU_create_process (utils/launch/launch.c:75): execvp error on file /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/hydra_pmi_proxy (No such file or directory) [mpiexec at crcfe01.crc.nd.edu] HYD_pmcd_pmiserv_proxy_init_cb (pm/pmiserv/pmiserv_cb.c:444): assert (!closed) failed [mpiexec at crcfe01.crc.nd.edu] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec at crcfe01.crc.nd.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec at crcfe01.crc.nd.edu] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion [mpiexec at crcfe01.crc.nd.edu] HYDU_create_process (utils/launch/launch.c:75): execvp error on file /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/hydra_pmi_proxy (No such file or directory) [mpiexec at crcfe01.crc.nd.edu] HYD_pmcd_pmiserv_proxy_init_cb (pm/pmiserv/pmiserv_cb.c:444): assert (!closed) failed [mpiexec at crcfe01.crc.nd.edu] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec at crcfe01.crc.nd.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec at crcfe01.crc.nd.edu] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion [mpiexec at crcfe01.crc.nd.edu] HYDU_create_process (utils/launch/launch.c:75): execvp error on file /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/hydra_pmi_proxy (No such file or directory) [mpiexec at crcfe01.crc.nd.edu] HYD_pmcd_pmiserv_proxy_init_cb (pm/pmiserv/pmiserv_cb.c:444): assert (!closed) failed [mpiexec at crcfe01.crc.nd.edu] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec at crcfe01.crc.nd.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec at crcfe01.crc.nd.edu] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion [mpiexec at crcfe01.crc.nd.edu] HYDU_create_process (utils/launch/launch.c:75): execvp error on file /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/hydra_pmi_proxy (No such file or directory) [mpiexec at crcfe01.crc.nd.edu] HYD_pmcd_pmiserv_proxy_init_cb (pm/pmiserv/pmiserv_cb.c:444): assert (!closed) failed [mpiexec at crcfe01.crc.nd.edu] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec at crcfe01.crc.nd.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec at crcfe01.crc.nd.edu] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion [mpiexec at crcfe01.crc.nd.edu] HYDU_create_process (utils/launch/launch.c:75): execvp error on file /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/hydra_pmi_proxy (No such file or directory) [mpiexec at crcfe01.crc.nd.edu] HYD_pmcd_pmiserv_proxy_init_cb (pm/pmiserv/pmiserv_cb.c:444): assert (!closed) failed [mpiexec at crcfe01.crc.nd.edu] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec at crcfe01.crc.nd.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec at crcfe01.crc.nd.edu] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion [mpiexec at crcfe01.crc.nd.edu] HYDU_create_process (utils/launch/launch.c:75): execvp error on file /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/hydra_pmi_proxy (No such file or directory) [mpiexec at crcfe01.crc.nd.edu] HYD_pmcd_pmiserv_proxy_init_cb (pm/pmiserv/pmiserv_cb.c:444): assert (!closed) failed [mpiexec at crcfe01.crc.nd.edu] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec at crcfe01.crc.nd.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec at crcfe01.crc.nd.edu] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion ------------------------------------------------ -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 3517506 bytes Desc: not available URL: From balay at mcs.anl.gov Thu Jan 25 19:09:59 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 25 Jan 2018 19:09:59 -0600 Subject: [petsc-users] Install Petsc from repository "--download-fblaslapack libraries cannot be used" In-Reply-To: References: Message-ID: Executing: mpicc -show stdout: gcc -I/opt/crc/o/openmpi/2.0.1/gcc/6.2.0/eth/include -pthread -Wl,-rpath -Wl,/opt/crc/o/openmpi/2.0.1/gcc/6.2.0/eth/lib -Wl,--enable-new-dtags -L/opt/crc/o/openmpi/2.0.1/gcc/6.2.0/eth/lib -lmpi Checking for program /afs/crc/x86_64_linux/p/paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/bin/mpiexec...found You have an incompatible 'mpiexec' in your PATH. Use mpiexec that matches mpicc that you are using. You can either fix your PATH - or use --with-mpiexec=/opt/crc/o/openmpi/2.0.1/gcc/6.2.0/eth/bin/mpiexec [assuming thats the correct one] Satish On Fri, 26 Jan 2018, John Haase wrote: > Satish, > > I ran the command > > PETSC_DIR="${HOME}/projects/petsc/petsc" > > PETSC_ARCH="gcc" > > ./configure \ > > --configModules=PETSc.Configure \ > > --optionsModule=config.compilerOptions \ > > --download-hypre=1 \ > > --with-ssl=0 \ > > --with-debugging=no \ > > --with-pic=1 \ > > --with-shared-libraries=1 \ > > --with-cc=mpicc \ > > --with-cxx=mpicxx \ > > --with-fc=mpif90 \ > > --download-fblaslapack=1 \ > > --download-metis=1 \ > > --download-parmetis=1 \ > > --download-superlu_dist=1 \ > > --download-scalapack=1 \ > > --download-mumps=1 \ > > CC=mpicc CXX=mpicxx FC=mpif90 F77=mpif77 F90=mpif90 \ > > CFLAGS="-fPIC -fopenmp" CXXFLAGS="-fPIC -fopenmp" \ > > FFLAGS="-fPIC -fopenmp" FCFLAGS="-fPIC -fopenmp" \ > > F90FLAGS="-fPIC -fopenmp" F77FLAGS="-fPIC -fopenmp" \ > > PETSC_DIR=${HOME}/projects/petsc/petsc > > > The process threw errors after the 2nd make statement, but still gives the > 3rd make statement. However, the third make statement throws a ton of > errors. See attached console output and configure.log > > Regards, > > John R. Haase > jhaase1 at nd.edu > > On Thu, Jan 25, 2018 at 7:01 PM, Satish Balay wrote: > > > > Configure Options: --configModules=PETSc.Configure > > --optionsModule=config.compilerOptions --download-hypre=1 --with-ssl=0 > > --with-debugging=no --with-pic=1 --with-shared-libraries=1 --with-cc=mpicc > > --with-cxx=mpicxx --with-fc=mpif90 --download-fblaslapack=1 > > --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 > > --download-scalapack=1 --download-mumps=1 CC=mpicc CXX=mpicxx FC=mpif90 > > F77=mpif77 F90=mpif90 CFLAGS="-fPIC -fopenmp" CXXFLAGS="-fPIC -fopenmp" > > FFLAGS="-fPIC -fopenmp" FCFLAGS="-fPIC -fopenmp" F90FLAGS="-fPIC -fopenmp" > > F77FLAGS="-fPIC -fopenmp" PETSC_DIR=~/project/petsc/petsc > > > > > > use ${HOME} instead of ~ above. > > > > > /usr/bin/ld: cannot find -lfblas > > > > And delete your build dir i.e 'rm -rf ~/project/petsc/petsc/arch- > > linux2-c-opt/' > > > > And see if the problem persists. > > > > Satish > > > > On Thu, 25 Jan 2018, John Haase wrote: > > > > > Hello all, > > > > > > I am trying to install an bleeding edge version of Petsc ( > > > https://bitbucket.org/petsc/petsc/pull-requests/843/added- > > api-snesgetforceiteration/diff). > > > However, when I try to install with the command > > > > > > ./configure --prefix=$PETSC_DIR/../installed --download-hypre=1 > > > --with-ssl=0 --with-debugging=no --with-pic=1 --with-shared-libraries=1 > > > --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > > --download-fblaslapack=1 > > > --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 > > > --download-scalapack=1 --download-mumps=1 CC=mpicc CXX=mpicxx FC=mpif90 > > > F77=mpif77 F90=mpif90 CFLAGS='-fPIC -fopenmp' CXXFLAGS='-fPIC -fopenmp' > > > FFLAGS='-fPIC -fopenmp' FCFLAGS='-fPIC -fopenmp' F90FLAGS='-fPIC > > -fopenmp' > > > F77FLAGS='-fPIC -fopenmp' PETSC_DIR=`pwd` > > > > > > I get the error "--download-fblaslapack libraries cannot be used". Any > > > suggestions? > > > > > > I've attached configure.log in case I'm not reporting something > > important. > > > > > > Thank you for your help > > > > > > John > > > > > > Regards, > > > > > > John R. Haase > > > jhaase1 at nd.edu > > > > > > > > From jhaase1 at nd.edu Thu Jan 25 19:45:16 2018 From: jhaase1 at nd.edu (John Haase) Date: Thu, 25 Jan 2018 20:45:16 -0500 Subject: [petsc-users] Install Petsc from repository "--download-fblaslapack libraries cannot be used" In-Reply-To: References: Message-ID: That works, thank you very much! Regards, John R. Haase jhaase1 at nd.edu On Thu, Jan 25, 2018 at 8:09 PM, Satish Balay wrote: > Executing: mpicc -show > stdout: gcc -I/opt/crc/o/openmpi/2.0.1/gcc/6.2.0/eth/include -pthread > -Wl,-rpath -Wl,/opt/crc/o/openmpi/2.0.1/gcc/6.2.0/eth/lib > -Wl,--enable-new-dtags -L/opt/crc/o/openmpi/2.0.1/gcc/6.2.0/eth/lib -lmpi > > Checking for program /afs/crc/x86_64_linux/p/ > paraview/5.3.0/ParaView-5.3.0-Qt5-OpenGL2-MPI-Linux-64bit/ > bin/mpiexec...found > > You have an incompatible 'mpiexec' in your PATH. Use mpiexec that matches > mpicc that you are using. > > You can either fix your PATH - or use --with-mpiexec=/opt/crc/o/ > openmpi/2.0.1/gcc/6.2.0/eth/bin/mpiexec [assuming thats the correct one] > > Satish > > > On Fri, 26 Jan 2018, John Haase wrote: > > > Satish, > > > > I ran the command > > > > PETSC_DIR="${HOME}/projects/petsc/petsc" > > > PETSC_ARCH="gcc" > > > ./configure \ > > > --configModules=PETSc.Configure \ > > > --optionsModule=config.compilerOptions \ > > > --download-hypre=1 \ > > > --with-ssl=0 \ > > > --with-debugging=no \ > > > --with-pic=1 \ > > > --with-shared-libraries=1 \ > > > --with-cc=mpicc \ > > > --with-cxx=mpicxx \ > > > --with-fc=mpif90 \ > > > --download-fblaslapack=1 \ > > > --download-metis=1 \ > > > --download-parmetis=1 \ > > > --download-superlu_dist=1 \ > > > --download-scalapack=1 \ > > > --download-mumps=1 \ > > > CC=mpicc CXX=mpicxx FC=mpif90 F77=mpif77 F90=mpif90 \ > > > CFLAGS="-fPIC -fopenmp" CXXFLAGS="-fPIC -fopenmp" \ > > > FFLAGS="-fPIC -fopenmp" FCFLAGS="-fPIC -fopenmp" \ > > > F90FLAGS="-fPIC -fopenmp" F77FLAGS="-fPIC -fopenmp" \ > > > PETSC_DIR=${HOME}/projects/petsc/petsc > > > > > > The process threw errors after the 2nd make statement, but still gives > the > > 3rd make statement. However, the third make statement throws a ton of > > errors. See attached console output and configure.log > > > > Regards, > > > > John R. Haase > > jhaase1 at nd.edu > > > > On Thu, Jan 25, 2018 at 7:01 PM, Satish Balay wrote: > > > > > > Configure Options: --configModules=PETSc.Configure > > > --optionsModule=config.compilerOptions --download-hypre=1 --with-ssl=0 > > > --with-debugging=no --with-pic=1 --with-shared-libraries=1 > --with-cc=mpicc > > > --with-cxx=mpicxx --with-fc=mpif90 --download-fblaslapack=1 > > > --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 > > > --download-scalapack=1 --download-mumps=1 CC=mpicc CXX=mpicxx FC=mpif90 > > > F77=mpif77 F90=mpif90 CFLAGS="-fPIC -fopenmp" CXXFLAGS="-fPIC -fopenmp" > > > FFLAGS="-fPIC -fopenmp" FCFLAGS="-fPIC -fopenmp" F90FLAGS="-fPIC > -fopenmp" > > > F77FLAGS="-fPIC -fopenmp" PETSC_DIR=~/project/petsc/petsc > > > > > > > > > use ${HOME} instead of ~ above. > > > > > > > /usr/bin/ld: cannot find -lfblas > > > > > > And delete your build dir i.e 'rm -rf ~/project/petsc/petsc/arch- > > > linux2-c-opt/' > > > > > > And see if the problem persists. > > > > > > Satish > > > > > > On Thu, 25 Jan 2018, John Haase wrote: > > > > > > > Hello all, > > > > > > > > I am trying to install an bleeding edge version of Petsc ( > > > > https://bitbucket.org/petsc/petsc/pull-requests/843/added- > > > api-snesgetforceiteration/diff). > > > > However, when I try to install with the command > > > > > > > > ./configure --prefix=$PETSC_DIR/../installed --download-hypre=1 > > > > --with-ssl=0 --with-debugging=no --with-pic=1 > --with-shared-libraries=1 > > > > --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > > > --download-fblaslapack=1 > > > > --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 > > > > --download-scalapack=1 --download-mumps=1 CC=mpicc CXX=mpicxx > FC=mpif90 > > > > F77=mpif77 F90=mpif90 CFLAGS='-fPIC -fopenmp' CXXFLAGS='-fPIC > -fopenmp' > > > > FFLAGS='-fPIC -fopenmp' FCFLAGS='-fPIC -fopenmp' F90FLAGS='-fPIC > > > -fopenmp' > > > > F77FLAGS='-fPIC -fopenmp' PETSC_DIR=`pwd` > > > > > > > > I get the error "--download-fblaslapack libraries cannot be used". > Any > > > > suggestions? > > > > > > > > I've attached configure.log in case I'm not reporting something > > > important. > > > > > > > > Thank you for your help > > > > > > > > John > > > > > > > > Regards, > > > > > > > > John R. Haase > > > > jhaase1 at nd.edu > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen.wornom at inria.fr Fri Jan 26 03:39:14 2018 From: stephen.wornom at inria.fr (Stephen Wornom) Date: Fri, 26 Jan 2018 10:39:14 +0100 (CET) Subject: [petsc-users] Novice: How to implement the petsc Gmres libraries in my CFD code In-Reply-To: <459539046.6751984.1516957212045.JavaMail.zimbra@inria.fr> Message-ID: <1533977207.6781492.1516959554228.JavaMail.zimbra@inria.fr> I am disparate to implement the petsc library gmres in an CFD code and need a little help to get going. 23 years ago, gmres was implemented/written in the code by Y. Saad, modified by A. Malevsky, version February 1, 1995. petsc, most likely started with the same version, added new solvers, ...etc and maintained and corrected bugs reported by users and I would like to use the latest petsc version of gmres Why? Normally we have no convergence problems using gmres. However sometimes gmres diverges instantaneously, no warning, just a negative density. Our code is parallel mpi, unstructured, uses a low Mach preconditioner, FV Roe scheme. Thus I would like to add an petsc option. Would someone outline, in detail, the steps that I need to follow? I need all the encourage that I can get to add the option in the code. petsc is installed on our computers Thanks in advance, Stephen IF ( petsc .EQ. 1 ) THEN c petsc CALL MatAssembleBegin c petsc CALL MatAssembleEnd ... add steps ELSE CALL GMRESASR ! from our code ENDIF This is from our code which may be useful information c----------------------------------------------------------------------- c flexible GMRES routine. This is a version of GMRES which allows a c a variable preconditioner. Implemented with a reverse communication c protocole for flexibility - c DISTRIBUTED VERSION (USES DISTDOT FOR DDOT) c explicit (exact) residual norms for restarts c written by Y. Saad, modified by A. Malevsky, version February 1, 1995 c----------------------------------------------------------------------- c This Is A Reverse Communication Implementation. c------------------------------------------------- c USAGE: (see also comments for icode below). CGMRES c should be put in a loop and the loop should be active for as c long as icode is not equal to 0. On RETURN fgmres will c 1) either be requesting the new preconditioned vector applied c to wk1 in case icode.eq.1 (result should be put in wk2) c 2) or be requesting the product of A applied to the vector wk1 c in case icode.eq.2 (result should be put in wk2) c 3) or be terminated in case icode .eq. 0. c on entry always set icode = 0. So icode should be set back to zero c upon convergence. c----------------------------------------------------------------------- c Here is a typical way of running fgmres: c c icode = 0 c 1 continue c CALL fgmres (n,im,rhs,sol,i,vv,w,wk1, wk2,eps,maxits,iout,icode) c c if (icode .eq. 1) then c CALL precon(n, wk1, wk2) <--- user's variable preconditioning c goto 1 c else if (icode .ge. 2) then c CALL matvec (n,wk1, wk2) <--- user's matrix vector product. c goto 1 c else c ----- done ---- c ......... -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jan 26 07:25:48 2018 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 27 Jan 2018 00:25:48 +1100 Subject: [petsc-users] Novice: How to implement the petsc Gmres libraries in my CFD code In-Reply-To: <1533977207.6781492.1516959554228.JavaMail.zimbra@inria.fr> References: <459539046.6751984.1516957212045.JavaMail.zimbra@inria.fr> <1533977207.6781492.1516959554228.JavaMail.zimbra@inria.fr> Message-ID: On Fri, Jan 26, 2018 at 8:39 PM, Stephen Wornom wrote: > I am disparate to implement the petsc library gmres in an CFD code and > need a little help to get going. > > 23 years ago, gmres was implemented/written in the code by Y. Saad, > modified by A. Malevsky, version February 1, 1995. > > petsc, most likely started with the same version, added new solvers, > ...etc and maintained and corrected bugs reported by users > and I would like to use the latest petsc version of gmres > > Why? > Normally we have no convergence problems using gmres. > However sometimes gmres diverges instantaneously, no warning, just a > negative density. > Our code is parallel mpi, unstructured, uses a low Mach preconditioner, FV > Roe scheme. > > Thus I would like to add an petsc option. > Would someone outline, in detail, the steps that I need to follow? > I need all the encourage that I can get to add the option in the code. > I normally divide this process into four steps: 1) Call PetscInitialize() and PetscFinalize() once in the code. We usually do this at the beginning and the end of the code, but you can do it anywhere you want as long as Init precedes any PETSc calls, and Fin follows them. 2) Convert the input vectors to PETSc Vecs This should be easy. We have wrapper constructors http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecCreateSeqWithArray.html http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecCreateMPIWithArray.html#VecCreateMPIWithArray There is an example of using these in Fortran90 to create a solver interface that takes raw arrays http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex13f90.F90.html 3) Convert the system matrix to a PETSc Mat If you do not want to use PETSc preconditioners, you can easily do this by wrapping just the action of the matrix: http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatShellSetOperation.html and there are Fortran examples listed at the bottom of that page. If instead you want to enable preconditioners, then you must put the values into a PETSc Mat structure, and you would use MatSetValues() in the naive way for that. 4) Create a solver, pass it the system matrix, rhs, and solution There are many examples of this. For instance http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex1f.F90.html Thanks, Matt > petsc is installed on our computers > Thanks in advance, > Stephen > > > IF ( petsc .EQ. 1 ) THEN > c petsc CALL MatAssembleBegin > c petsc CALL MatAssembleEnd > ... add steps > ELSE > CALL GMRESASR ! from our code > ENDIF > > > > This is from our code which may be useful information > c----------------------------------------------------------------------- > c flexible GMRES routine. This is a version of GMRES which allows a > c a variable preconditioner. Implemented with a reverse communication > c protocole for flexibility - > c DISTRIBUTED VERSION (USES DISTDOT FOR DDOT) > c explicit (exact) residual norms for restarts > c written by Y. Saad, modified by A. Malevsky, version February 1, 1995 > c----------------------------------------------------------------------- > c This Is A Reverse Communication Implementation. > c------------------------------------------------- > c USAGE: (see also comments for icode below). CGMRES > c should be put in a loop and the loop should be active for as > c long as icode is not equal to 0. On RETURN fgmres will > c 1) either be requesting the new preconditioned vector applied > c to wk1 in case icode.eq.1 (result should be put in wk2) > c 2) or be requesting the product of A applied to the vector wk1 > c in case icode.eq.2 (result should be put in wk2) > c 3) or be terminated in case icode .eq. 0. > c on entry always set icode = 0. So icode should be set back to zero > c upon convergence. > c----------------------------------------------------------------------- > c Here is a typical way of running fgmres: > c > c icode = 0 > c 1 continue > c CALL fgmres (n,im,rhs,sol,i,vv,w,wk1, wk2,eps,maxits,iout,icode) > c > c if (icode .eq. 1) then > c CALL precon(n, wk1, wk2) <--- user's variable preconditioning > c goto 1 > c else if (icode .ge. 2) then > c CALL matvec (n,wk1, wk2) <--- user's matrix vector product. > c goto 1 > c else > c ----- done ---- > c ......... > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Jan 26 08:55:09 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 26 Jan 2018 14:55:09 +0000 Subject: [petsc-users] Novice: How to implement the petsc Gmres libraries in my CFD code In-Reply-To: <1533977207.6781492.1516959554228.JavaMail.zimbra@inria.fr> References: <1533977207.6781492.1516959554228.JavaMail.zimbra@inria.fr> Message-ID: <6016B4BF-7B57-4FDC-990E-31F8E41E2616@anl.gov> > On Jan 26, 2018, at 3:39 AM, Stephen Wornom wrote: > > I am disparate to implement the petsc library gmres in an CFD code and need a little help to get going. > > 23 years ago, gmres was implemented/written in the code by Y. Saad, modified by A. Malevsky, version February 1, 1995. Off by a decade or so. > > petsc, most likely started with the same version, added new solvers, ...etc and maintained and corrected bugs reported by users > and I would like to use the latest petsc version of gmres > > Why? > Normally we have no convergence problems using gmres. > However sometimes gmres diverges instantaneously, no warning, just a negative density. A negative density likely has nothing to do with your gmres. You should run your code with a very tight GMRES convergence tolerance and see what happens and how and when the density goes negative; put in debugging code in your solver and your GEMRES that checks for negative density at each step. Does a tighter tolerance lead to a later time for the negative density? GMRES is GMRES, I don't think converting to the PETSc GMRES will make an difference with your negative density problem. > Our code is parallel mpi, unstructured, uses a low Mach preconditioner, FV Roe scheme. > > Thus I would like to add an petsc option. > Would someone outline, in detail, the steps that I need to follow? > I need all the encourage that I can get to add the option in the code. > petsc is installed on our computers > Thanks in advance, > Stephen > > > IF ( petsc .EQ. 1 ) THEN > c petsc CALL MatAssembleBegin > c petsc CALL MatAssembleEnd > ... add steps > ELSE > CALL GMRESASR ! from our code > ENDIF > > > > This is from our code which may be useful information > c----------------------------------------------------------------------- > c flexible GMRES routine. This is a version of GMRES which allows a > c a variable preconditioner. Implemented with a reverse communication > c protocole for flexibility - > c DISTRIBUTED VERSION (USES DISTDOT FOR DDOT) > c explicit (exact) residual norms for restarts > c written by Y. Saad, modified by A. Malevsky, version February 1, 1995 > c----------------------------------------------------------------------- > c This Is A Reverse Communication Implementation. > c------------------------------------------------- > c USAGE: (see also comments for icode below). CGMRES > c should be put in a loop and the loop should be active for as > c long as icode is not equal to 0. On RETURN fgmres will > c 1) either be requesting the new preconditioned vector applied > c to wk1 in case icode.eq.1 (result should be put in wk2) > c 2) or be requesting the product of A applied to the vector wk1 > c in case icode.eq.2 (result should be put in wk2) > c 3) or be terminated in case icode .eq. 0. > c on entry always set icode = 0. So icode should be set back to zero > c upon convergence. > c----------------------------------------------------------------------- > c Here is a typical way of running fgmres: > c > c icode = 0 > c 1 continue > c CALL fgmres (n,im,rhs,sol,i,vv,w,wk1, wk2,eps,maxits,iout,icode) > c > c if (icode .eq. 1) then > c CALL precon(n, wk1, wk2) <--- user's variable preconditioning > c goto 1 > c else if (icode .ge. 2) then > c CALL matvec (n,wk1, wk2) <--- user's matrix vector product. > c goto 1 > c else > c ----- done ---- > c ......... > From knepley at gmail.com Fri Jan 26 09:01:27 2018 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 27 Jan 2018 02:01:27 +1100 Subject: [petsc-users] Novice: How to implement the petsc Gmres libraries in my CFD code In-Reply-To: <6016B4BF-7B57-4FDC-990E-31F8E41E2616@anl.gov> References: <1533977207.6781492.1516959554228.JavaMail.zimbra@inria.fr> <6016B4BF-7B57-4FDC-990E-31F8E41E2616@anl.gov> Message-ID: On Sat, Jan 27, 2018 at 1:55 AM, Smith, Barry F. wrote: > > > > On Jan 26, 2018, at 3:39 AM, Stephen Wornom > wrote: > > > > I am disparate to implement the petsc library gmres in an CFD code and > need a little help to get going. > > > > 23 years ago, gmres was implemented/written in the code by Y. Saad, > modified by A. Malevsky, version February 1, 1995. > > Off by a decade or so. > > > > petsc, most likely started with the same version, added new solvers, > ...etc and maintained and corrected bugs reported by users > > and I would like to use the latest petsc version of gmres > > > > Why? > > Normally we have no convergence problems using gmres. > > However sometimes gmres diverges instantaneously, no warning, just a > negative density. > > A negative density likely has nothing to do with your gmres. You > should run your code with a very tight GMRES convergence tolerance and see > what happens and how and when the density goes negative; put in debugging > code in your solver and your GEMRES that checks for negative density at > each step. Does a tighter tolerance lead to a later time for the negative > density? > > GMRES is GMRES, I don't think converting to the PETSc GMRES will make > an difference with your negative density problem. > As long as we are making guesses, there are different algorithms for orthogonalization, some of which are better at dealing with ill-conditioned matrices. If you are using classical Gram-Schmidt, you could try Householder or modified GS. Thanks, Matt > > Our code is parallel mpi, unstructured, uses a low Mach preconditioner, > FV Roe scheme. > > > > Thus I would like to add an petsc option. > > Would someone outline, in detail, the steps that I need to follow? > > I need all the encourage that I can get to add the option in the code. > > petsc is installed on our computers > > Thanks in advance, > > Stephen > > > > > > IF ( petsc .EQ. 1 ) THEN > > c petsc CALL MatAssembleBegin > > c petsc CALL MatAssembleEnd > > ... add steps > > ELSE > > CALL GMRESASR ! from our code > > ENDIF > > > > > > > > This is from our code which may be useful information > > c----------------------------------------------------------------------- > > c flexible GMRES routine. This is a version of GMRES which allows a > > c a variable preconditioner. Implemented with a reverse communication > > c protocole for flexibility - > > c DISTRIBUTED VERSION (USES DISTDOT FOR DDOT) > > c explicit (exact) residual norms for restarts > > c written by Y. Saad, modified by A. Malevsky, version February 1, 1995 > > c----------------------------------------------------------------------- > > c This Is A Reverse Communication Implementation. > > c------------------------------------------------- > > c USAGE: (see also comments for icode below). CGMRES > > c should be put in a loop and the loop should be active for as > > c long as icode is not equal to 0. On RETURN fgmres will > > c 1) either be requesting the new preconditioned vector applied > > c to wk1 in case icode.eq.1 (result should be put in wk2) > > c 2) or be requesting the product of A applied to the vector wk1 > > c in case icode.eq.2 (result should be put in wk2) > > c 3) or be terminated in case icode .eq. 0. > > c on entry always set icode = 0. So icode should be set back to zero > > c upon convergence. > > c----------------------------------------------------------------------- > > c Here is a typical way of running fgmres: > > c > > c icode = 0 > > c 1 continue > > c CALL fgmres (n,im,rhs,sol,i,vv,w,wk1, wk2,eps,maxits,iout,icode) > > c > > c if (icode .eq. 1) then > > c CALL precon(n, wk1, wk2) <--- user's variable > preconditioning > > c goto 1 > > c else if (icode .ge. 2) then > > c CALL matvec (n,wk1, wk2) <--- user's matrix vector product. > > c goto 1 > > c else > > c ----- done ---- > > c ......... > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jan 26 09:41:37 2018 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 27 Jan 2018 02:41:37 +1100 Subject: [petsc-users] Is there a partition section in a distributed DM object? In-Reply-To: References: Message-ID: On Wed, Jan 24, 2018 at 3:17 PM, Jordan Wagner wrote: > Hi, > > At some point, I would like to export my distributed DM along with an > associated scalar field corresponding to the process rank for which each > point is owned. I know that what I need is the partition section, which is > more or less identical to Fig. 6 in Knepley and Karpeev (2009). I was > wondering if this particular section is already made behind the scenes when > distributing and just hiding somewhere in the DM object. > > I know it shouldn't be difficult to just make the section myself, but it > seems like something that would be needed a lot and already made. I haven't > been successful in finding it, so I just wanted to make sure I am not > missing anything. > We do not keep the partitioning around, since you do not need it after it is used. Second, it is stored as a PetscSection+IS, which is the inverse of what you want. I would just make a Vec with the rank. It should also be possible to use an IS, but I am not sure parallel I/O is completely right for IS whereas I know everything works for Vec. Thanks, Matt > Thanks very much > > jw > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bcyee at umich.edu Fri Jan 26 09:48:07 2018 From: bcyee at umich.edu (Ben Yee) Date: Fri, 26 Jan 2018 10:48:07 -0500 Subject: [petsc-users] Changing the operator for PCMG while leaving interpolation matrices unchanged In-Reply-To: References: Message-ID: Hi Barry, This seems to work for me. I defined the following function in our code repository, and I call it every time I use KSPSetOperators to update the operator for my KSP object with PCMG: PetscErrorCode PCMGSoftReset(PC pc) { PC_MG *mg = (PC_MG*)pc->data; PC_MG_Levels **mglevels = mg->levels; PetscErrorCode ierr; PetscInt i,n; PetscFunctionBegin; if (mglevels) { n = mglevels[0]->levels; for (i=0; iA);CHKERRQ(ierr); if (mglevels[i]->smoothd != mglevels[i]->smoothu) { ierr = KSPReset(mglevels[i]->smoothd);CHKERRQ(ierr); } ierr = KSPReset(mglevels[i]->smoothu);CHKERRQ(ierr); } } pc->setupcalled = PETSC_FALSE; PetscFunctionReturn(0); } (Note that I had to add "pc->setupcalled = PETSC_FALSE".) Thanks for the help! On Wed, Jan 24, 2018 at 12:19 PM, Smith, Barry F. wrote: > > This is off the top of my head and my not be useful. Copy PCReset_MG() > which is > > PetscErrorCode PCReset_MG(PC pc) > { > PC_MG *mg = (PC_MG*)pc->data; > PC_MG_Levels **mglevels = mg->levels; > PetscErrorCode ierr; > PetscInt i,n; > > PetscFunctionBegin; > if (mglevels) { > n = mglevels[0]->levels; > for (i=0; i ierr = VecDestroy(&mglevels[i+1]->r);CHKERRQ(ierr); > ierr = VecDestroy(&mglevels[i]->b);CHKERRQ(ierr); > ierr = VecDestroy(&mglevels[i]->x);CHKERRQ(ierr); > ierr = MatDestroy(&mglevels[i+1]->restrct);CHKERRQ(ierr); > ierr = MatDestroy(&mglevels[i+1]->interpolate);CHKERRQ(ierr); > ierr = VecDestroy(&mglevels[i+1]->rscale);CHKERRQ(ierr); > } > > for (i=0; i ierr = MatDestroy(&mglevels[i]->A);CHKERRQ(ierr); > if (mglevels[i]->smoothd != mglevels[i]->smoothu) { > ierr = KSPReset(mglevels[i]->smoothd);CHKERRQ(ierr); > } > ierr = KSPReset(mglevels[i]->smoothu);CHKERRQ(ierr); > } > } > PetscFunctionReturn(0); > } > > And change it to only destroy mats on each level ierr = > MatDestroy(&mglevels[i]->A);CHKERRQ(ierr); Keep everything else. > > Not that you will need to include #include in > your code. > > > On Jan 24, 2018, at 9:53 AM, Ben Yee wrote: > > > > Yes, the nonzero structure changes in a way that cannot easily be > predicted. I could overallocate, add some extra entries to the first > matrix, and reuse that matrix, but I would have to roughly double the > number of allocated entries, leading to an undesirable increase in memory. > I think that one alternative would be to just reset the KSP object when I > change the operator, but, if I'm not mistaken, this would destroy all the > interpolation operators and I'd have to redefine all of them. > > > > On Wed, Jan 24, 2018 at 10:41 AM, Smith, Barry F. > wrote: > > > > Ben > > > > This is a good question, I'll take a look at how difficult it would > be to add this feature. > > > > I take it the nonzero structure of the outer matrix changes over > time? > > > > > > Barry > > > > > > > On Jan 24, 2018, at 9:12 AM, Ben Yee wrote: > > > > > > Hi, > > > > > > I have multiple matrices (with different nonzero structures) for which > I would like to use the same KSP operator (KSPRICHARDSON with PCMG, > Galerkin process for generating coarse grid operator). I want to use the > same PCMG structure for all the matrices (same interpolation, same > restriction, same number of levels, same smoothing techniques, etc.), so I > tried using KSPSetOperators to just change the operator. My interpolation > matrices are MPIAIJ matrices that I have defined myself and provided to > PCMG. My hope was that, with the new operator I provided, PETSc would > recompute all of the coarse/intermediate grid operators using the Galerkin > process with the old interpolation matrices and the new fine-grid operator. > > > > > > However, I get the wrong solution after changing the operator with > KSPSetOperators. After some extensive digging around the PETSc source > code, it seems that the operators in mg.c are not updated by > KSPSetOperators. When I do a MatView on mglevels->A or the operator in > mglevels->smoothu or mglevels->smoothd in PCMGMCycle_Private() of mg.c, it > has the matrix values of the old matrix. From what I understand, the only > place where mglevels->A (used to compute the residual) is updated is in > PCApplyRIchardson_MG() of mg.c, from the following lines of code: > > > 74: for > > > (i=0; i > > > > > 75: if > > > (!mglevels[i]->A) { > > > > > > 76: KSPGetOperators > > > (mglevels[i]->smoothu,&mglevels[i]->A,NULL); > > > > > > 77: PetscObjectReference((PetscObject > > > )mglevels[i]->A); > > > > > > 78: > > > } > > > > > > 79: } > > > It seems to me that the pointer mglevels[i]->A is only set once, when > it is undefined at the beginning of the iteration scheme. I think there > are more issues as well, but this is the easiest one to point out. (For > example, it doesn't seem like it is possible to update the operator for > mglevels[i]->smoothd from KSPSetOperators on the outermost KSP object, and, > if I try to set it manually via PCMGGetSmoother, I get errors if the > nonzero structures of the coarse-grid operators have changed, which tends > to be the case when the fine-grid operator's nonzero structure changes). > > > > > > Is there something I'm doing wrong? Is there a way to do this so that > I can reuse PCMG with a new operator? > > > > > > Thanks! > > > > > > -- > > > Ben Yee > > > > > > NERS PhD Candidate, University of Michigan > > > B.S. Mech. & Nuclear Eng., U.C. Berkeley > > > > > > > > > > -- > > Ben Yee > > > > NERS PhD Candidate, University of Michigan > > B.S. Mech. & Nuclear Eng., U.C. Berkeley > > -- Ben Yee NERS PhD Candidate, University of Michigan B.S. Mech. & Nuclear Eng., U.C. Berkeley -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Jan 26 12:58:11 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 26 Jan 2018 18:58:11 +0000 Subject: [petsc-users] Changing the operator for PCMG while leaving interpolation matrices unchanged In-Reply-To: References: Message-ID: <677F7668-D0A2-4E11-98EA-F036A92F2BC8@anl.gov> Great > On Jan 26, 2018, at 9:48 AM, Ben Yee wrote: > > Hi Barry, > > This seems to work for me. I defined the following function in our code repository, and I call it every time I use KSPSetOperators to update the operator for my KSP object with PCMG: > > PetscErrorCode PCMGSoftReset(PC pc) > { > PC_MG *mg = (PC_MG*)pc->data; > PC_MG_Levels **mglevels = mg->levels; > PetscErrorCode ierr; > PetscInt i,n; > > PetscFunctionBegin; > if (mglevels) { > n = mglevels[0]->levels; > for (i=0; i ierr = MatDestroy(&mglevels[i]->A);CHKERRQ(ierr); > if (mglevels[i]->smoothd != mglevels[i]->smoothu) { > ierr = KSPReset(mglevels[i]->smoothd);CHKERRQ(ierr); > } > ierr = KSPReset(mglevels[i]->smoothu);CHKERRQ(ierr); > } > } > pc->setupcalled = PETSC_FALSE; > PetscFunctionReturn(0); > } > > (Note that I had to add "pc->setupcalled = PETSC_FALSE".) Thanks for the help! > > On Wed, Jan 24, 2018 at 12:19 PM, Smith, Barry F. wrote: > > This is off the top of my head and my not be useful. Copy PCReset_MG() which is > > PetscErrorCode PCReset_MG(PC pc) > { > PC_MG *mg = (PC_MG*)pc->data; > PC_MG_Levels **mglevels = mg->levels; > PetscErrorCode ierr; > PetscInt i,n; > > PetscFunctionBegin; > if (mglevels) { > n = mglevels[0]->levels; > for (i=0; i ierr = VecDestroy(&mglevels[i+1]->r);CHKERRQ(ierr); > ierr = VecDestroy(&mglevels[i]->b);CHKERRQ(ierr); > ierr = VecDestroy(&mglevels[i]->x);CHKERRQ(ierr); > ierr = MatDestroy(&mglevels[i+1]->restrct);CHKERRQ(ierr); > ierr = MatDestroy(&mglevels[i+1]->interpolate);CHKERRQ(ierr); > ierr = VecDestroy(&mglevels[i+1]->rscale);CHKERRQ(ierr); > } > > for (i=0; i ierr = MatDestroy(&mglevels[i]->A);CHKERRQ(ierr); > if (mglevels[i]->smoothd != mglevels[i]->smoothu) { > ierr = KSPReset(mglevels[i]->smoothd);CHKERRQ(ierr); > } > ierr = KSPReset(mglevels[i]->smoothu);CHKERRQ(ierr); > } > } > PetscFunctionReturn(0); > } > > And change it to only destroy mats on each level ierr = MatDestroy(&mglevels[i]->A);CHKERRQ(ierr); Keep everything else. > > Not that you will need to include #include in your code. > > > On Jan 24, 2018, at 9:53 AM, Ben Yee wrote: > > > > Yes, the nonzero structure changes in a way that cannot easily be predicted. I could overallocate, add some extra entries to the first matrix, and reuse that matrix, but I would have to roughly double the number of allocated entries, leading to an undesirable increase in memory. I think that one alternative would be to just reset the KSP object when I change the operator, but, if I'm not mistaken, this would destroy all the interpolation operators and I'd have to redefine all of them. > > > > On Wed, Jan 24, 2018 at 10:41 AM, Smith, Barry F. wrote: > > > > Ben > > > > This is a good question, I'll take a look at how difficult it would be to add this feature. > > > > I take it the nonzero structure of the outer matrix changes over time? > > > > > > Barry > > > > > > > On Jan 24, 2018, at 9:12 AM, Ben Yee wrote: > > > > > > Hi, > > > > > > I have multiple matrices (with different nonzero structures) for which I would like to use the same KSP operator (KSPRICHARDSON with PCMG, Galerkin process for generating coarse grid operator). I want to use the same PCMG structure for all the matrices (same interpolation, same restriction, same number of levels, same smoothing techniques, etc.), so I tried using KSPSetOperators to just change the operator. My interpolation matrices are MPIAIJ matrices that I have defined myself and provided to PCMG. My hope was that, with the new operator I provided, PETSc would recompute all of the coarse/intermediate grid operators using the Galerkin process with the old interpolation matrices and the new fine-grid operator. > > > > > > However, I get the wrong solution after changing the operator with KSPSetOperators. After some extensive digging around the PETSc source code, it seems that the operators in mg.c are not updated by KSPSetOperators. When I do a MatView on mglevels->A or the operator in mglevels->smoothu or mglevels->smoothd in PCMGMCycle_Private() of mg.c, it has the matrix values of the old matrix. From what I understand, the only place where mglevels->A (used to compute the residual) is updated is in PCApplyRIchardson_MG() of mg.c, from the following lines of code: > > > 74: for > > > (i=0; i > > > > > 75: if > > > (!mglevels[i]->A) { > > > > > > 76: KSPGetOperators > > > (mglevels[i]->smoothu,&mglevels[i]->A,NULL); > > > > > > 77: PetscObjectReference((PetscObject > > > )mglevels[i]->A); > > > > > > 78: > > > } > > > > > > 79: } > > > It seems to me that the pointer mglevels[i]->A is only set once, when it is undefined at the beginning of the iteration scheme. I think there are more issues as well, but this is the easiest one to point out. (For example, it doesn't seem like it is possible to update the operator for mglevels[i]->smoothd from KSPSetOperators on the outermost KSP object, and, if I try to set it manually via PCMGGetSmoother, I get errors if the nonzero structures of the coarse-grid operators have changed, which tends to be the case when the fine-grid operator's nonzero structure changes). > > > > > > Is there something I'm doing wrong? Is there a way to do this so that I can reuse PCMG with a new operator? > > > > > > Thanks! > > > > > > -- > > > Ben Yee > > > > > > NERS PhD Candidate, University of Michigan > > > B.S. Mech. & Nuclear Eng., U.C. Berkeley > > > > > > > > > > -- > > Ben Yee > > > > NERS PhD Candidate, University of Michigan > > B.S. Mech. & Nuclear Eng., U.C. Berkeley > > > > > -- > Ben Yee > > NERS PhD Candidate, University of Michigan > B.S. Mech. & Nuclear Eng., U.C. Berkeley From luis.saturday at gmail.com Fri Jan 26 13:18:16 2018 From: luis.saturday at gmail.com (saturday luis) Date: Fri, 26 Jan 2018 11:18:16 -0800 Subject: [petsc-users] A question on Schur complement Message-ID: Hi PETSc team: I want to consult for a good way of doing an approximated Schur complement matrix. So I just want to handle a matrix in the format C + B^t B, or sometimes B^t B. An easy way is to run with the MatCreateSchurComplement to create C + B^t inv A B, wherein A is an identity matrix. The downside is that the matrix free style does not allow for algebraic preconditioners. Another way is to generate the matrix B^t B. For my case, I checked the sparsity of the resulting matrix from MatMatMult and it is pretty sparse still. I found a similar approach in http://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex70.c.html where the B^t B matrix is created explicitly (line 367-388). However, there is a line of comment saying " in real life this matrix would be build directly without MatMatMult". So I just want to ask the best way of assemblying and solving the B^t B matrix. Thanks, Luis -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Jan 26 13:25:06 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 26 Jan 2018 19:25:06 +0000 Subject: [petsc-users] [petsc-maint] A question on Schur complement In-Reply-To: References: Message-ID: <31B851B9-1C21-4746-92F3-83B3388C1957@anl.gov> > On Jan 26, 2018, at 1:18 PM, saturday luis wrote: > > Hi PETSc team: > > I want to consult for a good way of doing an approximated Schur complement matrix. So I just want to handle a matrix in the format C + B^t B, or sometimes B^t B. > > An easy way is to run with the MatCreateSchurComplement to create C + B^t inv A B, wherein A is an identity matrix. The downside is that the matrix free style does not allow for algebraic preconditioners. > > Another way is to generate the matrix B^t B. For my case, I checked the sparsity of the resulting matrix from MatMatMult and it is pretty sparse still. I found a similar approach in > http://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex70.c.html > where the B^t B matrix is created explicitly (line 367-388). However, there is a line of comment saying " in real life this matrix would be build directly without MatMatMult". So I just want to ask the best way of assemblying and solving the B^t B matrix. Using the product is best. You can ignore the comment. The comment is attempting to indicate that one can build the matrix from which the preconditioner is to be constructed by explicitly forming a matrix based on the specific problem at hand. Barry > > Thanks, > > Luis From luis.saturday at gmail.com Fri Jan 26 13:36:39 2018 From: luis.saturday at gmail.com (saturday luis) Date: Fri, 26 Jan 2018 11:36:39 -0800 Subject: [petsc-users] Fwd: [petsc-maint] A question on Schur complement In-Reply-To: References: <31B851B9-1C21-4746-92F3-83B3388C1957@anl.gov> Message-ID: Thanks, Barry! Do you have any suggestions on solving the B^tB matrix? I noticed the solver option LSQR but haven't tried it yet. Luis 2018-01-26 11:25 GMT-08:00 Smith, Barry F. : > > > > On Jan 26, 2018, at 1:18 PM, saturday luis > wrote: > > > > Hi PETSc team: > > > > I want to consult for a good way of doing an approximated Schur > complement matrix. So I just want to handle a matrix in the format C + B^t > B, or sometimes B^t B. > > > > An easy way is to run with the MatCreateSchurComplement to create C + > B^t inv A B, wherein A is an identity matrix. The downside is that the > matrix free style does not allow for algebraic preconditioners. > > > > Another way is to generate the matrix B^t B. For my case, I checked the > sparsity of the resulting matrix from MatMatMult and it is pretty sparse > still. I found a similar approach in > > http://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples > /tutorials/ex70.c.html > > where the B^t B matrix is created explicitly (line 367-388). However, > there is a line of comment saying " in real life this matrix would be > build directly without MatMatMult". So I just want to ask the best way of > assemblying and solving the B^t B matrix. > > Using the product is best. You can ignore the comment. > > The comment is attempting to indicate that one can build the matrix > from which the preconditioner is to be constructed by explicitly forming a > matrix based on the specific problem at hand. > > Barry > > > > > > Thanks, > > > > Luis > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Jan 26 14:09:14 2018 From: jed at jedbrown.org (Jed Brown) Date: Fri, 26 Jan 2018 13:09:14 -0700 Subject: [petsc-users] A question on Schur complement In-Reply-To: References: Message-ID: <87po5w5qmt.fsf@jedbrown.org> saturday luis writes: > Hi PETSc team: > > I want to consult for a good way of doing an approximated Schur complement > matrix. So I just want to handle a matrix in the format C + B^t B, or > sometimes B^t B. > > An easy way is to run with the MatCreateSchurComplement to create C + B^t > inv A B, wherein A is an identity matrix. The downside is that the matrix > free style does not allow for algebraic preconditioners. MatCreateLRC is more natural in this context. Even if you solve with LSQR, you'll likely need some explicit representation for preconditioning. From fande.kong at inl.gov Fri Jan 26 14:15:06 2018 From: fande.kong at inl.gov (Kong, Fande) Date: Fri, 26 Jan 2018 13:15:06 -0700 Subject: [petsc-users] Context behind SNESComputeFunction call In-Reply-To: References: Message-ID: On Mon, Jan 8, 2018 at 2:15 PM, Smith, Barry F. wrote: > > > > On Jan 8, 2018, at 2:59 PM, Alexander Lindsay > wrote: > > > > Is there any elegant way to tell whether SNESComputeFunction is being > called under different conceptual contexts? > > > > E.g. non-linear residual evaluation vs. Jacobian formation from finite > differencing vs. Jacobian-vector products from finite differencing? > > Under normal usage with the options database no. > Hi Barry, How difficult to provide an API? Is it possible? > > If you have some reason to know you could write three functions and > provide them to SNESSetFunction(), MatMFFDSetFunction(), and > MatFDColoringSetFunction(). Note that these functions have slightly > different calling sequences but you can have all of them call the same > underlying common function if you are only interested in, for example, how > many times the function is used for each purpose. > If we use this way for the Jacobian-free Newton, the function evaluation will be called twice at the first linear iteration because the computed residual vector at the nonlinear step is not reused. Any way to reuse the function residual of the Newton step instead of recomputing a new residual at the first linear iteration? Fande, > > Barry > > > > > > > Alex > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Jan 26 16:10:54 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 26 Jan 2018 22:10:54 +0000 Subject: [petsc-users] Context behind SNESComputeFunction call In-Reply-To: References: Message-ID: <3D320626-58DC-49EC-A80B-614AC3BEA709@anl.gov> > On Jan 26, 2018, at 2:15 PM, Kong, Fande wrote: > > > > On Mon, Jan 8, 2018 at 2:15 PM, Smith, Barry F. wrote: > > > > On Jan 8, 2018, at 2:59 PM, Alexander Lindsay wrote: > > > > Is there any elegant way to tell whether SNESComputeFunction is being called under different conceptual contexts? > > > > E.g. non-linear residual evaluation vs. Jacobian formation from finite differencing vs. Jacobian-vector products from finite differencing? > > Under normal usage with the options database no. > > Hi Barry, > > How difficult to provide an API? Is it possible? > > > > If you have some reason to know you could write three functions and provide them to SNESSetFunction(), MatMFFDSetFunction(), and MatFDColoringSetFunction(). Note that these functions have slightly different calling sequences but you can have all of them call the same underlying common function if you are only interested in, for example, how many times the function is used for each purpose. > > If we use this way for the Jacobian-free Newton, the function evaluation will be called twice at the first linear iteration because the computed residual vector at the nonlinear step is not reused. Any way to reuse the function residual of the Newton step instead of recomputing a new residual at the first linear iteration? It does reuse the function evaluation. Why do you think it does not? If you look at MatMult_MFFD() you will see the lines of code /* compute func(U) as base for differencing; only needed first time in and not when provided by user */ if (ctx->ncurrenth == 1 && ctx->current_f_allocated) { ierr = (*ctx->func)(ctx->funcctx,U,F);CHKERRQ(ierr); } since the if is satisfied it does not compute the function at the base location. To double check I ran src/snes/examples/tutorials/ex19 with -snes_mf in the debugger and verified that the "extra" function evaluations are not done. Barry > > Fande, > > > > Barry > > > > > > > Alex > > From luis.saturday at gmail.com Fri Jan 26 16:28:14 2018 From: luis.saturday at gmail.com (saturday luis) Date: Fri, 26 Jan 2018 14:28:14 -0800 Subject: [petsc-users] A question on Schur complement In-Reply-To: <87po5w5qmt.fsf@jedbrown.org> References: <87po5w5qmt.fsf@jedbrown.org> Message-ID: Thanks, Jed. It seems that MatCreateLRC requires two dense matrices for U and V according to the page http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateLRC.html Not sure if it is applicable to sparse matrices. 2018-01-26 12:09 GMT-08:00 Jed Brown : > saturday luis writes: > > > Hi PETSc team: > > > > I want to consult for a good way of doing an approximated Schur > complement > > matrix. So I just want to handle a matrix in the format C + B^t B, or > > sometimes B^t B. > > > > An easy way is to run with the MatCreateSchurComplement to create C + B^t > > inv A B, wherein A is an identity matrix. The downside is that the matrix > > free style does not allow for algebraic preconditioners. > > MatCreateLRC is more natural in this context. > > Even if you solve with LSQR, you'll likely need some explicit > representation for preconditioning. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fande.kong at inl.gov Fri Jan 26 16:32:18 2018 From: fande.kong at inl.gov (Kong, Fande) Date: Fri, 26 Jan 2018 15:32:18 -0700 Subject: [petsc-users] Context behind SNESComputeFunction call In-Reply-To: <3D320626-58DC-49EC-A80B-614AC3BEA709@anl.gov> References: <3D320626-58DC-49EC-A80B-614AC3BEA709@anl.gov> Message-ID: On Fri, Jan 26, 2018 at 3:10 PM, Smith, Barry F. wrote: > > > > On Jan 26, 2018, at 2:15 PM, Kong, Fande wrote: > > > > > > > > On Mon, Jan 8, 2018 at 2:15 PM, Smith, Barry F. > wrote: > > > > > > > On Jan 8, 2018, at 2:59 PM, Alexander Lindsay < > alexlindsay239 at gmail.com> wrote: > > > > > > Is there any elegant way to tell whether SNESComputeFunction is being > called under different conceptual contexts? > > > > > > E.g. non-linear residual evaluation vs. Jacobian formation from finite > differencing vs. Jacobian-vector products from finite differencing? > > > > Under normal usage with the options database no. > > > > Hi Barry, > > > > How difficult to provide an API? Is it possible? > > > > > > > > If you have some reason to know you could write three functions and > provide them to SNESSetFunction(), MatMFFDSetFunction(), and > MatFDColoringSetFunction(). Note that these functions have slightly > different calling sequences but you can have all of them call the same > underlying common function if you are only interested in, for example, how > many times the function is used for each purpose. > > > > If we use this way for the Jacobian-free Newton, the function evaluation > will be called twice at the first linear iteration because the computed > residual vector at the nonlinear step is not reused. Any way to reuse the > function residual of the Newton step instead of recomputing a new residual > at the first linear iteration? > > It does reuse the function evaluation. Why do you think it does not? If > you look at MatMult_MFFD() you will see the lines of code > > /* compute func(U) as base for differencing; only needed first time in > and not when provided by user */ > if (ctx->ncurrenth == 1 && ctx->current_f_allocated) { > ierr = (*ctx->func)(ctx->funcctx,U,F);CHKERRQ(ierr); > } > > since the if is satisfied it does not compute the function at the base > location. To double check I ran src/snes/examples/tutorials/ex19 with > -snes_mf in the debugger and verified that the "extra" function evaluations > are not done. > In MatAssemblyEnd_SNESMF, if (j->func == (PetscErrorCode (*)(void*,Vec,Vec))SNESComputeFunction) { ierr = SNESGetFunction(snes,&f,NULL,NULL);CHKERRQ(ierr); ierr = MatMFFDSetBase_MFFD(J,u,f);CHKERRQ(ierr); } else { /* f value known by SNES is not correct for other differencing function */ ierr = MatMFFDSetBase_MFFD(J,u,NULL);CHKERRQ(ierr); } Will hit ierr = MatMFFDSetBase_MFFD(J,u,NULL);CHKERRQ(ierr), because SNES and MAT have different function pointers. In MatMFFDSetBase_MFFD(Mat J,Vec U,Vec F), if (F) { if (ctx->current_f_allocated) {ierr = VecDestroy(&ctx->current_f);CHKERRQ(ierr);} ctx->current_f = F; ctx->current_f_allocated = PETSC_FALSE; } else if (!ctx->current_f_allocated) { ierr = MatCreateVecs(J,NULL,&ctx->current_f);CHKERRQ(ierr); ctx->current_f_allocated = PETSC_TRUE; } Because F=NULL, we then have ctx->current_f_allocated = PETSC_TRUE. Then, the following if statement is true: if (ctx->ncurrenth == 1 && ctx->current_f_allocated) { ierr = (*ctx->func)(ctx->funcctx,U,F);CHKERRQ(ierr); } Fande, > > Barry > > > > > > Fande, > > > > > > > > Barry > > > > > > > > > > > > Alex > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Jan 26 16:35:05 2018 From: jed at jedbrown.org (Jed Brown) Date: Fri, 26 Jan 2018 15:35:05 -0700 Subject: [petsc-users] A question on Schur complement In-Reply-To: References: <87po5w5qmt.fsf@jedbrown.org> Message-ID: <87607o5jvq.fsf@jedbrown.org> saturday luis writes: > Thanks, Jed. > > It seems that MatCreateLRC requires two dense matrices for U and V > according to the page > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateLRC.html > > Not sure if it is applicable to sparse matrices. Oops, I thought the implementation did not rely on that property, but in fact it does. From bsmith at mcs.anl.gov Fri Jan 26 16:50:06 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Fri, 26 Jan 2018 22:50:06 +0000 Subject: [petsc-users] Context behind SNESComputeFunction call In-Reply-To: References: <3D320626-58DC-49EC-A80B-614AC3BEA709@anl.gov> Message-ID: So you are doing something non-standard? Are you not just using -snes_mf or -snes_mf_operator? Can you send me a sample code that has the extra function evaluations? Because if you run through regular usage with the debugger you will see there is no extra evaluation. Barry > On Jan 26, 2018, at 4:32 PM, Kong, Fande wrote: > > > > On Fri, Jan 26, 2018 at 3:10 PM, Smith, Barry F. wrote: > > > > On Jan 26, 2018, at 2:15 PM, Kong, Fande wrote: > > > > > > > > On Mon, Jan 8, 2018 at 2:15 PM, Smith, Barry F. wrote: > > > > > > > On Jan 8, 2018, at 2:59 PM, Alexander Lindsay wrote: > > > > > > Is there any elegant way to tell whether SNESComputeFunction is being called under different conceptual contexts? > > > > > > E.g. non-linear residual evaluation vs. Jacobian formation from finite differencing vs. Jacobian-vector products from finite differencing? > > > > Under normal usage with the options database no. > > > > Hi Barry, > > > > How difficult to provide an API? Is it possible? > > > > > > > > If you have some reason to know you could write three functions and provide them to SNESSetFunction(), MatMFFDSetFunction(), and MatFDColoringSetFunction(). Note that these functions have slightly different calling sequences but you can have all of them call the same underlying common function if you are only interested in, for example, how many times the function is used for each purpose. > > > > If we use this way for the Jacobian-free Newton, the function evaluation will be called twice at the first linear iteration because the computed residual vector at the nonlinear step is not reused. Any way to reuse the function residual of the Newton step instead of recomputing a new residual at the first linear iteration? > > It does reuse the function evaluation. Why do you think it does not? If you look at MatMult_MFFD() you will see the lines of code > > /* compute func(U) as base for differencing; only needed first time in and not when provided by user */ > if (ctx->ncurrenth == 1 && ctx->current_f_allocated) { > ierr = (*ctx->func)(ctx->funcctx,U,F);CHKERRQ(ierr); > } > > since the if is satisfied it does not compute the function at the base location. To double check I ran src/snes/examples/tutorials/ex19 with -snes_mf in the debugger and verified that the "extra" function evaluations are not done. > > In MatAssemblyEnd_SNESMF, > > if (j->func == (PetscErrorCode (*)(void*,Vec,Vec))SNESComputeFunction) { > ierr = SNESGetFunction(snes,&f,NULL,NULL);CHKERRQ(ierr); > ierr = MatMFFDSetBase_MFFD(J,u,f);CHKERRQ(ierr); > } else { > /* f value known by SNES is not correct for other differencing function */ > ierr = MatMFFDSetBase_MFFD(J,u,NULL);CHKERRQ(ierr); > } > > > Will hit ierr = MatMFFDSetBase_MFFD(J,u,NULL);CHKERRQ(ierr), because SNES and MAT have different function pointers. > > In MatMFFDSetBase_MFFD(Mat J,Vec U,Vec F), > > if (F) { > if (ctx->current_f_allocated) {ierr = VecDestroy(&ctx->current_f);CHKERRQ(ierr);} > ctx->current_f = F; > ctx->current_f_allocated = PETSC_FALSE; > } else if (!ctx->current_f_allocated) { > ierr = MatCreateVecs(J,NULL,&ctx->current_f);CHKERRQ(ierr); > > ctx->current_f_allocated = PETSC_TRUE; > } > > Because F=NULL, we then have ctx->current_f_allocated = PETSC_TRUE. > > Then, the following if statement is true: > > if (ctx->ncurrenth == 1 && ctx->current_f_allocated) { > ierr = (*ctx->func)(ctx->funcctx,U,F);CHKERRQ(ierr); > } > > > Fande, > > > > Barry > > > > > > Fande, > > > > > > > > Barry > > > > > > > > > > > > Alex > > > > From fande.kong at inl.gov Fri Jan 26 17:34:40 2018 From: fande.kong at inl.gov (Kong, Fande) Date: Fri, 26 Jan 2018 16:34:40 -0700 Subject: [petsc-users] Context behind SNESComputeFunction call In-Reply-To: References: <3D320626-58DC-49EC-A80B-614AC3BEA709@anl.gov> Message-ID: Hi Barry, I made minor changes on src/snes/examples/tutorials/ex2.c to demonstrate this issue. Please see the attachment. ./ex2 -snes_monitor -ksp_monitor -snes_mf_operator 1 *atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 FormFunction is called 0 SNES Function norm 5.414682427127e+00 0 KSP Residual norm 9.559082033938e-01 FormFunction is called FormFunction is called 1 KSP Residual norm 1.703870633386e-09 FormFunction is called FormFunction is called 1 SNES Function norm 2.952582481151e-01 0 KSP Residual norm 2.672054855433e-02 FormFunction is called FormFunction is called 1 KSP Residual norm 1.519298012177e-10 FormFunction is called FormFunction is called 2 SNES Function norm 4.502289047587e-04 0 KSP Residual norm 4.722075651268e-05 FormFunction is called FormFunction is called 1 KSP Residual norm 3.834927363659e-14 FormFunction is called FormFunction is called 3 SNES Function norm 1.390073376131e-09 number of SNES iterations = 3Norm of error 1.49795e-10, Iterations 3* "FormFunction" is called TWICE at "0 KSP". If we comment out MatMFFDSetFunction: */* ierr = MatMFFDSetFunction(Jacobian,FormFunction_MFFD,(void*)snes);CHKERRQ(ierr); */* ./ex2 -snes_monitor -ksp_monitor -snes_mf_operator 1 *atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 FormFunction is called 0 SNES Function norm 5.414682427127e+00 0 KSP Residual norm 9.559082033938e-01 FormFunction is called 1 KSP Residual norm 1.703870633386e-09 FormFunction is called FormFunction is called 1 SNES Function norm 2.952582481151e-01 0 KSP Residual norm 2.672054855433e-02 FormFunction is called 1 KSP Residual norm 1.519298012177e-10 FormFunction is called FormFunction is called 2 SNES Function norm 4.502289047587e-04 0 KSP Residual norm 4.722075651268e-05 FormFunction is called 1 KSP Residual norm 3.834927363659e-14 FormFunction is called FormFunction is called 3 SNES Function norm 1.390073376131e-09 number of SNES iterations = 3Norm of error 1.49795e-10, Iterations 3* "FormFunction" is called ONCE at "0 KSP". Hopefully, this example makes the point clear. Fande, On Fri, Jan 26, 2018 at 3:50 PM, Smith, Barry F. wrote: > > > So you are doing something non-standard? Are you not just using -snes_mf > or -snes_mf_operator? Can you send me a sample code that has the extra > function evaluations? Because if you run through regular usage with the > debugger you will see there is no extra evaluation. > > Barry > > > > On Jan 26, 2018, at 4:32 PM, Kong, Fande wrote: > > > > > > > > On Fri, Jan 26, 2018 at 3:10 PM, Smith, Barry F. > wrote: > > > > > > > On Jan 26, 2018, at 2:15 PM, Kong, Fande wrote: > > > > > > > > > > > > On Mon, Jan 8, 2018 at 2:15 PM, Smith, Barry F. > wrote: > > > > > > > > > > On Jan 8, 2018, at 2:59 PM, Alexander Lindsay < > alexlindsay239 at gmail.com> wrote: > > > > > > > > Is there any elegant way to tell whether SNESComputeFunction is > being called under different conceptual contexts? > > > > > > > > E.g. non-linear residual evaluation vs. Jacobian formation from > finite differencing vs. Jacobian-vector products from finite differencing? > > > > > > Under normal usage with the options database no. > > > > > > Hi Barry, > > > > > > How difficult to provide an API? Is it possible? > > > > > > > > > > > > If you have some reason to know you could write three functions and > provide them to SNESSetFunction(), MatMFFDSetFunction(), and > MatFDColoringSetFunction(). Note that these functions have slightly > different calling sequences but you can have all of them call the same > underlying common function if you are only interested in, for example, how > many times the function is used for each purpose. > > > > > > If we use this way for the Jacobian-free Newton, the function > evaluation will be called twice at the first linear iteration because the > computed residual vector at the nonlinear step is not reused. Any way to > reuse the function residual of the Newton step instead of recomputing a new > residual at the first linear iteration? > > > > It does reuse the function evaluation. Why do you think it does not? > If you look at MatMult_MFFD() you will see the lines of code > > > > /* compute func(U) as base for differencing; only needed first time in > and not when provided by user */ > > if (ctx->ncurrenth == 1 && ctx->current_f_allocated) { > > ierr = (*ctx->func)(ctx->funcctx,U,F);CHKERRQ(ierr); > > } > > > > since the if is satisfied it does not compute the function at the base > location. To double check I ran src/snes/examples/tutorials/ex19 with > -snes_mf in the debugger and verified that the "extra" function evaluations > are not done. > > > > In MatAssemblyEnd_SNESMF, > > > > if (j->func == (PetscErrorCode (*)(void*,Vec,Vec))SNESComputeFunction) > { > > ierr = SNESGetFunction(snes,&f,NULL,NULL);CHKERRQ(ierr); > > ierr = MatMFFDSetBase_MFFD(J,u,f);CHKERRQ(ierr); > > } else { > > /* f value known by SNES is not correct for other differencing > function */ > > ierr = MatMFFDSetBase_MFFD(J,u,NULL);CHKERRQ(ierr); > > } > > > > > > Will hit ierr = MatMFFDSetBase_MFFD(J,u,NULL);CHKERRQ(ierr), because > SNES and MAT have different function pointers. > > > > In MatMFFDSetBase_MFFD(Mat J,Vec U,Vec F), > > > > if (F) { > > if (ctx->current_f_allocated) {ierr = VecDestroy(&ctx->current_f); > CHKERRQ(ierr);} > > ctx->current_f = F; > > ctx->current_f_allocated = PETSC_FALSE; > > } else if (!ctx->current_f_allocated) { > > ierr = MatCreateVecs(J,NULL,&ctx->current_f);CHKERRQ(ierr); > > > > ctx->current_f_allocated = PETSC_TRUE; > > } > > > > Because F=NULL, we then have ctx->current_f_allocated = PETSC_TRUE. > > > > Then, the following if statement is true: > > > > if (ctx->ncurrenth == 1 && ctx->current_f_allocated) { > > ierr = (*ctx->func)(ctx->funcctx,U,F);CHKERRQ(ierr); > > } > > > > > > Fande, > > > > > > > > Barry > > > > > > > > > > Fande, > > > > > > > > > > > > Barry > > > > > > > > > > > > > > > > > Alex > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex2.c Type: text/x-csrc Size: 12999 bytes Desc: not available URL: From fande.kong at inl.gov Fri Jan 26 17:53:46 2018 From: fande.kong at inl.gov (Kong, Fande) Date: Fri, 26 Jan 2018 16:53:46 -0700 Subject: [petsc-users] How to generate manul.pdf from source files Message-ID: Hi, I want to generate manul.pdf from source files on my own desktop. In directory: ./src/docs/tex/manual make ALL LOC=/Users/kongf/projects/petsc * make: *** No rule to make target `/Users/kongf/projects/petsc/docs/manualpages/htmlmap', needed by `listing_kspex1tmp.tex'. Stop.* What else I need to install? Fande -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexlindsay239 at gmail.com Fri Jan 26 17:54:08 2018 From: alexlindsay239 at gmail.com (Alexander Lindsay) Date: Fri, 26 Jan 2018 17:54:08 -0600 Subject: [petsc-users] Context behind SNESComputeFunction call In-Reply-To: References: <3D320626-58DC-49EC-A80B-614AC3BEA709@anl.gov> Message-ID: We are doing something non-standard. We set different functions for snes and mffd. When a function is called through snes, we know we are doing a non-linear residual evaluation and we allow an update of our mechanical contact nodes. When a function is called through mffd, we know we are within the linear solve, and we do not allow our contact state to change. If after a contact update our non-linear residual is not deemed to be converged, then we would hope to be able to re-use it for the zeroth-linear residual. I know that you don't like this way of handling contact, but it integrates well for us into our multiphysics framework and works well in many cases. > On Jan 26, 2018, at 5:34 PM, Kong, Fande wrote: > > Hi Barry, > > I made minor changes on src/snes/examples/tutorials/ex2.c to demonstrate this issue. Please see the attachment. > > ./ex2 -snes_monitor -ksp_monitor -snes_mf_operator 1 > > > atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 > FormFunction is called > 0 SNES Function norm 5.414682427127e+00 > 0 KSP Residual norm 9.559082033938e-01 > FormFunction is called > FormFunction is called > 1 KSP Residual norm 1.703870633386e-09 > FormFunction is called > FormFunction is called > 1 SNES Function norm 2.952582481151e-01 > 0 KSP Residual norm 2.672054855433e-02 > FormFunction is called > FormFunction is called > 1 KSP Residual norm 1.519298012177e-10 > FormFunction is called > FormFunction is called > 2 SNES Function norm 4.502289047587e-04 > 0 KSP Residual norm 4.722075651268e-05 > FormFunction is called > FormFunction is called > 1 KSP Residual norm 3.834927363659e-14 > FormFunction is called > FormFunction is called > 3 SNES Function norm 1.390073376131e-09 > number of SNES iterations = 3 > > Norm of error 1.49795e-10, Iterations 3 > > "FormFunction" is called TWICE at "0 KSP". > > If we comment out MatMFFDSetFunction: > > /* ierr = MatMFFDSetFunction(Jacobian,FormFunction_MFFD,(void*)snes);CHKERRQ(ierr); */ > > > ./ex2 -snes_monitor -ksp_monitor -snes_mf_operator 1 > > atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 > FormFunction is called > 0 SNES Function norm 5.414682427127e+00 > 0 KSP Residual norm 9.559082033938e-01 > FormFunction is called > 1 KSP Residual norm 1.703870633386e-09 > FormFunction is called > FormFunction is called > 1 SNES Function norm 2.952582481151e-01 > 0 KSP Residual norm 2.672054855433e-02 > FormFunction is called > 1 KSP Residual norm 1.519298012177e-10 > FormFunction is called > FormFunction is called > 2 SNES Function norm 4.502289047587e-04 > 0 KSP Residual norm 4.722075651268e-05 > FormFunction is called > 1 KSP Residual norm 3.834927363659e-14 > FormFunction is called > FormFunction is called > 3 SNES Function norm 1.390073376131e-09 > number of SNES iterations = 3 > > Norm of error 1.49795e-10, Iterations 3 > > "FormFunction" is called ONCE at "0 KSP". > > Hopefully, this example makes the point clear. > > > Fande, > >> On Fri, Jan 26, 2018 at 3:50 PM, Smith, Barry F. wrote: >> >> >> So you are doing something non-standard? Are you not just using -snes_mf or -snes_mf_operator? Can you send me a sample code that has the extra function evaluations? Because if you run through regular usage with the debugger you will see there is no extra evaluation. >> >> Barry >> >> >> > On Jan 26, 2018, at 4:32 PM, Kong, Fande wrote: >> > >> > >> > >> > On Fri, Jan 26, 2018 at 3:10 PM, Smith, Barry F. wrote: >> > >> > >> > > On Jan 26, 2018, at 2:15 PM, Kong, Fande wrote: >> > > >> > > >> > > >> > > On Mon, Jan 8, 2018 at 2:15 PM, Smith, Barry F. wrote: >> > > >> > > >> > > > On Jan 8, 2018, at 2:59 PM, Alexander Lindsay wrote: >> > > > >> > > > Is there any elegant way to tell whether SNESComputeFunction is being called under different conceptual contexts? >> > > > >> > > > E.g. non-linear residual evaluation vs. Jacobian formation from finite differencing vs. Jacobian-vector products from finite differencing? >> > > >> > > Under normal usage with the options database no. >> > > >> > > Hi Barry, >> > > >> > > How difficult to provide an API? Is it possible? >> > > >> > > >> > > >> > > If you have some reason to know you could write three functions and provide them to SNESSetFunction(), MatMFFDSetFunction(), and MatFDColoringSetFunction(). Note that these functions have slightly different calling sequences but you can have all of them call the same underlying common function if you are only interested in, for example, how many times the function is used for each purpose. >> > > >> > > If we use this way for the Jacobian-free Newton, the function evaluation will be called twice at the first linear iteration because the computed residual vector at the nonlinear step is not reused. Any way to reuse the function residual of the Newton step instead of recomputing a new residual at the first linear iteration? >> > >> > It does reuse the function evaluation. Why do you think it does not? If you look at MatMult_MFFD() you will see the lines of code >> > >> > /* compute func(U) as base for differencing; only needed first time in and not when provided by user */ >> > if (ctx->ncurrenth == 1 && ctx->current_f_allocated) { >> > ierr = (*ctx->func)(ctx->funcctx,U,F);CHKERRQ(ierr); >> > } >> > >> > since the if is satisfied it does not compute the function at the base location. To double check I ran src/snes/examples/tutorials/ex19 with -snes_mf in the debugger and verified that the "extra" function evaluations are not done. >> > >> > In MatAssemblyEnd_SNESMF, >> > >> > if (j->func == (PetscErrorCode (*)(void*,Vec,Vec))SNESComputeFunction) { >> > ierr = SNESGetFunction(snes,&f,NULL,NULL);CHKERRQ(ierr); >> > ierr = MatMFFDSetBase_MFFD(J,u,f);CHKERRQ(ierr); >> > } else { >> > /* f value known by SNES is not correct for other differencing function */ >> > ierr = MatMFFDSetBase_MFFD(J,u,NULL);CHKERRQ(ierr); >> > } >> > >> > >> > Will hit ierr = MatMFFDSetBase_MFFD(J,u,NULL);CHKERRQ(ierr), because SNES and MAT have different function pointers. >> > >> > In MatMFFDSetBase_MFFD(Mat J,Vec U,Vec F), >> > >> > if (F) { >> > if (ctx->current_f_allocated) {ierr = VecDestroy(&ctx->current_f);CHKERRQ(ierr);} >> > ctx->current_f = F; >> > ctx->current_f_allocated = PETSC_FALSE; >> > } else if (!ctx->current_f_allocated) { >> > ierr = MatCreateVecs(J,NULL,&ctx->current_f);CHKERRQ(ierr); >> > >> > ctx->current_f_allocated = PETSC_TRUE; >> > } >> > >> > Because F=NULL, we then have ctx->current_f_allocated = PETSC_TRUE. >> > >> > Then, the following if statement is true: >> > >> > if (ctx->ncurrenth == 1 && ctx->current_f_allocated) { >> > ierr = (*ctx->func)(ctx->funcctx,U,F);CHKERRQ(ierr); >> > } >> > >> > >> > Fande, >> > >> > >> > >> > Barry >> > >> > >> > > >> > > Fande, >> > > >> > > >> > > >> > > Barry >> > > >> > > >> > > >> > > > >> > > > Alex >> > > >> > > >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jan 26 17:56:16 2018 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 27 Jan 2018 10:56:16 +1100 Subject: [petsc-users] How to generate manul.pdf from source files In-Reply-To: References: Message-ID: On Sat, Jan 27, 2018 at 10:53 AM, Kong, Fande wrote: > Hi, > > I want to generate manul.pdf from source files on my own desktop. > > In directory: ./src/docs/tex/manual > > make ALL LOC=/Users/kongf/projects/petsc > > * make: *** No rule to make target > `/Users/kongf/projects/petsc/docs/manualpages/htmlmap', needed by > `listing_kspex1tmp.tex'. Stop.* > > What else I need to install? > Do you have sowing? Matt > > Fande > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From fande.kong at inl.gov Fri Jan 26 18:00:11 2018 From: fande.kong at inl.gov (Kong, Fande) Date: Fri, 26 Jan 2018 17:00:11 -0700 Subject: [petsc-users] How to generate manul.pdf from source files In-Reply-To: References: Message-ID: I guess we install sowing automatically when we are on petsc/master > cat configure.log | grep "sowing" * Bfort not found. Installing sowing for FortranStubsTEST checkDependencies from config.packages.sowing(/Users/kongf/projects/petsc/config/BuildSystem/config/package.py:719)TESTING: checkDependencies from config.packages.sowing(config/BuildSystem/config/package.py:719)TEST configureLibrary from config.packages.sowing(/Users/kongf/projects/petsc/config/BuildSystem/config/package.py:744)TESTING: configureLibrary from config.packages.sowing(config/BuildSystem/config/package.py:744) Checking for a functional sowing Looking for SOWING at git.sowing, hg.sowing or a directory starting with ['petsc-pkg-sowing'] Found a copy of SOWING in git.sowingRemoving sowing.petscconfTEST checkSharedLibrary from config.packages.sowing(/Users/kongf/projects/petsc/config/BuildSystem/config/package.py:792)TESTING: checkSharedLibrary from config.packages.sowing(config/BuildSystem/config/package.py:792)sowing:* Thanks *,* Fande On Fri, Jan 26, 2018 at 4:56 PM, Matthew Knepley wrote: > On Sat, Jan 27, 2018 at 10:53 AM, Kong, Fande wrote: > >> Hi, >> >> I want to generate manul.pdf from source files on my own desktop. >> >> In directory: ./src/docs/tex/manual >> >> make ALL LOC=/Users/kongf/projects/petsc >> >> * make: *** No rule to make target >> `/Users/kongf/projects/petsc/docs/manualpages/htmlmap', needed by >> `listing_kspex1tmp.tex'. Stop.* >> >> What else I need to install? >> > > Do you have sowing? > > Matt > > >> >> Fande >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Jan 26 18:53:30 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Sat, 27 Jan 2018 00:53:30 +0000 Subject: [petsc-users] How to generate manul.pdf from source files In-Reply-To: References: Message-ID: <955A90AF-28D6-44A6-82ED-855ABE44520A@anl.gov> You need to run make alldocs in the root PETSc directory first. This builds many things used by the users manual (also builds the users manual). After you have run the make alldocs once you can rebuild just the users manual multiple times directly in the users manual directory. > On Jan 26, 2018, at 5:53 PM, Kong, Fande wrote: > > Hi, > > I want to generate manul.pdf from source files on my own desktop. > > In directory: ./src/docs/tex/manual > > make ALL LOC=/Users/kongf/projects/petsc > > make: *** No rule to make target `/Users/kongf/projects/petsc/docs/manualpages/htmlmap', needed by `listing_kspex1tmp.tex'. Stop. > > What else I need to install? > > Fande > > From balay at mcs.anl.gov Fri Jan 26 19:24:45 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 26 Jan 2018 19:24:45 -0600 Subject: [petsc-users] How to generate manul.pdf from source files In-Reply-To: <955A90AF-28D6-44A6-82ED-855ABE44520A@anl.gov> References: <955A90AF-28D6-44A6-82ED-855ABE44520A@anl.gov> Message-ID: Its: make alldoc LOC=$PETSC_DIR Satish On Sat, 27 Jan 2018, Smith, Barry F. wrote: > > You need to run > > make alldocs > > in the root PETSc directory first. This builds many things used by the users manual (also builds the users manual). After you have run the make alldocs once you can rebuild just the users manual multiple times directly in the users manual directory. > > > On Jan 26, 2018, at 5:53 PM, Kong, Fande wrote: > > > > Hi, > > > > I want to generate manul.pdf from source files on my own desktop. > > > > In directory: ./src/docs/tex/manual > > > > make ALL LOC=/Users/kongf/projects/petsc > > > > make: *** No rule to make target `/Users/kongf/projects/petsc/docs/manualpages/htmlmap', needed by `listing_kspex1tmp.tex'. Stop. > > > > What else I need to install? > > > > Fande > > > > > > From bsmith at mcs.anl.gov Fri Jan 26 23:47:48 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Sat, 27 Jan 2018 05:47:48 +0000 Subject: [petsc-users] Context behind SNESComputeFunction call In-Reply-To: References: <3D320626-58DC-49EC-A80B-614AC3BEA709@anl.gov> Message-ID: Thanks, I now understand the situation. I have a tentative solution for you that does not require complex new APIs. I have added one function to the master branch MatSNESMFGetSNES() and attach a modified example that utilizes this to avoid the extra function evaluations. Please let me know if it works for you and what you think, Barry > On Jan 26, 2018, at 5:54 PM, Alexander Lindsay wrote: > > We are doing something non-standard. We set different functions for snes and mffd. When a function is called through snes, we know we are doing a non-linear residual evaluation and we allow an update of our mechanical contact nodes. When a function is called through mffd, we know we are within the linear solve, and we do not allow our contact state to change. If after a contact update our non-linear residual is not deemed to be converged, then we would hope to be able to re-use it for the zeroth-linear residual. > > I know that you don't like this way of handling contact, but it integrates well for us into our multiphysics framework and works well in many cases. > > On Jan 26, 2018, at 5:34 PM, Kong, Fande wrote: > >> Hi Barry, >> >> I made minor changes on src/snes/examples/tutorials/ex2.c to demonstrate this issue. Please see the attachment. >> >> ./ex2 -snes_monitor -ksp_monitor -snes_mf_operator 1 >> >> >> atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 >> FormFunction is called >> 0 SNES Function norm 5.414682427127e+00 >> 0 KSP Residual norm 9.559082033938e-01 >> FormFunction is called >> FormFunction is called >> 1 KSP Residual norm 1.703870633386e-09 >> FormFunction is called >> FormFunction is called >> 1 SNES Function norm 2.952582481151e-01 >> 0 KSP Residual norm 2.672054855433e-02 >> FormFunction is called >> FormFunction is called >> 1 KSP Residual norm 1.519298012177e-10 >> FormFunction is called >> FormFunction is called >> 2 SNES Function norm 4.502289047587e-04 >> 0 KSP Residual norm 4.722075651268e-05 >> FormFunction is called >> FormFunction is called >> 1 KSP Residual norm 3.834927363659e-14 >> FormFunction is called >> FormFunction is called >> 3 SNES Function norm 1.390073376131e-09 >> number of SNES iterations = 3 >> >> Norm of error 1.49795e-10, Iterations 3 >> >> "FormFunction" is called TWICE at "0 KSP". >> >> If we comment out MatMFFDSetFunction: >> >> /* ierr = MatMFFDSetFunction(Jacobian,FormFunction_MFFD,(void*)snes);CHKERRQ(ierr); */ >> >> >> ./ex2 -snes_monitor -ksp_monitor -snes_mf_operator 1 >> >> atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 >> FormFunction is called >> 0 SNES Function norm 5.414682427127e+00 >> 0 KSP Residual norm 9.559082033938e-01 >> FormFunction is called >> 1 KSP Residual norm 1.703870633386e-09 >> FormFunction is called >> FormFunction is called >> 1 SNES Function norm 2.952582481151e-01 >> 0 KSP Residual norm 2.672054855433e-02 >> FormFunction is called >> 1 KSP Residual norm 1.519298012177e-10 >> FormFunction is called >> FormFunction is called >> 2 SNES Function norm 4.502289047587e-04 >> 0 KSP Residual norm 4.722075651268e-05 >> FormFunction is called >> 1 KSP Residual norm 3.834927363659e-14 >> FormFunction is called >> FormFunction is called >> 3 SNES Function norm 1.390073376131e-09 >> number of SNES iterations = 3 >> >> Norm of error 1.49795e-10, Iterations 3 >> >> "FormFunction" is called ONCE at "0 KSP". >> >> Hopefully, this example makes the point clear. >> >> >> Fande, >> >> On Fri, Jan 26, 2018 at 3:50 PM, Smith, Barry F. wrote: >> >> >> So you are doing something non-standard? Are you not just using -snes_mf or -snes_mf_operator? Can you send me a sample code that has the extra function evaluations? Because if you run through regular usage with the debugger you will see there is no extra evaluation. >> >> Barry >> >> >> > On Jan 26, 2018, at 4:32 PM, Kong, Fande wrote: >> > >> > >> > >> > On Fri, Jan 26, 2018 at 3:10 PM, Smith, Barry F. wrote: >> > >> > >> > > On Jan 26, 2018, at 2:15 PM, Kong, Fande wrote: >> > > >> > > >> > > >> > > On Mon, Jan 8, 2018 at 2:15 PM, Smith, Barry F. wrote: >> > > >> > > >> > > > On Jan 8, 2018, at 2:59 PM, Alexander Lindsay wrote: >> > > > >> > > > Is there any elegant way to tell whether SNESComputeFunction is being called under different conceptual contexts? >> > > > >> > > > E.g. non-linear residual evaluation vs. Jacobian formation from finite differencing vs. Jacobian-vector products from finite differencing? >> > > >> > > Under normal usage with the options database no. >> > > >> > > Hi Barry, >> > > >> > > How difficult to provide an API? Is it possible? >> > > >> > > >> > > >> > > If you have some reason to know you could write three functions and provide them to SNESSetFunction(), MatMFFDSetFunction(), and MatFDColoringSetFunction(). Note that these functions have slightly different calling sequences but you can have all of them call the same underlying common function if you are only interested in, for example, how many times the function is used for each purpose. >> > > >> > > If we use this way for the Jacobian-free Newton, the function evaluation will be called twice at the first linear iteration because the computed residual vector at the nonlinear step is not reused. Any way to reuse the function residual of the Newton step instead of recomputing a new residual at the first linear iteration? >> > >> > It does reuse the function evaluation. Why do you think it does not? If you look at MatMult_MFFD() you will see the lines of code >> > >> > /* compute func(U) as base for differencing; only needed first time in and not when provided by user */ >> > if (ctx->ncurrenth == 1 && ctx->current_f_allocated) { >> > ierr = (*ctx->func)(ctx->funcctx,U,F);CHKERRQ(ierr); >> > } >> > >> > since the if is satisfied it does not compute the function at the base location. To double check I ran src/snes/examples/tutorials/ex19 with -snes_mf in the debugger and verified that the "extra" function evaluations are not done. >> > >> > In MatAssemblyEnd_SNESMF, >> > >> > if (j->func == (PetscErrorCode (*)(void*,Vec,Vec))SNESComputeFunction) { >> > ierr = SNESGetFunction(snes,&f,NULL,NULL);CHKERRQ(ierr); >> > ierr = MatMFFDSetBase_MFFD(J,u,f);CHKERRQ(ierr); >> > } else { >> > /* f value known by SNES is not correct for other differencing function */ >> > ierr = MatMFFDSetBase_MFFD(J,u,NULL);CHKERRQ(ierr); >> > } >> > >> > >> > Will hit ierr = MatMFFDSetBase_MFFD(J,u,NULL);CHKERRQ(ierr), because SNES and MAT have different function pointers. >> > >> > In MatMFFDSetBase_MFFD(Mat J,Vec U,Vec F), >> > >> > if (F) { >> > if (ctx->current_f_allocated) {ierr = VecDestroy(&ctx->current_f);CHKERRQ(ierr);} >> > ctx->current_f = F; >> > ctx->current_f_allocated = PETSC_FALSE; >> > } else if (!ctx->current_f_allocated) { >> > ierr = MatCreateVecs(J,NULL,&ctx->current_f);CHKERRQ(ierr); >> > >> > ctx->current_f_allocated = PETSC_TRUE; >> > } >> > >> > Because F=NULL, we then have ctx->current_f_allocated = PETSC_TRUE. >> > >> > Then, the following if statement is true: >> > >> > if (ctx->ncurrenth == 1 && ctx->current_f_allocated) { >> > ierr = (*ctx->func)(ctx->funcctx,U,F);CHKERRQ(ierr); >> > } >> > >> > >> > Fande, >> > >> > >> > >> > Barry >> > >> > >> > > >> > > Fande, >> > > >> > > >> > > >> > > Barry >> > > >> > > >> > > >> > > > >> > > > Alex >> > > >> > > >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex3.c Type: application/octet-stream Size: 14125 bytes Desc: ex3.c URL: From fdkong.jd at gmail.com Sat Jan 27 11:18:02 2018 From: fdkong.jd at gmail.com (Fande Kong) Date: Sat, 27 Jan 2018 10:18:02 -0700 Subject: [petsc-users] Context behind SNESComputeFunction call In-Reply-To: References: <3D320626-58DC-49EC-A80B-614AC3BEA709@anl.gov> Message-ID: Hi Barry, Thanks so much! Does this requires us to attach a shell function "MatAssemblyEnd_SNESMF"? We need to maintain "MatAssemblyEnd_SNESMF" in the moose side? Could we just add a flag into the original routine "MatAssemblyEnd_SNESMF" in PETSc? *static PetscErrorCode MatAssemblyEnd_SNESMF(Mat J,MatAssemblyType mt)* *{* * PetscErrorCode ierr;* * MatMFFD j = (MatMFFD)J->data;* * SNES snes = (SNES)j->ctx;* * PetscBool refuse;* * Vec u,f;* * PetscFunctionBegin;* * ierr = MatAssemblyEnd_MFFD(J,mt);CHKERRQ(ierr);* * ierr = SNESGetReuseBaseMFFD(SNES snes, & refuse);CHKERRQ(ierr);* * ierr = SNESGetSolution(snes,&u);CHKERRQ(ierr);* * if (j->func == (PetscErrorCode (*)(void*,Vec,Vec))SNESComputeFunction || reuse) {* * ierr = SNESGetFunction(snes,&f,NULL,NULL);CHKERRQ(ierr);* * ierr = MatMFFDSetBase_MFFD(J,u,f);CHKERRQ(ierr);* * } else {* * /* f value known by SNES is not correct for other differencing function */* * ierr = MatMFFDSetBase_MFFD(J,u,NULL);CHKERRQ(ierr);* * }* * PetscFunctionReturn(0);* *}* We may need new APIs SNESGetReuseBaseMFFD and SNESSetReuseBaseMFFD, and a command line option "-snes_reuse_base_mffd". "reuse" is false by default, and it is consistent with current setting. Please let me know what do you think. If you agree, I can make a PR on this. Fande, On Fri, Jan 26, 2018 at 10:47 PM, Smith, Barry F. wrote: > > Thanks, I now understand the situation. > > I have a tentative solution for you that does not require complex new > APIs. I have added one function to the master branch MatSNESMFGetSNES() and > attach a modified example that utilizes this to avoid the extra function > evaluations. > > > > Please let me know if it works for you and what you think, > > Barry > > > > > On Jan 26, 2018, at 5:54 PM, Alexander Lindsay > wrote: > > > > We are doing something non-standard. We set different functions for snes > and mffd. When a function is called through snes, we know we are doing a > non-linear residual evaluation and we allow an update of our mechanical > contact nodes. When a function is called through mffd, we know we are > within the linear solve, and we do not allow our contact state to change. > If after a contact update our non-linear residual is not deemed to be > converged, then we would hope to be able to re-use it for the zeroth-linear > residual. > > > > I know that you don't like this way of handling contact, but it > integrates well for us into our multiphysics framework and works well in > many cases. > > > > On Jan 26, 2018, at 5:34 PM, Kong, Fande wrote: > > > >> Hi Barry, > >> > >> I made minor changes on src/snes/examples/tutorials/ex2.c to > demonstrate this issue. Please see the attachment. > >> > >> ./ex2 -snes_monitor -ksp_monitor -snes_mf_operator 1 > >> > >> > >> atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 > >> FormFunction is called > >> 0 SNES Function norm 5.414682427127e+00 > >> 0 KSP Residual norm 9.559082033938e-01 > >> FormFunction is called > >> FormFunction is called > >> 1 KSP Residual norm 1.703870633386e-09 > >> FormFunction is called > >> FormFunction is called > >> 1 SNES Function norm 2.952582481151e-01 > >> 0 KSP Residual norm 2.672054855433e-02 > >> FormFunction is called > >> FormFunction is called > >> 1 KSP Residual norm 1.519298012177e-10 > >> FormFunction is called > >> FormFunction is called > >> 2 SNES Function norm 4.502289047587e-04 > >> 0 KSP Residual norm 4.722075651268e-05 > >> FormFunction is called > >> FormFunction is called > >> 1 KSP Residual norm 3.834927363659e-14 > >> FormFunction is called > >> FormFunction is called > >> 3 SNES Function norm 1.390073376131e-09 > >> number of SNES iterations = 3 > >> > >> Norm of error 1.49795e-10, Iterations 3 > >> > >> "FormFunction" is called TWICE at "0 KSP". > >> > >> If we comment out MatMFFDSetFunction: > >> > >> /* ierr = MatMFFDSetFunction(Jacobian,FormFunction_MFFD,(void*)snes);CHKERRQ(ierr); > */ > >> > >> > >> ./ex2 -snes_monitor -ksp_monitor -snes_mf_operator 1 > >> > >> atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 > >> FormFunction is called > >> 0 SNES Function norm 5.414682427127e+00 > >> 0 KSP Residual norm 9.559082033938e-01 > >> FormFunction is called > >> 1 KSP Residual norm 1.703870633386e-09 > >> FormFunction is called > >> FormFunction is called > >> 1 SNES Function norm 2.952582481151e-01 > >> 0 KSP Residual norm 2.672054855433e-02 > >> FormFunction is called > >> 1 KSP Residual norm 1.519298012177e-10 > >> FormFunction is called > >> FormFunction is called > >> 2 SNES Function norm 4.502289047587e-04 > >> 0 KSP Residual norm 4.722075651268e-05 > >> FormFunction is called > >> 1 KSP Residual norm 3.834927363659e-14 > >> FormFunction is called > >> FormFunction is called > >> 3 SNES Function norm 1.390073376131e-09 > >> number of SNES iterations = 3 > >> > >> Norm of error 1.49795e-10, Iterations 3 > >> > >> "FormFunction" is called ONCE at "0 KSP". > >> > >> Hopefully, this example makes the point clear. > >> > >> > >> Fande, > >> > >> On Fri, Jan 26, 2018 at 3:50 PM, Smith, Barry F. > wrote: > >> > >> > >> So you are doing something non-standard? Are you not just using > -snes_mf or -snes_mf_operator? Can you send me a sample code that has the > extra function evaluations? Because if you run through regular usage with > the debugger you will see there is no extra evaluation. > >> > >> Barry > >> > >> > >> > On Jan 26, 2018, at 4:32 PM, Kong, Fande wrote: > >> > > >> > > >> > > >> > On Fri, Jan 26, 2018 at 3:10 PM, Smith, Barry F. > wrote: > >> > > >> > > >> > > On Jan 26, 2018, at 2:15 PM, Kong, Fande > wrote: > >> > > > >> > > > >> > > > >> > > On Mon, Jan 8, 2018 at 2:15 PM, Smith, Barry F. > wrote: > >> > > > >> > > > >> > > > On Jan 8, 2018, at 2:59 PM, Alexander Lindsay < > alexlindsay239 at gmail.com> wrote: > >> > > > > >> > > > Is there any elegant way to tell whether SNESComputeFunction is > being called under different conceptual contexts? > >> > > > > >> > > > E.g. non-linear residual evaluation vs. Jacobian formation from > finite differencing vs. Jacobian-vector products from finite differencing? > >> > > > >> > > Under normal usage with the options database no. > >> > > > >> > > Hi Barry, > >> > > > >> > > How difficult to provide an API? Is it possible? > >> > > > >> > > > >> > > > >> > > If you have some reason to know you could write three functions > and provide them to SNESSetFunction(), MatMFFDSetFunction(), and > MatFDColoringSetFunction(). Note that these functions have slightly > different calling sequences but you can have all of them call the same > underlying common function if you are only interested in, for example, how > many times the function is used for each purpose. > >> > > > >> > > If we use this way for the Jacobian-free Newton, the function > evaluation will be called twice at the first linear iteration because the > computed residual vector at the nonlinear step is not reused. Any way to > reuse the function residual of the Newton step instead of recomputing a new > residual at the first linear iteration? > >> > > >> > It does reuse the function evaluation. Why do you think it does > not? If you look at MatMult_MFFD() you will see the lines of code > >> > > >> > /* compute func(U) as base for differencing; only needed first time > in and not when provided by user */ > >> > if (ctx->ncurrenth == 1 && ctx->current_f_allocated) { > >> > ierr = (*ctx->func)(ctx->funcctx,U,F);CHKERRQ(ierr); > >> > } > >> > > >> > since the if is satisfied it does not compute the function at the > base location. To double check I ran src/snes/examples/tutorials/ex19 > with -snes_mf in the debugger and verified that the "extra" function > evaluations are not done. > >> > > >> > In MatAssemblyEnd_SNESMF, > >> > > >> > if (j->func == (PetscErrorCode (*)(void*,Vec,Vec))SNESComputeFunction) > { > >> > ierr = SNESGetFunction(snes,&f,NULL,NULL);CHKERRQ(ierr); > >> > ierr = MatMFFDSetBase_MFFD(J,u,f);CHKERRQ(ierr); > >> > } else { > >> > /* f value known by SNES is not correct for other differencing > function */ > >> > ierr = MatMFFDSetBase_MFFD(J,u,NULL);CHKERRQ(ierr); > >> > } > >> > > >> > > >> > Will hit ierr = MatMFFDSetBase_MFFD(J,u,NULL);CHKERRQ(ierr), because > SNES and MAT have different function pointers. > >> > > >> > In MatMFFDSetBase_MFFD(Mat J,Vec U,Vec F), > >> > > >> > if (F) { > >> > if (ctx->current_f_allocated) {ierr = VecDestroy(&ctx->current_f); > CHKERRQ(ierr);} > >> > ctx->current_f = F; > >> > ctx->current_f_allocated = PETSC_FALSE; > >> > } else if (!ctx->current_f_allocated) { > >> > ierr = MatCreateVecs(J,NULL,&ctx->current_f);CHKERRQ(ierr); > >> > > >> > ctx->current_f_allocated = PETSC_TRUE; > >> > } > >> > > >> > Because F=NULL, we then have ctx->current_f_allocated = PETSC_TRUE. > >> > > >> > Then, the following if statement is true: > >> > > >> > if (ctx->ncurrenth == 1 && ctx->current_f_allocated) { > >> > ierr = (*ctx->func)(ctx->funcctx,U,F);CHKERRQ(ierr); > >> > } > >> > > >> > > >> > Fande, > >> > > >> > > >> > > >> > Barry > >> > > >> > > >> > > > >> > > Fande, > >> > > > >> > > > >> > > > >> > > Barry > >> > > > >> > > > >> > > > >> > > > > >> > > > Alex > >> > > > >> > > > >> > >> > >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Jan 27 16:10:06 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Sat, 27 Jan 2018 22:10:06 +0000 Subject: [petsc-users] Context behind SNESComputeFunction call In-Reply-To: References: <3D320626-58DC-49EC-A80B-614AC3BEA709@anl.gov> Message-ID: Fande, I have done something similar in the branch https://bitbucket.org/petsc/petsc/pull-requests/845/added-matsnesmfsetreusebase-at-request-of/diff attached is your test case. I don't think a command line option makes sense for this functionality. Barry I didn't do it with a SNESSetReuseBaseMFFD() because I didn't want the functionality of the MatSNESMFCreate() function to "bleed over" into the SNES object. That is, it is not the business of the SNES object to even know about details of specific matrix types it might use. > On Jan 27, 2018, at 11:18 AM, Fande Kong wrote: > > Hi Barry, > > Thanks so much! > > Does this requires us to attach a shell function "MatAssemblyEnd_SNESMF"? We need to maintain "MatAssemblyEnd_SNESMF" in the moose side? > > > > Could we just add a flag into the original routine "MatAssemblyEnd_SNESMF" in PETSc? > > > static PetscErrorCode MatAssemblyEnd_SNESMF(Mat J,MatAssemblyType mt) > { > PetscErrorCode ierr; > MatMFFD j = (MatMFFD)J->data; > SNES snes = (SNES)j->ctx; > PetscBool refuse; > Vec u,f; > > PetscFunctionBegin; > ierr = MatAssemblyEnd_MFFD(J,mt);CHKERRQ(ierr); > > ierr = SNESGetReuseBaseMFFD(SNES snes, & refuse);CHKERRQ(ierr); > > ierr = SNESGetSolution(snes,&u);CHKERRQ(ierr); > if (j->func == (PetscErrorCode (*)(void*,Vec,Vec))SNESComputeFunction || reuse) { > ierr = SNESGetFunction(snes,&f,NULL,NULL);CHKERRQ(ierr); > ierr = MatMFFDSetBase_MFFD(J,u,f);CHKERRQ(ierr); > } else { > /* f value known by SNES is not correct for other differencing function */ > ierr = MatMFFDSetBase_MFFD(J,u,NULL);CHKERRQ(ierr); > } > PetscFunctionReturn(0); > } > > > > We may need new APIs SNESGetReuseBaseMFFD and SNESSetReuseBaseMFFD, and a command line option "-snes_reuse_base_mffd". "reuse" is false by default, and it is consistent with current setting. > > > Please let me know what do you think. If you agree, I can make a PR on this. > > Fande, > > > On Fri, Jan 26, 2018 at 10:47 PM, Smith, Barry F. wrote: > > Thanks, I now understand the situation. > > I have a tentative solution for you that does not require complex new APIs. I have added one function to the master branch MatSNESMFGetSNES() and attach a modified example that utilizes this to avoid the extra function evaluations. > > > > Please let me know if it works for you and what you think, > > Barry > > > > > On Jan 26, 2018, at 5:54 PM, Alexander Lindsay wrote: > > > > We are doing something non-standard. We set different functions for snes and mffd. When a function is called through snes, we know we are doing a non-linear residual evaluation and we allow an update of our mechanical contact nodes. When a function is called through mffd, we know we are within the linear solve, and we do not allow our contact state to change. If after a contact update our non-linear residual is not deemed to be converged, then we would hope to be able to re-use it for the zeroth-linear residual. > > > > I know that you don't like this way of handling contact, but it integrates well for us into our multiphysics framework and works well in many cases. > > > > On Jan 26, 2018, at 5:34 PM, Kong, Fande wrote: > > > >> Hi Barry, > >> > >> I made minor changes on src/snes/examples/tutorials/ex2.c to demonstrate this issue. Please see the attachment. > >> > >> ./ex2 -snes_monitor -ksp_monitor -snes_mf_operator 1 > >> > >> > >> atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 > >> FormFunction is called > >> 0 SNES Function norm 5.414682427127e+00 > >> 0 KSP Residual norm 9.559082033938e-01 > >> FormFunction is called > >> FormFunction is called > >> 1 KSP Residual norm 1.703870633386e-09 > >> FormFunction is called > >> FormFunction is called > >> 1 SNES Function norm 2.952582481151e-01 > >> 0 KSP Residual norm 2.672054855433e-02 > >> FormFunction is called > >> FormFunction is called > >> 1 KSP Residual norm 1.519298012177e-10 > >> FormFunction is called > >> FormFunction is called > >> 2 SNES Function norm 4.502289047587e-04 > >> 0 KSP Residual norm 4.722075651268e-05 > >> FormFunction is called > >> FormFunction is called > >> 1 KSP Residual norm 3.834927363659e-14 > >> FormFunction is called > >> FormFunction is called > >> 3 SNES Function norm 1.390073376131e-09 > >> number of SNES iterations = 3 > >> > >> Norm of error 1.49795e-10, Iterations 3 > >> > >> "FormFunction" is called TWICE at "0 KSP". > >> > >> If we comment out MatMFFDSetFunction: > >> > >> /* ierr = MatMFFDSetFunction(Jacobian,FormFunction_MFFD,(void*)snes);CHKERRQ(ierr); */ > >> > >> > >> ./ex2 -snes_monitor -ksp_monitor -snes_mf_operator 1 > >> > >> atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 > >> FormFunction is called > >> 0 SNES Function norm 5.414682427127e+00 > >> 0 KSP Residual norm 9.559082033938e-01 > >> FormFunction is called > >> 1 KSP Residual norm 1.703870633386e-09 > >> FormFunction is called > >> FormFunction is called > >> 1 SNES Function norm 2.952582481151e-01 > >> 0 KSP Residual norm 2.672054855433e-02 > >> FormFunction is called > >> 1 KSP Residual norm 1.519298012177e-10 > >> FormFunction is called > >> FormFunction is called > >> 2 SNES Function norm 4.502289047587e-04 > >> 0 KSP Residual norm 4.722075651268e-05 > >> FormFunction is called > >> 1 KSP Residual norm 3.834927363659e-14 > >> FormFunction is called > >> FormFunction is called > >> 3 SNES Function norm 1.390073376131e-09 > >> number of SNES iterations = 3 > >> > >> Norm of error 1.49795e-10, Iterations 3 > >> > >> "FormFunction" is called ONCE at "0 KSP". > >> > >> Hopefully, this example makes the point clear. > >> > >> > >> Fande, > >> > >> On Fri, Jan 26, 2018 at 3:50 PM, Smith, Barry F. wrote: > >> > >> > >> So you are doing something non-standard? Are you not just using -snes_mf or -snes_mf_operator? Can you send me a sample code that has the extra function evaluations? Because if you run through regular usage with the debugger you will see there is no extra evaluation. > >> > >> Barry > >> > >> > >> > On Jan 26, 2018, at 4:32 PM, Kong, Fande wrote: > >> > > >> > > >> > > >> > On Fri, Jan 26, 2018 at 3:10 PM, Smith, Barry F. wrote: > >> > > >> > > >> > > On Jan 26, 2018, at 2:15 PM, Kong, Fande wrote: > >> > > > >> > > > >> > > > >> > > On Mon, Jan 8, 2018 at 2:15 PM, Smith, Barry F. wrote: > >> > > > >> > > > >> > > > On Jan 8, 2018, at 2:59 PM, Alexander Lindsay wrote: > >> > > > > >> > > > Is there any elegant way to tell whether SNESComputeFunction is being called under different conceptual contexts? > >> > > > > >> > > > E.g. non-linear residual evaluation vs. Jacobian formation from finite differencing vs. Jacobian-vector products from finite differencing? > >> > > > >> > > Under normal usage with the options database no. > >> > > > >> > > Hi Barry, > >> > > > >> > > How difficult to provide an API? Is it possible? > >> > > > >> > > > >> > > > >> > > If you have some reason to know you could write three functions and provide them to SNESSetFunction(), MatMFFDSetFunction(), and MatFDColoringSetFunction(). Note that these functions have slightly different calling sequences but you can have all of them call the same underlying common function if you are only interested in, for example, how many times the function is used for each purpose. > >> > > > >> > > If we use this way for the Jacobian-free Newton, the function evaluation will be called twice at the first linear iteration because the computed residual vector at the nonlinear step is not reused. Any way to reuse the function residual of the Newton step instead of recomputing a new residual at the first linear iteration? > >> > > >> > It does reuse the function evaluation. Why do you think it does not? If you look at MatMult_MFFD() you will see the lines of code > >> > > >> > /* compute func(U) as base for differencing; only needed first time in and not when provided by user */ > >> > if (ctx->ncurrenth == 1 && ctx->current_f_allocated) { > >> > ierr = (*ctx->func)(ctx->funcctx,U,F);CHKERRQ(ierr); > >> > } > >> > > >> > since the if is satisfied it does not compute the function at the base location. To double check I ran src/snes/examples/tutorials/ex19 with -snes_mf in the debugger and verified that the "extra" function evaluations are not done. > >> > > >> > In MatAssemblyEnd_SNESMF, > >> > > >> > if (j->func == (PetscErrorCode (*)(void*,Vec,Vec))SNESComputeFunction) { > >> > ierr = SNESGetFunction(snes,&f,NULL,NULL);CHKERRQ(ierr); > >> > ierr = MatMFFDSetBase_MFFD(J,u,f);CHKERRQ(ierr); > >> > } else { > >> > /* f value known by SNES is not correct for other differencing function */ > >> > ierr = MatMFFDSetBase_MFFD(J,u,NULL);CHKERRQ(ierr); > >> > } > >> > > >> > > >> > Will hit ierr = MatMFFDSetBase_MFFD(J,u,NULL);CHKERRQ(ierr), because SNES and MAT have different function pointers. > >> > > >> > In MatMFFDSetBase_MFFD(Mat J,Vec U,Vec F), > >> > > >> > if (F) { > >> > if (ctx->current_f_allocated) {ierr = VecDestroy(&ctx->current_f);CHKERRQ(ierr);} > >> > ctx->current_f = F; > >> > ctx->current_f_allocated = PETSC_FALSE; > >> > } else if (!ctx->current_f_allocated) { > >> > ierr = MatCreateVecs(J,NULL,&ctx->current_f);CHKERRQ(ierr); > >> > > >> > ctx->current_f_allocated = PETSC_TRUE; > >> > } > >> > > >> > Because F=NULL, we then have ctx->current_f_allocated = PETSC_TRUE. > >> > > >> > Then, the following if statement is true: > >> > > >> > if (ctx->ncurrenth == 1 && ctx->current_f_allocated) { > >> > ierr = (*ctx->func)(ctx->funcctx,U,F);CHKERRQ(ierr); > >> > } > >> > > >> > > >> > Fande, > >> > > >> > > >> > > >> > Barry > >> > > >> > > >> > > > >> > > Fande, > >> > > > >> > > > >> > > > >> > > Barry > >> > > > >> > > > >> > > > >> > > > > >> > > > Alex > >> > > > >> > > > >> > >> > >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex3.c Type: application/octet-stream Size: 13541 bytes Desc: ex3.c URL: From fdkong.jd at gmail.com Sat Jan 27 17:10:00 2018 From: fdkong.jd at gmail.com (Fande Kong) Date: Sat, 27 Jan 2018 16:10:00 -0700 Subject: [petsc-users] Context behind SNESComputeFunction call In-Reply-To: References: <3D320626-58DC-49EC-A80B-614AC3BEA709@anl.gov> Message-ID: Thanks, Barry, On Sat, Jan 27, 2018 at 3:10 PM, Smith, Barry F. wrote: > > Fande, > > I have done something similar in the branch > https://bitbucket.org/petsc/petsc/pull-requests/845/added- > matsnesmfsetreusebase-at-request-of/diff attached is your test case. > I like this design. I will try out at the moose side. Fande, > > > I don't think a command line option makes sense for this functionality. > > Barry > > I didn't do it with a SNESSetReuseBaseMFFD() because I didn't want the > functionality of the MatSNESMFCreate() function to "bleed over" into the > SNES object. That is, it is not the business of the SNES object to even > know about details of specific matrix types it might use. > > > > > On Jan 27, 2018, at 11:18 AM, Fande Kong wrote: > > > > Hi Barry, > > > > Thanks so much! > > > > Does this requires us to attach a shell function > "MatAssemblyEnd_SNESMF"? We need to maintain "MatAssemblyEnd_SNESMF" in the > moose side? > > > > > > > > Could we just add a flag into the original routine > "MatAssemblyEnd_SNESMF" in PETSc? > > > > > > static PetscErrorCode MatAssemblyEnd_SNESMF(Mat J,MatAssemblyType mt) > > { > > PetscErrorCode ierr; > > MatMFFD j = (MatMFFD)J->data; > > SNES snes = (SNES)j->ctx; > > PetscBool refuse; > > Vec u,f; > > > > PetscFunctionBegin; > > ierr = MatAssemblyEnd_MFFD(J,mt);CHKERRQ(ierr); > > > > ierr = SNESGetReuseBaseMFFD(SNES snes, & refuse);CHKERRQ(ierr); > > > > ierr = SNESGetSolution(snes,&u);CHKERRQ(ierr); > > if (j->func == (PetscErrorCode (*)(void*,Vec,Vec))SNESComputeFunction > || reuse) { > > ierr = SNESGetFunction(snes,&f,NULL,NULL);CHKERRQ(ierr); > > ierr = MatMFFDSetBase_MFFD(J,u,f);CHKERRQ(ierr); > > } else { > > /* f value known by SNES is not correct for other differencing > function */ > > ierr = MatMFFDSetBase_MFFD(J,u,NULL);CHKERRQ(ierr); > > } > > PetscFunctionReturn(0); > > } > > > > > > > > We may need new APIs SNESGetReuseBaseMFFD and SNESSetReuseBaseMFFD, and > a command line option "-snes_reuse_base_mffd". "reuse" is false by > default, and it is consistent with current setting. > > > > > > Please let me know what do you think. If you agree, I can make a PR on > this. > > > > Fande, > > > > > > On Fri, Jan 26, 2018 at 10:47 PM, Smith, Barry F. > wrote: > > > > Thanks, I now understand the situation. > > > > I have a tentative solution for you that does not require complex new > APIs. I have added one function to the master branch MatSNESMFGetSNES() and > attach a modified example that utilizes this to avoid the extra function > evaluations. > > > > > > > > Please let me know if it works for you and what you think, > > > > Barry > > > > > > > > > On Jan 26, 2018, at 5:54 PM, Alexander Lindsay < > alexlindsay239 at gmail.com> wrote: > > > > > > We are doing something non-standard. We set different functions for > snes and mffd. When a function is called through snes, we know we are doing > a non-linear residual evaluation and we allow an update of our mechanical > contact nodes. When a function is called through mffd, we know we are > within the linear solve, and we do not allow our contact state to change. > If after a contact update our non-linear residual is not deemed to be > converged, then we would hope to be able to re-use it for the zeroth-linear > residual. > > > > > > I know that you don't like this way of handling contact, but it > integrates well for us into our multiphysics framework and works well in > many cases. > > > > > > On Jan 26, 2018, at 5:34 PM, Kong, Fande wrote: > > > > > >> Hi Barry, > > >> > > >> I made minor changes on src/snes/examples/tutorials/ex2.c to > demonstrate this issue. Please see the attachment. > > >> > > >> ./ex2 -snes_monitor -ksp_monitor -snes_mf_operator 1 > > >> > > >> > > >> atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 > > >> FormFunction is called > > >> 0 SNES Function norm 5.414682427127e+00 > > >> 0 KSP Residual norm 9.559082033938e-01 > > >> FormFunction is called > > >> FormFunction is called > > >> 1 KSP Residual norm 1.703870633386e-09 > > >> FormFunction is called > > >> FormFunction is called > > >> 1 SNES Function norm 2.952582481151e-01 > > >> 0 KSP Residual norm 2.672054855433e-02 > > >> FormFunction is called > > >> FormFunction is called > > >> 1 KSP Residual norm 1.519298012177e-10 > > >> FormFunction is called > > >> FormFunction is called > > >> 2 SNES Function norm 4.502289047587e-04 > > >> 0 KSP Residual norm 4.722075651268e-05 > > >> FormFunction is called > > >> FormFunction is called > > >> 1 KSP Residual norm 3.834927363659e-14 > > >> FormFunction is called > > >> FormFunction is called > > >> 3 SNES Function norm 1.390073376131e-09 > > >> number of SNES iterations = 3 > > >> > > >> Norm of error 1.49795e-10, Iterations 3 > > >> > > >> "FormFunction" is called TWICE at "0 KSP". > > >> > > >> If we comment out MatMFFDSetFunction: > > >> > > >> /* ierr = MatMFFDSetFunction(Jacobian,FormFunction_MFFD,(void*)snes);CHKERRQ(ierr); > */ > > >> > > >> > > >> ./ex2 -snes_monitor -ksp_monitor -snes_mf_operator 1 > > >> > > >> atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 > > >> FormFunction is called > > >> 0 SNES Function norm 5.414682427127e+00 > > >> 0 KSP Residual norm 9.559082033938e-01 > > >> FormFunction is called > > >> 1 KSP Residual norm 1.703870633386e-09 > > >> FormFunction is called > > >> FormFunction is called > > >> 1 SNES Function norm 2.952582481151e-01 > > >> 0 KSP Residual norm 2.672054855433e-02 > > >> FormFunction is called > > >> 1 KSP Residual norm 1.519298012177e-10 > > >> FormFunction is called > > >> FormFunction is called > > >> 2 SNES Function norm 4.502289047587e-04 > > >> 0 KSP Residual norm 4.722075651268e-05 > > >> FormFunction is called > > >> 1 KSP Residual norm 3.834927363659e-14 > > >> FormFunction is called > > >> FormFunction is called > > >> 3 SNES Function norm 1.390073376131e-09 > > >> number of SNES iterations = 3 > > >> > > >> Norm of error 1.49795e-10, Iterations 3 > > >> > > >> "FormFunction" is called ONCE at "0 KSP". > > >> > > >> Hopefully, this example makes the point clear. > > >> > > >> > > >> Fande, > > >> > > >> On Fri, Jan 26, 2018 at 3:50 PM, Smith, Barry F. > wrote: > > >> > > >> > > >> So you are doing something non-standard? Are you not just using > -snes_mf or -snes_mf_operator? Can you send me a sample code that has the > extra function evaluations? Because if you run through regular usage with > the debugger you will see there is no extra evaluation. > > >> > > >> Barry > > >> > > >> > > >> > On Jan 26, 2018, at 4:32 PM, Kong, Fande > wrote: > > >> > > > >> > > > >> > > > >> > On Fri, Jan 26, 2018 at 3:10 PM, Smith, Barry F. < > bsmith at mcs.anl.gov> wrote: > > >> > > > >> > > > >> > > On Jan 26, 2018, at 2:15 PM, Kong, Fande > wrote: > > >> > > > > >> > > > > >> > > > > >> > > On Mon, Jan 8, 2018 at 2:15 PM, Smith, Barry F. < > bsmith at mcs.anl.gov> wrote: > > >> > > > > >> > > > > >> > > > On Jan 8, 2018, at 2:59 PM, Alexander Lindsay < > alexlindsay239 at gmail.com> wrote: > > >> > > > > > >> > > > Is there any elegant way to tell whether SNESComputeFunction is > being called under different conceptual contexts? > > >> > > > > > >> > > > E.g. non-linear residual evaluation vs. Jacobian formation from > finite differencing vs. Jacobian-vector products from finite differencing? > > >> > > > > >> > > Under normal usage with the options database no. > > >> > > > > >> > > Hi Barry, > > >> > > > > >> > > How difficult to provide an API? Is it possible? > > >> > > > > >> > > > > >> > > > > >> > > If you have some reason to know you could write three functions > and provide them to SNESSetFunction(), MatMFFDSetFunction(), and > MatFDColoringSetFunction(). Note that these functions have slightly > different calling sequences but you can have all of them call the same > underlying common function if you are only interested in, for example, how > many times the function is used for each purpose. > > >> > > > > >> > > If we use this way for the Jacobian-free Newton, the function > evaluation will be called twice at the first linear iteration because the > computed residual vector at the nonlinear step is not reused. Any way to > reuse the function residual of the Newton step instead of recomputing a new > residual at the first linear iteration? > > >> > > > >> > It does reuse the function evaluation. Why do you think it does > not? If you look at MatMult_MFFD() you will see the lines of code > > >> > > > >> > /* compute func(U) as base for differencing; only needed first > time in and not when provided by user */ > > >> > if (ctx->ncurrenth == 1 && ctx->current_f_allocated) { > > >> > ierr = (*ctx->func)(ctx->funcctx,U,F);CHKERRQ(ierr); > > >> > } > > >> > > > >> > since the if is satisfied it does not compute the function at the > base location. To double check I ran src/snes/examples/tutorials/ex19 > with -snes_mf in the debugger and verified that the "extra" function > evaluations are not done. > > >> > > > >> > In MatAssemblyEnd_SNESMF, > > >> > > > >> > if (j->func == (PetscErrorCode (*)(void*,Vec,Vec))SNESComputeFunction) > { > > >> > ierr = SNESGetFunction(snes,&f,NULL,NULL);CHKERRQ(ierr); > > >> > ierr = MatMFFDSetBase_MFFD(J,u,f);CHKERRQ(ierr); > > >> > } else { > > >> > /* f value known by SNES is not correct for other differencing > function */ > > >> > ierr = MatMFFDSetBase_MFFD(J,u,NULL);CHKERRQ(ierr); > > >> > } > > >> > > > >> > > > >> > Will hit ierr = MatMFFDSetBase_MFFD(J,u,NULL);CHKERRQ(ierr), > because SNES and MAT have different function pointers. > > >> > > > >> > In MatMFFDSetBase_MFFD(Mat J,Vec U,Vec F), > > >> > > > >> > if (F) { > > >> > if (ctx->current_f_allocated) {ierr = > VecDestroy(&ctx->current_f);CHKERRQ(ierr);} > > >> > ctx->current_f = F; > > >> > ctx->current_f_allocated = PETSC_FALSE; > > >> > } else if (!ctx->current_f_allocated) { > > >> > ierr = MatCreateVecs(J,NULL,&ctx->current_f);CHKERRQ(ierr); > > >> > > > >> > ctx->current_f_allocated = PETSC_TRUE; > > >> > } > > >> > > > >> > Because F=NULL, we then have ctx->current_f_allocated = PETSC_TRUE. > > >> > > > >> > Then, the following if statement is true: > > >> > > > >> > if (ctx->ncurrenth == 1 && ctx->current_f_allocated) { > > >> > ierr = (*ctx->func)(ctx->funcctx,U,F);CHKERRQ(ierr); > > >> > } > > >> > > > >> > > > >> > Fande, > > >> > > > >> > > > >> > > > >> > Barry > > >> > > > >> > > > >> > > > > >> > > Fande, > > >> > > > > >> > > > > >> > > > > >> > > Barry > > >> > > > > >> > > > > >> > > > > >> > > > > > >> > > > Alex > > >> > > > > >> > > > > >> > > >> > > >> > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Sun Jan 28 09:27:25 2018 From: mfadams at lbl.gov (Mark Adams) Date: Sun, 28 Jan 2018 10:27:25 -0500 Subject: [petsc-users] using makefiles Message-ID: I want to add stuff to compile lines and I tried: include ${PETSC_DIR}/lib/petsc/conf/variables include ${PETSC_DIR}/lib/petsc/conf/rules CC_FLAGS := ${CC_FLAGS} -I$(OLYMPUS_INC_DIR) -I$(FEI_DIR) -DPROM_HAVE_METIS -DPROM_USE_PETSC -DPROM_NO_FEI -DPROM_HAVE_SILO # -I$(SILO_INC) FC_FLAGS := ${FC_FLAGS} -fdefault-integer-8 This a C++ code: 10:21 2 master *= ~/Codes/olympus-keaveny/Olympus$ make getcflags -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -Qunused-arguments -fvisibility=hidden -g -mavx2 -I./include -I./fei_prom -DPROM_HAVE_METIS -DPROM_USE_PETSC -DPROM_NO_FEI -DPROM_HAVE_SILO 10:21 master *= ~/Codes/olympus-keaveny/Olympus$ make /Users/markadams/Codes/petsc/arch-macosx-gnu-g/bin/mpicxx -o pfeap/pfeap.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -g -mavx2 -I/Users/markadams/Codes/petsc/include -I/Users/markadams/Codes/petsc/arch-macosx-gnu-g/include -I/Users/markadams/Codes/petsc/arch-macosx-gnu-g/include/pragmatic -I/Users/markadams/Codes/petsc/arch-macosx-gnu-g/include/eigen3 `pwd`/pfeap/pfeap.C /Users/markadams/Codes/olympus-keaveny/Olympus/pfeap/pfeap.C:7:10: fatal error: 'pfeap.hh' file not found #include "pfeap.hh" ^~~~~~~~~~ 1 error generated. It looks like CC_FLAGS are not getting used by CXX (-I./include is not making it to the compile line). How should I do this? I don't see a CXX_FLAGS. Thanks, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Jan 28 09:29:46 2018 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 29 Jan 2018 02:29:46 +1100 Subject: [petsc-users] using makefiles In-Reply-To: References: Message-ID: On Mon, Jan 29, 2018 at 2:27 AM, Mark Adams wrote: > I want to add stuff to compile lines and I tried: > > > include ${PETSC_DIR}/lib/petsc/conf/variables > include ${PETSC_DIR}/lib/petsc/conf/rules > > CC_FLAGS := ${CC_FLAGS} -I$(OLYMPUS_INC_DIR) -I$(FEI_DIR) > -DPROM_HAVE_METIS -DPROM_USE_PETSC -DPROM_NO_FEI -DPROM_HAVE_SILO # > -I$(SILO_INC) > FC_FLAGS := ${FC_FLAGS} -fdefault-integer-8 > > This a C++ code: > > 10:21 2 master *= ~/Codes/olympus-keaveny/Olympus$ make getcflags > > -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas > -fstack-protector -Qunused-arguments -fvisibility=hidden -g -mavx2 > -I./include -I./fei_prom -DPROM_HAVE_METIS -DPROM_USE_PETSC -DPROM_NO_FEI > -DPROM_HAVE_SILO > > 10:21 master *= ~/Codes/olympus-keaveny/Olympus$ make > > /Users/markadams/Codes/petsc/arch-macosx-gnu-g/bin/mpicxx -o > pfeap/pfeap.o -c -Wall -Wwrite-strings -Wno-strict-aliasing > -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -g -mavx2 > -I/Users/markadams/Codes/petsc/include -I/Users/markadams/Codes/ > petsc/arch-macosx-gnu-g/include -I/Users/markadams/Codes/ > petsc/arch-macosx-gnu-g/include/pragmatic -I/Users/markadams/Codes/ > petsc/arch-macosx-gnu-g/include/eigen3 `pwd`/pfeap/pfeap.C > > /Users/markadams/Codes/olympus-keaveny/Olympus/pfeap/pfeap.C:7:10: fatal > error: 'pfeap.hh' file not found > > #include "pfeap.hh" > > ^~~~~~~~~~ > > 1 error generated. > > It looks like CC_FLAGS are not getting used by CXX (-I./include is not > making it to the compile line). How should I do this? I don't see a > CXX_FLAGS. > It is CXX_FLAGS. Matt > > Thanks, > Mark > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Sun Jan 28 10:42:02 2018 From: mfadams at lbl.gov (Mark Adams) Date: Sun, 28 Jan 2018 11:42:02 -0500 Subject: [petsc-users] using makefiles In-Reply-To: References: Message-ID: Thanks, and did __FUNCT__ go away? On Sun, Jan 28, 2018 at 10:29 AM, Matthew Knepley wrote: > On Mon, Jan 29, 2018 at 2:27 AM, Mark Adams wrote: > >> I want to add stuff to compile lines and I tried: >> >> >> include ${PETSC_DIR}/lib/petsc/conf/variables >> include ${PETSC_DIR}/lib/petsc/conf/rules >> >> CC_FLAGS := ${CC_FLAGS} -I$(OLYMPUS_INC_DIR) -I$(FEI_DIR) >> -DPROM_HAVE_METIS -DPROM_USE_PETSC -DPROM_NO_FEI -DPROM_HAVE_SILO # >> -I$(SILO_INC) >> FC_FLAGS := ${FC_FLAGS} -fdefault-integer-8 >> >> This a C++ code: >> >> 10:21 2 master *= ~/Codes/olympus-keaveny/Olympus$ make getcflags >> >> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >> -fstack-protector -Qunused-arguments -fvisibility=hidden -g -mavx2 >> -I./include -I./fei_prom -DPROM_HAVE_METIS -DPROM_USE_PETSC -DPROM_NO_FEI >> -DPROM_HAVE_SILO >> >> 10:21 master *= ~/Codes/olympus-keaveny/Olympus$ make >> >> /Users/markadams/Codes/petsc/arch-macosx-gnu-g/bin/mpicxx -o >> pfeap/pfeap.o -c -Wall -Wwrite-strings -Wno-strict-aliasing >> -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -g -mavx2 >> -I/Users/markadams/Codes/petsc/include -I/Users/markadams/Codes/petsc/arch-macosx-gnu-g/include >> -I/Users/markadams/Codes/petsc/arch-macosx-gnu-g/include/pragmatic >> -I/Users/markadams/Codes/petsc/arch-macosx-gnu-g/include/eigen3 >> `pwd`/pfeap/pfeap.C >> >> /Users/markadams/Codes/olympus-keaveny/Olympus/pfeap/pfeap.C:7:10: fatal >> error: 'pfeap.hh' file not found >> >> #include "pfeap.hh" >> >> ^~~~~~~~~~ >> >> 1 error generated. >> >> It looks like CC_FLAGS are not getting used by CXX (-I./include is not >> making it to the compile line). How should I do this? I don't see a >> CXX_FLAGS. >> > > It is CXX_FLAGS. > > Matt > > >> >> Thanks, >> Mark >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Sun Jan 28 10:44:55 2018 From: mfadams at lbl.gov (Mark Adams) Date: Sun, 28 Jan 2018 11:44:55 -0500 Subject: [petsc-users] using makefiles In-Reply-To: References: Message-ID: Nevermind, On Sun, Jan 28, 2018 at 11:42 AM, Mark Adams wrote: > Thanks, and did __FUNCT__ go away? > > On Sun, Jan 28, 2018 at 10:29 AM, Matthew Knepley > wrote: > >> On Mon, Jan 29, 2018 at 2:27 AM, Mark Adams wrote: >> >>> I want to add stuff to compile lines and I tried: >>> >>> >>> include ${PETSC_DIR}/lib/petsc/conf/variables >>> include ${PETSC_DIR}/lib/petsc/conf/rules >>> >>> CC_FLAGS := ${CC_FLAGS} -I$(OLYMPUS_INC_DIR) -I$(FEI_DIR) >>> -DPROM_HAVE_METIS -DPROM_USE_PETSC -DPROM_NO_FEI -DPROM_HAVE_SILO # >>> -I$(SILO_INC) >>> FC_FLAGS := ${FC_FLAGS} -fdefault-integer-8 >>> >>> This a C++ code: >>> >>> 10:21 2 master *= ~/Codes/olympus-keaveny/Olympus$ make getcflags >>> >>> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >>> -fstack-protector -Qunused-arguments -fvisibility=hidden -g -mavx2 >>> -I./include -I./fei_prom -DPROM_HAVE_METIS -DPROM_USE_PETSC -DPROM_NO_FEI >>> -DPROM_HAVE_SILO >>> >>> 10:21 master *= ~/Codes/olympus-keaveny/Olympus$ make >>> >>> /Users/markadams/Codes/petsc/arch-macosx-gnu-g/bin/mpicxx -o >>> pfeap/pfeap.o -c -Wall -Wwrite-strings -Wno-strict-aliasing >>> -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -g -mavx2 >>> -I/Users/markadams/Codes/petsc/include >>> -I/Users/markadams/Codes/petsc/arch-macosx-gnu-g/include >>> -I/Users/markadams/Codes/petsc/arch-macosx-gnu-g/include/pragmatic >>> -I/Users/markadams/Codes/petsc/arch-macosx-gnu-g/include/eigen3 >>> `pwd`/pfeap/pfeap.C >>> >>> /Users/markadams/Codes/olympus-keaveny/Olympus/pfeap/pfeap.C:7:10: fatal >>> error: 'pfeap.hh' file not found >>> >>> #include "pfeap.hh" >>> >>> ^~~~~~~~~~ >>> >>> 1 error generated. >>> >>> It looks like CC_FLAGS are not getting used by CXX (-I./include is not >>> making it to the compile line). How should I do this? I don't see a >>> CXX_FLAGS. >>> >> >> It is CXX_FLAGS. >> >> Matt >> >> >>> >>> Thanks, >>> Mark >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Sun Jan 28 11:15:05 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Sun, 28 Jan 2018 11:15:05 -0600 Subject: [petsc-users] using makefiles In-Reply-To: References: Message-ID: Normally you would do: CPPFLAGS = I$(OLYMPUS_INC_DIR) -I$(FEI_DIR) -DPROM_HAVE_METIS -DPROM_USE_PETSC -DPROM_NO_FEI -DPROM_HAVE_SILO # -I$(SILO_INC) FPPFLAGS = -fdefault-integer-8 include ${PETSC_DIR}/lib/petsc/conf/variables include ${PETSC_DIR}/lib/petsc/conf/rules [CFPPFLAGS/FPPFLAGS are used for compile - and CFLAGS/FFLAGS are used for both compile and link] Satish On Sun, 28 Jan 2018, Mark Adams wrote: > I want to add stuff to compile lines and I tried: > > > include ${PETSC_DIR}/lib/petsc/conf/variables > include ${PETSC_DIR}/lib/petsc/conf/rules > > CC_FLAGS := ${CC_FLAGS} -I$(OLYMPUS_INC_DIR) -I$(FEI_DIR) -DPROM_HAVE_METIS > -DPROM_USE_PETSC -DPROM_NO_FEI -DPROM_HAVE_SILO # -I$(SILO_INC) > FC_FLAGS := ${FC_FLAGS} -fdefault-integer-8 > > This a C++ code: > > 10:21 2 master *= ~/Codes/olympus-keaveny/Olympus$ make getcflags > > -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas > -fstack-protector -Qunused-arguments -fvisibility=hidden -g -mavx2 > -I./include -I./fei_prom -DPROM_HAVE_METIS -DPROM_USE_PETSC -DPROM_NO_FEI > -DPROM_HAVE_SILO > > 10:21 master *= ~/Codes/olympus-keaveny/Olympus$ make > > /Users/markadams/Codes/petsc/arch-macosx-gnu-g/bin/mpicxx -o pfeap/pfeap.o > -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas > -fstack-protector -fvisibility=hidden -g -mavx2 > -I/Users/markadams/Codes/petsc/include > -I/Users/markadams/Codes/petsc/arch-macosx-gnu-g/include > -I/Users/markadams/Codes/petsc/arch-macosx-gnu-g/include/pragmatic > -I/Users/markadams/Codes/petsc/arch-macosx-gnu-g/include/eigen3 > `pwd`/pfeap/pfeap.C > > /Users/markadams/Codes/olympus-keaveny/Olympus/pfeap/pfeap.C:7:10: fatal > error: 'pfeap.hh' file not found > > #include "pfeap.hh" > > ^~~~~~~~~~ > > 1 error generated. > > It looks like CC_FLAGS are not getting used by CXX (-I./include is not > making it to the compile line). How should I do this? I don't see a > CXX_FLAGS. > > Thanks, > Mark > From aliberkkahraman at yahoo.com Sun Jan 28 13:09:30 2018 From: aliberkkahraman at yahoo.com (Ali Berk Kahraman) Date: Sun, 28 Jan 2018 22:09:30 +0300 Subject: [petsc-users] Parallel Matrix Causes a Deadlock Message-ID: <60ec2ddf-fe37-4946-1a00-afdde34ceb2c@yahoo.com> Hello All, The code takes a parallel matrix and calls a function using that matrix. That function fills the specified row of that matrix with the id of the process that part of the matrix belongs in. You can see the short code in the attachment, it is about 80 lines. The problem is that the code gets into a deadlock at some point, usually at the last row? of each process except for the last process (greatest pid). I use petsc with configure options "--with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack". I am a beginner with MPI, so I do not know what may be causing this. My apologies in advance if this is a very trivial problem. Best Regards to All, Ali Berk Kahraman M.Sc. Student, Mechanical Eng. Bogazici Uni., Istanbul, Turkey -------------- next part -------------- A non-text attachment was scrubbed... Name: PetscMPIProblem.c Type: text/x-csrc Size: 1880 bytes Desc: not available URL: From bsmith at mcs.anl.gov Sun Jan 28 13:13:42 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Sun, 28 Jan 2018 19:13:42 +0000 Subject: [petsc-users] Parallel Matrix Causes a Deadlock In-Reply-To: <60ec2ddf-fe37-4946-1a00-afdde34ceb2c@yahoo.com> References: <60ec2ddf-fe37-4946-1a00-afdde34ceb2c@yahoo.com> Message-ID: <66B07CCE-1FE2-4906-B280-A91558AEC2B4@anl.gov> You are assuming that all processes enter this loop an identical number of times for(loop2=ownbegin;loop2 On Jan 28, 2018, at 1:09 PM, Ali Berk Kahraman wrote: > > Hello All, > > > The code takes a parallel matrix and calls a function using that matrix. That function fills the specified row of that matrix with the id of the process that part of the matrix belongs in. You can see the short code in the attachment, it is about 80 lines. > > > The problem is that the code gets into a deadlock at some point, usually at the last row of each process except for the last process (greatest pid). I use petsc with configure options "--with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack". > > > I am a beginner with MPI, so I do not know what may be causing this. My apologies in advance if this is a very trivial problem. > > > Best Regards to All, > > > Ali Berk Kahraman > > M.Sc. Student, Mechanical Eng. > > Bogazici Uni., Istanbul, Turkey > > From aliberkkahraman at yahoo.com Sun Jan 28 13:15:13 2018 From: aliberkkahraman at yahoo.com (Ali Berk Kahraman) Date: Sun, 28 Jan 2018 22:15:13 +0300 Subject: [petsc-users] Parallel Matrix Causes a Deadlock In-Reply-To: <60ec2ddf-fe37-4946-1a00-afdde34ceb2c@yahoo.com> References: <60ec2ddf-fe37-4946-1a00-afdde34ceb2c@yahoo.com> Message-ID: <5e66c7be-83a2-cac5-30e9-b7254f198d6c@yahoo.com> Hello All, My apologies, I have closed this e-mail window and the first thing I read on the manual is "ALL processes that share a matrix MUST call MatAssemblyBegin () and MatAssemblyEnd () the SAME NUMBER of times". So I understand that petsc simply does not support unequal number of assembly calls. My question evolves then. I have a problem at hand where I do not know how many calls each process will make to MatAssembly routines. Any suggestions to make this work? On 28-01-2018 22:09, Ali Berk Kahraman wrote: > Hello All, > > > The code takes a parallel matrix and calls a function using that > matrix. That function fills the specified row of that matrix with the > id of the process that part of the matrix belongs in. You can see the > short code in the attachment, it is about 80 lines. > > > The problem is that the code gets into a deadlock at some point, > usually at the last row? of each process except for the last process > (greatest pid). I use petsc with configure options "--with-cc=gcc > --with-cxx=g++ --with-fc=gfortran --download-mpich > --download-fblaslapack". > > > I am a beginner with MPI, so I do not know what may be causing this. > My apologies in advance if this is a very trivial problem. > > > Best Regards to All, > > > Ali Berk Kahraman > > M.Sc. Student, Mechanical Eng. > > Bogazici Uni., Istanbul, Turkey > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Jan 28 13:24:15 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Sun, 28 Jan 2018 19:24:15 +0000 Subject: [petsc-users] Parallel Matrix Causes a Deadlock In-Reply-To: <5e66c7be-83a2-cac5-30e9-b7254f198d6c@yahoo.com> References: <60ec2ddf-fe37-4946-1a00-afdde34ceb2c@yahoo.com> <5e66c7be-83a2-cac5-30e9-b7254f198d6c@yahoo.com> Message-ID: In your code there is NO reason to call the MatAssemblyBegin/End where you do. Just pull it out and call it once. I submit this is the same for any other code. Please explain enough about your code (or send it) that has to call the assembly routines a different number of times. You just pull it above all the calls to MatSetValues(). Barry > On Jan 28, 2018, at 1:15 PM, Ali Berk Kahraman wrote: > > Hello All, > > > My apologies, I have closed this e-mail window and the first thing I read on the manual is "ALL processes that share a matrix MUST call MatAssemblyBegin() and MatAssemblyEnd() the SAME NUMBER of times". So I understand that petsc simply does not support unequal number of assembly calls. > > > My question evolves then. I have a problem at hand where I do not know how many calls each process will make to MatAssembly routines. Any suggestions to make this work? > > On 28-01-2018 22:09, Ali Berk Kahraman wrote: >> Hello All, >> >> >> The code takes a parallel matrix and calls a function using that matrix. That function fills the specified row of that matrix with the id of the process that part of the matrix belongs in. You can see the short code in the attachment, it is about 80 lines. >> >> >> The problem is that the code gets into a deadlock at some point, usually at the last row of each process except for the last process (greatest pid). I use petsc with configure options "--with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack". >> >> >> I am a beginner with MPI, so I do not know what may be causing this. My apologies in advance if this is a very trivial problem. >> >> >> Best Regards to All, >> >> >> Ali Berk Kahraman >> >> M.Sc. Student, Mechanical Eng. >> >> Bogazici Uni., Istanbul, Turkey >> > From aliberkkahraman at yahoo.com Sun Jan 28 14:16:33 2018 From: aliberkkahraman at yahoo.com (Ali Berk Kahraman) Date: Sun, 28 Jan 2018 23:16:33 +0300 Subject: [petsc-users] Parallel Matrix Causes a Deadlock In-Reply-To: References: <60ec2ddf-fe37-4946-1a00-afdde34ceb2c@yahoo.com> <5e66c7be-83a2-cac5-30e9-b7254f198d6c@yahoo.com> Message-ID: Dear Barry, I see what you are talking about, I have oversimplified my problem. The problem is that I will have to call MatGetValues on Wavelets matrix for every row in a loop. Thus, if I do not call assembly routine in the child function, it starts to give "not for unassembled matrix" error after the first loop. The relevant part of the code is appended on this e-mail. You can see the matgetvalues calls on line 81, and it is also in line 94's function GetWaveletPolyFit2DSingleRow_x1. Ali On 28-01-2018 22:24, Smith, Barry F. wrote: > In your code there is NO reason to call the MatAssemblyBegin/End where you do. Just pull it out and call it once. I submit this is the same for any other code. Please explain enough about your code (or send it) that has to call the assembly routines a different number of times. You just pull it above all the calls to MatSetValues(). > > Barry > > >> On Jan 28, 2018, at 1:15 PM, Ali Berk Kahraman wrote: >> >> Hello All, >> >> >> My apologies, I have closed this e-mail window and the first thing I read on the manual is "ALL processes that share a matrix MUST call MatAssemblyBegin() and MatAssemblyEnd() the SAME NUMBER of times". So I understand that petsc simply does not support unequal number of assembly calls. >> >> >> My question evolves then. I have a problem at hand where I do not know how many calls each process will make to MatAssembly routines. Any suggestions to make this work? >> >> On 28-01-2018 22:09, Ali Berk Kahraman wrote: >>> Hello All, >>> >>> >>> The code takes a parallel matrix and calls a function using that matrix. That function fills the specified row of that matrix with the id of the process that part of the matrix belongs in. You can see the short code in the attachment, it is about 80 lines. >>> >>> >>> The problem is that the code gets into a deadlock at some point, usually at the last row of each process except for the last process (greatest pid). I use petsc with configure options "--with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack". >>> >>> >>> I am a beginner with MPI, so I do not know what may be causing this. My apologies in advance if this is a very trivial problem. >>> >>> >>> Best Regards to All, >>> >>> >>> Ali Berk Kahraman >>> >>> M.Sc. Student, Mechanical Eng. >>> >>> Bogazici Uni., Istanbul, Turkey >>> -------------- next part -------------- A non-text attachment was scrubbed... Name: PetscRealProblem.c Type: text/x-csrc Size: 4580 bytes Desc: not available URL: From bsmith at mcs.anl.gov Sun Jan 28 17:16:05 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Sun, 28 Jan 2018 23:16:05 +0000 Subject: [petsc-users] Parallel Matrix Causes a Deadlock In-Reply-To: References: <60ec2ddf-fe37-4946-1a00-afdde34ceb2c@yahoo.com> <5e66c7be-83a2-cac5-30e9-b7254f198d6c@yahoo.com> Message-ID: <695142F0-03C8-444C-84CB-F338DD8CEC54@anl.gov> From your code it looks like you are accessing values in a single row, changing them and then putting them back in the same row at the same index locations? If this is the case you should use MatGetRow() to process the values, change the numerical entries in that row as needed and then call MatRestoreRow(). No need to call MatSetValues() and hence no business about unassembled matrices. Will this work for you? Barry > On Jan 28, 2018, at 2:16 PM, Ali Berk Kahraman wrote: > > Dear Barry, > > I see what you are talking about, I have oversimplified my problem. The problem is that I will have to call MatGetValues on Wavelets matrix for every row in a loop. Thus, if I do not call assembly routine in the child function, it starts to give "not for unassembled matrix" error after the first loop. The relevant part of the code is appended on this e-mail. You can see the matgetvalues calls on line 81, and it is also in line 94's function GetWaveletPolyFit2DSingleRow_x1. > > > Ali > > > On 28-01-2018 22:24, Smith, Barry F. wrote: >> In your code there is NO reason to call the MatAssemblyBegin/End where you do. Just pull it out and call it once. I submit this is the same for any other code. Please explain enough about your code (or send it) that has to call the assembly routines a different number of times. You just pull it above all the calls to MatSetValues(). >> >> Barry >> >> >>> On Jan 28, 2018, at 1:15 PM, Ali Berk Kahraman wrote: >>> >>> Hello All, >>> >>> >>> My apologies, I have closed this e-mail window and the first thing I read on the manual is "ALL processes that share a matrix MUST call MatAssemblyBegin() and MatAssemblyEnd() the SAME NUMBER of times". So I understand that petsc simply does not support unequal number of assembly calls. >>> >>> >>> My question evolves then. I have a problem at hand where I do not know how many calls each process will make to MatAssembly routines. Any suggestions to make this work? >>> >>> On 28-01-2018 22:09, Ali Berk Kahraman wrote: >>>> Hello All, >>>> >>>> >>>> The code takes a parallel matrix and calls a function using that matrix. That function fills the specified row of that matrix with the id of the process that part of the matrix belongs in. You can see the short code in the attachment, it is about 80 lines. >>>> >>>> >>>> The problem is that the code gets into a deadlock at some point, usually at the last row of each process except for the last process (greatest pid). I use petsc with configure options "--with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack". >>>> >>>> >>>> I am a beginner with MPI, so I do not know what may be causing this. My apologies in advance if this is a very trivial problem. >>>> >>>> >>>> Best Regards to All, >>>> >>>> >>>> Ali Berk Kahraman >>>> >>>> M.Sc. Student, Mechanical Eng. >>>> >>>> Bogazici Uni., Istanbul, Turkey >>>> > > From aliberkkahraman at yahoo.com Mon Jan 29 03:18:31 2018 From: aliberkkahraman at yahoo.com (Ali Berk Kahraman) Date: Mon, 29 Jan 2018 12:18:31 +0300 Subject: [petsc-users] Parallel Matrix Causes a Deadlock In-Reply-To: <695142F0-03C8-444C-84CB-F338DD8CEC54@anl.gov> References: <60ec2ddf-fe37-4946-1a00-afdde34ceb2c@yahoo.com> <5e66c7be-83a2-cac5-30e9-b7254f198d6c@yahoo.com> <695142F0-03C8-444C-84CB-F338DD8CEC54@anl.gov> Message-ID: <3e9ad274-9a20-c30c-d443-26188d13f3f6@yahoo.com> Yes, that is exactly what I am trying to do. In the documentation it says that MatGetRow cannot be used to alter the entries, it is only for examining them. So if I understand correctly, do you suggest: 1:MatGetRow, get the row 2:MatSetValues, set the values 3:Do not call assembly, instead call matRestoreRow, because the next MatGetRow will not need assembly (will it really not after matsetvalues on line 2?) (note: next MatGetRow will not need to reach these newly set values, it will work on a different row) 4:go to step 1 for the next row Also, I can spare memory for 1 additional matrix. Should I just generate a new identical matrix with same values, using either MatCopy or MatConvert , and use MatGetValues on the old matrix and MatSetValues on the new matrix, hence no need for assembly on the new matrix until everything is done? If you think this is possible, how expensive would this be? Which function do you suggest (Copy or Convert)? Thank you for your answers and attention so far Ali On 29-01-2018 02:16, Smith, Barry F. wrote: > From your code it looks like you are accessing values in a single row, changing them and then putting them back in the same row at the same index locations? > > If this is the case you should use MatGetRow() to process the values, change the numerical entries in that row as needed and then call MatRestoreRow(). No need to call MatSetValues() and hence no business about unassembled matrices. > > Will this work for you? > > Barry > > >> On Jan 28, 2018, at 2:16 PM, Ali Berk Kahraman wrote: >> >> Dear Barry, >> >> I see what you are talking about, I have oversimplified my problem. The problem is that I will have to call MatGetValues on Wavelets matrix for every row in a loop. Thus, if I do not call assembly routine in the child function, it starts to give "not for unassembled matrix" error after the first loop. The relevant part of the code is appended on this e-mail. You can see the matgetvalues calls on line 81, and it is also in line 94's function GetWaveletPolyFit2DSingleRow_x1. >> >> >> Ali >> >> >> On 28-01-2018 22:24, Smith, Barry F. wrote: >>> In your code there is NO reason to call the MatAssemblyBegin/End where you do. Just pull it out and call it once. I submit this is the same for any other code. Please explain enough about your code (or send it) that has to call the assembly routines a different number of times. You just pull it above all the calls to MatSetValues(). >>> >>> Barry >>> >>> >>>> On Jan 28, 2018, at 1:15 PM, Ali Berk Kahraman wrote: >>>> >>>> Hello All, >>>> >>>> >>>> My apologies, I have closed this e-mail window and the first thing I read on the manual is "ALL processes that share a matrix MUST call MatAssemblyBegin() and MatAssemblyEnd() the SAME NUMBER of times". So I understand that petsc simply does not support unequal number of assembly calls. >>>> >>>> >>>> My question evolves then. I have a problem at hand where I do not know how many calls each process will make to MatAssembly routines. Any suggestions to make this work? >>>> >>>> On 28-01-2018 22:09, Ali Berk Kahraman wrote: >>>>> Hello All, >>>>> >>>>> >>>>> The code takes a parallel matrix and calls a function using that matrix. That function fills the specified row of that matrix with the id of the process that part of the matrix belongs in. You can see the short code in the attachment, it is about 80 lines. >>>>> >>>>> >>>>> The problem is that the code gets into a deadlock at some point, usually at the last row of each process except for the last process (greatest pid). I use petsc with configure options "--with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack". >>>>> >>>>> >>>>> I am a beginner with MPI, so I do not know what may be causing this. My apologies in advance if this is a very trivial problem. >>>>> >>>>> >>>>> Best Regards to All, >>>>> >>>>> >>>>> Ali Berk Kahraman >>>>> >>>>> M.Sc. Student, Mechanical Eng. >>>>> >>>>> Bogazici Uni., Istanbul, Turkey >>>>> >> From knepley at gmail.com Mon Jan 29 04:55:32 2018 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 29 Jan 2018 21:55:32 +1100 Subject: [petsc-users] Parallel Matrix Causes a Deadlock In-Reply-To: <3e9ad274-9a20-c30c-d443-26188d13f3f6@yahoo.com> References: <60ec2ddf-fe37-4946-1a00-afdde34ceb2c@yahoo.com> <5e66c7be-83a2-cac5-30e9-b7254f198d6c@yahoo.com> <695142F0-03C8-444C-84CB-F338DD8CEC54@anl.gov> <3e9ad274-9a20-c30c-d443-26188d13f3f6@yahoo.com> Message-ID: On Mon, Jan 29, 2018 at 8:18 PM, Ali Berk Kahraman < aliberkkahraman at yahoo.com> wrote: > Yes, that is exactly what I am trying to do. > > In the documentation it says that MatGetRow cannot be used to alter the > entries, it is only for examining them. So if I understand correctly, do > you suggest: > You are right, so it is not quite as easy as Barry suggests. If you code up the algorithm below, you will get an error on the second iteration which says the matrix is not assembled. Look at the code for MatChop() here https://bitbucket.org/petsc/petsc/src/e49f4227a4aa635b95b6f78e8b19273a0cf7d2d5/src/mat/utils/axpy.c?at=master&fileviewer=file-view-default#axpy.c-385 It does what you are doing. We could perhaps make a generic version where you pass in a function to modify the row values. Thanks, Matt > 1:MatGetRow, get the row > > 2:MatSetValues, set the values > > 3:Do not call assembly, instead call matRestoreRow, because the next > MatGetRow will not need assembly (will it really not after matsetvalues on > line 2?) (note: next MatGetRow will not need to reach these newly set > values, it will work on a different row) > > 4:go to step 1 for the next row > > > Also, I can spare memory for 1 additional matrix. Should I just generate a > new identical matrix with same values, using either MatCopy or MatConvert , > and use MatGetValues on the old matrix and MatSetValues on the new matrix, > hence no need for assembly on the new matrix until everything is done? If > you think this is possible, how expensive would this be? Which function do > you suggest (Copy or Convert)? > > > Thank you for your answers and attention so far > > Ali > > > > On 29-01-2018 02:16, Smith, Barry F. wrote: > >> From your code it looks like you are accessing values in a single row, >> changing them and then putting them back in the same row at the same index >> locations? >> >> If this is the case you should use MatGetRow() to process the values, >> change the numerical entries in that row as needed and then call >> MatRestoreRow(). No need to call MatSetValues() and hence no business about >> unassembled matrices. >> >> Will this work for you? >> >> Barry >> >> >> On Jan 28, 2018, at 2:16 PM, Ali Berk Kahraman >>> wrote: >>> >>> Dear Barry, >>> >>> I see what you are talking about, I have oversimplified my problem. The >>> problem is that I will have to call MatGetValues on Wavelets matrix for >>> every row in a loop. Thus, if I do not call assembly routine in the child >>> function, it starts to give "not for unassembled matrix" error after the >>> first loop. The relevant part of the code is appended on this e-mail. You >>> can see the matgetvalues calls on line 81, and it is also in line 94's >>> function GetWaveletPolyFit2DSingleRow_x1. >>> >>> >>> Ali >>> >>> >>> On 28-01-2018 22:24, Smith, Barry F. wrote: >>> >>>> In your code there is NO reason to call the MatAssemblyBegin/End >>>> where you do. Just pull it out and call it once. I submit this is the same >>>> for any other code. Please explain enough about your code (or send it) that >>>> has to call the assembly routines a different number of times. You just >>>> pull it above all the calls to MatSetValues(). >>>> >>>> Barry >>>> >>>> >>>> On Jan 28, 2018, at 1:15 PM, Ali Berk Kahraman < >>>>> aliberkkahraman at yahoo.com> wrote: >>>>> >>>>> Hello All, >>>>> >>>>> >>>>> My apologies, I have closed this e-mail window and the first thing I >>>>> read on the manual is "ALL processes that share a matrix MUST call >>>>> MatAssemblyBegin() and MatAssemblyEnd() the SAME NUMBER of times". So I >>>>> understand that petsc simply does not support unequal number of assembly >>>>> calls. >>>>> >>>>> >>>>> My question evolves then. I have a problem at hand where I do not know >>>>> how many calls each process will make to MatAssembly routines. Any >>>>> suggestions to make this work? >>>>> >>>>> On 28-01-2018 22:09, Ali Berk Kahraman wrote: >>>>> >>>>>> Hello All, >>>>>> >>>>>> >>>>>> The code takes a parallel matrix and calls a function using that >>>>>> matrix. That function fills the specified row of that matrix with the id of >>>>>> the process that part of the matrix belongs in. You can see the short code >>>>>> in the attachment, it is about 80 lines. >>>>>> >>>>>> >>>>>> The problem is that the code gets into a deadlock at some point, >>>>>> usually at the last row of each process except for the last process >>>>>> (greatest pid). I use petsc with configure options "--with-cc=gcc >>>>>> --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack". >>>>>> >>>>>> >>>>>> I am a beginner with MPI, so I do not know what may be causing this. >>>>>> My apologies in advance if this is a very trivial problem. >>>>>> >>>>>> >>>>>> Best Regards to All, >>>>>> >>>>>> >>>>>> Ali Berk Kahraman >>>>>> >>>>>> M.Sc. Student, Mechanical Eng. >>>>>> >>>>>> Bogazici Uni., Istanbul, Turkey >>>>>> >>>>>> >>> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Jan 29 08:56:00 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Mon, 29 Jan 2018 14:56:00 +0000 Subject: [petsc-users] Parallel Matrix Causes a Deadlock In-Reply-To: <3e9ad274-9a20-c30c-d443-26188d13f3f6@yahoo.com> References: <60ec2ddf-fe37-4946-1a00-afdde34ceb2c@yahoo.com> <5e66c7be-83a2-cac5-30e9-b7254f198d6c@yahoo.com> <695142F0-03C8-444C-84CB-F338DD8CEC54@anl.gov> <3e9ad274-9a20-c30c-d443-26188d13f3f6@yahoo.com> Message-ID: <384C147F-0026-4D57-B046-219F17FC14A7@anl.gov> > On Jan 29, 2018, at 3:18 AM, Ali Berk Kahraman wrote: > > Yes, that is exactly what I am trying to do. > > In the documentation it says that MatGetRow cannot be used to alter the entries, it is only for examining them. So if I understand correctly, do you suggest: > > 1:MatGetRow, get the row > > 2:MatSetValues, set the values > > 3:Do not call assembly, instead call matRestoreRow, because the next MatGetRow will not need assembly (will it really not after matsetvalues on line 2?) (note: next MatGetRow will not need to reach these newly set values, it will work on a different row) > > 4:go to step 1 for the next row > > > Also, I can spare memory for 1 additional matrix. Should I just generate a new identical matrix with same values, using either MatCopy or MatConvert , and use MatGetValues on the old matrix and MatSetValues on the new matrix, hence no need for assembly on the new matrix until everything is done? If you think this is possible, how expensive would this be? Which function do you suggest (Copy or Convert)? You can do this with MatCopy, the only significant cost is in memory. > > > Thank you for your answers and attention so far > > Ali > > > > On 29-01-2018 02:16, Smith, Barry F. wrote: >> From your code it looks like you are accessing values in a single row, changing them and then putting them back in the same row at the same index locations? >> >> If this is the case you should use MatGetRow() to process the values, change the numerical entries in that row as needed and then call MatRestoreRow(). No need to call MatSetValues() and hence no business about unassembled matrices. >> >> Will this work for you? >> >> Barry >> >> >>> On Jan 28, 2018, at 2:16 PM, Ali Berk Kahraman wrote: >>> >>> Dear Barry, >>> >>> I see what you are talking about, I have oversimplified my problem. The problem is that I will have to call MatGetValues on Wavelets matrix for every row in a loop. Thus, if I do not call assembly routine in the child function, it starts to give "not for unassembled matrix" error after the first loop. The relevant part of the code is appended on this e-mail. You can see the matgetvalues calls on line 81, and it is also in line 94's function GetWaveletPolyFit2DSingleRow_x1. >>> >>> >>> Ali >>> >>> >>> On 28-01-2018 22:24, Smith, Barry F. wrote: >>>> In your code there is NO reason to call the MatAssemblyBegin/End where you do. Just pull it out and call it once. I submit this is the same for any other code. Please explain enough about your code (or send it) that has to call the assembly routines a different number of times. You just pull it above all the calls to MatSetValues(). >>>> >>>> Barry >>>> >>>> >>>>> On Jan 28, 2018, at 1:15 PM, Ali Berk Kahraman wrote: >>>>> >>>>> Hello All, >>>>> >>>>> >>>>> My apologies, I have closed this e-mail window and the first thing I read on the manual is "ALL processes that share a matrix MUST call MatAssemblyBegin() and MatAssemblyEnd() the SAME NUMBER of times". So I understand that petsc simply does not support unequal number of assembly calls. >>>>> >>>>> >>>>> My question evolves then. I have a problem at hand where I do not know how many calls each process will make to MatAssembly routines. Any suggestions to make this work? >>>>> >>>>> On 28-01-2018 22:09, Ali Berk Kahraman wrote: >>>>>> Hello All, >>>>>> >>>>>> >>>>>> The code takes a parallel matrix and calls a function using that matrix. That function fills the specified row of that matrix with the id of the process that part of the matrix belongs in. You can see the short code in the attachment, it is about 80 lines. >>>>>> >>>>>> >>>>>> The problem is that the code gets into a deadlock at some point, usually at the last row of each process except for the last process (greatest pid). I use petsc with configure options "--with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack". >>>>>> >>>>>> >>>>>> I am a beginner with MPI, so I do not know what may be causing this. My apologies in advance if this is a very trivial problem. >>>>>> >>>>>> >>>>>> Best Regards to All, >>>>>> >>>>>> >>>>>> Ali Berk Kahraman >>>>>> >>>>>> M.Sc. Student, Mechanical Eng. >>>>>> >>>>>> Bogazici Uni., Istanbul, Turkey >>>>>> >>> > From mbuerkle at web.de Mon Jan 29 19:34:40 2018 From: mbuerkle at web.de (Marius Buerkle) Date: Tue, 30 Jan 2018 02:34:40 +0100 Subject: [petsc-users] combine two matrices with different non-zero structure Message-ID: An HTML attachment was scrubbed... URL: From mail2amneet at gmail.com Mon Jan 29 22:05:00 2018 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Mon, 29 Jan 2018 20:05:00 -0800 Subject: [petsc-users] Error during PETSc build on Edison Message-ID: Hi, I am trying to build PETSc (v3.7.7) locally in my home directory on NERSC (Edison). However, I am getting the following error during the configuration stage. =============================================================================== TESTING: checkFortranNameMangling from config.compilers(config/BuildSystem/config/compilers.py:653) ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Unknown Fortran name mangling: Are you sure the C and Fortran compilers are compatible? Perhaps one is 64 bit and one is 32 bit? ------------------------------------------------------------------------------- Unknown Fortran name mangling: Are you sure the C and Fortran compilers are compatible? Perhaps one is 64 bit and one is 32 bit? ******************************************************************************* File "./config/configure.py", line 405, in petsc_configure framework.configure(out = sys.stdout) File "/global/u2/a/apbhalla/softwares/PETSc-BitBucket/PETSc/config/BuildSystem/config/framework.py", line 1091, in configure self.processChildren() File "/global/u2/a/apbhalla/softwares/PETSc-BitBucket/PETSc/config/BuildSystem/config/framework.py", line 1080, in processChildren self.serialEvaluation(self.childGraph) File "/global/u2/a/apbhalla/softwares/PETSc-BitBucket/PETSc/config/BuildSystem/config/framework.py", line 1061, in serialEvaluation child.configure() File "/global/u2/a/apbhalla/softwares/PETSc-BitBucket/PETSc/config/BuildSystem/config/compilers.py", line 1458, in configure self.executeTest(self.checkFortranNameMangling) File "/global/u2/a/apbhalla/softwares/PETSc-BitBucket/PETSc/config/BuildSystem/config/base.py", line 126, in executeTest ret = test(*args,**kargs) File "/global/u2/a/apbhalla/softwares/PETSc-BitBucket/PETSc/config/BuildSystem/config/compilers.py", line 675, in checkFortranNameMangling raise RuntimeError('Unknown Fortran name mangling: Are you sure the C and Fortran compilers are compatible?\n Perhaps one is 64 bit and one is 32 bit?\n'+mess) I using the following configure options ./config/configure.py --CC=cc --CXX=CC --FC=ftn --with-default-arch=0 --PETSC_ARCH=linux-opt --with-debugging=0 --with-c++-support=1 --with-hypre=1 --download-hypre=1 --COPTFLAGS="-O3" --CXXOPTFLAGS="-O3" --FOPTFLAGS="-O3" --with-x=0 I have swapped the Prgn-Env-intel with Prgn-Env-gnu. The compiler commands return the following: % cc --version gcc (GCC) 7.1.0 20170502 (Cray Inc.) Copyright (C) 2017 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. % which cc /opt/cray/pe/craype/2.5.12/bin/cc % CC --version g++ (GCC) 7.1.0 20170502 (Cray Inc.) Copyright (C) 2017 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. % which CC /opt/cray/pe/craype/2.5.12/bin/CC % ftn --version GNU Fortran (GCC) 7.1.0 20170502 (Cray Inc.) Copyright (C) 2017 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. % which ftn /opt/cray/pe/craype/2.5.12/bin/ftn Any suggestions? Thanks, -- --Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Jan 29 22:09:35 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 30 Jan 2018 04:09:35 +0000 Subject: [petsc-users] Error during PETSc build on Edison In-Reply-To: References: Message-ID: We always need configure.log (it has all the information about what went wrong). --with-c++-support shouldn't exist as an option. Barry > On Jan 29, 2018, at 10:05 PM, Amneet Bhalla wrote: > > Hi, > > I am trying to build PETSc (v3.7.7) locally in my home directory on NERSC (Edison). > However, I am getting the following error during the configuration stage. > > =============================================================================== TESTING: checkFortranNameMangling from config.compilers(config/BuildSystem/config/compilers.py:653) ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > ------------------------------------------------------------------------------- > Unknown Fortran name mangling: Are you sure the C and Fortran compilers are compatible? > Perhaps one is 64 bit and one is 32 bit? > ------------------------------------------------------------------------------- > Unknown Fortran name mangling: Are you sure the C and Fortran compilers are compatible? > Perhaps one is 64 bit and one is 32 bit? > ******************************************************************************* > File "./config/configure.py", line 405, in petsc_configure > framework.configure(out = sys.stdout) > File "/global/u2/a/apbhalla/softwares/PETSc-BitBucket/PETSc/config/BuildSystem/config/framework.py", line 1091, in configure > self.processChildren() > File "/global/u2/a/apbhalla/softwares/PETSc-BitBucket/PETSc/config/BuildSystem/config/framework.py", line 1080, in processChildren > self.serialEvaluation(self.childGraph) > File "/global/u2/a/apbhalla/softwares/PETSc-BitBucket/PETSc/config/BuildSystem/config/framework.py", line 1061, in serialEvaluation > child.configure() > File "/global/u2/a/apbhalla/softwares/PETSc-BitBucket/PETSc/config/BuildSystem/config/compilers.py", line 1458, in configure > self.executeTest(self.checkFortranNameMangling) > File "/global/u2/a/apbhalla/softwares/PETSc-BitBucket/PETSc/config/BuildSystem/config/base.py", line 126, in executeTest > ret = test(*args,**kargs) > File "/global/u2/a/apbhalla/softwares/PETSc-BitBucket/PETSc/config/BuildSystem/config/compilers.py", line 675, in checkFortranNameMangling > raise RuntimeError('Unknown Fortran name mangling: Are you sure the C and Fortran compilers are compatible?\n Perhaps one is 64 bit and one is 32 bit?\n'+mess) > > > > > > > I using the following configure options > > ./config/configure.py --CC=cc --CXX=CC --FC=ftn --with-default-arch=0 --PETSC_ARCH=linux-opt --with-debugging=0 --with-c++-support=1 --with-hypre=1 --download-hypre=1 --COPTFLAGS="-O3" --CXXOPTFLAGS="-O3" --FOPTFLAGS="-O3" --with-x=0 > > > I have swapped the Prgn-Env-intel with Prgn-Env-gnu. > > The compiler commands return the following: > > % cc --version > gcc (GCC) 7.1.0 20170502 (Cray Inc.) > Copyright (C) 2017 Free Software Foundation, Inc. > This is free software; see the source for copying conditions. There is NO > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. > > % which cc > /opt/cray/pe/craype/2.5.12/bin/cc > > > % CC --version > g++ (GCC) 7.1.0 20170502 (Cray Inc.) > Copyright (C) 2017 Free Software Foundation, Inc. > This is free software; see the source for copying conditions. There is NO > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. > > % which CC > /opt/cray/pe/craype/2.5.12/bin/CC > > > > > % ftn --version > GNU Fortran (GCC) 7.1.0 20170502 (Cray Inc.) > Copyright (C) 2017 Free Software Foundation, Inc. > This is free software; see the source for copying conditions. There is NO > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. > > % which ftn > /opt/cray/pe/craype/2.5.12/bin/ftn > > > > > Any suggestions? > > Thanks, > > > -- > --Amneet > > > From balay at mcs.anl.gov Mon Jan 29 22:12:04 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 29 Jan 2018 22:12:04 -0600 Subject: [petsc-users] Error during PETSc build on Edison In-Reply-To: References: Message-ID: 1. cray provides prebuilt petsc libraries. Does that work for you? 2. verylikely you need: --with-clib-autodetect=0 --with-cxxlib-autodetect=0 LIBS=-lstdc++ If it doesn't work - send configure.log Satish On Tue, 30 Jan 2018, Amneet Bhalla wrote: > Hi, > > I am trying to build PETSc (v3.7.7) locally in my home directory on NERSC > (Edison). > However, I am getting the following error during the configuration stage. > > =============================================================================== > > TESTING: checkFortranNameMangling from > config.compilers(config/BuildSystem/config/compilers.py:653) > > ******************************************************************************* > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > details): > > ------------------------------------------------------------------------------- > > Unknown Fortran name mangling: Are you sure the C and Fortran compilers are > compatible? > > Perhaps one is 64 bit and one is 32 bit? > > ------------------------------------------------------------------------------- > > Unknown Fortran name mangling: Are you sure the C and Fortran compilers are > compatible? > > Perhaps one is 64 bit and one is 32 bit? > > ******************************************************************************* > > File "./config/configure.py", line 405, in petsc_configure > > framework.configure(out = sys.stdout) > > File > "/global/u2/a/apbhalla/softwares/PETSc-BitBucket/PETSc/config/BuildSystem/config/framework.py", > line 1091, in configure > > self.processChildren() > > File > "/global/u2/a/apbhalla/softwares/PETSc-BitBucket/PETSc/config/BuildSystem/config/framework.py", > line 1080, in processChildren > > self.serialEvaluation(self.childGraph) > > File > "/global/u2/a/apbhalla/softwares/PETSc-BitBucket/PETSc/config/BuildSystem/config/framework.py", > line 1061, in serialEvaluation > > child.configure() > > File > "/global/u2/a/apbhalla/softwares/PETSc-BitBucket/PETSc/config/BuildSystem/config/compilers.py", > line 1458, in configure > > self.executeTest(self.checkFortranNameMangling) > > File > "/global/u2/a/apbhalla/softwares/PETSc-BitBucket/PETSc/config/BuildSystem/config/base.py", > line 126, in executeTest > > ret = test(*args,**kargs) > > File > "/global/u2/a/apbhalla/softwares/PETSc-BitBucket/PETSc/config/BuildSystem/config/compilers.py", > line 675, in checkFortranNameMangling > > raise RuntimeError('Unknown Fortran name mangling: Are you sure the C > and Fortran compilers are compatible?\n Perhaps one is 64 bit and one is > 32 bit?\n'+mess) > > > > > > I using the following configure options > > ./config/configure.py --CC=cc --CXX=CC --FC=ftn --with-default-arch=0 > --PETSC_ARCH=linux-opt --with-debugging=0 --with-c++-support=1 > --with-hypre=1 --download-hypre=1 --COPTFLAGS="-O3" --CXXOPTFLAGS="-O3" > --FOPTFLAGS="-O3" --with-x=0 > > > I have swapped the Prgn-Env-intel with Prgn-Env-gnu. > > The compiler commands return the following: > > % cc --version > > gcc (GCC) 7.1.0 20170502 (Cray Inc.) > > Copyright (C) 2017 Free Software Foundation, Inc. > > This is free software; see the source for copying conditions. There is NO > > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. > > > % which cc > > /opt/cray/pe/craype/2.5.12/bin/cc > > > % CC --version > > g++ (GCC) 7.1.0 20170502 (Cray Inc.) > > Copyright (C) 2017 Free Software Foundation, Inc. > > This is free software; see the source for copying conditions. There is NO > > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. > > > % which CC > > /opt/cray/pe/craype/2.5.12/bin/CC > > > > % ftn --version > > GNU Fortran (GCC) 7.1.0 20170502 (Cray Inc.) > > Copyright (C) 2017 Free Software Foundation, Inc. > > This is free software; see the source for copying conditions. There is NO > > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. > > > % which ftn > > /opt/cray/pe/craype/2.5.12/bin/ftn > > > > Any suggestions? > > Thanks, > > > From bsmith at mcs.anl.gov Mon Jan 29 22:12:44 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 30 Jan 2018 04:12:44 +0000 Subject: [petsc-users] combine two matrices with different non-zero structure In-Reply-To: References: Message-ID: Marius, We don't provide a way to insert a "generic" sparse matrix into a bigger matrices (dense matrices coming from element stiffness matrices yes) so I don't see any simple solution. Does the submatrix you pull out have any particular structure, what does it represent? Barry > On Jan 29, 2018, at 7:34 PM, Marius Buerkle wrote: > > Hi ! > > I have the followng problem. I create a Submatrix containing a subset of row/columns of the original matrix. After some matrix multiplications the non-zero strucutre of the resulting matrix changed. Now I want to insert this submatix back into the original one keeping only the non-zero entries which are present in the original matrix discarding the others which accumulated due to the matrix multiplications. Is there an easy or also not so easy way to do this ? > > Best, > Marius From balay at mcs.anl.gov Mon Jan 29 22:14:20 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 29 Jan 2018 22:14:20 -0600 Subject: [petsc-users] Error during PETSc build on Edison In-Reply-To: References: Message-ID: On Tue, 30 Jan 2018, Satish Balay wrote: > 1. cray provides prebuilt petsc libraries. Does that work for you? > > 2. verylikely you need: > > --with-clib-autodetect=0 > --with-cxxlib-autodetect=0 Also --with-fortranlib-autodetect=0 > LIBS=-lstdc++ > > If it doesn't work - send configure.log BTW: What compilers do you require? Intel or GNU? Satish From mail2amneet at gmail.com Mon Jan 29 22:19:41 2018 From: mail2amneet at gmail.com (Amneet Bhalla) Date: Mon, 29 Jan 2018 20:19:41 -0800 Subject: [petsc-users] Error during PETSc build on Edison In-Reply-To: References: Message-ID: Hi Satish --- I am trying GNU. I am configuring with these additional flags, and it seems to be past that point. I will let you know if it works fine all the way. The reason I am not using prebuilt PETSc on NERSC is that I want to be consistent with C++, C, and fortran compilers with all the 3rd party libraries I am building for a code (that requires them). On Mon, Jan 29, 2018 at 8:14 PM, Satish Balay wrote: > On Tue, 30 Jan 2018, Satish Balay wrote: > > > 1. cray provides prebuilt petsc libraries. Does that work for you? > > > > 2. verylikely you need: > > > > --with-clib-autodetect=0 > > --with-cxxlib-autodetect=0 > > Also > > --with-fortranlib-autodetect=0 > > > LIBS=-lstdc++ > > > > If it doesn't work - send configure.log > > BTW: What compilers do you require? Intel or GNU? > > > Satish > > -- --Amneet -------------- next part -------------- An HTML attachment was scrubbed... URL: From jpi at informatik.uni-kiel.de Tue Jan 30 08:07:34 2018 From: jpi at informatik.uni-kiel.de (Jaroslaw Piwonski (CAU)) Date: Tue, 30 Jan 2018 15:07:34 +0100 Subject: [petsc-users] petsc4py, cython In-Reply-To: References: <781ADC9E-82A2-4FEB-8D03-15F733D89354@informatik.uni-kiel.de> <183EC67F-4754-4146-A10C-841498BC9ABA@informatik.uni-kiel.de> Message-ID: <4C37A7D6-94BC-4DCA-8D5D-4D554EDE3397@informatik.uni-kiel.de> Dear Lisandro, > Am 18.01.2018 um 17:13 schrieb Lisandro Dalcin : > > In Cython code using C types from petsc4py, you should do: > > from petsc4py.PETSc cimport Vec, PetscVec > from petsc4py.PETSc cimport Mat, PetscMat > # etcetera > > now you can "cdef" either "PetscVec" (the low-level PETSc C type) and > "Vec" (the higher-level petsc4py Python type) > > Or you can just > > from petsc4py.PETSc cimport * > > to get all definitions, or even > > from petsc4py cimport PETSc > > then use properly-namespaced definitions "PETSc.PetscVec" (C type) and > "PETSc.Vec" (Python type). thank you! That was the missing link. Great! Best regards, Jaroslaw > > > -- > Lisandro Dalcin > ============ > Research Scientist > Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) > Extreme Computing Research Center (ECRC) > King Abdullah University of Science and Technology (KAUST) > http://ecrc.kaust.edu.sa/ > > 4700 King Abdullah University of Science and Technology > al-Khawarizmi Bldg (Bldg 1), Office # 0109 > Thuwal 23955-6900, Kingdom of Saudi Arabia > http://www.kaust.edu.sa > > Office Phone: +966 12 808-0459 ? Dr. Jaroslaw Piwonski Christian-Albrechts-Universit?t zu Kiel Institut f?r Informatik Algorithmische Optimale Steuerung ? CO2-Aufnahme des Meeres Christian-Albrechts-Platz 4 24118 Kiel Telefon: +49 431 880-7452 Telefax: +49 431 880-7618 -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothee.nicolas at gmail.com Tue Jan 30 10:57:30 2018 From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=) Date: Tue, 30 Jan 2018 17:57:30 +0100 Subject: [petsc-users] Petsc messing with OpenMP Message-ID: Dear petsc team, For a while, I have been wondering why I have never managed to print what threads are doing in an openMP region in my FORTRAN programs. Some people told me it was normal because threads will get confused all trying to write at the same time. However I realised today that the problem seems to be related to petsc. I have a super simple "hello world" example that reproduces the problem : program hello !$ use omp_lib implicit none integer nthreads, tid !$omp parallel private(nthreads, tid) tid = omp_get_thread_num() print *, 'hello world from thread = ', tid if (tid .eq. 0) then nthreads = omp_get_num_threads() print *, 'number of threads = ', nthreads end if !$omp end parallel end program hello If I compile it with mpif90 -qopenmp -o omp main.f90 Then I get no problem. But if I link petsc library (as is the case in my code) : mpif90 -qopenmp -o omp main.f90 -L/home/timotheenicolas/petsc-3.7.3/arch-linux2-c-debug/lib -lpetsc Then I get the following error after execution of export OMP_NUM_THREADS=2;./omp hello world from thread = 0 number of threads = 2 forrtl: severe (40): recursive I/O operation, unit -1, file unknown Image PC Routine Line Source omp 0000000000403BC8 Unknown Unknown Unknown omp 0000000000403572 Unknown Unknown Unknown libiomp5.so 00002AAAAD3146A3 Unknown Unknown Unknown libiomp5.so 00002AAAAD2E3007 Unknown Unknown Unknown libiomp5.so 00002AAAAD2E26F5 Unknown Unknown Unknown libiomp5.so 00002AAAAD3149C3 Unknown Unknown Unknown libpthread.so.0 00002AAAAD5BADC5 Unknown Unknown Unknown libc.so.6 00002AAAAD8C5CED Unknown Unknown Unknown This means that in my programs using petsc, I can't use prints to see exactly what the threads are doing, which is a pain when debug is required (that is to say, all the time). Is this issue expected ? Thx in advance Timothee NICOLAS -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jan 30 11:02:08 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Tue, 30 Jan 2018 17:02:08 +0000 Subject: [petsc-users] Petsc messing with OpenMP In-Reply-To: References: Message-ID: <613E404A-30F1-43EA-85AE-E1198C09910A@anl.gov> I don't know what you are trying to do with OpenMP and PETSc, nor do I understand why anyone would use OpenMP, but you cannot call virtually any PETSc function or operation while you are using threads. Best to use PETSc as intended, with one MPI process per core or hardware thread and not use OpenMP at all. http://www.mcs.anl.gov/petsc/miscellaneous/petscthreads.html Barry > On Jan 30, 2018, at 10:57 AM, Timoth?e Nicolas wrote: > > Dear petsc team, > > For a while, I have been wondering why I have never managed to print what threads are doing in an openMP region in my FORTRAN programs. Some people told me it was normal because threads will get confused all trying to write at the same time. However I realised today that the problem seems to be related to petsc. I have a super simple "hello world" example that reproduces the problem : > > program hello > !$ use omp_lib > implicit none > integer nthreads, tid > > !$omp parallel private(nthreads, tid) > tid = omp_get_thread_num() > print *, 'hello world from thread = ', tid > > if (tid .eq. 0) then > nthreads = omp_get_num_threads() > print *, 'number of threads = ', nthreads > end if > !$omp end parallel > > end program hello > > If I compile it with > > mpif90 -qopenmp -o omp main.f90 > > Then I get no problem. But if I link petsc library (as is the case in my code) : > > mpif90 -qopenmp -o omp main.f90 -L/home/timotheenicolas/petsc-3.7.3/arch-linux2-c-debug/lib -lpetsc > > > Then I get the following error after execution of export OMP_NUM_THREADS=2;./omp > > hello world from thread = 0 > number of threads = 2 > forrtl: severe (40): recursive I/O operation, unit -1, file unknown > Image PC Routine Line Source > omp 0000000000403BC8 Unknown Unknown Unknown > omp 0000000000403572 Unknown Unknown Unknown > libiomp5.so 00002AAAAD3146A3 Unknown Unknown Unknown > libiomp5.so 00002AAAAD2E3007 Unknown Unknown Unknown > libiomp5.so 00002AAAAD2E26F5 Unknown Unknown Unknown > libiomp5.so 00002AAAAD3149C3 Unknown Unknown Unknown > libpthread.so.0 00002AAAAD5BADC5 Unknown Unknown Unknown > libc.so.6 00002AAAAD8C5CED Unknown Unknown Unknown > > > This means that in my programs using petsc, I can't use prints to see exactly what the threads are doing, which is a pain when debug is required (that is to say, all the time). Is this issue expected ? > > Thx in advance > > Timothee NICOLAS > From timothee.nicolas at gmail.com Tue Jan 30 11:36:33 2018 From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=) Date: Tue, 30 Jan 2018 18:36:33 +0100 Subject: [petsc-users] Petsc messing with OpenMP In-Reply-To: <613E404A-30F1-43EA-85AE-E1198C09910A@anl.gov> References: <613E404A-30F1-43EA-85AE-E1198C09910A@anl.gov> Message-ID: Thank you for your answer, I am not responsible for the choices in this code to begin with, but in any case, as you can see in the example, I am not using any petsc call. Also in our code, even though openmp is used, this is never in interaction with petsc, the latter being used only at a specific (but crucial) place in the code, and without openMP. In that sense, the model we are using is NOT some 'Petsc with threads model', as described in the page you sent me. Nevertheless, it appears that just linking petsc is enough to perturb the behavior of openMP. I believe it may require further investigation. On top of the bug I report in my first message, I have noticed very weird openMP behavior lately, the kind of bugs that 'don't make sense' at first sight (would be too long/boring to give details at this stage). I am starting to realize that it may have something to do with petsc. Again, the webpage you have sent me seems to adopt the point of view that hybrid MPI/openMP is useless from the performance point of view, and/or that Petsc routines should not be threaded. It does not, however, explain why the behaviour of openMP may be changed by merely linking a program with petsc, without a single petsc call in the program. Best 2018-01-30 18:02 GMT+01:00 Smith, Barry F. : > > I don't know what you are trying to do with OpenMP and PETSc, nor do I > understand why anyone would use OpenMP, but you cannot call virtually any > PETSc function or operation while you are using threads. > > Best to use PETSc as intended, with one MPI process per core or hardware > thread and not use OpenMP at all. http://www.mcs.anl.gov/petsc/ > miscellaneous/petscthreads.html > > > Barry > > > > On Jan 30, 2018, at 10:57 AM, Timoth?e Nicolas < > timothee.nicolas at gmail.com> wrote: > > > > Dear petsc team, > > > > For a while, I have been wondering why I have never managed to print > what threads are doing in an openMP region in my FORTRAN programs. Some > people told me it was normal because threads will get confused all trying > to write at the same time. However I realised today that the problem seems > to be related to petsc. I have a super simple "hello world" example that > reproduces the problem : > > > > program hello > > !$ use omp_lib > > implicit none > > integer nthreads, tid > > > > !$omp parallel private(nthreads, tid) > > tid = omp_get_thread_num() > > print *, 'hello world from thread = ', tid > > > > if (tid .eq. 0) then > > nthreads = omp_get_num_threads() > > print *, 'number of threads = ', nthreads > > end if > > !$omp end parallel > > > > end program hello > > > > If I compile it with > > > > mpif90 -qopenmp -o omp main.f90 > > > > Then I get no problem. But if I link petsc library (as is the case in my > code) : > > > > mpif90 -qopenmp -o omp main.f90 -L/home/timotheenicolas/petsc-3.7.3/arch-linux2-c-debug/lib > -lpetsc > > > > > > Then I get the following error after execution of export > OMP_NUM_THREADS=2;./omp > > > > hello world from thread = 0 > > number of threads = 2 > > forrtl: severe (40): recursive I/O operation, unit -1, file unknown > > Image PC Routine Line > Source > > omp 0000000000403BC8 Unknown Unknown > Unknown > > omp 0000000000403572 Unknown Unknown > Unknown > > libiomp5.so 00002AAAAD3146A3 Unknown Unknown > Unknown > > libiomp5.so 00002AAAAD2E3007 Unknown Unknown > Unknown > > libiomp5.so 00002AAAAD2E26F5 Unknown Unknown > Unknown > > libiomp5.so 00002AAAAD3149C3 Unknown Unknown > Unknown > > libpthread.so.0 00002AAAAD5BADC5 Unknown Unknown > Unknown > > libc.so.6 00002AAAAD8C5CED Unknown Unknown > Unknown > > > > > > This means that in my programs using petsc, I can't use prints to see > exactly what the threads are doing, which is a pain when debug is required > (that is to say, all the time). Is this issue expected ? > > > > Thx in advance > > > > Timothee NICOLAS > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Jan 30 11:46:08 2018 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 30 Jan 2018 11:46:08 -0600 Subject: [petsc-users] Petsc messing with OpenMP In-Reply-To: References: <613E404A-30F1-43EA-85AE-E1198C09910A@anl.gov> Message-ID: Try building PETSc with -qopenmp flag and see if this makes a difference. i.e use CFLAGS=-qopenmp FFLAGS=-qopenmp CXXFLAGS=-qopenmp Potential explanation: Compilers do different things internally [for ex - link to different libraries] when different compile flags are used. When -qopenmp is used - its possible some openmp library gets linked in. And when -qopenmp is not used - some equivalnet stub library might get linked in. Only one lib is supporsed to be used at anytime. With the way petsc gets built - both might get used - causing some conflict. You can verify with: mpicc -v -qopenmp dummy.c mpicc -v dummy.c And 'ldd libpetsc.so' Satish On Tue, 30 Jan 2018, Timoth?e Nicolas wrote: > Thank you for your answer, > > I am not responsible for the choices in this code to begin with, but in any > case, as you can see in the example, I am not using any petsc call. Also in > our code, even though openmp is used, this is never in interaction with > petsc, the latter being used only at a specific (but crucial) place in the > code, and without openMP. In that sense, the model we are using is NOT some > 'Petsc with threads model', as described in the page you sent me. > Nevertheless, it appears that just linking petsc is enough to perturb the > behavior of openMP. I believe it may require further investigation. On top > of the bug I report in my first message, I have noticed very weird openMP > behavior lately, the kind of bugs that 'don't make sense' at first sight > (would be too long/boring to give details at this stage). I am starting to > realize that it may have something to do with petsc. > > Again, the webpage you have sent me seems to adopt the point of view that > hybrid MPI/openMP is useless from the performance point of view, and/or > that Petsc routines should not be threaded. It does not, however, explain > why the behaviour of openMP may be changed by merely linking a program with > petsc, without a single petsc call in the program. > > Best > > 2018-01-30 18:02 GMT+01:00 Smith, Barry F. : > > > > > I don't know what you are trying to do with OpenMP and PETSc, nor do I > > understand why anyone would use OpenMP, but you cannot call virtually any > > PETSc function or operation while you are using threads. > > > > Best to use PETSc as intended, with one MPI process per core or hardware > > thread and not use OpenMP at all. http://www.mcs.anl.gov/petsc/ > > miscellaneous/petscthreads.html > > > > > > Barry > > > > > > > On Jan 30, 2018, at 10:57 AM, Timoth?e Nicolas < > > timothee.nicolas at gmail.com> wrote: > > > > > > Dear petsc team, > > > > > > For a while, I have been wondering why I have never managed to print > > what threads are doing in an openMP region in my FORTRAN programs. Some > > people told me it was normal because threads will get confused all trying > > to write at the same time. However I realised today that the problem seems > > to be related to petsc. I have a super simple "hello world" example that > > reproduces the problem : > > > > > > program hello > > > !$ use omp_lib > > > implicit none > > > integer nthreads, tid > > > > > > !$omp parallel private(nthreads, tid) > > > tid = omp_get_thread_num() > > > print *, 'hello world from thread = ', tid > > > > > > if (tid .eq. 0) then > > > nthreads = omp_get_num_threads() > > > print *, 'number of threads = ', nthreads > > > end if > > > !$omp end parallel > > > > > > end program hello > > > > > > If I compile it with > > > > > > mpif90 -qopenmp -o omp main.f90 > > > > > > Then I get no problem. But if I link petsc library (as is the case in my > > code) : > > > > > > mpif90 -qopenmp -o omp main.f90 -L/home/timotheenicolas/petsc-3.7.3/arch-linux2-c-debug/lib > > -lpetsc > > > > > > > > > Then I get the following error after execution of export > > OMP_NUM_THREADS=2;./omp > > > > > > hello world from thread = 0 > > > number of threads = 2 > > > forrtl: severe (40): recursive I/O operation, unit -1, file unknown > > > Image PC Routine Line > > Source > > > omp 0000000000403BC8 Unknown Unknown > > Unknown > > > omp 0000000000403572 Unknown Unknown > > Unknown > > > libiomp5.so 00002AAAAD3146A3 Unknown Unknown > > Unknown > > > libiomp5.so 00002AAAAD2E3007 Unknown Unknown > > Unknown > > > libiomp5.so 00002AAAAD2E26F5 Unknown Unknown > > Unknown > > > libiomp5.so 00002AAAAD3149C3 Unknown Unknown > > Unknown > > > libpthread.so.0 00002AAAAD5BADC5 Unknown Unknown > > Unknown > > > libc.so.6 00002AAAAD8C5CED Unknown Unknown > > Unknown > > > > > > > > > This means that in my programs using petsc, I can't use prints to see > > exactly what the threads are doing, which is a pain when debug is required > > (that is to say, all the time). Is this issue expected ? > > > > > > Thx in advance > > > > > > Timothee NICOLAS > > > > > > > > From timothee.nicolas at gmail.com Tue Jan 30 12:00:54 2018 From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=) Date: Tue, 30 Jan 2018 19:00:54 +0100 Subject: [petsc-users] Petsc messing with OpenMP In-Reply-To: References: <613E404A-30F1-43EA-85AE-E1198C09910A@anl.gov> Message-ID: Thank you very much Barry, it works indeed to build petsc with the qopenmp flags! Best Timothee 2018-01-30 18:46 GMT+01:00 Satish Balay : > Try building PETSc with -qopenmp flag and see if this makes a difference. > > i.e use CFLAGS=-qopenmp FFLAGS=-qopenmp CXXFLAGS=-qopenmp > > Potential explanation: > > Compilers do different things internally [for ex - link to different > libraries] when different compile flags are used. > > When -qopenmp is used - its possible some openmp library gets linked > in. And when -qopenmp is not used - some equivalnet stub library might > get linked in. Only one lib is supporsed to be used at anytime. With > the way petsc gets built - both might get used - causing some > conflict. > > You can verify with: > > > mpicc -v -qopenmp dummy.c > mpicc -v dummy.c > > And 'ldd libpetsc.so' > > Satish > > On Tue, 30 Jan 2018, Timoth?e Nicolas wrote: > > > Thank you for your answer, > > > > I am not responsible for the choices in this code to begin with, but in > any > > case, as you can see in the example, I am not using any petsc call. Also > in > > our code, even though openmp is used, this is never in interaction with > > petsc, the latter being used only at a specific (but crucial) place in > the > > code, and without openMP. In that sense, the model we are using is NOT > some > > 'Petsc with threads model', as described in the page you sent me. > > Nevertheless, it appears that just linking petsc is enough to perturb the > > behavior of openMP. I believe it may require further investigation. On > top > > of the bug I report in my first message, I have noticed very weird openMP > > behavior lately, the kind of bugs that 'don't make sense' at first sight > > (would be too long/boring to give details at this stage). I am starting > to > > realize that it may have something to do with petsc. > > > > Again, the webpage you have sent me seems to adopt the point of view that > > hybrid MPI/openMP is useless from the performance point of view, and/or > > that Petsc routines should not be threaded. It does not, however, explain > > why the behaviour of openMP may be changed by merely linking a program > with > > petsc, without a single petsc call in the program. > > > > Best > > > > 2018-01-30 18:02 GMT+01:00 Smith, Barry F. : > > > > > > > > I don't know what you are trying to do with OpenMP and PETSc, nor do > I > > > understand why anyone would use OpenMP, but you cannot call virtually > any > > > PETSc function or operation while you are using threads. > > > > > > Best to use PETSc as intended, with one MPI process per core or > hardware > > > thread and not use OpenMP at all. http://www.mcs.anl.gov/petsc/ > > > miscellaneous/petscthreads.html > > > > > > > > > Barry > > > > > > > > > > On Jan 30, 2018, at 10:57 AM, Timoth?e Nicolas < > > > timothee.nicolas at gmail.com> wrote: > > > > > > > > Dear petsc team, > > > > > > > > For a while, I have been wondering why I have never managed to print > > > what threads are doing in an openMP region in my FORTRAN programs. Some > > > people told me it was normal because threads will get confused all > trying > > > to write at the same time. However I realised today that the problem > seems > > > to be related to petsc. I have a super simple "hello world" example > that > > > reproduces the problem : > > > > > > > > program hello > > > > !$ use omp_lib > > > > implicit none > > > > integer nthreads, tid > > > > > > > > !$omp parallel private(nthreads, tid) > > > > tid = omp_get_thread_num() > > > > print *, 'hello world from thread = ', tid > > > > > > > > if (tid .eq. 0) then > > > > nthreads = omp_get_num_threads() > > > > print *, 'number of threads = ', nthreads > > > > end if > > > > !$omp end parallel > > > > > > > > end program hello > > > > > > > > If I compile it with > > > > > > > > mpif90 -qopenmp -o omp main.f90 > > > > > > > > Then I get no problem. But if I link petsc library (as is the case > in my > > > code) : > > > > > > > > mpif90 -qopenmp -o omp main.f90 -L/home/timotheenicolas/petsc- > 3.7.3/arch-linux2-c-debug/lib > > > -lpetsc > > > > > > > > > > > > Then I get the following error after execution of export > > > OMP_NUM_THREADS=2;./omp > > > > > > > > hello world from thread = 0 > > > > number of threads = 2 > > > > forrtl: severe (40): recursive I/O operation, unit -1, file unknown > > > > Image PC Routine Line > > > Source > > > > omp 0000000000403BC8 Unknown Unknown > > > Unknown > > > > omp 0000000000403572 Unknown Unknown > > > Unknown > > > > libiomp5.so 00002AAAAD3146A3 Unknown Unknown > > > Unknown > > > > libiomp5.so 00002AAAAD2E3007 Unknown Unknown > > > Unknown > > > > libiomp5.so 00002AAAAD2E26F5 Unknown Unknown > > > Unknown > > > > libiomp5.so 00002AAAAD3149C3 Unknown Unknown > > > Unknown > > > > libpthread.so.0 00002AAAAD5BADC5 Unknown Unknown > > > Unknown > > > > libc.so.6 00002AAAAD8C5CED Unknown Unknown > > > Unknown > > > > > > > > > > > > This means that in my programs using petsc, I can't use prints to see > > > exactly what the threads are doing, which is a pain when debug is > required > > > (that is to say, all the time). Is this issue expected ? > > > > > > > > Thx in advance > > > > > > > > Timothee NICOLAS > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mbuerkle at web.de Tue Jan 30 18:53:29 2018 From: mbuerkle at web.de (Marius Buerkle) Date: Wed, 31 Jan 2018 01:53:29 +0100 Subject: [petsc-users] combine two matrices with different non-zero structure In-Reply-To: References: Message-ID: Barry, ? Thanks for you reply. The pulled matrix is symmetric but that's it. At the moment I am doing a copy of the matrix right after MatCreateSubMatrix to keep it's nonzero structure. To insert the matrix which I obtained after the multiplications back into the bigger matrix I use the initially copied Matrix to get the nonzero elements with MatGetRow and as I know the offset of the SubMatrix relative to the elements of the big-one I can use MatSetValues to insert the elements row by row. That kinda works but it is rather inefficient. Is there anyway to avoid copying the whole submatrix in the beginning, as I don't need the actual values but only the positions of the nonzero elements in one way or the other. Marius ? >Marius, > >We don't provide a way to insert a "generic" sparse matrix into a bigger matrices (dense matrices coming from element stiffness matrices yes) so I don't see any simple solution. >Does the submatrix you pull out have any particular structure, what does it represent? > >Barry >> >> Hi ! >> >> I have the followng problem. I create a Submatrix containing a subset of row/columns of the original matrix. After some matrix multiplications the non-zero strucutre of the resulting matrix changed. Now I want to insert this submatix back into the original one keeping only the non-zero entries which are present in the original matrix discarding the others which accumulated due to the matrix multiplications. Is there an easy or also not so easy way to do this ? >> >> Best, >> Marius ? From bsmith at mcs.anl.gov Tue Jan 30 19:55:28 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 31 Jan 2018 01:55:28 +0000 Subject: [petsc-users] combine two matrices with different non-zero structure In-Reply-To: References: Message-ID: <47BF666E-F79A-4C36-8105-BFCC89EE9988@anl.gov> Could you tell us exactly what matrix matrix products you are doing? Barry > On Jan 30, 2018, at 6:53 PM, Marius Buerkle wrote: > > Barry, > > Thanks for you reply. The pulled matrix is symmetric but that's it. At the moment I am doing a copy of the matrix right after MatCreateSubMatrix to keep it's nonzero structure. To insert the matrix which I obtained after the multiplications back into the bigger matrix I use the initially copied Matrix to get the nonzero elements with MatGetRow and as I know the offset of the SubMatrix relative to the elements of the big-one I can use MatSetValues to insert the elements row by row. That kinda works but it is rather inefficient. Is there anyway to avoid copying the whole submatrix in the beginning, as I don't need the actual values but only the positions of the nonzero elements in one way or the other. > > Marius > > > >> Marius, >> >> We don't provide a way to insert a "generic" sparse matrix into a bigger matrices (dense matrices coming from element stiffness matrices yes) so I don't see any simple solution. >Does the submatrix you pull out have any particular structure, what does it represent? >> >> Barry > > >>> >>> Hi ! >>> >>> I have the followng problem. I create a Submatrix containing a subset of row/columns of the original matrix. After some matrix multiplications the non-zero strucutre of the resulting matrix changed. Now I want to insert this submatix back into the original one keeping only the non-zero entries which are present in the original matrix discarding the others which accumulated due to the matrix multiplications. Is there an easy or also not so easy way to do this ? >>> >>> Best, >>> Marius > From bsmith at mcs.anl.gov Tue Jan 30 20:53:43 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 31 Jan 2018 02:53:43 +0000 Subject: [petsc-users] combine two matrices with different non-zero structure In-Reply-To: References: Message-ID: <65CABBFF-51E4-4B69-BDA1-21F3603AE382@anl.gov> > On Jan 30, 2018, at 6:53 PM, Marius Buerkle wrote: > > Barry, > > Thanks for you reply. The pulled matrix is symmetric but that's it. At the moment I am doing a copy of the matrix right after MatCreateSubMatrix to keep it's nonzero structure. To insert the matrix which I obtained after the multiplications back into the bigger matrix I use the initially copied Matrix to get the nonzero elements with MatGetRow and as I know the offset of the SubMatrix relative to the elements of the big-one I can use MatSetValues to insert the elements row by row. That kinda works but it is rather inefficient. How do you know it is inefficient? Are you doing some kind of search on each element of each row to see if it fits in the original nonzero pattern of the block? Anything else that is inefficient? Barry > Is there anyway to avoid copying the whole submatrix in the beginning, as I don't need the actual values but only the positions of the nonzero elements in one way or the other. > > Marius > > > >> Marius, >> >> We don't provide a way to insert a "generic" sparse matrix into a bigger matrices (dense matrices coming from element stiffness matrices yes) so I don't see any simple solution. >Does the submatrix you pull out have any particular structure, what does it represent? >> >> Barry > > >>> >>> Hi ! >>> >>> I have the followng problem. I create a Submatrix containing a subset of row/columns of the original matrix. After some matrix multiplications the non-zero strucutre of the resulting matrix changed. Now I want to insert this submatix back into the original one keeping only the non-zero entries which are present in the original matrix discarding the others which accumulated due to the matrix multiplications. Is there an easy or also not so easy way to do this ? >>> >>> Best, >>> Marius > From mbuerkle at web.de Tue Jan 30 21:15:59 2018 From: mbuerkle at web.de (Marius Buerkle) Date: Wed, 31 Jan 2018 04:15:59 +0100 Subject: [petsc-users] combine two matrices with different non-zero structure In-Reply-To: <47BF666E-F79A-4C36-8105-BFCC89EE9988@anl.gov> References: <47BF666E-F79A-4C36-8105-BFCC89EE9988@anl.gov> Message-ID: An HTML attachment was scrubbed... URL: From mbuerkle at web.de Tue Jan 30 21:21:58 2018 From: mbuerkle at web.de (Marius Buerkle) Date: Wed, 31 Jan 2018 04:21:58 +0100 Subject: [petsc-users] combine two matrices with different non-zero structure In-Reply-To: <65CABBFF-51E4-4B69-BDA1-21F3603AE382@anl.gov> References: <65CABBFF-51E4-4B69-BDA1-21F3603AE382@anl.gov> Message-ID: An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jan 30 21:34:38 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 31 Jan 2018 03:34:38 +0000 Subject: [petsc-users] combine two matrices with different non-zero structure In-Reply-To: References: <47BF666E-F79A-4C36-8105-BFCC89EE9988@anl.gov> Message-ID: To do this most efficiently you would compute exactly the part of F = B'*C*B you need (that matches the nonzero pattern of A) do a MatAXPY() with same numerical pattern and then just stick the values of D into the original matrix (without worrying about excluding the new nonzeros since there won't be any). MatPtAPNumeric_MPIAIJ_MPIAIJ_scalable() does the numerical triple product so the only special code you need is to rewrite MatPtAPSymbolic_MPIAIJ_MPIAIJ_scalable() so that that it takes another matrix (your A) that determines the nonzero pattern you want and from this constructs the needed symbolic information needed to do the (special, reduced) matrix triple product. If you can this great, if not then I suspect there is not much you can do to improve the efficiency of what you have already done. Barry > On Jan 30, 2018, at 9:15 PM, Marius Buerkle wrote: > > The matrix A is the extracted submatrix and I am basicallly doing something like this D=A+B'*C*B. Where A, B, and C are sparse but B'*C*C will clearly generate some addional nonzero entries in D. And D should go back into the bigger matrix but only the entries which are already nonzero in A. > > > Could you tell us exactly what matrix matrix products you are doing? > > Barry > > > > On Jan 30, 2018, at 6:53 PM, Marius Buerkle wrote: > > > > Barry, > > > > Thanks for you reply. The pulled matrix is symmetric but that's it. At the moment I am doing a copy of the matrix right after MatCreateSubMatrix to keep it's nonzero structure. To insert the matrix which I obtained after the multiplications back into the bigger matrix I use the initially copied Matrix to get the nonzero elements with MatGetRow and as I know the offset of the SubMatrix relative to the elements of the big-one I can use MatSetValues to insert the elements row by row. That kinda works but it is rather inefficient. Is there anyway to avoid copying the whole submatrix in the beginning, as I don't need the actual values but only the positions of the nonzero elements in one way or the other. > > > > Marius > > > > > > > >> Marius, > >> > >> We don't provide a way to insert a "generic" sparse matrix into a bigger matrices (dense matrices coming from element stiffness matrices yes) so I don't see any simple solution. >Does the submatrix you pull out have any particular structure, what does it represent? > >> > >> Barry > > > > > >>> > >>> Hi ! > >>> > >>> I have the followng problem. I create a Submatrix containing a subset of row/columns of the original matrix. After some matrix multiplications the non-zero strucutre of the resulting matrix changed. Now I want to insert this submatix back into the original one keeping only the non-zero entries which are present in the original matrix discarding the others which accumulated due to the matrix multiplications. Is there an easy or also not so easy way to do this ? > >>> > >>> Best, > >>> Marius > > > From mbuerkle at web.de Tue Jan 30 22:10:43 2018 From: mbuerkle at web.de (Marius Buerkle) Date: Wed, 31 Jan 2018 05:10:43 +0100 Subject: [petsc-users] combine two matrices with different non-zero structure In-Reply-To: References: <47BF666E-F79A-4C36-8105-BFCC89EE9988@anl.gov> Message-ID: An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jan 30 22:17:15 2018 From: bsmith at mcs.anl.gov (Smith, Barry F.) Date: Wed, 31 Jan 2018 04:17:15 +0000 Subject: [petsc-users] combine two matrices with different non-zero structure In-Reply-To: References: <47BF666E-F79A-4C36-8105-BFCC89EE9988@anl.gov> Message-ID: <553E147A-3834-431F-9862-443EDDE57D72@anl.gov> MaTAXPY with same nonzero pattern is much faster. > On Jan 30, 2018, at 10:10 PM, Marius Buerkle wrote: > > Thanks. While B'*C*B may not have the same nonzero pattern as A it does not matter as this is done by MaTAXPY with different nonzero pattern it should not matter when using MatPtAPNumeric_MPIAIJ_MPIAIJ_scalable() with a restricted reduction. I will give it a try. > > > > > To do this most efficiently you would compute exactly the part of F = B'*C*B you need (that matches the nonzero pattern of A) do a MatAXPY() with same numerical pattern and then just stick the values of D into the original matrix (without worrying about excluding the new nonzeros since there won't be any). > > MatPtAPNumeric_MPIAIJ_MPIAIJ_scalable() does the numerical triple product so the only special code you need is to rewrite MatPtAPSymbolic_MPIAIJ_MPIAIJ_scalable() so that that it takes another matrix (your A) that determines the nonzero pattern you want and from this constructs the needed symbolic information needed to do the (special, reduced) matrix triple product. > > If you can this great, if not then I suspect there is not much you can do to improve the efficiency of what you have already done. > > Barry > > > > On Jan 30, 2018, at 9:15 PM, Marius Buerkle wrote: > > > > The matrix A is the extracted submatrix and I am basicallly doing something like this D=A+B'*C*B. Where A, B, and C are sparse but B'*C*C will clearly generate some addional nonzero entries in D. And D should go back into the bigger matrix but only the entries which are already nonzero in A. > > > > > > Could you tell us exactly what matrix matrix products you are doing? > > > > Barry > > > > > > > On Jan 30, 2018, at 6:53 PM, Marius Buerkle wrote: > > > > > > Barry, > > > > > > Thanks for you reply. The pulled matrix is symmetric but that's it. At the moment I am doing a copy of the matrix right after MatCreateSubMatrix to keep it's nonzero structure. To insert the matrix which I obtained after the multiplications back into the bigger matrix I use the initially copied Matrix to get the nonzero elements with MatGetRow and as I know the offset of the SubMatrix relative to the elements of the big-one I can use MatSetValues to insert the elements row by row. That kinda works but it is rather inefficient. Is there anyway to avoid copying the whole submatrix in the beginning, as I don't need the actual values but only the positions of the nonzero elements in one way or the other. > > > > > > Marius > > > > > > > > > > > >> Marius, > > >> > > >> We don't provide a way to insert a "generic" sparse matrix into a bigger matrices (dense matrices coming from element stiffness matrices yes) so I don't see any simple solution. >Does the submatrix you pull out have any particular structure, what does it represent? > > >> > > >> Barry > > > > > > > > >>> > > >>> Hi ! > > >>> > > >>> I have the followng problem. I create a Submatrix containing a subset of row/columns of the original matrix. After some matrix multiplications the non-zero strucutre of the resulting matrix changed. Now I want to insert this submatix back into the original one keeping only the non-zero entries which are present in the original matrix discarding the others which accumulated due to the matrix multiplications. Is there an easy or also not so easy way to do this ? > > >>> > > >>> Best, > > >>> Marius > > > > > >